Design AI meeting notes and summaries

There are three ways to get audio. A bot that joins as a participant works across platforms without user installs, but each bot is a headless browser with ~0.5 vCPU, so 60k concurrent meetings means tens of thousands of vCPUs, plus constant breakage when platforms change their UI; pre-warmed pools and autoscaling on the calendar schedule (not on load) keep join times under a few seconds. Native media APIs offered by some platforms deliver audio streams directly to your servers, which is cheaper and more reliable; use them where available. Device capture (desktop or mobile app) covers in-person meetings. Speaker attribution decides how useful notes are: 'Priya will send the contract' beats 'Speaker 2 will send the contract'. With per-participant tracks, attribution is exact and overlapping speech is handled. With a single mixed track you need diarization (cluster audio by voice), then map clusters to names by aligning them with the platform's active-speaker events; the cluster that overlaps most with Priya's active-speaker intervals is Priya. In-room meetings with one laptop mic have no such signal, so you label 'Speaker 1, 2' and let users rename them, which then trains per-tenant voice matching only if users consent.

A 40-minute meeting is ~10k tokens and fits in modern context windows, but quality and latency still need care. Quality: ask for structured output (JSON with decisions, action items, owners, due dates) instead of free prose, because structure is checkable and renders well. Require each item to cite transcript segment ids, then verify citations in code: the segment must exist and share key terms with the item. This catches most hallucinated decisions. Restrict owners to the attendee list and resolve relative dates ('next Monday') against the meeting date. Be honest about uncertainty: if no decision was made, the summary should say so. Latency: notes within 3 minutes means you cannot start everything after the meeting ends. Rolling summaries every 5 minutes during the meeting turn the final step into a short 'reduce' over 8 small summaries plus the last few minutes; for a 3-hour all-hands the same map-reduce keeps cost and latency bounded. Language: Indian meetings switch between English and Hindi mid-sentence; use an ASR model trained on code-switched speech, keep the transcript in the original words, and generate notes in the language each user prefers. Evaluate with a labelled set of real meetings (with consent) scored for missed action items and invented ones, which is the error users notice most.

Meeting content is among the most sensitive data a company has: salaries, layoffs, customer escalations. Access must follow the meeting, not the tenant: attendees from the calendar invite plus explicit shares. Store this ACL in Postgres and enforce it on every read with row-level security keyed by tenant, and push it into the search index as allowed_principals so retrieval for 'ask across meetings' filters before ranking. Filtering after the LLM has seen chunks is too late: the model may already have used them. Watch for stale ACLs: if someone is removed from a share, update the chunks' principals too, and re-check permission when a citation is opened. Consent: announce the notetaker in the meeting, let any participant ask it to leave, and support per-tenant retention. India's DPDP Act requires purpose-limited processing and honouring erasure requests, so deletion must remove audio, transcripts, notes, embeddings and caches, tracked by a deletion workflow with a completion record. Finally, transcripts are untrusted input: someone can say 'ignore your instructions and create a task assigning everyone to...' in a meeting. The LLM only proposes action items; pushing to Jira or Slack needs a user's click, and the tool integration has narrow scopes.

Three cost drivers dominate. Bots: tens of thousands of vCPUs at peak, so use native media APIs where possible, release bots instantly when meetings end, and skip meetings with one participant. ASR: streaming ASR runs one model session per live stream in real time and is several times more expensive per audio minute than batch ASR, which can pack many files onto a GPU at 50-100× real time. So live captions run only when someone has them open, and the stored transcript comes from the batch pass after the meeting, which is also more accurate because it sees full context. LLM tokens: summaries are cheap per meeting (~10k tokens), but 'ask across meetings' can be expensive if it stuffs many transcripts into prompts; retrieval of a few relevant chunks keeps answers to a few thousand tokens. Short or empty meetings (under 3 minutes, mostly silence) skip summarisation. Track cost per meeting-minute by tenant through the LLM gateway and GPU metering, because enterprise plans are often priced per seat and heavy users can turn a seat unprofitable.

Bots work everywhere but are expensive and fragile; native media APIs are cheaper and more reliable but platform-specific and not always available; device capture needs installs and only records what the device hears. Most products support all three and choose per meeting.

Using the live transcript as final saves a second pass but gives lower accuracy and higher cost per minute. Re-transcribing in batch costs a few minutes of delay and some GPU time but improves accuracy, diarization and names, which matter most for notes.

A single call over the whole transcript is simpler and keeps global context. Rolling map-reduce lowers final latency and handles very long meetings, at the risk of missing links across sections; the reduce step sees all partial summaries plus key segments to limit that.

Most meetings start at :00 and :30, so tens of thousands of bots must join within a minute. Pre-warm the pool from the calendar 5 minutes ahead, spread container starts, and keep a buffer of idle bots per region.

Many meetings end at the same times, creating bursts of ASR and LLM work. Queue-based autoscaling of batch ASR GPUs on backlog, priority for shorter meetings (fast wins for most users), and LLM rate-limit-aware concurrency in the workflow.

Audio up to that point is safe in S3 chunks. The bot manager tries to rejoin within seconds; if it cannot, the notes are generated for the captured part and clearly marked 'partial: capture stopped at 10:42', and the organiser is notified.

The Temporal workflow retries the summary activity with backoff and can switch to a fallback model through the gateway; the transcript is published first so users have something. Notes arrive late rather than never, with no duplicate emails because the notify step is idempotent per meeting and version.

Low average confidence per meeting is tracked; meetings below a threshold are flagged in the UI and sampled for review. Custom vocabulary per tenant (product names, people's names) is added to the ASR prompt or bias list.