Building an always-on agent
How we'd split the system into areas that can be built and investigated on their own. It's our own breakdown, based on the architecture in the talk notes and the Under the hood page, not a description of Grok Bot's internals.
12 areas6 contracts6 build phasesP to present
01Map
How the areas fit together
Twelve areas in four zones, plus one cross-cutting band. Arrows are the contracts between them.
Edges
10Conversations and clientsstore, sync, apps, Slack
11Triggers and schedulingingress, queues, routines
Dconversation events
Ftriggers
Agent core · server
1Agent runtimedurable loop per task
2Model gatewaycalls, retries, cost
3Context and memoryprompts, compaction, memory
4Tool layer and routerregistry, routing, wake rules
ABtool calls, wake
AEtool calls, credential refs
Execution · per-user computer C
5Computer platformVM, disk, snapshots
6In-VM worker and gatewaydial-out command runner
7Computer use and live desktopscreens, browser, take-over
Access · apps and secrets
8ConnectorsMCP, plugins, policy
9Credentials and secretsvault, secret requests, redaction
12Safety, identity and opsreview model, approvals, tenancy, tracing, evals
Cross-cutting: every area above goes through itA Letters mark the contracts in section 03
Notes
- Edges bring work in and show results. The agent core decides. Execution and access do the work.
- Everything the agent knows lives on the server side. The computer holds working files and logins on its own disk, and can be rebuilt without losing a conversation.
02Areas
The twelve areas
What each one owns, what makes it hard, and whether to build it or reuse something.
| # | Area | Owns | Hardest part | Risk | Build or reuse |
|---|---|---|---|---|---|
| 1 | Agent runtime | Durable workflow per task: load state, the model and tool loop, interrupts, batching, helpers, when to stop | Idempotent steps; timeouts that cover a computer restore; versioning | Medium | Reuse Temporal |
| 2 | Model gateway | Provider calls, retries, fallbacks, token and cost accounting, fast vs. long-running tiers | Cache hit rate; what to do during an outage | Low | Thin layer on provider SDKs |
| 3 | Context and memory | Prompt builder (fixed order), compaction, memory layers and the write pass, skill library | Compaction quality over weeks; what to forget | High | Build |
| 4 | Tool layer and router | Tool registry, tool details on demand, server vs. computer routing, wake rules, error design, call IDs | Stable schemas so the prompt cache holds | Medium | Build |
| 5 | Computer platform | One VM per user: image, supervisor, durable disk, snapshot and restore, sleep policy, file sync, reset | Restore speed; cost of idle users | Medium | Buy first (sandbox vendor) |
| 6 | In-VM worker and gateway | Dial-out worker, protocol and auth, streamed commands, file ops, de-duplication log, reconnect after restore | Restores, retries and commands running twice | High | Build |
| 7 | Computer use and live desktop | A screen and Chrome per Bot, accessibility tree then pixels, screenshots, recordings, live view, take-over | Reliable clicking and grounding | High | Build on Chrome and VNC |
| 8 | Connectors | MCP client, plugin catalog, installs, per-tool switches, team connector policy | Vendor auth quirks | Low | Reuse MCP |
| 9 | Credentials and secrets | OAuth broker and token vault, secret requests, use by name, redaction, login hand-offs | Keeping values away from the model and the VM | Medium | Build flows on a vault |
| 10 | Conversations and clients | SendToUser contract and payloads, append-only store, signal-then-fetch sync, apps, Slack and email | Ordering and de-duplication across clients | Low | Build |
| 11 | Triggers and scheduling | Ingress routing, per-conversation queues, routines, webhooks, app events, Bot-to-Bot messages, when to stay quiet | Priority and batching; staying quiet | Medium | Build on a queue and cron |
| 12 | Safety, identity and ops | Review model, rules and approvals, provenance labels, network policy, audit; users, Bots, Team Bots, metering; tracing and evals | Prompt injection; eval coverage | High | Build |
03Contracts
Contracts to agree first
Six interfaces. With these fixed, the areas can be built in parallel against fakes of each other.
| Contract | Between areas | What it carries | |
|---|---|---|---|
| A | Tool call and result | 1, 4, 6, 8 | Call ID, tool, arguments, target (server or computer). Back: output, or an error that says what to do next. |
| B | Computer control plane | 5 with 4 and 6 | state, ensureAwake, snapshot, reset |
| C | Worker protocol | 6 (worker with gateway) | Connect with a stable ID, receive a call, return a result, heartbeat |
| D | Conversation events | 1 with 10 | Sequence numbers, step types, SendToUser payloads |
| E | Credential reference | 8, 9 and 6 | Ask for a secret or token by name; never see the value |
| F | Trigger envelope | 11 to 1 | Source, Bot, conversation, priority, provenance |
Notes
- A is the most important: it's the seam between the loop and everything that runs tools.
04Order
Build order
Each phase ends with something usable.
1
Talk to it
1Agent runtime2Model gateway4Tool layer and router10Conversations and clients
A loop on the server that replies through SendToUser. No computer yet.
2
Give it a computer
5Computer platform6In-VM worker and gateway
An always-on container with a dial-out worker, then sleep and wake.
3
Connect apps
8Connectors9Credentials and secrets
Connectors, the token vault and secret requests.
4
Make it remember
3Context and memory
Memory layers and compaction, measured with evals.
5
Let it start work
11Triggers and scheduling
Triggers, routines and the "worth messaging?" decision.
6
Use any app
7Computer use and live desktop
Computer use, live view and take-over.
12Safety, identity and opsReview model and approvals from phase 2, as soon as the agent can run commands
LaterTeam Bots and multi-tenant routing: they depend on almost everything else.
05Risk
Risk and build vs. reuse
Where the unknowns are, and where we can lean on existing tools.
Build ourselves
Reuse or buy first
High
3Context and memory6In-VM worker and gateway7Computer use and live desktop12Safety, identity and ops
none
Medium
4Tool layer and router9Credentials and secrets11Triggers and scheduling
1Agent runtime5Computer platform
Low
10Conversations and clients
2Model gateway8Connectors
Reuse means: Temporal for the runtime, a Firecracker-based sandbox vendor for the computer (E2B, Modal, Daytona or Fly Machines, for example), provider SDKs for the model gateway, and MCP for connectors.
Notes
- The three high-risk build areas are where to start spikes early: restores and retries (6), compaction over weeks (3), and reliable computer use (7).
- The rest is mostly known engineering: durable workflows, OAuth, MCP and sandbox vendors all exist.