How an always-on agent is built
Notes from Lee's talk on how Grok Bot is built (Lee works at SpaceX and was previously at Cursor). Grok Bot is an agent you message like a person. It lives on a server, gets its own Linux machine, keeps memory between conversations, and can be started by things other than you. The slides are redrawn below, with what Lee said about each one.
Runs on a server
The agent loop runs inside a durable workflow and saves after every step, so a crash just picks up where it stopped.
Its own computer
One Firecracker VM per user, with files, a shell and Chrome. It goes to disk when idle and comes back on the next tool call.
Wakes on events
Slack, email, calls, routines and webhooks can all start a turn. It only messages you if there's something to say.
Cheap to keep running
Prompt caching, helper agents, summarization and a small memory in the prompt keep long conversations affordable.
Overview
What Lee means by an always-on agent, and one request followed through the whole system.
From a terminal harness to an always-on agent
The week 1 harness was a model calling tools in a loop, running in your terminal. Close the laptop and it stops. Grok Bot keeps the same loop and moves it to a server.
| Terminal harness | Always-on agent ("bot") | |
|---|---|---|
| Where it runs | Your terminal. Stops when the laptop closes. | A server in the cloud. Survives restarts and crashes. |
| Computer | Your laptop | Its own Linux machine with a desktop, files, a browser and even 3D apps |
| Memory | The current session | Profile, dated logs, notes and files that build up over time |
| Triggered by | You typing a prompt | You, Slack, email, calls, meetings, routines, webhooks, other agents |
| When it talks | Replies to every message | Works in the background and messages you when there's an update |
| Time horizon | Minutes to hours | A few hours to a day today. Weeks or months is where it's heading |
| Safety | You approve each shell command | A separate model reviews commands (auto-review). It asks before irreversible actions. |
From the talk
- As the models got better, the interface got simpler. It's a chat, because everyone already knows how to text.
- People use it to hand off longer tasks, the way they would with a coworker, more than to ask questions.
- Lee mentioned several products like this taking off right now. The talk is about how Grok Bot does it.
Life of a message, end to end
One request from start to finish. The § links jump to the section with more detail.
- Something wakes the agentYour message, a Slack mention, a routine at 8:00, a webhook, a phone call, another agent, or a helper finishing.§09
- The router drops duplicates and picks the conversationThe input joins that conversation's queue. Your own 1:1 messages skip the line, and quick bursts get batched.§13
- A durable workflow starts the turnIt loads conversation state, memory, routines and skills from the database.§12
- The prompt is assembledStable cached prefix (tools, system prompt, memory snapshot), then append-only history, then the newest message.§22
- Model ↔ tools loop on the serverEvery step is saved, so a crash resumes mid-turn instead of starting over.§06
- If a tool needs the computer, the VM wakesThe Firecracker snapshot is restored in place, and logins and installed tools are still there.§10
- Heavy work goes to helpersComputer use, video, long tool chains and cloud coding agents run separately and come back with a short report.§19
- Risky actions get reviewedA separate model checks them. Irreversible ones (email, payments) ask you through a purpose-built UI.§21
- "Anything worth saying?"If yes, it calls SendToUser and the message goes to the right app. If not, nothing is sent. The model's own text is never shown to you.§14
- Apps syncThe server sends a tiny "something changed" signal. Each app fetches from the database and applies updates in sequence order.§11
- Everything goes back to sleepThe process stops after about 2 idle minutes and the VM snapshots to disk. A big context gets summarized while the cache is still warm.§25
How we got here
The background part of the talk: how working with AI has changed, and which model improvements made this kind of agent possible.
Four eras of developers working with AI
Lee's rough timeline. At each step you do less of the hands-on work yourself.
Copy-paste from chat
Paste code out of ChatGPT into your editor. That feels like decades ago, but it was only a few years.
You do everythingAgents in the terminal
A simple harness: edit files and run shell commands in a loop. Suddenly the model could do proactive work.
You prompt and waitDedicated agent apps
Many agents in parallel, some in the cloud. You're still reviewing every shell command, so you're the bottleneck.
You approve every stepAlways-on agents
They run in the background with their own computer and memory, pick up skills as they go, and keep working after you log off.
You steerFrom the talk
- Steering: you can say "go check this thread" and right after, "actually, do that on Thursday." Older models got confused when interrupted like that. Current ones work out that the two messages belong together.
- Auto-review: a second model checks commands before they run. Lee's point was that this is safer than a tired person clicking approve on hundreds of commands, and in blind comparisons the model usually does a better job.
Six model changes that made this possible
Lee said there are more, but these are the six covered in the talk.
Long-horizon instruction following
Models can keep going for hours or days without losing track, as long as the context gets compacted well.
Sub-agents
The model can hand work to a helper that starts with an empty context. The helper does the expensive part and reports back.
Reliable, powerful tools
Tool calls used to be unreliable. Lee called hallucination mostly solved now, so you can hand it real tools, up to a full Linux machine.
Computer use
It can click through a screen like a person, so sites and apps with no API or MCP server are still usable.
Files as memory
Memory can just be files, because models are good at finding things with grep and other shell commands.
Learning by being shown
Show it how to do something and it turns that into reusable skills, memories or routines.
Basically, everything is a file
Roughly what it stores about you (the file names are made up for illustration). The model decides when to open or search each file.
profile.mdYour profile, included in every prompt. Example: "prefers all-lowercase messages."log/A running log of what you worked on, like a coworker's notes. "What did I do last week?" gets answered by searching it.skills/How to do a specific task, written as a markdown file. Lots of people and companies publish these.plugins/One install per tool you use. It brings that tool's skills and MCP servers.routines/Prompts that run on a schedule, like cron jobs.Architecture of an always-on agent
First the three main pieces (apps, server, the agent's computer), then how it sleeps, wakes up, keeps apps in sync, recovers from crashes and orders incoming messages.
Three parts of the system
The agent runs on the server, uses tools on its own computer, and every app shows the same conversation.
From the talk
- Left: triggers like apps, Slack messages and emails. They all talk to the same server.
- Middle: input goes into a queue, which calls the agent loop: the same "model calling tools in a loop" from week 1. On shutdown, state is saved to the database and storage.
- Right: the computer just runs the tools. It can be put to sleep, rebuilt or patched without touching the server.
- Clients subscribe to the server for live updates, so every surface shows the same conversation.
Why the agent runs on the server
Lee framed it as thin client, thick server, which computing has gone back and forth on over the years.
- Thin client, thick server
- Close your laptop and the work keeps going
- Easily move between your computer and phone
- Ship code without App Store review cycles
- Offload heavy work to the server
- Agent loop is entirely decoupled from app code
- Easier to scale with hyper growth
- Agents that work for months can't be on device
From the talk
- A new agent version should just be a server deploy, with no waiting on App Store review.
- Heavy work like transcription and image generation is faster on servers, and there are well-known ways to scale servers when traffic jumps around.
- The main reason: if the agent loop lived on the virtual machine, you couldn't shut the VM down independently. With everything saved on the server, you can rebuild the VM (say, for an urgent zero-day Linux patch) without killing the loop or causing downtime.
How does it stay always on?
Time goes left to right. The agent process and the computer both start off asleep, and the computer only wakes if a tool needs it.
When nothing is happening, nothing runs. Everything that matters is saved, so any trigger can bring it back.
From the talk
- A trigger means loading and booting all of the agent's state on the server.
- The computer only starts if the agent decides it needs it, for example to use the browser or click around. When it's done, both go back to sleep.
- Why: running a computer per user 24/7 is expensive, and "there's not enough computers in the world" to do it for everyone.
What wakes the agent
Lots of things can start a turn. At the end, the agent decides whether to message you at all.
Every wake-up records where it came from, so the agent can tell whether you asked, a routine ran, or an outside service sent something.
From the talk
- Lee was most interested in the triggers that aren't you: Slack pings, phone calls, meetings, and background work that ran for hours or days and just finished.
- At SpaceX, bots now join meetings for employees who can't make it, take notes and report back.
- Then there's the "do I bug the user?" check. A coworker who messaged you about every bit of progress would get annoying. Often a daily update is enough, depending on the task.
The agent's computer
Each user gets one microVM. It sits on disk while idle and comes back with the same files and logins.
Going to sleep
- Files sync to cloud storage
- Memory and disk are saved as a snapshot
- The machine stops
Waking up
- The snapshot is restored in place
- Logins and installed tools are still there
- Conversation state comes from the server
From the talk
- Firecracker makes it easy to snapshot the machine's state, shut it down and start it back up.
- You can connect to a live view of the machine and take control yourself.
- Shell commands run in a secured, locked-down environment instead of on your personal laptop.
- Virtual desktops: with 5 or 10 bots, each gets its own screen and browser. It's shared infrastructure the bots call into.
What happens when you hit send
Notifications only say "something changed." The database always has the real data.
Unique IDs make retries safe
One turn at a time per conversation
The signal carries no data, so losing one is harmless
Every app ends up showing the same thing
From the talk
- The app should show and acknowledge your message right away.
- Clients hold subscriptions to the servers. On a signal they fetch the latest state, and de-duplication means retries never show a message twice.
- Worth digging into if you enjoy distributed systems: strong vs. eventual consistency.
Durable workflows vs. job queues
What happens when a server crashes in the middle of a turn.
Resume from any point
Every step is recorded, so a new server picks up from the last one.
Safe retries
Steps are written so running one twice does no harm. That's called idempotent.
Built in, open source
Timers, cancellation, and health checks come with engines like Temporal.
From the talk
- With a normal queue, a crash at step 3 means replaying steps 1–3, which is a real problem if they had side effects.
- This is mostly a solved problem in distributed systems. They use Temporal (open source) instead of rebuilding it, though you still have to run its infrastructure.
- Every helper sub-agent is also its own durable workflow (see §19).
Rules for handling new messages
Your own messages interrupt whatever it's doing. Everything else waits in line, and messages that arrive close together are handled in one turn.
From the talk
- Priorities differ: messaging your own bot vs. a group chat, a Slack thread or a background routine.
- "Change the date for this event" should stop the agent loop. Your message beats everything else.
- Everyone else goes to the back of the line. Three or four quick messages get batched into one turn, and the batching heuristic is yours to choose.
The harness
What changes compared to the week 1 harness: how it talks to you, how it finds tools, how it hands off work, and how it stays safe.
One tool for talking to you
All the work tools run on the server. The only one that reaches your app is SendToUser.
Thinking stays private
The model's regular text is never shown to you.
Quiet is allowed
If there's nothing new, it doesn't message you.
Rich messages
Text, files, multiple choice, and secure password entry.
One less step
The last message can end the turn without another model call.
From the talk
- Lee called this the biggest difference in the Grok Bot harness. The client is basically texting the server.
- Adding a new client (mobile, desktop, email, Slack) means implementing just this one tool.
Pairing tools with the right UI
The harness has dozens of tools. When you need to see or approve something, the tool shows a purpose-built control.
Sign-in forms
Fills your login only on the site that's open. The password never reaches the model.
$80.00Approval needed
Pay Starlink (starlink.com) for the Roam Unlimited – 300GB service plan.
› Show the breakdownPayment approvals
Purchases wait until you tap Allow or Deny.
Could we move Friday's design review from 11 AM to 2 PM? A client call came up…
Email drafts
You see the whole email and choose Send or Discard.
Reactions
Sometimes a reaction is the right reply. No message needed.
From the talk
- The app decides how to show each message. If the server says you need to log in, the app shows a sign-in form.
- Logins connect to password managers like 1Password. Payments go through services like Stripe Link or X Money.
- Anything you can't reverse, like sending an email or a Slack message, comes back as a draft for you to edit and approve.
Load tool details only when needed
Each square is a chunk of the context window (illustrative, like on the slide). Every connector you add, like Drive, Gmail or YouTube, brings tools with long schemas.
From the talk
- The context window is the model's working memory, with a hard token limit. Each tool brings a name, a schema and a description.
- Some call it "dynamic tool discovery," others the "tool search tool." Only tool names go into every message. When the model wants a tool, it reads the details from a file ("files all the way down").
- That saves context, which also saves cost and usage.
Try the cheapest, most reliable option first
The order it tries things in when it needs information. It only moves up when the cheaper option didn't work.
From the talk
- Lee compared it to a colleague who tries a few things on their own before coming to ask you.
Three ways to use a computer
A separate helper agent does the clicking, so screenshots don't fill up the main agent's context.
Read the page structure
It reads the page's list of buttons, links, and fields, and clicks them directly. Faster and more reliable than screenshots.
Click on pixels
Mouse and keyboard on a 1280×800 screen, for desktop apps, system dialogs, and anything else.
Ask a person
At logins, verification codes, captchas, or payments, it stops, explains where it is, and hands you the screen.
From the talk
- A screenshot works, but there are more efficient options. HTML is one. The accessibility tree cuts token usage a lot and points straight at the buttons, links and other elements to click.
- Pixel clicks are still needed for things like pop-up modals and dialogs.
- Worst case on the ladder: hand back to the user for a two-factor code or a captcha.
One main agent, many helpers
The main agent coordinates and tries to keep its own conversation short.
From the talk
- Helpers run the shell commands and expensive tool calls. None of that bloats the main context, and only the result comes back.
- This is why it seems to have unlimited context. It doesn't, but with these tricks (and models trained to use them) you don't run into the limit or see quality drop.
- Each helper is its own durable workflow, so if the video helper crashes or runs out of memory, it restarts without losing progress.
- Lee compared it to how you'd organize a team of engineers on a big problem.
Tips for designing agent tools
Practical rules they follow when adding tools.
Don't add or remove tools between turns
If a tool can't be used right now, have it return a clear error. Changing the tool list breaks the prompt cache.
Keep each tool's description and schema under 4,000 characters
Long schemas take up context, which costs money.
Use a dedicated tool for saving state instead of shell commands
That way reviewers can see what's changing. You can do almost anything through the shell, but if the model keeps running the same kind of shell command for one job, a dedicated tool is cheaper and easier to follow.
Write errors that say what to do next
Clear error messages help agents for the same reasons they help people.
The computer is asleep. Use Shell or Read to wake it, then try again.
How we keep it safe
If it's going to work on its own for long stretches, you need to trust what it does with your data and accounts.
Automatic review
A separate model checks risky actions before they run. If it blocks one, you get an approval request. No answer in 15 seconds means blocked.
Track the source
Text from emails and webhooks is labeled untrusted. Memories are treated as information, not instructions.
Asks first
It won't email or message people unless you asked it to. New routines need your OK.
Keep secrets hidden
Passwords go into environment variables, never to the model. Actions on your own laptop need your approval.
From the talk
- Even on its own isolated machine, you still want a model to review every shell command.
- An untrusted input, like a phishing email in your inbox or a webhook, must never be treated as a user instruction. The industry still has a lot of work to do on prompt injection.
- Non-reversible actions always ask first, and product workflows keep passwords out of the model conversation.
Context engineering
Lee used this interchangeably with "harness engineering": ways to use less context, which saves money and keeps the model working well over long sessions.
Keep the start of the prompt the same
Each call resends the whole conversation plus the new message, so ideally most of those tokens hit the provider's cache.
Stays the same until the next summary
Tool definitions
System prompt, including a snapshot of memory and routines
First message: environment details and a list of tool names
Conversation history
New messages are only ever added at the end
Newest message
The time, short notes on anything that changed, then your message
Providers charge less and respond faster when the start of a prompt matches a recent one. So stable content goes first and what changes goes last.
From the talk
- Model pricing gives a big discount on cached tokens, so you want to stay inside the cache window as long as possible.
- Keep tool definitions and system prompts as static as possible between calls. Then the only full-price delta is the newest message at the bottom.
- "There's a whole art and science to this."
Treat a cache miss like a bug
The rules from the slide. On the right is a made-up example of hashing each section.
- Until the next summary, the tools, system prompt, and first message must stay exactly the same
- If a deploy changes a tool description, running conversations get a note instead of the new text
- Everything is built in a fixed order and sorted, so nothing moves around between calls
- Tools are never added or removed for a single turn. Unavailable tools return an error instead
- Each section is hashed separately, so we can see exactly which part changed
- When you open the chat or start typing, the server prepares the prompt early to warm the cache
7c1e…a2unchanged3f9b…01unchangeda04d…9eunchangedappend-onlyprefix grows—newchangedcache missExample only. With a hash per section you can tell which part changed when the cache hit rate drops.
From the talk
- Back in January, when OpenClaw got popular and people ran it on model-provider subscription plans, harnesses weren't optimized for always-on usage. There were lots of cache misses, and many fixes had to go in.
- Keep a fixed section order so an A/B test of a new prompt section doesn't bust everyone else's cache.
- Warm the cache the way a website prefetches a link on hover.
Keep the main context small
The main agent has the longest conversation and gets summarized many times, so anything bulky is kept out of it.
Long outputs go to files
The context keeps only the file path
Screenshots go to a helper
The context gets what it found
Tool-heavy work goes to helpers
The context gets a short report
Old messages get summarized
The context keeps a short summary
Old memories stay searchable
Only the most important facts are in the prompt
Instructions become skills
The prompt keeps a one-line pointer
Heavy things live outside the main conversation. Only a pointer or a summary comes back.
From the talk
- Agents are very good at reading files, so dumping a long output to a file works as well as reading a skill or a memory.
- Lee described it as constant gardening: trimming what you don't need, for cost and so the model remembers the rest better.
Summarize while you're away
You'll have to summarize eventually, and it's cheapest while the provider's cache is still warm.
A big turn ends
The prompt was over 100,000 tokens
About 10 minutes pass
The provider's cache is still warm
Write a summary
Uses the same start of the prompt, so it reuses the cache
Your next message
Starts from the short summary
Skipped if background work is about to wake the agent anyway
The agent's last few messages to you are kept word for word
If a turn gets too long, it can summarize in the middle
From the talk
- With, say, a 200k-token window you eventually have to summarize, and all compression is lossy. Deciding what to keep is hard (Lee joked there are whole PhDs on it).
- Provider caches stay warm for roughly 10 to 60 minutes. Compacting inside that window is much cheaper.
- When you come back 30 minutes later, you continue from a cheap summary instead of a 100k-token history.
How the agent remembers
Four layers, from facts that are always in the prompt to things it only finds by searching. Lee compared it to keeping today's details in your head and letting the rest fade.
Profile
Core facts about you. Always in the prompt: up to 100 facts in 4,000 characters.
Log
Dated events. The 30 most relevant recent ones go in the prompt.
Notes
Short-term details that fade quickly.
Everything else
Found with a memory search tool when needed.
- Important facts rank higher, and older facts slowly rank lower
- After each turn, a pass adds new facts and replaces old ones. It can only remove facts it was shown
- Facts about you are shared by all your agents. Each agent writes to its own file
- The model is told that memories are information, not instructions
From the talk
- Some facts must be in every message ("the user always wants lowercase"). Otherwise one message would suddenly come back in uppercase.
- There's a kind of garbage collection of facts, keeping what's relevant right now.
- Scratchpad notes are designed to disappear and might only matter for the next 30 minutes or an hour. Everything else is found through files and shell search.
Where it's going
How they train the model for this product, how product fixes feed into training, and what Lee expects next.
Training the model for the product
A quick refresher on the three training stages, and where the harness comes in.
Pre-training
Learns language, code, and general knowledge from a huge amount of text.
Supervised fine-tuning
Learns the harness from examples:
- the tools and their formats
- when to send a quick reply
- what an ideal response looks like
Reinforcement learning
Practices real agent tasks and is rewarded for doing them well:
- following long instructions
- using skills and memories
- clicking the right spot (grounding)
- working with other bots
Main agent: fast replies
Inference and harness are tuned so it answers in seconds.
Helpers: long tasks
Tuned to finish hours of work reliably and control the computer quickly.
From the talk
- Supervised fine-tuning shows the model the harness so it understands which tools exist. RL teaches it to pick the right skill out of hundreds and to click the right spots in a browser.
- Serving matters too. The main agent needs low latency, but for a helper doing very long work, 10 minutes vs. 12 is negligible, so you optimize each one differently.
Product and model improve together
A daily loop of fixing the product, and a slower loop of training new models. Evals connect the two.
Every bot that doesn't work well becomes an eval. Evals improve the product today and the next model tomorrow.
From the talk
- Example failure: the model called a tool it wasn't supposed to. That becomes an eval, then a fix in the harness, prompt or inference, a check that the eval passes, and a ship.
- New models let people use the product in new ways. Bots in group chats are one example: there's very little training data for it, and Lee pointed to Moltbook and the Hugging Face incident as examples of it going wrong.
The new building blocks
The comparison: in 2005 every web app had a server, a database and a CRUD UI. Agent products all look alike now for similar reasons.
Many products will look alike but build these pieces very differently. Expect a long tail of specialized, best-in-class versions.
From the talk
- Lee credited the comparison to a post on X by Alex from OpenRouter, replying to people complaining that every AI app looks the same.
- Every new product will either have an agentic side or become headless data that other agents use.
- Lee's rough guess: each box has billions in VC money and about 20 startups behind it.
The next 6–12 months
What Lee expects, what's still unsolved, and a principle from Cursor that they kept at SpaceX.
Lee's predictions
- Agents running for weeks or months at a time
- Better recall of past conversations (graphs? files? something else)
- Digital colleagues get every tool a new human hire would
- Simpler interfaces: speech-to-speech, dedicated devices
- Skills learned on the job and encoded into files
- Companies write down bespoke processes so agents can read them
- A lot more tokens in use, so efficiency matters more
Still unsolved
- How do you evaluate an agent that runs for a month when models ship every month?
- Memory algorithms and representations
- Securing the harness and infrastructure against takeover
- When to stay silent vs. speak up. "I recorded the class you missed" could be helpful or annoying
Delete the product
Assume today's models are the worst you'll ever use. The UI might need rebuilding in six months, and Lee said this talk would have looked very different six months ago. So build for the next model, build as little as you can, and keep the interface simple.
Takeaways
Closing advice for the class.
Think about system design
Architecture decisions have big long-term effects. Lee still looks at the code ("I'm still pro looking at code"), mostly to understand the architecture being shipped.
Build a version yourself
Take the week 1 harness and try adding some of this: a sandbox or VM, durable workflows, a SendToUser tool. Lee mentioned there's probably an open-source clone on GitHub already.
Learn how LLMs work
You don't need deep ML, just enough intuition. It's like how a web developer should know how databases work: a lot of harness decisions depend on how the model behaves.