Building a Google Docs clone is a great system design exercise. The first version is easy, and then every step after it breaks something we already built.

The requirement sounds simple. Multiple users editing the same document. Let’s see how it goes.

The First Version

Our server needs to push. Someone else’s keystroke has to show up on our screen without us asking for it, so we are using websockets. That part is obvious and we are not going to spend time on it.

What Happens When We Add a Second Server?

It works fine, until we get some real traffic. What happens when a single server can’t handle the traffic? What if there is an error and the only server we have is down?

To handle that we deploy multiple servers behind a load balancer.

Here is our problem. Alice connects and lands on server A. Bob connects and lands on server B. They are both in the same document. Alice types something. Server A looks up who else is in that document, finds only Alice, and forwards her edit to nobody.

Each server only knows about its own connections, and a document’s users are now spread across all of them.

Redis Pub/Sub

We need our servers to tell each other about edits.

We could have every server hold a connection to every other server, but then each instance needs to know the current list of instances, and that list changes every time we deploy or autoscale. That is a service discovery problem we don’t want to own.

So instead we put a message bus in the middle, and the usual choice here is Redis Pub/Sub.

What Is Redis Pub/Sub?

Redis Pub/Sub is a live broadcast.

A client subscribes to a channel by name. From that moment, anything anyone publishes to that channel gets pushed down that subscriber’s connection. That is the whole feature.

We let Redis do all the filtering and the server only gets updates that it actually cares about.

The important part is what happens when nobody is listening. If we publish to a channel with no subscribers, the message is simply gone. Redis does not save it, does not retry it, and does not remember it ever existed. When we publish, all we get back is the number of subscribers that received the message.

This is called at most once delivery. It is what makes pub/sub fast and simple, since Redis is not writing anything to disk or tracking who acknowledged what. It also means pub/sub can never be our source of truth. It is a way to move messages between our servers, not a place anything is actually kept.

Using It for Our Documents

Now we can fix our two server problem. Each document gets it’s own Redis pub/sub channel. The server subscribes to the documents it is using. And when an edit comes in it publishes to that channel instead of writing to the local sockets.

We subscribe when the first local client opens a document and unsubscribe when the last one leaves.

One thing worth noting. When a server gets an update, it should not deliver it to its own sockets, it should just publish it. Redis will send it right back to the same server (and other subscribed servers) and only then can we start handling it like we handle every update. This saves us from maintaining a handler for local updates and a different handler for external updates.

Now edits reach everyone, no matter which server they are connected to, and our servers still don’t know anything about each other.

Two People Typing at the Same Time

We can move messages between any two users now. Unfortunately the messages are wrong.

Alice and Bob can edit the same text at the same time. Just like in Git, we get a conflict. In Git we give the second editor the choice of how to handle it. Of course we can’t do that in a text editor.

Use a CRDT (Conflict-free Replicated Data Type) and let a library like Yjs or Automerge deal with it.

What is worth knowing is how that choice changes our architecture. A CRDT gives every character we insert an id that never changes, so an operation becomes “insert this character after that character” instead of “insert at position 5”. Those operations commute, which means clients can apply them in any order and still end up with the same document.

That is a big deal for us. Order is the thing distributed systems usually fight about, and we just stopped caring about it. Our broadcast makes no promises about the order two servers see messages in, and now it doesn’t have to. No server has to be the authority on anything, so any server can accept any edit for any document, our fleet stays stateless, and we can deploy and autoscale without thinking about it.

Commuting does not mean we can lose messages though. Order doesn’t matter, but arrival still does. Every copy of the document has to see every operation eventually, and pub/sub is at most once, so sooner or later it drops one. Nobody gets an error when that happens. The two servers just quietly stop agreeing about what the document says. We will fix that in a minute.

Saving the Document

Everything so far only exists in memory. Restart a server or lose the last client editing a document and it is gone.

The obvious fix is to write to our database on every keystroke, but that is not going to work. A fast typist produces five to ten operations a second, and a busy document has a few of those at once. That is a lot of transactional writes for data that gets overwritten again a few milliseconds later.

What we want is write behind. Edits are applied in memory as they happen, and the expensive database write happens on its own schedule, well behind them.

So we snapshot the document into our real database every so often. Every N operations, or every few seconds of activity, and also immediately when the last client leaves, since that is the moment the document is about to disappear from memory. Loading is the same thing in reverse. When the first client opens a document nobody else is editing, we read the snapshot back and we are ready to go.

The tradeoff is right there in the interval. Whatever was typed since the last snapshot only exists in memory, so a crash takes it with it. Snapshot more often and we lose less but write to the database more. It is a number we should pick on purpose instead of ending up with by accident, and we can tighten it for documents that matter and loosen it for scratch pads.

CRDTs soften this too. Every client is holding its own full copy of the document, so a client that reconnects after a server died can merge what it has back in. The snapshot is our floor, not our only copy.

Catching Up

There is still a hole on the way in.

Carol opens the document and lands on server C. That server has never seen this document before, so its memory is empty, and so is Carol’s browser. It subscribes to the channel, which means it now hears every new edit while knowing nothing about the ninety pages that are already there.

So it reads the snapshot out of the database, and this is where write behind comes back around on us. That snapshot is from the last flush. Alice and Bob have been typing for the five seconds since, and none of that is on disk yet. Carol gets the document as it was five seconds ago, and then live edits start landing on it that were written against text she doesn’t have.

The current version does exist. It is sitting in memory on server A, where Alice is typing. It just hasn’t been written down.

So server C asks for it. It publishes a message on the document’s channel saying how much it already has, and server A is subscribed to that channel, so it answers with the rest. No new connections and nothing to discover, we are using the pub/sub that is already there.

What server C sends is called a state vector, and it is tiny. Each person’s operations are numbered in order, so “I have Alice’s first four hundred and Bob’s first thirty” is a couple of numbers.

This is not a compressed copy of the document. It is a bookmark. It says how far along we are and nothing whatsoever about what any of those operations were.

The document itself still has to travel. Server A compares those numbers against its own and sends back the operations that are missing, in full. Cold open and that is the whole document, which was always going to be true. Five seconds behind and it is five seconds of edits instead of ninety pages, which is the case we actually care about.

Carol then does the same thing with server C over her websocket, and everybody is current.

This is our dropped message fix too. A server that missed a broadcast is in exactly the same position server C was in, holding a copy that is quietly behind. Same question, same answer. A lost message stops being permanent and becomes something we recover from.

When Pub/Sub Becomes the Bottleneck

At a big enough scale, the pub/sub is the thing that starts hurting.

Every server subscribed to a document’s channel gets every message on that channel, and delivering them is work Redis does on its single thread. The fan out multiplies quickly. Two hundred servers each subscribed to five thousand documents is a million subscriptions on one instance.

It is also all shared, so a single popular document eats Redis time and slows down all of our users. This is the noisy neighbor problem.

Here is what helps, roughly in the order to try it:

Stop publishing on every keystroke. Batch operations over a short window, something like twenty to fifty milliseconds, and publish them as one message. Users cannot feel the difference and it can cut our message volume by an order of magnitude on an active document. This one is easy and it should be the first thing we do.

Watch out for Redis Cluster. This one surprises people. In a normal Redis Cluster, pub/sub messages are broadcast to every node in the cluster, because any client could be subscribed on any node. So adding nodes does not divide our pub/sub load, it multiplies it, and the thing we scaled out to fix gets worse. Redis 7.0 added sharded pub/sub, which sends a channel only to the shard that owns it. Use it, or the cluster migration will be a downgrade.

Consider document affinity. This is sticky sessions, except we stick on the document id instead of the user id. Every document hashes to one server, every connection for it gets routed there, and that server holds all of its connections in local memory without touching the pub/sub at all. The problem is when a pod restarts and we need to hand off the document and all of its connections.

What If Every Document Had Its Own Server?

Let’s look back at where all of this actually came from.

The first version worked. One server, a map of documents, a forwarding loop. It was maybe thirty lines and it was correct. Then we added a second server, and from that moment on every single thing in this article has been damage control. The pub/sub, the channel per document, the affinity routing. None of it is about collaborative editing. All of it is about the fact that our users are spread across machines that don’t know about each other.

So what if they weren’t? What if every document had its own server, and everybody editing that document connected to it?

That solves everything. There is nothing to broadcast, because all the sockets are already in one place. There is nobody to coordinate with, because there is nobody else. We would just go back to the version that worked.

The reason we don’t do this is that it sounds insane. A server per document means a server for every meeting note and every grocery list anyone ever typed. That is millions of processes, each holding an operating system’s worth of overhead, and almost all of them sitting idle at any given moment. Nobody is provisioning a container per document.

But that objection is about the cost of a server, not about the idea. So what happens if the thing we spin up per document gets small enough and cheap enough?

V8 Isolates

This is what Cloudflare Workers are built on.

Most serverless platforms give each application a container or a micro VM. That is a real machine boundary, which means real memory overhead and a real startup cost, usually somewhere between a few hundred milliseconds and a few seconds. It is why cold starts are something we have to design around.

Workers don’t do that. They run our code in a V8 isolate, which is the same mechanism Chrome uses to keep one tab’s JavaScript from touching another tab’s. An isolate is a sandbox inside a process rather than a process of its own, so a single process can host thousands of them at once. There is no container to pull, no OS to boot, and no runtime to initialize per instance.

The numbers are what matter here. An isolate starts in a couple of milliseconds and carries a few megabytes of overhead instead of a few hundred. Cloudflare can even start ours while the TLS handshake is still finishing, so in practice the cold start disappears into time we were already spending.

That completely changes what “one per document” costs. A million containers is a fantasy. A million isolates, mostly idle, is just Tuesday.

Durable Objects

A Worker on its own is stateless, so it isn’t enough for us yet. It has no identity and no memory between requests.

A Durable Object is the piece that adds both. It is a Worker with a name and with storage attached to it, and Cloudflare guarantees that for a given name, exactly one instance of it is alive anywhere in the world at a time. Every request for that name is routed to that one instance, wherever it came from.

So we derive the object’s name from the document id. Now every client editing a document connects to the same object, because there is no other object to connect to. We got our server per document.

Look at what that deletes. There is no pub/sub, because all the sockets are already in the same place. There is no load balancer to make sticky and no affinity routing to build, because routing by name is the platform’s job. And storage is attached to the object and lives on the same machine, so a snapshot is a local write instead of a background job reaching across the network to a shared database, which means we can afford to do it far more often and lose far less.

Server C never has to ask server A for the difference either, because there is no server C. Carol still has to be handed the ninety pages when she opens the document, but the current version is either in the object’s memory or in its own storage, so there is nobody to ask and nothing that can quietly fall behind.

Our whole diagram collapses back into the first version we wrote.

Why This Scales Better

The surprising part is that this is not a step backwards. Going from many servers to one server per document sounds like giving up on scale, and it is the opposite.

Our Redis design has a shared bottleneck in the middle. Every document’s traffic goes through the same pub/sub, so documents compete with each other for it, and the way we scale is by splitting that pub/sub into more pieces. With Durable Objects there is nothing in the middle to contend on. Documents are independent by construction, so a million documents is a million tiny objects instead of a million subscriptions on one Redis instance. Adding documents adds capacity instead of consuming it.

That also kills the noisy neighbor problem, since a popular document is now its own object burning its own CPU with no shared pub/sub to slow down for everybody else.

We also stop paying for documents nobody is using. With websocket hibernation, an object that has open connections but no traffic is evicted from memory while those connections stay alive, and it wakes back up when a message arrives. Since starting an isolate costs a couple of milliseconds, waking up is cheap enough to do constantly. Most documents are idle most of the time, and now that costs us nothing.

And because objects are created near whoever first used them, a document a team in Berlin works on lives in Berlin, instead of in whichever region we happened to pick for our cluster.

What It Doesn’t Solve

It does not solve concurrent editing for us.

It is tempting to think it does, since the object is single threaded and handles one message at a time, so every edit passes through one place in a definite order. But an order is not a resolution. Alice and Bob both wrote their edit at the same time.

The Tradeoffs

Deleting most of an architecture is not something we get for free. Here is what we are paying for it.

We are marrying a platform. Durable Objects are a Cloudflare product. The programming model, the storage API and the deployment story all come from them, so if we ever have to leave we are not migrating, we are rebuilding.

It is not Node. Workers run in a V8 isolate with web standard APIs, not in a Node process. There is no filesystem, and a good chunk of npm assumes Node built ins that aren’t there. There is a compatibility layer that covers a lot of it now, but “does our stack actually run on this” is a question we have to answer before we design around it, not after. For our case we are fine, since a CRDT library is pure JavaScript, but that is luck rather than a general rule.

One object is one thread. This is the same hot document problem as before, except now we cannot route our way out of it. The object is the unit, so a document with a thousand people typing in it has one single threaded process handling all of them, and no amount of scaling helps. It is the noisy neighbor fix seen from the other side, since nothing leaks out of an object and nothing can be spread out of one either. Worth noting that Google Docs caps simultaneous editors at around a hundred and sends everyone else to view only, so this limit is not unique to us. But we should know where our ceiling is instead of discovering it.

One instance is also one point of failure. The guarantee that exactly one object is alive is what makes the whole thing work, and it means there is no replica standing by. While that object is restarting, migrating between machines, or picking up a deploy, that document is unavailable and its sockets drop. It is usually brief, and clients reconnecting cover it, but our availability story for a single document is now “one thing, and it comes back quickly” rather than “any server can serve it”.

Location is decided once. The object is created near whoever opened the document first, and that is where it stays. That is great for a team in one city and less great for a team split between Berlin and Sydney, where somebody is always paying the round trip. We can give placement hints, but there is still exactly one location, and the first user picks it.

The cost model is different. We stop paying for idle servers and start paying per object, per request, and for how long objects stay awake. Hibernation makes idle documents genuinely cheap, which is the main thing. But it is a different shape of bill than “we already have some VMs and a Redis”, and it is worth modelling against our actual usage before assuming it is cheaper.

None of these are dealbreakers for a collaborative editor. The point is that we traded a pile of infrastructure we control for a primitive we don’t, and that is a trade with a direction. It is the right one surprisingly often, but it should be a decision.

Running It Ourselves

The vendor lock in argument took a real hit this month, when Ryan Dahl released celld.

celld is an open source implementation of Workers and Durable Objects that we run on our own machines. It is Apache 2.0, it takes the same Wrangler bundles and config, and it implements the same JavaScript APIs, so the code we wrote for Cloudflare is the code that runs here.

It is also extremely new, and the limitations are real. This is Ryan Dahl’s third JavaScript runtime after Node and Deno, So it is very promising.

So this is not what we build our editor on this quarter. But it does change what the lock in tradeoff means. Durable Objects are starting to look less like one vendor’s product and more like a shape that other people can implement, and that is a much better position for us to be standing in.

Summary

Let’s put the whole thing together.

A stateless websocket fleet behind a load balancer, each server holding its own local sockets and subscribing to a Redis channel per document. Edits are CRDT operations, which commute, so nobody has to be the authority on ordering and every server stays interchangeable. A connecting client tells the server what it already has and gets back the rest, which is also how a server recovers anything the pub/sub dropped. Documents live in memory and get snapshotted on an interval and when the last client leaves, and whatever was typed since then is what a crash costs us. When the pub/sub hurts, we batch, we shard it, and eventually we pin each document to one server.

That last fix is the tell. All of it exists because the sockets for one document are scattered across machines that don’t know about each other. Durable Objects remove the scattering instead of compensating for it, and the bill is a platform we don’t own and one thread per document.

Real time collaboration is not hard because pushing bytes is hard, it is hard because “everyone sees the same text” is a much bigger promise than it looks.