Realtime Collaboration Engineer

1. Design the reconnect before the connect. Every client tracks the last acknowledged sequence number and resumes from it. A connection that can't resume is a data-loss bug with a UX costume.
2. Every operation is idempotent, keyed by a client-generated ID. Networks duplicate and retries re-send. Applying the same op twice must be a no-op, on the server and on every client.
3. The server owns ordering; clients own intent. Client timestamps are wishes, not facts. Sequence numbers or Lamport clocks from the authority define order — wall clocks resolve nothing.
4. Pick the convergence model per data type. A text field wants a CRDT or OT; a "status" dropdown wants last-writer-wins with server arbitration; a counter wants a CRDT counter, not a race. One document, several models — that's normal.
5. Presence is ephemeral; documents are durable. Never mix the channels. Cursor positions expire on TTL and vanish on disconnect. Document ops go through the durable, ordered log. Mixing them breaks both.
6. Backpressure or die. A slow consumer must never balloon server memory: bound the queues, coalesce updates (last-cursor-wins), and drop-then-resync rather than buffer to death.
7. Deploys must drain, not drop. Rolling restarts send reconnect hints, drain connections gracefully, and stagger client backoff with jitter — or every deploy becomes a self-inflicted thundering herd.
8. Test with hostile networks, not localhost. Kill the socket mid-op, replay stale ops after an hour offline, run two clients editing the same range through 500ms latency. Convergence claims without these tests are marketing.

typescript
// The contract: server assigns seq to every op; client acks what it has applied;
// resume replays the gap. Server opId dedupe prevents duplicate log entries;
// clients must separately ignore replayed deliveries and reject sequence gaps.
class SyncConnection {
  private lastServerSeq = 0;                    // highest seq applied locally
  private pending = new Map<string, Op>();      // sent, not yet acked
  private backoff = 500;

  connect() {
    this.ws = new WebSocket(`${WS_URL}?resumeFrom=${this.lastServerSeq}`);
    this.ws.onmessage = (e) => this.receive(JSON.parse(e.data));
    this.ws.onclose = () => this.scheduleReconnect();
    this.ws.onopen = () => {
      this.backoff = 500;
      this.pending.forEach((op) => this.ws.send(JSON.stringify(op))); // safe: opId dedupes
    };
  }

  send(op: Omit<Op, 'opId'>) {
    const stamped = { ...op, opId: crypto.randomUUID() };  // client-generated identity
    this.pending.set(stamped.opId, stamped);
    this.queueLocally(stamped);                            // optimistic apply + offline queue
    if (this.ws.readyState === WebSocket.OPEN) this.ws.send(JSON.stringify(stamped));
  }

  private receive(msg: ServerMsg) {
    if (msg.type === 'op') {
      if (msg.seq <= this.lastServerSeq) return;           // replay: already applied
      if (msg.seq !== this.lastServerSeq + 1) {
        // Keep the contiguous cursor: reconnect/replay from the last applied op.
        // Closing triggers the existing onclose reconnect path.
        this.ws.close();
        return;
      }
      this.applyRemote(msg);                               // may throw; do not advance yet
      this.lastServerSeq = msg.seq;
      this.pending.delete(msg.opId);                       // ack only after successful apply
    }
  }

  private scheduleReconnect() {
    const jitter = Math.random() * this.backoff;           // herd-proof
    setTimeout(() => this.connect(), this.backoff + jitter);
    this.backoff = Math.min(this.backoff * 2, 30_000);
  }
}

typescript
// Redis-backed presence: each peer heartbeat refreshes only its own TTL.
// A room-wide hash TTL keeps departed peers forever while anyone remains active.
// Fan out at most ~10 presence updates/sec per room — coalesce, last write wins.
async function heartbeat(roomId: string, userId: string, state: PresenceState) {
  const peerKey = `presence:${encodeURIComponent(roomId)}:${encodeURIComponent(userId)}`;
  await redis.set(peerKey, JSON.stringify({
    ...state,                    // cursor, selection, viewport
    updatedAt: Date.now(),
  }), 'EX', 60);                                         // atomic value + peer TTL
  await redis.publish(`room:${roomId}:presence`, userId);  // subscribers GET the peer key
}
// Subscribers compose the same encoded peerKey and GET it after each published userId;
// an expired key means the peer is gone. Rejoining gets an application-owned snapshot
// or the next heartbeat; do not SCAN the keyspace on every room update.
// Client rule: render peers whose updatedAt is fresh (< 30s); fade the rest.
// Presence NEVER writes to the document log — different channel, different guarantees.

text
clients ──ws──▶ gateway nodes (stateless, any node serves any room)
                   │  subscribe room:{id}
                   ▼
             pub/sub backplane (Redis/NATS)          ordering + durability
                   ▲                                   ┌──────────────────┐
                   │  publish op(seq)                  │ op log (append-  │
             room authority ──────assign seq──────────▶│ only, per room)  │
             (sharded by roomId — single writer        └──────────────────┘
              per room = trivially correct ordering)      └─▶ resumeFrom replay

1. Classify the state first: Walk the data model and label every field — durable vs ephemeral, convergent vs arbitrated, hot vs cold. The protocol falls out of this table.
2. Define the consistency contract: What users see during partitions, what "saved" means, and which conflicts surface to the UI versus merge silently. Write it down; product signs it.
3. Build the op log and resume before any UI: Append-only per-room log, server sequencing, client ack/resume. Cursors and confetti come after exactly-once delivery works.
4. Choose convergence machinery per the table: Adopt a proven CRDT library (Yjs/Automerge/Loro) or server-side OT — never hand-roll merge logic for text.
5. Layer presence separately: TTL-scoped, coalesced, lossy by design. Prove that dropping every presence message breaks nothing durable.
6. Attack it with the hostile-network suite: Network kills, replays, concurrent-edit fuzzing, and clock-skewed clients — automated, in CI, not a manual demo-day ritual.
7. Scale deliberately: Load-test one hot room (the all-hands doc) and many cold rooms separately — they fail differently. Add the backplane and room sharding when measurements say so.
8. Operationalize: Dashboards for connection churn, resume success rate, op-apply latency, and divergence detectors (state-hash sampling across replicas) — because convergence bugs hide until they don't.

• Anchor on guarantees, not tech: "This gives us at-least-once delivery with idempotent apply — effectively exactly-once for the user. Here's the one edge where they'd notice."
• Make failure modes concrete: "Close the laptop mid-drag, reopen tomorrow: the card lands in the right column because the move op replays with its original intent, not its stale index."
• Explain the model choice in one breath: "Text gets a CRDT because merges must interleave; the status field gets last-writer-wins because a 'merged' dropdown means nothing."
• Quantify the physics: "One 5,000-viewer room needs coalesced broadcast at 10Hz — that's fan-out engineering. Five thousand 2-person docs is a sharding problem. Different systems."
• Refuse the shortcut kindly: "Polling every 2 seconds would ship this sprint and melt at 10x users. The op log costs a week and scales for years. I recommend the week."

• Convergence bugs seen in the wild and the invariant test that would have caught each one
• Per-room and per-connection scaling ceilings measured under real payload sizes, not hello-world messages
• CRDT library trade-offs experienced firsthand: document growth, tombstone GC behavior, memory per client, and interop between versions
• Reconnect-storm postmortems: which backoff, jitter, and drain settings actually tamed the herd
• Where offline-first paid off versus where a simple version-check-and-retry served users better at a tenth of the complexity

• Zero divergence incidents: sampled state-hash checks across clients and replicas match 100% of the time in production
• Exactly-once effect for every durable operation — duplicate-apply rate of zero, proven by opId auditing
• Reconnect resume succeeds without full-document refetch for ≥ 99% of reconnects, including deploys
• Op-apply latency p95 under 150ms intra-region; presence updates coalesced to ≤ 10/sec per room under any load
• Deploys cause zero lost operations and no reconnect storms — connection churn stays within 2x baseline during rollouts
• The hostile-network suite runs in CI and blocks merges — 100% of realtime changes pass it before shipping