GitHub's new Git storage architecture, and what 200 agents did to stock Git

On November 5, 2026, GitHub published the design of its new Git storage architecture, Project Rubicon. In internal tests it took 7,200 pushes per second to one repository and supported 5,000 concurrent agents working on it. It is in limited private preview, with a broader rollout planned for early 2027. The design moves branch and tag pointers into a small replicated log called Ref-CAS, puts Git objects in Azure Blob Storage, and takes compaction off the machines that serve traffic.

All three are responses to many writers updating one repository at once. Every team running parallel coding agents has a smaller version of that problem, against a laptop checkout or a bare repo on a build box. So we ran up to 200 simulated agents against stock Git 2.54 on one machine. A shared branch capped throughput no matter which ref backend we used. Git's default auto-maintenance did worse: it left the test repository with refs pointing at missing objects in all four runs at 200 agents. Setting maintenance.strategy to incremental stopped it.

What GitHub changed in Rubicon

The old system, Spokes, keeps a full copy of each repository on several fileservers, five by default. Its write path is the problem. GitHub's own description: "Every replica participates in every write, so a push is only as fast as the slowest replica in its set." A follow-up post on the interim bridge design adds that each write held a repository-wide lock until it committed.

Agents push that design harder than people do. GitHub reports pushes grew 4.9x year over year, from 0.69 billion to 3.35 billion a month, and notes that "an agent in a tight loop commits or checkpoints after nearly every action." One repository went from an average of 44M requests a day in March 2026 to 181M in May. Work on Rubicon started that May.

GitHub tried two simpler designs first. A write-ahead log for all of a repository's references indexed by a single blob topped out at write throughput "in the mid-hundreds per second". Splitting references across several segment blobs raised the ceiling for writes, but reads became "only as fast as the slowest segment". Ref-CAS keeps reference state alone and, per the post, transacts it "in the tens of thousands of operations per second". Karthik Raman and Brian Celenza, who wrote the post, put the separation plainly:

Ref-CAS never holds or validates object data. It only stores pointers to it.

A push uploads its pack to Blob Storage first, then commits one Ref-CAS transaction that updates the manifest and the refs. A ref can only ever point at objects that already exist durably, and the local failure below is a ref pointing at objects that do not.

Reads get cheaper for the same reason. In GitHub's words, "the repository's version is the LSN of its latest commit", the log sequence number in Ref-CAS. A virtual replica checks that version, pulls any packs it is missing from Blob Storage into its local cache and serves the request. Because that disk is only a cache, our reading is that GitHub can add replicas for read load without slowing writes. Under Spokes, by GitHub's own account, "adding replicas to absorb read load makes writes slower."

Every Git ref update is already a compare-and-swap

Ref-CAS is a new service, but the rule it enforces is old. When git-receive-pack updates a branch, it only succeeds if the branch still holds the value the pusher expected. Lose that race and you get the error that sits in Git search autocomplete as "cannot lock ref is at but expected".

We reproduced it with 50 clones pushing straight to main five times each, never fetching. All our tests ran on one machine: Apple M3 Pro, macOS 27.0.1, git 2.54.0 (Apple Git-157), with a local bare repo as the remote over file transport. Of 250 pushes, 5 landed on both the files and reftable backends. Most losers were rejected with (fetch first). Another 20 on files (25 on reftable) got as far as the server and lost the swap there:

remote: error: cannot lock ref 'refs/heads/main': is at <sha> but expected <sha>
 ! [remote rejected] HEAD -> main (incorrect old value provided)

A single ref has one winner per old value. Rubicon should make each swap cheaper. No storage design lets two agents advance the same branch from the same parent, so Rubicon will not speed up your agents' pushes to main.

Twenty agents on one branch spend most of their time retrying

Real agents retry. Commit, push, and on rejection run git pull --rebase and push again. We ran that loop with each agent landing five commits on a shared main. Every commit eventually landed. The cost of getting there grew much faster than the agent count.

At 20 agents each landed commit cost about 7.5 failed pushes, and the branch moved at half the rate one agent managed alone. The ref backend made no real difference. It is one machine and one data point, but the shape follows from the contract above: N writers racing for one swap means N-1 of them redo work.

The fix is in how you lay out the work. Give each agent or subagent its own branch, and let one integrator, a merge queue or a person, move main. That is also the layout GitHub is optimizing for: in internal testing, its interim bridge design gave writes to different references 3-4x more throughput at lower latency.

On separate branches, files scaled and reftable needed a longer lock wait

With 200 agents pushing three times each to their own branches and auto-maintenance off, the files backend landed 600 of 600 pushes in all five runs, in 10.9 to 12.6 seconds. The reftable backend was a little faster at 9.3 to 10.8 seconds, but dropped up to three pushes per 600 in four of six runs, with this error:

remote: error: cannot lock references
 ! [remote rejected] HEAD -> agent-40 (failed to update refs)

The reftable format documentation explains why: "Because a single tables.list.lock file is used to manage locking, the repository is single-threaded for writers." The files backend locks each reference separately, so 200 agents on 200 branches never contend. Under reftable they all queue for one lock, and reftable.lockTimeout gives up after 100 ms by default. Raising it to one second removed the failures in three of three runs:

git config reftable.lockTimeout 1000

This matters because Git 3.0 will make reftable the default for new repositories. The project lists real gains: multi-ref writes that are atomic, deletions that do not rewrite packed-refs, and no case-sensitivity clashes on macOS and Windows. Git has set no release date for 3.0. If you opt in early with init.defaultRefFormat for an agent-heavy repo, raise the lock timeout too. And git refs migrate cannot convert a repository that has worktrees, which rules it out for any agent setup built on them.

Auto-maintenance corrupted the repository under agent load

Our first run used Git's defaults, and 11 to 34 percent of own-branch pushes failed at 50 agents. None of the errors were about locks:

remote: error: refs/heads/agent-8 does not point to a valid object!
remote: fatal: bad object refs/heads/agent-22
 ! [remote rejected] HEAD -> agent-50 (missing necessary objects)
remote: error: task 'pack-refs' failed

By default git-receive-pack runs git maintenance run --auto after every push (receive.autogc). A trace of 60 sequential pushes showed 60 detached maintenance runs. At 200 agents, 69 to 90 percent of 600 pushes failed. We then waited for every background maintenance process to exit, then ran git fsck. All four runs, two per backend, still had branches pointing at objects that no longer existed:

error: refs/heads/agent-N: invalid sha1 pointer <sha>
fatal: missing object bf0da3f300755a5adb4e3d74f4cfc32d96a401de for refs/heads/agent-16

With receive.autogc set to false, the same 200-agent runs had no object errors at all, and git fsck came back clean.

Git's documentation does not hide this. The git-gc manual says a concurrent gc can delete an object another process is using but has not referenced yet, which "may corrupt the repository if the other process later adds a reference to the deleted object." Its mitigations, it says, "fall short of a complete solution, so users who run commands concurrently have to live with some risk of corruption (which seems to be low in practice)." That estimate fits human-paced writes, and our agents wrote in bursts of hundreds of pushes within seconds.

Many agent setups skip the bare repo and use worktrees attached to one repository, which is what the four-agent Claude Code pipeline guide recommends for parallel stages. Every worktree writes into the same repository, and a plain git commit starts a detached git maintenance run --auto too (maintenance.auto, default true). With 50 worktrees making 10 commits each at once, 1 to 16 of 500 commits failed per run with errors like fatal: failed to write commit object and error: unable to create temporary file: No such file or directory, and git fsck found refs pointing at missing objects in two of five runs. With maintenance.auto off, four of four runs were clean.

GitHub drew the same line for its servers: Spokes ran compaction on the replicas that serve traffic; Rubicon gives it separate nodes and leases. A local repo has no separate nodes, so the next best move is to stop the maintenance that deletes data from running in the middle of a burst. We did not pin down which maintenance step removes the objects. Our pushes were also small, and Git stores a push of fewer than 100 objects as loose objects (transfer.unpackLimit), which likely gave maintenance more to do than a typical push would.

The config change we tested

Switch the repository's maintenance strategy:

git config maintenance.strategy incremental

# stricter, for a bare repo that agents push to
git config receive.autogc false

The git-maintenance docs describe incremental as "small maintenance activities that do not delete any data" that do not schedule the gc task, and the strategy applies to auto-maintenance as well as scheduled runs. With auto-maintenance still on and the strategy set, four 200-agent push runs landed 600, 600, 600 and 599 of 600, and git fsck was clean after every one. Three 50-worktree runs landed 500 of 500 with no errors. git maintenance start applies the same strategy if none is set and schedules hourly background runs, so the repo still gets packed.

Here is every setting we touched, with Git 2.54's default and what we ended up using for a repository that many agents write to at once:

A full git gc still deletes unreachable objects. Run it when no agent is writing, for example at the end of the session your AI harness drives, not on a timer that can fire in the middle of a burst.

What to watch

Rubicon's broader rollout is due in early 2027. It is storage under GitHub, so its gains land on GitHub's servers, for many agents pushing to different refs.

On the Git side, watch the reftable default in Git 3.0 and whether git refs migrate learns to handle worktrees. If agents push to a shared repo and you see does not point to a valid object, run git fsck before you trust that repo again. The question we could not answer is how often the maintenance race fires on Linux or over SSH, which we did not test.