What we already know, and how we know it.
The systems we build today carry the lessons of systems we have already had to keep running, scale, change, integrate and recover.
Three kinds of knowing follow, in order: what experience taught us, what experiments can prove, and what only time can prove.
Problems we've already had to solve.
Eight situations that arose in systems that had to keep running. None of this is a capability we offer, and none of it is something a reader can check — it is our account of what happened and what it left behind.
When everyone arrives at once
Problem
A transaction was not one database update. It could involve authorization, eligibility, risk, balances and several other checks before it was complete.
At peak moments many users reached those same paths at once. Transactions began waiting on each other, database contention grew, and the system slowed down.
What changed
We separated the work. Processing was distributed across independent components instead of forcing every transaction through the same blocking path.
Concurrency stopped being only a question of how much traffic the system could receive. It became a question of how little unrelated work had to wait for unrelated work.
Distributed processing · independent services
When the system cannot stop
Problem
Some systems do not get a convenient maintenance window. Users are active, transactions are happening, and releases still have to go out.
Taking the application offline every time something changed was not an acceptable operating model.
What changed
We learned to change a running system. Traffic could be moved across running instances while a new version was introduced, without taking the user experience offline.
That made releasing without downtime part of the architecture rather than a deployment trick.
Kubernetes · blue-green releases
When software has to keep living
Problem
A successful release is not the end of the system. Some of the systems we have worked on have operated continuously for years, and during those years the software did not stay frozen.
New versions kept moving into production the whole time.
What changed
We built the ability to evolve into the operating model. Release, recovery, distribution of traffic and continued operation became part of the system itself.
The goal stopped being to deploy software. It became to keep changing software that had to stay alive.
When one product has many screens
Problem
The user might arrive from an iPhone, an Android phone, a tablet or a desktop. Building an independent product for every screen would have multiplied both the software and the work of keeping it alive.
What changed
We built the experience to adapt: the interface responds to the device while the underlying product stays the same.
The screen stopped being the product. It became one way into it.
Responsive interface · one product
When the infrastructure belongs to someone else
Problem
Some systems run on infrastructure owned and operated by the customer. We still had to deploy them, support them, secure them, observe them and work within real service-level commitments.
What changed
We learned to treat the operating environment as part of the architecture. Deployment, monitoring, recovery, security and support could not assume the infrastructure was ours.
The system had to work where the customer needed it to live.
Two-factor authentication · monitoring · service-level commitments
When your system is only one piece of the system
Problem
Production software rarely lives alone. Ours has had to connect to customer platforms, internal systems, external providers and teams with their own responsibilities and release schedules.
Connecting two APIs was only one part of the problem.
What changed
We learned to engineer the boundaries too. Dependencies, ownership, failures, releases and coordination across teams became part of delivering the system.
Integration turned out to be an operational problem as much as a technical one.
When the requirements are not the product
Problem
Business requirements could describe what needed to happen without defining how the finished application should work for the person using it.
Someone still had to decide how the screens behaved, how people moved through them, and how business rules became a usable product.
What changed
We learned to work from the problem, not only from the specification. The work grew to include mocks, interaction design, product decisions, architecture, implementation, deployment and operation.
Some of the most important parts of the finished product were discovered while building it.
When repeated manual work should stop being manual
Problem
Information had to move, actions had to be coordinated, and repeatable operations still depended on someone carrying the same steps again and again.
What changed
Where the rules were clear enough, we moved that work into the system. Repeatable processes became automated parts of the product instead of recurring operational tasks.
Some claims can be measured.
Some lessons came from years of production systems. Others came from reducing a hard question until we could test it directly. Every entry below was measured on a stated date and published on this site, so it can be opened and checked.
- 01Problem
Records had to keep arriving with the remote database down.
What changedTen thousand writes taken while it was stopped, and not one of them lost when it came back.
- 02Problem
An institution's code had to run on somebody else's phone, without being trusted.
What changedFour ways to do it, measured on real devices — and a sandbox that closed eight of eight attempts to get out.
- 03Problem
A shared game, with no server in the room.
What changedA bingo night running on the phones in one hall: one history per handset, nothing outside it taking part.
- 04Problem
One domain had to reach four screens without being rewritten for each.
What changedA console, a browser, a desktop window and a phone — and the diff over the domain came back empty.
- 05Problem
A microservice that could not leave its datacenter.
What changedThe same ordering service running on three machines that a printed QR code admitted.
- 06Problem
After a crash, who can say whether a message actually arrived?
What changedWe killed the sender on purpose, between deciding to send and sending. The record says only what it can know — and when it can’t know, it says nothing. Anything else would be a guess.
Built is not the same as running.
A laboratory cannot show you that something lasts. Every result in the ledger above was measured on a stated date and published so it could be checked, which is the right standard for a claim about how a system behaves — and the wrong instrument entirely for a claim about how long it keeps behaving that way.
Two generations of one system have been answering that second question for us, and they are in production side by side: the older was never switched off to make room for the newer. Whose they are and what they do are not ours to say.
In that time they have changed, scaled, integrated with other systems, moved through new versions and kept serving users while the work around them kept moving. What can be said beyond that is structural, and it is the part that does not resemble how this is normally done.
- Releases several times a week.
- Sometimes several times a day.
- New versions introduced without taking the user experience offline.
Counted in the newer of the two generations
- Fourteen versions of the domain — and it still starts by replaying its own history from the first one, rather than from a database left in whatever shape the last change gave it.
- Four different builds of the software underneath it, live at the same time. Each service stays on the one it was proven on, and none of them forced the domain to be rewritten to keep up.
- No cache and no lock service anywhere in it. The state is held in memory and the history is the record, so there is no second copy that has to be kept true.
- Eighty-two third-party packages in the whole of it, in a system that has changed every single year since it went live — the quietest of them still had hundreds of changes.
Some things can be measured in a test. Others take years to prove.