The Container Push That Taught One Team to Use a Checklist

Posted by

–

What follows is an invented composite, stitched together from afternoons that many small engineering teams will recognise. A four-person team ran a handful of services in containers, shipped most days and prided itself on moving quickly. Nobody on the team was careless. The trouble was that nobody had ever written down what “ready to push” meant, and the story of how they fixed that is a useful case study in building a pre-deploy checklist for container releases, one short card at a time.

The afternoon that started it

On a Thursday, the developer who owned the payments service built a fresh container image and pushed it to the team’s registry with the tag the team used for “current”. Another service pulled that tag on its next restart. The image the developer had pushed was, unfortunately, built from a feature branch that was never meant to leave their laptop. Within minutes the payments service restarted with half-finished code.

The fix was quick: they restored the previous image and restarted. The damage was a short window of failed requests and a very awkward conversation. But the retrospective produced the useful line: “We did not do anything unusual. We did what we always do. And that is the problem.”

What the team realised

Over coffee, the tech lead listed what could have caught the mistake. A look at the branch name before building. A check of which tag was about to be overwritten. A glance at who else depended on that tag. Each of these took seconds, and all of them had been skipped because the push felt routine. The step was easy to perform and difficult to reverse, which is an uncomfortable combination.

She proposed a rule: before any push to a shared tag, whoever is doing it reads a short card aloud and answers each line. The team agreed to try it for a month. Crucially, they agreed on its scope. It would guard the push, not the whole development process. Code review, testing and branch policy were discussed in other forums.

The first version of the card

They wrote it on a whiteboard and photographed it. It had six lines.

  1. Which branch is this built from? Must be the release branch, confirmed by reading the build log.
  2. Which tag am I about to write? Say it aloud. If it is a shared tag, say who else uses it.
  3. What does that tag point to now? Note the existing image so it can be restored.
  4. Did the automated checks pass on this exact image? Not an earlier build; this one.
  5. Is someone around for the next hour? A named person, not “the team”.
  6. Do the people who depend on this know? A one-line message is enough.

It was imperfect, and they knew it. But it was theirs, and it addressed the actual failure they had experienced.

The second month: trimming and tightening

At the monthly review, the team found two interesting things. First, the card had caught three problems: a build from the wrong branch, a push to a tag that a different service relied on, and a push at the end of a day when nobody would be available. None were dramatic. All would have caused at least an evening of trouble.

Second, one line never fired. The team debated removing it and decided to rewrite it instead, replacing it with a line about confirming that a configuration file for the new version existed in the target environment. That change came from a near miss in the second week. The card stayed at six lines.

The principle they settled on was simple: every line must come from an incident or a near miss, and the list must stay short enough to read in two minutes. When someone proposed adding a new line, somebody else was asked which line it would replace.

What changed in how the team worked

The most striking effect was not fewer incidents, though that happened. It was a change in how people talked. Developers began saying “I am running the card” instead of “I am pushing the image”, which signalled to everyone nearby that a risky step was in progress. Colleagues learned not to interrupt. The phrase became a small ritual that carried real information.

The team also began to treat tags with more respect. Because the card forced them to say who relied on a given tag, they noticed that two services depended on “current” in ways nobody had documented. They moved to specific version tags for deployments and kept the shared tag for convenience only. That change came directly from the habit of reading line two aloud. The wider guidance on keeping a tidy registry is covered in this article on managing container images properly across an engineering team, which expands on what the team stumbled towards.

An objection from the sceptic on the team

Not everybody was convinced at the start. One engineer, who preferred to move fast and fix forward, argued that a card was a crutch for people who could not hold six things in their heads. The tech lead did not argue the point. She asked him to run the card once, out loud, on his next push and to tell the team what happened.

He came back a week later, slightly sheepish. On his push, line three had made him look at what the shared tag currently pointed to, and he had realised that another team member had pushed a hotfix to it an hour earlier. His build would have silently overwritten the fix. He did not say he had been wrong; he simply started reading the card before every push, and he was the one who suggested putting it into the script so nobody could forget. The sceptics on a team are often the strongest advocates once they have seen the card work on their own work.

Lessons worth borrowing

Write the card from your own history. The team did not start with a template. They started with what hurt. That gave the card legitimacy.

Keep it to the launch. The moment the list grew to cover reviews and testing, it would have become a process document that people skim.

Make saying it aloud a habit. Spoken words engage a different kind of attention than silent reading. Even when working alone, read the target name aloud.

Review regularly. Treat the card as a living thing. Add what hurt recently; drop what never matters.

Several of the mistakes in the story are of the type described in the roundup of frequent beginner errors when working with Docker, which is worth a skim if you are new to containers and want to know what to watch for.

The career angle

A small habit contributes more to a professional reputation than people tend to expect. The developer who made the original mistake became, within a few months, the person colleagues asked to run releases. She had not become more talented; she had become reliably careful, and everyone could see it. On a team of four, that visibility counts. For people thinking about their personal brand in the engineering world, being known as the person whose launches are calm is as useful as any certificate.

A modest public footprint helps too. Writing a short post about a card you built, or sharing a sanitised version of it, gives other engineers something useful and shows your approach. Keep it practical and honest, and avoid exaggerating what it prevented. Specific, modest claims earn more trust than dramatic ones.

A look at other communities

Many fields that deal with final actions end up in the same place. Readers who follow crypto may have met CryptoBlockCoins, a blog on cryptocurrency and blockchain that sorts its writing into coin posts, tutorials and daily logs, for an audience where mistakes can be permanent and care before acting is a recurring topic. The parallel holds even if the subject itself is not your thing.

The epilogue

Half a year later, the whiteboard photo had been replaced by a pinned message in the release channel, and the card now ran as a prompt in the push script. New joiners were handed it during their first week. The retrospective note from that Thursday was still in the shared folder, and the team kept it as a reminder of why the six lines exist.

The point most worth keeping is how narrow the card is. Do not try to write a process that prevents every error. Write a small card that guards the single step you cannot undo, build it from your own near misses and read it aloud every time. Start with your next push to a shared tag.

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x