The Week a Fast Delivery Team Discovered ITOps

Posted by

–

Here is a team that never existed but that you have probably worked beside. Four engineers at a small online learning company, call them the platform crew, shipped container images to production several times a day and were proud of it. Releases were fast, dashboards were green and morale was high, right up to the Tuesday when a perfectly ordinary deployment took the checkout page down for forty minutes during a promotion. Everything that followed is a story about ITOps, told the way I would tell it over coffee.

For vocabulary I lean on xOps School’s Ops Atlas, where the ITOps page frames the work as keeping IT services up, including the less glamorous half: requests, repeat-cause removal, change control and knowing what you own. The crew had mastered delivery and skipped most of that half.

Tuesday: forty minutes of silence

The deployment itself was fine. What broke was a database schema tweak that rode along with it, scheduled by a different team the same afternoon. Neither group knew the other’s plan. The promotion email had gone out an hour earlier, so traffic was at its peak, and the checkout page returned errors until someone rolled back by hand.

In the postmortem, the crew’s lead (I will keep her nameless; she hates attention) said something that changed the team: “We have a pipeline, but we do not have a calendar.” Two changes, each reasonable alone, had collided. The Atlas argues for one forward calendar that flags overlaps, with an example where a payroll freeze window blocks anything colliding with it. The crew started with a shared sheet and a rule: every planned change goes on the sheet before it goes anywhere else.

Wednesday: the argument about approvals

Management’s first instinct was to demand sign-off from a committee for every release. The crew pushed back, and they were right to. The Atlas reports that years of delivery research turned up nothing tying outside sign-off to fewer failed changes, and points toward peer review. It describes the board’s role migrating away from gatekeeping and toward shaping the process and sharing information.

But the crew also learned that the story has a twist. The Atlas also points out that EU financial firms face rules demanding approvers who are independent, plus retrospective sign-off on emergency changes. Their company was not a bank, so peer review stood, but they wrote down the exception and moved on. If you are reading this at a firm that is regulated, ask your compliance team which side you are on; treat this as general information rather than advice.

The compromise was elegant in its smallness. Routine releases using the usual image build, a recipe they had run hundreds of times, became standard changes that needed no meeting. Anything touching the database or the payment flow went to a short huddle. And emergencies could proceed immediately, with a review the next day.

Thursday: the pipeline learns manners

The crew’s pipeline already raised a ticket for each deployment. The Atlas has a firm opinion here: pipelines may create change records but should never approve them. So the crew changed one setting. The job still files the record automatically, but approval comes from a human reviewer or a written policy rule, never from the job itself.

They also added a tripwire. A routine recipe that slipped twice within its latest twenty runs lost its pre-approval until someone looked at why. This idea, taken from the Atlas examples of change models with failure limits, caught a flaky image-build step within a fortnight. Our walkthrough of managing container images across an engineering team covers the build side of the same issue.

Friday: scoring risk without drama

On Friday afternoon, the crew sat down to write a score. Three factors: how critical the system is, whether a data schema changes and whether the work lands outside the usual window. They gave the factors weights and chose a total that triggers review. In the Atlas worked example, a change scoring 6 from its three components went to the board. The crew’s numbers were different, but the habit was the same: argue about the weights once, then let the sheet decide.

The results were modest and real. Review meetings dropped from daily to twice a week. Nobody missed the others.

The following month: failures get a second look

The next piece was a habit rather than a tool. Every failed change and every emergency change got a ten-minute review within two days, with one question: what would have made this boring? The Atlas lists that review as part of change mechanics, and for good reason. Emergencies are where hidden weaknesses live.

The crew also noticed they were drifting toward a classic trap. After a bad release, someone proposed a seven-step checklist for every deployment. The lead vetoed it, citing the Atlas anti-pattern of adding process in response to failure. Instead they fixed the specific gap the postmortem had found, the missing calendar, and left everything else alone.

Month three: the register nobody wanted

Then came the dull chore. A customer asked which services stood between their data and the internet, and the crew spent a day assembling the answer from memory. Afterward they built a register: service, owner, dependencies, renewal dates and last-confirmed date. It started with the nine services that mattered and stopped there.

The Atlas warns that registers decay when they try to contain everything and recommends holding what decisions require. Its ten-item health check, with a couple of stale rows, an orphan and a repeated serial, reads uncomfortably like a typical first audit. The crew’s first audit found a retired service still listed as live, which prompted an overdue cleanup. If you are new to container work and want to avoid the beginner pitfalls that make registers messy, this list of common container mistakes for newcomers is a good companion.

Month four: alerts that raise their hand

Finally, the crew asked how they knew when something was wrong. The answer was a patchwork of dashboards and an alert rule set in an open-source monitoring stack. When a certificate expired quietly, no alert fired, because the alerting component had been misconfigured weeks earlier. The Atlas calls out exactly this: if the event layer is treated as a non-production dependency, a silent failure hides outages.

The fix was cheap. The alerting pipeline got an owner, entered the change calendar and began sending a daily test message to the on-call channel. A missing message is now itself an alarm.

What changed in the way they talked

The subtlest shift was linguistic. Before the bad Tuesday, the crew said “ship it” and “roll it back”. Afterwards they said “is it standard or normal?”, “what is its score?” and “who owns the register row?”. Those phrases sound bureaucratic until you hear how much argument they replace. A new engineer joining in month five picked up the routine within a week, simply by reading the sheet and listening to the Friday review.

One more detail: they never adopted a large ticketing platform. The Atlas is clear that a tool records the practice, and the crew’s practice fitted on a shared board and two spreadsheets. They may well move to a proper tool later; if they do, the habits will transfer, and the tool will only record what already happens.

What the crew would tell you

  • A pipeline is not a process. Fast delivery still needs a calendar, a score and a review habit.
  • Skip heavyweight service management for a four-person team; borrow only what fixes an observed pain.
  • Let machines file change records and let people or written rules approve them.
  • Keep the register small and confirmed.
  • Test your alarms as if they were production, because they are.

Six months on, the crew ships just as often, and the promotion days are pleasantly boring. If you are building a DevOps career, the lesson I would underline is that operations skills are what make you trusted with bigger systems. Anyone can make deployments fast; the people who also make them uneventful get the interesting jobs.

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x