{"id":3120,"date":"2026-11-16T06:40:00","date_gmt":"2026-11-16T06:40:00","guid":{"rendered":"https:\/\/www.rajeshkumar.xyz\/blog\/?p=3120"},"modified":"2026-11-18T15:40:00","modified_gmt":"2026-11-18T15:40:00","slug":"itops-story","status":"publish","type":"post","link":"https:\/\/www.rajeshkumar.xyz\/blog\/itops-story\/","title":{"rendered":"The Week a Fast Delivery Team Discovered ITOps"},"content":{"rendered":"<p>Here is a team that never existed but that you have probably worked beside. Four engineers at a small online learning company, call them the platform crew, shipped container images to production several times a day and were proud of it. Releases were fast, dashboards were green and morale was high, right up to the Tuesday when a perfectly ordinary deployment took the checkout page down for forty minutes during a promotion. Everything that followed is a story about ITOps, told the way I would tell it over coffee.<\/p>\n<p>For vocabulary I lean on xOps School&#8217;s Ops Atlas, where <a href=\"https:\/\/www.xopsschool.com\/itops\/\">the ITOps page<\/a> frames the work as keeping IT services up, including the less glamorous half: requests, repeat-cause removal, change control and knowing what you own. The crew had mastered delivery and skipped most of that half.<\/p>\n<h2>Tuesday: forty minutes of silence<\/h2>\n<p>The deployment itself was fine. What broke was a database schema tweak that rode along with it, scheduled by a different team the same afternoon. Neither group knew the other&#8217;s plan. The promotion email had gone out an hour earlier, so traffic was at its peak, and the checkout page returned errors until someone rolled back by hand.<\/p>\n<p>In the postmortem, the crew&#8217;s lead (I will keep her nameless; she hates attention) said something that changed the team: &#8220;We have a pipeline, but we do not have a calendar.&#8221; Two changes, each reasonable alone, had collided. The Atlas argues for one forward calendar that flags overlaps, with an example where a payroll freeze window blocks anything colliding with it. The crew started with a shared sheet and a rule: every planned change goes on the sheet before it goes anywhere else.<\/p>\n<h2>Wednesday: the argument about approvals<\/h2>\n<p>Management&#8217;s first instinct was to demand sign-off from a committee for every release. The crew pushed back, and they were right to. The Atlas reports that years of delivery research turned up nothing tying outside sign-off to fewer failed changes, and points toward peer review. It describes the board&#8217;s role migrating away from gatekeeping and toward shaping the process and sharing information.<\/p>\n<p>But the crew also learned that the story has a twist. The Atlas also points out that EU financial firms face rules demanding approvers who are independent, plus retrospective sign-off on emergency changes. Their company was not a bank, so peer review stood, but they wrote down the exception and moved on. If you are reading this at a firm that is regulated, ask your compliance team which side you are on; treat this as general information rather than advice.<\/p>\n<p>The compromise was elegant in its smallness. Routine releases using the usual image build, a recipe they had run hundreds of times, became standard changes that needed no meeting. Anything touching the database or the payment flow went to a short huddle. And emergencies could proceed immediately, with a review the next day.<\/p>\n<h2>Thursday: the pipeline learns manners<\/h2>\n<p>The crew&#8217;s pipeline already raised a ticket for each deployment. The Atlas has a firm opinion here: pipelines may create change records but should never approve them. So the crew changed one setting. The job still files the record automatically, but approval comes from a human reviewer or a written policy rule, never from the job itself.<\/p>\n<p>They also added a tripwire. A routine recipe that slipped twice within its latest twenty runs lost its pre-approval until someone looked at why. This idea, taken from the Atlas examples of change models with failure limits, caught a flaky image-build step within a fortnight. Our walkthrough of <a href=\"https:\/\/www.rajeshkumar.xyz\/blog\/master-docker-image-management-best-practices-for-engineering-teams\/\">managing container images across an engineering team<\/a> covers the build side of the same issue.<\/p>\n<h2>Friday: scoring risk without drama<\/h2>\n<p>On Friday afternoon, the crew sat down to write a score. Three factors: how critical the system is, whether a data schema changes and whether the work lands outside the usual window. They gave the factors weights and chose a total that triggers review. In the Atlas worked example, a change scoring 6 from its three components went to the board. The crew&#8217;s numbers were different, but the habit was the same: argue about the weights once, then let the sheet decide.<\/p>\n<p>The results were modest and real. Review meetings dropped from daily to twice a week. Nobody missed the others.<\/p>\n<h2>The following month: failures get a second look<\/h2>\n<p>The next piece was a habit rather than a tool. Every failed change and every emergency change got a ten-minute review within two days, with one question: what would have made this boring? The Atlas lists that review as part of change mechanics, and for good reason. Emergencies are where hidden weaknesses live.<\/p>\n<p>The crew also noticed they were drifting toward a classic trap. After a bad release, someone proposed a seven-step checklist for every deployment. The lead vetoed it, citing the Atlas anti-pattern of adding process in response to failure. Instead they fixed the specific gap the postmortem had found, the missing calendar, and left everything else alone.<\/p>\n<h2>Month three: the register nobody wanted<\/h2>\n<p>Then came the dull chore. A customer asked which services stood between their data and the internet, and the crew spent a day assembling the answer from memory. Afterward they built a register: service, owner, dependencies, renewal dates and last-confirmed date. It started with the nine services that mattered and stopped there.<\/p>\n<p>The Atlas warns that registers decay when they try to contain everything and recommends holding what decisions require. Its ten-item health check, with a couple of stale rows, an orphan and a repeated serial, reads uncomfortably like a typical first audit. The crew&#8217;s first audit found a retired service still listed as live, which prompted an overdue cleanup. If you are new to container work and want to avoid the beginner pitfalls that make registers messy, <a href=\"https:\/\/www.rajeshkumar.xyz\/blog\/common-docker-mistakes-beginners-should-avoid\/\">this list of common container mistakes for newcomers<\/a> is a good companion.<\/p>\n<h2>Month four: alerts that raise their hand<\/h2>\n<p>Finally, the crew asked how they knew when something was wrong. The answer was a patchwork of dashboards and an alert rule set in an open-source monitoring stack. When a certificate expired quietly, no alert fired, because the alerting component had been misconfigured weeks earlier. The Atlas calls out exactly this: if the event layer is treated as a non-production dependency, a silent failure hides outages.<\/p>\n<p>The fix was cheap. The alerting pipeline got an owner, entered the change calendar and began sending a daily test message to the on-call channel. A missing message is now itself an alarm.<\/p>\n<h2>What changed in the way they talked<\/h2>\n<p>The subtlest shift was linguistic. Before the bad Tuesday, the crew said &#8220;ship it&#8221; and &#8220;roll it back&#8221;. Afterwards they said &#8220;is it standard or normal?&#8221;, &#8220;what is its score?&#8221; and &#8220;who owns the register row?&#8221;. Those phrases sound bureaucratic until you hear how much argument they replace. A new engineer joining in month five picked up the routine within a week, simply by reading the sheet and listening to the Friday review.<\/p>\n<p>One more detail: they never adopted a large ticketing platform. The Atlas is clear that a tool records the practice, and the crew&#8217;s practice fitted on a shared board and two spreadsheets. They may well move to a proper tool later; if they do, the habits will transfer, and the tool will only record what already happens.<\/p>\n<h2>What the crew would tell you<\/h2>\n<ul>\n<li>A pipeline is not a process. Fast delivery still needs a calendar, a score and a review habit.<\/li>\n<li>Skip heavyweight service management for a four-person team; borrow only what fixes an observed pain.<\/li>\n<li>Let machines file change records and let people or written rules approve them.<\/li>\n<li>Keep the register small and confirmed.<\/li>\n<li>Test your alarms as if they were production, because they are.<\/li>\n<\/ul>\n<p>Six months on, the crew ships just as often, and the promotion days are pleasantly boring. If you are building a DevOps career, the lesson I would underline is that operations skills are what make you trusted with bigger systems. Anyone can make deployments fast; the people who also make them uneventful get the interesting jobs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A fictional four-person platform crew learns, over one rough week and the months after, why fast delivery still needs calendars, change scores and tested alarms.<\/p>\n","protected":false},"author":6,"featured_media":3121,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[713],"tags":[],"series":[],"class_list":["post-3120","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops-careers"],"_links":{"self":[{"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/posts\/3120","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/comments?post=3120"}],"version-history":[{"count":1,"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/posts\/3120\/revisions"}],"predecessor-version":[{"id":3243,"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/posts\/3120\/revisions\/3243"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/media\/3121"}],"wp:attachment":[{"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/media?parent=3120"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/categories?post=3120"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/tags?post=3120"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/www.rajeshkumar.xyz\/blog\/wp-json\/wp\/v2\/series?post=3120"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}