The company and people in this case study are invented, though release engineers will probably recognise the situation from their own jobs. It follows a platform team at a mid-sized online retailer whose deployment process included a single manual security sign-off before every production release. The sign-off was meant to keep releases safe. Instead it created a queue, encouraged rubber-stamping and still let real problems through. What follows is how the team replaced that one gate with small checks distributed across the deployment pipeline, and how the engineer who led the change turned the experience into a turning point in her career.
The organisation and the original process
The company, which we will call Kirana Cart, ran roughly forty services across web, mobile and warehouse systems. Releases moved through a pipeline that built container images, ran automated tests and deployed to staging. Before any change reached production, however, a member of the three-person security team had to review a release ticket and click “approve”.
On paper, this gave the organisation a clear control point. In practice, the security team received dozens of release tickets a week. Each ticket summarised the change in a paragraph and linked to a diff. Reviewers had minutes per ticket. They checked for obvious red flags, approved, and moved on.
Shalini Rao, a senior platform engineer, had been at Kirana Cart for three years. She had built much of the deployment pipeline herself, and she was increasingly uneasy about how the sign-off actually worked.
The incident that forced a rethink
The turning point came with a routine release of the order-tracking service. The change included an upgraded image-processing library pulled in to generate shipment thumbnails. The release ticket described it as “minor dependency update plus UI tweak”. It was approved within ten minutes.
Two weeks later, an external researcher reported that the library version carried a publicly known vulnerability allowing crafted image files to trigger code execution. The fix itself was straightforward: upgrade again and redeploy. But the investigation revealed three uncomfortable facts.
- The vulnerability had been public before the release. Any automated dependency check would have flagged it at build time.
- The reviewer could not reasonably have caught it. Nothing in a one-paragraph ticket indicated risk, and manually cross-checking every library version against vulnerability databases was impossible at that volume.
- The same library had been pulled into two other services in the meantime, both also approved through the same gate.
The sign-off had functioned exactly as designed. It simply was not capable of catching this class of problem, at this point in the process.
Diagnosing the real weakness
Shalini wrote a short internal analysis. Its central argument was precise: a single human gate at the end of the pipeline was being asked to detect problems that had entered much earlier, at specific, identifiable steps. The dependency had entered at build. Earlier incidents in the company’s history had entered at commit, through a credential accidentally included in a configuration file, and at deployment, through a container running with broader permissions than needed.
In each case, the relevant information existed at the step where the problem began and had largely disappeared by the time a reviewer saw the release ticket. The gate was positioned where the context was thinnest.
Her proposal was not to remove security oversight, but to move most of it to where context was richest, and to automate it there.
The redesign: checks distributed across the pipeline
Over the following quarter, the platform and security teams rebuilt the release process around stage-level controls. Three changes did most of the work.
At commit: secrets blocked automatically
Every repository gained a pre-receive hook that scanned incoming changes for credentials and rejected pushes containing them. Existing secrets were moved into HashiCorp Vault, and services were updated to retrieve credentials at runtime. Developers received the feedback in their own terminal, seconds after the mistake, while they still had full context.
At build: dependencies and images scanned
The build stage began scanning both application dependencies and resulting container images with Trivy. Builds failed on critical known vulnerabilities and produced warnings for lower severities. Every build also generated a software bill of materials, stored alongside the image. When the next widely publicised library vulnerability appeared, the team identified every affected service in minutes by searching those records, rather than grepping repositories by hand. Shalini’s earlier article on managing container images well across engineering teams was updated to describe the scanning step.
Before deployment: a policy gate replaced the manual click
The manual sign-off was retired for standard changes and replaced with an automated policy check. It verified that the build had passed its scans, that the image was signed, and that deployment manifests did not request privileged access. Changes touching payment flows or authentication still required a human security review, but those reviews were now a handful per week rather than dozens, and reviewers had the scan results in front of them. The overall approach drew on the principles in designing safe deployment pipelines for business applications.
Results after six months
The outcomes were measured qualitatively and through incident records rather than any invented headline figure, but they were clear.
- The release queue disappeared. Standard changes deployed as soon as their checks passed.
- Human reviews became meaningful. Security engineers spent their time on high-risk changes with full context, rather than skimming tickets.
- Late-discovered vulnerabilities dropped noticeably. Most dependency issues were now caught and resolved at build, by the developer who introduced them.
- Rollbacks became rarer and calmer. When they were needed, the team’s existing rollback approach for continuous deployment worked better because changes were smaller and better documented.
Perhaps the most telling change was cultural. Developers stopped viewing security as an obstacle at the end and started seeing it as feedback built into the work they were already doing.
What did not go smoothly
A precise account should include the friction. The first fortnight of build-stage scanning failed a large share of builds, because years of unreviewed dependencies suddenly became visible at once. Developers were frustrated, and some asked for the scanner to be switched off. The team responded by running in warning-only mode for a short period, triaging the backlog service by service, and only then switching critical findings to blocking. The security team also had to accept a loss of perceived control: retiring the manual click felt, to some, like giving up oversight. Shalini addressed this by showing that every automated decision was logged with its evidence, which actually gave the security team more visibility than the old tickets had.
The career dimension
For Shalini, the project changed how others in the industry saw her. She had been known internally as a capable pipeline engineer. After the redesign, she became the person colleagues consulted about secure delivery.
She was deliberate about building on that. She wrote up the case, carefully anonymised, as a conference talk and a series of blog posts. She framed her experience precisely: not “I improved security” but “I moved security controls from a single manual gate to the pipeline stages where problems originate, and here is what changed.” That specificity made her story credible to hiring managers and peers.
She also formalised her knowledge. Having learned much of the discipline through experience, she wanted structured grounding in areas she had touched less, such as threat modeling and policy-as-code at scale. She enrolled with DevSecOpsSchool, a DevSecOps training and certification provider, whose curriculum is built on placing a security control inside each stage of the delivery lifecycle. She chose the Certified DevSecOps Architect track, whose focus on organisation-wide security design, running workloads across several clouds, encoding rules as policy and zero-trust thinking matched the direction she wanted her career to take.
Within a year she had moved into a principal engineer role focused on platform security.
Lessons for engineers building their careers
The technical lessons of the Kirana Cart story are well established: controls work best at the step where risk enters, and a single end-stage gate is positioned where context is weakest. The career lessons are worth stating just as clearly.
- Look for processes that rely on one late check. They are common, they fail predictably, and fixing them is visible, high-value work.
- Diagnose before proposing. Shalini’s analysis connected specific past incidents to specific pipeline steps. That precision persuaded leadership.
- Keep humans where judgement matters. Automating routine checks is not about removing people; it is about giving them the right work.
- Tell the story specifically. In talks, posts and interviews, describe what you moved, where, and what changed. Vague claims about “improving security” carry little weight.
- Fill your gaps deliberately. Practical experience combined with structured learning builds a profile that is both credible and well-rounded.
Closing note
Kirana Cart is invented, but the pattern is real and widespread: a manual approval at the end of a pipeline, carrying the full weight of security, and quietly failing to bear it. Teams that redistribute that weight into small, automated checks at each stage tend to ship faster and more safely. Engineers who lead that change, and who can explain it precisely, tend to find their careers moving in a new direction too.
