Failure stories that became success or not (QA, DevOps, ...)

From CitconWiki
Jump to navigationJump to search

Journey from failure to success

Three stages of the journey described, from v1.0 to v3.0


  • Started new job
  • Day 3 - noticed no log of test cases. No one knew what they were doing. Not even starting from nothing - starting from -5!.
  • No DevOps, no DB team. Most infrastructure on AWS. All manual (testing, deployment)
  • 1 monolithic app (no microservices)
  • Deployment - all manual processes. 3-4 hrs.
  • Step-by-step written down -> reduced downtime.
  • Move from excel for test cases
  • Project lasts 6 months - 1yr
  • 6 people. 0 to 5000 test cases in 1.5 years. (manual)
    • someone mentioned coverage is more important than quantity
  • Talked about automation for 1 year but didn't have time.
  • QA team was always referred to as a bottleneck - got the sense that this was derogatory.


  • Introduced Infrastructure as code. This took deployment time down from 3 hours to 5-10mins for deployment
  • Moved from monolith to microservices
  • Introduced Jenkins
  • Matrix of compatibility for product components

Someone asked if replaced one problem with another?

    • Yes, but the problem was smaller and better!


  • Containerised apps
  • zero downtime
  • devops -> platform engineering
  • Jenkins -> Argo
  • Deliver faster, more reliable
  • Product owner doesn't care how frequently deployed (1/month, 1/week, 5/day etc.)- as long as a project is delivered in 1 year for example

Additional info

  • The journey from v1.0 to v3.0 took 6 years
  • Asked if any advice for a department to move them from v1.0 to v3.0:
    • Befriend someone in top management to push ideas faster
    • Remove those with high performance, low loyalty from the company - Low performance and loyal is preferable.