WarStoriesAndSuccesses

Test strategies and flow and sharing of strategies

• Test pipelines and integration strategies across teams

• Architecture
• It’s so important to assume you will miss something .. so MTTR is critical

War stories

• Defeat: why can’t anyone deploy anything
• Victory: 3 months to change a web page —> changes in 1 day
  • Text link to 404 for the purpose of recording clicks of interest for house viewing
• PJ: Company daily release process but often would break and often take 3 people most of the day to get it out
  • Eventually, every time someone made a change of the code it would be deployed to prod and monitored
    • End to end test bankruptcy
  • Success
  • Monolithic system (with seams but tight coupling) was broken down into component based independently deployable systems
  • Be prepared, this process will take 3 years (or more)
  • How do you sell this idea?
    • The key to our success is shortening our feedback cycles
    • From Tony later: show other companies that are already past 3y to show what’s possible (and build trust)
• Tim: traditionally siloed functional org

   * Idea to cash was ~ 200d
   * 91% wait time
   * First response was to be mad at the person pointing at the problem
   * Improvement is slow, but it is improving
   * It was important to just get people talking about it even if they got upset initially
   * Value stream map made it all possible

• Success: Operations team and developer teams

   * Devs write run-books over a couple months, then throw it to the operators
   * Big bang release every couple of months
   * Build a more intimate relationship between Dev and Ops
       * Moved to central Jenkins instance from individual snowflakes
       * This integrated the dev workflow with the ops workflow (common environment)
   * Decoupled infra track from application track for deployments
   * Over a 6 month period — still monthly releases but far less stressful on everyone involved
   * Enjoyable pizza

• Frederick: Goal: faster release cycle — get to weekly

   * Current was monthly with bi-weekly patch releases
   * Visualized value stream maps with both
   * Took similarities and combined to 1 flow like the patch flow — currently at every 2w
   * Giant monolith
   * There were a lot of fears and resistance but enough trust was built to give it a try
   * Now trying to work towards TBD

• Small web team gets security scan result

   * Pen testers test quarterly — first report was very ugly
   * Introduced PR process, but devs weren’t able to spot security vulnerabilities 
   * Secure Code Warrior gamefies this .. points for vulnerabilities found
   * Got some training (e.g. OWASP) and testers were more involved
   * Making progress
   * “Why should I train you (freelancer), what if you leave"
       * “What if you don’t train us and we stay” (response)

• Failure:

   * Cultural, hierarchical issues with a Dev silo, QA silo (which moved closer) and DevOps (eww) silo which was moved even further away
   * Improvements were made in delivering more often, but then there was some realization/questioning of this and so deliveries are now getting slowed/spaced out