WarStoriesAndSuccesses: Difference between revisions
From CitconWiki
Jump to navigationJump to search
No edit summary |
No edit summary |
||
| (One intermediate revision by the same user not shown) | |||
| Line 19: | Line 19: | ||
*** From Tony later: show other companies that are already past 3y to show what’s possible (and build trust) | *** From Tony later: show other companies that are already past 3y to show what’s possible (and build trust) | ||
* Tim: traditionally siloed functional org | * Tim: traditionally siloed functional org | ||
** Idea to cash was ~ 200d | |||
** 91% wait time | |||
** First response was to be mad at the person pointing at the problem | |||
** Improvement is slow, but it is improving | |||
** It was important to just get people talking about it even if they got upset initially | |||
** Value stream map made it all possible | |||
* Success: Operations team and developer teams | * Christophe - Success: Operations team and developer teams | ||
** Devs write run-books over a couple months, then throw it to the operators | |||
** Big bang release every couple of months | |||
** Build a more intimate relationship between Dev and Ops | |||
*** Moved to central Jenkins instance from individual snowflakes | |||
*** This integrated the dev workflow with the ops workflow (common environment) | |||
** Decoupled infra track from application track for deployments | |||
** Over a 6 month period — still monthly releases but far less stressful on everyone involved | |||
** Enjoyable pizza | |||
* Frederick: Goal: faster release cycle — get to weekly | * Frederick: Goal: faster release cycle — get to weekly | ||
** Current was monthly with bi-weekly patch releases | |||
** Visualized value stream maps with both | |||
** Took similarities and combined to 1 flow like the patch flow — currently at every 2w | |||
** Giant monolith | |||
** There were a lot of fears and resistance but enough trust was built to give it a try | |||
** Now trying to work towards TBD | |||
* Small web team gets security scan result | * Small web team gets security scan result | ||
** Pen testers test quarterly — first report was very ugly | |||
** Introduced PR process, but devs weren’t able to spot security vulnerabilities | |||
** Secure Code Warrior gamefies this .. points for vulnerabilities found | |||
** Got some training (e.g. OWASP) and testers were more involved | |||
** Making progress | |||
** “Why should I train you (freelancer), what if you leave" | |||
*** “What if you don’t train us and we stay” (response) | |||
* Failure: | * Jonathon - Failure: | ||
** Cultural, hierarchical issues with a Dev silo, QA silo (which moved closer) and DevOps (eww) silo which was moved even further away | |||
** Improvements were made in delivering more often, but then there was some realization/questioning of this and so deliveries are now getting slowed/spaced out | |||
Latest revision as of 04:15, 18 May 2019
Test strategies and flow and sharing of strategies
- Test pipelines and integration strategies across teams
- Architecture
- It’s so important to assume you will miss something .. so MTTR is critical
War stories
- Defeat: why can’t anyone deploy anything
- Victory: 3 months to change a web page —> changes in 1 day
- Text link to 404 for the purpose of recording clicks of interest for house viewing
- PJ: Company daily release process but often would break and often take 3 people most of the day to get it out
- Eventually, every time someone made a change of the code it would be deployed to prod and monitored
- End to end test bankruptcy
- Success
- Monolithic system (with seams but tight coupling) was broken down into component based independently deployable systems
- Be prepared, this process will take 3 years (or more)
- How do you sell this idea?
- The key to our success is shortening our feedback cycles
- From Tony later: show other companies that are already past 3y to show what’s possible (and build trust)
- Eventually, every time someone made a change of the code it would be deployed to prod and monitored
- Tim: traditionally siloed functional org
- Idea to cash was ~ 200d
- 91% wait time
- First response was to be mad at the person pointing at the problem
- Improvement is slow, but it is improving
- It was important to just get people talking about it even if they got upset initially
- Value stream map made it all possible
- Christophe - Success: Operations team and developer teams
- Devs write run-books over a couple months, then throw it to the operators
- Big bang release every couple of months
- Build a more intimate relationship between Dev and Ops
- Moved to central Jenkins instance from individual snowflakes
- This integrated the dev workflow with the ops workflow (common environment)
- Decoupled infra track from application track for deployments
- Over a 6 month period — still monthly releases but far less stressful on everyone involved
- Enjoyable pizza
- Frederick: Goal: faster release cycle — get to weekly
- Current was monthly with bi-weekly patch releases
- Visualized value stream maps with both
- Took similarities and combined to 1 flow like the patch flow — currently at every 2w
- Giant monolith
- There were a lot of fears and resistance but enough trust was built to give it a try
- Now trying to work towards TBD
- Small web team gets security scan result
- Pen testers test quarterly — first report was very ugly
- Introduced PR process, but devs weren’t able to spot security vulnerabilities
- Secure Code Warrior gamefies this .. points for vulnerabilities found
- Got some training (e.g. OWASP) and testers were more involved
- Making progress
- “Why should I train you (freelancer), what if you leave"
- “What if you don’t train us and we stay” (response)
- Jonathon - Failure:
- Cultural, hierarchical issues with a Dev silo, QA silo (which moved closer) and DevOps (eww) silo which was moved even further away
- Improvements were made in delivering more often, but then there was some realization/questioning of this and so deliveries are now getting slowed/spaced out
