https://citconf.com/wiki/api.php?action=feedcontributions&feedformat=atom&user=Treaz 2026-05-08T13:37:22Z User contributions MediaWiki 1.35.11 https://citconf.com/wiki/index.php?title=MDD_Monitoring_Driven_Development&diff=16635 2022-10-18T08:47:36Z

<p>Treaz: </p> <hr /> <div><br /> == From the Rubber Chicken to MDD ==<br /> <br /> @jtf&#039;s &quot;presentation&quot; <br /> <br /> # James Shore&#039;s Rubber Chicken <br /> ** physical token you had to get to commit (push) to main (it was svn back then), and you ran the build/tests before commit<br /> ** had to use a separate physical machine (solving the &#039;It works on my machine&#039; problem)<br /> # CI<br /> ** can run more stuff now (fast tests, slow tests) - but separate build for deploy<br /> # pipelines with artifact passing<br /> # promoting to test/prod<br /> # CD - blue green deploy - rolling back based on KPIs &#039;&#039;&#039;CI + monitoring now controls production&#039;&#039;&#039;<br /> ** if any step fails, the change is automatically to be reverted<br /> ** if it made to prod, but business metrics down<br /> *** not reverting code<br /> *** take out from production cluster to investigate<br /> <br /> == State of the monitoring (first) ==<br /> <br /> * metrics used in monitoring are not specific (high level business metric down, must have been this change)<br /> * just like adding tests after writing the code is hard, so is adding monitoring/metrics<br /> * who tried monitoring first?<br /> ** zsoldosp - checklist item in issue template, but too many issues it didn&#039;t apply, so it kinda got ignored after on that project<br /> ** PJ/intent media<br /> *** monitoring can stop deploy/rollout<br /> *** stopped doing acceptance tests in favor of monitoring<br /> ** aparker / TIM - failure analyses: we built it, now that we know how it works, let&#039;s figure out <br /> *** how could it fail<br /> *** what impact it would have<br /> *** how would we know (from customers? )<br /> *** it it worth adding it? (metric, alert)<br /> <br /> == alerting ==<br /> <br /> * how many alerts should we create<br /> ** high level? e.g.: number failed API requests?<br /> ** more specific - e.g.: we know it after debugging that it failed &#039;coz the middleware failed. Should we monitor the middleware?<br /> * metrics vs. monitoring <br /> ** monitoring triggers somene to look at it<br /> ** metrics - kinda like classic OPs - collect data, don&#039;t attach metrics, just eyeball &quot;looks to be an unusual shape, let&#039;s investigate&quot;<br /> * who should we call (e.g.: if only high level metrics, who should the alerts wake up?)<br /> * (pagerduty.com)<br /> <br /> == &quot;Failure Friday&quot; practice ==<br /> * during work hours!<br /> * we think this should be redundant, so let&#039;s shut this off and see the team recover<br /> * important: do it when you expect the exercise to be successful<br /> <br /> <br /> == Feature validation / AB testing ==<br /> <br /> not the same as monitoring<br /> <br /> <br /> == Alert thresholds ==<br /> <br /> * it&#039;s not always binary (on/off) <br /> * normal is not the same as yesterday/last week / last year<br /> ** seasonality - e.g.: black friday, but can be different for each industry. And you kinda know it &quot;Mondays are usually about this many pageloads&quot;<br /> ** event driven - e.g.: if you publish tips, it depends on what happens in the world<br /> * factor into<br /> ** what can we measure<br /> ** what should be alert on (i.e.: wake people up). Some things can wait till next business days - use different channels<br /> <br /> == Improving Alerts ==<br /> <br /> * make them actionable<br /> ** link to wiki of runbook how to fix<br /> ** write it for your future self who alerted at 2am at a party, not with your present knowledge of the context of the feature you just implemented<br /> * metrics you don&#039;t use is inventory, thus not useful<br /> <br /> (question: any logging frameworks that would only flush logs on exceptions? but then on DEBUG level?)<br /> <br /> * should we alert on causes (disk full) or symptoms (user can&#039;t login) (symptoms more useful? some tools allow dependencies, i.e.: if this is down, these others will be down too, don&#039;t alert on those)<br /> <br /> == Workshop on MDD - 2 minutes to dropped jaws ==<br /> <br /> Story: Given that currently our support lines are overwhelmed, if we added an FAQ about it, support calls would drop back to managable levels<br /> <br /> what can we measure?<br /> <br /> * nr of FAQ views<br /> * # of calls<br /> * ask support reps to ask if caller read the FAQ &amp; feed that back to the system?<br /> * instead of &quot;was this helpful&quot; &quot;yes/no&quot; maybe we could have &quot;yes/Call support (link/phone number)&quot; (talk to UX before doing this at home :-))<br /> <br /> =&gt; the way you think of validation/measuring changes the product<br /> <br /> == Monitoring Embedded into Business ==<br /> <br /> * SRE handbook only focuses on the tech<br /> * if decision makers use monitoring data, it&#039;s important for the business, thus no need to justify why monitoring<br /> <br /> == Links ==<br /> <br /> * My Philosophy on Alerting (based my observations while I was a Site Reliability Engineer at Google) by Rob Ewaschuk: https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit<br /> * Patrick Debois: Codifying devops practices: https://jedi.be/blog/2012/05/12/codifying-devops-area-practices/<br /> * Doing the impossible fifty times a day: http://timothyfitz.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/<br /> <br /> == good questions to ask: ==<br /> * what does this data mean?<br /> * If we are not wachting it -&gt; delete it? <br /> * Should we try &quot;Failure Friday&quot;?<br /> * Should we use &quot;Daily Red&quot;?<br /> * Is this indicator fast enough (leading or lagging indicator) to react?</div>

Treaz https://citconf.com/wiki/index.php?title=Self_care_part_2&diff=16613 2022-10-15T10:25:47Z

<p>Treaz: </p> <hr /> <div>Attendees:<br /> * [[Horia Constantin]]<br /> * [[Katri Ordning]]<br /> <br /> Practiced for 5 mins doing 4:6 breathing<br /> <br /> Practiced 1 session of Wim Hof Method breathing.<br /> <br /> Discussed the simple ritual of daily journaling of unpleasant thoughts.</div>

Treaz https://citconf.com/wiki/index.php?title=Self_care_part_2&diff=16612 2022-10-15T10:25:35Z

<p>Treaz: </p> <hr /> <div>Attendees:<br /> * [[Horia Constantin]]<br /> * [[Katri Ordning]]<br /> <br /> Practiced for 5 mins doing 4:6 breathing<br /> Practiced 1 session of Wim Hof Method breathing.<br /> Discussed the simple ritual of daily journaling of unpleasant thoughts.</div>

Treaz https://citconf.com/wiki/index.php?title=Self_care_part_2&diff=16611 2022-10-15T10:24:58Z

<p>Treaz: Created page with &quot;Attendees: * Horia Constantin * Kata Practiced for 5 mins doing 4:6 breathing Practiced 1 session of Wim Hof Method breathing. Discussed the simple ritual of daily jo...&quot;</p> <hr /> <div>Attendees:<br /> * [[Horia Constantin]]<br /> * [[Kata]]<br /> <br /> Practiced for 5 mins doing 4:6 breathing<br /> Practiced 1 session of Wim Hof Method breathing.<br /> Discussed the simple ritual of daily journaling of unpleasant thoughts.</div>

Treaz https://citconf.com/wiki/index.php?title=Horia_Constantin&diff=16600 2022-10-15T06:00:14Z

<p>Treaz: </p> <hr /> <div>https://horiaconstantin.com/<br /> <br /> https://mastodon.online/web/@treaz<br /> <br /> https://twitter.com/ConstantinHoria</div>

Treaz https://citconf.com/wiki/index.php?title=Horia_Constantin&diff=16564 2020-05-30T11:02:26Z

<p>Treaz: </p> <hr /> <div>https://horiaconstantin.com/<br /> <br /> https://twitter.com/ConstantinHoria</div>

Treaz