https://citconf.com/wiki/index.php?title=Using_Machine_Learning_To_Solve_Test_Passed_Site_Down_Problem&feed=atom&action=historyUsing Machine Learning To Solve Test Passed Site Down Problem - Revision history2024-03-28T15:00:23ZRevision history for this page on the wikiMediaWiki 1.35.11https://citconf.com/wiki/index.php?title=Using_Machine_Learning_To_Solve_Test_Passed_Site_Down_Problem&diff=16121&oldid=prevMagnuslassi: Created page with "Attendees: <br> PJ <br> Markus <br> Brett <br> Jeeva <br> Magnus Lassi <br> Erik <br> Dan <br> Girish <br> Magnus Stahre <br> Joe Bishop <br> Paul Duvall <br> Problem stateme..."2015-10-04T17:34:11Z<p>Created page with "Attendees: <br> PJ <br> Markus <br> Brett <br> Jeeva <br> Magnus Lassi <br> Erik <br> Dan <br> Girish <br> Magnus Stahre <br> Joe Bishop <br> Paul Duvall <br> Problem stateme..."</p>
<p><b>New page</b></p><div>Attendees: <br><br />
PJ <br><br />
Markus <br><br />
Brett <br><br />
Jeeva <br><br />
Magnus Lassi <br><br />
Erik <br><br />
Dan <br><br />
Girish <br><br />
Magnus Stahre <br><br />
Joe Bishop <br><br />
Paul Duvall <br><br />
<br />
Problem statement: Worked at pre-prod time fails in production - same as production<br />
Cause: Data in production starts changing<br />
<br />
Question: Does the problem statement assume you’re using Continous Delivery (CD)?<br />
PJ: No, just that it was a certified artifact deployed to production.<br />
<br />
Possible solutions:<br />
- Use AI to dynamically evolve pre-prod verification<br />
<br />
Real world examples (PJ): We had an interface contract regarding user id’s that we received from a 3rd party that the user id’s would be numeric. In production we started receiving user id’s that were alphanumeric which was unexpected.<br />
<br />
Another real world example: We had agree on a JSON structure where the id’s sent would be 6 digits but they sent 8 digits in production.<br />
[ {PJ}, {DAWN} ]<br />
[ {id: “125799846”} ]<br />
<br />
We had also agreed on the user names being US alphanumerics but we received some names with umlauts and breaks the system:<br />
[ {PJ}, {DAWN}, {Hänk} ]<br />
<br />
In a world where the data is constantly evolving, machine learning would be beneficial. It would be usable in a dynamic test suite to get the test data updated.<br />
<br />
If you rely on existing data, you restrict yourself. You don’t allow the system to evolve.<br />
<br />
If you allow each new message to let the system learn, no one on the team needs to know.<br />
<br />
- Totally doable with existing data<br />
- to evolve need a feedback loop<br />
<br />
Could use a batch process to look at data in production and feed it to test systems.<br />
We know we failed a request because we logged it somewhere.<br />
<br />
Why haven’t we evolved from automated tests to an evolving feedback loop? Why don’t we automatically update the test suite (when failure occurs due to unexpected data in production)?<br />
The gap is that the automated test suite isn’t updated automatically. It seems reasonable that using AI / machine learning to solve this problem.<br />
<br />
Reason’s we don’t do this today:<br />
- Hard to know what to change<br />
- Lack of knowledge about this possibility<br />
- Lack of trust that it will do what we want it to do<br />
- If we can understand the change then we can test better<br />
- Risk of lack of trust in test suite : false negatives / false positives, who’s testing the tester<br />
<br />
No matter what requests the systems receive it should handle it gracefully. Returning an error is acceptable, crashing the system is not.<br />
<br />
Take every input for the last hour -> expected output -> non-system-crash<br />
JSON -> dynamic -> non-system crash<br />
<br />
If we can understand the change, we can do forward looking testing before the future data gets there.<br />
<br />
In some systems the answer isn’t known until runtime. A good way to unit test these cases is to assert a given range.<br />
<br />
mutation testing - modify the production code, your unit tests should fail. We’re testing our tests.<br />
<br />
Big potential challenge to get organization to accept prod dat to flow down automatically to test systems.<br />
<br />
Does anyone feel using AI is a waste of time for this?<br />
Consensus is no but challenging to implement. It can be a potential political challenge to convince peers and management.<br />
<br />
It could be as easy as: take a feed of data + known post conditions + feed it to an AI model<br />
<br />
The hypothesis is that is shouldn’t be a large task versus manually doing this work with people.<br />
<br />
No-one gave a reason NOT to try machine learning to this problem.<br />
<br />
Tools:<br />
- datatomic: http://cognitect.com/datomic<br />
- quick check (property testing tool) https://en.wikipedia.org/wiki/QuickCheck<br />
- Amazon machine learning (they have industry algorithms, you supply data)<br />
- Spock<br />
- Jester (mutation testing tool) http://programmers.stackexchange.com/questions/189939/is-there-a-modern-replacement-for-a-mutation-testing-tool-like-jester-for-java<br />
- F#</div>Magnuslassi