Difference between revisions of "Fragile Test"
From CitconWiki
Jump to navigationJump to search| Line 1: | Line 1: | ||
| − | + | = Less Fragile Tests = | |
| − | Tests | + | == Tests == |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | * Asserts | |
| − | + | ** Component | |
| − | + | ** Integration | |
| − | + | * Functional | |
| − | + | ** UAT | |
| − | + | * Non-Functional | |
| − | + | ** Perf | |
| − | + | ** Security | |
| − | + | ** Usability | |
| − | + | * See ISO 9126 | |
| − | |||
| − | |||
| − | |||
| − | + | == Good Tests == | |
| − | |||
| − | |||
| − | |||
| − | |||
| + | * Pass Repeatedly | ||
| + | ** Timing (e.g. Selenium) | ||
| + | ** External Dependencies | ||
| + | *** External Systems | ||
| + | **** Reliability | ||
| + | **** Availability | ||
| + | *** Data | ||
| + | **** Set Values | ||
| + | **** Incorrect State | ||
| + | ** Heavy Mocking | ||
| + | *** Utility to build mocks | ||
| + | *** Easy switching mock to real | ||
| − | + | == Bad Tests == | |
| − | + | * Hidden cause of failure | |
| − | + | ** Visible Replay using TimeSnapper | |
| − | + | * Visible Replay | |
| − | + | * Dump state data at failure | |
| − | Data | + | |
| − | + | == Tool advice == | |
| − | + | * Time Snapper | |
| − | + | * EasyMock | |
| − | + | * Pretend Classes | |
| + | * DBUnit gets prod snapshot regularly (security issues: scrubbing) Table dependancies makes it a significant task | ||
| + | |||
| + | == Data == | ||
| + | * Domain (name, addr, etc class pulled from DB | ||
| + | * (abstracted data from test) | ||
| + | * Data Driven Approach | ||
| + | * Fitness Integration Testing | ||
Strong correlation between importance of tests depending on the types of testing and what kind of thing you are doing (Product type development = high focus on test vs bespoke = low focus | Strong correlation between importance of tests depending on the types of testing and what kind of thing you are doing (Product type development = high focus on test vs bespoke = low focus | ||
| − | Suggestion | + | == Suggestion == |
| − | + | * Use a virtual machine, setup - refresh test machine using VMWARe to simulate MSG based, 20 mins business transactions | |
| − | Test Data Repository | + | == Test Data Repository == |
| − | + | * Used keyword substitution to create the real data | |
| − | Typical Data | + | == Typical Data == |
| − | + | * Edge / Border cases, Exception testing | |
| − | Broken windows | + | == Broken windows == |
| − | + | * Continually failing, low priority tests/defects cause a 'Broken windows' effect which degrades the value of CI. | |
| − | |||
Revision as of 21:37, 28 July 2007
Less Fragile Tests
Tests
- Asserts
- Component
- Integration
- Functional
- UAT
- Non-Functional
- Perf
- Security
- Usability
- See ISO 9126
Good Tests
- Pass Repeatedly
- Timing (e.g. Selenium)
- External Dependencies
- External Systems
- Reliability
- Availability
- Data
- Set Values
- Incorrect State
- External Systems
- Heavy Mocking
- Utility to build mocks
- Easy switching mock to real
Bad Tests
- Hidden cause of failure
- Visible Replay using TimeSnapper
- Visible Replay
- Dump state data at failure
Tool advice
- Time Snapper
- EasyMock
- Pretend Classes
- DBUnit gets prod snapshot regularly (security issues: scrubbing) Table dependancies makes it a significant task
Data
- Domain (name, addr, etc class pulled from DB
- (abstracted data from test)
- Data Driven Approach
- Fitness Integration Testing
Strong correlation between importance of tests depending on the types of testing and what kind of thing you are doing (Product type development = high focus on test vs bespoke = low focus
Suggestion
- Use a virtual machine, setup - refresh test machine using VMWARe to simulate MSG based, 20 mins business transactions
Test Data Repository
- Used keyword substitution to create the real data
Typical Data
- Edge / Border cases, Exception testing
Broken windows
- Continually failing, low priority tests/defects cause a 'Broken windows' effect which degrades the value of CI.
