Fragile Test

Less Fragile Tests

Tests

• Asserts
  • Component
  • Integration
• Functional
  • UAT
• Non-Functional
  • Perf
  • Security
  • Usability
• See ISO 9126

Good Tests

• Pass Repeatedly
  • Timing (e.g. Selenium)
  • External Dependencies
    • External Systems
      • Reliability
      • Availability
    • Data
      • Set Values
      • Incorrect State
  • Heavy Mocking
    • Utility to build mocks
    • Easy switching mock to real
      • use same data, settings

Bad Tests

• Hidden cause of failure
  • Visible Replay using TimeSnapper
• Visible Replay
• Dump state data at failure

Tool advice

• Time Snapper
• EasyMock
• Pretend Classes
• DBUnit gets prod snapshot regularly (security issues: scrubbing) Table dependancies makes it a significant task

Data

• Domain (name, addr, etc class pulled from DB
• (abstracted data from test)
• Data Driven Approach
• Fitness Integration Testing

Strong correlation between importance of tests depending on the types of testing and what kind of thing you are doing (Product type development = high focus on test vs bespoke = low focus

Suggestion

• Use a virtual machine, setup - refresh test machine using VMWARe to simulate MSG based, 20 mins business transactions

Test Data Repository

• Used keyword substitution to create the real data

Typical Data

• Edge / Border cases, Exception testing

Broken windows

• Continually failing, low priority tests/defects cause a 'Broken windows' effect which degrades the value of CI.