Test Tools The Next Generation - Revision history

PaulJulius: Formatting.

2015-09-12T08:12:37Z

Formatting.

PaulJulius: Formatting

2015-09-12T08:07:21Z

Formatting

PaulJulius: Formatting.

2015-09-12T08:04:46Z

Formatting.

PaulJulius: Notes taken by PJ during the initial session

2015-09-12T08:02:55Z

Notes taken by PJ during the initial session

New page

CITCON 2015 Europe in Helsinki

Session 1 - Auditorium

Testing Tools of the Future - Proposed by Gojko Adzic

Given how cheap resources (machines, people, etc), what can we do now that we could not do before.

Existing Examples:
https://www.usertesting.com/

https://aws.amazon.com/device-farm/

Reminder of Jason Huggins's

* Why not just do production monitoring, like canary testing?
- It's a matter of risk

Third interesting trend: Webkits really grown up

Gojko uses Phantom for DOM testing all the time.

Wraith, from BBC, is a testing tool that runs the test in lots of browsers and then diff the visuals

Gatling, another tool that was very active

James Shore released Quixote, CSS Unit Testing, tool

Gauge was released by ThoughtWorks, as replacement for Twist

----------------
Classes of Problems:
----------------
* Fragmentation
- Issues of testing across multiple devices
* Resilience
- Issues related to the ephemeral nature of the infrastructure
* Costs
** of Execution
** of Maintenance
* Infrastructure
- How to test the infrastructure
----------------

* What about something that enables getting end users involved?
- Session recording?
- Prototyping?
- Exploration toolkits
- Paper prototyping
- Goal: Get feedback from the customer as soon as possible to guide your development

* Mechanical Turk, how much can you control it?

*****
PJ
*****
Proposing "AI Testing", using Machine Learning
because the greatest cost in testing is maintaining a legacy test suite.
*****

Mark Streibeck from Google on Testing
- If the test fails, and you have to change the test, then it is a BAD test
- If the test fails and you change the code, the it is a GOOD test

Microsoft Reverse Code Coverage tool?
To speed up feedback, by running

A/B Testing can be more difficult in Finance applications

One Company that Gojko knows of wrote a test, but as soon as it passes, they deleted the test

False positives, test designs

Is a "test suite" a _tall_ factory?

Pascal community used mathematical modeling

Google captures a list of system attributes/capabilities

Unit testing: if the unit test is hard to write, then that indicates a code smell

References
=================
Quixote, CSS Unit Testing
Galen Framework
Wraith, from BBC
Device Farm from AWS - https://aws.amazon.com/device-farm/
UserTesting.com
Gauge
Concordian
Applitools
Visual Review
Domreactor
Hotjar
Text Test - for testing log files

@@ Line 47: / Line 47: @@
 * What about something that enables getting end users involved?
-  - Session recording?
+** Session recording?
-  - Prototyping?
+** Prototyping?
-  - Exploration toolkits
+** Exploration toolkits
-  - Paper prototyping
+** Paper prototyping
-  - Goal: Get feedback from the customer as soon as possible to guide your development
+** Goal: Get feedback from the customer as soon as possible to guide your development
 * Mechanical Turk, how much can you control it?
-*****
-PJ
+PJ Proposing "AI Testing", using Machine Learning because the greatest cost in testing is maintaining a legacy test suite.
-*****
 Mark Streibeck from Google on Testing
-- If the test fails, and you have to change the test, then it is a BAD test
+* If the test fails, and you have to change the test, then it is a BAD test
-- If the test fails and you change the code, the it is a GOOD test
+* If the test fails and you change the code, the it is a GOOD test
 Microsoft Reverse Code Coverage tool?

@@ Line 39: / Line 39: @@
 ** Issues related to the ephemeral nature of the infrastructure
 * Costs
-  ** of Execution
+** of Execution
-  ** of Maintenance
+** of Maintenance
 * Infrastructure
-  ** How to test the infrastructure
+** How to test the infrastructure
 ----------------

@@ Line 17: / Line 17: @@
 * Why not just do production monitoring, like canary testing?
-  - It's a matter of risk
+** It's a matter of risk
 Third interesting trend: Webkits really grown up
@@ Line 35: / Line 35: @@
 ----------------
 * Fragmentation
-  - Issues of testing across multiple devices
+** Issues of testing across multiple devices
 * Resilience
-  - Issues related to the ephemeral nature of the infrastructure
+** Issues related to the ephemeral nature of the infrastructure
 * Costs
    ** of Execution
    ** of Maintenance
 * Infrastructure
-   - How to test the infrastructure
+   ** How to test the infrastructure
 ----------------