CitconWiki - User contributions [en]

Mark Waite

2008-04-10T16:39:38Z

MarkEWaite: New page: Mark Waite is a software development manager at [http://www.ptc.com Parametric Technology Corporation], working in the Fort Collins, CO office. Prior to working for PTC, Mark worked for [...

Mark Waite is a software development manager at [http://www.ptc.com Parametric Technology Corporation], working in the Fort Collins, CO office. Prior to working for PTC, Mark worked for [http://www.cocreate.com CoCreate Software]. CoCreate was purchased by PTC in November 2007.

Mark is interested in software testing, software development, and in agile techniques. He was the team manager when CoCreate switched to Extreme Programming in March 2003 (XP by 3/03). His [http://blog.360.yahoo.com/markwaite business blog] usually contains notes about things learned while managing, testing, or programming. His [http://markwaite.blogspot.com/ personal blog] may become the new home for business information, since Yahoo (the home for the business blog) has stopped development and fixes on Yahoo 360.

Functional tests take a long time

2008-04-06T23:20:49Z

MarkEWaite:

Long Running Functional Tests

These are the few notes I took from the "long running functional tests" discussions.

'''Problems'''

* Complete features take 1 day
* Functional test takes 15 hours

'''Alternatives, Risks, and Trade-offs'''

* Parallel testing vs. pipelined testing
** Compile
** Fast unit tests
** Slow unit tests
** Functional tests

* Incremental feedback during test runs
** Show failures sooner, but
** Does not typically lead to stopping the tests because we want to know all the results from that set of code

Functional tests take a long time

2008-04-06T23:20:25Z

MarkEWaite: New page: Long Running Functional Tests These are the few notes I took from the "long running functional tests" discussions. Problems * Complete features take 1 day * Functional test takes 15 hou...

Long Running Functional Tests

These are the few notes I took from the "long running functional tests" discussions.

Problems

* Complete features take 1 day
* Functional test takes 15 hours

Alternatives, Risks, and Trade-offs

* Parallel testing vs. pipelined testing
** Compile
** Fast unit tests
** Slow unit tests
** Functional tests

* Incremental feedback during test runs
** Show failures sooner, but
** Does not typically lead to stopping the tests because we want to know all the results from that set of code

CITCONDenver2008Sessions

2008-04-06T23:18:59Z

MarkEWaite: /* 11:15 Topics */

CITCON Denver Sessions

== 10:00 Topics ==

#[[What's an agile tester?]]
#[[Scaling continuous integration to the enterprise]]

== 11:15 Topics ==

#[[Making builds faster, more efficient]]
#[[Functional tests take a long time]]

Scaling continuous integration to the enterprise

2008-04-06T23:17:42Z

MarkEWaite: /* Enterprise Scale Continuous Integration */

== Enterprise Scale Continuous Integration ==

These were the notes captured from the discussions. We started by describing the problems we had seen in trying to run continuous integration on very large code bases, and in large organizations which are coming from monthly or weekly integration cycles to continuous integration cycles.

'''Problem Definition'''

* 300 devs, 1 build break per year per developer means:
** build will be broken every day
** slows us down
** distrust of the source master
** various defensive behaviors so programmers can get work done even though the source master is frequently broken
*** Only sync from the master when it is "known good"
*** Private branches to insulate from noise on the master
*** Project branches to insulate from noise on the master

* 2 hour build, 3 day acceptance test
** A 2 hour build means that the time between a submit and feedback on the result of the build could be as much as 4 hours (I just missed the start of the current build, my build will start in almost two hours and will require two hours to confirm success or failure)

* hard to assign failure due to multiple commits per build (many developers may submit during the 2 hour build window, so it becomes harder to diagnose which submit caused a build failure)

* long cycle time on failure (hours before you know you broke something)

* failures affect more people, are more expensive

* Understand root cause of failure is not obvious

* How to handle 300 applications, each with a few devs, how to scale to many projects and still manage level

* How to manage many branches to many mains

* Managing build time dependencies (unexpected, undetected coupling)
** incorrect incremental builds

'''Addressing the problems, alternatives, risks, and trade-offs'''

* subcomponents
** reduces build time, but
** increases integration time
* build acceleration technology
** parallel build, multi-machine, multi-core (Electric Accelerator, for instance)
** bug fast machines (although disc I/O may dominate)
* modularize to get recent successful build, not compile
** faster, less built (narrow the impact to a smaller team)
* Use "pre-flight" build (production build with many changes, not yet on the source master)
** integration race conditions
** faster hardware
** parallel builds

'''Alternatives (2)'''

* 3 day acceptance test
** throw bodies at the problem (but it is not scalable)
** review the acceptance process for automation opportunities
** increase automated testing inside the application (at the interfaces)
** modularize tests, make them independent so they can run in parallel
** accept human tests less frequently, automation running continuously
** use assistive automation to support more effective exploratory testing
*** Brian Marick has some work going in this area
*** Michael Bolton describes his use of Watir as assistive automation

Lisa Crispin suggested that Jared Richardson had done the continuous
integration work for SAS and might share insights and ideas.

'''Alternatives (3)'''

300 applications, small teams on each

* Either many independent CI systems or an enterprise CI system
** unified view
** shared configuration
** reuse between teams
** security
** usable for small teams

* Dependency management
** component level dependencies managed by tools
*** Anthill / Codestation
*** maven
*** ivy
** scheduling builds, which build should be run first
** how do I express the rules by which I select a component
*** version (specific version, pattern match a version, relational operator to version string, etc.)
*** acceptance test results

Scaling continuous integration to the enterprise

2008-04-06T23:13:56Z

MarkEWaite: /* Enterprise Scale Continuous Integration */

== Enterprise Scale Continuous Integration ==

These were the notes captured from the discussions. We started by describing the problems we had seen in trying to run continuous integration on very large code bases, and in large organizations which are coming from monthly or weekly integration cycles to continuous integration cycles.

'''Problem Definition'''

* 300 devs, 1 build break / year
** build will be broken every day
** slows us down
** creates distrust of the source master

* 2 hour build, 3 day acceptance test

* hard to assign failure due to multiple commits per build

* long cycle time on failure (hours before you know you broke something)

* failures affect more people, are more expensive

* Understand root cause of failure is not obvious

* How to handle 300 applications, each with a few devs, how to scale to many projects and still manage level

* How to manage many branches to many mains

* Managing build time dependencies (unexpected, undetected coupling)
** incorrect incremental builds

'''Addressing the problems, alternatives, risks, and trade-offs'''

* subcomponents
** reduces build time, but
** increases integration time
* build acceleration technology
** parallel build, multi-machine, multi-core (Electric Accelerator, for instance)
** bug fast machines (although disc I/O may dominate)
* modularize to get recent successful build, not compile
** faster, less built (narrow the impact to a smaller team)
* Use "pre-flight" build (production build with many changes, not yet on the source master)
** integration race conditions
** faster hardware
** parallel builds

'''Alternatives (2)'''

* 3 day acceptance test
** throw bodies at the problem (but it is not scalable)
** review the acceptance process for automation opportunities
** increase automated testing inside the application (at the interfaces)
** modularize tests, make them independent so they can run in parallel
** accept human tests less frequently, automation running continuously
** use assistive automation to support more effective exploratory testing
*** Brian Marick has some work going in this area
*** Michael Bolton describes his use of Watir as assistive automation

Lisa Crispin suggested that Jared Richardson had done the continuous
integration work for SAS and might share insights and ideas.

'''Alternatives (3)'''

300 applications, small teams on each

* Either many independent CI systems or an enterprise CI system
** unified view
** shared configuration
** reuse between teams
** security
** usable for small teams

* Dependency management
** component level dependencies managed by tools
*** Anthill / Codestation
*** maven
*** ivy
** scheduling builds, which build should be run first
** how do I express the rules by which I select a component
*** version (specific version, pattern match a version, relational operator to version string, etc.)
*** acceptance test results

Scaling continuous integration to the enterprise

2008-04-06T23:11:10Z

MarkEWaite:

== Enterprise Scale Continuous Integration ==

'''Problem Definition'''

* 300 devs, 1 build break / year
** build will be broken every day
** slows us down
** creates distrust of the source master

* 2 hour build, 3 day acceptance test

* hard to assign failure due to multiple commits per build

* long cycle time on failure (hours before you know you broke something)

* failures affect more people, are more expensive

* Understand root cause of failure is not obvious

* How to handle 300 applications, each with a few devs, how to scale to many projects and still manage level

* How to manage many branches to many mains

* Managing build time dependencies (unexpected, undetected coupling)
** incorrect incremental builds

'''Addressing the problems, alternatives, risks, and trade-offs'''

* subcomponents
** reduces build time, but
** increases integration time
* build acceleration technology
** parallel build, multi-machine, multi-core (Electric Accelerator, for instance)
** bug fast machines (although disc I/O may dominate)
* modularize to get recent successful build, not compile
** faster, less built (narrow the impact to a smaller team)
* Use "pre-flight" build (production build with many changes, not yet on the source master)
** integration race conditions
** faster hardware
** parallel builds

'''Alternatives (2)'''

* 3 day acceptance test
** throw bodies at the problem (but it is not scalable)
** review the acceptance process for automation opportunities
** increase automated testing inside the application (at the interfaces)
** modularize tests, make them independent so they can run in parallel
** accept human tests less frequently, automation running continuously
** use assistive automation to support more effective exploratory testing
*** Brian Marick has some work going in this area
*** Michael Bolton describes his use of Watir as assistive automation

Lisa Crispin suggested that Jared Richardson had done the continuous
integration work for SAS and might share insights and ideas.

'''Alternatives (3)'''

300 applications, small teams on each

* Either many independent CI systems or an enterprise CI system
** unified view
** shared configuration
** reuse between teams
** security
** usable for small teams

* Dependency management
** component level dependencies managed by tools
*** Anthill / Codestation
*** maven
*** ivy
** scheduling builds, which build should be run first
** how do I express the rules by which I select a component
*** version (specific version, pattern match a version, relational operator to version string, etc.)
*** acceptance test results

Scaling continuous integration to the enterprise

2008-04-06T23:07:16Z

MarkEWaite: Notes from the enterprise scale conitnuous integration discussions

== Enterprise Scale Continuous Integration ==

'''Problem Definition'''

- 300 devs, 1 build break / year
= build will be broken every day
= slows us down
= creates distrust of the source master

- 2 hour build, 3 day acceptance test

- hard to assign failure due to multiple commits per build

- long cycle time on failure (hours before you know you broke something)

- failures affect more people, are more expensive

- Understand root cause of failure is not obvious

- How to handle 300 applications, each with a few devs, how to scale to many projects and still manage level

- How to manage many branches to many mains

- Managing build time dependencies (unexpected, undetected coupling)
= incorrect incremental builds

Addressing the problems, alternatives, risks, and trade-offs

- subcomponents
= reduces build time, but
= increases integration time
- build acceleration technology
= parallel build, multi-machine, multi-core (Electric Accelerator, for instance)
= bug fast machines (although disc I/O may dominate)
- modularize to get recent successful build, not compile
= faster, less built (narrow the impact to a smaller team)
- Use "pre-flight" build (production build with many changes, not yet on the source master)
= integration race conditions
= faster hardware
= parallel builds

Alternatives (2)

- 3 day acceptance test
= throw bodies at the problem (but it is not scalable)
= review the acceptance process for automation opportunities
= increase automated testing inside the application (at the interfaces)
= modularize tests, make them independent so they can run in parallel
= accept human tests less frequently, automation running continuously
= use assistive automation to support more effective exploratory testing
* Brian Marick has some work going in this area
* Michael Bolton describes his use of Watir as assistive automation

Lisa Crispin suggested that Jared Richardson had done the continuous
integration work for SAS and might share insights and ideas.

Alternatives (3)

300 applications, small teams on each

- Either many independent CI systems or an enterprise CI system
= unified view
= shared configuration
= reuse between teams
= security
= usable for small teams

- Dependency management
= component level dependencies managed by tools
* Anthill / Codestation
* maven
* ivy
= scheduling builds, which build should be run first
= how do I express the rules by which I select a component
* version (specific version, pattern match a version, relational operator to version string, etc.)
* acceptance test results