CitconWiki - User contributions [en]

Virtualisation Services

2014-03-03T22:50:11Z

Paulflewelling: /* Testing Coverage */

== Service Virtualisation ==

Current tools on the market:

IBM Green Hat, CA LISA

Definition of Service Virtualisation: System need to develop or test against many other (external/internal) interfaces.

Allow for integration testing by providing Artifical Intelligent Stubs which need to be trained.

What we do is we use simulations/virtualisations to create stubs for those services.

Not running on actual system, very fast to setup.

'''''CASE STUDY'''''
Royal Bank of Scotland have successfully implemented SV in CI

'''BUSINESS NEEDS'''
* Consolidate 4 messaging hubs, deliver 35 major business change projects within 3 years.
* Enables payment/confirmation messages, clearing services, supporting regulatory services.
* Needs to cater fro a new currency if Greece kicked out of the Euro.

'''TECHNICAL ENV'''
* Waterfall
* Contractors/Vendors doing different things
* Manual testing (3 week regression)
* Complex env in and out (a lot of external 3rd parties)
* Regression tests expensive against actual 3rd party services
* Manual deployment

'''PROBLEMS'''
* Not meeting business needs
* External systems not available (functionality not developed)
* Expensive testing
* Expensive deployment

'''STRATEGIES'''
* Delay integration until services available - too long
* Write own stub - maintenance involved in keeping up-to-date slow and expensive
* Evaluated SV tools

Selected IBM Green Hat because of complex message support.

Greenhat tool essentially a way of setting up stubs for complex messages.

SOAP/XML/JMS/SWIFT/MQ/etc

Greenhat is a simulator/intelligent service

Allows them to use a cyclical build/test env pipeline to production. Using Service Virtualisation (SV) for the first two environments, DEV and SIT. Actual services are used for UAT and PROD.

How do you confirm that you virtualization is covering all tests it needs. And is this the same as the actual service?

The suggestion is that this is done with collaboration of the 3rd party. Have to complete explicit audit tests and baselining, i.e. tests against the real service and the actual service and compare the results.

It's all just speculation and collaboration if the actual service doesn't exist.

Level of risk against each service in your pipeline allows you to determine how much effort you put into virtualization.

The suggestion is that if you have faith in the virtual service then you can skip UAT.

For RBS this has significantly sped up their testing because don't have to necessarily wait for UAT env.

Automated testing reduced regression testing from 3 weeks to 4 hours.

[Some discussion over the differences between automatic deploy and potential automatic deploy.]

[Then some discussion over 4 hour regression test not being viable in dev and how to handle it, i.e. use targeted testing for the part of the app that you're working on]

[Now some discussion about how to version the virtualisations]

The suggestion is that you create different virtualisations for different versions unless the version is inherent in the message that you are sending, i.e. unless the version is part of the message.

Nowe some discussion over version control and rolling back versions, this is a problem for the team to manage and is down to their workflow.

Royal Bank of Scotland (RBS) are using automatic configuration as well as automatic deployment.

'''RESULTS'''
* Deliver change more quickly and frequently
* Reduce incidents and defect costs reduce over time
* UAT/Pre-prod minimised
* Increase in test efficiency (coverage / time taken)

==== Testing Coverage ====

Can be determined through impact analysis on changes by comparison of schema's on services and requirements/functional spec traceability with tests.

Also overtime confidence will be gained on defects not being found in production.

Virtualisation Services

2014-03-03T22:48:09Z

Paulflewelling: /* Service Virtualisation */

== Service Virtualisation ==

Current tools on the market:

IBM Green Hat, CA LISA

Definition of Service Virtualisation: System need to develop or test against many other (external/internal) interfaces.

Allow for integration testing by providing Artifical Intelligent Stubs which need to be trained.

What we do is we use simulations/virtualisations to create stubs for those services.

Not running on actual system, very fast to setup.

'''''CASE STUDY'''''
Royal Bank of Scotland have successfully implemented SV in CI

'''BUSINESS NEEDS'''
* Consolidate 4 messaging hubs, deliver 35 major business change projects within 3 years.
* Enables payment/confirmation messages, clearing services, supporting regulatory services.
* Needs to cater fro a new currency if Greece kicked out of the Euro.

'''TECHNICAL ENV'''
* Waterfall
* Contractors/Vendors doing different things
* Manual testing (3 week regression)
* Complex env in and out (a lot of external 3rd parties)
* Regression tests expensive against actual 3rd party services
* Manual deployment

'''PROBLEMS'''
* Not meeting business needs
* External systems not available (functionality not developed)
* Expensive testing
* Expensive deployment

'''STRATEGIES'''
* Delay integration until services available - too long
* Write own stub - maintenance involved in keeping up-to-date slow and expensive
* Evaluated SV tools

Selected IBM Green Hat because of complex message support.

Greenhat tool essentially a way of setting up stubs for complex messages.

SOAP/XML/JMS/SWIFT/MQ/etc

Greenhat is a simulator/intelligent service

Allows them to use a cyclical build/test env pipeline to production. Using Service Virtualisation (SV) for the first two environments, DEV and SIT. Actual services are used for UAT and PROD.

How do you confirm that you virtualization is covering all tests it needs. And is this the same as the actual service?

The suggestion is that this is done with collaboration of the 3rd party. Have to complete explicit audit tests and baselining, i.e. tests against the real service and the actual service and compare the results.

It's all just speculation and collaboration if the actual service doesn't exist.

Level of risk against each service in your pipeline allows you to determine how much effort you put into virtualization.

The suggestion is that if you have faith in the virtual service then you can skip UAT.

For RBS this has significantly sped up their testing because don't have to necessarily wait for UAT env.

Automated testing reduced regression testing from 3 weeks to 4 hours.

[Some discussion over the differences between automatic deploy and potential automatic deploy.]

[Then some discussion over 4 hour regression test not being viable in dev and how to handle it, i.e. use targeted testing for the part of the app that you're working on]

[Now some discussion about how to version the virtualisations]

The suggestion is that you create different virtualisations for different versions unless the version is inherent in the message that you are sending, i.e. unless the version is part of the message.

Nowe some discussion over version control and rolling back versions, this is a problem for the team to manage and is down to their workflow.

Royal Bank of Scotland (RBS) are using automatic configuration as well as automatic deployment.

'''RESULTS'''
* Deliver change more quickly and frequently
* Reduce incidents and defect costs reduce over time
* UAT/Pre-prod minimised
* Increase in test efficiency (coverage / time taken)

== Testing Coverage ==

Can be determined through impact analysis on changes by comparison of schema's on services and requirements/functional spec traceability with tests.

Also overtime confidence will be gained on defects not being found in production.

Virtualisation Services

2014-03-03T04:50:47Z

Paulflewelling: /* Service Virtualisation */

== Service Virtualisation ==

Current tools on the market:

IBM Green Hat, CA LISA

Definition of Service Virtualisation: System need to develop or test against many other (external/internal) interfaces.

Allow for integration testing by providing Artifical Intelligent Stubs which need to be trained.

What we do is we use simulations/virtualisations to create stubs for those services.

Not running on actual system, very fast to setup.

'''''CASE STUDY'''''
Royal Bank of Scotland have successfully implemented SV in CI

'''BUSINESS NEEDS'''
* Consolidate 4 messaging hubs, deliver 35 major business change projects within 3 years.
* Enables payment/confirmation messages, clearing services, supporting regulatory services.
* Needs to cater fro a new currency if Greece kicked out of the Euro.

'''TECHNICAL ENV'''
* Waterfall
* Contractors/Vendors doing different things
* Manual testing (3 week regression)
* Complex env in and out (a lot of external 3rd parties)
* Regression tests expensive against actual 3rd party services
* Manual deployment

'''PROBLEMS'''
* Not meeting business needs
* External systems not available (functionality not developed)
* Expensive testing
* Expensive deployment

'''STRATEGIES'''
* Delay integration until services available - too long
* Write own stub - maintenance involved in keeping up-to-date slow and expensive
* Evaluated SV tools

Selected IBM Green Hat because of complex message support.

Greenhat tool essentially a way of setting up stubs for complex messages.

SOAP/XML/JMS/SWIFT/MQ/etc

Greenhat is a simulator/intelligent service

Allows them to use a cyclical build/test env pipeline to production. Using Service Virtualisation (SV) for the first two environments, DEV and SIT. Actual services are used for UAT and PROD.

How do you confirm that you virtualization is covering all tests it needs. And is this the same as the actual service?

The suggestion is that this is done with collaboration of the 3rd party. Have to complete explicit audit tests and baselining, i.e. tests against the real service and the actual service and compare the results.

It's all just speculation and collaboration if the actual service doesn't exist.

Level of risk against each service in your pipeline allows you to determine how much effort you put into virtualization.

The suggestion is that if you have faith in the virtual service then you can skip UAT.

For RBS this has significantly sped up their testing because don't have to necessarily wait for UAT env.

Automated testing reduced regression testing from 3 weeks to 4 hours.

[Some discussion over the differences between automatic deploy and potential automatic deploy.]

[Then some discussion over 4 hour regression test not being viable in dev and how to handle it, i.e. use targeted testing for the part of the app that you're working on]

[Now some discussion about how to version the virtualisations]

The suggestion is that you create different virtualisations for different versions unless the version is inherent in the message that you are sending, i.e. unless the version is part of the message.

Nowe some discussion over version control and rolling back versions, this is a problem for the team to manage and is down to their workflow.

Royal Bank of Scotland (RBS) are using automatic configuration as well as automatic deployment.

RESULTS
* Deliver change more quickly and frequently
* Reduce incidents and defect costs reduce over time
* UAT/Pre-prod minimised
* Increase in test efficiency (coverage / time taken)

== Testing Coverage ==

Can be determined through impact analysis on changes by comparison of schema's on services and requirements/functional spec traceability with tests.

Also overtime confidence will be gained on defects not being found in production.

CITCONANZ2014Sessions

2014-03-03T04:10:52Z

Paulflewelling: /* 3:15 Topics */

CITCON ANZ 2014 Auckland Sessions

Back to the [[Main Page]]

== 10:00 Topics ==

# [[Virtualisation Services]]
# [[CI & TDD for legacy Systems]]
# [[CI in thirty minutes]]
# [[Reusing validation tests as monitoring]]
# [[Mobile Testing]]

== 11:15 Topics ==

# [[Vagrant/Packer for Continuous Delivery of Application Infrastructure]]
# [[?]]
# [[?]]
# [[?]]
# [[Tips and tricks for CI and CD]]

== Lunch Topics ==

# [[?]]

== 2:00 Topics ==

# [[How do you change the team culture from waterfall to shepherding change to production + Communication model]]
# [[?]]
# [[Managing Software Technical Debt]]
# [[Test Execution Time and Running Web-UI Tests in Parallel]]
# [[?]]

== 3:15 Topics ==

# [[Continuous Deployment - Why and How]]
# [[?]]
# [[?]]
# [[?]]
# [[?]]

== 4:30 Topics ==

# [[?]]
# [[?]]
# [[Deployment Contracts]]
# [[Beyond Given-When-Then]]
# [[?]]

== Table View ==

{| class="wikitable"
|-
! Room name
! 10:00
! 11:15
! 2:00
! 3:15
! 4:30
|-
| Cube
| [[?]]
| [[?]]
| [[?]]
| [[?]]
| [[?]]
|-
| Cube Bar
| [[?]]
| [[?]]
| [[?]]
| [[?]]
| [[?]]
|-
| Cube Hall
| [[CI in thirty minutes]]
| [[?]]
| [[?]]
| [[?]]
| [[?]]
|-
| Oku Wairangi
| [[?]]
| [[?]]
| [[Test Execution Time and Running Web-UI Tests in Parallel]]
| [[?]]
| [[Beyond Given-When-Then]]
|-
| Yellow Circle
| [[?]]
| [[Tips and tricks for CI and CD]]
| [[?]]
| [[?]]
| [[?]]
|}

Continuous Deployment - Why and How

2014-03-03T04:10:00Z

Paulflewelling:

Are you doing Continuous Deployment?
How are you doing it?

Define continuous: automated deployment process through all your envs through to production deployment.

3/4 doing continuous deployment

What's the motivation?

'''''1st Example (from group)'''''
1 of our projects configured to deploy continuously. Allows us to iterate faster, speed up delivery, be more Agile.

Potential benefits for us to deliver overall better quality of product

'''''2nd Example (from group)'''''
Only to development environment.

As soon as it's check into master it's deployed.

(This is Continuous Integration)

Any reason why you don't deploy to production? The tester has to give the ok before deploy to production.

'''''3rd Example'''''
Increase the value of what we're delivering.

Envs were a bit of a mess before we started continuous deployment.

Made our environment management much better.

'''What's preventing other people?'''

* Database migrations
* Client requiring notifications of outages
* Don't have tests
* Managing the media and making announcements

Feature flags can be used to managed the latter, for example Java Togglz http://www.togglz.org

Other people have managed using feature flags. A/B testing very similar to feature flags.

Use UI toggles to manage if features are on or off.

If we don't manage using feature flags, then we have features outside of the main branch that have to be maintained.

Virtualisation services are one way of managing the complexity you've got with turning on/off features in the test envs when you've to manage connections to things like SWIFT (Society for Worldwide Interbank Financial Telecommunication) http://www.swift.com/index.page?lang=en

If you need to change code in production to switch on a feature, does that mean you have to raise a change request? (takes 5 days)

Depends if large organisation and need to have change request. Some orgs just need to know what happened, i.e. they have a record.

Don't need a change request for a feature toggle.

Feature toggles are used in Production so that different groups can test different features.

Another comment: Testers are the people that hate feature toggles the most. Which combination of feature toggles was it that cause the particular bug?

Also the little things that catch you with features toggles, i.e. I told you months ago to turn that toggle on.

'''Challenges with Continuous Deployment'''

* A lot of investment at the beginning
* Haven't even started down the road of functional testing
* It's only going to be good as the testing that we do
* How to manage data migrations in production, have to invest some effort into how we do that, i.e. multiple steps.
* Who manages the continuous delivery? Developers? Testers? DevOps?

We picked some of the biggest monolithic systems, if I had a choice I would have selected one of our web apps. Instead we chose one of our core banking systems with mainframe, OracleDB, etc, etc. Essentially we selected the system with the greatest strategic importance and the one where we would get our arses kicked by our competitors.

Of the 4 things that you've done are you trying to make it a similar process for the next project that you change?

Some of the code we're dealing with is very old and so are the people that manage it. People change is the biggest challenge.

Basically we're making a conveyor belt so that each project gets shipped in the same way. Unfortunately for some people it is still down to the people who are on the project.

Manage multiple different projects, java, python, PHP in the same way.

How do you get the infrastructure guys to care about this? Well I'm the head of the infrastructure team, so I care about it. The see was planted long ago, moved to Puppet for CM 3 years ago. It's been a long steady process.

Another team using Chef now deploy to Amazon. Takes an hour to configure.

Some team used a bottom up approach, do it in their spare time, etc. IBM used a top down approach.

Others have a R&D time allocated each week.

Management buy in is important to success, otherwise things will get in the way.

Continuous Deployment - Why and How

2014-03-03T04:07:56Z

Paulflewelling: Created page with "Are you doing Continuous Deployment? How are you doing it? Define continuous: automated deployment process through all your envs through to production deployment. 3/4 doing ..."

Are you doing Continuous Deployment?
How are you doing it?

Define continuous: automated deployment process through all your envs through to production deployment.

3/4 doing continuous deployment

What's the motivation?

'''''1st Example (from group)'''''
1 of our projects configured to deploy continuously. Allows us to iterate faster, speed up delivery, be more Agile.

Potential benefits for us to deliver overall better quality of product

'''''2nd Example (from group)'''''
Only to development environment.

As soon as it's check into master it's deployed.

(This is Continuous Integration)

Any reason why you don't deploy to production? The tester has to give the ok before deploy to production.

'''''3rd Example'''''
Increase the value of what we're delivering.

Envs were a bit of a mess before we started continuous deployment.

Made our environment management much better.

What's preventing other people?
* Database migrations
* Client requiring notifications of outages
* Don't have tests
* Managing the media and making announcements

Feature flags can be used to managed the latter, for example Java Togglz http://www.togglz.org

Other people have managed using feature flags. A/B testing very similar to feature flags.

Use UI toggles to manage if features are on or off.

If we don't manage using feature flags, then we have features outside of the main branch that have to be maintained.

Virtualisation services are one way of managing the complexity you've got with turning on/off features in the test envs when you've to manage connections to things like SWIFT (Society for Worldwide Interbank Financial Telecommunication) http://www.swift.com/index.page?lang=en

If you need to change code in production to switch on a feature, does that mean you have to raise a change request? (takes 5 days)

Depends if large organisation and need to have change request. Some orgs just need to know what happened, i.e. they have a record.

Don't need a change request for a feature toggle.

Feature toggles are used in Production so that different groups can test different features.

Another comment: Testers are the people that hate feature toggles the most. Which combination of feature toggles was it that cause the particular bug?

Also the little things that catch you with features toggles, i.e. I told you months ago to turn that toggle on.

Challenges with Continuous Deployment
* A lot of investment at the beginning
* Haven't even started down the road of functional testing
* It's only going to be good as the testing that we do
* How to manage data migrations in production, have to invest some effort into how we do that, i.e. multiple steps.
* Who manages the continuous delivery? Developers? Testers? DevOps?

We picked some of the biggest monolithic systems, if I had a choice I would have selected one of our web apps. Instead we chose one of our core banking systems with mainframe, OracleDB, etc, etc. Essentially we selected the system with the greatest strategic importance and the one where we would get our arses kicked by our competitors.

Of the 4 things that you've done are you trying to make it a similar process for the next project that you change?

Some of the code we're dealing with is very old and so are the people that manage it. People change is the biggest challenge.

Basically we're making a conveyor belt so that each project gets shipped in the same way. Unfortunately for some people it is still down to the people who are on the project.

Manage multiple different projects, java, python, PHP in the same way.

How do you get the infrastructure guys to care about this? Well I'm the head of the infrastructure team, so I care about it. The see was planted long ago, moved to Puppet for CM 3 years ago. It's been a long steady process.

Another team using Chef now deploy to Amazon. Takes an hour to configure.

Some team used a bottom up approach, do it in their spare time, etc. IBM used a top down approach.

Others have a R&D time allocated each week.

Management buy in is important to success, otherwise things will get in the way.

Managing Software Technical Debt

2014-03-03T02:16:21Z

Paulflewelling: Created page with "Ways to determine * Complexity / Class Usage can be used to determine where to look (bug hunting missions) * Fuck metric, the more the number of fucks to a comment, the more s..."

Ways to determine
* Complexity / Class Usage can be used to determine where to look (bug hunting missions)
* Fuck metric, the more the number of fucks to a comment, the more serious the problem
* Source code analysis, manually (peer review) and automatically (tools like code climate)
* Annotate the code, for example https://code.google.com/p/gag/
* http://www.techdebt.org

Discussion of technical debt talked about different debts
* Hacks you were asked for
* Hacks you did to get things done

Gource is one way to analyse code.

There are also tools like Code Climate.

There was a mention of Rich Hickey's talk/presentation "Simple Made Easy": http://www.infoq.com/presentations/Simple-Made-Easy

We discussed the fact that Technical Debt, while it's something that developers know all about and experience sometimes quite tangibly, it is often not visible to the business or product owner and, the conversation focussed on how to make it more visible.

What followed was then a discussion looking at various tools from ticket management systems, some alternatives like http://www.techdebt.org which are mentioned above, through to using something like a whiteboard with the basic system architecture which is annotated with post-its showing where there are/were code smells, bugs, critical problems, upgrades required. Something like a Technical Debt Heatmap.

CITCONANZ2014Sessions

2014-02-27T01:02:08Z

Paulflewelling: /* 2:00 Topics */

CITCON ANZ 2014 Auckland Sessions

Back to the [[Main Page]]

== 10:00 Topics ==

# [[Virtualisation Services]]
# [[CI & TDD for legacy Systems]]
# [[CI in thirty minutes]]
# [[Reusing validation tests as monitoring]]
# [[Mobile Testing]]

== 11:15 Topics ==

# [[Vagrant/Packer for Continuous Delivery of Application Infrastructure]]
# [[?]]
# [[?]]
# [[?]]
# [[Tips and tricks for CI and CD]]

== Lunch Topics ==

# [[?]]

== 2:00 Topics ==

# [[How do you change the team culture from waterfall to shepherding change to production + Communication model]]
# [[?]]
# [[Managing Software Technical Debt]]
# [[Test Execution Time and Running Web-UI Tests in Parallel]]
# [[?]]

== 3:15 Topics ==

# [[?]]
# [[?]]
# [[?]]
# [[?]]
# [[?]]

== 4:30 Topics ==

# [[?]]
# [[?]]
# [[Deployment Contracts]]
# [[Beyond Given-When-Then]]
# [[?]]

== Table View ==

{| class="wikitable"
|-
! Room name
! 10:00
! 11:15
! 2:00
! 3:15
! 4:30
|-
| Cube
| [[?]]
| [[?]]
| [[?]]
| [[?]]
| [[?]]
|-
| Cube Bar
| [[?]]
| [[?]]
| [[?]]
| [[?]]
| [[?]]
|-
| Cube Hall
| [[CI in thirty minutes]]
| [[?]]
| [[?]]
| [[?]]
| [[?]]
|-
| Oku Wairangi
| [[?]]
| [[?]]
| [[Test Execution Time and Running Web-UI Tests in Parallel]]
| [[?]]
| [[Beyond Given-When-Then]]
|-
| Yellow Circle
| [[?]]
| [[Tips and tricks for CI and CD]]
| [[?]]
| [[?]]
| [[?]]
|}