CitconWiki - User contributions [en]

Takeaways from Citcon 2012

2012-09-23T00:55:36Z

Kentbye: Created page with "'''Takeaways from Citicon Portland 2012''' * Break addiction to avoidance. Focus on speed. * Opportunity that's here to improve things * Testing. We all have the same problems..."

'''Takeaways from Citicon Portland 2012'''
* Break addiction to avoidance. Focus on speed.
* Opportunity that's here to improve things
* Testing. We all have the same problems. Work towards.
* Experienced similar problems in testing gives him heart, and have more vision.
* Ideas from other processes and tools that didn't know existed.
* Enjoyed open space format, and choose own topics
* My team doesn't check in code often enough.
* Anti-pattern notes and the taxonomy
* At different stages and could empathasize
* Andy's session on the Quagmire on change management.
* There's still a lot of work to be done. Lots of common struggles.
* Get other to improve themselves. Need to improve your own self first.
* Continuous integration is a process and mindset of people than the technology that you use to do it. Lots of ways to do it. The continuous part is the hard part.
* CI who check in stuff, but don't turn it on. Most favorite phrase was Voodoo charm was to say something to shut down thinking
* Expect more from team of developers, and trust that they can do their jobs
* Intrigued by idea of CI testing and sysadmin setups and testing the client that sets up the system and whether the system is working right. Lots of complexities that didn't think about before
* Felt shockingly like same problems as 2 1/2 years ago. Problems are pretty universal and not just the software. Would help for whatever industry he's in.
* Problems are hard, and he still doesn't have an answer. Aha moment with Jez Humble. Stop thinking about optimizing between mean time to failure. But optimize to mean time form failure and that might be just as good.
* It's still a people problem
* Enjoyed open space because lots of things to learn. It's all about people, and getting richer conversations.
* Enjoyed brainstorming environment.
* Meeting Adam in person was a highlight. Anti-patterns and smells was also interesting. There's a diversity of ingenuity to solve these problems. Would like to produce catalog of ingenious solutions.
* Watching people pair with James Short, and learned a lot and it was very cool.
* Software development and devops need to collaborate
* 1-on-1 session on what the real problem is, and what they need to solve. Perhaps start a CI user group.

CITCONNA2012Sessions

2012-09-23T00:54:55Z

Kentbye:

CITCON United States Portland 2012 Sessions

Back to the [[Main Page]]

== 10:00 Topics ==

#[[CI Anti-Patterns]]
#
#[[Test Scope unit vs functional vs dev vs QA]]
#[[Consolidated dashboard reporting of unit tests]]
#

== 11:15 Topics ==

#[[Detox the testing pyramid]]
#[[Bringing Automation to Manual Testers]]
#[[Out of the quagmire]]
#[[Sani Opinions pros cons]]
#

== 2:00 Topics ==

#
# [[Anti-Automated Test Patterns]]
#
# [[Lets Play TDD]]
#

== 3:15 Topics ==

#
#
#
#
#

== 4:30 Topics ==

# [[Why won't this work? Antidotes to resistance]]
#
#
#
#

[[Takeaways from Citcon 2012]]

== Table View ==

{| class="wikitable"
|-
! #
! 10:00
! 11:15
! 2:00
! 3:15
! 4:30
|-
| 1
| [[CI Anti-Patterns]]
| [[Detox the testing pyramid]]
|
|
|
|-
| 2
|
| [[Bringing Automation to Manual Testers]]
|
|
|
|-
| 3
| [[Test Scope unit vs functional vs dev vs QA]]
| [[Out of the quagmire]]
|
|
|
|-
| 4
| [[Consolidated dashboard reporting of unit tests]]
| [[Sani Opinions pros cons]]
| [[Lets Play TDD]]
|
|
|-
| 5
|
|
|
|
|
|}

Why won't this work? Antidotes to resistance

2012-09-23T00:35:00Z

Kentbye:

Common reasons of '''Why can't this work?'''
* We're afraid of change.
* Sunk cost. Because we're already paying for X product, and we want to use it. But could save money by dropping it.
* "Structure of magic 1 & 2" NLP book that pulls out structure of language
* Because our teams are in separate locations.
* No permission support. "Our managers won't let us."
* Our developers will rebel against a new process
* Everyone will quit.
* We don't have time. We're under pressure to deliver.
* It's legacy. Lots of technical debt. Lots of stuff runs on batch processes.
* Because that's not scrum. That's not agile.
* What's in it for me? Why?
* It might make me look bad.
* I don't own it. Not my problem.
* Where's data that will prove that this work.
* We don't know how
* Only works on toy projects
* We don't have a problem
* That's too much new stuff. Quit teaching me these new things.
* Hard sell to have people try synchronous integration
* People should just do the right thing without it.
* That's not enterprise technology.
* We should detect that in code review
* We have smart people we shouldn't be telling them what to do
* Can't prove the ROI on my stuff. Seen it on the data, but can't prove that
* It's too much work.
* Because we've followed the path of least resistance
* Resisting planning ahead and gantt does
* My ____ doesn't care
* It'll slow us down.
* That's hard
* We only have a few users

'''Antidotes'''

What is "this"? Missing referential indexes - "This" is generalized, and not specific enough.

Most of these sound like false objections. Sometimes we should honor where these are coming from.
Asking for help is an awesome thing to do. These are not constructive objections.

First objection you hear is always a false objection, and it's not the real objection. The first answer is trying to get rid of you.

Objections arise from a failure to establish value. Not establish problem that is serious enough to solve. Objections come from emotional reasons, and they are not from a rational perspective, and need to be considered from an emotional perspective.

Need to live in other person's shoes, and see what their concerns are form an emotional POV.

List of eight common sales objections. If you can't talk to these eight points, then you won't be able to close sales.

Politely ignore in order to dig deeper. If you push someone, then they'll just strengthen their unreasonable position. e.g. "Managers told us we can't do unit testing." "Must be frustrating to not be able to test your code." Reflect their perspective, believe it and be empathetic. Assume positive intent, and that they're doing the best thing that they know how to do in the situation. In their mind, they're doing it for noble reasons.

Showing empathy and really understanding what they're going through and understanding their pain and what they're going through is really important. Sympathy can reinforce their pain / I'm better than you, and I'll give you pity and hand out. Empathy and compassion is a lot better. Compassion may be a better word.

Near enemy. Compassion is the genuine compassion. Pity is that it's too bad that you're suffering / It sucks to be you. Compassion insights action to help. Compassion includes wanting to help.

One big motivation is reducing suffering.

"Slight of mouth" patterns of taking language and reframing it.

Assume what other people think is mind reading. Looking for invalid attributions.

Get backup support to come support what you're advocating for. Senior business person, and want to do the right thing even if it takes longer. 2nd best answer is no.

Rejecting help, they believe that they're doing the right thing for the business. That it'll be a waste of time, and we should just get to work and get stuff done to complete our deadline. We're already behind. Upper level management felt like a big problem was there, and they wanted to invest money to fix it, but developers were resistant.

CTO should say we should do TDD, and there's resistant.

Interviewing each department, and understanding pain points of each department. In the end there were commonalities of not knowing how to get things to go through.

Meet with individual contributors, and lack of development environments. Met with VP who had budgetary problem, and didn't believe that it's a problem. But didn't believe that this wasn't their biggest problem.

Could be that it was asking VP to do something when other people need to change. Seemed to be mental block, and wasn't able to do it after 3 tries. Steamroller of reality wasn't happening, it changed a year later.

Plant seeds. Sometimes you just have to let go.

Unrealistic expectation of how fast you can learn agile stuff. It takes time.

Celebrate the successes.

Sometimes the most successful technique is to go away for a while. Don't become too dependent. Let them know that he'll be going away. Job is not to do it for them. Your job is to take this on and decide if you continue to do this. Tell them that this iteration is yours. Take well-timed bathroom breaks. A planning meeting, and they're not showing up (Take a walk for 15 minutes) This is setting up "responsibility boundaries"

Allow small failures. And all sorts of failures. Age-appropriate failures.

Do "One small thing" pattern

"Try and see" pattern. See if this comes to fruition. Try it, and see what works and see what works out.

Ask permission to be rigorous. Try to see it and see how it works.

Leverage inertia

Ask permission to help them

Don't inflict help.

''Notes by Kent Bye''

Why won't this work? Antidotes to resistance

2012-09-23T00:34:27Z

Kentbye: Created page with "Common reasons of '''Why can't this work?''' * We're afraid of change. * Sunk cost. Because we're already paying for X product, and we want to use it. But could save money by..."

Common reasons of '''Why can't this work?'''
* We're afraid of change.
* Sunk cost. Because we're already paying for X product, and we want to use it. But could save money by dropping it.
* "Structure of magic 1 & 2" NLP book that pulls out structure of language
Because our teams are in separate locations.
* No permission support. "Our managers won't let us."
* Our developers will rebel against a new process
* Everyone will quit.
* We don't have time. We're under pressure to deliver.
* It's legacy. Lots of technical debt. Lots of stuff runs on batch processes.
* Because that's not scrum. That's not agile.
* What's in it for me? Why?
* It might make me look bad.
* I don't own it. Not my problem.
* Where's data that will prove that this work.
* We don't know how
* Only works on toy projects
* We don't have a problem
* That's too much new stuff. Quit teaching me these new things.
* Hard sell to have people try synchronous integration
* People should just do the right thing without it.
* That's not enterprise technology.
* We should detect that in code review
* We have smart people we shouldn't be telling them what to do
* Can't prove the ROI on my stuff. Seen it on the data, but can't prove that
* It's too much work.
* Because we've followed the path of least resistance
* Resisting planning ahead and gantt does
* My ____ doesn't care
* It'll slow us down.
* That's hard
* We only have a few users

'''Antidotes'''

What is "this"? Missing referential indexes - "This" is generalized, and not specific enough.

Most of these sound like false objections. Sometimes we should honor where these are coming from.
Asking for help is an awesome thing to do. These are not constructive objections.

First objection you hear is always a false objection, and it's not the real objection. The first answer is trying to get rid of you.

Objections arise from a failure to establish value. Not establish problem that is serious enough to solve. Objections come from emotional reasons, and they are not from a rational perspective, and need to be considered from an emotional perspective.

Need to live in other person's shoes, and see what their concerns are form an emotional POV.

List of eight common sales objections. If you can't talk to these eight points, then you won't be able to close sales.

Politely ignore in order to dig deeper. If you push someone, then they'll just strengthen their unreasonable position. e.g. "Managers told us we can't do unit testing." "Must be frustrating to not be able to test your code." Reflect their perspective, believe it and be empathetic. Assume positive intent, and that they're doing the best thing that they know how to do in the situation. In their mind, they're doing it for noble reasons.

Showing empathy and really understanding what they're going through and understanding their pain and what they're going through is really important. Sympathy can reinforce their pain / I'm better than you, and I'll give you pity and hand out. Empathy and compassion is a lot better. Compassion may be a better word.

Near enemy. Compassion is the genuine compassion. Pity is that it's too bad that you're suffering / It sucks to be you. Compassion insights action to help. Compassion includes wanting to help.

One big motivation is reducing suffering.

"Slight of mouth" patterns of taking language and reframing it.

Assume what other people think is mind reading. Looking for invalid attributions.

Get backup support to come support what you're advocating for. Senior business person, and want to do the right thing even if it takes longer. 2nd best answer is no.

Rejecting help, they believe that they're doing the right thing for the business. That it'll be a waste of time, and we should just get to work and get stuff done to complete our deadline. We're already behind. Upper level management felt like a big problem was there, and they wanted to invest money to fix it, but developers were resistant.

CTO should say we should do TDD, and there's resistant.

Interviewing each department, and understanding pain points of each department. In the end there were commonalities of not knowing how to get things to go through.

Meet with individual contributors, and lack of development environments. Met with VP who had budgetary problem, and didn't believe that it's a problem. But didn't believe that this wasn't their biggest problem.

Could be that it was asking VP to do something when other people need to change. Seemed to be mental block, and wasn't able to do it after 3 tries. Steamroller of reality wasn't happening, it changed a year later.

Plant seeds. Sometimes you just have to let go.

Unrealistic expectation of how fast you can learn agile stuff. It takes time.

Celebrate the successes.

Sometimes the most successful technique is to go away for a while. Don't become too dependent. Let them know that he'll be going away. Job is not to do it for them. Your job is to take this on and decide if you continue to do this. Tell them that this iteration is yours. Take well-timed bathroom breaks. A planning meeting, and they're not showing up (Take a walk for 15 minutes) This is setting up "responsibility boundaries"

Allow small failures. And all sorts of failures. Age-appropriate failures.

Do "One small thing" pattern

"Try and see" pattern. See if this comes to fruition. Try it, and see what works and see what works out.

Ask permission to be rigorous. Try to see it and see how it works.

Leverage inertia

Ask permission to help them

Don't inflict help.

''Notes by Kent Bye''

CITCONNA2012Sessions

2012-09-23T00:34:06Z

Kentbye:

CITCON United States Portland 2012 Sessions

Back to the [[Main Page]]

== 10:00 Topics ==

#[[CI Anti-Patterns]]
#
#[[Test Scope unit vs functional vs dev vs QA]]
#[[Consolidated dashboard reporting of unit tests]]
#

== 11:15 Topics ==

#[[Detox the testing pyramid]]
#[[Bringing Automation to Manual Testers]]
#[[Out of the quagmire]]
#[[Sani Opinions pros cons]]
#

== 2:00 Topics ==

#
# [[Anti-Automated Test Patterns]]
#
# [[Lets Play TDD]]
#

== 3:15 Topics ==

#
#
#
#
#

== 4:30 Topics ==

# [[Why won't this work? Antidotes to resistance]]
#
#
#
#

== Table View ==

{| class="wikitable"
|-
! #
! 10:00
! 11:15
! 2:00
! 3:15
! 4:30
|-
| 1
| [[CI Anti-Patterns]]
| [[Detox the testing pyramid]]
|
|
|
|-
| 2
|
| [[Bringing Automation to Manual Testers]]
|
|
|
|-
| 3
| [[Test Scope unit vs functional vs dev vs QA]]
| [[Out of the quagmire]]
|
|
|
|-
| 4
| [[Consolidated dashboard reporting of unit tests]]
| [[Sani Opinions pros cons]]
| [[Lets Play TDD]]
|
|
|-
| 5
|
|
|
|
|
|}

Bringing Automation to Manual Testers

2012-09-22T22:11:49Z

Kentbye:

Bringing automation to manual testers (with no budget)

There are 9 development cross-functional agile teams. 2 testers and 5 developers. Tester vs. developer? Staff of testers who only do manual testers. Defect leakage is lower. No problem form an efficiency POV.

If not have automation training, then job satisfaction is down b/c Google says manual testing is dead. Have no training budget for tools or training?

What could he do? Do a number of activities to create hand-on experience to do small tests on their own. Brian isn't from a testing background. It should really come from the testers. Would love to have someone from testing department to step up and lead it. Brian is being a facilitator.

Potential Tactics
* Technology association Oregon to have a panel discussion to host at their office. Invite local experts who have implemented automation, and get testers to come to hear success stories. Hear about tools to use, Learning Curve.
* Brown bags: Do a series of internal presentations about automation with the tools and code base.
* Academy classes: Talk for a topic and ask questions.
* Use power shell as a testing tool. Invoke domain business objects and do some testing on.

Need to foster hands-on experience
* Record and play automation tool. 3 classes to work through an application of record and playback -- all doing the same one. Everyone who wanted to do testing could do that. See scripts that was generated, and talk about the script after the fact
* Then train the testers in programming over 6-8 weeks, and have homework. Train control structures and basic OO programming. Could buy books.
* Organize a user group to show others.

* Identify places where automation would be helpful.
* Create opportunities

How much free time do testers have? Are they storming or forming?
If they have homework, then they'd have to do extra time.

Introduced test automation, and cut regression test time by a certain amount of %.
Personnel turn-over.
If need 20 hours to automate test, then is there going to get pushback.

Commit to a story that's focused on quality during each sprint.
Tag stories that you can track and report on them.

Did some training classes where the students train each other in Java. People who are novices are better at training each other. Need one person who knows what they're doing.

Did training of OO analysis and design at a bank. Cultural differences, and they didn't take well to outside training. Start small and engage testers at developing their own curriculum. Take into account the culture, and it's more likely to succeed. Start small and grow it rather than starting with a big bang. Time is ripe, and there's enough momentum out there that needs to be pushed a bit.

Testing new feature, is it difficult and take significantly more time to test it automatically.

Anti-pattern is short-term thinking rather than long-term planning. If product manager isn't supportive from a timeline perspective, then the long-term benefits will be the first to go.

Has to be a commitment to the long-term to get out of the short-term, fire-fighting mentality. And slowly quality slowly degrades.

IT is on board for Agile, but didn't educate business about agile. Need to slow down in the short-term in order to go faster in the long-term.

Implemented agile, and the deliverable times is down to 2 weeks, and the QA became the bottleneck. Then had to go back up to 3 weeks. Need to automate the testing. Review the tools. Have the engineer go to QA staff and train them. QA staff is heads down on their 3-week cycles.

Fostering the sense of leadership, and have the developers start to evaluate

Gauge their interest. How easy for them to pick up.

In choosing tools: Easier to train on? Or better to get a tool best for their technology stack?
Potentially have a BDD.

Developers will train the QA staff to take the test for Java certification course done by Oracle.

Strategy for training people: Local tech user group to do their own training. Struggling how to teach novices who don't have full time to learning this. Start with 1-day workshop to get development environment set up. Added a 2nd user group meeting for new users with 1/2 trainers and 1/2 people who are novices. Ask experts how to do a specific task. As an expert, you do them so much that don't think about them as concepts. As a practice, then you write down the question so that you can teach it to them the next time.

If someone asks it, then others are thinking it.

* Python and Ruby groups are having a 2nd meeting every month with an occasional workshop. Beginning Ruby meet-up. Lots of people who want to learn. Experienced people know that you can learn bad things, and so testing helps to learn the good ones. Went over xUnit and BDD and some Cucumber stuff, and there will be more sessions like that in the future.

A ladder of tasks and competencies for people contribute and start to being able to contribute. In Drupal, there's the Drupal ladder.

* Tester pull down developer code, and build it and run it.
* Look at unit test, and be able to read it. Start to turn Blackbox testers into Whitebox testers
* Open hatch: One Barrier to entry is using version control. How to make a commit to git, and it's like a video game. Make a pull, and make a change. And then they get points or stars.

* Automate low-hanging fruit. Start with some easy tests to get experience so that they can get through rough spots. Start small. Case studies: Need to have small projects where automation written didn't need to be preserved beyond task that they could do within the course of their regular testing.
* Optimizing whatever is really repetitive and whatever would do the most impact in the least amount of work. Automate what you're doing a lot, and look for something has a decent bang for your buck.

* SQA User Group is just getting started up -- http://www.sqaug.org/
* Work on acceptance criteria as a cross-functional team, and help business see the benefit so that we can invest more time and energy into QA.

* Started using Fitness tool to automated business-facing acceptance testing, and then the product managers would look at it within English. But the product manager didn't care or look at the test. Testers and developers had lots of conversations which were really helpful.

Business object layer exposed through an API, and the testers could give a scripting language so that they could reach the test objects. Testers could write business-facing acceptance tests with Cucumber, and then it'd be a good step towards automation.

Java has a lot of odd rules for non-programmers, and it'd be intimidating because it's not intuitive.

Programmers know more than one language. Learn the easiest things to learn, and once you have the concepts, then going to more complicated languages like Java become easier. Don't try to learn everything at once. Make it cumulative.

If you need budget, then ask "How much is QA saving you?" instead of "How much is QA costing?"

''Notes by Kent Bye''

Anti-Automated Test Patterns

2012-09-22T22:10:21Z

Kentbye:

'''Anti-Automated Test Patterns'''

* '''Ice cream cone'''
Inverse of triangle of having lots of manual testing at the top, and not a lot of unit tests at the bottom. Break it down by risk.

* '''Happy Path'''
Basic function of the system, but it's not really testing anything serious. False sense of being complete. Don't have complete code coverage that you need. Need to do more comprehensive testing.

* '''Local Hero'''
Wrote something that will always pass in environment because of how you've written it. Seems risky because you expect one thing, but explodes in the real world. Learn from the regressions that you get out of it. Get input from customer service. May not be looking at the business requirements properly.

Perhaps deploy to a staging environment? May have lost the customer focus. Have customer service to testing.

Beautiful thing in real world, but you're not testing it in the same way that the users are using it. They may be doing more complicated tasks than you're testing for.

* '''2nd Class Citizen'''
A lot of duplicate code, and so the code bloats up, and it becomes a maintenance headache. Same thing over and over again. You might need to refactor your automated test. May need to reuse some of the code that you've written.

If it's a high-risk thing, then perhaps refactor. If it's used often, then use re-use. Do evaluation before you start to refactor. Don't pay off all of your tech debt all at once.

* '''Chain Gang'''
Set up. Task and then Tear down. Merge two tests into one set up and tear down. Create dependencies within a chain of tests. Set up and tear down can be a PITA.
Can be acceptable in some time if the tests are passing and you feel comfortable that they're valid. Advice is to evaluate the risk in the gang, and if there are problems, then split it up.
Avoid it, but when you need it, then use it.

* '''The Mockery'''
Use the real stuff. Over mock everything. Need something. Mock it. Not using the real world or the real servers you'll be working with. If mocking REST and SOAP calls, and then it fails on the live data. If there's a lot of mockery.

The opposite of mockery is the local hero.

For example, with a maven repo, grab top layer, and it'll work. Customer pulls from inside, and then it broke the code, and weren't testing it. Purpose of mock objects is to avoid inspector.

* '''The Inspector'''
It knows everything about the system. Object knows everything and tightly coupled. If changes, then it breaks. Joined models together, and if you remove it, then it breaks.

White box testing is something that if you test it, then it breaks. Perhaps only use it on edge cases.

Break it up, and make it less dependent and decoupled.

* '''The Golddigger'''
Greedy and they want everything in terms of resources. They need to have 50 things set up. Lots of time required. Can you really break it up? It takes 2 hours to set up. Pre-set up while doing a deployment. Deploy what the things that the gold digger needs.

At what point in the process do you right that test? Do it at the end? Or do it at the beginning?

* '''Anal probe / Contract Violator'''
Write test and get to the internals, and then override OO fundamentals with private and public stuff. Heavily white, and get into internals of anything. Are you testing it in a realistic way. Playing with the innards of the code, and if things change, then you're screwed. Violating an object.

Would exploratory or ad hoc testing be enough for this test? Might be blinded by real part of the test.

If something is inside that you need to test, then it's a design problem where it needs to exposed of whatever needs to be available. Need to reword code.

* '''Test with no name'''
There's a bug and name it non-sensical name like "Test CR2386." Solution is to use better names and do it right. Name doesn't tell you anything. This is more of a bad practice than an anti-pattern.

* '''The Slow Poke'''
Takes a lot of time to run. Could potentially run in parallel. Or potentially break it up. Set up your own environment and make it aware. Make not put in CI or CD. Only run on release candidate instead of a daily build. It's likely to be an integration test, but it could be at any level. Database dependencies and network latencies. Can't always run integration test. Could you mock something?

* '''The Giant / God Complex Test / Boss Hog'''
If it's a big test, that is consuming. Way too much code, and may be a part of a chain gang. It's very complex

* '''Wait & See'''
Using Sleep. Love and hate relationship with sleep. Press button, and sleep. You're not checking the validity of the system. You're going to race condition. It'll cause flickering. Solution is to not use sleep. If you have sleeps everywhere, and make sure you don't have interrupts.

* '''China Vase'''
Code is fragile. Selenium is too fragile. Biggest issue in industry. Every one keeps complaining about that's it's too long or it's too fragile. How do we deal with it? Break down into more stable pieces. Might have some other anti-patterns happens. More concerned with the China passes than fails.

* '''Flickering Lights'''
Flickers between passing and failing. Didn't write the test correctly. Too much mockery or too much golddigging. Had two different load balances, and direct to a working and not working breaking. It can also usually it's an environmental issue. It can be demoralizing, and testers get used to living with red lights. Psychologically keep pushing button until it passes, which is a bad habit. If I hit restart it 3 times, and then I'll investigate it. [laugh] If it doesn't pass 1st time, then investigate it.

* '''The Pig'''
Tests that don't clean up after themselves, which can lead to flickering lights. Dependencies that relate to each other.

* '''Edge Play'''
Playing on the edges too much. Waste the test cycles of testing things that the user doesn't do. High-risk, and you might only run it once. Reduce it form main test suites or take it out.

* '''Customer Don't Do that'''
Testing things that customer doesn't actually do.

* '''Fear the automator'''
The fear that manual testers that automation will eliminate their jobs. Deliberate sabotage, and party when automated test fails. It's a management issue, and will loose morale and testing cycles.

* '''The Metrics Lie'''
Management will want to know how much test cycles are saving. Wanting ROI metrics, and sacrifice. Opening tickets on small tasks just to get the metrics to be higher. "Get lots of bugs now!" to justify bugs. "If you find a bug, then cover it up"

* '''Test doesn't test anything'''
Happens in unit a lot. Who's responsible for that? Developer or tester? Whomever wrote it or is maintaining it. Who's' responsible for what? SOMEONE is responsible. If it doesn't do what it's supposed to do, then someone will need to take responsible.

* '''Who owns this?'''
No one owns it, and it gets ignored. Transparency and communication is the solution. Project team leads report to each other, and they have to fight it out. Denying responsibility, and have to prove it's not yours. Manager should know, but sometimes there's no management structure.

* '''How are these related?'''
Boss hog and slow poke are connected
Inspector and Gold Digger are connected.
China Vase and Flickering Lights are connected
If you're seeing flickering lights, then root cause could be Pig
2nd class citizen and the Flickering lights would be related.
Gold Digger and Inspector are the same thing.
Chain gang would lead to Boss Hog
Mockery is connected to flickering lights
Local hero goes with flickering lights. Works fine in staging, but not production
Ice cream cone is independent. Break it down by risk is the answer to anything.

* '''Bad practices'''
Test with no name, wait and see, 2nd class citizen

''Notes by Kent Bye''

Anti-Automated Test Patterns

2012-09-22T22:09:46Z

Kentbye:

Anti-Automated Test Patterns

2012-09-22T22:08:01Z

Kentbye:

Anti-Automated Test Patterns

* '''Ice cream cone'''
Inverse of triangle of having lots of manual testing at the top, and not a lot of unit tests at the bottom. Break it down by risk.

'''Happy Path'''
Basic function of the system, but it's not really testing anything serious. False sense of being complete. Don't have complete code coverage that you need. Need to do more comprehensive testing.

'''Local Hero'''
Wrote something that will always pass in environment because of how you've written it. Seems risky because you expect one thing, but explodes in the real world. Learn from the regressions that you get out of it. Get input from customer service. May not be looking at the business requirements properly.

Perhaps deploy to a staging environment? May have lost the customer focus. Have customer service to testing.

Beautiful thing in real world, but you're not testing it in the same way that the users are using it. They may be doing more complicated tasks than you're testing for.

'''2nd Class Citizen'''
A lot of duplicate code, and so the code bloats up, and it becomes a maintenance headache. Same thing over and over again. You might need to refactor your automated test. May need to reuse some of the code that you've written.

If it's a high-risk thing, then perhaps refactor. If it's used often, then use re-use. Do evaluation before you start to refactor. Don't pay off all of your tech debt all at once.

'''Chain Gang'''
Set up. Task and then Tear down. Merge two tests into one set up and tear down. Create dependencies within a chain of tests. Set up and tear down can be a PITA.
Can be acceptable in some time if the tests are passing and you feel comfortable that they're valid. Advice is to evaluate the risk in the gang, and if there are problems, then split it up.
Avoid it, but when you need it, then use it.

'''The Mockery'''
Use the real stuff. Over mock everything. Need something. Mock it. Not using the real world or the real servers you'll be working with. If mocking REST and SOAP calls, and then it fails on the live data. If there's a lot of mockery.

The opposite of mockery is the local hero.

For example, with a maven repo, grab top layer, and it'll work. Customer pulls from inside, and then it broke the code, and weren't testing it. Purpose of mock objects is to avoid inspector.

'''The Inspector'''
It knows everything about the system. Object knows everything and tightly coupled. If changes, then it breaks. Joined models together, and if you remove it, then it breaks.

White box testing is something that if you test it, then it breaks. Perhaps only use it on edge cases.

Break it up, and make it less dependent and decoupled.

'''The Golddigger'''
Greedy and they want everything in terms of resources. They need to have 50 things set up. Lots of time required. Can you really break it up? It takes 2 hours to set up. Pre-set up while doing a deployment. Deploy what the things that the gold digger needs.

At what point in the process do you right that test? Do it at the end? Or do it at the beginning?

'''Anal probe / Contract Violator'''
Write test and get to the internals, and then override OO fundamentals with private and public stuff. Heavily white, and get into internals of anything. Are you testing it in a realistic way. Playing with the innards of the code, and if things change, then you're screwed. Violating an object.

Would exploratory or ad hoc testing be enough for this test? Might be blinded by real part of the test.

If something is inside that you need to test, then it's a design problem where it needs to exposed of whatever needs to be available. Need to reword code.

'''Test with no name'''
There's a bug and name it non-sensical name like "Test CR2386." Solution is to use better names and do it right. Name doesn't tell you anything. This is more of a bad practice than an anti-pattern.

'''The Slow Poke'''
Takes a lot of time to run. Could potentially run in parallel. Or potentially break it up. Set up your own environment and make it aware. Make not put in CI or CD. Only run on release candidate instead of a daily build. It's likely to be an integration test, but it could be at any level. Database dependencies and network latencies. Can't always run integration test. Could you mock something?

'''The Giant / God Complex Test / Boss Hog'''
If it's a big test, that is consuming. Way too much code, and may be a part of a chain gang. It's very complex

'''Wait & See'''
Using Sleep. Love and hate relationship with sleep. Press button, and sleep. You're not checking the validity of the system. You're going to race condition. It'll cause flickering. Solution is to not use sleep. If you have sleeps everywhere, and make sure you don't have interrupts.

'''China Vase'''
Code is fragile. Selenium is too fragile. Biggest issue in industry. Every one keeps complaining about that's it's too long or it's too fragile. How do we deal with it? Break down into more stable pieces. Might have some other anti-patterns happens. More concerned with the China passes than fails.

'''Flickering Lights'''
Flickers between passing and failing. Didn't write the test correctly. Too much mockery or too much golddigging. Had two different load balances, and direct to a working and not working breaking. It can also usually it's an environmental issue. It can be demoralizing, and testers get used to living with red lights. Psychologically keep pushing button until it passes, which is a bad habit. If I hit restart it 3 times, and then I'll investigate it. [laugh] If it doesn't pass 1st time, then investigate it.

'''The Pig'''
Tests that don't clean up after themselves, which can lead to flickering lights. Dependencies that relate to each other.

'''Edge Play'''
Playing on the edges too much. Waste the test cycles of testing things that the user doesn't do. High-risk, and you might only run it once. Reduce it form main test suites or take it out.

'''Customer Don't Do that'''
Testing things that customer doesn't actually do.

'''Fear the automator'''
The fear that manual testers that automation will eliminate their jobs. Deliberate sabotage, and party when automated test fails. It's a management issue, and will loose morale and testing cycles.

'''The Metrics Lie'''
Management will want to know how much test cycles are saving. Wanting ROI metrics, and sacrifice. Opening tickets on small tasks just to get the metrics to be higher. "Get lots of bugs now!" to justify bugs. "If you find a bug, then cover it up"

'''Test doesn't test anything'''
Happens in unit a lot. Who's responsible for that? Developer or tester? Whomever wrote it or is maintaining it. Who's' responsible for what? SOMEONE is responsible. If it doesn't do what it's supposed to do, then someone will need to take responsible.

'''Who owns this?'''
No one owns it, and it gets ignored. Transparency and communication is the solution. Project team leads report to each other, and they have to fight it out. Denying responsibility, and have to prove it's not yours. Manager should know, but sometimes there's no management structure.

'''How are these related?'''
Boss hog and slow poke are connected
Inspector and Gold Digger are connected.
China Vase and Flickering Lights are connected
If you're seeing flickering lights, then root cause could be Pig
2nd class citizen and the Flickering lights would be related.
Gold Digger and Inspector are the same thing.
Chain gang would lead to Boss Hog
Mockery is connected to flickering lights
Local hero goes with flickering lights. Works fine in staging, but not production
Ice cream cone is independent. Break it down by risk is the answer to anything.

'''Bad practices'''
Test with no name, wait and see, 2nd class citizen

"Notes by Kent Bye"

Anti-Automated Test Patterns

2012-09-22T22:07:21Z

Kentbye: Created page with "Anti-Automated Test Patterns '''Ice cream cone''' Inverse of triangle of having lots of manual testing at the top, and not a lot of unit tests at the bottom. Break it down by..."

Anti-Automated Test Patterns

'''Ice cream cone'''
Inverse of triangle of having lots of manual testing at the top, and not a lot of unit tests at the bottom. Break it down by risk.

'''Happy Path'''
Basic function of the system, but it's not really testing anything serious. False sense of being complete. Don't have complete code coverage that you need. Need to do more comprehensive testing.

'''Local Hero'''
Wrote something that will always pass in environment because of how you've written it. Seems risky because you expect one thing, but explodes in the real world. Learn from the regressions that you get out of it. Get input from customer service. May not be looking at the business requirements properly.

Perhaps deploy to a staging environment? May have lost the customer focus. Have customer service to testing.

Beautiful thing in real world, but you're not testing it in the same way that the users are using it. They may be doing more complicated tasks than you're testing for.

'''2nd Class Citizen'''
A lot of duplicate code, and so the code bloats up, and it becomes a maintenance headache. Same thing over and over again. You might need to refactor your automated test. May need to reuse some of the code that you've written.

If it's a high-risk thing, then perhaps refactor. If it's used often, then use re-use. Do evaluation before you start to refactor. Don't pay off all of your tech debt all at once.

'''Chain Gang'''
Set up. Task and then Tear down. Merge two tests into one set up and tear down. Create dependencies within a chain of tests. Set up and tear down can be a PITA.
Can be acceptable in some time if the tests are passing and you feel comfortable that they're valid. Advice is to evaluate the risk in the gang, and if there are problems, then split it up.
Avoid it, but when you need it, then use it.

'''The Mockery'''
Use the real stuff. Over mock everything. Need something. Mock it. Not using the real world or the real servers you'll be working with. If mocking REST and SOAP calls, and then it fails on the live data. If there's a lot of mockery.

The opposite of mockery is the local hero.

For example, with a maven repo, grab top layer, and it'll work. Customer pulls from inside, and then it broke the code, and weren't testing it. Purpose of mock objects is to avoid inspector.

'''The Inspector'''
It knows everything about the system. Object knows everything and tightly coupled. If changes, then it breaks. Joined models together, and if you remove it, then it breaks.

White box testing is something that if you test it, then it breaks. Perhaps only use it on edge cases.

Break it up, and make it less dependent and decoupled.

'''The Golddigger'''
Greedy and they want everything in terms of resources. They need to have 50 things set up. Lots of time required. Can you really break it up? It takes 2 hours to set up. Pre-set up while doing a deployment. Deploy what the things that the gold digger needs.

At what point in the process do you right that test? Do it at the end? Or do it at the beginning?

'''Anal probe / Contract Violator'''
Write test and get to the internals, and then override OO fundamentals with private and public stuff. Heavily white, and get into internals of anything. Are you testing it in a realistic way. Playing with the innards of the code, and if things change, then you're screwed. Violating an object.

Would exploratory or ad hoc testing be enough for this test? Might be blinded by real part of the test.

If something is inside that you need to test, then it's a design problem where it needs to exposed of whatever needs to be available. Need to reword code.

'''Test with no name'''
There's a bug and name it non-sensical name like "Test CR2386." Solution is to use better names and do it right. Name doesn't tell you anything. This is more of a bad practice than an anti-pattern.

'''The Slow Poke'''
Takes a lot of time to run. Could potentially run in parallel. Or potentially break it up. Set up your own environment and make it aware. Make not put in CI or CD. Only run on release candidate instead of a daily build. It's likely to be an integration test, but it could be at any level. Database dependencies and network latencies. Can't always run integration test. Could you mock something?

'''The Giant / God Complex Test / Boss Hog'''
If it's a big test, that is consuming. Way too much code, and may be a part of a chain gang. It's very complex

'''Wait & See'''
Using Sleep. Love and hate relationship with sleep. Press button, and sleep. You're not checking the validity of the system. You're going to race condition. It'll cause flickering. Solution is to not use sleep. If you have sleeps everywhere, and make sure you don't have interrupts.

'''China Vase'''
Code is fragile. Selenium is too fragile. Biggest issue in industry. Every one keeps complaining about that's it's too long or it's too fragile. How do we deal with it? Break down into more stable pieces. Might have some other anti-patterns happens. More concerned with the China passes than fails.

'''Flickering Lights'''
Flickers between passing and failing. Didn't write the test correctly. Too much mockery or too much golddigging. Had two different load balances, and direct to a working and not working breaking. It can also usually it's an environmental issue. It can be demoralizing, and testers get used to living with red lights. Psychologically keep pushing button until it passes, which is a bad habit. If I hit restart it 3 times, and then I'll investigate it. [laugh] If it doesn't pass 1st time, then investigate it.

'''The Pig'''
Tests that don't clean up after themselves, which can lead to flickering lights. Dependencies that relate to each other.

'''Edge Play'''
Playing on the edges too much. Waste the test cycles of testing things that the user doesn't do. High-risk, and you might only run it once. Reduce it form main test suites or take it out.

'''Customer Don't Do that'''
Testing things that customer doesn't actually do.

'''Fear the automator'''
The fear that manual testers that automation will eliminate their jobs. Deliberate sabotage, and party when automated test fails. It's a management issue, and will loose morale and testing cycles.

'''The Metrics Lie'''
Management will want to know how much test cycles are saving. Wanting ROI metrics, and sacrifice. Opening tickets on small tasks just to get the metrics to be higher. "Get lots of bugs now!" to justify bugs. "If you find a bug, then cover it up"

'''Test doesn't test anything'''
Happens in unit a lot. Who's responsible for that? Developer or tester? Whomever wrote it or is maintaining it. Who's' responsible for what? SOMEONE is responsible. If it doesn't do what it's supposed to do, then someone will need to take responsible.

'''Who owns this?'''
No one owns it, and it gets ignored. Transparency and communication is the solution. Project team leads report to each other, and they have to fight it out. Denying responsibility, and have to prove it's not yours. Manager should know, but sometimes there's no management structure.

'''How are these related?'''
Boss hog and slow poke are connected
Inspector and Gold Digger are connected.
China Vase and Flickering Lights are connected
If you're seeing flickering lights, then root cause could be Pig
2nd class citizen and the Flickering lights would be related.
Gold Digger and Inspector are the same thing.
Chain gang would lead to Boss Hog
Mockery is connected to flickering lights
Local hero goes with flickering lights. Works fine in staging, but not production
Ice cream cone is independent. Break it down by risk is the answer to anything.

'''Bad practices'''
Test with no name, wait and see, 2nd class citizen

"Notes by Kent Bye"

CITCONNA2012Sessions

2012-09-22T22:02:47Z

Kentbye:

CITCON United States Portland 2012 Sessions

Back to the [[Main Page]]

== 10:00 Topics ==

#[[CI Anti-Patterns]]
#
#[[Test Scope unit vs functional vs dev vs QA]]
#[[Consolidated dashboard reporting of unit tests]]
#

== 11:15 Topics ==

#[[Detox the testing pyramid]]
#[[Bringing Automation to Manual Testers]]
#[[Out of the quagmire]]
#[[Sani Opinions pros cons]]
#

== 2:00 Topics ==

#
# [[Anti-Automated Test Patterns]]
#
#
#

== 3:15 Topics ==

#
#
#
#
#

== 4:30 Topics ==

#
#
#
#
#

== Table View ==

{| class="wikitable"
|-
! #
! 10:00
! 11:15
! 2:00
! 3:15
! 4:30
|-
| 1
| [[CI Anti-Patterns]]
| [[Detox the testing pyramid]]
|
|
|
|-
| 2
|
| [[Bringing Automation to Manual Testers]]
|
|
|
|-
| 3
| [[Test Scope unit vs functional vs dev vs QA]]
| [[Out of the quagmire]]
|
|
|
|-
| 4
| [[Consolidated dashboard reporting of unit tests]]
| [[Sani Opinions pros cons]]
|
|
|
|-
| 5
|
|
|
|
|
|}

CI Anti-Patterns

2012-09-22T22:01:37Z

Kentbye:

What are you going to monitor?
How will you know what you're monitoring happens?
What will you do if it happens?

Continuous deployment - Will the site work while you deploy the version? Or take the site down while you deploy? What happens to the users on the site while you're deploying?

If you're deploying manually, then you could think about these questions, but it's not mandatory.

Ops job was to be a buffer between devs and remote system administrators.

This particular Ops team were not sysadmins and there no monitoring in place. They were doing market fixes, and provide a buffer between devs and remote sysadmins.

Internal staging site, but it has a different topology. Production has 3 machines behind a load balancer. B/c of remote relationship, could change the app, except through the database.

Company is successful, but there's a lack of growing up. They plug gaps with people instead of systems.

Co-workers checked in code that didn't work. Code should compile before you check it in. Read Scott Adams, "Goals are for losers, winners build systems." Now have common deployment contract across deployments, and it can run transformations and run rules. They now have blue/green deployments. Ask it, are you ready to shut down? Now have init scripts to start and stop the service. Now have metrics in place.

Internal API to load data warehouse, and the traffic is much higher than live site. Run extract of data, and it slows down the production site. Run extract during running hours, and be able to monitor that performance isn't impact.

PJ session on Anti-patterns
Anti-patterns: Read book CI, and they want a plan. Provide a 2-year plan. Work with them for a year and half, and after 1.5 years, then they say that need 5 years. What they have in place, and what do they need to have in place. If they don't have it in place, then it's an anti-pattern.
What have you seen where people think they have it's right, but it's actually wrong.

Want to do continuous delivery. Deliver to production every 2 weeks. Have a build script? CI running? Have developers check in frequently? 300-400 devs, and they want a roadmap. One year into it, they kind of have a CI in place. Check in, build happens and it's red or green. No industry standard CI practices, like not always include unit tests. Not doing CI if not running unit tests. If you say CI, then assume including unit tests in the build. Lack of commitment to a green build. Can check in, and have red for weeks of a time, and it's acceptable within the org. You'll never achieve continuous delivery if you have don't commit to green build.

Need to fix very quickly within a couple of hours. Whoever broke it is responsible to fix it.

If you can't break your CI process. Run your build before you check in. Use rubber chicken.

USB nerd control that will target developer.

Break build, but if not fixed in 15 minutes, then we'll roll back.
If unit tests fail, then it's reverted.

How to prevent people from not checking in. Can't solve stupidity or malice. Only build systems that support good intentions.

Unstable test problems. Build will fail. Elaborate build radiator and mark test as flickering. Then call it a "bad test." Not obviously my problem. Next time runs, then it'll go green. Run 3-4 times to get different results. Tests that have non-deterministic behavior is an anti-pattern.

How to determine a non-deterministic test? Run again and it works.

Data-dependencies like data or class dependencies. Create object and it's not created first time. Race conditions is another cause. Tests that don't clean up after themselves. Run the tests in a random order can help make sure there's no dependencies.

Suggestion to write less end-to-end and write more unit tests.

Run forwards. Run backwards. Run in random order. Detect if not clean-up. Sometimes leak database connections. Detect when database leakages were happening. Red, yellow and green systems.
Build scripts? CI? Frequent check-ins?
Couldn't do deployment consistently. Needed to solidify system. Started with monthly deployments. Do a server every two days, and then move forward. Then moved to bi-weekly. Bring ops and developers and build their own deployment system.

Needed ops and dev collaboration in order to get to CI.

What makes CD unique from CI?

Why create user stories because tech spec was huge. Create spec and meet it, but it didn't bring any value. Then needed user story. Similarly, developer would meet requirement, but it still wouldn't get to CD.

Dev build something useful to the ops team.

Cucumber and nagios that provide ops-friendly output. Is it useful to bridge the gap between ops and devs? Yes.
Ops Not familiar with Chef or Puppet. Only familiar with web sphere and native web sphere tools.

Being able to reproduce infrastructure from the command-line. Need to collaborate with script automation and site operations. Way to communicated with them was to write cucumber test ATDD. Collaborated with them to create tests. Cucumber were for the infrastructure
"Give have a VM with an operating system with a Chef, when I run install_websphere.rv, then I go to this URL and should see an admin screen."

Use cucumber to monitor and do a virtual install. More often you're install an application, and with web sphere installers they were monitoring how well it was going. Given deploy_foo.sh, then I should NOT see X message. Or I should see Y message. Checking the log for details if something failed should not be a person. Use cucumber. Put cucumber output to nagios for ops people.

Will human every check log? Just for exploratory purposes. From systems POV, then look at log and know what's up. Suggestion to look at log to see if we're blind to anything not testing is covering. Showing all logs all the time is an anti-pattern. Don't plug gaps with people, do it with systems.

Deployment monitoring was an issue. Monitoring failed from the beginning. Eventually did hooks within system. Ping system for health check. Put output into a nagios alert.

Direction: Should only do manual testing as exploratory testing to discover unknown things that might be wrong. Regression testing is a confirmation that it works. See places that only 5% of unit testing and 95% of testing is manual regression testing. "Testing" could either be "checking" or "exploring." Jeff would insist that "testing" means exploring, but can't change industry systems. Only thing should not already be automated is looking at system or new ways to understand the system.

Continual thing, then have automated ways to find it. If a human is testing, then find out what needs to be tested. If new feature, then have humans who didn't design it, then there's usability testing. Can't do UX testing in CI. Do regular checks in system. If roll out new feature, then have humans use it in order to figure out what needs checking.

There is a test framework by Lou Wellon Falco to test visual appearance. "Approval testing." Do test, and then it records state of system. If it does change, then you detect it. Hybrid between manual and automated testing.

Area that's "hard" to test. You should test everything. Hard to test, then there could be tests. Layout in the browser isn't done well.

Take current build that's a golden build. Create a number of test cases, and take a snapshot of current state. New version of code on the other side, and then compare the test results according to the DOM. Spot the differences. Then a human can detect a CSS problem. Can do this cross-browser as well. Much less to spot UI issues.

Identify stuff that's in way of CI, and then identify ingenuity of solutions because of commitment to CD. Non-deterministic tests usually have bugs. If code is right, then why try to write tests. There's a barrier to commitment to CD/CI. Need to share ingenuity so that it's easier to do CD.

Treat test code as serious as production code. If the test is MORE difficult to write than production code, then it becomes hard to justify.

Shore: "Agile doesn't work if you don't have self-discipline." If you have a non-deterministic failure, then within a couple of weeks. Then you're accumulating debt.

But if you find a non-determinsitic failure, then put in another hour into it. Then they will eventually give up. May put 6-8 hours into a non-deterministic issue, and then give up.

1/2 of flicking tests are poorly written test and 1/2 are really difficult problem in code.

Turn on Code coverage before and after to detect issues.

Writing a book on How to detect flickering tests would be a best seller.
a
Database needs follow evolutionary design pattern. Duplicate data, maintain it, and then migrate it. Book on "Database continuous integration: Evolutionary database design." If make change to database, then write a delta script. ~Liqui-base.

Don't want downtime. Need to decouple structure of database. Address field. Split into two address field. Write migration that creates new stuff. Write new data, but only read old data. Need to have multiple versions of your code talking to the database. Mention of a "Refactoring databases" book.

Anti-pattern: Ivory tower DBA. Submit ticket to make change to database. DBA is a bottleneck to the organization. Very hard to reproduce the database. In order to reproduce the database, then you have to take db and reproduce it entirely. Takes a lot of time. If use a evolutionary database, then it's easier to grab 3% of database or a specific portion of db.

Instead of integrate with database, then integrate with services. Decouple database per service. Avoid having JOIN is reporting software. Pretend have persistent memory would it be okay objects to break database. Have a well-defined API. Have code that could read the alpha database.
Versioned by class. Was there code complexity to deal with it? No, there were abstractions that dealt with it.

NoSQL databases will defend them on a version. NoSQL migrations can be really difficult.

Collaboration between Dev and Ops

"Failure Mode and Effects Analysis" -- Failure analysis: What could go wrong with system? If it went wrong, then how would we know? How quickly could we fix it? Do risk and impact analysis, and add issues to the backlog.

Blue/Green deployments as a prerequisite? CD is defined differently per organization.

Do releases at 11 pm, after US market close and before AU market open.

Do releases under load on the site while users are on the system.

Have to be testing what you're releasing. Need the packages of what you're going to deploy. If not testing it, then you'd have put a commit to bump version number. Made and snapshots. Code signing process brings ambiguity to if it's the same.

''Notes by Kent Bye.''

Bringing Automation to Manual Testers

2012-09-22T21:05:09Z

Kentbye: Created page with "Bringing automation to manual testers (with no budget) There are 9 development cross-functional agile teams. 2 testers and 5 developers. Tester vs. developer? Staff of tester..."

CI Anti-Patterns

2012-09-22T21:04:26Z

Kentbye: Created page with "What are you going to monitor? How will you know what you're monitoring happens? What will you do if it happens? Continuous deployment - Will the site work while you deploy..."

CITCONNA2012Sessions

2012-09-22T21:03:28Z

Kentbye: Created page with "CITCON United States Portland 2012 Sessions Back to the Main Page == 10:00 Topics == #CI Anti-Patterns == 11:15 Topics == #[[Bringing Automation to Manual Testers..."

CITCON United States Portland 2012 Sessions

Back to the [[Main Page]]

== 10:00 Topics ==

#[[CI Anti-Patterns]]

== 11:15 Topics ==

#[[Bringing Automation to Manual Testers]]

== 2:00 Topics ==

CITCONNA2012Registrants

2012-09-21T06:24:20Z

Kentbye:

This page is generated from registration data. For changes or corrections please post to the CITCON mailing list.

[[Aaron Rhodes]] 
[[Adam Yuret]] 
[[Alex Yamashita]] 
[[Amy Julius]] 
[[An Doan]] 
[[Andrew Parker]] 
[[Antony Marcano]] 
[[Balaji Subramanian]] 
[[Brett Jones]] 
[[Brian Myers]] 
[[Bruce R Smith]] 
[[Dan Ivy]] 
[[Dan Pape]] 
[[Dan Post]] 
[[Daniel Johnson]] 
[[David Amick]] 
[[Diana]] 
[[Elisabeth Hendrickson]] 
[[Elizabeth Flanagan]] 
[[Fred Obermann]] 
[[Geoff Goodman]] 
[[George Breedlove]] 
[[Hang Dao]] 
[[Igal Koshevoy]] 
[[James Eisenhauer]] 
[[James Rucker]] 
[[James Shore]] 
[[Jason LaPier]] 
[[Jason Larsen]] 
[[Jay Riddle]] 
[[Jeff Rogers]] 
[[Jeffrey Fredrick]] 
[[Jeremy Haage]] 
[[Jesse Cooke]] 
[[Jesse Joe Bernardo]] 
[[Jim Dewson]] 
[[John Wilger]] 
[[Julias Shaw]] 
[[Justin Myers]] 
[[Kent Bye]] 
[[Khai Do]] 
[[Kirsten Comandich]] 
[[Kurt Ruff]] 
[[Lucian]] 
[[Lynn Scaglione]] 
[[Maggie Leake]] 
[[Maher M Hawash]] 
[[Marcos David Vacca]] 
[[Merlyn]] 
[[Michael P Karlovich]] 
[[Monica Farrell]] 
[[Nigel Syhaphonh]] 
[[Novak Banda]] 
[[Paul Giron]] 
[[Paul Julius]] 
[[Paul Jungwirth]] 
[[Paula Hannan]] 
[[Ravi Gadad]] 
[[Ruth Struck]] 
[[Ryan Souza]] 
[[Sabrina Rusi]] 
[[Saran Chinnaraj]] 
[[Shawn Dowling]] 
[[srinivasarao]] 
[[Steve Boone]] 
[[Stuart Celarier]] 
[[Thai Hai Pham]] 
[[Tom Miller]] 
[[Ward Cunningham]]