My Experience at QA or the Highway 2015 – Columbus


QA or the Highway
Date – February 17, 2015
Location – Columbus State Community College – Columbus, Ohio

Personal Reflections
This was the second conference that I was able to attend with my wife, Mindy.  It was an excellent day of learning.  The conference was well put together.  The speakers were dynamic and provided excellent content for the sessions that I attended.  I thank all of the individuals and vendors who made this event possible. I am already looking forward to the conference next year.

During the sessions, Mindy and my brother, David, were able to take some picture of the speakers in action. Mindy put together a glimpse of the conference in a slideshow for your enjoyment.

On February 24th, if you are in the Columbus/Dublin area, The Software QA & Testing meetup will be discussing “Lessons Learned from QA or the Highway 2015.”  I am looking forward to hearing about the experience of other conference attendees.

And now, to the recap.

The conference was well attended with approximately 350 testers.   The one day conference consisted of an opening keynote, 5 breakout sessions (with an option to chose from one of four tracks), and the closing keynote.   The talks were conducted by local, regional, and national speakers.  Experience of the speakers ranged from first time presenters to seasoned veterans.

Links to the presentations, presenters, and schedule:

Here is my account of the sessions that I attended.

Opening Keynote

David L. Kaufman kicked off the breakfast and morning with the opening keynote.
David spoke on the the topic of leadership in his presentation, “Leadership 2.0.”

Key takeaways and “lines”:

– Leadership 1.0 – “Leadership is the power or ability to direct the operation, activity, or performance of others.” – Webster’s Dictionary
– Leadership 2.0 – “Leadership is the ability to attract and empower other’s to achieve their potential.” – Dave Kaufman
– Lessons Learned during the leadership process – Purpose and people first.  Clear goals, roles, and process is key to success.  Focus on needs over wants.  Culture trumps strategy.  Structure follows strategy.  Effectiveness over efficiency.
– Leadership 2.0 Principles – Leadership is an Attitude, not a position.  Empowering others requires authority, capability, and motivation.  Everyone has a voice that must be heard.  Face to face generates trust.
– Participation leads to shared meaning which leads to ownership.
– At times, it’s necessary to go slow to go fast.
– Qualities of 2.0 – Everyone sees work as contribution.  Everyone feels valued and appreciated.  Trust throughout the organization.  Everyone knows their role.  Everyone succeeds, everyone benefits.  All seek to understand.
– The importance of personal vision, goal setting, and personal mission.
– Qualities of 2.0 Leader – Never satisfied, better future, continual improvement.  Coach & collaborate rather than control and command.  Garner trust through chemistry, competence, and character.  Humility.  Self confidence to lead.  Seek to understand before being understood.
– Culture changes from the top down.
– 2.0 Leader Keep Daily Challenges – 1. Keep all words positive 2. Learn one new thing 3. Give your best effort

Session #1

Joseph Ours presented “Bad Metric, Bad.”
In this presentation,  Joe looked at common testing metrics and explained why they are not helpful. He discussed the real purpose of metrics and metrics programs.

Key takeaways and “lines”:

– A list of common metrics and how they are commonly used (Counts, Metric (Counts/Counts), Trends)
– Common problems with the metrics – They fail to communicate, they are often inaccurate, they do not measure a control, they are not tied to organizational objectives
– Candy will get volunteers from the audience to participate for demonstrations.
– Challenges with counting – Label does not equal content, lacks reference for context, lack of consistency
– Challenges with Metrics (Measure over Measure) – Implied Derivation and Forecasting, Counts over Counts, Implies Velocity
– Trend is a change in a measure (or metric) over interval w/ three components – Direction/movement, speed/size, cause
– Challenge of Trends – Affected by challenges of counting, Timed Based, Interavls and Activity Pause
– Purpose of Metrics – Measure of Performance, Conformance of Best Practice, Deviation from Goal
– How to Leverage Metrics – Link metrics to goals, Use trends over absolute numbers, Use sort tracking periods, Change metrics when they stop driving change, Account for error and confidence

Session #2

Raj Subramanian spoke on “When Cultures Collide – A tester’s story”.
In his lively presentation, Raj talked about his personal experience coming to the United States and the cultural challenges he faced with communication.

Key takeaways and “lines”:

– Raj has an interesting Texas accent.
– Everyone has different point of view based on where they come from.
– Ask clarifying questions.  Be straight to the point.  Try to summarize conversations to make sure everyone understands.
– Use of words between countries can be interesting.  Interesting comparison of words used for the same meaning between U.S. and India.
– Be careful how other regions may reaction to the word choice in your postings and writings.
– Learn from the culture you are visiting.  As others to review your communication before sending.
– Be careful of using idioms and phrases.
– Interesting book to checkout – When Cultures Collide by Richard Lewis
– The use of body language and tone of voice can drastically change the message being delivered.
– Put yourself in the shoes of the person coming to a new culture.
– How to improve in a new culture?  Get out of your comfort zone, be social, read as much as you can, learn pop culture and sports
– Continually look for proactive and reactive feedback.

Break for lunch.  Time to recharge, network, and get ready for some more presentations.

Session #3

Jim Holmes up was after lunch to talk about “Developer Tester Collaboration”.
In this presentation, Jim talked about how testers can help developers and how developers can help testers.

Key takeaways and “lines”:

– Developers and Testers should spend more talking.  Each can learn from the other.
– Projects will run more smoothly, and you’ll deliver better software to your customers.
– Excellent example of how Lisa Crispin included a tester in every conversation who was working remotely (Laptop + Skype + wheeled desk = Collaboration).
– Developers can help Testers understand the architecture, aid in better automation coverage, provide details of units tests (allowing testers to reduce downstream UI tests and perform more exploratory testing), use backend APIs to help with test data setup, and can help create testable user interfaces (to make automation easier).
– Testers can help Developers with improved acceptance criteria, letting development know tests up front, and providing testing ideas for edge cases earlier for better error handling.

Session #4

Justin Rohrman was up next with “API checks for fun and profit.”

Key takeaways and “lines”:

– An overview of API testing, its purpose, and an intro on REST.
– The differences between checking vs. testing (definitions from James Bach & Michael Bolton along with illustrations).
– We hunt bugs like a shark stalking the ocean.
– API is like D&D – No UI.
– An introduction to 3 tools in Justin’s toolkit w/ demos – HTTPie, Postman, Frisbyjs

Session #5

Shawn Wallace closed out the session with his presentation, “STOP! You’re automating too much! (and other tales from the field)”.
Shawn talked about the practical lessons learned from several years of building solutions that leveraged test automation in both large and small environments.

Key takeaways and “lines”:
– Test automation is a good idea as it saves time and money.
– How do I get Started automating a system that is already in place?  1. Test KEY use cases 2. Test Defects 3. Test new features
– Start slow, over time, the regression suite will build out.
– What should I automate? The behavior of the system. Regression is a side effect.
– Don’t forget your testing triangle.  Test the system at the boundaries.
– Get to know your developers, they like to help
– Look at the code check-ins, see what code is changing.
– Act as a team to discuss the feature and define the acceptance criteria.
– A look at a case study in pain.  5500 scenarios and 550hrs to execute them.  The process to lessen the number of scenarios and execution time.
– Things to think about: How will I run my tests?  What machines?  How are they chained together?  What do I do with all this data?  Do I always run all my tests? Do I need to keep my tests forever?  How will I know if I am successful?
– Don’t be overconfident based on automation.

After a series of sponsor drawings, gifts, and winners, it was time for the Closing Keynote.

Closing Keynote

Gareth Bowles closed out the day with the closing keynote in “I Don’t Test Often…But When I Do, I Test in Production.”
In Gareth’s presentation, he covered the complexities of Netflix, their need to test in production, and some of the way they do so.

Key takeaways and “lines”:
– Netflix is leading Internet television network with more than 57 million members in 50 countries enjoying more than 1 billion hrs of TV shows and movies each month
– Netflix accounts for 34% of downstream U.S. internet traffic
– Over 2 billion requests per day come into the Netflix API and 12 billion request per day to API dependencies, over 1000 devices supported
– The Netflix system is so complex that no single body knows how everything works
– A look into outages that have impacted Netflix over the years
– Because of the complexity and scale of Netflix, it’s not possible to create an adequate test environment, so Netflix will often deliberately test in production to stress the system.
– The Simian Army is used for production testing.
– Chaos Monkey (2009) simulates failures inherent to running in the cloud during business hour.  Randomly terminates clusters.
– Chaos Gorilla simulates an entire zone being down to test multizone redundancy.
– Chaos King simulates an entire region going down.
– Latency Monkey tests degraded instances don’t impact other services.
– Conformity Monkey ensures that rules on instances are in place and emails when issues are encountered.
– Cool stickers were available for pickup form the audience.
– Some of these tools are available at and
– Failure Injection testing and Canary monitoring methods were also discussed.
– Excellent talk, Netflix is doing some cool stuff.