You can ignore this article unless you want your software development organization to grow, or go faster, or both.
That might seem provocative, but the truth is that many organizations are fine as they are. They are satisfactorily people-driven, with reliance on experts. That is how most small teams operate, and it works well as you grow as long as you have enough experts and you’re ok with the time it takes for the experts to figure things out.
But read on if you’re concerned that your current processes might not scale, and thus you have or anticipate a need for an alternative to people-driven problem identification.
Agile Challenges Quality
Teams that are very agile strive to go faster and release software more frequently, the theory being that incremental improvements increase customer satisfaction and allow the team to change direction more quickly based on customer feedback. The holy grail of agile is continuous delivery, where incremental improvements can be delivered to production “immediately”, which might result in many releases each week or even each day.
To go quickly, especially, as the team size and the software code base grows, a development team must increasingly rely on automation. Automated build systems run automated tests and when tests pass there is automated deployment. The team has to rely on test automation to move fast, because there isn’t time to manually test everything.
But as the team size and the code base size grow, it gets increasingly hard to know that you have covered all areas that need testing. Recognizing this, many advanced large agile organizations like Facebook and LinkedIn also then rely on A/B testing in production, meaning that they release new software to a subset of users and monitor the results. If bugs are found they can rollback and fix, and if no bugs are found they can complete rollout. Essentially, real users become the last line of QA.
The questions that one must ask in these situations are:
• Do you know where your key quality risks are, and are they acceptable to release?
As software development teams become more agile, with greater and greater reliance on automation, the role of the QA team will inevitably change
• Can you detect small problems, which can add up over time, as well as big ones?
• When problems are found, how quickly can you identify the root cause?
Implementing Quality Tracking
If you are trying to be very agile, especially at scale with a large development team or a large code base, the need for excellent quality tracking and reporting becomes clear. Quality tracking tells you what has been tested, the results of testing, and when issues are found then good quality tracking can tie the issues back to features and even to specific code changes and the developers involved. In this way, quality tracking becomes your key to quickly assess risk and to speed up problem resolution. Quality tracking is also an excellent tool to help assess the overall health of your software development team.
Data that can be captured and reported in your quality tracking system might include:
• Features in each release (from your agile project tracking system)
• Code changes made for each feature, including when and by whom (from your source code repository)
• Tests created and run for each feature, along with results (from your test case management system and results gathered from your test runners)
• Defects found in testing or production, along with severity and association to specific features (from your defect tracking system and/or your customer support system)
• Results from other test or production monitoring systems, such as performance monitoring (from your performance monitoring tools)
Gathering this data, you will essentially be creating a Business Intelligence (BI) hub for software quality tracking. As stated above, while any software development team could benefit from such a BI system, it is imperative for larger teams trying to be very agile, because without such a system you are flying blind. While for very small teams, or teams that are not concerned with being very agile, it might be too much work to build or deploy a BI quality tracking system, for teams that really need it the investment in time and tools will be more than offset by the time savings and risk reduction it will yield.
There are many open source and commercial BI systems that could be used to ingest and report data from the various systems involved. As the need for such systems becomes clearer, it is likely that solutions specifically designed for this purpose will also appear. One such solution available now is qTest Insights from my company QASymphony. Our Insights product leverages integration with JIRA and other agile management tools and provides you with quality, coverage, and velocity analysis along with the ability to create custom dashboards.
Analytics Will Transform QA
As software development teams become more agile, with greater and greater reliance on automation, the role of the QA team will inevitably change. In the future, analytics will be an important tool to make the QA team more efficient and effective. Analytics from test results and from production, along with historical analysis of past code changes and results, will help inform and guide the QA team towards the areas that need more attention. Armed with this knowledge, the QA team will also be positioned as trusted experts to advise the entire software development team on steps that can ensure better quality. Teams that begin by implementing a BI quality tracking hub today will be one step closer towards realizing this future.