We at 99tests have seen more than 23,000 bugs being logged by over 5000 testers in the past two year. Here are some of key finding we have uncovered over the past couple of year. Specially the effect of a Crowd Testing team on finding high quality bugs in a short time.
When does the performance of a Crowd Testing team over take the performance of a normal QA team?
We have seen that a Crowd Testing team has to be composed of a minimum of 20 testers of diverse backgrounds to see the positive effects of the Wisdom of Crowds. Now, in the book Wisdom of Crowds by James Surowiecki states that for a group of diverse individuals to come to an accurate prediction or decision, four conditions that need to be met, first is diversity of the crowd, the second is independence in thinking, third is decentralization and the fourth is aggregation.
Now coming back to Software Testing, one of the inherent issues with in house QA teams, is familiarity with the product, this means that since they have seen the product evolve over time, they become resistant to looking at the product with a fresh set of eyes, so traditional QA teams are very good at executing test cases and letting the developers know what works. This is where a Crowd Testing team out performs in finding bugs, that a new user would find. The second key trait of Crowd Testing Teams is the team size. Once the number of testers is two to three time the size of normal QA team, we start seeing the kinds of bugs which only a crowd team can produce. The third key point in forming a crowd testing team, is the diversity of the testers, i.e. the team of 20-30 testers needs to be composed of very highly skilled exploratory testers, average testers and novice testers. This team dynamic ensures that a high bar is set for the quality of the bugs by top testers and the team follows their lead in logging high quality bugs. The third key trait for good crowd testing teams is time pressure, so having a fixed time in which to find the maximum number of bugs is important condition for getting the best results. The fourth key trait is the incentives or rewards need to matched to the testers who find the most number and high quality bugs.