Crowd is the ‘in’ word in today’s highly interconnected equally crowded & world. From crowd funding of startups to crowd sourcing ideas, we have come a long from managing isolated silos of knowledge to harvesting knowledge from across the world, thanks to innovation in technology and emerging openness among corporations to experiment with people outside of their comfort zones
Of late, crowd testing has been an important area where the collective wisdom of the crowd has proven more effective & efficient than in-house testing of software apps. With more software components being exposed directly to clients, quality of software becomes paramount in determining customer loyalty & retention. Software testing today is the critical cog in the wheel of software development which can make or break businesses. Crowdtesting as a methodology that leverages the power of a vibrant independent testing community across demographics ranging from a few dozens to few thousand testers with one passion – Find & fix bugs
More than software components being more client facing, it’s the adoption of trends like cloud computing & BYOD (Bring Your Own Device) that has ensured that software is now consumed across mediums, channels, devices & I.T system configurations. In such a scenario, software has to be compatible across a matrix of configurations. In-house testing would fail in such a case where critical elements may be overseen due to lack of resources & time
With crowdtesting, the app is subjected to real world scenarios in advance which make refactoring a stress free experience. In a well organized crowdtesting environment, usually all bugs are spotted & logged well before app release resulting in a bug free app & a happy customer
Despite these winning points for crowdtesting, there are a few points that an enterprise software firm should consider before hiring a crowdtesting partner with a ready to deploy army of crowd testers
1. Data sensitivity, authorization & confidentiality
How much of my data is sensitive & can cause severe breach of privacy?
How do I know if the tester isn’t a shady hacker?
How many levels of access does the crowdtesting platform have?
Do I have options to recall & reset the code to pre- test scenarios?
All these questions should be raised with the crowdtesting platform to understand the safety measures they undertake for glitch free delivery
2. Is their testing process robust?
For enterprise products, the biggest challenge would be to draw a common ground between exploratory testing & inside out approach of crowdtesting. A good crowdtesting platform would rope in people who have extensive experience handling apps with multiple levels of complexity. Crowdtesting methodology is typically modeled on existing systems like Agile, and hence adopting it to a larger enterprise wont be tedious. Compare your internal testing methodology with the approach the crowdtesting platform provides and take a call
3. Learn from the masters
‘Big bounty’ programs have always been a hit among ace testers. It’s a program where external bug reporters work as your internal security experts and get paid for every critical bug they find.
Researchers at University of California have categorically suggested that it’s worthwhile to hire external crowdtesters to do the job of bug hunting.
The study also suggests that major players like FireFox, HP, Google chrome & Facebook have been actively running crowdtesting programs & rewarding best performers & that every software company must have a round the year reward program for crowdtesters
“Our vulnerability reward programs have been very successful in helping us fix more bugs and better protect our users, while also strengthening our relationships with security researchers,” – Google security research team
4. How do I choose my testers from the crowd?
The backbone of any crowd platform is the strong community ecosystem. Watch out if the platform has an active forum where testers share knowledge & rate each others’ work. An active community is also an indication of transparency. Also ensure that the platform itself isn’t dictating community rules & allows the community to have a free hand in sharing content they like
Also, watch out if the platform has a mechanism to rank testers. In a highly competitive world, peer ranking is seen as a real & accurate indicator of one’s capabilities. You can ask the platform to choose crowdtesters based on your requirements too. The more transparent the platform is, the more assured you can be that your work would be done in a hassle free manner
5. Leveraging based on risk & priority
We recommend crowdtesting for apps that have high Defect exposure factor (DEF). DEF is defined as
DEF = (No of releases per year * Average number of daily critical stakeholder visits.
These stakeholders could be your customers, your customer’s customers whose experience can make or break the app’s sale-ability
6. Is their pricing flexible?
TaaS – Testing as a service. Crowdtesting platforms typically operate in the cloud. Their flexibility helps them to be flexible with pricing as well. Go for a platform that has a hybrid pricing model of charging minimal price for accessing the platform, generating reports etc & a performance linked pricing that links payouts to no of bugs delivered
7. Watch out for new trends
There are niche vendors who are carving a market for themselves in the crowdtesting market. Some venture into specialized testing methods like performance testing. We would see a scenario where crowdtesting would be synonymous with community testing.
Since crowdsourcing is getting crowded, crowdtesting platforms need to stand apart from competition. Watch out for crowdtesting platforms that have any of the following initiatives
- Crowdtesting contests or testing hackathons
- A vibrant idea sharing community
- Platforms that share strong & relevant content that makes you smart
- Emergence of social media – Give priority to firms that have active social media groups that act as a great storehouse of knowledge & content