Up until the latter half of last year, we, the QA team at GameChanger, dedicated many hours to manually running regression tests on our apps. We found ourselves caught in a time crunch before weekly releases, falling behind in testing new features, and struggling to maintain a high standard of quality for our products.
An obvious solution would have been to create a suite of automated tests so that we could forego manual regression testing. Although we are starting to work on this as part of our complete test strategy, we still have a need for manual testing for many reasons. For instance, automated tests may not pick up on certain types of UI bugs like a flashing screen and they certainly cannot provide usability feedback.
So we came up with the idea of hiring manual testers for regression testing…
At first, we hired a company that supplied us with on-demand testers who were randomly assigned to us when we needed to run our regression test suites. This meant that tests were always executed by someone who had never seen our apps before. However, without a deep understanding of functionalities within our apps, these testers were unable to effectively test and find bugs. This solution was a swing-and-a-miss.
Assembling the Ultimate Team of Part-Time Testers
We needed our QA team to focus on creating test plans, testing new features, and strategizing and implementing ways to attain and maintain higher standards of quality. So we set out to hire our own team of part-time testers to run our daily regressions and beta regressions.
When we first thought about hiring testers, we were constantly weighing the importance of hiring someone with solid QA experience versus someone who was very familiar with our apps. At the time, we were so impressed with the enthusiasm of GameChanger users applying for the role, we ended up hiring avid GameChanger users with at least some QA experience.
Contrary to the outsourced testers, our newly hired testers were real users who were personally invested in our products and able to provide real-world feedback. They were also easy to train, able to run tests faster, quickly find release-blocking bugs, and were more adept at finding complex bugs. The idea of having real users to test products was further solidified by a talk at the STAREAST Conference last year describing how other companies found success in hiring their loyal customers as testers.
How our QA Team has Benefitted from our Part-Time Testers
After a couple months of tweaking the job post for part-time testers and interviewing candidates, we were very fortunate to find a strong team of GameChanger users with QA experience. At first, we had them run our daily regressions and beta regressions. In time, we found that our testers were so proficient at finding bugs in our apps that we started to give them new features to test as well. If a new feature is ready to test at the end of the day, we are able to have our testers run test cases for the feature that evening. Any bugs that are found are ready for our engineers to work on the next morning, increasing the velocity of our product teams.
While it is still our job as QAs to thoroughly test features, having the features tested sooner and having another set of eyes on them increases the speed and quality at which we can release features. We can also get feedback from our testers who are real users before features get shipped.
Another huge benefit of having a team of testers is that we gain more coverage on a spectrum of devices and OS versions. In the past, an in-house QA would run an entire regression by him/herself on one device on one OS. We would have needed more QAs to provide the same coverage we have today. Now, we have testers testing our apps on various devices and OSs on a daily basis. The risk of our customers finding bugs related to specific devices and/or OSs has greatly reduced.
Getting to Know our Remote Part-Time Testers
A few months ago, a group of women at GameChanger, including myself, decided to participate in the Grace Hopper Celebration conference by submitting topics to the conference that focused on women in technology. As we discussed potential topics to submit, I thought about doing some research on our testers on the basis that they were mostly women. Gender aside, I wanted to better understand what drives our testers to put forth incredible amounts of effort every day to ensure that our apps adhere to high standards of quality.
Getting a more holistic picture of our testers’ backgrounds and their thoughts could help us understand both the positives and negative aspects of their roles at GameChanger. The hope was to use these findings to help us optimize and enhance how we work with our testers. So I sent out a questionnaire to our team of part-time testers with some open-ended questions and found some common themes in their responses.
Some facts about our testers:
- All but one of our testers are users of our apps
- All but one of our testers are women
- All of our female testers have families with children
General themes that came up in their responses to the questionnaire:
- All of those testers were drawn to the role in part because they love our apps and are long-time users of it
- Only one tester does not have an additional full-time or part-time job
- Being able to spend more time with family was evident in the women’s responses
- Most testers mentioned that lack of physical face-to-face contact was a negative but daily communication via messaging was a plus
- The women:
- One of the themes that came up for the women was an increased sense of productiveness, accomplishment, and self-importance
Some of the quotes from our testers’ responses that really struck a chord with me:
“This position had nothing to do with the hours/money I really like GameChanger as an app for softball and thought it would be fun to get a job there”
“I love the app and wanted to be part of something new (change it up a little from my normal job)”
“I like being productive, using my brain, actually speaking to adults once in awhile, and earning an income.”
“…after 17 years in the same job you get comfortable and you get scared that you couldn’t leave that job and find something else (not capable enough etc.), but know that I really do have the experience and knowledge and work ethic to gain a new position was pretty powerful (personally). This may sound odd, […] but it gave me the little extra spark that (if something was to happen with my regular job) I would be ok.”
“I had been a home-based employee the last 8 of those years […] I have a child with epilepsy and when she was diagnosed 7 years ago we decided that it would be best if I stayed closer to her schools in the event something happened.”
Changes in How We Work with Our Testers
Since acquiring the responses to the questionnaire, we have made conscious decisions as to how we work with our testers.
Being Sensitive to Personal Schedules
The most obvious theme from the responses was that all our testers have family and/or other job obligations. We try to be cognizant of their busy schedules when we communicate with them and when we schedule tests for them to run. We are also flexible as to the time of day they test and if they need time off.
To help our testers plan their days, we have set up a calendar with daily assignments that stays quite consistent from week to week. We have found that once we implemented a consistent and transparent schedule that they could follow, it seemed to have lessened anxiety around balancing testing with their other jobs and personal tasks such as picking up their kids from school and going to their kids’ ball games.
Meeting with Testers
Lack of face-to-face contact was one of the negative aspects of working remotely that we also wanted to address. After being aware of this, we have made a conscious effort to either meet with our testers on video chat or by phone in order to keep our lines of communication more open and personable. In addition, our testers we use Slack to keep us connected at all times.
Changes to How We Select our Testers
When the time comes to hire additional testers, we will be focusing on candidates who are users of our apps. This approach has already proven to be a huge benefit to our company. Our testers are always excited to test out new features and promote them to their friends and family. They truly care about maintaining high standards for our products as they use them regularly to track teams that their families are involved in.
Maintaining Good Relationships with Our Testers
Companies should show appreciation to all their employees and we wanted to make sure that our testers felt valued even though they work remotely. We want them to be happy in their roles, continue to care deeply about our products, and be proud to work at GameChanger.
Showing appreciation for our colleagues is already part of our company culture for those of us who work in the office. For our remote workers, we try to convey our appreciation on a daily basis through our communication. We try to check in on how they are doing personally and how they feel about our processes and the workload that we give them. We also try to respond to any of their needs and concerns as quickly as possible. In essence, we treat them as part of the GameChanger family.
Conclusions
While automated test suites are something we are currently looking into building, manual testing is a necessity for our apps due to complex user interfaces and our company’s extremely agile environment.
We are happy to say that we now have a strong group of testers. Our testers have contributed to much lower crash rates in our apps and a lower incidence of hotfixes. We have also seen a significant reduction in customer complaints about bugs. Our QA team is now able to focus on releasing quality features and implementing strategies that help maintain high standards for our apps.
Since our testers are real users of our products, they are also great promoters of our products within their communities. User feedback from them is also very valuable to us.
Even more rewarding is that not only do our testers benefit our company, but they also find that their roles benefit themselves and their families.
We are so grateful to have such a great team of part-time testers and hope that we can continue to work with this amazing group of people.