As you build the product, it’s important to test internally and externally to get out of the development bubble. Dogfooding is a quick internal sanity check that ensures the product team stays empathetic to users. On the other hand, beta testing can be more in-depth since it is the last comprehensive user review of your product before you hand it off to be shipped. In this piece, we’ll discuss helpful processes and documents to use for both types of testing.
Eat Your Own Dogfood
Once you’ve iterated your product to the point where it’s usable, testing it with your team is a great low-cost (or no-cost) reality check.
Since the cost of defects increases the longer they stay in the codebase, this feedback loop — known as dogfooding, alpha testing, or user acceptance testing — might even get you some net gains. The importance of dogfooding might best be understood by what happens when you don’t do it. For example, part of Facebook Home’s failure was that its product team used mostly iPhones while the product was based on Android. As a result, the team left out important widgets, docks, and app folders needed by Android users.
At UXPin, we eat our own dogfood. We use JIRA to keep track of bugs and a highly organized Google Docs spreadsheet for each team member to create a “user diary” of good experiences, annoyances, and times to complete certain tasks. We also use UXPin to prototype new iterations of UXPin (a very meta UX design process).
Compared to earlier phases, the main difference is that dogfooding may not be done in as controlled a setting. Bug tracking software, user diaries, and alpha test surveys are all great for capturing feedback in this looser environment.
1. Bug Tracking Software
Most of the data you’ll get during dogfooding will likely be bug reports. While we’ve talked about the project tracking capability of JIRA, it truly shines at allowing for easy visibility and collaboration when documenting and squashing bugs.
2. User Diary
As we described in The Guide to Usability Testing, user diaries are great at capturing organic information as someone uses the product freely. Using Google Docs, you can create a simple template and then include open-ended prompts such as:
- Where were you when using the product?
- What tasks did you hope to achieve? How long did it take you?
- Describe something the product did to make your life easier.
- Include a screenshot of something that frustrated you.
3. Alpha Test Survey
Using a free site like Survey Monkey, you can create a survey to gauge how team members feel about the usability and functionality of certain features and the product as a whole. This sample survey captures both qualitative and quantitative information.
According to Luke Freiler, CEO of beta testing company Centercode, the goal of an alpha test is to find bugs, crashes, and missing features so tailor your survey questions accordingly. The qualitative and quantitative data from an alpha test will also set the benchmarks for the product and emulate testing done with ~80% of real users.
Now that you’ve fed the dogfood to your own team, it’s time to try it with real users in a beta test.
Feed Your Dogfood to Others
Unlike an alpha test which is designed to improve product quality, a beta test is designed to incorporate customer feedback into the product.
As we described in our Guide to Minimum Viable Products, logging and then acting on customer feedback is one of the lowest-cost ways to prevent feature bloat. Since some bugs may still be discovered since alpha testing won’t cover all edge cases, JIRA can still come in handy for documentation. Similarly, the surveys you ran during internal testing can also be adapted for real users. Below, we’ll look at how other tactics like beta selection criteria, qualification surveys, and user forums can supplement your beta test.
1. Beta Selection Criteria
Before finding applicants, it’s important to prioritize the must-have qualifications, the nice-to-have qualifications, and subjective qualifications for testers. This will help you form a “scorecard” which you can use to filter applicants after they complete your qualification survey. Centercode has a simple selection checklist in their Beta Recruitment Kit to get you started.
2. Beta Qualification Survey
Whether your beta test is public or private, you will want a send a survey to potential testers to ensure they fit the user profiles outlined in your PRD. Centercode’s Beta Recruitment Kit also contains a quick qualification worksheet survey which you can then translate into a live survey on Survey Monkey. You can also reference this live example from McAfee or the sample below.
In our Guide to UX Design & Process Documentation, we also explain how to create different surveys to improve your product after launching.
3. Beta Testing Forums
Using a forum not only fosters earlyvangelism by building a community, it allows your team to actually respond to beta testers. Beta testers are volunteering their time because they believe in the product, so it’s important to make them feel appreciated. Furthermore, a forum will serve as a large-scale user diary of sorts for you to track the experiences of all beta testers.
Use What Works and Scrap the Rest
Building your product is an ongoing process and the last thing you probably want to do is throw more paperwork into your sprints.
But a certain level of documentation is necessary to keep some order in all the chaos. User requirements coming from product management need to be translated. Dependencies among different technical entities have to be understood. And internal and external testing feedback must be captured to justify expensive-but-necessary changes. In between all this, you’ll need to answer stakeholder questions like “How is everyone staying on the same page?” and “How will we realize our goals in the set time limit?”
The templates and tactics we’ve listed all serve as helpful reference points as you prepare for the ultimate product test — your launch date.
For more smart ways on incorporating documentation into the design process, download the Guide to UX Design & Process Documentation. Expert advice is featured from Aarron Walter, Laura Klein, Ian McAllister, and dozens others. Visual examples are also shown from companies like Vurb, MailChimp, Apple, Google, and many more.
