Notes on a roundtable discussion at GDC 2019
- Run by Chris Rando (@Rodneysquint)
- IGDA QA community
- QA Mixer first floor of Marriot Marquis
- How do we articulate how QA prevents bugs
- Bug prevention
- Doing acceptance testing—QA sets the criteria to decide when something’s done (rather than just doing some testing when the devs say it’s done)
- QA involved in game design documentation process
- Identify things that could go wrong; helps devs think through this stuff
- Help devs design things to be testable
- QA can advise how they would design the feature to prevent the bugs
- Account for the potential problems now
- Have test plans at the beginning of the project… not the end!
- Spreadsheet for each component
- Have QA teach devs what sort of things they should be doing
- Rather than just addressing bugs, we want to find it before it even gets to the stage where you could write a JIRA ticket
- As early in the dev cycle as possible—before you write the bug is the best time to find it
- Integrate the community and social media teams early too—outside perspective from people who might know better what people are going to do with your code
- Produce tools early that validate data; prevent bugs introduced by bad/unexpected data
- Share your criteria/testing content with the dev team so that they can test themselves
- May want devs to generate the test content for you
- Bug prevention matters because shipping bugs is risky; makes you look bad
- QA is one aspect of risk management
- Ask yourself: if such-and-such bug happens, what are the costs associated with that?
- Can you give yourself a way to fix that bug if it does get introduced
- Define quality as a team
- What does that mean for how you build and test
- What are our quality targets?
- How you prevent bugs
- Schedule things far enough ahead of time that you can either fix the bug you find or mitigate it
- Help devs understand how bugs get introduced (challenge their assumptions, point out their blind spots)
- Make sure you’re making the right feature for your users
- Communicate with engineering to get the test data you need (often creating this test data finds bugs)
- May need debugging “cheats” to isolate certain features
- QA needs to receive a design document so they can understand what features are supposed to do
- Have a tool that can send devs back the current game state to reproduce bugs
- Empower QA to use dev tools/art tools to isolate issues
- Gather telemetry from playtests
- At the end of a project, QA gets ownership over check-ins, decide what goes in
- Devs have to give a risk assessment on what they want to get in
- QA has to have a focused, specific test for every branch before it gets merged in
- Figure out where bugs can hide in pushing a new release
- E.g., data changed at the same time as code; they got tested individually, but not together
- Collective quality ownership: devs aren’t off the hook until tests pass; might be motivated to help put together test plans
- Actively manage knowledge across the project—help people understand edge cases
- Loop QA in on deliberate changes that are happening so that they don’t file bugs expecting the old behavior
- Retrospectives to look at bugs (or categories of bugs) that keep coming up
- Why do these bugs crop up? (Dev not understanding something?)
- Look at changes in processes/tools that would help prevent the bugs
- An introduction to testing (how to choose edge cases to test)
