Preventing Bugs: The Dark Matter of QA

Notes on a roundtable discussion at GDC 2019

  • Run by Chris Rando (@Rodneysquint)
  • IGDA QA community
  • QA Mixer first floor of Marriot Marquis
  • How do we articulate how QA prevents bugs
  • Bug prevention
    • Doing acceptance testing—QA sets the criteria to decide when something’s done (rather than just doing some testing when the devs say it’s done)
    • QA involved in game design documentation process
      • Identify things that could go wrong; helps devs think through this stuff
      • Help devs design things to be testable
        • QA can advise how they would design the feature to prevent the bugs
        • Account for the potential problems now
      • Have test plans at the beginning of the project… not the end!
        • Spreadsheet for each component
    • Have QA teach devs what sort of things they should be doing
    • Rather than just addressing bugs, we want to find it before it even gets to the stage where you could write a JIRA ticket
      • As early in the dev cycle as possible—before you write the bug is the best time to find it
    • Integrate the community and social media teams early too—outside perspective from people who might know better what people are going to do with your code
    • Produce tools early that validate data; prevent bugs introduced by bad/unexpected data
    • Share your criteria/testing content with the dev team so that they can test themselves
      • May want devs to generate the test content for you
    • Bug prevention matters because shipping bugs is risky; makes you look bad
      • QA is one aspect of risk management
      • Ask yourself: if such-and-such bug happens, what are the costs associated with that?
        • Can you give yourself a way to fix that bug if it does get introduced
    • Define quality as a team
      • What does that mean for how you build and test
      • What are our quality targets?
  • How you prevent bugs
    • Schedule things far enough ahead of time that you can either fix the bug you find or mitigate it
    • Help devs understand how bugs get introduced (challenge their assumptions, point out their blind spots)
    • Make sure you’re making the right feature for your users
    • Communicate with engineering to get the test data you need (often creating this test data finds bugs)
    • May need debugging “cheats” to isolate certain features
    • QA needs to receive a design document so they can understand what features are supposed to do
    • Have a tool that can send devs back the current game state to reproduce bugs
    • Empower QA to use dev tools/art tools to isolate issues
    • Gather telemetry from playtests
    • At the end of a project, QA gets ownership over check-ins, decide what goes in
      • Devs have to give a risk assessment on what they want to get in
    • QA has to have a focused, specific test for every branch before it gets merged in
    • Figure out where bugs can hide in pushing a new release
      • E.g., data changed at the same time as code; they got tested individually, but not together
    • Collective quality ownership: devs aren’t off the hook until tests pass; might be motivated to help put together test plans
    • Actively manage knowledge across the project—help people understand edge cases
    • Loop QA in on deliberate changes that are happening so that they don’t file bugs expecting the old behavior
    • Retrospectives to look at bugs (or categories of bugs) that keep coming up
      • Why do these bugs crop up? (Dev not understanding something?)
      • Look at changes in processes/tools that would help prevent the bugs
  • An introduction to testing (how to choose edge cases to test)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s