Automated Testing Process Roundtable

Notes from a roundtable discussion moderated by Andrew Fray at GDC 2019

  • For more, see
  • First steps getting started with testing
    • First decide what you want out of testing; if you want a rock-solid engine, need unit tests (despite it being hard on a legacy engine)
    • Unit testing
    • Analyze how long it takes you to go through a checklist of manual tests; see what you can automate there (prioritized by time)
    • Focus on tests that never have false positives early on
    • Identified places where you could save money, make your testers happier
    • “How much time can I cut from QA tester’s?”
      • Answer: None. But you can instead put them working on better, more intelligent tests.
    • Mandate tests for each bug fix
    • Track the build success rate: how many times do 
    • Start with AI running around a level and making sure they don’t get into problems (e.g., do they violate assertions)
  • How to choose which tests get run all the time, versus what only get run pre-merge or something?
    • Sea of Thieves has a specific branch that just runs the tests 24 hours a day: are there any tests that are unreliable?
      • If so, remove them from the normal daily tests.
      • Are there tests that have just never broken? Consider removing those from daily tests
  • On a legacy code base, should you do integration testing first or unit testing in the engine?
    • Identify safety-critical systems and test those first
    • Start with the ones that have had bugs in the past
    • Integration/end-to-end tests get more coverage; if you have to choose one, do that
    • Don’t write unit tests around legacy code unless you need to refactor it, lest you fail to completely understand the system (and/or locking in behavior that isn’t really critical)
    • Refactoring is the best time to write tests
    • Prioritize unit tests of code that is used a lot or changed a lot (fundamental stuff)
  • Mocking legacy code in an engine
    • Mock using real data—TCAS data; do you turn at the right speed, etc.
    • Use manual QA tester to create reference data, use that as mock data
    • Use AI to fly the planesIf you need to have more than 1 mock, script the entire game rather than adding more mocks—instead, do a global test of the whole game focused on that specific area
  • Inventive uses of automated testing system
    • Run the same test repeatedly with a bunch of assertions, use build farm to see if your assertions hold
    • Use automated tests for benchmarking
    • Golden image testing: diff between a reference image and the image generated from the current build; can go back to tech artist and ask if this should have changed
    • Tests-as-benchmarks useful for artists to help them understand if what they’re doing is too slow
  • Who should be responsible for fixing tests that break over time?
    • If you made the feature change, doesn’t matter what level, the checkin won’t be allowed if you broke any tests
    • Ownership of tests is important—if you write the test, put your name on it.
      • One person or team responsible per test; those are the people who get notified of breakages
      • Ownership gives people a first point of contact when someone makes a change that breaks a test they don’t understand
    • Regardless of who wrote the test, it should be written in a way that they can identify what’s going on
    • When a test breaks, you may want to ask if it’s still a test you want, if it’s still testing something valuable
  • Reviewing results: when the newness of the testing system fades, people started ignoring emails—how do you keep people keep using the tests?
    • Dashboard showing build status
    • Simplify or anthropomorphize your results: a very angry face, very happy face
    • @-mention responsible persons when they break the test in Slack rather than sending an email
    • Make sure when people check in tests that they write some rationale behind it, so that you understand why it was worth having
  • Good resources for getting started with testing
    • Changed from having automated tests run by devs to having QA run them
      • QA would request automated tests for new stuff
    • Gary Bernhart talks on YouTube
    • Not every talk/paper on the internet will apply to you
    • Ministry of Testing
    • Michael Feathers’ Working Effectively with Legacy Code
    • Decompose your problems into pure functions (trivial to test)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s