Automated Testing Process Roundtable

Notes from a roundtable discussion moderated by Andrew Fray at GDC 2019

• For more, see autotestingroundtable.com
• First steps getting started with testing
  • First decide what you want out of testing; if you want a rock-solid engine, need unit tests (despite it being hard on a legacy engine)
  • Unit testing
  • Analyze how long it takes you to go through a checklist of manual tests; see what you can automate there (prioritized by time)
  • Focus on tests that never have false positives early on
  • Identified places where you could save money, make your testers happier
  • “How much time can I cut from QA tester’s?”
    • Answer: None. But you can instead put them working on better, more intelligent tests.
  • Mandate tests for each bug fix
  • Track the build success rate: how many times do 
  • Start with AI running around a level and making sure they don’t get into problems (e.g., do they violate assertions)
• How to choose which tests get run all the time, versus what only get run pre-merge or something?
  • Sea of Thieves has a specific branch that just runs the tests 24 hours a day: are there any tests that are unreliable?
    • If so, remove them from the normal daily tests.
    • Are there tests that have just never broken? Consider removing those from daily tests
• On a legacy code base, should you do integration testing first or unit testing in the engine?
  • Identify safety-critical systems and test those first
  • Start with the ones that have had bugs in the past
  • Integration/end-to-end tests get more coverage; if you have to choose one, do that
  • Don’t write unit tests around legacy code unless you need to refactor it, lest you fail to completely understand the system (and/or locking in behavior that isn’t really critical)
  • Refactoring is the best time to write tests
  • Prioritize unit tests of code that is used a lot or changed a lot (fundamental stuff)
• Mocking legacy code in an engine
  • Mock using real data—TCAS data; do you turn at the right speed, etc.
  • Use manual QA tester to create reference data, use that as mock data
  • Use AI to fly the planesIf you need to have more than 1 mock, script the entire game rather than adding more mocks—instead, do a global test of the whole game focused on that specific area
• Inventive uses of automated testing system
  • Run the same test repeatedly with a bunch of assertions, use build farm to see if your assertions hold
  • Use automated tests for benchmarking
  • Golden image testing: diff between a reference image and the image generated from the current build; can go back to tech artist and ask if this should have changed
  • Tests-as-benchmarks useful for artists to help them understand if what they’re doing is too slow
• Who should be responsible for fixing tests that break over time?
  • If you made the feature change, doesn’t matter what level, the checkin won’t be allowed if you broke any tests
  • Ownership of tests is important—if you write the test, put your name on it.
    • One person or team responsible per test; those are the people who get notified of breakages
    • Ownership gives people a first point of contact when someone makes a change that breaks a test they don’t understand
  • Regardless of who wrote the test, it should be written in a way that they can identify what’s going on
  • When a test breaks, you may want to ask if it’s still a test you want, if it’s still testing something valuable
• Reviewing results: when the newness of the testing system fades, people started ignoring emails—how do you keep people keep using the tests?
  • Dashboard showing build status
  • Simplify or anthropomorphize your results: a very angry face, very happy face
  • @-mention responsible persons when they break the test in Slack rather than sending an email
  • Make sure when people check in tests that they write some rationale behind it, so that you understand why it was worth having
• Good resources for getting started with testing
  • Changed from having automated tests run by devs to having QA run them
    • QA would request automated tests for new stuff
  • Gary Bernhart talks on YouTube
  • Not every talk/paper on the internet will apply to you
  • Ministry of Testing
  • Michael Feathers’ Working Effectively with Legacy Code
    • Key points
  • Decompose your problems into pure functions (trivial to test)