Notes from a roundtable discussion moderated by Andrew Fray at GDC 2019
- For more, see autotestingroundtable.com
- First steps getting started with testing
- First decide what you want out of testing; if you want a rock-solid engine, need unit tests (despite it being hard on a legacy engine)
- Unit testing
- Analyze how long it takes you to go through a checklist of manual tests; see what you can automate there (prioritized by time)
- Focus on tests that never have false positives early on
- Identified places where you could save money, make your testers happier
- “How much time can I cut from QA tester’s?”
- Answer: None. But you can instead put them working on better, more intelligent tests.
- Mandate tests for each bug fix
- Track the build success rate: how many times do
- Start with AI running around a level and making sure they don’t get into problems (e.g., do they violate assertions)
- How to choose which tests get run all the time, versus what only get run pre-merge or something?
- Sea of Thieves has a specific branch that just runs the tests 24 hours a day: are there any tests that are unreliable?
- If so, remove them from the normal daily tests.
- Are there tests that have just never broken? Consider removing those from daily tests
- Sea of Thieves has a specific branch that just runs the tests 24 hours a day: are there any tests that are unreliable?
- On a legacy code base, should you do integration testing first or unit testing in the engine?
- Identify safety-critical systems and test those first
- Start with the ones that have had bugs in the past
- Integration/end-to-end tests get more coverage; if you have to choose one, do that
- Don’t write unit tests around legacy code unless you need to refactor it, lest you fail to completely understand the system (and/or locking in behavior that isn’t really critical)
- Refactoring is the best time to write tests
- Prioritize unit tests of code that is used a lot or changed a lot (fundamental stuff)
- Mocking legacy code in an engine
- Mock using real data—TCAS data; do you turn at the right speed, etc.
- Use manual QA tester to create reference data, use that as mock data
- Use AI to fly the planesIf you need to have more than 1 mock, script the entire game rather than adding more mocks—instead, do a global test of the whole game focused on that specific area
- Inventive uses of automated testing system
- Run the same test repeatedly with a bunch of assertions, use build farm to see if your assertions hold
- Use automated tests for benchmarking
- Golden image testing: diff between a reference image and the image generated from the current build; can go back to tech artist and ask if this should have changed
- Tests-as-benchmarks useful for artists to help them understand if what they’re doing is too slow
- Who should be responsible for fixing tests that break over time?
- If you made the feature change, doesn’t matter what level, the checkin won’t be allowed if you broke any tests
- Ownership of tests is important—if you write the test, put your name on it.
- One person or team responsible per test; those are the people who get notified of breakages
- Ownership gives people a first point of contact when someone makes a change that breaks a test they don’t understand
- Regardless of who wrote the test, it should be written in a way that they can identify what’s going on
- When a test breaks, you may want to ask if it’s still a test you want, if it’s still testing something valuable
- Reviewing results: when the newness of the testing system fades, people started ignoring emails—how do you keep people keep using the tests?
- Dashboard showing build status
- Simplify or anthropomorphize your results: a very angry face, very happy face
- @-mention responsible persons when they break the test in Slack rather than sending an email
- Make sure when people check in tests that they write some rationale behind it, so that you understand why it was worth having
- Good resources for getting started with testing
- Changed from having automated tests run by devs to having QA run them
- QA would request automated tests for new stuff
- Gary Bernhart talks on YouTube
- Not every talk/paper on the internet will apply to you
- Ministry of Testing
- Michael Feathers’ Working Effectively with Legacy Code
- Decompose your problems into pure functions (trivial to test)
- Changed from having automated tests run by devs to having QA run them