Tooling for Small Team Workflows

Notes on a talk by Irina Z. from Bungie

  • Bungie uses multidisciplinary small teams of 4 to 9 people
  • Many small autonomous teams working on dedicated feature branches
    • See “Pipeline support for feature branches in destiny” talk for more
  • Setting stability expectations
    • Previously had exhaustive testing and manual sign off to push to the single shared branch 
      • Changes could take days to get out of a small team 
      • Wanted to get changes shared out between teams faster
      • Needed automated testing a background process
    • Don’t want teams to diverge too far
      • Consolidate changes between teams automatically onto master
      • First runs auto tests
      • Build verification team runs manual tests of master as well; fixes get checked in on their branch off master, and fixes get committed to their branch and merged back to master
        • BVT branch is their release!
    • Each team sets requirements for stability before pushing to their shared branch
    • Branch owner is the person made responsible for the branch—responsible for fixing test failures, integration conflicts, build failures, etc.
      • Can control automated testing requirements for the build
      • Schedules builds
      • Keeps branch up to date with other teams
    • Created one stop UI for viewing bugs reported on the branch, build status, pull requests, etc.
      • They use PerForce
    • Testing levels before a change goes out to the team:
      • None
      • Automated only
      • Automated plus manual (slowest to share changes)
    • Categorized tests, all run through NUnit, into unit tests, integration tests (chosen per branch based on applicability)
      • Categories: required for anything on main, versus required on this branch
      • Platform it applies to
      • Optional (allowed to fail and continue checkin)—useful for tests you just wrote and aren’t sure they’re correct everywhere; also saves the committer from a notification
        • Automatic retries (only alert after multiple failures)
    • Team am has to investigate all issues introduced on their branch, at least far enough to prove the break came from another branch
    • Branch owner sets the recommended build (includes content recommendation!) for their branch
      • 3 levels of testing
        • Untested
        • Automated tests approved
        • Human approved
  • Scheduling builds
    • At least 3 types of builds:
      • Continuous build of the branch itself
      • Continuous integration into master
      • Periodic content build, which updates the recommended builds
    • Set per branch who gets notified first about failures, and who gets notified if the problem persists too long—often branch owner
  • Keeping up with changes
    • In big integration merges, wanted a list of all the changes going in to master (same when build verification team pulls from master)
    • Every time build verification team pulls, they get the complete commit log and note all new changes coming in
    • Safe cherry picking between branches via finding common roots for all files that have been touched along with that file
  • Branch owner skills
    • Source control
    • Build distribution
    • Automated testing
    • Coding
    • Debugging both test and build failures
    • Paying attention to automated communication—don’t send too much such that people tune out, make it clear who is responsible for what, make it clear what the urgency is
  • Tooling UX feedback
    • Recent changes and issues being gathered in one place was valuable—used to be manual for your branch

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s