Release trains.

Mobile apps under constant development are usually released on a fixed schedule. The release train has become common practice. It’s been used by every team I’ve worked on since around 2013.

Separately, continuous delivery is becoming a sensible default across swathes of the software industry. So why do mobile apps require a different approach? Let’s demystify the mobile release train.

Constraints

Mobile app distribution is largely controlled by Apple’s App Store and Google Play, each with their own approval process and mechanism for distributing builds to users’ devices. A simplified release flow involves creating a release candidate, testing that build, then submitting it to the app store.

Release candidate, testing, and submission to the store

Mobile app releases have long and unavoidable feedback loops: 1–2 days for app store review, 1–2 hours for the first users to download and install the new build, and perhaps a day to collect enough data (e.g. crash rates) to confirm a stable rollout.

Ultimately, the new build — often 10s or 100s of MBs of data — must traverse unreliable mobile networks to be installed on users’ devices (or not, depending on their auto-update settings). This means significant lag in release adoption. It’s common to see new builds account for only 70% of sessions a week after release.

Trains

A release train is just a fixed schedule for releases, defining exactly when key steps (like building a release candidate) will happen. It provides mile markers for a predictable planning cadence across multiple teams. Ad-hoc releases simply don’t scale when there are multiple teams contributing to the codebase, each with separate plans to release their changes in the same app build.

Code cut is followed by stabilisation and rollout

Each new feature targets a specific release. If development is running late, the train leaves as scheduled and the feature gets released on the next train instead. (See Trunk-based mobile for version control practices that support this workflow.) Regular releases lower risk by keeping the change set small. Releasing regularly, even without new features, provides fast feedback to validate bug fixes and refactors.

Development is continuous with staggered releases

One or two-week release cycles are common in the industry. SoundCloud has shared details about automation and quality tooling that supports their release train, and Facebook has validated the approach at massive scale, gradually moving from monthly to weekly releases over several years.

Definitions

It’s useful to develop common language for the phases of your release train. I’ve found the following terminology to be widely understood.

Code cut

The code cut event really defines the release train schedule (e.g. every Monday at noon). It means code changes are no longer generally accepted for the current release. A series of automated steps typically precedes the generation of a release candidate build:

  1. Create a release branch
  2. Run tests and static analysis
  3. Import translations
  4. Bump version numbers

Ideally this is a non-blocking event, allowing code changes for the subsequent release cycle to start being merged right away.

Stabilisation

A stabilisation phase can involve a variety of QA processes: a beta release, manual regression testing, or release to an internal dogfooding program. It depends on the needs of the team, but the goal is always to find major defects before they end up in production, potentially forever.

Teams should develop guidelines for code changes that can still be accepted during stabilisation. Bug fixes? Of course. But what if the bug isn’t a regression since the last release? What about copy and localisation changes?

Rollout

Once the build is submitted to the app store, it’s all about monitoring. The app stores support staged rollouts, allowing teams to gain confidence in the health of the release with a subset of the userbase.

App store review, staged rollout, and monitoring

When major issues are discovered, it’s usually necessary to generate and submit a new “hotfix” build. This requires another app store review. If the team uses remote feature flags, disabling the defective feature and publishing a fix on the next release train may be an option.

Release captains

Some very large organisations have dedicated release engineering teams. At smaller companies, it’s common practice to assign an engineer to a rotating release captain role. Their responsibilities often include:

  • Perform any manual release steps
  • Coordinate bug fixes
  • Monitor health metrics during rollout
  • Make hotfix decisions

Conclusion

Release trains offer structure to mobile apps under constant development. Smaller releases reduce risk, and releasing more often forces the refinement of the release process. I’ve found a two-week release cycle is achievable in most cases, but moving to a weekly cadence demands significant investment in release automation, which will depend on the size and priorities of the team.

Mobile teams can adopt some continuous delivery practices for internal builds or enterprise apps, but given the overhead of releasing to production, a regular release (at least every other week) is a decent goal.