Continuous integration is widely adopted in backend and web development. The core principles can be applied to mobile development too, but there are some specific challenges to consider. Let’s explore a model for developing mobile apps with a trunk-based version control workflow. This is based on practices I’ve seen work well in growing mobile teams at SoundCloud and Deliveroo.
“Trunk” refers to the main line of the code, such as the default main branch in a GitHub project.
Motivation
Development workflows with long-lived feature branches — including the GitFlow model — delay integration until a feature is completed. On a small team, the overhead of maintaining feature branches is relatively low. Once you have several developers committing to the same codebase with frequent app releases, things can get messy fast.
Merge conflicts
The longer a branch is active, the harder it gets to merge code changes back to a trunk that’s moved on. Even with comprehensive automated test coverage, it’s easy to make mistakes resolving merge conflicts and introduce subtle bugs.
Mobile release cycles tend to be in the range of 1–4 weeks, so if you use a branch to develop a release it will diverge significantly. This means large or frequent merges to reconcile changes intended for the current release with features still in development for the next version.
The work of resolving merge conflicts from long-running branches is time consuming and error prone. An alternative mechanism for isolating unfinished code from production could make the development process much more efficient.
Release chaos
With feature branches, the lifespan of a branch depends on the size of a feature. It could be a single developer adding commits over time, or many contributors branching away from a feature branch and back again.
When it’s time to release features that were developed in isolation, more branches means more coordination overhead and more room for error. In this diagram, there are four different histories of the codebase to reconcile, creating a stressful release process with the potential for integration issues and delays from failing builds.
Going trunk-based
Trunk-based development rejects long-lived feature branches and emphasises frequently integrating code to a releasable trunk. Executed well, it reduces time spent fixing merge conflicts, lets the whole team work from a single source of truth, and derisks the release process.
A trunk-based branching model is a key enabler for continuous integration:
- Code changes are merged to trunk frequently
- Trunk is releasable*
- A build server validates commits to trunk
* Meaning that no unfinished or unstable changes are exposed to users. You can still run a regular mobile release cycle with a stabilisation phase.
Feature flags
The first thing to get comfortable with is shipping unfinished code behind feature flags. These flags can be configured at build-time and baked into the app binary, or configured remotely and initialised based on an API response at app start.
Build flags
Build flags are used for developing new features on trunk and keeping work-in-progress invisible to users. Gradle’s BuildConfig is a neat way to configure static flags for Android apps, but there are plenty of other options including reading flag states from a file or simply defining a set of booleans in code.
Extending this approach, you can set default feature flag states based on the build configuration. A feature that’s almost finished but lacking polish can be enabled in debug builds for internal testing but remain disabled for release builds.
Another common strategy is to configure feature flag states at build-time and allow them to be overridden at run-time via a debug menu (which is only available in debug builds, or perhaps enabled for staff accounts).
Branch by abstraction
Not all changes can be guarded by a simple if/else statement. Large refactors, like completely replacing legacy components, can be achieved with the branch by abstraction pattern.
Introducing an abstraction layer above the old and new components allows switching between two implementations with a feature flag. Once the work is completed, the old component can be removed.
At this point, the feature flag can also be retired. Removing the abstraction itself is optional. This raises an important consideration: many feature flags become obsolete by design and need to be cleaned up. Managing feature flags in a central file or registry enables periodic sweeps to remove old flags and the associated dead code.
Remote flags
Remote feature flags go further, allowing app releases to be completely decoupled from feature rollouts. Features can be launched dark (i.e. ready for release but turned off) and enabled after a stable rollout of the new build.
But there are dozens of older versions of the code running on users’ devices, each with unfinished versions of features hidden behind flags, and you definitely don’t want to enable those!
Mobile adaptation 1: careful management of versioning and feature flags
There are two common approaches to this problem:
- Equip remote flags with a parameter for the minimum version of the app where the feature is stable
- Use build flags for work-in-progress, then convert them to remote flags when the feature is completed
The second option is arguably safer. In practice, it’s possible to combine both types of feature flag by checking the build configuration before the remote configuration. Third-party tools like LaunchDarkly are also available to support the management and delivery of remote feature flags.
Code review
Some advocates of trunk-based development push for always pair-programming and committing directly to trunk. Pairing has all sorts of benefits, but mobile changes are inherently risky.
Mobile adaptation 2: branches for code review
Time to recover from mobile app defects is high, with bugs often taking hours to manifest in a release rollout and potentially days to patch depending on the app store review process. Short-lived branches with pull requests for code review can be a nice safety net, even on teams that pair frequently.
Short-lived branches
Short-lived branches don’t need to contain entire features (although all tests should pass for the subset of functionality). Branches become a tool for code review, rather than a place to store unfinished work. Development of a new feature might be broken down into several pull requests:
- Add a feature flag
- Integrate the API
- Build the UI layer
- Add translations and UI tweaks
- Enable the flag
This more incremental approach shortens feedback loops. Code reviews get smaller and changes become visible to the whole team sooner. Branches lasting more than a few days might still be cause for concern, though. Metrics like average branch lifetime and code review turnaround time can help develop good habits.
Quality
Continuous integration is sometimes described as “removing distance between developers”, but it doesn’t only help developers.
After the third stage in the previous example, a product manager, tester, or designer might decide to enable the feature in a debug build and give feedback. If everybody is working from the same build, the coordination effort required to try out early versions of features drops to almost zero.
Releases
With all major changes protected by feature flags, the number of threads of communication involved in cutting app releases reduces. The question becomes “what features are ready to be enabled?” rather than “which features need to be merged in time for the release?”.
In the mobile world, it’s not feasible or desirable to ship apps more often than weekly. App store reviews can take days, the phased rollouts that ensure stability add hours or days, and users wouldn’t thank you for a new app binary to download every day either.
Mobile adaptation 3: continuous integration with a release train
So continuous delivery is out of scope (at least for production builds), but continuous integration practices can still be applied within the constraints of a mobile release train.
Cutting release builds
One strategy for triggering releases is pushing a tag to trunk. In practice, this can be supported by a short code freeze (hours rather than days) that allows feature flag updates and bug fixes to be merged, but holds back larger code changes that might break the build and delay the release schedule.
Another strategy is to create a release branch. This is still within the spirit of continuous integration if the release branch is short-lived and isn’t used for any new development.
Using a release branch works well if you have a stabilisation phase or beta release. Fixes applied to the branch can either be cherry-picked back to trunk, or the branch can be merged once the release has achieved a stable rollout.
Hotfixes
Releasing from trunk is all well and good until you need to hotfix a bug in production and trunk has progressed.
Teams that regularly publish hotfixes (because their release cadence is long, for example) might find short-lived release branches a better fit. However, if you release by tagging trunk, you can also branch from the tagged commit and publish a hotfix from that branch. It all depends on the preferences of the team and their release train tooling.
Conclusion
Accelerate suggests trunk-based development is “correlated with higher delivery performance”, hypothesising that “having multiple long-lived branches discourages both refactoring and intrateam communication”. These benefits are absolutely not limited to backend development. Most large mobile teams use some version of these practices.
Trunk-based development shortens feedback loops, reduces time wasted resolving merge conflicts, and derisks releases. With a little tooling and some process adaptations, the core principles are easily applied to large mobile codebases with dozens of developers.
