The epiphany of integration points is that they control product development. They are the leverage points to improve the system. When timing of integration points slip, the project is in trouble.
—Dantar Oosterwal, The Lean Machine
Continuous Integration (CI) is the process of taking features from the Program Backlog and developing, testing, integrating, and validating them in a staging environment where they are ready for deployment and release. It is the second element in the four-part Continuous Delivery Pipeline, as shown in Figure 1.
CI is perhaps the most essential technical practice for each Agile Release Train (ART). It provides a quality heartbeat that reduces risk and establishes a fast, reliable, and sustainable development pace.
With Continuous integration, the “system always runs,” meaning it’s potentially deployable, even during development. CI is most readily applied to software solutions where small, tested vertical threads can deliver value independently. In larger, multi-platform software systems the challenge is harder. Each platform has its own technical constructs, and the platforms themselves must be continuously integrated to prove new functionality.
In complex, heterogeneous systems—those with mechanical subsystems, software, electrical/electronic subsystems, suppliers, subassemblies, and the like—CI is harder still. However, integrating and testing collaborating components together frequently is the only practical way to fully validate a Solution. As Oosterwal points out, “integration points control product development.” 
As a result, teams need a balanced approach, one that allows them to build quality in and receive fast feedback from the integrated increments. For the software elements, truly continuous integration can be more easily applied. For larger and complex heterogeneous systems, in particular, CI requires economic trade-offs between frequency, scope of integration, and testing.
A Three-tiered Solution Integration Strategy
The level of effort and strategy for CI depends on the size and complexity of the solution. When building at the Large Solution Level, CI becomes a three-tiered approach, as indicated in Figure 2.
Integrate and Test Stories
The role of the CI process is to pull features from the program backlog to design, implement, integrate, test, and validate them in a staging environment. But as we know, features are too big and abstract to be coded directly. Instead, features are converted into Stories during Program Increment (PI) Planning, as Figure 3 illustrates.
Thereafter, each story is defined, coded, tested, and integrated into the baseline. To support this, team members integrate their individual work frequently, applying automated continuous integration environments as illustrated in Figure 4 .
Automate Story Tests
Developing incrementally means that the system must be continually tested as well, otherwise there is no way to assure that the new stories work compatibly with all the existing functionality. To this end, teams apply Test-First Development, which includes creating unit and acceptance tests for each story that gets implemented. To avoid building ‘technical debt,’ Agile teams develop and apply these tests in the same iteration in which they implement the story, as Figure 5 illustrates. Passing versus not-yet-passing and broken automated tests are the real indicators of progress.
Integrate and Test the System
While critical, such local story integration and testing isn’t enough. In order to fully test the features, system-level integration and testing is required. With the support of the System Team, the work of all teams on the ART must be frequently integrated to assure that the system is evolving as anticipated, as Figure 6 illustrates.
System-level testing happens as frequently as possible during the iteration, ideally daily. However, whatever the circumstances, such full-system integration must be accomplished at least once per iteration. Otherwise, the late discovery of defects and issues reflects all the way back to earlier iterations, causing substantial rework and delays. All this work comes together in the System Demo, which demonstrates the accumulated features and is the true indicator of ART progress.
Automate Feature Tests
As with user stories, features must be continuously tested against their acceptance criteria, to assure that they work not just once, but continuously into the future as new functionality is developed. Again, the philosophy of test-first is applied, in this case by automating as many of the feature tests as possible.
Integrate and Test the Solution
Finally, building large solutions—those that require multiple ARTs and Suppliers to develop them—requires an additional level of integration to pull all the work together. Full or partial integration happens over the course of the PI, with a full solution integration occurring at least once per PI, as illustrated in Figure 7.
The solution integration and demo are the joint responsibility of the ART and solution system teams. The Solution Demo event is where the aggregated results are made visible to the Customer and other stakeholders.
In order to be able to demonstrate the full solution routinely, ARTs typically make additional investments in integration, testing, and supporting infrastructure. In many cases, the extent of integration and testing may be less than 100 percent and may need to vary across multiple early integration points. To assist, teams can leverage virtualization, environment emulation, mocks, stubs, reduced test suites, etc. Time and effort for such extensive integration and demonstration may need to be explicitly allocated during PI planning.
Synchronize with Suppliers
Suppliers bring unique contributions that can have a large impact on lead-time and value delivery. Their work must be continuously integrated as well. Tips for accomplishing this integration include:
- Plan the integration points together
- Adopt an integration cadence; establish objective evaluation milestones
- Foster collaboration between the system teams of ARTs and suppliers
- Participate in Pre- and Post-PI Planning and solution demos
- Foster collaboration and synchronization between ART and supplier Architecture/Engineering and Systems teams
Optimizing Integration Trade-offs
Each ART’s goal is to fully integrate features across all teams in each iteration. As we described, however, that can be difficult due to solution complexity and heterogeneity, and availability of specialty testers, laboratories, equipment, third-party components, and so on. Given these challenges, teams may feel that the goal of integrated iteration is not realistic—at least initially. But that can’t be an excuse for accepting late-in-the cycle integration and testing. Below are some suggestions for how to achieve most of the benefits, even when full, fast CI isn’t immediately practical.
- Integrate different aspects of the solution at different intervals
- Integrate all assets, but run only a subset of tests
- Use virtual or emulated environments, stubs, and mocks, until the actual functionality is available
Enabling Continuous Integration
Continuously integrating large and complex systems is a journey that takes time. The following section provides some suggestions for building a successful CI culture and practice.
- Integrate often – The more frequently teams integrate, the quicker they find problems. The harder it is to do, the more often they need to do it—eliminating impediments and adding automation along the way. This results in faster learning cycles and less rework.
- Make integration results visible – When the integration process breaks, everybody should know how and why it broke. And when it’s fixed, new tests should be added to detect the problem earlier and prevent it from happening again.
- Fixing failed integrations is top priority – If teams just keep working during an integration failure, it doesn’t create the right sense of urgency and importance to fix the problem. To highlight the problem, teams often use flashing lights to draw attention to a broken build, and visible indicators displaying the percentage of the time the system remains broken.
- Establish common cadence – Integration points are easier when all the teams are moving at the same consistent rhythm. That’s why all iterations and PIs use the same cadence within an ART and/or Solution Train. If full CI can’t be accomplished during the course of an iteration, teams can make near-term trade-offs on what can be integrated, while continuously improving their techniques and infrastructure toward this goal.
- Develop and maintain proper infrastructure – Effective continuous integration depends on the availability of test and staging environments (see Continuous Deployment). Infrastructure is, of course, an investment. But Lean-Agile Leaders take the long view and make the investments necessary today to increase velocity for the marathon ahead.
- Apply supportive engineering practices – Continuous integration is easier when the system is designed with those concerns in mind. Test-first development and designing for testability call for better modularity and separation of concerns, as well as the use of primary interfaces and physical test points.
It’s also important to note that continuous system integration and frequent solution integration represent a special challenge for groups early in their transition to SAFe. They just haven’t done it previously, nor have they built the necessary infrastructure. But the current state cannot be an excuse to simply reduce the scope of integration. Most of these challenges will disappear in a better, future state, but only if the teams start now.
Learn More Dantar P. Oosterwal. The Lean Machine: How Harley-Davidson Drove Top-Line Growth and Profitability with Revolutionary Lean Product Development. Kindle Edition.  Leffingwell, Dean. Scaling Software Agility: Best Practices for Large Enterprises (Agile Software Development Series). Pearson Education. Kindle Edition.  Kim, Gene and Jez Humble, Patrick Debois, John Willis. The DevOps Handbook: How to Create World-Class Agility, Reliability, and Security in Technology Organizations. IT Revolution Press. Kindle Edition.
Last update: 13 September, 2017