By Zubin Irani, cPrime CEO

While the benefit of Agile has been noted by those within firms that create embedded software, or firmware, the practical application of it to combined hardware and software development has been difficult to envision. But waterfall methodologies create at times extremely lengthy development cycles (years, for complex systems), and this has caused decision makers to search for ways to reduce time-to-market without sacrificing quality.  These same decision makers must first work with the challenges unique to their industry, when determining which methodologies and project frameworks would work best.

Challenges unique to a hardware/software development

One of the challenges for combined software and hardware development is that software can normally be developed fairly rapidly, and the development broken down into smaller chunks, or iterations. Hardware, on the other hand, may require three to six months or more to show a working component or feature. If the software must wait for the hardware to be created for final testing, this can create testing delays.

Kevin Thompson, PhD, PMP, CSP, and the Agile Practice Lead at cPrime states, “Historically, the view was, ‘Let’s make the hardware first, then develop the software, and get it to work well’. The problem with this approach is that it takes a long time, and at the end, the hardware might not work, which creates a whole new set of problems.”

Hardware must also often follow strictly defined process models, meet compliance standards, and it can be difficult to make late changes (which would be prohibitively expensive) to hardware. The priority for embedded software, for example, would be to write the hardware drivers first, to allow evaluation of the new device and to allow testing¹.  Testing is more complex when software must work within a small physical space, and time itself perfectly to prevent race and other timing issues, requiring testing at some point on the physical hardware².

If problems occur, with waterfall methods it can be difficult to determine which is at fault:  the hardware or the software. Because hardware often isn’t available until near the end of a project for actual deployment and testing, virtual versions of the hardware such as mocks, simulations and emulations may be developed, if the budget permits³. But this is not the same as testing on the real hardware, under live conditions where heat, movement and other environmental impacts must be taken into consideration, and reliability under these condition ensured (because you really do want your auto safety brake system to work, even on a bumpy, icy, wet road).

Problems with Waterfall for Embedded/Firmware Software developers

The problems with waterfall can be summed up as “technologies, customer requests and market realities change faster than waterfall can keep up with.” This is certainly true when complex projects that use waterfall have been known to last for years. By the time the embedded software is integrated with the hardware, the original project specifications may have changed enormously –or the technology may be outdated.

Other common waterfall problems include features being built that are never used, late product delivery, and the inability to prioritize development to what the customer really wants, especially as time goes on.  “Customers often don’t know fully what they want, until actual work on a project is started, and they can see and preview features,” says Dr. Thompson.  This is especially true when an entirely new type of device is being developed, with no actual way of knowing exactly how the product will work.  Requirements during projects of this type evolve, as the teams come to understand the technology and the device, and as they develop the device and demonstrate it to the customer⁴.

In spite of the rigid processes, designs, specifications and tests used by waterfall, the resulting software quality may end up being lower than with Agile, which tests every build from the beginning, and employs continuous, integrated testing throughout. When using Agile practice, the resulting software can be so reliable that it is used to help determine if a hardware defect is present, instead of the other way around.

Special Challenges for Agile Implementation in Combined Software and Hardware

A major problem seen when companies who create hardware and the software that runs it face when trying to “go Agile” is that they often attempt to take methods and practices developed for software (such as Scrum, an Agile project management framework), and try to use it for everything, including hardware development.  Scrum is based upon Sprints of relatively short lengths (two weeks to 30 days), with highly defined tasks that must be completed during the Sprint. The nature of software development makes this an excellent framework for rapid progress; but Scrum isn’t necessarily the best framework for hardware development.

If the products are in a highly regulated industry, then the documentation must follow industry requirements for specification and design, as well as normal testing and functional requirements documentation.  This makes it extremely difficult to use Scrum by itself, since the processes for hardware are frequently much more rigid, defined, and design-oriented than those normally defined by Scrum.

On the software side, because software must interface, communicate with, and control hardware, development issues using Agile are more complex for combined software/hardware projects, and the Stories (definition of the functions for a specific feature) that the software developers define for each Sprint are accordingly more complex.

Large projects with large amounts of dependencies can be even more challenging.  One method of dealing with hardware that isn’t ready to test, is to decouple software and hardware development, via an abstraction layer, to allow software development to continue more rapidly.  The challenge is to find a method that allows the rapid development of software, with concurrent development of the hardware, that can best meet the requirements of each process.

Solution: Use Different Agile Approaches for Software and Hardware

One approach for companies who develop hardware and software to consider is using different methodologies for each. “Scrum works very well for software development,” says Dr. Thompson, “while hardware development can be better managed with an approach developed for non-software Agile projects, such as Commitment-Based Project Management (CBPM).”

With CBPM, instead of working on a Sprint timetable, which may be unrealistic for the delivery of working hardware features, the team members each make a personal commitment to develop one hardware feature by a specific date. The team meets regularly, for accountability that they are on target. Progress reviews, with any corrections needed, take place at regular intervals.

If hardware production falls behind, re-planning occurs to address any roadblocks and implement changes to get things back on track. Both Scrum and CBPM are team-oriented approaches, and can improve communication both internally among team members, and with the Product Owners (team member who represents the stakeholders, ensuring that development create the features they want). While the Scrum master (development facilitator) will assign hours to Sprints for software development, the individual hardware team members will define the work that they must complete to meet their commitment by the time they specify, using Performance Against Commitment (PAC) charts to show progress. Both methods allow Product Owners and end users high visibility into how things are going, and where things are going next⁵. Problems that arise can be addressed quickly.

With CBPM, the emphasis is on delivery of at least a component or piece of the hardware that works, in order to allow the development or testing of the software that will work on it. This is very different from waterfall, where the entire hardware must all be built first. While Scrum for software is based upon two to four-week Sprints with small portions of the software completed at a time, hardware development requires a different approach.

Dr. Thompson says, “The question must be, “What are the basic, smallest chunks of hardware functionality that I can deliver? Can I split the hardware up, so that it’s somewhat modularized? And for each of these ‘modular’ deliveries, what’s the best, the most Agile way to deliver the work?” This transformation in thinking, and the use of CBPM, is at the heart of using Agile in a non-software environment such as hardware production. Instead of defining a four-week Sprint, CBPM has a continuous execution of the project. The goal isn’t a completed Sprint (and associated tasks), but instead, to show actual progress. The result is faster velocity.

With Agile, both hardware and software features are broken down into smaller chunks – only the methodology is a bit different for each. Once software is working, it can be deployed either on any available hardware “modules”, or in a test or simulation environment. This allows the early identification and fixing of race issues and bugs that arise, and reduces the amount of “fixing” and lengthy hours reworking that must occur during late integration.

For CBPM, the Product Owners, or users, collaborate in defining when the deliverable is completed. This differs from Scrum, where “finished” is defined as completing the backlog for that Sprint. Monitoring of progress can performed in the manner that works best for both the hardware team and the Product Owners, with a performance against commitments chart created to show progress.

By the time the software is tried out on the hardware, with Agile the software quality problems are often resolved, and the only issues are related to integration. This is because with Agile, the code is tested, and these tests then help with the design of further code. These tests in themselves become part of the documentation for the system⁶. Automated regression testing allows the addition of new features or alterations to existing code with more confidence, since issues are revealed quickly.

A significant benefit of Agile for embedded software development is that because the Sprints are shorter, it is easier to create more accurate project completion estimates. This allows for improved risk management. The nature of Agile prevents or reduces technical debt, and performance goes up. A concrete example is a three-year long Agile embedded project studied in Europe, where the Agile project team out-performed 95^th percentile “top” waterfall development teams⁷.

Working Both Together: What It Looks Like 

The exploration required at the beginning of each Agile project is the perfect time to divide responsibilities between software and hardware development teams and to define the processes (Scrum or CBPM) to be used by each. As the system is architected, with its subsystems, and the interfaces for communication between hardware and software are defined, the decision whether to use an emulator, or other approaches for testing are made.

The goal is concurrent work, with hardware creating components, and software developing and testing, at times independently of the hardware at first, and then on the hardware components as they are developed. This requires a large amount of collaboration between engineering, design, developers and Product Owners (who provide feedback). As Stories are implemented, first in smaller ‘chunks” or subsystems/modules, and then with larger, more integrated parts of the whole, the system design will begin to evolve, and to incorporate Product Owner and user feedback into the design.

Daily Scrum meetings help facilitate communication among software team members. Once the Sprint is completed, a Sprint retrospective helps to identify what went right, what didn’t, and how to improve things during the next Sprint. Meanwhile, the hardware teams are using performance against commitment to communicate progress, as well as regular team meetings and “map days” to create project commitments⁸.

When working on a hybrid project, such as hardware/software, it is important to realize that the processes will be different for each. Hardware will more naturally fall into a waterfall type process, while software will fit into a Scrum methodology. Constraint-based Planning will identify the critical milestones in one project, on which the other depends, and plans the latter around the former⁹. If the software is ready well ahead of the hardware (which is common), the development team can continue to develop new features until the hardware is ready, and then deploy and test the software on the hardware as it becomes ready.

Hybrid Project Example

cPrime: Challenges Met, and Benefits Enjoyed

cPrime has worked with numerous clients in a combined software and hardware production environment. “We’ve seen the best results when management supports the transition to Agile,” says Dr.Thompson. “All transitions to a new process involve issues, and you need to plan carefully to execute the transition itself.”

This is why when helping companies go Agile, cPrime consultants first meet with key decision-makers, to clarify exactly how the transition should occur, with agreement among all parties. The basis of the initial transition plan is an Agile assessment. Then, team members are defined, including cPrime Scrum and Agile experts who have helped firms that employ hardware and software technologies make extremely successful transitions.

Our firm has found that a phased approach to transition, with small changes that reflect a focus on implementing Agile methods that bring the greatest return first, works best. Staff are taught by on-site consultants how to create user Stories and define “done” for Scrum; or in using maps to define delivery dates, and user feedback as criteria for done with hardware development.

“cPrime offers the companies we work with the benefit of the fact that we understand how things are supports to work,” says Thompson. “We’ve done coaching in transitions, and have worked with companies that have hardware/software networking devices.” The result is improved velocity, improved morale, and improved customer satisfaction.

To learn more about cPrime’s Enterprise Agile Transformation solutions. Contact us at learn@cprime.com.

Sources

1. Punkka, Timo. “Agile Methods and Firmware Development,” paper for the Helsinki University of Technology, Software Business and Engineering Institute
2. ibid
3. Michael J. Karlesky, William I. Bereza, and Carl B. Erickson, PhD. “Effective Test Driven Development for Embedded Software,” April 2006.
4. Punkka, Timo. “Agile Methods and Firmware Development,” paper for the Helsinki University of Technology, Software Business and Engineering Institute
5. Esque, Timm J. “Commitment-based Project Management,” available online at  http://ensemblemc.com/wp-content/uploads/2011/10/2010-Ensemble-paper_Commitment-Based-Project-Management.pdf
6. Michael J. Karlesky, William I. Bereza, and Carl B. Erickson, PhD. “Effective Test Driven Development for Embedded Software,” April 2006.
7. Abrahamsson. Information Technology for European Advancement (ITEA) Innovation Report, “Speeding up embedded software development.”
8. Esque, Timm J. “Commitment-based Project Management,” available online at  http://ensemblemc.com/wp-content/uploads/2011/10/2010-Ensemble-paper_Commitment-Based-Project-Management.pdf
9.   Thompson, Kevin, “Scrum in the Enterprise: How to Manage the Common Issues of Scrum Projects,”

http://www.cprime.com/articles/Scrum-in-the-enterprise.html