Resource Type: Case Study

The Client

In an era of online shopping and next-day delivery, this US-based Cprime client facilitates business continuity for thousands of brands worldwide. Their complex automated storage, retrieval, warehousing, and logistics solutions improve the flow of goods, materials, and information, allowing companies to meet the ever-growing expectations of global consumers.

Industry expertise, robotics, artificial intelligence, and machine learning have made the company a global leader in materials handling and logistics. Its diverse clientele come from industries that include ecommerce, retail, food and beverage, manufacturing, and third-party logistics providers.

Challenge: Streamlining Modern Warehouse Design Pipelines to Enhance Pre-Sales Efficiency

Designing a modern automated warehousing solution is astonishingly complex. In an unoptimized environment, the sheer number of factors and volume of calculations required to create a design prototype for a new logistics facility can take weeks.

“A modern warehouse is essentially a collection of robots. It’s all spectacularly complicated,” explains Kevin Ryan, Cprime Head of DevOps, EMEA. “The scope, size, and throughput of modern warehouses, and the number of unique products in any given warehouse have grown beyond our human ability to calculate. Even machines struggle to crunch the numbers efficiently.”

While lengthy design calculation times were the company’s primary pain point, system complexity also resulted in an inability to react rapidly to change. Any updates to a customer’s operating parameters or facility specification would further delay pre-sales project modeling; the company had to recalculate each time-consuming potential design from scratch. The delays ranged from hours to weeks.

“If customers can’t get answers from us quickly, the dialogue stalls and the entire process can grind to a halt. We wanted to change that,” says one of the project’s executive sponsors.

“They were seeing new projects and revenues delayed by up to a quarter as they waited for a mathematical model. There was a potential cost of millions for each delay,” says Ryan. “The company needed a development framework that would allow them to put a pre-sales mathematical model of a warehouse in front of a customer faster, which meant boosting efficiency.”

To complicate matters, any development framework the company adopted had to be a good fit with Azure Batch, the cloud platform providing the computing power to manage the calculations. The company also needed an implementation partner that could work closely with the cloud specialists on-site to modernize, scale, and streamline its infrastructure to meet the growing demand of the world’s largest retailers and manufacturers. 

Solution: Leveraging the GitLab Toolchain with Cprime Expert DevOps Services

Evaluating its options, the automated warehouse designer selected GitLab to provide the backbone for its new continuous integration and continuous development (CI/CD) initiative. Recognized for being robust, reliable, and secure, GitLab was the ideal CI/CD, collaboration, issue tracking, quality automation analysis, and code review tool. The GitLab toolchain contained everything the company needed to speed up development and enhance their predictive modeling sales tools and in-production logistics engines.

To implement the solution, GitLab recommended the Expert-Led GitLab Solutions and DevOps Services of Cprime, their Certified Training Partner.

“The Cprime DevOps team has invested heavily in GitLab professional services engineer certifications covering everything from CI/CD, project management, DevOps, and inner sourcing, to security and system admin,” explains Ryan. “But the requirements for this engagement went beyond professional GitLab implementation services. The company hired Cprime to show them how to take the GitLab toolchain and achieve elite DevOps CI/CD performance.” 

“The requirements for this engagement went beyond professional GitLab implementation services. The company hired Cprime to show them how to take the GitLab toolchain and achieve elite DevOps CI/CD performance.” — Kevin Ryan, Cprime Head of DevOps, EMEA

Discovering Context and Objectives Using Spikes

The two-person Cprime team began Discovery to define the company’s context and objectives clearly. They started by creating research tasks to identify and resolve questions and explore potential solutions.

“By creating technical spikes, we formed an understanding of their codebase and what they needed to achieve,” says Ryan. “Using those prototypes, we rapidly gathered feedback and verified we were on the right track.”

Tailoring Toolsets and Processes for the Azure Environment

After achieving consensus with the company’s management and DevOps teams, the Cprime team tailored the new GitLab instance to work specifically within the customer’s Microsoft Azure environment and security context.

“We created secure containers, CI/CD pipelines, and process templates,” says Ryan. “The result was a universal base implementation they could reuse across multiple projects to speed up development long term.”

Infrastructure as Code With Helm and Terraform

To ensure a consistent, easily deployed development environment, the Cprime team also worked with the company to leverage GitLab CI/CD’s native Terraform integration. Using the Terraform Helm provider allowed the company to begin centrally deploying Helm charts. This allowed it to manage high- and low-level Azure Kubernetes components, easily implement and modify Infrastructure-as-Code (IaC) workflows, and simplify the creation of common and complex infrastructure patterns.

Throughout the process, Cprime worked to bridge the gaps, coordinating and driving innovation between company stakeholders, DevOps teams, and third-party Azure infrastructure architects.

“As implementation specialists, Cprime took the work of all the teams, rationalized it, and worked as the engine that put it into production,” says Ryan. “It was all about streamlining collaboration and automating the processes to speed up development.”

“As implementation specialists, Cprime took the output of all the teams, rationalized it, and put it into production using GitLab. It was all about streamlining collaboration and automating processes to accelerate development.” — Kevin Ryan, Cprime Head of DevOps, EMEA

Results: Increased Deployment Frequency and Reduced Error Rates 

Because of the combined efforts of Cprime and the project teams, the warehouse design company established high levels of process maturity based on DevOps Research and Assessment (DORA) core metrics. Across the board, the company’s deployment frequency increased while it slashed its change lead times, change failure rates, and mean error recovery times.

Cumulative Change Dramatically Reduces Turnaround Times

As an indicator of the efficacy of the cumulative changes, the company reduced simulation times from over 11 hours to as few as 24 minutes. Turnarounds on proposal submissions showed similarly dramatic improvements.

“We’ve reduced the time to solution—from receiving customer data to running designs and calculations and ultimately giving out a warehouse proposal—from eight weeks down to one day,” the executive sponsor states. “The productivity increase has scaled our capacity to answer RFQs and engage with new and existing customers.”

Likewise, by using GitLab project templates, Terraform, and Helm scripts to automate their CI/CD pipeline, the company’s development team recently set a new turnaround record—they received a new specification and deployed the corrected warehouse design in only 90 minutes.

“We’ve reduced the time to solution—from having customer data to running designs and calculations and ultimately giving out a warehouse proposal—from eight weeks down to one day. The productivity increase has scaled our capacity to answer RFQs and engage with new and existing customers.” — Executive Sponsor

Getting There With Cprime

During the engagement, Cprime exceeded the client’s expectations for GitLab professional engineering services. Based on Cprime’s performance and the clear successes of phase one, the company has extended the relationship indefinitely.

“We started with a very short engagement and, based on our performance, over a year later we support all their GitLab and Azure-related activities,” says Ryan. “The company learned what I want all potential Cprime customers to understand: if you want to become a high-performing, elite development organization, Cprime will get you there.”

If you’d like to see similar results in your own organization, explore our flexible GitLab and DevOps solutions.

The Client

When a prestigious European technical university set out to build a moon robot, one of its goals was to highlight the potential of its new generation of engineers. Using a fresh student perspective, the institution sought to create a semi-autonomous rover that could move freely about the moon’s surface while collecting important information about radiation levels.

Unlike previous remote moon exploration vehicles, however, the university’s initiative focused on creating a smaller, more lightweight system. The new rover would have a form factor that scientists could easily convey into space and deploy as a swarm to gather data across a vast surface area. The rover’s affordability, autonomy, and improved mobility would allow exploration teams to undertake missions across terrain impractical for larger, heavier, and more expensive self-driving vehicles.

Challenge: Breaking New Ground — Creating a Modern CI/CD Testing Platform for the Aerospace Industry

Software development is often a process of trial and error, and developing hardware, software, and firmware systems for off-planet exploration exponentially increases the potential risk of something going wrong. Due to the narrow margins for error in space, the rover project team of 50 student engineers and support staff wanted a modern, proactive testing and development process to increase their chances of developing a robust, reliable lunar vehicle.

“Having your code break on the moon is obviously a massive problem. It’s not something that you can fix by hitting a button and rebooting the system,” says Kevin Ryan, Agile and DevOps Coach for Cprime EMEA. “The problem was that embedded system workflows, especially in the aerospace and space industries, are decades behind the processes used to develop modern cloud-based industrial infrastructures. Not that the students lacked skills—there was a lack of precedent in the entire industry. Cprime’s role was to help them establish that precedent and significantly reduce marginal risks.”

“Having your code break on the moon is obviously a massive problem. It’s not something that you can fix by hitting a button and rebooting the system.” — Kevin Ryan, Agile and DevOps Coach for Cprime EMEA

Aligning processes with modern cloud methodologies

To improve their quality control protocols, the university team needed to align their processes with contemporary cloud methodologies and software development best practices. That meant implementing the tooling and ways of working for a CI/CD (continuous integration, continuous deployment) pipeline. The testing pipeline would allow them to rapidly iterate upon their code and design while maintaining quality and continuity across the project.

“As a team, they focused on writing complex, embedded machine learning and AI code. Coming directly from academia, they weren’t up to speed on code development automation infrastructures,” says Ryan. “Industrialized development just wasn’t a part of their scope.”

Solution: Partnering with Cprime to Evaluate Tools and Create New Development Opportunities

Based on a recommendation from one of their commercial partners, the university team turned to Cprime, GitLab Certified Training Partners. The initial brief was simple, but left room for expansion.

“They had chosen GitLab—a complete DevOps platform—but they wanted Cprime to show them what using it should look like,” says Ryan. “They needed to understand the fundamental building blocks. We began by implementing the development environment and went from there as they learned the tool’s potential applications and use cases.”

The Cprime trainers experienced a learning curve nearly as steep as the student engineers themselves. The project was more than a simple tooling upgrade or transformation. The Cprime team needed to dive deep to define the project’s practical requirements and apply their newfound knowledge to a viable solution.

“Due to the unfamiliar environment, it has been a little like a research project—an incremental process to figure out how their DevOps journey should work,” says Ryan. “There was no Googling for a quick solution. As trainers, we needed to understand what they do before we could even hope to make reasonable and intelligent proposals to move them forward.”

A single repository of distinct projects with clear relationships

After clarifying the engineering team’s requirements, Cprime began putting together the components for a successful CI/CD pipeline. The team adapted course materials and explored key principles—from classes like GitLab with Git Basics and GitLab for CI/CD Pipelines—with the engineers to help them create an elite DevOps toolchain to solve their testing issues.

“We began by advocating for a mono repo rather than a poly repo approach to mitigate complexity and aid communication. Then we developed systems for quality assurance, build, and testing automation. We also automated dependency management and documentation generation,” says Ryan. “Once they were comfortable with the basic processes, we moved into more hardcore QA tools like Google Test, the C++ testing and mocking framework. Now, they are doing some groundbreaking stuff and actually deploying machine-level firmware to their space robot.”

“Together we have put together a toolchain that ensures they check, double check, and triple check every one and zero in their code base. It has pushed the project forward and allowed them to increase their benchmarks while simplifying each subsequent success.” — Kevin Ryan, Agile and DevOps Coach for Cprime EMEA

The right tools for success

To reduce context switching and promote a more familiar, consistent development and testing environment, Cprime also introduced the GitLab Workflow extension for Visual Studio (VS).

“Using modern technologies like Docker and VS development containers has made a huge difference,” says Ryan. “Because each machine you compile code on can yield a different result, just having an identical environment for everyone with all the dependencies, tooling, and  build infrastructure in place has improved auditability and repeatability.”

Results: Increased Quality and Reduced Margins for Error

After 18 months of working with Cprime, the university teams have a unique CI/CD pipeline suitable for the aerospace industry and low-level firmware and embedded systems development. It is a development and testing framework that rivals any cutting-edge development and testing automation infrastructure for the cloud.

By guiding them toward a uniform development ecosystem with repeatable processes, Cprime helped the engineering team eliminate nearly 75% of the trial and error caused by compiling and testing code on inconsistent systems.

“That translates directly into higher quality firmware and software, and a reduced margin for error,” says Ryan. “We have put together a toolchain that ensures they check, double check, and triple check every one and zero in their code base. It has pushed the project forward and allowed them to increase their benchmarks while simplifying each subsequent success.”

If you’d like to see similar results in your organization, explore our flexible
GitLab training and GitLab implementation solutions.

The Challenge: Lack of Alignment and Planning Struggles Slowed the Agile Scaling Journey

As they progressed in their efforts to scale their Agile practice, they identified opportunities for improvement across the board, both in processes and in the underlying cultural and strategic approach they were taking to scaled agility.

Interdependent teams working in silos

“We had four delivery teams working with two products,” says the R&D department’s Product Director, “and the products are highly interdependent. The data coming out of each feed the other in a cycle. So the delivery teams worked hard on their own product backlog items (PBIs) but there really wasn’t a mechanism for keeping all four teams aligned at all times.”

Additionally, the effectiveness of planning was limited because different teams were planning independently in parallel, which often left risks and dependencies undetected until they became blockers.

Project instead of product mindset

This leading pharmaceutical manufacturer had already engaged Cprime a year earlier to provide extensive training courses around various Agile disciplines and Lean Portfolio Management (LPM). As a result of her own experience in these training classes, the Product Director recognized that the delivery teams were approaching their workflow from a project perspective rather than focusing on value streams, which is a core foundation of SAFe and LPM. To progress with their scaling journey, this would need to change. This core adjustment in what was guiding the work would require tremendous change involving many people. It was no small task.

 

“We had four delivery teams working with two products and the products are highly interdependent. The data coming out of each feed the other in a cycle. So the delivery teams worked hard on their own product backlog items (PBIs) but there really wasn’t a mechanism for keeping all four teams aligned at all times.” – R&D department’s Product Director

 

Limited understanding of the big picture

Finally, a general lack of alignment between the business and technical teams resulted in priorities and tasks being set down without the teams really understanding the bigger picture of what they were building and why. This is a situation that we see affecting nearly every organization in one form or another. While it can be a formidable challenge, the delivery teams were eager to overcome it.

The Solution: Bringing in a Release Train Engineer (RTE) to Solidify SAFe

Heading into 2022, they reached out to Cprime to source a coach and SAFe expert who could help guide the teams through the coming changes. Ryan Evans, RTE and Agile Coach, took on this challenging and exciting engagement.

An initial assessment and gap analysis

“Although I am an RTE,” Ryan says, “running the Agile Release Train (ART) wasn’t my first task. I needed to thoroughly understand where the teams were in terms of Agile maturity and workflow so I could give the ART the best chance at success.”

Ryan worked closely with the Product Director to perform a large-scale assessment of the delivery teams and processes. In the end, the assessment functioned as a gap analysis.

“I’ll never forget opening Ryan’s report,” she says, “to see forty-six items listed as opportunities for improvement. But it was great to see everything out on the table. Then, Ryan and I discussed each item and decided which were most important and would have the most impact. Then we could focus our efforts accordingly.”

Ryan adds, “While the gap analysis turned up many necessary improvements, we could boil nearly all down to a few main themes: getting all the teams unified into one ART and working on one cadence so they could plan and execute the work as a cohesive unit, and applying foundational SAFe concepts pragmatically to better organize and prioritize the work being done. So, that’s what we did.”

Changes to enhance PI Planning

Ryan began working with the teams in February 2022, five weeks before the end of the first quarter. With an eye on optimizing the Q2 PI Planning session, he focused on making changes that would get meaningful results without causing undue disruption.

“We had about five weeks to prepare for the new PI Planning,” Ryan recalls, “so it was a blur of activity. We needed to coordinate a much larger and longer session than they’d done before, and we wanted to do some preliminary training so the teams and leadership could get the most out of the event. But we completed it and ran an enhanced planning session involving all four teams, as well as subject matter experts and other stakeholders from several other business units—about one hundred attendees on three continents.”

The primary goals of these changes were:

• Getting on the same page — Bringing all four teams together for a single, all-encompassing planning session
• Getting on the same schedule — Establishing a unified cadence so everyone could effectively plan, deliver, and continually improve in concert
• Understanding our dependencies — With the use of Mural, the teams could visualize all their dependencies and when they needed to be delivered
• Determining the priority of work — Using the concept of Weighted Shortest Job First (WSJF), the teams could determine the order in which they should tackle work
• Formalizing the I&P — Establishing a consistent Innovation & Planning (I&P) sprint at the end of each PI to support continuous improvement
• Laying a solid foundation — This first enhanced PI planning event would set a baseline on which to measure improvement, and a blueprint for future engagements

During the event, Ryan served as RTE, the facilitator who kept the process on track and moving forward. With his coaching, the Product Owners and Agile Facilitators (Scrum Masters) helped facilitate and took the lead in their areas of responsibility.

The Product Director recalls, “Going into that Q2 planning event, I think we were all far better prepared than we’d been in the past to have meaningful conversations and make appropriate decisions. Understanding what we needed to accomplish, how to best do so, and especially why, was vital to the event’s success.”

Working the SAFe framework

Another objective Ryan and the team focused on, both before and after the enhanced Q2 PI planning event, was formalizing the teams’ use of the SAFe framework to better support their scaled Agile practice as an ART. This was another byproduct of ongoing training and slow-but-steady adjustments to existing roles and processes to better align with best practices that have proved successful in thousands of previous engagements.

“To me,” the Product Director says, “one of the greatest benefits of working with Ryan on this was his supporting our use of SAFe as a framework, not a rigid checklist of requirements. There were several concepts we agreed were necessary and beneficial changes we needed to implement, but others that we discussed and decided just weren’t right for the company at this time. And that’s ok. We could pick what would work best for our unique situation.”

The primary goals of these adjustments were:

• Formalizing roles and responsibilities — Team members and leaders were trained to take on or better carry out their respective roles in a SAFe environment.
• Formalizing Agile concepts — This included the definition of ready, definition of done, acceptance criteria, capacity planning, feature refinement, and prioritization via WSJF. These concepts were already in play, but teams weren’t applying them consistently, so they needed a common understanding and commitment.
• Formalizing Agile ceremonies — Establishing a consistent schedule of daily, weekly, and per-sprint meetings provided stability and unity while eliminating a lot of unnecessary ad hoc meetings.
• Supporting ongoing improvement — Providing necessary SAFe training to get buy-in and help everyone understand the how AND the why behind the changes, as well as ongoing assessments to first establish a baseline and then track increased maturity.

Ryan says, “There was a notable rise in the teams’ confidence level as these principles sunk in and they started seeing the real world value of adhering to a more unified and consistent way of working. Even those who were hesitant at first have really embraced the process.”

The Results: An unprecedented 96% predictability rate on top of greater transparency and trust

Two PIs further along since that first enhanced session in Q2, the team has experienced an unprecedented improvement in predictability based on committed PI Objectives completed.

“In all my years running ARTs, I’ve never seen a percentage this high,” Ryan says, referring to the 95.7 percent predictability rate of committed PI Objectives the teams achieved in the last quarter. “It’s a testament to the strides they’ve made in understanding capacity and velocity, as well as what they can legitimately commit to versus what work is less predictable because of dependencies or blockers outside their control.”

The Product Director concurs, “While that number is phenomenal, we know there’s still work we can do to increase the number of stories we’re able to commit to as we work to smooth out dependencies internally and with outside teams. But, the progress so far has been excellent.”

 

“While the gap analysis turned up many necessary improvements, we could boil nearly all down to a few main themes: getting all the teams unified into one ART and working on one cadence so they could plan and execute the work as a cohesive unit, and applying foundational SAFe concepts pragmatically to better organize and prioritize the work being done. So, that’s what we did.” -Ryan Evans, STE, Cprime RTE and Agile Coach

 

Qualitative improvements

Besides quantitative results showing dramatic improvement from baselines set in March, the teams have reported several qualitative improvements as well, including:

• Greater confidence planning, committing to, and delivering work, but more importantly, greater confidence in stating when the teams cannot commit to deliver work
• More transparency and visibility into the work within and outside the teams
• Stronger engagement with and understanding of the big picture strategy
• New-found stability to withstand inevitable change within and outside the teams
• Enhanced collaboration within the development teams and with stakeholders in other departments
• Greater trust that the teams will finish the work that is ready and in the queue

Additionally, in November, a fifth delivery team joined to do a Large Solution PI planning session. This showed a new level of coordination and collaboration between the teams as both continue to mature in their scaled Agile practice. As a result, the company has renewed the engagement so the progress can continue.

The Product Director concludes, “Working with Ryan has been great. We got on well from the get go, and I know that every new decision or experiment is going to be a give and take where we will do what’s best for my teams and the company as a whole.”

Want to see similar results for your organization? Explore our flexible Scaled Agility and Learning solutions.

SAFe and Scaled Agile Framework are registered trademarks of Scaled Agile, Inc.