How to Implement a DevSecOps Pipeline

Security is a major concern in software development, but often isn’t treated with the same priority as other concerns. By adopting a DevSecOps pipeline, companies can rectify this problem and bring security into the forefront where it belongs. Complete details on implementing such a pipeline is more than one blog post can cover, but in this one, we’ll give you the overview you need to get started.

Introduction to DevOps, Security, and DevSecOps

DevOps is a set of practices that seek to reduce the development cycle by combining software development with IT operations. This arrangement traditionally leaves security as an afterthought. As the whole point of DevOps is to reduce the development lifecycle, relying on a separate team to tackle security concerns as the project nears completion is an unnecessary bottleneck. Therefore, software produced under this paradigm is either not as secure as it could be, or not as rapidly developed.

This isn’t a necessary trade off to make, however. By switching from a purely DevOps paradigm to a DevSecOps pipeline, companies can integrate security into the development process right from the beginning and end up with a much more secure product that still benefits from rapid development time. Because security usually comes at the end of the development cycle (the right side of the timeline), and DevSecOps moves it to the beginning (the left side of the timeline), you’ll often hear the philosophy referred to as shifting left.

Bringing DevOps to Security

Security may be the last aspect of development that has been brought into the mindset behind DevOps, but the benefits are the same. Security as a final stage problem creates a bottleneck in two ways: The security team can’t get to work until the product is already done, but worse, there may have been bad practices throughout the development cycle that harm the security of the end result. By shifting left, not only is everything done at once, but security becomes a part of the design constraint of the software. Just like the move from traditional development to DevOps, making the move to DevSecOps will result in getting better results in a shorter time period.

Automated testing tools have gone a long way in making it feasible to fold security into a traditional DevOps operation. Now, it’s possible for teams to put a high priority on testing for security without causing a massive slowdown in development times. Having the time to bring security into the prioritization means being able to bring the security experts into the design right from the start. This solves both of the problems discussed above. Throughout the rest of this post, we’ll discuss the tools, procedures, and best practices needed to successfully transition from DevOps to DevSecOps.

1. Threat Modeling

In a traditional DevOps workflow, functionality is king. The team is focused on making a product that does what it’s supposed to do and does it well. With DevSecOps, you need to move beyond that thinking. Like the way a good QA person will look at the functionality and try to find ways to break it, a good security person will look at it and try to find ways to break into or abuse it. So, in the same way the QA team makes your software less buggy, the security team makes it more secure.

We can continue this comparison. Bugs are much easier to find if they are caught early, which is why the QA team was always an integral part of the development process. Bringing the security team into that same level of priority means they’ll be able to trace security problems quicker and prevent them from becoming major problems that are harder to fix later down the line.

2. Using OWASP

The Web Application Security Project (OWASP) is an online community that provides a ton of great resources for security teams. In addition to raising awareness of the top ten security threats faced by web applications, the community publishes a number of testing guides to help take the mystery out of security. OWASP also maintains a list of tools for source code analysis, vulnerability scanning, security pipelines, testing, and more. These tools can be implemented into your development workflow to aid in the threat modeling and automated tests that are required to successfully integrate security into DevOps.

3. Identity and Access Management (IAM)

Controlling access and ensuring that only the proper people have privileges that could put the security of your software or the data of your business or customers at risk a huge part of security, be it DevSecOps or traditional. Identity and access management (IAM) is important for both the software you are developing and the systems you are using to develop it.

IAM isn’t just about providing secure passwords to people who need access to your system. In fact, compliance with the European data protection law, GDPR, requires much more. You need to build in ways to detect unauthorized access and promptly alert someone that the data has been compromised. Integrating security into DevOps means making this sort of functionality a core part of the design, and testing of it a regular occurrence.

This means that everyone on the team should be on the same page regarding the use of IAM as one of the building blocks of your application. They should also be versed in the types of patterns and antipatterns that can make or break an IAM strategy. Like everything else in DevOps, IAM should be a combined effort of the whole team.

4. Security Focused Deployment Patterns

Like development itself, the job of the security team isn’t over when the product is ready to ship. The process of deployment and the timeframe afterward are also important factors in keeping your software and your users’ data secure. There are some best practices for deployment that will aid in keeping any problems that crop up from a new release under control.

Canary deployment

Releasing under a canary deployment model is an increasingly common practice, both for catching security problems and for catching bugs in general. The concept behind canary deployment is that you don’t release to the entire user base at once, instead opting to roll out your new versions to an increasingly large group of them until it has been fully deployed. This means if there are any glaring problems, the number of users that are affected is minimized.

Dark launches

A dark launch has a similar goal as canary deployment: limit the damage in situations where problems emerge. As opposed to canary launches, dark launches usually go out to everyone. The difference here is that the new features are enabled with a toggle that is initially set to off. The ability to turn the new features on, check for problems, and turn it off if there are any helps to minimize the risk of rolling out the new version.

Keeping dependencies up to date

Just as the software your computers run on need to be updated with the latest security patches, so to do the dependencies that your software relies on. You should keep track of when the latest third-party tools you have used to build your software are released and be sure to quickly implement any of them that have crucial bug fixes, particularly those related to security.

Scanning for trouble

Automated security testing doesn’t stop when development does either. You always want to be on the lookout for threats. Thankfully, there are plenty tools available to help you perform security scans. These tools include OpenSCAP for testing security baselines, OpenVAS for vulnerability scanning, w3af to test your web applications for security problems, and many more.

5. DevSecOps and Operations

Just as with DevOps, everyone comes together to make DevSecOps work. The operations side of the equation has plenty to do in order to ensure your product is as secure as it can be. Like many other aspects of DevOps, there may be some crossover here. That’s a good thing. What makes these paradigms so successful is that there is so much that can be accomplished jointly. Some examples of that for operations include maintaining release archives so the company can quickly roll back if needed, or maintaining and monitoring logs and other alerts.

One security specific operation is incident response planning. What happens if something goes wrong? Proper planning, and emergency drills to ensure everyone remembers what to do and stays sharp, will mean a prompt response should a real security threat emerge.

6. Setting DevSecOps Policy

A common framework in DevOps is GRC, which stands for governance, risk, and compliance. Governance refers to the leadership, guidance, and specific policies that a company puts into place to achieve their goals. Risk is any uncertainty that could cause problems. Compliance is a way of ensuring that a set of guidelines or standards are applied. When everyone comes together to make it happen, as is the DevOps way, GRC becomes a guiding principle that allows a company to reliably achieve objectives and maintain its integrity.

Tools such as Chef InSpec allow businesses to turn policies surrounding security, compliance, and more into automated tests that can make enforcing those policies easier, using language that everyone can understand.

7. Managing Change in DevSecOps

There’s always some level of risk involved in change. That risk is only amplified when security practices and procedures are a part of the change. Still, managing change in DevSecOps looks a lot like managing change in any DevOps environment. ITIL change management still provides the overarching framework that is augmented with specific policies that are relevant to a given business.

A change-advisory board may be used to help facilitate the changes. In DevOps, however, there are trade-offs to consider when deciding to bring on a CAB. While they can be helpful in helping to facilitate change, change-advisory boards can also quickly become a bottleneck that will break the pace of a DevOps team and negate the benefits of the paradigm. Learning when to use them, and when not to, is crucial to keeping your team running at peak efficiency.

Some project management software will also have tools that will help you automate parts of your change management pipeline. This type of automation should be taken advantage of whenever possible.

8. Monitoring Key Metrics

Keeping an eye on important metrics is an effective way of keeping everyone on track. There are metrics that can help you follow the progress of every aspect of development. Some important security-related metrics include:

• Number of security tickets opened — It stands to reason that this number should be as low as possible. It can also be helpful to track the number of issues found during development. The goal is to trend towards more issues found during development rather than after deployment.
• Number of failed security tests — A failed security test means more work finding and fixing the problem. One of the benefits of having security combined with DevOps should be a reduction in such failures.
• Remediation time — This is the time it takes from discovery of a vulnerability to the time that it is fixed. Working to keep this number low will avoid vulnerabilities becoming a bottleneck in your processes.
• Deployment time — This is similar to remediation time, except instead of focusing on when the vulnerability is fixed, it focuses on when the release is pushed to users. Quick release times give everyone the peace of mind of knowing that security problems aren’t in the wild for long.

There are many more metrics that can be tracked, and your team can work together to find the ones that make the most sense for your project and your company culture.

Learn More

We’ve introduced a lot of concepts in this post, but implementing a DevSecOps pipeline is a fairly complex subject that can’t be adequately covered in a single blog post. If you want a step-by-step explanation of the processes, tool, and pitfalls you’ll encounter while building your DevSecOps operation, consider signing up for Cprime’s DevSecOps Boot Camp. We’ll tell you everything you need to know in order to get your new team working together to build more secure products. If you have any questions about the boot camp, please do not hesitate to contact us.

John Garrison