DevOps is hard. Our perspective at Cprime is that DevOps requires that we build our own story by using desired business outcomes as a key component to map that journey. It’s only natural that questions will arise along the way. Below are a few questions and answers to help map the next, right step. Stay tuned for more questions and answers.
My company has the “always been this way” problem. How do you overcome this?
It’s only natural that teams view new ways of working with skepticism; many process improvements in the past have been ham-fisted attempts to improve efficiency and cycle time. DevOps, on the other hand, is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes. It’s only human to want to keep doing the same behaviors, despite outside influences that tell you and your team that it is time to change your practices.
Many organizations new to DevOps practices find the new discipline overwhelming at first glance. The seminal Accelerate: The Science of Lean Software and DevOps book lays out 24 distinct DevOps Capabilities. Two dozen capabilities are likely overwhelming for most developers to gain an easily understandable grasp around quickly, so many new teams focus on symptoms or signals that can show if the desired behaviors are being exhibited. The DORA-Four metrics of Deployment Frequency, Mean Lead Time for Changes, Change Failure Rate and Time to Recovery are great signals on how your organization is executing DevOps in practice.
Changing team workflows and behaviors is more tricky: It kind of goes down to the cultural aspect of showing the value of what you’re trying to propose. Most teams are responsive to a win-win situation where both the organization realizes better metrics and the team realizes a smoother and easier day-to-day workflow. It’s one thing to talk about a big game and say “hey, this is gonna change how we do everything at our organization”. But then it’s another thing to show “hey, this is how we reduced this amount of time in the release process for releasing our application or this is how we reduced the amount of time in this particular process” either through automation or some level of process adoption or change.
Should teams with monolithic architectures be transitioned to more of a web services architecture and if not microservices to be successful with a DevOps practice?
We have this tendency as people to jump to extremes when the answer might lie somewhere in the middle. In monoliths everything you have is all in one spot. If there’s a problem there, it takes everything down. But microservices may be too granular with too many places to track things. In reality, you might have to find something in the middle where you group some things together and then you have a lot of microservices on the side. It should depend on what you are trying to deploy. What is your application? What is the purpose of it?
If you choose to transition, you need to do it incrementally. I see a lot of organizations that say, Well, we have a monolith causing problems, therefore we need microservices. And again, you might end up with true microservices two years down the road but you probably don’t jump from one extreme to the other overnight. I also hear “well, we have a monolith and it means that every time we change anything, we have to retest everything so the answer is microservices” and what we were able to show them is there are ways to structure how you test, and to some degree, ways to structure your code within the monolith where you can make that test latency problem go away.
Something else to pay attention to is that microservices may make development much easier for individual teams. And if you don’t have the right platform tooling, it can make things much, much harder for operations, because now instead of managing one thing, you’re managing a thousand things. So you trade development complexity for operational complexity. And then the final point I want to make goes back to our old friend, Mr. Conway and Conway’s Law that decomposing a monolith also has implications for how you structure your teams. And the process of restructuring teams is not a simple thing.
Tooling for DevOps is overwhelming. How do I choose?
I don’t have a good answer for that because I agree. It is overwhelming. It is constantly changing. And one of the things that we’re seeing now is a lot of companies are going after the same ground and are really overlapping. And the advice I would give is to approach it the way you would approach any tooling assessment and this is one we really need agility and speed and learning so we can fail fast. If you’re looking for something like a monitoring tool, take your time, understand what you’re trying to accomplish, assume that you’re going to have to do a fair amount of due diligence. Definitely at Cprime, this is somewhere we help clients with guided trials of tooling or even run through tool selection workshops to help weigh the decisions.
I think one of the first things you need to identify first is what is your problem? If I have a nail stuck in wood, I’m not going to go grab a wrench for that. I mean, I could do it, but it’s not necessarily the best tool for the job, right? Let the pain be your guide. Take an inventory of the multiple problems and then do some value stream mapping, do some workflow mapping, do some root cause analysis or some constraint identification and then rank that pain. There’s a good chance it’ll help you narrow your focus on what you need to bring on.
I’m going to create a little controversy here. You might want to take a sort of traditional IT procurement approach and say, “Okay, before we’re going to invest a million dollars in the observability tool dijour, we’re actually going to do some serious due diligence and some serious piloting and some serious type of RFP management” and that kind of stuff. There are a few cases where this isn’t true. There are a few cases where the answer’s pretty simple. There are industry leaders that everybody uses, they work well, they’re safe. But in a lot of cases that’s, that’s not the case.
What are some of the potholes in trying to build a DevOps practice from the tools that you already have?
It really all depends. I’ve worked with organizations that have had old tools. For example, they worked with a Jenkins that was 10 years old? It did the job for them. It really needed to be upgraded and that was what we ended up getting them to do. But after upgrading, they kind of realized maybe Jenkins isn’t the right thing for the job. 10 years ago, they had a team of five or six people. Now they have 40 people. And, and so it really all depends, if your original are showing process errors and overall functionality, it may be letting you know you need to upgrade or just look at another option.
Should our metrics focus on quantity (ex. number of bug fixes) or flow (ex. the time it takes to resolve bugs)
This is kind of one of those answers that kind of falls in between for me. Ideally, it’d be a mixture of both. You need to understand your average time to say,fix bugs, but you also have to take into fact from the developer’s perspective that not all bugs are equal. Some bugs are harder to fix than others. Some are really simple.
In my experience, if your organization is focused on how many, that’s pretty close to a root cause for agility problems, which is really what DevOps is about. One of the core principles of the Agile manifesto is that customer satisfaction comes from continuously delivering valuable software. And competitive advantage these days is about how quickly you can respond. And if you look at what DevOps is about at heart, it’s about delivering quality plus speed so that you can continuously deliver customer value, not just fix the stuff that you broke. But in, in my mind, optimizing for flow is what this stuff is all about. It’s what makes agile and agility go. It’s after that when we’re doing DevOps.
You also have to be very careful what you pick to measure. There’s an old adage that you can’t expect what you don’t inspect. For police officers if you tell, if you tell a police officer you didn’t write enough tickets today, tomorrow, that police officer is going to go out and write a whole bunch of speeding tickets, and then suddenly you have way too much enforcement of speeding tickets and not enough enforcement of the other things that a law enforcement officer should be doing. The metrics we select need to be designed to enforce behavior. Quality is another important metric to consider.
What are some of the best outcomes to tie success metrics to?
It’s not about huge leaps and bounds. It’s about gradual success. You’re going to have to visualize gradual success. Let’s say for example, releasing software. If you’re going from once a quarter, you’re not going to just start releasing every 15 minutes. In order to do that, you have to know where you are now. So you have to be able to look at where you are now and graph it, visualize it in some capacity, and then show your improvement over time with those gradual fixes. To me, that’s what success looks like, is gradual fixes to inevitably reach your end goal, which would be releasing software faster.
We can agree that the outcome at a high level is the flow of customer value. And I think to achieve success at that, you have to think about flow bottlenecks at every level and end to end. So I can’t tell you how many so-called agile/dev application teams I’ve worked with that have two week sprints and 10 day user stories. If stories are too big, you have flow problems. I can’t tell you how many teams I’ve seen, it takes 18 hours to run the automated test suite, but nobody is thinking about test suite execution as a scalable high performance activity. Just like any other piece of architecture, the truth is that we need QA engineers to be software engineers, and we need them to think about things like scalability and performance the way any other engineer does.
I can’t tell you how many teams I’ve worked with where the CI/CD platform is not a mission critical resource. If it’s four o’clock in the morning and you have a production bug that’s causing data corruption, you need to fix it quickly. If your Jenkins server or whatever your CI/CD platform is offline and the support team for it isn’t available, well then you have a problem. You need to look at every piece of the puzzle of how this is impacting flow and how it is impacting flow as part of what we’re actually selling to our customers. So it really is a mission critical activity. Everything we do to deliver software is actually part of our product now.
How does DevOps, CI/CD and agile engineering patterns fit into ITSM or change management?
Change management is a business process, right? People and the behaviors that they’re exhibiting are what we want to change, not adding more process and paperwork into the solution. So at most large companies, if there’s a change management process, it is because there’s a culture of not performing at a predictable rhythm, a predictable amount of output, let’s call it features, but it could be anything.
You start implementing DevOps practices that make your releases predictable so that the business people can see you’re releasing on a predictable schedule. You implement practices that allow you to make sure that you’re releasing reliable, tested code.
There’s a couple different angles of attack for change management? One of them is a cultural issue and this very heavyweight granular change management processes and systems can grow over time as a result of fear. So there’s a whole conversation that you can have around the human factors of change management. But as it relates to DevOps, I actually think it’s a much cooler thing to think about. How can you apply engineering to change management? Because the reality is, especially in large organizations, you often need some degree of change management.
Think about the way you might provision architecture. You might have a library of certain configurations or instances that you use, and then you might be able to push button deploy those types of things. Or you might even be able to set up triggers that provision that type of stuff. There are ways you can kind of do this with change management as well. Cprime is a big Atlassian partner for example and now you can use tools like JSM to pre-provision change requests and scenarios and automate them or make them available to go live with, with the push of a button. This list could go on but it’s very interesting to think about what are the common change scenarios or requests or needs that come up.
How can we advocate for agility and DevOps in our organization?
A way to be advocates for agility and spreading DevOps is to focus on outcomes. Where the outcome is flow and also, to put everything in terms of an incremental maturing process. This makes us become more effective at whatever it is we’re trying to do.
Don’t try to change the world all at once within your organization. Start with one group at a time. Show your value, prove your success with that one group. And, you know, with that proof, you’re able to spread the message to everybody else around you.
It starts with putting yourself in a place of influence, whatever your role or where you think that position may be based on your scope of jurisdiction. Even though you may be in a position that has no power and no influence, you still influence behavioral change in some capacity. Make, make the right thing seem easy and obvious to people no matter what it is. And you’ll get behavioral change.
Lastly, demonstrate value. Now there are multiple types of value, but I would always say that at some point in, at least in a commercial enterprise, you should certainly be able to tie a business case that’s monetary at some level. Now, there’s different ways to do this. You may be using more automation or faster cycle time to reduce risk, you may maybe deploying something new that that generates new revenue. You may be able to form an engineering pattern that reduces costs. These are all very viable things and people who have their hands on the levers of power are going to love any of those types of business cases.
Learn how we can help: