How DevOps Transforms Organizations

Maciej Kłodaś (MK): Hi everyone, my name is Maciej, I’m the leader of Analytics Experience Competency Group at C&F and this is C&F Talks, a place where experts discuss their challenges and observations from the perspective of an IT vendor. My guest today is Konrad Madej, head of DevOps Competency Center at C&F. Hello Konrad.

Konrad Madej (KM): Hello Maciej, hello everyone.

MK: Today’s topic is something different, more technical, I would say. And I was in the office yesterday and I was talking to some guys and they knew that we’ll be recording a new episode today. About DevOps. And everybody said that, okay, they know the expression, but essentially nobody knows what DevOps really is.

It’s a kind of an enigmatic expression, because it’s kind of a merge of development and operations. So it’s something that, I don’t know, enables some processes, some tools that enable both the development and then IT operations. So I don’t know, bringing closer the gap between these two areas. So tell us, what is DevOps exactly?

KM: Okay, so you’ve got a really good point about being technical or maybe not so technical today, because DevOps really isn’t a purely technical thing. If you start thinking about DevOps, it’s actually more than that. Exactly as you said, DevOps is more about people, processes, and accompanying tools and automations that are created alongside of the DevOps development.

So whenever you think about DevOps, it’s more how the entire organization creates software, how the entire processes are built to make sure that we are doing this in an optimized way. Same way as we produce cars, we’re trying to reduce waste by optimizing processes, making sure that we are delivering things on time. The same way that we are thinking about producing cars and delivering parts to the line where the cars are produced, the same way we want to deliver software, to deliver things on time, at a particular moment of time, so that it’s the most useful.

And then just be aware of the entire process, how it works, so that we can deliver everything in a smooth manner without wasting time during our wait.

The switch in the way businesses approach IT

MK: Okay, so it’s not a magical technology that you implement and everybody’s happy, everything works. But until today, things were working as they were working. We didn’t have DevOps in place, and now it’s gradually evolving or being implemented in different organizations.

You are working closely with our clients from different industries, mainly pharma. So what is your perspective on the DevOps state as it is right now with our clients?

KM: I will answer in the usual consultant way. And I would say “it depends”, because it really is dependent on where our customers are in their journey. So the biggest thing that changed during the past 20 years or so is that companies that were focusing on their major business goal, like pharma industry, producing drugs, insurances, making sure that they can provide insurances to the customer, all of them suddenly became software development.

So they need to learn this new discipline and they need to really excel at it. And in the beginning, most companies treated software development and basically IT as a whole as a necessary cost to focus on their major goal. Right now, many of the customers have already acknowledged the fact that this is their advantage on the market.

The way that they produce software, the way that they can introduce good practices to their teams, the same way as they do for their regular core business will be important and crucial to their survival on the market and have this competitive advantage among their competitors. So in the past years, they were just trying to mimic whatever they could. They could see and they could adapt the good practices from the software development companies.

But as they started to develop their own ways of working, they’ve also started to notice that maybe they need some custom ways of working and delivering solutions to their customers.

MK: I guess it’s all about the maturity and awareness of things that need to happen in order to improve the process with our clients. Right. So what are the main challenges you identified while seeing how companies work?

KM: We still can see that there are many companies that still treat this IT thing as a necessary cost and they’re just trying to make sure that it actually works somehow. And the moment when they will notice that it is their competitive advantage will come, surely. But they didn’t acknowledge the fact that this software development thing is something that they really need to focus on as well alongside of their regular core business.

So they would work like they would do it in the 90s. So they would have software development team where developers produce software, then they would have operational team and just throwing the ready-made software over the fence from the developers to the operational team and just make sure that everything works.

But that obviously means that those teams are kind of separated. They don’t share knowledge to each other. And also, to some extent, they compete with each other, right? They’re trying just to throw over the fence the same issues that they have.

Like you don’t have proper running environment. You didn’t deliver software on time. You did software that has a lot of bugs, et cetera, et cetera.

MK: It’s pointing fingers. Whose fault was that?

KM: Yeah. And it really depends still on the structure of the organization itself. So DevOps at its core, it’s basically about making sure that people can work together and making sure that we are providing tools and processes to them that will make those things possible or easier.

The most popular myths about DevOps

MK: All right. So let’s try to break it down into key elements of the whole thing, right? So there are myths about DevOps, right?

One of them, the most popular one is that DevOps is for startups. What other myths? And I believe that might be a challenge when you are trying to convince your client to implement DevOps strategies. So what are those most popular myths?

KM: The first one you’ve already mentioned that it is for startups. It really isn’t. I mean, it’s obviously maybe a bit easier to implement DevOps at startups just because those organizations are smaller and they are ready to experiment and also acknowledge the failure.

So that’s why it’s usually perceived that it’s easier to implement the DevOps approach in startups rather than in mature organization, big organizations.

MK: They are newer, they are fresh, the learning curve is not so steep.

KM: Yeah. And they’re missing all these establishment that a mature organization has. So that would be the one. Another one is that DevOps, it’s about tools. If you ask many people about what DevOps is, they will tell you that, well, there are tools for DevOps.

MK: I know. Azure DevOps.

KM: Yeah. For example, that would be one of them. And obviously there are tools for DevOps, but it doesn’t mean that DevOps is tools.

DevOps requires some tools to be implemented that make sure that whole process will work smoothly. But tools is just one of the means to be able to implement DevOps approach fully in the organization. The other one is about that DevOps is actually just tools.

Whereas DevOps is really more about culture itself and the way we work with people and people work with each other than the tools that they are using. Because you have a lot of tools that cover the same areas. But if you don’t have proper approach, proper culture implemented in your organization, they won’t work.

How to implement DevOps effectively?

MK: So this is awareness also, right? So this is a change management that the company needs to take and needs to cascade to those teams to build this awareness that, okay, the way we would work in the future will benefit all of you. All right.

If you would say, okay, what are the main steps of DevOps implementation? Where would you start?

KM: So again, it depends. It depends what your role in the organization is. If you are just talking about a small team of development, like in startup, that would be a bit different approach than if you need to manage the whole software development department in large organization and you need make sure that you can properly implement DevOps in an organization of this size.

And in the middle, we also have like separate departments that we would like also to deploy DevOps. So yeah, if you think about the DevOps implementation in the organization, it really depends on your role and where you are right now. The maturity of the organization. Yeah. This is pretty crucial part.

So what we usually do when we start implementing DevOps, I mean, if we just have, if customer comes to us and asks, we would like to implement DevOps, then we would ask why. We would ask a lot of questions. What do they understand for the DevOps term and how they would like to approach DevOps implementation, what they think about it.

I mean, how they imagine they would implement this DevOps in their organization. Then having that knowledge, we have some idea what customer might expect from DevOps, right? And then we can tell him that this is good or bad approach.

Maybe it’s not really what DevOps is and maybe what he thought he or she would like to implement wouldn’t fulfill his or her goals in terms of implementation, why they would like to do it.

MK: Okay. So about the maturity. So I assume that there are a lot of maturity levels. So you are conducting some kind of an assessment when you are starting working with clients, you’re asking questions. So what are these maturity levels of companies?

KM: There’s no like strict framework saying what kind of maturity levels you might have. It depends on like even big providers like Google or Microsoft, they have their own assessment framework. So they would tell you what kind of maturity of DevOps or other ops like MLOps implementation you have in the organization.

And I think that we could start with like four levels maybe where the level one is that you don’t have DevOps at all. I mean, people work in some way, but it’s very unstructured. They don’t deliver in a structured and predictable way.

So like more or less like in startup where you just start working and you just focus on the goal, what you need to deliver, but you are not really focusing on how you are delivering that. So you don’t put too much attention on the tools that you are using to the processes. You are just trying to make this first version of your beautiful product and just push it to the market.

So that would be the first step. That would be the first layer where you don’t have anything working just yet. You are experimenting a lot and there’s basically no predictability of delivery of what you are doing. You can deliver the product once, but then if you need to make some change or fix or deliver a new version, that might be difficult because of that.

Then the second level of maturity would be where you have already implemented some tools, some processes. You can see that there is a value of it because you gain some predictability of delivery, but you are not measuring anything just yet. You can just see that, okay, we have some processes here in the development.

Our developers start to use Git for version control, for example. Okay. That’s great. We now know what we’ve done, which version we push to production.

MK: We can roll it back.

KM: We can roll it back. We can fix it if necessary, because we knew exactly which version went to production. But we are just not measuring anything in this process.

We don’t know how often we would be able to deliver a new version of software. We don’t know how much time we would need to spend to develop a new feature, for example.

The agile methodology in software development

MK: Okay. But is it about sprint planning, this agile framework implementation? Is it a part of DevOps or is it a part of software development in general?

KM: It’s part of software development in general, but the fact that we are using agile ways of working makes DevOps more useful in our way, right? Because we are focusing on delivering value more often, which means that we need to have some processes in place to be able to do so, right? Whether we have like waterfall process that where software delivery took one year and it was completely unpredictable.

MK: Black box, right?

KM: Yeah. Even though we had a plan in the beginning. That didn’t mean that we would be able to deliver according to the plan. And in fact, many of the projects were either late or out of budget.

MK: Constantly out of budget.

KM: Okay. I don’t think that this actually changed that much, but we are able to act on it much, much earlier when we work using agile practices to deliver software. We are able to deliver many, many more versions.

MK: And you can control the process more.

KM: Yeah. And see what kind of version you are delivering with the fast paced work that we are right now living in. It’s much more difficult to control the outcome in like one or two years, right?

And we want to make sure that we are delivering things basically now. We also want to make sure that what we are doing impacts how fast we can introduce new changes to the software, to the solutions that we have in the organizations.

It really needs to be quick right now, because it might be that something that we need to deliver must happen within next month or two. Take COVID for example, where fast reaction was crucial to deliver products like in pharma industry, the vaccines.

MK: It was a boom. Yes. I remember that

KM: Yeah. So it was crucial back then to be able to deliver something fast. You didn’t have like whole year to plan and release something really big. Instead, you need to work on it gradually because situation was also so unpredictable that you didn’t know what would happen within a month or two months.

MK: So I assume that companies that were more mature in terms of DevOps played it out better during COVID and during this fast paced development of digital tools to make this work happen rather than companies that were like back in the 90s, waterfall-like.

KM: Yes, that definitely changes a lot in terms of how fast you can act. You can even create new products from scratch because if you have those processes in the organization already established, it doesn’t really matter if you are using those for the existing products or for the new products. You can easily adapt it to develop new products, even if they’re like not one to one comparing to what you are developing right now.

MK: Okay. So level three?

KM: Level three is where you already are enlightened in terms of DevOps and you know what kind of needs you need to measure.

So getting back to our example, we would like to know that if we deliver a new feature to our software, it will be done or we can do it within two weeks time or one month time. So based on the history of changes that we’ve done, we can measure how fast we can move on with delivering new features and how much time do we need to develop something completely new or just small features and deliver it to production to the customers.

So we know what to measure and we know how to measure it. And we have all the beautiful dashboards saying that we’ve done that and we can see how much time do we need to spend on delivering to production, for example.

MK: So I assume that level four would be an organization which is DevOps driven of some sort.

KM: Yes. And that would mean for the organization that they not only have all the processes established, they know how to use them, but they also are fully aware of where their weaknesses are and how to fix them. So they have this feedback loop.

If you take a look at this, I think that pretty famous right now DevOps infinity loop where you have planning, developing, monitoring and deployment, then you would see that the most crucial part of it is that this is continuous process. So that means that we not only have those processes deployed in your organization, but you are in the position where you can constantly improve them and adapt to the changing external conditions, internal conditions of your organization. So you can constantly adapt.

And even in terms of making sure that we can keep up with the pace that we’ve created, even though the conditions change, right? Our environment changes, the market changes, and we still could be able to keep up.

Real-life examples of DevOps implementation

MK: We’ve been talking about building a white paper, a kind of a playbook of all these rules and maturity levels, benefits of DevOps. So this is a document that we will share with you. The link is in the description of our episode. But this is theory, right? It all sounds great. I’ve heard that before.

But how does it look in real life? Do you have some interesting use cases from your experience? How it looked when you are implementing all of these processes and tools and everything with our clients?

KM: Of course. I mean, as you may imagine, DevOps, because of DevOps being so complex, and there is a lot of tooling available, a lot of approaches to do DevOps in the organizations, there is no one answer to how to implement DevOps. It depends on what kind of processes you have, how do you work, what your organization is, on which level we are working.

I could give you one example of the company that we’ve worked with, where we wanted to make sure that they have good process of pushing their software to production, coming from different vendors. They’ve used a lot of vendors to help them develop software. And they didn’t have one established way of getting the ready-made software from their vendors and pushing it to production and making sure that someone is responsible for operating it.

So we had to not only assess what their processes were and how they work with different vendors, we needed to standardize the whole process and also make sure that the vendors that they were working with were also aligned with that. So it was a lot of the cultural slash people management thing, rather than introducing the tools themselves.

MK: So it’s not only an internal process that you are implementing with the client, but also you need to somehow communicate that and build this change management and control this process implementation with the vendors that are cooperating with the client, right?

KM: Yes, DevOps basically covers the entire software development process. If you think about it, it can help you with, so you can add a lot of prefixes to DevOps. It can be Biz-DevOps, Biz-Sec-DevOps, whatever works for you.

But there’s a lot of these names on the market. You could see them in many different places, but basically the idea is that we want to make people cooperate with each other. And the same comes to vendors.

If you work with vendors a lot, those are, you need to treat them as internal teams too, to some extent, right? So they are working on some requirements that are coming from your business, that are delivered to the developer’s team. And then they will create something, some artifacts, like working software, libraries, Docker images, whatever you require from them.

And you need to then deploy it and operate it in some kind of environment. And again, this might be your internal team of operations team, or there might be another vendor that is responsible for operating the software. So you just want to make sure that the whole process is well aligned, works nicely, and there are no frictions between those teams. Especially if they are external teams, that is a bit more difficult to manage.

MK: Well, different cultures, different way of implementing software, right?

KM: Sure. It depends on what kind of vendors you are working with, how your agreements are built and how they understand their responsibilities. All those factors are pretty crucial to build the nice and frictionless cooperation between those teams. So in that particular project, we had to make sure that the software that vendors deliver is of proper quality expected by the customer.

So we need to build in measurements of the security measures, but not only security, because we also wanted to make sure that the software itself is of proper quality. Because the way that customers work is that they had external vendors that develop software for them, but at some point of time, they would take it over all the sources of the software and they would continue to work with it by themselves.

So we just wanted to make sure that the vendor not only delivers the components of the proper quality, so they are secured, bug free, etc. But the source code itself would also be of the expected quality.

MK: Knowledge transfer, handover, everything.

KM: Yes, but this is on the latter part. The knowledge transfer is the last step that you are doing when you are actually taking over. What we had to do is that we set all the standards for the vendors that they would need to work according to those standards so that we know that when the time comes and we need to take it over, it will be of proper quality. Because you cannot just build in this quality at the very last step when you take it over.

MK: So some code annotations, the quality of the code itself.

KM: So for example, what we did is to use static code analysis tools to make sure that the code is, well, some level of quality. We did build the CI process for that so that we build the software in some specific manner that we would expect it to. We also added some security tools that we’re looking for security holes in the software vulnerabilities in the software that they were building so that we know that what they are delivering is already safe when it’s designed.

We are not just pushing this security to the very last step of the software building or delivery, right? It’s not that we want to report bugs that, hell, we just noticed that there is a vulnerability in your source code and you need to fix it. They should be able to fix it very early in the process because it’s also much cheaper than do it in the very last step.

The benefits of DevOps

MK: So if you would wrap it up around the benefits of DevOps, because I can read from everything you say, it’s saving time, saving money, better cooperation, better quality, everything. So what are these benefits? Can you name them?

KM: So you just named a lot of them, actually.

MK: All right. Perfect

KM: Did my job. But yeah, basically what we are striving for is to make sure that we are able to develop software cheaper and faster.

That’s the two major goals and of the proper quality as well. Right. So all those processes that we are working on is to make sure that we are able to deliver changes to the production in a fast pace.

It doesn’t mean that for customer it needs to be continuous deployment because in many areas like pharma, it’s virtually impossible to do that because of the compliance that you need to take care of. But you still want to make sure that if something goes wrong, you need to provide some hotfix or you want to deliver some new feature fast, you can do it. Right.

So that’s why we develop those tools, those processes, according to the customer needs. So we don’t need to aim for being able to deliver something constantly and provide this continuous delivery abilities. But we want to make sure that processes are aligned with the customer needs.

And maybe it’s just cheaper for the customer to not only implement, but also operate the process where we can deliver software every week rather than do it on a day by day basis. So that would be one, deliver it fast. If you do it fast, you obviously also need to reduce all the waste that is on the way.

So you should deliver it also cheaper than if you need to wait for software to be delivered, for software to be tested. Basically, every time you need to wait for something, this is a waste from the process perspective. So by reducing those waiting times, we also make sure that we can deliver software cheaper.

And by building in quality from the very beginning, we also make sure that this software is delivered of a proper expected quality. Again, also depends on the customer. Some want to spend more money on the security, some want to spend less.

But this is something that you need to think from the very beginning of the process. In the DevOps world, we call it shifting left. And it’s like shifting left to the very beginning, all those concerns that are important from the customer perspective.

But technically and like historically, we didn’t think about them in that way. So like quality, we had development, we had a testing. So the development went first, then testing afterwards.

Now, when you start implementing DevOps, you need to include those teams that test software from the very beginning, because you also want to make sure that you are not wasting time or money on inappropriate testing. So there might be some tests that can be automated and make it very cheap as a unit test in the CI pipelines. So cheap to run, cheap to design and execute, versus some manual tests that need to be taken after certain feature is delivered.

How to prevent the main risks associated with DevOps

MK: So you said costs and time. I’m very skeptical. So I don’t know if DevOps is the thing for me. What are the risks and what are the costs of implementing DevOps?

KM: Obviously, there are risks, you’d think. But one thing that you need to remember is that it’s risky because you might not be thinking about what you really want to achieve. So if you hear DevOps, you might think, okay, let’s implement everything.

Right, while in reality, it’s like a very evolutionary way of implementing approach and change the approach to developing software or even analytical products, because we also can talk about the DataOps, we can talk about MLOps. So whatever ops you can think of, it’s about incremental development and checking what kind of impact this produces. So when we talked about this maturity levels of DevOps implementation, we talked that the very last step is where you have feedback loops, right?

And this is the ideal scenario where you have everything already implemented working and you can benefit from having KPIs and measurements and are able to react and gather feedback from the production and implement changes in your process. But if you implement something from scratch, if you want to have the best results and avoid those risks of spending too much money on the development of the DevOps itself, you also need to implement this feedback loop from the very beginning, right? So even in a small scope, if you implement, I don’t know, version control, you implement some tool to manage your work like Jira or Azure DevOps for manager tasks, you also want to make sure that these tools are actually implemented in a way that helps you not make people annoyed by need of using them.

So risks are that one, you don’t know or organization that helps you implement DevOps can’t answer really what your goal is, why you are doing so. It’s like with any other project, if you don’t know why you are doing it, it’s really just a cost and you are not achieving anything. But if you start implementing it, you also want to make sure that you are using proper tools and you implement those tools in a proper manner and be able to validate that very early in the process that it works for you.

It works for the people that actually use those tools and processes. And that you are going in the good direction. So if you think about how you develop your products, you need to treat the DevOps implementation as a product as well.

MK: Can you estimate how long does it take to implement DevOps practices in the organization?

KM: Can I answer “it depends”?

MK: Yeah, sure. Of course. It depends. Perfect. Okay. Depends on what?

KM: It depends on where you are right now. So what we usually do when we start working with customers is we perform an assessment to be able to see where our customers are in the software development area or data development area, to be able to tell what kind of means they can take right now as low hanging fruits to implement immediately, what goals they can take short- to mid-term and what would be their long-term goals in terms of DevOps implementation. So I don’t know, if you implement DevOps in small teams like startup-like, it might take a couple of days to see some visible results.

Depends on where the team actually is right now in terms of their maturity. In large organizations, it can be years if you think about implementing everything on the organizational level, multi-department level.

MK: Okay. But I assume that you can take it step by step, start small and evolve. And usually it happens so that you implement DevOps strategy on a certain level, division level, team level, and then climbing up to the organizational level or how it happens?

KM: So again, it depends. But you should have a plan. I mean, if you are about to deploy DevOps on the organizational level, you should attack all those layers.

So you need to provide teams means to deploy certain aspects on their team levels. Then you have things that you need to deploy on the department level. And then you need to have a plan of what you want to achieve on the organizational level.

So it’s like multi-level implementation that happens simultaneously. But long-term plan, if you intend to deploy DevOps on the organizational level, you obviously need to have a plan on the organizational level, like strategic plan. Then on the department level, it’s more strategic slash tactical goals that you need to have.

And on the team level, you have tactical goals. So you implement it step by step on the team level with the tactical approach. But you need to have this strategic plan in mind when you work and think on those tactical goals on the lower levels.

So you will implement it incrementally, just as I said, with the feedback loops built in from the very beginning. But if you want to deploy it on an organization level and see the most benefit from DevOps implementation, you will need to work on an organization level as well.

MK: OK, so it might happen bottom up. It might happen top down. It depends on the situation and maturity of the organization, I assume.

Identifying whether your team needs DevOps

All right, so imagine I’m a manager of a team or a division, and I don’t really know where I’m at in terms of maturity. My team produces software. I have a lot of users that are benefiting from the software.

We need to build new features, new whatever tools, and introduce it to a larger group of users. So what would you say I should look for in terms of where DevOps might help or identify elements that I can identify in my team to say, OK, my team needs DevOps. So what would you say are the generic points to implement or to identify where DevOps is needed essentially here?

KM: So I would start with asking if there are any aspects that you are unhappy with in terms of your software delivery processes, right? And then basically we would just need to walk one by one and see what kind of needs you have, what kind of issues you have right now. I would ask you about what kind of development teams do you have, if you have any testing teams, how they operate, how do you manage teams that need to deploy and operate the software in production.

Again, if you have any issues with any of those steps, if you can see something that is obvious.

MK: Oh, yeah. The development takes more time than we assessed at the very beginning or estimated. We have a lot of bugs and well, we pay a lot more than we have predicted to pay. So this is the point. So I need DevOps?

KM: Yeah, This is the usual point. So, yeah, I would say that you need DevOps or you need some way of working to help you orchestrate the entire process, right? So you need to make sure that developers are able to work efficiently. So make sure that they can focus on software development itself, not need to think about other things.

They are not distracted by other things. And same applies to any other teams in your process.

MK: OK. I also have a lot of dependencies. I mean, we work across teams, cross divisions. We have different teams scattered across the organization, also external vendors. So this is also a scenario to think of DevOps implementation.

KM: Oh, yes, for sure. I mean, the more relationships and dependencies across teams and people you have, the bigger areas to have to be addressed there and to establish some specific rules, how those teams should work together and how they need to cooperate with each other to make sure that they can focus on their single goal of delivering the product. You need to make sure that everyone feels responsible and they feel ownership of what they are delivering.

MK: OK. I would say that everything you said seems to me that everyone needs DevOps in their organization of one way or the other.

You can implement or improve the processes and, you know, and optimize the way of working with all of the organization. There is no ideal setup, I would say, I think. And from your perspective and your experience, we see that even the most mature or leaders in the industry need DevOps strategies and processes to optimize the way they are producing software or cooperating.

All of these also use cases will be placed in the playbook that is and will be available to download with the link in the description. So thank you very much, Konrad. I think this is it for today and hopefully we see each other in the next episode.

KM: Thank you