Internal Developer Platforms: The Missing Dashboard in Your Data Platform

Maciej Kłodaś (MK), Konrad Madej (KM)

Maciej Kłodaś [MK]
Hello, my name is Maciej. I’m the leader of Analytics Experience Competency Group at CNF. And this is CNF Talks, a place where experts discuss ideas, challenges from the perspective of an IT partner.

My guest today is Konrad Madej, the leader of DevOps Tech Practice at CNF. Hello, Konrad.

Konrad Madej [KM]
Hello, Maciej. Hello, everyone.

Maciej Kłodaś [MK]
This is the second time you’re the guest at CNF Talks, so this is my big pleasure to have you here. And well, today’s topic is very interesting, also close to my heart, because I’ve seen solutions, maybe not like that, but an idea of a solution like that, but only on the front end side of things.

And we’ll be talking about data. So tell me. Our clients, big companies tend to implement big, modern data platforms, and they think that everything’s fine because it’s modern, it’s new.

But it’s not like there are no challenges with that data platforms, right? There are a lot of different process obstacles, and you are here to address that. So tell me.

[KM]
Yeah, hopefully. Yeah. Thanks for having me again on the talk. Yes, today we would like to talk more about how we can make data platforms more developer friendly, maybe even business user friendly.

There are solutions in the software engineering that already address that issue. But in data world that historically is a little bit behind of the technology advances, those solutions are not so popular just yet. And we see that this is like emerging trend of doing so.

And they definitely help increasing developer experience across platform that cost companies a lot of millions of dollars.

[MK]
What is the developer experience? Because these platforms are modern, this experience should be perfect.

[KM]
True. So maybe imagine a car. If you think about the car right now, when you need to start it, it’s like push the brake pedal, push the button, and you’re done. If you think about the cars from 1920s, 1930s, there was like whole procedure of how to start the car, right? You use different knobs, different pedals before the car even starts.

[MK]
Flywheel.

[KM]
Yeah, that was the tricky one. So if you think about the data platforms as a car, so this is like vehicle for developers to work on the company data, to deliver some value.

They are modern right now, right? But imagine the car without any dashboard, without those buttons to start it, and you still need to go around the car and try to figure out how to start the car. Especially if you are new to the organization, there can be a lot of friction to even learn how to build new products using this million dollar platform. So in software development, this problem was addressed a while ago already by building internal developer platforms.

Those are solutions that are like dashboard, single pane of glass for developers where they can start working on building new solutions or maintaining solutions from slightly from the DevOps perspective, but also on the data side from the platform perspective. Because in the data area, we talk a bit less about the DevOps like in regular software development. We talk more about building platforms and using platforms as components to build solutions.

So by building this developer platform, we want to give developers eyes and ears to be able to effectively drive this vehicle that data platform is.

[MK]
Okay, but what is the real problem right now? What’s the situation like? What does this experience look like?

[KM]
So imagine being this data engineer that is tasked with creating new data product on the platform. And if you need to start a completely new solution from scratch, usually you need to try to figure out how to do it.

There is maybe a ticket that you need to create to get resources to be able to build your solution. There might be a repository that you need to create. There might be some template for this repository, but usually from our experience and from what we’ve seen, there might be like project that was done really, really well.

And other teams started using that one as their template. But this project was created by a specific team for specific conditions. It’s not meant to be a template for others, right? So there is really no one that maintains that template.

And over time, it might become obsolete. It might become not well adjusted to what we are actually doing in our organization. And people still keep using that one because they know that, well, it was good, right?

[MK]
Or it might not be so good.

[KM]
It might not be so good at all, as you said, or it might be obsolete, right? Because the team that created this template, it was for their specific needs. It wasn’t meant to be a template.

[MK]
Listen, people are working like that for years right now, right? So is this a real problem, everyone’s problem? People were working like this and are working like this.

[KM]
The problem that occurred recently is the scale. Because it’s not a problem if you are starting a new data product once per year and it takes like one or two weeks. Of course, it’s wasted time.

But if you think about how much time you would need to spend to automate it, it might be much more, right? For one engineer to try to automate something that is done once per year might not be an efficient solution. But if you think about creating new data products a couple of times per month.

[MK]
Like without clients, for instance?

[KM]
Yeah, this time really adds up, right? If one engineer spent one week per year on starting a new project, that was fine.

If you think about starting 10 projects per quarter, right? And then every single time someone needs to spend one week triggering the project, that’s a lot of time, right? It’s 10 weeks per quarter, it’s 40 weeks per year. So it’s almost like one year of wasted time. That’s why we want to start automating this to the point where it might still be a semi-manual process.

In a large organization, it will still require some approvals, it might require some manual steps. But we want to decrease the time to the point where it’s efficient, right? So it might take one day, it might take two days, but it’s still in the scale of the one year and number of projects started by the team. It still provides a lot of benefits and a lot of efficiency.

[MK]
Okay, so assuming that every organization has the same problem, but it’s kind of invisible because everybody’s working the way they work and they get the job done. So how do you spot the problem in large organizations?

[KM]
So there are different approaches that we take. Sometimes it’s like looking at the organization top down.

And then you can see how many teams you have, how many products are created in the specific time unit like month or year. What we do more often is we are working with bottom up. So we are working with smaller teams that are working on a platform.

And then we try to figure out what kind of problems they have. For example, they might have problems with, as I just said, with creating new projects. But it might be that there are like data science teams that have more problems investigating and experimenting in the sandbox environments.

So they need to have a sandbox environment that is easy to create and also very easy to destroy. Because those quite often are really experiments that take a week or two just to figure out whether this product even makes sense. They can build it. And then after that initial experiment, they would like to dispose of the environment, make sure that there is no flows running over time. There’s no data building up to the point where you can see how much it actually costs. But usually those pipelines are there.

[MK]
Nobody’s caring for them. They are just lying there and they’re burning money.

[KM]
Exactly. That’s also the problem. Because at some point of time, we would like to automate platforms to the extent where people can easily not only create things, but also destroy things. Or even make this destroy automated so that after one week time, when the experiment is done, it will be just removed.

[MK]
OK, so we can schedule it.

[KM]
Yes. And that also resolves another problem that we’ve seen in the organization that when the platform teams start working on the platform, they are really focused on creating new capabilities of the platform to deliver new values for the development teams.

When the platform is mature, they kind of spin from that to be operators of the platform. And they focus on managing the queue of the tasks that are provided to them by the development teams. So developers create requests to create the projects, maybe change the password, maybe rotate the keys.

A lot of stuff that piles up on the platform. So effectively, instead of be focused on constantly develop the platform and deliver new capabilities, they are focused on taking task by task and resolve people’s issues. So it takes time, not only for them, but also it takes time to wait for the issue to be resolved.

[MK]
Correct. OK, so what about the governance?

[KM]
So this is tricky because this issue is resolved partially by the platform itself. But the thing is that tools like IDP can help enforce those rules and standards that are defined for the platform.

For example, we can tell that for the particular problems, the solution would be to use… We have a platform build on Azure and we are using Databricks. There are people that maybe just came to our organization, they don’t know the full stack, they haven’t learned everything and they just try to figure out things from scratch, how to work on Databricks, how to create those data pipelines. And we know what we would like to do, right? I mean, we might have already a piece of documentation that describes how those solutions should be built.

When we have solutions like IDP, it’s not just a piece of documentation, it’s a writing code that someone can take and implement, use as a template and implement on top of it. So all those rules that we have defined are already built into that, so that all the standards that we have in the organization and we expect developers to align to them will be implemented as we expect them to be, right?

[MK]
Okay, but do you need a platform for that? You can produce a very crisp manual, a guideline and everybody should follow that? So that should do the trick.

[KM]
Yeah, so we basically are back to the initial point that I’ve made, that it really all depends on the scale.

If you have a platform, but you have very few development teams and you have automation and the teams are mature enough so they know what they are doing, they don’t spend a lot of time on creating new solutions, initiating some new data products, etc. You might not need that, right? It might be actually a burden time on building this kind of solution. But if you have enough scale, so the developers spend actually three, four days on initiating new solutions, they need to wait for the platform team for a week to do something for them, it might be a good case to actually start building IDP.

And when we are thinking about building IDP, it’s not that we need to build something really big in the beginning, right? In the platform, like in the products, we usually talk about minimum viable product. In the platform, we quite often think about thinnest viable platform. When we start from a single use case that is real pain for the teams and try to automate that one.

We build very slim platform. We can take any ready-made solution like software development. There is, for example, Backstage that is developed by Spotify and it’s a framework that allows you to create this kind of platform.

You can take a single use case that you have, deploy it onto the platform and see how that works for the team. If people are actually using that one, if this really is your golden path to go from A to B. And that means that it needs to be good. I mean, for people to use it, it needs to fulfill its purpose, right? Because otherwise people will start to try looking for the shortcuts, looking for alternative solutions and it won’t be really useful for any of the parties because people will not use it and you as a platform team will be frustrated because no one uses your product.

[MK]
Okay, but usually when it gets to those teams, it’s about autonomy. So those teams tend to build their stuff the way they build it, right? So this platform will centralize it?

[KM]
No, absolutely not. I mean, this platform is a single place for developers, but it also allows teams to build their own solutions to fulfill their own specific needs.

I can give an example. In one organization, we’ve been working with a specific data science team that build their solutions on the platform. And there was already a framework that was meant to be used by the data products, right? And data product team have been using that one successfully because it was really, really nice.

But for the data science solutions, it wasn’t so great because they have different expectations. They build a slightly different kind of products and they wanted to do something by their own. So we developed for them a blueprint describing how to work with their kind of solutions.

And on top of that, we started building automations for them. And that’s a great use case where you can have a single place where people can go, can see what’s available. Because this eventually will be a self-service portal, right? So everyone can go there, see what’s available, how to build different stuff, what kind of blueprints, what kind of solutions they can use it for, and just pick whatever is for their specific needs.

[MK]
Okay, from your perspective and from your experience, what is the most expensive symptom you’ve seen across our clients?

[KM]
I think that’s the one that we are discussing from the very beginning where we are actually creating new products. And then people, especially if you have newcomers to your organization, they don’t know how to do that. They don’t know how to start working with that one.

And what we usually do is we are offering our customer to prepare the blueprints that will describe how to build solutions on the platform. We are defining certain standards for the teams that we expect them teams to adhere to. And then this builds a great foundation for building this IDP.

Because that one, this is what IDP is built upon. So you need to have solutions already in place. If you don’t have anything built already, you need to discover how teams are using the platform, how they are building, what tools they are using, maybe what kind of libraries they are using in Python or any other language that they have.

And then just follow the path that teams are using. What we usually see is that whenever a team needs to start a new data product, they need to create a ticket somewhere. Whether it’s Jira service now, whatever.

Or even sending an Excel file to the platform team with fielding fields. And that often cause the back and forth communication because, oh, you didn’t fill that field properly.

[MK]
And delays.

[KM]
And eventually it takes even weeks to even start a new data product, data science product.

[MK]
Which is basically wasted time, right? Because you are just waiting for something that should be available from the very beginning. So tell me, because you said you need to have scale in order for this investment to be viable.

So what is the cost of such implementation?

[KM]
I would say it as a consultant, as I usually do. It depends. But the thing is that if you are really committed to building this kind of platform, you need to think about it as a product.

It can’t be just a project that you assign some people and after a month or so they will be let go. You need to have a team that will build and then operate this tool. And from our recent example, we had a team of three people building the IDP for three months.

And after three months we had not one pipeline, but a number of pipelines that are designed to build solutions for development teams. The thing with that customer though was that he knew what he wanted. So that was easier to build.

And we knew what kind of golden paths we had to build. So we knew the solutions that needed to be created. There was already some pieces of the infrastructure existing for that, like Terraform code to build the infrastructure necessary for the projects.

So we really needed to just build the orchestration layer on top of that. So create a portal where developers can go, can see documentation of what can get out of it and click, fill in a couple of fields. And within 20 to 30 minutes they had a ready solution to start building a new product.

[MK]
Okay, but depending on the cost, I assume that the bigger organization, the bigger pipeline of projects, of data products, the bigger the cost. But what is the return of investment? I mean, when do you see the breakeven point?

[KM]
So that also depends on not only the scale of the organization, but the repeatability of the solutions that we have out there, right? As we mentioned initially, if you need to wait a week as a data engineer to create a new product and this solution will give it to you in one day, that’s like four days saved every single time you need to create it, right? So if you create, let’s say, 10 new data products per quarter, which is 40 per year, isn’t really unusual for a large organization. 40 times four days is 160 days.

So that gives you some perspective on how much time you could potentially spend on building the platform. Because the problem is that it won’t show how much money you can get out of this platform. It shows you how much money you can save.

[MK]
It’s usually harder to sell this internally.

[KM]
Harder to sell, yeah, correct. So you need to have really good proof that it actually saves people’s time, right? On the other hand, if you show how much time nowadays, at any given day developers in the organization need to wait for, have something new or do certain action, it’s fairly easy to show how much you can save by having this solution.

[MK]
But if it’s fairly easy to see that, because this is everybody’s problem, why didn’t anyone think about it earlier?

[KM]
People start thinking about it and, I mean, they do it, but not in an organized manner. Because they try to automate certain stuff, but they usually don’t do a central solution. Because usually in organizations you already have this kind of solutions, like CMDB for registering products, but they don’t operate on the infrastructure and product level to be able to show you what exactly you have per product.

What kind of resources do you have, do you need to create to be able to start working on your product or maintain your product? This is the problem that’s been there for a while. Organizations try to solve it, but they don’t often know that this kind of solutions exist. And to be honest, for the data platforms, for the data world, there isn’t much solution that really are aligned with the data platforms world.

Because usually if you think about IDP and if you ask people about the IDP, the immediate answer will be about software development, about creating new solutions, running it on Kubernetes, GitHub Kubernetes and that sort of stuff. Very rarely someone will tell you that this kind of solution is really usable for the data platforms and for the data teams. And on the other hand, providers like AWS, Microsoft, also creates perception that if you just use their tools, like use on Fabric, use built on Databricks, you will get everything that you need to be able to create solution end to end.

[MK]
It’s not true?

[KM]
It’s not really true.

[MK]
Surprise, surprise. Oh my goodness, really?

[KM]
But yeah, usually in organizations, you will find that the technological stack for building these data products is not only Databricks, it’s not only Fabric, right? Usually it’s a combination of a number of different tools that you have.

And the complexity of those solutions is also much higher than you can see on the demos from providers like Databricks or Microsoft, right? Where those solutions that they usually show to people are fairly easy. If you try and start scaling it up, it will become more complex.

[MK]
But okay, about the complexity.

I assume that you might have like 80% or 90% of fairly standard projects. But what about this 20 or 10% of non-standard projects and non-standard needs for the teams that will need to customize this approach? What about that?

[KM]
At least initially, we most likely shouldn’t be taking care of them because it’s not worth the time. There is like this chart from XKCD comic where it shows how much time you need to do the action and how often you need to do the action.

So it’s worth automating it, right? We can probably show it somewhere. But basically, if you have some actions that are happening once or twice per year, it’s not really worth automating them.

[MK]
But it comes with a risk.

For instance, security risks. When you have this IDP platform, I assume that you are taking care of the security and compliance, right? What about potential attacks on those non-standard elements?

[KM]
Well, the fact that it’s not automated doesn’t mean it’s not documented somewhere, right? And it doesn’t mean that there isn’t a team behind the platform that will still operate on top of it. So whenever we build standard or non-standard solution, there is always someone that will take care of that one, right? Even for the standard ones, you know that you have those guardrails built in, into those templates.

So it’s easier to enforce whatever is required. For non-standard solution, for custom solution, there will be some work required to be able to adhere to those standards. But, you know, it’s still there.

It will just remain on the level that we see right now, where everything is treated like a custom solution, right?

[MK]
But you can’t monitor those custom solutions. You can’t monitor them the way you can when they use the IDP platform, yes?

[KM]
Yes and no. So you probably won’t be able to automatically create all of that, right? But if you think about registering those solutions in IDP platform to be able to see that they are there, it’s perfectly possible and you can and you probably should do it as well.

Because we focus a lot on saying that IDP goes mostly for the automation part, which in reality is just one of the pillars of the IDP platform. Because when you think about IDP is that, for one, it tries to automate stuff, right? So it helps you orchestrate whatever processes that you already have to be able to work on your products, to maintain them, etc. Then another pillar is to create the catalog of whatever is built.

So create a catalog of source code repositories, of infrastructure pieces that are created for a particular product.

[MK]
So all the governance stuff.

[KM]
Yes.

And then you have great visibility onto what product use which components, which isn’t really that common even nowadays. Especially if you think, I mean, on the infrastructure, it can be fairly easy because usually you have all the solutions that are necessary to tag the resources and to be able to relate them to the specific products. But think about GitHub repository.

How often do you know which source code repository is used for a specific product?

[MK]
Not so often.

[KM]
And IDP allows you to gather all of that in one place. So with automation, it happens automatically.

So when you create a new product, for example, everything that is created alongside of this process will be automatically registered into IDP. So after creation of the new product, you are fully synchronized. You have all the data that you need to have for this particular product.

If you do some custom stuff, you can still integrate with already existing solutions to be able to import those information into IDP. So even for products that are not standard, you still have capabilities of having them fully registered, knowing who owns the solution, who is technical leader of the solution, what kind of infrastructure was created for that, even if it’s completely custom.

[MK]
Okay, but this is theory.

It looks and sounds really, really nice. But how does it look in practice? So what does it really change?

[KM]
It provides you with additional or, again, single place that will show you all the components that are created for your solutions. So I’ve mentioned CMDB before, right? That is Configuration Management Database.

In many organizations, it is used on the product level. So we know that, well, I have created this and that product. You probably went through some enterprise architecture review.

So you know that this solution adheres to the standards of the organization. Maybe if we are talking about building some data products on the ready-made platform, you don’t need to go through that because it’s already pre-approved with some specific configurations and architectures. But you know it’s there, right? What you usually don’t know is what kind of components you have.

Specific solutions. And this is the gap that IDP tries to close. And that doesn’t mean that this IDP exists in the void. You need to create everything from scratch. What you want to do is to integrate with those systems that are already in the organizations to be able to link information from CMDB, from the cloud, to be able to gather information. And again, that doesn’t mean that you will be copying information from CMDB. You just want to create link that, hey, for this specific product, we already have this entry in CMDB that describes all of this technical slash architectural stuff for it, right? And you know that this product also involves creating specific storage in Azure, specific repository in GitHub, specific repository in Databricks or whatever else is needed for the solution. But in a single spot, you can gather all of that knowledge and be able to review that as needed. For example, you need to do some kind of assessment or you need to do audit of what is running in your platform because, you know, you might want to just cut some costs out of it.

[MK]
There is auditing, I guess, every once a year, at least to see how much money you spend on the product, right? How it usually starts. But it’s long and painful.

[KM]
It is.

[MK]
But why?

[KM]
You need to find out what kind of solutions you have running on the platform and what kind of resources they involve, whether anyone uses them. Usually it’s a lot of manual work to do that, right? Even to start and create this inventory of solutions that are out there. With IDP, this step usually can be omitted because you will have this inventory already in place.

So you won’t need to recreate it every single time there is audit needed or there is like inventory is needed for some specific purpose. Moreover, if you’re working on solutions and you have different teams for building and for maintaining solution, these transitions between teams can also be made easier, right? Because when you think about putting your solution to the service, to the DataOps team, it takes some time for the DataOps team to learn how the solution is built, what kind of workflows are there, how it works, whether it’s built up on standards, etc., etc. With IDP in place, this knowledge is at least partially already out there because it’s cataloged properly. And because there was some automations used to create a solution, you can also expect that it adheres to certain standards. So that also makes this transition processes easier because there is a lot of knowledge inside of this solution. So team don’t need to go and ask for it specifically.

[MK]
Okay, so you can just generate a report from IDP instead of conducting a very long and painful audit.

[KM]
Correct.

[MK]
Can you decommission a product directly from the IDP?

[KM]
Technically, you should be able to do that. I mean, you can build this kind of capability into IDP. Not everyone would like to have it in place, right? Because it can be a risky process. So it needs to be properly gated if you want to have it automated. But I mean, for sure you can automate this process. The question would be, how often do you do that? And if it’s worth fully automating or just having proper documentation for it. But IDP for sure will give you information about what you need to decommission, right? Which is also not always very clear.

[MK]
Okay. And you said that IDP is a product and you need to treat it as a product. So you need to have a dedicated team to maintain it, to keep everything tidy, right?

[KM]
Correct.

[MK]
Who is responsible for blueprints and for new blueprint versioning or whatever?

[KM]
That’s really up to you. I mean, depends on how you organize your teams working on the IDP. The IDP team will be definitely responsible for making sure that products is up and running, it’s up to date. And all the templates that are out there are also up and running and also up to date. But that doesn’t mean that they need to create everything by themselves. Because as we talked before, there will be situations where you want to have development teams creating their own templates and making sure that this is usable for them.

So the IDP team will be more of a governance team that will make sure that everything that is built in the IDP platform is up to the rules and up to the standards that we expect.

[MK]
Okay. And what about the more demanding projects like machine learning or AI projects?

[KM]
It really boils down to more or less the same things in the infrastructure. I mean, you will be using different infrastructure pieces for creating ML and AI products. But all in all, you will just have separate templates to create AI solutions. So it doesn’t really matter if it’s…

[MK]
Yeah, we’re still talking about data and about some pieces of software, right?

[KM]
So it’s not so much different of what we are creating as a data product or data science product on the platform.

[MK]
Right. So you said that it’s not about earning money, it’s about saving money. So it’s not so easy to show those savings and to sell it internally. So how do you sell this idea to a CTO, for instance?

[KM]
So I think that we have two major personas that we can talk to from two different angles. One will be the data platform owner, which is really interested in making sure that his people or her people are working on a platform development rather than just managing the queue of tickets that they are getting from the development team. And the second persona would be the development team leaders that are working on the platform because they also want to make sure that their engineers are focused on delivering value rather than making sure that all the configurations and all the new products are set up in time.

So as a data development leader, I won’t be responsible for any budget. So I won’t say that, well, it will save my budget here, but it will definitely save time of my engineers. So we will be able to create more products. So I can definitely go and influence data platform owner or CFO or whoever else I have in my organization to say that, hey, if we build this kind of platform, it will allow me to build five, ten products more per year because of the time that we won’t be spending on creating tickets in ServiceNow or in Jira. And we’ll be focused on developing products instead. As a platform owner, I will be definitely interested in building this kind of product because it will allow my people to focus on working on the platform development rather than managing the tickets and just resolving created products, which is really repeatable actions. So I’m not really interested in doing so. So again, I can tell that, well, if we have this product and it will cost X for development because we need to hire two or three people to be able to build it and run it, I will be able to refocus my people to actually develop new capabilities for maybe those AI slash ML capabilities that we always wanted to have on the platform. So major selling point would be for me showing that if we have this, by this I mean IDP, we can deliver new solutions faster within predictable time frame. And we can also create stricter SLAs for our development teams when we are delivering them new capabilities. So when we are delivering them new place to work on new product, new data science product, new AI product, etc.

[MK]
Okay. So essentially we can build more with the same capacity.

[KM]
Yes.

[MK]
Okay, cool. So how do you implement such platform then?

[KM]
So, yeah, as we discussed earlier, we want usually to start with something that is really thin. We don’t want to make it a big bank deployment. Instead, we are usually just picking a single problem that is really painful for either platform team or development teams, or ideally for both of them. So things like creating new data product, right? Because on one hand side, you have this development team that will need to wait one week to be able to get all the necessary resources. On the platform side, you have team that already have cute number of tickets that they need to work on. And then it takes, let’s say one day to create all those infrastructure, which underneath means that they need to run number of Terraform pipelines to be able to create this infrastructure needed, and then send messages to development team about what they’ve created and where the development team can find it. And having this single scenario, we implement it and we check how that works, if that teams are happy, if platform team is happy. And then if they are happy and they should be, right? But if we build it properly, we can build next capabilities. So we can start integrating with the mentioned CMDB, we can integrate with Azure to gather all the information about what we are building. So to build this database with information about what do we have on the platform. And then from there, we can build automations next solutions so that those solutions that are maybe not that often created, but still we need to create them and it takes a lot of time, we can automate them. And, you know, even adding some simple stuff like change my database password, this is also something that can be make a self-service action. If you think about the organizations, it was like a couple of years ago, it required you to write to IT to be able to change your password on your domain. It wasn’t that easy as it is right now. So it’s the same things basically, that you need to communicate to someone from the platform team, they do something for you, and then you receive information that it’s done.

Our goal would be to make this available on the portal so you can access it, you can do it by yourself, but also be able to learn about it. So the IDP also provides you with a place where everything is documented because that also could be a problem in organizations that they have documentation, but it’s spread across different places and it’s difficult to find, to learn about what do I need to do actually to create a new solution. And IDP also allows you to in a single spot so you can learn about what you need to do and just do it.

[MK]
I have one question now, which might be challenging, because with every self-service capability like BI self-service, one of the biggest challenges is that, well, everybody can build solutions right now, but there is no approval process, so they are building multiple different reports, for instance, right? And the cost grows of upkeeping those reports and they are not being used at all. So what happens with IDP platform? It’s self-service. Is there any kind of approval process or people can just, you know, build their solutions and then abandon them and the cost is there.

[KM]
So the fact that this is self-service doesn’t mean that there wouldn’t be any approvals in place if it’s needed, right? It’s still because effectively IDP allows you to orchestrate this process end to end, which means that for organizations it is still and will be always important to be able to control who can do what. So initially you can get some of the automations to be used only by the specific teams, only by the specific people. So not that everyone is able to do everything, but maybe for some like solutions that require spending some money, like creating new workspace for BI, you can create a simple approval so that someone is still responsible for approving and have a chance to review what people are requesting and maybe tell them that it’s not a good idea to create that one, right? So the fact that we have IDP doesn’t mean that we have no control over what is created and by whom.

[MK]
Okay. I feel secure now. So imagine that I see the challenge in my organization and we start off on Monday. So how fast will I see results and value?

[KM]
So again, depends on where you are. If you have already well-defined process and you know exactly what step needs to be taken to create something, setting up new IDP, even as a, you know, a demo or POC instance to show the actual value, this is something that can be done within week, two weeks time. So if we are talking about large organization, it basically means that it’s lightning fast and then you can see some value out of it. If you commit that you want to really build this kind of stuff, within three months, you can have like fully workable solutions that will start delivering value and showing on the KPIs that, well, we’ve saved that and that much of time by not creating this manually, but doing this automatically instead. And then obviously you will need someone to, if we’re speaking about treating this kind of solution as a product, you will need someone to maintain the solution, build new capabilities. So this is really up to you, but I think that at least, you know, one or two people should be there to be able to run this solution over time to properly maintain it, add new capabilities and make sure that it’s safe and sound as we move forward with developing platform, developing new products, et cetera.

[MK]
It all sounds super beautiful, but what are the biggest risks from your perspective?

[KM]
I think in large organization, the biggest risks are usually not technical ones, but organizational ones, because you need to really have buy-in from various parties to be able to successfully build, deploy and maintain these kinds of solutions, right? So you need to have the platform team, development teams, maybe even some external teams that are doing things for you, like, I don’t know, maybe just creating for you GitHub projects, because there are situations where their team is responsible for this kind of very narrow things in the organizations to maintain them properly. And if you want to show that you are able to automate it and create new solution within one hour, you need to have buy-in from every single party and you need to make sure that you are actually able to automate it to the point that it wouldn’t require someone to approve every single step that is not necessary anymore, right? So from my perspective, having proper buy-in and convince people within organizations that where we are building will benefit everyone is the biggest challenge. And quite often it also causes the failure of this kind of projects, building this kind of products, because team still want to govern the pieces of infrastructure, pieces of their responsibility by themselves. They won’t allow anyone else to doing so automatically, because, you know, it’s their field, right?

[MK]
So they are just scared, like they might be scared of AI that will take their jobs.

[KM]
Correct. So if you have some kind of automation, people are usually scared that it will just take the job from them. And we are talking several different teams that might become unnecessary, maybe not unnecessary, but they will not need as many people to just operate the queues in Jira or in ServiceNow, right? Instead, they will be focusing on developing something. For sure, it can cause some disruption in their work and they will try to hold you off with building this kind of solution.

[MK]
Okay. From what I see with, and this is a pattern I spot, when we talk about different solutions like that, programs, I would say, they’re always, the biggest challenge is always people and organizations. So the change management there is one of the crucial points to address during the implementation process. My last question is about how you see the future. Do you see vendors looking in the direction of IDPs or, you know, clients building awareness that such platform is indeed something that they need to implement?

[KM]
Yeah, for sure. I mean, having these solutions already well established in the software development market will definitely cause these teams and these people to start looking at the new areas that are not properly addressed till now, right? So definitely the existing solutions will start to be used by the data platforms, by the data world in general. But also we can see that vendors like Snowflakes, Databricks or Microsoft, they are starting to address this solution, this problem more widely by creating the fabric that covers everything end to end and tries to properly govern everything from start to end. These vendors are starting to address that one. So they definitely see the problem. And by that you can also expect that there will be more solutions coming up on the market and addressing this specific problem, right? So we won’t need to adjust the software that is already out there for software development, but instead you will have some dedicated solutions that are specifically created for the data world and for the data products.

[MK]
Okay. All right. Thank you very much. This was a very interesting topic. Thank you for being here again. Hope to see you soon. So thanks for being here. This is all in today’s episode of CNF Talks. Be sure to leave a comment, leave us your feedback and tune in for the next episode of CNF Talks.