Internal Developer Platforms for data teams: Closing the gap between infrastructure and delivery 

Ask an engineer how long it takes to stand up a new sandbox, and you get two numbers that do not agree. The hands-on part, creating the repository, wiring the pipeline, requesting access, could run to a few hours. But the calendar says a week and a half to two and a half weeks. Why? 

The time goes into the spaces between steps: a half-specified request that bounces back for clarification, or an approval sitting unread in an inbox. No cluster is warming up while that happens. Nobody logs that time, so nobody counts it.

That difference costs real money, and it slips past because it rarely reaches an invoice. Nobody is billed for the week a request spent in a queue. Since the accounts never show it, it runs unnoticed for years while it eats a large share of the engineering capacity already on the payroll.  Studies put the loss at 30 to 50 percent of engineering time, and Google’s SRE research measured about a third. In most data organizations nobody has put a name to it. 

Where the weeks actually go: the hidden coordination overhead in data teams 

Look closely and there is rarely one big blocker to point at. The delay is spread across a dozen small handoffs, each one a place where a request stops moving and waits for a person to act. None of those pauses is dramatic on its own but added up, they are the difference between hours and weeks, and four of them account for most of it. 

Provisioning runs on tickets, which leaves the platform team acting as a routing desk. Every environment, every access grant, every new project arrives as a request that a person has to read, chase for missing detail, and pass along. That team spends its days moving other people’s work through a queue instead of building anything of its own. 

Standards get reinvented on each new product. With no single place where the agreed conventions live and get applied, every team settles its own naming, its own access patterns, its own quality checks, and each version drifts a little further from the last. 

Approvals get stuck in email. The security or governance check itself is usually fast once someone sits down to it. What burns the time is the routing: the request lands in an inbox, gets forwarded, waits again, and quiet days stack up between steps that each take minutes. 

Audits then pay for all of it. With no system of record, showing who approved what and when means digging back through mail threads and chat history. 

It is worth stating plainly, because the same shape shows up at every intake point: new environments, access requests, new data products, onboarding a new hire. Short stretches of real work with long stretches of waiting and re-deciding wrapped around them. 

Cognitive load: why more tooling backfires 

Faced with delays like these, the reflex is to buy a tool: a slicker ticketing system, or one more portal.  

That usually makes things worse, and Team Topologies, by Matthew Skelton and Manuel Pais, explains why in a way worth borrowing. 

Their term is cognitive load: the sum of what a team has to hold in its head to get work done. They split it three ways, and the split is the whole point. 

Intrinsic load is the difficulty of the problem itself: writing the business logic, shaping the data model, settling the API contract. This is the work you want people spending their time on, and cutting it means cutting the value. 

Extraneous load is imposed from outside, by tooling and process: configuring pipelines, hand-writing YAML, working out which access role opens which resource. None of it is the job anyone was hired to do. This is the part worth going after. 

Germane load is different from both. It is the effort of learning the way your organization does things: its conventions, its patterns, the examples a new joiner copies from. You want to support this one, because it is what turns a new hire into a productive one. Good docs and worked examples bring it down. 

Once you hold those three apart, the buy-another-tool instinct falls apart with them. Every tool a team has to learn and maintain adds to the second kind. Hand developers one more portal to figure out and you have made the pile bigger. 

How an IDP differs from a wiki, a portal, and a catalog 

An Internal Developer Platform is meant to take that second kind of load away. The label gets blurry, though, so three common mix-ups are worth clearing before the definition. 

The first is the wiki. A wiki records how a process is supposed to work. A platform runs the process, from the form through to the provisioned resource, with the approvals applied along the way. 

The second is the developer portal. The portal is the screen people log into. The platform is the machinery underneath it that provisions, applies the standards, and keeps the record.

People reach the platform through the portal, which is a different thing from the portal being the platform. 

The third is the catalog on its own. A catalog that only lists what exists is a register: handy, but for someone trying to ship, one more screen to check and keep current. It starts earning its place once it is wired to the workflows that create and change the things it lists. 

The common thread runs underneath all three. Raw infrastructure, the clusters, pipelines, secret stores, monitoring, is the foundation and no more than that. A team still has to know which tools apply, how to set them up, which rules hold, and who to call when something breaks. Closing that gap is the job of an IDP. 

What an Internal Developer Platform actually is: three pillars, five planes 

With the problem named, the definition falls out of it.

An IDP stands on three pillars. The catalog holds the source of truth for what exists: services, data products, APIs, pipelines, environments and their owners, modeled so you can query them and follow how they connect. Templates encode the golden paths: ready-made, automated routes through the common requests, roughly the 80% of work that looks the same every time. A golden path carries a request from a form to a running, registered, governed result, with no ticket opened. Guardrails are the standards and approvals baked into those routes, so the rules apply at the moment something is created and the audit trail comes out as a side effect.

Seen from the build side, the platform is five planes: a developer plane where people work with it, an integration and delivery plane that provisions and ships, a security plane, an observability plane, and a resource plane of the compute and data underneath. Backstage, the open-source framework most platforms are built on, is only the portal across the top. One plane out of five. 

What holds it together is composition. The platform kicks off the work and keeps track of it; your existing tools carry it out. GitHub still runs the pipelines, the cloud still provisions the infrastructure, the identity provider still handles access. Nothing there gets replaced; what goes away is the manual coordination between them, which is where the weeks were going. 

For data teams there is a second gap. Generic developer-platform tooling grew up around microservices, and it has no native concept of a data product, a pipeline, a dataset, or a data contract. The tooling that ships with the data platforms, Unity Catalog, Purview, Snowflake Horizon, understands those things but governs only its own platform. In an enterprise running several, the IDP is the layer that spans them, with the same catalog, golden paths, and guardrails across the whole estate. 

This is the model the field has converged on. Gartner forecasts that 80% of large software engineering organizations will run dedicated platform teams by the end of 2026, up from 45% in 2022. DORA’s numbers say most of that has already happened: 90% of organizations use some form of internal platform, and 76% run a dedicated platform team. For most data organizations the question has shifted from whether to add a platform layer to how to do it. 

From tickets to governed self-service 

Go back to the start. The platform is fast, and processes around it are slow. Once that is clear, the useful question is what the operating model has to become. The answer is governed self-service: the routine requests handled by routes that carry their own standards, with the platform team owning those routes rather than working a ticket queue. The result is governed velocity: speed and safety by default, instead of one traded against the other. 

All of it leans on the catalog, since a golden path is only as trustworthy as the record of what exists behind it. The next article in this series takes that on directly: how an IDP models your estate as a graph, and how it keeps that graph honest as the systems underneath it change. 

There is one more reason not to let this sit. DORA’s 2025 research found that AI amplifies whatever a team already has, and that the quality of the internal platform is one of the things that decide whether AI spending returns anything at the organizational level. On a weak platform, AI barely shifts organizational performance; a strong one is what turns individual speed-ups into organizational gains. With 90% of software professionals now using AI at work, up 14% year on year, slow coordination is no longer only a delivery drag. It caps what the AI budget can ever give back. 

The bottom line 

The delay that makes new data products slow to start is a coordination problem, and more tooling does not fix a coordination problem. An Internal Developer Platform addresses it at the source: a catalog for what exists, golden paths for the work that repeats, and guardrails that keep the standards in place while teams move. The platform team stops running a queue and starts owning the routes the work runs on. 

If you want a quick read on whether your organization needs one, ask a single question: are several teams independently solving the same setup problem? When the answer is yes, the gap is structural, and a platform is what closes it.  

Proving it does not take a fleet-wide rollout. A pilot is enough: one team, one painful flow, the first golden path running end to end. The baseline comes from your own volumes and costs in the first week, and the result shows up within a quarter. If several teams in your organization are solving the same setup problem, C&F can help you scope the first golden path. 

FAQ

Is an Internal Developer Platform just Backstage? No. Backstage is the portal, the layer people log into, and it is one part of a larger platform. A full IDP spans five planes: the developer control plane people work in, the integration and delivery plane that provisions and ships, and the security, observability, and resource planes underneath. Backstage gives you the portal and the catalog; the platform is everything wired behind it. 

What is a golden path? A golden path is a pre-approved, automated route through a common request, such as spinning up a sandbox or scaffolding a new data product. It carries the request from a form to a running, registered, governed result with no ticket opened. Golden paths cover roughly the 80% of work that repeats, and less common cases still go through normal infrastructure access. 

Does an IDP replace the tools we already use? No. The platform orchestrates by composition: it triggers and tracks the work while your existing tools carry it out. GitHub still runs the pipelines, your cloud still provisions infrastructure, your identity provider still handles access. What the platform removes is the manual coordination between them. 

Will one more platform just add to what developers have to learn? That is the real risk with a portal bolted on top of everything else, which is why adoption matters more than features. A platform earns its place by making the default path the easiest one, so teams reach for it instead of working around it. Done well, it lowers the tooling and process load a team carries rather than adding a new layer of it.