What are internal developer platforms?
Internal developer platforms (IDPs) are developed by platform engineers to include a curated set of tools, implementations, services, and infrastructure that streamlines critical functionality and automation for development teams.
An IDP provides unified access to tools, resources, and documentation for development teams. IDPs can assist with onboarding and can help deliver the necessary tools, access roles, and connectivity to enable development teams to produce high-quality software. IDPs address key developer challenges by:
Improving the developer experience: Offering centralized access to tools, APIs, and infrastructure helps developers react and respond to increasing complexity and scale.
Enhancing visibility: Providing a single source of truth for service ownership, dependencies, and documentation, so development teams can quickly locate the information they need.
Standardizing development practices: Building in governance, scorecards, and golden paths helps ensure best practices and compliance across teams. \
Accelerating onboarding: Assisting new developers who can quickly familiarize themselves with available resources and services reduces time to productivity.
Reducing operational overhead: Providing self-service capabilities can help minimize the need for manual intervention from engineers or DevOps teams.
Key features for IDPs
For platform engineers, an IDP should include the building blocks of system components, access controls, connectivity to services, and the following essential features:
Software catalog: A centralized inventory of microservices, APIs, and dependencies, enabling easy discovery and governance. This catalog should keep development teams working with a set of cohesive and approved tools.
Self-service workflows: Automated workflows for provisioning infrastructure, deploying services, and managing access. These automated workflows can be built using scripting and pipelines.
Scorecards and compliance tracking: Built-in mechanisms for enforcing engineering standards, such as security and reliability. These controls should maintain code standards such as effective naming conventions, avoiding global variables, opening and closing connections, and using authentication keys and certificates.
Observability and insights: Integrated monitoring and alerting to help teams quickly detect and troubleshoot issues. Development teams should use observability reports to maintain best practices and code standards.
Golden paths and best practices: Opinionated guidance on preferred tools and workflows to encourage consistency. Documentation for this guidance should be quickly accessible for development teams.
RBAC: Permissions and security policies that align with organizational needs. Access controls should be regularly audited for security risks and to maintain compliance requirements.
Integrating IDPs with existing infrastructure and tooling
An IDP is meant, by design, to act as the bridge between development teams and the components of the software development cycle. Historically, development teams would request access or implementation of these components from DevOps, NetOps, and other infrastructure teams.
Implementations in this manner might fail to install the right components or services and could grant unnecessarily broad access privileges. IDPs consolidate these requests into formal, organized, automated building blocks by integrating with the following components:
CI/CD pipelines: To enable seamless deployments and visibility into build statuses, automated build scripts can grant the necessary access for development teams.
Observability platforms: Platforms such as Datadog’s observability platform can provide real-time monitoring and insights, enabling valuable tracking and insights into systems, components, security, and compliance.
Infrastructure management tools: Using automated scripts, development teams can be granted the right access controls to resources and services to allow self-service provisioning of cloud resources. Documentation can provide examples of connectivity and correct usage.
Service meshes and API gateways: An IDP can provision the right access rights and provide documentation to facilitate service discovery and governance.
Identity and access management (IAM) systems: IDPs can authenticate and provision development team access to enforce RBAC.
Use cases for IDPs using platform engineering
Platform engineering applies the concepts of IDPs to specific software development efforts and scenarios. Specific use cases include:
Microservices management: Platform engineering simplifies the implementation and scaling of microservices by creating an IDP for service discovery, load balancing, and configuration management. Without platform engineering, teams must manually configure and maintain microservices, which can lead to inconsistencies, incorrect or too-broad access rights, and operational overhead.
Cloud-native development: Platform engineering with an IDP automates cloud resource provisioning and governance, ensuring that infrastructure is efficiently managed, security risks are managed, and development adheres to compliance requirements. Without this approach, developers might need to manually request and configure cloud resources, slowing down development cycles and increasing operational, security, and compliance risks.
Security and compliance enforcement: Platform engineering through an IDP implements security guardrails, such as automated security scanning, policy enforcement, and access controls. Without platform engineering, teams might struggle with fragmented security practices, making it harder to enforce consistent compliance policies across teams.
Incident response and SRE practices: Platform engineering with an IDP enhances monitoring and debugging workflows by integrating real-time observability reporting, automated remediation strategies, and structured on-call rotation management. Without platform engineering, teams might rely on ad-hoc troubleshooting, leading to slower incident resolution times and higher mean-time-to-recovery (MTTR).
DevOps and CI/CD automation: Platform engineering through an IDP reduces manual tasks in deployment and release management by introducing automated pipelines, version control strategies, and rollback capabilities. Without platform engineering, teams might be required to manually deploy changes, increasing deployment risks and potentially delaying releases.
Industry shifts affecting platform engineering and IDPs
Changes in the industry are encouraging the deployment of platform engineering and IDPs in the following areas:
Rise of Kubernetes and containerization: Kubernetes has become the standard for container orchestration, with more than 60 percent of enterprises adopting it for complex deployments. Platform engineering enhances the implementation of these complex environments.
Focus on developer productivity: IDPs are gaining prominence as companies streamline workflows and enhance collaboration. Crucial to the success of organizations and their business goals, IDPs help development teams focus on accelerating software delivery.
Security-first mindset: With rising cybersecurity threats, embedding security into software delivery has become imperative. A 2024 Red Hat report indicates that two-thirds of organizations have delayed application deployments due to Kubernetes security concerns, highlighting the need for integrated security measures. Platform engineering and IDPs incorporate role-based access assignment to reduce security risks.
Adoption of AI and automation: AI-driven tools are being used to optimize performance, observability, and incident response. Platform engineering can utilize automation and scripting using AI-based tools.
Shift to multicloud and hybrid environments: As multicloud strategies become prevalent, platform engineering teams are developing solutions for consistent deployments across diverse environments. The average Kubernetes adopter now operates more than 20 clusters, often spanning multiple environments, to ensure availability and security.
Challenges affecting platform engineering and IDPs
Platform engineering and IDPs offer significant advantages but also present several challenges:
Balancing standardization with flexibility: Platform engineers must balance standardized workflows while permitting flexibility for teams to innovate and work efficiently within different environments.
Managing tool sprawl: Platform engineers must navigate and integrate an ever-expanding set of tools for diverse environments and services while avoiding redundancy and complexity.
Ensuring developer adoption: Building a self-service platform is only effective if developers use it. Encouraging adoption often requires internal advocacy, documentation, and ongoing support.
Scaling infrastructure efficiently: As technology and complexity for development increase, platform engineers need to ensure that infrastructure and tooling scale smoothly without introducing bottlenecks or degrading performance.
Addressing security and compliance concerns: Platform engineers must ensure security best practices and compliance requirements are built into development workflows while maintaining ease of use and developer autonomy.
Learn more
Discover Datadog Internal Developer Portal, which combines real-time telemetry with ownership data, repositories, and documentation, providing teams with a single source of truth for everything they build and run: “From Chaos to Clarity: Introducing Datadog Internal Developer Portal.”
