Datadog Scorecards Improvement

_YEAR
2026
_{PROJECT TEAM}
1 designer
1 product manager
2 engineers

The Scorecards feature in Datadog’s Internal Developer Portal (IDP) lets platform teams define engineering standards and evaluate whether services, repositories, and other entities meet them.

For example, a rule might check whether a service has an owner or an on-call team. It could also check for monitoring, documentation, or appropriate deployment settings.

Each rule shows an overall score alongside individual entity results, such as pass or fail, so teams can see what needs improvement.

Users could see whether something passed or failed, but it was hard to understand why.

Built-in rules were easy to enable, but their exact evaluation logic was not visible. Custom rules were flexible, but required users to manage evaluation logic outside Scorecards through Workflows or APIs.

One customer told us that most of the actual work happened elsewhere. To them, Scorecards was mainly a dashboard for reporting results.

I started by looking at how users created and maintained rules. I sent a survey to 86 users, received 12 responses, and followed up with six interviews to better understand their workflows and challenges. Participants included architects, SREs, platform engineers, and software engineers.


I looked across the entire workflow, including how users worked with Scorecards, wrote rules, managed external systems, and investigated failures. I also explored what they would need to write and manage rules directly in code.

Alongside the user research, I reviewed how related products approached policies, templates, evaluation logic, troubleshooting, and change history.

Four themes consistently emerged. Rule logic was often difficult to understand because it was hidden or managed elsewhere. External systems also required significant effort to set up and maintain, including mapping results to the right entities, handling API and data changes, and identifying entities missing from evaluation. When creating a rule, users also struggled to understand how rules, levels, scope, and evaluation methods worked together. Finally, they wanted a way to test new rules on real entities before applying them.

Together, these findings pointed to a connected workflow for understanding, creating, testing, and maintaining standards. A query editor would be one part of that experience. It couldn’t solve the whole problem on its own.

I turned the research into three design principles. Make Datadog’s rules easy to understand by connecting each result to the rule’s source, logic, evaluated data, and execution. Let users create rules directly with data already in Datadog, reducing the need for separate pipelines. And provide guidance so users can write and test their evaluation logic with confidence.

These principles also helped define the MVP. Even as we reduced the scope, we wanted to keep the core workflow connected: understand a rule, write it, and see how it works.

The existing rule management page showed which rules were available, but it was difficult to understand where the logic came from or how each rule was evaluated.

^{︎︎︎ Old rule management page}

I kept the familiar layout and added the rule source and evaluation method directly to the table, making these differences easier to identify at a glance.

The rule detail page was an opportunity to make evaluation logic more transparent. The existing experience showed basic information and results, but the logic itself was missing and the information felt disconnected.

^{︎︎︎ Old rule detail side panel}

I first explored expanding the existing side panel and allowing users to edit rule logic directly within it. This would keep users in context, but there wasn’t enough space to review complex logic, investigate issues, and support new functionality. As more information was added, the panel quickly became crowded.

^{︎︎︎ Early exploration of expanding the side panel}

I moved the rule details to a dedicated page instead, organizing the experience into clear overview, outcomes, and evaluation sections. This brought the rule source, execution details, and evaluation logic together in one connected experience.

Users had to create a rule with its basic information first and configure its evaluation logic afterward.  That meant they were creating a rule before they could fully understand how it would work.

^{︎︎︎ Old rule configuration page}

I explored several approaches before reaching the final design.

I first explored a step for defining evaluation logic through form fields, with the underlying code shown in a separate view. Technical constraints and differences across rules made this approach difficult to implement, so I moved toward letting users write and edit the code directly.

I also explored how the preview could support rule creation. It started as a way to show which entities were in scope, then evolved to provide guidance and display query results. This allowed users to understand what to write and see how their logic evaluated each entity before applying the rule.

In the final design, users could choose an evaluation method and configure it within the creation flow. For Datadog SQL (DDSQL), they could write and test queries, use Datadog’s AI features to help generate SQL, and preview results before completing setup.

^{︎︎︎ Updated rule configuration page}

The creation flow adapts based on the evaluation method users choose. They can write and test DDSQL directly in Datadog, configure evaluations through Workflows, or use the Scorecards API to submit results from external systems.

Each option provides the relevant setup experience while keeping evaluation methods within a consistent rule creation flow.

The final design brings evaluation setup into the rule creation flow. Users can choose an evaluation method and complete the relevant configuration in one place. For DDSQL-based rules, they can write and test queries and preview results on real entities before applying the rule.

The updated detail page brings together the rule’s source, evaluation method, results, and available logic and execution details. The management page makes each rule’s source and evaluation method easier to compare at a glance.

Together, these changes are designed to make rules easier to understand and reduce the need to build and maintain evaluation logic in external systems.

I shared this direction with customers and received positive feedback. They valued having the evaluation logic visible on the same page, which gave them a clearer understanding of how a rule worked. They also liked having example scripts they could run right away, which they felt would make it easier to start writing their own logic.

Overall, the feedback reinforced that making rules more transparent and easier to create addressed real customer needs. One customer described the direction as a “game changer” for how they could use Scorecards.

Looking back, this project reinforced a few lessons about understanding technical users and adapting to constraints along the way. Two stood out in particular.