Kaggle is making AI benchmark creation effortless

As AI models evolve from simple chatbots into reasoning agents that write code, use tools and solve complex problems, traditional benchmarks are no longer enough. The community needs dynamic, rigorous evaluations — built by the people who use these models in the real-world.

That’s why we launched Kaggle Benchmarks. Since then, the global AI community has created more than 10,000 evaluation tasks, creating the trustworthy, transparent public leaderboards that help labs measure and accelerate AI progress.

Today, we are taking the next step by launching local development for Kaggle Benchmarks.

Use Kaggle Benchmarks from your local development environment

Until now, creating evaluation tasks meant working exclusively in Kaggle's web-based notebook editor, instead of developers’ preferred stack to build with.

Our new update enables developers to create, validate, push, run and download tasks directly from their local development environments like Antigravity, VSCode, Cursor and coding agents. This update is designed to meet developers where they work, making the journey from idea to evaluation faster and more intuitive.

Build evaluation tasks in natural language with AI coding agents

Local development also unlocks a powerful new workflow: using AI coding agents to write benchmark tasks through the write-kaggle-benchmarks skill. This skill comprises a set of structured instructions that teaches a coding agent how to build tasks using the kaggle-benchmarks SDK and the Kaggle CLI.

To add this skill to your agent, simply ask your agent to:

• “Install the write-kaggle-benchmarks skill: https://github.com/Kaggle/kaggle-skills”

Once installed, you can describe an evaluation in plain language and get a working task on Kaggle. For example, you can tell your agent:

• Using the write-kaggle-benchmarks skill, build a task that asks the model if "300+140=460 is correct?"

These powerful capabilities are driven by the new commands that we have built for Benchmarks in the Kaggle CLI.

Understand why community-driven evaluations matter

We built Kaggle Benchmarks to democratize trustworthy AI evaluations. We believe that if a capability can be measured, labs will race to improve it. By providing these clear, objective signals, our hope is to empower AI labs to drive model improvements in the areas that matter most.

For AI to truly benefit humanity, evaluations must reflect the full diversity of real-world challenges. We believe this launch is a significant step toward enabling anyone, anywhere, to build the evaluations that will shape the future of AI.

Ready to build? Try Kaggle Benchmarks today.

Additional resources

• Read the docs for kaggle-cli on GitHub
• Install the write-kaggle-benchmark skill
• 🎁 Idea → eval? Show us. Post your task + workflow by tagging @kaggle on X or LinkedIn by July 1st for a chance to win Kaggle swag and a social shoutout
• Watch the product demo on YouTube