AI Coding Challenge Yields Surprising Results — and a New Standard for Open Models
Brazilian Engineer Wins K Prize with Just 7.5% Accuracy
The Laude Institute made waves on Wednesday by announcing the first winner of the K Prize, a new AI coding competition focused on real-world programming tasks. The contest, launched by Databricks and Perplexity AI co-founder Andy Konwinski, awarded $50,000 to Eduardo Rocha de Andrade, a Brazilian prompt engineer.
However, the most headline-grabbing detail wasn’t the payout — it was the performance. Andrade secured the top spot with a score of just 7.5%, highlighting both the difficulty of the test and the limitations of current AI coding models.
“We’re glad we built a benchmark that is actually hard,” said Konwinski. “Benchmarks should be hard if they’re going to matter.”
What Makes the K Prize Different?
Unlike typical AI coding benchmarks, the K Prize emphasizes real-world, real-time testing. It’s modeled after SWE-Bench, a GitHub-based benchmark that evaluates models on software engineering tasks. But while SWE-Bench uses a static set of GitHub issues — allowing models to train on the test itself — the K Prize uses a “contamination-free” design.
To maintain this purity, K Prize submissions were locked in by March 12, and the organizers built the final exam set using only GitHub issues flagged after that deadline. This means no pre-training advantages — only the true capabilities of the models on unseen tasks.
Tougher Than Existing Benchmarks
The competition’s outcome has shed light on a growing concern in the AI world: benchmark saturation. In comparison, SWE-Bench currently shows a 75% top score on its “Verified” test and 34% on its more difficult “Full” version.
Konwinski acknowledged the stark contrast:
“Scores would be different if the big labs had entered with their biggest models. But that’s kind of the point. K Prize runs offline with limited compute, so it favors smaller and open models. I love that. It levels the playing field.”
He added that the challenge is not just about evaluating performance — it’s about fostering innovation in the open-source AI ecosystem. Konwinski has even pledged $1 million to the first open-source model that can break the 90% accuracy threshold on the test.
An Industry-Wide Wake-Up Call
The low winning score may surprise those familiar with the range of AI tools available today. But experts argue that many current benchmarks have become too predictable — and no longer reflect real-world performance.
“I’m quite bullish about building new tests for existing benchmarks,” said Sayash Kapoor, a Princeton researcher who recently published a paper advocating for more rigorous evaluation tools. “Without such experiments, we can’t actually tell if the issue is contamination, or even just targeting the SWE-Bench leaderboard with a human in the loop.”
For Konwinski, the K Prize offers a necessary dose of realism for the tech industry:
“If you listen to the hype, it’s like we should be seeing AI doctors and AI lawyers and AI software engineers, and that’s just not true,” he said. “If we can’t even get more than 10% on a contamination-free SWE-Bench, that’s the reality check for me.”
What’s Next for the K Prize?
According to Konwinski, more insights will emerge as the K Prize continues with new rounds:
“As we get more runs of the thing, we’ll have a better sense,” he told TechCrunch, “because we expect people to adapt to the dynamics of competing on this every few months.”
The next round of submissions is expected to draw wider participation and potentially more powerful models — though the offline, compute-limited format will continue to ensure a level playing field for open-source developers.
0 Comments