DevBench: A Comprehensive Benchmark for Software Development

Existing coding benchmarks focus on simplified or isolated aspects of coding, such as single-file generation or repository issue debugging, and fall short of measuring the full spectrum of real-world programming activity. We evaluate LLMs across the entire software development lifecycle, covering software design, environment setup, implementation, acceptance testing, and unit testing, across four programming languages and multiple domains.

Empirical studies show that current LLMs, including GPT-4, fail to solve the challenges the benchmark presents.

The arXiv version of this work is titled “Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study”, where the benchmark is referred to as DevEval.