SWE-Explore: Benchmarking How Coding Agents Explore Repositories
Abstract
SWE-Explore introduces a benchmark for evaluating coding agents' repository exploration capabilities by requiring ranked lists of relevant code regions within line budgets, demonstrating that agentic exploration outperforms traditional retrieval methods.
Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tasks as a holistic, binary prediction problem (e.g., resolved or unresolved), neglecting fine-grained agent capabilities such as repository understanding, context retrieval, code localization, and bug diagnosis. In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration, a critical capability of coding agents. Given a repository and an issue, SWE-Explore asks an explorer to return a ranked list of relevant code regions under a fixed line budget. SWE-Explore covers 848 issues across 10 programming languages and 203 open-source repositories. For each instance, we derive line-level ground truth from independent agent trajectories that successfully solved the same issue, distilling the specific code regions their solution paths actually consulted. We evaluate exploration along coverage, ranking, and context-efficiency dimensions, showing that these metrics strongly track downstream repair behavior. Across a broad set of retrieval methods, general coding agents, and specialized localizers, we find that agentic explorers form a clear tier above classical retrieval. While file-level localization is already strong for modern methods, line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers.
Community
Is your coding agent good at exploring repositories?
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution (2026)
- RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades (2026)
- SWE-TRACE: Optimizing Long-Horizon SWE Agents Through Rubric Process Reward Models and Heuristic Test-Time Scaling (2026)
- Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills (2026)
- RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations (2026)
- CodeTracer: Towards Traceable Agent States (2026)
- BLAgent: Agentic RAG for File-Level Bug Localization (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Congrats on #1 Paper of the day!
Your analysis gives me many ideas for improving my coding agent Outrider: https://github.com/remyxai/outrider
Compared to the coding agents you've evaluated, Outrider applies agentic exploration in the task of cross-repo analysis to recommend new candidate methods to improve a target code base.
The candidate space is continuously evolving and "correctness" is multi-dimensional. In the cross-repo recommendation task, knowledge should compound as state accumulates between runs.
Get this paper in your agent:
hf papers read 2606.07297 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper