Paper
A 'Finding Inspiring Papers' Benchmark Created by 184 Researchers
ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research
- What
- We created ScholarCatalyst, a dataset where first authors of 207 recent computer science papers indicated which prior papers actually helped their research, and presented a retrieval task to find them.
- Key result
- Agentic search (AI that iteratively performs tasks on its own) did not show better performance than basic search methods (recall of 0.42 versus 0.48 based on the top 20 results). Agents based on Claude Fable 5.1 only reached 0.51.
- Why it matters
- This indicates that current AI still lacks the expert intuition needed to identify necessary prior ideas from vast research literature, pointing to the need for new training methods. While this can be viewed as a step forward for AI assisting scientific research, actual commercialization or industrial effects cannot be confirmed from the abstract.AI summary
