Devin fails a Google Scholar paper scrape

Answer.AI tested Devin on twenty tasks over a month and published both successful and failed attempts. One research-related task asked it to collect the most recent twenty-five papers from an author on Google Scholar, using Playwright where appropriate and skipping documents behind paywalls.

The agent could browse the web, write extraction code and run it in its computing environment. In this attempt, however, it became absorbed in parsing HTML and failed to return the intended collection. The appendix says it got stuck and went to sleep. The documented result is an unsuccessful retrieval workflow, not a completed literature review.

The larger experiment helps put that failure in context. The same team had early success moving Notion data to Google Sheets, including documentation research and human credential setup. Across the twenty tasks, the authors classified three as successes, fourteen as failures and three as inconclusive. Those are their experiment categories, not a universal failure rate.

The useful lesson is an observed failure to recognize and escape an unproductive approach. The team supplied a bounded target and a paywall fallback, yet still had to assess the incomplete work rather than trust elapsed autonomous effort.

Prompt
Suggested prompt (written by Agents In Use, not the author's)

Source: www.answer.ai · Hamel Husain, Isaac Flath and Johno Whitaker