Research & KnowledgeDone with Devin Jan 8, 2025
Try Devin on research tasks (mostly failed)
Answer.AI had Devin fetch an author's recent papers and research transcript summarization; it got stuck parsing HTML and returned generic results, rated a failure.
- What They Did
- Answer.AI gave Devin, which works through Slack in its own Docker environment with a browser, editor and shell, two tasks. One was grabbing an author's 25 most recent Google Scholar papers with Playwright, skipping paywalls. The other was researching transcript summarization with accurate timestamps. They then rated the results.
- How It Went
- Both failed. On the scraping task Devin got stuck parsing HTML and went to sleep. On the research task it returned related but unhelpful material and code that missed the core problem.
- Worth Knowing
- Across 20 tasks Devin had 14 failures and 3 successes. One research success, a Discord bot, worked because the goal was loose pseudocode-level guidance.
Try it yourself with Devin
Research [author or topic] by finding their [number] most recent papers and writing a one-paragraph summary of each, with links to the sources. Then give me a short overview of the main themes across them. If a page can't be read, tell me which one and why instead of guessing, and stop once you have a cited summary I can check.
Do this with a skill.
These SkillShop playbooks do this exact job.