Test Devin on pulling Notion data into Sheets

Answer.AI testers found Devin completed only 3 of 20 tasks, succeeding at moving Notion data to Google Sheets and building a planet tracker.

What They Did

Three Answer.AI data scientists (Hamel Husain, Isaac Flath, Johno Whitaker) gave Devin 20 tasks. Devin takes commands through Slack and works in a Docker container with a terminal, browser, code editor and planner. Early tasks included pulling data from a Notion database into Google Sheets and building a planet tracker for checking historical Jupiter and Saturn positions.

How It Went

Devin finished 3 of 20 tasks: the Notion-to-Sheets move, the planet tracker, and a Discord bot research task. Three were inconclusive and 14 failed. Simple-looking tasks often took days, and it was hard to predict which would work.

Worth Knowing

Devin starts at $500 per month. It kept pushing on impossible tasks, such as a Railway deployment, for over a day instead of stopping, so set time limits.