Software & DevOpsDone with Devin, Cursor Jan 8, 2025
Test Devin on integrations, migration and security review
Answer.AI tried Devin on an observability integration, an nbdev migration and a vulnerability review; it found some real issues, hallucinated others, and failed the migration.
- What They Did
- Answer.AI gave Devin, which works through Slack in its own Docker environment with a browser, editor and shell, about 20 real tasks. These included generating and uploading synthetic data to Braintrust, migrating a Python project to nbdev, and a security review of a repo under 700 lines. People checked the output and often took over.
- How It Went
- The Braintrust integration was abandoned and redone in Cursor. The nbdev migration failed on basic setup. The security review found some real issues but also false positives and hallucinated ones. Overall 14 failures, 3 successes and 3 inconclusive.
- Worth Knowing
- Devin pressed on with impossible tasks for days. The authors suggest a single focused LLM call for something like a small-repo security review.
Try it yourself with Devin
Take on three tasks in my repo [repo name]: add an integration with [observability tool], migrate [a module or tool] to [target tool or version], and review the code for security vulnerabilities. For each one, tell me how you verified your results, and clearly mark any finding you could not confirm. Open separate pull requests for the work and do not merge anything.