Daniel Stenberg on auditing curl with Claude Mythos

Lounge Linux lounge
1

Daniel Stenberg, developer of curl, has written a blog post on using Claude Mythos to find a vulnerability in curl.

TL;DR: Curl has been using many static analysers for years and all in all do a lot to ensure the project is secure. He was a critic of LLMs as the Curl bug bounty program got slopped really hard (they’ve since stopped said program). By now, they’ve used multiple LLM-driven security scanning tools to successfully find vulnerabilities. While they still report false positives, Stenberg seems to be quite happy with them from what I understand.
Claude Mythos reported five vulnerabilities from which three were false positives and the “issues” Mythos raised were already documented oddities. One was a plain old bug without security consequences and the fifth actually a vulnerability, although a low-severity one.

2

This is good. So out of 5 issues identified by LLM, 2 were issues. Out of those 2 issues, 1 was a bug without security consequences. The remaining was a vulnerability. I hope both of them are getting fixed.

3 bugs are false positives. So about 60% of the issues identified were bogus. Only 40% of them were issues.

These are types of figures that we need to understand the efficacy of LLMs. I wonder if Windows, Oracle Databases, and other closed source code will release these types of figures. Not holding my breath on such a disclosure.

I was understand why Anthropic Mythos was so scary was its ability to take those bugs and chain them together to create exploits at a pace/speed which was unprecedented. Am I missing something over here? Or this understanding stretched?