Manual vs AI Smart Contract Auditors : Winning Journey
Here's the story of how I competed with two AI audit bots in the last few weeks and why it's not the right time to favor AI audits over manual ones.
We've been hearing the rumors for quite some time now – AI is set to snatch away auditors' jobs. There have been attempts by AI developers to replace us, but they haven't quite hit the mark. But will they eventually succeed? Well, maybe, to some extent. But it's not going to happen overnight. My guess? We're looking at a good 3-5 years before AI can even come close to matching the skills of a decent junior smart contract auditor.
Now, when I say junior auditor, I'm talking about someone who can dive into code and spot around 30-40% of the significant issues hiding within a codebase.
If someone is using checklists, it’s a very slow bot, not an auditor.
So I competed with an AI bot that claimed to do audits better than humans and I won and won by a margin.
Here is my story….
Competition Timeline
About a month ago I came across this ad while searching for something about smart contract auditing.
That is indeed a bold claim. Personally, I wouldn't put my faith in just one manual audit, let alone an AI-powered one. And if you dig a little deeper into their website, you'll find this…
Claiming to be "more accurate than a human audit" is quite bold. So, I decided to see for myself. I took to Twitter to express my doubts, hoping for a good discussion. But instead, I was met with immediate backlash. It quickly became clear that this approach wasn't leading to constructive conversation.
The Solution? A Face-Off: I Challenged Them to a Head-to-Head Audit on a Codebase, and They Agreed.
Now, there was a need for an intermediary that both parties could trust. In the mix of it all, @fellows decided to join with his AI bot. He suggested Django, co-founder of gasbotxyz and an OG EVM whitehat, to serve as the intermediary, and both parties agreed.
After some obstacles, it was finally time to start the audit. The codebase I selected for this challenge was a new EVM<>MEM bridge by Mem_tech that I audited some time ago. It wasn't a simple bridge, so I had to do some digging to understand it. In the process, My team found two major bugs where funds could be drained from the protocol.
Both AI bots were given a 2-hour window to run their bots and submit their reports. Meanwhile, my report was already public, but only Django and I were aware of it. In hindsight, it was a bit of a rookie mistake on my part.
Here is the link to the codebase :
Results
Both AI bots submitted their reports, and I'll admit, I was feeling a bit nervous at first. What if their claims were true, and they found something I missed? But as I opened the reports, it became clear.
Both AI bots missed all of the high and medium severity issues, except for the bug that I intentionally introduced and had informed Django about beforehand. Fellow's AI never claimed to outperform manual review in the first place, but Bunzz did, and they failed miserably.
Missed bugs can be seen here:
Then came the post from Bunzz's official account declaring victory. It was a dishonest approach, to say the least. But for those of us who care about protocol security, integrity matters. Attaching all the reports here so you can compare how my report uncovered actual bugs that could have led to the loss of funds, while the bots could only find one meaningful issue that was intentionally introduced. You can confirm this by comparing the provided code above with the original commit here:
Original MEM Commit
Links to Audit Reports
Fellows's AI Report
Huge respect for Fellows for his high personal integrity.
What’s Next?
But it doesn’t end there, yes there is more to it. After this, I went down a rabbit hole of AI auditors and came to know there is a whole cult of them and no they are not here to provide security to projects, what are they here for? you may ask. That is the topic for part 2 where I will show you what is happening in the whole AI-Auditor industry. Though there are some good players, shoutout to fellows for being one of them…..
If you want to make sure your smart contracts are secure, contact me here