Unpacking malware with AI (safely)

In my last post, I showed how to unpack a random malware sample I found online. In this post, I’ll show how to let an AI agent do it - safely, with a real sandbox that neither the malware nor the agent can escape.
(Contrary to popular belief, it’s possible.)

Sandboxing setup

Handling malware is very dangerous, so you need a VM. Easy.

Using AI agents on their own has some limited risk - they can accidentally run $ rm -rf /, or just do other accidental stuff that they shouldn’t. In a regular project, the LLM risk is really not that high, as most errors are harmless or recoverable.

However, mixing both is an explosive combination, so I needed a decent setup. The easiest way is to just install the agent harness in the malware VM, but then you’re putting LLM credentials where you put malware. Not good.

The next approach would be to simply run the agent harness on the host and let it connect to the guest VM. This opens up a door for the malware to escape: it just takes one bad agent decision to potentially infect your host (e.g. “let me copy this file to host’s %TEMP% and run because {silly hallucinated reason}”). Not good.

Therefore, the obvious solution is to run two VMs, one running the malware, and another one running the agent, and connect them with an SSH tunnel. To save time, I used whatever I had installed already:

• a Windows 11 guest in VMware to run IDA and the malware
• a Multipass VM with an agent
• OpenSSH for the communication tunnel

Setting all this up took relatively little technical effort - the agents are just smart enough for it if you nudge them correctly.

For the MCP server, I used ida-pro-mcp. Note that if you want to allow your agent to run and debug the target malware, you must run idalib-mcp with the --unsafe flag. I had ChatGPT 5.6 Sol set this up and it initially refused to pass the flag because, you see, it’s unsafe! Better not do it! Anyway, after nudging it more, it conceded.

Model choice and prompt

Initially I tried DeepSeek V4.1 Flash with the omp harness, but it quickly became obvious that the model just wasn’t smart enough. It was not following instructions properly, it was running the malware before setting breakpoints and then waiting for them to hit (which they never did), and so on, and I quickly grew tired, and ended it.

Next, I tried Grok with Grok Build, hoping it would not trip cyber guardrails. As usual, it didn’t, and it started dutifully fulfilling its task. Here’s the full prompt I used:

Use the IDA Pro MCP. There’s a packed malware loaded. Unpack it using the following technique:

1. Run in IDA with local Windows debugger. Break on CRT entry point (not app entry point)
2. Put software breakpoints on NtAllocateVirtualMemory and NtProtectVirtualMemory. Condition: break on executable pages only.
3. Set HWBP on write on pages where the malware could be unpacked.
4. When HWBPs hit, read code at RIP, let any loop finish copying data if necessary, then check page for MZ header
5. No MZ -> continue. MZ -> dump as a .bin file on desktop and stop debugging (killing debuggee)

Do not write a script to automate this - do it yourself manually.

The last line about not automating with a script was to prevent Grok from doing what DeepSeek had attempted earlier: writing a Python script to execute the instructions as written, literally. That could potentially work, but it was not in the spirit of this run.

Lastly, I didn’t consider OpenAI or Anthropic models for reversing for obvious reasons.

LLM reversing

Grok started reading the documentation to figure out how to use IDA through the MCP server. It tried launching the binary under the debugger, but for some reason IDA crashed, and Grok started trying to figure out why. I had to interject:

After a few more minutes of thinking, Grok seemed to have completed the task successfully:

Grok successfully launched the binary under the debugger, set the correct breakpoints, ignored the false positive hits, identified the correct page to dump, placed a HWBP and found the MZ header on write, and then dumped the file. But then I checked the output, and it was almost all zeroes. It turns out Grok was too eager to dump the PE file, so as soon as it saw a header (and nothing else) written, it dumped it out. I told it to try again, and it did, this time seemingly successfully:

If you look carefully though, you’ll notice that this time, Grok had dumped the mapped file, not the original unpacked file. It had done so by placing a breakpoint on execution:

This is very different from what I had asked for, and is a suboptimal result - although I’m still impressed that it found the OEP. Still, I nudged it again, telling it to follow the instructions as written and to dump the unpacked-but-not-yet-mapped file, and it did:

I verified the dumped file by opening it in IDA:

This one loaded without warnings, and overall looked correctly dumped. I was happy to see that - it took a few nudges, but it was still a job well done.

Takeaways

Despite knowing that this is possible, and despite all the technical hurdles and nudging it took, I still felt quite amazed to watch it happen. Once everything was set up, the unpacking process took minutes and the result was excellent.

There’s a lot of talk about cybersecurity dangers as a result of AI, but I don’t see nearly as much talk about how awesome of a tool it is for defense, as demonstrated in this post. With a slightly smarter model and enough compute, you could automatically unpack and classify a lot more malware, a lot better than we do now.

Traditional security pipelines could then be significantly improved: malware sandboxes, end-user AVs, corporate EDRs and security pipelines, etc. Currently, they rely on a lot of fragile pattern matching, and LLMs (and the upcoming non-LLM classifiers like Jev) are a direct and effective fix for that. They will be the paradigm-shifting improvement that will harden safety more than any single advancement so far. They are the ultimate tool in the cybersecurity defense kit.

That’s it for this post, hope you enjoyed it!