How does a malware sandbox work?

Ever wondered what a malware sandbox is, how it works, what information it produces, or what that information is useful for?

In this series, I’ll discuss exactly that. In this post I’ll go through the overall process and generalize a lot; in the next posts, I’ll go progressively deeper into the technical details.

What’s a malware sandbox?

It’s a VM (virtual machine) that powers up, runs malware, exports behavioral information, then gets killed and reset to a pre-run snapshot (i.e. known good pre-infection state), ready for the next run.

There are exceptions, but for the vast majority of purposes it’s just a regular virtual machine with some code around it to turn it on, transfer the malware into the guest operating system, run it, log activity, export it, and finally analyze it and produce a report.

What information does it produce?

Different sandboxes log different things, but generally they log library, API, and system calls, and produce a memory dump. Consider this pseudocode example:

// ransomware example
function encrypt_user_files(key) {
    files = list_all_user_files()

    for (file in files) {
        contents = read_file(file)
        contents = encrypt(contents, key)
        write_file(file, contents)
    }

    store_key_in_malicious_remote_server(key, machine_id)
}

function main() {
    secret_key = "some long unique secret key"
    encrypt_user_files(secret_key)
    demand_money_from_user()
}

(This would be ransomware, a type of malware that encrypts your files and then asks you to pay a ransom to decrypt them.)

On Windows, this code would call several library functions and syscalls under the hood, including CreateFileW() to open the file, ReadFile() to read the original contents, WriteFile() to write the encrypted buffer, and WinHttpSendRequest() (or similar) to send the key to the malicious server (also called a “Command and Control server”, or C2/CnC for short).

Running this malware in a sandbox would show multiple calls to those file- and network-related functions:

// Launching the malware
CreateProcessW(totally_not_malicious.exe) // library call
  -> NtCreateUserProcess(...) // syscall

// Random initialization code, OS-dependent
LdrInitializeThunk(...)
RtlUserThreadStart(...)

// malware loop: encrypts important_document.pdf
CreateFileW(important_document.pdf)
ReadFile(...)
WriteFile(..., <encrypted buffer>)
CloseHandle(...)

// malware loop: encrypts family_photo.jpg
CreateFileW(family_photo.jpg)
ReadFile(...)
WriteFile(..., <encrypted buffer>)
CloseHandle(...)

// etc
// ..
// ..

// Finally, the encryption key is sent back to the server
WinHttpSendRequest(example.com, {"key": "c29tZSBsb25nIHVuaXF1ZSBzZWNyZXQga2V5"})

// ... more calls ...

The sandbox logs all relevant API calls, and in this trivial example, you’d see the initialization, the core encryption loop, and the encryption key that was sent to the malicious server in the call to WinHttpSendRequest. Real malware isn’t quite as trivial, but the same concept applies.

If you want to see what a real output looks like, see this example (source, requires login).

Sandboxes will also often dump the entire memory of the guest into a file for further analysis. Consider this pseudocode example:

// Load malicious server IP from config
malicious_ip = config.get_malicious_ip()

if (!healthy(malicious_ip)) {
    // Load backup IP
    malicious_ip = config.get_backup_malicious_ip()
}

connect(malicious_ip)

Here, if the main malicious IP is healthy and responding, the malware will never connect to the backup malicious IP, and therefore the sandbox will not be able to extract it.

A memory dump would contain the entire config, so an extractor can be written for all relevant fields. This would also allow automation: drop the newest version of the malware into the sandbox, get back the full updated config.

All of this information is extracted, stored, processed, and sold in various forms, including as threat intelligence. The cybersecurity market is big, and a lot of it consumes this information directly or indirectly.

What else can it do?

Automating malware analysis is one feature, but a sandbox can do more than that.

Helps develop new detection signatures.
Runtime information helps malware analysts understand patterns of behavior and write detection rules for it. It also greatly helps with the process of reverse engineering, as it already gives away a lot of what the malware is doing before you have to read a single line of assembly.

Detects obfuscated malware significantly better than regular AVs.
Regular antivirus (AV) software generally works by matching signatures against a file. Malware authors avoid detection by wrapping malicious binaries in layers of obfuscation and encoding to evade these static signatures.

When the malware is run in a sandbox, it eventually deobfuscates its code so it can run and exhibit its bad behavior - allowing code-based and behavioral signatures, respectively.
(Modern AVs do a lot more than just static scans, but the concept still holds.)

Detects never-previously-seen malware.
Behavioral rules detect patterns of behavior - so they can flag malicious behavior from never-previously-seen samples. Consider the pseudocode example shown above: a binary which, when launched without any interaction, reads the user’s files, calls a crypto API, overwrites files, then communicates with a foreign IP. Clearly ransomware-like behavior - enough to surface to a human for review. These rules can occasionally produce false positives, but they are very useful for discovery.

Automates extraction of Indicators of Compromise (IOC).
Malicious IPs, domain names, file names, file hashes, registry keys, etc., are all Indicators of Compromise. These allow a company to quickly tell if any of its machines got infected with a particular kind of malware - simply by checking if a file with a given hash was downloaded or executed, or if a malicious domain was resolved, or if a given registry key was created, etc. Sandboxes extract this information automatically and cyber vendors integrate it into their threat intelligence platforms.

Sample to report

Here’s the overall process, from submitting a sample to obtaining a final report.

Note: this is a highly simplified general overview, and can vary a lot between different sandboxes.

VM setup

First, the VM is prepared by installing Windows, Windows updates, runtimes and libraries (e.g. .NET, Visual C++ redistributables), and optionally, miscellaneous files and programs to make it appear like a regular user’s machine.

The sandbox wants to look like a regular user’s machine because malware generally doesn’t like being analyzed. If alerted, it may choose to simply exit in an attempt to evade detection.

Next, the sandbox’s plumbing is installed in the VM:

• Communication server: allows the host to send files to the guest, and the guest to send back behavioral logs to the host.
• Stealth: hides or modifies various artifacts that indicate a virtualized environment, e.g. MAC addresses (vendor-specific to the virtualization software), driver filenames (e.g. VBoxMouse.sys), various VM tools (e.g. VMware Tools), network checks, and so on.
• Optionally, a mouse mover to simulate user activity, a screenshot tool to send regular screenshots, or other sandbox-specific tools.

Next, internet access - keeping it off can break some malware such as downloaders¹, but it prevents alerting the authors, which is useful for high-value targets such as state actors and APTs². Keeping it on lets the malware do more and thus the sandbox sees more, but it also leaks the sandbox’s existence and IP. Some sandboxes take a third approach and instead keep internet access off, but fake successful responses to some requests (e.g. DNS resolutions, basic HTTP requests)³.

Ultimately, it’s a strategic choice and different companies take different approaches, depending on their targets and scale of operation.

Lastly, a snapshot of the VM is taken. This is the known good snapshot to which the entire OS will be restored after each run, cleaning up the mess made by the previous malware execution, and restoring all sandbox machinery to its original state.

Sandbox execution

When a sample is submitted to the sandbox for analysis, the VM is powered on, the known good snapshot is loaded, and the sample is transferred to the VM. The sample is then launched like this:

1. Run suspended: create a process, map the sample to memory, and have it ready to go - but don’t actually run any threads yet. On Windows, this is done by calling CreateProcess() with the CREATE_SUSPENDED flag.
2. Inject a monitor DLL into the suspended process, which will hook⁴ functions; that is, the DLL goes through all of the desired functions to monitor and it replaces them with a stub that first logs the call and the parameters, and then jumps to the original function. This can also steer or completely replace function behavior.⁵
   • In addition to logging and steering behavior, the monitor also hooks process creation functions so that child processes are tracked too.
3. Stealth: hide the injected DLL from the list of loaded DLLs in the PEB. Some malware will go through its own list of loaded DLLs and, if any of them lives outside of the default system paths, it’ll assume it’s being monitored and exit.
4. Resume execution - now with all desired functions hooked and monitored.

Only executables (.exe) can be launched in this way. Library (.dll) code instead runs with rundll32.exe, a small stub program that loads the passed DLL and calls the passed function - but the rest of the instrumentation is the same. Microsoft Office documents, which are also a common malware vector, run in a very similar way: the applicable MS Office product is launched suspended, the monitor is injected, and the process is resumed, allowing the packaged malware to run while being monitored.

Notice that all of this is happening in user mode. Hooking a function simply means modifying the library that contains it to add a redirect to your own code. But the malware doesn’t actually have to go through library code: it could instead directly call the kernel. Not through the syscall stubs in the OS library - but through dynamically constructing its own assembly code containing syscall instructions, and calling it. This would effectively bypass user mode hooks and logging. Most malware doesn’t do this, but there’s always the possibility that an APT might do this and go undetected. This is why some sandboxes add a kernel monitor too.⁶

Next, after the malware is launched, the sandbox waits either until it exits or a timeout is reached. Then, the communication server lets the host know that the analysis is over, the entire guest OS memory is dumped into a file, and the VM is shut down.

Analysis and report

After the execution, we have a behavioral log and a memory dump. Sandboxes will also often keep dropped files, which are files created or downloaded by the malware, and network captures, which could be either raw network sniffs, or fully intercepted communications obtained at the API level that bypass standard TLS.

Next, the detection rules are run. This is where a lab’s intellectual property really is, so there’s a wide variety of detection types and methods that could happen here, but in general it’s going to be behavioral rules (did the malware call functions A and B with parameters X and Y? did it spawn process C, create a mutex D, and edit registry key E?), memory rules (is there a chunk of code that looks like this pattern?), network-level rules (is there a communication from A to B, with contents roughly looking like pattern C?), and config extraction.

Behavioral rules are essentially patterns for a sequence of actions or API calls. As an example, consider the following pattern:

1. User receives an email with a Microsoft Word attachment.
2. User downloads attachment, double-clicks it, and launches Microsoft Word (winword.exe)
3. The loaded document contains a macro, a VBA-based automation script that the Office suite supports. The macro is executed⁷.
4. The macro runs a new process by executing a shell command - usually running a cmd.exe or a PowerShell instance, which downloads a malicious binary and executes it.

Here, the behavioral rule you’d write would look something like the following pseudocode:

executions
    # find Microsoft Word
    .find(process_name="winword.exe")

    # Find any cmd or powershell that was launched by Word
    .find_children(process_name=["cmd.exe", "powershell.exe"])

    # where the shell spawned at least 1 more process
    .find_children().length(min=1)

Alternatively, you could write a detection rule on API calls:

api_calls
    # Find all API calls for the given process ID
    .filter(process_id=pid)

    # Find a call to CreateMutexA("Global\\TrickBot")
    .filter(name="CreateMutexA", parameters={"lpName": "Global\\TrickBot"})

(The example is pseudocode but the detection is real: a banking trojan called TrickBot would register a global mutex with its own name, which made detection trivial - just find the API call creating the mutex.)

Memory rules are essentially regex matching with extra steps over a dump of memory. Consider the following machine code:

x86 assembly                   | binary    
-------------------------------|-----------
push    ebp                      # 55
mov     ebp, esp                 # 89 e5
mov     eax, DWORD PTR [ebp+8]   # 8b 45 08
imul    eax, eax                 # 0f af c0
pop     ebp                      # 5d
ret                              # c3

Let’s assume this code is unique to a given malware. The signature for it would look like:

// YARA rule
rule SomeMalware {
    meta:
        author = "APSecurity"
        date = "..."
        description = "Matches the malware family SomeMalware."
    strings:
        // matches a unique function or code chunk in the malware
        $c1 = {55 89 e5 8b 45 08 0f af c0 5d c3}
    condition:
        $c1 // c1 must match
}

Note that this is often not the case: functions aren’t that unique, and they can also change with new builds, invalidating literal byte-matchers like the example above. YARA allows you to match on several conditions, byte and string patterns, and more - so rules can get more complex than this (example), but that’s essentially what they do.

Network-level rules are most often written in Snort or Suricata. Snort is older (although still widely used), so let’s look at a simple Suricata rule:

alert http $HOME_NET any -> $EXTERNAL_NET any (
    msg:"ET MALWARE W32.Qakbot Request for Compromised FTP Sites";
    flow:established,to_server;
    http.uri; content:"/cgi-bin/jl/ad03.pl?pv=2&d=";
    reference:url,[some url];
    reference:url,[some url];
    classtype:trojan-activity;
    sid:2012972;
    rev:3;
    metadata:created_at 2011_06_09, signature_severity Major, updated_at 2020_04_20;
)

This rule says that a request to the path /cgi-bin/jl/ad03.pl?pv=2&d= under certain conditions (outbound, active connection) matches W32.Qakbot behavior. I won’t get into further details because network-level rules are generally uninteresting in the context of a sandbox - in my experience, they are less effective than other types of rules and produce more false positives (detect legitimate behavior as malicious) as well as false negatives (miss real malicious behavior). There are some exceptions to this, of course.

Lastly, there’s the config extraction stage from the dumped OS memory. Load the dump into an API that can parse it (e.g. Volatility), then write a custom extractor to find the interesting process, pattern-match a memory location, extract a memory region, optionally decrypt or decode, and obtain the config. This process can vary a lot depending on the way the config is stored.

After the analysis is done, a report is generated. This report contains all of the relevant information from the analysis: API calls, process information, network connections, registry activity, filesystem activity, hashes and other IOCs, etc. That’s the unit of business value generated by the sandbox.

This information can then be sold or bought in many ways, for example:

• Feed it into a threat intelligence platform to track malware authors, attribute attacks, and build a view of the malware scene.
  • Many (most?) of the large cybersecurity companies build and maintain a platform like this.
  • These platforms are used by SOC analysts for their investigations. They can answer questions such as “is this executable bad?” without requiring the SOC analyst to have the skills or time to reverse engineer it.
• Buy the threat intelligence platform’s IOC feed to easily check if any of the machines on your corporate fleet are infected.
  • Many cybersecurity companies buy these platforms’ feeds.
• Sell the collected samples and information to malware researchers who are on the hunt for new malware builds or other malware artifacts.
  • VirusTotal does this through their various products. They allow you to search their massive corpus of data for binaries with YARA rules and download the matching samples. A very useful (and very expensive) service.
• Discover more of the malware’s infrastructure and inspect it for information about its authors or their network; or even attack it to recover customer information. Note that I’m stating that this can be monetized and that some companies do it, not that it’s legal or that I encourage it.

That’s it for this post. In the next one in the series, I’ll get into deeper technical details, so stay tuned.