esReven Project Manager

Table of content

esReven - Auto-record on QEMU

The auto-record feature in the Project Manager Python API allows you to perform a record without having you starting and stopping the record manually, detecting automatically when the record should start/stop instead. Also, when using the autorun feature, you will be able to completly bypass any manual interaction with the guest to perform the record.

Note that the auto-record functionality is currently in preview and exclusive to QEMU. As a result, it may contain bugs, and the interface is subject to change in a later version. In particular, the API we currently offer requires manipulating fairly low-level concepts.

This documents aims at explaining how to use the auto-record API. To do so, it covers the following topics:

You can find a complete example of using the auto-record feature in the Project Manager Python API examples named automatic-binary-record.py.

General

Concept

Auto-record sequence diagram

To perform auto-records, the Project Manager uses a recorder, which will communicate with the hypervisor in order to monitor the execution in the guest. So, the recorder knows what is happening on the guest, and is able to ask the Project Manager to start or stop the record after a specific event occurred. (e.g the start of a binary or a special sequence of ASM instructions executed by the guest).

The recorder requires initialization before it can enter the ready state, and this operation can take time depending of the state of the guest.

As a result, you will need to monitor the status of the recorder. You can retrieve the current status of the recorder from the field recorder_status of an auto-record task (retrieved by calling the get_task method of the Project Manager Python API)

Here is the list of possible statuses:

You should not assume that the recorder_status will only go from the RECORDING to the RECORDED status: you will encounter cases where the status will reach RECORDED at some point, and then go back to RECORDING. This may happen for instance when recording a binary because the recorder has detected that it may reduce the trace size without losing useful information, by stopping the current record and starting a new one. Similarly, code that uses the ASM stub is free to start and stop several records.

When the recorder is ready, the Project Manager will insert a CD-ROM into the guest containing the input files of the scenario and the mandatory file for the autorun.

Generic arguments

The auto-record feature is accessible via the Project Manager Python API via one method per auto-record type. All the methods use a set of common arguments described here:

Each of these methods returns a JSON object with a task key containing the representation of a task. With this data and the get_task method of the Project Manager Python API, you can pull the task's data repeatedly to know when the auto-record is finished, and get the recorder_status (a status listed in the previous section) (See the Project Manager Python API examples).

How to wait the end of the auto-record

from reven2.preview.project_manager import ProjectManager

# Connecting to the Project Manager with its URL.
project_manager = ProjectManager(url="https://user:password@url.to.project.manager:8880")

# Launching the auto-record and retrieving its task
auto_record_task = project_manager.auto_record_XXX(
    # ...
)['task']
print("Launched auto-record task with id: %d" % auto_record_task['id'])

# Waiting for the end of the task
while not auto_record_task['finished']:
    sleep(1)

    # retrieve the task with updated information
    auto_record_task = pm.get_task(auto_record_task['id'])
    print("Recorder status: %s" % auto_record_task['display_recorder_status'])

print("Auto-record finished with status: %s" % auto_record_task['status'])

Autorun

When using either of the automatic recording methods, the Project Manager will allow you to automatically launch a binary, script or command on the guest. This allows for a fully automated record without any interaction from the user.

Note that although autorun isn't mandatory by itself, if you don't use it you will have to interact manually with the guest, e.g. to launch the binary you want to record.

To enable autorun on the guest, you will have to configure your guest to automatically execute the script contained in a CD-ROM following some instructions.

To enable the autorun on the automatic record, you will have to use these arguments:

NOTE: When the autorun is enabled, all the content of the folder will be copied to C:\reven\ on Windows and /tmp/reven on Linux and executed from there.

Commiting the record to a scenario

The autorecord will generate a record the same way you can do it with start_record and stop_record and will also need to be saved in a scenario by using commit_record.

You can do something as follow:

# Auto record stuff...

# - `auto_record_task` contains the JSON of the task after the end of the auto record
# - `session` contains the JSON of the session
# - `scenario` contains the JSON of the scenario

pm.commit_record(session['id'], scenario['id'], record=auto_record_task['record_name'])

# Now you have a scenario with a record, you can replay it.

Examples

Launching a custom Windows script
project_manager.auto_record_XXX(
    autorun_binary="my_script.bat",
    autorun_args=[
        "1",
        "\"C:\\Program Files\"",
    ],
    autorun_files=[
        42, # Id of `my_script.bat`
    ]
    # ...
)

This example will launch the script my_script.bat also embedded in the CD-ROM like the following:

"my_script.bat" 1 "C:\Program Files"

Warning: To access my_script.bat from the guest, you need to put its id in the autorun_files argument

Launching a binary already in the guest
project_manager.auto_record_XXX(
    autorun_binary="C:\\Program Files\\VideoLAN\\VLC\\vlc.exe",
    # ...
)

This example will launch the executable C:\Program Files\VideoLAN\VLC\vlc.exe that is already present on the guest

Prepare the autorun

Windows

To allow autorun on Windows, you need to:

NOTE: If autorun does not work despite following the instructions of this section, check that the ShellHardwareDetection service is running. If it is not, enable the service and reboot the virtual machine.

If AutoPlay is disabled on your guest, you can still use a bat script which must be already launched on the guest when you start the automatic recording.

@echo off
title Wait CDROM Windows

:Main
timeout 2 > NUL
dir D:\ 1>NUL 2>NUL || GOTO Main

D:\autorun.bat

Warning: This script assumes that the CD-ROM will be mounted in D:, if it's not the case on your guest you should change it accordingly

Linux

Linux doesn't have an AutoPlay feature like Windows, so you will have to use a script to reproduce this feature. Like in Windows, this script must be already launched on the guest when you start the automatic recording if you want to use autorun.

#!/usr/bin/env bash

MOUNT_PATH=/media/cdrom0

while ! mount ${MOUNT_PATH} &> /dev/null
do
    sleep 0.3
done

source ${MOUNT_PATH}/autorun.sh

Warning: This script was tested on a Debian guest, it may require modification in order to work for other distributions

Binary recording

Binary recording

Requirements:

To record a specific binary from the start of it until it exits, crashs or BSOD, you have to use the auto_record_binary method of the Project Manager Python API.

When using this auto-record, the recorder will resolve and watch specific Windows symbols, such as CreateProcessInternalW that is used to spawn new processes.

This method will record at least from the CreateProcessInternalW function and try to reduce the record by re-starting it at some important points using heuristics. Generally the first instruction of your trace will be the first instruction of the binary (on the entry point), but for some binaries the trace might instead start at the CreateProcessInternalW.

To indicate which binary you want to record, this method takes an extra parameter called binary_name, which should contain either the full name of the binary with the extension (and optionally with some parts of its path) or the id of a file that was previously uploaded to the Project Manager.

Note that autorun_binary and binary_name are different, autorun_binary is used to automatically execute something on the guest but won't start the record by itself, binary_name is an indication of which binary should be recorded.

Examples

project_manager.auto_record_binary(
    binary_name="my_binary.exe",
    # ...
)

Will record C:\my_directory\my_binary.exe but not C:\my_directory\my_second_binary.exe.

project_manager.auto_record_binary(
    binary_name="my_directory/my_binary.exe",
    # ...
)

Will record C:\my_directory\my_binary.exe but not C:\my_second_directory\my_binary.exe.

project_manager.auto_record_binary(
    binary_name="directory/my_binary.exe",
    # ...
)

Won't record C:\my_directory\my_binary.exe nor C:\my_second_directory\my_binary.exe.

Warning: Because the auto-record of a binary is based on the arguments of the function CreateProcessInternalW, you should provide a binary_name that is present in the command line used to launch it. See the next example.

project_manager.auto_record_binary(
    binary_name="my_directory/my_binary.exe",
    # ...
)

Here is some batch command lines to see when the record will be started and when not:

Rem This won't be recorded
cd my_directory
my_binary.exe

Rem This will be recorded
my_directory\my_binary.exe

Rem This won't be recorded
my_directory\my_binary

Rem This will be recorded
my_first_directory\my_directory\my_binary.exe

Rem This will be recorded
C:\my_first_directory\my_directory\my_binary.exe

If you want to use autorun to directly launch the binary instead of launching it yourself on the guest you can do something like the following:

project_manager.auto_record_binary(
    autorun_binary="my_binary.exe",
    binary_name="my_binary.exe",
    # ...
)

Or using ids:

project_manager.auto_record_binary(
    autorun_binary=5, # 5 is the id of the file "my_binary.exe" already uploaded to the Project Manager
    binary_name=5,
    # ...
)

Recording via ASM stubs

Requirements:

Automatic recording using ASM stubs allows you to directly control the record from the guest, e.g. to only record the execution of a specific function.

Using the ASM stubs described in the next section you are able to start, stop, etc the record automatically from within the VM. However, note that this ability comes with its own drawbacks, such as needing to modify a binary to insert the ASM stubs on the function you want to record.

NOTE: Auto-recording using ASM stubs should be OS agnostic, and was tested successfully on Windows 10 x64 and Debian Stretch amd64

ASM stubs

The ASM stub works by hijacking an instruction in the guest so that it is interpreted differently in the hypervisor when used with a specific CPU context. In Reven, the ASM stub is the instruction int3 (bytecode 0xcc), when executed with a magic value in rcx.

So, when calling int3 you need to set these registers accordingly:

The list of commands is:

Helpers for various languages (C/C++, Rust, Python, ...) are available from the Downloads page of the Project Manager.

Examples

Recording a function execution

If you have an executable my_binary.exe with these sources:

void function_to_record() {
    // ...
}

int main() {
    // ...

    // Start the record
    __asm__ __volatile__("int3\n" : : "a"(0x0), "b"(1), "c"(0xDECBDECB));

    function_to_record();

    // Stop/commit the record
    __asm__ __volatile__("int3\n" : : "a"(0x1), "c"(0xDECBDECB));
    __asm__ __volatile__("int3\n" : : "a"(0x2), "b"(1), "c"(0xDECBDECB));

    // ...
}

You can record it like the following:

project_manager.auto_record_asm_stub(
    autorun_binary="my_binary.exe",
    # ...
)

Recording two executables

If you want to record the execution of my_binary.exe and also my_second_binary.exe you can use something like the following.

With a binary start_record.exe:

int main() {
    // Start the record
    __asm__ __volatile__("int3\n" : : "a"(0x0), "b"(1), "c"(0xDECBDECB));
    return 0;
}

With a binary stop_record.exe:

int main() {
    // Stop/commit the record
    __asm__ __volatile__("int3\n" : : "a"(0x1), "c"(0xDECBDECB));
    __asm__ __volatile__("int3\n" : : "a"(0x2), "b"(1), "c"(0xDECBDECB));
    return 0;
}

With a script my_script.bat:

@echo off
C:\reven\start_record.exe

C:\reven\my_binary.exe
C:\reven\my_second_binary.exe

C:\reven\stop_record.exe

You will be able to record them by uploading all these files into the Project Manager and calling auto_record_asm_stub like the following:

project_manager.auto_record_asm_stub(
    autorun_binary="my_script.bat",
    autorun_files=[
        42, # Id of `my_script.bat`
        43, # Id of `start_record.exe`
        44, # Id of `stop_record.exe`
        45, # Id of `my_binary.exe`
        46, # Id of `my_second_binary.exe`
    ]
    # ...
)

Mixing modes: binary recording + ASM stubs

The binary recording mode allows overriding its operations with ASM stub commands during the recording phase. This allows for more control in binary recording mode while still keeping the OS-related automation such as automatic recording stop on process stop or OS crash.

Binary recording mode automatically allows mixing ASM stubs, there is no option necessary to select.

Here is an example where the program will manually restart the recording even though the Binary recording mode had started it ealier:

Binary recording

In general, when mixing modes, ASM stub commands have priority over the binary recording heuristics while still allowing levegaring the binary automation:

This mode makes it easier to record various situations, notably non-deterministic crashes: restarting a recording from within the program is easy, and the binary recorder will still catch the crash.