RTEVAL(8)

NAME
rteval - evaluate latency performance of a realtime Linux system

SYNOPSIS
rteval [OPTIONS]

DESCRIPTION
The rteval program is used to evaluate the realtime performance of a Linux kernel on a particular hardware system. It runs measurement modules (cyclictest or timerlat) to measure scheduling and timer latency while simultaneously running system load modules (kernel compile, hackbench, stress-ng) to stress the system.

The program runs for a specified duration (default 60 seconds, typically 12+ hours for production testing) and generates an XML report containing system information, raw measurement data, and statistical analysis including min/max latency, standard deviation, and histograms.

Results are stored in a timestamped directory (rteval-YYYYMMDD-N) containing the XML report and optional logs.

OPTIONS

  General Options

-h, --help
    Display help message and exit.

-V, --version
    Print rteval version and exit.

-d, --duration DURATION
    Specify length of test run in seconds (default: 60.0). Can be specified as a floating point number (e.g., 1.5 for 90 seconds) or with time suffixes.

-f, --inifile FILE
    Initialization file for configuring loads and behavior.

-w, --workdir DIRECTORY
    Top directory for rteval data (default: current directory).

-i, --installdir DIRECTORY
    Place to locate installed templates (default: /usr/share/rteval).

-l, --loaddir DIRECTORY
    Directory for load source tarballs (default: ./loadsource).

-a, --annotate STRING
    Add an annotation string which is stored in the report.

-D, --debug
    Turn on debug prints (default: False).

-L, --logging
    Log the output of measurement and load modules in the report directory.

-q, --quiet
    Turn on quiet mode (default: False).

-v, --verbose
    Turn on verbose prints (default: False).

-s, --sysreport
    Run sysreport to collect system data (default: False).

  Load Control Options

--noload
    Only run the measurements (don't run load modules).

-O, --onlyload
    Only run the loads (don't run measurement threads).

--loads-cpulist CPULIST
    CPU list where load modules will run (e.g., 0-3,8-11).

  CPU Isolation and Cpuset Options

--cpusets
    Use cgroup v2 cpusets to isolate measurement and housekeeping workloads.

--housekeeping CPULIST
    Isolated CPUs reserved for system tasks (not used by rteval).

--warn-non-isolated-core-sharing
    Warn about measurement and load CPUs sharing cores even when neither is isolated.

--cleanup-cpusets
    Remove any leftover rteval cpusets from previous runs and exit.

  Measurement Module Options

--measurement-module MODULE
    Select measurement module: cyclictest or timerlat (overrides config file).

--measurement-cpulist CPULIST
    CPU list where measurement modules will run.

--measurement-run-on-isolcpus
    Include isolated CPUs in default cpulist.

--measurement-idle-set IDLESTATE
    Idle state depth to set on CPUs running measurement modules.

  Cyclictest Module Options

--cyclictest-interval INTV_US
    Base interval of the threads in microseconds (default: 100).

--cyclictest-buckets NUM
    Histogram width (default: 3500).

--cyclictest-priority PRIO
    Run cyclictest with the given priority (default: 95).

--cyclictest-breaktrace USEC
    Send a break trace command when latency > USEC.

--cyclictest-threshold USEC
    Exit rteval if latency > USEC.

  Timerlat Module Options

--timerlat-interval INTV_US
    Base interval or period of threads in microseconds (default: 100).

--timerlat-priority PRIO
    Run rtla timerlat with this priority (default: 95).

--timerlat-buckets NUM
    Number of histogram buckets (default: 3500).

--timerlat-stoptrace USEC
    Stop trace if thread latency higher than USEC.

--timerlat-trace FILE
    File to save trace to.

--timerlat-dma_latency USEC
    Set /dev/cpu_dma_latency to USEC (set to None to disable) (default: 0).

  Hackbench Load Module Options

--hackbench-jobspercore NUM
    Number of working threads per CPU core (default: 5).

--hackbench-runlowmem True|False
    Run hackbench on machines where low memory is detected.

  Kcompile Load Module Options

--kcompile-source TARBALL
    Source tar ball (default: linux-6.17.7.tar.xz).

--kcompile-jobspercore NUM
    Number of working threads per core (default: 2).

  Stress-ng Load Module Options

--stressng-option OPTION
    Stressor specific option.

--stressng-arg ARG
    Stressor specific arg.

--stressng-timeout T
    Timeout after T seconds.

  Report Analysis Options

-H, --raw-histogram XMLFILE [XMLFILE ...]
    Generate raw histogram data for one or more already existing XML reports.

-Z, --summarize XMLFILE [XMLFILE ...]
    Summarize one or more already existing XML reports.

-S, --source-download [KERNEL_VERSION]
    Download a source kernel from kernel.org and exit.

EXAMPLES

  Basic Usage
Run a 60-second test with default settings (cyclictest measurement, kernel compile and hackbench loads):

    sudo rteval

  Long Duration Test
Run a 12-hour test with timerlat measurement:

    sudo rteval -d 43200 --measurement-module timerlat

  Measurement Only
Run timerlat measurement without any load modules:

    sudo rteval -d 600 --measurement-module timerlat --noload

  With Logging and Annotation
Run with logging enabled and add an annotation to the report:

    sudo rteval -d 3600 -L -a "Testing kernel 6.17.7"

  CPU Isolation with Cpusets
Run with cpusets to isolate measurement on cores 0-3, loads on cores 4-15, and reserve core 0 for housekeeping:

    sudo rteval --cpusets --measurement-cpulist 1-3 \
           --loads-cpulist 4-15 --housekeeping 0

  Custom Timerlat Configuration
Run timerlat with custom interval and stop trace threshold:

    sudo rteval -d 1800 --measurement-module timerlat \
           --timerlat-interval 1000 --timerlat-stoptrace 100

  Analyze Existing Reports
Summarize multiple existing XML reports:

    rteval -Z rteval-20260601-1/summary.xml \
              rteval-20260602-1/summary.xml

FILES

rteval-YYYYMMDD-N/
    Report directory created for each run, containing:

    summary.xml
        XML report with system info, measurements, and statistics

    logs/
        Directory containing measurement and load module logs (with -L option)

MEASUREMENT MODULES

cyclictest
    Measures scheduling latency by running high-priority SCHED_FIFO threads that sleep for a specified interval and measure wakeup latency. This is the traditional measurement tool for realtime Linux testing.

timerlat
    Uses the rtla (Real-Time Linux Analysis) tool's timerlat tracer to measure timer latency. This is a more modern approach that leverages kernel tracing infrastructure and can provide more detailed analysis of latency sources.

LOAD MODULES

kcompile
    Performs parallel kernel compilation to create CPU and I/O load. Compiles with allmodconfig to maximize compilation work.

hackbench
    Scheduler stress test that creates pairs of threads exchanging data via pipes or sockets, exercising the kernel scheduler.

stress-ng
    Flexible stress testing tool with many stressor options for CPU, memory, I/O, and more.

NOTES

Root privileges are required to run rteval as it needs to set real-time scheduling priorities and access system resources.

For production testing, run for at least 12-24 hours to capture rare latency events. The default 60-second duration is only suitable for quick validation.

When using cpusets (--cpusets), ensure your kernel has cgroup v2 support and the cpuset controller is available.

The timerlat measurement module requires a kernel with rtla support (kernel 5.15+ with CONFIG_OSNOISE_TRACER enabled).

SEE ALSO
cyclictest(8), rtla(1), rtla-timerlat(1), stress-ng(1)

AUTHORS
rteval was written by Clark Williams, David Sommerseth, and John Kacur.

This manual page was written for rteval 3.10 in June 2026.

REPORTING BUGS
Report bugs to linux-rt-users@vger.kernel.org

CC: John Kacur <jkacur at redhat.com>, Clark Williams <williams at redhat.com>