Skip to content

Bios shenanigans #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
uncleDecart wants to merge 2 commits into
base: main
Choose a base branch
from
+473 −18
Open

Bios shenanigans #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4d124c2
Add BIOS scraping from Redfish
uncleDecart Jan 27, 2026
8424b6e
Update README
uncleDecart Jan 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
209 changes: 191 additions & 18 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,14 @@
# rtos_bench: benchmarking suite to analyse real-time (RT) performance of an operating system

A lightweight, configurable Python framework for running system and application benchmarks using Hydra for flexible experiment management.
This repository provides a single entry-point script to run various performance tests with reproducible configurations defined in conf/config.yaml.
A comprehensive Python framework for benchmarking, analyzing and validating real-time (RT) performance of operating systems. Combines Docker-containerized benchmarks with statistical analysis tools based on Extreme Value Theory (EVT) to determine if a system meets real-time requirements.

## Key Features

- **Containerized Benchmarks**: Run reproducible RT benchmarks (Caterpillar, Cyclictest, iperf3, CODESYS) in Docker or on your host system
- **Intel RDT Integration**: Full support for Cache Allocation Technology (CAT) and Memory Bandwidth Allocation (MBA)
- **Statistical RT Validation**: EVT-based analysis with Region of Acceptance (RoA) for probabilistic WCET estimation
- **BIOS Collection via Redfish**: Automatically capture BIOS settings from BMC/iDRAC before benchmarks
- **Jupyter Analysis Notebooks**: Interactive reports for analyzing benchmark results and RT readiness

## Prerequisites:

Expand All @@ -19,12 +26,32 @@ curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync
```

## How to run benchmark
4. Additional system requirements:
- Docker (for containerized execution)
- `intel-cmt-cat` package (for pqos/Intel RDT support)
- Root access (required for pqos, IRQ affinity, and some metrics)



## Quick Start

```bash
uv run main.py

# Install dependencies and virtual environment (venv)
uv sync

# Build all Docker images first
sudo .venv/bin/python3 main.py run.command=build

# Run a benchmark (e.g. caterpillar)
sudo .venv/bin/python3 main.py run.command=caterpillar

# Analyze results in Jupyter
uv run jupyter-lab
```

> **Note**: Benchmarks require root access for pqos, IRQ affinity configuration, and hardware monitoring. Use `sudo .venv/bin/python3/main.py` instead of `uv run main.py`

## How to run jupyter notebook (analysis software)

```
Expand Down Expand Up @@ -56,7 +83,6 @@ After that you can open any report and run it, just double-click on it like here
├── iperf3/
├── mega-benchmark/
├── codesys-jitter-benchmark/
├── data/ # Store experiments here
├── outputs/ # Where we run experiment bundles
├── notebooks/ # Jupyter notebooks to analyse data
├── src/ # libraries
Expand All @@ -73,24 +99,171 @@ All experiment parameters are controlled via Hydra’s configuration file at:
conf/config.yaml
```

## Example configuration
You can override any configuration parameter from the command line:

```bash
sudo .venv/bin/python3 main.py run.command=cyclictest run.t_core="3,5"
```

## Run Configuration

```yaml
run:
command: "caterpillar"
llc_cache_mask: "0x000f"
t_core: "3"
stressor: true
tests_path: "tests"
command: "caterpillar" # Benchmark to run
t_core: "9,11" # Target CPU cores
numa_node: "1" # NUMA node for cpuset-mems (should be same as NUMA node for t_core)
stressor: true # Enable stress workload
metrics: true # Enable metrics monitoring
docker: true # Run inside Docker container
cat_clos_pinning:
enable: true # Pin test PID to CLOS
clos: 1 # CLOS ID to use
```

| Parameter | Type | Description |
| ---------------------------- | ------- | -------------------------------------------------------------------------------------------------------------- |
| `run.command` | str | Benchmark to run: `caterpillar`, `cyclictest`, `iperf3`, `mega-benchmark`, `codesys-jitter-benchmark`, `codesys-opcua-pubsub`, or `build`. |
| `run.t_core` | str | Target CPU cores for running the benchmark (e.g., `"3,5,7,9"` or `"9,11"`) |
| `run.numa_node` | str | NUMA node for cpuset-mems (should be same as NUMA node for t_core) |
| `run.stressor` | bool | Enables additional stress workload during the benchmark |
| `run.metrics` | bool | Enable real-time metrics monitoring (CPU temp, IRQs, memory, etc.) |
| `run.docker` | bool | Run benchmark inside Docker container (if `false`, runs on host) |
| `run.cat_clos_pinning.enable`| bool | Enable pinning test PID to specified CLOS (caterpillar/cyclictest only) |
| `run.cat_clos_pinning.clos` | int | CLOS ID to pin the test process to |

## Intel RDT/CAT Configuration (pqos)

Configure Intel Resource Director Technology (Cache Allocation Technology, Memory Bandwidth Allocation):

```yaml
pqos:
interface: "os" # 'os' for resctrl (recommended), 'msr' for direct access
reset_before_apply: true # Reset all allocations before applying new ones

classes:
- id: 1
description: "real-time workload"
l3_mask: "0x00ff" # L3 cache mask (8 cache ways)
l2_mask: "0x00ff" # L2 cache mask
mba: 100 # Memory Bandwidth Allocation (%)
pids: [] # PIDs to assign to this class
cores: [] # CPU cores to assign to this class
- id: 0
description: "background worker"
l3_mask: "0x7f00" # Different cache ways for isolation
l2_mask: "0xff00"
mba: 10
cores: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]
pids: [115, 118]
```

| Parameter | Type | Description |
| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------- |
| `command` | str | Benchmark to run. One of: `caterpillar`, `cyclictest`, `iperf3`, `mega-benchmark`, `codesys-jitter-benchmark`. |
| `llc_cache_mask` | str | Hexadecimal mask for Last-Level Cache (LLC) configuration. |
| `t_core` | str/int | Target CPU core for running the benchmark (i.e. '3,5,7,9') |
| `stressor` | bool | Enables additional stress workload during the benchmark. |
| `tests_path` | str | Path to the directory containing benchmark implementations. |
| Parameter | Type | Description |
| -------------------------- | ------ | ------------------------------------------------------------------ |
| `pqos.interface` | str | Interface mode: `os` (resctrl, required for PIDs) or `msr` (direct)|
| `pqos.reset_before_apply` | bool | Reset all allocations before applying new configuration |
| `pqos.classes[].id` | int | Class of Service (CLOS) ID |
| `pqos.classes[].l3_mask` | str | Hexadecimal L3 cache way mask |
| `pqos.classes[].l2_mask` | str | Hexadecimal L2 cache way mask |
| `pqos.classes[].mba` | int | Memory Bandwidth Allocation percentage (10-100) |
| `pqos.classes[].cores` | list | CPU cores to assign to this CLOS leave empty if not used |
| `pqos.classes[].pids` | list | Process IDs to assign to this CLOS leave empty if not used |


## IRQ Affinity Configuration

Configure IRQ and RCU task affinity to isolate real-time cores:

```yaml
irq_affinity:
enabled: true
housekeeping_cores: "0-1" # Cores for handling IRQs and RCU
```

## BIOS Settings Collection via Redfish

Automatically collect BIOS settings from servers with Redfish-enabled BMC (e.g., Dell iDRAC) before running benchmarks:

```yaml
bios:
enable: true
redfish:
host: "192.168.1.100" # BMC/iDRAC IP address
username: "root"
password: "YOUR_PASSWORD"
verify_ssl: false # Set to true for valid SSL certificates
timeout: 15

output:
format: "json" # Output format: json, yaml, or text
file: "${hydra:run.dir}/bios.json"
pretty: true
```

| Parameter | Type | Description |
| ------------------------- | ------ | ------------------------------------------------------------------ |
| `bios.enable` | bool | Enable/disable BIOS settings collection |
| `bios.redfish.host` | str | BMC/iDRAC hostname or IP address |
| `bios.redfish.username` | str | Username for Redfish API authentication |
| `bios.redfish.password` | str | Password for Redfish API authentication |
| `bios.redfish.verify_ssl` | bool | Verify SSL certificates (set `false` for self-signed certs) |
| `bios.redfish.timeout` | int | Connection timeout in seconds |
| `bios.output.format` | str | Output format: `json`, `yaml`, or `text` |
| `bios.output.file` | str | Path to save BIOS settings (supports Hydra interpolation) |
| `bios.output.pretty` | bool | Enable pretty-printing for JSON output |

## Test-Specific Configuration

### Caterpillar
```yaml
caterpillar:
n_cycles: 7200 # Number of measurement cycles
```

### Cyclictest
```yaml
cyclictest:
loops: 100000 # Number of test loops
```

## Metrics Monitoring

When `run.metrics: true`, the following monitors collect data during benchmark execution:

| Monitor | Output File | Description |
| ---------------- | ------------------------------ | ---------------------------------------- |
| CPU Monitor | `cpu_monitor.csv` | Per-core CPU temperatures |
| IRQ Monitor | `irq_monitor.csv` | Interrupt counts per CPU |
| MemInfo Monitor | `meminfo_monitor.csv` | Memory statistics from `/proc/meminfo` |
| SoftIRQ Monitor | `softirq_monitor.csv` | Software interrupt statistics |
| CPUStat Monitor | `cpustat_monitor.csv` | CPU usage statistics |
| PQOS Monitor | `pqos_monitor.csv` | Intel RDT monitoring data |

Configure monitoring intervals in the config:

```yaml
cpu_monitor:
path: "${hydra:run.dir}/cpu_monitor.csv"
interval: 1.0
```

## Output Files

Each benchmark run creates a timestamped directory in `outputs/` containing:

- `output.csv` - Benchmark results
- `sysinfo.json` - System information snapshot (includes Hydra configuration)
- `bios.json` - BIOS settings (if enabled)
- `*_monitor.csv` - Various metrics logs (if enabled)
- `.hydra/` - Hydra configuration logs

## Security Note

⚠️ **Important**: The Redfish password is stored in the configuration file. Consider:
- Using environment variables for sensitive credentials
- Restricting file permissions on `config.yaml`
- Not committing passwords to version control

## References

- [Dealing with Uncertainty in pWCET Estimations](https://dl.acm.org/doi/abs/10.1145/3396234) - Region of Acceptance methodology
- [Probabilistic-WCET Reliability](https://dl.acm.org/doi/10.1145/3126495) - EVT hypothesis validation
16 changes: 16 additions & 0 deletions conf/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,22 @@
benchmark_output_path: "${hydra:run.dir}/output.csv"
sysinfo_collector_file: "${hydra:run.dir}/sysinfo.json"

# --- Configuration for getting BIOS data from redfish

bios:
enable: true
redfish:
host: "0.0.0.0"
username: "root"
password: "SECRET_PASSWORD"
verify_ssl: false
timeout: 15

output:
format: "json"
file: "${hydra:run.dir}/bios.json"
pretty: true

# --- Intel CAT specific configuration ---

pqos:
Expand Down
6 changes: 6 additions & 0 deletions main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@

from omegaconf import DictConfig, OmegaConf

from src.bios_settings import process_bios_settings

from src.metrics import (
CPUmonitor,
InterruptMonitor,
Expand Down Expand Up @@ -91,6 +93,10 @@ def main(cfg: DictConfig):
collector.gather_all(cfg)
collector.dump_to_file(cfg.sysinfo_collector_file)

# Collect BIOS settings via redfish
if cfg.bios.enable:
process_bios_settings(cfg.bios)

runner = DockerTestRunner(cfg)

if cfg.run.command == "build":
Expand Down
Loading