August 19, 2026 1 min read

OS Optimizations for DGX Spark

Some optimizations for stable inferencing performance on DGX spark

While working on Drove and from years of using Apache Mesos it became apparent that container performance gets severely affected by how and where they are spun up and what the rest of the operating system is doing at that time.

How CPU and Memory affects container performance

Processes running inside containers and other busy OS processes fighting each other for cores - Under load when all parts of the system is under heavy load, multiple processes from the operating system will start stealing cycles from across cores.
Processes across different containers fighting for each other - When multiple containers are running on the same server, and they are under load they will fight for CPU cycles. The last thing we want would be to have a process start on one core and then move to another once it comes back on some interrupt.
NUMA - On multi-processor servers that frequently are setup using NUMA (Non uniform memory architecture), the distance between the cores executing the code and where the memory being allocated, significantly affect performance for memory bound applications.
Memory Pressure and Swappiness - Swap is meant to be a fallback for the system to extend it’s memory when it’s running out of memory. For desktop systems this is set a safe level. However, as we would be deploying models on these boxes and would want to use most of the memory for it, the operating system would start swapping. This will affect system performance. The other thing we would want the system to do is to aggressively reclaim cached memory pages instead of the caches like the file-system caches lingering in memory while the rest of the system is getting starved.
Other shared resources (network, disk, firewall etc)

This is a larger topic and I shall try to cover some of this in more depth later in another note where I discuss how we extract performance from containers in Drove. For this discussion however, most of these points are irrelevant because:

The DGX Sparks’ main USP is the Unified Memory architecture.
We are not planning to run multiple competing containers on these machines. At least not in the scale a typical Drove executor or a Kubernetes node would.

We take the above assumptions which would hold true for majority of the users of this hardware and try to see how we can improve the performance a bit.

Core layout for DGX Spark CPU

The DGX Spark has 20 cores arranged in the following mode:

CPU IDs	Max MHz	Role
0–4	2808	Slow cores (Cortex-A78C)
5–9	3900	Fast cores (Cortex-X925)
10–14	2808	Slow cores (SMT siblings of 0–4)
15–19	3900	Fast cores (SMT siblings of 5–9)

Strategy

Our strategy would be consisting of the following:

Assign all os processes including docker engine etc to the slow cores.
Move network interrupt handling to slow cores.
Shut down to GUI to reduce random Wayland/Gnome related handling and memory usage.
Move inferencing related services on the performance cores.

Base Setup

Hardware

2x MSI EdgeXpert
Connected using 400G QFSP Cable
1 Gbps connectivity on management port
Laptop on wifi

Software

My setup is the one published by @Mia in this GitHub Repo.

The three OS related changes are done are mentioned in the following sections.

Localizing the OS Cores

The trick lies in configuring CPU Affinity in systemd. We can force all processes to start on defined cores and manually place the relevant docker on the performance core.

sudo mkdir -p /etc/systemd/system.conf.d
echo -e '[Manager]\nCPUAffinity=0-4 10-14' | sudo tee /etc/systemd/system.conf.d/cpu-affinity.conf

Effect: All services started by systemd run on slow cores (0–4, 10–14). Fast cores (5–9, 15–19) remain available for vLLM worker processes and NCCL.

Note You must reboot the node after this change for it to take effect.

Setting IRQ Affinity

We want to offload the network handling for the NVIDIA GPU and RoCE (mlx/ConnectX) hardware interrupt handling to slow cores. This will prevent IRQ handling from stealing cycles from the fast cores during inference.

We do this by creating a systemd service on both nodes as follows:

sudo tee /etc/systemd/system/irq-affinity.service << 'EOF'
[Unit]
Description=Set NVIDIA and RoCE IRQ affinity to slow cores
After=multi-user.target
Wants=multi-user.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/bash -c 'for irq in $(grep -iE "nvidia|mlx5|connectx|roce" /proc/interrupts 2>/dev/null | awk -F: "{print \$1}" | tr -d " "); do echo "0-4,10-14" > /proc/irq/$irq/smp_affinity_list 2>/dev/null; done'

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now irq-affinity.service

Headless boot/Turn off GUI

In most practical scenarios, we would spin up the cluster and connect to it from another node. In this situation, running a GUI on these nodes seems like a needless wastage of precious memory. We would want to preserve as much of the memory possible to the inferencing itself.

To achieve this we need to change the init default.target symlinked from graphical.target to multi-user.target.

sudo systemctl set-default multi-user.target

To get back to the GUI set the runlevel back to graphical.

sudo systemctl set-default graphical.target

Note This will promptly turn off the GUI. So plan accordingly.

Changing swap settings

What we would try to do here is to tell the system to start swapping right at the end when it is almost out of memory. If you are using the DGX boxes only for inferencing, this is unlikely to happen. And Even if you are setting the memory usage limit in vllm etc to 85%-90% of the system ram, the amount of memory left is still in GBs for rest of the system to use.

sudo tee /etc/sysctl.d/99-vm-tune.conf << 'EOF'
# VM tuning for VLLM inference workloads
vm.swappiness=1
vm.vfs_cache_pressure=200
EOF
sudo sysctl -w vm.swappiness=1
sudo sysctl -w vm.vfs_cache_pressure=200

This will setup for the next boot as well as in the current session. You can validate this by running the following:

cat /proc/sys/vm/swappiness           # should print: 1
cat /proc/sys/vm/vfs_cache_pressure   # should print: 200

Final Verification

Please complete a full reboot of all nodes before getting to this step

You can run the following commands to verify your changes:

# Check CPU affinity
cat /etc/systemd/system.conf.d/cpu-affinity.conf

# Check IRQ affinity service
systemctl is-enabled irq-affinity.service
systemctl is-active irq-affinity.service

# Check IRQ pinning at runtime
grep -iE "nvidia|mlx5|roce" /proc/interrupts | while read line; do
  irq=$(echo "$line" | awk -F: '{print $1}' | tr -d ' ')
  echo "IRQ $irq: $(cat /proc/irq/$irq/smp_affinity_list 2>/dev/null)"
done

# Check default target
systemctl get-default

# Check memory settings
cat /proc/sys/vm/swappiness           # should print: 1
cat /proc/sys/vm/vfs_cache_pressure   # should print: 200

Changes to Dockerfile

We need to pin the cores for the container to the actual performance cores using the following command:

The change is simple. Add the following to container params in docker-compose.dspark.yml.

  ...
  cpuset: "5-9,15-19" # Pin to performance cores
  ulimit:
    ...

Testing Methodology

The benchmarks were run using llama-benchy using the following commands:

Depth Sweep

This will test processing performance as context depth grows. Data is loaded into KV Cache and then prompt processing happens.

llama-benchy --base-url http://<endpoint>/v1 \
  --model deepseek-v4-flash-0731 \
  --pp 128 512 2048 --tg 32 128 \
  --depth 0 4096 16384 32768 --runs 3 \
  --skip-coherence --format md \
  --save-result /path/to/output.md

Concurrency

This will test how performance varies as number of parallel sessions grow.

llama-benchy --base-url http://<endpoint>/v1 \
  --model deepseek-v4-flash-0731 \
  --pp 512 --tg 128 --depth 0 --runs 3 \
  --concurrency 1 2 4 6 \
  --skip-coherence --format md \
  --save-result /path/to/output.md

Results of experiments

We saw some improvements across the board. With better performance for more parallel tasks.

Model: deepseek-v4-flash-0731

Aggregate tok/s — OS ON vs OS OFF

Prompt	Conc	OS OFF (baseline)	OS ON	Δ
256	1	67.3	69.0	+2.4%
256	2	104.6	111.1	+6.2%
256	4	154.6	141.7	−8.3%
256	6	188.3	209.3	+11.2%
2048	1	44.2	44.1	−0.2%
2048	2	58.1	58.4	+0.5%
2048	4	75.2	69.2	−8.0%
2048	6	79.2	76.8	−3.0%
8192	1	20.8	28.8	+38.2%
8192	2	24.8	39.6	+59.8%
8192	4	26.5	43.2	+62.9%
8192	6	27.2	46.0	+69.1%
32768	1	6.6	30.6	+362.0%
32768	2	7.4	37.3	+403.8%
32768	4	7.5	41.9	+460.6%
32768	6	7.5	44.6	+491.9%
131072	1	1.6	26.8	+1539.5%
131072	2	1.8	26.4	+1372.6%
131072	4	1.8	33.3	+1754.6%
131072	6	1.8	1.7	−7.5%

Median TTFT (s) — OS ON vs OS OFF

Prompt	Conc	OS OFF (baseline)	OS ON	Δ
256	1	0.253	0.249	−1.6%
256	2	0.362	0.394	+8.9%
256	4	0.558	0.573	+2.6%
256	6	1.000	0.828	−17.2%
2048	1	1.187	1.227	+3.4%
2048	2	1.980	1.980	+0.0%
2048	4	3.060	3.094	+1.1%
2048	6	4.634	4.652	+0.4%
8192	1	4.454	2.333	−47.6%
8192	2	7.909	4.036	−49.0%
8192	4	12.594	6.660	−47.1%
8192	6	16.802	8.850	−47.3%
32768	1	17.687	2.445	−86.2%
32768	2	32.101	4.224	−86.8%
32768	4	49.502	7.431	−85.0%
32768	6	66.364	9.695	−85.4%
131072	1	76.697	3.117	−95.9%
131072	2	140.453	6.806	−95.2%
131072	4	211.810	9.408	−95.6%
131072	6	282.220	304.776	+8.0%

Key takeaways

	Short prompts (256–2048)	Long prompts (8K–128K)
Throughput	±11%	+38% to +1755%
TTFT	±17%	−47% to −96%
131K / c=6	—	KV pool ceiling (~1.8 tok/s, no config helps)

The numbers look dramatic because of the low values. It would not be prudent to expect to a 2000% bump. All these settings do is to reduce entropy in the system.

Similar changes, however have dramatic impact on server machines. But that is a story for another day.

Appendix

To reverse the changes you have done run the following:

# 1. Remove CPU affinity pinning
sudo rm /etc/systemd/system.conf.d/cpu-affinity.conf
sudo systemctl daemon-reexec

# 2. Remove IRQ affinity service
sudo systemctl disable --now irq-affinity.service
sudo rm /etc/systemd/system/irq-affinity.service
sudo systemctl daemon-reload

# 3. Restore graphical boot (optional)
sudo systemctl set-default graphical.target

# 4. Revert swappiness changes
sudo rm /etc/sysctl.d/99-vm-tune.conf
sudo sysctl -w vm.swappiness=60
sudo sysctl -w vm.vfs_cache_pressure=100

# 5. Reboot to apply all changes
sudo reboot