Sub-microsecond SEC 15c3-5 risk checks on AWS F2 via FPGA MMIO | Hacker News Reader

Sub-microsecond SEC 15c3-5 risk checks on AWS F2 via FPGA MMIO

github.com

Most SEC Rule 15c3-5 pre-trade risk controls implemented in software incur 2 to 5 µs of OS thread scheduling jitter and kernel context-switching latency. During market micro-bursts, this tail variance leads directly to adverse execution selection and fill slippage.

We built an immutable FPGA appliance substrate for AWS EC2 (ami-06e581f3a6d6c3e32, targeting Xilinx UltraScale+ silicon) that pins risk-gate state validation directly to physical PCIe MMIO registers.

Empirical telemetry logged directly on the AWS F2 substrate:

* Direct PCIe TLP Non-Posted Read (BAR2): 1.220 µs P50 | 1.231 µs P99 (11 ns tail jitter spread)

* Bare-Metal Kernel-Bypass Logic: 240 ns P50 | 251 ns P99 | 380 ns P99.9

* Multi-Channel Saturated Line Rate: 14.95M ops/sec across 16 pinned channels

* Local Zero-Copy Binary Socket IPC (/tmp/oikonomia.sock): 19.77 µs

* Core Power & Dissipation: 18.2W at 42.5°C nominal core temp

The public repository contains our synthetic benchmark harness, terminal outputs, and the 1-page institutional diligence memo: https://github.com/joeiko3424-svg/oikonomia-benchmarks

Curious how other low-latency and systems engineers here are handling hypervisor-induced jitter and memory-bus contention when deploying trading or risk infrastructure in public cloud environments.

OS thread scheduling jitter and kernel context-switching latency

This goes away with isolcpus. Have you tried that?

Yes, isolcpus (alongside nohz_full, rcu_nocbs, and strict IRQ affinity via smp_affinity) is the standard Day-1 playbook for any low-latency trading desk. But it only solves scheduler contention in user-space. It does absolutely nothing to eliminate the physical realities of running a software stack.

Even on a perfectly isolated, tickless core with kernel-bypass (DPDK/Solarflare OpenOnload), you are still bound by three hard latency floors that software tuning cannot fix:

System Management Interrupts (SMIs) & IPIs: The BIOS/UEFI can and will pause the CPU underneath the OS for hardware management. Similarly, TLB shootdowns (Inter-Processor Interrupts) force core stalls regardless of scheduler isolation.

The Cache-Miss Death Spiral: A single L3 cache miss costs ~60-100ns to fetch from main memory. If your risk validation logic touches cold data during a micro-burst, your P99 tail latency explodes, isolated CPU or not.

PCIe to L1 Transit: To evaluate a payload in software, the packet must cross the PCIe bus to the NIC, DMA into host memory, trigger an event, and move from L3 to L1 cache before the ALU can even look at it.

We moved this out-of-band entirely because we are operating in a time domain where an isolated CPU is still too slow.

Since posting this, our latest regression on the F2 silicon hit a 160ns P50 latency with an 84.0ns hardware drop shunt.

84 nanoseconds is less time than it takes an isolated CPU core to fetch a missed line from RAM. You cannot isolcpus your way out of physics. When the SEC requires pre-trade risk validation, the only way to avoid the latency tax is to evaluate the packet in flight at the PCIe gate level.

Reply on news.ycombinator.com