The rate-limit case against a real limiter

Historical experiment (2026-09-27). Its machines, installed tools, and deployment topology describe that run, not vero prerequisites. For current setup see Getting started and Running the examples.

Chapter 10.4 calls this the most valuable experiment the dossier did not run: its 3% came from a token bucket with no time window, and a real limiter with a window and a burst allowance might be more forgiving, which would weaken the case for holds. This is one data point, nginx limit_req, run on 2026-09-27 between 07:05:31Z and 07:05:35Z.

Result: it does not weaken the case. Against nginx, four edge-less requests racing for the budget passed 135 of 300 runs at --jobs 8 and 300 of 300 at --jobs 1; the loser was whichever request arrived last, spread over all four positions. The serial plan with holds passed 300/300. And without a reset between runs, the second run onward gave [429 429 429 429] every time, as chapter 7.1 measured. The failure stays schedule-dependent under a real limiter.

Historical setup

• Limiter: docker.io/library/nginx:1.27-alpine, pulled as nginx@sha256:65645c7bb6a0661892a8b03b89d0743208a18dd2f3f17a54ef4b76fb8e2f2a10, one worker process, on the k3s cluster in namespace testing-platform, from deploy/experiments/real-limiter/.
• Config, the part that matters:

  limit_req_zone $http_x_api_key zone=perkey:10m rate=1r/m;
  limit_req_status 429;
  location /orders {
    limit_req zone=perkey burst=2 nodelay;
    try_files /orders.json =404;
  }

  One leaky bucket per X-Api-Key, one request a minute with a burst of two served at once, so four requests in quick succession give 200 200 200 429. Checked by hand with curl on one key: 200 200 200 429 429. (return 200 would not do: return runs before limit_req sees the request, so the file is served with try_files.)
• Path: vero in the worker container → the veth gateway $POD (10.233.1.1) → NodePort 30561 → kube-proxy → the pod. A NodePort, not kubectl port-forward. No real network latency: this is a single node, so like the dossier's numbers it measures ordering, not a WAN.
• vero at commit edb9a03 plus this experiment's plans, go1.26.7.
• Plans in testdata/experiments/real-limiter/. nginx has no reset endpoint, so variants A and B use a fresh key per run (X-Api-Key: {{ run.id }}, lifecycle unique); variant C uses one fixed key for every run.

Historical commands — do not use as current setup

Current-tree warning: make infra-up now deploys Postgres using NodePort 30561, the same port this limiter used. Do not follow the recorded command block as current setup: the deployments would collide. The block below is a transcript of the experiment, not an install recipe.

make infra-up                                   # the namespace
kubectl apply -k deploy/experiments/real-limiter
export VERO_LIMITER_URL=http://$POD:30561 VERO_LIMITER_KEY=g02-fixed-<unix time>
bin/vero run --jobs 8 --repeat 300 --events a.ndjson testdata/experiments/real-limiter/serial-holds.yaml
bin/vero run --repeat 300 --compare-jobs --jobs 8 --events b.ndjson testdata/experiments/real-limiter/four-edgeless.yaml
bin/vero run --jobs 1 --repeat 300 --events c.ndjson testdata/experiments/real-limiter/serial-no-reset.yaml
kubectl delete -k deploy/experiments/real-limiter

The 429 positions were counted from the request events of each run in the event logs.

Results

Every B run had exactly one 429. vero's --compare-jobs output for B flagged all four tasks FLAKY at --jobs 8 only and printed the holds hint for each.

Against the dossier

The parallel pass rate is much higher than the dossier's 3% with either limiter, and about the same with both. That points at the client, not the limiter: vero's dispatcher starts the four ready tasks in plan order microseconds apart, so the fourth is more often the last to arrive, where the dossier raced four bare goroutines. The limiter type barely moved it (41% to 45%).

Direction of the effect

• The case for holds and repeat.mode: serial stands. A leaky bucket with a burst allowance did not make the racing version reliable: it failed 55% of the time at --jobs 8, never at --jobs 1, and the losing request was spread across all four positions. Nothing about a real limiter's window made "the fourth request" meaningful when the four were concurrent.
• The case for lifecycle is stronger than the dossier stated it. With a limiter that refills one request a minute, a second run a few milliseconds later is [429 429 429 429], and it stayed that way for 299 runs. A plan against a real limiter needs unique keys or a reset hook; a refill window does not rescue it at test speed.
• What this does not show. Everything ran on one node. Over a real network with jitter, arrival order is noisier and the parallel pass rate may move toward the dossier's 3% or away from it; either way it stays a draw the plan does not control. Envoy, Kong and cloud gateways were out of scope.

Historical teardown

kubectl delete -k deploy/experiments/real-limiter removed the ConfigMap, Service and Deployment; the namespace and D03's Postgres stayed.