Start
How a run works
vero run plan.yaml does four things in order: it loads the plan and refuses it whole if anything
is wrong, it builds a graph of the tasks, it schedules that graph across --jobs workers, and it
reports what each task became. This page follows a run through those four stages. The
plan format is the shape of the file; this is what happens to it.
Load
Everything the loader can reject, it rejects before the first socket opens. A plan that fails to
load sends nothing and exits 64. The stages run in a fixed order, structure first, then
templates, the graph, assertions, holds, secrets, script runtimes and lints, and every stage
runs even after an earlier one found errors, so one load reports every problem it can see.
Each error names the file, the line and the path in the plan:
$ vero plan dup.yaml
dup.yaml:18: a plan file holds one YAML document; found a second
dup.yaml:7: defaults: field defaults not found in the plan
dup.yaml:13: tasks.a.steps[0].http: merge keys (<<) are not supported; write the fields out
dup.yaml:15: tasks.a.steps[0].http.url: duplicate key "url" at line 15, first defined at line 14
dup.yaml:17: tasks.a.steps[0].timeout: "2x" is not a duration; write it like 500ms, 30s or 1m30s
...
dup.yaml: 7 load error(s); nothing ranThe YAML pass is strict on purpose. A key the type does not have is an error with a suggestion,
field jsn not found in an http step (did you mean json?). A key written twice at any depth is an
error, where YAML would keep the last one. Merge keys are refused. A scalar in a string position
that YAML would read as something else is refused with what it would have become, never coerced:
header X-Trace is an int (0755 decodes as 493); quote it: "0755". An empty value is
is empty (null); write a quoted string, "" if empty is meant. An assertion written as a mapping
instead of a string draws an assertion is an expression string such as status == 200, not a mapping; a key/value assertion can decay into one that asserts nothing. A duration is written
500ms, 30s or 1m30s and must be positive. A file with two documents, or none, is refused;
an empty file is the file is empty.
A plan without a lifecycle block does not load. The error lists the three strategies, because
"I did not think about this" and "unique is correct here" must not look the same in a file
someone else reads:
the plan needs a lifecycle block saying how a second run avoids the first run's leftovers (chapter 7.2), strategy one of:
unique: every run namespaces what it creates with run.id; fits when the plan controls the names
reset: a hook puts the target in a known state before any task; the only answer for state keyed by something the plan cannot vary, such as a rate-limit budget
isolated: each run gets its own target (a schema, a container, a tenant), brought up by the plan's own tasksWith reset, the lifecycle.reset steps run before any task, one request in flight, no cookie
jar. A reset step is an exec, an http or a fixture step, needs a timeout (a reset step needs a timeout; a reset that hangs holds the whole run), may not read task results, and takes
neither repeat nor results. Without assertions of its own it is held to status < 400 for
http and exitCode == 0 for exec and fixture. If a reset step does not pass, no task starts,
every task is not_run with the reason not run: lifecycle reset step 1 failed: ..., and the exit
is 3.
A plan is one file with no includes, no loops and no conditionals beyond when. That is what
lets vero plan print the whole graph and count every request before anything is sent, and it is
what lets a reader see the whole test in one place. A plan that needs a table of rows has
matrix, which expands to one task per row at load. A plan that needs computation has a
script step, and the script's requests are the one thing the count cannot know.
The graph
A task has two kinds of edge. needs is a dependency: the task runs after the tasks it needs have
passed. holds is a claim on a named resource: the task never overlaps another holder. The
graph is what vero plan --graph prints:
$ vero plan --graph examples/rate-limit.yaml
task login
task rate-limit-trips-on-fourth
needs login derived:template at tasks.rate-limit-trips-on-fourth.steps[0].http.headers.Authorization
holds ratelimit/orders-api-key exclusive
task pagination
needs login derived:template at tasks.pagination.steps[0].http.headers.Authorization
needs login derived:template at tasks.pagination.steps[1].http.headers.Authorization
needs login derived:template at tasks.pagination.steps[2].http.headers.Authorization
holds ratelimit/orders-api-key exclusive
task filtering
needs login derived:template at tasks.filtering.steps[0].http.headers.Authorization
needs login derived:template at tasks.filtering.steps[1].http.headers.Authorization
needs login derived:template at tasks.filtering.steps[2].http.headers.Authorization
holds ratelimit/orders-api-key exclusive
resource ratelimit/orders-api-key exclusive
held by rate-limit-trips-on-fourth, pagination, filteringneeds
No task in that plan writes needs. Every edge is derived:template: the task's header reads
{{ tasks.login.results.token }}, and the reference is the declaration. An edge has one of three
origins, and the graph shows each with the place in the plan that created it:
explicit, from aneeds:list, for ordering with no data flowing.derived:template, from a{{ tasks.<name>.results.<key> }}in another task's strings.derived:when, fromtasks.<name>in awhenguard, since the guard needs that task's result to be known.
Two references to the same task collapse into one edge with every origin kept. A step reading a
result an earlier step of its own task produced is not an edge: the steps of one task share a
scratch namespace. A main task may not reference a finally task in any of the three ways, and a
guard may not read its own task's results:
finally-ref.yaml:9: tasks.a.when[0]: "tasks.a.results.x == 200" reads task a's own results; a guard runs before the task does
finally-ref.yaml:15: tasks.b.needs[0]: main task b references finally task cleanup; finally tasks run after every main task has finished, so nothing in tasks can depend on themA cycle is refused naming every task that could not run, with the levels that could:
cycle.yaml: the task graph has a cycle and nothing will run
scheduled levels: [[a] [d]]
unreachable (in a cycle): [b c]holds
holds is not ordering. It says "not during", never "after". Two tasks that hold one
exclusive resource run in whichever order the scheduler reaches them, and never at the same
time. A shared claim overlaps other shared claims on the same resource and nothing exclusive.
A bare name in holds takes the resource's declared mode, exclusive when the resource says
nothing, and { name, mode } overrides it for one task. A resource named in holds must be
declared: task a holds resource budget, which is not declared in resources (the plan declares no resources).
The rate-limit plan is why the second kind of edge exists. Three tasks spend one API key's
budget of three requests, and one of them asserts that its fourth request is refused. Run as
independent tasks on several workers, the 429 lands on whichever request arrives last, and the
plan passes about half the time. With the three tasks holding ratelimit/orders-api-key, no two
of them overlap, and the plan passes 50 runs out of 50 at --jobs 8
(the example, against nginx).
No needs edge could say that: the three tasks have no order between them, and adding one would
serialise the whole plan.
The scheduler
vero run starts --jobs workers (4 on a four-core machine, GOMAXPROCS) and a dispatcher. The
dispatcher keeps the tasks whose every upstream has passed in a ready list, in plan order, and
walks that list each time a worker is free. A ready task starts when its resources are free and
stays in its place otherwise, so a later task that holds nothing can start past it.
A task takes all of its resources at once or none of them, and it takes them in the dispatcher,
before a worker is assigned. A task waiting for a resource never occupies a worker. A task that
holds any resource runs with one request in flight at a time: a repeat in parallel mode
inside a holder sends its requests one after another, because the point of holding a budget is to
know who is spending it.
Shared and exclusive claims follow a reader-writer rule with writer preference. While an exclusive claim is waiting on a resource, new shared claims on it are refused, so a steady stream of readers cannot starve a writer. Resources are released when the task completes, whatever the outcome: a failure, an error, a timeout and a panic all release.
When a task ends, each of its dependents loses one upstream. A dependent whose every upstream has
ended is settled at once. If every upstream passed, it joins the ready list. If one did not, the
dependent is blocked by the first upstream that did not pass, or skipped if its own when
guard is false. The block propagates: a task downstream of a blocked task is blocked too, and
nothing in that chain ever reaches a worker. Here broken fails, optional skips, and the tasks
behind them settle three different ways:
FAIL broken › step 1 › assert #1 settle.yaml:19
assert status == 200 failed
status = 500
BLOCKED after-broken blocked by broken, which failed
BLOCKED needs-optional blocked by optional, which was skipped (when guard false: env.feature == "on"). The run did not finish, so the exit code is 3. If this task is optional too, give it the same when guard so it is skipped in its own right
1 passed 1 failed 0 errored 0 timed out 4 skipped 3 blocked 0 not run (9 tasks, 2 requests, 0.0s)after-broken is blocked by a failure, and the failure's exit code already covers it.
needs-optional is blocked by a skip, which nothing else would report, so the run exits 3. A
fourth task, wants-optional, carries the same guard as optional and is skipped in its own
right; it is the fix the message suggests. The fifth skipped task is tagged slow.
A task
A worker runs a task in a fixed order. The tag check comes first: a task tagged slow is
skipped with excluded by tag slow; run with --include-tag slow to include it, and slow is
the only tag excluded by default. Then the when guards are evaluated against what has been
published so far; a result no task has published reads as nil, so tasks.x.results.y != nil is
false and the task skips rather than sending a request built from nothing. A guard that cannot be
evaluated at all makes the task errored, since that is a mistake in the plan.
Then the task starts. Its timeout, if set, covers all of its steps, and each step's own deadline
still applies inside it. A task with cookies: true gets a cookie jar for this task only. The
steps run in order, and the task stops at the first step that does not pass. The step's state
becomes the task's, failed, errored or timed_out, with step N: in front of the reason.
Results are published only when the whole task passed. A downstream task never sees a partial result from a task that failed at its third step; it is blocked instead. Inside the task, a later step reads an earlier step's result as soon as the step passes.
Finally
finally tasks are their own graph. They run after the main graph has drained, in the same
workers, under --finally-timeout (30 s by default), and under a context the run's cancellation
does not reach. So they run after a run timeout, and after the first SIGINT.
Their edges to main tasks are satisfied by the main graph having drained, whatever its states, so
an explicit needs: [create-order] on a finally task orders nothing. What blocks a finally task
is data: a template that reads a result of a main task that published nothing, because the task
failed, skipped or was blocked. A when guard on the finally task reads that result as nil and
skips the task instead, which is how teardown says "only if there is something to tear down".
In the run above, cleanup reads {{ tasks.broken.results.code }} and is blocked by broken;
cleanup-guarded has when: ['tasks.broken.results.code != nil'] and skips:
TEARDOWN
BLOCKED cleanup blocked by broken, which failedA main task may not reference a finally task. With --only, a finally task is kept only when
every main task it references is selected, and the summary says which were left out.
Ending
A run ends when every task has settled, when --timeout (or the plan's timeout) fires, or on
a signal. A run timeout ends running steps timed_out with run timeout 300ms fired while it was running: ... and marks tasks that had not started not_run: run timeout 300ms; finally still
runs. The first SIGINT or SIGTERM takes the same path:
vero: interrupt: stopping; unfinished tasks are not run and finally still runs. Send it again to exit now.Running steps end timed_out with the cause cancelled (interrupt), unstarted tasks are
not run: cancelled (interrupt), and finally runs under its own deadline. A second signal
prints vero: second interrupt: exiting now with code 3; teardown was skipped and exits at once.
Every task now has one of the seven states, and the exit code is computed from them in one place, in the order 64, 3, 2, 1, 4, 0. The run prints its failure blocks and the summary line, then writes the exports it was asked for. The states and their codes are on States and exit codes; the flags, the output and the exports are on the command line page.
This page is docs/content/guides/how-a-run-works.md in the repository.