States and exit codes

A task ends in one of seven states, and the run's exit code is computed from them in one place. There are seven because "did not pass" has four different fixes: an assertion to look at, a server to fix, a guard that was false, or a run that ended before the task started.

The seven states

The summary line spells two of them with a space, timed out and not run. The event log, JUnit and TAP use the underscore forms.

How a task reaches each state

A task passes when every step ran and every assertion held. Only then are its results published; a downstream task never sees a partial result from a task that did not pass.

A task fails when a step's assertion evaluated and did not hold, or evaluated to n/a because it compared a timing phase that did not happen. The task stops at that step.

A task errors when a step could not complete: a connection refused, a TLS handshake the run does not trust, a response that is not the JSON it claims to be, a process killed by a signal vero did not send. A when guard that cannot be evaluated also errors the task, with the reason when guard could not be evaluated: <guard>: <error>, because a guard that reads a result the plan never declared is a mistake in the plan, not a condition that was false.

A task times out when a deadline fired while a step was running: the step's own, the task's, the run's, or a signal. The reason names which: run timeout 300ms fired while it was running: deadline exceeded waiting for first byte, task timeout 200ms ..., cancelled (interrupt) ..., finally timeout 30s .... A step cut off by a deadline did not observe an answer, so it is never failed or errored.

A task is skipped when its own guard was false, when guard false: env.feature == "on", or when its tag is excluded, excluded by tag slow; run with --include-tag slow to include it. A skipped task did nothing wrong, and the run can still exit 0.

A task is blocked when an upstream task did not pass. The reason says which and why:

• blocked by wrong, which failed (or errored, timed out). The upstream's own code covers the run, so this adds nothing to the exit code.
• blocked by optional-feature, which was skipped (when guard false: env.feature == "on"). The run did not finish, so the exit code is 3. If this task is optional too, give it the same when guard so it is skipped in its own right. Nothing failed, but a task that was meant to run did not, and a green exit would hide that.
• blocked by nightly, which was excluded by tag slow; the run did not finish, so the exit code is 3. Run with --include-tag slow to run both.

A task whose own guard is false is skipped, never blocked, even when its upstream was skipped too. That is the fix the second reason suggests.

A task is not run when the run ended before it started: not run: run timeout 300ms, not run: cancelled (interrupt), or, after a reset hook that did not pass, not run: lifecycle reset step 1 failed: .... Nothing is known about it.

Exit codes

When several apply, the first in the order 64, 3, 2, 1, 4 wins: a run that did not finish outranks one that failed, because an incomplete run may hide more failures. A finally task blocked by a main task that failed adds nothing; the main task's code already says it. With --repeat, the exit code is the worst over all runs. The exports never change it: JUnit, TAP and the HTML report are written after the code is known, and a file that cannot be written raises the code to at least 2.

The HTML report opens with the code and the meaning in the second column, in those words.

One run, every state

This plan reaches six of the seven states against the fixture server, with the feature flag unset so optional-feature skips:

apiVersion: vero.plan/v1
kind: Plan
metadata: { name: states }
env:
  baseUrl: http://127.0.0.1:18080
  feature: ${FEATURE:-off}
lifecycle: { strategy: isolated }
tasks:
  - name: health
    steps:
      - http: { method: GET, url: "{{ env.baseUrl }}/healthz" }
        assert: [status == 200]
  - name: optional-feature
    when: ['env.feature == "on"']
    steps:
      - http: { method: GET, url: "{{ env.baseUrl }}/healthz" }
        assert: [status == 200]
        results: { seen: status }
  - name: uses-it
    steps:
      - http: { method: GET, url: "{{ env.baseUrl }}/fixtures/status?code={{ tasks.optional-feature.results.seen }}" }
        assert: [status == 200]
  - name: wrong
    steps:
      - http: { method: GET, url: "{{ env.baseUrl }}/fixtures/status?code=503" }
        assert: [status == 200]
        results: { x: status }
  - name: after-wrong
    steps:
      - http: { method: GET, url: "{{ env.baseUrl }}/fixtures/status?code=200", headers: { X-Prev: "{{ tasks.wrong.results.x }}" } }
        assert: [status == 200]
  - name: nobody
    steps:
      - http: { method: GET, url: "http://127.0.0.1:1/x" }
        assert: [status == 200]
  - name: slow-one
    steps:
      - http: { method: GET, url: "{{ env.baseUrl }}/fixtures/slow?ms=3000" }
        timeout: 500ms
        assert: [status == 200]
finally:
  - name: cleanup
    steps:
      - http: { method: GET, url: "{{ env.baseUrl }}/fixtures/status?code=204" }
        assert: [status == 204]

vero run prints the three failure blocks, the two blocked lines and the summary, and exits 3:

BLOCKED  uses-it  blocked by optional-feature, which was skipped (when guard false: env.feature == "on"). The run did not finish, so the exit code is 3. If this task is optional too, give it the same when guard so it is skipped in its own right
BLOCKED  after-wrong  blocked by wrong, which failed

2 passed  1 failed  1 errored  1 timed out  1 skipped  2 blocked  0 not run   (8 tasks, 5 requests, 0.5s)   teardown: 1 passed

Codes 1, 2 and 3 all apply here, and 3 wins. Give uses-it the same when guard and the run exits 2, the errored and timed-out tasks outranking the failed one. With --timeout 300ms the slow task is cut by the run instead of its own deadline; the states and the code are the same, and the timing line says deadline hit waiting for first byte either way.

In JUnit and TAP

JUnit XML can say passed, failed, errored and skipped, and most CI systems read <skipped> as green. So skipped, blocked and not_run all go out as <skipped> with a message naming the cause: the guard, blocked by <task>, or not run: run ended first (<cause>). Wrong but labelled beats wrong and silent, and the exit code does not read this file. One <testsuite> per run carries the run id, the plan file, a UTC timestamp and three properties: vero.runId, vero.summary (the summary line) and, after --only, vero.only (what was selected and left out). A finally task's classname ends in .finally. From the run above:

<testcase name="optional-feature" classname="states.tasks" time="0.000">
  <skipped message="when guard false: env.feature == &#34;on&#34;"></skipped>
</testcase>
<testcase name="uses-it" classname="states.tasks" time="0.000">
  <skipped message="blocked by optional-feature"></skipped>
</testcase>
<testcase name="wrong" classname="states.tasks" time="0.001">
  <failure type="failed" message="assert #1 failed: status == 200">wrong › step 1 › assert #1  plan.yaml:27&#xA;status == 200    failed&#xA;  status = 503&#xA;</failure>
</testcase>
<testcase name="nobody" classname="states.tasks" time="0.000">
  <error type="errored" message="step 1: Get &#34;http://127.0.0.1:1/x&#34;: dial tcp 127.0.0.1:1: connect: connection refused">...</error>
</testcase>
<testcase name="slow-one" classname="states.tasks" time="0.501">
  <error type="timed_out" message="step 1: deadline exceeded waiting for first byte">...</error>
</testcase>
<testcase name="cleanup" classname="states.finally" time="0.001"></testcase>

A passed (unverified) task is a plain pass with a <system-out> note, because JUnit has no such state. The dialect is Jenkins and Surefire's, checked against their schema in the repository's tests.

TAP version 14 has the same three labels after # SKIP, a not ok with a YAML block for each failure, and finally <task> as the name of a teardown task:

TAP version 14
1..8
ok 1 - health
ok 2 - optional-feature # SKIP when guard false: env.feature == "on"
ok 3 - uses-it # SKIP blocked by optional-feature
not ok 4 - wrong
  ---
  message: "assert #1 failed: status == 200"
  severity: failed
  detail: |
    wrong › step 1 › assert #1  plan.yaml:27
    status == 200    failed
      status = 503
  ...
ok 5 - after-wrong # SKIP blocked by wrong
not ok 6 - nobody
  ---
  message: "step 1: Get \"http://127.0.0.1:1/x\": dial tcp 127.0.0.1:1: connect: connection refused"
  severity: errored
  detail: |
    step 1: Get "http://127.0.0.1:1/x": dial tcp 127.0.0.1:1: connect: connection refused
  ...
not ok 7 - slow-one
  ---
  message: "step 1: deadline exceeded waiting for first byte"
  severity: timed_out
  detail: |
    step 1: deadline exceeded waiting for first byte
  ...
ok 8 - finally cleanup

A # SKIP is an ok in TAP. The label is what keeps it from reading as a pass, and the exit code, not the file, decides the build. With --repeat, each run becomes a subtest, # Subtest: run <id>, and the outer ok or not ok follows that run's exit code.