Part 7: Shutdown & Signals

WaterWall has exactly one place where process teardown runs: worker 0, the main thread, exactly once. Registered shutdown callbacks, worker joining, node teardown and global-state destruction all happen there.

That constraint is not stylistic. The global-state shutdown callback stops and joins the other workers, so running it from one of those workers can join the current thread or wait for a lock the current thread holds. Running it from a POSIX signal handler or a Windows console-handler thread is worse: those contexts may interrupt code that owns arbitrary locks.

At the same time, an orderly shutdown requested by another thread must not be converted into an immediate _Exit() that skips cleanup. Route, DNS and iptables restoration, node onStop hooks and device pre-down scripts all live in the registered cleanup path.

1. The three APIs

All three are declared in wlibc.h, so every translation unit sees them.

bool           requestProgramShutdown(int exit_code);
_Noreturn void abortProgramNow(int exit_code);
_Noreturn void terminateProgram(int exit_code);   // legacy compatibility

requestProgramShutdown(code) — orderly, thread-safe, returns

Records the exit status, schedules the shutdown sequence on worker 0, and returns to its caller. It never waits for teardown, which is exactly what lets worker 0 join the requesting thread afterwards.

The caller must unwind normally immediately after. Because the function returns, every migrated call site needs an explicit return, goto cleanup, or error propagation — the old _Noreturn behaviour used to hide missing control flow.

It returns true when this call queued the first request or when a shutdown was already in flight and the request coalesced into it. It returns false only when the worker-0 handoff is unavailable and no shutdown was accepted. Handle false by failing closed:

if (! requestProgramShutdown(0))
{
    abortProgramNow(1);
}
return;

abortProgramNow(code) — immediate _Exit(), no cleanup

For corrupted state, broken invariants, and failures reached while arbitrary locks may still be held. It arbitrates the exit code through the same policy and then leaves without touching the registered cleanup path.

terminateProgram(code) — legacy compatibility

Orderly shutdown only on the main thread. Off the main thread it is still a hard abort that skips registered cleanup, and it now logs the offending TID/WID saying so (and asserts in debug builds). Treat any off-main terminateProgram() as an unmigrated call site.

2. Shutdown phase state machine

RUNNING ──requestProgramShutdown()/signal/console──▶ REQUESTED
REQUESTED ──worker 0, exactly once──▶ STOPPING
STOPPING ──workers stopped and joined──▶ FINALIZING ──▶ exit()

The phase (program_shutdown_phase_e) is the single source of truth. GSTATE.application_stopping_flag is a published compatibility flag derived from it:

• it stays false while shutdown is merely REQUESTED, so the control wakeup can still be delivered and the requester can unwind;
• worker 0 sets it true when it enters STOPPING.

Only worker 0 performs REQUESTED -> STOPPING, and only the pass that wins that transition runs cleanup.

3. Exit-code arbitration

Deterministic, independent of thread scheduling:

• the initial exit code is zero;
• the first non-zero code wins;
• a later zero can never overwrite a recorded error;
• a later non-zero can never replace an earlier non-zero;
• a stop signal records 128 + signum under the same rule;
• a second SIGINT/SIGTERM (or console stop event) forces an immediate _Exit(128 + signum) as the operator escape hatch;
• duplicate programmatic requests coalesce and never force an immediate exit.

signalmanagerSetExitCode() is the arbitration entry point; it is not a last-writer-wins store.

4. Shutdown-control transport

Control traffic must survive after normal application work is refused.

• POSIX: a dedicated self-pipe. The async handler writes the signal number; requestProgramShutdown() writes a 0 request marker. Worker 0 watches the read end at high priority, drains the whole batch, arbitrates the exit code, and then runs the shutdown. The handler itself touches only volatile sig_atomic_t statics — never the manager object, which can be freed during late teardown.
• Windows: wloopPostControlEvent(), which is wloopPostEvent() without the isApplicationTerminating() gate. Both copy the event, so a stack-allocated wevent_t is fine.

wloopPostEvent() keeps rejecting ordinary application work during shutdown. Never route shutdown control through it.

5. Worker stop and join

bool workerRequestStop(worker_t *worker);        // thread-safe, never waits
void workerDestroyOwnResources(worker_t *worker); // owner thread only, once
bool workerJoin(worker_t *worker);

Rules:

• worker 0 only requests a remote stop, then joins;
• a worker thread destroys its own event-loop-local resources after its loop returns;
• worker 0 destroys its own resources on worker 0;
• pseudo-workers (no event loop — currently lwIP) are identified by worker->has_event_loop, not by a null loop, because a normal worker clears its loop during teardown;
• every spawned worker is asked to stop before the first join, so shutdown does not serialize on one slow worker.
• a failed join leaves thread_valid set and prevents worker-resource/global destruction. Terminal shutdown hard-aborts at that point because a possibly-live worker can still access those resources.

worker_t::lifecycle is an atomic, monotonically advancing worker_lifecycle_e (Initialized → Running → StopRequested → Exited → Joined). A per-worker control_mutex serializes a remote stop request against the owner detaching its own loop, so a stop racing a worker's self-exit is safe in both orderings and nothing is destroyed twice.

Under the hood, wloopRequestStop() sets an atomic stop_requested on the loop with release ordering and wakes the poller through its eventfd/pipe/socketpair. It enqueues nothing and allocates nothing. The loop checks the flag with acquire ordering before each iteration, so a request issued before wloopRun() started is honored too. There are no cross-thread writes to wloop_t::flags.

6. Device stop, join, and destruction

Device Stop and Destroy have different ownership jobs:

• Stop unpublishes new work, wakes blocked I/O, verifies every owned thread join, and only then drains thread-owned queues;
• Destroy runs later, after external producers are quiescent, and frees retained process-visible resources.

The global order is deliberately:

nodemanagerStop()
  -> request/stop/join normal workers
  -> drain queued lwIP work
  -> release protocol state and stop/join tcpip_thread
  -> destroy pseudo-worker and worker-0 resources
  -> nodemanagerDestroy()

tundeviceWrite() and rawdeviceWrite() can run on event-loop workers and the lwIP path. Device destruction therefore cannot reclaim a writer queue merely because its own writer thread joined; it waits until the global order above has stopped every producer context.

The lwIP pseudo-worker owns WaterWall pools, but tcpip_init() also creates a real OS thread. Node Stop first detaches PacketsToConnection netifs and listeners. Global shutdown next joins every ordinary worker, because those workers can still enter the lwIP core lock from tunnel callbacks. It then posts a cooperative callback behind already queued lwIP work. That callback releases active/bound/listening TCP and UDP state, including retained out-of-order pbufs, and stops tcpip_thread; the shutdown owner joins it before destroying the lwIP pseudo-worker or any global resource. A custom pbuf records its originating worker and pool; after ordinary workers have stopped, a foreign-thread release destroys the buffer rather than inserting it into the originating thread's pool.

Reader delivery ownership

A device_reader_session_t owns one packed lifetime gate. CLOSED and the successful-entry count share one atomic modification order: an Enter CAS before Close contributes to the count and Close waits for Leave; a Close first makes the Enter reject. Open-release publishes installed session fields to Enter-acquire, and Leave-release publishes completed callback work to Close's acquire observation of zero.

Close-and-quiesce is a lifecycle-owner operation. Calling it from inside the same guarded callback is a programming error, because the owner would wait for itself. There is no release-build "skip the wait" path.

Each queued reader message retains a session reference until delivery or queue cleanup. Delivery enters the gate, compares a relaxed generation tag, calls the device only for the current generation, leaves the gate, recycles the message, and finally releases its reference. Generation zero is invalid and generation wrap is refused, so a pre-restart message cannot become current again. The old reader producer thread must be joined before a new generation opens: gate quiescence covers callbacks, not a producer still able to post.

Writer generations

Writer submission intentionally has no gate or per-send refcount. A producer performs one acquire load of an immutable published generation and calls chanTrySend() on that generation's channel. Stop exchanges publication to null, closes the selected channel, joins the writer, drains leftovers, and retires the closed generation.

A producer that loaded the old pointer before Stop may finish its send attempt against the allocated, now-closed channel; it can never cross into a replacement queue. A call paused before the pointer load may select a later generation. Retired generations remain allocated until device destruction, after all workers and pseudo-workers have stopped. This trades one queue allocation per restart (up to 128K pointer slots for Linux TUN/Raw) for a writer hot path with no lifetime RMW, TLS scan, hazard pointer, or quiescence wait. Repeated runtime restart is therefore safe but intentionally not memory-neutral.

Every device join follows the same failure rule: a failed join retains the thread handle/joinable flag and all resources the thread may access. It blocks drain, retirement, restart, and destruction until a later join succeeds. After a successful join, a restartable device explicitly releases the debug thread ownership of its reader/writer buffer pool. The next thread generation then claims the pool on first access; resetting ownership while the old thread can still run is forbidden.

Linux Capture classifies reader exit under reader_state_mutex. Stop sets reader_stop_requested while holding that mutex before clearing the relaxed loop flag or waking the reader. The exiting wrapper takes the same mutex: whichever side acquires it first decides expected versus unexpected exit. Neither an acquire load nor any other atomic load is a "fresh value" primitive.

TUN keeps one relaxed lifecycle atomic for STARTING, UP, STOPPING, and FAILED arbitration. It orders only that value. Thread creation/join, the reader gate, and writer-generation publication provide the ordering for companion fields.

7. Worker-originated shutdown, step by step

1. release every buffer the callback owns;
2. unlock every lock it holds;
3. call requestProgramShutdown();
4. return immediately through normal control flow;
5. the worker loop later observes its stop request;
6. the worker destroys its own resources and returns from its thread routine;
7. worker 0 joins it.

The canonical shape is:

if (! requestProgramShutdown(1))
{
    abortProgramNow(1);
}
return;

abortProgramNow() is the handoff-failure fallback only. Reaching for it first turns an ordinary resource failure into a teardown that skips every registered cleanup callback.

Cleanup before the request, and nothing after it

Steps 1 and 2 are not advice, they are the precondition. Never call requestProgramShutdown() while holding a tunnel mutex, a read/write lock, a line lock, or any transient resource this callback must release itself — worker 0 runs the real teardown and cannot take a lock the failing worker still owns.

Recycle only what the current callback still owns. A buffer already pushed into a line-state queue or stream stays reachable and is released by normal line/tunnel teardown; recycling it again is a double free.

After an accepted request the callback is done. Do not create a line, send a packet, arm another timer, mutate completion state, or call a later startup step. When a helper reports "shutdown requested" to its caller, that caller must return too — a timer failure that a caller reads as success leaves the callback running as though a timer were armed. packetsenderWaitUntilDeadline() shows the shape: it returns a three-valued result so ready, waiting and shutdown-requested cannot collapse into one boolean.

Requesting a shutdown is not closing a line

requestProgramShutdown() schedules global teardown on worker 0. It does not make any current line logically dead, and it does not run before this callback returns: the worker callback unwinds normally, through every suspended and re-entrant frame between it and the event loop. Those frames keep observing lineIsAlive(line).

So a shutdown request never substitutes for the owner Finish postcondition in Part 2. When a Finish handler for a normal line the tunnel created reaches a terminal failure, the order is:

1. record the diagnostic and the failure verdict;
2. detach the line's producers (io callbacks, timers, idle items, maps);
3. destroy this tunnel's line state;
4. lineDestroy() the line, so it is dead before the handler returns;
5. requestProgramShutdown(code), with the abortProgramNow(code) fallback if the worker-0 handoff is refused;
6. return.

Steps 2–4 come before step 5 for the same reason as every other cleanup here: this callback is the last frame that still owns that state. TesterClient's incomplete-response branch (testerclientFailOwnedLine() in tunnels/TesterClient/common/helpers.c) is the worked example, and tests/unittests/testerclient_orderly_shutdown_test.c holds an outer reference across the callback to prove the line is dead when it returns.

The exception is the same one as everywhere else: a persistent worker packet line is never destroyed at runtime, so a terminal failure on one hard-aborts instead (Category D).

onStart schedules work; it does not run it

Do not infer execution context from a filename. instance/start.c contains both kinds of code:

• what onStart itself runs is a main-thread startup context (Category E);
• what onStart enqueues with a worker-message API is a runtime worker callback (Category B), even though it is defined in the same file.

By the time such a task runs, the workers are up and the shutdown handoff exists, so a failure inside it must request an orderly shutdown, not terminateProgram(). authenticationclientStartPingTimer(), keepaliveclientStartWorkerTimer() and packetstostreamStartWorkerTimer() are all in instance/start.c and all Category B.

A worker that fails this way must not clean up on behalf of anyone else. Timers other workers already published stay in their slots; each worker deletes its own slot in onWorkerStop during the orderly shutdown.

8. Classifying a termination call site

Do not classify by log level or by whether the exit code is non-zero. For each site, record which threads can reach it, what it owns, and whether the containing function can return.

Category	Situation	Action
A	Expected successful completion (test drivers, report writers)	requestProgramShutdown(0) + return, after the success marker is committed
B	Orderly runtime failure with structurally valid local state (worker loop error, unrecoverable device I/O)	mark the component failed, release buffers/handles, requestProgramShutdown(code), exit through the normal cleanup path
C	Per-line / per-connection failure	usually close only the affected line; do not broaden into a tunnel behaviour change without a focused test showing process termination is wrong there
D	Invariant violation or unsafe state (suspected corruption, impossible callback, a lock intentionally still held, unknown state validity)	abortProgramNow() with an explicit diagnostic
E	Main-thread startup/configuration failure	terminateProgram() is fine; before the signal manager exists, plain exit() after unwinding what was initialized

When migrating a _Noreturn site, re-check: functions that omitted a return because the call could not return; buffers already recycled before the call; device lifetime gates whose comments assume the callback cannot resume; callbacks that may destroy a line re-entrantly; non-void return values; and worker/device routines that must publish an exited/failed state.

Reaching Category B from a device I/O routine

A device thread only reaches Category B by returning. Its wrapper (tundeviceNoteUnexpectedThreadExit()) is what publishes FAILED and requests the shutdown, so an error the routine swallows with continue never gets there: the device stays advertised as usable while every packet is discarded. A process that exits can be restarted by a supervisor; one that spins cannot be noticed.

So POSIX device I/O errors are classified against a whitelist (ww/devices/tun/tun_io_error.h), never by guessing:

Class	Errors	Action
Transient	EINTR, EAGAIN/EWOULDBLOCK, ENOBUFS, ENOMEM	retry / go back to the poll loop
Packet-local (writes only)	EINVAL	drop that packet, keep the device
Terminal	everything else, including EIO, EBADF, ENODEV, ENXIO, EMSGSIZE	return from the routine

Two rules that are easy to get backwards:

• EINVAL on write must stay packet-local. The kernel rejects a malformed frame that way, and frames reaching a TUN writer come from remote peers. Making it terminal hands any peer a one-packet process kill.
• EMSGSIZE on write is terminal. Oversized packets are already dropped before the syscall, so EMSGSIZE means the configured MTU exceeds the device's real MTU — an operator misconfiguration that never recovers.

Capture errno (or GetLastError()) into a local immediately after the failing call. Buffer recycling and the loggers both run before the classification and either can overwrite it, which silently reclassifies a device loss as a transient.

A failed test verdict is not an impossible internal state

TesterClient and TesterServer reach termination for two unrelated reasons, and the distinction decides the category:

• A valid runtime failure verdict — the peer sent an early, extra, duplicate, malformed-size, mismatching, trailing or overflowing payload, finished before verification completed, or the watchdog expired. The test driver reached a real result, so this is Category B: recycle the buffer the callback still owns, then go through the one verdict helper (testerclientFail() / testerserverFail()) and return.

  One extra step when the verdict is reached inside a Finish handler for a normal line the tester owns: the verdict helper only reports, it does not close anything, so that branch must go through testerclientFailOwnedLine() instead. It detaches the worker slot, destroys the line state, marks the line dead, and only then reports the verdict — the owner postcondition and the Category-B shape at once.

• An impossible internal state — a payload generator asked for a payload its own validated configuration can never produce, a buffer pool reporting zero writable capacity, a close scheduled before verification, a response counter disagreeing with its own mask. Nothing a peer sends can produce these, so they are Category D: log and abortProgramNow(1) directly, never through the verdict helper.

Keeping those helpers to a single meaning is what makes both readable, so a generator whose every failure is an invariant becomes non-nullable: its callers use the returned buffer without a NULL check.

Packet-line death stays a hard abort

A worker packet line is process-lifetime state: it is allocated in tunnelchainFinalize() and destroyed only in tunnelchainDestroy(). If one dies at runtime, that contract has already been violated, and orderly teardown can no longer assume line state is structurally valid. Those sites hard-abort even though they run on a worker — initializeLineOnTargetWorker(), wireguarddeviceQueueWorkerPacketInit(), packetsenderStartWorker(), packetsenderSendReadyPackets(), testerserverResponseSendTask(), and any Finish received on a packet line.

The opposite case is an ordinary line being rejected. A neighbouring connector refusing one transport line is operational, not process-wide: wireguarddeviceQueueWorkerTransportLineInit() logs a warning and returns, leaves the slot NULL, and lets wireguarddeviceEnsureTransportLine() retry on a later output. It calls none of the three process APIs.

WireGuardDevice shows both halves in one handler, which is why it has to check the exact role first: wireguarddeviceHandleTransportLineFinish() treats a Finish on one of its own per-worker transport lines as an ordinary owned-line close (clear the slot, lineDestroy()), and a Finish on the chain's worker packet line as the Category-D violation above.

9. Shutdown callback registry

• registration and removal are rejected once the list is frozen for the traversal;
• the capacity check happens inside the registry mutex;
• worker 0 freezes and snapshots the list, then runs each callback exactly once in LIFO order;
• the registry mutex is never held while a callback runs;
• a callback influences the result through signalmanagerSetExitCode(); it must not try to continue the sequence by re-entering the shutdown API.

The global-state callback is registered first, so LIFO runs it last — after every other subsystem callback has had the workers available.

10. Signal-mask policy

Only the main thread has the handled stop signals unblocked. Every application-owned thread keeps them blocked. This is enforced in threadCreate(), which blocks them around every spawn so the child inherits the mask with no delivery window — including threads created after signalmanagerStart() (device, lwIP, TUN/raw/capture threads).

Fatal crash signals (SIGSEGV, SIGILL, SIGABRT, …) restore the default disposition and re-raise. They never run ordinary cleanup.

11. Tests

ctest --preset linux --output-on-failure -R 'shutdown|signal_manager|wloop_stop|node_manager_stop|thread_transfer'

• shutdown_manager_test — off-main orderly requests, once-only cleanup on the main thread, exit-code arbitration, request coalescing, requester unwinding (probe mutex), and handoff failure.
• shutdown_signal_test — SIGTERM idle and against a busy worker, SIGINT status, a signal batched behind a programmatic request, the second-signal escape hatch, and proof that callbacks do not run in the async handler's context.
• wloop_stop_request_test — stop wakes a blocked loop, survives being requested before the loop runs, is idempotent, and is safe racing the loop's own exit.
• node_manager_stop_once_test — nodemanagerStop() idempotence and onStop running exactly once.
• lwip_shutdown_test — retained TCP out-of-order pbuf release plus cooperative tcpip_thread join.
• buffer_pool_thread_transfer_test — post-join pool ownership transfer between two simultaneously allocated, distinct thread identities.

Every successful shutdown path exits the process, so these tests fork a child per scenario and report back over an inherited pipe.

Call-site policy tests

python3 tests/tunnels_abort_policy_test.py --mutation-test
python3 tests/tunnels_orderly_shutdown_policy_test.py --mutation-test
python3 tests/line_ownership_policy_test.py --mutation-test
ctest --preset linux --output-on-failure -R 'orderly_shutdown|abort_runtime|owned_line|line_ownership'

• tunnels_abort_policy_test.py — every audited Category-D site still hard-aborts with exit code 1, and the Category-C/E exclusions never drift into one.
• tunnels_orderly_shutdown_policy_test.py — every Category-B site keeps its request, its fallback, its pre-request cleanup anchors and its caller's immediate return; tester invariants keep hard-aborting; the retryable WireGuard path calls none of the three APIs; main-thread startup sites stay Category E.
• line_ownership_policy_test.py — every production lineCreate() site stays classified, no borrowing tunnel starts calling lineDestroy(), every registered owner close path still destroys its line, and every registered packet-lifecycle anchor still hard-aborts instead of destroying the packet line. It also sweeps every tunnels/*/{upstream,downstream}/fin.c and rejects any handler whose whole body is discard statements unless it is registered in SILENT_FINISH_ALLOWED with a rationale, so an intentional terminal absorber is distinguishable from a lost propagation or owner close. Transparent middle handlers remain ordinary same-direction forwarding code. This is where the "a shutdown request is not line destruction" rule is pinned in source: it requires the runtime tests below to keep both their content and their add_test registration, because a test file nothing builds proves nothing.

Both scan function bodies with a lexical C scanner, so a commented-out or quoted lookalike cannot satisfy a candidate, and both run a mutation mode that removes each load-bearing element in turn and requires the checker to catch it.

The runtime counterparts inject the real failure. *_orderly_shutdown_test linker-wraps wtimerAdd and the three process APIs, then asserts the whole contract: exactly one request with exit code 1, no abort, no terminate, every transient buffer recycled exactly once, nothing forwarded, no state published, and the callback returning.

The failure paths that leave the process run in a forked child, and the child's exit code says which contract the abort satisfied, so reaching the right function through the wrong path still fails:

• a Category-B fallback must follow exactly one refused requestProgramShutdown(1);
• a Category-D invariant must abort with no request at all.

Tasks that onStart only enqueues are driven the way the runtime drives them — through the public onStart entry point and a real worker message queue — so per-worker ownership and onWorkerStop cleanup are exercised rather than assumed.