Part 7: Application Termination

WaterWall has one orderly process-termination authority. The application shutdown controller records requests and exit status; the main thread performs the ownership-based shutdown sequence after worker 0 leaves normal event-loop dispatch. Workers, loops, tunnels, devices, queues, and protocol code respond to explicit local lifecycle operations from their owners. They do not poll the process controller.

Fatal termination is separate. abortProgramNow() deliberately skips orderly cleanup and is reserved for corruption, invariant failure, a failed orderly handoff whose state cannot safely continue, or heap allocation failure. The memory manager invokes this fatal path directly on OOM, without normal logging, cleanup callbacks or an orderly-shutdown request.

A critical runtime invariant failure must emit LOGF and call abortProgramNow(); returning an ordinary failure or requesting asynchronous shutdown does not make unsafe continuation valid. Caller-only preconditions normally use Debug assertions instead; see caller preconditions and failure handling.

1. Process shutdown controller

The controller advances monotonically through:

Running
  -> Requested
  -> ClosingAdmissions
  -> QuiescingWorkers
  -> QuiescingExternalProducers
  -> DrainingWorkers
  -> StoppingComponents
  -> DestroyingWorkers
  -> Finalizing

requestProgramShutdown(code) is the public orderly-request facade. The first request records three independent facts: diagnostic origin, cleanup scope, and exit status. Cleanup scope comes from the runtime-commit milestone, never from the origin: every request accepted before commit selects StartupRollback, and every request accepted afterward selects ProcessShutdown. A signal during startup therefore remains a signal while still receiving rollback cleanup.

The first request is one controller-to-worker transaction. While holding the controller mutex, it selects the scope and installs worker 0's level-triggered quiesce request, including normal/message admission closure. Only then does it publish Requested. If worker 0 cannot accept the local request, the controller remains Running. Once accepted, a degraded wake does not roll the request back. The lock order is controller mutex followed by worker control mutex; worker failure paths release their worker lock before reporting globally.

After all startup stages and required task admissions succeed, but before workers_run_flag is published, the main startup owner calls the runtime-commit operation. Commit and request acceptance serialize on the controller mutex, so a request either wins and causes rollback or loses to a committed runtime. Repeated requests coalesce. A later first non-zero status may raise a successful status until origin, scope, and status are frozen together on entry to Finalizing.

Requested promises only that the request record exists and the main execution controller must enter shutdown. It does not mean that admission has closed, a callback has returned, a line is dead, or an object is safe to reclaim.

Only SignalManager's OS handoff and the main coordinator inspect the process-wide phase. There is no application_stopping_flag, leaf isApplicationTerminating() check, or secondary component-visible shutdown state.

2. Request handoff and worker 0

On POSIX, the async signal handler stores the first signal in a volatile sig_atomic_t mailbox and writes a best-effort byte to a nonblocking self-pipe. A full pipe already represents a pending wake. The handler takes no application lock and performs no cleanup. Worker 0 consumes the mailbox and publishes the typed controller request. Mailbox take-and-clear blocks the exact graceful signal set, publishes a captured signal before unblocking, and restores the caller's exact previous mask. The startup-failure boundary performs mailbox take, signal publication, and StartupFailure publication while that same set remains blocked. A signal already captured when the mask is acquired wins the immutable origin; one delivered only after acquisition is ordered after the boundary and cannot replace an accepted startup failure. The main startup path also consumes the mailbox at stage checkpoints.

On Windows, a process-lifetime atomic entry gate combines handler admission and the active entrant count. A handler acquires it before reading manager or controller storage. Destruction closes the gate, unregisters the OS handler, releases waiters, and waits only for admitted entrants. A late entrant uses the manager-free fatal-status fallback. Console close/logoff/shutdown handling may wait a bounded time where the OS contract requires it.

The callback does not destroy its event loop, workers, nodes, or global state. wloopRun() observes the local request between callback roots and returns a typed result. runMainThread() then calls the coordinator outside the event dispatch stack.

A second operator signal may use SignalManager's immediate-exit escape hatch. abortProgramNow() performs only lock-free status arbitration and immediate _Exit/ExitProcess; it emits no diagnostic that could block on a full stream. The fatal-status integer is compile-time required to be always lock-free on the C11 path. The Windows fallback uses the corresponding lock-free Interlocked primitive. Its operations remain relaxed because the integer publishes no other state. On POSIX, fatal crash signals restore their default disposition, unblock and re-raise the signal, with _Exit(128 + signum) only as a fallback. Windows fatal handlers terminate with the arbitrated status because POSIX signal/core semantics do not apply there.

3. Worker lifecycle

Every ordinary event worker follows:

Initialized
  -> Running
  -> QuiesceRequested
  -> Quiesced
  -> DrainRequested
  -> Drained
  -> TeardownRequested
  -> Exited
  -> Joined

The main coordinator publishes requests. The owner worker publishes acknowledgements only after the represented work is complete. A worker mutex and condition variable carry drain and teardown releases after normal event-loop dispatch has ended; lifecycle commands are not smuggled through a normal event queue.

Worker 0 uses the same semantics but executes its quiesce, drain, and teardown stages directly on the main thread. lwIP engines belong to ordinary workers; there is no extra stack worker or thread. Unused workers need no engine.

All registered workers follow this lifecycle. Their identity remains valid after loop/message-queue detachment until they unbind. Admission closes before detachment, independently of worker ownership. Teardown must use its saved loop reference where required and must not infer a non-worker thread from loop == NULL.

WIO stores and closes its own descriptor directly. Loop cleanup settles these descriptors and all pending/native roots before destroying the existing WIO pools. Splice buffers permanently own independent pipe pairs. Complete buffers move between the shared master, bounded worker caches and active owners; foreign destruction returns healthy entries directly to the master; they hold no reference to a source WIO or its descriptor. The shared pipe inventory is owned through GSTATE.splice_inventory, with its implementation private to bufio. Its control object is published before imports share global state; all chains finalize padding and inventory limits before demand can lazily allocate complete buffers. Healthy entries remain allocated until teardown. It survives all worker and master buffer pools and closes only after all active buffers and worker/device caches have returned during final global teardown. Global-state copies borrow this same object; the owner frees it and clears its pointer only after all users have settled.

A quiesced worker:

• admits no new normal asynchronous work;
• is running no normal callback root and will begin no later one;
• has settled accepted queued and timed worker messages;
• has detached component timers and watchers for its WID;
• has returned from normal loop dispatch;
• still owns the loop, queue, tunnel, line, and pool resources required for drain and teardown.

4. Normal callback boundaries

A normal callback root is one scheduler or external-producer entry into ordinary application behavior: an I/O callback, timer, idle, custom event, queued worker message, device delivery, or lwIP delivery. Synchronous nested tunnel callbacks belong to the admitted root for that exact wloop_t; a callback on loop A has no inherited authority on loop B. Direct writes follow the same rule on NIO and IOCP: an independent owner-thread caller acquires target-loop admission across the system send, interest registration, queue insertion, and asynchronous-operation publication. A synchronous write callback is a target-loop callback root: it runs with the target loop as its only normal authority and restores any outer source-loop authority on return. Admission is released before invoking user code, so closed admission suppresses any later root and the backend settles the transferred buffer or queued operation exactly once.

NIO direct writes delegate callback admission to wloopInvokeWriteCallback(); the caller needs no separate dispatch precheck before that helper. The guard suppresses an independent callback if admission closes after its send completes, while preserving synchronous nested callbacks within an admitted root. Post-I/O delivery checks and checks between callbacks remain necessary: an active event handler can contain more than one callback or queue entry.

WIO access, writes, close and associated callbacks belong to the owning event worker during normal operation. An independent write is a call on that worker outside an already-admitted normal callback; it is not a foreign-thread call. Serialization and target-loop admission do not grant access to another worker's WIO. Foreign producers submit through the adapter's worker-message interface; the owner resolves and uses its WIO. Canceled messages settle detached buffers without accessing a foreign WIO or pool. wioClose() does not dispatch foreign callers. wioCloseAsync() defers close through the owner loop's cleanup-control queue; its name and queue do not confer foreign WIO access. Exclusive startup, explicit detach/attach and teardown retain their existing contracts. Zero-length TCP writes and existing stream-queue branches cross the same target loop gate. Rejection sends no bytes, does not close the WIO as a side effect, and consumes the transferred buffer once. UDP zero-length datagrams remain valid datagrams.

The event loop checks its local quiescence request:

1. before polling;
2. immediately after polling;
3. before each timer, idle, ready I/O, custom event, completion, and worker message callback root;
4. between entries in captured batches and queue drains.

If a request arrives during a callback, that callback returns normally. The loop does not begin another normal root. A level-triggered request plus a poller wake provides progress even when no ordinary event can be admitted.

Raw wevent_t custom posting is private to event-loop and worker plumbing and may carry no application-owned payload. Normal posts are rejected once admission closes. Posting stores a local copy whose loop metadata is always normalized to the destination; caller storage cannot spoof callback authority. A narrower control queue admits required WIO cleanup until worker quiescence drains it, then closes its own admission before Quiesced is published. Control callbacks run with no inherited normal authority, so normal work they intentionally start must acquire fresh admission. Arbitrary ownership-bearing application tasks must use an API with an explicit run-or-cancel contract.

5. Run-or-cancel ownership

An accepted worker message executes exactly one of its callback or cleanup paths. Queue admission and closure linearize under the worker lifetime lock and queue lock. Pending queued and timed messages are canceled on their owner worker before it acknowledges quiescence; queue destruction never silently drops them.

Posting APIs distinguish the two ownership styles used by the runtime:

• cleanup-on-refusal APIs reject and execute the registered cleanup immediately;
• retain-on-refusal APIs return ownership to the caller when the message was not accepted;
• after acceptance, the task system owns the arguments until either callback or cleanup settles them.

Delayed submission makes the same distinction during timer setup. A retain-on-refusal call rejected by closed timer admission returns its payload to the caller without invoking cleanup. An already accepted setup wrapper instead settles through cleanup if admission closes before it can arm.

Timed-message and line-task records share one fixed allocation geometry and GSTATE.masterpool_messages. Checkout and return use the current event worker's message_pool, sized from RAM_PROFILE, with ordinary generic-pool batch refill and shrink. Unregistered callers and callers whose local pool has been destroyed use the shared master directly. A record carries no allocating-worker pool pointer: it may return to a different worker's cache or settle after its source worker exits. Message caches survive queued-message, engine and loop cleanup; the master survives all workers and external message producers. Their generic pool constructor preserves the master's fixed, userdata-independent allocation callbacks so direct foreign checkout remains valid. Ordinary queued messages still live by value in the target deque.

Each worker's ordinary message deque initially allocates 2,048 slots for L1/L2 RAM profiles (numeric settings 5/6, including the server alias), and 512 slots for smaller profiles. One ring slot remains empty, giving 2,047 or 511 usable entries. Initialization reserves one fewer entry than the intended slot count because STC rounds capacity + 1 upward. This reserve is independent of the 256-message drain batch; increasing initial storage does not enlarge callback batches. The deque can grow with backlog and retains its capacity during normal message delivery. workerMessagesInit() has no recoverable allocation result; valid initial storage follows the fatal heap-allocation policy.

Timed worker messages use wtimerAdd() and normal fail-fast allocation. Owner workers acquire timeout records from their existing generic pools backed by GSTATE.masterpool_timers, sized from RAM_PROFILE as described in Part 2. No recoverable allocator or pool interface is involved. Explicit pool ownership routes expiry, explicit deletion, quiesced/deferred cleanup, pending-event release, and loop destruction back to the pool only when no pending callback retains the allocation. Loop cleanup finishes before local timer pools are destroyed, and the master remains alive until all workers are destroyed. Timer allocation failure terminates the process; closed admission still reports AdmissionClosed.

Checked submissions return worker_message_submit_result_e: accepted, rejected with caller ownership retained, or rejected after cleanup ran. Cleanup receives a worker_message_cancel_reason_e that distinguishes unavailable targets, closed admission, enqueue failure, quiescence, and teardown. Intentional quiescence and teardown cancellation are normal ownership settlement, not new process failures.

Same-worker inline execution remains valid after a concurrent admission close only when it is a synchronous nested operation inside an already-admitted normal callback root. Shutdown cleanup cannot regain normal admission merely because its TLS WID matches the target.

Cleanup releases ownership. It must not restart application work, call an unbounded neighbor graph, or request another shutdown merely because intentional quiescence canceled the task.

Public line-task settlement

The public lineSchedule*Task*() family builds on this run-or-cancel machinery. Its cleanup maps worker-message reasons to line_task_cancel_reason_e, adds LineDead when dispatch finds a physically retained but logically dead line, and releases its scheduling record before invoking re-entrant user code. The scheduler keeps the line reference through notification, then settles any buffer and finally releases the line.

RejectedSettled means synchronous notification and internal cleanup are complete. AcceptedAsync only transfers ownership to an asynchronous path. In particular, foreign delayed submission has two independently progressing lanes after setup admission:

submitter lane -> setup admission commits result -> return/observe AcceptedAsync
owner lane     -> accepted setup may be canceled -> Quiesced cancellation
                 cancellation may precede, race with, or follow submitter observation

Caller state and pending latches must therefore be published before submission; observing AcceptedAsync is not a synchronization point with cancellation.

Pending setup can instead be canceled with Quiesced; an armed timer can be canceled before expiry; detached queue cleanup reports Teardown. These are terminal settlement, not permission to rearm, restart a retry loop, or request shutdown again. Required progress must remain represented by an authoritative owner inventory or Stop path. Passing a null cancellation callback changes only notification, never record, buffer, or line-reference cleanup.

6. Lifecycle stages and external-node ABI

Every tunnel has a total, context-bearing lifecycle callback table. Process shutdown invokes these exact stages for every tunnel. Within a chain, NodeManager currently uses the chain's storage order; that order is not a topological source-to-tail guarantee.

Stage	Context	Contract
onQuiesceRequest	main thread	Close local source, producer, rearm, and component admission and issue stop wakes. It is bounded and nonblocking.
onWorkerQuiesce	owner worker	Detach this WID's component timers, watchers, async handles, and queued handles while loop and tunnel state remain alive.
onQuiesceWait	main thread	After all workers are quiesced, join or detach external callback roots and prove local gates empty. Preserve resources required by line drain.
onWorkerStop	owner worker	Drain authoritative source inventories and internally owned normal lines. Never invent an inventory for borrowed lines.
onStop	main thread	After all workers are drained, close final writers, sockets, PCBs, netifs, and shared state. Touch no worker-affine timer or watcher.
onDestroy	main thread	Reclaim the instance after callbacks, messages, lines, external threads, and worker references are gone.

All callbacks receive a read-only ww_lifecycle_context_t. Supported scopes are process shutdown, startup rollback, owner-driven private-child stop, and local device restart. The context also specifies graceful or abortive close policy. Components choose behavior from this context, not from ambient process state.

Live removal of a running configuration is not supported by this lifecycle. It would require its own per-configuration execution lease and publication/removal contract.

Private child tunnels are driven through the owned-child lifecycle helpers. A child inserted into a normal NodeManager chain is not driven twice by its parent; the central traversal already owns it.

External nodes

Lifecycle v2 intentionally changed the external tunnel_t layout and callback signatures. The complete external-node ABI is now version 7, covering packet-local sbuf checksum requests and removal of the line checksum field/accessors, public line-task signatures, removal of generic buffer ownership callbacks, and the by-value socket_filter_option_t layout without shared_balance_table, the three-master buffer-pool constructor and the complete-buffer splice inventory APIs. A dynamic node library must export the historical descriptor symbol:

uint32_t waterwallNodeLifecycleAbiVersion(void)
{
    return WW_EXTERNAL_NODE_ABI_VERSION;
}

The loader validates this symbol before resolving or invoking a node getter. Libraries with a missing or otherwise mismatched descriptor are rejected. WW_EXTERNAL_NODE_LIFECYCLE_ABI_VERSION remains an alias for source compatibility, while lifecycle callbacks retain their v2 semantics. WaterWall never calls an old callback or line-task prototype through a new function boundary and does not normalize an incompatible layout.

7. Ordered shutdown sequence

The main coordinator performs:

1. call every onQuiesceRequest and close normal admission for every worker;
2. request quiescence for all ordinary workers, quiesce worker 0, then wait for every Quiesced acknowledgement;
3. call every onQuiesceWait to join or detach external producers;
4. release every worker into drain; SocketManager drains its UDP source-owner table before NodeManager invokes onWorkerStop in chain storage order;
5. wait for every Drained acknowledgement, then call every onStop;
6. release ordinary workers into message/engine/loop/pool teardown, wait for Exited, and join non-main workers;
7. finalize shared lwIP pools, ports and quotas and erase the ISN secret after all engines/workers exit;
8. run late, non-owning exit observers and the explicit application finalizer;
9. freeze exit status and destroy nodes, SocketManager, pools, shortcuts, worker synchronization, signal infrastructure, and global storage;
10. exit with the frozen status.

Core settings remain application-owned through step 8; the application finalizer calls destroyCoreSettings() at that existing position. Before a controller exists, parse failure releases temporary input and partially constructed settings and returns directly. The Linux packed launcher has no shutdown coordinator: failures release launcher resources, while successful execveat replaces its entire image with the ordinary executable. Bootstrap uses inherited signal dispositions; after execution the ordinary signal setup, startup rollback, and shutdown sequence apply. On Linux there is no resident launcher or loadable WaterWall runtime module.

The Windows packed public process owns a private kill-on-close Job and joins it before creating any descendant. Children inherit membership at creation; there is no suspended-child create-to-assignment gap. Neither breakaway flag is set and no Job handle reaches the runtime. An incompatible enclosing Job fails startup; this works without nested Jobs on Windows 7 when the public process starts outside an enclosing Job. The sole ordinary Job handle is intentionally retained through process exit because closing it would terminate the public process itself.

Independent --stop-event, --ready-event, and --controller-process capabilities replace hosted mode. The launcher validates object type/access and uses explicit handle inheritance. A process-lifetime observer is armed before configuration reading, translates stop/controller death to a private event, and enforces one 60-second startup budget followed by at most 15 seconds of stop/cleanup. The runtime observes the exact controller at startup checkpoints and publishes readiness only after commit. Deadline termination skips runtime/CRT cleanup; termination initiation does not establish completion.

Resolve dynamically loaded deadline APIs before admitting startup or recovery observers. Deadline callbacks must use cached entry points without loading DLLs or resolving symbols, so a stalled loader cannot block forced termination.

--console:hidden detaches the console; --console:visible allocates one when needed and routes diagnostics there while preserving config stdin. Console requests use the same stop path. The observer never blocks on logs or config I/O. Ordinary clients without a controller capability remain independent of parent exit.

--session-file:PATH creates a private, non-reusable record before runtime creation. --recover:PATH uses its bound Job/process identity, requests stop and verifies active-process accounting. A missing Job name is not proof of termination. If the Job cannot be opened, recovery requires a final one-member barrier receipt plus proof that the exact public process exited. Runtime adapter creation publishes intent before the OS call and its actual GUID before returning to callers. Recovery checks only those GUIDs; unresolved creation identity remains unverified. Built-in Windows TUN IP, route, and DNS operations target the owned adapter. Loop prevention uses the physical-interface egress pin on participating sockets and adds no physical-interface host routes. Shared WinDivert support still serves raw and capture devices. Recovery holds a read-only mapping of the journal across settlement and refreshes that view afterward; ReadFile must not be used to observe mapped effect writes, because Windows does not guarantee coherence between those access methods. An orderly-return event and launcher receipt distinguish cleanup from an arbitrary exit code. Full DWORD runtime status is retained independently of cleanup outcome. The public result marks runtime status known only when it was actually obtained from a terminated runtime. Failed creation, or cancellation of a never-resumed child followed by a confirmed wait, can settle without a runtime receipt. Admission intent remains published until the creation owner proves that boundary. Recovery reports the first unmet completion condition through cleanup_detail, separately from the causal end reason and any file residue. An abrupt runtime exit can settle without an orderly receipt when recovery authenticates the record, independently establishes process inactivity, and observes every recorded adapter GUID absent from GetIfTable2. Retained driver files and possible residue after abrupt exit are reported through file_residue; they do not veto verified process/interface settlement. Runtime status and termination still describe the actual exit, including an unknown status when the launcher could not finalize it. Missing process evidence, unresolved adapter creation identity, failed interface queries, or a still-present adapter remain unverified. This includes a disconnected adapter and does not itself prove active network effects. A missing launcher receipt can therefore still prevent process certification after the Job's name has disappeared. Recovery never treats name absence as inactive Job accounting. Historical cleanup status is separate from permission to attempt a new session. Startup never scans old journals or maintains a persistent unverified blacklist. A settled result permits retirement of client-owned session inputs even with file residue. An unverified result retains those diagnostics and uncertain resources, but clients may launch a replacement with a fresh journal and let its current ownership and initialization checks decide whether it can become ready.

On Windows, the TunDevice constructor reserves its case-insensitive logical device-name before egress-pin selection or socket preparation. A SHA-256 digest of invariant-uppercase UTF-16 names scopes a machine-wide named-pipe ownership handle created with FILE_FLAG_FIRST_PIPE_INSTANCE and a protected administrators/ SYSTEM ACL. The constructor transfers it to the device on creation, or releases it on rollback. The device retains it until teardown. Built-in DNS helpers inherit only a reduced duplicate through an explicit handle list, preserving ownership if the runtime dies first. The last handle close releases ownership automatically; no journal or graceful receipt is needed. Different logical device names remain independent.

With ownership acquired, startup removes only present network devices carrying that exact WaterWall creation tag and Wintun hardware identity, including the empty/missing hardware identity of a creation stub. The tag is supplied to Wintun as the tunnel type and identifies its device description independently of the session record. Untagged/foreign devices are not reclaimed. Phantom registry residue does not block initialization. Ownership and removal each retry for up to five seconds between OS calls; selected transient creation failures get up to three attempts. Blocked OS calls remain covered by the public process deadline. Class-installer removal requires a build matching the OS architecture; unsupported WOW64 removal reports that requirement. Windows restart-required flags become ERROR_SUCCESS_REBOOT_REQUIRED, not successful preparation. Current ownership conflicts, occupied interface aliases, access failures, and failed device/network operations remain normal startup errors.

Applications should use --restricted-config when relying on this built-in ownership boundary; scripts and external nodes can create untracked effects. Client-owned firewall policy remains a separate decision. Recovery does not persist its observations as a replacement launcher receipt; clients retain a successful result until session retirement. Recovery reports the old session; new startup owns reconciliation of its explicitly tagged devices. The full argument, result, permission, deadline and recovery contract is in core/tools/windows_launcher/LIFECYCLE.md.

Completed Windows process termination closes that process's handles and ends its threads and process-local state, including socket egress bindings. Wintun's driver handles owner-process exit and handle closure by unregistering packet buffers and reporting disconnection, independently of WaterWall's graceful handlers. This packet-session boundary must not be equated with adapter removal: modern Windows software-device removal is asynchronous, and Windows 7 uses a separate adapter and orphan-cleanup path. Inactive device properties, interface configuration, driver installations, extracted files, logs, and the recovery record can remain. Client-owned firewall policy has its own lifetime. User scripts and external nodes fall outside the built-in TUN effect inventory. The interface contract's forced-termination table documents the precise boundaries and primary Windows/Wintun sources; these mechanisms do not replace native qualification of the shipped driver and OS.

Logging remains available through late cleanup diagnostics. SignalManager's exit observer registry is not a general ownership stack: observers may emit final diagnostics or adjust the still-unfrozen status, but they do not own workers, nodes, devices, or queues.

8. Lines during drain

The line rules from Part 2 do not change. Source admission closes before inventories drain, and SocketManager's UDP owner table drains before tunnel worker hooks. Chain storage order is not guaranteed to follow callback topology, however. An onWorkerStop hook must therefore drain only its authoritative source or owned-line inventory; it must not require a borrowed-line registry to be empty based on an assumed neighbour order. A final onDestroy diagnostic may require that borrowed registry to be empty after all worker hooks and normal-line owners have completed their drains.

The event loop, chain, tunnel state, tail resources needed for Finish, and line and buffer pools remain alive throughout worker drain. Only a normal line's creator calls lineDestroy(), and every owner drain leaves its normal lines logically dead. Runtime shutdown never destroys a packet line.

Final synchronous payload flushed before Finish can reach a protocol server whose outgoing branch has not started yet. TrojanServer and VlessServer reject initial requests and creation of new UDP backends once the line's owner loop closes normal admission. They retain ordinary forwarding on already-initialized TCP, fallback and UDP associations until association cleanup. TcpConnector also rejects late Init before creating a socket or idle table, clears its local state and finishes toward the borrowed line's owner. These checks belong at creation boundaries; they do not require a quiescence check in every transform's payload callback or a duplicate per-worker shutdown flag.

VmessServer inventories its owned UDP backends per worker for both native and packetaddr associations. Worker stop drains that authoritative inventory while borrowed carriers may still await their source owner's hook; it never assumes a neighbour's drain order. Packetaddr carrier closure publishes Closing admission before the first backend callback, removes each exact destination and ownership link before notification, and restarts lookup after callbacks that can retire siblings. Ready replies own independent encoded bytes. New backend creation requires normal owner-loop admission, so quiescence cannot recreate drained lines.

UDP send retirement

Incomplete pipe-backed datagram assembly retires only the affected socket. The socket owner runs the complete udpSendBuffer() assembly and retirement; the helper does not transfer ownership. WIO settles the input buffer before closing and marks the socket closed before the synchronous close callback, which may destroy associated lines or the WIO. Continuations must retain and recheck their line, and must not access a reclaimed WIO. Close callbacks detach every cached socket pointer before peer cleanup. UdpConnector removes its shared socket and peer bindings, finishes affected borrowed lines through their owners, and guards queue/DNS continuations against reentrant line death. During a source-initiated Finish, the close-time flush retains local cleanup authority and never reflects Finish.

Before the first peer Finish notification, retirement must reject sends through every affected binding, including still-live siblings. Binding selection and each actual write through a cached binding must honor the socket's Closing state before using its WIO or refreshing idle activity. Settle rejected buffers exactly once through the existing buffer APIs; leave binding removal and owner notification to the close handler. Draining is separate: existing bindings on a healthy Draining socket remain usable, while new bindings are refused. A regression test should send on a still-live sibling during the first Finish notification, not only verify that all lines are dead after close returns.

The persistent SocketManager endpoint collection owns every static listening WIO and UDP side-data allocation exactly once, including shared sockets, listed ports, per-port ranges and redirect sockets. Its separately allocated endpoints keep WIO userdata and udpsock_t.owner_slot = &endpoint->listen_io stable when the pointer collection grows. Close each WIO on its owner loop while loop storage is alive. Successful earlier binds remain inventoried when a later setup fails; failed binds never publish a sharable endpoint. Initialize endpoint userdata and UDP side-data before enabling callbacks.

Static UdpListener side-data survives through shutdown and carries an atomic retirement gate plus an immutable diagnostic FD snapshot. Socket-owner close clears its WIO and listener slot, then queues each worker's authoritative idle inventory drain. Queued ingress checks the gate; retired writes never refresh idle deadlines. Drain notifications never run inline inside an expiry callback. If a notification is refused/canceled, the existing finite idle deadlines (at most the current five-minute peer timeout) or normal worker shutdown drain settle the lines; no new retry queue, peer, replacement socket or foreign-pool access is introduced. UdpStatelessSocket uses the same inventory settlement and retains its local-recycle versus detached-buffer-destruction distinction. Endpoint metadata, scope strings and UDP side-data must outlive queued ingress, writes, cancellation, worker-local idle drains and the owned peer lines. Final manager destruction releases them only after worker settlement; the existing shutdown ordering is the lifetime proof, without a new endpoint reference count. Dynamic endpoints remove their value handles and owned normal lines together on socket close; quiescence disables the close callback before detaching userdata and leaves line drain to the endpoint owner. Persistent packet lines are never destroyed here.

Per-worker dynamic endpoint inventory

A dynamic UDP endpoint provider keeps an authoritative registry per owner WID. Each entry owns its WIO and, after first ingress, its provider-created normal line; consumers retain only a value handle, never a line_t *. onQuiesceRequest closes endpoint admission. onWorkerQuiesce detaches read callbacks and WIO userdata on the owner loop while all drain resources remain alive.

onWorkerStop then closes every remaining registry entry through one idempotent path: remove it from the registry before callbacks, detach/close its WIO, clear the endpoint-to-line link, destroy local line state, send Finish away from the source, and make the provider-owned normal line logically dead. A stale or duplicate value handle is a no-op. The provider must not depend on a wrapper or neighbour visiting first, and its final destroy assertion verifies that every worker registry is empty.

Endpoint close may synchronously finish adjacent SOCKS UDP remote lines. Those remote drains retain their separate provider-client line across each callback, detach a remote from its registry before finishing it, and stop immediately if a nested endpoint close already made the client line logically dead. No shutdown path reconstructs or reuses line state after that observation.

Socks5Server remote registries compare the complete original SOCKS address kind, bytes and port. Each owned UDP remote holds an immutable key separately from its mutable routing context; the client map borrows that key. Publish the key, parent reference and map entry before next Init. Detach the exact key/line entry before callbacks or freeing the key, and roll back unpublished setup/insertion failures without sending Finish into a next side that never received Init. Unused line roles keep valid empty queues/maps and do not reserve their backing storage. Only control TCP lines allocate a handshake stream at initialization.

ReverseClient keeps every pair it creates in a per-worker owner registry from publication until close, including connecting, unused, and active pairs. Its shared idle table is only a starvation timeout index; activation removes an item from that index without removing ownership. Expiry, adjacent Finish, re-entrant Init failure, and worker drain all use one pair-close path that unregisters before neighbor callbacks and makes both created lines dead. Worker drain first settles idle items, then closes every pair still present in the owner registry.

MuxClient's per-worker owned-parent list is independent of its selection slots: retirement from selection never ends parent ownership, including a zero-child parent waiting to hand off final encoded output. Worker stop discards that output without waiting for Resume and closes every owned parent and finishes attached borrowed children toward their source owners. Previously detached borrowed children may await a later source-owner Finish, so their accounting in the embedded worker-state slots survives worker stop. MuxServer's worker-local child idle table inventories every owned child across attached and detached states; draining that table closes all children before destroying it. Borrowed server parents keep valid state until their owners finish them; their outgoing backlog can still own queue capacity until parent cleanup. Both MUX nodes publish worker quiescence before any owner drain and discard residual queues in terminal cleanup without creating protocol payload or flow-control work. Their payload entry points also recycle final bytes flushed by neighbours before Finish: closing normal loop admission does not suppress those synchronous calls, and a late Open must not recreate MuxServer's already-drained child inventory.

Mux parent teardown also destroys the fixed-header input stream, its active head, cached bytes and incoming queue charges while the owner pool is alive. Staging batches settle locally on refusal; committed batches belong to parent output and are discarded by the existing terminal path. No callback occurs during staging.

Mux detachment rebinds every child queue, including empty queues, from the parent-only budget to the worker's detached budget before unlinking and publishing detached child membership, all before outward callbacks. Each pop/discard settles its bound budget; delivery observes the released usage. Output remains parent-owned after its producer child dies. Borrowed queues may outlive an individual worker-stop hook, so their worker budgets remain alive until final borrowed-child cleanup. Destroy or rebind every child queue before freeing its parent state. Local stream/output destruction checks settlement, and final instance destruction preserves its owned-parent, child-admission and detached-inventory checks after owner cleanup. Queue charge does not imply that recycled private pipes were physically closed.

IdleTable expiration messages temporarily own delivery, not the owner's index. If delivery is canceled while the item is still attached, cleanup clears the pending marker and restores the item to the expiration heap with its original deadline. Cleanup never substitutes for the worker-affine expiration callback. Explicit removal, worker detachment, or table destruction leaves the item message-owned so cancellation frees it once instead of restoring it. Capacity reserved for attached items permits restoration without allocating in cancellation cleanup. Draining one worker's inventory must preserve those heap reservations for attached items whose expiration messages belong to other workers, even when those items are temporarily absent from the heap. Staging and index growth follow the fatal heap-allocation policy; they have no OOM retry or rollback path. Item creation rejects duplicate keys or unrepresentable index geometry before publication and exposes a handle only after both the map and expiration heap own it.

Worker-local and shared idle tables have different shutdown boundaries. Quiesce each local_idle_table_t timer on its exact owner worker before line drain. An owning component drains the entire local table inline and destroys it while that worker's loop, callbacks, lines, and pools remain alive. A registry of borrowed lines instead remains available until the line owners remove its entries, then the empty table is destroyed on that same worker. Tunnel storage order cannot prove that a neighboring owner has already drained. Never transfer a local table or item handle to another worker. A local natural-expiration callback may extend or remove its own active item, and callbacks may manipulate other active items under their usual lifetime rules. A local drain callback is terminal for its already-detached item and must not refresh or remove it. Never synchronously drain, quiesce, or destroy the same local table from either callback; defer those lifecycle operations until the outer expiration or drain dispatch returns. For idle_table_t, drain each owner WID's active inventory on that WID before final shared-table destruction. An item already owned by a posted expiration message is detached by owner drain rather than called back or freed there; the message callback or cleanup retains delivery and table ownership until it settles exactly once.

An orderly shutdown request is not line destruction. A Finish handler for an owned normal line must still detach producers, destroy line state, and make the line logically dead before returning, even when it also requests process shutdown.

UserController shaping settlement

Quiescence closes shaping admission and rearming, removes owner-worker timers/continuations, and cancels fair-turn participation while AuthenticationClient user tables and locks still exist. The true line creator then drains its normal lines while controller state and pools remain alive. UserController borrows those lines and never calls lineDestroy(). Finish first discards retained/prepared buffers, releases user/client queue costs and tickets, and releases connection/IP reservations once, then propagates only away from the sender. Final destruction requires empty inventories and budgets.

A canceled line continuation is notification only: it may run synchronously, on a foreign thread, or after logical death. It must not access owner-only queues, restart a timer or emit callbacks. Owner-worker inventory settlement releases that work. Retained reservations reference their unique old runtime, so a removed/recreated numeric user ID cannot receive its releases. Same-ID refresh instead moves bucket credit, refill fractions, timestamps and ticket metadata with runtime. Config changes settle the old rate to the change time before applying the new capacity; unchanged refresh adds no burst. Existing managed lines keep cached drains during ordinary client outage or credential rejection; new-login admission retains its separate AuthenticationClient gate. Required local timer/continuation failure closes only the affected line.

Borrowed QUIC session timers

QuicClient/QuicServer confine each QUICHE engine, carrier parser and alarm root to lineGetWID() and that worker's loop. Their clocks use fresh monotonic time for protocol deadlines/RTT, with wall time reserved for certificate/protocol uses. A single owner-loop timer tracks the earliest QUIC alarm or readiness deadline with safe upward millisecond rounding and bounded work per dispatch. Partial input never extends the total handshake/stream-opening deadline.

The per-worker session registry manages borrowed contexts, not line ownership. Quiescence closes new admission and detaches alarm roots on their owner workers before acknowledgment. Normal line owners subsequently drain while chains, protocol state and pools remain alive; a borrowed registry is not required to be empty based on an assumed worker-hook topology order. Final destruction requires no sessions, timers or active driver references before shared configuration is released. Reentrant Finish detaches local state immediately; an active driver frame defers C++ engine deletion until it returns safely. No callbacks or alarms may survive retirement, and received Finish never waits for acknowledgment from a carrier whose lifetime belongs to another node.

9. Timers, watchers, and deferred timer reclamation

AuthenticationClient owns its worker-0 normal control line. Positive reconnect requests preserve the first monotonic due time; zero-delay retries use force-queued work, with a mutex-protected pending latch published before submission and settled once by run or cancellation. Cancellation touches only independently protected tunnel state and never restarts work. An independent timer tracks the earliest outstanding request even when optional ping/pull/push jobs are disabled. Quiescence detaches all timer roots; Stop drains the owned line before final state reclamation. Nonterminal stats-push refusal adds a monotonic not-before deadline of at least 30 seconds, honoring a larger configured reconnect interval. The reconnect timer owns the due retry even with all intervals zero; opening a line, first-use work, explicit requests and Resume cannot bypass it. Session success retains refusal state until the immutable batch receives its matching ACK. Paused writes and this deadline do not rearm the short progress timer. Transport/session clearing preserves its unresolved immutable usage batch and outstanding relative catch-up debt. The client replays that batch before the reconnect pull. Authentication and ACK request a full refresh. If a non-accounting refresh fails after a usable table was installed, new batches may use that unchanged active/baseline pair between backed-off traversal attempts; no batch starts during an active traversal. Due pull retries still receive execution under continuing traffic. Initial synchronization remains mandatory. After the reconnect replay/required-refresh barrier, capture owed relative deltas and unacknowledged observations through that refresh, then drain a finite pass in bounded batches with requested post-ACK pulls. Never retain a rebased aggregate as an absolute recovery target. Keep captured relative debt until matching ACK even when saturation hides it in the rebased visible counters. Traffic after the cut follows configured online pushing/first-use triggers. Disconnect retains both the immutable batch and outstanding relative debt. After the next replay/refresh, merge retained debt with visible unacknowledged deltas without adding their overlap twice. Missing accounts retain orphan obligations, which are excluded from runnable batches and reattach if the same durable ID returns. Ledger-capacity refusal preserves all obligations. Orphan evidence retains local first-use origin and elapsed age, so a later return does not restart expiry. Continuous account presence migrates its handle-generation floor; removal and reappearance reject older handles without letting their cleanup decrement new reservations. A verified server proof that a missing-account batch did not commit may release the immutable attempt and reuse its sequence, but consumes no debt or checkpoint. Only matching ACK consumes represented debt; final tunnel destruction releases its RAM storage without guaranteeing delivery. A nonzero worker-0 progress timer continues recovery with optional jobs disabled, yields between batches and retries recoverable snapshot/send refusals. Every accepted nonfinal user page schedules this continuation before requesting the next page, so a synchronous adjacent peer cannot import a whole traversal in one response drain. Resume and explicit demand respect the published continuation. The last data page may validate immediately because validation imports no new rows. Retained-debt capture budget refusal preserves its work and uses a monotonic retry delay of at least one second; unchanged capacity must not create a one-millisecond wakeup loop. If required scheduling cannot be established during normal operation, request orderly shutdown. Admission refusal after the owner loop has begun quiescence is expected: never escalate a successful shutdown or rearm work. Quiescence cancels this root without rearming; Stop drains the owned line before the arrays/table/locks are reclaimed. Cached new admission survives ordinary outages after initial valid table/clock synchronization, while explicit credential rejection is sticky until validated authorization. Existing connections continue cached enforcement and accounting. First-use observations, local origins/elapsed age and ACK markers survive session clearing and table migration; they are never reset to make reconnect succeed. An empty batch attempt still allows an ordinary queued pull to proceed. The authenticated server identity's last epoch/sequence/digest checkpoint outlives session expiry/replacement; lost continuity stops automatic synchronization rather than discarding or relabeling an uncertain batch. Both accounting owners are runtime state, with no crash/restart recovery guarantee. An ACK means acceptance in server memory, not a disk save. Periodic/final database saving does not persist producer checkpoints; client restart may lose unreported usage, and server restart loses retry checkpoints and may lose acknowledged usage since its last successful save. Shutdown does not guarantee final usage delivery. Preserve continuity refusal for an uncertain batch instead of replaying it into a new checkpoint.

Failed complete synchronization has a process-local streak and monotonic not-before deadline. The first ordinary connection failure retains the configured retry interval, including force-queued zero. Further failures start at max(reconnect-interval-ms,1000), double to max(base,60000) and add bounded jitter up to one quarter of the computed delay. The effective deadline is the maximum of all retry constraints, including stats refusal. Direct requests, opening, first use, Resume and periodic timers recheck it; a dedicated retry wakeup works with optional intervals zero. Successful Session/Pong/intermediate page never resets that streak. Only complete validated installation resets it. Do not extend deadlines on traffic or poll a short progress timer while blocked/paused. Status exposes fixed categories and remaining delay without remote error text.

A paged traversal owns staging and clock ranges independently of a pending page. Each page/final validation retains its own absolute request deadline; progress does not extend it. Disconnect/stale cursor/malformed page/limit/timeout releases staging while retaining the active cache and immutable accounting owners. Quiescence closes request admission, cancels published timers/messages and stops page work. Owner-worker Stop settles the owned normal control line; final instance cleanup releases remaining staging only after callback/message roots have settled. Foreign cancellation callbacks never access worker-only page state or restart synchronization.

A pending NIO remains allocated through dispatch even when a callback calls wioFree(). Close releases its owned descriptor and detaches its array slot; the pending-list release recycles the WIO through its pool. Loop cleanup also releases pending NIO membership, reclaiming detached destroyed WIOs before their pools are torn down.

A pending allocation keeps its original descriptor identity and loop until this release. Descriptor-number reuse retires a closed pending NIO and allocates a fresh WIO; detach/attach rejects pending objects. These rules preserve stale event suppression without clearing the flag that protects allocation lifetime.

Component timers and watchers are worker-affine. onQuiesceRequest closes their rearm/admission state; onWorkerQuiesce deletes the component's published timer or watcher pointer on its owner worker.

MuxClient embeds a trailing worker-state array in its checked tunnel allocation. Each slot holds its selectable timer/counter parent, owned-parent inventory, detached-child accounting, quiescence gate, and a local keepalive table. The first eligible parent transport Est creates that table on the owner worker through the existing fail-fast constructor. No startup messages, selection scan, or temporary keepalive snapshot is needed. An established selectable parent has a recurring ping item and a separate reply-deadline item only while awaiting Pong. Each expiry callback retains its exact parent across reentrant retirement or payload delivery. Closing a sibling removes its items before the heap visits it; no saved sibling pointer remains to inspect afterward.

The first probe waits one ping-interval after transport Est. Waiting does not stall the parent or start a reply deadline. New probes wait for unpaused output, an idle pump and an empty FIFO. A due unsent probe retries after 1000 ms without moving its due timestamp, allocating a frame, or starting a reply deadline; missed wakeups coalesce. Probe state and its independent deadline item are published before direct forwarding because a matching reply or Finish may arrive inline. A matching Pong cancels only the deadline item. Unanswered discovery expires without suspicion or retirement, then permits another due probe. Once peer support is confirmed, deadline expiry retires occupied parents or closes idle parents, including during Pause. A full retirement quota leaves the parent selectable and retries the deadline item after 1000 ms, preserving the original probe time and suspicion. The table's approximately one-second wakeup precision also applies when configured intervals or tolerance are shorter.

Selection retirement removes both items but preserves parent ownership and outstanding-token state. A late matching Pong can restore an unretired selectable parent; retirement never returns it to selection. Replacement creation remains a new-child selection action. Watchdog-retired parents close after their last child leaves without waiting for blocked final output. Worker quiescence closes admission and stops the table before any owner drain. Worker Stop drains the separate owned-parent inventory including retired parents, then destroys the empty table on that worker. Final destruction requires empty ownership and detached accounting, and no table. MuxServer responds through the bounded parent FIFO and obeys Pause, creating no health timer or child for a Ping.

Each KeepAliveClient instance embeds a trailing worker-state array in its tunnel allocation, sized once at construction for the fixed event-worker count. Each slot contains a local idle table, a borrowed-line count, and its quiescence gate. The first transport Est on that worker creates the table through its existing fail-fast constructor. The table holds one recurring ping item per established line and a separate deadline item only while sensitive mode awaits a pong. Item keys use the addresses of their distinct owner handle slots. No deadline-shortening API, shared registry mutex, line-list scan, or snapshot allocation is needed. Ping callbacks retain their exact line across payload delivery; reentrant Finish removes either the current item or a sibling item before the heap visits it. Worker quiescence stops expiry and closes table admission, preserving active handles for owner drain. The last borrowed line releases the quiesced table on its worker; final instance destruction requires zero borrowed lines and no table in every slot. There are no per-line timer roots. Both modes schedule the first ping one ping-interval after downstream transport Est, without blocking application traffic. Repeated Est does not reset the schedule. Its optional reply watchdog starts only when a ping is sent, publishing one outstanding ping and its monotonic deadline before the payload callback, and acknowledges only an empty pong on that exact line. The deadline item expires independently of the ping schedule. A valid pong removes only its deadline item. Timeout destroys local state before finishing upstream and then a still-live downstream owner. It never destroys or recreates the borrowed line. Either directional Pause leaves timer checks and elapsed reply deadlines running; expiry does not require Resume. Due probes may send during Pause. An active encoder or an outstanding pong still defers a probe, retaining one due timestamp rather than queued frames. A blocked ping item retries after one second without extending the outstanding reply deadline. Idle-table checks may deliver a new earlier deadline about one second late; subsecond wakeup precision is not required. Both KeepAlive nodes send Pong replies directly during parsing, including during Pause. The decoder serializes nested input under its existing bounds and holds exact-line references across callbacks. Default Pause/Resume callbacks relay every notification; Finish stops parsing and discards retained input. Finish also clears the pending ping wait, worker quiescence detaches the timer, and owners drain lines before final registry destruction.

Elapsed-time deadlines can use wloopNowMonotonicMS(loop), the loop's cached high-resolution clock converted to milliseconds. The getter does not refresh time: it reflects the latest wloopUpdateTime() and can lag during a long callback. Read it only on the loop's owning thread. For line-bound work, obtain the loop with getWorkerLoop(lineGetWID(line)); event callbacks can use their supplied loop. wloopNowMS() follows wall-clock adjustments and is a different clock domain. Keep timestamp writes and deadline comparisons in one domain. Tests that invoke callbacks without running the loop must advance its cached clock explicitly. Both idle-table implementations follow this coarse timing contract. Shared-table operations read the executing worker's own loop cache, including the receiving owner's expiration recheck; they never borrow the table timer owner's cache from another thread.

wloopNowLoopRunTime(loop) exposes the same cached monotonic timestamp in microseconds; despite its name, it is not elapsed time since loop creation. TrojanClient, VlessClient and VmessClient use it for their first-payload deadlines to preserve microsecond precision. PingClient and PingServer use the millisecond getter for reply-ID idle bookkeeping; the shared generator ignores older worker timestamps. Do not substitute cached time for fresh measurements, precise pacing within a callback, or authentication expiry checks shared across threads. Code running outside event workers, including startup, retains a thread-independent clock source and must not read another worker's loop cache. lwIP runs on owner workers but retains its fresh monotonic sys_now() clock for protocol deadline arithmetic.

Timers that generate protocol output or release delayed work follow their exact node's flow-control role. Ordinary middle nodes may submit due output during advisory Pause without a consumer-pause gate. Preserve FIFO, protocol readiness and hard retained-storage limits. A protocol reply deadline started on submission may therefore include time spent in a receiving adapter's queue. Adapters, exact-line heads/tails and SpeedLimit retain their producer-control responsibilities. All roles stop on Finish, logical death or quiescence, and owner cleanup cancels remaining timer roots.

VmessClient's first-payload timer belongs to the exact direct TCP or owned UDP carrier line's event worker. The first transport Est publishes one absolute deadline; duplicates do not extend it. Clear the authoritative timer slot before an expiring one-shot callback can emit output or close the association. Early wakeup resets the same deadline; reset refusal closes the affected association without retaining or deleting a timer already owned by pending dispatch. Direct TCP due output follows its ordinary-middle advisory-Pause role; an idle owned UDP carrier waits for Resume of that exact carrier. Empty/oversized local input does not cancel the idle obligation.

Request commitment and every line teardown cancel the first-payload timer on its owner worker. No separate response-wait timer is created for silent backends. Normal dispatch admission prevents emissions and new associations after quiescence; generic loop timer quiescence preserves deferred timer storage until source-owner drain can delete its published handle. Drain borrowed application lines and dependent owned carriers while tunnel state, helpers and pools remain alive. Detach associations, settle queues and erase session keys before zeroing line state; an owned carrier's Finish returns with that exact line logically dead. No client-wide autonomous line inventory or authentication pump is required.

StreamFragmenter's one per-line timer can represent initial ClientHello assembly. That deadline may run before Est or during advisory Pause; expiry makes original bytes ready in its FIFO. Its optional startup Est wait and FIFO order still apply, but due output does not wait for Resume. Finish discards the candidate and detaches the timer without replay. Once shaping and all retained work settle, the timer is gone and the node destroys and zeros its line state before permanent bypass.

The process-global resource sampler follows this order on worker 0: stop its periodic timer during worker quiescence, then allow the last published snapshot to age stale naturally before global-state destruction. Worker-local idle tables are likewise quiesced before their callbacks' reachable tunnel or line state is drained; owner-worker stop removes remaining items and destroys the empty table.

Generic loop quiescence deactivates any remaining timer and clears its callback and userdata. The allocation is kept on a loop-owned deferred list because line state may still contain the timer pointer and delete it during the subsequent owner-line drain. wtimerDelete() recognizes that state and performs the final free exactly once. Loop destruction frees only residual deferred timers after drain.

A due one-shot is marked for reclamation before its callback runs. A successful wtimerReset() revives it. If self-rearm is refused, the callback clears every authoritative owner slot and settles represented work, then returns without deleting or retaining the executing timer; pending-event release performs the single reclamation. This differs from a callback suppressed by quiescence, whose allocation remains available for owner-worker cleanup.

Resetting a pending timer suppresses its old expiration without clearing the pending lifetime reference. The dispatcher may still hold the current or next timer pointer. Deleting after a successful reset is safe, including from the timer's own callback: allocation release waits until dispatch settles that reference. If the timer is kept, its replacement schedule remains active and its callback can run at the new deadline. Quiescence and raw loop cleanup settle the same reference before reclaiming or retaining the timer for owner cleanup.

On IOCP, cancellation is not completion. Quiescence suppresses user callbacks, but kernel operation records remain alive until canceled completions retire. WIO memory is reclaimed only after those references have drained.

10. External devices and lwIP

Reader activation belongs in a startup callback after construction, with worker bootstrap queued before delivery can be posted. See Part 4's startup ordering and foreign-thread reader rules.

TUN, raw, and capture devices split stop into request and wait operations. The request phase closes local delivery/rearm admission and signals external readers without joining. The wait phase joins or quiesces external callback roots while device and tunnel state remain valid. Final stop releases remaining interface, writer, route, DNS, and platform resources after worker line drain.

Linux Capture prepares inactive NFQUEUE readers before installing rules. While its rules are installed, the coordinator services bounded receive/verdict batches with NF_ACCEPT. After every rule succeeds, capture activates and queues reader activation behind worker packet Init. During later startup commands, while workers remain gated, coordinator service uses NF_DROP and posts detached packets through the bounded session FIFO without invoking tunnel callbacks. During normal operation, each Netlink descriptor has one worker owner. Stop closes reader/session admission; owner quiescence detaches WIOs and verdict timers. BringDown waits for admitted callbacks and detached watchers before reclaiming native ownership on the coordinator. Unsent verdicts remain available for NF_ACCEPT; already-dispatched packets follow ordinary ownership.

Only after this handoff may supervised rule cleanup pump the queues. Remove NOTRACK before NFQUEUE, settle residual packets within the device packet/time bounds, then close descriptors. Pending rules retain their ownership ledger for retry; closed queues make remaining queue-bypass rules fail open, but cannot undo NOTRACK. A failed verdict or drain is reported and closes the queue rather than leaving a bound queue without a reader. Session/generation references make late worker notifications inert after stop, restart or device destruction.

procRunArgvWithDeadline() supports an optional bounded nonblocking parent-thread progress callback. With it, quiet-output poll waits are sliced at 5 ms and child reaping/termination grace also service progress. The original absolute command deadline remains authoritative. Each Linux Capture reader generation also registers a caller-owned progress record on its lifecycle coordinator thread. Every supervised command services these records, including another device's cleanup and commands without an explicit progress callback. POSIX execCmd() also services registered progress while waiting for user scripts, retaining their existing unlimited duration. Capture residual drains service the registry between their own bounded passes. After all owner watchers detach, a device can reclaim coordinator ownership for acceptance before its own sequential QuiesceWait hook runs. This keeps all still-bound queues serviced during intervening cleanup. The generation unregisters before its device is freed; late worker tokens retain no command registration. Registration, BringUp, BringDown and destruction share the same coordinator thread.

Registered Capture progress skips active worker-owned queues. During startup, workers_run_flag == false provides the separate exclusive-context proof that queued worker activation has not begun. Coordinator service follows capture admission: inactive devices accept, while activated devices drop and queue packets for later worker delivery. A full session FIFO stops native reads until capacity returns. Otherwise closed/quiesced reader admission and zero attached watchers prove the handoff. Progress must never race worker reads, borrow their pools, or invoke callbacks through the tunnel chain.

On Linux and BSD, built-in TUN configuration commands use the deadline-aware process supervisor (7-second command deadline, 250-ms termination grace, 64-KiB stdout cap). Route and Linux DNS cleanup retain an unresolved record when a supervised mutation may have taken effect but ownership cannot be proved. Such a record makes cleanup fail visibly; it does not authorize deleting potentially pre-existing host configuration. The limits apply per command, not to the whole shutdown sequence or user-configured scripts. Attached Linux interfaces capture systemd-resolved's structured DNSEx and Domains properties using supervised busctl commands before mutation. The lifecycle-owned snapshot preserves ordered servers/domains, IPv6 addresses, ports and server names. Each attempted field mutation retains its own pending restoration, including failed or timed-out commands. Cleanup compares the current value with the captured baseline and the requested value, restores only a still-matching changed field, and reads back uncertain restoration. Unreadable or conflicting state remains visibly pending; one failed field does not prevent cleanup of the other. Attached-interface cleanup never invokes link-wide revert or changes unrelated resolver properties. The saved link index is checked before reads and writes. Final device destruction releases snapshot storage after the policy owner's bounded cleanup attempts. Per-interface configuration must be serialized with other network managers: D-Bus read/compare/write does not provide a cross-process atomic transaction.

Every TUN backend calls deviceReaderSessionEndRequest() in the request phase and defers deviceReaderSessionEndWait() to BringDown. Linux closes its reader generation and capacity-notification admission, and each owner detaches its WIO in onWorkerQuiesce. BringDown waits for admitted reads and detached watchers before retiring pending records. Refcounted queued tokens can outlive the device; a closed generation permits only token settlement, never device access. Required runtime resume/stop notification failure must not silently strand a disabled reader or an attached watcher.

Windows closes receive/rearm admission, disables its one-shot thread-pool wait, waits for admitted worker receives, disables the wait again to settle a concurrent rearm, and quiesces native callbacks before ending Wintun. Queued notifications retain only the closed reader generation. The session owns its read event; the backend must not close that event separately. Its separate stop event also serves DNS command cancellation and remains alive through those operations.

Darwin/BSD stop pipes remain nonblocking and close-on-exec. Their reader must join before pool ownership reset and fragment retirement. Every backend closes worker-output admission in the same request phase and settles admitted writes before native resources close. Down is published only after required ownership is gone. UdpStatelessSocket has no external producer session; its worker messages and async DNS requests settle before worker quiescence, and it closes its WIO on the recorded owner.

Linux receive scratch belongs exclusively to each queue's worker. Segment work reserves input storage, work context and its output allowance before transferring scratch to the destination FIFO. All ordinary work, including assured input, copies only the compact IP packet, reserves one slot without an output allowance, and leaves scratch with the reader. Capacity refusal retains the current record and returns to the event loop; workers never wait for their own queued deliveries. Stop discards unadmitted records after receive roots quiesce. Shared fragment storage uses detached allocations under its mutex and retains one device-wide set of limits.

Ordinary batches, checksum work and GSO work share each worker's session FIFO in both offload modes. A GSO step allocates, segments, completes checksums, and delivers bounded output while the existing delivery gate is held. An unfinished aggregate remains at the head for a later callback. Completed small entries share the callback's remaining packet/byte budget. Each entry separately enters the gate and checks its generation, so shutdown or stale work cannot admit later delivery in the batch. Credits remain charged through each segment's delivery and are released once when the work finishes or is discarded. Refusal or cancellation destroys owned input and work storage and settles credits without device access, normal delivery, or borrowing a buffer pool, including on a foreign thread.

Ordinary unfragmented TCP/UDP checksum work uses the same FIFO work admission, generation check and delivery gate. It copies only its compact packet and retains the validated checksum plan independently of reader scratch. Completion runs on the destination worker before publication; cancelled or stale work only destroys its private packet/context and settles its reservation.

BSD and Darwin external readers configure their 8 MiB allocation-charge and 1,024-packet budget and ordered worker FIFO before opening a generation or allocating message records. Full-queue input drops without waiting for its own credits; stop joins the reader before retiring its pool.

Reader-session internals retain one lifetime owner: device_reader_session.c coordinates generation, admission and final reference release; device_reader_budget.c owns reservations and capacity notification; device_reader_dispatch.c owns message settlement and ordered FIFO continuations. Their shared private message/queue representation adds no independent lifetime. The final session reference releases dispatch storage, fragment storage, the message allocator, and then the capacity notifier in that order. Budget layout configuration still precedes the first message-pool allocation.

A scheduled FIFO token holds a reader-session reference through continuation or cancellation. The queue and capacity event therefore survive late cleanup after device teardown. Retained old-generation work cannot deliver into a reopened generation. A failed join retains everything the reader may still access; it never forces cleanup of storage reachable by a live reader.

Device delivery uses local generation/session gates. A producer entering before gate closure may finish; a later entrant is refused and releases its retained ownership. An old reader generation is quiesced (and external reader threads joined) before restart opens a new one. TUN and raw devices own bounded per-worker output FIFOs. Their published writer generation is detached before admission closes. Native writes hold that generation's operation gate; close waits for them without serializing writes against one another. After admitted callbacks and pending native operations retire, queued buffers are destroyed and the native device/session can close. Late callbacks or cancellation retain only a generation reference and cannot write into a reopened device or access reclaimed device state. Cancellation never borrows an event worker's pool. Destroy still requires external producer contexts to have quiesced.

Raw Linux output retains an active batch and its unsent suffix across worker callbacks. Each native callback attempts at most one sendmmsg(). EINTR yields through an ordinary worker message; EAGAIN schedules one 1 ms retry that checks poll(POLLOUT, 0) before sending. Lack of readiness schedules another timed callback; terminal readiness failures publish device failure. None of these callbacks blocks the event loop. Closing the writer generation makes pending retry tokens inert, and the coordinator discards their retained buffers after admitted callbacks finish. NOTRACK removal follows this settlement.

The Windows raw backend uses overlapped WinDivert writes with a one-shot native completion notification. The notification only posts to the recorded worker; that worker reads completion results and recycles finished buffers. Stop first closes writer admission and waits for admitted callbacks, then disables and quiesces completion notifications and cancels/completes outstanding native operations. Only afterward may the helper destroy queued buffers or close the WinDivert handle. Cancellation is not evidence that the kernel has released its buffer. These native roots require no dedicated application writer thread.

Device fragment storage is session-local. Close admission, wait for admitted delivery, quiesce worker producers or join external readers, reset exclusive pool ownership where applicable, and retire assembly or staged buffers before reopening. No incomplete assembly crosses generations. Pool retirement detaches its pointer before delayed queue cleanup can release the last session reference. EndWait waits for admitted delivery roots only; packets already handed downstream carry no device receipt and follow ordinary packet ownership. See Part 4 for policy, expiry and resource limits.

Every failed join retains the native handle and resources the thread can access. Destroying them underneath a potentially live thread is never an orderly fallback.

Worker-engine slot initialization and lazy engine control allocation use the WaterWall allocator. Heap exhaustion terminates at that boundary; engine lookup still refuses unavailable runtime, foreign ownership or closed admission. Bounded lwIP packet/protocol pools retain their ordinary exhaustion results.

After stack work, the engine recomputes its next deadline and advances the owner wakeup only when needed. It preserves an existing earlier or already-pending wakeup; repeated packet input must not postpone due protocol work indefinitely. Due dispatch remains bounded and yields back to the loop.

Worker quiescence closes engine admission and detaches its timer root before wloopQuiesceNormalWork(). No new engine, flow or netif may appear afterward. Node worker drain detaches local stack roots while engine and pools remain live. PTC destroys its owned normal lines; CTP preserves borrowed state until the source's Finish, even if CTP is visited first. Main node stop only releases shared services after all workers acknowledge drain.

If a required bridge control submission is refused, the owner-engine callback first detaches its producer and records terminal reconciliation for owner drain. It then uses signalmanagerRequestShutdownPreservingAcceptedStatus(1): an already accepted shutdown keeps its selected status, while a refusal that initiates shutdown selects failure. Unavailable orderly handoff still hard-aborts. This request never replaces PCB, registry, buffer or line settlement.

Pretend UDP input validates its transport checksum before publishing a child. An accepted PTC child belongs to its exact engine/netif/generation even before the first receive callback creates a normal line. Route cleanup drains both mapped flows and any remaining exact-route children before removing its netif.

workerPerformTeardown() settles detached worker messages, releases the engine, then destroys the saved loop and pools. Message detachment already cleared worker->loop, so engine cleanup uses its saved explicit loop reference. Retained pbufs and deferred timer roots must settle before their storage disappears. Shared pool/port/quota finalization and ISN-secret erasure occur after all engines and workers exit; destroying one engine must not affect another's live state.

11. Choosing a termination API

Classify the state, not the log level:

Situation	Action
Expected runtime completion	Commit the result, release local ownership, call requestProgramShutdown(0), and return.
Orderly runtime failure with structurally valid state	Publish component failure, release callback-local ownership, call requestProgramShutdown(code), and unwind normally.
Per-line failure	Close that line through its owner contract; do not make it process-wide without a product requirement.
Invariant violation or unsafe/corrupt state	Log and call abortProgramNow(code).
WaterWall memory-manager heap allocation failure	The memory manager calls abortProgramNow(1) directly, without logging or orderly cleanup.
Startup failure	Return/record startup status bottom-up, release local resources and locks, and let the top startup owner select StartupRollback once.
Pre-controller command/configuration failure	Release parser-owned state and return an exit code directly.

Do not call requestProgramShutdown() while holding a transient lock or buffer that the current callback must release. Publishing the request does not preempt the callback. After an accepted request, return through normal control flow and do not arm new work.

The ambiguous terminateProgram() API does not exist. Runtime sites use a typed orderly request or the independent lock-free fatal path; startup sites propagate status to their owner.

Major startup stages return ww_startup_result_t. One synchronous top-level stage may install a thread-local ww_startup_context_t as a bounded dynamic error collector, with nested scopes around historical fixed-ABI tunnel callbacks. The context is restored deterministically before worker event loops run and asynchronous callbacks must not use it. APIs that already have a useful result type should still return errors explicitly.

After imported-tunnel loading, configuration parse and installation, and SocketManager startup, the main thread consumes the signal mailbox regardless of whether the stage result already failed. The shared startup-failure boundary uses the mask-protected SignalManager arbitration operation described above. A signal already in the mailbox therefore retains Signal as the immutable origin while both causes still select StartupRollback; a signal deferred after mask acquisition is handled later and cannot retroactively replace StartupFailure. Later stages are not entered.

Registered API admission and detached jobs

NodeManager closes global API dispatch admission at the beginning of nodemanagerQuiesceRequest(), before invoking any target's quiesce hook. Closing only APIServer admission in its own hook is insufficient: target storage order must not leave a window for new callbacks into a quiescing target.

Each connection worker cancels APIServer timer/continuation roots and marks pending jobs abandoned before acknowledging quiescence. Worker-message cancellation settles queued jobs; already-running bounded handlers return before their worker quiesces. Detached cancellation may run synchronously, on a foreign thread, or after its connection is logically dead. It can release independently synchronized job ownership and reservations, but cannot inspect owner-only session state, emit tunnel callbacks or schedule replacement work. Running abandoned jobs stay charged until they actually settle.

At final release, detached jobs securely clear their owned request/response copies before freeing them. Response buffers returned by handlers are cleared before pool reuse. Request ownership has already transferred to the handler; dispatch cannot touch that buffer again unless the handler returns it. APIServer clears owned request bodies/tokens, retained frames and output on normal completion and owner cleanup, and clears consumed or discarded input before recycling. These boundaries do not guarantee erasure of configuration, parser-internal partial trees, neighboring transport or OS copies. Credential-bearing AuthenticationServer parsers supply the lexical parser's cleanup callback so fully parsed trees rejected for duplicate keys or trailing input are wiped before deletion; callers also wipe trees returned successfully. Secure clearing targets the owned secret buffers and inline tokens; ordinary metadata structures need only their normal reset/release. These are management-operation boundaries, not blanket wiping of VPN traffic. Keep ordinary lookup/index comparisons and constant-time bearer-token verification in their respective roles; secret cleanup does not require replacing unrelated comparisons. Decoded API tokens are cleared immediately after credential verification, including refusal; schema-invalid envelopes clear unused tokens during parsing. Anonymous reservations settle on credential verification or owner cleanup. Quiescence closes both the total and anonymous memory domains, and destruction requires both to be empty.

AuthenticationServer conditional edits, account creation and password replacement use this same admission and detached-job lifetime. Queued work rejected after global admission closes cannot mutate. Already-running bounded handlers settle before target storage is reclaimed, and a committed edit or new account remains committed if shutdown abandons its reply. Cancellation, connection loss or a late gateway timeout cannot roll back an in-memory commit. A returned node-operation failure means that operation did not commit; a lost or late delivery after dispatch leaves the caller's outcome unknown. Preserve the generic completion rules and require a fresh account read before deciding on another patch; never fetch a new edit token and replay a write automatically. For creation, reread the original ID; its presence cannot prove which caller created it or that the redacted password matches. If absent, an explicit retry keeps the same ID/inputs, and uniqueness still prevents replacement or duplicate publication. Never auto-select a new ID or treat id_exists as proof of an earlier successful request.

For password replacement, rereading supplies a current token and public account state, not evidence of which password was installed. The caller must explicitly decide whether to resubmit the retained desired password with that token. A successful resubmission establishes it only at that transaction's commit point; request IDs remain correlation labels without automatic retry or durable records.

AuthenticationServer normal shutdown forces a final loaded-database save; StartupRollback never does. Periodic clean saves may skip only with a successful process-local primary checkpoint, a usable nonempty regular primary and no due history slot. Configuration/new-statistics commits mark dirty; exact replay and session enrollment do not. Producer creation forces synchronous save with its candidate present, restores prior dirty state on rollback and marks clean after the successful revision bump. Primary success clears dirty even when optional history fails; its slot stays due. A skipped tick changes neither recovery bytes nor primary-save telemetry. Missing primary forces the existing repair; health probe errors fail visibly. No disk thread, persisted accounting checkpoint or stronger crash guarantee is implied.

Startup restricts existing primary/recovery permissions independently of choosing a database candidate. A valid primary still requires restricting an existing backup, but not reading or parsing its contents; an absent backup is allowed. Restriction failure stops startup without replacing either file. Existing file ownership need not match the process user; restriction changes no ownership or process privileges, so root execution retains normal filesystem access. Startup parses an isolated database candidate before publication. It accepts one leading UTF-8 BOM only at the database boundary and retains the original bytes for archival; strict UTF-8, unique decoded keys and complete-input validation remain mandatory. Rejection diagnostics report categories, source offsets and account index/durable ID without source strings or credentials. Missing or structurally malformed primary data may use only a fully validated recovery backup. Native/I/O/construction failures and otherwise valid accounts exceeding the synchronization export allowance stop startup with both database paths/contents preserved. Before restoring a malformed (including empty) primary, preserve its exact already-read bytes in an exclusively created private sibling corruption archive; successful flush/close precede atomic restoration. Archive failure prevents restoration; failed restoration retains the archive and backup. Archives have operator lifetime and never enter recovery or history pruning. History pruning retains the exact current successful snapshot plus the lexically latest eligible other period names; clock reversal does not remove the snapshot just written. Successful snapshot writes advance their period slot even when pruning fails; pruning retries with the next snapshot. Optional history-write failure backs off for one minute without postponing dirty primary saves. Ordinary save health checks still refuse an empty previous primary without rotating its backup.

Producer paging snapshots are retained by sessions under exclusive server state ownership. Quiescence and session destruction release all references before the server store is reclaimed; final destruction requires zero snapshot charges. Idle expiry and terminal page validation release references during ordinary operation. Snapshot storage is credential-bearing and is securely cleared before free.

An enabled AuthenticationServer audit writer outlives all API and producer submitters. Global API quiescence and owner-line drain finish those submitters before node destruction closes audit admission, signals its writer and joins it. The writer drains accepted records and closes its checked file before queue, condition, mutex and node storage are freed, including startup rollback. It never borrows a worker pool or posts callbacks into an event loop. A failed join is a fatal invariant, not permission to free reachable storage. File-output failure remains component degradation; it must not request account rollback or claim that an already committed write failed. Shutdown drain is not crash recovery, and accepted records cannot be recovered after an earlier writer fault.

Rejected administrative operations share one node-wide five-second monotonic sample gate and may enter only below 16 of the 256 FIFO slots; committed admission uses all free slots. Intentional omissions saturate separate cumulative and pending suppression counters without changing health. An admitted rejection carries the pending node-wide count. After submitters quiesce, the writer may append one fixed server-sourced rejection summary after accepted records, using the same sequence rules and fixed storage. It adds no timer or callback root. Actual writer failure is checked before sampling and still records loss/unhealthy state; failure to write a final summary loses one record, not the number of intentionally suppressed attempts it represents. Queue locks cover bounded metadata only, never rendering or file I/O, and audit failure never undoes an account commit.

APIServer owns no accepted normal line. During Stop the true source owners drain their lines and invoke its borrowed-line cleanup; it must never call lineDestroy() or invent a second line inventory for owner cleanup. Instance registry, session lookup, job and budget storage outlive every reachable callback. Final destruction requires no live sessions, jobs or reservations. Startup rollback removes descriptors before freeing instances even when no worker began. A session whose required timer/continuation cannot be installed is settled and closed locally; an API client's resource refusal does not require process shutdown.

12. Validation

Focused validation should cover the behavior changed:

ctest --preset linux-unit-release --output-on-failure \
  -R 'line_task_scheduling|worker_context_helpers|shutdown|signal|worker.*quies|wloop_(stop|wakeup)|external_node_abi|quiescence_gate|reader_session|lwip'

# Focused race coverage on supported Linux toolchains
ctest --preset linux-tsan-unit --output-on-failure

Important fixtures prove:

• timer_pool proves warm reuse across every memory profile, shared-master reuse, fatal allocation failure, and pending/deferred timer reclamation;

• line_task_scheduling proves direct/foreign, buffered, line-dead, admission closure, quiescence, teardown, and re-entrant settlement;

• worker_context_helpers proves the underlying worker-message result/reason mapping and callback-or-cleanup ownership contract;

• durable degraded-wake acceptance, saturated POSIX wake handling, request coalescing, first-reason selection, and final status arbitration;

• the worker-0 callback returns before coordination;

• two-worker quiesce, drain, teardown, and join ordering;

• callback-batch suppression after the first quiesce request;

• exact callback-or-cleanup settlement for queued and timed messages;

• component request passes complete before any wait pass;

• SocketManager source-owner drain precedes tunnel worker hooks;

• the real shutdown_production_sequence composition is a multi-worker smoke test for UDP source ownership, mux carrier/child ownership, queued work, and coordinator process exit; it is not an external-producer wait or complete phase-ledger proof;

• focused ReverseClient units drain connecting and active pairs, cover re-entrant Init close and adjacent Finish before drain, and require an empty owner registry before destroying the timeout index; the reverse TCP bridge remains the smoke test;

• device/session enter-versus-close and join-versus-reclamation races;

• external-node ABI acceptance and pre-getter rejection of missing or otherwise mismatched descriptors;

• no production leaf reads a process-wide shutdown phase.

The linux-tsan-unit preset builds the focused concurrency units plus the real production-sequence and reverse-teardown fixtures. Windows CI owns native execution of the console lifetime and IOCP admission tests. Local completion should still use the smallest focused set that covers the changed ownership boundary. The full production and unit lanes remain a release checkpoint.

The trusted TUN/PTC checksum mode is selected before receive activation and normal worker dispatch. Keep it immutable through shutdown and local device restart: queued partial packets and PCBs retain that contract until settlement. Existing session generations, delivery gates, detached-output cleanup and owner drain provide lifetime protection. Never clear a neighbor's mode during destroy.

PTC's certified inline packet output holds the existing output gate through the TUN payload callback; it does not bypass stopping checks. All other packet output retains one detached sbuf through run-or-cancel worker delivery. Cleanup destroys that sbuf without dereferencing the tunnel or borrowing a worker pool. Both paths leave normal-line drain, Finish propagation and engine teardown unchanged.

Each TUN worker flush checks its writer generation before entering native I/O. The Linux batch callback also checks device stop state before emitting each record, and releases every dequeued buffer before returning. The lifecycle owner closes admission and waits from outside those callbacks; a callback must never close or wait on its own writer gate. A fatal native result marks the device failed, then requests shutdown only after the writer gate is released. The shared atomic GSO-refusal latch disables later aggregation for the device generation without changing published checksum capabilities; BringUp resets it.