WaterWall Developer Guide

This is a source-oriented guide for people and coding agents who implement, fix, or review tunnels under tunnels/. It has seven core parts and a dedicated API section. This page (Part 1) builds the mental model. The remaining core parts go deep on the contracts you must not break; the API guides cover external clients and registered node operations.

WaterWall is a chain-based tunneling runtime, so a correct tunnel is never "just" a parser or an encoder. A tunnel must preserve callback direction, line lifetime, per-line state, buffer padding, packet-line behavior, and composability with whatever nodes happen to sit before or after it in a user's configuration.

The single most important rule

Do not design a new lifecycle model as an incidental fix for one tunnel. Start from the intended contracts in this guide and the current mechanics in ww/net/, then compare relevant mature tunnels that solve a similar problem. A reference implementation is evidence, not proof: verify it against the contract, its callers, and its neighbors, because a repeated pattern can contain a repeated bug. An explicitly requested lifecycle redesign is cross-cutting work, not forbidden work; account for every affected implementation, test, and contract document.

How This Guide Is Organized

Part	Topic	When you need it
Part 1 (this page)	Overview, core objects, directions, lifecycle	Always read first
Part 2: Lines, Callbacks & Lifetime Safety	Init/Est/Payload/Pause/Resume/Finish, re-entrancy, line references, clean close	Any change that touches connection flow
Part 3: Buffers, Padding & Shift Buffers	sbuf_t, buffer pools, required_padding_left, prepending headers	Any change that frames, prepends, or rewrites payload
Part 4: Packet Lines & Packet Tunnels	Packet-line semantics, pure packet tunnels, packet/stream bridges	Any layer-3 / packet work
Part 5: Anatomy of a Tunnel & Workflow	Directory layout, node.c metadata, create.c, line state, HTTP rules	When you add or restructure a tunnel
Part 6: Building, Testing & Reviewing	CMake presets, ctest, validation, review checklist, agent output	Before you call a change "done"
Part 7: Application Termination	Shutdown controller, worker quiescence, lifecycle stages, signals, and destruction order	Any change to startup, shutdown, worker control, or asynchronous ownership
API protocol and Node operations	Client framing, authentication and envelopes; registered handlers and their ownership/budget contracts	When building an API client or exposing a node operation

Read These Files First

Before changing tunnel behavior, read the source that implements the contract. These files are short and dense, and they define the current mechanics. They are not proof that the implementation is defect-free. If source, tests, and this guide disagree, investigate and report the mismatch rather than silently normalizing to the existing code.

Area	Files
Tunnel callbacks and chaining	ww/net/tunnel.h, ww/net/tunnel.c
Line lifetime and per-line state	ww/net/line.h, ww/net/line.c
Chain finalization and packet lines	ww/net/chain.h, ww/net/chain.c
Packet-tunnel defaults	ww/net/packet_tunnel.h, ww/net/packet_tunnel.c
Shift buffers and padding	ww/bufio/shiftbuffer.h, ww/bufio/shiftbuffer.c
Stream inspection/extraction	ww/bufio/buffer_stream.h, ww/bufio/buffer_stream.c
Fixed-header splice stream	ww/bufio/splice_stream.h, ww/bufio/splice_stream.c
Splice representation helpers	ww/bufio/splice_buffer.h, ww/bufio/splice_buffer.c
Buffer pools	ww/bufio/buffer_pool.h, ww/bufio/buffer_pool.c
Node metadata	ww/objects/node.h

Reuse Existing Containers and Helpers

Before implementing a private data structure, buffer utility, or timeout scheduler, check the facilities already available in ww. Prefer an existing helper when its ownership, worker affinity, and timing contracts fit the node.

WaterWall includes the STC library for typed C data structures. It provides hash maps (stc/hmap.h), sets (stc/hset.h), deques (stc/deque.h), and priority queues (stc/pqueue.h), among other containers. ww/CMakeLists.txt configures the dependency and exports its include path to nodes. Follow existing container definitions for the i_type, i_key, and, where needed, i_val configuration. Mutable containers still require owner-worker confinement or synchronization. The pinned STC headers are copied to the build-owned patched-deps/stc tree. Its compact hash metadata saturates long probe distances; lookup, displacement and erase reconstruct the exact distance for saturated buckets. Never edit the dependency cache or allow distance truncation to turn an occupied bucket empty. User credential, name, identifier and active-IP indexes use one immutable secret process seed from the runtime CSPRNG. Direct object fixtures must establish the same secure-random, fast-random and crypto prerequisites as normal startup.

ww/bufio/ provides reusable helpers beyond raw buffers:

Need	Helpers
Inspect and extract bytes across queued ordinary buffers	buffer_stream.h
Queue buffers in FIFO order while preserving entry boundaries	buffer_queue.h
Extract an exact ordinary/splice range from an active head and unbound queue	buffer_queue_range.h
Account for queued bytes, retained storage, and entry limits	buffer_budget.h
Parse fixed headers with ordinary or splice input	splice_stream.h, splice_buffer.h
Queue context events	context_queue.h
Reuse buffers and other objects	buffer_pool.h, generic_pool.h, threadsafe_generic_pool.h

Read each helper's API contract and Part 3 before choosing it; the helpers differ in buffer ownership, representation, and worker requirements.

For coarse idle expiry or keepalive scheduling, consider the tables in ww/base/local_widle_table.h and ww/base/widle_table.h instead of maintaining a custom linked list and timer. They already provide lookup, deadline refresh, expiration callbacks, removal, and drain operations. See Choosing an Idle Table for worker ownership, timing, and item lifetime, and Part 7 for shutdown.

Good Reference Tunnels

Use the closest existing patterns as comparisons, not as templates to copy blindly. These tunnels exercise useful contract shapes, but their behavior still has to be checked against the exact role being changed:

Reference	Read it for
TcpListener, UdpListener	Chain-head adapters that own an OS resource and create/destroy their normal lines
TcpConnector, UdpConnector	Chain-tail adapters that own an OS resource but borrow the line created upstream
TlsClient, EncryptionClient	Stateful protocol wrapping, per-line state, clean finish with final bytes
MuxClient	Internal line ownership, parent/child lines, re-entrant safety
PacketsToStream, StreamToPackets, PacketsToConnection	Packet-to-stream boundaries and packet-line anchored bridges
PingClient, PingServer	Direct paired ICMP Echo transforms, same-direction decoded data, and optional opposite-direction exact replies
template	An explicit callback skeleton for reference; production nodes retain applicable defaults

The Runtime Model

WaterWall composes tunnel instances into an ordered chain. A typical stream chain looks like this:

TcpListener -> ObfuscatorClient -> TlsClient -> TcpConnector

Each assembled chain supports at most 64 tunnel instances (kMaxChainLen). The count includes internal helper tunnels and all branches merged into that chain; it is independent of the configuration's total node count. Inserting or merging beyond this limit must reject startup with a diagnostic stating the 64-node maximum. A chain with exactly 64 instances remains valid.

Chains track MuxClient and MuxServer presence independently through the mux_client_tunnel_present and mux_server_tunnel_present one-bit booleans. Insertion sets the matching flag, and merging retains both kinds of presence. Consumers that need any mux node use the logical OR of these flags. Both TcpConnector and TcpListener use the large socket send/receive defaults (16 MiB per option) whenever either MuxClient or MuxServer is present, independently of core misc.tcp-tune or sysctl success. With no Mux node in the chain, omitted options remain at the kernel default. Explicit buffer options take precedence independently for each direction; connector destinations inherit the effective top-level value for options they omit.

The first and last nodes are usually adapters. They own an operating-system resource: a TCP socket, a UDP socket, a TUN device, a raw socket, and so on. They are the only nodes that read from or write to the outside world.

Everything between the adapters is a middle tunnel. Middle tunnels must stay composable: they transform callbacks and payloads without assuming which adapter is on either side. A middle tunnel that only works when a specific adapter sits next to it is broken, even if its own tests pass.

Est carries adapter transport establishment through that composition. Middle tunnels must forward it promptly, independently of their protocol handshake, request header, first payload or output Pause. Keep protocol readiness in separate state; see Est Is Transport Readiness for reentrancy and mappings between distinct line lifecycles.

Pause/Resume are repeatable advisory state notifications. Ordinary middle nodes forward them transparently, without retaining pause state solely for forwarding or gating due scheduled output. Adapters and exact-line heads/tails control their producers and enforce receive/write limits; SpeedLimit and stream shaping in UserController combine their own holds with external pressure. UserController datagram shaping relays external pressure but does not pause producers for local rate backlog alone. Every node still preserves FIFO, protocol readiness and its own retained-storage limits. See Flow-Control Roles and Admission. Valid payload can be submitted after Init without waiting for Est; required protocol headers, authentication and exact-line lifetime still apply.

On a splice-capable stream chain, logical payload bytes may be ordinary resident memory or a real prefix followed by bytes in one buffer's private pipe. Logical length therefore does not prove pointer accessibility. Mux uses a fixed-header splice stream to cache headers and extract exact bodies; BufferStream and HTTP parsers require ordinary memory. Opaque forwarding can preserve the wrapper. See the BufferStream and splice contract in Part 3.

 -------------- chain --------------------------------------------------

   ------------          ------------          ------------
   |          | -- up -> |          | -- up -> |          |
   | Tunnel 1 |          | Tunnel 2 |          | Tunnel 3 |
   |          | <- down- |          | <- down- |          |
   ------------          ------------          ------------
   (adapter)              (middle)              (adapter)

 -----------------------------------------------------------------------

Each connection is one line_t. A line has two ends, described in ww/net/line.h as Down-end <----> Up-end. The chain head (first adapter) faces the Down-end; the chain tail (last adapter) faces the Up-end. Pressure signals travel symmetrically: a blocked Down-end writer requests Pause toward the Up-end, and vice versa. They do not guarantee that all in-flight or scheduled payload stops; receivers enforce their own hard limits.

Core configuration ownership

The ordinary application owns the sole settings object and its normal WaterWall allocations. getCoreSettings() returns borrowed settings. Runtime construction borrows DNS and logging strings; consumers must not free them or destroy settings after construction. destroyCoreSettings() remains the application finalizer after workers and external activity have stopped. Parse failures use a synchronous startup error scope and local cleanup before a shutdown controller exists.

Core startup runs the optional misc.tcp-tune phase after runtime logging is ready and before node configurations are loaded. It defaults on for native Linux and off elsewhere. Regardless of misc.splice, four fixed settings apply 128 MiB socket ceilings, tcp_rmem 4096 87380 134217728 and tcp_wmem 4096 87380 134217728; the RAM profile does not select these values or change backlogs. When both misc.tcp-tune and misc.splice are enabled, the same phase also reads net.ipv4.tcp_mem and targets 3:4:6 pages with the middle threshold at approximately one quarter of total host physical RAM. Compute one integral unit from host pages divided by 16; never assume 4 KiB pages or use container quotas for this global setting. Preserve the current triple if any target threshold would decrease, and verify changed values by reading them back. Independent best-effort commands warn on failure and never fail startup. The application copies the configured flag into ww_construction_data_t.tcp_tune_enabled; runtime construction publishes GSTATE.tcp_tune_enabled before node startup. Automatic Mux socket sizing depends on chain presence independently of this flag or sysctl success. Explicit node buffer settings retain precedence. BBR has an independent admission flag; splice also gates the RAM-based tcp_mem step. RawSocket's separate skip-sysctl batch tunes backlogs and other TCP parameters; buffer and tcp_mem writes belong exclusively to core tcp-tune. See the core setting and fixed targets.

On Linux, omitted or non-positive misc.workers values use get_nprocs() to count online logical CPUs. Offline configured CPUs do not contribute to this default; the existing maximum of 254 workers still applies.

The optional Linux packed launcher reads the selected core input once and checks JSON syntax and an object root, including restricted lexical checks when selected. It does not construct settings. It releases its CRT input buffer and temporary JSON after creating a sealed snapshot, restores the ordinary executable into a separate sealed memfd, and replaces itself with execveat. No heap objects or settings pointers cross execution. The ordinary application's startup adapter consumes the inherited snapshot, closes it before runtime initialization, and passes the exact byte length to the authoritative parser. Restricted mode is selected again from the original arguments and independently validates all bytes. Normal-mode historical NUL-terminated parser acceptance remains unchanged.

The plain-C input reader always returns CRT storage, released only with waterwallStartupOptionsFreeCoreJson(). JSON allocator selection is private to the JSON component: CRT in the launcher and the normal allocator in WaterWall. Temporary input and JSON are released after full settings parsing. Node configuration parsing remains ordinary runtime work.

The core misc.mtu setting is exposed as CORE_DEFAULT_MTU, accessing global_ww_state.mtu_size, solely as a construction-time default seed. The core parser retains its 68..65535 range and DEFAULT_MTU_PROFILE fallback of 1500. IpManipulator, PacketsToConnection, TcpOverUdpClient, TcpOverUdpServer and ConnectionToPackets resolve mtu; TunDevice resolves device-mtu. Each instance strictly parses its optional override, validates the final value against its own constraints, and stores an immutable MTU. An unsupported inherited value fails construction; a supported explicit override can replace it. Runtime consumers, including lazy netifs, later KCP handles and delayed trick work, read instance state, never core MTU storage. Overrides do not write back to core state. This is neither path-MTU measurement nor chain-wide MTU propagation.

Restricted Configuration Input

--restricted-config selects a process-lifetime policy before reading core JSON. Only the startup owner selects it; JSON and environment settings cannot disable it. This is a configuration boundary, not a sandbox or a claim that Windows VPN policy, drivers, or host-controlled shutdown are ready for deployment.

In this mode, core input is limited to 2 MiB; node input and resolved node JSON are each limited to 8 MiB. Limits exclude the terminating NUL. Reads, comment stripping, substitution output, and temporary replacement serialization stay bounded. Node files still support // comments and outside-string $name$ substitutions; core JSON retains its ordinary JSON syntax. Variables and resolved JSON must consume their complete document, have valid UTF-8, unique decoded object keys, no embedded or decoded NUL, and at most 128 nested containers. The existing 512-node ceiling still applies.

Built-in node lookup remains available. An unknown type must fail before the external loader or its search paths are consulted. TunDevice rejects either script field, including null or variable-resolved fields, before storing a command or starting a device. Parser diagnostics report categories and byte positions rather than source tails; restricted parser and NodeManager diagnostics omit caller-controlled identifiers. A resolved-JSON position refers to the resolved document, and a variables position is relative to that object.

The client still owns protected inputs and artifacts, writable log paths, and its network-policy enforcement. The Windows packed public executable owns its internal process containment and lifecycle deadlines (see Part 7). In particular, the parser switch does not repair Windows driver extraction or DNS/route cleanup. Keep stdin finite and close its writers: the read is synchronous. The Windows launcher observer independently bounds stalled configuration transfer; the runtime shutdown coordinator itself cannot interrupt a pre-controller read.

API-facing configuration strings retain lexical evidence before the ordinary configuration parser discards source bytes or serializes variable substitutions. An embedded-NUL suffix must never turn into a valid C-string token or selector. These checks cover operator settings and public selector names; unrelated node settings retain their existing parsing rules. Runtime JSON error state is local to each parsing thread, since envelopes and operation bodies can be parsed on different event workers. Global JSON allocator hooks remain startup-only.

The Four Core Objects

Object	What it is
node_t	Parsed configuration plus metadata: type, next, flags, layer_group, required_padding_left, createHandle. One per node in the JSON config. Defined in ww/objects/node.h.
tunnel_t	The runtime instance of a node. Holds the twelve callback function pointers, next/prev links, tunnel state, line-state size, and chain offsets. Defined in ww/net/tunnel.h.
line_t	One connection: a normal connection line, a logical line, or a worker packet line. Holds routing context, user/auth markers, the owning worker id, a refcount, an alive flag, and every tunnel's per-line state. Defined in ww/net/line.h.
tunnel_chain_t	The ordered set of tunnels. During finalization it computes total line-state size, total left padding, and the per-worker packet lines. Defined in ww/net/chain.h.

The trusted packet-checksum capability and its active bit are node-instance flags, not line or buffer state. Only exclusive startup publishes a directly adjacent TUN/PTC pair after device setup succeeds; lazy worker routes consume that decision. See Part 4.

Tunnel State vs Line State

Each tunnel can own two kinds of state:

• Tunnel state (tstate) — one block per tunnel instance, shared by every connection that flows through it. Size is fixed at tunnelCreate() time.
• Line state (lstate) — one slot per line_t, private to this tunnel for that one connection.

Access them with the existing helpers (never reach into the structs by hand):

my_tstate_t *ts = tunnelGetState(t);        // tunnel-wide state
my_lstate_t *ls = lineGetState(line, t);    // this tunnel's state for this line

How does lineGetState() find the right slot? During chain indexing, the runtime assigns each tunnel an lstate_offset and reserves lstate_size bytes inside every line (see tunnelDefaultOnIndex in ww/net/tunnel.c). lineGetState() is then just line->tunnels_line_state + t->lstate_offset. This is why line state must be initialized in Init and treated as dead after you destroy it: the memory is shared real estate inside the line, sized once for the whole chain.

Tunnel-wide state sizes are cache-line aligned. Per-line state slots are aligned and rounded to 32 bytes (kTunnelLineStateAlignment); their types must require no greater alignment. The line-state area starts on a cache-line boundary, and the complete line allocation is rounded once to a cache line so separate lines stay isolated. When you zero line state, zero only its aligned slot, exactly as existing tunnels do:

memoryZeroAligned32(ls, tunnelGetCorrectAlignedLineStateSize(sizeof(my_lstate_t)));

Callback Directions

Direction ownership is the first thing to get right, and the most common thing to get wrong.

Flow	Meaning	Forward with
Upstream	request, outbound, forward path (toward the Up-end / next)	tunnelNextUpStream*
Downstream	response, inbound, backward path (toward the Down-end / prev)	tunnelPrevDownStream*

In an ordinary stream chain drawn left to right, normal connection lines are usually owned on the left (prev) side: their creator sends upstream Init into the rest of the chain. The right-side socket adapter borrows those lines. Persistent packet lines instead belong to the chain's worker-local packet pipeline and stay alive until chain destruction. Reverse/Bridge direction changes and internal transport lines have specific ownership contracts; see ownership and topology in Part 2.

Every tunnel exposes twelve flow callbacks, one per {event} x {direction}:

fnInitU  fnEstU  fnPayloadU  fnPauseU  fnResumeU  fnFinU    // upstream handlers
fnInitD  fnEstD  fnPayloadD  fnPauseD  fnResumeD  fnFinD    // downstream handlers

You almost never call those pointers directly. You call the forwarding helpers, which simply invoke the matching handler on the neighbor:

// "Next, upstream": calls self->next->fnPayloadU(self->next, line, buf)
tunnelNextUpStreamPayload(t, line, buf);

// "Prev, downstream": calls self->prev->fnPayloadD(self->prev, line, buf)
tunnelPrevDownStreamPayload(t, line, buf);

Lifecycle forwarding follows the same naming:

tunnelNextUpStreamInit(t, line);      // start the rest of the chain forward
tunnelNextUpStreamFinish(t, line);    // close the forward direction

tunnelPrevDownStreamEst(t, line);     // tell the backward side it is established
tunnelPrevDownStreamFinish(t, line);  // close the backward direction

tunnelCreate() rejects upstream Est and downstream Init by default with LOGF and abortProgramNow(1). A node whose contract supports either callback must explicitly install its handler. Other flow defaults pass through to the neighbor in the same direction, including payload forwarding (see ww/net/tunnel.c). packettunnelCreate() explicitly preserves forwarding for both Init and Est directions, while requiring payload overrides. Adapter constructors additionally reject callbacks on their disabled chain edge.

Do not reverse directions

Do not flip the forwarding helper for the received buffer because a server-side tunnel "feels opposite." Draw the actual chain, mark the Up-end and Down-end, and classify every emitted buffer separately. In Ping v2, the decoded original continues in the received request's callback direction. Only with the receiving endpoint's send-replies: true does a separately cloned exact Echo Reply travel in the opposite direction as an acknowledgement; false or omission skips that allocation and output. PingServer therefore sends enabled replies downstream from its upstream request handler, and PingClient sends them upstream from its downstream request handler. The endpoint settings are independent; both retain request tracking, replay suppression, and consumption of correlated replies. Reverse-direction application data starts a separate Echo Request. See Part 4.

Which Forwarding Function Do I Call?

You are handling...	and you want to send the event...	call
anything	further up the chain (toward next)	tunnelNextUpStream{Init,Payload,Est,Pause,Resume,Finish}
anything	back down the chain (toward prev)	tunnelPrevDownStream{Init,Payload,Est,Pause,Resume,Finish}

The mirror helpers tunnelUpStream* / tunnelDownStream* (without Next/Prev) start dispatch at a tunnel you hold a pointer to directly, skipping any tunnels bypassed for that line — used when a tunnel drives a branch it bound below itself (a route target, fallback, or helper branch). Resolve such a branch's real entry with tunnelGetBranchEntry(); do not assume the raw node you were handed is the callable head.

All inter-tunnel flow delivery must use these helpers. Never invoke or capture another tunnel's fn* callback fields directly; constructors may still assign handlers. See Dispatch Through Helpers, including the rule for reference guards and scheduled callbacks.

The Tunnel Lifecycle

A tunnel instance moves through a fixed set of lifecycle hooks. They are distinct from the per-connection flow callbacks above — these run at configuration, startup, and shutdown time, once per tunnel (or once per worker), not per packet.

The runtime drives them in roughly this order (see the header comment in ww/net/tunnel.h and the defaults in ww/net/tunnel.c):

createHandle(node)         // node.c -> create.c: allocate, assign callbacks, parse settings
      |
   onChain                 // bind to next/prev, insert into the chain (default walks node->next)
      |
   solve + onSolvedTopology // inspect solved snapshot; repeats after a topology change
      |
   onIndex                 // assign chain_index + lstate_offset, reserve line-state bytes
      |
   onPrepare               // pre-start preparation (was "onChainingComplete")
      |
   onStart                 // chain is fully built; start real work / bootstrap packet lines
      |
    ...                     // runtime: flow callbacks run here
      |
   onQuiesceRequest        // main thread: close admission; request producer stop; do not wait
      |
   onWorkerQuiesce         // each owner worker: detach timers/watchers/worker-affine handles
      |
   onQuiesceWait           // main thread: join/detach already-admitted external callback roots
      |
   onWorkerStop            // each owner worker: drain owned normal lines and worker resources
      |
   onStop                  // main thread: release remaining shared component resources
      |
   onDestroy               // main thread: reclaim the instance after all references are gone

What each hook is for:

Hook	Default	Typical override
createHandle	(required)	Allocate the tunnel with tunnelCreate(node, sizeof(tstate), sizeof(lstate)), assign callbacks that differ from the constructor defaults, parse and validate JSON settings, allocate per-worker resources.
onChain	tunnelDefaultOnChain builds the chain by following node->next	Rarely overridden. Adapters keep the default.
onSolvedTopology	NULL (optional)	Consume the current solved-topology snapshot and, when necessary, materialize layer-dependent private topology before indexing. The snapshot is not necessarily final. The hook may run more than once, so it must tolerate repeated calls. Return true if and only if topology changed; that immediately makes the snapshot stale and NodeManager re-solves before invoking another hook.
onIndex	tunnelDefaultOnIndex assigns the line-state offset	Runs once after topology expansion and layer resolution reach a fixed point. The final solved edge domains may be consumed here, but chain topology must not be changed. Rarely overridden.
onPrepare	no-op	Pre-start setup that needs the chain to exist.
onStart	no-op	Begin listening/connecting; bootstrap packet lines (see Part 4).
onQuiesceRequest	no-op	Main-thread, bounded, nonblocking admission closure and producer-stop request. Do not join or release line-drain resources here.
onWorkerQuiesce	no-op	On each owner worker, detach that WID's timers, watchers, and worker-affine handles before the worker acknowledges quiescence.
onQuiesceWait	no-op	On the main thread after workers quiesce, join or detach external callback roots while resources needed by line drain remain alive.
onWorkerStop	no-op	On each owner worker during drain, close only authoritative source inventories and internally owned normal lines. Chain storage order is not a neighbour-order guarantee; validate borrowed registries only at final onDestroy, after every owner drain.
onStop	no-op	On the main thread after every worker drains, release remaining shared resources; touch no worker-affine timer or watcher.
onDestroy	tunnelDestroy frees the instance	Free settings, SSL contexts, pools, then call tunnelDestroy.

Across startup configurations, NodeManager builds and finalizes every chain before running any onPrepare or onStart callback. This fixes maximum buffer headroom before core configures the empty splice inventory and its capacity ceiling. Complete splice buffers and their private pipes are created lazily on checkout. After inventory configuration, nodemanagerStartConfigs() prepares all configurations before starting producers. A preparation failure therefore starts none of them. Shutdown intent is consumed between configurations so a stop prevents subsequent preparation or startup. Packet bootstrap remains queued before readers start and worker loops remain behind their startup gate.

An external reader must remain inactive during createHandle. Constructors must not start reader threads or enable callbacks that can post packets or other work to workers immediately. Allocate inactive resources there, use onPrepare for setup requiring the finalized chain, and start reading in onStart after the required worker bootstrap has been queued. If opening a device or library handle also starts its reader, defer that operation too. A worker startup gate delays execution but does not repair messages queued before their Init. See external readers and packet startup in Part 4 for ordering and foreign-thread ownership, and Part 7 for stopping and joining external readers before releasing their state.

Most tunnels override only the lifecycle stages for resources they actually own, plus the relevant flow callbacks. Keep the chaining and indexing defaults unless the tunnel's topology contract requires something else, and verify any deviation against the solved-topology rules rather than copying a reference mechanically.

Layer results have two distinct consumption points. Use onSolvedTopology only when a tunnel must act before indexing—for example, to insert private topology whose placement depends on the solved adjacent domains. A callback that changes topology must stop using that snapshot and return true. The topology becomes final only after a complete callback pass reports no changes. Use onIndex when the tunnel only needs to read the final solution; that solution remains stable for onIndex and all later lifecycle callbacks. Static internal topology whose presence and placement are known before solving still belongs in onChain.

A Minimal Tunnel Skeleton

The template tunnel explicitly lists the available callbacks below. Production nodes assign only callbacks whose functional behavior differs from the chosen constructor defaults; inherited callbacks need no node-specific implementation file, declaration, or registration. This is the template's create.c, lightly trimmed:

tunnel_t *templateTunnelCreate(node_t *node)
{
    tunnel_t *t = tunnelCreate(node, sizeof(template_tstate_t), sizeof(template_lstate_t));
    if (! t)
    {
        return NULL;
    }

    t->fnInitU    = &templateTunnelUpStreamInit;
    t->fnEstU     = &templateTunnelUpStreamEst;
    t->fnFinU     = &templateTunnelUpStreamFinish;
    t->fnPayloadU = &templateTunnelUpStreamPayload;
    t->fnPauseU   = &templateTunnelUpStreamPause;
    t->fnResumeU  = &templateTunnelUpStreamResume;

    t->fnInitD    = &templateTunnelDownStreamInit;
    t->fnEstD     = &templateTunnelDownStreamEst;
    t->fnFinD     = &templateTunnelDownStreamFinish;
    t->fnPayloadD = &templateTunnelDownStreamPayload;
    t->fnPauseD   = &templateTunnelDownStreamPause;
    t->fnResumeD  = &templateTunnelDownStreamResume;

    t->onPrepare = &templateTunnelOnPrepair;
    t->onStart   = &templateTunnelOnStart;
    t->onStop    = &templateTunnelOnStop;
    t->onDestroy = &templateTunnelDestroy;

    return t;
}

Part 5 walks the full directory layout behind this skeleton, including node.c metadata, line_state.c, and where each callback lives.

What "Correct" Means Here

A correct tunnel change preserves every applicable item below at once:

• tunnel composability (works regardless of neighbors)
• line lifecycle rules (init in Init, destroy once, owner-only destruction)
• direction ownership (Next*/Prev* never reversed)
• buffer padding assumptions (required_padding_left honored)
• shift-buffer usage (sbufShiftLeft only with available left capacity)
• reference safety (no use-after-free of line_t or line state)
• packet-line semantics (worker-local, never destroyed at runtime)
• existing chain behavior (no regression in other layouts)

If a proposed change cannot explain how it preserves each of these, it is not ready. The rest of this guide is, in effect, a detailed expansion of that list.

Worker Thread Identity & WID

Worker thread identity in WaterWall is explicit and invalid by default:

• WID means a registered worker slot index, not an OS/application thread identifier (getTID()).
• Every registered worker owns an event loop during normal operation; there is no separate registered non-event role. getWorkersCount() counts all registered worker slots.
• Unregistered threads (Darwin/BSD TUN readers, Windows capture threads, native completion callbacks, helper threads, or test threads) observe getWID() == kInvalidWID (UINT8_MAX).
• workerWIDIsRegistered(wid) validates a slot; currentThreadIsWorker() checks the calling thread's registered identity. Neither proves ownership of another worker's state: use currentThreadOwnsWorker(wid) for that exact-owner check.
• Auxiliary threads must post ownership-bearing work through the worker message queue and must never borrow a worker-local pool directly. Raw loop custom events are reserved for loop-owned, non-owning operations.
• lwIP runs on ordinary event workers. There is no stack thread or extra worker slot; the configured maximum remains 254 event workers.
• Native/library threads use getTID() for logging/diagnostics and remain WID-invalid unless explicitly bound via workerBindCurrentThread().

Worker identity survives loop/message-queue detachment during teardown until the thread unbinds. A registered identity does not prove that a loop or pool is still available, that message admission is open, or that a normal callback is active; check the required resource and lifecycle boundary separately. A null loop never identifies a different worker role.

During normal operation, a wio_t belongs exclusively to its event worker: only that worker accesses it, writes through it, closes it, and runs its associated callbacks. Serializing foreign calls or entering target-loop admission does not transfer ownership. An independent write is an owner-thread call outside an already-admitted normal callback. Exclusive startup, explicit detach/attach, and teardown keep their existing lifecycle contracts.

Cross-worker producers submit a buffer, destination and supported adapter handle through the adapter's worker-message interface. The owner resolves its WIO and performs the send and any retirement. Cancellation may physically destroy a detached buffer, but must not access the owner's WIO or borrow its local pool. wioClose() does not dispatch a foreign caller; wioCloseAsync() requests a deferred close on the owner loop, without granting foreign WIO access.

Process resource snapshots follow the same ownership rule. Host and cgroup providers run only during synchronous global-state initialization and the worker-0 background sampler. On Linux, the sampler publishes the maximum used percentage and minimum remaining allowance across the visible finite cgroup hierarchy. effective_total_bytes is the minimum of host physical RAM and every applicable finite cgroup ceiling, independently of current usage. The diagnostic cgroup_limit_bytes can identify a different ancestor with higher pressure and must not be used as the effective total. Ambiguous or hidden constraints remain unavailable for conservative consumer fallback. Mount discovery requires canonical filesystem paths for cgroup roots, without imposing that requirement on unrelated namespace filesystem roots such as net:[inode] or mnt:[inode]. A complete cgroup v1/v2 walk with no finite limits publishes host-only RAM when the mount metadata and kernfs root inode prove that the real hierarchy root is visible. An apparent / mount or membership path alone does not establish that proof. Every sample still validates the cached boundary identity and reads ancestor accounting, so new finite limits apply without waiting for topology rediscovery. Hot worker paths consume that coherently published, bounded-atomic tuple; they never open system files, refresh stale host state, allocate, wait, or take the sampler mutex.

lwIP engine ownership

The runtime lazily creates at most one engine per event worker. Every PTC/CTP instance on that worker attaches its own netif to that engine. Stack entry must select the explicit engine and prove its owner WID and saved loop identity; private TLS selects an already-owned context and never grants foreign access. The worker binding and saved loop remain fixed throughout an entered scope. Private module-field access reuses that validated scope, with missing-scope and module-index guards in every build and repeated owner validation in Debug. Raw stack entry and PCB/netif ownership checks retain their full owner guards. Nodes detach their objects during owner drain; only runtime teardown destroys engines. Recursive packet input and timeout dispatch are refused.

TCP/UDP lists, input scratch, protocol tables and timers belong to the engine. Shared backing pools, port occupancy, generated packet IDs, node fake DNS and aggregate drain quotas retain short synchronization. Those locks must not cover protocol traversal, custom-pbuf destruction, neighbour callbacks or worker posts. Bounded engine-local caches serve TCP segments, pooled-heap classes and PTC RX wrappers. Only atomically detached free storage can be reclaimed from another engine under allocation pressure; its live protocol state remains owner-only.

Choosing a Worker-Context Helper

Never write getWID() into an affinity check, an array index, or a resource lookup. Pick the narrowest helper for the context you are already in — the helper encodes the domain, so an unregistered thread cannot slip through an equality comparison that happens to match.

Context	Use
Callback has a line_t *	lineIsOnCurrentWorker(line), lineGetWID(line), lineGetBufferPool(line), lineReuseBuffer(line, buf)
Worker-message callback has worker_t *worker	worker->wid, worker->buffer_pool, worker->loop — do not re-read TLS
Event/timer/io callback has a loop or event	getLoopWorkerWID(weventGetLoop(ev)) once, then reuse the local
Genuine event-loop callback with no line or worker	getCurrentWorker(), getCurrentWorkerWID(), getCurrentWorkerBufferPool(), getCurrentWorkerLoop()
Lifecycle callback given an explicit wid	currentThreadOwnsWorker(wid)
Code that needs worker 0 by design	explicit WID 0 plus currentThreadOwnsWorker(0); never infer it from default TLS
Externally reachable or otherwise fallible API	tryGetCurrentWorker() / currentThreadIsWorker(), then return an error, NULL, or queue
lwIP raw callback	The selected engine owner; validate explicit engine/WID ownership before stack entry
Auxiliary device thread	device-owned pools/channels only; post to an explicit event worker
Logging only	raw getWID() is fine, passed through workerWIDForLog() printed with %d (-1 means unregistered) alongside getTID()

The checked getCurrentWorker*() accessors are for internal callbacks whose contract already guarantees a registered worker. They abort rather than fall back to worker 0, so anything reachable from a device thread, a library callback, or a public API must use the predicate form and fail cleanly.

Line-bound work has a dedicated cross-thread boundary. The four lineSchedule*Task*() functions may be called from any thread, but only while the caller has an independently synchronized live handle: a reference plus a gate or registry operation that prevents lineDestroy() until the scheduler has acquired its own reference. A raw pointer, or a reference that preserves only allocation memory, cannot be used to race logical destruction. Submission does not lend the caller the target worker's state or pool.

The callback always runs on the line's event worker. Immediate and zero-delay submissions are queued; a positive delay submitted elsewhere first queues timer setup to the owner. Consequently AcceptedAsync means that ownership entered an asynchronous path, not that the timer is armed or that the task will run. See Part 2 for the complete result, cancellation, and lifetime contract.

// Line callback: affinity and pool ownership both come from the line.
if (! lineIsOnCurrentWorker(l))
{
    sendWorkerMessageForceQueueWithCleanup(lineGetWID(l), doWork, cleanup, t, l, buf);
    return;
}
lineReuseBuffer(l, buf);

// Worker-message callback: the message was delivered to exactly one worker.
static void onWorkerMessage(worker_t *worker, void *arg1, void *arg2, void *arg3)
{
    line_t *packet_line = tunnelchainGetWorkerPacketLine(tunnelGetChain(t), worker->wid);
    ...
}

// Timer callback: derive the owner from the loop, validate once, reuse it.
const wid_t wid = getLoopWorkerWID(weventGetLoop(timer));
ts->worker_timers[wid] = NULL;

// lwIP output: queue arbitrary neighbours, including on the same worker.
// PTC may directly call an immediate certified enqueue-only packet sink.
// A neighbour must not recursively inject into active stack input scratch.
queueToPacketWorker();

// Device thread: device-owned resources only, delivery by explicit WID.
sbuf_t *buf = bufferpoolGetLargeBuffer(dev->reader_buffer_pool);
sendWorkerMessageForceQueueWithCleanup(target_wid, deliver, cleanup, dev, buf, NULL);

Cross-thread buffer cleanup is not the same as recycling. reuseBuffer(buf) means "reuse on the current event worker" and rejects a caller that owns no worker-local pool. When you are not on the buffer's owner, sbufDestroy(buf) is the correct release — do not convert those paths to reuseBuffer().

Ordinary tunnel APIs (<node>TunnelApi()) run on an event worker and take ownership of their message. Consume it through tunnelapiRecycleMessage() or tunnelapiUnsupportedMessage() in ww/objects/node.h; both destroy the buffer and report kApiResultError when the caller is not an event worker, instead of borrowing worker 0's pool.

tests/worker_identity_source_policy_test.py enforces these rules: it rejects new getWorkerXxx(getWID()) nesting, raw getWID() == 0 ownership decisions, raw current-WID affinity comparisons, and tunnel-API recycle boilerplate.

Registered external node operations

ww/api/node_api.h defines opt-in operation descriptors alongside node instances; existing TunnelApi entrypoints are not automatically public. A descriptor fixes the operation name, capability, request content type, response content type and encoding, and event-worker execution policy. Register only after successful constructor setup and freeze descriptors before runtime dispatch. Remove them on normal destruction, startup rollback and callback-table rejection before the instance or its dynamic library is reclaimed. Public node/tunnel layouts and the external-node ABI do not carry this registry.

The common dispatcher receives already decoded, authenticated and authorized opaque bytes. The handler consumes an ordinary request buffer from its execution worker's pool and returns an ordinary result buffer from that same worker, or null for an empty body. It must respect its response and temporary-work allowance, perform finite synchronous work, and never retain borrowed call-context pointers, block on network I/O or wait for another worker. It must not infer permissions from command text. Response type/encoding come from the immutable descriptor.

Dispatch and reply delivery are force-queued, including same-worker calls. A detached job owns heap bytes, cancellation state, synchronized reservations, owner WID and non-reused session/request tokens. Its foreign execution and cancellation paths must never dereference a raw connection line or owner-worker state. The target worker copies a bounded result into detached storage and recycles its result locally; only the connection owner resolves the live token and materializes output from its own pool. A line reference alone does not make foreign scheduling safe against logical destruction.

Publish pending ownership before checked submission because refusal cleanup can run synchronously. Cancellation is notification/freeing only: it must not post new work, restart timers or call tunnel callbacks. Disconnect or timeout abandons a request but cannot interrupt a running handler or release its job/storage reservation early. Keep the reservation until execution/cancellation settles. Client request IDs correlate replies; they are neither lifetime tokens nor exactly-once execution or retry guarantees.

Choosing an Idle Table

Choose local_idle_table_t when item creation, lookup, refresh, removal, expiration callback, drain, and destruction all belong to one event worker. It has no synchronization or cross-worker delivery: never pass the table or one of its borrowed item handles to another worker.

Choose idle_table_t only when one table must index items owned by multiple event workers and expiration must be posted to each item's recorded owner. Its indexes are synchronized, but a returned item handle is still a borrowed, worker-owned reference; the shared table is not a general-purpose thread-safe handle container. Do not choose it merely because its name or mutex sounds safer. Shared refresh is intentionally lock-free and leaves heap reordering to the timer scan.

Both tables have the same logical lifetime rule: successful removal clears lookup visibility and invalidates every raw handle before returning. A consumer that stores a handle clears that authoritative slot before removal and never reads the item afterward. Their physical reclamation differs: local removal normally frees immediately, while a shared item can remain heap- or message-owned until safe settlement.

Idle ages measure elapsed time in cached monotonic milliseconds. The local table uses its owner's wloopNowMonotonicMS(loop). The shared table uses wloopNowMonotonicMS(getCurrentWorkerLoop()) for creation, refresh, timer scans and posted-expiration rechecks. Each operation reads only its executing worker's own cache; the table's timer loop does not supply time for other workers. All worker caches share the underlying monotonic clock domain, so their deadlines remain comparable. Time reflects the latest loop update and can lag during a long callback; idle tables provide coarse cleanup, without a fresh-sample guarantee. Clock conversions widen seconds before scaling to microseconds, so supported 32-bit time_t targets retain the full 64-bit elapsed reading without intermediate overflow. A 64-bit return type alone does not widen arithmetic performed before the return. Never compare an idle deadline with a wall-clock timestamp. Refresh only extends an existing deadline. Independent schedules can use separate items with distinct keys, canceling a completed obligation without moving another item's deadline.

Random Generation Contracts

WaterWall provides three related random-generation and selection facilities in ww/libc/wfrand.h and ww/libc/wpercentgate.h:

1. Operating-System Secure Random (secureRandomBytes):

   • Backed directly by OS entropy sources (getrandom() syscall on Linux, arc4random_buf() on BSD/Darwin, BCryptGenRandom on Windows, /dev/urandom fallback on Unix).
   • Initialized and probed by createGlobalState() before workers start.
   • Returns a recoverable failure without providing a fallback if the OS provider cannot fill the complete request.
   • Use it when a value intentionally needs an independent OS draw, including the process fast-random master key, bearer credentials, and private or rotating secret material kept outside the process fast stream.

2. Fast Random Stream Family (fastRand, fastRand32, fastRand64, getRandomBytes, fastRandRange32, fastRandJittered32, roll100):

   • This is a cryptographically secure pseudorandom generator backed by per-thread RFC 8439 ChaCha20 streams and a 256-bit OS-generated process master key.
   • It is the normal runtime choice for nonces, IVs, salts, unpredictable identifiers, local hash seeds, randomized protocol fields, and padding.
   • fastRand(), fastRand32(), and fastRand64() provide 15, 32, and 64 random bits. Use getRandomBytes() or multiple words when a value needs more; two fastRand64() calls provide 128 bits.
   • All raw integer and byte calls consume the same continuous per-thread stream. Range, jitter, and percentage helpers apply modulo mapping and are convenience samplers rather than exact-uniform cryptographic range samplers.
   • The process master key is generated once during startup by frandGlobalInit().
   • Each thread derives an independent, contiguous ChaCha20 stream using a 96-bit nonce containing domain tag "WWFR" and an atomically allocated 64-bit stream identifier.
   • Outputs are buffered in a thread-local 64-byte block cache; after thread initialization, ordinary calls use only TLS state and perform no syscall, lock, allocation, or atomic operation.
   • Requests larger than the cache drain its remainder and generate whole 64-byte ChaCha20 blocks directly into caller memory.
   • Stream renewal: if a thread consumes $2^32$ blocks (counter exhaustion), it atomically requests a new stream identifier and initializes a new nonce without making an OS syscall.
   • Fork safety: the POSIX pthread_atfork child handler uses only lock-free state invalidation. The child treats the inherited process key and surviving TLS stream as unusable, then reseeds from OS entropy on its next random request.
   • Lifecycle: frandGlobalInit() is called during startup immediately after secure random initialization; frandInit() initializes worker TLS state; frandThreadCleanup() erases thread state on exit; frandGlobalCleanup() clears process state in destroyGlobalState() and startup rollback paths. Calling random functions before global initialization or after global cleanup is fatal.

3. Non-Cryptographic Percentage Gates (noncrypto_percent_gate_t, nonCryptoPercentGateInit, nonCryptoPercentGateStep):

   • This is a purpose-specific, caller-owned PCG32 stream for hot decisions that need random-like percentage behavior but do not need unpredictability or cryptographic security. It does not expose general-purpose random integers or bytes.
   • Initialize each gate once with its fixed percentage and a seed. Initialization precomputes the comparison cutoff; an ordinary step performs only the PCG state update, output permutation, and cutoff comparison. It makes no syscall, allocation, lock, or atomic operation.
   • Keep one independently seeded gate for every logical decision stream and ownership domain. Unrelated decisions, directions, node instances, or workers must not share a gate: shared state couples their observed rates. A gate has one owner and must not be accessed concurrently.
   • The sequence deliberately permits runs of selected and rejected events and approaches the configured percentage over time. It is not an exact-frequency scheduler, an exact-uniform cryptographic sampler, or a source of security.
   • A seed may be drawn once from fastRand64() to keep gates from starting in lockstep, but that does not make PCG output secure. State disclosure or sufficient observation may reveal future decisions.
   • Use this facility only when predictability cannot affect authentication, association, abuse resistance, wire camouflage, classifier resistance, protocol identifiers, nonces, secrets, hash-flood defenses, or another adversarial boundary. Those purposes remain on the fast CSPRNG family.
   • Zero and one hundred percent are exact and do not advance gate state. Percentages from one through ninety-nine consume one PCG output per decision.

The lwIP port uses this same fast CSPRNG on every host. LWIP_RAND() delegates to fastRand32() and therefore requires initialized fast-RNG TLS, but it does not require an event-worker WID. Shared stack bootstrap runs after main-thread RNG initialization and before worker spawning. Protocol execution uses each ordinary worker's initialized TLS stream, which is erased during that worker's teardown.

TCP initial sequence numbers use a separate immutable 32-byte process secret. It is generated from getRandomBytes() after the fast RNG and crypto runtime are ready and before lwIP starts. The hook adds the millisecond clock converted to nominal 4-microsecond units to a keyed BLAKE2s result over a canonical address-family, local/remote address, and local/remote port tuple. Shutdown erases the secret only after all engines and their TCP PCBs are gone and workers have joined; startup rollback follows the same ordering.

Glossary

Term	Meaning
Adapter	A chain-head or chain-end tunnel that owns an OS resource (socket, TUN, raw socket). Resource ownership does not imply line_t ownership: a source such as TcpListener creates normal lines, while a tail such as TcpConnector borrows them.
Middle tunnel	A non-adapter tunnel that transforms callbacks/payloads and must stay composable.
Line	One line_t; a single connection (or a worker packet line). Holds all tunnels' per-line state.
Upstream	The forward/outbound direction, toward next (the Up-end).
Downstream	The backward/inbound direction, toward prev (the Down-end).
tstate	Tunnel state: per-instance, shared across all lines.
lstate	Line state: per-line, private to one tunnel.
Packet line	A persistent, per-worker line_t used by layer-3 chains; not a per-connection line.
WID	Registered worker slot index (0 .. WORKERS_COUNT - 1). Is kInvalidWID on unregistered threads.
kInvalidWID	Sentinel value (UINT8_MAX) representing an unregistered or auxiliary thread.
Re-entrancy	An inter-tunnel callback synchronously calling back into your tunnel, possibly closing the line before control returns.
Reflection	Incorrectly forwarding a callback back toward a direction that has already finished. A common crash source.

---

Continue to Part 2: Lines, Callbacks & Lifetime Safety.