WaterWall Developer Guide

This is a source-oriented guide for people and coding agents who implement, fix, or review tunnels under tunnels/. It is split into six parts. This page (Part 1) builds the mental model. The remaining parts go deep on the contracts you must not break.

WaterWall is a chain-based tunneling runtime, so a correct tunnel is never "just" a parser or an encoder. A tunnel must preserve callback direction, line lifetime, per-line state, buffer padding, packet-line behavior, and composability with whatever nodes happen to sit before or after it in a user's configuration.

The single most important rule

Do not design a new lifecycle model for one tunnel. Start from the runtime contracts in ww/net/, then copy the shape of a mature tunnel that already solves a similar problem. In this codebase, correctness comes from matching the existing lifecycle pattern, not from inventing a more general one.

How This Guide Is Organized

Part	Topic	When you need it
Part 1 (this page)	Overview, core objects, directions, lifecycle	Always read first
Part 2: Lines, Callbacks & Lifetime Safety	Init/Est/Payload/Pause/Resume/Finish, re-entrancy, locking, clean close	Any change that touches connection flow
Part 3: Buffers, Padding & Shift Buffers	sbuf_t, buffer pools, required_padding_left, prepending headers	Any change that frames, prepends, or rewrites payload
Part 4: Packet Lines & Packet Tunnels	Packet-line semantics, pure packet tunnels, packet/stream bridges	Any layer-3 / packet work
Part 5: Anatomy of a Tunnel & Workflow	Directory layout, node.c metadata, create.c, line state, HTTP rules	When you add or restructure a tunnel
Part 6: Building, Testing & Reviewing	CMake presets, ctest, validation, review checklist, agent output	Before you call a change "done"

Read These Files First

Before changing tunnel behavior, read the source that owns the contract. These are short, dense, and authoritative.

Area	Files
Tunnel callbacks and chaining	ww/net/tunnel.h, ww/net/tunnel.c
Line lifetime and per-line state	ww/net/line.h, ww/net/line.c
Chain finalization and packet lines	ww/net/chain.h, ww/net/chain.c
Packet-tunnel defaults	ww/net/packet_tunnel.h, ww/net/packet_tunnel.c
Shift buffers and padding	ww/bufio/shiftbuffer.h, ww/bufio/shiftbuffer.c
Buffer pools	ww/bufio/buffer_pool.h, ww/bufio/buffer_pool.c
Node metadata	ww/objects/node.h

Good Reference Tunnels

Pick the closest existing pattern and stay near it. These tunnels are mature and exercise specific contracts cleanly:

Reference	Read it for
TcpListener, TcpConnector	Adapter behavior: a real socket on one end, line creation/destruction
TlsClient, EncryptionClient	Stateful protocol wrapping, per-line state, clean finish with final bytes
MuxClient	Internal line ownership, parent/child lines, re-entrant safety
PacketsToStream, StreamToPackets, PacketsToConnection	Packet-to-stream boundaries and packet-line anchored bridges
PingClient, PingServer	Direct paired packet transforms and same-direction forwarding
template	The minimal skeleton every tunnel starts from

The Runtime Model

WaterWall composes tunnel instances into an ordered chain. A typical stream chain looks like this:

TcpListener -> ObfuscatorClient -> TlsClient -> TcpConnector

The first and last nodes are usually adapters. They own an operating-system resource: a TCP socket, a UDP socket, a TUN device, a raw socket, and so on. They are the only nodes that read from or write to the outside world.

Everything between the adapters is a middle tunnel. Middle tunnels must stay composable: they transform callbacks and payloads without assuming which adapter is on either side. A middle tunnel that only works when a specific adapter sits next to it is broken, even if its own tests pass.

 -------------- chain --------------------------------------------------

   ------------          ------------          ------------
   |          | -- up -> |          | -- up -> |          |
   | Tunnel 1 |          | Tunnel 2 |          | Tunnel 3 |
   |          | <- down- |          | <- down- |          |
   ------------          ------------          ------------
   (adapter)              (middle)              (adapter)

 -----------------------------------------------------------------------

Each connection is one line_t. A line has two ends, described in ww/net/line.h as Down-end <----> Up-end. The chain head (first adapter) faces the Down-end; the chain tail (last adapter) faces the Up-end. Backpressure is symmetric: if a write on the Down-end blocks, the Up-end is paused, and vice versa.

The Four Core Objects

Object	What it is
node_t	Parsed configuration plus metadata: type, next, flags, layer_group, required_padding_left, createHandle. One per node in the JSON config. Defined in ww/objects/node.h.
tunnel_t	The runtime instance of a node. Holds the twelve callback function pointers, next/prev links, tunnel state, line-state size, and chain offsets. Defined in ww/net/tunnel.h.
line_t	One connection: a normal connection line, a logical line, or a worker packet line. Holds routing context, user/auth markers, the owning worker id, a refcount, an alive flag, and every tunnel's per-line state. Defined in ww/net/line.h.
tunnel_chain_t	The ordered set of tunnels. During finalization it computes total line-state size, total left padding, and the per-worker packet lines. Defined in ww/net/chain.h.

Tunnel State vs Line State

Each tunnel can own two kinds of state:

• Tunnel state (tstate) — one block per tunnel instance, shared by every connection that flows through it. Size is fixed at tunnelCreate() time.
• Line state (lstate) — one slot per line_t, private to this tunnel for that one connection.

Access them with the existing helpers (never reach into the structs by hand):

my_tstate_t *ts = tunnelGetState(t);        // tunnel-wide state
my_lstate_t *ls = lineGetState(line, t);    // this tunnel's state for this line

How does lineGetState() find the right slot? During chain indexing, the runtime assigns each tunnel an lstate_offset and reserves lstate_size bytes inside every line (see tunnelDefaultOnIndex in ww/net/tunnel.c). lineGetState() is then just line->tunnels_line_state + t->lstate_offset. This is why line state must be initialized in Init and treated as dead after you destroy it: the memory is shared real estate inside the line, sized once for the whole chain.

State sizes are cache-line aligned. When you zero line state, zero the aligned region, exactly as existing tunnels do:

memoryZeroAligned32(ls, tunnelGetCorrectAlignedLineStateSize(sizeof(my_lstate_t)));

Callback Directions

Direction ownership is the first thing to get right, and the most common thing to get wrong.

Flow	Meaning	Forward with
Upstream	request, outbound, forward path (toward the Up-end / next)	tunnelNextUpStream*
Downstream	response, inbound, backward path (toward the Down-end / prev)	tunnelPrevDownStream*

Every tunnel exposes twelve flow callbacks, one per {event} x {direction}:

fnInitU  fnEstU  fnPayloadU  fnPauseU  fnResumeU  fnFinU    // upstream handlers
fnInitD  fnEstD  fnPayloadD  fnPauseD  fnResumeD  fnFinD    // downstream handlers

You almost never call those pointers directly. You call the forwarding helpers, which simply invoke the matching handler on the neighbor:

// "Next, upstream": calls self->next->fnPayloadU(self->next, line, buf)
tunnelNextUpStreamPayload(t, line, buf);

// "Prev, downstream": calls self->prev->fnPayloadD(self->prev, line, buf)
tunnelPrevDownStreamPayload(t, line, buf);

Lifecycle forwarding follows the same naming:

tunnelNextUpStreamInit(t, line);      // start the rest of the chain forward
tunnelNextUpStreamFinish(t, line);    // close the forward direction

tunnelPrevDownStreamEst(t, line);     // tell the backward side it is established
tunnelPrevDownStreamFinish(t, line);  // close the backward direction

A tunnel that does not override a given callback gets the framework default, which is a pure pass-through to the neighbor in the same direction (see tunnelDefaultUpStreamPayload / tunnelDefaultDownStreamPayload in ww/net/tunnel.c). So an "obfuscation only on upstream" tunnel can leave its downstream payload handler at the default and let bytes flow straight through.

Do not reverse directions

Do not flip forwarding helpers because a server-side tunnel "feels opposite." Draw the actual chain, mark the Up-end and Down-end, then decide separately whether that callback encodes or decodes. In a direct PingClient -> PingServer pair, both nodes forward the request upstream even though one encodes and the other decodes; the inverse transformations both forward the response downstream. See Part 4.

Which Forwarding Function Do I Call?

You are handling...	and you want to send the event...	call
anything	further up the chain (toward next)	tunnelNextUpStream{Init,Payload,Est,Pause,Resume,Finish}
anything	back down the chain (toward prev)	tunnelPrevDownStream{Init,Payload,Est,Pause,Resume,Finish}

The mirror helpers tunnelUpStream* / tunnelDownStream* (without Next/Prev) invoke a handler on a tunnel you hold a pointer to directly — used when a tunnel drives a branch it bound below itself (a route target, fallback, or helper branch). Resolve such a branch's real entry with tunnelGetBranchEntry(); do not assume the raw node you were handed is the callable head.

The Tunnel Lifecycle

A tunnel instance moves through a fixed set of lifecycle hooks. They are distinct from the per-connection flow callbacks above — these run at configuration, startup, and shutdown time, once per tunnel (or once per worker), not per packet.

The runtime drives them in roughly this order (see the header comment in ww/net/tunnel.h and the defaults in ww/net/tunnel.c):

createHandle(node)         // node.c -> create.c: allocate, assign callbacks, parse settings
      |
   onChain                 // bind to next/prev, insert into the chain (default walks node->next)
      |
   onIndex                 // assign chain_index + lstate_offset, reserve line-state bytes
      |
   onPrepare               // pre-start preparation (was "onChainingComplete")
      |
   onStart                 // chain is fully built; start real work / bootstrap packet lines
      |
    ...                     // runtime: flow callbacks run here
      |
   onStop  /  onWorkerStop // stop tunnel / stop per-worker resources
      |
   onDestroy               // free tunnel and its state

What each hook is for:

Hook	Default	Typical override
createHandle	(required)	Allocate the tunnel with tunnelCreate(node, sizeof(tstate), sizeof(lstate)), wire up the twelve callbacks, parse and validate JSON settings, allocate per-worker resources.
onChain	tunnelDefaultOnChain builds the chain by following node->next	Rarely overridden. Adapters keep the default.
onIndex	tunnelDefaultOnIndex assigns the line-state offset	Rarely overridden.
onPrepare	no-op	Pre-start setup that needs the chain to exist.
onStart	no-op	Begin listening/connecting; bootstrap packet lines (see Part 4).
onStop	no-op	Stop accepting work, begin teardown.
onWorkerStop	no-op	Release worker-local resources for one worker id.
onDestroy	tunnelDestroy frees the instance	Free settings, SSL contexts, pools, then call tunnelDestroy.

Most tunnels only override onPrepare, onStart, onStop, and onDestroy, plus the flow callbacks. The chaining and indexing defaults are correct for almost everything; leave them alone unless the source of a reference tunnel proves you need otherwise.

A Minimal Tunnel Skeleton

Every tunnel starts from the same shape (this is the template tunnel's create.c, lightly trimmed):

tunnel_t *templateTunnelCreate(node_t *node)
{
    tunnel_t *t = tunnelCreate(node, sizeof(template_tstate_t), sizeof(template_lstate_t));

    t->fnInitU    = &templateTunnelUpStreamInit;
    t->fnEstU     = &templateTunnelUpStreamEst;
    t->fnFinU     = &templateTunnelUpStreamFinish;
    t->fnPayloadU = &templateTunnelUpStreamPayload;
    t->fnPauseU   = &templateTunnelUpStreamPause;
    t->fnResumeU  = &templateTunnelUpStreamResume;

    t->fnInitD    = &templateTunnelDownStreamInit;
    t->fnEstD     = &templateTunnelDownStreamEst;
    t->fnFinD     = &templateTunnelDownStreamFinish;
    t->fnPayloadD = &templateTunnelDownStreamPayload;
    t->fnPauseD   = &templateTunnelDownStreamPause;
    t->fnResumeD  = &templateTunnelDownStreamResume;

    t->onPrepare = &templateTunnelOnPrepair;
    t->onStart   = &templateTunnelOnStart;
    t->onStop    = &templateTunnelOnStop;
    t->onDestroy = &templateTunnelDestroy;

    return t;
}

Part 5 walks the full directory layout behind this skeleton, including node.c metadata, line_state.c, and where each callback lives.

What "Correct" Means Here

A correct tunnel change preserves all of the following at once:

• tunnel composability (works regardless of neighbors)
• line lifecycle rules (init in Init, destroy once, owner-only destruction)
• direction ownership (Next*/Prev* never reversed)
• buffer padding assumptions (required_padding_left honored)
• shift-buffer usage (sbufShiftLeft only with available left capacity)
• lock / refcount safety (no use-after-free of line_t or line state)
• packet-line semantics (worker-local, never destroyed at runtime)
• existing chain behavior (no regression in other layouts)

If a proposed change cannot explain how it preserves each of these, it is not ready. The rest of this guide is, in effect, a detailed expansion of that list.

Glossary

Term	Meaning
Adapter	A chain-head or chain-end tunnel that owns an OS resource (socket, TUN, raw socket). Creates and destroys lines.
Middle tunnel	A non-adapter tunnel that transforms callbacks/payloads and must stay composable.
Line	One line_t; a single connection (or a worker packet line). Holds all tunnels' per-line state.
Upstream	The forward/outbound direction, toward next (the Up-end).
Downstream	The backward/inbound direction, toward prev (the Down-end).
tstate	Tunnel state: per-instance, shared across all lines.
lstate	Line state: per-line, private to one tunnel.
Packet line	A persistent, per-worker line_t used by layer-3 chains; not a per-connection line.
Re-entrancy	An inter-tunnel callback synchronously calling back into your tunnel, possibly closing the line before control returns.
Reflection	Incorrectly forwarding a callback back toward a direction that has already finished. A common crash source.

---

Continue to Part 2: Lines, Callbacks & Lifetime Safety.