Part 4: Packet Lines & Packet Tunnels

WaterWall handles layer-3 packet flows (TUN devices, WireGuard-style paths, raw packet processing) as well as ordinary streams. The packet model reuses the line_t type but applies a completely different lifecycle. Treating a packet line like a per-connection line is one of the most damaging mistakes you can make in this codebase.

Read this part before touching anything under a layer-3 node. Source of truth: ww/net/chain.c, ww/net/packet_tunnel.c, ww/net/line.h, and the packet tunnels themselves.

What a Packet Line Is

A packet line is a real line_t, but unlike a connection line it is:

allocated once per worker for any chain that contains a layer-3 node,
shared by every packet that worker processes,
not tied to a TCP/UDP connection lifecycle,
not closed during normal runtime,
destroyed only when the chain itself is destroyed.

The source is explicit. In tunnelchainFinalize() (ww/net/chain.c), when the chain contains a packet node, one packet line is created per worker:

tc->packet_lines = (line_t **) memoryAllocate(sizeof(line_t) * tc->workers_count);

for (wid_t i = 0; i < tc->workers_count; i++)
{
    // ... create the worker line pool ...
    if (tc->contains_packet_node)
    {
        tc->packet_lines[i] = lineCreateForWorker(0, tc->line_pools, i);
    }
}

And they are torn down only in tunnelchainDestroy():

for (uint32_t i = 0; i < tc->workers_count; i++)
{
    if (tc->packet_lines && tc->packet_lines[i])
    {
        lineDestroy(tc->packet_lines[i]);
    }
}

Reach a worker's packet line with tunnelchainGetWorkerPacketLine(chain, wid).

Never destroy a packet line at runtime

Do not call lineDestroy() on a packet line during normal operation, and never design logic that expects packet lines to be recreated on Finish. Some packet adapters (TunDevice, UdpStatelessSocket) assert in debug builds that the packet line is still alive after forwarding a packet. If your change can let a packet line die at runtime, that is almost certainly a bug.

Runtime `Finish` on a Packet Line

Part 2 requires an owner's Finish handler for a normal line to return with ! lineIsAlive(line). Packet lines are excluded from that postcondition, without exception. A packet line is owned by the chain, not by any tunnel, and tunnelchainDestroy() is the only place that may ever destroy it. That part is absolute: whatever else your handler does, it must never call lineDestroy() on a packet line.

What a Finish on a packet line means, though, depends entirely on the hop that receives it, the direction it arrived from, and the role that handler plays. There are three categories. Chain position is useful context, but it is not enough by itself to select one.

Category 1 — packet-lifecycle anchor: reject impossible close

A packet adapter or bridge may anchor the worker's packet-pipeline state on the packet line and have no notion of that role closing. Here “adapter” means a packet-pipeline boundary or lifecycle endpoint, not a tunnel that owns the line_t. Nothing in a correct chain finishes that exact role, so a Finish arriving in that handler and direction means a neighbour broke the contract:

never call lineDestroy();
fail loudly — LOGF(...) then abortProgramNow(1), the project's packet-line contract-violation policy;
never convert it into normal connection teardown;
leave the real destruction to tunnelchainDestroy().

That is exactly what the anchoring adapters do:

void tundeviceTunnelDownStreamFinish(tunnel_t *t, line_t *l)
{
    discard t;
    LOGF("TunDevice: unexpected downstream Finish on worker packet line %u", (unsigned int) lineGetWID(l));
    abortProgramNow(1);
}

Examples include TunDevice and RawSocket; the upstream packet sides of PacketsToConnection and PacketsToStream; the downstream packet side of StreamToPackets; PacketSplitStream; the packet modes of TesterClient and TesterServer; and the packet-line role of WireGuardDevice. These are per-handler classifications, not labels for the whole tunnel: the same bridge may own normal lines on its other side and close those normally.

tests/unittests/packet_adapter_finish_test.c proves the covered anchoring calls do not return.

Category 2 — transparent middle transform: forward lifecycle

A middle tunnel that transforms packet payloads does not thereby own or terminate the line lifecycle. It must forward Init, Est, Pause, Resume, Finish, and any other callback unrelated to its transform in the same direction:

upstream callbacks continue with tunnelNextUpStream*;
downstream callbacks continue with tunnelPrevDownStream*;
a received callback is never reflected toward its sender.

PingClient inherits these lifecycle pass-throughs from packettunnelCreate(); PingServer installs equivalent explicit Finish forwarding. Keeping the transform separate from the lifecycle allows flexible chains such as:

UdpListener <-> PingServer <-> UdpConnector

Here PingServer can decapsulate the ICMP-wrapped packet on the relevant payload path while the ordinary UDP line lifecycle continues through it unchanged. packettunnelCreate() therefore keeps the standard lifecycle pass-throughs and only makes unimplemented packet payload callbacks fatal.

Category 3 — intentional terminal absorber: handle locally and document

A packet line is still a line_t, and a chain may legally terminate one at an ordinary stream adapter. That adapter borrows the packet line exactly as it would borrow a connection line, and it closes its own side for its own reasons.

The shipped example is PacketSender -> UdpConnector, which tests/cases/udp_connector_listener_packet_loss_multiworker exercises:

PacketSender ──(worker packet line)──> UdpConnector ──> network

UdpConnector opens a UDP socket per line and expires it when the flow goes idle. Expiry sends tunnelPrevDownStreamFinish() down the worker packet line, and it lands on packetsenderTunnelDownStreamFinish(). That is not a violation and must not abort. PacketSender owns no per-line state and has no prev, so it absorbs the close without propagating it or destroying the packet line. It must still stop that worker's timer and send loop because receiving Finish permanently closes the upstream destination.

This classification is directional. PacketSender downstream is an absorber, while its upstream handler remains fatal because the chain head has no prev that could legally send it a callback. PacketReceiver is another intentional absorber: it owns no per-line state, may terminate either end of a chain, and reports completion through its timer and report path rather than Finish.

Propagate the close, release what you own, report a violation — or state in the handler that you own no per-line state and have no onward direction, so absorbing the close really is the whole job. A body that is nothing but discard statements does not distinguish the last case from a bug.

The legitimate absorbers today are BlackHole (a chain-end sink that borrows every line), PacketReceiver (a chain end that reports by timer and file) and PacketSender downstream (Category 3 above).

tests/line_ownership_policy_test.py walks every tunnels/*/{upstream,downstream}/fin.c and rejects a handler whose whole body is discard statements unless it appears in SILENT_FINISH_ALLOWED with a written rationale, so a lost propagation or a lost owner close cannot arrive looking like a deliberate no-op.

So the question to ask is never only “is this a packet line?” or “is this tunnel at a chain end?” It is “what role does this exact handler and direction play, what does the tunnel keep on this line, and who is still waiting on it?”

Dual-role handlers must check the exact role first

Some nodes see both a persistent packet line and normal lines they own. The handler must decide which one it was given before doing anything:

WireGuardDevice is a packettunnelCreate() tunnel on its packet side, but it also creates one normal transport line per worker for its encrypted side. A Finish on a transport line is an ordinary owned-line close (destroy the slot, lineDestroy()); a Finish on the packet line is a violation and aborts.
PacketsToStream and PacketsToConnection anchor bridge state on the packet line while owning the normal lines they create behind it.

Use the chain to identify the role — never a name or a guess:

if (tunnelchainIsWorkerPacketLine(tunnelGetChain(t), l))
{
    LOGF("MyNode: unexpected Finish on worker packet line %u", (unsigned int) lineGetWID(l));
    abortProgramNow(1);
}
/* from here on this is a normal line: apply the Part 2 owner postcondition */

Any intentional classification that differs by line role or direction must be documented at the handler and covered by a regression test.

Packet Lines Are Shared Worker State

Because one packet line serves many unrelated packets on a worker, anything stored on it is worker-local scratch state, not per-connection identity. Be especially careful with:

routing_context.src_ctx and routing_context.dest_ctx
recalculate_checksum (a per-packet scratch flag, see lineSetRecalculateChecksum)
any tunnel line state stored on the packet line

These fields are overwritten packet-to-packet. Do not treat them as a stable peer, flow, or session. If you store state on a packet line, that state must make sense as one of:

per-worker state,
packet-pipeline scratch state,
bridge state anchored to the worker packet side.

If you need real per-flow or per-connection lifecycle behind packet traffic, create normal lines behind the packet side — do not try to make the packet line behave like a connection.

Two Kinds of Packet Tunnels

Not every tunnel that touches packet lines is the same kind of tunnel. There are two distinct categories, and they have different rules.

1. Pure Packet Tunnels (`packettunnelCreate`)

Some tunnels are created with packettunnelCreate() instead of tunnelCreate(). From ww/net/packet_tunnel.c:

tunnel_t *packettunnelCreate(node_t *node, uint16_t tstate_size, uint16_t lstate_size)
{
    assert(lstate_size == 0); // packet tunnels don't have lines
    tunnel_t *t = tunnelCreate(node, tstate_size, 0);
    // ... installs packet-tunnel default callbacks ...
}

Key facts:

packettunnelCreate() asserts lstate_size == 0. Pure packet tunnels have no per-line state.
They are expected to override the packet payload callbacks.
The default payload callbacks are intentionally invalid: they call LOGF(...) and abortProgramNow(1), so they crash loudly if you forget to override them:

static void packettunnelDefaultUpStreamPayload(tunnel_t *self, line_t *line, sbuf_t *payload)
{
    LOGF("Unexpected call to default up stream payload for a packet tunnel, "
         "this function must be overridden");
    abortProgramNow(1);
}

Init, Est, Pause, Resume, and Finish inherit the standard same-direction lifecycle pass-throughs. An individual tunnel overrides one only when its exact handler role requires an anchoring rejection, explicit forwarding, or an intentional terminal absorption as classified above.

Examples: IpOverrider, IpManipulator, PingClient, PingServer, WireGuardDevice.

For pure packet tunnels:

Do not add connection-style line lifecycle logic casually.
lstate_size == 0 does not mean "packet lines only". A kNodeLayerAnything packet tunnel also carries normal connection lines.
Some default callbacks deliberately abort; know which ones you must override.

2. Packet-Line Anchored Bridges (`tunnelCreate`)

Other tunnels are ordinary tunnelCreate() tunnels that use the worker packet line as an anchor for worker-local bridge state. They cross the boundary between packet-oriented runtime and connection-oriented runtime.

Examples and what they anchor:

Bridge	Role
`PacketsToStream`	packet line to a worker-local stream-facing line; stores that line and a read buffer
`StreamToPackets`	a stream line to the packet line; stores the active stream line and read buffer per worker
`PacketsToConnection`	packet traffic to lwIP to normal WaterWall connection lines behind the packet side
`PacketSplitStream`	splits packet traffic across stream paths

State stored here is worker-local bridge state — still not per-connection state. If you need closable per-connection behavior behind packet traffic, create or manage normal lines behind the packet line instead of closing the packet line.

Packet-Line Initialization

Packet lines are initialized differently from connection lines.

For a chain whose head is a layer-3 packet node, the node manager sends one upstream Init on each worker's packet line during startup — once per worker, not once per packet. So packet-line Init is a worker/bootstrap event, not a connection-open event.

But if a bridge relies on packet lines while the chain head is not itself a layer-3 packet node, that automatic init does not happen, and the bridge must trigger it. StreamToPackets does exactly this in onStart(), queueing the bootstrap onto every worker so packet-side tunnels see their Init after startup rather than re-entering init paths inline:

static void streamtopacketsQueueWorkerPacketInit(void *worker, void *arg1, void *arg2, void *arg3)
{
    tunnel_t *t = arg1;
    line_t   *l = tunnelchainGetWorkerPacketLine(tunnelGetChain(t), getWID());
    tunnelNextUpStreamInit(t, l);
    assert(lineIsAlive(l));
}

void streamtopacketsTunnelOnStart(tunnel_t *t)
{
    for (wid_t wi = 0; wi < getWorkersCount(); wi++)
    {
        sendWorkerMessageForceQueue(wi, streamtopacketsQueueWorkerPacketInit, t, NULL, NULL);
    }
}

So if you create or modify a packet bridge: do not assume packet-line init happened automatically. Verify whether the init comes from the node manager (chain head is layer-3) or whether your tunnel must trigger it in onStart().

Paired Tunnel Direction (the PingClient / PingServer Rule)

Transform role and callback forwarding direction are separate decisions. The direct pair is:

plain request -> PingClient -> PingServer -> plain request
plain response <- PingClient <- PingServer <- plain response

On upstream payload, PingClient encapsulates and PingServer decapsulates. Both forward with tunnelNextUpStreamPayload().
On downstream payload, PingServer encapsulates and PingClient decapsulates. Both forward with tunnelPrevDownStreamPayload().

Calling a downstream helper from PingServer's upstream callback would reflect the request toward its sender. Likewise, encapsulating in PingServer's upstream callback would transform the request twice.

The procedure for any packet tunnel:

Draw the packet flow with arrows.
Place the encode/decode step in the callback direction that flow requires.
Use tunnelNextUpStreamPayload() only for upstream forwarding and tunnelPrevDownStreamPayload() only for downstream forwarding.
Do not treat packet lines as per-connection lines.
Do not call lineDestroy() on packet lines at runtime.
Do not add stream-style lifecycle to a pure packet tunnel unless its design requires it.

Testing Packet Tunnels

Packet tunnel tests must exercise the real intended direction. The canonical Ping test is one direct packet chain:

TesterClient(packet-mode=true)
    -> PingClient
    -> PingServer
    -> TesterServer(packet-mode=true)

The upstream request must be restored before TesterServer sees it, and the downstream response must be restored before TesterClient sees it. Add at least one test that would fail if PingServer upstream encapsulated instead of decapsulating.

The packet-line Finish contract has its own coverage, and a new packet-side handler belongs in it:

tests/unittests/packet_adapter_finish_test.c calls the anchoring adapters' Finish handlers for real in a forked child and requires the child to die rather than return.
tests/unittests/packetsender_orderly_shutdown_test.c proves that PacketSender cancels a pending worker timer and stops a ready multi-packet batch when a payload emits downstream Finish re-entrantly, without destroying the worker packet line.
tests/line_ownership_policy_test.py fails if a fin.c becomes an unexplained bare return, or if a contract test loses its ctest registration.

Rules for Packet-Oriented Changes (Summary)

Do not close packet lines during runtime, and never lineDestroy() one.
The owner Finish postcondition from Part 2 does not apply to packet lines.
Classify each packet-line Finish handler and direction as one of three roles: an anchor rejects an impossible close with LOGF + abortProgramNow(1); a transparent middle transform forwards it in the same direction; an intentional terminal absorber documents why there is nothing to forward or release.
PacketSender is direction-specific: downstream absorbs the legitimate close from UdpConnector, while upstream remains fatal. PacketReceiver is also an intentional terminal absorber.
A handler that can receive both a packet line and a normal line must branch on the exact role (tunnelchainIsWorkerPacketLine()) before any cleanup.
packettunnelCreate() only means lstate_size == 0; several such nodes are kNodeLayerAnything and carry ordinary connection lines too, which is why its lifecycle callbacks stay plain pass-throughs.
A Finish handler that does nothing must say why: propagate, release, report, or document that you own no per-line state and have no onward direction.
Do not run normal per-connection Finish logic on packet lines unless the source clearly expects it.
Treat packet-line state as worker-local shared state.
Treat routing context and recalculate_checksum as mutable per-packet metadata, never as durable session identity.
For per-flow lifecycle, create normal lines behind the packet side.
A packettunnelCreate() tunnel has no per-line state (lstate_size == 0).
A bridge that anchors state on the packet line keeps that state across many packets on the worker.
If your code depends on packet-line init, verify where that init comes from.

The single most common mistake to avoid:

Using packet lines as if they were normal connection lines with an ordinary
Init -> Payload -> Finish per-connection lifecycle. They are not.

Continue to Part 5: Anatomy of a Tunnel & Workflow.

What a Packet Line Is​

Runtime Finish on a Packet Line​

Category 1 — packet-lifecycle anchor: reject impossible close​

Category 2 — transparent middle transform: forward lifecycle​

Category 3 — intentional terminal absorber: handle locally and document​

Dual-role handlers must check the exact role first​

Packet Lines Are Shared Worker State​

Two Kinds of Packet Tunnels​

1. Pure Packet Tunnels (packettunnelCreate)​

2. Packet-Line Anchored Bridges (tunnelCreate)​

Packet-Line Initialization​

Paired Tunnel Direction (the PingClient / PingServer Rule)​

Testing Packet Tunnels​

Rules for Packet-Oriented Changes (Summary)​