Part 4: Packet Lines & Packet Tunnels

WaterWall handles layer-3 packet flows (TUN devices, WireGuard-style paths, raw packet processing) as well as ordinary streams. The packet model reuses the line_t type but applies a completely different lifecycle. Treating a packet line like a per-connection line is one of the most damaging mistakes you can make in this codebase.

Read this part before touching anything under a layer-3 node. Source of truth: ww/net/chain.c, ww/net/packet_tunnel.c, ww/net/line.h, and the packet tunnels themselves.

What a Packet Line Is

A packet line is a real line_t, but unlike a connection line it is:

• allocated once per worker for any chain that contains a layer-3 node,
• shared by every packet that worker processes,
• not tied to a TCP/UDP connection lifecycle,
• not closed during normal runtime,
• destroyed only when the chain itself is destroyed.

"Worker-owned" here means persistent, worker-local packet-pipeline state. It does not make a node on the left or right its connection owner. Normal lines created inside a packet chain, including WireGuardDevice's companion transport lines and UdpStatelessSocket's incoming peer lines, still belong to their exact creators and close normally. Classify the actual line_t, not just its worker ID or whether the chain contains packet nodes.

The source is explicit. In tunnelchainFinalize() (ww/net/chain.c), when the chain contains a packet node, one packet line is created per worker (abridged here without the surrounding pool cleanup):

line_t **packet_lines = memoryAllocate(sizeof(*packet_lines) * tc->workers_count);
// ... create and validate every worker's line pool ...
tc->packet_lines = packet_lines;

for (wid_t i = 0; i < tc->workers_count; i++)
{
    if (tc->contains_packet_node)
    {
        packet_lines[i] = lineCreateForWorker(0, tc->line_pools, i);
    }
}

And they are torn down only in tunnelchainDestroy():

for (uint32_t i = 0; i < tc->workers_count; i++)
{
    if (tc->packet_lines && tc->packet_lines[i])
    {
        lineDestroy(tc->packet_lines[i]);
    }
}

Reach a worker's packet line with tunnelchainGetWorkerPacketLine(chain, wid).

Never destroy a packet line at runtime

Do not call lineDestroy() on a packet line during normal operation, and never design logic that expects packet lines to be recreated on Finish. Packet adapters such as TunDevice assert in debug builds that the packet line is still alive after forwarding a packet. If your change can let a packet line die at runtime, that is almost certainly a bug.

Runtime Finish on a Packet Line

Part 2 requires an owner's Finish handler for a normal line to return with ! lineIsAlive(line). Packet lines are excluded from that postcondition, without exception. A packet line is owned by the chain, not by any tunnel, and tunnelchainDestroy() is the only place that may ever destroy it. That part is absolute: whatever else your handler does, it must never call lineDestroy() on a packet line.

What a Finish on a packet line means, though, depends entirely on the hop that receives it, the direction it arrived from, and the role that handler plays. There are three categories. Chain position is useful context, but it is not enough by itself to select one.

Category 1 — packet-lifecycle anchor: reject impossible close

A packet adapter or bridge may anchor the worker's packet-pipeline state on the packet line and have no notion of that role closing. Here “adapter” means a packet-pipeline boundary or lifecycle endpoint, not a tunnel that owns the line_t. Nothing in a correct chain finishes that exact role, so a Finish arriving in that handler and direction means a neighbour broke the contract:

• never call lineDestroy();
• fail loudly — LOGF(...) then abortProgramNow(1), the project's packet-line contract-violation policy;
• never convert it into normal connection teardown;
• leave the real destruction to tunnelchainDestroy().

That is exactly what the anchoring adapters do:

void tundeviceTunnelDownStreamFinish(tunnel_t *t, line_t *l)
{
    discard t;
    LOGF("TunDevice: unexpected downstream Finish on worker packet line %u", (unsigned int) lineGetWID(l));
    abortProgramNow(1);
}

Examples include TunDevice and RawSocket; the upstream packet sides of PacketsToConnection and PacketsToStream; the downstream packet side of StreamToPackets; PacketSplitStream; the packet modes of TesterClient and TesterServer; and the packet-line role of WireGuardDevice. These are per-handler classifications, not labels for the whole tunnel: the same bridge may own normal lines on its other side and close those normally.

tests/unittests/tunnels/abort/tunnels_abort_runtime_test.c proves the covered anchoring calls do not return.

Category 2 — transparent middle transform: forward lifecycle

A middle tunnel that transforms packet payloads does not thereby own or terminate the line lifecycle. It must forward Init, Est, Pause, Resume, Finish, and any other callback unrelated to its transform in the same direction:

• upstream callbacks continue with tunnelNextUpStream*;
• downstream callbacks continue with tunnelPrevDownStream*;
• a received callback is never reflected toward its sender.

Layer 3 has no per-connection TCP window, but that does not erase local scheduling or queue pressure. Packet/stream bridges and active packet producers still follow Part 2's flow-control roles. Transparent packet transforms relay signals; a documented packet endpoint may absorb them. Keep packet loss/admission policy explicit and never destroy the persistent packet line to implement per-flow pressure handling.

PingClient and PingServer inherit these lifecycle pass-throughs from packettunnelCreate(). Payload support is a separate contract: both Ping v2 endpoints require the chain's exact worker packet line for buffer ownership and worker validation. Their supported edge orders are:

TunDevice -> PingClient -> RawSocket
RawSocket -> PingServer -> TunDevice

An ordinary normal-line topology such as UdpListener <-> PingServer <-> UdpConnector is therefore not a supported Ping v2 composition. packettunnelCreate() keeps standard lifecycle pass-throughs, but an implemented payload handler may still enforce a stricter packet-line role.

Category 3 — intentional terminal absorber: handle locally and document

A packet line is still a line_t, and a chain may legally terminate one at a packet-capable terminal adapter. That adapter borrows the packet line without owning it and may close its own side for a documented reason.

One shipped example is an active BlackHole behind PacketSender:

PacketSender ──(worker packet line)──> BlackHole(mode=active)

During upstream Init, an active BlackHole sends tunnelPrevDownStreamFinish() down the worker packet line, and it lands on packetsenderTunnelDownStreamFinish(). That is not a violation and must not abort. PacketSender owns no per-line state and has no prev, so it absorbs the close without propagating it or destroying the packet line. It must still stop that worker's timer and send loop because receiving Finish permanently closes the upstream destination.

An adapter that requires normal connection lines is not a packet-line terminal. UdpConnector is layer-4-only: a packet path must use PacketSender -> PacketsToStream -> UdpConnector, with StreamToPackets on the receiving side when the payload returns to layer 3. The bridge owns the normal line and handles its recreation without closing the persistent packet line.

This classification is directional. PacketSender downstream is an absorber, while its upstream handler remains fatal because the chain head has no prev that could legally send it a callback. PacketReceiver is another intentional absorber: it owns no per-line state, may terminate either end of a chain, and reports completion through its timer and report path rather than Finish. Its required protocol-number selection controls per-source/protocol counters; expected-packets-per-ip applies to each selected protocol, while all matched arrivals remain counted even when they exceed that expectation. ALL selects protocol numbers 0–254, matching PacketSender.

Propagate the close, release what you own, report a violation — or state in the handler that you own no per-line state and have no onward direction, so absorbing the close really is the whole job. A body that is nothing but discard statements does not distinguish the last case from a bug.

The legitimate absorbers today are BlackHole (a chain-end sink that borrows every line), PacketReceiver (a chain end that reports by timer and file), PacketSender downstream (Category 3 above), and UdpStatelessSocket downstream on borrowed transport or packet lines. UdpStatelessSocket owns no peer resources on those borrowed lines and has no previous node to notify. Its owned normal peer lines remain distinct: Finish removes their idle entries and destroys them.

Review each tunnels/*/{upstream,downstream}/fin.c against its exact line role. An intentional no-op needs a written rationale in the handler; runtime tests must cover the propagation or owner close required by nonterminal handlers.

So the question to ask is never only “is this a packet line?” or “is this tunnel at a chain end?” It is “what role does this exact handler and direction play, what does the tunnel keep on this line, and who is still waiting on it?”

Dual-role handlers must check the exact role first

Some nodes see both a persistent packet line and normal lines they own. The handler must decide which one it was given before doing anything:

• WireGuardDevice is a packettunnelCreate() tunnel on its packet side, but it also creates one normal transport line per worker for its encrypted side. A Finish on a transport line is an ordinary owned-line close (destroy the slot, lineDestroy()); a Finish on the packet line is a violation and aborts.
• PacketsToStream and PacketsToConnection anchor bridge state on the packet line while owning the normal lines they create behind it.

Use the chain to identify the role — never a name or a guess:

if (tunnelchainIsWorkerPacketLine(tunnelGetChain(t), l))
{
    LOGF("MyNode: unexpected Finish on worker packet line %u", (unsigned int) lineGetWID(l));
    abortProgramNow(1);
}
/* from here on this is a normal line: apply the Part 2 owner postcondition */

Any intentional classification that differs by line role or direction must be documented at the handler and covered by a regression test.

Packet Lines Are Shared Worker State

Because one packet line serves many unrelated packets on a worker, anything stored on it is worker-local scratch state, not per-connection identity. Be especially careful with:

• routing_context.src_ctx and routing_context.dest_ctx
• any tunnel line state stored on the packet line

These fields are overwritten packet-to-packet. Do not treat them as a stable peer, flow, or session. If you store state on a packet line, that state must make sense as one of:

• per-worker state,
• packet-pipeline scratch state,
• bridge state anchored to the worker packet side.

Checksum repair intent belongs to each ordinary packet's sbuf_t, not the shared line. Queued, duplicated and retained packets carry independent requests, including during reentrant callbacks on the same worker.

If you need real per-flow or per-connection lifecycle behind packet traffic, create normal lines behind the packet side — do not try to make the packet line behave like a connection.

Packet checksum repair boundaries

Use sbufSetRecalculateChecksum() / sbufGetRecalculateChecksum() for an outstanding repair request on an ordinary resident packet. False does not prove validity and must preserve intentional checksum corruption. Complete copies, growth and reset follow Part 3.

packettunnelTakeChecksumRequest(buf) clears and returns that buffer's request. packettunnelConsumeChecksumRequest(buf) takes it and validates exact IPv4 structure before performing requested repair. Validation is unconditional: malformed input, IPv6 and trailing bytes are refused even when unrequested. These are terminal boundaries; refusal discards the packet. A transform that can refuse and pass unchanged must preflight and preserve bytes and intent on refusal.

TunDevice, RawSocket and packet/connection bridges consume the actual packet's request. Aligned copies retain intent locally until finalization; raw-byte ConnectionToPackets messages complete repair before queueing. Packet/stream bridges finish repair before serialization; no local flag is serialized. Ping repairs inner packets before encapsulation and requested outer packets before validation, then clears outer intent when stripping to the inner packet. Generated replies, originals and duplicates own independent requests.

Mark the surviving allocation after replacement and each generated packet according to existing repair behavior. Incremental IpOverrider/ProtoSwap address corrections preserve earlier requests unless a complete repair satisfies them. Buffer-owning queues and affinity messages need no parallel request boolean. Raw-byte SmuggleFin handoffs retain one explicitly and restore it onto the newly constructed packet. No request grants checksum assurance or enables offload; trusted TUN/PTC input still honors explicit repair requests.

Device implementation boundaries

Raw and capture handles are opaque. Production callers use the public lifecycle, write, and status APIs; backend layouts belong to private headers. A status load requires a live handle and supplies no lifetime reference. Capture's status accessor preserves the relaxed up publication used during startup and stop. Receive callbacks consume buffers on the selected event worker. A successful raw write transfers buffer ownership to the queue; refusal leaves it with the caller. Stop requests do not join threads or authorize destruction.

Linux TUN keeps lifecycle/open/rollback coordination in tun_linux.c, worker receive and GSO adaptation in tun_linux_reader.c, native writes in tun_linux_io.c, fixed multiqueue setup in tun_linux_queues.c, and interface/route/DNS configuration in tun_linux_config.c. Capture keeps lifecycle coordination in capture_linux.c, reader and residual drain work in capture_linux_io.c, Netlink parsing/transactions/verdicts in capture_linux_nfqueue.c, and firewall identities, command supervision and reconciliation in capture_linux_rules.c. Port-aware Linux capture predicates live in capture_linux_steering.c. These are private implementation boundaries, not interchangeable device backends.

TunDevice makes a best-effort attempt to place the first byte of each received IP packet on a 32-byte boundary, independently of GSO and trusted-checksum mode. Ordinary reads and Wintun copies use the aligned start of a padded buffer. Linux receives virtio metadata before that start, in reserved headroom; Darwin receives the utun family word separately with readv(). These layouts preserve the onward chain's padding without inserting bytes into the IP packet.

This placement preference is not a negotiated capability or an input requirement for other nodes. A path that cannot preserve it may still deliver a valid packet. Consumers must not infer alignment from the source node or checksum trust flags: check the actual address before using aligned accesses. Other packet carriers, header removal and cursor shifts can change alignment, and an aligned IP header does not imply an aligned TCP/UDP payload. Existing alignment checks and copy fallbacks remain necessary where their own operations require them.

device_fragment_policy.h defines only the policy enum. Pure packet hashing is provided by device_flow_hash.h; worker reduction and posting remain in device_flow_affinity.h. Checksum provenance and validity helpers live in device_packet_checksum.h; their callers retain their distinct trust and repair policies. Hash values, fragment keys and packet-admission rules do not depend on this source organization.

Worker-owned TUN receive

Linux and Windows TUN receive runs on ordinary event workers, without a dedicated application reader thread. Each native receive source has exactly one owner. Linux installs a nonblocking readiness watcher through a worker message queued behind packet Init; poller creation/registration refusal propagates through wioAdd() without publishing an active watcher. Windows uses a one-shot Wintun read-event wait whose foreign callback only posts a coalesced notification to worker 0. Windows workers copy and release ring packets before forwarding ordinary buffers. An empty ring rearms the native wait; a budget-limited drain queues a continuation. Windows callbacks admit at most 128 received records or 256 KiB, counting discarded records too. Linux callbacks attempt at most 256 reads and stop after 256 KiB; a single native record can exceed the remaining byte quantum.

TunDevice's boolean multi-queue setting defaults to true. In that mode a newly created Linux device attempts one queue per event worker. All queues and fixed eBPF steering must be established before interface activation. Unsupported multiqueue/steering or refused extra-queue setup leaves one queue owned by worker 0; extra descriptors are closed before publication. Existing legacy interfaces keep the single-queue attachment path. Queue count and steering stay fixed through a run; there is no automatic live queue migration. Queue mode is independent of checksum/GSO framing, and startup logs both choices. With multi-queue: false, Linux skips IFF_MULTI_QUEUE, eBPF steering and extra queue attachment, and uses one queue owned by worker 0. The configured processing worker count remains unchanged; complete native TCP/UDP flows retain per-flow receive locality across those workers, while opaque records rotate. Consumers still establish any required ownership at their own processing boundary. Native writes use the same resulting queue set. Invalid types, including null, are rejected at node construction. Other platforms accept the boolean and retain their existing receive arrangement.

Ingress steering uses the same symmetric address/port/protocol hash as consumer affinity for unfragmented IPv4 TCP/UDP packets with both ports present. With all worker queues active, these packets begin processing on their receiving worker, and connections between the same hosts can use different queues. This steering preserves locality; it is not a consumer's proof of ownership after tuple changes. IPv4 header and declared-length bounds are checked before reading ports, including when IP options are present.

Complete, structurally valid IPv4 packets other than TCP/UDP use approximate round-robin receive queues. Each interface owns a BPF_MAP_TYPE_PERCPU_ARRAY with one bounded 32-bit cursor per kernel CPU. Ordinary loads, increments and stores advance that CPU's cursor modulo the queue count; no BPF_ATOMIC instruction is needed. Selection adds the CPU ID before reducing modulo the queue count, so CPUs need not all begin at queue zero. Overlapping updates can lose a cursor step: there is no exact global round-robin sequence or equal-count guarantee. All complete packets continue processing on their receiving worker when multiple receive queues are active. With one receive queue, complete native TCP/UDP records with visible ports use the symmetric flow hash before entering a worker FIFO. The same flow stays in one FIFO even when other workers are delayed. Opaque or unclassifiable records rotate across workers using the session-local cursor. External-reader backends use the same policy. Single-reader classification recognizes unfragmented IPv4 and plain IPv6 TCP/UDP with both ports inside the declared packet length; it does not walk IPv6 extension headers or guess opaque transport fields. Unsupported per-CPU map or steering setup uses the existing one-reader fallback. Device readers do not inspect their chain or assign transport-state ownership. Consumers that need it select a worker from the tuple visible at their processing boundary, including after an earlier decoder.

All IPv4 fragments, including fragment zero, use the symmetric address-pair and protocol fallback, excluding ports and fragment IDs. TCP/UDP packets too short to contain both ports use that fallback too. Every fragment of one datagram therefore has one receive producer. Ordinary packets and fragmented datagrams of the same transport flow can have different readers and can reorder across those readers. The shared fragment table preserves assembly and processing affinity for preserved fragments; completed packets follow the ordinary reader placement policy after reassembly. Opaque round robin also permits reordering across readers. Kernel write-learned queue migration remains bypassed. Malformed headers and unsupported IP versions select queue zero and retain the existing downstream validation policy.

All Linux readers share one session budget and destination FIFO set. Ordinary posts use one message slot; an independent 512-packet / 8 MiB allocation budget also covers offload work. After each bounded drain, inactive lifecycle or a closed reader generation removes native read interest; owner quiescence still detaches the watcher. Nonblocking deviceReaderSessionTryPost() retains the whole input batch only on capacity refusal; all other results transfer ownership. A reader publishes its waiting flag before attempting admission, retains its pending input, disables read interest, and resumes from a coalesced capacity notification. It must never call the legacy blocking admission path. Notification admission is independently closed and quiesced before its device context is reclaimed. A Linux reader may open a synchronous receive-burst scope on its admitted current worker. A newly claimed same-worker FIFO accumulates input without posting a wake, then drains one bounded quantum when the scope ends, before the read callback returns. An existing scheduled token retains its ownership. Tokens claimed by foreign producers or outside a successful scope use the ordinary queued path; foreign appends can join an already locally claimed token. New work never overtakes a scheduled FIFO head or unfinished GSO work.

deviceReaderSessionTryBeginBurst() requires an enabled worker FIFO and returns false outside the supplied worker's admitted normal callback. The caller keeps a live session reference, never nests scopes for the same queue, and calls deviceReaderSessionEndBurst() on every successful scope before yielding, including capacity refusal, read failure and closure. End clears the scope before delivery callbacks; unfinished work keeps its queued continuation and token reference. The native reader's generation gate remains held through End.

The shared Linux fragment table uses its existing mutex and device-wide entry, range and storage limits. Its assembly and discard paths use detached allocations with finalized padding, never another worker's pool. Released fragment pointers belong to the caller's result array and survive subsequent offers. Other devices retain their exclusively owned reader-pool mode. Each Linux queue additionally retains at most one current receive allocation and one pending fragment-release batch (at most 17 packets); these bounded reader allocations are separate from the shared queued-work and fragment-storage budgets.

Worker-owned Linux capture receive

RawSocket's Linux input uses NFQUEUE, with no dedicated application reader thread. multi-queue defaults to true: capture prepares one independently bound Netlink socket/queue per event worker and installs fixed classic BPF predicates through iptables. A refused extra queue or steering installation falls back to a single queue owned by worker 0 only after partial owned rules have been removed and residual packets settled. Incomplete cleanup fails startup. multi-queue: false skips extra queues and steering. The configured processing worker count remains unchanged in either mode. Windows accepts the setting and retains its existing capture reader.

Unfragmented IPv4 TCP/UDP records with four transport bytes use a symmetric address/port/protocol receive hash. Distinct port pairs between the same two IPs can therefore select different readers. All fragments, including fragment zero, and short TCP/UDP records retain the address/protocol fallback. Structurally valid unfragmented packets of every other protocol use random NFQUEUE selection: disjoint classic BPF predicates feed cascading iptables statistic --mode random rules. For N queues, conditional probabilities are 1/N, 1/(N-1) and so on, with an unconditional final queue. Each queue has approximately the same share, without exact global counts or round-robin order. There is no shared nth-selection counter; kernel RNG implementations may still use local locking. This is the default and needs no protocol mapping or chain inspection. IPv4 length and IHL bounds are checked before ports are read. Steering does not validate opaque TCP/UDP contents or checksums. Invalid IPv4 shapes select queue zero for ordinary capture validation.

The capture hash is a compact 32-bit classic BPF predicate, distinct from TUN's 64-bit eBPF and WaterWall's consumer affinity. With multiple receive queues, complete packets of every protocol begin processing on their reader worker. A single receive queue assigns complete visible TCP/UDP flows to their symmetric hash worker before dispatch. Opaque or unclassifiable records still round-robin across configured workers using a session-local cursor. Preserved fragments retain the existing shared fragment association; reassembled packets use the ordinary reader placement policy. A consumer establishes required transport ownership using the tuple visible at its own boundary, after any protocol or address restoration. A stateless RawSocket -> IpOverrider -> RawSocket relay therefore preserves parallel processing even for one opaque protocol-50 peer pair.

Each multiqueue source range has one flow rule and one opaque rule per receive queue. Opaque rules use independent owned identities and rollback records; only one NOTRACK rule is installed per range. Protocol exclusions apply equally to both receive families and NOTRACK. Queue identities stay fixed through a run. Preserve fixed kernel TCP/UDP receive steering rather than replacing it with CPU fanout. This receive-locality rule does not require software flow ownership at the reader. Opaque distribution can reorder packets before they reach a consumer's affinity boundary; affinity cannot reconstruct that earlier order. Visible single-reader TCP/UDP keeps its per-flow FIFO to avoid creating such gaps in ordered input, including segmented ClientHello capture.

Each worker performs a bounded native drain (256 receive/verdict steps or 256 KiB) and yields. Receive, verdict sends, dispatch admission and retry paths never block the loop. A packet stays private until its NF_DROP verdict succeeds; verdict backpressure retains the exact packet/id, enables writable readiness and has an absolute 250 ms deadline. Interrupted primitives yield through a queued continuation. Shutdown can change an unsent DROP to ACCEPT; an accepted packet must never also be published through RawSocket.

Worker queues share one 512-packet / 8 MiB allocation-charge dispatch budget and one synchronized fragment table with detached storage. Capacity refusal retains at most one current record and the bounded fragment-release batch per reader, disables native read interest and resumes from coalesced credit notifications. The baseline kernel allowances of 65,536 packets and 64 MiB of requested receive buffering are divided over the active queue set. Startup kernel queue tuning can increase them as described below. Kernel socket limits can reduce the actual buffer allowance.

Before capture activation, and after owner watchers have detached, the lifecycle coordinator alone services inactive queues with NF_ACCEPT. Supervised firewall commands invoke bounded nonblocking progress callbacks during normal waits and failure cleanup. Every Capture generation is registered, so waiting for one device or another component also services other eligible Capture queues. After capture activates, service while workers remain behind their startup gate uses NF_DROP and queues detached packets through the bounded session FIFO. Normal worker activation is queued behind packet Init; the kernel queues and session FIFO buffer traffic during that ownership handoff. See Part 7.

Startup kernel queue sizing

TunDevice and RawSocket expose kernel-queue-tune, a strict boolean defaulting to true on Linux and false elsewhere. Other platforms accept valid settings without changing their queues. Numeric overrides are optional whole numbers in 1..INT_MAX; omission selects automatic sizing, while zero, negatives, fractions, strings, booleans and null are invalid. Validate settings even when tuning or capture is disabled and on non-Linux platforms.

Tuning uses the final active queue count after fallback and never lowers a larger existing allowance. TunDevice's kernel-queue-length is packets per queue. RawSocket's kernel-queue-packets and kernel-receive-buffer are device-wide totals divided over active queues; the latter is requested SO_RCVBUF bytes. Each explicit value replaces its corresponding automatic target. Explicit values and larger existing allowances are not constrained by the automatic caps.

Configured RAM profile	TUN packets per queue	Capture preferred packets per queue	Capture preferred receive-buffer request per queue
S1 / S2	1,024	1,024	1 MiB
M1 (client)	2,048	2,048	2 MiB
M2 (client-larger)	4,096	2,048	2 MiB
L1 / L2 (server)	8,192	4,096	4 MiB

For N active queues, TUN's automatic target is min(profile_target, 131072 / N). Capture retains device-wide baseline totals of 65,536 packets and 64 MiB of requested receive buffering. Its automatic per-queue target for each resource is max(baseline_total / N, min(profile_preferred, automatic_total_cap / N)), with automatic total caps of 131,072 packets and 128 MiB. An explicit total selects max(baseline_total / N, override_total / N). Division is integer division. These are startup sizing policies based on the configured RAM profile, not host RAM detection or runtime load adaptation. Each device has separate allowances.

With tuning disabled, leave TUN's existing queue length unchanged and retain Capture's baseline setup, ignoring valid overrides. Tuning is independent of Capture's skip-sysctl and core misc.tcp-tune; it adds no global sysctl writes. The userspace receive FIFO and its 512-packet / 8 MiB allocation-charge budget, callback quanta and output backlogs remain separate from kernel queue sizing.

Tuning/query failures are best effort: warn and retain otherwise usable queues. An unavailable old TUN value must not cause an unchecked reduction. Startup logs must also run when tuning is disabled and report the actual queue count and auto/override/disabled source. TUN reports old, requested and read-back txqueuelen, marking unavailable values. Capture reports ACK-confirmed NFQUEUE limits, marking unconfirmed limits unknown, plus requested and read-back SO_RCVBUF. Linux doubles SO_RCVBUF bookkeeping values, and host socket caps can reduce the request; neither the requested byte total nor packet count is actual RSS or a packet-payload byte budget. Increased depth can absorb bursts while adding latency and memory use; it does not establish a throughput gain. Before increasing SO_RCVBUF, apply the request to a disposable socket in the same network namespace and read back its effective buffer. Preserve the current queue buffer unless that probe confirms an increase. The probe must be closed on every result. This avoids relying on a host sysctl file being exposed in the namespace and does not authorize a sysctl change or SO_RCVBUFFORCE.

Device ingress fragment policy

Every fresh Linux TunDevice attempts checksum offload independently of gso. It probes an exclusive new interface with IFF_VNET_HDR, selects a 10-byte little-endian virtio header and requests TUN_F_CSUM first. The existing gso setting defaults to true and controls TCPv4 segmentation only. When requested, a second ioctl replaces the mask with TUN_F_CSUM | TUN_F_TSO4; a rejected upgrade keeps checksum-only mode on that descriptor.

Final mode	TUN framing	Active features
Raw fallback	IP bytes	None negotiated
Checksum only	10-byte little-endian virtio header + IP	TUN_F_CSUM
Checksum and TCPv4 GSO	Same virtio framing	TUN_F_CSUM | TUN_F_TSO4

Unavailable checksum/virtio setup, scratch geometry, capacity-notifier or FIFO resources can cause a close/reopen into raw mode before publication. Worker FIFO and budget resources remain required for raw worker reads; failure to establish them also in raw mode fails startup. Open, MTU, nonblocking and other genuine construction failures still fail construction. A GSO output-allowance failure disables TSO4 with a successful TUN_F_CSUM downgrade before publishing checksum-only state; if that ioctl fails, the descriptor is closed and reopened in checksum-only or raw mode. The session budget is configured once; partial common setup remains session-owned through raw fallback and cleanup. Startup logs both the active mode and requested GSO. That active mode fixes framing in both directions through stop and restart.

The checksum probe is always exclusive. Requested gso:false preserves the nonexclusive attachment path for pre-existing raw interfaces, including its resolver-baseline ownership; requested gso:true still rejects name conflicts. No offload is negotiated on a nonexclusively attached interface.

Checksum-only mode rejects every non-NONE GSO type before trust or work admission. TUN_F_CSUM permits deferred checksums but does not require NEEDS_CSUM on each record: completed, unmarked and applicable DATA_VALID input keep their existing validation and admission rules.

Outside the direct trusted pair described below, ordinary GSO_NONE TCP/UDP records with matching NEEDS_CSUM coordinates complete their checksum on the selected event worker before packet publication. This covers complete IPv4 datagrams and plain IPv6 TCP/UDP without extension headers. The reader retains generic checksum completion for other metadata and keeps fragment handling before worker selection. No accepted record is rejected merely because it cannot use worker completion. Compact packet storage and the preflighted checksum plan travel through the existing worker FIFO; reader draining remains batched and completion cannot alter the transport tuple. Ordinary GSO_NONE records of either IP family remain accepted. For TCPv4 GSO, the reader validates the entire aggregate, selects a worker using the ordinary reader-placement policy, and transfers the aggregate buffer and conversion plan to that worker. In single-reader mode this is the visible TCP flow's hash worker, matching ordinary records of the same flow. The selection survives capacity retries, and all segments of the record keep it; a later consumer establishes flow ownership. Outside the intact delivery path below, the worker allocates independent MTU-sized packets, copies headers and payload, uses the GSO payload size, adjusts TCP sequence/FIN/PSH/CWR, and completes IPv4 lengths, IDs, and both IPv4 and TCP checksums before normal TunDevice delivery, logging, or packet publication in ordinary mode. The direct trusted pair delivers assured TCPv4 aggregates intact and omits transport completion as specified below. IPv4 and TCP option bytes retain the negotiated contract's copying semantics when segmenting. A partial TCP seed must retain the kernel's pseudoheader contribution while changing its length; the IPv4 header's current destination can be a source-route hop rather than TCP's final destination. Valid negotiated TCPv4 GSO traffic is not filtered by a TCP-option allowlist. No virtio metadata or checksum-assurance flag reaches the chain; ordinary packet repair requests remain separate from private TUN work.

Ordinary NEEDS_CSUM completion uses tunLinuxOffloadCompleteChecksum() to sum the metadata-selected span, including its existing checksum-field seed. That helper stores an arithmetic zero as 0xffff, matching Linux skb_checksum_help() / CSUM_MANGLED_0 for any accepted protocol. The rule applies on both the worker and reader completion paths. TCP GSO segment completion instead preserves a computed TCP zero. Keep this offload encoding outside the generic checksum backend.

Every Linux receive mode has one FIFO per event worker. Ordinary batches, checksum work and GSO work share that FIFO, so a later ordinary packet cannot overtake unfinished GSO work on the same worker. Each callback shares a budget of 256 packets and 256 KiB across ordinary packets, GSO segments, and intact aggregates. An intact aggregate consumes one packet from that budget and its full IPv4 length from the byte budget. Completed small aggregates share remaining capacity with subsequent FIFO entries; an aggregate boundary alone does not force rescheduling. An unfinished aggregate remains the FIFO head and schedules a continuation without needing another TUN read event. Queue locking covers work/batch admission and FIFO maintenance, not each generated segment.

The kernel's requested gso_max_segs value of 2,048 is best effort: failure is logged and GSO continues. It limits some producers, but is not an ingress validity bound: forwarded/injected GSO can exceed it. Before publication, the device validates and stores scratch capacity, final padding and allocation charge from checked best-fit pool geometry without retaining a dummy packet allocation. Each reader allocates its own scratch under the fatal heap-allocation policy. Unsupported geometry or OS offload setup refusal can fall back to raw-IP framing. Segment admission transfers scratch itself to worker work and obtains a replacement; the reader does not copy aggregate payloads. Every non-segment record, including assured ordinary input, copies only its IP bytes into the queried best-fit allocation and retains scratch. Its reservation covers that actual allocation and work metadata, one packet slot and no output allowance.

The session's 512 packet-slot and 8 MiB allocation-charge budget covers queued ordinary packets and GSO work. Each GSO item reserves its actual aggregate allocation, work context, and one maximum output allocation allowance, with two packet slots. It retains that allowance through segmented or intact delivery and releases the reservation after finishing or being discarded. Workers do not wait for capacity. When admission is full, the owning worker keeps the unadmitted record, suspends native read interest, and returns to its event loop until a session capacity notification schedules a retry. Reader drain staging and fragment storage have separate existing bounds; session and FIFO-message metadata are bounded separately by packet count. The 8 MiB limit is an admission charge, not a total process-memory bound. There is no drop-only segment-count limit. Ordinary records retain the existing fragment policy; reassembly can publish a complete datagram larger than the TUN MTU without re-entering the GSO converter.

TunDevice and RawSocket accept an optional fragment-policy, either reassemble or preserve-fragments. Omission selects reassemble; an explicit invalid value, including null, rejects startup. The policy is immutable for the reader session and is passed to all TUN and capture backends. Write-only RawSocket instances have no ingress to assemble.

Fragment policy does not impose a node-name or topology allowlist. Normal chain and layer validation remains in force. Reassembled ordinary packets may pass through transforms, branches, packet/stream bridges, egress devices or sinks; each component remains responsible for its input and output contracts.

Boundary	Contract
Device ingress, reassemble	Publish complete IPv4 datagrams after bounded assembly; ordinary unfragmented input follows the existing path.
Device ingress, preserve-fragments	Preserve fragment bytes/boundaries and fragment-aware worker selection.
PacketsToConnection, upstream packet input	Must receive a complete IPv4 datagram before fake DNS, routing or lwIP input.
ConnectionToPackets, downstream packet input	Must receive a complete IPv4 datagram before flow lookup, worker injection or lwIP input.
Packet egress	Enforce existing packet-size and device-MTU limits; ingress reassembly does not provide output fragmentation.

At either packet-to-stack input, MF or a nonzero fragment offset causes a bounded input drop; DF alone is allowed. Reject after structural normalization and before stack delivery. CTP also filters destination/protocol before rejecting fragments and never reads transport ports from rejected fragments. Rejection must recycle the input, settle any copied message and ingress gate exactly once, preserve the chain-owned packet line, and leave unrelated connections running. The first drop logs a warning; subsequent drops increment the instance counter, with its total logged at destruction. Structural packet validation/drop behavior remains unchanged.

CTP's two packet-handoff directions share a 16 MiB / 4096-allocation instance budget, charged before allocation with metadata and explicit alignment padding. Output saturation returns ERR_MEM; return saturation drops the packet. Keep force-queued delivery and settle every terminal path. A custom pbuf retains its charge until actual free, using budget storage independent of tunnel lifetime. Egress transfer has one transient line-pool buffer per executing worker; the next receiver owns its retention limit after transfer. First-refusal diagnostics and per-direction destruction totals report overload without logging every packet.

PTC fake-DNS cache entries protect the latest successful answer's advertised TTL using a shared monotonic clock. Refresh on every committed answer and refuse new names with SERVFAIL when no expired, unpinned slot exists. Failed transactions preserve names, addresses, leases and LRU order. Accepted names contain ASCII letters, digits, hyphens and underscores within DNS labels; reject embedded NUL, literal label dots and other unsupported bytes before cache mutation so length-aware policy and C-string resolution share one canonical lowercased identity.

Reassemble before these consumers, regardless of whether fragments originate from a device or another node. preserve-fragments is not rejected at startup merely because a stack is downstream: the actual delivered packet is checked. A device policy affects only that device's reads. In particular, a direct ConnectionToPackets/PacketsToConnection pair cannot exchange fragmented output without an intervening reassembly boundary. Both nodes may still emit IPv4 fragments on their packet output sides.

Reassembly accepts the existing 20-byte IPv4-header subset, exact total length, valid header checksum, nonempty payload, aligned nonfinal fragments, and payload ends at most 65,515. The active direct TUN/PTC pair omits only the header checksum verification; TunDevice configures that trust on its fresh fragment table before publishing the reader, and the choice remains fixed across session generations. Other device fragment tables retain verification. Ordinary unfragmented/non-IPv4 input bypasses assembly. One session owns all incomplete bytes; no incomplete datagram is published. Completion requires fragment zero, consistent final end and gap-free coverage. Fragment zero supplies the header including TOS, TTL, reserved and DF bits; MF and offset are cleared, total length and IPv4 checksum are rebuilt. Transport bytes and checksums are unchanged. The direct trusted Linux TUN/PTC pair skips whole-datagram transport checksum verification after reassembly. Other routes retain their downstream consumer's validation policy. Overlaps are dropped without overwriting accepted bytes; conflicting ends and range exhaustion poison the identity. Raw overlapping zero with staged tails also poisons, preserving the former local rejection rule.

Each session has at most 256 entries, 16 disjoint ranges per entry, and 8,192 accepted fragments per datagram (enough for a maximum IPv4 datagram composed of minimum-size fragments). Normalized storage is bounded to 4 MiB of actual sbuf allocation charge, including capacity, padding, header and explicit alignment overhead. This accommodates dozens of full datagrams while bounding sparse-tail attacks. Growth reserves old plus new allocations simultaneously and copies only accepted ranges. Raw mode retains at most 16 pre-zero tails per entry and 256 in total, under a separate 32 MiB actual allocation-charge limit, allowing the full 16-tail staging window even with 1 MiB device receive buffers plus overhead. Fixed entry and range arrays bound metadata separately. These limits exclude pool caches and RSS.

Entries expire after 15 seconds without deadline refresh. Refusal/expiry keeps one fixed 15-second local poison interval; retries do not extend it. Opportunistic sweeping can retain bounded expired storage while idle. Retirement releases all storage after reader quiescence (or external-thread join) and any required pool-owner reset; poison survives restart until its local deadline. No downstream timer epoch or residue receipt controls this owner. IPv4 identification reuse still has the protocol's inherent delayed-fragment ambiguity.

Completed packets retain onward padding and use ordinary reader placement: the receiving worker with multiple queues, or visible native TCP/UDP flow hashing and opaque session-local rotation with a single reader. Queued session messages still reject stale generations. Once delivered, complete packets follow ordinary buffer ownership, including delays, duplication and drops; there is no late device-generation receipt. Raw mode preserves fragment bytes and boundaries and stages tails until fragment zero supplies port-based worker affinity, but makes no downstream residue or post-publication generation guarantee.

Assembly may produce complete datagrams up to 65,535 bytes. Global packet/stream and raw-egress limits and device-MTU checks remain; an oversized packet may be dropped at those boundaries. Transforms must handle the complete packets they receive within their own contracts. Fake DNS sees only complete UDP queries; fragments from any source are dropped at the packet-to-stack input.

Linux capture conntrack policy

RawSocket uses one bypass-conntrack setting for capture and raw output, defaulting to true. When enabled, Linux capture pairs each source-range NFQUEUE rule in filter INPUT with a CT --notrack rule in raw PREROUTING. NOTRACK additionally matches addrtype --dst-type LOCAL, so ordinary transit traffic, broadcast, and multicast are not exempted. The INPUT source-only capture match remains unchanged. Matching occurs before DNAT; selected local-unicast traffic must not depend on host conntrack-based NAT or connection-state filtering. Setting the option to false installs neither capture nor output NOTRACK rules; NFQUEUE capture and its drop-and-dispatch behavior remain active. Keep normal raw priority: NOTRACK must not move fragment reassembly or change the capture payload limit. PMTU/ICMP delivery remains a separate responsibility.

dont-capture-protocols is a set of literal IPv4 Protocol bytes (integers 0..255); omission and [] exclude nothing and duplicates are idempotent. Apply the same exclusion predicate to NFQUEUE and incoming NOTRACK, so an excluded protocol from a configured source reaches ordinary host conntrack and firewall processing. Filtering only after queue delivery is too late to preserve conntrack. This option never changes the marked raw socket's OUTPUT exemption.

Linux builds a fixed-size classic BPF bitmap lookup once during capture creation and stores its text on the device. Nonempty exclusions require the iptables bpf match (xt_bpf); empty exclusions add no match. Use the identical predicate for insertion, rollback, deletion, and cleanup retries in both tables. Literal protocol zero must work despite iptables -p 0 meaning all protocols. Read only the IPv4 Protocol byte; exclusions must not interpret or validate opaque transport headers or change normal fragment reassembly. Windows includes the same literal exclusions in its WinDivert filter using ip.Protocol comparisons. The constructor consumes the borrowed exclusion set synchronously; neither backend retains a pointer into the node's settings.

Prepare the inactive queue readers before rule installation and service them with NF_ACCEPT from the lifecycle coordinator during commands. Install all INPUT rules before any enabled NOTRACK rules and activate worker capture only after all selected rules succeed. The lifecycle owner removes NOTRACK before NFQUEUE, including startup rollback, normal stop, and cleanup following reader failure. Attempt queue cleanup even if NOTRACK cleanup fails. Account for each rule kind independently, including commands that may have committed before a timeout; reconcile those outcomes against the corresponding table and chain. Retain failed cleanup ownership for retry and report residual rules. Do not mutate the rule ledger from a normal worker receive callback or run blocking firewall commands in the nonblocking stop-request phase.

Rule removal affects future packets. Already-marked or queued packets remain untracked. --queue-bypass cannot undo NOTRACK; abrupt process death or final cleanup failure can leave WWCAP_NOTRACK_... rules in raw PREROUTING requiring administrator removal. This implementation does not reclaim another process's rules. skip-sysctl suppresses Capture's global sysctl tuning batch only, never either rule kind or the independent per-device kernel-queue-tune setting.

Linux raw output conntrack policy

RawSocket defaults bypass-conntrack to true. A configured mark field is rejected while it is enabled, including zero or null. Disabling the option preserves explicit output marks and installs neither capture nor output NOTRACK exemptions. The Windows backend accepts the setting without changing WinDivert behavior.

The Linux raw device selects a random mark from values at least 0x00010000, preferring the high half of the 32-bit space. Selection checks iptables-save mark comparisons/literal assignments and IPv4 ip rule fwmark selectors. It preserves their match results for an unmarked packet, including masked and inverted comparisons: the socket mark affects routing before raw OUTPUT. These snapshots provide best-effort avoidance, not a global reservation of marks on other sockets, native nftables/tc/eBPF rules, or future/concurrent policy changes. Do not claim that a high range is reserved for WaterWall.

Set SO_MARK and install the exact-mark CT --notrack rule in raw OUTPUT before publishing the writer. Inspection, selection, socket-option, or rule installation failure must fail startup and roll back. Keep the ordinary raw hook priority and packet-size policy. No per-packet firewall commands are needed; normal routing/firewall, MTU, and ICMP-delivery responsibilities remain. Host conntrack-based NAT cannot be relied upon for the exempted output.

Only the lifecycle owner runs firewall commands. Stop requests remain nonblocking. Close raw-output admission and wait for admitted native sends before removing the exemption. Queued and partially sent batches are discarded only after native access has ended; stale retry callbacks cannot use a restarted socket. There is no raw writer thread to join. Record a rule as pending before insertion because a failed or timed-out command may have committed. Remove by exact mark and per-installation WWRAW_NOTRACK_... comment, reconciling failed deletion against raw OUTPUT. Retain pending cleanup for retry during teardown and report the mark/comment when it remains unresolved. Abrupt process death can leave a rule that an administrator must remove; never flush foreign rules.

Linux capture packet validation

Linux NFQUEUE capture must preserve deliberately invalid IPv4/TCP/UDP checksums and opaque transport bytes, including XORed headers, unusual TCP flags, invalid TCP data offsets, and inconsistent UDP lengths. Transport interpretation belongs to the receiving tunnel. The capture boundary requires a complete IPv4 packet: at least 20 bytes, version 4, a bounded IHL, and an IPv4 total length matching the captured payload. Netlink envelope/attribute bounds, kernel sender verification, packet IDs, actual truncation detection, and the 1,500-byte capture limit remain enforced. Truncation/oversize logging counts only those size failures; malformed Netlink messages or invalid basic IPv4 structure use a separate diagnostic.

Only explicit NFQA_SKB_CSUMNOTREADY metadata authorizes checksum completion. For an unfragmented packet, capture attempts to finish this pending kernel work when its headers can be interpreted safely. If completion cannot be performed, the bytes remain unchanged and capture still admits the packet. Without that metadata, capture neither validates nor rewrites checksums. Fragments are not normalized here: the selected shared fragment policy validates them and may stage, reassemble, or reject them. Windows capture retains its checksum-provenance validation policy.

Shared lwIP checksum backend

The shared lwIP build, including both packet/connection bridges, routes its generic Internet sums through LWIP_CHKSUM = wwLwipChecksum. The adapter calls calcGenericChecksum() and removes the final complement without swapping bytes or normalizing zero. For example, bytes 01 02 produce raw sum bytes 01 02 and completed checksum bytes fe fd. lwIP retains responsibility for pbuf-chain parity, IPv4/IPv6 pseudoheaders, checksum fields, and protocol zero semantics.

calcGenericChecksum() returns the final complemented checksum in network byte order, including a legitimate zero result; it performs no zero normalization. Verification includes the checksum field and any required pseudoheader seed and expects a result of zero. Verification and raw-sum callers must preserve that arithmetic result. Applying 0 -> 0xffff in the generic backend would break those consumers.

Protocol and offload helpers own output encoding. Enabled UDP generation stores an arithmetic zero as 0xffff; in IPv4 UDP, a transmitted zero means checksum disabled (RFC 768). Ordinary TCP and IPv4 header generation preserve a computed zero. Ordinary TUN NEEDS_CSUM completion follows the Linux offload rule described above. These encoding decisions belong after calculation in the appropriate helper, never in calcGenericChecksum() or wwLwipChecksum().

The adapter accepts a nonnegative int length and readable storage for that span; zero length permits a null pointer and returns raw zero. Spans exceeding 65,535 bytes use even 65,534-byte chunks, combining raw sums with end-around carry. It neither traverses pbuf chains nor allocates or copies payload bytes. It must call only the generic backend path, never packet-checksum helpers that can re-enter lwIP.

The checksum function pointer starts at the portable implementation, including for standalone lwIP callers before application startup. checkSumInit() selects AVX2 when compiled and supported, otherwise SSE3 or portable C. Selection is a startup-only operation before concurrent checksum users are published; checksum calls perform no lazy initialization or locking.

The portable backend reduces its 64-bit payload accumulator before adding saved overflow carries. It combines the resulting 32-bit sum and the generic checksum seed in a 64-bit temporary before final folding, preserving end-around carry at both stages.

Checksum-on-copy and generation/verification support remain enabled globally. Ordinary PCBs cache and combine payload contributions across writes and retransmissions. Per-netif checksum controls initially enable every check and generator. Only the direct trusted pair below changes checksum policy; IPv4 header generation remains enabled on every route.

Direct TunDevice / PacketsToConnection checksums

Only a reciprocal immediate Linux TunDevice -> PacketsToConnection pair may select trusted IPv4 header and TCP/UDP transport checksums. Both nodes advertise kNodeFlagSupportsTrustedPacketChecksums. TunDevice selects its private receive policy and publishes kNodeFlagTrustedPacketReceiveChecksumsActive on both instances before starting device threads. This happens automatically in raw, checksum-only and GSO modes, independently of gso; no extra setting is needed. Non-Linux devices, interposed nodes and a pair of TUN devices retain ordinary verification. PTC reads the published receive mode when each owner-worker route is created lazily, regardless of node start order. The mode remains immutable through quiescence, pending work settlement and teardown.

This pair assumes all IPv4 input supplied by the kernel TUN interface has valid IPv4 header and TCP/UDP checksums, including forwarded input that the kernel may not have transport-verified. Corrupted payload can therefore be accepted, acknowledged and forwarded through a new connection with a fresh valid checksum. This is the kernel-boundary contract, independent of checksum metadata assurance. Other packet consumers and devices do not inherit it.

The active TUN validates IPv4 structure without verifying its header checksum, including fragment admission and trusted ordinary/GSO writes. It skips TCP/UDP receive verification for unmarked ordinary records, after reassembly, and before GSO segmentation. The active PTC netif and fake DNS omit incoming IPv4 header and TCP/UDP checksum verification. Structure, exact lengths, metadata consistency, partial-fragment rejection, reassembly limits, repair requests, alignment copies, flow lookup, receive credit and output admission remain enforced. IPv6 and UDP-Lite verification remain enabled. Header generation remains enabled: lwIP output, reassembly, segmentation and GSO aggregation still produce valid IPv4 header checksums.

Complete TCP/UDP records with matching NEEDS_CSUM coordinates, or applicable DATA_VALID, carry positive assurance only in TUN-private queued work. Simultaneous flags are rejected. Receive trust does not manufacture positive metadata assurance for unmarked records. Ordinary records, generated GSO segments and intact aggregates share the existing bounded worker FIFO; no checksum-assurance marker is added to an sbuf or line. The sbuf repair request is separate and never certifies received bytes.

With active GSO on this pair, preflighted, assured TCPv4 GSO is delivered intact as one complete, unfragmented IPv4 packet of at most 65,535 bytes. PTC's custom pbuf wraps the aggregate for lwIP input without userspace segmentation or transport checksum completion. The aggregate retains its position in the device worker FIFO; PTC establishes the current tuple's flow worker internally before stack input. lwIP sees one larger TCP segment; no GSO metadata or original segment count is passed into the stack. Unmarked GSO uses the existing segmentation path without pre-segmentation transport checksum verification. Supported partial GSO seeds retain the kernel pseudoheader destination; segmentation adjusts the transport length and completes IPv4 headers. Generic offload coordinates retain materialization; partial IP fragments are rejected. Preserved ordinary fragments reach PTC's bounded input-drop path. IPv4 UDP checksum zero remains disabled and TCP zero remains a valid checksum. UDP lengths must be bounded by the IP payload. Positive offload assurance and partial writes require an exact UDP span; a shorter declaration retains ordinary conversion instead of certifying its unchecked suffix.

Receive trust and transmit capability are independent. After successful Linux virtio/checksum setup, validated scratch geometry and worker FIFO setup, TunDevice also publishes kNodeFlagTrustedPacketChecksumsActive on both instances. This transmit mode requires actual negotiated checksum support and ready resources, including checksum-only mode with gso:false or a rejected GSO upgrade; raw fallback does not qualify. PTC's ww_trusted_receive_checksums controls the receive exemption; ww_partial_transport_checksum controls only the negotiated transmit policy. Never enable partial transmit checksums solely because receive trust is active.

Only transmit-enabled, exactly bound IPv4 PCBs copy TCP payload without checksum-on-copy work. These copy-only writes keep ordinary copying below 64 bytes and call WaterWall's wwMemoryCopyLarge() entry point for larger spans, without requiring aligned payload addresses. Uncomputed segment caches remain invalid through append and split operations. When a full output sum is needed, an invalid cache selects a checksum over the actual pbuf chain. Raw TUN retains complete software-generated output checksums and checksum-on-copy.

In negotiated transmit mode, complete IPv4 TCP/UDP output fitting the actual netif MTU may omit its transport sum. The transport paths call ip_output_if with no IP options and account for its generated IP_HLEN header; other callers must supply their actual header length. Output needing fragmentation receives a full transport checksum first, including oversized UDP and exceptional TCP retransmissions. Fake DNS uses the same immutable route MTU and header geometry. Its receive exemption applies only on the receive-trusted instance; other instances still validate UDP before cache mutation.

The transmit-enabled ordinary TUN writer replaces each complete supported packet's checksum field with the folded, non-complemented pseudoheader seed and emits ten-byte little-endian virtio metadata: NEEDS_CSUM, GSO_NONE, exact checksum start and offset, and zero GSO fields. A zero partial UDP seed is valid. Fragments and other protocols retain zero metadata; no fragment requests whole-datagram completion. The kernel can deliver partial packets locally or complete/offload them when forwarding. DATA_VALID is never used to ask a TUN writer to repair bytes. ConnectionToPackets, raw/capture devices and ordinary PTC netifs retain full checksum behavior, including when sharing an engine with an active route.

PTC output dispatch and TUN TCP aggregation

PTC detaches one owned best-fit sbuf from lwIP output. Only its reciprocal immediate enqueue-only TunDevice sink may receive it inline on the exact packet worker; other neighbours stay deferred. The same rule covers fake-DNS fragments. Normal-line callbacks retain their deferred reentry boundary.

TUN output is deferred onto the submitting event worker, with one pending flush per worker FIFO. Workers may write the native device concurrently; no application mutex covers native writes. FIFO order is preserved within each worker, and direct PTC flows stay on their owning packet worker. There is no total order between independent workers. Admission remains bounded across the whole device.

On an active direct trusted Linux pair with active TCPv4 GSO, each worker collects only already-available FIFO entries, without a fill timer or per-flow cache. It combines only adjacent exact IPv4 TCP data packets with IHL 20, valid headers, original lengths within MTU, identical flow/ACK/window/options/DF/TTL/TOS and reserved/urgent semantics, consecutive IDs and contiguous wrapping sequence numbers. Only ACK and final PSH are supported. The first payload sets gso_size; a shorter final member or PSH closes the record. An aggregate is at most 65,535 IPv4 bytes and 64 members. Controls, UDP, fragments and incompatible packets keep ordinary order and framing; no packet-line or sbuf GSO marker is introduced.

A private header carries the combined IPv4 length/checksum, first ID/sequence, final PSH and folded non-complemented pseudoheader seed for the complete TCP span. Ten-byte little-endian virtio metadata requests TCPv4 GSO and checksum completion at TCP offset 16. The native writev() receives that header and original payload slices as one record. EINTR retries retain ownership; a short result is never replayed. Linux TUN's pre-injection EINVAL rejection falls back to originals and atomically disables aggregation for that device generation with one diagnostic, leaving framing and checksum trust unchanged. Already-admitted writes on other workers may complete. Checksum-only writers never form GSO records; trusted ordinary TCP/UDP writes retain partial seeds and NEEDS_CSUM with GSO_NONE, while untrusted ordinary writes use zero metadata and complete IP bytes. Invalid generated metadata remains a defect. Other I/O errors retain the device's existing transient, packet-local and fatal handling.

Consumer-owned packet worker affinity

packet_affinity.h provides a fixed consumer continuation, bounded shared fragment storage and one FIFO per destination worker. Complete IPv4 TCP/UDP packet-line input selects the existing symmetric flow hash. The consumer decides where that requirement begins: device receive steering is a locality choice, not a substitute for ownership at a later transformed tuple.

Single-reader native TCP/UDP receive locality must be preserved even when a consumer applies affinity later. Assigning ordered segments 1,2,3 to workers 0,1,0 can deliver 1,3,2 when worker 1 is delayed. The consumer FIFO cannot recover the earlier receive order because segment 2 has not reached it yet. ECH-SNI then treats the sequence gap as a capture failure and marks the current flow generation passthrough. Keeping each visible flow in one receive FIFO prevents this artificial gap; it does not repair network reordering. Opaque protocols such as 50 retain parallel distribution. The testReaderSkewPreservesClientHello ECH regression exercises this requirement through the real reader, consumer dispatcher and capture paths.

PTC invokes the dispatcher before its stack entry. IpManipulator installs an upstream entry boundary only for first-sni, smuggle-sni, overlap-sni, synfin-sni, ech-sni-trick or smuggle-fin; downstream it dispatches after protocol/port restoration and before stateful processing. Plain ProtoSwap encoding and decoding stay on the arrival worker; protocol/checksum conversion does not require a flow owner. PingClient/PingServer and WireGuardDevice also forward decoded inner packets on their arrival worker. PacketsToStream and StreamToPackets map decoded packets to the receiving stream worker's packet line; StreamToPackets rechecks the authorized source generation before publication and retains return-carrier rendezvous selection. The next stateful consumer establishes its own ownership. TesterClient/TesterServer do so at packet input when packet-ipv4.worker-affine-flow is enabled, before accessing the selected worker's verification state. Stateless relay paths need no affinity, and opaque protocols without a native TCP/UDP tuple stay local. CTP retains its existing connection-owner lookup and output routing rather than adding a generic input hash before the exact-owner lookup.

A flowAffinityTunnelCreate() helper is available for an actual chain boundary. It is internal, has no JSON-registered node or user setting, and dispatches only its constructor-selected upstream or downstream payload direction. Opposite payload and lifecycle callbacks pass through. Before forwarding Finish, the helper closes that exact packet worker lane and discards its queued payload; already-published task/cancel tokens still settle before destruction. It owns its metadata and dispatcher, not a neighbour or normal line. Its owner inserts it during mutable construction and destroys it after runtime settlement. PTC and IpManipulator use the dispatcher internally, preserving their exact processing boundaries and physical adjacency.

Each context accepts at most 512 retained packets and 8 MiB of allocation charge across all destinations, counting queue records and the active packet through continuation return. Fragment staging has its separate existing bounds. Capacity refusal drops the new packet. All accepted local, foreign and reentrant producers join the same destination FIFO; a local arrival never bypasses older admitted work. An empty local queue can drain synchronously inside an admitted normal callback, inheriting its packet-batch scope. Otherwise one coalesced worker token yields after 256 packets or 256 KiB and requeues unfinished work; one packet can cross the remaining byte quantum. Each deferred drain opens a packet-batch scope so consumers retain ACK batching.

A scheduled destination retains its worker packet line; records own their sbufs and carry their checksum requests. Worker-message refusal or cancellation drops that destination's accepted FIFO. Cancellation uses detached destruction and no worker pool or owner-only tunnel state. Context mutex and queue storage remain live until producers, callbacks and messages settle. Context destruction then releases staged fragments and synchronization storage.

Preserved TCP/UDP fragments use the leading fragment's ports to select the same worker as complete packets in that flow. Later fragments arriving first are held within the existing shared fragment limits, not hashed independently by IP ID. The helper preserves fragment bytes and boundaries; pending fragment checksum requests are completed before storage. It never moves normal lines. Other protocols and one-worker execution continue directly. FIFO order is admission order at this boundary; the dispatcher cannot reconstruct global packet arrival order across independently running readers.

ProtoSwap checksum representation

ProtoSwap normalizes the TCP/UDP checksum's source and destination address contributions to zero before mapping, then restores the current addresses' contributions when decoding. The actual IP addresses stay intact. The mapped checksum still uses the native TCP (6) or UDP (17) pseudo-header protocol, not the mapped number. Address rewrites between endpoints therefore preserve valid checksums after decoding, without payload scans, packet expansion, or per-flow state. Both endpoints must use this checksum convention.

Without an existing recalculation request, the incremental adjustment preserves invalid checksum residuals and detection of payload corruption. Disabled IPv4 UDP checksum zero stays zero; enabled UDP arithmetic zero is transmitted as 0xffff. TCP zero is an ordinary checksum value.

Every fragment receives the protocol mapping. Only the fragment containing both transport checksum bytes receives the address adjustment, including a later TCP fragment after an 8- or 16-byte first fragment. Fragmentation may occur before encoding or between endpoints; checksum conversion itself needs no reassembly or ordering state. Plain ProtoSwap forwards decoded fragments on the current worker; any required fragment association belongs to the later consumer. Fragments of one datagram must receive consistent final addresses. Malformed IPv4/transport lengths, partial checksum fields and invalid local fragment geometry pass through unchanged, with their pending request preserved. Trailing buffer bytes are excluded from the transform; final writers still validate the packet length.

Native shaping and port ghost run before upstream normalization. An existing checksum request is completed under the native protocol and actual addresses before encoding, in either callback direction; a failed request prevents mapping. For fragments, that request repairs only the IPv4 header. Decoding restores the address contribution and native protocol before downstream port restoration or TCP/TLS parsing. Existing downstream checksum requests are handed onward; upstream decoding can complete them immediately. ProtoSwap creates no new recalculation request. Arbitrary payload edits while mapped require a transform that understands this checksum convention.

The shared updateIpv4TransportChecksumAddresses() helper takes an effective TCP/UDP protocol separately from the visible IP protocol and explicit old/new address contributions in network byte order. This byte-oriented helper must never manage an sbuf's checksum request. It applies RFC 1624 equation 3 with 16-bit complements and end-around carry, and changes at most the two transport checksum bytes. For whole packets, validate TCP minimum header/data offset or UDP declared length against IPv4 total length using the effective protocol. For fragments, compute fragment_start = fragment_offset * 8 and locate TCP offset 16 or UDP offset 6 relative to the original IP payload. Never form a later-fragment TCP-header pointer or require every fragment to contain a full transport header.

Preflight the complete transition before any mutation or request consumption. A no-match or rejected transition must leave bytes and the request unchanged. A successful encoder must not hand a normalized packet with an unsettled request to a generic writer: mapped protocol 1, for example, would select ICMP checksum logic. Preserve the post-transition request per duplicate or delayed packet; never restore a consumed request from an earlier cached boolean. An already mapped upstream packet is decoded exactly once at this node and forwarded native.

Two Kinds of Packet Tunnels

Not every tunnel that touches packet lines is the same kind of tunnel. There are two distinct categories, and they have different rules.

1. Pure Packet Tunnels (packettunnelCreate)

Some tunnels are created with packettunnelCreate() instead of tunnelCreate(). From ww/net/packet_tunnel.c:

tunnel_t *packettunnelCreate(node_t *node, size_t tstate_size, size_t lstate_size)
{
    if (UNLIKELY(lstate_size != 0))
    {
        return NULL; // pure packet tunnels have no per-line state
    }

    tunnel_t *t = tunnelCreate(node, tstate_size, 0);
    if (! t)
    {
        return NULL; // invalid or overflowing state size
    }

    // ... explicitly installs upstream Est and downstream Init forwarding ...
    // ... installs mandatory packet-payload defaults ...
    return t;
}

Key facts:

• packettunnelCreate() requires lstate_size == 0 and returns NULL when that contract or state-size validation fails. Heap allocation follows the fatal policy. Pure packet tunnels have no per-line state, and their constructors must check the result before assigning callbacks or accessing tunnel state.
• They are expected to override the packet payload callbacks.
• The default payload callbacks are intentionally invalid: they call LOGF(...) and abortProgramNow(1), so they crash loudly if you forget to override them:

static void packettunnelDefaultUpStreamPayload(tunnel_t *self, line_t *line, sbuf_t *payload)
{
    LOGF("Unexpected call to default up stream payload for a packet tunnel, "
         "this function must be overridden");
    abortProgramNow(1);
}

• Init, Est, Pause, Resume, and Finish use same-direction lifecycle pass-throughs. packettunnelCreate() explicitly installs upstream Est and downstream Init forwarding because tunnelCreate() rejects those events. An individual tunnel overrides one only when its exact handler role requires an anchoring rejection, explicit forwarding, or an intentional terminal absorption as classified above.

Examples: IpOverrider, IpManipulator, PingClient, PingServer, WireGuardDevice.

For pure packet tunnels:

• Do not add connection-style line lifecycle logic casually.
• lstate_size == 0 does not mean "packet lines only". A kNodeLayerAnything packet tunnel also carries normal connection lines.
• Some default callbacks deliberately abort; know which ones you must override.

2. Packet-Line Anchored Bridges (tunnelCreate)

Other tunnels are ordinary tunnelCreate() tunnels that use the worker packet line as an anchor for worker-local bridge state. They cross the boundary between packet-oriented runtime and connection-oriented runtime.

Examples and what they anchor:

Bridge	Role
PacketsToStream	packet line to a worker-local stream-facing line; stores that line and a read buffer
StreamToPackets	borrowed stream lines to packet lines; each stream has a parser, with a shared source registry and per-flow return-carrier selection
PacketsToConnection	packet traffic to lwIP to normal WaterWall connection lines behind the packet side
PacketSplitStream	splits packet traffic across stream paths

State stored on a packet line is worker-local bridge state, not per-connection state. If you need closable per-connection behavior behind packet traffic, create or manage normal lines behind the packet line instead of closing the packet line.

Stream/packet bridge flow control

PacketsToStream owns its normal stream lines and keeps its bridge state on each worker packet line. StreamToPackets borrows normal stream lines and stores its parser state on those lines; its packet lines are forwarding targets. Both are endpoints for the exact lines they join. Neither may use permanent tunnel bypass: payload still requires IPv4 boundary parsing, carrier ownership and mapping between different line identities.

Keep their endpoint Pause state. PacketsToStream maps stream pressure to the packet source and drops additional packets while paused. StreamToPackets excludes a paused carrier from return selection and drops return packets if no eligible carrier remains. Repeated notifications are idempotent, not counted holds. Sensitive-mode Ping and Pong may cross Pause; they do not release ordinary packet admission or suspend the Pong deadline.

After creating a live, unpaused stream line, PacketsToStream sends packet-side Resume so a Pause from an earlier carrier cannot strand the packet producer. Replacement Init must settle first; a Pause during Init prevents that release. If creation happened while handling a packet or heartbeat, settle that buffer before Resume can reenter and release more input. Reset, failed Init and shutdown do not send a recovery Resume.

Obfuscator stream and packet roles

With tls_record_header enabled, ObfuscatorClient and ObfuscatorServer reserve normal-line receive state through tunnelCreate(). Both supported Init directions initialize that state before forwarding; upstream Est remains pass-through. Exact worker packet lines bypass this state and preserve one-packet/one-record validation, checksum requests, and oversized-datagram drops. Packet Finish remains same-direction forwarding and never destroys the packet line.

Ordinary stream deliveries split into records with at most 65,535 body bytes. XOR and skip operate per record. One encoded aggregate is sent per delivery; complete received records become one decoded aggregate, with partial tails kept for later input. Receive retention is bounded by 2 MiB plus one maximum wire record (65,540 bytes), for a fixed total of 2,162,692 bytes. This is independent of pool sizes, RAM profile and splice settings. Pause/Resume use transparent defaults in both directions, including repeats. Complete records decode into the current aggregate without waiting for Resume; only an incomplete suffix remains in the read stream. XOR and record processing remain necessary for every payload, so these nodes do not permanently bypass. Aggregate transformation follows Part 2's FIFO and advisory Pause rules. Malformed records and overflow close the borrowed normal line through its owner. Record the Init source direction before forwarding. Error close destroys local state, finishes the opposite side, then the still-live source side while holding a line reference. This supports both upstream and downstream Init without destroying a right-side owner's line before left-side cleanup. Received Finish releases state and continues only away from its sender.

Packet-Line Initialization

Packet lines are initialized differently from connection lines.

For a chain whose head is a layer-3 packet node, the node manager sends one upstream Init on each worker's packet line during startup — once per worker, not once per packet. So packet-line Init is a worker/bootstrap event, not a connection-open event.

But if a bridge relies on packet lines while the chain head is not itself a layer-3 packet node, that automatic init does not happen, and the bridge must trigger it. StreamToPackets does exactly this in onStart(), queueing the bootstrap onto every worker so packet-side tunnels see their Init after startup rather than re-entering init paths inline:

static void streamtopacketsQueueWorkerPacketInit(void *worker, void *arg1, void *arg2, void *arg3)
{
    discard worker;
    discard arg2;
    discard arg3;

    tunnel_t *t = arg1;
    line_t   *l = tunnelchainGetWorkerPacketLine(tunnelGetChain(t), getCurrentWorkerWID());

    if (UNLIKELY(! lineCallWithRef(l, tunnelNextUpStreamInit, t)))
    {
        LOGF("StreamToPackets: worker packet line died during packet-side init");
        abortProgramNow(1);
    }
}

void streamtopacketsTunnelOnStart(tunnel_t *t)
{
    for (wid_t wi = 0; wi < getWorkersCount(); wi++)
    {
        if (UNLIKELY(
                sendWorkerMessageForceQueueWithCleanup(wi, streamtopacketsQueueWorkerPacketInit, NULL, t, NULL, NULL) !=
                kWorkerMessageSubmitAccepted))
        {
            LOGF("StreamToPackets: failed to admit required packet-side Init on worker %u", (unsigned int) wi);
            startupFailureRecord(1);
            return;
        }
    }
}

So if you create or modify a packet bridge: do not assume packet-line init happened automatically. Verify whether the init comes from the node manager (chain head is layer-3) or whether your tunnel must trigger it in onStart().

External readers and packet startup

A constructor must not start a reader or enable external callbacks that can immediately enqueue packet delivery. Creating an inactive device handle is different from bringing it up and launching its reader. If a creation/open API also starts reading, call that API from the later startup callback instead of createHandle. TunDevice keeps device creation and tundeviceBringUp() in onStart; the latter queues receive activation behind packet Init and opens worker-output admission on Linux and Windows. Darwin/BSD retain their external reader. Native output remains deferred onto event workers.

The risk is initialization order, even when workers have not started dispatching. createHandle runs before chain finalization establishes packet lines, line-state offsets and buffer padding. An early reader can create buffers with insufficient padding or enqueue delivery before the packet side's Init. Workers wait behind workers_run_flag until startup finishes, but queued messages retain their order: a Payload queued ahead of Init can later access uninitialized state and crash. Do not rely on a quiet device, low packet rates, or sleeps to avoid this.

Use this startup sequence:

1. In createHandle, validate settings and allocate inactive state. Keep reader threads and automatic callback delivery disabled.
2. In onPrepare, finish setup that needs finalized topology or buffer padding, while keeping the reader inactive.
3. In onStart, bind the publication target and finish reader-visible settings, buffer geometry and lifetime gates. Ensure the required packet Init has been successfully queued for every destination worker before starting the reader. Required bootstrap admission failure must fail startup; do not start reading.

For a layer-3 head, NodeManager queues Init after finalization and before onPrepare and onStart. For a stream-headed chain, a bridge such as StreamToPackets or ConnectionToPackets queues its packet bootstrap in onStart. If another node owns that bootstrap, explicitly establish its ordering before reader activation; choosing onStart alone does not establish ordering between different nodes' hooks. Queue admission is not Init completion, but placing Init ahead of delivery in the same worker FIFO preserves the required order.

An auxiliary reader thread is not an event worker. Use a device-owned buffer pool and an explicit destination worker; never index worker state with its raw getWID(), borrow a worker-local pool, or invoke tunnel flow callbacks directly from that thread. Post ownership-bearing delivery through sendWorkerMessageForceQueueWithCleanup() or the existing device_reader_session APIs. Resolve the packet line and invoke tunnel callbacks inside the destination worker callback, using its supplied worker identity. Cancellation cleanup must release message-owned storage and references without borrowing worker-local state. Follow Part 1's worker-context rules and Part 7's reader lifetime rules to close admission and join the reader before reclaiming its reachable state.

Paired Tunnel Direction (the PingClient / PingServer Rule)

Transform role and callback forwarding direction are separate decisions. Ping wire v2 uses two independent Echo sessions:

client data: PingClient -- Echo Request --> PingServer
             PingClient <-- optional exact Echo Reply -- PingServer

server data: PingClient <-- Echo Request -- PingServer
             PingClient -- optional exact Echo Reply --> PingServer

• PingClient upstream and PingServer downstream originate type-8 requests with their local data. Their peer decodes the request in its corresponding callback and forwards the inner packet in the same callback direction.
• Inbound parsing validates exact IPv4 framing and the IPv4 header checksum before classifying the configured peer/local/ICMP tuple. Carrier-only restrictions, including a 20-byte outer header and no outer fragmentation, apply only after that match. Unrelated valid IPv4 packets, including options and fragments, pass through unchanged in the received direction.
• Validation failures expose a concrete diagnostic reason without changing classification or forwarding. Packet drops log at ERROR with packet length, inner/carrier limits, wrapper overhead and available/required left headroom. The existing five-second gate stays shared per node across reasons and workers.
• send-replies is a boolean configured independently on each endpoint and defaults to false. Only true allocates a reply clone and emits the received request's exact type-0 reply back to its sender before delivering the original request. False or omission skips reply allocation and output. A reply is an acknowledgement only; the sender consumes a matching reply instead of delivering its payload regardless of its own send-replies setting. Data delivery does not wait for a reply.
• Correlation and replay suppression belong to one bounded synchronized tracker per Ping node. The arrival worker runs correlation and reply generation; its WID is not request/reply correlation identity. Both remain active when reply generation is disabled. Duplicate requests deliver inner data once and receive another reply only when local reply generation is enabled.
• With reply generation enabled, a request receiver publishes two distinct buffers. The generated reply clone travels back toward the request sender, while the decoded original continues in the received direction. With it disabled, only the decoded original is published. Decoded originals and replies remain on the arrival worker. Inner fragments are forwarded immediately with their bytes and boundaries intact; Ping does not hold later fragments for a leading fragment or select a transport-flow worker.
• The shared codec keeps its checked input APIs. At the nodes' reply and strip call sites, the validated envelope, exact independent clone and locally retained original already establish success. Assert those caller guarantees rather than adding packet-drop or acknowledgement fallback paths. Execute the codec calls outside assertions so Release builds still perform them.
• Consequently, PingServer's upstream request handler sends an enabled reply clone with tunnelPrevDownStreamPayload() and the decoded original with tunnelNextUpStreamPayload(). PingClient's downstream request handler sends the clone with tunnelNextUpStreamPayload() and the decoded original with tunnelPrevDownStreamPayload().

Sending the original received request back toward its sender would be an invalid reflection. An enabled, separately generated exact Echo Reply travels in that opposite direction. Likewise, encapsulating the original in PingServer's upstream callback would transform it twice.

The procedure for any packet tunnel:

1. Draw the packet flow with arrows.
2. Place the encode/decode step in the callback direction that flow requires.
3. Use tunnelNextUpStreamPayload() only for upstream forwarding and tunnelPrevDownStreamPayload() only for downstream forwarding.
4. Do not treat packet lines as per-connection lines.
5. Do not call lineDestroy() on packet lines at runtime.
6. Do not add stream-style lifecycle to a pure packet tunnel unless its design requires it.

Testing Packet Tunnels

Packet tunnel tests must exercise the real intended direction. The canonical Ping test is one direct packet chain:

TesterClient(packet-mode=true)
    -> PingClient
    -> PingServer
    -> TesterServer(packet-mode=true)

The upstream request must be restored before TesterServer sees it; reverse data must become a separate Echo Request and be restored before TesterClient sees it. Exercise omitted/false and true send-replies settings at both endpoints, including asymmetric settings. Enabled replies must precede inner delivery and be consumed by the originator; disabled replies must cause no clone or reply output. Add coverage that would fail if PingServer upstream encapsulated instead of decoding, if an inner request were delivered twice, if reply consumption or deduplication depended on local reply generation, or if server data were put into the original request's reply.

The packet-line Finish contract has its own coverage, and a new packet-side handler belongs in it:

• tests/unittests/tunnels/abort/tunnels_abort_runtime_test.c calls the anchoring adapters' Finish handlers in child processes; the CMake runner requires exact exit status 1 rather than a return or signal.
• tests/unittests/tunnels/packet_io/packetsender_orderly_shutdown_test.c proves that PacketSender cancels a pending worker timer and stops a ready multi-packet batch when a payload emits downstream Finish re-entrantly, without destroying the worker packet line.

Rules for Packet-Oriented Changes (Summary)

• Do not close packet lines during runtime, and never lineDestroy() one.
• The owner Finish postcondition from Part 2 does not apply to packet lines.
• Classify each packet-line Finish handler and direction as one of three roles: an anchor rejects an impossible close with LOGF + abortProgramNow(1); a transparent middle transform forwards it in the same direction; an intentional terminal absorber documents why there is nothing to forward or release.
• PacketSender is direction-specific: downstream absorbs a legitimate close from a packet-line terminal such as an active BlackHole, while upstream remains fatal. PacketReceiver is also an intentional terminal absorber.
• A handler that can receive both a packet line and a normal line must branch on the exact role (tunnelchainIsWorkerPacketLine()) before any cleanup.
• packettunnelCreate() only means lstate_size == 0; several such nodes are kNodeLayerAnything and carry ordinary connection lines too, which is why its lifecycle callbacks stay plain pass-throughs.
• A Finish handler that does nothing must say why: propagate, release, report, or document that you own no per-line state and have no onward direction.
• Do not run normal per-connection Finish logic on packet lines unless the exact handler-role contract explicitly requires it.
• Treat packet-line state as worker-local shared state.
• Treat routing context as mutable packet-line scratch metadata, never as durable session identity.
• For per-flow lifecycle, create normal lines behind the packet side.
• A packettunnelCreate() tunnel has no per-line state (lstate_size == 0).
• A bridge that anchors state on the packet line keeps that state across many packets on the worker.
• If your code depends on packet-line init, verify where that init comes from.

The single most common mistake to avoid:

Using packet lines as if they were normal connection lines with an ordinary
Init -> Payload -> Finish per-connection lifecycle. They are not.

---

Continue to Part 5: Anatomy of a Tunnel & Workflow.

Worker-owned lwIP bridge engines

PTC validates a complete IPv4 packet, settles its checksum request, rejects fragments and establishes affinity from the current TCP/UDP tuple before any engine/netif or flow lookup. Its continuation selects the destination packet line's engine and verifies exact current worker ownership. Earlier packet nodes establish their own state ownership at their own tuple boundaries; they do not select PTC's owner. The dispatcher stays inside PTC, so the immediate TUN topology, startup-negotiated checksum trust and certified inline output remain intact.

CTP selects the borrowed normal line's owner engine, which can differ from the packet hash worker. Return input resolves the full node flow key, WID and generation, force-queues detached bytes to that owner, then rechecks the entry, engine and exact netif before injection. No flow-registry lock encloses stack execution or neighbour publication.

lwIP allocation pools remain process-wide. The eight malloc-pool classes in ww/lwip/lwippools.h reserve four blocks per rounded flow share, about 18.5 MiB before allocator overhead at the default TCP/UDP flow targets. This capacity is fixed at build time and independent of the runtime RAM profile. Both the header and CMake reject flow targets whose derived pool counts exceed 65,535 entries.

Netif indices and generations are engine-local and may coincide across engines. Identity includes the engine and exact netif. Multiple nodes may use identical tuples on different netifs. Each engine has its own 255-index namespace, including its loopback; netif exhaustion refuses the affected attachment promptly.

Both bridges still require complete IPv4 datagrams and reject fragments at input. Device reassembly precedes worker selection by default. Output fragmentation and MTU behavior remain supported; generated IPv4 IDs and IPv6 fragment IDs use separate shared sequences. Finite-width wrap is unchanged. Direct bridge pairs still require reassembly when exchanging fragments.

Ordinary UDP input validates the declared UDP length before checksum verification or pretend-child publication. It must be at least UDP_HLEN and no greater than the IP-bounded transport bytes. A shorter valid span trims the pbuf, including chained storage, before checksumming and delivery. The application receives exactly the declared span minus the UDP header; a header-only span delivers one empty datagram. Disabled or trusted checksums do not bypass these structural checks. UDP-Lite retains its separate checksum-coverage semantics.

Device receive dispatch opens a generic packet_batch_t scope around its existing 256-packet and 256-KiB worker quantum; an oversized ordinary packet travels alone. The dispatcher has no lwIP dependency. Consumers join through persistent, owner-local participants that supply synchronous begin/end callbacks. Repeated joins of one participant share one begin/end pair. Empty scopes invoke no hooks; participants and their callback contexts must remain alive until scope end returns. Scopes cannot nest or span an event-loop yield. End clears the active scope and all participant memberships before flushing callbacks, which cannot open another scope; joins during a flush have no active scope.

PTC joins its engine before stack entry, coalescing ordinary established, in-order TCP ACK intent before packet construction. Multiple PTC instances on one worker share the engine's one participant. A newly created engine can join immediately. Device-only traffic does no lwIP batch work, even when another configuration uses lwIP on the same worker. Output retains each connection's own interface and chain. Worker-owned readers enqueue even when the destination is their current worker. Linux readers collect a bounded burst before ending their receive-burst scope. An eligible same-worker FIFO then drains inline through the existing ACK quantum, so its packets share ACK state without a receive-dispatch self-wake. Other FIFO tokens and unfinished quanta remain queued; same-worker input is not delivered during native read collection. Other workers may drain their queues concurrently. wwLwipEngineBeginAckBatch() and wwLwipEngineEndAckBatch() require the owner outside stack entry or dispatch. They must pair synchronously, without nesting or an event-loop yield. End flushes current cumulative ACK/window state before returning; batching introduces no cross-burst timer or delay policy.

Eligible receive-credit updates share the queue. Accepted payload that advances the receive sequence into an existing out-of-order gap requests an immediate ACK, including partial gap progress and a trimmed duplicate prefix, with or without a device batch. A receive-window update reaching the existing threshold bypasses batching when the last successfully output wire advertisement is unknown or its right edge no longer exceeds rcv_nxt, using TCP sequence comparisons. The record uses the transmitted 16-bit window and negotiated scaling (SYN is unscaled); 1–127 internal bytes at scale 7 still advertise zero. Capture occurs before output consumes the packet, and publication requires actual ERR_OK output, which records local acceptance without claiming peer receipt. Failed or skipped output and RST do not publish an advertisement. Split credit returns therefore cannot conceal a closed wire window. Existing connection-control, duplicate, out-of-order and SACK paths retain their existing behavior. Eligibility and gap state are captured before duplicate trimming or out-of-order queue changes can make recovery input appear ordinary. Piggybacked or immediate ACK output clears the normal ACK flags, suppressing a redundant batch flush. Pending PCBs unlink before purge, free or flush output; failed output retains lwIP's existing ACK retry flags and timer.