PacketsToConnection

PacketsToConnection is a packet-to-transport bridge built on lwIP.

It accepts raw IPv4 packets on the packet side, injects them into lwIP, and exposes the resulting transport flows as normal Waterwall line_t connections toward the next tunnel.

Before entering lwIP, the node establishes TCP/UDP worker affinity from the current packet tuple, after any preceding address or port transformations. Receive placement is not a flow-ownership requirement. The internal dispatcher uses bounded per-destination FIFOs; it does not insert another tunnel between TunDevice and PTC, so direct checksum and output capabilities remain available. Replies retain their existing connection owner. Affinity serializes admitted work but does not reconstruct packet order lost before admission.

Instance MTU

Optional integer mtu defaults to core misc.mtu at construction (core fallback 1500). Its range is 68..65535 bytes: raw IPv4 MTU installed on every lazy worker-local virtual interface. Each instance stores its validated value immutably; other instances and later core changes do not change its behavior. Null, wrong types, fractions, zero, negative or out-of-range values reject construction. An unsupported inherited default also fails; supply a supported node override. This is not measured path MTU or MTU negotiation. Existing lwIP IPv4 fragmentation follows this interface MTU.

What It Is

This node is for chains that start from packet traffic and then want to enter Waterwall's normal connection-oriented world.

Conceptually:

• packet side -> PacketsToConnection -> normal Waterwall service chain
• service-chain responses -> PacketsToConnection -> raw IP packets back to the packet side

It is closer to a small in-process transport stack bridge than to a framing adapter.

What It Is Not

• It is not a TUN device manager.
• It does not create or configure routes or OS policy rules.
• It does not replace TunDevice.
• It does not serialize packets onto a stream like PacketsToStream.

TunDevice owns the real packet adapter side.

PacketsToConnection owns the lwIP transport bridge side.

Current Protocol Support

• IPv4: supported
• TCP: supported
• UDP: supported as per-flow Waterwall lines
• IPv6: not supported
• ICMP: not supported

Unsupported packets are dropped conservatively.

Checksum Finalization Boundary

PacketsToConnection is the final consumer of the packet buffer's checksum recalculation request. It takes and clears that request before any validation or drop path, normalizes shifted input into mutable aligned storage, and then repairs the supported checksums before typed parsing or lwIP ingress. A malformed, IPv6, stopping, or refused packet therefore cannot leak its request into the next unrelated packet on the worker line.

For a complete IPv4 TCP or UDP packet, finalization repairs the IPv4 header and transport checksum.

The packet input accepts complete IPv4 datagrams. MF or a nonzero fragment offset causes a bounded drop before stack delivery; DF alone is allowed. Reassemble at device ingress or before this node when fragmented datagrams must be delivered. The first unsupported-fragment drop logs a warning; subsequent drops increment the instance counter, whose total is logged at destruction. Rejection releases the input resources, preserves the packet line and leaves other connections running. Malformed packet structure follows the existing validation/drop path.

Flow Model

Packet ingress

Upstream payload must be a full IPv4 packet.

PacketsToConnection does not create one lwIP netif per destination IP.

Instead, it keeps a worker-local route context for each packet worker that sends packets into the tunnel. For IPv4, that route context owns one lwIP netif with NETIF_FLAG_PRETEND enabled.

The destination IP is preserved in the packet and in the lwIP PCB tuple, but it is not used as the key for creating separate route contexts. In the current source, the IPv4 route-context lookup is keyed by packet worker id.

Each worker-local route is published only after its pretend netif, TCP wildcard listener, and UDP wildcard PCB have all been created successfully. Later packets reuse that complete route; listener setup is not repeated per packet.

The Waterwall lwIP pretend patch lets those wildcard PCBs accept packets for arbitrary destination addresses and ports on the pretend netif while still preserving the original local destination tuple.

TCP flow creation

When lwIP accepts a TCP connection:

• PacketsToConnection creates a real Waterwall line
• fills the line source/destination address context from the TCP tuple
• schedules upstream Init
• forwards later payload with upstream Payload

Downstream payload from the next tunnel is written back into lwIP with tcp_write().

The TCP listener is not per destination port. It is bound as a pretend listener on the worker-local netif, and accepted TCP PCBs carry the original destination IP and port from the packet.

UDP flow creation

UDP is tracked per worker-local route context and per 4-tuple:

• source IPv4 address
• destination IPv4 address
• source port
• destination port

When the first datagram for a UDP flow arrives:

• PacketsToConnection creates a real Waterwall line for that flow
• fills source/destination address context
• schedules a checked owner-worker Open operation that sends upstream Init and arms the one idle timer
• forwards datagrams with upstream Payload

Open is required control and is queued before the intentionally lossy payload operation. If Open admission is refused, the node first detaches the PCB, timer, and flow-map entry inside the owner engine, records the owned line in its preallocated terminal-reconciliation registry, and then requests orderly shutdown as escalation. Cleanup therefore does not depend on the refused queue or on Stop discovering a still-registered flow. A later payload refusal is still an ordinary UDP drop because the timer already bounds the flow.

Downstream payload from the next tunnel is sent back through the connected UDP PCB, using the original local destination address and port as the packet source.

UDP lines are closed by idle timeout, not by packet half-close semantics.

Like TCP, the initial UDP listener is not per destination port. The pretend UDP path creates or reuses a connected per-flow UDP PCB carrying the original source and destination tuple, and PacketsToConnection maps that tuple to a Waterwall line.

Worker And Lifetime Model

This tunnel sits on a shared packet line on the packet side and creates normal Waterwall lines behind it.

Important internal rules:

• the packet line is not closed during runtime
• packet-line Init is a startup/bootstrap event, not a per-flow open
• generated TCP/UDP Waterwall lines are owned by the packet worker that accepted the flow
• lwIP callbacks hand work back to the owning line worker through the typed-result lineScheduleTask() and lineScheduleTaskWithBuf() contract. Required control refusal makes the producer terminal on the owner engine; optional cancellation is deliberately null and cleanup stays on the owner. Buffered delivery transfers its copied buffer on every result
• UDP idle close uses one cancellable owner-worker timer. The timer owns exactly one line reference while armed, activity resets the same timer, and close cancels it and drops the reference before line state is zeroed
• packets emitted back to the packet side use the worker packet line for that packet worker
• the worker-local netifs use the instance mtu, so a response larger than it leaves as IPv4 fragments rather than as one oversized raw packet the packet topology could not carry. Fake-DNS replies go out through that same netif rather than straight at the neighbour: echoed question names and answer records make reply sizes variable, and replies can exceed small IPv4 MTUs. PTC accepts the core's legal minimum MTU of 68 bytes. The reply is copied into a stack-owned PBUF_RAM and submitted as a UDP datagram through ip4_output_if(), so lwIP constructs the IPv4 header, allocates the shared identification value, and applies fragmentation with stack-owned offsets, MF flags, per-fragment lengths, and checksums. That publication happens inside the owner engine, which is legal because netif output only calls a certified enqueue-only TUN sink inline; arbitrary neighbours stay deferred
• packet emission is protected by an admission gate held through the previous-neighbour callback, and normal-line Init/Payload/Resume/Finish work uses a second gate held through the next-neighbour callback. Global shutdown closes both gates in onQuiesceRequest() and waits for admitted callbacks in onQuiesceWait() before any component stops, so an already-admitted callback finishes before its neighbour can stop and queued work that runs later is cancelled. A refused next-side ordinary work recycles anything it owns and leaves teardown to the owner path; an initialized line still receives its one explicit teardown Finish during the owner-worker drain
• netif output copies each pbuf packet directly into one best-fit, padded sbuf. A reciprocal immediate TunDevice that certifies enqueue-only packet delivery receives that same buffer inline on the owner packet worker; other neighbours receive it through deferred delivery. No arbitrary callback can reenter stack input. Quiesce closes output admission; canceled messages destroy their detached sbuf without borrowing a worker pool. Fake-DNS mapping transactions and reverse-name copying use one node mutex shared across workers. No cache-entry pointer escapes that scope. The mutex is released before stack output, neighbour callbacks or worker posts
• top-level packet parsing reads only the version byte before normalizing cursor alignment. Shifted packet buffers are copied to aligned sbuf storage before any typed IPv4/UDP access; fake-DNS additionally verifies the IPv4 header and any nonzero UDP checksum before it can answer or mutate its mapping cache, except on an active direct TUN pair. Structural checks remain active in both modes
• upstream packet input rejects IPv4 fragments before fake DNS, route/listener creation, or lwIP input. Reassembled packets follow ordinary buffer ownership; no per-buffer fragment settlement claim is needed
• every generated TCP/UDP line is registered in a per-worker owner list until its one close path unlinks it. onWorkerStop drains that worker's lines and detaches PCB, route-map, callback and UDP-idle producers in its engine; it preserves whether next-side Init completed, destroys PTC line state, sends exactly one next-side teardown Finish when required, and calls lineDestroy(). Configuration Stop waits for all worker drains; terminal worker shutdown performs the same drain locally without relying on new message admission
• Stop explicitly detaches every owned per-flow UDP PCB, wildcard UDP PCB, and TCP listener before removing its netif. WaterWall's lwIP patch treats that as a raw-owner precondition and rejects netif_remove() if either raw owner is still present; it never silently frees a pointer the tunnel could still retain. A failed conversion from a full TCP PCB to a listener also closes or aborts that full PCB before the route is retried

This keeps line destruction and tunnel state teardown on the normal Waterwall line side instead of relying on ad-hoc cross-worker message ownership.

The internally created TCP/UDP lines are normal connection lines. They are separate from the persistent worker packet line and may be closed and destroyed during normal runtime.

Finish / Close Behavior

When the network side closes a flow:

• a connected TCP PCB moves to an instance-owned closer, even when the application queue is empty
• the closer copies only the suffix tcp_write() has not accepted; bytes already copied into lwIP are never duplicated
• receive credit is returned, peer data is consumed, and a TX-only tcp_shutdown() sends FIN after every accepted byte
• this tunnel destroys its own line state
• if upstream Init was already sent, upstream Finish is propagated
• the internally created line is then destroyed by PacketsToConnection

When the next tunnel sends downstream Finish:

• PacketsToConnection transfers a connected TCP PCB to the same graceful closer
• destroys its own line state
• destroys the internally created line

It does not reflect that downstream Finish back toward upstream.

The closer has monotonic delivering, FIN-pending, and closing phases. Delivery and peer-close deadlines are bounded at 30 and 60 seconds, and one instance may retain at most 4096 closers and 16 MiB of copied suffixes. Allocation, capacity, terminal write, or deadline failure resets that flow; Stop aborts all active closers before route/netif removal.

That matches Waterwall's direction rules for internally created lines.

The shared packet line is not destroyed by these flow closes. It remains alive until the tunnel chain is destroyed.

Pause / Resume

TCP

TCP backpressure is integrated with Waterwall:

• if lwIP cannot accept all downstream bytes, remaining data is queued
• upstream is paused with tunnelNextUpStreamPause()
• lwIP sent callbacks flush the queue, and tcp_poll supplies bounded progress when the first write accepted nothing
• upstream is resumed with tunnelNextUpStreamResume() when writable again
• acknowledgement records keep their totals and remaining-byte arithmetic at 32 bits, so one stream buffer may exceed 65535 bytes even though each individual lwIP sent callback reports a 16-bit count. Empty TCP buffers are consumed without entering the write or acknowledgement queues

For receive-side backpressure:

• downstream Pause stops immediate tcp_recved()
• downstream Resume releases the deferred receive credit

UDP

UDP has no true stream backpressure model.

Current behavior:

• downstream Pause marks the flow paused
• inbound UDP datagrams received while paused are dropped
• downstream Resume clears the paused state
• downstream application payloads are limited to the IPv4 UDP maximum of 65507 bytes; larger buffers are dropped without narrowing through lwIP's 16-bit APIs, while an empty UDP payload remains a valid header-only datagram

This is intentional and should be treated as a current limitation of the UDP path.

JSON Settings

PacketsToConnection currently supports:

• udp-idle-timeout-ms (int, default: 300000) Controls how long an idle UDP flow line is kept alive before the tunnel closes it.
• max-pending-bytes (int, default: 2097152) Budget for TCP payload retained while lwIP has not yet taken or fully acknowledged it. Valid range 1024 .. 67108864. Admission includes a fixed 1 MiB of delivery headroom; exceeding the combined bound sheds that flow.
• fake-dns (bool or object, default: false) Enables an in-tunnel fake DNS responder for IPv4 A queries. Mapped fake-IP destinations are converted back into domain destinations on generated TCP/UDP Waterwall lines.

The minimum allowed udp-idle-timeout-ms value is 1.

The default TCP pending budget is 2 MiB, matching lwIP's send capacity. This is a per-flow limit, not a preallocation; the shared lwIP pools can impose pressure before an individual flow reaches that limit.

Pause is advisory between tunnels, so it cannot be the only bound on retained memory: the next tunnel may already have a payload callback queued, may race the pause, or may simply ignore it. The admission bound is therefore P + L, where P = max-pending-bytes and L is a fixed 1 MiB of delivery headroom, independent of pool sizes, RAM profile and splice settings. The extra L covers already-delivered input before Pause takes effect. Each acknowledgement record charges its original payload length until fully acknowledged, covering queued and outstanding writes. Production stays paused while this charge exceeds P, even after the write queue becomes empty; Resume waits for sufficient acknowledgement progress. Admission is checked before either queue takes ownership. A payload that would pass this bound is dropped, the flow's TCP connection is reset rather than drained, and the line is closed toward the next tunnel. UDP retains nothing downstream, so the limit does not apply to it.

One TCP flow may also retain at most 1,024 ACK/pause records, independently of the byte limit. This is an internal allocation-safety cap, not a JSON setting: it bounds record overhead for streams split into tiny callbacks. Reaching the next entry resets and closes only that TCP flow. It does not affect UDP flows.

fake-dns object fields:

• enabled (bool, default: true)
• address (IPv4 string, default: "198.18.0.2")
• port (int, default: 53)
• network (IPv4 string, default: "100.64.0.0")
• netmask (contiguous IPv4 /1 through /31 prefix mask, default: "255.192.0.0")
• cache-size (int, default: 10000, maximum: 262144)
• ttl (int seconds, default: 1)

The legacy key mapdns is accepted as an alias for fake-dns. The key fake_dns is also accepted. Exactly one of these spellings may be present; duplicate aliases are rejected even when their values are equal. A disabled object is still validated field by field but does not allocate fake-DNS tables.

address must be a concrete, non-loopback IPv4 unicast source. Wildcard, loopback, multicast, and limited-broadcast addresses are rejected so lwIP never rewrites or emits an unusable reply identity. The listener may share the fake network prefix only when its host index is outside [0, cache-size); it cannot overlap an address allocated to a fake record. The first index immediately after that interval remains valid.

Every setting above is validated rather than defaulted on error. Only an omitted key selects its default; a value of the wrong JSON type, a fractional integer, or one outside the accepted range fails node creation and names the exact JSON path. A rejected DNS query also leaves the cache alone: the whole question section is parsed and validated before the first mapping is created. All unique A/IN names in one query are committed as one cache transaction; if they cannot coexist within the lease policy, the response is SERVFAIL with no fake answers and the old forward map, reverse records, and LRU order remain unchanged. Duplicate questions reuse one mapping. The mask must be contiguous and leave at least one host bit: /32 is rejected because it provides no indexed address space, while /0 necessarily normalizes to the zero range and is rejected by the rule below. A cache may contain at most 2^(32 - prefix) records, including both endpoint addresses of the prefix (/31 therefore supports exactly two). The configured network is masked to its prefix, record index zero maps to that masked address, and indexes carry across IPv4 octets normally. Any range whose first record would be the reserved 0.0.0.0 failure sentinel is rejected. The product limit of 262144 records bounds the eagerly allocated record index and reserved name-map storage to roughly 16 MiB on a 64-bit build. Allocation multiplication, signed map capacity, load-factor growth, and the map's next power-of-two bucket count are all validated before either allocation is attempted.

Names use ASCII letters, digits, hyphens and underscores inside labels. Label boundaries are represented only by DNS label lengths. The parser rejects embedded NUL, literal label dots, control bytes and other unsupported bytes before touching the cache. Accepted names are lowercased, and their explicit byte length equals their C-string length, so policy matching and resolution receive the same destination identity.

Each successfully issued answer protects its mapping until the monotonic issuance time plus its advertised ttl. Every answer refreshes that lease. LRU eviction may reuse only expired, unpinned entries; if all slots are protected, a new mapping is refused with SERVFAIL without changing existing mappings or leases. Expired entries remain available until capacity requires reuse. The supported TTL range remains 0..2147483647 seconds: zero permits immediate reuse, while a long TTL can prevent new-name admission for a long time in a full cache. Clients must not keep using a fake address after its answer TTL, and downstream queueing should remain bounded.

Example

{
  "name": "ptc",
  "type": "PacketsToConnection",
  "settings": {
    "udp-idle-timeout-ms": 120000,
    "max-pending-bytes": 2097152,
    "fake-dns": {
      "address": "198.18.0.2",
      "network": "100.64.0.0",
      "netmask": "255.192.0.0",
      "cache-size": 10000
    }
  },
  "next": "service-chain-entry"
}

Typical Placement

Example packet-to-service chain:

TunDevice <--> PacketsToConnection <--> TcpConnector

or:

WireGuardDevice <--> PacketsToConnection <--> HttpClient

The key idea is that the previous side is packet-oriented, while the next side is normal Waterwall connection-oriented chaining.

Limitations

• IPv6 is not implemented
• upstream packet input must be a complete IPv4 datagram, including traffic addressed to fake DNS. Unsupported fragments are dropped before DNS or stack processing
• the zero-copy receive path wraps each incoming packet in a pooled descriptor. TCP may retain it for out-of-order delivery. The pool remains sized from the shared lwIP budget, TCP flow target and fixed reserve; per-PCB TCP_OOSEQ_MAX_PBUFS and TCP_OOSEQ_MAX_BYTES limits still apply. Exhaustion drops a packet with a rate-limited warning. The process-global pool is initialized once and is not reset by Stop, Destroy or hot replacement
• packet transforms may leave the buffer cursor at any byte alignment, and lwIP reads typed IP/TCP headers straight out of it, so that exceptional case is copied into an aligned buffer and travels through the same custom-pbuf wrapper as everything else. It is deliberately not copied into an lwIP PBUF_RAM: with pooled heap allocation the largest class is 16 KiB including overhead, which silently dropped every valid misaligned IPv4 packet above roughly that size while the aligned path beside it had no such ceiling
• ICMP is not implemented
• UDP pause is lossy, not queued
• IPv4 route/netif contexts are created on demand per packet worker, not per destination IP, and are not currently exposed as a tuning surface
• each worker engine has 255 netif indices, shared by its loopback and attached nodes. Indices may coincide across engines; engine plus exact netif identifies an attachment. Capacity exhaustion refuses creation promptly
• TCP and UDP listeners are pretend wildcard PCBs on the worker-local netif, not one listener per destination port
• fake-DNS configuration validates its contiguous prefix, usable address range, 262144-record product cap, reserved name-map geometry, and record-array byte count before allocating or enabling the cache. Every multi-question response pins existing answers, stages all new mappings and victims, and commits them together; allocation, insertion, or coexistence failure returns no fake answer and leaves existing usage, LRU order, forward map, and reverse records unchanged
• this node assumes packet payload really is IP traffic and does not validate every malformed edge case beyond conservative basic checks
• this node is not a packet-framing tunnel and does not add per-payload headers; its node declares no extra left-padding requirement

Difference From PacketsToStream

Use PacketsToStream when you only need to preserve packet boundaries over a stream transport.

Use PacketsToConnection when you need lwIP to reconstruct transport flows and expose them as normal Waterwall lines.

Node Metadata

Source-backed metadata:

Property	Value
node flags	kNodeFlagSupportsTrustedPacketChecksums
can_have_prev	true
can_have_next	true
layer_group	kNodeLayer3 | kNodeLayer4
layer_group_prev_node	kNodeLayer3
layer_group_next_node	kNodeLayer4
required_padding_left	0 bytes

The checksum capability bit lets PacketsToConnection participate in the direct trusted-checksum path with an eligible adjacent Linux TunDevice. Receive trust activates automatically during pair setup; transmit checksum offload additionally requires negotiated support and ready resources.

Engine resources and lifecycle

The runtime lazily creates one raw lwIP engine per ordinary event worker, shared by all bridge instances on that worker. There is no separate stack thread or protocol core mutex. PTC preserves the delivered packet worker; CTP preserves the application line's owner and queues return packets after tuple/generation lookup. Normal-line callbacks and arbitrary packet neighbours remain deferred, including same-worker delivery. Only PTC's immediate certified enqueue-only packet sink has the direct-delivery exception.

Flow/payload pools remain process-wide with their existing capacities. Each active engine adds bounded control tables and 17 timeout records; unused workers allocate no engine. Automatic TCP and UDP ports use separate shared occupancy accounts, held through PCB reclamation, including TIME_WAIT. Generated IPv4 and IPv6 fragment IDs use separate shared sequences with their existing finite-width wrap behavior. Worker-local detached drain lists retain each node's aggregate 4,096-object and 16 MiB allowance. Pool/quota locks never cover stack traversal or output.

Worker quiescence detaches engine timers before normal-loop admission closes. Nodes remove their local stack roots during owner drain; runtime teardown then releases engines after detached messages settle and before loops and pools disappear. Shared pools, occupancy accounts and the process ISN secret finalize after all workers exit.

Direct TunDevice checksum policy

A reciprocal immediate Linux TunDevice -> PacketsToConnection pair automatically trusts incoming IPv4 header and TCP/UDP checksums in raw, checksum-only and GSO modes, independently of gso. Each lazy worker netif reads the published receive mode; other PTC netifs, ConnectionToPackets, interposed nodes and other platforms retain ordinary verification. The kernel-input trust assumption is described in Developer Guide Part 4. No extra setting is needed.

TUN, the active PTC netif and fake DNS omit incoming IPv4 header and TCP/UDP checksum verification, including unmarked input. Structural parsing, exact lengths, metadata validation, alignment copies, fragment guards and repair requests remain enforced. IPv6 and UDP-Lite verification remain enabled. Positive offload assurance stays private to TUN work; receive trust does not mark unassured input as metadata-verified.

Receive trust and transmit offload are independent. Raw output retains full software checksums and TCP checksum-on-copy. Only when Linux checksum offload and its resources are ready may eligible transport output and TCP checksum-on-copy work be omitted. Absent caches fall back to full pbuf checksums when required. IPv4 output header checksums are still generated. Oversized output, including fake DNS replies, is checksummed before IP fragmentation. In negotiated offload mode, TUN writes complete packets as NEEDS_CSUM with the correct pseudoheader seed; fragments retain ordinary metadata. Modes are immutable through quiescence and owner-worker drain.