Skywire Development Update — September 5, 2026

The day’s through-line ran from a bare continue statement to a visor’s reward eligibility. Three readers in the transport discovery skipped a failed batch silently, and because those counts decide online status and uptime feeds the reward threshold, a short read could push live visors under the bar while still returning a plausible number. Fixing that fed a larger rebuild of the reward server’s statistics pages, which moved off HTTP-over-dmsg onto CXO feeds and gained panels for the dmsg substrate, service reachability, transports per visor, and the route setup nodes that negotiate every route in the network and had never appeared on any chart.

Elsewhere: profiling a host visor carrying 460 live WebRTC peer connections found pion’s mDNS and default interceptors accounting for 48% of its 18,111 goroutines, none of it ever asked for; autoconnect was found trying stcpr against at most five of the network’s fifteen public visors, so which ones a given visor reached over a direct carrier was a lottery; and --direct turned out never to have bypassed the route finder that three separate code comments claimed it did. The day closed with three services moving onto one shared CXO aggregator core and nine packages leaving the tree for modules of their own.

Skywire: Counts That Decide Rewards

TPD assembles its transport counts from batched reads over the transport index, and three of those readers skipped a failed batch with a bare continue. With a batch size of ten thousand against an index observed between roughly 4,400 and 15,800 members, that is one or two batches — so losing one silently removed up to ten thousand transports while still returning a number that looked ordinary, and downstream a short read is indistinguishable from the network shrinking, which is how a stable network reads as a collapsing one.

4542 handles each caller according to what a wrong answer costs there. The per-peer counts that decide online status now return an error and propagate it, because uptime feeds the reward threshold and a short read marks live visors offline at their operators’ expense; refusing to answer is the only safe option. The summary still returns its partial result but flags it as partial with the count of missing batches, so a publisher or a chart can treat the sample as untrustworthy rather than as data. The scope is stated honestly in the change itself: this is necessary for any count to be trustworthy and is not proven sufficient for the index oscillation that motivated it, since Redis currently reports no failed batch reads at all, and instrumenting index cardinality on a timer is the next step so the next swing carries evidence rather than being reconstructed afterwards.

4536 fixes the visible face of the same class of problem, a cliff at the right edge of the visors-online chart: the final five-minute slot is normally still being written, and sampled minutes apart it read 808 in one fetch and 114 in another while the preceding hour sat steady near 880 — the network had not moved, the bucket was incomplete. Trailing slots below half the median of the preceding hour are now dropped, bounded to six slots so a real outage is never eaten. It also fixes a section that had been vanishing outright, its struct field and renderer both present while nothing ever populated it, which is precisely the failure mode the design exists to make impossible.

Skywire: The Statistics Page, Rendered as a Terminal

4543 moves the statistics pages onto CXO. They had fetched every TPD aggregate over HTTP-over-dmsg, and those fetches were failing intermittently with EOF — a 2.7KB body and a 1,400-character URL were both observed failing on the live site — because the server’s long-lived dmsg client periodically loses its sessions and every request issued in that window dies outright, where a CXO subscriber reads a snapshot it already holds, from memory, whole-object-or-nothing. The pages read the feeds first over the client the server already has, with HTTP kept behind them and the source printed on the page, because a stale snapshot and a fresh fetch are not the same claim. Per-visor bandwidth is summed locally by edge public key, which deletes the enormous request rather than shortening it, and TPD’s daily aggregate gains a feed of its own on a five-minute cadence, since its 2.7KB body costs a thirty-day store query to produce.

Two reads stay on HTTP because no feed can carry them, and the page says so. On top of that base, 4545 adds a transports-per-visor histogram, an architecture pie rasterized in-tree, and the per-visor uptime timeline — the histogram gated rather than drawn unconditionally, because a distribution taken mid-refill draws a network whose visors each hold half the transports they hold, and where a line chart at least shows the fluctuation a histogram renders one sample as a static shape. It refuses to draw on the store’s partial-read flag, on the feed’s completeness stamp, or when the per-key index fails to sum to twice the aggregate, and prints the reason instead. The architecture panel prints its provenance too, since it comes from operator surveys rather than TPD, whose heartbeat carries a version string and no architecture field at all, making the surveyed population a subset of the visors being counted.

4539 adds the dmsg substrate, which nothing on the site had shown: per-server spare capacity, fullest first, and registered client entries — and the labeling is the point, since the discovery’s figure is capacity remaining, not load, and reading it as load inverts the meaning and makes the fullest server look like the emptiest. That inversion is how a saturated server was recently reported as having no clients, while it sat at its 600-session maximum attracting around 750. 4541 checks a premise nothing had been checking: the deployment services deliberately publish no discovery entry, so they are reached through seeded delegated-server entries and are equally reachable through whichever server a caller already holds a session with — which means a service connected to only some of the servers stays perfectly reachable for clients on those servers and silently unresolvable for the rest, with no error anywhere and no entry for the discovery to report a gap against. Health already carries live connected-server state, so comparing it against the full list gives coverage per service and names the servers whose clients cannot reach it; transport discovery currently reads nine of nine.

4554 adds the route setup nodes, whose absence was the starkest gap: every route in the network is negotiated by one, and a setup node refusing setups leaves no mark on any existing chart, since the transports stay registered and the visors stay online while routing across them quietly stops working. The panel enumerates nodes from the deployment config rather than a hardcoded key and draws request volume, success rate, latency percentiles, failures by reason and the route-length histogram, with an unreachable node an explicit row and every empty sample saying why — a zero percent success rate over zero attempts is an idle node, not a failing one.

4559 then explains why four panels had been rendering “unavailable” in production at all: the reward server started its dmsg client with the constructor that seeds no service keys, so every stats fetch did a discovery lookup for a service that deliberately answers none, and the split was visible on the page with everything sourced from CXO rendering and everything still on HTTP failing. It also removes a probe of the retired standalone uptime tracker, which had been reporting a service that is not supposed to exist as unreachable on every page load.

Skywire: Half the Goroutines Belonged to pion

Profiling a host visor with 460 live WebRTC peer connections showed pion accounting for roughly half of its 18,111 goroutines: 6,960 in mDNS and 1,856 in the interceptor package, 48% between them, and both were being paid without ever having been asked for. Constructing the API registers the default interceptors unless a registry is supplied — passing only a media engine does not suppress them — and the ICE layer defaults an unset mDNS mode to query-only, which creates the mDNS connection.

The existing comment claiming an empty media engine suppressed all this was simply wrong. 4544 turns both off, with the reasoning that skywire uses WebRTC for data channels only and addresses peers by public key, exchanging ICE candidates over a dmsg signaling stream, so neither the media interceptors nor .local name resolution is ever used; the remaining SRTP goroutines are not reachable from here, since pion starts them unconditionally once DTLS connects. The test asserts no such goroutines exist after a peer connection gathers, and was confirmed to fail when either half is reverted. The measured drop needs a visor restart and has not been observed live. There is one behavioral trade, which is why it initially shipped with an escape hatch: disabling mDNS also discards remote mDNS candidates, so a browser visor on the same LAN as a native visor loses that direct path.

4546 removed the escape hatch a few merges later on the grounds that it was the wrong shape — whether a data-channel-only mesh carrier resolves .local peer names is a policy decision, not a per-deployment knob, and that narrow same-LAN case is not worth 38% of the process. Loopback is unaffected either way, so the local development loop is untouched. 4549 removes a second, smaller leak in the same carrier: each WebRTC transport held its dmsg signaling stream open for the transport’s whole lifetime, one reserved dmsg port apiece, with nothing reading it — the pump goroutine is bound to the dial context and has already returned by the time the DataChannel opens. Measured on a host visor: four pumps against 35 transports, so 31 streams open and unread, a third of that visor’s 94 dmsg ports. Signaling only carries the offer, answer and trickle candidates, so the stream is closed at handoff; a peer still sending candidates is unaffected, and an ICE restart was already impossible since the reader was gone.

Skywire: The Carrier Nobody Tried, and a Direct Dial That Was Not

Autoconnect only ever attempted stcpr against the five-or-fewer public visors an earlier phase had already selected from a shuffle, so which public visors a given visor reached over a direct carrier was a lottery and every other one fell through to the WebRTC last resort. That is measurable: of one host visor’s 88 WebRTC transports, 81 were inbound, and none of those 81 peers had any other transport to it — yet asking their visors to dial stcpr succeeded eight times out of eight, and a watcher that caught four more WebRTC transports as they were created found stcpr and squicr available to all four at that exact moment. WebRTC had never been a fallback from failure; those peers simply never tried the cheaper carrier.

4550 uncaps it, and the blast radius is small and known: there are fifteen public visors on the network and all fourteen reachable ones accept stcpr on request, verified against every one including the three with the lowest transport counts, so a visor’s public-visor stcpr transports rise from at most five to at most about fifteen. The legacy budgets on the other direct carriers go with it; QUIC and the WebRTC last resort keep their caps. 4553 fixes a claim that three separate comments made and nothing implemented: --direct was documented at the hop-count downgrade, at the transport-ensuring call and at the app-server flag site as bypassing the route finder, but the flag was consumed exactly once, to skip the transport-creation hooks.

Every direct dial over an existing transport still made a route-finder round trip, and the finder handed back, among other candidates, the very transport the dial was already holding. A local one-hop route is now built instead, guarded so that it only applies when a transport to the destination is already known and neither the per-dial options nor the visor’s global minimum-hop setting asked for more than one hop — so an operator on three hops is never silently handed one — with setup-labeled and closed transports excluded and nothing-found falling through to the finder rather than failing the dial. The setup-node handshake is unchanged, since installing rules on the remote edge has no peer-direct path.

4551 makes the population visible with no new data source and no extra request, since pv already fetches the uptime rows its online filter needs and those rows carry a version that was simply never displayed; a visor with no row renders a dash rather than an empty column, so a gap reads as a gap. Verified live: fifteen public visors spanning v1.3.91 through v1.3.94-0.

Skywire: Router Arithmetic, and a Dial Storm From One Tab

4555 fixes a throughput calculation that returned garbage for a zero-width sampling window. Go leaves float-to-integer conversion undefined for NaN and for out-of-range values, and on amd64 it yields the smallest representable integer — so two samples inside a single clock tick produced either infinity or NaN and returned a large negative number. That value does not stay local: it goes into a ping packet and is sent to the peer, so a nonsense negative throughput propagated into the remote’s view of the leg. Windows CI hits this deterministically because its timer granularity is coarse, but the window is only ever usually non-zero on Linux, never guaranteed. A zero-width or backwards window — an NTP step — now returns zero and deliberately does not open a new sampling window, so the bytes counted so far survive for the next real measurement rather than being swapped away by a poll that measured nothing.

4556 cleans up the last Windows failure behind it, a test that required a gap’s age to exceed zero immediately after the call that opened it, which needs the clock to tick between two adjacent statements; the test now asserts the invariant it actually meant, since every production caller compares that age against a threshold where a brand-new gap reading zero is correct and must not trip a stall detector.

4558 traces a production setup node failing about half its route setups to a single browser visor, which accounted for 244 requests against one exit, 84 against another and 62 against a third. Two compounding defects, both in code with the right machinery already sitting next to it: the auto-select loop retried on a fixed two-second timer with no backoff and exited only on success, which against a mesh that cannot yet satisfy the pool is an unbounded dial storm rather than a retry, and a failed probe simply continued rather than cooling the exit down, so the loop re-picked the same failing exits every round. Backoff now runs from two to sixty seconds and failed exits are cooled, which is what lets successive rounds diverge. The file is js/wasm-only, so the effect on setup-node load can only be confirmed against a deployed browser visor.

Skywire: dmsg Says What It Does Not Do

Three changes make the dmsg client explicit about capabilities it had been expressing by accident. 4564 adds a way to say “do not register”, which previously could only be obtained by handing the client a discovery whose writes are silent no-op successes: the registration loop then ran on its normal interval, got a nil error every time, and the client believed it had advertised itself while being unresolvable by lookup — no error, no warning. The flag is gated inside the publish itself rather than only in the loop, so the first-session callback and the nudge path are covered too, and the semantics are worth being exact about, since the name invites the wrong reading: not registering does not make a client undialable, only unresolvable by lookup. Anyone who knows or is seeded with the servers it sits on still reaches it, which is the right trade for a read-only consumer with a fixed destination set and for a deployment running with no discovery at all.

4563 closes the one remaining hole in the dial ladder — a peer that is in no discovery, was never seeded, and sits on a server this client holds no session to — with an opt-in sweep that establishes a session to each server in turn and tries the destination through it. It is opt-in and the change argues it should stay that way: the existing automatic fallback costs one stream per session already held, whereas a sweep costs a full Noise handshake per server, so firing it automatically on every failed resolution would turn one typo or one dead key into a handshake storm against the whole fleet and defeat the dial-failure backoff that exists for exactly that reason. Reaching the one case it fixes needs a peer that is unregistered, unseeded and on an unconnected server, so it is not verified live.

4565 adds a resolver seam ahead of the discovery, which is what makes CXO-backed entry resolution buildable at all, since the CXO package imports the dmsg client and the dependency can therefore never run the other way — the resolver has to be injected by something that imports both. Each call is bounded at 500ms, and that bound is the point rather than a detail: a resolver sits on the dial path and a CXO-backed one can be cold, syncing or wedged, so without it installing one would make every dial wait on a feed sync, and a resolver that answers “I do not know” quickly is always better than one that answers slowly. Installing one is a per-deployment judgment with the cost stated in numbers — a resident copy of a mapping currently holding 930 client entries and growing with the network — right for a setup node or a hypervisor, wrong for a microcontroller or a browser tab. The change also removes the DHT lookup hooks nothing had ever set, leftovers from what CXO replaced.

4566 then found the CXO lookup path had never worked in the deployment it ships against: the peer resolution read only one config field, and on a dmsg-only deployment that field is unset while the discovery key lives in another, already in the right form — so the feed spec always failed and the resolver silently degraded to HTTP on every lookup. The transport discovery’s equivalent tries both fields, which is exactly why its feeds work and this one never had. 4561 writes the audit down where the next caller will see it, as a decision matrix over what is really a two-by-two: how servers are found, crossed with whether the client publishes its own entry. It records two structural findings rather than fixing them — that the visor builds its client inline and does not use the shared package at all, so the most exercised dmsg setup in the codebase is not the shared one, and that two exported constructors have no non-test callers.

4540 makes one common log line useful: it announced that a discovery lookup had failed without saying for whom, while the destination key was right there. Since the deployment services deliberately publish no entries, every dial to one logs this and then succeeds on the fallback — so the expected case and a genuinely unreachable peer looked identical, which is what made twenty-one of them in thirty minutes of reward-server output uninvestigable.

Skywire: One Aggregator Instead of Three, and Nine Packages Out of the Tree

No visor in the deployment was registering over CXO, and the reason was undiagnosable from either end: the discovery logged a generic rejection and the visor logged nothing at all. 4570 makes the visor say it, registering the publisher for state reporting so visor state names the rejected key directly — the difference between inferring a cause and reading it — and correcting two comments that described the feature as opt-in via a config field that does not exist. 4571 then reads the answer off that surface: the aggregator built its CXO node from a bare config, and with a zero secret key the node constructor mints a random keypair, so the aggregator presented an identity no visor had allowlisted.

The visors were behaving correctly, allowing the configured discovery key and refusing anything else; allowlisting the random key would have been the wrong direction, since it is not the discovery’s identity and nothing should trust it. It needs a discovery redeploy to take effect and has not been confirmed end to end for that reason. That bug is what motivated 4573, which writes the shared fan-in lifecycle once: three services each carried a hand-copied version of node construction, the connect-driven subscribe loop, grace-gated orphan-feed reclaim, cleanup and close, and the copies had drifted invisibly — the transport discovery had already hit the same class of bug once and a listener-collision variant before that. In the shared core the service secret key is a required positional argument rather than an optional config field, so a caller that forgets it does not compile, and the constructor then asserts that the node’s identity is the key’s public key, so a wrong key fails at startup instead of being silently refused by every visor for months.

4574, 4575 and 4577 migrate the three services onto it one at a time, removing 198, 162 and 238 lines respectively with the ingest paths unchanged, and each gains a test constructing the real aggregator over an in-process dmsg environment and asserting it presents its service’s own key — the address resolver had been binding its key correctly all along but had no test for it, which left it one edit away from the same failure. Separately, nine packages left the tree entirely. 4567 moves the HTTP downloader behind skywire-cli got to its own module, 4568 replaces three in-tree copies of code that already lived in modules elsewhere — removing 4.5k lines of source and about 9k of vendor, and picking up glyphs the copied font renderer had been drawing as nothing — and 4579 moves six more, another 4.8k lines, each with the reasoning for why no existing library covers it recorded alongside.

4560 updates the 0magnet dependencies, pinning one explicitly because resolving it to latest moved it backwards to an older tag, and noting the six forks that cannot move that way because each declares its upstream module path in its own go.mod. 4548 clears the lint debt that had been failing every test run on develop since the docs command merged — four unchecked errors and a doc comment that read as a malformed compiler directive — which had left the workflow red for days, so nothing merged in that window was actually checked by it.