symphony

High-performance TLS termination proxy with SNI-based routing, written in Rust (via napi-rs) and exposed as a Node.js native module.

Designed for Linux (x64 + arm64, glibc + musl), and will run on MacOS as well. Pre-built binaries are published for all targets.

Overview

symphony sits in front of your services and:

Terminates TLS per route using per-route certificates (falls back to a listener-level default cert)
Routes by SNI hostname — exact matches, wildcard prefixes (*.example.com), and a catch-all default
Proxies TCP — either terminating TLS (decrypt + forward plaintext) or passing raw TLS bytes through
Balances over Unix Domain Sockets (UDS) using least-connections weighted by thread CPU utilisation, with optional IP session affinity
Limits routes with per-route token-bucket rate caps to prevent any one route from starving others
Protects connections with per-IP token-bucket rate limiting, concurrency limits, CIDR allowlist/blocklist, JA3 fingerprint blocking, TLS handshake timeout, and SNI-required enforcement
Suspends routes — hold incoming connections and fire an event; your code decides whether to proxy or reject each one
Hot-swaps routes and per-listener protection config (CIDR lists, JA3 blocklist, rate limits, concurrency caps, handshake timeout, requireSni) without restarting or dropping existing connections
Scales to ~1 million concurrent connections via SO_REUSEPORT, tokio's multi-thread runtime, and lock-free data structures

Installation

npm install symphony

Pre-built binaries are downloaded automatically for your platform during install. No Rust toolchain required.

Quick start

import { SymphonyProxy } from 'symphony';
import { readFileSync } from 'node:fs';

const proxy = new SymphonyProxy({
  listeners: [{ port: 443 }],
  routes: [
    {
      sni: 'api.example.com',
      upstreams: [{ kind: 'tcp', host: '127.0.0.1', port: 3000 }],
      terminateTls: true,
      cert: {
        certChain: readFileSync('/etc/ssl/api.pem', 'utf8'),
        privateKey: readFileSync('/etc/ssl/api-key.pem', 'utf8'),
      },
    },
  ],
});

await proxy.start();
console.log('proxy listening on :443');

Configuration reference

`ProxyConfig`

Field	Type	Default	Description
`listeners`	`ListenerConfig[]`	required	One entry per listening address
`routes`	`RouteConfig[]`	required	SNI routing table
`workerThreads`	`number`	CPU count	Tokio worker threads; also controls `SO_REUSEPORT` socket count per listener
`readBufferSize`	`number`	`8192`	Per-direction copy buffer size in bytes, clamped to `[512, 1048576]`. See Copy buffers and per-connection memory
`clientReadBufferSize`	`number`	`readBufferSize`	Overrides `readBufferSize` for the client→upstream direction only
`upstreamReadBufferSize`	`number`	`readBufferSize`	Overrides `readBufferSize` for the upstream→client direction only
`tcpKeepalive`	`TcpKeepaliveConfig`	on, see below	TCP keepalive for accepted sockets. See Dead-peer detection
`halfCloseTimeoutMs`	`number`	`300000`	Reclaim a connection once its write half to the client is shut down and the surviving client→upstream direction has carried nothing for this many ms. Minimum `1`; `0` disables. See Dead-peer detection

`TcpKeepaliveConfig`

Field	Type	Default	Range	Description
`enabled`	`boolean`	`true`	—	`false` turns dead-peer detection off entirely
`idleMs`	`number`	`300000`	`≥ 1000`	Quiet time before the first probe
`intervalMs`	`number`	`30000`	`≥ 1000`	Gap between probes
`retries`	`number`	`5`	`1–127`	Unanswered probes before the connection is dropped

Out-of-range values are rejected by the constructor rather than clamped. The two timings reach the kernel as whole seconds (TCP_KEEPIDLE / TCP_KEEPINTVL), so a sub-second value would be installed as 0 and refused by setsockopt after SO_KEEPALIVE had already been enabled — leaving a listener that reports a schedule it is not running. retries has the same problem above TCP_KEEPCNT's ceiling of 127.

`ListenerConfig`

Field	Type	Default	Description
`host`	`string`	`'0.0.0.0'`	Bind address
`port`	`number`	required	Bind port
`mode`	`'tls' \| 'http'`	`'tls'`	Listener protocol. `'http'` serves plaintext for ACME HTTP-01 challenges; everything else gets a 301 to `https://<host><uri>`. See HTTP listener.
`defaultCert`	`CertConfig`	—	Fallback cert for routes without their own cert
`mtls`	`MtlsConfig`	—	Listener-level mTLS, used when a route doesn't override it
`maxConnections`	`number`	`0` (unlimited)	Drop new connections when active count reaches this
`idleTimeoutMs`	`number`	`60000`	Close connections silent for this many ms
`protection`	`ProtectionConfig`	—	IP-level protection
`failoverIngress`	`{ trustedSources: string[] }`	—	Makes this listener the receiving end of peer failover. See Peer failover.

`RouteConfig`

Field	Type	Default	Description
`sni`	`string`	required	Hostname for exact match, or `'*.suffix'` for wildcard, or `''` for default (maximum 255 UTF-8 bytes; no control characters; exact duplicates remain last-wins, later wildcard duplicates are skipped, even when the winning entry is rejected)
`metricsGroup`	`string`	—	Stable grouping key for aggregating several domain routes as one tenant (maximum 128 UTF-8 bytes; no control characters)
`upstreams`	`Upstream[]`	required	Destination(s); multiple UDS upstreams are load-balanced
`terminateTls`	`boolean`	required	`true` = decrypt TLS; `false` = TCP passthrough
`cert`	`CertConfig`	—	Per-route cert, overrides listener `defaultCert`
`mtls`	`MtlsConfig`	—	Per-route mTLS, overrides listener `mtls`
`suspended`	`boolean`	`false`	Hold connections and emit `'suspended'` events
`suspendTimeoutMs`	`number`	`30000`	Drop held connections after this ms if not resolved
`maxConnectionsPerSecond`	`number`	—	Route-wide new-connection rate cap (token bucket). Connections are silently dropped when exhausted.
`burst`	`number`	`maxConnectionsPerSecond`	Token bucket burst ceiling for the route rate limit
`http2`	`boolean`	`false`	Advertise `h2` in ALPN so clients negotiate HTTP/2. Raw H2 frames flow through to the upstream unchanged. Requires `terminateTls: true`.
`sourceAddressHeader`	`'proxyProtocol' \| 'proxyProtocolV2' \| 'xForwardedFor' \| 'none'`	`'proxyProtocol'` for UDS, `'none'` for TCP	How the real client IP is forwarded to the upstream. See Source address forwarding.
`forwardFingerprint`	`'ja3' \| 'ja4' \| 'none' \| ('ja3' \| 'ja4')[]`	`'none'`	Forward the client TLS fingerprint(s) downstream. See Forwarding the fingerprint.
`protocol`	`'http' \| 'opaque'`	`'opaque'`	The route's application protocol. Required to be `'http'` before `sourceAddressHeader: 'xForwardedFor'` or a header-carried `forwardFingerprint` is accepted — see Source address forwarding.
`allowedSources`	`string[]`	— (any client)	Only clients whose TCP peer address is in one of these CIDRs reach the route. See Per-route source allowlist.
`failover`	`FailoverConfig`	—	Forward new connections to peer nodes while the local upstream is down, and optionally a share of them while the local Harper threads are much busier than a peer's. See Peer failover and Spill-over.

`Upstream`

// TCP upstream
{ kind: 'tcp', host: string, port: number }

// Unix Domain Socket upstream
{
  kind: 'uds',
  path: string,
  ipAffinity?: boolean,      // route same-IP connections to same socket
  ipAffinityTtlMs?: number,  // evict affinity entry after this ms idle (default 300000)
  pid?: number,              // Linux PID of the worker process (enables CPU monitoring)
  tid?: number,              // Linux TID of the worker thread (must be set with pid)
}

`CertConfig`

{ certChain: string | Buffer, privateKey: string | Buffer }

Both fields accept PEM-encoded strings or Buffer. The cert chain may include intermediate certificates.

`MtlsConfig`

{ clientCaCert: string | Buffer, requireClientCert?: boolean }

requireClientCert defaults to true. Set to false to accept connections without a client cert while still validating those that do present one.

`ProtectionConfig`

Field	Type	Default	Description
`rateLimit`	`{ connectionsPerSecond, burst? }`	—	Per-second token bucket per source IP
`sustained`	`{ connectionsPerMinute, burst? }`	—	Per-minute token bucket per source IP (independent of `rateLimit`)
`penaltyBox`	`{ durationMs? }`	—	Block an IP for `durationMs` after any rate limit exhaustion
`maxConcurrentPerIp`	`number`	`0` (unlimited)	Max simultaneous connections per source IP
`allowlist`	`string[]`	`[]`	CIDRs that bypass all checks. IPv4 clients of a dual-stack listener are matched as IPv4 (see below)
`blocklist`	`string[]`	`[]`	CIDRs that are always blocked. IPv4 clients of a dual-stack listener are matched as IPv4 (see below)
`ja3Blocklist`	`string[]`	`[]`	JA3 MD5 hex fingerprints to block (32 chars each)
`ja4Blocklist`	`string[]`	`[]`	JA4 TLS fingerprints to block (36-char strings; see JA4 blocking)
`tlsHandshakeTimeoutMs`	`number`	`10000`	Abort slow TLS handshakes
`requireSni`	`boolean`	`false`	Reject connections without an SNI extension

A dual-stack ([::]) listener reports an IPv4 client as ::ffff:a.b.c.d. Protection treats it as a.b.c.d: only IPv4 allowlist/blocklist entries match it (::/0 does not; use 0.0.0.0/0 alongside it), it shares rate-limit, penalty and concurrency state with that IPv4 address, and blockedIps() reports the IPv4 form. An IPv4-mapped entry (::ffff:10.0.0.0/104) is stored as its IPv4 form (10.0.0.0/8); one with a prefix under 96 is rejected. The 'blocked' event still reports the address as accepted.

TLS & mTLS

Per-route certificates

Each route can have its own certificate. Routes without a cert use the listener's defaultCert.

const proxy = new SymphonyProxy({
  listeners: [{
    port: 443,
    defaultCert: { certChain: wildcardCert, privateKey: wildcardKey },
  }],
  routes: [
    // Uses its own cert
    { sni: 'special.example.com', cert: { certChain: specialCert, privateKey: specialKey }, ... },
    // Falls back to listener defaultCert
    { sni: '*.example.com', ... },
  ],
});

mTLS

const proxy = new SymphonyProxy({
  listeners: [{
    port: 443,
    mtls: { clientCaCert: readFileSync('ca.pem', 'utf8'), requireClientCert: true },
  }],
  routes: [
    {
      sni: 'internal.example.com',
      terminateTls: true,
      cert: { certChain, privateKey },
      // Inherits listener mTLS; or override per-route:
      // mtls: { clientCaCert: ..., requireClientCert: false },
    },
  ],
});

TLS passthrough

Set terminateTls: false to forward raw TLS bytes to the upstream without decryption. No cert needed.

{ sni: 'passthrough.example.com', terminateTls: false, upstreams: [{ kind: 'tcp', host: '10.0.0.5', port: 443 }] }

Routing

Routes are checked in order: exact match → wildcard suffix → default (empty sni).

routes: [
  { sni: 'api.example.com', ... },        // exact
  { sni: '*.example.com', ... },          // matches foo.example.com, bar.example.com
  { sni: '', ... },                        // catch-all default
]

The default route serves a connection that sends no SNI (including a plain-TCP client on a passthrough route, which sends no ClientHello at all) and one whose SNI matches no exact or wildcard route. It is never used for:

a name whose own exact or wildcard route was rejected: that name resolves to nothing;
a failover ingress listener, which closes an unmatched name before the handshake so the forwarding node tries its next peer;
an ACME challenge on an HTTP listener: only a route configured for the Host can answer one, so the default route's upstream can't validate certificates for names it doesn't serve.

requireSni: true still blocks a connection with no SNI before any route is chosen. A failover policy on the default route is ignored (logged and counted in failingRoutes): peers would refuse every connection it forwarded.

Suspended routes

Use suspended routes to inspect or authorize connections before proxying them:

proxy.on('suspended', async (conn) => {
  // conn.id, conn.sni, conn.peerIp, conn.peerPort, conn.listener
  const allowed = await checkAuthority(conn);

  if (allowed) {
    proxy.resolveConnection(conn.id, {
      upstreams: [{ kind: 'tcp', host: '127.0.0.1', port: 3000 }],
      terminateTls: false,
    });
  } else {
    proxy.resolveConnection(conn.id, null); // reject — TCP close
  }
});

// Route declared as suspended
{ sni: 'gated.example.com', suspended: true, upstreams: [], terminateTls: true, cert: { ... } }

Connections not resolved within suspendTimeoutMs are dropped automatically. Calling resolveConnection with an unknown or already-expired ID is a no-op.

HTTP listener (ACME + HTTPS redirect)

Setting mode: 'http' on a listener switches it from the default SNI-routed TLS proxy to a plaintext HTTP/1.1 handler intended for port 80. It serves two purposes:

ACME HTTP-01 challenges — requests under /.well-known/acme-challenge/ are proxied to the route matched by the Host header (using the same wildcard rules as SNI matching). The upstream is the route's first upstream, so a single route table covers both :443 TLS routing and :80 ACME proxying.
HTTPS redirect — every other request gets HTTP/1.1 301 Moved Permanently with Location: https://<host><request-target>.

Requests with no Host header return 400 Bad Request. ACME requests for hosts with no matching route return 404 Not Found.

new SymphonyProxy({
  listeners: [
    { host: '0.0.0.0', port: 443 },                         // TLS proxy
    { host: '0.0.0.0', port: 80, mode: 'http' },            // ACME + redirect
  ],
  routes: [
    {
      sni: 'api.example.com',
      upstreams: [{ kind: 'uds', path: '/var/harper/api.sock' }],
      terminateTls: true,
      cert: { certChain, privateKey },
    },
  ],
});

defaultCert, mtls, and protection on an HTTP-mode listener are ignored.

UDS load balancing

Provide multiple uds upstreams for a route. symphony picks the socket with the lowest score, where score is:

score = active_connections × 1000 + cpu_utilisation_permille

Active connections are the primary factor; CPU utilisation (0–1000, representing 0–100%) is a tiebreaker that steers new connections away from overloaded threads when connection counts are equal.

upstreams: [
  { kind: 'uds', path: '/run/app/worker-0.sock' },
  { kind: 'uds', path: '/run/app/worker-1.sock' },
  { kind: 'uds', path: '/run/app/worker-2.sock' },
]

If the picked socket refuses the connection (not listening, file missing, accept queue full, or connect timeout), symphony tries each remaining socket once, best score first, within the same connect timeout. One dead worker socket then costs a failed connect instead of the client's connection.

IP session affinity

Add ipAffinity: true to any UDS upstream entry to pin source IPs to the same socket:

upstreams: [
  { kind: 'uds', path: '/run/app/worker-0.sock', ipAffinity: true, ipAffinityTtlMs: 300000 },
  { kind: 'uds', path: '/run/app/worker-1.sock', ipAffinity: true },
]

The same ipAffinity / ipAffinityTtlMs values apply to all sockets in the set (values from the first entry are used for the shared balancer).

Thread CPU utilisation monitoring

When each UDS upstream serves a known worker thread, symphony can read its CPU utilisation from /proc/{pid}/task/{tid}/stat and incorporate it into socket selection:

upstreams: [
  { kind: 'uds', path: '/run/app/worker-0.sock', pid: 12345, tid: 12346 },
  { kind: 'uds', path: '/run/app/worker-1.sock', pid: 12345, tid: 12347 },
  { kind: 'uds', path: '/run/app/worker-2.sock', pid: 12345, tid: 12348 },
]

Symphony samples /proc/{pid}/task/{tid}/stat every 250 ms and computes the thread's CPU utilisation over the interval. Sockets without pid/tid keep a CPU score of 0 and fall back to pure least-connections. A thread that can no longer be read (process exit, crash) scores 0 until it is found again; symphony never keeps a value for a thread that is gone.

pid/tid may be the ids the worker sees in its own PID namespace, as Harper in a container reports them (process.pid and gettid()). Before sampling a socket, symphony connects to it and closes at once, reads the listening process's host pid from the socket credentials (SO_PEERCRED), and, when that process's NSpid: shows the configured pid as its namespaced pid, maps the configured tid to the host thread the same way. Every 30 s it checks that the socket file is still the same one (its inode) and connects again only when it is not; an unresolved socket (around a restart) is retried every 5 s. All of this runs in the background, so sampling never waits on it. The thread's start time is recorded with it, so a reused id is detected on the next sample and the socket is looked up again. Configured ids that do not name the socket's owner at all are still sampled as given, for socket selection only; they never count toward a route's load (see spill-over).

Per-route rate limiting

Use maxConnectionsPerSecond on a route to cap the rate of new connections accepted for that route, independent of source IP. This prevents a single busy route from starving other routes under high load:

routes: [
  {
    sni: 'api.example.com',
    maxConnectionsPerSecond: 500,  // route-wide cap; burst defaults to this value
    burst: 1000,                   // allow short bursts up to 1000 conn/s
    upstreams: [{ kind: 'uds', path: '/run/app/api.sock' }],
    terminateTls: true,
    cert: { certChain, privateKey },
  },
  {
    sni: 'admin.example.com',
    maxConnectionsPerSecond: 20,
    upstreams: [{ kind: 'uds', path: '/run/app/admin.sock' }],
    terminateTls: true,
    cert: { certChain, privateKey },
  },
]

Connections that exceed the limit are silently dropped (TCP RST). This is a global token bucket per route — not per IP. For per-IP rate limiting use protection.rateLimit.

Per-route source allowlist

allowedSources restricts a route to a set of client addresses. Several tenants can share one listener and still each have their own list, since the list belongs to the route the SNI selects:

routes: [
  {
    sni: 'tenant-a.example.com',
    allowedSources: ['203.0.113.0/24', '198.51.100.7', '2001:db8:1::/48'],
    upstreams: [{ kind: 'uds', path: '/run/tenant-a.sock' }],
    terminateTls: true,
    cert: { certChain, privateKey },
  },
]

Entries are IPv4 or IPv6 CIDRs; a bare IP is a single host. Absent or empty = any client.
The client address is the TCP peer symphony sees. Inbound PROXY headers and X-Forwarded-For are never consulted. An IPv4 client on a dual-stack ([::]) listener (::ffff:a.b.c.d) is matched against the IPv4 entries only, so ::/0 does not admit it; an IPv4-mapped entry (::ffff:10.0.0.0/104) is treated as its IPv4 form (10.0.0.0/8).
A refused connection is closed (TCP RST) right after route selection: before the TLS handshake, the upstream connect, the route's maxConnectionsPerSecond bucket and any 'suspended' event. It is counted as source_not_allowed in the route's and the listener's blockedByReason, and no 'blocked' event is emitted.
On a suspended route, the route's own list decides whether the connection is suspended at all. A resolveConnection() route may carry its own allowedSources, checked in addition before the handshake and upstream connect; an invalid entry there rejects the resolution with an 'error' event.
updateConfig() changes apply to new connections. The list is checked once, when a connection is admitted, so established connections are not dropped when their address is removed.
An invalid entry drops the route (it refuses every connection and is counted in failingRoutes) instead of failing the whole new SymphonyProxy() / updateConfig() call for every other route. A rejected route never falls through to a wildcard or default route.
The list applies on every listener that resolves the route, including an HTTP listener's ACME challenge proxy. A list on a route that answers ACME HTTP-01 challenges therefore also refuses the CA's validators, whose addresses are not published.
It restricts a route, not the upstream behind it. The client picks the SNI, so a second route reaching the same upstream (an alias, or a wildcard that also matches) is a way around a list it doesn't carry — and symphony does not compare the HTTP Host header with the SNI. Give every route to a restricted upstream the same list.
This is unrelated to protection.allowlist, which exempts listener-wide addresses from protection checks rather than restricting anything.
An older symphony ignores the field, admitting every client. A config producer should not count on enforcement until every process accepting on the port runs a version that supports it.

Peer failover

Every node of a cluster serves the same hostname with the same certificate. While a route's local upstream is down (Harper restarting, crashed, deploying), symphony can send that route's new connections to another node instead of failing them, so a restart needs no load-balancer change.

// On every node: the public listener, plus an ingress listener that only peers may reach.
listeners: [
  { port: 443 },
  { port: 9443, failoverIngress: { trustedSources: ['10.0.1.0/24'] } },
],
routes: [{
  sni: 'app.example.com',
  terminateTls: true,
  cert,
  upstreams: [{ kind: 'uds', path: '/run/harper/http.sock' }],
  failover: {
    peers: [{ host: '10.0.1.12', port: 9443 }, { host: '10.0.1.13', port: 9443 }],
    // drain: true,              // forward every new connection, whatever the local health
    // probeIntervalMs: 1000,
    // peerCooldownMs: { min: 5000, max: 60000 },
    // spill: {},                // also move load to less-busy peers (see Spill-over below)
  },
}],

When a connection is forwarded. After the ClientHello peek, protection, route lookup, allowedSources and the route rate limit, and before the TLS handshake: if the route has failover, is not suspended, and either drain is set or every slot of its local upstream (or of its h2 upstream) is refusing connections. A slot is marked refusing when a connect to it is refused, finds no socket file, finds its accept queue full, times out, or finds its host or network unreachable or down (EHOSTUNREACH, ENETUNREACH, ENETDOWN, EHOSTDOWN; a TCP connect to a stopped container's bridge address fails this way after a few seconds of unanswered ARP). A background probe connects to each refusing slot every probeIntervalMs and closes at once; a successful connect clears the slot. A routeHealth event ({ route, metricsGroup, status: 'up' | 'down' }) reports each transition. A connect proves the socket accepts, not that the application behind it is ready.

A passthrough route (terminateTls: false) also forwards a connection when its local connect fails, on a client listener and unless the route is suspended. Nothing has been read from the client at that point, so the connection that discovers a dead upstream is served by a peer instead of being lost. A slot that fails in one of the ways above is marked, and the next local slot is tried first. Any other connect error, such as EACCES, goes to the peers at once and marks nothing. Either way the failure counts upstream_connect.

How. The connection is forwarded as raw TLS: the peer's symphony terminates it with its own copy of the certificate, so client certificates and ALPN are negotiated with the peer. symphony sends a PROXY v2 header (the real client address) and the ClientHello, then waits for the peer's first byte before reading anything from the client — the ClientHello stays unread on the client socket until then. A peer that refuses, times out, or closes without sending a byte received nothing a retry could duplicate, so the same ClientHello is replayed to the next peer. The peer's first byte commits the connection to it; nothing after that is retried. Peers are tried round-robin, skipping peers in cooldown (per route and peer, starting at min, doubling per consecutive failure up to max, reset by a success), at most once each.

Local upstream down: all attempts share the listener's TLS handshake timeout. If every peer is cooling down, the one whose cooldown ends first is tried once. If no peer takes the connection it is closed (exhausted).
Local connect failed (passthrough): the connect and the peer attempts share one TLS handshake timeout. The connect gets at most half of it, and the peers get whatever it left, as above. After a drain or spill attempt no peer took, the two split what is left of the timeout, and a peer tried before may be tried again.
Draining (drain, local upstream up): peers in cooldown are skipped, the attempts share at most half the handshake timeout (and at most 2 s), and if no peer takes the connection before answering it is served by the local upstream with what is left of the handshake timeout (localFallbacks).

The ingress listener (failoverIngress) is the receiving end:

A connection from outside trustedSources is closed before any byte is read. An empty list admits nobody.
A PROXY v2 header is required (v1, malformed or slow headers are refused). Its source address replaces the TCP peer address for protection, allowedSources and the header sent to the upstream. Its TLVs are ignored. A PROXY v2 LOCAL header is a peer's load query when it names routes, and refused (proxy_header_local) otherwise.
A connection is never forwarded again, and a name with no exact or wildcard route is closed rather than served by a default route. If the route's local upstream is down or draining, the connection is closed before the handshake, which the forwarding node treats as "try the next peer".

Limits and security.

Only new connections are forwarded; established ones close with their upstream.
A TLS-terminating route loses the connection that discovers a dead upstream: it connects after the handshake, which has already read the ClientHello, so there is nothing to replay. Only a failed connect marks a slot (the probe only clears marks), so on a quiet route that is the first connection after every outage. A passthrough route forwards that connection instead.
At a peer, a failed local connect on the ingress listener closes with no bytes, which sends the forwarding node on to its next peer. A suspended route's resolved upstream is not covered either.
On a passthrough route, a TCP upstream slower to accept than half the handshake timeout counts as down: its connections go to peers and the slot is marked. A probe has 1 s to clear the mark, so the route stays failed over while connects take longer than that.
A trusted source can assert any client address, which bypasses per-IP limits and allowedSources. It grants no identity: the peer verifies client certificates itself. On hosts where containers egress with the host's address, block container traffic to the ingress port.
Peers must be IP literals; symphony does not resolve names.
Roll out failoverIngress on every node before any node's routes gain failover: an older listener given a forwarded connection reads the PROXY header as TLS and fails it.
An invalid failover serves the route without failover, counted in failingRoutes. An invalid trustedSources CIDR fails construction or updateConfig().
Both failover (with the routes) and trustedSources (updateConfig({ failoverIngress: [{ port, trustedSources }] })) are hot-swappable; adding or removing failoverIngress on a listener needs a restart (symphony-server recreates the proxy).

Metrics: each failover route's routeMetrics[].failover carries localUp, refusing/total slots, replays, budget exhaustion (exhausted), localFallbacks, and per-peer forwarded counts and failures by reason (refused, timeout, zero_byte_close, error; zero-valued reasons omitted). Ingress refusals are listener block reasons (untrusted_source, proxy_header_invalid, proxy_header_local, proxy_header_v1, proxy_header_timeout); a forwarded connection refused for a down route is the route error locally_down.

On a passthrough route, upstream_connect counts failed local connects, including those a peer then served. Compare it with forwarded and exhausted to see whether clients were affected. A connection no peer took counts one upstream_connect, not two.

Spill-over to a less-loaded peer

DNS without load feedback (latency or geoproximity records with no health checks) keeps sending a busy node its full share. With failover.spill, a route whose local Harper threads are much busier than the same route's on a peer forwards a share of its new client connections to that peer, through the same forwarding path as failover, while its own upstream stays up.

failover: {
  peers: [{ host: '10.0.1.12', port: 9443 }, { host: '10.0.1.13', port: 9443 }],
  spill: {
    // highWater: 0.75,     // local load at which spilling may start
    // margin: 0.2,         // how far below the local load a peer must be to raise the share
    // step: 0.05,          // change of the share per second, up and down
    // maxFraction: 0.5,    // largest share of new connections spilled
    // maxPeerRttMs: 20,    // only peers this close (measured); default: no limit
  },
},

Load. A route's load is the mean CPU (0–1 of one core) of its worker threads, smoothed over about 3 s, from the thread CPU monitor — so its UDS upstreams need pid/tid, and only threads tied to their socket's owner count. A route with a TCP upstream, or whose threads cannot be measured, never spills. With an h2 upstream set, the busier of the two counts. Per-thread CPU is relative to one core: if an instance runs more threads than it has CPU quota, it saturates below 1.0, so lower highWater to match.

Peer loads. About every probeIntervalMs (at least 100 ms), each node sends each peer one PROXY v2 LOCAL header on its failoverIngress port, naming the spill routes it shares (custom TLV 0xE3, up to 64 per query). The peer answers SYMPHONY-LOAD/1 and one line per name: its load in permille for the exact route configured for that name, or - when it would not take a spilled connection (no exact route, draining, local upstream down, or load not measurable). The connect time of the query is the peer's RTT. A report is usable for 3 × probeIntervalMs (at least 3 s); a failed query leaves it to expire, while an answer of - clears it at once. A peer that does not answer — including symphony 1.5.0 and earlier, which closes LOCAL headers — is never a target.

The rule. Every 250 ms, and after each answer, the controller takes the least-loaded peer with a usable report that is not cooling down (and within maxPeerRttMs, when set), and with L the local load and P that peer's:

condition	share of new connections
`L ≥ highWater` and `P ≤ L − margin`	rises by `step`/s, up to `maxFraction`
`L ≥ highWater − margin/2` and `P ≤ L − margin/2`	held
otherwise, or `L` unknown	spilling stops at once; the share decays by `step`/s

Each new connection spills with that probability. Whatever the share, nothing spills once the local or peer reading behind the decision is older than its freshness window, so a stalled poll stops spilling by itself. Because spilling needs the peer at least margin/2 below, two nodes reading each other's loads correctly cannot spill to each other at once; with delayed readings both can briefly, and each share then moves at most step per second.

What a spilled connection costs. It is relayed through the busy node for its whole life: an extra network hop, and cross-region egress if the peer is far. If the peer refuses, times out, or closes before answering (within half the handshake timeout, at most 1 s), the connection is served locally with what is left of the handshake timeout, and the peer cools down. After the peer's first byte the connection is committed, as with failover: a peer that then fails (for example its upstream went down after its last report) loses that connection. Connections that arrived on an ingress listener never spill, and only new connections move. Where DNS steering with load feedback exists (Akamai GTM), it remains the better tool; spill-over complements it for spikes of a few seconds.

Exact-SNI routes only: a wildcard's load is not each matching name's load on a peer, where an exact route can override it. An invalid spill (or one on a wildcard route) disables only spill-over, counted in failingRoutes; down and drain failover still apply.
The spilled share, its target, the local load, and each peer's last report and RTT are in routeMetrics[].failover.spill and .peers[] (spilled, load, rttMs), and on the admin endpoint as symphony_failover_spill_fraction, symphony_failover_local_load, symphony_failover_spilled_total, symphony_failover_peer_load and symphony_failover_peer_rtt_seconds. Load queries answered are the listener counter loadQueries (symphony_listener_load_queries_total).

Source address forwarding

Use sourceAddressHeader on a route to control how the real client IP is communicated to the upstream. The PROXY protocol carriers work whether or not TLS is terminated (they prefix the connection); 'xForwardedFor' requires terminateTls: true and a plaintext HTTP/1 upstream, since it rewrites the HTTP request.

Value	Behaviour
`'proxyProtocol'`	Sends a PROXY protocol v1 (text) header (`PROXY TCP4 <src-ip> <dst-ip> <src-port> 0\r\n`) before any application data. Default for UDS upstreams.
`'proxyProtocolV2'`	Sends a PROXY protocol v2 (binary) header before any application data. v2 adds a TLV section — the carrier for `forwardFingerprint` below and for mTLS client cert forwarding. Keep it opt-in: the consumer must speak v2 (nginx/HAProxy do; Harper core's UDS reader parses v1 only before Harper 5.2).
`'xForwardedFor'`	Reads and rewrites every request on the connection (not just the first), inserting an `X-Forwarded-For` header after the request line — a pipelined or keep-alive request must be parsed too, or a later request could smuggle a spoofed header past the first-request-only rewrite. Default for TCP upstreams (disabled). Requires `protocol: 'http'` on the route (below).
`'none'`	Does not forward source address information. Default for TCP upstreams.

Declaring the route protocol

sourceAddressHeader: 'xForwardedFor', and forwardFingerprint under any mode other than 'proxyProtocolV2', rewrite an HTTP/1 request — they need a route that says it actually carries HTTP. Set protocol: 'http' to opt in:

{
  sni: 'app.example.com',
  upstreams: [{ kind: 'uds', path: '/run/app/worker.sock' }],
  terminateTls: true,
  cert: { certChain, privateKey },
  sourceAddressHeader: 'xForwardedFor',
  protocol: 'http', // required — omitting this is a config error, not a silent no-op
}

protocol defaults to 'opaque' — a route for a non-HTTP application protocol (MQTT, or any other raw TCP/TLS protocol), limited to the PROXY-protocol carriers, which work on any byte stream. The declaration exists because ALPN can't stand in for it: a native protocol that negotiates no ALPN (MQTT does not) is indistinguishable at the TLS layer from an HTTPS client that simply didn't offer one. A route that requests a header-injection mode without declaring protocol: 'http' is rejected with a descriptive error — it never silently stops injecting the header, since a backend that silently sees the wrong (or no) client IP is worse than a route that refuses to serve traffic at all.

This declaration only helps when a header could actually be injected in the first place. A passthrough route (terminateTls: false) never decrypts the stream, so a header-carried mode has no carrier at all regardless of protocol — declaring 'http' on a passthrough route wouldn't make it work, and is rejected with a distinct "no carrier" error instead of steering you toward a declaration that can't help. Use sourceAddressHeader: 'proxyProtocolV2' there instead (it works on any byte stream, passthrough included).

Like a cert that fails to build, a route rejected for either reason is isolated to just that route: it's dropped (its SNI resolves to nothing) rather than failing new SymphonyProxy() or updateConfig() for every other route on the same port-set — one tenant's config mistake doesn't take the rest down with it. The rejection is still loud: it's logged (symphony: skipping route '<sni>': ...) and counted in failingRoutes. A rejected declaration is dropped on a hot-swap too, since it won't heal until the config is edited. A cert that fails to build on a hot-swap (typically a cert and key rotating as two separate files) instead keeps the route's last-good version serving until the pair is consistent again, but only while the route still asks for the client-certificate policy that version enforces (passthrough, no mtls, or the same clientCaCert bundle and requireClientCert). A route whose policy changed is dropped, so an old verifier never outlives the edit that replaced it.

This is a breaking change for a hand-written route that already uses sourceAddressHeader: 'xForwardedFor' (or a header-carried forwardFingerprint) without protocol: 'http' — add the declaration when upgrading, or that route will stop serving traffic (silently, aside from the log line) rather than failing loudly at startup.

PROXY protocol (default for UDS)

Most backends that consume PROXY protocol (nginx, HAProxy, HarperDB) read the header once per connection before parsing application data.

{
  sni: 'api.example.com',
  upstreams: [{ kind: 'uds', path: '/run/app/worker.sock' }],
  terminateTls: true,
  cert: { certChain, privateKey },
  // sourceAddressHeader: 'proxyProtocol',  // this is already the default for UDS
}

X-Forwarded-For (for Bun and other HTTP backends)

Bun's built-in HTTP server does not support PROXY protocol. Use 'xForwardedFor' instead — symphony injects the header into every HTTP request on the connection, not just the first:

{
  sni: 'app.example.com',
  upstreams: [{ kind: 'uds', path: '/run/bun/worker.sock' }],
  terminateTls: true,
  cert: { certChain, privateKey },
  sourceAddressHeader: 'xForwardedFor',
  protocol: 'http',
}

In your Bun server:

Bun.serve({
  unix: '/run/bun/worker.sock',
  fetch(req) {
    const clientIp = req.headers.get('x-forwarded-for');
    // ...
  },
});

Forwarding the fingerprint downstream

symphony computes the client's JA3/JA4 fingerprint from the ClientHello (the same value used for ja3Blocklist/ja4Blocklist). Set forwardFingerprint to also hand it to the upstream, so a backend behind symphony can make its own bot/abuse decisions on it.

`forwardFingerprint`	Value forwarded
`'ja3'`	The JA3 MD5 hex (32 chars)
`'ja4'`	The JA4 fingerprint
`['ja3', 'ja4']`	Both — e.g. while an existing JA3-keyed blocklist or threat-intel feed is migrated to JA4. List order doesn't matter; JA3 is always emitted first.
`'none'` (default)	Nothing forwarded

The carrier depends on sourceAddressHeader, and every selected fingerprint uses the same one:

With 'proxyProtocolV2', each fingerprint rides a PROXY v2 TLV — type 0xE0 for JA3, 0xE1 for JA4 (in HAProxy's 0xE0–0xEF private range). This works even in passthrough (terminateTls: false), since the header prefixes the raw TLS bytes. No protocol declaration is needed — the TLV carries it regardless of the route's application protocol.
Otherwise, symphony injects an X-JA3 / X-JA4 HTTP header per selected fingerprint. This requires protocol: 'http' on the route (see Declaring the route protocol) and a plaintext HTTP/1 upstream (terminateTls: true and not http2); it is skipped for HTTP/2 upstreams (use 'proxyProtocolV2' there). For that HTTP/1 case, the client-supplied copy of each forwarded header is stripped so the injected value is authoritative and can't be spoofed — this guarantee does not extend to an h2-negotiated connection on an http2: true route: injection and stripping are both skipped there, so a client-supplied X-JA3/X-JA4 reaches the upstream unmodified. Use 'proxyProtocolV2' wherever h2 is possible.

A config that requests a header-carried forwardFingerprint without declaring protocol: 'http' is rejected (and isolated to just that route) — the same fail-loud rule as sourceAddressHeader: 'xForwardedFor'. A config that requests forwardFingerprint with no viable carrier at all — passthrough (terminateTls: false) without sourceAddressHeader: 'proxyProtocolV2', where there's neither an HTTP request to inject a header into nor a v2 TLV — is also rejected, with a distinct "no carrier" error: no protocol declaration could fix a passthrough route's inability to inject a header, so it isn't steered toward one. A route that could have carried the header but silently won't for some connections — http2: true, where ALPN negotiation is per-connection and some clients may still land on HTTP/1 — logs a startup warning instead, since that outcome isn't guaranteed.

// TLV carrier — works for any upstream that speaks PROXY v2, including passthrough
{
  sni: 'app.example.com',
  upstreams: [{ kind: 'tcp', host: '10.0.0.5', port: 8080 }],
  terminateTls: true,
  cert: { certChain, privateKey },
  sourceAddressHeader: 'proxyProtocolV2',
  forwardFingerprint: ['ja3', 'ja4'], // 0xE0 and 0xE1 TLVs
}

// HTTP-header carrier — for HTTP/1 backends that read request headers
{
  sni: 'app.example.com',
  upstreams: [{ kind: 'uds', path: '/run/app/worker.sock' }],
  terminateTls: true,
  cert: { certChain, privateKey },
  sourceAddressHeader: 'xForwardedFor',
  forwardFingerprint: 'ja3', // upstream reads X-JA3 alongside X-Forwarded-For
  protocol: 'http',
}

Forwarding mTLS client certificates

When symphony terminates TLS on a route with mtls configured, the verified client certificate would normally be invisible to the upstream. With sourceAddressHeader: 'proxyProtocolV2' and terminateTls: true, the v2 TLV section carries the connection's TLS facts, once per connection, before any application data:

TLV	Type	Content
ALPN	`0x01`	Negotiated ALPN protocol (e.g. `h2`)
Authority	`0x02`	SNI hostname from the ClientHello
SSL	`0x20`	`client` bit field (`0x01` = TLS, `0x02` = client cert presented on this connection), `verify` (u32 BE, `0` only for a verified cert), and sub-TLVs `0x21` (TLS version) and `0x23` (cipher suite)
Client cert chain	`0xE2` (custom range, after JA3 `0xE0` / JA4 `0xE1`)	One TLV per certificate, DER-encoded, leaf first

{
  sni: 'api.example.com',
  upstreams: [{ kind: 'uds', path: '/run/app/worker.sock' }],
  terminateTls: true,
  cert: { certChain, privateKey },
  mtls: { clientCaCert, requireClientCert: true },
  sourceAddressHeader: 'proxyProtocolV2',
}

A certificate chain is only forwarded when the mtls verifier accepted it — rustls aborts the handshake otherwise — so an upstream receiving a 0xE2 TLV over a trusted (e.g. UDS) link can treat it as a verified client identity. A pathological chain that cannot fit the v2 header's 16-bit length budget is omitted with a warning (the SSL TLV still signals that a verified cert was presented). On passthrough routes (terminateTls: false) the TLS facts are unavailable; the header carries the source address, SNI authority, and any configured fingerprint TLV.

Protection

Recommended starting values for public-facing deployments

protection: {
  rateLimit: { connectionsPerSecond: 50, burst: 100 },
  sustained: { connectionsPerMinute: 300, burst: 300 },
  penaltyBox: { durationMs: 600_000 }, // 10 minutes
  maxConcurrentPerIp: 200,
  allowlist: ['10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16'],
  requireSni: true,
  tlsHandshakeTimeoutMs: 5000,
}

Sustained rate limits

Use sustained to enforce a per-minute cap that is independent of the per-second rateLimit bucket. Both buckets are checked on every connection; exhausting either one blocks the connection. This lets you allow short bursts while still capping total volume over longer windows:

protection: {
  rateLimit: { connectionsPerSecond: 50, burst: 200 }, // allow short bursts
  sustained: { connectionsPerMinute: 600, burst: 600 }, // but cap at 10/s long-term
}

Penalty box

When penaltyBox is configured, exhausting any rate limit places the source IP in a penalty box for durationMs (default 10 minutes). While penalized, all connections from that IP are rejected outright without touching the token buckets — protecting the proxy from re-assembling per-connection state under a sustained attack.

Extension semantics: while an IP is boxed, symphony still debits its token buckets on each connection attempt (to measure whether the attack is continuing). If a bucket is exhausted (the IP is still sending at the excess rate), the penalty deadline is reset to now + durationMs — effectively extending the penalty by a full durationMs from the moment of continued excess. If the IP stops attacking, the buckets refill and debits succeed; the deadline is not extended, so the IP is readmitted once the original deadline expires.

protection: {
  rateLimit: { connectionsPerSecond: 50, burst: 100 },
  penaltyBox: { durationMs: 600_000 }, // 10 minutes (default)
}

Blocked events from penalized IPs have reason: 'penalty_boxed'. They appear under penaltyBoxed in blockedIps():

const info = proxy.blockedIps();
// info.penaltyBoxed — IPs currently in the penalty box

Penalty state is stored on per-IP runtime state and survives a configuration hot-swap. penaltyBox can be added or removed via updateConfig without restarting.

Hot-swap note on active deadlines: changing durationMs via updateConfig affects new penalty stamps immediately, but IPs already in the box retain their current deadline until it expires or is re-stamped by a continued rate-limit hit. There is no retroactive recalculation of existing deadlines.

JA3 blocking

JA3 fingerprints the TLS ClientHello by hashing a canonical string of the version, cipher suites, extensions, elliptic curves, and EC point formats using MD5. That MD5 is by specification, not a security claim — it allows lists of known-bad fingerprints to be compared cheaply. Collect JA3 fingerprints from your logs (available in the blocked event ja3 field) and add known-bad clients:

ja3Blocklist: [
  'e7d705a3286e19ea42f587b344ee6865', // example known-bad scanner
]

Limitation: Chrome and other modern browsers randomize the order of ClientHello extensions on each connection, so a single browser can produce many different JA3 hashes. This makes per-browser JA3 blocking unreliable. Use ja4Blocklist where extension-order randomization is a concern.

Upgrade note: earlier versions filtered only one of the 16 GREASE values when computing JA3, so hashes for clients that send other GREASE values (e.g. Chrome) were nonstandard and unstable. JA3 values collected from earlier symphony versions may no longer match and should be re-collected.

JA4 blocking

JA4 is the randomization-resistant successor to JA3. It sorts the cipher and extension lists before hashing, so it produces a stable fingerprint regardless of ClientHello field ordering. Fingerprints are 36-char lowercase ASCII strings in the form t<ver><sni><cc><ec><alpn>_<sha256/12>_<sha256/12>.

Collect JA4 values from the blocked event ja4 field and configure them:

ja4Blocklist: [
  't13d1516h2_8daaf6152771_b186095e22b6', // example Chrome fingerprint
]

Matching is case-insensitive. JA4 fingerprints are always emitted as lowercase. A blocklist entry that isn't a structurally valid JA4 (or JA3) string is rejected at construction rather than silently installed as an entry that could never match.

Fail-closed reassembly: when a ja3Blocklist or ja4Blocklist is configured, symphony reassembles the full ClientHello (bounded by size and a short timeout) before fingerprinting. A connection whose ClientHello can't be fully reassembled — e.g. a client that fragments it to expose SNI while withholding later extensions — is blocked (blocked reason incomplete_handshake) rather than allowed on a partial/empty fingerprint. Without enforcement configured, fingerprinting stays best-effort and an incomplete hello is not blocked.

License scope: Symphony implements core JA4 (TLS client fingerprinting) only. Core JA4 is BSD-licensed. The JA4+ suite of variants (JA4S, JA4H, JA4SSH, etc.) carries a separate FoxIO proprietary license and is not implemented here.

Hot-swapping protection config

Protection config is per-listener and fully hot-swappable via updateConfig. Push a new config atomically to each listener by port — no restart needed, in-flight connections are unaffected:

// Block a new CIDR range without restarting
proxy.updateConfig({
  protection: [
    {
      port: 443,
      protection: {
        blocklist: ['198.51.100.0/24'],  // added under attack
        rateLimit: { connectionsPerSecond: 50, burst: 100 },
        requireSni: true,
      },
    },
  ],
});

The entire ProtectionConfig is replaced atomically (one pointer swap). Any field not included in the new config reverts to its default. Existing per-IP rate-limit token buckets are preserved across a swap; if burst decreases, tokens are capped at the new ceiling on the next refill — no underflow.

Transitions (none→some, some→none): when using symphony-server, adding or removing a protection block in the config file triggers a seamless proxy recreate via SO_REUSEPORT — no bind gap, existing connections unaffected. Only contents-only changes (same presence, different CIDR/rate/etc.) stay on the pure hot-swap path. When calling updateConfig() directly, a listener that was started without protection cannot gain it and returns an error; restart the listener to change protection presence.

Security & compliance

For deployment hardening and security-questionnaire guidance — private key file handling and rotation, protecting the (hot-reloaded) config file with restrictive permissions and file-integrity monitoring, why an MD5 dependency appears (JA3 fingerprinting, not a cryptographic use), the current TLS parameters and FIPS/PCI attestation roadmap, and the shared-responsibility split for DDoS — see SECURITY.md.

Metrics & monitoring

In-process (`proxy.metrics()`)

const m = proxy.metrics();
// Proxy-wide
// m.activeConnections     — connections being proxied right now
// m.blockedConnections    — total rejected since start (protection + maxConnections)
// m.pendingSuspended      — connections held waiting for resolveConnection()
// m.suspendedResolved     — suspended connections that were resolved with a route
// m.suspendedUnresolved   — suspended connections that timed out or were rejected
// m.routes                — routes in the live table, including the default route
// m.failingRoutes         — rejected cert/protocol/carrier/allowedSources routes, duplicate wildcards, or invalid failover (or failover.spill) policies

// Per listener, in configuration order
for (const l of m.listeners) {
  // l.address, l.mode ('tls' | 'http')
  // l.activeConnections, l.accepted
  // l.bytesReceived — bytes read from clients   (client → upstream)
  // l.bytesSent     — bytes written to clients  (upstream → client)
  //   Counted where the proxy sees the bytes. On a terminated-TLS route that is the plaintext
  //   stream, and the handshake — which precedes the counter — is excluded. On a passthrough
  //   route the proxy has no plaintext view and forwards wire bytes, so the handshake records
  //   are part of the stream and are counted.
  // l.blockedByReason — [{ reason: 'rate_limited', count: 12 }, ...]
  // l.errorsByReason  — [{ reason: 'upstream_connect', count: 3 }, ...]
}

// Per configured route, aggregated across this proxy's listeners
for (const r of m.routeMetrics) {
  // r.route        — configured exact SNI or wildcard pattern; never a client-provided hostname
  // r.metricsGroup — optional RouteConfig.metricsGroup, or ''
  // r.activeConnections, r.connections
  // r.bytesReceived, r.bytesSent
  // r.blockedByReason — sparse post-route refusals (`source_not_allowed`)
  // r.errorsByReason — sparse post-route failures; zero-valued reasons are omitted
}

const blocked = proxy.blockedIps();
// blocked.rateLimited — IPs with a depleted per-second or sustained token bucket
// blocked.concurrencyLimited — IPs at their maxConcurrentPerIp limit
// blocked.cidrBlocklist — the configured static CIDR blocklist
// blocked.penaltyBoxed — IPs currently in the penalty box

Every listener reason is reported on every call, including reasons still at zero, so a dashboard series exists before the first incident rather than appearing mid-outage. l.blocked / l.errors are summed from the very reason values reported alongside them, and m.blockedConnections from the listener values in the same snapshot — so a reading taken mid-traffic is internally consistent rather than only adding up while the proxy is idle. Route blocks and errors are sparse to avoid multiplying every zero-valued reason by every configured tenant route.

Route counters begin only after resolution succeeds. Listener-level protection blocks, maxConnections, malformed/no SNI, unmatched routes, and HTTP header failures therefore have no route series. Exact routes are labeled with their configured SNI, and wildcard routes with the configured wildcard pattern (*.example.com) rather than the client-provided hostname, keeping label cardinality bounded by configuration. Counters survive route hot reloads while both sni and metricsGroup are unchanged; changing the group starts a new series rather than moving historical counters between tenants. Removing or rejecting a route removes its series; adding that route again starts fresh counters. tls_handshake and route_rate_limited are attributed after SNI selection but before client authentication, so they describe traffic aimed at a route rather than confirmed tenant sessions. Creating route metric objects and rendering them are linear in configured routes (four base samples per route); deployments with thousands of routes should size their scrape interval accordingly.

Block reasons: max_connections, cidr_blocked, ja3_blocked, ja4_blocked, incomplete_handshake, no_sni, rate_limited, too_many_connections, penalty_boxed, source_not_allowed. Only source_not_allowed is route-scoped (also in r.blockedByReason and symphony_route_blocked_total).

Error reasons: no_route, route_rate_limited, suspend_unresolved, tls_handshake, tls_missing_cert, upstream_connect, idle_timeout, half_closed, peer_timeout, stream, http_header.

symphony_listener_half_closed_connections is a gauge alongside the active one, counting the subset of active connections whose upstream half has closed. A rising floor there is a retention leak while it is still measured in connections — see Dead-peer detection.

idle_timeout counts terminations by idleTimeoutMs, which today is a total duration cap rather than an idleness one — see #34. Busy long-lived connections land in this bucket, not just quiet ones.

Out-of-process (`symphony-server` admin endpoint)

When symphony runs as its own process there is no JS API to call, so the server bin can expose the same numbers over HTTP. Add an admin block to the config file:

{
  "version": 1,
  "admin": {
    "socketPath": "/run/symphony/admin.sock",
    "socketMode": 432,
    "port": 9095,
    "host": "127.0.0.1"
  },
  "proxies": [ ... ]
}

Both bindings are optional; give either or both. Omit the admin block entirely and nothing is exposed. socketPath may be relative to the config file's directory, and is chmodded to socketMode (default 0o660) after bind. host defaults to 127.0.0.1. Route and group labels can identify tenants, so keep access scoped to the host collector. Host Manager's standard Docker bridge does not expose host loopback to tenant containers, but other host-local processes and downstream metric systems remain inside this endpoint's trust boundary.

Route	Response
`GET /metrics`	Prometheus text exposition (v0.0.4)
`GET /metrics.json`	the same snapshot as JSON
`GET /health`	`{ ok, pid, version, ports }`

$ curl --unix-socket /run/symphony/admin.sock http://localhost/metrics
# HELP symphony_build_info Always 1; the version is carried in the label.
# TYPE symphony_build_info gauge
symphony_build_info{version="0.5.0"} 1
...
symphony_listener_accepted_total{proxy="80,443",listener="0.0.0.0:443",mode="tls"} 148213
symphony_listener_blocked_total{proxy="80,443",listener="0.0.0.0:443",mode="tls",reason="rate_limited"} 27
symphony_listener_errors_total{proxy="80,443",listener="0.0.0.0:443",mode="tls",reason="upstream_connect"} 4
symphony_route_connections_total{proxy="80,443",route="api.example.com",group="tenant-1"} 148000
symphony_route_errors_total{proxy="80,443",route="api.example.com",group="tenant-1",reason="upstream_connect"} 3

The proxy label is the port-set of the proxy entry the listener belongs to (each config entry gets its own route table). Blocked and error counts are only ever emitted with their reason label — the labelled series sum to the total, so use sum without(reason) rather than looking for a separate unlabelled metric. Likewise the proxy-wide active-connection gauge is sum without(listener) (symphony_listener_active_connections).

Route series use route (the configured SNI/pattern) and group (the configured metricsGroup, or an empty string). Their active/connection/byte values aggregate across all listeners in that proxy; Host Manager uses one proxy per external port, so its existing proxy label keeps route traffic separated by port. Route error samples appear only after a reason is non-zero.

With a generation in the config (Knowing which config symphony-server applied), /metrics.json adds appliedGeneration and failedGeneration, and /metrics adds:

symphony_config_applied_generation_info{generation="4171"} 1
symphony_config_failed_generation_info{generation="4172",ports="8443,9443"} 1

Each appears only while its field is set. A generation can only be a label, so each new value starts a new series: change it when the config changes, not on a timer.

The endpoint is strictly read-only and best-effort: it never blocks proxying, and a bind failure is logged and retried every 5s rather than aborting startup. That matters during a version upgrade, where the incumbent still holds the socket while the replacement is already serving traffic through SO_REUSEPORT — the successor picks up the admin endpoint once the old process exits. A socket file left behind by a SIGKILLed process is reclaimed automatically, but only after a connect probe proves nobody is listening on it.

The Prometheus renderer is internal to the standalone server and is not exported from the package root: a snapshot carries that process's pid, timestamps, and port-set grouping, which an embedded consumer would have to synthesise. An embedded proxy has proxy.metrics() directly. If a caller genuinely needs Prometheus text from an embedded proxy, open an issue — the right shape is a ProxyMetrics-based renderer, not this one.

Hot config updates

// Replace the entire route table atomically — in-flight connections are unaffected.
proxy.updateConfig({
  routes: newRoutes,
});

What can be hot-swapped: routes (destinations, TLS certs, suspension state, failover), per-listener protection (CIDR allowlist/blocklist, JA3 blocklist, rate limits, sustained rate limits, penalty box, concurrency caps, handshake timeout, requireSni), and a failover ingress listener's trustedSources.

What requires a restart: bind address, port, idle timeout, worker threads. When calling updateConfig() directly, protection presence (None↔Some) cannot change at runtime — the listener must be restarted. The symphony-server bin handles this automatically via seamless recreate.

start() binds every listener before it accepts a connection on any of them. If one can't bind, it rejects and leaves nothing listening; calling it again retries.

Replacing a running `symphony-server`

A supervisor upgrades symphony-server by starting the new process on the same --config and --status paths while the old one still runs; SO_REUSEPORT lets both accept on the same ports. status.json names the newest live process (pid, version, startedAt, configPath, ports, and listeners — the host:port of every listener it has bound). The supervisor waits for the file to name the new process, then sends the old one SIGTERM. A new process that could not bind a listener the running one has (a listener cert it can't read, say) does not take the file, so a supervisor never retires the only process serving that listener; host-manager reads that as a failed upgrade and keeps the old process.

If the old process is not stopped (a handoff that failed partway, or a supervisor that restarted during it), it stops itself: once a newer process has held status.json for 10 seconds, serving the same config on every listener the old process has, the old process runs its normal SIGTERM stop and exits. Each second it checks again, and it starts the 10 seconds over if the newer process exits, stops naming itself in the file, or stops covering one of its listeners. It never exits for a process it can't verify through /proc, so on platforms without /proc it waits for its supervisor. A process that stops never deletes status.json.

Only a process with this behavior stops itself. A symphony-server from 1.4.0 or earlier that is already running keeps running until something stops it, so clearing out processes left over from earlier upgrades needs the supervisor (or an operator) to stop them once.

Knowing which config `symphony-server` applied

reloadedAt in status.json says when a reload finished. It does not say which config that reload read, or whether it applied. To find out, set a top-level generation in the config:

{ "version": 1, "generation": "4172", "proxies": [ ... ] }

generation is an opaque string the writer chooses: a counter or a hash of the content. Use a new value whenever the content changes, because symphony compares nothing and orders nothing; it only reports back the string it read. A value that isn't a string is logged and treated as no generation, and the rest of the config still applies.

Each reload then writes these to status.json:

Field	Meaning
`appliedGeneration`	The generation of the last config in which every proxy entry applied. It comes from the config that reload read, so a reload that read the previous config and finishes after a new write reports the previous generation.
`failedGeneration`	`{ generation, ports }`, present when the last reload skipped a proxy entry: the generation it read and the sorted listener ports of what it skipped. The next reload that applies every entry removes it.

A writer waiting for generation G reads them like this:

`status.json` shows	Meaning
`appliedGeneration` is `G`	Every proxy entry applied `G`.
`failedGeneration.generation` is `G`, and `appliedGeneration` is not	`G` applied with errors. The listed ports keep what they were serving; every other entry has `G`. Nothing retries until the next reload (a config write, a change to a referenced cert file, or `SIGHUP`).
Neither is `G`	Not applied yet. A symphony older than this feature always reads this way, so never treat a missing field as applied.

Both fields can name G. G applied, then a later reload of the same config (a rotated cert, say) skipped a proxy. appliedGeneration is still true, since the skipped proxy keeps G's routes.
An entry applies or doesn't as a whole. A route whose cert can't be built is isolated inside its proxy and doesn't count as a skip. failingRoutes reports it.
An entry whose listeners don't name its ports is a skip. That is a missing or empty array, or a listener with no numeric port. It can't be matched to a running proxy, so a reload of that config removes no proxy, and ports lists only the ports it did name.
A running proxy stays while a skipped entry may replace it. That is any proxy sharing a port with a skipped entry, so moving a port to a new entry under a bad cert doesn't take the port down. If a config splits one proxy into two and only one applies, the port that applied is served by both the old and the new proxy until the other applies, and status.json lists it twice.
A config that can't be read changes nothing. An unreadable file, invalid JSON or a missing proxies array leaves status.json as it was, so a writer sees "not applied yet".
Without a generation neither field appears. Dropping it from the config removes both.
The fields describe the process status.json names, in memory. A new process reports what its own first reload did. /metrics.json and the symphony_config_*_generation_info samples carry the same fields for the process that answered, which can be the outgoing one during an upgrade, so compare pid.

Building from source

Requirements: Rust stable (1.70+), Node.js 18+, @napi-rs/cli.

npm install
npm run build:debug    # builds a dev .node file
npm run build          # release build (LTO, stripped)

On Linux the addon's Rust allocations go to a statically linked jemalloc, which returns memory to the OS after connections close; glibc's malloc keeps the peak resident. Its symbols are prefixed, so Node's own allocations stay on the system allocator. To build on the system allocator instead, for example to bisect a memory problem: npm run build -- --cargo-flags=--no-default-features.

Cross-compilation

Use the napi-rs Docker images (same ones used in CI):

# x64 musl (Alpine)
docker run --rm -v $(pwd):/build -w /build \
  ghcr.io/napi-rs/napi-rs/nodejs-rust:lts-alpine \
  npm run build -- --target x86_64-unknown-linux-musl

# arm64 glibc
docker run --rm -v $(pwd):/build -w /build \
  ghcr.io/napi-rs/napi-rs/nodejs-rust:lts-debian-aarch64 \
  npm run build -- --target aarch64-unknown-linux-gnu

Dead-peer detection

A client that disappears without closing — a mobile network dropping, a host powering off — sends no FIN, so nothing in the ordinary proxying path ever learns it is gone. Two independent mechanisms bound how long such a connection is retained; both are on by default.

TCP keepalive. SO_KEEPALIVE is off per socket by default, and the net.ipv4.tcp_keepalive_* sysctls govern only sockets that opted in — so without this a silent peer is indistinguishable from a parked subscriber for the life of the process. symphony sets the option and its timings on the listening socket, which every accepted socket inherits, so the accept path pays no extra syscalls. At the defaults a peer that stops answering is reclaimed 300s + 30s × 5 = 7.5 minutes after the connection's last activity. Lower idleMs to reclaim sooner, at one probe per connection per idleMs: across 300k connections the default is roughly 1000 packets/s.

This bounds idle sockets. A socket with data outstanding is instead bounded by the kernel's retransmission limit (net.ipv4.tcp_retries2, ~15 minutes at its default), because keepalive probes are not sent while anything is unacknowledged.

The half-close bound. copy_bidirectional returns only once both directions have finished. When the upstream closes, symphony drains the response and shuts down its write half to the client — the socket enters FIN-WAIT-2 — and then waits for a FIN the client may never send. Because the fd stays open the socket is not orphaned, so tcp_fin_timeout does not apply either. halfCloseTimeoutMs reclaims that connection once the surviving client→upstream direction has carried nothing for the configured window.

Two properties keep this from cutting live sessions:

The clock starts when our FIN is actually out, not when the upstream EOFs. Those are not the same instant: on a terminated-TLS route a write is accepted into rustls' buffer long before it reaches a slow client's socket, so the drain happens inside the shutdown. Starting at the EOF would run the deadline against a flush still in progress. Once the FIN is out, nothing further can be sent to that client, so there is no response left for the bound to truncate. A client half-close shuts down the write half to the upstream instead and arms nothing — there the response is still in flight, and a slow upstream is idleTimeoutMs's territory.
The deadline resets on activity. A client still uploading to an upstream that closed early keeps its connection; only a genuinely quiet one is reclaimed.

It is still a policy, not a protocol guarantee: TCP permits an upstream to shutdown(SHUT_WR) and keep reading indefinitely, so a protocol whose client legitimately pauses longer than halfCloseTimeoutMs between writes after its upstream has stopped replying will be cut. It is on by default because the alternative — another knob the fleet has to opt into — is what left idleTimeoutMs pinned at 0 and produced the sockets above. Set halfCloseTimeoutMs: 0 on a listener carrying such a protocol.

Like the buffer sizes, both settings are frozen at construction, so editing either makes symphony-server recreate the proxy — a reconnect event on a high-connection-count listener, not a live tune.

Copy buffers and per-connection memory

Each proxied connection holds two copy buffers — one per direction — for its entire life, whether or not it is transferring anything. So buffer memory scales with connection count, not with traffic:

buffer bytes = (clientReadBufferSize + upstreamReadBufferSize) × connections

Each unset override falls back to readBufferSize, so with symmetric sizing that is just 2 × readBufferSize × connections. At the 8192-byte default it is 16 KiB per connection: 4.0 GiB at 262k connections, 5.1 GiB at 333k. The asymmetric MQTT setting recommended below is 1024 + 4096 = 5 KiB per connection, which is where the general form matters. The knob is per proxy, and the right value is the opposite for the two shapes of traffic symphony carries:

Traffic	Connections	Payloads	Suggested
Native MQTT (`8883`)	100k–1M	hundreds of bytes	`clientReadBufferSize: 1024`, `upstreamReadBufferSize: 4096`
HTTPS (`443`)	thousands	mixed	leave at the default
Operations API (`9925`)	tens	can be large	leave at the default
Replication (`9933`)	~6	bulk streams	leave, or raise — 64 KiB across 6 connections is 768 KB total

MQTT is worth splitting by direction: after SUBSCRIBE a client sends almost nothing but PINGREQ, while the broker carries the whole fan-out. 1024/4096 is 5 KiB per connection against the default's 16 KiB — 3.5 GiB saved at 333k connections — and buys more downstream headroom than a symmetric 2048 would.

Going small costs CPU, not correctness: a payload larger than the buffer is simply copied in more iterations. On a TLS-terminating listener those extra iterations are not even syscalls, since the reads come out of rustls's already-decrypted buffer.

Two limits on where these settings apply:

Only the plain proxying path. A route that injects HTTP headers (sourceAddressHeader: 'xForwardedFor', or a header-carried forwardFingerprint) is framed per-request by proxy_http1_rewriting, which uses its own fixed 8 KiB buffers — the same size as the default here, so the default is unaffected, but a raised or lowered value has no effect on those routes. PROXY-protocol routes, including every UDS route, take the plain path and are governed normally.
A config reload cannot change these. They are frozen when the proxy is constructed, so changing one makes symphony-server recreate the proxy rather than hot-swap it. SO_REUSEPORT means there is no bind gap, but established connections on the old proxy are not drained — stop() waits 100 ms and connection tasks are detached — so they are all dropped. On a high-connection-count listener, treat a buffer-size edit as a reconnect event, not a live tune.

Upgrading: before this setting was applied to the copy loop, readBufferSize had no effect — every connection got 8 KiB per direction regardless of what the config said, and the default documented here was 65536. A config that leaves it unset is unaffected. A config that sets it explicitly now gets what it asked for, so a value copied from the old documented default becomes 128 KiB per connection instead of 16 KiB. Drop or remove such a value before upgrading a high-connection-count deployment.

Two caveats when sizing a node from this:

Buffers are one term, not the whole per-connection cost. A TLS-terminating listener also carries rustls session state per connection, plus kernel socket memory (bounded in aggregate by net.ipv4.tcp_mem). Measure the actual slope — activeConnections from the admin endpoint against process RSS — before deciding a connection target is reachable. npm run benchmark:memory-oop (__test__/bench-copy-memory-oop.ts) does this with symphony in its own process under parked, active, slow-reader and churn load; the other benchmark* scripts host the proxy inside their own load generator, so their RSS is mostly the client's.
Buffer memory is not what caps connection count. File descriptors (2 per proxied connection, against RLIMIT_NOFILE) and nf_conntrack_max bind first; see Linux kernel tuning.

Linux kernel tuning

To reach ~1 million concurrent connections, the following system settings are required.

File descriptor limits

# Per-process (set before starting Node)
ulimit -n 2097152

# System-wide persistent — /etc/security/limits.conf
*  soft  nofile  2097152
*  hard  nofile  2097152

symphony attempts to raise RLIMIT_NOFILE automatically at startup (to 2 × maxConnections + 1024), but the hard limit must be raised by the OS first.

Kernel networking

# /etc/sysctl.d/99-symphony.conf

# TCP connection tracking
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1

# Socket buffers (tune to your bandwidth)
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Accept queue depth per socket
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535

# Max open files system-wide
fs.file-max = 4194304

Apply with:

sudo sysctl --system

musl note

On musl-libc systems (Alpine), the hard RLIMIT_NOFILE is often capped at 1048576 rather than the glibc default of 1073741816. symphony will log a warning if the desired limit exceeds the hard limit and fall back to the hard limit.

Name		Name	Last commit message	Last commit date
.cargo		.cargo
.github/workflows		.github/workflows
__test__		__test__
npm		npm
src		src
ts		ts
.gitignore		.gitignore
.prettierrc		.prettierrc
CLAUDE.md		CLAUDE.md
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
build.rs		build.rs
package-lock.json		package-lock.json
package.json		package.json
rust-toolchain.toml		rust-toolchain.toml
rustfmt.toml		rustfmt.toml
tsconfig.json		tsconfig.json
tsconfig.test.json		tsconfig.test.json

Latest commit

History

Folders and files

Repository files navigation

symphony

Overview

Installation

Quick start

Configuration reference

ProxyConfig

TcpKeepaliveConfig

ListenerConfig

RouteConfig

Upstream

CertConfig

MtlsConfig

ProtectionConfig