socket: distinguish eagain from closed, and retry transparently via write-select by harmon25 · Pull Request #2345 · atomvm/AtomVM

harmon25 · 2026-06-30T14:49:31Z

Distinguish transient send backpressure from a closed connection in socket:send/2,3, and wait for write-readiness with a proper write select instead of leaking {error, eagain} or {ok, Rest} to callers.

Ran into this building httpd, a fork/standalone implementation of atomvm_lib/httpd: large or many subsequent requests failed with the socket closing early, because lwIP ERR_MEM backpressure was reported as {error, closed}.

Commits

Fix socket send error reporting: lwIP ERR_MEM / BSD EAGAIN|EWOULDBLOCK is now {error, eagain}, a closed connection is {error, closed} instead of {ok, Data}, other errors return the real errno or lwIP error, and sending on a socket that was closed returns {error, closed} as in OTP.
Add enif_select_write and ERL_NIF_SELECT_CANCEL: enif_select_write was declared but never implemented. A select event now keeps independent read and write state (ref, message, pid), so a resource can have a pending read and a pending write select at the same time. ERL_NIF_SELECT_CANCEL (with the *_CANCELLED flags) cancels one direction and keeps the other armed. Also fixes the ref ticks computation. C tests in tests/test-enif.c.
otp_socket: Add nif_select_write/2: BSD uses enif_select_write; lwIP uses tcp_sent/tcp_poll and tcp_sndbuf/tcp_sndqueuelen. A socket has one selecting process and monitor per direction, like resources.c: select replaces only its own direction, close and select_stop handle both, socket_down releases only the dead owner's direction (and no longer aborts the lwIP connection), and aborts go to each direction's own pid. Includes the lwIP callback-lifetime, lock, ERR_MEM pacing, closed-select and connecting-socket fixes from review.
otp_socket: Fix lwIP connection teardown: a peer reset or a read_write shutdown can free the pcb; forget it, report {error, closed} once received data is consumed, wake both select owners, and register the error callback on accepted sockets too. Fixes lwIP sockets: a peer reset does not wake processes blocked in socket:send/2 or recv #2432.
socket: Retry sends on backpressure: socket:send/2 and sendto/3 wait on nif_select_write/2 and retry, so callers get ok | {error, Reason} (or {error, {Reason, RestData}} after a partial send, as in OTP).
Add tests for socket write select and send: backpressure with write select, closing an accepted socket selected by another process, close with a blocked reader and writer, a dead writer leaving the reader working, reader cleanup keeping a blocked writer, and synchronized rather than sleep-based backpressure tests.
socket: Add send/3 and sendto/4: timeout, nowait or a select handle, returning {error, timeout} / {error, {timeout, RestData}} or {select, ...}, plus nif_select_write_stop/1.
Change socket:send callers to the new semantics: ssl waits for write-readiness instead of busy looping on want_write; socket_dist_controller terminates the connection on send errors and keeps the unsent frame out of the stop reason; epmd handles a failed ALIVE2 response instead of crashing; dead {ok, Rest} branches are removed.
gen_tcp_socket: Send asynchronously: sends are queued and one packet at a time is sent with nowait, so gen_tcp:recv/2,3, close/1 and controlling_process/2 keep working while the peer does not read.
gen_udp_socket: Send asynchronously: same for datagrams on BSD sockets, where the UDP send buffer can fill.
otp_socket: Fix select owner lifetime: selecting on an already closed socket no longer leaves a monitor and reference armed, and queued lwIP sent/error handlers hold a reference while notifying outside the lwIP lock.
ESP32: Fix a lost select wakeup: a select event registered while the select thread rebuilt its poll set is no longer missed.
Document socket write select in the changelog.

Known limitations

socket:send/2 now blocks, so socket_dist_controller can block on a peer that stops reading and miss the tick that would detect it: socket_dist_controller: sends can block the controller and hide a dead peer #2424.
The lwIP changes are built for ESP32 but have not been run on RP2 or ESP32 hardware by the author in this round.

Testing

test-enif on generic_unix, natively and under valgrind (including the new ERL_NIF_SELECT_CANCEL tests).
The full test_estdlib suite on generic_unix, natively and under valgrind, including test_tcp_socket, test_udp_socket, test_gen_tcp, test_gen_udp, test_epmd and test_ssl, run repeatedly.
ESP32 build with ESP-IDF 5.5.4, which compiles the lwIP code paths.
The gcc-10 -m32 CI job will re-run with the synchronized backpressure tests once a maintainer approves the workflow runs for this fork PR.
Earlier, before the review changes, on real ESP32-S3 hardware with atomvm_httpd: 3 concurrent slow readers (curl --limit-rate 1k against a 64KB response) logged only eagain retries (never closed) and every response completed with the full body.

These changes are made under both the "Apache 2.0" and the "GNU Lesser General Public License 2.1 or later" license terms (dual license).

SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

petermm · 2026-07-02T16:30:38Z

thank you for pushing the httpd side!

this is obviously correct - however it does increase the leakage of underlying implementation, and now any caller will also need to implement their own unique retry mechanism (gen_tcp_socket.erl, socket_dist_controller.erl, ssl.erl - and atomvm_lib httpd) - and is it 5ms with exponential backoff or xyz - and we end up with complexity, duplicated and slow (sleeping) code..

I believe the solution is to implement nif_select_write/2, with proper write-select and no sleeping/guessing, eg. handle the happy path out of the box, similar to otp:

see https://github.com/petermm/AtomVM/tree/nif_select_write/2 (based on this PR)

harmon25 · 2026-07-04T14:35:04Z

Thanks @petermm! I agree, that sounds like a better approach.

Should we continue from your branch, or should i incorporate those changes here?

harmon25 · 2026-07-04T21:10:59Z

@petermm Thanks for the review and for prototyping this on nif_select_write/2 — agreed this is the right fix, and I've implemented it on top of this branch (6 new commits, keeping the existing eagain/closed distinction as the low-level primitive underneath):

enif_select_write — was declared in erl_nif.h but never implemented or called anywhere. Implemented it, and reworked enif_select/enif_select_read/enif_select_write so a resource can hold independent pending read and write selects (each with its own ref/message/pid) on the same event at the same time, instead of one direction silently overwriting the other's state (needed since a socket can have a permanent recv select active at the same time as a transient send-backpressure select, e.g. socket_dist_controller). Also fixed a latent ref-ticks bug found along the way (using term_is_local_reference(ref) instead of ref != UNDEFINED_ATOM). Added C-level unit test coverage for this in tests/test-enif.c (there was none before for enif_select at all).
otp_socket.c — wired up nif_select_write/2 for both BSD (enif_select_write) and lwIP (new tcp_sent/tcp_poll callbacks + a SocketStateSelectingWrite state, using tcp_sndbuf/tcp_sndqueuelen to detect room again).
socket:send/2,3 — now retries partial sends and {error, eagain} internally by waiting on nif_select_write, so callers just see ok | {error, Reason} — no more leaking {ok, Rest}/{error, eagain}.
ssl.erl — replaced the 4 want_write busy-loops with a real wait-for-writable.
socket_dist_controller.erl — now actually terminates the connection on socket:send/2 errors instead of silently ignoring them (tick + outgoing data loop).
Added a real backpressure test to test_tcp_socket.erl (fills the send buffer via raw nif_send/2 until genuine EAGAIN, then verifies nif_select_write/2 correctly wakes up once the peer drains, and that socket:send/2 completes transparently afterward).

Verified: test-enif, the full test_estdlib suite (including the new backpressure test) on generic_unix, and a full ESP32 (esp32s3) build via ESP-IDF 5.5.4.

One thing worth calling out from writing the test: I initially had it drain only a small chunk (64KB) of the peer's backlog and expected the socket to become writable again, but that hung — traced it down (gdb backtraces + an independent raw C poll(2) repro, unrelated to any AtomVM code) to the sandbox's kernel/buffer accounting not reliably crossing the write low-water mark on a small partial drain, only on a full drain. Not an AtomVM bug, just a test design fix, but flagging in case it's relevant to anyone else validating this on constrained targets (rp2/esp32) with small buffers — might be worth a similar "drain fully, not partially" pattern if there's ever a device-side test for this.

bettio

First of all I apologize for the long wait before this review. Here I have some comments.

I would like to ask you make your branch git history linear without those merges from release-0.7. Ideally your commits should be amended using git commit --fixup and they should sit on top of latest release-0.7. See also GIT_WORKFLOW_AND_HISTORY_STYLE_GUIDE.md

Blocking sends require read and write readiness to coexist on one socket. Sharing their owner lets a writer replace a reader, misroutes close notifications, and cancels both directions when one selector exits. Give each direction its own owner, monitor and reference, initialized for both opened and accepted sockets. Implement ERL_NIF_SELECT_CANCEL so selector death cancels only its direction, disposes its pending message, and leaves the other direction armed. Retain socket resources while lwIP write/error events are queued and detach callbacks on close. Notify established-socket waiters of terminal errors through direction-specific abort messages, independently of the connect trap. Keep distribution output event-driven so backpressure cannot block incoming traffic or ticks. Preserve pending frame suffixes and queue ticks behind them to maintain byte ordering. Send empty binaries through the native socket path so closed sockets still fail and connected UDP sockets transmit zero-length datagrams. Add regressions for accepted-socket close, writer death, both selector registration orders, exact-data backpressure recovery, cancellation cleanup, and empty UDP sends. Move the changelog entry to beta.1. See also: atomvm#2345 Signed-off-by: Peter M <petermm@gmail.com>

bettio

Let's fix the CodeQL error and it is good for merge.

bettio

I need to apologize, I think I found other issues just before merging.

harmon25 · 2026-10-15T19:03:13Z

Pushed fixes for all six comments, rebased on the latest release-0.7. Thanks for the careful review. #2422 touches the same area (nif_select_read_stop/1), so whichever lands second will need a small rebase.

bettio

I need to apologize here, I have one more review round here. Still I believe we are getting super close.

harmon25 · 2026-10-16T21:42:18Z

Pushed fixes for all six comments, rebased on the latest release-0.7. Thanks again for the careful review.

bettio

We are getting close to green light, but there are still some issues.

bettio · 2026-10-20T08:27:49Z

-    {reply, socket:send(State#state.socket, Packet), State};
+    Reply =
+        case socket:send(State#state.socket, Packet) of
+            % gen_tcp:send/2 does not report the data that was not sent


Let's avoid narration comment, such this one.

Removed in gen_tcp_socket: Send asynchronously.

bettio · 2026-10-20T08:29:27Z

+            // after tcp_shutdown succeeded so that a failed shutdown leaves the
+            // socket intact, and under the lwIP lock so no callback can run in
+            // between.
+            socket_tcp_detach_callbacks(rsrc_obj->tcp_pcb, rsrc_obj->socket_state);


tcp_shutdown(pcb, 1, 1) is lwIP's tcp_close_shutdown(pcb, 1). It frees the pcb and still returns ERR_OK when the pcb is in CLOSED or SYN_SENT state, and in ESTABLISHED/CLOSE_WAIT whenever rcv_wnd != TCP_WND_MAX or refused_data is set (the RST path).
Since tcp_recved only runs when data is copied out in recv (line 2310), any shutdown(S, read_write) with unread data takes that path, and socket_tcp_detach_callbacks then writes callback_arg, recv, sent, poll and err into freed memory.
Detach before tcp_shutdown, as socket_dtor and nif_socket_close do. The only non-freeing failure is ERR_MEM from tcp_close_shutdown_fin, which sets TF_CLOSEPEND so lwIP finishes the close itself; keeping the callbacks detached in that case is fine.

Detaching now happens before tcp_shutdown, in the same commit that closes the pcb. The later otp_socket: Fix lwIP connection teardown refactors the same path through socket_tcp_pcb_gone, which also handles an already freed pcb.

bettio · 2026-10-20T08:31:30Z

+%% Receives at least Remaining bytes (a recv can return more than needed).
+recv_at_least(_Socket, Remaining) when Remaining =< 0 ->
+    ok;
+recv_at_least(Socket, Remaining) ->


{recv_failed,timeout,69620736} is thrown here.
Remaining is larger than ?LARGE_PAYLOAD_SIZE, so the server read fewer bytes than Sent, i.e. less than the client kernel had already accepted, then saw nothing for 10 s. That is not writer slowness: queued kernel data cannot stall, so it points at a lost read-select wakeup or an accounting bug specific to that build. The job passed on e2e3ec3 and fails identically on 42a027c and c34bb11, so it tracks the nif_select_read_stop round.
I could not reproduce on 64-bit (native, pinned to one core, valgrind, valgrind --fair-sched=yes); it needs a 32-bit run with the test instrumented before merge.

I traced this to the new test racing the writer's write-select arming: the reader cleanup could run before the writer owned the write direction. The writer now fills the buffer with the raw NIF, reports that it observed eagain, and the reader cleanup starts only then; recv_at_least uses a generous timeout and the drain raises on timeout. The gcc-10 -m32 job is re-running, and I will follow up if it still fails.

bettio · 2026-10-20T08:33:06Z

+    LWIP_END();
+    if (can_send) {
+        // Clear flag to avoid sending a message again.
+        rsrc_obj->socket_state &= ~SocketStateSelectingWrite;


nif_socket_select updates socket_state under LWIP_BEGIN/END (lines 1283, 1286 and the read branch), while this handler does a read-modify-write of the same word with no lock.
RP2 builds are SMP by default, so a reader doing |= SelectingRead on one core while this clears the write bit on the other loses a bit: the reader is never notified, or a second write notification goes to a consumed ref. tcp_recv_handler has the same pre-existing pattern, but two processes selecting both directions is now the normal case. Move the flag update and the write_select.process_id read inside the locked region.

The flag update and the owner read are now inside the LWIP_BEGIN()/LWIP_END() region in tcp_sent_handler.

bettio · 2026-10-20T08:35:03Z

+    if (LIKELY(rsrc_obj) && (rsrc_obj->socket_state & SocketStateSelectingWrite)) {
+        struct LWIPEvent event;
+        event.handler = tcp_sent_handler;
+        event.tcp_sent.rsrc_obj = rsrc_obj;


The queued event carries a raw rsrc_obj. Between the enqueue in lwIP context and tcp_sent_handler, socket_down can clear the flag and drop the owner's reference, which can be the last one, so the handler dereferences freed memory. tcp_recv_cb avoids this by queuing tpcb and reading tpcb->callback_arg, which socket_tcp_detach_callbacks NULLs under the lock. Do the same here (also at line 1145), or enif_keep_resource in the callback and release in the handler.

The event handlers now check the resource against the lwip_sockets list under the lwIP lock, and take a reference while notifying outside the lock, so socket_down dropping the last reference cannot free it under them.

bettio · 2026-10-20T09:01:04Z

+            Total = recv_until_closed(ServerSocket, 0),
+            Owner ! {drained_all, Total},
+            ok = socket:close(ServerSocket)
+    after 60000 ->


Under valgrind on a loaded runner, draining 69 MB can exceed this, and the close from another process turns a slow run into a confusing {recv_failed, closed, _} or an abort on the writer. Teardown always sends drain_all, so use infinity or a much larger value.

The acceptor's 60 s timeout is gone, and recv_at_least uses a generous timeout. A stalled drain now raises {recv_failed, Reason, Remaining} through the linked acceptor instead of closing quietly.

bettio · 2026-10-20T09:01:42Z

+    Reader = spawn(fun() ->
+        Self ! {recv_result, self(), socket:recv(Client, 0, 30000)}
+    end),
+    timer:sleep(100),


Lines 913 and 921 assume the reader and writer are blocked after 100/200 ms, and test_send_error_after_partial_send (lines 871, 876) assumes the writer made progress within 200 ms of the drain, otherwise it gets a plain {error, closed} and fails. Under valgrind on slow CI these are flakiness sources. Have the writer report when it has observed eagain (e.g. fill via raw nif_send in the writer itself before socket:send), and retry the drain until a partial send is observed.

The sleeps are replaced by explicit synchronization: the reader reports that it started, the writer fills the buffer with nif_send/2 and reports once it observed eagain, and the partial-send test reads past the setup bytes. No timing assumptions remain.

bettio · 2026-11-06T12:06:33Z

+            RAISE_ERROR(OUT_OF_MEMORY_ATOM);
+        }
+    }
+    if ((rsrc_obj->socket_state & SocketStateSelectingWrite)


This notifies the process selecting for write, but SocketStateSelectingWrite is only cleared by the SocketStateClosed store at line 938 and tcp_pcb only at line 930, both after the LWIP_END() at line 925.

tcp_sent_handler does not take socket_lock, so on an SMP build an event already queued by tcp_sent_cb/tcp_poll_cb can pass the SocketStateSelectingWrite check at line 1099 in that window and call lwip_tcp_can_send under the lwIP lock at line 1103, reading snd_buf of a pcb that tcp_close may already have freed (for example in CLOSED or SYN_SENT, or on the RST path with unread data).

Please clear tcp_pcb and set the closed state inside the locked block at lines 922-925, as socket_dtor does for the pcb at line 326.

nif_socket_close now clears tcp_pcb and stores SocketStateClosed inside the LWIP_BEGIN()/LWIP_END() region, before releasable resources can be notified.

bettio · 2026-11-06T12:17:17Z

+    ok | {error, Reason :: term()}.
 sendto(Socket, Data, Dest) when is_binary(Data) ->
-    ?MODULE:nif_sendto(Socket, Data, Dest);
+    sendto_all(Socket, Data, Dest);


On BSD sockets, sendto/3 now waits on nif_select_write when sendto(2) returns EAGAIN (sendto_all, line 799), where before the PR the same condition surfaced as an immediate {error, closed}.

gen_udp_socket:handle_call({sendto, ...}) at gen_udp_socket.erl:178 performs the send inside the gen_server, and send/4 calls it with an infinite timeout, so the server, and with it every other operation on that gen_udp socket, now waits until the socket is writable. OTP's gen_udp_socket sends from the calling process instead, so only the sender blocks there.

Let's handle it: for example a nowait send with the pending datagram kept in the gen_server state, as suggested for gen_tcp_socket. The lwIP UDP path is unaffected, as it never reports would-block.

gen_udp_socket now sends with nowait and keeps the pending datagram and following ones in its state, so recv, close and controlling_process keep working while a BSD UDP send buffer is full.

bettio · 2026-11-06T12:20:08Z

+            recv_until_closed(Socket, Acc + byte_size(Data));
+        {error, closed} ->
+            Acc;
+        {error, timeout} ->


{error, timeout} ends the drain quietly with a partial count that no caller checks (they only match is_integer(N)), and the acceptor then closes the server socket. In a healthy run a recv never waits 30 s here, since the client is either still sending or has already closed, so this only fires when the drain has stalled.
If that happens while test_send_backpressure is still in socket:send/2 at line 647, the failure shows up there (a badmatch on {error, closed} or {error, {closed, Rest}}, or a hang) instead of as the stalled drain it is. Since the acceptor is linked, raising here as recv_at_least does at line 901 would keep such a failure diagnosable.

recv_until_closed now raises {recv_failed, Reason, Acc} on a timeout, so the linked acceptor fails the test instead of quietly closing the server socket with a partial count.

bettio · 2026-11-06T12:54:55Z

I added a number of patches here that fix a number of reported issues. Feel free to use them / take them as suggestions.

0017-tests-fail-the-backpressure-drain-when-recv-times-ou.patch
0016-tests-synchronize-the-socket-backpressure-tests-expl.patch
0015-ESP32-store-the-poll-count-before-releasing-the-sele.patch
0014-otp_socket-do-not-call-tcp_recved-on-a-pcb-that-is-g.patch
0013-otp_socket-clear-the-pcb-and-the-state-of-a-closed-l.patch
0012-otp_socket-handle-the-lwIP-error-callback-on-connect.patch
0011-otp_socket-fix-the-closed-checks-of-accept-and-recv-.patch
0010-otp_socket-release-the-socket-lock-when-tcp_close-fa.patch
0009-otp_socket-deliver-the-closed-notification-when-sele.patch
0008-otp_socket-update-the-write-select-flag-under-the-lw.patch
0007-otp_socket-keep-a-write-select-pending-when-tcp_writ.patch
0006-otp_socket-detach-the-lwIP-callbacks-before-tcp_shut.patch
0005-otp_socket-return-closed-when-sending-on-a-closed-so.patch
0004-otp_socket-report-the-actual-error-of-a-failed-send.patch
0003-gen_tcp_socket-send-asynchronously-one-packet-at-a-t.patch
0002-socket-add-send-3-and-sendto-4-with-a-timeout-nowait.patch
0001-socket_dist_controller-do-not-put-the-unsent-data-in.patch

bettio · 2026-11-06T14:04:00Z

+    // Another process selecting the same direction is replaced, not queued.
+    struct SocketSelect *select = socket_get_select(rsrc_obj, direction);
+    if (select->process_id != ctx->process_id) {
+        socket_select_disown(env, rsrc_obj, select);


On lwIP, nif_socket_connect parks the process trapped in connect in the write slot, without a monitor (line 3078). A write select from another process while the socket is still connecting reaches this line, disowns that process and takes the slot, and only then raises badarg in the lwIP branch (line 1298): the connecting process is never answered. Please reject a write select on a SocketStateTCPNew socket before this point (patch 0009 does that).

Fixed in otp_socket: Add nif_select_write/2: a write select on SocketStateTCPNew is rejected before the write slot is touched, so the process trapped in socket:connect/2 is not disowned.

bettio · 2026-11-06T16:29:29Z

Few additional patches (that you can use as a fix suggestions) for this PR.

0018-gen_udp_socket-Fix-send-blocking-the-server.patch
0019-epmd-Fix-a-crash-when-the-ALIVE2-response-fails.patch
0020-gen_tcp_socket-Remove-a-narration-comment.patch

Blocking sends require read and write readiness to coexist on one socket. Sharing their owner lets a writer replace a reader, misroutes close notifications, and cancels both directions when one selector exits. Give each direction its own owner, monitor and reference, initialized for both opened and accepted sockets. Implement ERL_NIF_SELECT_CANCEL so selector death cancels only its direction, disposes its pending message, and leaves the other direction armed. Retain socket resources while lwIP write/error events are queued and detach callbacks on close. Notify established-socket waiters of terminal errors through direction-specific abort messages, independently of the connect trap. Keep distribution output event-driven so backpressure cannot block incoming traffic or ticks. Preserve pending frame suffixes and queue ticks behind them to maintain byte ordering. Send empty binaries through the native socket path so closed sockets still fail and connected UDP sockets transmit zero-length datagrams. Add regressions for accepted-socket close, writer death, both selector registration orders, exact-data backpressure recovery, cancellation cleanup, and empty UDP sends. Move the changelog entry to beta.1. See also: atomvm#2345 Signed-off-by: Peter M <petermm@gmail.com>

socket:send/2,3 reported any error other than backpressure as {error, closed}, and a send on a socket that was already closed as {error, ebadf}. Report the actual errno (BSD) or lwIP error instead, and {error, closed} for a socket that was closed, as in OTP. Backpressure (EAGAIN/EWOULDBLOCK or lwIP ERR_MEM) stays {error, eagain}, and a connection closed by the peer is reported as {error, closed} rather than {ok, Data}. Co-authored-by: Davide Bettio <davide@uninstall.it> Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

enif_select_write was declared in erl_nif.h but never implemented, and ERL_NIF_SELECT_CANCEL was documented as unimplemented. A select event used to hold a single ref, message and pid shared by the read and the write direction, so selecting one direction silently overwrote the other. Keep them independent, so a resource can have a pending read select and a pending write select at the same time, each with its own owner. Implement ERL_NIF_SELECT_CANCEL (with the *_CANCELLED result flags) to cancel a single direction while keeping the event and the other direction armed. Also fix the ref ticks computation, that tested ref against UNDEFINED_ATOM instead of checking it is a local reference. Signed-off-by: harmon25 <dougwright1@gmail.com>

A select event registered between the count being computed and the lock being released marked the count dirty only to have it overwritten, so its fd was not polled until another event was registered or unregistered. Store the count before releasing the lock, and count the events again whenever the poll set is rebuilt: enif_select sets read/write under the lock but marks the count dirty after releasing it, so a cached count can be smaller than the entries written. Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

Add a write select to the socket NIFs, so a process can wait until a socket can accept more data instead of polling after {error, eagain}. BSD sockets use enif_select_write. lwIP sockets use the tcp_sent and tcp_poll callbacks, and tcp_sndbuf/tcp_sndqueuelen to tell when the send buffer has room again. A socket has one selecting process and monitor per direction, like resources.c: selecting in one direction replaces the previous owner of that direction only, close and select_stop handle both, and socket_down releases only the direction of the dead process. A reader and a writer can therefore block on the same socket without disturbing each other, and an abort is sent to the owner of each direction. On lwIP the callbacks notify the owner of their direction, and a connect traps its caller on the write slot. The selecting state is initialized by a helper shared by all the ways a socket is created, including accept. The lwIP paths also need fixes found while testing them: - detach the callbacks before a pcb is closed or shut down, and pace a retry after tcp_write fails for lack of memory; - update the write select flag under the lwIP lock and keep a queued event from outliving the resource; - answer a select on a closed socket with the closed notification, and reject one on a connecting socket; - release the socket lock on error paths, and test SocketStateClosed with equality, as it is zero. Co-authored-by: Davide Bettio <davide@uninstall.it> Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

lwIP frees the pcb before calling the error callback, for example on a reset, and tcp_shutdown(pcb, 1, 1) can free it right away too. The socket kept using it, and processes blocked in recv or send were never woken up. Forget the pcb, make the socket report closed once the data already received is consumed, and notify the owners of both directions. Accepted sockets now register the error callback as well. Also clear the pcb and the state of a closed socket under the lwIP lock, and do not call tcp_recved on a pcb that is gone. Fixes: atomvm#2432 Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

Make socket:send/2 and socket:sendto/3 retry partial sends and {error, eagain} internally, waiting on nif_select_write/2 for the socket to become writable again. Callers get ok | {error, Reason} and do not need their own retry and backoff loop. Signed-off-by: harmon25 <dougwright1@gmail.com>

- A write select on a socket with a full send buffer wakes up when the peer drains it, and socket:send/2 then completes transparently. - Closing an accepted socket selected by another process. It would have caught accepted sockets not initializing their selecting state, under valgrind. - A process blocked in recv and another blocked in send on the same socket are both aborted when a third process closes it. - When the process blocked in send dies, the one blocked in recv still receives data. - The reader cleanup after a half-close keeps a blocked writer. - Synchronize the backpressure tests explicitly instead of sleeping, use a generous drain timeout, and fail the drain when recv times out. Co-authored-by: Davide Bettio <davide@uninstall.it> Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

socket:send/2 blocks until all the data is accepted, with no way to bound the wait when the peer does not read. As in OTP, send/3 and sendto/4 take a timeout, returning {error, timeout} or {error, {timeout, RestData}} when it expires, or nowait or a select handle, returning {select, SelectInfo} or {select, {SelectInfo, RestData}} and delivering a {'$socket', Socket, select, SelectHandle} message when the socket is writable again. Add nif_select_write_stop/1, which cancels a write select without disowning a process selecting for read, used when the timeout expires, and make the select stops return ok on a closed socket, as closing it already stopped the selects. Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

socket:send/2 no longer returns {ok, Rest} or {error, eagain}. - ssl: wait for write readiness in the handshake, close notify, send and recv loops instead of busy looping on want_write. - socket_dist_controller: terminate the connection when socket:send/2 fails on a tick or on outgoing data, instead of ignoring the error, and do not put the unsent frame in the stop reason. - epmd: handle a failed ALIVE2 response instead of crashing the server. - Remove the {ok, Rest} branches that can no longer happen from the examples. Co-authored-by: Davide Bettio <davide@uninstall.it> Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

socket:send/2 ran inside the gen_server and blocks until the peer reads, so gen_tcp:recv/2,3, gen_tcp:close/1 and gen_tcp:controlling_process/2 from other processes hung meanwhile. Send with nowait instead: when the socket is not writable, the rest of the packet waits for the write select notification and the following packets wait in a queue, so the process serving the socket stays responsive. A pending send fails with {error, closed} when the socket is closed or the controlling process terminates. Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

socket:sendto/3 waits for the socket to be writable when sendto(2) returns EAGAIN on BSD sockets, and gen_udp_socket performed the send inside the gen_server, so a full UDP send buffer stalled every other operation on the socket until it drained. Send with nowait and keep the datagram waiting for the write select in the gen_server state, one at a time and in order. The caller still blocks until its datagram is accepted, but the server keeps serving the socket, and pending sends fail with {error, closed} when the socket is closed. Signed-off-by: Davide Bettio <davide@uninstall.it> Signed-off-by: harmon25 <dougwright1@gmail.com>

A select on an already closed socket sends the closed notification but left the owner, its monitor and the monitor's reference armed. No event will ever answer it, so drop the owner in nif_socket_select and in the closed-fd path of the stop NIFs. Queued lwIP sent and error handlers also used the resource after releasing the lwIP lock. The selecting owner can terminate in between, dropping the last reference, so hold one while notifying it. Signed-off-by: harmon25 <dougwright1@gmail.com>

Signed-off-by: harmon25 <dougwright1@gmail.com>

harmon25 · 2026-11-12T04:59:57Z

All outstanding review comments are addressed and the branch is rebased onto the current release-0.7 as a 13-commit linear series.

Locally on generic_unix, test-enif and the full test_estdlib suite pass natively and under valgrind (run repeatedly for the backpressure tests), and the ESP32 build with ESP-IDF 5.5.4 compiles the lwIP paths. The workflow runs for this fork PR are currently waiting for maintainer approval; the gcc-10 -m32 job in particular is the one to watch for the backpressure test synchronization.

harmon25 marked this pull request as ready for review July 1, 2026 19:47

harmon25 changed the title ~~Improve socket:send/2,3 error handling for transient send backpressure~~ socket:send/2,3: distinguish eagain from closed, and retry transparently via write-select Jul 4, 2026

harmon25 changed the title ~~socket:send/2,3: distinguish eagain from closed, and retry transparently via write-select~~ socket:: distinguish eagain from closed, and retry transparently via write-select Jul 6, 2026

harmon25 changed the title ~~socket:: distinguish eagain from closed, and retry transparently via write-select~~ socket: distinguish eagain from closed, and retry transparently via write-select Jul 6, 2026

bettio mentioned this pull request Oct 5, 2026

BSD sockets: connect without blocking the scheduler #2416

Open

bettio requested changes Oct 5, 2026

View reviewed changes

petermm mentioned this pull request Oct 10, 2026

Socket: nif_select_write/2 #2422

Open

harmon25 force-pushed the eagain_tcp branch from 80bf4f9 to 2883e4e Compare October 10, 2026 18:40

harmon25 mentioned this pull request Oct 10, 2026

socket_dist_controller: sends can block the controller and hide a dead peer #2424

Open

harmon25 requested a review from bettio October 10, 2026 18:47

github-advanced-security Bot found potential problems Oct 12, 2026

View reviewed changes

Comment thread tests/test-enif.c Fixed

bettio requested changes Oct 13, 2026

View reviewed changes

Comment thread tests/test-enif.c Resolved

harmon25 force-pushed the eagain_tcp branch from 2883e4e to becf7d2 Compare October 13, 2026 17:00

harmon25 requested a review from bettio October 13, 2026 17:05

bettio approved these changes Oct 13, 2026

View reviewed changes

harmon25 force-pushed the eagain_tcp branch from becf7d2 to e2e3ec3 Compare October 14, 2026 18:08

bettio requested changes Oct 14, 2026

View reviewed changes

harmon25 force-pushed the eagain_tcp branch from e2e3ec3 to 42a027c Compare October 15, 2026 17:38

harmon25 requested a review from bettio October 15, 2026 19:05

bettio requested changes Oct 16, 2026

View reviewed changes

harmon25 force-pushed the eagain_tcp branch from 42a027c to c34bb11 Compare October 16, 2026 17:52

harmon25 mentioned this pull request Oct 16, 2026

lwIP sockets: a peer reset does not wake processes blocked in socket:send/2 or recv #2432

Open

harmon25 requested a review from bettio October 16, 2026 21:49

bettio requested changes Nov 6, 2026

View reviewed changes

bettio reviewed Nov 6, 2026

View reviewed changes

harmon25 and others added 13 commits November 11, 2026 23:54

Document socket write select in the changelog

c4a2824

Signed-off-by: harmon25 <dougwright1@gmail.com>

harmon25 force-pushed the eagain_tcp branch from c34bb11 to c4a2824 Compare November 12, 2026 04:56

Conversation

harmon25 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

Uh oh!

petermm commented Jul 2, 2026

Uh oh!

harmon25 commented Jul 4, 2026

Uh oh!

harmon25 commented Jul 4, 2026

Uh oh!

bettio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bettio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bettio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harmon25 commented Oct 15, 2026

Uh oh!

bettio left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harmon25 commented Oct 16, 2026

Uh oh!

bettio left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harmon25 commented Jun 30, 2026 •

edited

Loading

bettio left a comment •

edited

Loading