[wesbiggs]

Problem

Once Ethernet is initialized (a cable is not needed), NativeEthernet runs the whole FNET stack from a 1 ms IntervalTimer (PIT) ISR (FNET_POLL_TIME in NativeEthernet.h). Measured with late-ISR counters in isrPHI2, about 39 phi2 interrupts per second came in late (>1.1 µs) even though the GPIO interrupt has the higher priority. Masking IRQ_ENET changed nothing; masking IRQ_PIT removed them.

Under Kernal Replace, late interrupts hurt most on the first read after a $01 write, where KernalCheck senses HIRAM before it drives the bus. Programs that switch $01 constantly got corrupt reads, NMIs or froze:

• My test RPG project game (keeps its map under the I/O area and the KERNAL)
• CCGMS with SwiftLink over TCP, running under a replaced KERNAL (its NMI handler writes $01)

Change

While Kernal Replace serves the KERNAL and the stack is up, IRQ_PIT is masked and FnetLoopPollService() (Teensy.ino, FLASHMEM) calls fnet_poll() from loop() every ms. Late phi2 interrupts drop to about one every 2 s. "Serves the KERNAL" means the Kernal Replace Special IO, or fKernRepl set under Fab04_GlobalKernalReplace.

• Starts at PreStart. loop() only sees the new handler ~0.4 s after a launch, and the ~15 late interrupts in that window were still enough to crash Quest for the King. InitHndlr_KERNALReplace_PreStart() therefore starts the loop poll at once (FnetLoopPollStart()), and SetUpMainMenuROM() ends it.
• AT commands. Under Fab04_GlobalKernalReplace, SwiftLink can run while the KERNAL is replaced. connect, DHCP and DNS block waiting for the stack, so ProcessATCommand() unmasks IRQ_PIT while an AT command runs. The wrapper is compiled only under that flag.
• Remote commands. A remote command that waits for its input bytes runs inside loop(), so with the PIT masked nothing polled the stack while it waited. A payload longer than one TCP segment could then stall until SerialTimoutMillis. CmdInputPoll() (SerUSBIO.ino) calls FnetLoopPollService() in the waits of SerialAvailabeTimeout, DiscardCmdInput and ReceiveSpanBytes; it compiles to nothing without Fab04_KernalReplace.
• TR+ only. Everything is guarded by Fab04_KernalReplace, so the plain TR build is unchanged (same link report as main).
• Scope. Other handlers are unchanged, plain SwiftLink on the stock KERNAL included: its late-interrupt rate hasn't been measured, so it isn't in this PR.
• Docs. docs/Architecture/Constraints.md describes the mechanism and its two consequences: TCP is serviced only as often as loop() comes round, and new blocking network calls that can run while the KERNAL is replaced need the PIT back.

Memory (link report, before → after)

	TR+ (--no-extensions)	TR
RAM1 free for local variables	85532 → 85532	unchanged
RAM1 padding	31800 → 31784	unchanged
RAM2 variables	17600 → 17600	unchanged
FLASH code	269004 → 269284	unchanged

Testing

• Hardware: real C64 (breadbin), TR+ (PCB v0.4), with Ethernet initialized.
  • RPG plays and loads/saves under Kernal Replace.
  • CCGMS with SwiftLink over TCP under a replaced KERNAL: no stalls or flow-control problems; a repeatable first-page receive glitch remains, cause not yet found.
  • Tested as part of the teensyrom-vdrive firmware build (the same mechanism, with VDrive's handlers in the list).
• Host: npm test passes; TR and TR+ firmware build, and TR+ also builds with Fab04_GlobalKernalReplace defined.
• No host test added: the change is interrupt masking and polling cadence, which the existing harnesses can't exercise meaningfully.

🤖 Generated with Claude Code

[SensoriumEmbedded]

Thank you for tracking the late PHI2 interrupts down to the Ethernet poll timer and for getting them out from under Kernal Replace, great find! Ethernet interference has been something worked around from the beginning. I'll be looking at implementing this across the board to help in other area as well. :)

I reviewed this merged onto main and built both targets, then ran the TR+ build of aaa2b8c merged onto current main (eae5075) on hardware. Your follow-up commit (polling from the remote-command waits) landed while I was writing this up; I've read it and it looks right to me: those waits all run from loop() with nothing of the stack on the call stack, and it closes the case I had on my list, a TCP payload longer than one segment stalling until the time-out. I haven't run that path myself, so there's one specific test request under 3. As always, feel free to push back on any that are inappropriate or would take significant memory/code to resolve.

Memory: on the TR+ main image this adds 256 B of flash (FnetLoopPollService 136, FnetLoopPollStart 44, the call in PreStart 44, two veneers, SetUpMainMenuROM, CmdInputPoll and its call sites the rest) and nothing to RAM1 beyond alignment: 12 B of ITCM and 6 B of DTCM, both inside the current steps, so padding and free for local variables are unchanged. Plain TR's main image gains 8 B of flash for the empty CmdInputPoll, nothing else; MinimalBoot and the extension image are byte-identical.

There are 3 things I'd like you to look at before merging. One is a hang path in the Global build's browser, one is the loop() call running on every TR+ whatever it's doing, and the third is the send-side twin of your follow-up commit; suggestions are in-line and below.

1. The Global build's browser still blocks with the PIT masked (Swift_ATcommands.c:476)

The wrapper re-enables the PIT around ProcessATCommand() only. Browser-mode input goes through ProcessBrowserCommand() instead (IOH_Swiftlink.c:634), and a page by host name ends in WebConnect() → client.connect(host, port) → NativeEthernet's DNSClient::getHostByName(), which spins in while (resolveDone == 0) {} with the flag set only from fnet_service_poll(). That poll runs from the PIT timer or from loop(), and loop() isn't running while the handler blocks, so under a replaced KERNAL the browser's first host-name connect never returns. Repro: Global build, KERNAL replaced, SwiftLink browser, open any page by name.

Suggested fix: do the unmask and re-mask once, at the dispatch, so both paths are covered, and drop the ProcessATCommandInner wrapper:

//IOH_Swiftlink.c, polling handler, where a complete command line is dispatched
#ifdef Fab04_GlobalKernalReplace
            if (FnetLoopPoll) NVIC_ENABLE_IRQ(IRQ_PIT); //connect/DNS block in here waiting for the stack
#endif
            if(BrowserMode)
            {
               ProcessBrowserCommand();
               SendPETSCIICharImmediate(PETSCIIwhite);
            }
            else
            {
               ATRespCode Resp = ProcessATCommand();
               if (!BrowserMode) SendATresponse(Resp);
            }
#ifdef Fab04_GlobalKernalReplace
            if (FnetLoopPoll) NVIC_DISABLE_IRQ(IRQ_PIT);
#endif

(with extern volatile bool FnetLoopPoll; moving to that file). It's Global-only, so the shipping build never reaches it; keeping the Global build working matters because it's the one that debugs this kind of thing.

2. FnetLoopPollService() runs on every loop() pass on every TR+, and want masks the PIT with no kernal loaded (Teensy.ino:391, :423)

Two things in one place:

• The call at Teensy.ino:391 runs on every pass whatever handler is active, menu, SwiftLink, EasyFlash included, and it's a FLASHMEM function. Constraints.md's rule is that a feature costs nothing while it's off.
• want's CurrentIOHandler == IOH_KernalReplace term is also true when PreStart returned before a kernal was loaded (file missing, wrong size, allocation failed) and for a generic CRT that inherits Kernal Replace as its Special IO: the handler is Kernal Replace, nothing is served, and the PIT stays masked for the whole session for no reason. It also keeps FnetLoopPoll true for the window after SetUpMainMenuROM() re-enables the PIT (:460) while the handler is still Kernal Replace, so for those passes the PIT ISR and loop() both poll.

FnetPollHold already says exactly "a kernal is being served": set at the end of a successful PreStart, cleared with the menu. Suggested fix (in-line): gate the call so nothing runs unless the hold is on or there's a timer to switch back, and derive want from FnetPollHold alone. Three small things then fall out: FnetLoopPollStart() can return before setting the hold when the stack isn't initialized, so a Kernal Replace session with Ethernet off costs loop() nothing and the stack test in want goes with it; the Global fKernRepl term is redundant with the hold (both are armed in PreStart and cleared in SetUpMainMenuROM()); and SetUpMainMenuROM() can clear FnetLoopPoll beside the hold, since it re-enables the PIT itself, so the loop poll stops with it instead of one pass later. FnetLoopPollStart() would then read:

FLASHMEM void FnetLoopPollStart()
{  //start polling from loop() right away: loop() only sees the new handler ~0.4 s after the launch, and late phi2 interrupts in that time crashed programs
   if (fnet_netif_get_default() == 0) return; //stack not initialized: nothing to poll
   FnetPollHold = true;
   if (!FnetLoopPoll) { NVIC_DISABLE_IRQ(IRQ_PIT); FnetLoopPoll = true; }
}

While you're there, it sits at the top of the file beside the flags; moving it down next to FnetLoopPollService() keeps the mechanism in one place.

3. Long replies over TCP can spin in the library with nothing polling (SerUSBIO.ino:1532)

Your follow-up covers a command waiting for its bytes; the mirror is a command sending a long reply. ReadC64MemCommand() writes its payload a byte at a time, and once the socket's send buffer is full each write() lands in NativeEthernet's EthernetClass::socketSend(), which spins in while (socketSendAvailable(s) < len) {} with no poll. Buffer space comes back with the peer's ACKs, which the ENET interrupt handles, so normally the spin ends; but if one of the TR's data segments is lost, the retransmit timer lives in fnet_poll(), and with the PIT masked and loop() inside that spin nothing ever calls it. Main's PIT timer would have retransmitted; here the main loop stays in the spin for good (a later close from the peer doesn't end it). It needs a lost segment during a ReadC64Mem over TCP under Kernal Replace, so it's rare on a wired LAN, but it's the same hole your commit closed on the receive side.

Suggested fix: poll before the write whenever the channel has no room, with one SerialTimoutMillis budget for the whole reply so a channel that never drains (host gone) costs no more than that, and the library spin is only ever entered with the stack freshly polled (USB serial has its own write time-out and is unaffected):

   uint32_t Began = millis();
   for(uint32_t ByteNum = 0; ByteNum < DMALength; ByteNum++)
   {
      while (!CmdChannel->availableForWrite() && millis() - Began < SerialTimoutMillis) CmdInputPoll();
      CmdChannel->write(DMABuf[ByteNum]);
   }

A specific request on this one: your follow-up commit and item 3 both sit on the TCP path under Kernal Replace, and I haven't run that path with either, and the PR's testing section doesn't say whether the follow-up was run after the fix. Once the changes are in, could you run a few hundred WriteC64Mem commands with 4 KB payloads and a few hundred ReadC64Mem reads of 8-16 KB over TCP while Kernal Replace serves the KERNAL, and confirm no time-outs and the TR still answering afterwards? Your 1-in-550 number is the kind of rate only a long run shows, and that run is the evidence I'd like to merge on.

Hardware testing: Just to document. I'll hit these at some point, not a request or requirement! :)

Your testing covers an RPG under Kernal Replace with the stack up on a C64, and CCGMS over SwiftLink under a replaced KERNAL in the Global build.

I ran the TR+ build of this head merged onto current main on an NTSC C128 in C64 mode with the stack initialized: the $01-switching test from the PR #78 review under Kernal Replace with JiffyDOS gave the same results as before; BASIC and the keyboard with the stock, JiffyDOS and JaffyDOS images; a file launch over TCP during a Kernal Replace program; after the session, Set RTC from Net in the menu (its host-name lookup only completes from the PIT's poll while the menu blocks, so the timer is back) and a TCP connection; then CCGMS over SwiftLink on the stock KERNAL. On an NTSC C64 the same $01 test passed under Kernal Replace with the stack down. I also built the suggestions above on top of this head and ran them on the NTSC C64 with the stack up: a Kernal Replace session still polls from loop() and the menu gets its timer back afterwards (Set RTC from Net completes), and Kernal Replace selected with the kernal file missing now leaves the timer alone.

• A long run of a $01-heavy program under Kernal Replace with the stack up, on a C64, NTSC and PAL
• A reset over TCP during a Kernal Replace program, and the TCP commands with NFC enabled
• After 1: the Global build's browser opening a page by host name under a replaced KERNAL

Thank you!

[SensoriumEmbedded]

See 1 in the main comment: the browser's ProcessBrowserCommand() path blocks in DNS the same way and doesn't come through here. Suggest the unmask/re-mask pair moves to the dispatch in IOH_Swiftlink.c around both calls, and this wrapper (and ProcessATCommandInner) goes away.

[wesbiggs]

Done

[SensoriumEmbedded]

Nothing to do, and nothing to switch back, unless the hold is on or the timer is still masked:

Suggested change

	FnetLoopPollService();
	if (FnetPollHold || FnetLoopPoll) FnetLoopPollService();

[wesbiggs]

addressed

[SensoriumEmbedded]

FnetPollHold is set only once PreStart has a kernal loaded and cleared with the menu, so it's the one condition needed; the handler term also fires with nothing served (missing file, a generic CRT inheriting Kernal Replace) and keeps this true after SetUpMainMenuROM() has re-enabled the PIT. With FnetLoopPollStart() skipping the hold when there's no netif (main comment), the stack test goes too, and under Global fKernRepl is armed and cleared at the same two sites as the hold:

Suggested change

	bool want = FnetPollHold || (CurrentIOHandler == IOH_KernalReplace)
	#ifdef Fab04_GlobalKernalReplace
	|| (fKernRepl != NULL)
	#endif
	;
	want = want && (fnet_netif_get_default() != 0); //stack initialized
	bool want = FnetPollHold; //set by FnetLoopPollStart once a kernal is served with the stack up, cleared with the main menu

[wesbiggs]

done

[SensoriumEmbedded]

The PIT is re-enabled a few lines down, so the loop poll can stop here too instead of on the next loop() pass:

Suggested change

	FnetPollHold = false;
	FnetPollHold = FnetLoopPoll = false; //the PIT timer is re-enabled below

[wesbiggs]

done

[wesbiggs]

Thanks for the careful review. I’ve pushed 55f7134, which covers all of it, with one change to your suggestion on the hold.

1. Browser path: done. The unmask/re-mask is now in the SwiftLink dispatch in IOH_Swiftlink.c, around both ProcessBrowserCommand() and ProcessATCommand(), and the ProcessATCommandInner wrapper is gone.
2. loop() cost: loop() now calls FnetLoopPollService() only when FnetPollHold || FnetLoopPoll. I kept setting the hold unconditionally in FnetLoopPollStart(), though, rather than only when the network interface is already up. The stack often comes up after launch: a SwiftLink program running ATC/DHCP while the KERNAL is replaced (Global builds, and VDrive+SwiftLink on my side). If the hold depends on the stack being up at PreStart, those programs keep the PIT poll for the whole run, which is the CCGMS case that started this. With the hold always set, a launch without a network costs one fnet_netif_get_default() call per loop pass, and loop() moves the poll off the PIT once the stack exists.
3. want: agreed. It’s now FnetPollHold && fnet_netif_get_default() != 0. The CurrentIOHandler == IOH_KernalReplace term was actually wrong: when the KERNAL file fails to load, PreStart returns early and nothing is served, but that term still turned the loop poll on. fKernRepl was redundant.
4. SetUpMainMenuROM(): it now clears FnetLoopPoll with the hold.
5. Send side: good catch, and it’s worse than a stall. NativeEthernet’s socketSend() is while(socketSendAvailable(s) < len){} with no timeout, so with the PIT masked, a full send buffer plus a lost segment hangs the Teensy for good. ReadC64MemCommand() now sends through CmdSendSpan(), which writes whatever availableForWrite() allows and calls fnet_poll() while the buffer is full. I used a 2 s no-progress timeout (SerialTimoutMillis*4, the same as SendFileData) rather than one fixed budget for the whole reply, so that a large reply on a slow link isn’t cut off while it’s still making progress. SendFileData() had the same wait with no poll, so I added CmdInputPoll() to it as well.

These changes are all in the latest commit here.

[SensoriumEmbedded]

Thank you for the quick turnaround, and for taking the send side further than I'd suggested: the no-progress time-out and the SendFileData() wait are both good additions.

I reviewed 55f7134 against the previous head. Items 1 to 3 all look right to me, and you're right about the hold: a stack that comes up mid-session needs it, and without a network it costs one call per loop() pass. Agreed.

There is 1 (very minor) thing I'd like you to look at before merging, suggestion in-line, plus the test run below.

1. The stall message goes to Serial (SerUSBIO.ino:1553)

When the command came in over USB, Serial is also the reply channel, so after a stall the message lands in the binary stream behind the partial reply, and the host reads it as part of its next answer. As a plain string literal it also sits in DTCM. Printf_dbg compiles out in normal builds and fixes both.

Testing

Your comment doesn't mention a run on 55f7134. Could you run the test from my first review on it: a few hundred WriteC64Mem commands with 4 KB payloads and a few hundred ReadC64Mem reads of 8-16 KB over TCP while Kernal Replace serves the KERNAL, and confirm no time-outs and the TR still answering afterwards? CmdSendSpan() is new on that path, and that run is the evidence I'd like to merge on. If you don't have time, let me know and I'll try to fit it in.

A C128 in C64 mode is fine for this. Leaving the Kernal Replace program at a BASIC prompt and writing to free RAM keeps the C64 side out of it, since pass or fail is on the TCP side.

Thank you!

[SensoriumEmbedded]

Debug-only, so it stays out of the reply stream and out of DTCM:

Suggested change

	if (!CmdSendSpan(DMABuf, DMALength)) Serial.printf("DMA Read: reply stalled, %lu bytes\n", DMALength); //Ack already sent, the host times out
	if (!CmdSendSpan(DMABuf, DMALength)) Printf_dbg("DMA Read: reply stalled, %lu bytes\n", DMALength); //Ack already sent, the host times out

[wesbiggs]

Thanks — applied your suggestion in 81ed96b.

I ran the test on 142f57b. Its firmware matches 81ed96b; the two commits after it only touch tools/bench/tcpstress.py, which I added so the run can be repeated: python3 tools/bench/tcpstress.py <TR IP>. Each run sends 300 WriteC64Mem commands of 4 KiB and 300 ReadC64Mem reads of 8–16 KiB, shuffled, over one TCP connection. It checks every read, plus a full read-back at the end, against what was written to $2000-$9FFF, then checks the TR still answers on the same connection and on a new one.

Setup: TR+ in a C128 in C64 mode, NTSC, with Kernal Replace serving a patched KERNAL at a BASIC prompt. The host was a Raspberry Pi on wired Ethernet.

Three runs, 1,800 commands in all: no time-outs, no mismatches, and the TR answered on both connections after each run. About 475 KiB/s in every run, and the slowest command was 32 ms, always a 16 KiB read. Three earlier runs from a Mac over Wi-Fi also passed, at 154–191 KiB/s; one read in those took 2.9 s but came back correct, and it didn't recur on the wired link. Not tested on PAL or a real C64.

🤖 Generated with Claude Code

[SensoriumEmbedded]

Thank you for the run, and for adding tcpstress.py. 1,800 commands with no time-outs or mismatches is exactly what I was after.

Squash-merging now. Thanks again for tracking the late interrupts down to the Ethernet poll timer and getting them out from under Kernal Replace! :)