Skip to content

arm64: the FP/SIMD registers belong to the thread and to the domain - #381

Merged
daniel-rossier merged 1 commit into
mainfrom
fix/arm64-fpsimd-context
Nov 15, 2026
Merged

daniel-rossier merged 1 commit into
mainfrom
fix/arm64-fpsimd-context

Conversation

@daniel-rossier

Copy link
Copy Markdown
Contributor

On arm64 nothing saved v0–v31, FPSR and FPCR: __switch_to and __switch_domain_to carry x19–x29, lr and sp, the exception frame x0–x30. The threads of a process, the guests sharing a CPU and the kernel itself (built without -mgeneral-regs-only, it used v0/v1 in printk, sprintf, fork, the hypercall path) all wrote the same registers. A preempted thread got another context's values back.

The visible failure was a process killed in musl's fmt_fp: the loop which writes the integer part of a floating-point number goes through v0/v1 at each step, a corrupted value never reaches zero, and the loop writes upward until it leaves the stack.

Change

  1. struct fpsimd_state (v0–v31, FPSR, FPCR), saved and loaded by fpsimd_save() / fpsimd_restore().
  2. Thread switch. The scheduler saves the state of the thread it leaves and loads the next one's. A new thread starts with its creator's, as fork() requires.
  3. Domain switch. AVZ does the same in vcpu_t. The injector already copies vcpu_t into a capsule's context, so the state travels with a snapshot; the size of that context changes, both ends must be rebuilt.
  4. The arm64 kernel is built with -mgeneral-regs-only, so the registers stay those of the context which entered it until a switch. strtod.c and roxml_xpath.c compute in doubles and are exempted; nothing in the kernel calls them.
  5. CHANGELOG: one sentence in the v6.4.0 entry.

Checks

SOO-NG on virt64 under QEMU, 180 s per node, a capsule whose threads hold a pattern in d0, d1, d8, d31 and read it back in user mode, or format doubles of known text (wrong reads / reads):

Setting Before After
one capsule, one thread 37 / 29 742 0 / 27 421
one capsule, two threads 8 535 / 17 923 0 / 14 218
two capsules on a node 2 975 / 15 535 0 / 13 392
two threads formatting doubles killed at once 0 / 19.8 M

Before, the values found were the other thread's pattern, the other capsule's, and kernel addresses.

  • All 17 configurations build.
  • After the change only fpsimd_save and fpsimd_restore touch an FP/SIMD register in the kernel image.
  • clang-format check clean.

Limits

  • Tested on SOO-NG's copy of these sources with this diff applied, not on a standalone SO3 nor on a board.
  • Migration of a capsule with the new context not exercised.
  • A signal handler still runs on the interrupted thread's registers.
  • arm32 is unchanged: vfp_save_state() / vfp_restore_state() have no caller there either.

On arm64 nothing saved v0-v31, FPSR and FPCR: __switch_to and
__switch_domain_to carry x19-x29, lr and sp, the exception frame x0-x30.
The threads of a process, the guests sharing a CPU and the kernel itself
(built without -mgeneral-regs-only, it used v0/v1 in printk, sprintf,
fork, the hypercall path) all wrote the same registers. A preempted
thread got another context's values back.

The visible failure was a process killed in musl's fmt_fp: the loop which
writes the integer part of a floating-point number goes through v0/v1 at
each step, a corrupted value never reaches zero, and the loop writes
upward until it leaves the stack. Seen twice in SOO-NG campaigns on a
capsule printing "%.2f".

Now:
 - struct fpsimd_state (v0-v31, FPSR, FPCR), saved and loaded by
   fpsimd_save()/fpsimd_restore();
 - the scheduler saves it for the thread it leaves and loads the next
   one's; a new thread starts with its creator's, as fork() requires;
 - AVZ does the same for the domains, in vcpu_t, which the injector
   already copies into the capsule's context: the state travels with a
   snapshot (size of that context changes, both ends must be rebuilt);
 - the arm64 kernel is built with -mgeneral-regs-only, so the registers
   stay those of the context which entered it until a switch. strtod.c
   and roxml_xpath.c compute in doubles and are exempted; nothing in the
   kernel calls them.

Test, SOO-NG on virt64 under QEMU, 180 s per node, a capsule whose
threads hold a pattern in d0, d1, d8, d31 and read it back in user mode,
or format doubles of known text:

                                        before          after
  one capsule, one thread               37 / 29742      0 / 27421
  one capsule, two threads              8535 / 17923    0 / 14218
  two capsules on a node                2975 / 15535    0 / 13392
  two threads formatting doubles        killed at once  0 / 19.8 M

(wrong reads / reads.) Before, the values found were the other thread's
pattern, the other capsule's, and kernel addresses.

All 17 configurations build; after the change only fpsimd_save and
fpsimd_restore touch an FP/SIMD register in the kernel image.

Limits: tested on SOO-NG's copy of these sources, not on a standalone
SO3 nor on a board; migration of a capsule with the new context not
exercised; a signal handler still runs on the interrupted thread's
registers; arm32 is unchanged (vfp_save_state()/vfp_restore_state() have
no caller there either).
@daniel-rossier

Copy link
Copy Markdown
Contributor Author

Rebased on main after #380 (x86_64). Two conflicts, both in Makefiles next to the x86 additions: the arm64 flags in so3/so3/Makefile and the strtod.o line in lib/Makefile. The roxml_xpath.o exemption is now limited to arm64, so the x86 build is unchanged. Rebuilt after the rebase: virt64, virt64_avz_soo, virt64_capsule, rpi4_64_avz, virt32, pc64.

@daniel-rossier
daniel-rossier force-pushed the fix/arm64-fpsimd-context branch from daaf4f2 to 775f37d Compare November 15, 2026 09:07
@daniel-rossier
daniel-rossier merged commit 6b89c5d into main Nov 15, 2026
5 of 6 checks passed
@daniel-rossier
daniel-rossier deleted the fix/arm64-fpsimd-context branch November 15, 2026 09:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant