[heios]

macFUSE 5 can run file systems in user space through FSKit, which needs neither its kernel extension nor Reduced Security. This PR uses that backend on macOS 26 and later when the kernel extension is not loaded, so the macFUSE build can mount volumes on a Mac with default security settings. Related to #1900, which asks for FSKit through FUSE-T; this covers the macFUSE build only.

The backend is chosen once per mount, before the FUSE process forks. FSKit is used when macOS is 26 or later, the kernel extension is not loaded (no vfs.generic.macfuse.version.number sysctl) and the installed macFUSE is 5.2 or later. Otherwise the mount goes through the kernel extension as before. An extension that is approved but not yet loaded cannot be told apart from an unapproved one without a privileged load attempt, so such a system now mounts through FSKit. An extension loaded later is picked up by the next mount.

What FSKit needed:

• -o backend=fskit is passed to the auxiliary mount.
• FSKit loads no kernel extension, so the version sysctl does not exist after the mount. A missing sysctl after a successful mount is now taken as FSKit instead of raising HigherFuseVersionRequired.
• FSKit ignores direct_io and fails a read that ends before the advertised file size with EIO. control and aux-device-info already advertise 64 KiB, so their reads are now padded with zeros up to that size. This part is shared code, so Linux and FreeBSD serve the same padded files.
• FSKit extensions are enabled per user, and macFUSE refuses FSKit mounts from root. When the elevated service mounts a volume through FSKit, the FUSE process opens the volume as root, gives the mount point to the requesting user and drops to that user (setgroups, setgid, setuid) before fuse_main. Dropping from root marks the process set-id, so the user's other processes cannot attach to it or get its task port. The signal-handler process forked earlier stays root, as before. Mounts through the kernel extension are unchanged.
• The host file of the volume is marked close-on-exec on macOS for both backends, so that helpers such as hdiutil do not keep it open after the service exits.

Testing

macOS 27.0.1 on Apple Silicon, macFUSE 5.5.0, kernel extension not loaded:

• command line (--text), without sudo: create a 64 MB exFAT container, mount, write 20 MiB, unmount, mount again and compare SHA-256. The auxiliary mount shows fskit, the kernel extension stays unloaded, and no process or mount is left after the unmount;
• the same commands with sudo: the mount belongs to my user and the FUSE process runs with my uid, while the signal-handler process stays root;
• GUI: create, mount, dismount and quit. No administrator password is asked for a file container.

Linux, because of the padded reads (Debian 13 container, kernel 6.8, aarch64), console build with libfuse 2.9.9 and with libfuse 3.17.2: FAT and exFAT containers, with kernel crypto and with nokernelcrypto, create, mount, --list, write 20 MB, unmount, mount again and compare. control and aux-device-info read back as 65536 bytes. With the libfuse 2 build, test_volume_read.py, test_linux_process.py, test_linux_mount_read_only.py and test_linux_mount_points.py pass, and the three Tests/test.*.hc mounts from the Linux workflow work.

Not tested:

• the fallback with the kernel extension loaded. As far as I can tell from the code it is unchanged, but I was hesitant to relax the security of my machine to test this, sorry for that;
• macOS 26 itself, Intel Macs and FreeBSD.

Two things I noticed with FSKit: on the first mount macOS asks once whether VeraCrypt may access files on a network volume, and in text mode macFUSE prints fuse: forking a threaded process is unsafe, the child may crash or deadlock. I do not know whether the kernel backend prints the second one too.