[f-brinkmann]

Which issue(s) are closed by this pull request?

Closes #147 and fixes #140. Also fixes failing tests in #148

Changes proposed in this pull request:

Improved writing of SOFA attributes and string variables to be fully conform with the SOFA standard

• SOFA attributes and string variables are now always written as netCDF char array as required by the SOFA standard
  • To this end all conversion between python/numpy and netCDF data types is done in sofar.io._format_value_for/from_netcdf and the use of the netCDF functions stringtochar and chartostring that caused the error in the first place is now avoided.
• SOFA attributes and string variables can be read even if tey were written as a netCDF string to be user friendly and backwards compatible
• Remove restriction on numpy 2.5
  • this required the new restriction on netCDF4 >= 1.7.5 due to numpy and
  • the restriction to Python >= 3.11 due to netCDF (Python 3.10 is close to end of life)
• Failing ruff checks are fixed in Fix ruff #150 to keep this PR more focused.

All changes are fully backwards compatible because they only changed how data are handled internally in Sofa and SofaStream.

AI disclaimer:

The fixes were done by an LMM. I checked the SOFA standard before hand. I discussed and instructed the LMM with care, and checked all output.

[Copilot]

🟡 Changes recommended

UTF-8 byte-array comparisons can raise decoding errors, and custom netCDF string variables lack dimension metadata needed for reliable read/write round trips.

2 open issues

• Normalize NC_STRING dimensions to preserve text data · New
• UTF-8 byte-string comparisons raise UnicodeDecodeError · New

What changed in this PR

This PR addresses #147 by improving UTF-8 handling in SOFA file I/O, writing text as standard-compliant netCDF character data while accepting netCDF string data when reading.

Changes:

• Updates text encoding, decoding, comparison, and byte-length calculation.
• Adds tests for UTF-8 data and netCDF attribute types.
• Raises minimum requirements to Python 3.11 and netCDF4 1.7.5, removes the NumPy upper bound, and updates CI and documentation.

File	Description
tests/​test_io.py	Adds encoding and netCDF type checks.
sofar/​utils.py	Changes string comparisons to Unicode.
sofar/​sofastream.py	Accepts netCDF string variables during verification.
sofar/​sofa.py	Calculates string lengths in UTF-8 bytes.
sofar/​io.py	Updates text serialization and decoding.
README.md	Updates the minimum Python version.
pyproject.toml	Revises Python and dependency requirements.
HISTORY.rst	Documents text handling and support changes.
.circleci/​config.yml	Removes unsupported Python versions from CI.

_{Review effort: Balanced}

[hoyer-a]

As far as i can judge, this looks good!

[artpelling]

dropping support is not really deprecation. Maybe rephrase?

[artpelling]

Suggested change

	if file[var].dtype in ["S1", str]:
	if file[var].dtype == str:
	dimensions += "S"
	dtype = "string"
	elif file[var].dtype == "S1":

I think Copilot has a point here. For string, we need to append S otherwise, we might get a shape mismatch in write_sofa(). Maybe we also should test this?

[artpelling]

Same here.

Suggested change

	dimensions = "".join(list(netcdf_variable.dimensions))
	if netcdf_variable.dtype == str:
	dimensions += "S"

[artpelling]

Is this really necessary? I think disabling this breaks cases where _Encoding was used.

[artpelling]

You can try this:

   from pathlib import Path
   from tempfile import TemporaryDirectory

   import netCDF4
   import numpy as np
   import sofar as sf

   with TemporaryDirectory() as tmp:
       path = Path(tmp) / "encoding.sofa"
       sf.write_sofa(path, sf.Sofa("SimpleFreeFieldHRIR"))

       with netCDF4.Dataset(path, "a") as nc:
           nc.createDimension("L", 4)
           var = nc.createVariable("Legacy", "S1", ("L",))
           var[:] = np.frombuffer(b"M\xf6p", dtype="S1")
           var._Encoding = "latin-1"

       print(sf.read_sofa(path, verify=False, verbose=False).Legacy)

which worked on main but fails now with UnicodeDecodeError because _Encoding is ignored and UTF-8 is assumed.