From e23863c5ebacdc5ce1a4e0c59e253bf8a458f103 Mon Sep 17 00:00:00 2001 From: Brad Barnett <127794626+bdbarnett@users.noreply.github.com> Date: Wed, 4 Nov 2026 01:14:39 -0600 Subject: [PATCH 1/2] Install docs teach omip; a build fails on a name micropython-lib uses omip looks on this index first, so a package name shared with micropython-lib would hide theirs. tools/pins.py overlap compares a built index, the untested one included, with micropython-lib's, and the publish workflow runs it after verify. If their index can't be read it warns and carries on. Ran the unit tests, and the check against a local build and the live micropython-lib index: 171 names here, 129 there, none shared. With aioble planted in the local index it fails and names it. --- .github/workflows/publish.yml | 3 +++ README.md | 12 +++++++--- docs/get-started/install.md | 38 +++++++++++++++++++++++++++---- docs/how-it-works/index.md | 2 +- docs/reference/packages.md | 2 +- tests/test_pins.py | 39 +++++++++++++++++++++++++++++++ tools/pins.py | 43 +++++++++++++++++++++++++++++++++-- 7 files changed, 127 insertions(+), 12 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 2a1ba81..439c6e8 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -112,6 +112,9 @@ jobs: - name: Check the index against the pins run: python tools/pins.py verify build/index + - name: Check no package name is also micropython-lib's + run: python tools/pins.py overlap build/index + - name: Prepare the Pages artifact run: touch build/index/.nojekyll diff --git a/README.md b/README.md index 39197fe..25eccf6 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,17 @@ The `mip` index for OmniPython's libraries, board configs and drivers, at **http On a board with Wi-Fi, in the REPL: ```python -import mip -mip.install("pydevices", index="https://pydevices.github.io/lib") +import omip +omip.install("pydevices") ``` -From a computer, onto a board on USB: +`omip` gets the board online if it isn't, and brings what a package needs from micropython-lib. If `import omip` fails, your firmware doesn't have it yet, so copy it on from a computer: + +```bash +mpremote mip install github:PyDevices/pydevices/lib/omip.py +``` + +From a computer, onto a board on USB, with no Wi-Fi at all: ```bash mpremote mip install --index https://pydevices.github.io/lib pydevices diff --git a/docs/get-started/install.md b/docs/get-started/install.md index d72e0d1..4930b2a 100644 --- a/docs/get-started/install.md +++ b/docs/get-started/install.md @@ -1,17 +1,45 @@ # Install a package -Every install names the index: `https://pydevices.github.io/lib`. [What's in the index](../reference/packages.md) lists the packages. +[What's in the index](../reference/packages.md) lists the packages. ## On a board with Wi-Fi -Connect to your network, then in the REPL: +In the REPL: + +```python +import omip +omip.install("pydevices") +``` + +If the board is offline, `omip` joins the network in `secrets.py` or `settings.toml`, and with neither it asks you, saving what you type only if you say so. It looks for the package here and then on micropython-lib, and brings what the package needs from both. It takes the arguments of `mip.install`, so `omip.install("pydevices", mpy=False)` gets `.py` source instead of `.mpy` files for your board's bytecode version. + +If it can't finish, `omip` raises `OSError`. A program with no one at the REPL passes `prompt=False`, so it never waits for an answer. + +### Getting omip + +Firmware that freezes pydevices has it, once its pydevices release does. On any other MicroPython firmware, copy the one file on from a computer: + +```bash +mpremote mip install github:PyDevices/pydevices/lib/omip.py +``` + +Or, on a board that's already online: ```python import mip -mip.install("pydevices", index="https://pydevices.github.io/lib") +mip.install("github:PyDevices/pydevices/lib/omip.py") ``` -You get `.mpy` files for the MicroPython bytecode version your board runs, or `.py` source with `mpy=False`. +On CircuitPython, copy [`omip.py`](https://github.com/PyDevices/pydevices/blob/main/lib/omip.py) to `lib/` on the CIRCUITPY drive. To let `import mip` work there too, copy [`mip.py`](https://github.com/PyDevices/pydevices/blob/main/board_configs/cp/mip.py) beside it; don't put that file on MicroPython, whose own `mip` it would hide. Installing on the board needs a writable drive, which `omip` explains if it isn't, and `aioble` and the other micropython-lib packages are left out, because CircuitPython's libraries come from `circup`. + +## With plain mip + +Every `mip` install names the index, and you connect to your network first: + +```python +import mip +mip.install("pydevices", index="https://pydevices.github.io/lib") +``` ## From a computer, onto a board on USB @@ -37,7 +65,7 @@ mip.install("esp32-s3-touch-lcd-7", index="https://pydevices.github.io/lib") ## Packages from micropython-lib -Packages that come from [micropython-lib](https://github.com/micropython/micropython-lib) (`aioble`, `sdcard`, `bmi270`, `usb-device-*`) aren't copied here. `mip` takes every file of a package from the one index it was given, so install those from the default index first, then ours: +Packages that come from [micropython-lib](https://github.com/micropython/micropython-lib) (`aioble`, `sdcard`, `bmi270`, `usb-device-*`) aren't copied here. A package that needs one names it under `upstream` in its JSON, and `omip` brings it. `mip` and `mpremote` ignore that key, because they take every file of a package from the one index they were given, so with them install those from the default index first, then ours: ```python mip.install("aioble") diff --git a/docs/how-it-works/index.md b/docs/how-it-works/index.md index 9a74843..bc3bf6c 100644 --- a/docs/how-it-works/index.md +++ b/docs/how-it-works/index.md @@ -8,7 +8,7 @@ A release in any of the source repositories sends this repository a request nami ## pins.json, then a build -[`publish.yml`](../../.github/workflows/publish.yml) records each request in [`pins.json`](../../pins.json) as soon as it arrives. It then rebuilds every package from the releases pinned there with pydevices' `tools/libbuild.py`, and deploys the result to GitHub Pages. +[`publish.yml`](../../.github/workflows/publish.yml) records each request in [`pins.json`](../../pins.json) as soon as it arrives. It then rebuilds every package from the releases pinned there with pydevices' `tools/libbuild.py`, and deploys the result to GitHub Pages. Before it deploys, the build checks that no package name here is also one of micropython-lib's, because `omip` looks here first and a match would hide theirs. If micropython-lib's index can't be read, the build warns and carries on. When a package moves to a new repository, its entry in `pins.json` names the new one and lists the old under `formerly`. A pydevices release made before the move still names the old repository in its `lib-lock.json`; the build takes the package from the new one at the pinned tag. diff --git a/docs/reference/packages.md b/docs/reference/packages.md index 4dc4906..82cde8b 100644 --- a/docs/reference/packages.md +++ b/docs/reference/packages.md @@ -12,4 +12,4 @@ The packages come from these repositories: A board config's package is named after its folder in [board_configs](https://github.com/PyDevices/pydevices/tree/main/board_configs), and a driver's after its folder in [drivers](https://github.com/PyDevices/pydevices/tree/main/drivers). -Packages from micropython-lib aren't here; [install those first](../get-started/install.md#packages-from-micropython-lib) from the default index. +Packages from micropython-lib aren't here, and no package here has a name micropython-lib uses: the build fails if one does, so neither can hide the other. `omip` brings them; with `mip`, [install those first](../get-started/install.md#packages-from-micropython-lib) from the default index. diff --git a/tests/test_pins.py b/tests/test_pins.py index 3b70734..2ef60ce 100644 --- a/tests/test_pins.py +++ b/tests/test_pins.py @@ -9,6 +9,8 @@ import sys import tempfile import unittest +from contextlib import redirect_stdout +from io import StringIO from pathlib import Path ROOT = Path(__file__).resolve().parent.parent @@ -210,3 +212,40 @@ def test_a_root_that_names_another_repository_is_refused(self): {"repository": "PyDevices/audioinstruments", "ref": "v0.5.0", "formerly": ["PyDevices/audiocomponents"]}, {"repository": "PyDevices/somewhere-else", "ref": "v0.4.0"}, ) + + +class Overlap(unittest.TestCase): + """overlap compares a built index with an upstream one, here a folder served from this machine.""" + + def setUp(self): + self._tmp = tempfile.TemporaryDirectory() + self.addCleanup(self._tmp.cleanup) + self.dir = Path(self._tmp.name) + self.index = self.dir / "index" + (self.index / "untested").mkdir(parents=True) + (self.index / "index.json").write_text(json.dumps({"packages": [{"name": "bledev"}]})) + (self.index / "untested" / "index.json").write_text(json.dumps({"packages": [{"name": "bledev"}, {"name": "gt911"}]})) + self.upstream = self.dir / "upstream" + self.upstream.mkdir() + + def theirs(self, *names): + (self.upstream / "index.json").write_text(json.dumps({"packages": [{"name": n} for n in names]})) + return self.upstream.as_uri() + + def test_no_shared_name_passes(self): + with redirect_stdout(StringIO()) as out: + pins.overlap(self.index, self.theirs("aioble", "sdcard")) + self.assertIn("none shared", out.getvalue()) + + def test_a_shared_name_fails_and_names_it(self): + with self.assertRaisesRegex(SystemExit, "bledev"): + pins.overlap(self.index, self.theirs("aioble", "bledev")) + + def test_a_name_only_on_the_untested_index_counts(self): + with self.assertRaisesRegex(SystemExit, "gt911"): + pins.overlap(self.index, self.theirs("gt911")) + + def test_an_unreadable_upstream_warns_and_passes(self): + with redirect_stdout(StringIO()) as out: + pins.overlap(self.index, (self.dir / "not-there").as_uri()) + self.assertIn("::warning::", out.getvalue()) diff --git a/tools/pins.py b/tools/pins.py index de0b538..063903c 100644 --- a/tools/pins.py +++ b/tools/pins.py @@ -28,8 +28,9 @@ GitHub later replaces in the queue is still in ``pins.json`` for the build that replaces it. ``fetch`` checks the pinned sources out, ``lock`` writes the pinned tags into -the checkout's ``lib-lock.json`` for its builder, and ``verify`` checks a built -index has every package at the version pinned. +the checkout's ``lib-lock.json`` for its builder, ``verify`` checks a built +index has every package at the version pinned, and ``overlap`` checks that none +of its names is also one of micropython-lib's. Standard library and git only. """ @@ -43,10 +44,14 @@ import subprocess import sys import time +import urllib.request from pathlib import Path PINS = Path(__file__).resolve().parent.parent / "pins.json" +#: micropython-lib's index, whose names ours must not share. +UPSTREAM_INDEX = "https://micropython.org/pi/v2" + #: Profiles whose tag names a folder of the library repository. FOLDER_PROFILES = ("palettes", "pdwidgets") @@ -227,6 +232,35 @@ def verify(index: Path) -> None: print("the index matches pins.json") +def package_names(index: Path) -> set[str]: + """Every package name a built index publishes, its untested index included.""" + names: set[str] = set() + for path in (index / "index.json", index / "untested" / "index.json"): + if path.is_file(): + names.update(p["name"] for p in json.loads(path.read_text(encoding="utf-8"))["packages"]) + return names + + +def overlap(index: Path, upstream: str = UPSTREAM_INDEX) -> None: + """No package here is named as one of micropython-lib's. + + omip looks here first, so a shared name would hide theirs. Failing to read + their index only warns: their outage shouldn't hold up a release of ours. + """ + url = f"{upstream.rstrip('/')}/index.json" + try: + with urllib.request.urlopen(url, timeout=30) as response: + theirs = {p["name"] for p in json.loads(response.read())["packages"]} + except (OSError, ValueError, KeyError) as exc: + print(f"::warning::can't read {url}, so names weren't compared with micropython-lib's: {exc}") + return + ours = package_names(index) + shared = sorted(ours & theirs) + if shared: + raise SystemExit(f"these package names are micropython-lib's too, and omip would hide theirs: {', '.join(shared)}") + print(f"{len(ours)} package names here, {len(theirs)} in micropython-lib, none shared") + + def main() -> None: parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) commands = parser.add_subparsers(dest="command", required=True) @@ -245,6 +279,9 @@ def main() -> None: k.add_argument("root", type=Path) v = commands.add_parser("verify", help="check a built index against pins.json") v.add_argument("index", type=Path) + o = commands.add_parser("overlap", help="check a built index's names against micropython-lib's") + o.add_argument("index", type=Path) + o.add_argument("--upstream", default=UPSTREAM_INDEX) args = parser.parse_args() if args.command == "apply": apply(args.profile, args.repository, args.ref) @@ -254,6 +291,8 @@ def main() -> None: fetch(args.out, args.root_ref) elif args.command == "lock": lock(args.root) + elif args.command == "overlap": + overlap(args.index, args.upstream) else: verify(args.index) From 487d22c8c42e7b31c401d1dd06f08e42d96e5b44 Mon Sep 17 00:00:00 2001 From: Brad Barnett <127794626+bdbarnett@users.noreply.github.com> Date: Wed, 4 Nov 2026 01:27:14 -0600 Subject: [PATCH 2/2] Install docs: omip waits for the release that puts it on the index This leaves the overlap check, which fails a publish if a package name is also on micropython-lib. --- README.md | 12 +++--------- docs/get-started/install.md | 38 +++++-------------------------------- docs/how-it-works/index.md | 2 +- docs/reference/packages.md | 2 +- 4 files changed, 10 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 25eccf6..39197fe 100644 --- a/README.md +++ b/README.md @@ -9,17 +9,11 @@ The `mip` index for OmniPython's libraries, board configs and drivers, at **http On a board with Wi-Fi, in the REPL: ```python -import omip -omip.install("pydevices") +import mip +mip.install("pydevices", index="https://pydevices.github.io/lib") ``` -`omip` gets the board online if it isn't, and brings what a package needs from micropython-lib. If `import omip` fails, your firmware doesn't have it yet, so copy it on from a computer: - -```bash -mpremote mip install github:PyDevices/pydevices/lib/omip.py -``` - -From a computer, onto a board on USB, with no Wi-Fi at all: +From a computer, onto a board on USB: ```bash mpremote mip install --index https://pydevices.github.io/lib pydevices diff --git a/docs/get-started/install.md b/docs/get-started/install.md index 4930b2a..d72e0d1 100644 --- a/docs/get-started/install.md +++ b/docs/get-started/install.md @@ -1,46 +1,18 @@ # Install a package -[What's in the index](../reference/packages.md) lists the packages. +Every install names the index: `https://pydevices.github.io/lib`. [What's in the index](../reference/packages.md) lists the packages. ## On a board with Wi-Fi -In the REPL: - -```python -import omip -omip.install("pydevices") -``` - -If the board is offline, `omip` joins the network in `secrets.py` or `settings.toml`, and with neither it asks you, saving what you type only if you say so. It looks for the package here and then on micropython-lib, and brings what the package needs from both. It takes the arguments of `mip.install`, so `omip.install("pydevices", mpy=False)` gets `.py` source instead of `.mpy` files for your board's bytecode version. - -If it can't finish, `omip` raises `OSError`. A program with no one at the REPL passes `prompt=False`, so it never waits for an answer. - -### Getting omip - -Firmware that freezes pydevices has it, once its pydevices release does. On any other MicroPython firmware, copy the one file on from a computer: - -```bash -mpremote mip install github:PyDevices/pydevices/lib/omip.py -``` - -Or, on a board that's already online: - -```python -import mip -mip.install("github:PyDevices/pydevices/lib/omip.py") -``` - -On CircuitPython, copy [`omip.py`](https://github.com/PyDevices/pydevices/blob/main/lib/omip.py) to `lib/` on the CIRCUITPY drive. To let `import mip` work there too, copy [`mip.py`](https://github.com/PyDevices/pydevices/blob/main/board_configs/cp/mip.py) beside it; don't put that file on MicroPython, whose own `mip` it would hide. Installing on the board needs a writable drive, which `omip` explains if it isn't, and `aioble` and the other micropython-lib packages are left out, because CircuitPython's libraries come from `circup`. - -## With plain mip - -Every `mip` install names the index, and you connect to your network first: +Connect to your network, then in the REPL: ```python import mip mip.install("pydevices", index="https://pydevices.github.io/lib") ``` +You get `.mpy` files for the MicroPython bytecode version your board runs, or `.py` source with `mpy=False`. + ## From a computer, onto a board on USB `mpremote` fetches the package on your computer and copies it over, so the board needs no network: @@ -65,7 +37,7 @@ mip.install("esp32-s3-touch-lcd-7", index="https://pydevices.github.io/lib") ## Packages from micropython-lib -Packages that come from [micropython-lib](https://github.com/micropython/micropython-lib) (`aioble`, `sdcard`, `bmi270`, `usb-device-*`) aren't copied here. A package that needs one names it under `upstream` in its JSON, and `omip` brings it. `mip` and `mpremote` ignore that key, because they take every file of a package from the one index they were given, so with them install those from the default index first, then ours: +Packages that come from [micropython-lib](https://github.com/micropython/micropython-lib) (`aioble`, `sdcard`, `bmi270`, `usb-device-*`) aren't copied here. `mip` takes every file of a package from the one index it was given, so install those from the default index first, then ours: ```python mip.install("aioble") diff --git a/docs/how-it-works/index.md b/docs/how-it-works/index.md index bc3bf6c..9a74843 100644 --- a/docs/how-it-works/index.md +++ b/docs/how-it-works/index.md @@ -8,7 +8,7 @@ A release in any of the source repositories sends this repository a request nami ## pins.json, then a build -[`publish.yml`](../../.github/workflows/publish.yml) records each request in [`pins.json`](../../pins.json) as soon as it arrives. It then rebuilds every package from the releases pinned there with pydevices' `tools/libbuild.py`, and deploys the result to GitHub Pages. Before it deploys, the build checks that no package name here is also one of micropython-lib's, because `omip` looks here first and a match would hide theirs. If micropython-lib's index can't be read, the build warns and carries on. +[`publish.yml`](../../.github/workflows/publish.yml) records each request in [`pins.json`](../../pins.json) as soon as it arrives. It then rebuilds every package from the releases pinned there with pydevices' `tools/libbuild.py`, and deploys the result to GitHub Pages. When a package moves to a new repository, its entry in `pins.json` names the new one and lists the old under `formerly`. A pydevices release made before the move still names the old repository in its `lib-lock.json`; the build takes the package from the new one at the pinned tag. diff --git a/docs/reference/packages.md b/docs/reference/packages.md index 82cde8b..4dc4906 100644 --- a/docs/reference/packages.md +++ b/docs/reference/packages.md @@ -12,4 +12,4 @@ The packages come from these repositories: A board config's package is named after its folder in [board_configs](https://github.com/PyDevices/pydevices/tree/main/board_configs), and a driver's after its folder in [drivers](https://github.com/PyDevices/pydevices/tree/main/drivers). -Packages from micropython-lib aren't here, and no package here has a name micropython-lib uses: the build fails if one does, so neither can hide the other. `omip` brings them; with `mip`, [install those first](../get-started/install.md#packages-from-micropython-lib) from the default index. +Packages from micropython-lib aren't here; [install those first](../get-started/install.md#packages-from-micropython-lib) from the default index.