Skip to content

Enable CI, disable CD - #4974

Merged
yiyi-wang-rv merged 2 commits into
developfrom
enable-ci-disable-cd
Nov 5, 2026
Merged

yiyi-wang-rv merged 2 commits into
developfrom
enable-ci-disable-cd

Conversation

@yiyi-wang-rv

@yiyi-wang-rv yiyi-wang-rv commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Why

CI on this repository has to come back without any secrets, because our infrastructure
was compromised through leaked secrets. CD goes away completely. This follows
runtimeverification/evm-semantics#2878 and its podman follow-ups #2880 and #2881.

The last run that reached the self-hosted runners (#4938, Oct 26) failed in every
self-hosted job:

  • K: Ubuntu Jammy (Frontend) Package: mkdir: cannot create directory 'debian': Permission denied.
    K: Source Build & Test: Unable to create parent directory for the index file: /opt/workspace/target/spotless-index.
    The runners now use rootless Podman, so the container user cannot write to the
    bind-mounted checkout.
  • K: Nix Build & Test (Linux and MacM1): Cachix Auth token "CACHIX_PUBLIC_TOKEN" has been revoked.

What changes

Removed CD and every secret

  • develop.yml (version bump, push to master), master-pr.yml (retargets PRs to
    develop with JENKINS_GITHUB_PAT), master-push.yml (GitHub prerelease),
    release.yml (Cachix, Docker Hub, Maven, PyPI, gh-pages, dependents dispatch) and
    update-deps.yml (flake/submodule sync), plus the check-cachix-pin.sh helper.
    No release automation is left in the repository.
  • K: Nix Build & Test runs both of its self-hosted runners on their own Nix. It no
    longer pushes to Cachix, and it drops the install-nix-action and cachix-action
    steps, as the evm-semantics Nix job does. install-nix-action already did nothing
    on the runners, which have Nix installed. cachix-action would fail anyway, because
    the runners still hold the now-revoked token from earlier runs.
  • with-k-docker downloads the LLVM and Haskell backend .debs from their public
    release URLs with curl instead of gh release download with github.token.

Hardening

  • permissions: contents: read on both workflows, and persist-credentials: false on every
    checkout, including the one inside test-package.
  • Every action is pinned to the commit SHA of its latest release: checkout v7.0.1,
    setup-python v7.1.0, setup-uv v10.3.3, setup-java v6.1.0, upload-artifact v7.0.2,
    download-artifact v8.0.2, install-nix-action v31.11.3, cachix-action v17,
    zizmor-action v0.8.1.
  • No ${{ }} expressions are left in run: blocks. The composite actions take their
    inputs through env:, and the pyk jobs use ${GITHUB_SHA}.
  • run-actionlint.yml ran an unpinned install script piped from main. It is replaced
    by lint-workflows.yml, adapted from evm-semantics, which runs on every PR:
    • actionlint: pinned to 1.7.12, with the download's checksum verified.

    • zizmor: unlike evm-semantics, it reports findings as annotations and fails the
      check
      , instead of uploading SARIF to code scanning. Uploading SARIF needs
      security-events: write and never fails the check.

      .github/zizmor.yml disables one audit, self-repository. That audit asks for
      the new uses: $/... syntax, which actionlint 1.7.12 rejects.

    • The actionlint job keeps the name Run actionlint, which develop's branch
      protection requires. zizmor is not a required check yet.

Self-hosted jobs on Podman

  • with-docker copies the sources into the container with docker cp instead of
    bind-mounting them, then chowns them to the image user. test-package copies the
    built package back out.

    The test container still bind-mounts the workspace. It runs as root, which Podman
    maps to the runner user, so it can write there.

  • The container runs as the image's own github-user (uid 1000, the Dockerfiles' default),
    in a WORKDIR that the image now creates. The runner's user and UID are no longer
    passed in, and no longer have to match.

  • Each container is force-removed before it starts and torn down with docker rm -f || true.
    Otherwise a container left over from a canceled run of the same commit blocks the
    next run (evm-semantics#2881). The two package jobs now include the job id in their
    container names, so they cannot remove each other's containers.

No more Docker Hub pulls

CI pulled from Docker Hub without logging in: ubuntu:<distro> in four Dockerfiles and
in the package test container, plus runtimeverificationinc/z3. Docker Hub rate-limits
anonymous pulls per IP, and the runners share their IPs.

  • Ubuntu base images now come from public.ecr.aws/docker/library/ubuntu, AWS's
    mirror of the Docker official images. No login is needed. A private ECR registry
    would need AWS credentials, so it was ruled out.
  • Z3 is installed from the upstream release (z3-<ver>-x64-glibc-2.35.zip), with its
    SHA-256 pinned in .github/workflows/Dockerfile, instead of the
    runtimeverificationinc/z3 image. The binary needs only libc, libm, libstdc++ and libgcc_s.

Smaller changes

  • The base image is passed as BASE_IMAGE instead of being built from BASE_OS/BASE_DISTRO.
    BASE_DISTRO is still used for the local stack:/maven: image tags. Stage names are
    lowercase, and the repeated ARGs are gone.
  • type: keys are removed from composite-action inputs, since composite actions do not
    support them. Boolean defaults are now the strings 'true', which is what they were
    already compared against.
  • actionlint.yaml lists only the runner labels still in use: normal and MacM1.

Tested

CI on this PR passed: all 18 Test PR jobs, Run actionlint and zizmor. That includes
the self-hosted package, source-build, Nix and pyk jobs that failed on Oct 26. The logs
confirm that:

  • Ubuntu images were pulled from public.ecr.aws and nothing was pulled from Docker Hub.
  • The Z3 checksum verified in all three image builds.
  • No job used a secret.

Notes for reviewers

  • The develop → master flow stops. With develop.yml and master-pr.yml gone,
    merging to develop no longer bumps the version or updates master. PRs opened
    against master, the default branch, are not retargeted, and get only Lint Workflows,
    not Test PR.
  • Bumping deps/z3 now needs a manual checksum update. Update Z3_SHA256 in
    .github/workflows/Dockerfile, and Z3_ZIP if the asset name changed, or the image
    build fails at the checksum check.
  • The only token CI still uses is the automatic, per-run GITHUB_TOKEN, read-only.
  • PR code runs on persistent self-hosted runners. zizmor's auditor persona flags all
    8 of those jobs; that is by design. There are no secrets on them any more, but an
    approved PR can still leave state behind on a runner, for example in the shared
    stack:/maven: images or in the Nix store. Keep "require approval" for outside
    contributors on.
  • Required checks: every required check on develop keeps its name. Consider adding
    zizmor once this is merged.
  • This makes Pin secret-bearing third-party actions and the kup flake input #4944 unnecessary, since it only pins actions in release.yml.

Follow-ups

  • Decide what replaces release/CD: version bump, tags, GitHub releases, Docker images,
    Cachix, PyPI, Maven and gh-pages. package/docker/Dockerfile.ubuntu-*, which only
    release.yml used, still pull runtimeverificationinc/z3 from Docker Hub.
  • Dependency-bump automation (update-deps.yml) is gone. Flake/submodule syncs for
    _update-deps/* branches are manual for now.

🤖 Generated with Claude Code

yiyi-wang-rv and others added 2 commits November 4, 2026 09:00
Remove every workflow that publishes or pushes (version bump on develop,
PR retargeting, prerelease on master, release, dependency sync) and every
use of secrets, so CI can run safely again. Pin all actions to commit
SHAs, make the PR token read-only, and pass composite-action inputs via
env instead of interpolating them into shell.

Fix the self-hosted jobs on the rootless Podman runners: copy sources into
the container instead of bind-mounting them, pre-remove stale containers,
and tear down with `docker rm -f`. Run the Nix job on the runners' own Nix
without cachix-action, which would read the revoked token stored on the
runner. Stop pulling from Docker Hub: use the ECR Public mirror for Ubuntu
base images and the checksummed upstream Z3 release instead of
runtimeverificationinc/z3. Download the backend .debs from public release
URLs without a token.

Replace run-actionlint.yml with lint-workflows.yml: pinned actionlint, and
zizmor failing the check on findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Branch protection on develop requires a check named 'Run actionlint',
which came from the removed run-actionlint.yml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yiyi-wang-rv
yiyi-wang-rv requested a review from a team as a code owner November 4, 2026 01:30

@palinatolmach palinatolmach left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thanks Yiyi!

@yiyi-wang-rv
yiyi-wang-rv merged commit 9a7bcd9 into develop Nov 5, 2026
38 checks passed
@yiyi-wang-rv
yiyi-wang-rv deleted the enable-ci-disable-cd branch November 5, 2026 00:26
palinatolmach pushed a commit to runtimeverification/mir-semantics that referenced this pull request Nov 12, 2026
### Why

CI on this repository should run without any repository secrets, and CD
is removed. This
does for mir-semantics what runtimeverification/k#4974,
runtimeverification/llvm-backend#1223
and runtimeverification/haskell-backend#4166 did.

It supersedes #1094 and keeps its commits. #1094 removed CD and every
secret, but its
self-hosted jobs failed in `Set up Docker` with exit code 125. The
commit on top fixes that
and adds the same hardening as the other repositories.

### What changes

**From #1094: CD and every secret removed**
- `release.yml`, `p-token-release.yml` and `master-push.yml`: the
release pipeline (version bump,
  GitHub releases, Docker Hub images, Cachix).
- `update-dependencies.yml`: dependency-bump automation.
- `test-cachix-pin.yml` and `.github/scripts/check-cachix-pin.sh`.
- `JENKINS_GITHUB_PAT` removed from the checkouts. The only submodule,
`stable-mir-json`, is public.
- `cachix-action` removed from the `Nix Tests` job.
- Actions bumped to their latest releases.

**Self-hosted jobs on rootless Podman**
- **The CI image's base is now fully qualified,** as
`docker.io/runtimeverificationinc/kframework-k`.
Rootless Podman refuses the short name when no unqualified-search
registry is configured, and
  that was the exit code 125 in #1094.
- **`with-docker` no longer overrides `USER`.** `USER` is exported, and
rootless Podman looks up the
runner user's subordinate UIDs by `$USER`. With `USER=github-user` it
found none and fell back to
a single mapping, where only root exists in the container. The image
build then failed at its
  first `apt-get`. The container user is now `CONTAINER_USER`.
- **The checkout is copied with `docker cp ./.`** instead of `docker cp
.`. On the runners' Podman,
a bare `.` landed in a subdirectory, and `make` found no Makefile. K and
llvm-backend use `./.`.
- **Each job and matrix entry gets its own container name.** All six
`Integration Tests` entries
used `mir-semantics-ci-<sha>`, so two of them on one runner removed each
other's container.
- **Containers are cleaned up reliably.** Each container is
force-removed before it starts and torn
down with `docker rm -f || true`. A container left over from a canceled
run of the same commit
  no longer blocks the next run.
- **`Nix Tests` no longer runs `install-nix-action`** on the self-hosted
runners, which already have
Nix. On those runners it only printed `Aborting: Nix is already
installed`.

**Hardening**
- `permissions: contents: read` on every workflow, and
`persist-credentials: false` on every
  checkout.
- In `test-stable-mir-ui.yml` this also covers the rust-lang/rust
checkout. Before, it was copied
    into the container with the token still in its git config.
- Every action is pinned to the commit SHA of its latest release:
checkout v7.0.1, setup-uv v10.3.3,
install-nix-action v31.11.3, setup-buildx-action v4.4.1,
build-push-action v7.4.0,
  upload-artifact v7.0.2, zizmor-action v0.8.1.
- No `${{ }}` expressions are left in `run:` blocks. They go through
`env:` instead.
- New `lint-workflows.yml` and `.github/zizmor.yml`, identical to K's:
- **actionlint**: pinned to 1.7.12, with the download's checksum
verified.
  - **zizmor**: reports findings as annotations and fails the check.

  Neither is a required check yet.

**Fixes**
- **`smir-parse-tests` ran no tests in CI.** It looked for the tests
under `$(PWD)`, which
`docker exec` does not set, so `Integration with stable-mir-json` passed
without parsing anything.
It now uses `$(CURDIR)`, and fails if it finds no tests. Its 35 tests
pass.
- `Test Kmir Image` appended its summary to a file literally named
`GITHUB_STEP_SUMMARY`. Its job
  outputs are gone too, since only the release pipeline used them.
- `test-stable-mir-ui.yml` copied the rust-lang/rust checkout into the
container twice: once with
the workspace, and again into `rust/rust`. Runs of the same commit,
which share a container name,
  no longer overlap.
- `nix/README.md` no longer says CI uploads to the binary cache.

### Tested

CI on this PR passed: all 14 checks, including all 8 self-hosted jobs.
The Docker jobs ran on
github-runner-01, -02 and -03.
- The integration test suites all pass:
  - `LLVM Concrete Tests`: 60 passed
  - `Haskell Exec SMIR`: 40 passed
  - `Haskell Termination`: 20 passed
  - `Haskell Proofs`: 106 passed
  - `Verify Rust Std`: 16 passed
  - `Remainder`: 129 passed, 7 skipped
- `Unit Tests`: 40 passed.
- `Integration with stable-mir-json` parses all 35 `smir-parse-tests`
programs, instead of none.
- `Nix Tests (normal)` and `Nix Tests (MacM1)` pass on the runners' own
Nix, with no Cachix step.
- Every job's `GITHUB_TOKEN` was read-only (`Contents: read`). No job
used a secret, and no token
  value appears in any log.

### Notes for reviewers

- **Releases stop.** Merging to `master` no longer bumps the version,
tags, publishes a GitHub
release, pushes `runtimeverificationinc/kmir` to Docker Hub, or pushes
to Cachix.
  `package/version.sh` is kept for manual use.
- **The CI images are frozen at K 7.1.337.**
`.github/workflows/Dockerfile` and `Dockerfile.kmir`
build `FROM
runtimeverificationinc/kframework-k:ubuntu-jammy-<deps/k_release>`. Only
K's release
pipeline published that image, and that pipeline is gone, so bumping
`deps/k_release` past 7.1.337
will break `Set up Docker` and `Test Kmir Image` until the image is
published some other way.
- **`Nix Tests (normal)` relies on the runner's own Nix configuration**
for the `k-framework` cache.
The workflow cannot add it: the runner user is not a trusted Nix user,
so `cachix use` was already
  refused there. If the job times out, the runner admins can add
`extra-substituters = https://k-framework.cachix.org` and its public key
to `/etc/nix/nix.conf`.
  It is a public cache, so no token is needed.
- **The only token CI still uses is the automatic, per-run
`GITHUB_TOKEN`**, read-only.
- **PR code runs on persistent self-hosted runners.** zizmor's auditor
persona flags those jobs; that
  is by design. Keep "require approval" for outside contributors on.
- **Required checks:** all 12 required checks on `master` keep their
names.

### Follow-ups

- Decide what replaces release/CD: version bump, tags, GitHub releases,
Docker images, Cachix.
- Publish `kframework-k` images somewhere other than K's removed release
pipeline, or build the CI
  image from a public base plus the K release `.deb`.
- Dependency bumps (K, stable-mir-json, uv, rust toolchain) are manual
for now.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Everett Hildenbrandt <everett.hildenbrandt@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants