Skip to content

fix: honor RQJob dependencies - #3

Merged
paulmueller merged 1 commit into
DCOR-dev:masterfrom
paulmueller-agent:fix-rqjob-depends-on
Nov 5, 2026
Merged

paulmueller merged 1 commit into
DCOR-dev:masterfrom
paulmueller-agent:fix-rqjob-depends-on

Conversation

@paulmueller-agent

Copy link
Copy Markdown

RQJob dependencies (depends_on) have never had an effect:

  1. The dependency job IDs were built as f"{jid}_{dep}", but jid already ends with _, so they pointed to <pkg>_<pos>__<job> instead of <pkg>_<pos>_<job>.
  2. enqueue_all_jobs enqueued jobs in registration order. ckanext-dcor_schemas registers job_set_s3_resource_metadata after the jobs that depend on it, so the dependency did not exist yet.

RQ silently drops dependencies it cannot find (Job.fetch_dependencies filters out missing jobs), so all jobs were started right away. This PR fixes the IDs and enqueues jobs via get_all_job_methods_in_order.

The new test enqueues three jobs through CKAN's Redis connection and checks that RQ registers the dependencies and defers the dependent jobs.

Behavior change on deployment: dcor_schemas jobs will actually wait for their dependencies, e.g. job_set_sha256 (queue dcor-long) waits for job_set_s3_resource_metadata. If a dependency fails, RQ does not run the jobs that depend on it (they stay deferred), whereas until now they ran anyway.

🤖 Generated with Claude Code

The job IDs in `depends_on` contained a superfluous underscore and
jobs were enqueued in registration order, so RQ ignored all
dependencies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@paulmueller-agent
paulmueller-agent force-pushed the fix-rqjob-depends-on branch 2 times, most recently from b109a7e to e50f0e6 Compare November 5, 2026 10:34
@codecov

codecov Bot commented Nov 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 96.99%. Comparing base (b2048f1) to head (e50f0e6).

Additional details and impacted files
@@            Coverage Diff             @@
##           master       #3      +/-   ##
==========================================
- Coverage   97.00%   96.99%   -0.01%     
==========================================
  Files          12       12              
  Lines         567      566       -1     
==========================================
- Hits          550      549       -1     
  Misses         17       17              
Flag Coverage Δ
develop 96.99% <100.00%> (-0.01%) ⬇️
pypi 96.99% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paulmueller
paulmueller merged commit b5b40bd into DCOR-dev:master Nov 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants