feat: migrate credly badge poll to one sync unit per template by ulemons · Pull Request #5206 · linuxfoundation/crowd.dev

ulemons · 2026-11-04T17:18:33Z

Second PR of the Credly per-template sync units migration (linuxfoundation/lfx-cdp-insights-ops#1005, part 2 of the plan after #5203). Collapses runBadgePoll from one org-wide unit fanning out at emit time into one sync unit per (project integration, badge template) — the actual fix for the serial-backfill bottleneck.

What changes

runBadgePoll is now scoped to a single template (ctx.channel.channelId), fetched with filter[badge_template_id] and a new contract assertion (apiClient.ts) that every returned badge actually belongs to it, per the framework's "assert every returned record belongs to its channel" rule. The routing map, per-template locking and the release-queue loop inside the sync are gone — a unit only ever has one target.
CredlyDeps drops loadTemplateRouting/lockTemplate/the release-queue methods, gains segmentId — threaded through a new DepsFactoryInput.segmentId field, populated from the resolveRunSegmentId call every run already makes (syncRunActivities.ts/completenessActivities.ts). No behavior change for other connectors.
onboardCredlyTemplate (new, modeled on onboardGithubRepo): onboards one sync unit per newly-mapped template from credlyTemplateResolve.ts's applyDecisions, arming watermark.backfill from the template's creation date only on a genuine first creation (never resets an existing unit's progress). hasActiveSyncUnitsForChannelElsewhere gets a platform parameter so Credly can reuse it instead of copying it, per the README.
credlyTemplateMappings keeps its exact role as the resolver's own decision record (mapped/held/why) — it stops being read for per-badge routing at run time, and is now purely what creates sync units, matching the framework rule that a held queue may live outside the emit path as long as its only output is units created.
Removes what the new architecture leaves dead: seed-credly-sync-unit.ts (its channel was the organization id — no longer a valid template filter, would create a broken unit if run) and the entire credlyTemplateReleases queue + DAL + its --release-unreleased CLI mode (nothing drains it anymore). --release <id> is rewritten to arm a template's own live unit directly via mergeSyncUnitWatermark instead of queuing to a table nothing reads.

Caught in review and fixed before this PR: a dropped lock (lockCredlyTemplateMapping is now called right after a unit goes live, restoring the "freeze a mapping once it's syncing" invariant the old lockBeforeEmit provided) and a watermark gap (a freshly onboarded unit's live-poll cursor now starts exactly where its backfill's top ends, instead of defaulting to "now - 1 day" and potentially missing badges in between).

Not fixed, tracked separately: the credlyTemplateReleases table itself is left in Postgres, unused — dropping it is a deliberate migration for a later PR, not bundled here. The admission-time budget race flagged on #5203 is unchanged by this PR; it becomes relevant once real concurrency exists (next PR), not before.

Why

Investigated with Mouad why Credly can't parallelize its backfill like GitHub. GitHub's parallelism comes from one independent sync unit per repo x sync; Credly's API supports filter[badge_template_id], so it can follow the same pattern. This is also required by the connector framework rules (services/libs/connectors/README.md, #5192): a mapping table driving runtime routing duplicates what sync_units already does, and fan-out is only for providers that can't filter by channel.

Deployment note

The existing org-wide badgePoll unit's channelId is the organization id, not a template — once this code deploys, that unit would send filter[badge_template_id]=<organization id> to Credly. It needs to be paused before or at deploy time; the next PR (one-time migration: seed per-template units for all currently mapped templates, retire this unit) handles the cutover.

Tests

onboardCredlyTemplate.test.ts (new): create path, owned-by-another-integration skip, already-present rename.
badgePoll.test.ts: rewritten for the single-template scope (window/backfill/replacement-chain math preserved, routing/locking/release-loop tests removed as the behavior no longer exists), plus a new case for a replacement badge belonging to another template.
apiClient.test.ts: new cases for the per-template contract assertion.
credlyTemplateResolve.test.ts: new applyDecisions coverage (none existed before) for onboarding, locking, and the watermark it seeds.
Whole connectors + connectors_worker suite: 746 tests pass; tsc -b clean; pnpm lint --deny-warnings clean. DB-backed integration tests (syncUnits.test.ts, credly/badges.test.ts, credly/index.test.ts) need a running Postgres, not available in this environment — unaffected by this diff, same pre-existing dependency as before.

Refs linuxfoundation/lfx-cdp-insights-ops#1005

🤖 Generated with Claude Code

cursor · 2026-11-04T17:18:44Z

PR Summary

Medium Risk
Changes Credly ingestion architecture (sync unit shape, onboarding, watermark/backfill, and mapping locks) and removes the release queue drain path; incorrect cutover could mis-filter or duplicate sync work until the org-wide unit is retired.

Overview
This PR replaces the org-wide Credly badgePoll unit (one channel, runtime template routing, release queue) with one sync unit per mapped badge template, aligned with GitHub-style parallelism and the connector framework.

runBadgePoll now scopes to ctx.channel.channelId, always calls Credly with filter[badge_template_id], and emits to the unit’s integration/segment without per-emit overrides. Runtime routing maps, template locking inside the sync, and the release-queue loop are removed; CredlyDeps is simplified to segmentId plus badge state helpers, with segmentId passed through DepsFactoryInput from existing run resolution in sync/completeness activities.

Template resolution (applyDecisions) now onboards a badgePoll sync unit (new onboardCredlyTemplate, modeled on GitHub repo onboarding) with a backfill watermark from the template creation date, then locks the mapping once the unit exists. The old releasesToQueue / queueCredlyTemplateRelease path is dropped; dry-run CLI flags are updated for single-template runs.

Safety/contract: fetchBadgesPage asserts every returned badge matches the requested template when filtered. Cross-template replacement badges are skipped in the per-template unit. hasActiveSyncUnitsForChannelElsewhere takes an explicit platform so Credly can reuse channel ownership checks. Credly emitsOwnSegment is removed because segment is fixed per unit.

Deploy note: the legacy org-wide unit must be paused before deploy (its channel is not a template id).

^{Reviewed by Cursor Bugbot for commit f965b2b. Bugbot is set up for automated code reviews on this repo. Configure here.}

Copilot

🟡 Changes recommended

Non-atomic onboarding and unsafe cursor seeding can leave mappings inconsistent or permanently miss badges, while expiry processing and removed-template cleanup are also missing.

4 open issues

What changed in this PR

Moves Credly badge polling to one sync unit per project integration and badge template, aligning ingestion with the connector framework’s channel-based routing model and preparing independent backfills.

Changes:

Scopes badge polling to a template and validates returned badges against that scope.
Adds template onboarding with backfill watermarks, mapping freezes, and segment-aware dependencies.
Removes the release queue and org-wide seed command, updates operational tools, and adds budget admission support.

File	Description
`services/libs/data-access-layer/src/credly/releases.ts`	Removes release-queue data access.
`services/libs/data-access-layer/src/credly/releases.test.ts`	Removes obsolete release-queue tests.
`services/libs/data-access-layer/src/connectors/syncUnits.ts`	Adds channel-ID lookup and platform-scoped ownership checks.
`services/libs/data-access-layer/src/connectors/syncUnits.test.ts`	Tests lookup and updated ownership checks.
`services/libs/connectors/src/types.ts`	Adds dependency segment context and run-cost estimates.
`services/libs/connectors/src/pool/tokenPool.ts`	Exports the probe staleness threshold.
`services/libs/connectors/src/connectors/octolens/deps.test.ts`	Updates fixtures for segment context.
`services/libs/connectors/src/connectors/credly/syncs/badgePoll.ts`	Replaces org-wide routing with template-scoped polling.
`services/libs/connectors/src/connectors/credly/syncs/badgePoll.test.ts`	Adapts polling tests to single-template units.
`services/libs/connectors/src/connectors/credly/index.ts`	Registers budget support and removes fan-out declaration.
`services/libs/connectors/src/connectors/credly/discover.test.ts`	Verifies updated manifest capabilities.
`services/libs/connectors/src/connectors/credly/deps.ts`	Replaces routing and release dependencies with segment context.
`services/libs/connectors/src/connectors/credly/deps.test.ts`	Tests segment requirements and remaining data access.
`services/libs/connectors/src/connectors/credly/budget.ts`	Defines the fixed-window Credly budget probe.
`services/libs/connectors/src/connectors/credly/budget.test.ts`	Tests budget size and refresh timing.
`services/libs/connectors/src/connectors/credly/apiClient.ts`	Checks that filtered badges belong to the requested template.
`services/libs/connectors/src/connectors/credly/apiClient.test.ts`	Covers template-filter contract validation.
`services/apps/connectors_worker/src/onboardGithubRepo.ts`	Passes GitHub’s platform to ownership checks.
`services/apps/connectors_worker/src/onboardGithubRepo.test.ts`	Verifies the platform argument.
`services/apps/connectors_worker/src/onboardCredlyTemplate.ts`	Creates template polling units with ownership protection.
`services/apps/connectors_worker/src/onboardCredlyTemplate.test.ts`	Covers creation, ownership conflicts, and existing units.
`services/apps/connectors_worker/src/credlyTemplateResolve.ts`	Onboards mapped templates and seeds their cursors.
`services/apps/connectors_worker/src/credlyTemplateResolve.test.ts`	Adds decision-application and onboarding coverage.
`services/apps/connectors_worker/src/bin/seed-credly-sync-unit.ts`	Removes the org-wide seed command.
`services/apps/connectors_worker/src/bin/resolve-credly-templates.ts`	Arms backfills directly on template units.
`services/apps/connectors_worker/src/bin/dry-run-credly-badge-poll.ts`	Requires a template and destination for dry runs.
`services/apps/connectors_worker/src/activities/syncRunActivities.ts`	Passes the resolved segment into dependencies.
`services/apps/connectors_worker/src/activities/dispatcherActivities.ts`	Uses connector-specific admission cost estimates.
`services/apps/connectors_worker/src/activities/completenessActivities.ts`	Supplies segment context during validation.
`services/apps/connectors_worker/package.json`	Removes the obsolete seed script entry.
`docs/knowledge/sys-connectors.md`	Updates documented Credly table responsibilities.

_{Review effort: Balanced}

-      }
-      return stored
-    })
+    const outcome = await upsertCredlyTemplateDecision(qx, { ...decision, integrationId })


+      if (onboarded.outcome === 'created' || onboarded.reason === 'already_present') {
+        await lockCredlyTemplateMapping(qx, decision.badgeTemplateId, { segmentId, integrationId })


+        platform: PlatformType.CREDLY,
+        channelId: badgeTemplateId,
+        channelName: badgeTemplateName,
+        syncName: 'badgePoll',


cursor

Stale Bugbot comment from a previous run.

Copilot

🟡 Changes recommended

Non-atomic decision updates, onboarding, and mapping freezing allow concurrent resolvers to leave a live unit syncing one segment while its unfrozen decision points to another.

6 open issues

_{Review effort: Balanced}

Copilot

🔵 Needs a closer look

The mapping decision, unit creation, and lock commit separately, allowing concurrent runs or crashes to leave a live syncing unit with an unlocked mapping pointing to a different integration.

5 open issues

1 resolved since last review

Seed cursors at the visibility-lag boundary

_{Review effort: Balanced}

Credly's single org-wide badgePoll unit, fanned out to segments at emit time via a routing table, serialized its entire backfill behind the shared dispatcher queue — the same bottleneck PR #5189 worked around instead of fixing. Credly's API filters by badge_template_id, so it can follow GitHub's pattern instead: one sync unit per (project integration, badge template), the channel itself the mapping. - runBadgePoll is scoped to one template (ctx.channel.channelId), fetched with the template filter and a contract assertion that every returned badge actually belongs to it (apiClient.ts). No more routing map, locking, or release-queue loop inside the sync. - CredlyDeps drops loadTemplateRouting/lockTemplate/the release-queue methods, gains segmentId — threaded through a new DepsFactoryInput field populated from the same resolveRunSegmentId call every run already makes. - onboardCredlyTemplate (new, modeled on onboardGithubRepo) onboards one sync unit per newly-mapped template from credlyTemplateResolve.ts, arming watermark.backfill from the template's creation date only on a genuine first creation. hasActiveSyncUnitsForChannelElsewhere gets a platform parameter so Credly can reuse it instead of copying it. - The template-mapping decision stays in credlyTemplateMappings exactly as before (the resolver's own record of why a template is mapped or held); it only stops being consulted for per-badge routing at run time, and is still the only thing creating or decommissioning units. - Restores the lock credlyTemplateMappings relied on to freeze a mapping once its unit is live, which the new onboarding path had dropped — found by review, otherwise a later re-decision could silently repoint the mapping while badges kept landing on the old, still-live unit. - Removes what the new architecture leaves with no real caller: the single org-wide seed-credly-sync-unit.ts script (its channel was the organization id, no longer a valid template filter) and the entire credlyTemplateReleases queue plus its CLI modes. --release is rewritten to arm a template's own live unit directly instead of queuing to a table nothing drains anymore. Refs linuxfoundation/lfx-cdp-insights-ops#1005 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

The dead-code removal (seed-credly-sync-unit.ts, the credlyTemplateReleases queue and its CLI modes) moves to its own PR, stacked on this one, so the per-template rewrite can be reviewed and tested on its own. Refs linuxfoundation/lfx-cdp-insights-ops#1005 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

A fresh template's backfill top and live cursor were seeded at raw now, skipping the WINDOW_LAG_MS margin every other window in badgePoll.ts respects. A badge Credly stamps just before onboarding but doesn't surface until shortly after fell before the new cursor and was never revisited. Refs linuxfoundation/lfx-cdp-insights-ops#1005 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

A template mapped before per-template sync units existed was already fully covered by the retired org-wide poller, so this one-time script seeds its unit live-only (since: now - WINDOW_LAG_MS, no backfill) instead of re-walking years of history Credly has already given us once. The day-to-day resolver keeps using a full backfill for a template mapped for the first time, correctly, since it has no prior coverage to assume. Idempotent and dry-run by default: skips any template that already has a unit (including the 14 onboarded manually while testing #5206/#5207/#5211 today), and only --apply writes anything. Retiring the old org-wide unit stays a deliberate manual step, since its timing relative to the deploy matters and a script can't safely infer it. Also fixes findSyncUnitByChannelId to take a platform parameter, same as findSyncUnitByChannelName and the earlier hasActiveSyncUnitsForChannelElsewhere widening: this script is its second real caller, and without it two different platforms could collide on the same channelId under one integration. Restores the stale-mapping warning (integration deleted or moved) that the release-queue removal silently dropped. Refs linuxfoundation/lfx-cdp-insights-ops#1005 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9bc1cb3. Configure here.}

A cross-template replacement is not malformed data, just out of scope for this channel's own unit. Throwing aborted the whole page before it was saved, and since the mismatch is deterministic every retry hit the same wall. Skip it like the existing not-found branch instead. Refs linuxfoundation/lfx-cdp-insights-ops#1005 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

A template mapped before per-template sync units existed was already fully covered by the retired org-wide poller, so this one-time script seeds its unit live-only (since: now - WINDOW_LAG_MS, no backfill) instead of re-walking years of history Credly has already given us once. The day-to-day resolver keeps using a full backfill for a template mapped for the first time, correctly, since it has no prior coverage to assume. Idempotent and dry-run by default: skips any template that already has a unit (including the 14 onboarded manually while testing #5206/#5207/#5211 today), and only --apply writes anything. Retiring the old org-wide unit stays a deliberate manual step, since its timing relative to the deploy matters and a script can't safely infer it. Also fixes findSyncUnitByChannelId to take a platform parameter, same as findSyncUnitByChannelName and the earlier hasActiveSyncUnitsForChannelElsewhere widening: this script is its second real caller, and without it two different platforms could collide on the same channelId under one integration. Restores the stale-mapping warning (integration deleted or moved) that the release-queue removal silently dropped. Refs linuxfoundation/lfx-cdp-insights-ops#1005 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

Nothing in the codebase reads or writes this table anymore: the DAL (releases.ts) and its CLI modes (--release-unreleased, the drain loop) were removed in #5207 once badgePoll.ts moved to one sync unit per template, each backfilling itself. Do not merge/deploy this before the rest of the stack (#5206, #5207, #5211, #5212) is live in production: prod still runs the old code that reads this table until then, and dropping it first would break it. Refs linuxfoundation/lfx-cdp-insights-ops#1005 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

Copilot

🟡 Changes recommended

Non-atomic decision updates, unit creation, and mapping freezes can leave live units emitting under incorrect mappings, while skipped cross-template renewals can be permanently lost.

4 open issues

2 resolved since last review

_{Review effort: Balanced}

+              { badgeId, replacementTemplateId: replacement.badge_template.id, badgeTemplateId },
+              'credly replacement badge belongs to a different template, skipping it here',
+            )
+            continue


A template mapped before per-template sync units existed was already fully covered by the retired org-wide poller, so this one-time script seeds its unit live-only (since: now - WINDOW_LAG_MS, no backfill) instead of re-walking years of history Credly has already given us once. The day-to-day resolver keeps using a full backfill for a template mapped for the first time, correctly, since it has no prior coverage to assume. Idempotent and dry-run by default: skips any template that already has a unit (including the 14 onboarded manually while testing #5206/#5207/#5211 today), and only --apply writes anything. Retiring the old org-wide unit stays a deliberate manual step, since its timing relative to the deploy matters and a script can't safely infer it. Also fixes findSyncUnitByChannelId to take a platform parameter, same as findSyncUnitByChannelName and the earlier hasActiveSyncUnitsForChannelElsewhere widening: this script is its second real caller, and without it two different platforms could collide on the same channelId under one integration. Restores the stale-mapping warning (integration deleted or moved) that the release-queue removal silently dropped. Refs linuxfoundation/lfx-cdp-insights-ops#1005 Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>

ulemons self-assigned this Nov 4, 2026

Copilot AI balanced review requested due to automatic review settings November 4, 2026 17:18

Copilot started reviewing on behalf of ulemons November 4, 2026 17:19

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings November 4, 2026 17:27

Copilot started reviewing on behalf of ulemons November 4, 2026 17:28

View session

ulemons mentioned this pull request Nov 4, 2026

chore: retire the credly release-queue mechanism #5207

Merged

cursor Bot reviewed Nov 4, 2026

View reviewed changes

Comment thread services/apps/connectors_worker/src/bin/resolve-credly-templates.ts Resolved

Copilot AI reviewed Nov 4, 2026

View reviewed changes

Comment thread services/libs/connectors/src/connectors/credly/syncs/badgePoll.ts Resolved

Comment thread services/libs/connectors/src/connectors/credly/syncs/badgePoll.ts Resolved

ulemons force-pushed the feat/issue-1005-credly-per-template-sync-units branch from 6cabafd to 65e7324 Compare November 4, 2026 18:01

Copilot AI balanced review requested due to automatic review settings November 4, 2026 20:24

Copilot started reviewing on behalf of ulemons November 4, 2026 20:24

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

This was referenced Nov 4, 2026

fix: make credly decision/onboarding/lock and decommissioning atomic #5211

Merged

feat: migrate already-mapped credly templates to per-template units #5212

Merged

ulemons added 3 commits November 5, 2026 09:56

ulemons force-pushed the feat/issue-1005-credly-per-template-sync-units branch from f87a0b4 to 9bc1cb3 Compare November 5, 2026 08:56

cursor Bot reviewed Nov 5, 2026

View reviewed changes

Comment thread services/libs/connectors/src/connectors/credly/syncs/badgePoll.ts Resolved

ulemons mentioned this pull request Nov 5, 2026

chore: drop the unused credlyTemplateReleases table #5217

Draft

1 of 2 tasks

Copilot AI balanced review requested due to automatic review settings November 5, 2026 10:46

Copilot started reviewing on behalf of ulemons November 5, 2026 10:47

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

Comment thread services/libs/connectors/src/connectors/credly/syncs/badgePoll.ts

{ badgeId, replacementTemplateId: replacement.badge_template.id, badgeTemplateId },

'credly replacement badge belongs to a different template, skipping it here',

)

continue

ulemons merged commit 5b836c4 into main Nov 5, 2026
14 checks passed

ulemons deleted the feat/issue-1005-credly-per-template-sync-units branch November 5, 2026 11:27

Navigation Menu

Repository navigation

feat: migrate credly badge poll to one sync unit per template - #5206

feat: migrate credly badge poll to one sync unit per template#5206
ulemons merged 4 commits into
mainfrom
feat/issue-1005-credly-per-template-sync-units

ulemons commented Nov 4, 2026

Uh oh!

cursor Bot commented Nov 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

cursor Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		if (onboarded.outcome === 'created' \|\| onboarded.reason === 'already_present') {
		await lockCredlyTemplateMapping(qx, decision.badgeTemplateId, { segmentId, integrationId })

Conversation

ulemons commented Nov 4, 2026

What changes

Why

Deployment note

Tests

Uh oh!

cursor Bot commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🔵 Needs a closer look

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cursor Bot commented Nov 4, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading