Skip to content

RHINENG-31827: resolve RBAC from Clowder v2 with gateway bearer auth - #2389

Open
SteveHNH wants to merge 6 commits into
RedHatInsights:masterfrom
SteveHNH:clowder-v2-rbac-auth
Open

SteveHNH wants to merge 6 commits into
RedHatInsights:masterfrom
SteveHNH:clowder-v2-rbac-auth

Conversation

@SteveHNH

@SteveHNH SteveHNH commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Moves the manager's RBAC lookup to Clowder v2 and adds east-west gateway bearer auth for when RBAC is reached through a ClowdAppRef. Only RBAC is migrated; vmaas, content-sources and patchman's own private endpoints are unchanged.

 initServicesFromClowder
+  initRbacFromClowder            # v2 rbac/service -> uri, authenticated, ca_certificate
   for endpoint in v1 endpoints
-    case "rbac": prefer v2 uri, else v1 buildURL
     case "vmaas" / "content-sources-backend" / ...   # unchanged
 RBAC() middleware
+  validate RBAC address at startup
+  trust ca_certificate (added to system pool) if set
   per request
+    if rbac authenticated
+      token = getBearerToken()     # shared with Kessel gRPC
+      on error -> 502, RBAC not called
     GET /access  x-rh-identity
+                 Authorization: Bearer <token>   (authenticated only)

Kessel credentials (first commit): one lazily built OAuth2ClientCredentials, token endpoint from OIDC discovery. Previously KESSEL_AUTH_OIDC_ISSUER was passed in as the token endpoint, which can't work with the realm URL; hidden so far because KESSEL_INSECURE=true drops call credentials.

State Result
No v2 rbac endpoint and no RBAC_ADDRESS manager fails at startup naming the dependency
v2, authenticated: false (today: same cluster as RBAC) v2 uri, no token, CA if provided
v2, authenticated: true, Kessel creds set bearer added alongside x-rh-identity
v2, authenticated: true, creds missing 502 on RBAC-checked requests

Evidence

  • Before: RBAC address came from v2 uri or v1 fallback; no bearer support; authenticated endpoint would be called without a token.
    After: new/updated tests pass locally:
ok  app/base/utils           TestInitRbacFromClowder (v2 fields set; nil/empty/wrong-key -> empty)
ok  app/manager/middlewares
  TestRBACAuthenticatedAddsBearer              Authorization: Bearer + x-rh-identity sent
  TestRBACUnauthenticatedNoToken               no token fetched, identity unchanged
  TestRBACAuthenticatedTokenFailureFailsClosed 502, RBAC called 0 times
  TestRBACAndKesselShareCredentials            same credentials object
  TestRbacTLSConfigTrustsCA                    TLS server trusted via ca_certificate
  TestSetupClient / TestSetupClientSecureAuth  discovery used; missing issuer errors

Full DB-backed suite left to CI.

Merge Danger

Door: two-way

Revert restores the v1 RBAC fallback; no schema or data changes.

Blast Radius: manager

  • The manager now panics at startup (not on first request) if neither v2 rbac/service nor RBAC_ADDRESS is present. rbac is a required dependency, so Clowder should always write it.
  • Kessel gRPC with KESSEL_INSECURE=false and auth enabled now performs OIDC discovery against KESSEL_AUTH_OIDC_ISSUER. No change where KESSEL_INSECURE=true.
  • Bearer path only activates when RBAC's v2 entry is authenticated: true (ClowdAppRef); patchman isn't served by one today. Enabling it later needs patchman's SSO service account added to the RBAC gateway AuthorizationPolicy in app-interface.

Secure Coding Practices Checklist GitHub Link

Secure Coding Checklist

  • Input Validation
  • Output Encoding
  • Authentication and Password Management
  • Session Management
  • Access Control
  • Cryptographic Practices
  • Error Handling and Logging
  • Data Protection
  • Communication Security
  • System Configuration
  • Database Security
  • File Management
  • Memory Management
  • General Coding Practices

Summary by Sourcery

Migrate RBAC resolution to Clowder v2 and support authenticated gateway access while preserving secure failure behavior and existing non-RBAC service resolution.

New Features:

  • Resolve the manager's RBAC endpoint and authentication metadata from Clowder v2 dependencies.
  • Authenticate requests to gateway-protected RBAC endpoints with shared Kessel OAuth2 bearer credentials.
  • Trust RBAC-provided CA certificates for secure endpoint connections.

Bug Fixes:

  • Fail closed with a 502 response when credentials cannot be obtained for an authenticated RBAC endpoint.
  • Prevent authorization tokens from appearing in debug HTTP request logs.
  • Use OIDC discovery to obtain the correct Kessel token endpoint.

Enhancements:

  • Validate the RBAC dependency during middleware initialization and isolate authentication metadata when RBAC_ADDRESS overrides the Clowder endpoint.

Tests:

  • Add coverage for Clowder v2 RBAC configuration, bearer authentication, credential sharing, token failures, TLS CA trust, and authorization-header redaction.

The OAuth2 client was built at package init with KESSEL_AUTH_OIDC_ISSUER
passed as the token endpoint. The shared env sets that to the SSO realm
URL, so token fetches could never succeed; it went unnoticed because
KESSEL_INSECURE=true drops call credentials.

Build one credentials object lazily, resolve the token endpoint through
OIDC discovery, and fail with a clear error when issuer, client id or
secret is missing. Failures are not cached so a transient discovery
error is retried. The object is exposed via getBearerToken so HTTP
callers behind the east-west gateway can share it with the gRPC client.
Clowder writes dependencyEndpoints.v2 on every cluster, so read RBAC
only from v2 rbac/service and drop the v1 fallback. The v2 entry also
says whether the endpoint sits behind the east-west gateway
(authenticated) and which CA to trust.

When authenticated, send an SSO service-account bearer token alongside
x-rh-identity; RBAC still evaluates the identity, the gateway only
checks the bearer. If no token can be obtained the manager returns 502
without calling RBAC, so it never makes an unauthenticated call to an
authenticated endpoint. A provided ca_certificate is added to the
system trust store.

The RBAC address is now validated when the middleware is built, so a
missing dependency fails at startup instead of on the first request.
RBAC_ADDRESS still overrides the address for local runs.
@SteveHNH
SteveHNH requested a review from a team as a code owner November 11, 2026 17:01
@sourcery-ai

sourcery-ai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Data structures

Add RBAC authentication and CA-certificate fields to the shared core configuration used by the manager.

  • base/utils/config.go L76-77

Implementation

Build shared Kessel and RBAC OAuth2 credentials lazily through OIDC discovery, bound discovery and token requests, and pass the request context through Kessel client setup.

  • manager/middlewares/kessel.go L9, L22-74, L135

Attach a gateway bearer token to authenticated RBAC requests, trust the configured CA, and fail closed if token acquisition fails.

  • manager/middlewares/rbac.go L10-14, L46-73, L210-231 (+1 small edits)

Redact Authorization during debug request dumps while retaining the real header on the outgoing request.

  • base/utils/http.go L68-76

Configuration

Resolve RBAC only from Clowder v2 and discard its authentication metadata when RBAC_ADDRESS overrides the Clowder endpoint.

  • base/utils/config.go L193-196, L284-294 (+3 small edits)

Update the application configuration to support the Clowder v2 RBAC endpoint and gateway authentication setup.

  • conf/cdappconfig.json L70 (removed lines), L72-81

Tests

Test Clowder v2 RBAC resolution, environment override handling, and preservation or removal of endpoint authentication metadata.

  • base/utils/config_clowder_v2_test.go L11-90, L98-99

Verify that debug HTTP dumps redact Authorization without changing the header sent to the server.

  • base/utils/http_test.go L4-8, L13, L145-170

Test Kessel client setup with shared discovered credentials, including missing issuer handling and authenticated setup.

  • manager/middlewares/kessel_test.go L96-103, L108-111, L120-172 (+2 small edits)

Test authenticated and unauthenticated RBAC calls, fail-closed token errors, shared credentials, and custom CA trust.

  • manager/middlewares/rbac_auth_test.go L1-111

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 4 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="manager/middlewares/rbac.go" line_range="52" />
<code_context>
-	client := api.Client{
+	headers := map[string]string{xRHIdentity: identity}
+	if bearer != "" {
+		headers["Authorization"] = "Bearer " + bearer
+	}
+	return &api.Client{
</code_context>
<issue_to_address>
**Bearer sent to overridden RBAC host**

When clowder marks RBAC authenticated and `RBAC_ADDRESS` points to an untrusted or plaintext destination, `initServicesFromEnv` can replace the Clowder address without changing `RbacAuthenticated`, so the middleware sends the service-account bearer to the override; an HTTP or untrusted `RBAC_ADDRESS` exposes the token to that host.

Bind authentication metadata to the selected endpoint and reject authenticated overrides that are not trusted HTTPS destinations.
</issue_to_address>

### Comment 2
<location path="manager/middlewares/rbac.go" line_range="52" />
<code_context>
-	client := api.Client{
+	headers := map[string]string{xRHIdentity: identity}
+	if bearer != "" {
+		headers["Authorization"] = "Bearer " + bearer
+	}
+	return &api.Client{
</code_context>
<issue_to_address>
**Bearer tokens enter trace logs**

When trace logging is enabled and an authenticated RBAC request is made, `makeClient` enables request debugging at trace level, and `utils.CallAPI` dumps the outgoing request including headers without redaction, so every authenticated RBAC call writes the gateway bearer token to application logs.

Redact the Authorization header from request dumps, or disable request dumping for authenticated RBAC calls.
</issue_to_address>

### Comment 3
<location path="manager/middlewares/kessel.go" line_range="39" />
<code_context>
+	if cfg.KesselAuthOIDCIssuer == "" || cfg.KesselAuthClientID == "" || cfg.KesselAuthClientSecret == "" {
+		return nil, errors.New("KESSEL_AUTH_OIDC_ISSUER, KESSEL_AUTH_CLIENT_ID and KESSEL_AUTH_CLIENT_SECRET must be set")
+	}
+	discovery, err := auth.FetchOIDCDiscovery(ctx, cfg.KesselAuthOIDCIssuer, auth.FetchOIDCDiscoveryOptions{})
+	if err != nil {
+		return nil, errors.Wrap(err, "OIDC discovery failed")
</code_context>
<issue_to_address>
**Hung discovery stalls authorization**

When the OIDC discovery endpoint accepts the request but does not respond, and the incoming request has no deadline, `getCredentials` waits for `auth.FetchOIDCDiscovery` before reaching token retrieval; with no application-level deadline, a stalled OIDC endpoint leaves the RBAC-checked request waiting until its client disconnects, tying up a request handler.

Apply a bounded timeout to OIDC discovery and token acquisition so a stalled identity provider fails promptly.
</issue_to_address>

### Comment 4
<location path="manager/middlewares/rbac_auth_test.go" line_range="87-90" />
<code_context>
+	_, err := getBearerToken(t.Context())
+	require.NoError(t, err)
+	first := credentials
+	_, conn, err := setupClient(t.Context())
+	require.NoError(t, err)
+	defer conn.Close()
+	assert.Same(t, first, credentials)
+}
+
</code_context>
<issue_to_address>
**Kessel auth is not verified**

When `setupClient` stops passing its credentials to `OAuth2ClientAuthenticated` but still builds a client, when `setupClient` creates a client without attaching OAuth2 call credentials, `TestRBACAndKesselShareCredentials` still passes because `assert.Same` checks only the shared global pointer, and `TestSetupClientSecureAuth` checks only that the client is non-nil. The tests pass while secure Kessel calls are unauthenticated.

Exercise the client against a gRPC test server that requires and verifies the bearer token.

Also at `manager/middlewares/kessel_test.go:168-172`.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 4 findings to address first, and when Clowder marks RBAC as authenticated, the middleware now sends an SSO service-account bearer token to the configured RBAC URL; if that destination or its authentication configuration is wrong, the token may already have been disclosed, and reverting cannot undo that exposure. Token-fetch failures also fail closed and deny requests until corrected.

Blocking findings: manager/middlewares/rbac.go:52, manager/middlewares/rbac.go:52, manager/middlewares/kessel.go:39, manager/middlewares/rbac_auth_test.go:90


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread manager/middlewares/rbac.go Resolved
Comment thread manager/middlewares/rbac.go Resolved
Comment thread manager/middlewares/kessel.go Resolved
Comment thread manager/middlewares/rbac_auth_test.go Resolved
SteveHNH and others added 4 commits November 11, 2026 12:11
At trace level CallAPI dumps outgoing requests including headers, so
authenticated RBAC calls would write the gateway bearer token to logs.
The header is masked in place only for the dump; the request still
carries the real token.
The request context passed in is a gin.Context, which is never
cancelled, and discovery runs while holding credentialsMu. A stalled
SSO endpoint would therefore hang every RBAC-checked request, not just
one. Bound each discovery/token HTTP call to 10s.
authenticated and ca_certificate describe the Clowder v2 endpoint. If
RBAC_ADDRESS points elsewhere, keeping them would send the service
account bearer token to a host Clowder never vouched for. An override
equal to the v2 address keeps the v2 settings.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.09%. Comparing base (02d44ea) to head (32ee2c9).

Files with missing lines Patch % Lines
manager/middlewares/rbac.go 86.66% 4 Missing ⚠️
manager/middlewares/kessel.go 93.33% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2389      +/-   ##
==========================================
+ Coverage   66.82%   67.09%   +0.26%     
==========================================
  Files         157      157              
  Lines        9080     9133      +53     
==========================================
+ Hits         6068     6128      +60     
+ Misses       3010     3003       -7     
  Partials        2        2              
Flag Coverage Δ
unittests 67.09% <92.00%> (+0.26%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants