Skip to content

DOC-7289: Explain Databricks identities, grants, and compute profiles for Feature Form - #4503

Open
epps wants to merge 1 commit into
mainfrom
DOC-7289-featureform-databricks-3.3.0
Open

epps wants to merge 1 commit into
mainfrom
DOC-7289-featureform-databricks-3.3.0

Conversation

@epps

@epps epps commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Expands the Register Databricks providers page so Feature Form 3.3.0 users can answer common setup and operations questions without contacting support:

  • Explains which identity Databricks runs use for every catalog they read or write, and when unity-catalog provider credentials are used.
  • Adds a table of the Unity Catalog privileges each principal needs, including WRITE VOLUME for incremental transformations and READ on the Redis secret scope.
  • Clarifies how datasets refer to catalogs, what output_catalog controls, and when each catalog provider must be registered.
  • Documents installing the Feature Form Spark runtime on existing clusters.
  • Shows how to use different cluster settings for different workloads, how provider updates affect queued applies, and where to set Spark properties.
  • Describes how applies map to Databricks runs and how to sequence groups of resources.
  • Adds guidance on sharing or separating artifact Volumes, and on migrating from legacy artifact storage.
  • Explains how the Redis password reaches Databricks runs without appearing in run definitions or artifacts.
  • Adds tenant-isolation guidance for Databricks providers.
  • Replaces inconsistent identity terms with "Databricks run principal", and adds troubleshooting rows.

Verification

  • git diff --check passes.
  • Reviewed the rendered page content locally.
  • Confirmed every new in-page anchor resolves to a heading on the page.
  • Only content/develop/ai/featureform/register-providers/databricks.md changed.

🤖 Generated with Claude Code


Note

Low Risk
Documentation-only changes to a Feature Form how-to page; no runtime, security, or data-path impact.

Overview
Updates the Register Databricks providers page so operators understand identities, privileges, and operational limits in one place.

The doc now distinguishes Databricks run principal vs catalog and compute provider principals, explains when unity-catalog credentials are used (server-side checks, dataframe reads, artifact Volume I/O), and adds a Unity Catalog privileges table (inputs, output catalog, artifact Volume including WRITE VOLUME for incremental work, Redis secret scope, artifact-store principal).

Registration and catalog guidance are tightened: dependency order for output vs input catalogs, how datasets reference catalog_provider vs output_catalog, and that all workload providers must share the same workspace host (with a note on where mismatches surface).

New operational sections cover installing the featureform-spark wheel on existing clusters (Docker copy steps, version match), using multiple databricks providers or ff provider update instead of per-resource job configs, where Spark conf lives, how each apply step maps to Databricks runs and sequencing with ff apply, artifact Volume sharing vs tenant isolation, migrating off legacy DBFS/Workspace Files, Redis password resolution without leaking secrets in run defs, and workspace-level RBAC for multi-tenant Databricks.

Troubleshooting and health-check tables are extended (run principal grants, Volume grants, runtime install, unsupported job config, queued apply after provider update).

No code or behavior changes.

Reviewed by Cursor Bugbot for commit c69a038. Bugbot is set up for automated code reviews on this repo. Configure here.

… Feature Form

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

DOC-7289

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🧠 Redis Memory

Found 5 related items from repository history (5 new this commit):

Memory updated at c69a038

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants