Propagate auth env to experimental.python subprocess#5074
Merged
simonfaltum merged 10 commits intomainfrom May 8, 2026
Merged
Conversation
When workspace.profile is set (via --profile, DATABRICKS_CONFIG_PROFILE, or workspace.profile in databricks.yml), bundle commands that run the experimental.python phase need to pass the resolved profile through to the Python subprocess. Otherwise the Databricks SDK inside Python re-invokes the CLI via `databricks auth token --host <host>` without a profile hint, and the CLI cannot disambiguate profiles sharing the same host in ~/.databrickscfg. Mirrors what the Terraform phase already does via b.AuthEnv() and fixes the remaining path reported on issue #4649. Co-authored-by: Isaac
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
had a problem deploying
to
test-trigger-is
GitHub Actions
Failure
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
andrewnester
approved these changes
May 7, 2026
| assert.Equal(t, int64(1), b.Metrics.PythonUpdatedResourcesCount) | ||
| } | ||
|
|
||
| func TestPythonMutator_propagatesAuthEnv(t *testing.T) { |
Contributor
There was a problem hiding this comment.
nit: we could make it an acceptance test instead
Member
Author
There was a problem hiding this comment.
Done in 9f0f669, moved to acceptance/bundle/python/propagates-auth-env. Mutator captures DATABRICKS_CONFIG_PROFILE from os.environ and writes it to a file we then cat in the script, so it asserts the env actually reaches the python subprocess.
Replace the unit test that stubbed process.Background with an acceptance test under acceptance/bundle/python/propagates-auth-env/ that runs a real python mutator and asserts DATABRICKS_CONFIG_PROFILE is captured from the subprocess env. Co-authored-by: Isaac
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
simonfaltum
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
pietern
reviewed
May 8, 2026
| # so the SDK inside python re-invokes the CLI without a profile and fails on | ||
| # multi-profile ambiguity. | ||
| trace cat captured_env.txt | ||
| echo "" |
Contributor
There was a problem hiding this comment.
Nit: let Python write the newline. I was confused as to why this was necessary.
| export DATABRICKS_CONFIG_FILE=.databrickscfg | ||
| unset DATABRICKS_HOST | ||
| unset DATABRICKS_TOKEN | ||
| unset DATABRICKS_CONFIG_PROFILE |
Contributor
There was a problem hiding this comment.
Can use a comment why they need to be cleared.
|
|
||
| ### Bundles | ||
|
|
||
| * Propagate authentication environment (including `DATABRICKS_CONFIG_PROFILE`) to the `experimental.python` subprocess so bundle validate/deploy no longer fails with a multi-profile host ambiguity error when several profiles in `~/.databrickscfg` share the same host. |
Contributor
There was a problem hiding this comment.
"to Python subprocesses" -- the Python support is no longer experimental.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When
bundle deploy/bundle validateruns with a profile set (via--profile,DATABRICKS_CONFIG_PROFILE, orworkspace.profileindatabricks.yml) and the bundle usesexperimental.python, the Python subprocess does not inheritDATABRICKS_CONFIG_PROFILE. The Databricks SDK inside Python then re-invokes the CLI viadatabricks auth token --host <host>without a profile, and fails with a multi-profile ambiguity error when~/.databrickscfghas several profiles sharing the same host.This is the remaining scenario on #4649. The Terraform path was fixed in #4624; the
experimental.pythonpath was missed because the Python mutator spawned its subprocess viaprocess.Backgroundwithout any auth env setup.Changes
Before:
bundle/config/mutator/python/python_mutator.gocalledprocess.Backgroundwith no environment propagation. The Python subprocess only saw whatever env vars the user happened to export;DATABRICKS_CONFIG_PROFILEfrom--profileordatabricks.ymlwas silently dropped.Now: The mutator calls
b.AuthEnv(ctx)(same call the Terraform path uses) and passes the resulting map throughprocess.WithEnvs(...). The Python SDK and any subprocesses it spawns now see the same auth configuration as the CLI itself.Test plan
TestPythonMutator_propagatesAuthEnvasserts thatDATABRICKS_CONFIG_PROFILEis present in the Python subprocess env whenworkspace.profileis set.go test ./bundle/config/mutator/python/).go test ./bundle/...).make checksandmake lintclean.