Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion acceptance/bundle/help/bundle-generate-pipeline/output.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

>>> [CLI] bundle generate pipeline --help
Generate bundle configuration for an existing Delta Live Tables pipeline.
Generate bundle configuration for an existing Spark Declarative Pipeline.

This command downloads an existing Lakeflow Spark Declarative Pipeline's configuration and any associated
notebooks, creating bundle files that you can use to deploy the pipeline to other
Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/help/bundle-open/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Open a deployed bundle resource in the Databricks workspace.

Examples:
databricks bundle open # Prompts to select a resource to open
databricks bundle open my_job # Open specific job in Workflows UI
databricks bundle open my_job # Open specific job in Jobs UI
databricks bundle open my_dashboard # Open dashboard in browser

Use after deployment to quickly navigate to your resources in the workspace.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ resources:

variables:
notebook_dir:
description: Directory with DLT notebooks
description: Directory with SDP notebooks
default: non-existent
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ include:

variables:
notebook_dir:
description: Directory with DLT notebooks
description: Directory with SDP notebooks
default: notebooks
4 changes: 2 additions & 2 deletions acceptance/bundle/paths/pipeline_globs/root/databricks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ include:

variables:
notebook_dir:
description: Directory with DLT notebooks
description: Directory with SDP notebooks
default: notebooks
file_dir:
description: Directory with DLT files
description: Directory with SDP files
default: files
2 changes: 1 addition & 1 deletion acceptance/bundle/run_as/pipelines_legacy/output.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

>>> [CLI] bundle validate -o json
Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC.
Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the Spark Declarative Pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC.
at experimental.use_legacy_run_as
in databricks.yml:8:22

Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/telemetry/deploy-experimental/output.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@

>>> [CLI] bundle deploy
Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC.
Warning: You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the Spark Declarative Pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC.
at experimental.use_legacy_run_as
in databricks.yml:5:22

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ on CI/CD setup.
## Manually deploying to Databricks with Declarative Automation Bundles

Declarative Automation Bundles can be used to deploy to Databricks and to execute
dbt commands as a job using Databricks Workflows. See
dbt commands as a job using Databricks Jobs. See
https://docs.databricks.com/dev-tools/bundles/index.html to learn more.

Use the Databricks CLI to deploy a development copy of this project to a workspace:
Expand All @@ -117,7 +117,7 @@ is optional here.)
This deploys everything that's defined for this project.
For example, the default template would deploy a job called
`[dev yourname] my_dbt_sql_job` to your workspace.
You can find that job by opening your workpace and clicking on **Workflows**.
You can find that job by opening your workpace and clicking on **Jobs**.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can find that job by opening your workpace and clicking on **Jobs**.
You can find that job by opening your workpace and clicking on **Jobs & Pipelines**.


You can also deploy to your production target directly from the command-line.
The warehouse, catalog, and schema for that target are configured in `dbt_profiles/profiles.yml`.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The 'my_default_scala' project was generated by using the default-scala template
This deploys everything that's defined for this project.
For example, the default template would deploy a job called
`[dev yourname] my_default_scala_job` to your workspace.
You can find that job by opening your workspace and clicking on **Workflows**.
You can find that job by opening your workspace and clicking on **Jobs**.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can find that job by opening your workspace and clicking on **Jobs**.
You can find that job by opening your workspace and clicking on **Jobs & Pipelines**.

4. Similarly, to deploy a production copy, type:
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The 'my_default_sql' project was generated by using the default-sql template.
This deploys everything that's defined for this project.
For example, the default template would deploy a job called
`[dev yourname] my_default_sql_job` to your workspace.
You can find that job by opening your workpace and clicking on **Workflows**.
You can find that job by opening your workpace and clicking on **Jobs**.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can find that job by opening your workpace and clicking on **Jobs**.
You can find that job by opening your workpace and clicking on **Jobs & Pipelines**.

4. Similarly, to deploy a production copy, type:
```
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
-- This query is executed using Databricks Workflows (see resources/my_default_sql_sql.job.yml)
-- This query is executed using Databricks Jobs (see resources/my_default_sql_sql.job.yml)

USE CATALOG {{catalog}};
USE IDENTIFIER({{schema}});
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
-- This query is executed using Databricks Workflows (see resources/my_default_sql_sql.job.yml)
-- This query is executed using Databricks Jobs (see resources/my_default_sql_sql.job.yml)
--
-- The streaming table below ingests all JSON files in /databricks-datasets/retail-org/sales_orders/
-- See also https://docs.databricks.com/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"project_name": "my_jobs_as_code",
"include_notebook": "yes",
"include_python": "yes",
"include_dlt": "yes"
"include_sdp": "yes"
}
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Warning: Ignoring Databricks CLI version constraint for development build. Requi
"libraries": [
{
"notebook": {
"path": "/Workspace/Users/[USERNAME]/.bundle/my_jobs_as_code/dev/files/src/dlt_pipeline"
"path": "/Workspace/Users/[USERNAME]/.bundle/my_jobs_as_code/dev/files/src/sdp_pipeline"
}
}
],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ The 'my_jobs_as_code' project was generated by using the "Jobs as code" template
This deploys everything that's defined for this project.
For example, the default template would deploy a job called
`[dev yourname] my_jobs_as_code_job` to your workspace.
You can find that job by opening your workspace and clicking on **Workflows**.
You can find that job by opening your workspace and clicking on **Jobs**.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can find that job by opening your workspace and clicking on **Jobs**.
You can find that job by opening your workspace and clicking on **Jobs & Pipelines**.

3. Similarly, to deploy a production copy, type:
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ where = ["src"]
dev-dependencies = [
"databricks-bundles==x.y.z",

## Add code completion support for DLT
## Add code completion support for Spark Declarative Pipelines
# "databricks-dlt",

## databricks-connect can be used to run parts of this project locally.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
"libraries": [
{
"notebook": {
"path": "src/dlt_pipeline.ipynb",
"path": "src/sdp_pipeline.ipynb",
},
},
],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
"source": [
"# Default notebook\n",
"\n",
"This default notebook is executed using Databricks Workflows as defined in resources/my_jobs_as_code.job.yml."
"This default notebook is executed using Databricks Jobs as defined in resources/my_jobs_as_code.job.yml."
]
},
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
}
},
"source": [
"# DLT pipeline\n",
"# SDP pipeline\n",
"\n",
"This Lakeflow Spark Declarative Pipeline definition is executed using a pipeline defined in resources/my_jobs_as_code.pipeline.yml."
]
Expand Down Expand Up @@ -72,7 +72,7 @@
"notebookMetadata": {
"pythonIndentUnit": 2
},
"notebookName": "dlt_pipeline",
"notebookName": "sdp_pipeline",
"widgets": {}
},
"kernelspec": {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import (

type captureUCDependencies struct{}

// If a user defines a UC schema in the bundle, they can refer to it in DLT pipelines,
// If a user defines a UC schema in the bundle, they can refer to it in SDP pipelines,
// UC Volumes, Registered Models, Quality Monitors, or Model Serving Endpoints using the
// `${resources.schemas.<schema_key>.name}` syntax. Using this syntax allows TF to capture
// the deploy time dependency this resource has on the schema and deploy changes to the
Expand Down Expand Up @@ -110,7 +110,7 @@ func (m *captureUCDependencies) Apply(ctx context.Context, b *bundle.Bundle) dia
if p == nil {
continue
}
// "schema" and "target" have the same semantics in the DLT API but are mutually
// "schema" and "target" have the same semantics in the SDP API but are mutually
// exclusive i.e. only one can be set at a time.
p.Schema = resolveSchema(b, p.Catalog, p.Schema)
p.Target = resolveSchema(b, p.Catalog, p.Target)
Expand Down
4 changes: 2 additions & 2 deletions bundle/config/mutator/resourcemutator/run_as.go
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ func setRunAsForAlerts(b *bundle.Bundle) {
}
}

// Legacy behavior of run_as for DLT pipelines. Available under the experimental.use_run_as_legacy flag.
// Legacy behavior of run_as for SDP pipelines. Available under the experimental.use_run_as_legacy flag.
// Only available to unblock customers stuck due to breaking changes in https://github.com/databricks/cli/pull/1233
func setPipelineOwnersToRunAsIdentity(b *bundle.Bundle) {
runAs := b.Config.RunAs
Expand Down Expand Up @@ -228,7 +228,7 @@ func (m *setRunAs) Apply(_ context.Context, b *bundle.Bundle) diag.Diagnostics {
return diag.Diagnostics{
{
Severity: diag.Warning,
Summary: "You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the DLT pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC.",
Summary: "You are using the legacy mode of run_as. The support for this mode is experimental and might be removed in a future release of the CLI. In order to run the Spark Declarative Pipelines in your DAB as the run_as user this mode changes the owners of the pipelines to the run_as identity, which requires the user deploying the bundle to be a workspace admin, and also a Metastore admin if the pipeline target is in UC.",
Paths: []dyn.Path{dyn.MustPathFromString("experimental.use_legacy_run_as")},
Locations: b.Config.GetLocations("experimental.use_legacy_run_as"),
},
Expand Down
8 changes: 4 additions & 4 deletions bundle/internal/schema/annotations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -327,9 +327,9 @@ github.com/databricks/cli/bundle/config.Root:
```
"run_as":
"description": |-
The identity to use when running Declarative Automation Bundles workflows.
The identity to use when running Declarative Automation Bundles resources.
"markdown_description": |-
The identity to use when running Declarative Automation Bundles workflows. See [\_](/dev-tools/bundles/run-as.md).
The identity to use when running Declarative Automation Bundles resources. See [\_](/dev-tools/bundles/run-as.md).
"scripts":
"description": |-
PLACEHOLDER
Expand Down Expand Up @@ -420,7 +420,7 @@ github.com/databricks/cli/bundle/config.Workspace:
The Databricks account ID.
"artifact_path":
"description": |-
The artifact path to use within the workspace for both deployments and workflow runs
The artifact path to use within the workspace for both deployments and job runs
"auth_type":
"description": |-
The authentication type.
Expand Down Expand Up @@ -450,7 +450,7 @@ github.com/databricks/cli/bundle/config.Workspace:
Experimental feature flag to indicate if the host is a unified host
"file_path":
"description": |-
The file path to use within the workspace for both deployments and workflow runs
The file path to use within the workspace for both deployments and job runs
"google_service_account":
"description": |-
The Google service account name
Expand Down
4 changes: 2 additions & 2 deletions bundle/internal/schema/annotations_openapi_overrides.yml
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,7 @@ github.com/databricks/cli/bundle/config/resources.ModelServingEndpoint:
github.com/databricks/cli/bundle/config/resources.Pipeline:
"_":
"markdown_description": |-
The pipeline resource allows you to create Delta Live Tables [pipelines](/api/workspace/pipelines/create). For information about pipelines, see [_](/dlt/index.md). For a tutorial that uses the Declarative Automation Bundles template to create a pipeline, see [_](/dev-tools/bundles/pipelines-tutorial.md).
The pipeline resource allows you to create Spark Declarative [Pipelines](/api/workspace/pipelines/create). For information about pipelines, see [_](/dlt/index.md). For a tutorial that uses the Declarative Automation Bundles template to create a pipeline, see [_](/dev-tools/bundles/pipelines-tutorial.md).
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Nit] The new text reads "create Spark Declarative [Pipelines](...)" which splits the product name across a link boundary. "Spark Declarative" on its own doesn't mean anything. The original worked because "Delta Live Tables" was a self-contained product name and "pipelines" was a generic noun.

Consider "create [Spark Declarative Pipelines](/api/workspace/pipelines/create)" to keep the full product name inside the link.

"markdown_examples": |-
The following example defines a pipeline with the resource key `hello-pipeline`:

Expand Down Expand Up @@ -454,7 +454,7 @@ github.com/databricks/cli/bundle/config/resources.RegisteredModel:
github.com/databricks/cli/bundle/config/resources.Schema:
"_":
"markdown_description": |-
The schema resource type allows you to define Unity Catalog [schemas](/api/workspace/schemas/create) for tables and other assets in your workflows and pipelines created as part of a bundle. A schema, different from other resource types, has the following limitations:
The schema resource type allows you to define Unity Catalog [schemas](/api/workspace/schemas/create) for tables and other assets in your jobs and pipelines created as part of a bundle. A schema, different from other resource types, has the following limitations:

- The owner of a schema resource is always the deployment user, and cannot be changed. If `run_as` is specified in the bundle, it will be ignored by operations on the schema.
- Only fields supported by the corresponding [Schemas object create API](/api/workspace/schemas/create) are available for the schema resource. For example, `enable_predictive_optimization` is not supported as it is only available on the [update API](/api/workspace/schemas/update).
Expand Down
10 changes: 5 additions & 5 deletions bundle/phases/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,12 @@ func approvalForDeploy(ctx context.Context, b *bundle.Bundle, plan *deployplan.P

types := []deployplan.ActionType{deployplan.Recreate, deployplan.Delete}
schemaActions := filterGroup(actions, "schemas", types...)
dltActions := filterGroup(actions, "pipelines", types...)
pipelineActions := filterGroup(actions, "pipelines", types...)
volumeActions := filterGroup(actions, "volumes", types...)
dashboardActions := filterGroup(actions, "dashboards", types...)

// We don't need to display any prompts in this case.
if len(schemaActions) == 0 && len(dltActions) == 0 && len(volumeActions) == 0 && len(dashboardActions) == 0 {
if len(schemaActions) == 0 && len(pipelineActions) == 0 && len(volumeActions) == 0 && len(dashboardActions) == 0 {
return true, nil
}

Expand All @@ -56,10 +56,10 @@ func approvalForDeploy(ctx context.Context, b *bundle.Bundle, plan *deployplan.P
}
}

// One or more DLT pipelines is being recreated.
if len(dltActions) != 0 {
// One or more SDP pipelines is being recreated.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] Nice rename from dltActions to pipelineActions. The comment here could match: since the variable already says "pipeline", you could simplify to // One or more pipelines is being recreated. and drop the "SDP" abbreviation.

if len(pipelineActions) != 0 {
cmdio.LogString(ctx, deleteOrRecreatePipelineMessage)
for _, action := range dltActions {
for _, action := range pipelineActions {
cmdio.Log(ctx, action)
}
}
Expand Down
Loading
Loading