fix: Azure end-to-end reliability + warehouse permissions + financial overlays#24
Open
kavyaparashardatabricks wants to merge 6 commits into
Open
fix: Azure end-to-end reliability + warehouse permissions + financial overlays#24kavyaparashardatabricks wants to merge 6 commits into
kavyaparashardatabricks wants to merge 6 commits into
Conversation
…nctions - Makefile.shared: replace all bare `python` with `python3` for macOS compat - setup_demo.py: fix Azure workspace creation (ARM REST API instead of SDK), fix warehouse_type enum, add max_num_clusters, remove invalid managed_resource_group_id - generate_abac.py: pick most complete HCL/SQL block from LLM response (not first), strip SQL-style comments from HCL output, clean up stray commas after policy removal, remove orphaned tag assignments when policies are dropped for missing functions - financial_services.yaml: add 6 missing core masking functions (mask_redact, mask_email, mask_phone, mask_date_to_year, mask_credit_card_full, filter_aml_compliance) Co-authored-by: Isaac
…p cleanup - Switch ARM API version from 2025-10-01-preview to 2024-05-01 (GA) in both setup_demo.py and azure_provider.py - Add try/except with retry on transient 5xx errors in workspace polling loops (setup_demo.py and azure_provider.py) - Extract _cleanup_stray_commas() helper in generate_abac.py, replacing 7 duplicate regex sites; also fix pre-existing bug where valid HCL trailing commas after } were incorrectly stripped - Remove generic masking functions (mask_redact, mask_email, mask_phone, mask_date_to_year, mask_credit_card_full) from financial_services.yaml to avoid definition drift with the base system - Normalize filter_aml_compliance to use lowercase compliance_officer Co-authored-by: Isaac
…aces Switch from 2024-05-01 (GA) to 2025-10-01-preview, the latest published ARM API version for Microsoft.Databricks/workspaces. Both support computeMode=Serverless; preview includes the most recent schema updates. Applied in shared/scripts/cloud_providers/azure_provider.py and shared/examples/aus_bank_demo/setup_demo.py. Co-authored-by: Isaac
…rlay CVV columns (cvv, cvc, security_code) were missing from the identifiers list, so the LLM had no guidance to tag or mask them. Similarly, AML risk flag columns (aml_risk_flag, aml_flag) had a row filter function but no identifier definition with column hints. Adds: - CVV identifier + mask_cvv_redact (full redaction per PCI-DSS) - AML Risk Flag identifier + mask_aml_flag (restricted to compliance/fraud) - Updated prompt_overlay with detection hints for both Co-authored-by: Isaac
…rivileges Azure policy at db_fe management group requires an 'owner' tag on all resources. Added the tag to storage accounts, access connectors, and workspaces. Also grant the test admin group CREATE_CATALOG and CREATE_EXTERNAL_LOCATION on the metastore so members can see and manage UC objects without transferring metastore ownership from the SP. Co-authored-by: Isaac
Without this, groups assigned to the Genie Space get "You do not have permission to use the SQL Warehouse" because no databricks_permissions resource existed for the warehouse. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
databricks_permissionsresource grantingCAN_USEon the SQL warehouse to all ABAC groups — without this, Genie Space users got "You do not have permission to use the SQL Warehouse"CREATE_CATALOG+CREATE_EXTERNAL_LOCATIONon the metastore so members can see and manage catalogsownertag: All Azure resources (storage accounts, access connectors, workspaces) now include the requiredownertag — fixesRequestDisallowedByPolicyfrom thedb_femanagement group policymask_cvv_redact) and AML Risk Flag (mask_aml_flag) identifiers with column hints and prompt guidance2025-10-01-previewfor serverless workspaces, added 5xx retry in polling loops_cleanup_stray_commas()helper, fixed false-positive HCL comma stripping after}Test plan
setup_demo.pymake generate,make validate,make apply ENV=devall passCAN_USEon warehouse via Terraform applytest_no_change_on_valid_hclregression covered)This pull request and its description were written by Isaac.