Describe the feature
The V1 incremental materialization path (use_materialization_v2: false, the default) calls adapter.get_relation_config() unconditionally during every incremental run. This triggers 8 sequential metadata queries against information_schema and system tables, even when the user does not use tags, constraints, column masks, or other configuration change features.
The V2 path already wraps this in a process_config_changes() macro that respects the incremental_apply_config_changes config flag, but the V1 path calls get_relation_config() directly with no way to skip it.
Queries issued per incremental model on Unity Catalog:
SELECT ... FROM system.information_schema.table_tagsSELECT ... FROM system.information_schema.column_tagsSELECT ... FROM information_schema.columns WHERE is_nullable = 'NO'SELECT ... FROM information_schema.key_column_usage (PRIMARY KEY)SELECT ... FROM information_schema.key_column_usage (FOREIGN KEY)SELECT ... FROM system.information_schema.column_masksSHOW TBLPROPERTIESDESCRIBE TABLE EXTENDED
Describe alternatives you've considered
- Users could override
fetch_tags, fetch_column_tags, etc. with empty macros in their project, but this is fragile and version-dependent.
- Using V2 with
incremental_apply_config_changes: false works, but V2 has its own DDL overhead.
Additional context
The fix is minimal: replace the inline config change detection/application code in the V1 path with the existing process_config_changes() macro (already used by V2), which respects the incremental_apply_config_changes flag. This also removes code duplication and brings V1 in line with V2's behavior.
Who will this benefit?
All users running incremental models on Unity Catalog with the default V1 materialization path. The impact is most significant for:
- Projects with many incremental models (overhead compounds per model)
- Small-to-medium data volumes where metadata overhead is a large fraction of total execution time
Are you interested in contributing this feature?
Yes, the fix is ready.
Describe the feature
The V1 incremental materialization path (
use_materialization_v2: false, the default) callsadapter.get_relation_config()unconditionally during every incremental run. This triggers 8 sequential metadata queries againstinformation_schemaandsystemtables, even when the user does not use tags, constraints, column masks, or other configuration change features.The V2 path already wraps this in a
process_config_changes()macro that respects theincremental_apply_config_changesconfig flag, but the V1 path callsget_relation_config()directly with no way to skip it.Queries issued per incremental model on Unity Catalog:
SELECT ... FROM system.information_schema.table_tagsSELECT ... FROM system.information_schema.column_tagsSELECT ... FROM information_schema.columns WHERE is_nullable = 'NO'SELECT ... FROM information_schema.key_column_usage(PRIMARY KEY)SELECT ... FROM information_schema.key_column_usage(FOREIGN KEY)SELECT ... FROM system.information_schema.column_masksSHOW TBLPROPERTIESDESCRIBE TABLE EXTENDEDDescribe alternatives you've considered
fetch_tags,fetch_column_tags, etc. with empty macros in their project, but this is fragile and version-dependent.incremental_apply_config_changes: falseworks, but V2 has its own DDL overhead.Additional context
The fix is minimal: replace the inline config change detection/application code in the V1 path with the existing
process_config_changes()macro (already used by V2), which respects theincremental_apply_config_changesflag. This also removes code duplication and brings V1 in line with V2's behavior.Who will this benefit?
All users running incremental models on Unity Catalog with the default V1 materialization path. The impact is most significant for:
Are you interested in contributing this feature?
Yes, the fix is ready.