Skip to content

[DNM] Adding Upstream Spark#11893

Draft
zhouyuan wants to merge 13 commits intoapache:mainfrom
zhouyuan:wip_upstreeam_spark
Draft

[DNM] Adding Upstream Spark#11893
zhouyuan wants to merge 13 commits intoapache:mainfrom
zhouyuan:wip_upstreeam_spark

Conversation

@zhouyuan
Copy link
Copy Markdown
Member

@zhouyuan zhouyuan commented Apr 8, 2026

What changes are proposed in this pull request?

This patch adds a spark-main profile which builds Gluten against Spark nightly builds
TPCH/DS tests added

With this patch Gluten should be able to catch-up with Spark release, e.g., if Spark cut the branch for release Gluten could also move the spark-main shim layer to the target release shim easily

How was this patch tested?

pass GHA

Was this patch authored or co-authored using generative AI tooling?

@github-actions github-actions bot added CORE works for Gluten Core VELOX INFRA TOOLS labels Apr 8, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 8, 2026

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 8, 2026

Run Gluten Clickhouse CI on x86

@zhouyuan zhouyuan requested review from baibaichen and philo-he April 8, 2026 19:22
@philo-he
Copy link
Copy Markdown
Member

philo-he commented Apr 8, 2026

@zhouyuan, thanks for proposing this patch.
Perhaps we should move the Spark main tests to a nightly CI job. Since the Spark snapshot is under active development, keeping these tests in our regular CI may frequently cause stability issues and build failures that possibly blocking PR merging. Thanks.

@baibaichen
Copy link
Copy Markdown
Contributor

@zhouyuan would you please follow #11352?

@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@zhouyuan zhouyuan force-pushed the wip_upstreeam_spark branch from a39fe5a to 35ac892 Compare April 15, 2026 17:05
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

@zhouyuan
Copy link
Copy Markdown
Member Author

@philo-he @baibaichen Thanks for the inputs. I have monitored this for almost 2 weeks and there are two API changes from Spark in this timeframe which requires modifying the whole shim layers, moving the tests to nightly is a better option.

zhouyuan added 12 commits April 16, 2026 18:00
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
Signed-off-by: Yuan <yuanzhou@apache.org>
@zhouyuan zhouyuan force-pushed the wip_upstreeam_spark branch from 35ac892 to 43d3012 Compare April 16, 2026 17:05
Signed-off-by: Yuan <yuanzhou@apache.org>
@github-actions
Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core INFRA TOOLS VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants