chore(spark): remove unneccessary ‘compat’ functions#816
Merged
bestbeforetoday merged 1 commit intosubstrait-io:mainfrom Apr 15, 2026
Merged
chore(spark): remove unneccessary ‘compat’ functions#816bestbeforetoday merged 1 commit intosubstrait-io:mainfrom
bestbeforetoday merged 1 commit intosubstrait-io:mainfrom
Conversation
When the Spark converter code was refactored to support all of spark v3.0, v3.5 and v4.0, a number of functions with incompatible APIs were abstracted into the ‘SparkCompat’ trait. In the event, `ScalarSubquery` and `Aggregate` didn’t need this treatment, so this commit restores the original (simpler) code. Signed-off-by: Andrew Coleman <andrew_coleman@uk.ibm.com>
bestbeforetoday
approved these changes
Apr 15, 2026
|
|
||
| override def createScalarSubquery(plan: LogicalPlan): ScalarSubquery = { | ||
| // Spark 4.0 simplified constructor - no exprId needed | ||
| ScalarSubquery(plan) |
Member
There was a problem hiding this comment.
Note for others: I was confused that this single argument ScalarSubquery construction for Spark 4.0 is replaced by ScalarSubquery(plan, exprId = NamedExpression.newExprId). Checking deeper, NamedExpression.newExprId is the default for the exprId parameter if none is supplied so the two are functionally equivalent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the Spark converter code was refactored to support all of spark v3.0, v3.5 and v4.0, a number of functions with incompatible APIs were abstracted into the ‘SparkCompat’ trait. In the event,
ScalarSubqueryandAggregatedidn’t need this treatment, so this commit restores the original (simpler) code.