Skip to content

chore: Bump github.com/substrait-io/substrait-go from v8.1.1 to v9.0.0 - #1399

Open
kadinrabo wants to merge 7 commits into
apache:mainfrom
kadinrabo:kr/substrait-go-v9-alpha2
Open

kadinrabo wants to merge 7 commits into
apache:mainfrom
kadinrabo:kr/substrait-go-v9-alpha2

Conversation

@kadinrabo

@kadinrabo kadinrabo commented Oct 21, 2026 •

Copy link
Copy Markdown

Rationale for this change

Upgrade substrait-go to the latest major version (v9). I'm visiting from @substrait-io and want to bump this so I can update the dependency in iceberg-go.

v9 finishes moving protobuf out of substrait-go core, so substrait-protobuf/go drops out of arrow-go entirely.

What changes are included in this PR?

  • Updated github.com/substrait-io/substrait-go from v8.1.1 to v9.0.0, and import paths from v8 to v9 across arrow/compute/exprs
  • Dropped github.com/substrait-io/substrait-protobuf/go v0.85.0, no longer reachable
  • extensions.ID is now extensions.ExtensionID, with conversions where v9 asks for TypeID, FunctionID or TypeVariationID
  • types.Behavior* became the types.CastFailBehavior* enum, and types.PrecisionTimestamp is a plain struct

Are these changes tested?

Existing tests pass (go test ./arrow/compute/...). Also verified iceberg-go builds and tests green against this branch.

Are there any user-facing changes?

Users importing arrow/compute/exprs will transitively depend on substrait-go v9 instead of v8, and no longer get substrait-protobuf. Exported signatures taking extensions.ID now take extensions.ExtensionID.

@zeroshade

Copy link
Copy Markdown
Member

happy to move forward with this once we have a stable tag instead of the alpha one.

@kadinrabo kadinrabo changed the title chore: Bump github.com/substrait-io/substrait-go from v8.1.1 to v9.0.0-alpha.2 chore: Bump github.com/substrait-io/substrait-go from v8.1.1 to v9.0.0 Nov 3, 2026
@kadinrabo
kadinrabo marked this pull request as ready for review November 3, 2026 20:33
@kadinrabo
kadinrabo requested a review from zeroshade as a code owner November 3, 2026 20:33
@zeroshade

Copy link
Copy Markdown
Member

Why does substrait-go do so many breaking changes and major version updates? 😦

I'm unsure on this because of the breaking changes to public APIs that this is doing (the change from extensions.ID to extensions.ExtensionID. We should probably use aliases or other changes in here to hide the public substrait types from the arrow-go public API to avoid these in the future.

@vbarua

vbarua commented Nov 4, 2026

Copy link
Copy Markdown

happy to move forward with this once we have a stable tag instead of the alpha one.

For this PR, we just published the first official v9 release: https://pkg.go.dev/github.com/substrait-io/substrait-go/v9

@kadinrabo was checking that the alpha build compiled correctly for arrow-go before we published it. He's updated this to use the official release.

Why does substrait-go do so many breaking changes and major version updates? 😦

Honestly, because we accidentally published the first v1 release before the library should have been considered stable, and we never put the cat back in the bag. If the constant major updates are becoming a problem, we can consider starting fresh with a new name to limit these issues in the future.

We should probably use aliases or other changes in here to hide the public substrait types from the arrow-go public API to avoid these in the future.

Would it make sense to add the aliases in this PR if this is going to be breaking already? It would probably be good the decouple the arrow-go API from the substrait-go one.

Comment thread arrow/compute/exprs/types.go Outdated

type substraitToArrow = func(*expr.ScalarFunction, []compute.Datum) (fname string, args []compute.Datum, opts compute.FunctionOptions, err error)
type arrowToSubstrait = func(fname string) (extensions.ID, []*types.FunctionOption, error)
type arrowToSubstrait = func(fname string) (extensions.ExtensionID, []*types.FunctionOption, error)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be extension.FunctionID?

This applies here and to other places where the tigher ID bound would make sense.

Comment thread arrow/compute/exprs/types.go Outdated
}

func (e *extensionSet) EncodeTypeVariation(dt arrow.DataType) (extensions.ID, uint32, bool) {
func (e *extensionSet) EncodeTypeVariation(dt arrow.DataType) (extensions.ExtensionID, uint32, bool) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this return an extensions.TypeVariationID?

Comment thread arrow/compute/exprs/types.go Outdated
}

func (e *extensionSet) EncodeType(dt arrow.DataType) (extensions.ID, uint32, bool) {
func (e *extensionSet) EncodeType(dt arrow.DataType) (extensions.ExtensionID, uint32, bool) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this return a extensions.TypeID?

Comment thread arrow/compute/exprs/types.go Outdated

func (e *extensionSet) DecodeTypeArrow(anchor uint32) (extensions.ID, arrow.DataType, bool) {
id, ok := e.DecodeType(anchor)
func (e *extensionSet) DecodeTypeArrow(anchor uint32) (extensions.ExtensionID, arrow.DataType, bool) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was going to say that this should return a TypeID, but actually it can also return a TypeVariationID based on the code. That actually points to a potential issue.

The given anchor could be both a valid Type or TypeVariation extension as they are numbered indepently, but if both are set we would only ever return the TypeID.

@benbellick

Copy link
Copy Markdown
Contributor

Hi @zeroshade! I see that changes to substrait-go's API are having an adverse impact on arrow-go's API compatibility. We don't want to inadvertently cause breaking changes to your API. @kadinrabo and I are discussing some ideas for how to move forwards. We anticipate more breaking changes to substrait-go in the future, so we can't make any guarantee yet on frequency of major version bumps to substrait-go.

How do you generally decide when to make a major release of arrow-go? We will follow up with some concrete suggestions, but first wanted to get a sense of how negatively these breaking changes will impact you.

On our side, we have internal work blocked on removing arrow-go's dependency on substrait-protobuf. That gives us some motivation to get this upgrade through, but this can wait until we figure out a plan to do upgrades in a more compatible way. Thanks!

@zeroshade

Copy link
Copy Markdown
Member

Hey @benbellick,

I've been trying to avoid major version releases as much as possible. The ideal scenario here would be to not have any public API breaks. But if it's impossible to avoid that, then let's look to insulate the public arrow-go API from future substrait-go breaks. We can do the major version release if we can ensure that we'll avoid having to do more major versions for future substrait changes.

@benbellick

Copy link
Copy Markdown
Contributor

Hi @zeroshade! I agree that removing substrait-go types from Arrow's public API would be ideal. But that seems difficult for functions like ExecuteScalarExpression, which lets libraries like Iceberg build Substrait expressions and pass them to Arrow for execution.

First, I think we should reduce the exposure of substrait-go's API where we can. For example, Arrow probably doesn't need to expose function/type anchor bookkeeping. I put together a draft exploring that cleanup.

Beyond that, I see a few options:

  1. Keep existing execution functions and add new ones for each major version. For example, keep ExecuteScalarExpression accepting v8 expressions and add ExecuteScalarExpressionV9 for v9. We'd maintain some duplicate code, but users could upgrade when they're ready. Iceberg already builds its own expressions, so we wouldn't need to duplicate Arrow's builders for this use case.

  2. Move the Arrow/Substrait code into its own independently versioned module. We'd still have breaking changes, but they wouldn't force a major release of all of Arrow.

  3. Remove Substrait types from the public API entirely. I'm not sure how to do this cleanly without moving execution elsewhere or giving Arrow its own expression API, which goes against the idea of sharing Substrait's model.

We'd want to make any necessary breaking changes together in one major release, with a plan to avoid repeating this in the future. After that, adding execution support for newer Substrait versions wouldn't need to break existing callers.

What do you think? Any preference between those three? I think 1 is the simplest, though I'm not sure how much code we'd need to duplicate for each new major version. It would also mean keeping dependencies on multiple substrait-go major versions rather than replacing the old one with the new one.

@Sunyue

Sunyue commented Nov 17, 2026

Copy link
Copy Markdown

Hi, below are the arrow-go API that we are using which takes or returns substrait's types, please consider not removing those:
ExprBuilder.WrapLiteral, ExprBuilder.Cast, ExprBuilder.CallScalar, Builder.BuildExpr

@zeroshade

Copy link
Copy Markdown
Member

Keep existing execution functions and add new ones for each major version. For example, keep ExecuteScalarExpression accepting v8 expressions and add ExecuteScalarExpressionV9 for v9. We'd maintain some duplicate code, but users could upgrade when they're ready. Iceberg already builds its own expressions, so we wouldn't need to duplicate Arrow's builders for this use case.

I like this idea, though perhaps we can come up with a better name than just tacking V9 onto it 😄

Move the Arrow/Substrait code into its own independently versioned module. We'd still have breaking changes, but they wouldn't force a major release of all of Arrow.

This might be the better future-looking option though, though we wouldn't move the existing code. Instead we'd deprecate the current ones (to be removed at the next major version release) but this would enable better APIs and avoid requiring major version releases for Arrow itself.

It would also mean keeping dependencies on multiple substrait-go major versions rather than replacing the old one with the new one.

While this would make development harder, the Go compiler should be able to drop the unused substrait-go major versions if a dependency is only using one of them instead of both.

It might be best to start with option 1 so we can solve the current issue, and then start working on Option 2 and creating a separately versioned Arrow/Substrait module

@benbellick

Copy link
Copy Markdown
Contributor

@zeroshade Great! I will work with @kadinrabo to prep a few PRs (all of which will cause breaking changes) to merge before the next major release. These would reduce the exposure of the substrait-go API. Then once that is in, we can merge this one.

If you are interested in the module splitting btw, that is exactly what we did inside of substrait-go to remove the dependency on the generated protobuf code.

@zeroshade

Copy link
Copy Markdown
Member

I will work with @kadinrabo to prep a few PRs (all of which will cause breaking changes) to merge before the next major release.

I thought we were just going to do the additive methods and not do the breaking changes? i.e. option 1, keeping the existing functions and adding new functions for v9 so users can upgrade when ready and we avoid breaking changes for now?

@benbellick

Copy link
Copy Markdown
Contributor

@zeroshade Ah I may have misunderstood. For us, the primary motivation of this work is to remove the dependence on substrait-protobuf. Adding v9 support alongside v8 wouldn't achieve that, since keeping the v8 integration would keep the old dependency around. So, removing the dependency would have to be a breaking change.

The suggestion above was meant to be about preventing future changes to the substrait-go API (e.g. v10) from causing further breaking changes in arrow-go. If we decide to make a breaking change to remove the old dependency, I think it makes sense to take advantage of the moment to also reduce the API boundary where we can.

However, I think the first thing to settle is whether we can make a breaking change to remove the dependency on substrait-protobuf. If you're not ready to make that change now, do you have a sense of when you'd consider another major release?

@zeroshade

Copy link
Copy Markdown
Member

Is the goal for removing the dependency of substrait-protobuf to reduce the size of binaries that are compiled with it? Because if we add the new methods and dependency, the Go compiler should be able to potentially optimize and remove the unused symbols/dependency when you only call the new methods.

Or is the goal something else?

My preference is to see if there's a way to avoid the breaking change now if possible, but if we can't then I would say we can prep the changes and target the breaking change for release at the end of the year.

@benbellick

Copy link
Copy Markdown
Contributor

@zeroshade This actually isn't about binary size. We have a large monorepo that depends on substrait-go, arrow-go, and iceberg-go, and we want to upgrade our Substrait version at a faster cadence.

These protobuf updates often include breaking changes to the generated Go API, so we can't simply override the transitive dependency in our monorepo. We need the libraries using it to be updated as well. That can mean:

  1. Bump substrait-protobuf in substrait-go, make any necessary compatibility changes, and cut a release.
  2. Update substrait-go in arrow-go, make any necessary compatibility changes, and cut a release.
  3. Update arrow-go in iceberg-go, make any necessary compatibility changes, and cut a release.
  4. Update those dependencies together in our monorepo.

We can't use different versions of the protobuf-dependent libraries side by side, because their generated definitions would conflict in protobuf's global registry.

With substrait-go/v9, Arrow no longer depends on the generated protobuf code. Our code could then use substrait-go/v10 while Arrow and Iceberg stay on substrait-go/v9, since we don't pass expressions between our code and their integration.

E.g. this is why I updated arrow-go here, followed by iceberg-go here.

@zeroshade

Copy link
Copy Markdown
Member

gotcha, then yea. we can target doing a major version release at the end of the year and prep the changes to make it so future updates won't require more major version releases from arrow-go.

@benbellick

benbellick commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Great, thanks so much :)

I don't want to have to cause a breaking change but I appreciate your flexibility here. This will help us a lot!

@benbellick

Copy link
Copy Markdown
Contributor

I made an issue to track this work: #1622

We will start working on this.

@benbellick

benbellick commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

@zeroshade Here’s an example of one way we could change the public API to reduce the impact of future substrait-go upgrades on Arrow Go: #1500 and #1627. Would you mind taking a look and letting me know what you think?

I’m not really sure what the strategy is for deprecations here generally, but I thought it would make sense to make this a stack where the first PR just marks the APIs as deprecated and the second removes them.

@benbellick

Copy link
Copy Markdown
Contributor

Hey @zeroshade, small update!

I wanted to see what it would look like to maintain multiple versions of the API on different underlying substrait-go versions. I have two attempts here:

  1. Have a single source of truth on v9, with adapters for the v8 API (benbellick#6)
  2. Maintain all of the duplicated code in both places (benbellick#2)

(These are AI-generated and drafts, so I'm not asking for a real review here or anything, just sharing the general shape of the solutions.)

As you can see, they both require quite a bit of code 😅. Even if we reduce the scope of the expr API, the API surface is still quite large, and I think it will be a relatively large maintenance burden for Arrow maintainers.

Instead, I took a stab at generating a PR that breaks this out into a separate module (benbellick#7), which to me seems a lot simpler ultimately. Like you suggested, it deprecates the existing package rather than moving it, and bumping the new module to substrait-go v9 would be a follow-up. We can version them separately, as we do in substrait-go.

I'm unsure whether this is a reasonable goal to achieve before EOY for the next major release. Do you have any opinion on how we should move forward?

Thanks a lot 🙇

@zeroshade

Copy link
Copy Markdown
Member

Thanks for putting those drafts together, that makes the trade-off a lot clearer. I agree the side-by-side versions are too much code to maintain, so let's go with the separate module.

One thing I want to check before we commit to a major release: I don't think the separate module needs one. Go only compiles packages that are actually imported, so if the v9 code lives in the new module and arrow/compute/exprs stays on v8 as a deprecated package, anything that imports only the new module never builds the v8 code or substrait-protobuf. I tried it: a program that depends on arrow-go main without importing exprs compiles no substrait-go/v8 packages and doesn't have the module in its binary. substrait-go/v8 would still be listed in arrow-go's go.mod, but it would no longer constrain which substrait-protobuf you can build against, as long as iceberg-go moves to the new module. Does that cover your monorepo case, or is there something about the go.mod entry itself that still causes trouble?

If that works, the plan would be:

  1. Merge refactor(arrow/compute/exprs): deprecate Substrait anchor APIs #1500 (deprecations only).
  2. Add the new module, on v9 from the start if that's practical, and deprecate arrow/compute/exprs in place.
  3. Retarget this PR at the new module instead of arrow/compute/exprs.
  4. Leave refactor(arrow/compute/exprs)!: remove deprecated Substrait anchor APIs #1627 and the removal of the old package for whenever the next major release happens, with no deadline attached.

That takes the end-of-year pressure off. The part I still need to think through is the release process for a second module, since it would need its own tags and votes.

@Sunyue the builder APIs you listed (WrapLiteral, Cast, CallScalar, BuildExpr) would stay where they are in arrow/compute/exprs until the next major release, and the new module would carry the same ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants