Repository navigation
Conversation
|
happy to move forward with this once we have a stable tag instead of the alpha one. |
|
Why does substrait-go do so many breaking changes and major version updates? 😦 I'm unsure on this because of the breaking changes to public APIs that this is doing (the change from |
For this PR, we just published the first official v9 release: https://pkg.go.dev/github.com/substrait-io/substrait-go/v9 @kadinrabo was checking that the alpha build compiled correctly for arrow-go before we published it. He's updated this to use the official release.
Honestly, because we accidentally published the first v1 release before the library should have been considered stable, and we never put the cat back in the bag. If the constant major updates are becoming a problem, we can consider starting fresh with a new name to limit these issues in the future.
Would it make sense to add the aliases in this PR if this is going to be breaking already? It would probably be good the decouple the arrow-go API from the substrait-go one. |
|
|
||
| type substraitToArrow = func(*expr.ScalarFunction, []compute.Datum) (fname string, args []compute.Datum, opts compute.FunctionOptions, err error) | ||
| type arrowToSubstrait = func(fname string) (extensions.ID, []*types.FunctionOption, error) | ||
| type arrowToSubstrait = func(fname string) (extensions.ExtensionID, []*types.FunctionOption, error) |
There was a problem hiding this comment.
Should this be extension.FunctionID?
This applies here and to other places where the tigher ID bound would make sense.
| } | ||
|
|
||
| func (e *extensionSet) EncodeTypeVariation(dt arrow.DataType) (extensions.ID, uint32, bool) { | ||
| func (e *extensionSet) EncodeTypeVariation(dt arrow.DataType) (extensions.ExtensionID, uint32, bool) { |
There was a problem hiding this comment.
Should this return an extensions.TypeVariationID?
| } | ||
|
|
||
| func (e *extensionSet) EncodeType(dt arrow.DataType) (extensions.ID, uint32, bool) { | ||
| func (e *extensionSet) EncodeType(dt arrow.DataType) (extensions.ExtensionID, uint32, bool) { |
|
|
||
| func (e *extensionSet) DecodeTypeArrow(anchor uint32) (extensions.ID, arrow.DataType, bool) { | ||
| id, ok := e.DecodeType(anchor) | ||
| func (e *extensionSet) DecodeTypeArrow(anchor uint32) (extensions.ExtensionID, arrow.DataType, bool) { |
There was a problem hiding this comment.
I was going to say that this should return a TypeID, but actually it can also return a TypeVariationID based on the code. That actually points to a potential issue.
The given anchor could be both a valid Type or TypeVariation extension as they are numbered indepently, but if both are set we would only ever return the TypeID.
|
Hi @zeroshade! I see that changes to How do you generally decide when to make a major release of On our side, we have internal work blocked on removing |
|
Hey @benbellick, I've been trying to avoid major version releases as much as possible. The ideal scenario here would be to not have any public API breaks. But if it's impossible to avoid that, then let's look to insulate the public arrow-go API from future substrait-go breaks. We can do the major version release if we can ensure that we'll avoid having to do more major versions for future substrait changes. |
|
Hi @zeroshade! I agree that removing First, I think we should reduce the exposure of Beyond that, I see a few options:
We'd want to make any necessary breaking changes together in one major release, with a plan to avoid repeating this in the future. After that, adding execution support for newer Substrait versions wouldn't need to break existing callers. What do you think? Any preference between those three? I think 1 is the simplest, though I'm not sure how much code we'd need to duplicate for each new major version. It would also mean keeping dependencies on multiple |
|
Hi, below are the arrow-go API that we are using which takes or returns substrait's types, please consider not removing those: |
I like this idea, though perhaps we can come up with a better name than just tacking
This might be the better future-looking option though, though we wouldn't move the existing code. Instead we'd deprecate the current ones (to be removed at the next major version release) but this would enable better APIs and avoid requiring major version releases for Arrow itself.
While this would make development harder, the Go compiler should be able to drop the unused substrait-go major versions if a dependency is only using one of them instead of both. It might be best to start with option 1 so we can solve the current issue, and then start working on Option 2 and creating a separately versioned Arrow/Substrait module |
|
@zeroshade Great! I will work with @kadinrabo to prep a few PRs (all of which will cause breaking changes) to merge before the next major release. These would reduce the exposure of the If you are interested in the module splitting btw, that is exactly what we did inside of |
I thought we were just going to do the additive methods and not do the breaking changes? i.e. option 1, keeping the existing functions and adding new functions for v9 so users can upgrade when ready and we avoid breaking changes for now? |
|
@zeroshade Ah I may have misunderstood. For us, the primary motivation of this work is to remove the dependence on The suggestion above was meant to be about preventing future changes to the However, I think the first thing to settle is whether we can make a breaking change to remove the dependency on |
|
Is the goal for removing the dependency of Or is the goal something else? My preference is to see if there's a way to avoid the breaking change now if possible, but if we can't then I would say we can prep the changes and target the breaking change for release at the end of the year. |
|
@zeroshade This actually isn't about binary size. We have a large monorepo that depends on These protobuf updates often include breaking changes to the generated Go API, so we can't simply override the transitive dependency in our monorepo. We need the libraries using it to be updated as well. That can mean:
We can't use different versions of the protobuf-dependent libraries side by side, because their generated definitions would conflict in protobuf's global registry. With E.g. this is why I updated |
|
gotcha, then yea. we can target doing a major version release at the end of the year and prep the changes to make it so future updates won't require more major version releases from arrow-go. |
|
Great, thanks so much :) I don't want to have to cause a breaking change but I appreciate your flexibility here. This will help us a lot! |
|
I made an issue to track this work: #1622 We will start working on this. |
|
@zeroshade Here’s an example of one way we could change the public API to reduce the impact of future I’m not really sure what the strategy is for deprecations here generally, but I thought it would make sense to make this a stack where the first PR just marks the APIs as deprecated and the second removes them. |
|
Hey @zeroshade, small update! I wanted to see what it would look like to maintain multiple versions of the API on different underlying
(These are AI-generated and drafts, so I'm not asking for a real review here or anything, just sharing the general shape of the solutions.) As you can see, they both require quite a bit of code 😅. Even if we reduce the scope of the expr API, the API surface is still quite large, and I think it will be a relatively large maintenance burden for Arrow maintainers. Instead, I took a stab at generating a PR that breaks this out into a separate module (benbellick#7), which to me seems a lot simpler ultimately. Like you suggested, it deprecates the existing package rather than moving it, and bumping the new module to I'm unsure whether this is a reasonable goal to achieve before EOY for the next major release. Do you have any opinion on how we should move forward? Thanks a lot 🙇 |
|
Thanks for putting those drafts together, that makes the trade-off a lot clearer. I agree the side-by-side versions are too much code to maintain, so let's go with the separate module. One thing I want to check before we commit to a major release: I don't think the separate module needs one. Go only compiles packages that are actually imported, so if the v9 code lives in the new module and If that works, the plan would be:
That takes the end-of-year pressure off. The part I still need to think through is the release process for a second module, since it would need its own tags and votes. @Sunyue the builder APIs you listed ( |
Rationale for this change
Upgrade substrait-go to the latest major version (v9). I'm visiting from @substrait-io and want to bump this so I can update the dependency in
iceberg-go.v9 finishes moving protobuf out of substrait-go core, so
substrait-protobuf/godrops out of arrow-go entirely.What changes are included in this PR?
github.com/substrait-io/substrait-gofrom v8.1.1 to v9.0.0, and import paths fromv8tov9acrossarrow/compute/exprsgithub.com/substrait-io/substrait-protobuf/gov0.85.0, no longer reachableextensions.IDis nowextensions.ExtensionID, with conversions where v9 asks forTypeID,FunctionIDorTypeVariationIDtypes.Behavior*became thetypes.CastFailBehavior*enum, andtypes.PrecisionTimestampis a plain structAre these changes tested?
Existing tests pass (
go test ./arrow/compute/...). Also verifiediceberg-gobuilds and tests green against this branch.Are there any user-facing changes?
Users importing
arrow/compute/exprswill transitively depend on substrait-go v9 instead of v8, and no longer get substrait-protobuf. Exported signatures takingextensions.IDnow takeextensions.ExtensionID.