The direction of Embulk

[kou]

This is extracted from #42 .

ClearCode proposes the following:

• Make the current Embulk releasable as 1.0.0
  • We know that EEP-0008 still has remaining items from what we were trying to do with the 0.11→1.0.0 transition. However, since 0.11 is already stable and usable, we will make it 1.0.0 with minimal changes to avoid breaking compatibility.
  • Making it 1.0.0 is for marketing purposes only, to appeal that it is stable. (Maybe it will make earning money easier?)
  • The minimal changes are changes around CI and release; we will touch the implementation as little as possible.
  • Keep it usable on environments that are not EOL.
  • We may address the remaining items after the 1.0.0 release, but we will decide based on the users who provide money and/or development resources.
• Make the plugins that users who attended Embulk users meetup used also releasable as 1.0.0
  • If possible, we will also take over archived repositories and tidy up their CI and release setup.

[kou]

Can we update the Embulk SPI version to v1.0 too...? Does it break existing plugins...?

[hiroyuki-sato]

I think we need to discuss the proposed approach, but I personally think it would be better to remove msgpack from embulk-spi 1.0 as originally planned.

Reason 1

We spent quite a lot of time defining the milestones toward Embulk v1.0, and there are statements in several places saying that msgpack will be removed after "Embulk v1.0".

So, if we release embulk-spi v1.0 without removing msgpack, I think we would need to update those statements in several places, at least if we want to keep everything consistent.

There may be a question of whether we really need to update all of them. However, I think it would be better to spend our time on other work rather than on those updates.

https://github.com/embulk/embulk/blob/master/docs/eeps/eep-0002.md#at-some-point-after-embulk-v10

https://dev.embulk.org/topics/get-ready-for-v0.11-and-v1.0-updated.html

Cleanup dependencies: SPI

Note that Embulk SPI versions will not align with Embulk core versions. The Embulk SPI may stay 0.* for a while even after the Embulk core v1.0 is released. It would go 1.0 when the Embulk SPI drops some deprecated things, such as the dependency on org.msgpack:msgpack-core.

https://github.com/embulk/embulk-spi/blob/main/src/main/java/org/embulk/spi/json/JsonString.java#L180-L182

Reason2

There do not seem to be many plugins using embulk-spi v0.11 in the first place.
It has been about two years since Embulk entered maintenance mode, and there do not seem to be many actively maintained plugins, at least among the publicly available ones.

output-kafka、{output|output}-iceberg are using embulk-spi v0.11

https://github.com/joker1007/embulk-output-kafka/blob/master/build.gradle#L55
https://github.com/shin1103/embulk-output-iceberg/blob/main/build.gradle#L65
https://github.com/shin1103/embulk-input-iceberg/blob/main/build.gradle#L66

Most of the others are still using v0.10. For example, I checked three trocco repositories that have had recent changes, and they are using v0.10.

https://github.com/trocco-io/embulk-output-bigquery_java/blob/master/build.gradle#L35
https://github.com/trocco-io/embulk-output-databricks/blob/main/build.gradle#L30
https://github.com/trocco-io/embulk-input-kintone/blob/master/build.gradle#L26

So, if we are going to bump the version anyway, I don't think it would be a problem to include the changes we have been planning, such as removing msgpack and adding the JsonArray comparison changes.

Side node.

There may be a concern that embulk-spi is considered unstable because its version is still below 1.0, and that we should release 1.0 as soon as possible.

However, I don't think many users care very much about the embulk-spi version itself. From a user's point of view, I think things like being able to run Embulk on Java 11 or later, or getting a new version of a plugin, are more valuable than embulk-spi becoming 1.0.

[kou]

There may be a concern that embulk-spi is considered unstable because its version is still below 1.0, and that we should release 1.0 as soon as possible.

I agree that some users value things that work over specifications. However, at the recent meetup, there were also users who value being able to keep Embulk running with as little maintenance cost as possible. For example, some users said that their version upgrades have stalled because they cannot secure enough resources for verification. I think that the more Embulk can guarantee that there are no incompatible changes, the more we may be able to reduce maintenance costs, including verification costs.

EEP-0008 defines the versioning strategies to follow v1.0:

https://github.com/embulk/embulk/blob/master/docs/eeps/eep-0008.md#versioning-strategies-to-follow-v10

It clearly states that "incompatible changes" are indicated by the MAJOR version (although the removal of msgpack-core is mentioned as an exception).

By making the SPI 1.0, Embulk can guarantee that there are no incompatible changes during 1.y. This guarantees that plugins will not stop working when users update the Embulk core, so I think users can reduce their verification costs for updates.

EEP-0008 was written before Embulk entered maintenance mode, and I think the situation is different now that Embulk is in maintenance mode. Can we reconsider it?

EEP-0001 says the following:

https://github.com/embulk/embulk/blob/master/docs/eeps/eep-0001.md#status-of-an-eep

Note that any transition can happen between these status except for "Replaced" and "Active".

Since EEP-0008 is "Accepted", I think it is possible to reconsider it.

However, I don't think many users care very much about the embulk-spi version itself. From a user's point of view, I think things like being able to run Embulk on Java 11 or later, or getting a new version of a plugin, are more valuable than embulk-spi becoming 1.0.

I think there are users who need the removal of msgpack, but there are also existing users who would rather keep running Embulk with as few incompatibilities as possible than have msgpack removed. I think it is valuable for those existing users to be able to keep using Embulk longer with as little maintenance cost as possible.

I expect that users will be divided into two groups: users who want to keep using the current Embulk 0.11/SPI 0.11, and users who want to use newer Java and related libraries even if some compatibility is sacrificed. For the former group, I want to make both Embulk and the SPI 1.0 while keeping them as close to the current state as possible, guarantee that no incompatible changes will be introduced, and continue maintaining them. For the latter group, I want to make both Embulk and the SPI 2.0, allow incompatible changes, and continue maintaining them. Since their needs are different, I think we will maintain them in separate codebases (branches).

Reason 1

I'll take on the update work.

Reason 2

Just to be sure, let me confirm your intention. Are you saying that it is OK to introduce incompatible changes because many plugins use SPI 0.10? If so, I would rather avoid introducing incompatible changes so that the many plugins built with SPI 0.10 can keep working.

By the way, is it correct that Embulk 0.11 already accepts plugins built with both SPI 0.10 and 0.11? If so, I think it means that what I was wondering about, whether it is technically possible for Embulk 1.0 to accept plugins built with SPI 0.10, 0.11, and 1.0, is possible.

[hiroyuki-sato]

Target users?

Again, I am not strongly against the idea of releasing embulk-spi 1.0 as it is now in v0.11.

I am only saying that, before doing that, we should check the EEPs and think about whether we really need to do it.

For example:

• Which users want to keep compatibility with embulk-spi v0.11?
• Which plugins will have a problem if the Value feature of msgpack-java is removed?

I think the discussion will be more useful if we have clear answers to these questions.

My answer to these questions is that there may be no such users or plugins.

embulk-spi v1.0

Next, I think it would be good to get comments from @dmikurube again about the original plan for releasing embulk-spi 1.0.

Option 1

• Bump the version to embulk-spi 1.0-SNAPSHOT.
• Remove msgpack-java.
• Add the fix for JsonArray equality.
• Release embulk-spi 1.0.

At least, this does not affect the existing embulk-spi v0.11.

Option 2

• Remove msgpack-java.
• Add the fix for JsonArray equality.
• Release a new version of embulk-spi 0.11. (What would the version number be? embulk-spi does not use patch versions.)

For example, if we release something like embulk-spi v0.11.1, it may affect existing users.

However, embulk-spi has a rule that it does not use patch versions. Please see EEP-008.

So, as I understand it, we are not supposed to release a version like v0.11.1.

v0.9 user

I think we should make this point clear.

at the recent meetup, there were also users who value being able to keep Embulk running with as little maintenance cost as possible

About this point, I was also at the meetup, so I think I understand the situation.

They are using Embulk v0.9, not Embulk v0.11.

In other words, they are not using embulk-spi. They are not the embulk-spi users that we are talking about here.

Supporting them with the code base would mean continuing to support Embulk v0.9 with things such as:

• Jackson 2.6.7
• JRuby 9.1.5.0
• google-api-client versions that are no longer supported

We thought about whether we could continue supporting these old versions, and our answer was no. That is why we started developing the Embulk v0.10 series and later released v0.11 as the stable version.

If we want to target these users, I think the question should be whether we can continue maintaining Embulk v0.9.

They also said that they use the following plugins:

• embulk-input-jdbc
• embulk-output-bigquery
• embulk-filter-proc
• embulk-parser-csv

As far as I know, these plugins are compatible with both Embulk v0.9 and v0.11. So, if they upgrade Embulk, I think they should still work.

One of the participants explained why they do not upgrade:

It is difficult to check that nothing will change.

https://x.com/onunu_/status/2100499475466973623?s=20

I don't think what they need is to keep Embulk v0.9 forever. I think what they need is help moving to a newer version while checking compatibility together.

In that case, the target version would probably be either Embulk v0.11 or Embulk 1.0, which has not been released yet.

I think the important point is to keep compatibility when they make that upgrade.

Embulk SPI v0.10 and v0.11

By the way, is it correct that Embulk 0.11 already accepts plugins built with both SPI 0.10 and 0.11? If so, I think it means that what I was wondering about, whether it is technically possible for Embulk 1.0 to accept plugins built with SPI 0.10, 0.11, and 1.0, is possible.

Java plugins that work with Embulk v0.11 use embulk-spi v0.10 or v0.11. They cannot work without it.
embulk-spi v0.10 and v0.11 は

For this point, I don't want to give incorrect information, so I would like to wait for @dmikurube comment.

However, if I remember correctly:

https://github.com/embulk/embulk/releases/tag/v0.11.0

v0.10.50 = Embulk v0.11.0.

[dmikurube]

It seems to me that you/we are discussing different things, because several different topics are mixed in this discussion. So first, let me share my view on each topic separately.

About versioning

For embulk/embulk-spi#6 and the JsonDouble/JsonLong topic, I intended to release the fixed SPI as 0.12, and go on for a while. You can see that such an SPI version like 0.12 has been addressed in EEP-8. It is true that the fixes are "incompatible" strictly speaking, but I think those fixes are in an acceptably "compatible" range. Furthermore, these are still 0 versions.

At the same time, note that the Embulk plugin SPI version and the Embulk core version are not aligned. As you can see also in EEP-8, the Embulk core v1.0.* is planned to be coupled with the Embulk SPI 0.11 (or 0.12).

Those versions are designed not to be aligned. For example, the Embulk core v3.0.0 can be released based on the Embulk SPI v1.1, v1.2 or v2.0. Compatibility breakage in SPI means really a hard change. So, I expect that upgrading SPI's MAJOR version wouldn't happen so casually, only except for v1.0 removing msgpack-core. (Another SPI MAJOR version bump-up may happen when the SPI starts to be built on newer Java? It's TBD.)

As a preparation before the Embulk core v1.0, we may still release the new Embulk core v0.11.6 with the Embulk SPI v0.12. That (core v0.11.* and SPI v0.12) does not match EEP-8, but as @kou said, updating the EEP is acceptable, and this is in an acceptable range, too.

About msgpack-core

The removal of msgpack-core must go ahead. Withdrawing the removal is unacceptable.

We decided this removal because we expected that keeping msgpack-core would be a big pain in the future, both for us and for users, and we still expect so. Keeping this means that it can cause a much more catastrophic compatibility problem due to the future status of msgpack-core. Removing msgpack-core is planned to mitigate such a future pain, and we prepared for the removal during the v0.10.* series, which was the unstable "development" series.

About the compatibility between v0.10 v0.9 and v0.11

The v0.10 was an unstable "development" series to migrate the v0.9-style to the v0.11-style. In short, plugins that are based on later versions of Embulk v0.10 (esp. v0.10.38 or later? -- I have to recall it) have higher possibility to work with Embulk v0.11, but nothing is guaranteed about v0.10-based plugins.

Please read through the EEPs and understand the history first. Please. That's the starting point for any discussion.

In terms of v0.9 vs. v0.11, yes and no, depending on the combination and the plugin.

If the v0.11-style plugin depends on much older libraries that are equal to Embulk v0.9 (e.g. Jackson 2.6), the v0.11-style plugin still works on Embulk v0.9.25. Otherwise, it may or may not work.

The v0.9-based plugins MAY work on early v0.11.* releases (e.g. v0.11.0), but in general, they are not expected to work on Embulk v0.11. This is because of library version mismatch. It is very difficult to expect what happens when a plugin can observe both an old library and a new library in its classloader namespace.

The "compatibility" was really fragile until v0.9 (and v0.10, the development series) and that was the reason why we had to make the change for v0.11.

---

And then,

The position of ClearCode in Embulk, as of now

Every design decision had its background, the context, and the history. Please remember that you're totally new in the Embulk project, in both respects as a user and a developer. At this point, prioritize learning the background, the context, and the history, not to prioritize proposing your own ideas and wants. Keep it in mind that you've not established any trust in this project so far. You are neither an owner nor a maintainer of the project (yet).

As a candidate to take over the project, your business needs are something we may consider, but they are not the only thing our decisions are based on.

Furthermore, please do not block us. We are and have been doing things that we thought were necessary to keep this project going. We didn't finish "things that we thought were necessary" just because we decided to let Embulk into the "maintenance mode". If it's back from the maintenance mode, they still need to be done.

For questions, we can answer them straightforward. For "proposals", especially for one-sided context-less proposals, we have to estimate your motivation and background, and we have to guess what needs to be explained to you, by ourselves. Such a "proposal" blocks everything. It would delay the "take over" as a result, and also, it demotivates us.

If you want to "take full control" immediately, not to "take over", you could just fork it. I won't stop you from forking. I won't help with it, though.

To be honest, I am starting to feel that it's not very realistic for a team that has been neither a user nor a developer to "take over" the project.

[kou]

I understand. Let's pause this discussion until ClearCode has carefully read through the existing EEPs.