Skip to content

[C++] Expose Arrow's *FromJson, Assertion and Random generator helper functions #33538

Closed as not planned

Description

@asfimport

Helper functions would be useful when testing in projects that use Arrow. BatchesWithSchema, MakeBasicBatches, Finishes and ResultWith could be considered as well.

  • FromJSON helpers (from arrow/testing/gtest_util.h)
  • Assertion helpers (from arrow/testing/gtest_util.h), both macros (e.g., ASSERT_OK_AND_ASSIGN) and functions (e.g., AssertArraysEqual)
  • Random generators (from arrow/testing/random.h)

Reporter: Rok Mihevc / @rok
Assignee: Bryce Mecum / @amoeba
Watchers: Rok Mihevc / @rok

Note: This issue was originally created as ARROW-18371. Please see the migration documentation for further details.

Activity

  1. asfimport commented on Nov 21, 2022

    @asfimport
    Author

    Rok Mihevc / @rok:
    *FromJSON functions seem clear cut. How about also adding BatchesWithSchema and MakeBasicBatches. @pitrou @westonpace

  2. asfimport commented on Nov 21, 2022

    @asfimport
    Author

    Antoine Pitrou / @pitrou:
    Definitely not. These are functions generating ad hoc data tailored for specific tests, with little consistency.
    We could expose the Random generation class, though, possibly together with some API cleanup.

  3. asfimport commented on Nov 21, 2022

    @asfimport
    Author

    Li Jin / @icexelloss:

    Definitely not. These are functions generating ad hoc data tailored for specific tests, with little consistency.

    To clarify, do you mean the {Array,{{Exec,Record}Batch}FromJSON or BatchesWithSchema/MakeBasicBatches

  4. asfimport commented on Nov 21, 2022

    @asfimport
    Author

    Rok Mihevc / @rok:
    I assume the comment is regarding BatchesWithSchema and MakeBasicBatches.

  5. asfimport commented on Nov 21, 2022

    @asfimport
    Author

    Antoine Pitrou / @pitrou:

    I assume the comment is regarding BatchesWithSchema and MakeBasicBatches.

    Yes, this is what I meant. Sorry for the imprecision.

  6. asfimport commented on Nov 21, 2022

    @asfimport
    Author

    Weston Pace / @westonpace:
    MakeBasicBatches I agree is a definite no. The new source types being added in ARROW-17610 should remove any real need for BatchesWithSchema so I think that is a no too.

    I agree the random data generation could be quite useful. I think it would also be quite interesting to expose random data generation as an exec node but probably shouldn't add more work :)

  7. asfimport commented on Nov 30, 2022

    @asfimport
    Author

    Rok Mihevc / @rok:
    Random array generators would indeed be useful.
    How about macros from gtest_util.h such as ASSERT_OK_AND_ASSIGN? These macros would potentially be useful when testing another library with Arrow data.

  8. asfimport commented on Nov 30, 2022

    @asfimport
    Author

    Antoine Pitrou / @pitrou:
    We would have to prefix those macros with ARROW_.

  9. asfimport commented on Dec 15, 2022

    @asfimport
    Author

    Yaron Gvili / @rtpsw:
    The discussion went beyond the current subject of \*FromJSON - let's update the subject or split the discussion.

    I agree that MakeBasicBatches is not needed. While I'm less sure about whether BatchesWithSchema is indeed not needed, I'll take your opinion. Regarding random data generation, I'd prioritize having the functions; the node could come later. Regarding the assertion macros, they are specific to gtest and I'm wondering whether we could make them flexible and less opinionated - that would be a separate issue. Note that I have my own subset of these macros specific to Catch2, so this flexibility can be obtained.

  10. amoeba commented on Jan 11, 2023

    @amoeba
    Member

    Just catching up here,

    It looks like there's rough consensus around what we'd like to promote to the public API:

    • FromJSON helpers (from arrow/testing/gtest_util.h)
    • Assertion helpers (from arrow/testing/gtest_util.h), both macros (e.g., ASSERT_OK_AND_ASSIGN) and functions (e.g., AssertArraysEqual)
    • Random generators (from arrow/testing/random.h)

    The main question I have is whether we want to keep the above in libarrow_testing or do the necessary refactor to include them in libarrow or an existing/new target lib. I think the former makes more sense due to the dependence on gtest (unless we abstract them as @rtpsw suggests). Any advice here would be helpful since I'm not overly familiar with the conventions here.

    PS: @rok can you update the issue title to cover the new scope?

  11. changed the title [-][C++] Expose *FromJSON helpers[/-] [+][C++] Expose Arrow's testing helper functions[/+] on Jan 11, 2023
  12. amoeba commented on Jan 13, 2023

    @amoeba
    Member

    take

  13. amoeba commented on Feb 7, 2023

    @amoeba
    Member

    While working on this, it came up that test helpers like arrow::Finishes and arrow::ResultWith (from arrow/testing/matchers.h) might also be useful if they were lifted up into the public API. @rok @pitrou @westonpace any thoughts? I can file a separate issue if so.

  14. westonpace commented on Feb 8, 2023

    @westonpace
    Member

    Yes, those seem generally useful for anyone using the fluent API.

  15. westonpace commented on Feb 8, 2023

    @westonpace
    Member

    Please do go ahead and file an issue.

  16. changed the title [-][C++] Expose Arrow's testing helper functions[/-] [+][C++] Expose Arrow's *FromJson, Assertion and Random generator helper functions[/+] on Feb 8, 2023
  17. added
    Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
    on Feb 18, 2026
  18. thisisnic commented on Feb 18, 2026

    @thisisnic
    Member

    This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.

  19. removed
    Status: stale-warningIssues and PRs flagged as stale which are due to be closed if no indication otherwise
    on May 15, 2026
  20. rok commented on Nov 13, 2026

    @rok
    Member

    *FromJson helpers were exposed as public (#46180). I believe random generators were not.

  21. pitrou commented on Nov 13, 2026

    @pitrou
    Member

    I still believe at least a subset of random generation functionality could be exposed (including in Python).

  22. rok commented on Nov 13, 2026

    @rok
    Member

    Agreed. Do these cover it? #47255, #47288

  23. rok commented on Nov 13, 2026

    @rok
    Member

    Then there are also #33181 #38748

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions