You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[C++] Expose Arrow's *FromJson, Assertion and Random generator helper functions #33538
Helper functions would be useful when testing in projects that use Arrow. BatchesWithSchema, MakeBasicBatches, Finishes and ResultWith could be considered as well.
Antoine Pitrou / @pitrou:
Definitely not. These are functions generating ad hoc data tailored for specific tests, with little consistency.
We could expose the Random generation class, though, possibly together with some API cleanup.
Weston Pace / @westonpace: MakeBasicBatches I agree is a definite no. The new source types being added in ARROW-17610 should remove any real need for BatchesWithSchema so I think that is a no too.
I agree the random data generation could be quite useful. I think it would also be quite interesting to expose random data generation as an exec node but probably shouldn't add more work :)
Rok Mihevc / @rok:
Random array generators would indeed be useful.
How about macros from gtest_util.h such as ASSERT_OK_AND_ASSIGN? These macros would potentially be useful when testing another library with Arrow data.
Yaron Gvili / @rtpsw:
The discussion went beyond the current subject of \*FromJSON - let's update the subject or split the discussion.
I agree that MakeBasicBatches is not needed. While I'm less sure about whether BatchesWithSchema is indeed not needed, I'll take your opinion. Regarding random data generation, I'd prioritize having the functions; the node could come later. Regarding the assertion macros, they are specific to gtest and I'm wondering whether we could make them flexible and less opinionated - that would be a separate issue. Note that I have my own subset of these macros specific to Catch2, so this flexibility can be obtained.
Assertion helpers (from arrow/testing/gtest_util.h), both macros (e.g., ASSERT_OK_AND_ASSIGN) and functions (e.g., AssertArraysEqual)
Random generators (from arrow/testing/random.h)
The main question I have is whether we want to keep the above in libarrow_testing or do the necessary refactor to include them in libarrow or an existing/new target lib. I think the former makes more sense due to the dependence on gtest (unless we abstract them as @rtpsw suggests). Any advice here would be helpful since I'm not overly familiar with the conventions here.
PS: @rok can you update the issue title to cover the new scope?
While working on this, it came up that test helpers like arrow::Finishes and arrow::ResultWith (from arrow/testing/matchers.h) might also be useful if they were lifted up into the public API. @rok@pitrou@westonpace any thoughts? I can file a separate issue if so.
changed the title [-][C++] Expose Arrow's testing helper functions[/-][+][C++] Expose Arrow's *FromJson, Assertion and Random generator helper functions[/+]on Feb 8, 2023
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
Activity
asfimport commentedon Nov 21, 2022
Rok Mihevc / @rok:
*FromJSON functions seem clear cut. How about also adding BatchesWithSchema and MakeBasicBatches. @pitrou @westonpace
asfimport commentedon Nov 21, 2022
Antoine Pitrou / @pitrou:
Definitely not. These are functions generating ad hoc data tailored for specific tests, with little consistency.
We could expose the Random generation class, though, possibly together with some API cleanup.
asfimport commentedon Nov 21, 2022
Li Jin / @icexelloss:
To clarify, do you mean the {Array,{{Exec,Record}Batch}FromJSON or BatchesWithSchema/MakeBasicBatches
asfimport commentedon Nov 21, 2022
Rok Mihevc / @rok:
I assume the comment is regarding BatchesWithSchema and MakeBasicBatches.
asfimport commentedon Nov 21, 2022
Antoine Pitrou / @pitrou:
Yes, this is what I meant. Sorry for the imprecision.
asfimport commentedon Nov 21, 2022
Weston Pace / @westonpace:
MakeBasicBatchesI agree is a definite no. The new source types being added in ARROW-17610 should remove any real need forBatchesWithSchemaso I think that is a no too.I agree the random data generation could be quite useful. I think it would also be quite interesting to expose random data generation as an exec node but probably shouldn't add more work :)
asfimport commentedon Nov 30, 2022
Rok Mihevc / @rok:
Random array generators would indeed be useful.
How about macros from gtest_util.h such as ASSERT_OK_AND_ASSIGN? These macros would potentially be useful when testing another library with Arrow data.
asfimport commentedon Nov 30, 2022
Antoine Pitrou / @pitrou:
We would have to prefix those macros with
ARROW_.asfimport commentedon Dec 15, 2022
Yaron Gvili / @rtpsw:
The discussion went beyond the current subject of
\*FromJSON- let's update the subject or split the discussion.I agree that
MakeBasicBatchesis not needed. While I'm less sure about whetherBatchesWithSchemais indeed not needed, I'll take your opinion. Regarding random data generation, I'd prioritize having the functions; the node could come later. Regarding the assertion macros, they are specific togtestand I'm wondering whether we could make them flexible and less opinionated - that would be a separate issue. Note that I have my own subset of these macros specific toCatch2, so this flexibility can be obtained.amoeba commentedon Jan 11, 2023
Just catching up here,
It looks like there's rough consensus around what we'd like to promote to the public API:
arrow/testing/gtest_util.h)arrow/testing/gtest_util.h), both macros (e.g.,ASSERT_OK_AND_ASSIGN) and functions (e.g.,AssertArraysEqual)arrow/testing/random.h)The main question I have is whether we want to keep the above in
libarrow_testingor do the necessary refactor to include them inlibarrowor an existing/new target lib. I think the former makes more sense due to the dependence on gtest (unless we abstract them as @rtpsw suggests). Any advice here would be helpful since I'm not overly familiar with the conventions here.PS: @rok can you update the issue title to cover the new scope?
[-][C++] Expose *FromJSON helpers[/-][+][C++] Expose Arrow's testing helper functions[/+]amoeba commentedon Jan 13, 2023
take
amoeba commentedon Feb 7, 2023
While working on this, it came up that test helpers like
arrow::Finishesandarrow::ResultWith(fromarrow/testing/matchers.h) might also be useful if they were lifted up into the public API. @rok @pitrou @westonpace any thoughts? I can file a separate issue if so.westonpace commentedon Feb 8, 2023
Yes, those seem generally useful for anyone using the fluent API.
westonpace commentedon Feb 8, 2023
Please do go ahead and file an issue.
[-][C++] Expose Arrow's testing helper functions[/-][+][C++] Expose Arrow's *FromJson, Assertion and Random generator helper functions[/+]thisisnic commentedon Feb 18, 2026
This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.
rok commentedon Nov 13, 2026
*FromJsonhelpers were exposed as public (#46180). I believe random generators were not.pitrou commentedon Nov 13, 2026
I still believe at least a subset of random generation functionality could be exposed (including in Python).
rok commentedon Nov 13, 2026
Agreed. Do these cover it? #47255, #47288
rok commentedon Nov 13, 2026
Then there are also #33181 #38748