Repository navigation
ENH: add quantile #340
Description
It was decided not to add quantile to array-api (data-apis/array-api#795), partly because scipy added array API support to their quantile (scipy/scipy#22352) and partly due to complications (many methods, weight support, nan support).
This RFC is mostly motivated by usage in scikit-learn (as seen from data-apis/array-api#795 (comment) its not used much in other packages).
Although scipy does have an array API quantile, due to our min scipy version, we are currently not able to use it (e.g., scikit-learn/scikit-learn#31671).
More generally, it seems to be a commonly enough used function (which is implemented many array libraries) to warrant considering adding it to here, and having a consistent, single source.
Not sure who would be interested but maybe:
cc @betatim @ogrisel @lorentzenchr (and @rgommers who suggested it)
Activity
lorentzenchr commentedon Jul 10, 2025
For quantiles, you can have look at my blog post https://lorentzen.ch/index.php/2023/02/11/quantiles-and-their-estimation/.
For weighted quantiles, I did not find any peer reviewed reference. Therefore, the only certain connection I find is via the (weighted) empirical CDF.
lucascolley commentedon Jul 10, 2025
There was scipy/scipy#22794 (comment), we can ping Matt if that would be helpful. I can't find the further discussions that I can remember right now.
lucyleeow commentedon Jul 14, 2025
Sorry to clarify, does that mean we should normalize weights (to
n) ? Have looked into how normalizing/not normalizing affects weighted quantile results?lorentzenchr commentedon Jul 14, 2025
For a weighted CDF, the scale of the weights does not matter, only cumsum/total sum matters. Weights are therefore usually rescaled such that
np.sum(weights) == 1.You then find this behaviour (independence of scale of weights / invariance under multiplicative constant of weights) in
np.quantile(.., weights=.., method="inverted_cdf").lucyleeow commentedon Jul 14, 2025
Excuse my ignorance, but in regard to the different types of weights (frequency, sampling, analytical), how 'should' they be handled differently with regards to calculating weighted quantile then..?
lorentzenchr commentedon Jul 14, 2025
I find the sheer mention of different types of weights very unfortunate in the context of quantile estimation. The already linked post Weights in statistics explains pretty well that if you estimate the mean, different assumptions lead to different estimates of the variance of the estimated mean.
For quantile estimation, we are not talking about an estimation of the variance of the estimated quantile.
ogrisel commentedon Oct 3, 2025
Related: adding
argpartitionsupport toarray-api-extrashould make it possible to implementO(n)complexity weighted percentiles with array API support:partitionandargpartitionfunctions #449cakedev0 commentedon Oct 3, 2025
Hum not just weighted percentile: for non-weighted percentile you need partition too for a O(n) implementation (I think?).
Edit: Typically, current scipy implementation of quantile is based on
xp.sort: https://github.com/scipy/scipy/blob/b1296b9b4393e251511fe8fdd3e58c22a1124899/scipy/stats/_quantile.py#L57ogrisel commentedon Oct 6, 2025
Maybe we can first finalize the PR and its review in scikit-learn and then decide where to contribute this code upstream? Both
array-api-extraand SciPy would naturally make sense.lucyleeow commentedon Oct 21, 2025
@cakedev0 following on the discussion in #341 here, as I think it may be a better fit.
It may have been mentioned in #341 but @betatim and I have discussed maybe only (initially) adding one quantile method, for the reasons;
quantilearray-api#795 (comment), quantile is not used much in other packages) but we are interested in having an array API compliant quantile that supports weights and (later) NaN's . Would be useful to get input from other packages though.WDYT @ogrisel ?
cakedev0 commentedon Oct 21, 2025
Thank you for this detailed overview of the situation! It's incredibly helpful!
Based on that, here are the steps I propose:
method="linear"and propagate NaNs. This way, we can delegate for all major array libs (including dask and jax).inverted_cdfandaverage_inverted_cdfand weights. Before doing that, I think it would maybe make sense to add the support for methodaverage_inverted_cdfwith weights in numpy.And from there, we can either add the support for more methods and/or the support for
nan_policy="raise"|"omit".Note:
Indeed, you have this decorator above scipy's quantile function:
For the first step, we'll delegate anyway so we don't care. And for weights support, we'll see, no
take_along_axisseems hard to deal with.ogrisel commentedon Oct 21, 2025
Note:
daskis focused on very large virtual arrays that span multiple nodes on a cluster of machines. Therefore, dask developers might be reluctant to support operations that are fundamentally not scalable, like sorting arrays and doing big data shuffle.For such large arrays, people would probably rely on approximate quantiles (either via subsampling or dedicated sketching algorithms) instead of computing exact quantiles.
lucyleeow commentedon Oct 21, 2025
Not sure if delegating is possible, it relies on all packages having consistent behaviour, which we'd need to check. Just taking a quick look at tensorflow tfp.stats.quantiles - and they take
num_quantiles, instead ofq.Also it seems that scipy and numpy
axisbehaviour also differs: scipy/scipy#23832cakedev0 commentedon Oct 21, 2025
I think tensorflow is not a supported backend of
array-api-extra(see https://github.com/data-apis/array-api-extra/blob/main/src/array_api_extra/_lib/_backends.py).I already wrote the delegation code for numpy, cupy, torch, dask and jax, and based on the docs, it should be fine (but we'll see if the tests pass).
Ah, thanks for the pointer, I've been scratching my head on this broadcasting thing for a while, this should help ^^
lucyleeow commentedon Oct 21, 2025
Ah interesting, I'm sure they had a good reason. Tensorflow is listed as a stakeholder (in array API standard): https://data-apis.org/array-api/latest/purpose_and_scope.html#stakeholders
If we aren't interested in tensorflow, then seems reasonable, depending on how much tweaking is required for each individual package (as this may be difficult to maintain going forward). Do you want to open a draft PR and we can take a look?
lucyleeow commentedon Oct 21, 2025
Out of interest, why exclude tensorflow @lucascolley ?
lucascolley commentedon Oct 21, 2025
I don't think we are concerned with supporting tensorflow in array-api-extra at least, since it isn't supported by array-api-compat: https://data-apis.org/array-api-compat/supported-array-libraries.html. I'm missing historical context for the relationship between tensorflow and the consortium.
lucyleeow commentedon Oct 21, 2025
For a general guide to broadcasting, this is nice: https://quansight-labs.github.io/ndindex/indexing-guide/other-topics.html#broadcasting
cakedev0 commentedon Oct 21, 2025
Sorry, what I meant by "broadcasting thing" was not broadcasting in general but what was done in numpy/scipy for the quantile function related to broadcasting.
In numpy, there is an error:
ValueError: q must be a scalar or 1d. (but it's a bit weird, because it works with 2D q, see issue above).So I propose to support only scalar and 1d for
q. This should be enough for what is needed in scikit-learn right? And it would make the implementation quite easier I think.lucascolley commentedon Oct 21, 2025
yes, a good rule of thumb is to only implement a sensible subset of the APIs based on what we need, for more complex functions.
cakedev0 commentedon Oct 21, 2025
Ok great! I should be able to make my draft PR readable and tested soon 👍
cakedev0 commentedon Oct 22, 2025
I'm extending the scope of my PR. In the end, I prefer doing most of it in one-go to reduce context switchs (and I feel this will be true for reviewers too), because it's not so hard now that I have a clear understanding of the scope and the desired shapes/broadcasting behavior.
New scope:
"linear", "inverted_cdf", "averaged_inverted_cdf"nan_policy="omit"for weighted quantile. And usual behavior isnan_policy="propagte"(numpy's behavior for instance). So I might implement both. Or just always omit. What do you think?lucyleeow commentedon Oct 23, 2025
A thought (which may or may not be much of a concern): will it be odd that we only support 'linear' if the array library is one of the supported backends but 'average_inverted_cdf' (and not 'linear') otherwise?
I does mean when writing code, I probably wouldn't use 'linear' because it's not as widely supported as 'average_inverted_cdf'.
We could of course add support for 'linear', but in that case, we may as well add support for the other continuous methods as well.
But I am not 100% we have full agreement that limited support of method(s) would be the way to go, WDYT @betatim @ogrisel and maybe @lesteve
cakedev0 commentedon Oct 23, 2025
I'm not sure to understand what you mean. For me, the case "otherwise" is empty, because in my understanding "otherwise" means no
array-api-compatand hence no ways to get a workingxp.lucyleeow commentedon Oct 23, 2025
Ah I may have some mis-understanding. In array API extra, are the 'extra' functions ONLY for array libraries supported by array-api-compat?
I thought they were meant to be written general enough to support 'any' library that supports the standard?
cakedev0 commentedon Oct 23, 2025
Hum... My understanding is that any library that supports the standard can be trivially added to
array-api-compatso I think it's fair to assume that the set "libraries that support the standard but aren't inarray-api-compat" is empty.But that's just my understanding, so I'll let @lucascolley or someone else confirm.
cakedev0 commentedon Oct 23, 2025
They are. They are tested against
array_api_strictandarray_api_strictest(unless explicitly skipped, a few tests skip strictest, but none skip strict I think).