You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Document the user-facing changes from the sample-based NDV tracking issue (pingcap/tidb#67449):
statistics.md: add the "Sampling-based NDV collection" section. It describes how TiDB selects the NDV rate, the NDVRATE option and its persistence, and the limitations.
system-variables.md: add tidb_analyze_sampled_ndv_threshold and tidb_analyze_store_batch_size.
SHOW ANALYZE STATUS and ANALYZE_STATUS: Job_info can show ndvrate.
statistics.md: correct the note on the WITH syntax in the automatic update section. It said that the automatic update does not use the options of a manual ANALYZE statement. This has not been correct since v5.4.0, which added the persistence of ANALYZE configurations.
All facts are checked against the current master code of TiDB and the design docs. The TiKV PRs do not add configuration items, so they need no TiKV doc changes.
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
The changes in this PR were primarily made by an AI agent on behalf of the PR author.
Do your changes match any of the following descriptions?
Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch
Summary by CodeRabbit
Documentation
Documented NDVRATE as an ANALYZE TABLE option for estimating distinct values, including sampling behavior, defaults, persistence, and compatibility considerations.
Updated ANALYZE syntax and analysis-status references to include NDVRATE and supported defaults.
Added reference information for tidb_analyze_sampled_ndv_threshold, which controls when NDV estimation uses sampling.
Documented tidb_analyze_store_batch_size and how it controls batching of ANALYZE tasks sent to TiKV.
pingcap/tidb#71520 and tikv/tikv#20106 let ANALYZE estimate NDV from a
sample of rows on large tables. pingcap/tidb#67608 and
pingcap/tidb#71699 add the NDVRATE option, and pingcap/tidb#72453 sets
the default threshold for new clusters.
Document the tidb_analyze_sampled_ndv_threshold variable, the NDVRATE
option, its persistence, and the limits of the feature.
This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign lilin90 for approval. For more information see the Code Review Process. Please ensure that each of them provides their approval before proceeding.
The full list of commands accepted by this bot can be found here.
Details
Needs approval from an approver in each of these files:
It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.
The documentation adds NDVRATE syntax and sampled NDV estimation rules, including related system variables, persistence, and status reporting. It also documents the ANALYZE Region task batch-size setting and its behavior.
Documents NDVRATE syntax and sampling rules, the tidb_analyze_sampled_ndv_threshold setting, persistence of the NDV rate, and its appearance in ANALYZE status information.
ANALYZE Region task batching
Layer / File(s)
Summary
Batch size setting and behavior statistics.md, system-variables.md, system-variable-reference.md
Documents tidb_analyze_store_batch_size, its defaults and scope, and how TiDB batches Region tasks in requests to TiKV.
The documentation is accurate in its main rules, but the sample-size sentence for very large tables can mislead readers about how many rows are sampled. Two minor wording and consistency points also remain open. None of these block merge, but the authors should be aware of them.
Pre-merge checks: 4
Check name
Status
Explanation
Title check
The title clearly summarizes the main documentation changes: sampling-based NDV collection and batched ANALYZE requests.
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Description check
The pull request description is complete and follows the repository template. It explains the documentation changes, identifies the affected master version, provides related code and design links, rec…
Full details: Description check
Explanation
The pull request description is complete and follows the repository template. It explains the documentation changes, identifies the affected master version, provides related code and design links, records AI involvement, and addresses the optional change categories.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.
The sampling-based NDV section said that the row count from the
statistics is "much smaller" than the row count in PD. TiDB uses the PD
row count when it is more than 5 times the row count from the
statistics, so give that condition.
The rule named the point of 20 times the threshold, which the reader
can calculate from the formula, and the sentence was hard to read. Say
only that the NDV sample has approximately the threshold number of
rows and that the NDV rate has a minimum of 0.05.
Reviewing files that changed from the base of the PR and between ea78a49 and b107c2f.
Files selected for processing: 1
statistics.md
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.
pingcap/tidb#72511 extends the rule of pingcap/tidb#71721. ANALYZE now
also uses the row count as the NDV for a clustered integer primary key,
and for a primary key or a composite unique index whose columns are all
NOT NULL. The rule does not apply to partial indexes.
List these cases in the limitations of sampling-based NDV collection.
The section explained how TiDB and TiKV implement the feature: the
design motivation, the name of the estimator, the FMSketch data, and
each case in which TiDB increases the NDV rate. A reader cannot use
these details. The caution to upgrade all instances first applies only
if you enable the feature during a rolling upgrade.
Keep the rules, the options, and the errors, warnings, and notes that a
user can see.
The description explained how TiKV runs and merges the Region tasks of
a batch. Keep what the variable controls, its benefit, and its cost.
Also remove the statement that this variable and tidb_store_batch_size
do not affect each other.
The rule said that the NDV sample has approximately the threshold
number of rows. This is not correct when the minimum NDV rate of 0.05
applies. In that case, the sample is 5% of the rows, which is more than
the threshold.
The note said that the automatic update does not record the options of
a manual ANALYZE statement. This has not been correct since v5.4.0,
which added the persistence of ANALYZE configurations. The note also
conflicts with the persisted NDVRATE value that the sampling-based NDV
section describes.
Say that the automatic update uses the options if the persistence
feature is enabled.
The persistence paragraph did not say what occurs without a persisted
NDVRATE value. TiDB persists only a value that an ANALYZE statement
specifies. In other cases, each ANALYZE operation calculates the NDV
rate from the current row count.
The section gave the calculation of the NDV rate in each ANALYZE
operation as the last sentence of the persistence paragraph. Put this
default at the start of the section, before the NDVRATE option.
Also say that TiDB persists the value that you specify, and that
WITH DEFAULT NDVRATE goes back to the calculated rate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
missing-translation-statusThis PR does not have translation status info.size/LDenotes a PR that changes 100-499 lines, ignoring generated files.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is changed, added, or deleted? (Required)
Document the user-facing changes from the sample-based NDV tracking issue (pingcap/tidb#67449):
statistics.md: add the "Sampling-based NDV collection" section. It describes how TiDB selects the NDV rate, theNDVRATEoption and its persistence, and the limitations.system-variables.md: addtidb_analyze_sampled_ndv_thresholdandtidb_analyze_store_batch_size.ANALYZEsynopsis: addNDVRATEand theWITH DEFAULT <option>forms (parser, planner, executor: support ANALYZE ... WITH DEFAULT <option> tidb#69956).SHOW ANALYZE STATUSandANALYZE_STATUS:Job_infocan showndvrate.statistics.md: correct the note on theWITHsyntax in the automatic update section. It said that the automatic update does not use the options of a manualANALYZEstatement. This has not been correct since v5.4.0, which added the persistence ofANALYZEconfigurations.All facts are checked against the current
mastercode of TiDB and the design docs. The TiKV PRs do not add configuration items, so they need no TiKV doc changes.Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit
NDVRATEas anANALYZE TABLEoption for estimating distinct values, including sampling behavior, defaults, persistence, and compatibility considerations.ANALYZEsyntax and analysis-status references to includeNDVRATEand supported defaults.tidb_analyze_sampled_ndv_threshold, which controls when NDV estimation uses sampling.tidb_analyze_store_batch_sizeand how it controls batching ofANALYZEtasks sent to TiKV.