Skip to content

statistics: document sampling-based NDV collection and batched ANALYZE requests - #24667

Open
0xPoe wants to merge 12 commits into
pingcap:masterfrom
0xPoe:docs-sampled-ndv
Open

0xPoe wants to merge 12 commits into
pingcap:masterfrom
0xPoe:docs-sampled-ndv

Conversation

@0xPoe

@0xPoe 0xPoe commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

What is changed, added, or deleted? (Required)

Document the user-facing changes from the sample-based NDV tracking issue (pingcap/tidb#67449):

  • statistics.md: add the "Sampling-based NDV collection" section. It describes how TiDB selects the NDV rate, the NDVRATE option and its persistence, and the limitations.
  • system-variables.md: add tidb_analyze_sampled_ndv_threshold and tidb_analyze_store_batch_size.
  • ANALYZE synopsis: add NDVRATE and the WITH DEFAULT <option> forms (parser, planner, executor: support ANALYZE ... WITH DEFAULT <option> tidb#69956).
  • SHOW ANALYZE STATUS and ANALYZE_STATUS: Job_info can show ndvrate.
  • statistics.md: correct the note on the WITH syntax in the automatic update section. It said that the automatic update does not use the options of a manual ANALYZE statement. This has not been correct since v5.4.0, which added the persistence of ANALYZE configurations.

All facts are checked against the current master code of TiDB and the design docs. The TiKV PRs do not add configuration items, so they need no TiKV doc changes.

Which TiDB version(s) do your changes apply to? (Required)

Tips for choosing the affected version(s):

By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.

For details, see tips for choosing the affected versions.

  • master (the latest development version)
  • v9.0 (TiDB 9.0 versions)
  • v8.5 (TiDB 8.5 versions)
  • v8.1 (TiDB 8.1 versions)
  • v7.5 (TiDB 7.5 versions)
  • v7.1 (TiDB 7.1 versions)
  • v6.5 (TiDB 6.5 versions)

What is the related PR or file link(s)?

AI agent involvement

  • The changes in this PR were primarily made by an AI agent on behalf of the PR author.

Do your changes match any of the following descriptions?

  • Delete files
  • Change aliases
  • Need modification after applied to another branch
  • Might cause conflicts after applied to another branch

Summary by CodeRabbit

  • Documentation
    • Documented NDVRATE as an ANALYZE TABLE option for estimating distinct values, including sampling behavior, defaults, persistence, and compatibility considerations.
    • Updated ANALYZE syntax and analysis-status references to include NDVRATE and supported defaults.
    • Added reference information for tidb_analyze_sampled_ndv_threshold, which controls when NDV estimation uses sampling.
    • Documented tidb_analyze_store_batch_size and how it controls batching of ANALYZE tasks sent to TiKV.

0xPoe added 3 commits November 9, 2026 14:09
pingcap/tidb#69956 lets `ANALYZE ... WITH DEFAULT <option>` clear a
persisted option, but the synopsis does not show this form.
pingcap/tidb#69686 and pingcap/tidb#70587 batch the Region tasks of a
full-sampling ANALYZE by TiKV node, and tikv/tikv#19854 merges their
results in TiKV. The new variable sets the batch size.
pingcap/tidb#71520 and tikv/tikv#20106 let ANALYZE estimate NDV from a
sample of rows on large tables. pingcap/tidb#67608 and
pingcap/tidb#71699 add the NDVRATE option, and pingcap/tidb#72453 sets
the default threshold for new clusters.

Document the tidb_analyze_sampled_ndv_threshold variable, the NDVRATE
option, its persistence, and the limits of the feature.
@ti-chi-bot ti-chi-bot Bot added the missing-translation-status This PR does not have translation status info. label Nov 9, 2026
@ti-chi-bot

ti-chi-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lilin90 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 9, 2026
@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • Resume reviews
  • Trigger review
Walkthrough

The documentation adds NDVRATE syntax and sampled NDV estimation rules, including related system variables, persistence, and status reporting. It also documents the ANALYZE Region task batch-size setting and its behavior.

Changes: Sampled NDV estimation

Layer / File(s) Summary
NDV estimation options and behavior
statistics.md, sql-statements/sql-statement-analyze-table.md, sql-statements/sql-statement-show-analyze-status.md, information-schema/information-schema-analyze-status.md, system-variables.md, system-variable-reference.md
Documents NDVRATE syntax and sampling rules, the tidb_analyze_sampled_ndv_threshold setting, persistence of the NDV rate, and its appearance in ANALYZE status information.

ANALYZE Region task batching

Layer / File(s) Summary
Batch size setting and behavior
statistics.md, system-variables.md, system-variable-reference.md
Documents tidb_analyze_store_batch_size, its defaults and scope, and how TiDB batches Region tasks in requests to TiKV.

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Suggested reviewers: qiancai

Merge Risk: Low · previous review b107c

The documentation is accurate in its main rules, but the sample-size sentence for very large tables can mislead readers about how many rows are sampled. Two minor wording and consistency points also remain open. None of these block merge, but the authors should be aware of them.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly summarizes the main documentation changes: sampling-based NDV collection and batched ANALYZE requests.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The pull request description is complete and follows the repository template. It explains the documentation changes, identifies the affected master version, provides related code and design links, rec…
Full details: Description check

Explanation

The pull request description is complete and follows the repository template. It explains the documentation changes, identifies the affected master version, provides related code and design links, records AI involvement, and addresses the optional change categories.

Finishing Touches

Generate unit tests (beta)

  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Review info

Run configuration

  • Source mode: Auto
  • Repository: pingcap/docs
  • Branch: master
  • File: .coderabbit.yaml
  • Commit: 16b9550
  • Layers: Repository YAML
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 33f27986-f227-439a-a815-f8deda5cee7d
Commits

Reviewing files that changed from the base of the PR and between 16b9550 and b5208c0.

Files selected for processing: 6

  • information-schema/information-schema-analyze-status.md
  • sql-statements/sql-statement-analyze-table.md
  • sql-statements/sql-statement-show-analyze-status.md
  • statistics.md
  • system-variable-reference.md
  • system-variables.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.

Comment thread statistics.md Resolved
Comment thread statistics.md Resolved
Comment thread statistics.md Resolved
Comment thread statistics.md Resolved
Comment thread statistics.md Resolved
Comment thread system-variables.md Resolved
The sampling-based NDV section said that the row count from the
statistics is "much smaller" than the row count in PD. TiDB uses the PD
row count when it is more than 5 times the row count from the
statistics, so give that condition.
The rule named the point of 20 times the threshold, which the reader
can calculate from the formula, and the sentence was hard to read. Say
only that the NDV sample has approximately the threshold number of
rows and that the NDV rate has a minimum of 0.05.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review info

Run configuration

  • Source mode: Auto
  • Repository: pingcap/docs
  • Branch: master
  • File: .coderabbit.yaml
  • Commit: 16b9550
  • Layers: Repository YAML
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 879b1940-6e35-4e5a-a844-62e9347c4712
Commits

Reviewing files that changed from the base of the PR and between ea78a49 and b107c2f.

Files selected for processing: 1

  • statistics.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.

Comment thread statistics.md Resolved
pingcap/tidb#72511 extends the rule of pingcap/tidb#71721. ANALYZE now
also uses the row count as the NDV for a clustered integer primary key,
and for a primary key or a composite unique index whose columns are all
NOT NULL. The rule does not apply to partial indexes.

List these cases in the limitations of sampling-based NDV collection.
The section explained how TiDB and TiKV implement the feature: the
design motivation, the name of the estimator, the FMSketch data, and
each case in which TiDB increases the NDV rate. A reader cannot use
these details. The caution to upgrade all instances first applies only
if you enable the feature during a rolling upgrade.

Keep the rules, the options, and the errors, warnings, and notes that a
user can see.
The description explained how TiKV runs and merges the Region tasks of
a batch. Keep what the variable controls, its benefit, and its cost.

Also remove the statement that this variable and tidb_store_batch_size
do not affect each other.
The rule said that the NDV sample has approximately the threshold
number of rows. This is not correct when the minimum NDV rate of 0.05
applies. In that case, the sample is 5% of the rows, which is more than
the threshold.
The note said that the automatic update does not record the options of
a manual ANALYZE statement. This has not been correct since v5.4.0,
which added the persistence of ANALYZE configurations. The note also
conflicts with the persisted NDVRATE value that the sampling-based NDV
section describes.

Say that the automatic update uses the options if the persistence
feature is enabled.
The persistence paragraph did not say what occurs without a persisted
NDVRATE value. TiDB persists only a value that an ANALYZE statement
specifies. In other cases, each ANALYZE operation calculates the NDV
rate from the current row count.
The section gave the calculation of the NDV rate in each ANALYZE
operation as the last sentence of the persistence paragraph. Put this
default at the start of the section, before the NDVRATE option.

Also say that TiDB persists the value that you specify, and that
WITH DEFAULT NDVRATE goes back to the calculated rate.

@0xPoe 0xPoe left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔢 Self-check (PR reviewed by myself and ready for feedback)

/cc @mjonss

@ti-chi-bot
ti-chi-bot Bot requested a review from mjonss November 11, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

missing-translation-status This PR does not have translation status info. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant