Skip to content

Windows certs show up verified sooner, which matters for ESP - #55293

Open
getvictor wants to merge 9 commits into
mainfrom
victor/55239-windows-scep-status-probe
Open

getvictor wants to merge 9 commits into
mainfrom
victor/55239-windows-scep-status-probe

Conversation

@getvictor

@getvictor getvictor commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Related issue: Resolves #55239

Checklist for submitter

If some of the following don't apply, delete the relevant line.

  • Changes file added for user-visible changes in changes/, orbit/changes/ or ee/fleetd-chrome/changes.
    See Changes files for more information.

  • Timeouts are implemented and retries are limited to avoid infinite loops

Testing

  • Added/updated automated tests
  • QA'd all new/changed functionality manually

Summary by CodeRabbit

  • Bug Fixes
    • Windows custom SCEP, NDES, and Smallstep profiles now show verified or failed status within a few minutes of installation, rather than waiting for the next certificate inventory, which could take up to an hour.
    • Windows hosts are released from the setup experience after certificate installation completes, so setup no longer remains in progress while waiting for the next certificate inventory.

@getvictor
getvictor deployed to Docker Hub October 9, 2026 22:07 — with GitHub Actions Active
@getvictor
getvictor requested a balanced review from Copilot October 9, 2026 22:07
@getvictor

Copy link
Copy Markdown
Member Author

@coderabbitai full review

@getvictor

Copy link
Copy Markdown
Member Author

/agentic_review

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: fleetdm/fleet/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 37d80fb4-1632-4241-b38b-8dabfed1b7fd

📥 Commits

Reviewing files that changed from the base of the PR and between 6f0351d and 79bc288.


📒 Files selected for processing (1)
  • server/datastore/mysql/windows_scep_status_probe.go

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.



Walkthrough

The change adds delayed Windows SCEP status probes after proxied installation acknowledgments. The probes request certificate status and error details, and the service delays sending each probe for 30 seconds. Probe results can verify a profile, fail or retry an install, or keep it verifying and schedule another probe. The change also updates profile-status rollups when probe handling changes a profile.

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 79bc2

The reviewed probe flow keeps unresolved certificate installs from releasing ESP and updates status when results arrive. No merge-blocking issue was identified in the inspected paths.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 48.48% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 33 functions across 11 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check Passed The title clearly and concisely describes the primary change: Windows certificates reach verified status sooner to support timely ESP release.
Description check Passed The description includes the related issue, changes file, retry and timeout confirmation, automated test updates, and manual QA confirmation. It is sufficiently complete for this change.
Linked Issues check Passed The changes implement the coding requirements in #55239. Fleet queues delayed Windows SCEP status probes after proxied install acknowledgements. Each probe requests Status and ErrorCode for the co…
Out of Scope Changes check Passed The changes remain connected to #55239. Probe command identifiers, delayed delivery, reserved-command validation, probe result handling, profile status updates, and rollup updates support the Windows …

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR

🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@qodo-free-for-open-source-projects

qodo-free-for-open-source-projects Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Late replies undo certificate verification ✓ Resolved
Description
A repeated install ACK makes updateMDMWindowsHostProfileStatusFromResponseDB move an already
verified SCEP profile back to verifying, while enqueueWindowsSCEPStatusProbeDB skips the first
probe because its command row still exists. If that probe has already succeeded and been
acknowledged, the profile has no pending probe to verify it again and must wait for another
certificate inventory.
Code

server/datastore/mysql/windows_scep_status_probe.go[R106-108]

+	if exists {
+		return nil
+	}
Evidence
Successful probe processing marks the matching verifying profile as verified and schedules no
successor. The response path still matches later replies by command UUID and downgrades a successful
proxied-SCEP install ACK without checking the profile's current status; the new enqueue helper then
sees the already-existing probe command and returns without creating pending work.

server/datastore/mysql/windows_scep_status_probe.go[139-149]
server/datastore/mysql/microsoft_mdm.go[1441-1460]
server/datastore/mysql/microsoft_mdm.go[1784-1818]
server/datastore/mysql/windows_scep_status_probe.go[100-112]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A repeated install ACK can move a probe-verified SCEP profile back to verifying, but the existing, acknowledged first probe prevents another from being queued.
## Fix Focus Areas
- server/datastore/mysql/microsoft_mdm.go[1774-1819]
- server/datastore/mysql/windows_scep_status_probe.go[97-112]
## Recommended Fix
Use the current profile status when handling a successful install ACK. Only transition a newly acknowledged delivery into verifying and queue its first probe; preserve the status of a delivery already resolved by a probe or certificate inventory. Add a regression test that sends a repeated install ACK after probe success.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

2. Duplicate install ACKs can fail the response save ✓ Resolved
Description
enqueueWindowsSCEPStatusProbeDB checks whether the probe command exists with a plain SELECT and
inserts it afterwards. Nothing locks the row in between, and
mdmWindowsInsertCommandForEnrollmentIDsDB returns an alreadyExists error on a duplicate key. If
two transactions handle an ACK for the same install at once (for example, a device resends a message
while the first copy is still being processed), the second one hits the duplicate key.
MDMWindowsSaveResponse then rolls back every command result in that message, not just the probe.
Code

server/datastore/mysql/windows_scep_status_probe.go[R101-111]

+	var exists bool
+	if err := sqlx.GetContext(ctx, tx, &exists,
+		`SELECT EXISTS (SELECT 1 FROM windows_mdm_commands WHERE command_uuid = ?)`, cmd.CommandUUID); err != nil {
+		return ctxerr.Wrap(ctx, err, "check for existing windows scep status probe")
+	}
+	if exists {
+		return nil
+	}
+	if err := ds.mdmWindowsInsertCommandForEnrollmentIDsDB(ctx, tx, []uint{enrollmentID}, cmd); err != nil {
+		return ctxerr.Wrap(ctx, err, "enqueue windows scep status probe")
+	}
Evidence
The probe enqueue checks existence first and inserts after, and the insert treats a duplicate key as
an error instead of ignoring it. The caller in MDMWindowsSaveResponse returns that error from the
transaction, so the results insert and the acked_at stamp for the whole message are rolled back.

server/datastore/mysql/windows_scep_status_probe.go[100-113]
server/datastore/mysql/microsoft_mdm.go[1224-1232]
server/datastore/mysql/microsoft_mdm.go[1577-1583]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
enqueueWindowsSCEPStatusProbeDB checks for the probe with SELECT EXISTS and then inserts it. A concurrent transaction can insert the same probe in between, so the second insert fails with alreadyExists and the whole MDMWindowsSaveResponse transaction rolls back.
## Fix Focus Areas
- server/datastore/mysql/windows_scep_status_probe.go[100-113]
## Recommended Fix
If mdmWindowsInsertCommandForEnrollmentIDsDB returns an alreadyExists error (check with fleet.IsAlreadyExists or the existing helper), return nil so a probe another transaction already queued is not treated as a failure. Alternatively, insert with INSERT IGNORE and queue the probe only when a row was actually inserted.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Status probes fill the host's MDM command list ⊘ Outdated
Description
Each probe is queued through mdmWindowsInsertCommandForEnrollmentIDsDB as an ordinary
windows_mdm_commands row with a scep-status- UUID. The Windows command listing query does not
filter that prefix out. Each SCEP delivery that is still enrolling can add up to
windowsSCEPStatusProbeMaxAttempts (15) Get commands to the host's command history in the API and
UI, which can hide the commands admins actually sent.
Code

server/datastore/mysql/windows_scep_status_probe.go[R109-111]

+	if err := ds.mdmWindowsInsertCommandForEnrollmentIDsDB(ctx, tx, []uint{enrollmentID}, cmd); err != nil {
+		return ctxerr.Wrap(ctx, err, "enqueue windows scep status probe")
+	}
Evidence
The Windows branch of the command-listing SQL joins windows_mdm_commands with the queue and results
tables without excluding any command_uuid prefix. Because the probes are stored in those same
tables, they show up in the listing.

server/datastore/mysql/mdm.go[99-138]
server/datastore/mysql/windows_scep_status_probe.go[24-28]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
SCEP status probe Get commands are stored as regular Windows MDM commands, so they appear in the host's MDM command list (up to 15 per SCEP delivery).
## Fix Focus Areas
- server/datastore/mysql/mdm.go[99-138]
## Recommended Fix
Add `AND wmc.command_uuid NOT LIKE 'scep-status-%'` (built from fleet.WindowsSCEPStatusProbeCmdUUIDPrefix) to both Windows branches of the command-listing query, or confirm that showing the probes there is intended.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Tip of the day
💡 Did you know, you can commit Qodo's fix in one click with committable suggestions (GitHub & GitLab)

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread server/datastore/mysql/windows_scep_status_probe.go Outdated
Comment thread server/datastore/mysql/windows_scep_status_probe.go Outdated
Comment thread server/datastore/mysql/windows_scep_status_probe.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

  • Copilot's review of this pull request may be incomplete because some of the changed files are excluded by your Copilot content exclusion settings. See Excluding content from Copilot for details.

🟡 Changes recommended

Probe scoping, duplicate acknowledgments, and failure handling can produce behavior inconsistent with the linked issue.

3 open findings
What changed in this PR

Adds Windows MDM SCEP status probes to verify certificates sooner, especially during Autopilot ESP.

Changes:

  • Queues delayed SCEP status and error-code probes.
  • Applies probe results to profile status and adds retry handling.
  • Adds datastore and service tests.
File Description
changes/​55239-windows-scep-status-probe Changes entry; contents unavailable for review.
server/​fleet/​microsoft_mdm.go Defines probe identifiers and delay.
server/​service/​microsoft_mdm.go Delays recently queued probes.
server/​service/​microsoft_mdm_test.go Tests delayed probe delivery.
server/​datastore/​mysql/​windows_scep_status_probe.go Builds, queues, and processes probes.
server/​datastore/​mysql/​windows_scep_status_probe_test.go Tests probe lifecycle and results.
server/​datastore/​mysql/​microsoft_mdm.go Integrates probes with response processing.
server/​datastore/​mysql/​microsoft_mdm_test.go Updates helper invocation.
server/​datastore/​mysql/​host_certificates_test.go Updates helper invocation.
Files excluded by content exclusion policy (1)
  • changes/55239-windows-scep-status-probe

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread server/datastore/mysql/microsoft_mdm.go
Comment thread server/datastore/mysql/windows_scep_status_probe.go Outdated
Comment thread server/fleet/microsoft_mdm.go
Comment thread server/datastore/mysql/windows_scep_status_probe.go Fixed
…erate duplicate probe insert

- A repeated install ACK no longer sends a delivery the probe already verified back to verifying.
- Queuing a probe that already exists is a no-op instead of failing the response save.
- Parse the SCEP ErrorCode as a 32-bit value (CodeQL integer conversion).
@getvictor
getvictor marked this pull request as ready for review October 12, 2026 15:46
@getvictor
getvictor requested a review from a team as a code owner October 12, 2026 15:46
@codecov

codecov Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.95455% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.20%. Comparing base (fc58877) to head (79bc288).
⚠️ Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
server/datastore/mysql/microsoft_mdm.go 78.88% 19 Missing ⚠️
...erver/datastore/mysql/windows_scep_status_probe.go 85.52% 11 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #55293    +/-   ##
========================================
  Coverage   77.19%   77.20%            
========================================
  Files        4299     4300     +1     
  Lines      266562   266785   +223     
  Branches    15682    15682            
========================================
+ Hits       205779   205968   +189     
- Misses      60597    60631    +34     
  Partials      186      186            
Flag Coverage Δ
backend 78.49% <82.95%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jbelbo jbelbo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, left a few comments

}

switch status {
case windowsSCEPStatusSucceeded:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the issued cert doesn't carry the renewal ID Status=1 still marks this verified, and the 1h backstop only looks at verifying rows, so it never fires. Before, the admin got "Fleet did not detect the SCEP certificate". Now it will show verified while automatic renewal can't find the cert. Is that intended? Limiting the probe to the ESP, as Copilot suggested in one of the comments, would also keep osquery as the check everywhere else.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. I'll fix.

Comment thread server/datastore/mysql/microsoft_mdm.go Outdated
if hp.OperationType == fleet.MDMOperationTypeInstall && payload.Status != nil && *payload.Status == fleet.MDMDeliveryVerified {
if _, ok := scepProxyProfileUUIDs[hp.ProfileUUID]; ok {
// A repeated ACK must not undo a verification this delivery already got.
alreadyVerified := hp.Status != nil && *hp.Status == fleet.MDMDeliveryVerified

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on the ACK fix: it covers verified, but the same late ACK after the probe failed the delivery (failed, or reset for a retry) still flips it back to verifying, and since the attempt-1 probe already exists no new one is queued, so it waits for the 1h backstop. I think only moving a row to verifying while it's still pending will cover this case too.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll fix this one. It is unlikely (since it is a misconfiguration) so I'll widen the backstop to catch this case.

scope = "./User"
}
// The SCEP node ID is the profile UUID.
node := fmt.Sprintf("%s/Vendor/MSFT/ClientCertificateInstall/SCEP/%s", scope, profileUUID)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every delivery checks the same SCEP/<profile_uuid>/Status node, so I'm wondering what a probe sees after a retry or a renewal. If Windows doesn't reset Status when the new enrollment starts, could the probe pick up the previous attempt's 16 or 1? Did you get to try a redelivery on a device?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The device runs the Exec that starts the new enrollment before it sends the ACK, so any probe, which always goes out after the ACK, sees the new attempt. Also, older answers are discarded anyway. A probe is tied to its install command's UUID, so an answer about an earlier delivery can't change a newer one.

This branch is being deployed

1 in progress deployment
Docker Hub — 77545b2c Deployed Oct 12, 2026 by getvictor via publish #108717
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows setup experience (ESP) can be held up to an hour waiting for SCEP profile verification

4 participants