Repository navigation
Evaluate Agent Cop quality checks - #10396
Gerardo Lecaros (glecaros) wants to merge 3 commits into
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for this isolated advisory evaluation, which adds no production workflow or CI gate.
No open issues 🎉
What changed in this PR
Adds an isolated Agent Cop evaluation for Agent Framework’s .NET and Python quality checks. The report recommends against required enforcement for now; no production workflow or CI gate is added.
Changes:
- Adds three advisory checks for dependency boundaries and sample README presence.
- Adds representative fixtures and 14 tests covering detections, false positives, missed violations, and parser failures.
- Documents results, local timings, limitations, and reproduction steps.
| File | Description |
|---|---|
experiments/agent-cop/tests/test_experiment.py |
Runs 14 checks against temporary fixture copies. |
experiments/agent-cop/README.md |
Records results, limitations, and reproduction steps. |
experiments/agent-cop/fixtures/samples/without-readme/Example.csproj |
Seeds a sample missing its README. |
experiments/agent-cop/fixtures/samples/with-readme/README.md |
Provides same-directory sample documentation. |
experiments/agent-cop/fixtures/samples/with-readme/Example.csproj |
Seeds a documented sample. |
experiments/agent-cop/fixtures/samples/shared-readme/README.md |
Provides shared parent documentation. |
experiments/agent-cop/fixtures/samples/shared-readme/child/Example.csproj |
Demonstrates the shared-README false positive. |
experiments/agent-cop/fixtures/python/type-ignore/example.py |
Reproduces the Python parser failure. |
experiments/agent-cop/fixtures/python/required/pyproject.toml |
Seeds a required framework dependency. |
experiments/agent-cop/fixtures/python/optional/pyproject.toml |
Demonstrates the optional-dependency false positive. |
experiments/agent-cop/fixtures/python/normalized/pyproject.toml |
Exercises distribution-name normalization. |
experiments/agent-cop/fixtures/python/clean/pyproject.toml |
Provides an allowed dependency case. |
experiments/agent-cop/fixtures/python/bare-except/example.py |
Seeds a bare-except violation. |
experiments/agent-cop/fixtures/dotnet/property/Abstractions.csproj |
Seeds a missed property-expanded reference. |
experiments/agent-cop/fixtures/dotnet/malformed/Abstractions.csproj |
Exercises malformed XML handling. |
experiments/agent-cop/fixtures/dotnet/inherited/project/Abstractions.csproj |
Provides the project inheriting a reference. |
experiments/agent-cop/fixtures/dotnet/inherited/Directory.Build.props |
Supplies the missed inherited reference. |
experiments/agent-cop/fixtures/dotnet/forbidden/Abstractions.csproj |
Seeds four direct reference violations. |
experiments/agent-cop/fixtures/dotnet/allowed/Abstractions.csproj |
Covers allowed packages and reference operations. |
experiments/agent-cop/checks/python-core-dependencies.cop |
Checks core dependencies on framework distributions. |
experiments/agent-cop/checks/dotnet-sample-readmes.cop |
Checks for same-directory sample READMEs. |
experiments/agent-cop/checks/dotnet-abstractions.cop |
Checks direct project and framework package references. |
Review effort: Balanced
There was a problem hiding this comment.
MAF Automated Review — Iteration 1
Result: No findings
Scope: full PR (1 commit(s)): d9cb725116b7
Model: gpt-5.6-sol
Overview
This PR adds an isolated, explicitly non-blocking evaluation of three Agent Cop rules and does not connect them to production or CI enforcement. The fixture runner copies inputs into per-test temporary directories, bounds external execution, checks exact exit/result behavior, and documents the evaluated version and known false positives, false negatives, and parser failure. No publishable Critical, High, or Medium issue was established.
Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
No publishable findings remained after source verification for this scope.
Run Cop's built-in C# and Python packages over dotnet/src and python/packages, triage the 8,274 findings against MAF's existing analyzer policy, and add a reproducible scan script. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Summary
dotnet/srcandpython/packages, then triage the findings against MAF's existing analyzer policy. Full results report.scan_repository.pyto rerun the scan against an isolatedgit archivesnapshot and save raw findings, stderr, and a per-rule summary.Results on Agent Framework
varfindings against.editorconfig, 295 C# documentation findings on internal types, 75 Python**kwargsfindings contrary topython/CODING_STANDARD.md, and 126type()comparisons, mostly intentional exact-type checks.Reproduce
Run the scan script as documented in the report, using the pinned baseline commit
cf5ea1979562efaf993fb3d37958de0c52d612cc.Tracking: AB#5887155