' + f'{escape(result.name)}{escape(result.summary)}' + f'{"Required" if result.required else "Optional"}' + f'{result.duration_seconds:.2f}s' + f'{result.status.value}⌄
' + f'Policy weight: {result.weight:g}
{detail or "No additional evidence.
"}From f2f33a9b5d2e0947e4b574eb6b14dd8369cbc93a Mon Sep 17 00:00:00 2001 From: iwebslogtech <5895515+iwebslogtech@users.noreply.github.com> Date: Sat, 7 Nov 2026 14:47:42 +0000 Subject: [PATCH 1/2] Improve scan reliability, tooling, and reports Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 9 +- README.md | 42 ++++- action.yml | 16 +- docs/roadmap.md | 58 ++++++ pyproject.toml | 9 +- src/codeproof/__init__.py | 2 +- src/codeproof/adapters.py | 279 ++++++++++++++++++++++++++--- src/codeproof/cli.py | 179 +++++++++++++++--- src/codeproof/config.py | 98 +++++++++- src/codeproof/core.py | 112 ++++++++++-- src/codeproof/discovery.py | 57 ++++-- src/codeproof/files.py | 49 +++++ src/codeproof/models.py | 47 ++++- src/codeproof/report_template.html | 64 +++++++ src/codeproof/reporting.py | 147 +++++++++++++-- tests/test_adapters.py | 92 ++++++++++ tests/test_config.py | 5 +- tests/test_core.py | 4 +- tests/test_discovery.py | 25 +++ tests/test_policy.py | 97 ++++++++++ tests/test_reports_cli.py | 85 +++++++++ 21 files changed, 1356 insertions(+), 120 deletions(-) create mode 100644 docs/roadmap.md create mode 100644 src/codeproof/files.py create mode 100644 src/codeproof/report_template.html create mode 100644 tests/test_adapters.py create mode 100644 tests/test_discovery.py create mode 100644 tests/test_policy.py create mode 100644 tests/test_reports_cli.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3599f56..9c36207 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -3,9 +3,16 @@ on: [push, pull_request] jobs: test: runs-on: ubuntu-latest + strategy: + matrix: + python-version: ["3.10", "3.12"] steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 - with: {python-version: "3.12"} + with: + python-version: ${{ matrix.python-version }} - run: pip install -e ".[dev]" + - run: ruff check src tests + - run: ruff format --check src tests + - run: mypy src - run: pytest diff --git a/README.md b/README.md index d292bf3..1fc1900 100644 --- a/README.md +++ b/README.md @@ -19,15 +19,47 @@ pip install -e ".[checkers]" codeproof scan . ``` + + + + ## Commands + ```bash -codeproof doctor -codeproof init -codeproof inventory . +codeproof tools # supported checks and execution behavior +codeproof doctor # availability and install guidance +codeproof init # write a default policy +codeproof inventory . # public symbols, not coverage codeproof generate-tests . --output tests/generated -codeproof scan . --format all --fail-under 70 +codeproof scan . --format all --fail-under 70 --jobs 3 +codeproof scan . --config path/to/policy.yml --checks ruff,pytest,syntax +``` + +Reports are stored under `.codeproof/reports/`. Open `codeproof-report.html` for offline search, status filters, and expandable evidence. JSON and Markdown remain available. Paths are positional; the earlier `--path` option is also supported. Exit codes: **0** policy passed, **1** policy failed, **2** invalid usage or configuration. + +### Policy and additional tools + +Partial policies overlay the six default checks; explicitly disable any you do not want. A selected or configured check cannot bypass a required tool gate. Remove a tool from `required_tools` only when intentionally changing the policy. For a quick execution-free syntax-only scan, use: + +```yaml +# .codeproof.yml +required_tools: [] +source_paths: [src] +checks: + syntax: {enabled: true, weight: 10} ``` -Reports are stored under `.codeproof/reports/`. + +```bash +codeproof scan . --checks syntax +``` + +To add dependency auditing, install `pip install 'codeproof-ai[audit]'`, provide a root `requirements.txt`, and add `pip-audit: {enabled: true, weight: 15}` under `checks`. This opt-in tool queries vulnerability services and may invoke package tooling while resolving dependencies. Unsupported manifests are reported unavailable; the developer environment is never audited as a fallback. + +`jobs: 1` (default) runs serially; `--jobs 2` through `--jobs 8` allow concurrency. Tests and imports can share resources, so choose concurrency to suit the repository. Invalid thresholds, negative/nonfinite weights, unknown tools, and unknown policy keys are rejected. + +Generated tests check symbol exposure and execute imports; review them before running. Existing files are preserved unless `--force` is passed. See the [audit and prioritized roadmap](docs/roadmap.md) for delivered changes, compatibility notes, and follow-up work. + + ## Confidence model The score is the weighted percentage of included checks that pass. Missing required tools fail policy. Missing optional tools are marked unavailable and excluded from the denominator. A passing report is evidence that configured checks passed, not proof of correctness. diff --git a/action.yml b/action.yml index c6d49dc..2cb0ea1 100644 --- a/action.yml +++ b/action.yml @@ -3,13 +3,23 @@ description: Evidence-backed Python repository verification inputs: path: {description: Repository path, required: false, default: .} fail-under: {description: Minimum score, required: false, default: "70"} + jobs: {description: Concurrent check workers (1-8), required: false, default: "1"} runs: using: composite steps: - shell: bash - run: python -m pip install "${{ github.action_path }}[checkers]" + env: + CODEPROOF_ACTION_PATH: ${{ github.action_path }} + run: python -m pip install "${CODEPROOF_ACTION_PATH}[checkers]" - shell: bash - run: codeproof scan "${{ inputs.path }}" --fail-under "${{ inputs.fail-under }}" + env: + CODEPROOF_TARGET: ${{ inputs.path }} + CODEPROOF_THRESHOLD: ${{ inputs.fail-under }} + CODEPROOF_JOBS: ${{ inputs.jobs }} + run: codeproof scan "$CODEPROOF_TARGET" --fail-under "$CODEPROOF_THRESHOLD" --jobs "$CODEPROOF_JOBS" - if: always() uses: actions/upload-artifact@v4 - with: {name: codeproof-report, path: "${{ inputs.path }}/.codeproof/"} + with: + name: codeproof-report + path: "${{ inputs.path }}/.codeproof/reports/" + include-hidden-files: true diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..7b10c8a --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,58 @@ + + +# CodeProof AI — repository audit and delivery roadmap + +## Product direction + +CodeProof is a local Python CLI and reusable GitHub Action. Its strongest use cases are developer preflight checks, CI policy gates, and portable evidence for code review. Keep the local CLI as the execution engine and make its reports useful to maintainers before introducing a hosted product. A score summarizes configured checks; it cannot establish correctness or replace behavioral tests. + +Audit baseline: `7043a711daefaed5a2f973d51ec856a5ef00d703`. The initial package had six adapters, a weighted score, an AST symbol inventory, exposure-test generation, three report formats, and two tests. There was no application server, database, hosted account system, or test-coverage analysis. The inventory should not be described as identifying untested behavior. + +## Audit findings and delivered repairs + +| Priority | Finding and evidence | Delivered improvement | +|---|---|---| +| P0 | `core.py` checked required gates only for executed results; disabling or omitting one bypassed its gate | Emit required skipped results and fail policy; no weighted evidence cannot pass, including at threshold zero | +| P0 | `config.py` accepted invalid weights/timeouts and replaced all default checks when a partial file existed | Validate policy ranges, unknown settings/tools and version; merge partial overrides with defaults | +| P0 | `action.yml` interpolated inputs directly into shell scripts | Pass inputs through quoted environment variables; upload report artifacts including the hidden report directory | +| P0 | Semgrep ran without `--error`, permitting successful exit despite findings | Findings now produce a failed check; this may lower previously misleading scores | +| P1 | Documented positional CLI paths were interpreted as options | Support documented positional paths and retain the existing `--path` option for compatibility | +| P1 | `adapters.py` returned almost exclusively raw output and import timeouts escaped the adapter | Normalize Ruff/Bandit/Semgrep/pip-audit findings; retain raw diagnostics; contain launch and timeout errors; enforce total import time budget | +| P1 | All adapters ran serially | Configurable worker pool (1–8), stable result order; serial remains default | +| P1 | HTML had only an unstyled summary table | Offline responsive report, policy explanation, search, status filters, expandable findings and raw evidence; keyboard focus styles and live count | +| P1 | No dependency vulnerability or execution-free syntax tools | Opt-in `pip-audit` for root requirements.txt; built-in `syntax` without execution or bytecode writes; discoverable tools and install guidance | +| P1 | Test generation wrote only the first symbol per module without force, duplicated on force, and mishandled package imports | Group symbols by module, replace once on force, preserve existing files, handle package exports and avoid module filename collisions | +| P1 | File discovery descended into environments and caches | Shared deterministic traversal prunes common generated trees, skips symlinks and deduplicates overlapping roots | +| P1 | Only grade and total weights had tests | Regression coverage for policy gates, config, parallel execution, adapter errors, structured evidence, generation, escaping and CLI workflows | + +## Compatibility and intentional changes + +- Existing six default checks and their weights (100 total) remain. Both new tools are opt-in; all report filenames and JSON schema 1.0 fields remain. Existing `--path` callers remain supported. +- A partial YAML policy now overlays defaults. Previously, specifying only `fail_under` could run no checks. To run a smaller set, disable the other checks explicitly or use `--checks`; remove a tool from `required_tools` only when that policy change is intended. +- Disabled or absent required checks fail policy, matching the README's required-tool guarantee. Empty evidence fails even if the threshold is zero. Invalid or misspelled policies now fail with CLI exit code 2. +- Semgrep findings now fail the check. Optional unavailable checks remain excluded from the score; required unavailable checks block policy. +- Generated filenames now use length-prefixed module components to avoid collisions. Existing generated files are preserved; review and remove obsolete files before regenerating into the same directory. +- Concurrent execution is opt-in because tests and imports execute repository code and may share resources. Each tool has a time budget; concurrency is not a sandbox or a process/memory quota. + +## Prioritized roadmap + +| Phase | Work | Exit criteria | Effort | +|---|---|---|---| +| 1 — Delivered | Policy integrity, actionable reports, opt-in tools, bounded workers, generation repairs and regression tests | Local CLI and Action retain compatibility; policy bypass regressions pass; report controls work on desktop and mobile | This change | +| 2 — CI adoption | SARIF export and GitHub annotations; machine-readable doctor; configurable local Semgrep rules; lockfile support for pip-audit; supported Python-version matrix | SARIF validates against its schema; annotations point to source lines; offline configuration completes without rule downloads; locked dependency fixtures yield stable findings | M | +| 3 — Faster large repositories | Benchmarked discovery, streamed/limited subprocess output, process-tree cancellation, bounded findings, incremental cache | Representative small/medium/large fixtures publish timings; no orphan processes after timeout; cache keys include source, config, dependency locks, tool/Python versions; changed inputs invalidate results | L | +| 4 — Stronger evidence | Real pytest counts and coverage ingestion; mypy normalization; severity-aware policy; changed-code baselines; versioned report schema | Inventory remains distinct from coverage; original evidence retained; high-severity gates cannot be offset by unrelated passes; schema migration and consumer contract tests pass | L | +| 5 — Team workflows | Report history and comparison; reviewable policy templates; distributable static viewer; accessible navigation/contrast audit | Compare two saved reports without rerunning code; policy changes show a clear diff; keyboard and screen-reader checks pass; report artifacts require no external assets | M | + +M and L are relative planning estimates, not delivery dates. Phase 2 should precede persistent history or a hosted service. Hosted authentication, queues, object storage and multi-tenant execution need separate product demand and a security design; this repository does not currently contain those facilities. + +## Engineering tools and delivery discipline + +- Use Ruff format/lint, mypy with PyYAML stubs, pytest regression tests, and Bandit for changes to process execution and reporting. Keep dependency auditing as an explicit extra rather than a mandatory network dependency for every local scan. +- Add packaging smoke tests so the HTML template is verified inside built wheels. Pin a reproducible CI dependency set in the next phase and document an update cadence. +- Measure p50/p95 scan time, peak memory, adapter error rates, policy pass rates with missing tools, and report size on synthetic fixtures before implementing caching or claiming speedups. Worker parallelism is delivered; no benchmarked speedup is claimed. +- Preserve the schema and exit-code contract (0 pass, 1 policy fail, 2 usage/config error). Add integration coverage for supported checker versions and hostile JSON/text fixtures. + +## Remaining limits + +Dependency auditing currently supports only a root requirements.txt and requires the optional pip-audit executable and network access. Resolution can invoke package build tooling; use isolated runners for untrusted requirements. Semgrep auto may retrieve rules. Tool processes still capture output in memory before retaining the final 50 KB; nested child processes are not comprehensively cancelled. The import adapter skips standalone package `__init__.py` files, as before. Syntax checks do not establish runtime importability, and generated tests only check symbol exposure. Reports are snapshots; filters never alter the policy score. Live GitHub Action execution requires pushing and running the workflow. Broader minimum-Python and accessibility coverage remain roadmap work. diff --git a/pyproject.toml b/pyproject.toml index 51aa695..19cdaca 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,10 +11,17 @@ license={text="MIT"} dependencies=["typer>=0.12,<1","rich>=13.7,<15","pydantic>=2.7,<3","PyYAML>=6,<7"] [project.optional-dependencies] checkers=["pytest>=8,<10","ruff>=0.6,<1","mypy>=1.11,<2","bandit>=1.7,<2","semgrep>=1.80,<2"] -dev=["codeproof-ai[checkers]"] +audit=["pip-audit>=2.7,<3"] +dev=["codeproof-ai[checkers]", "types-PyYAML>=6,<7"] [project.scripts] codeproof="codeproof.cli:app" [tool.hatch.build.targets.wheel] packages=["src/codeproof"] [tool.pytest.ini_options] testpaths=["tests"] + +[tool.ruff] +target-version = "py310" + +[tool.ruff.lint] +select = ["E4", "E7", "E9", "F", "I", "B", "UP"] diff --git a/src/codeproof/__init__.py b/src/codeproof/__init__.py index 40692a7..3dc1f76 100644 --- a/src/codeproof/__init__.py +++ b/src/codeproof/__init__.py @@ -1 +1 @@ -__version__="0.1.0" +__version__ = "0.1.0" diff --git a/src/codeproof/adapters.py b/src/codeproof/adapters.py index 0224e9b..42f5d02 100644 --- a/src/codeproof/adapters.py +++ b/src/codeproof/adapters.py @@ -1,25 +1,258 @@ -import json,os,shutil,subprocess,sys,time +"""Tool boundaries: normalize findings and contain process failures.""" + +import json +import os +import shutil +import subprocess +import sys +import time from pathlib import Path -from .models import CheckResult,Finding,Status -def run(name:str,root:Path,cfg:str|None,timeout:int,sources:list[str],tests:list[str])->CheckResult: - src=[x for x in sources if (root/x).exists()] or ["."] - commands={"ruff":["ruff","check",*src,"--output-format","json"],"mypy":["mypy",*src,"--no-error-summary"],"pytest":["pytest",*([x for x in tests if (root/x).exists()] or ["."]),"-q"],"bandit":["bandit","-r",*src,"-f","json","-q"],"semgrep":["semgrep","scan","--config",cfg or "auto","--json",*src]} - if name=="imports": return imports(root,src,timeout) - cmd=commands.get(name) - if not cmd:return CheckResult(name=name,status=Status.skipped,summary="unknown adapter") - if not shutil.which(cmd[0]):return CheckResult(name=name,status=Status.unavailable,command=cmd,summary=f"{cmd[0]} is not installed") - started=time.perf_counter() + +from .files import python_files +from .models import CheckResult, Finding, Status + + +def parse_findings(name: str, output: str) -> list[Finding]: + """Retain raw evidence if a tool emits text or an unfamiliar JSON schema.""" try: - cp=subprocess.run(cmd,cwd=root,text=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,timeout=timeout,env={**os.environ,"NO_COLOR":"1"}) - return CheckResult(name=name,status=Status.passed if cp.returncode==0 else Status.failed,command=cmd,duration_seconds=round(time.perf_counter()-started,3),exit_code=cp.returncode,summary="passed" if cp.returncode==0 else "reported findings",raw_output=(cp.stdout or "")[-50000:]) - except subprocess.TimeoutExpired as e:return CheckResult(name=name,status=Status.error,command=cmd,summary=f"timed out after {timeout}s") -def imports(root:Path,sources:list[str],timeout:int)->CheckResult: - started=time.perf_counter(); findings=[]; count=0 - for src in sources: - base=root/src - for p in base.rglob("*.py") if base.exists() else []: - if p.name=="__init__.py":continue - module=".".join(p.relative_to(base).with_suffix("").parts); count+=1 - cp=subprocess.run([sys.executable,"-c",f"import sys;sys.path.insert(0,{str(base)!r});__import__({module!r})"],cwd=root,text=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,timeout=timeout) - if cp.returncode:findings.append(Finding(level="error",message=(cp.stdout or "failed")[-800:],path=str(p.relative_to(root)),rule="import")) - return CheckResult(name="imports",status=Status.passed if not findings else Status.failed,duration_seconds=round(time.perf_counter()-started,3),summary=f"{count} modules checked; {len(findings)} failed",findings=findings,evidence={"modules_checked":count}) + data = json.loads(output) + if name == "ruff": + return [ + Finding( + level="warning", + message=item["message"], + path=item.get("filename"), + line=item.get("location", {}).get("row"), + rule=item.get("code"), + ) + for item in data + ] + if name == "bandit": + return [ + Finding( + level=item.get("issue_severity", "warning").lower(), + message=item["issue_text"], + path=item.get("filename"), + line=item.get("line_number"), + rule=item.get("test_id"), + ) + for item in data.get("results", []) + ] + if name == "semgrep": + return [ + Finding( + level=item["extra"].get("severity", "warning").lower(), + message=item["extra"]["message"], + path=item.get("path"), + line=item.get("start", {}).get("line"), + rule=item.get("check_id"), + ) + for item in data.get("results", []) + ] + if name == "pip-audit": + return [ + Finding( + level="warning", + message=f"{dep['name']} {dep['version']}: {vuln.get('description', vuln['id'])}", + rule=vuln["id"], + path="requirements.txt", + ) + for dep in data.get("dependencies", []) + for vuln in dep.get("vulns", []) + ] + except (ValueError, KeyError, TypeError, AttributeError): + pass + return [] + + +def run( + name: str, + root: Path, + cfg: str | None, + timeout: int, + sources: list[str], + tests: list[str], +) -> CheckResult: + if name == "syntax": + return syntax(root, sources) + if name == "imports": + return imports(root, sources, timeout) + src = [x for x in sources if (root / x).exists()] or ["."] + commands = { + "ruff": ["ruff", "check", *src, "--output-format", "json"], + "mypy": ["mypy", *src, "--no-error-summary"], + "pytest": [ + "pytest", + *([x for x in tests if (root / x).exists()] or ["."]), + "-q", + ], + "bandit": ["bandit", "-r", *src, "-f", "json", "-q"], + "semgrep": [ + "semgrep", + "scan", + "--config", + cfg or "auto", + "--json", + "--error", + *src, + ], + "pip-audit": [ + "pip-audit", + "-r", + "requirements.txt", + "--format", + "json", + "--progress-spinner", + "off", + ], + } + cmd = commands.get(name) + if not cmd: + return CheckResult(name=name, status=Status.skipped, summary="unknown adapter") + if name == "pip-audit" and not (root / "requirements.txt").is_file(): + return CheckResult( + name=name, + status=Status.unavailable, + summary="Requires requirements.txt at the project root; no environment audit performed", + ) + if not shutil.which(cmd[0]): + return CheckResult( + name=name, + status=Status.unavailable, + command=cmd, + summary=f"{cmd[0]} is not installed; install codeproof-ai[{'audit' if name == 'pip-audit' else 'checkers'}]", + ) + started = time.perf_counter() + try: + cp = subprocess.run( + cmd, + check=False, + cwd=root, + text=True, + errors="replace", + capture_output=True, + timeout=timeout, + env={**os.environ, "NO_COLOR": "1"}, + ) + findings = parse_findings(name, cp.stdout or "") + return CheckResult( + name=name, + status=Status.passed if cp.returncode == 0 else Status.failed, + command=cmd, + duration_seconds=round(time.perf_counter() - started, 3), + exit_code=cp.returncode, + summary=f"{len(findings)} findings" + if findings + else ( + "passed" + if cp.returncode == 0 + else "Check failed; expand output for details" + ), + findings=findings, + raw_output=((cp.stdout or "") + ("\n" + cp.stderr if cp.stderr else ""))[ + -50000: + ], + ) + except (subprocess.TimeoutExpired, OSError) as exc: + return CheckResult( + name=name, + status=Status.error, + command=cmd, + duration_seconds=round(time.perf_counter() - started, 3), + summary=f"timed out after {timeout}s" + if isinstance(exc, subprocess.TimeoutExpired) + else str(exc), + ) + + +def syntax(root: Path, sources: list[str]) -> CheckResult: + started = time.perf_counter() + findings: list[Finding] = [] + count = 0 + for path, _ in python_files(root, sources): + count += 1 + try: + compile(path.read_bytes(), str(path), "exec") + except (SyntaxError, UnicodeError, OSError) as exc: + findings.append( + Finding( + level="error", + message=str(exc), + path=str(path.relative_to(root)), + line=getattr(exc, "lineno", None), + rule="syntax", + ) + ) + return CheckResult( + name="syntax", + status=Status.failed + if findings + else (Status.passed if count else Status.skipped), + duration_seconds=round(time.perf_counter() - started, 3), + summary=f"{count} files compiled; {len(findings)} errors", + findings=findings, + evidence={"files_checked": count}, + ) + + +def imports(root: Path, sources: list[str], timeout: int) -> CheckResult: + started = time.perf_counter() + findings: list[Finding] = [] + count = 0 + for path, base in python_files(root, sources): + if path.name == "__init__.py": + continue + remaining = timeout - (time.perf_counter() - started) + if remaining <= 0: + return CheckResult( + name="imports", + status=Status.error, + summary=f"timed out after {timeout}s", + findings=findings, + ) + module = ".".join(path.relative_to(base).with_suffix("").parts) + count += 1 + try: + cp = subprocess.run( + [ + sys.executable, + "-c", + f"import sys;sys.path.insert(0,{str(base)!r});__import__({module!r})", + ], + check=False, + cwd=root, + text=True, + errors="replace", + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + timeout=remaining, + ) + except (subprocess.TimeoutExpired, OSError) as exc: + return CheckResult( + name="imports", + status=Status.error, + duration_seconds=round(time.perf_counter() - started, 3), + findings=findings, + summary=f"timed out after {timeout}s" + if isinstance(exc, subprocess.TimeoutExpired) + else str(exc), + ) + if cp.returncode: + findings.append( + Finding( + level="error", + message=(cp.stdout or "failed")[-800:], + path=str(path.relative_to(root)), + rule="import", + ) + ) + return CheckResult( + name="imports", + status=Status.failed + if findings + else (Status.passed if count else Status.skipped), + duration_seconds=round(time.perf_counter() - started, 3), + summary=f"{count} modules checked; {len(findings)} failed", + findings=findings, + evidence={"modules_checked": count}, + ) diff --git a/src/codeproof/cli.py b/src/codeproof/cli.py index 48c7683..8974f8a 100644 --- a/src/codeproof/cli.py +++ b/src/codeproof/cli.py @@ -1,37 +1,166 @@ +import shutil +from enum import Enum from pathlib import Path -import shutil,typer,yaml +from typing import Annotated + +import typer +import yaml from rich.console import Console from rich.table import Table -from .config import load,DEFAULT + +from .config import DEFAULT, TOOL_INFO, CheckConfig, load from .core import scan as execute -from .discovery import symbols,generate +from .discovery import generate, symbols from .reporting import write -app=typer.Typer(no_args_is_help=True);console=Console() + +app = typer.Typer( + no_args_is_help=True, + help="Evidence-backed Python repository checks. Start with doctor or init.", +) +console = Console() + + +class Format(str, Enum): + all = "all" + json = "json" + markdown = "markdown" + html = "html" + + +def policy(root: Path, config: Path | None = None): + if not root.is_dir(): + raise typer.BadParameter(f"Repository directory does not exist: {root}") + try: + return load(root, config) + except (ValueError, TypeError, OSError, yaml.YAMLError) as exc: + raise typer.BadParameter(f"Invalid policy: {exc}") from exc + + +@app.command() +def tools(): + """List supported tools, execution behavior, and installation extras.""" + table = Table("Tool", "Purpose", "Behavior", "Install extra") + for name, (purpose, description, extra) in TOOL_INFO.items(): + table.add_row(name, purpose, description, extra or "built in") + console.print(table) + + @app.command() def doctor(): - t=Table("Tool","Status") - for n in ["ruff","mypy","pytest","bandit","semgrep"]:t.add_row(n,"available" if shutil.which(n) else "missing") - console.print(t) + """Check tool availability and show how to install missing tools.""" + table = Table("Tool", "Status", "Next step") + for name, (_, _, extra) in TOOL_INFO.items(): + available = extra is None or bool(shutil.which(name)) + table.add_row( + name, + "available" if available else "missing", + "Ready" if available else f"pip install 'codeproof-ai[{extra}]'", + ) + console.print(table) + console.print( + "Dependency auditing needs requirements.txt. Semgrep auto and pip-audit use the network." + ) + + @app.command() -def init(path:Path=Path("."),force:bool=False): - p=path.resolve()/".codeproof.yml" - if p.exists() and not force:raise typer.BadParameter("policy exists; use --force") - p.write_text(yaml.safe_dump(DEFAULT.model_dump(mode="json"),sort_keys=False));console.print(f"Created {p}") +def init( + path: Annotated[Path, typer.Argument()] = Path("."), + force: bool = False, + path_option: Annotated[Path | None, typer.Option("--path", hidden=True)] = None, +): + """Create a commented policy without overwriting an existing one.""" + root = (path_option or path).resolve() + if not root.is_dir(): + raise typer.BadParameter(f"Repository directory does not exist: {root}") + dest = root / ".codeproof.yml" + if dest.exists() and not force: + raise typer.BadParameter("policy exists; use --force") + dest.write_text( + "# Weights are relative. Required checks must pass even with weight 0.\n" + "# Add syntax or pip-audit under checks to opt in; jobs: 1 is serial.\n" + + yaml.safe_dump(DEFAULT.model_dump(mode="json"), sort_keys=False), + encoding="utf-8", + ) + console.print(f"Created {dest}") + + @app.command() -def inventory(path:Path=Path(".")): - root=path.resolve();cfg=load(root);t=Table("Symbol","File","Line") - data=symbols(root,cfg.source_paths) - for n,p,l in data:t.add_row(n,str(p),str(l)) - console.print(t);console.print(f"{len(data)} public symbols") +def inventory( + path: Annotated[Path, typer.Argument()] = Path("."), + path_option: Annotated[Path | None, typer.Option("--path", hidden=True)] = None, +): + """List public top-level symbols; this is an inventory, not coverage analysis.""" + root = (path_option or path).resolve() + cfg = policy(root) + table = Table("Symbol", "File", "Line") + data = symbols(root, cfg.source_paths) + for name, file, line in data: + table.add_row(name, str(file), str(line)) + console.print(table) + console.print(f"{len(data)} public symbols") + + @app.command("generate-tests") -def generate_tests(path:Path=Path("."),output:Path=Path("tests/generated"),force:bool=False): - root=path.resolve();cfg=load(root);made=generate(root,cfg.source_paths,root/output,force);console.print(f"Created {len(made)} files") +def generate_tests( + path: Annotated[Path, typer.Argument()] = Path("."), + path_option: Annotated[Path | None, typer.Option("--path", hidden=True)] = None, + output: Path = Path("tests/generated"), + force: bool = False, +): + """Generate reviewable exposure checks. These are not behavioral tests.""" + root = (path_option or path).resolve() + cfg = policy(root) + made = generate(root, cfg.source_paths, root / output, force) + console.print( + f"Created {len(made)} files; review before running (imports execute code)." + ) + + @app.command() -def scan(path:Path=Path("."),format:str="all",fail_under:float|None=None,output:Path=Path(".codeproof/reports")): - if format not in {"all","json","markdown","html"}:raise typer.BadParameter("invalid format") - root=path.resolve();report=execute(root,load(root),fail_under);t=Table("Check","Status","Weight","Summary") - for r in report.results:t.add_row(r.name,r.status.value,str(r.weight),r.summary) - console.print(t);console.print(f"Score {report.score}/100 ({report.grade}); policy {'PASS' if report.policy_passed else 'FAIL'}") - for p in write(report,root/output,format):console.print(f"Report: {p}") +def scan( + path: Annotated[Path, typer.Argument()] = Path("."), + path_option: Annotated[Path | None, typer.Option("--path", hidden=True)] = None, + format: Format = Format.all, + fail_under: float | None = typer.Option(None, min=0, max=100), + output: Path = Path(".codeproof/reports"), + config: Path | None = None, + jobs: int | None = typer.Option(None, min=1, max=8), + checks: str | None = typer.Option( + None, help="Comma-separated tools to run. Required tools remain policy gates." + ), +): + """Run checks and write reports. Exit 0: policy passes; 1: fails; 2: invalid input.""" + root = (path_option or path).resolve() + cfg = policy(root, config) + if checks is not None: + selected = {name.strip() for name in checks.split(",") if name.strip()} + if not selected or selected - TOOL_INFO.keys(): + raise typer.BadParameter( + "--checks must name supported tools; see codeproof tools" + ) + for name in sorted(selected): + cfg.checks.setdefault(name, CheckConfig(weight=10)) + for name, check in cfg.checks.items(): + check.enabled = name in selected + with console.status("Running repository checks…"): + report = execute(root, cfg, fail_under, jobs) + table = Table("Check", "Status", "Required", "Time", "Summary") + for result in report.results: + table.add_row( + result.name, + result.status.value, + "yes" if result.required else "no", + f"{result.duration_seconds:.2f}s", + result.summary, + ) + console.print(table) + console.print( + f"Score {report.score}/100 ({report.grade}); policy {'PASS' if report.policy_passed else 'FAIL'} (threshold {report.fail_under:g})" + ) + for dest in write(report, root / output, format.value): + console.print(f"Report: {dest}") raise typer.Exit(0 if report.policy_passed else 1) -if __name__=="__main__":app() + + +if __name__ == "__main__": + app() diff --git a/src/codeproof/config.py b/src/codeproof/config.py index e13d346..a307bbb 100644 --- a/src/codeproof/config.py +++ b/src/codeproof/config.py @@ -1,11 +1,95 @@ +"""Validated scan policies, with partial files layered over safe defaults.""" + from pathlib import Path +from typing import Literal + import yaml -from pydantic import BaseModel,Field +from pydantic import BaseModel, ConfigDict, Field, model_validator + +TOOL_INFO = { + "imports": ("Import smoke", "Executes each source module in a subprocess", None), + "ruff": ("Lint", "Fast Python lint checks", "checkers"), + "mypy": ("Types", "Static type analysis", "checkers"), + "pytest": ("Tests", "Executes the repository test suite", "checkers"), + "bandit": ("Security", "Python security analysis", "checkers"), + "semgrep": ( + "Patterns", + "Security rules (auto rules may use the network)", + "checkers", + ), + "syntax": ( + "Syntax", + "Compiles source without executing it or writing bytecode", + None, + ), + "pip-audit": ( + "Dependencies", + "Audits requirements.txt against vulnerability services", + "audit", + ), +} + + class CheckConfig(BaseModel): - enabled:bool=True; weight:float=1; config:str|None=None + model_config = ConfigDict(extra="forbid") + enabled: bool = True + weight: float = Field(default=1, ge=0, allow_inf_nan=False) + config: str | None = None + + class Config(BaseModel): - version:int=1; fail_under:float=70; source_paths:list[str]=Field(default_factory=lambda:["src"]); test_paths:list[str]=Field(default_factory=lambda:["tests"]); timeout_seconds:int=300; required_tools:list[str]=Field(default_factory=lambda:["ruff","pytest"]); checks:dict[str,CheckConfig]=Field(default_factory=dict) -DEFAULT=Config(checks={"imports":CheckConfig(weight=10),"ruff":CheckConfig(weight=15),"mypy":CheckConfig(weight=15),"pytest":CheckConfig(weight=30),"bandit":CheckConfig(weight=15),"semgrep":CheckConfig(weight=15,config="auto")}) -def load(root:Path, explicit:Path|None=None)->Config: - p=explicit or root/".codeproof.yml" - return DEFAULT.model_copy(deep=True) if not p.exists() else Config.model_validate(yaml.safe_load(p.read_text()) or {}) + model_config = ConfigDict(extra="forbid") + version: Literal[1] = 1 + fail_under: float = Field(default=70, ge=0, le=100, allow_inf_nan=False) + source_paths: list[str] = Field(default_factory=lambda: ["src"]) + test_paths: list[str] = Field(default_factory=lambda: ["tests"]) + timeout_seconds: int = Field(default=300, ge=1) + jobs: int = Field(default=1, ge=1, le=8) + required_tools: list[str] = Field(default_factory=lambda: ["ruff", "pytest"]) + checks: dict[str, CheckConfig] = Field(default_factory=dict) + + @model_validator(mode="after") + def valid_policy(self): + unknown = (set(self.checks) | set(self.required_tools)) - TOOL_INFO.keys() + if unknown: + raise ValueError("Unknown tools: " + ", ".join(sorted(unknown))) + if not self.source_paths or not self.test_paths: + raise ValueError("source_paths and test_paths must not be empty") + if any(not path.strip() for path in self.source_paths + self.test_paths): + raise ValueError("Source and test paths must not be blank") + return self + + +DEFAULT = Config( + checks={ + "imports": CheckConfig(weight=10), + "ruff": CheckConfig(weight=15), + "mypy": CheckConfig(weight=15), + "pytest": CheckConfig(weight=30), + "bandit": CheckConfig(weight=15), + "semgrep": CheckConfig(weight=15, config="auto"), + } +) + + +def load(root: Path, explicit: Path | None = None) -> Config: + path = explicit if explicit is not None else root / ".codeproof.yml" + if not path.exists(): + if explicit is not None: + raise ValueError(f"Policy file not found: {path}") + return DEFAULT.model_copy(deep=True) + data = yaml.safe_load(path.read_text(encoding="utf-8")) + if data is None: + data = {} + if not isinstance(data, dict): + raise TypeError("Policy must be a YAML mapping") + defaults = DEFAULT.model_dump() + overrides = data.get("checks", {}) + if not isinstance(overrides, dict): + raise TypeError("checks must be a YAML mapping") + checks = defaults["checks"] + for name, settings in overrides.items(): + if not isinstance(settings, dict): + raise TypeError(f"Check {name} must be a YAML mapping") + checks[name] = {**checks.get(name, {}), **settings} + return Config.model_validate({**defaults, **data, "checks": checks}) diff --git a/src/codeproof/core.py b/src/codeproof/core.py index b767863..a92b010 100644 --- a/src/codeproof/core.py +++ b/src/codeproof/core.py @@ -1,19 +1,97 @@ -from datetime import datetime,timezone -from pathlib import Path +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime, timezone + from . import __version__ from .adapters import run -from .models import ScanReport,Status -def grade(x:float)->str:return "A" if x>=90 else "B" if x>=80 else "C" if x>=70 else "D" if x>=60 else "F" -def scan(root,cfg,fail_under=None): - results=[] - for name,c in cfg.checks.items(): - if not c.enabled:continue - r=run(name,root,c.config,cfg.timeout_seconds,cfg.source_paths,cfg.test_paths);r.weight=c.weight;r.required=name in cfg.required_tools;results.append(r) - included=[r for r in results if r.status not in (Status.unavailable,Status.skipped) or r.required] - denominator=sum(r.weight for r in included);score=100*sum(r.weight for r in included if r.status==Status.passed)/denominator if denominator else 0 - threshold=cfg.fail_under if fail_under is None else fail_under - required_ok=all(not r.required or r.status==Status.passed for r in results) - unavailable=[r.name for r in results if r.status==Status.unavailable and not r.required] - limitations=["Passing configured checks is not proof of correctness.","pytest and import checks may execute repository code."] - if unavailable:limitations.append("Optional checks excluded from score: "+", ".join(unavailable)) - return ScanReport(project=root.name,target=str(root),generated_at=datetime.now(timezone.utc).isoformat(),codeproof_version=__version__,score=round(score,2),grade=grade(score),policy_passed=score>=threshold and required_ok,fail_under=threshold,results=results,limitations=limitations) +from .models import CheckResult, ScanReport, Status + + +def grade(x: float) -> str: + return ( + "A" + if x >= 90 + else "B" + if x >= 80 + else "C" + if x >= 70 + else "D" + if x >= 60 + else "F" + ) + + +def scan(root, cfg, fail_under=None, jobs=None): + threshold = cfg.fail_under if fail_under is None else fail_under + workers = cfg.jobs if jobs is None else jobs + if not 0 <= threshold <= 100: + raise ValueError("fail_under must be between 0 and 100") + if not 1 <= workers <= 8: + raise ValueError("jobs must be between 1 and 8") + selected = [(name, check) for name, check in cfg.checks.items() if check.enabled] + + def execute(item): + name, check = item + result = run( + name, + root, + check.config, + cfg.timeout_seconds, + cfg.source_paths, + cfg.test_paths, + ) + result.weight = check.weight + result.required = name in cfg.required_tools + return result + + # map preserves policy order even when workers finish out of order. + with ThreadPoolExecutor(max_workers=workers) as pool: + results = list(pool.map(execute, selected)) + for name in dict.fromkeys(cfg.required_tools): + if not any(result.name == name for result in results): + results.append( + CheckResult( + name=name, + status=Status.skipped, + required=True, + summary="Required check is disabled or absent from the policy", + ) + ) + included = [ + r + for r in results + if r.status not in (Status.unavailable, Status.skipped) or r.required + ] + denominator = sum(r.weight for r in included) + score = ( + 100 * sum(r.weight for r in included if r.status == Status.passed) / denominator + if denominator + else 0 + ) + required_ok = all(not r.required or r.status == Status.passed for r in results) + unavailable = [ + r.name for r in results if r.status == Status.unavailable and not r.required + ] + limitations = [ + "Passing configured checks is not proof of correctness.", + "pytest and import checks may execute repository code.", + ] + if unavailable: + limitations.append( + "Optional checks excluded from score: " + ", ".join(unavailable) + ) + if denominator == 0: + limitations.append( + "No weighted checks contributed evidence; policy cannot pass." + ) + return ScanReport( + project=root.name, + target=str(root), + generated_at=datetime.now(timezone.utc).isoformat(), + codeproof_version=__version__, + score=round(score, 2), + grade=grade(score), + policy_passed=score >= threshold and required_ok and denominator > 0, + fail_under=threshold, + results=results, + limitations=limitations, + ) diff --git a/src/codeproof/discovery.py b/src/codeproof/discovery.py index ef78d1b..f010d17 100644 --- a/src/codeproof/discovery.py +++ b/src/codeproof/discovery.py @@ -1,19 +1,46 @@ import ast +from collections import defaultdict from pathlib import Path -def symbols(root:Path,sources:list[str]): - out=[] - for src in sources: - base=root/src - for p in base.rglob("*.py") if base.exists() else []: - try:tree=ast.parse(p.read_text()) - except SyntaxError:continue - for n in tree.body: - if isinstance(n,(ast.FunctionDef,ast.AsyncFunctionDef,ast.ClassDef)) and not n.name.startswith("_"):out.append((n.name,p.relative_to(root),n.lineno)) + +from .files import python_files + + +def symbols(root: Path, sources: list[str]): + out = [] + for path, _ in python_files(root, sources): + try: + tree = ast.parse(path.read_bytes()) + except (SyntaxError, UnicodeError): + continue + for node in tree.body: + if isinstance( + node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef) + ) and not node.name.startswith("_"): + out.append((node.name, path.relative_to(root), node.lineno)) return out -def generate(root:Path,sources:list[str],output:Path,force=False): - output.mkdir(parents=True,exist_ok=True);made=[] - for name,p,line in symbols(root,sources): - src=next((s for s in sources if str(p).startswith(s+"/")),sources[0]);mod=".".join(Path(p).relative_to(src).with_suffix("").parts);dest=output/f"test_{mod.replace('.','_')}_generated.py" - text=(dest.read_text() if dest.exists() else f'import importlib\nmodule=importlib.import_module("{mod}")\n')+f'\ndef test_{name}_is_exposed():\n assert hasattr(module,"{name}")\n' - if force or not dest.exists():dest.write_text(text);made.append(dest) + + +def generate(root: Path, sources: list[str], output: Path, force=False): + output.mkdir(parents=True, exist_ok=True) + groups = defaultdict(list) + bases = {path.relative_to(root): base for path, base in python_files(root, sources)} + for name, path, _ in symbols(root, sources): + parts = list((root / path).relative_to(bases[path]).with_suffix("").parts) + if parts[-1] == "__init__": + parts.pop() + module = ".".join(parts) + if module: + groups[module].append(name) + made = [] + for module, names in groups.items(): + # Length-prefix components to avoid collisions such as a_b.c and a.b_c. + slug = "_".join(f"{len(part)}_{part}" for part in module.split(".")) + dest = output / f"test_{slug}_generated.py" + if dest.exists() and not force: + continue + text = f"import importlib\n\nmodule = importlib.import_module({module!r})\n" + for name in names: + text += f"\n\ndef test_{name}_is_exposed():\n assert hasattr(module, {name!r})\n" + dest.write_text(text, encoding="utf-8") + made.append(dest) return made diff --git a/src/codeproof/files.py b/src/codeproof/files.py new file mode 100644 index 0000000..842eda8 --- /dev/null +++ b/src/codeproof/files.py @@ -0,0 +1,49 @@ +"""Shared deterministic traversal; never descend into caches or environments.""" + +import os +from pathlib import Path + +EXCLUDED = { + ".git", + ".venv", + "venv", + "env", + "__pycache__", + ".pytest_cache", + ".mypy_cache", + ".ruff_cache", + ".codeproof", + "node_modules", + "build", + "dist", +} + + +def python_files(root: Path, sources: list[str]): + seen = set() + for source in sources: + base = root / source + if base.is_file(): + candidates = [base] if base.suffix == ".py" else [] + else: + candidates = [] + for directory, folders, names in os.walk(base, followlinks=False): + folders[:] = sorted( + name + for name in folders + if name not in EXCLUDED + and not (Path(directory) / name).is_symlink() + ) + candidates.extend( + Path(directory) / name + for name in sorted(names) + if name.endswith(".py") + ) + for path in candidates: + if path.is_symlink(): + continue + resolved = path.resolve() + if not resolved.is_relative_to(root.resolve()) or resolved in seen: + continue + seen.add(resolved) + yield path, base if base.is_dir() else base.parent diff --git a/src/codeproof/models.py b/src/codeproof/models.py index bdf46fb..b35a30f 100644 --- a/src/codeproof/models.py +++ b/src/codeproof/models.py @@ -1,11 +1,48 @@ from enum import Enum from typing import Any + from pydantic import BaseModel, Field -class Status(str,Enum): - passed="passed"; failed="failed"; unavailable="unavailable"; error="error"; skipped="skipped" + + +class Status(str, Enum): + passed = "passed" + failed = "failed" + unavailable = "unavailable" + error = "error" + skipped = "skipped" + + class Finding(BaseModel): - level:str="info"; message:str; path:str|None=None; line:int|None=None; rule:str|None=None + level: str = "info" + message: str + path: str | None = None + line: int | None = None + rule: str | None = None + + class CheckResult(BaseModel): - name:str; status:Status; command:list[str]=Field(default_factory=list); duration_seconds:float=0; exit_code:int|None=None; summary:str=""; findings:list[Finding]=Field(default_factory=list); evidence:dict[str,Any]=Field(default_factory=dict); raw_output:str|None=None; weight:float=0; required:bool=False + name: str + status: Status + command: list[str] = Field(default_factory=list) + duration_seconds: float = 0 + exit_code: int | None = None + summary: str = "" + findings: list[Finding] = Field(default_factory=list) + evidence: dict[str, Any] = Field(default_factory=dict) + raw_output: str | None = None + weight: float = 0 + required: bool = False + + class ScanReport(BaseModel): - schema_version:str="1.0"; project:str; target:str; generated_at:str; codeproof_version:str; score:float; grade:str; policy_passed:bool; fail_under:float; results:list[CheckResult]; limitations:list[str]=Field(default_factory=list) + schema_version: str = "1.0" + project: str + target: str + generated_at: str + codeproof_version: str + score: float + grade: str + policy_passed: bool + fail_under: float + results: list[CheckResult] + limitations: list[str] = Field(default_factory=list) diff --git a/src/codeproof/report_template.html b/src/codeproof/report_template.html new file mode 100644 index 0000000..e62b987 --- /dev/null +++ b/src/codeproof/report_template.html @@ -0,0 +1,64 @@ + + +
+ +A clearer picture of your repository’s health.
@@passed@@ of @@total@@ checks passed
Threshold: @@threshold@@ / 100
All required checks must pass
@@findings@@ normalized findings
Expand a check for its evidence
Inspect the evidence behind your score.
No matching checks. Try another search or filter.
Policy: {"PASS" if r.policy_passed else "FAIL"}
| Check | Status | Weight | Summary |
|---|
{escape(item.message)}
" + + escape(str(result.command)) + + "" + ) + if result.raw_output: + detail += "
" + escape(result.raw_output) + "" + if result.evidence: + detail += "
" + escape(str(result.evidence)) + "" + cards.append( + f'
Policy weight: {result.weight:g}
{detail or "No additional evidence.
"}Your code quality, security, and test evidence. In one place.
Find issues earlier. Understand what changed.
Give every release a clear quality signal.
Your latest completed runs
Ready for your next check
Start a scan to build your first report.
Up to 50 runs saved locally, including interrupted scans.
See new and resolved findings between two completed scans.
Choose two runs to see what changed.
Use the same engine in your terminal and CI pipeline.
Checks contribute a weighted score. Required checks must pass, even when the score exceeds your threshold.
Scan presets include required policy tools. Disabled required tools still fail the policy. Scan settings never overwrite your repository policy.
Practical workflows for individual developers, maintainers, and engineering teams.