Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions docs/ts-pbt/CORE_CAPABILITY_SHORTLIST.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# #356 core corpus: capability-first source screening

This is a read-only source inventory for a successor to development v1 (`ae7297699cc5036b54b8e3c54b1b42a2ca515705`). It was assembled before comparative outcomes. It does **not** add cases to the frozen v1 manifest, register a native suite, or assert direct symbolic support. Admission must follow original-runtime validation and a #395/#351–#352 capability result. Keep every v1 case and its outcomes on the development side.

| Source, pinned revision, license | Original human-written assertion and generator | Operations / capability decision |
| --- | --- | --- |
| [helpers4 `isNonEmpty.spec.ts`](https://github.com/helpers4/typescript/blob/9deb33a6f6cdab3c03404217aa8e87626930733e/helpers/array/isNonEmpty.spec.ts), [`isNonEmpty.ts`](https://github.com/helpers4/typescript/blob/9deb33a6f6cdab3c03404217aa8e87626930733e/helpers/array/isNonEmpty.ts), [`LICENSE`](https://github.com/helpers4/typescript/blob/9deb33a6f6cdab3c03404217aa8e87626930733e/LICENSE) — `9deb33a6f6cdab3c03404217aa8e87626930733e`, LGPL-3.0-or-later, copyright 2025 baxyz | `fc.array(fc.anything(), { minLength: 1 })`; `expect(isNonEmpty(arr)).toBe(true)`. A second property uses `fc.array(fc.integer(), { minLength: 1 })`, then `if (isNonEmpty(arr)) expect(arr[0]).toBeDefined()`. | First oracle: original SUT checks `value != null && value.length > 0`; no dependent generator, Set, flat, every or Math. **Preferred numeric-array candidate** with bounded dense integer-array subset length 1..8. `anything` distribution and non-numeric original support remain native-only outside the common subset. Check `arr[0]` separately before admitting the second assertion. Vendoring requires LGPL license and modification notices; no source was copied in this inventory. |
| [IgniteUI `math.property.spec.ts`](https://github.com/IgniteUI/igniteui-webcomponents/blob/d468f25bb12f72e05c5f27243fffd77c37e2107e/src/internals/utils/math.property.spec.ts), [`math.ts`](https://github.com/IgniteUI/igniteui-webcomponents/blob/d468f25bb12f72e05c5f27243fffd77c37e2107e/src/internals/utils/math.ts), [`orderedPair`](https://github.com/IgniteUI/igniteui-webcomponents/blob/d468f25bb12f72e05c5f27243fffd77c37e2107e/src/internals/testing/fast-check-setup.spec.ts), [`LICENSE`](https://github.com/IgniteUI/igniteui-webcomponents/blob/d468f25bb12f72e05c5f27243fffd77c37e2107e/LICENSE) — `d468f25bb12f72e05c5f27243fffd77c37e2107e`, MIT, copyright 2025 INFRAGISTICS | `fc.property(number, orderedPair(finite), (value,[min,max]) => expect(wrap(min,max,value)).to.be.within(min,max))`; `number` permits infinities but not NaN, `finite` excludes both, `orderedPair` sorts a generated tuple via `Math.min/Math.max`. | `wrap` SUT uses only `<`, `>` and argument returns; inclusive range oracle is meaningful. **Concrete-only under the current #395 contract**: the joint sorted-pair support is not registered/projected. A bounded integer subset must retain the declared `min <= max` relation and disclose changed distribution; inventing an unrelated precondition is not an import. `clamp` in the same suite also has idempotence and in-range preservation assertions, but its SUT uses `Math.min/Math.max`, so it is not a simpler core admission. |
| [es-toolkit `chunk.spec.ts`](https://github.com/toss/es-toolkit/blob/4504e985a433fe1e348d803a233b2c02d41e4b3c/src/array/chunk.spec.ts) — MIT | Preserves order after flattening, non-last chunk size, and nonempty bounded chunks. | Existing v1 family; `flat`/`every` and array structure need explicit symbolic capability. Retain in inventory, possibly concrete-only. |
| [fast-check `ArrayArbitrary.spec.ts`](https://github.com/dubzzz/fast-check/blob/85eeab9e87c9d37e66cc7819260e3df1e72305ae/packages/fast-check/test/arbitraries/ArrayArbitrary.spec.ts) — MIT | Deliberately faulty identity deduplicator; original `expect(filtered).toHaveLength(new Set(filtered).size)`. | Existing v1 family and #395 registration path; `Set` still requires symbolic capability. The local Set-based correct baseline is not an upstream implementation. |
| [fleet-sdk `zigZag.spec.ts`](https://github.com/fleet-sdk/fleet/blob/8fd834a94a7534069ed78af3538f518068ca1e2e/packages/serializer/src/coders/zigZag.spec.ts) — MIT | `fc.integer({min:MIN_I32,max:MAX_I32})`; decode(encode(n)) equals n. | Exclude from core direct comparison: SUT uses BigInt conversion, shifts, bitwise operations and helper imports, beyond the current bounded numeric subset. Retain for later inventory. |

## Admission rule for the next development revision

Select the first original oracle whose **entire tested call and predicate** have supported semantics on an explicitly bounded common subset, based on APIs and language operations before viewing search outcomes. Keep the original native generator and assertion callback for a native control; record any bounded subset and changed distribution as an adaptation, not as original support. Run the intended-correct implementation with the original oracle, then check concrete replay and direct symbolic capability against the same subset. `UNSUPPORTED`, `UNKNOWN`, timeout and failed mapping never count as a negative fault result. If the preferred helpers4 source cannot be redistributed with its LGPL obligations or fails the direct symbolic check, keep it concrete-only and record the reason; do not silently replace its assertion with a weaker one. A later admitted set and settings become development v2, preserving v1 and all pilot traces. Final held-out eligibility is decided by project/family rules independent of these outcomes.
27 changes: 27 additions & 0 deletions docs/ts-pbt/CORE_NEAREST_WORK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# #404 nearest-work and provisional claim ledger

Status: early input to #405, before pilot data or a comprehensive novelty review. `Yes` below means the cited primary source explicitly supports the stated mechanism; `unverified` means this pass did not establish its absence. Language alone is not a novelty argument.

| Work / primary evidence | Reused test or input structure | Feedback / inference | Symbolic challenge and replay | Boundary for this roadmap |
| --- | --- | --- | --- | --- |
| [JQF/Zest, ISSTA 2019](https://doi.org/10.1145/3293882.3330576) and [JQF project](https://github.com/rohanpadhye/JQF) | Parameterized Java property tests and pluggable generators | Coverage and validity feedback guide semantically valid generation | No symbolic relation challenge established by the cited overview | Reusing PBT generators and feeding coverage back are prior work; compare beyond coverage. |
| [Daikon project](https://plse.cs.washington.edu/daikon/) | Program execution traces | Reports **likely** point-specific invariants from observed values | No testing loop established by the cited overview | Observed relations are hypotheses, not facts or new by themselves. |
| [DSD-Crasher, TOSEM 2008 abstract](https://plse.cs.washington.edu/daikon/pubs/CsallnerSX2008-abstract.html) | Program tests/observations | Dynamic invariants restrict static analysis; a final dynamic step confirms predictions | Yes: dynamic–static–dynamic testing | Dynamic invariant plus static search plus concrete confirmation is direct prior work. Our proposed distinction needs a measured, assertion-specific target/feedback mechanism, not a generic D–S–D claim. |
| [QSYM, USENIX Security 2018](https://www.usenix.org/conference/usenixsecurity18/presentation/yun) | Fuzzer inputs for binaries | Hybrid fuzzing with concolic execution and fuzzer validation | Yes, bidirectional hybrid testing; not a user PBT oracle in the cited abstract | Returning symbolic inputs to a fuzzer and validating them is prior work. The intended TypeScript property/oracle semantics must show incremental value. |
| [HypoFuzz project](https://hypofuzz.com/) | Existing Python Hypothesis tests | Fuzzing backend and coverage dashboard; finer mechanism requires source review | Symbolic relation challenge unverified here | Existing PBT reuse and coverage are established beyond JQF; do not claim those features as novel. |

The supplied private 2026 Go/gopter + usvm-go thesis is **not redistributed** here. Its recorded mechanisms include uncovered-line guidance, input-corpus transfer and `behaviorKey` novelty filtering. Author/title/public citation must be verified with the owner before referencing it in a public paper. The output-novelty control in the pilot is required partly to test this boundary. The full #404 review must also inspect generator-choice search, metamorphic and stateful/model-based PBT, internal-context summaries, and newer property-directed testing against their primary papers; the table above is deliberately incomplete.

## Claim → required evidence

| Proposed statement | Evidence needed before making it | Current status |
| --- | --- | --- |
| The bounded core loop works | A real observation yields a nontrivial assertion-relevant relation; it changes an actual target; a replay-confirmed refutation changes later generated inputs; exact branch signal affects scheduling; all costs and mismatches are logged. | Unmeasured; fixtures and protocol only. |
| It finds more distinct faults or confirms them faster than baselines | Paired equal-budget development results, then independently frozen held-out results, with original oracle, common support, uncertainty and regressions. Compare direct property search and sequential portfolio. | Unmeasured; superiority provisional. |
| Assertion focus adds value beyond generic invariants | Focused versus unfocused inference, equal vocabulary/target budget and controlled target ordering. | Planned contrast only. |
| Observed relations add value beyond coverage/templates/output novelty | Coverage-only, observation-independent templates and output-novelty controls on the same eligible cases and budget. | Planned contrast only. |
| Extensions add value | End-to-end #400–#403 enable/disable comparisons on their supported strata after the core checkpoint. | Out of this early checkpoint. |
| A concrete witness is valid | Original-runtime replay of that exact input against the original predicate, with admissibility and execution outcome recorded. This confirms the witness only. | No witness from a search campaign yet. |
| Any bounded symbolic conclusion is sound | A separate argument for the model, supported semantics, binding/guards, solver assumptions and bounded scope; replay cannot establish general soundness or completeness, and finite unsuccessful search proves no property. | No proof claim or proof argument. |

The final ledger must link each empirical statement to a pinned implementation revision, corpus manifest, raw run IDs and table-regeneration command. #405 may report a negative or inconclusive pilot without turning functional completion into an improvement claim. No manuscript or external publication is in this PR.
19 changes: 19 additions & 0 deletions docs/ts-pbt/CORE_PILOT_PROTOCOL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# #405 core development comparison: preregistration draft

Status: protocol prepared before comparative runs. Use `usvm-ts-pbt/corpus/development/v1/manifest.json` as the initial development selection. #357's final held-out protocol will be frozen later; no results are claimed here. Record the implementation commit, corpus revision, property registration, configuration and machine/runtime versions with every campaign.

## Common denominator and controls

For each eligible case, retain the **same original TypeScript predicate/assertions**, declared support and precondition, exact input value semantics, and original-runtime confirmation in every mode. Fast-check distribution bias is measured separately from hard support. Case admission is per capability: only the intersection that all compared modes execute faithfully enters a paired comparison. Publish both that intersection and the full inventory with `supported`, `unsupported`, `UNKNOWN`, timeout and error counts. A hypothesis refutation that passes the predicate is not a fault. A false specification is a separate diagnosis.

Run six core configurations under a common total wall-clock deadline per case and seed: `PBT_ONLY`, `SYMBOLIC_ONLY`, `SEQUENTIAL` (fixed PBT then symbolic allocation, no feedback), `COVERAGE_FEEDBACK` (exact branch signal), `RELATION_FEEDBACK` (observations and symbolic challenges), and `COMBINED_FEEDBACK` (coverage plus relation, replay and returned-input rounds). Freeze the scheduler, phase allocations and target tie-break rules before observing comparisons. All symbolic modes retain direct original-property search and concrete confirmation. Covered branches remain searchable when fault search needs them. Report the native fast-check suite run with identical oracle, generator support and deadline as a bridge-overhead control; record any distribution mismatch explicitly.

Preplanned focused contrasts, on eligible cases only: (1) relation feedback versus observation-independent templates with **identical formula vocabulary and target budget**, preselected without observing outcomes; (2) property-focused versus unfocused observations/inference, fixed or randomized target ordering; (3) combined feedback versus a one-round/no-return variant; (4) combined feedback versus output-behavior novelty selection. These are planned contrasts, not a Cartesian product and not winner-dependent selections. #400–#403 get separate enable/disable comparisons in the final eligible strata. A relation-derived witness demonstrates functional causality only; compare it with direct original-property search before any advantage claim.

## Time, sampling and analysis

Set numeric deadlines, allocations, memory limit, observation points/templates, search bounds and seeds in a versioned run configuration **before** running the first comparison. For the pilot, start with at least ten independent campaign seeds per configuration and case; choose the final repetition count using development-only variance and save the change as a new protocol revision. Pair seeds across configurations, rotate/interleave run order, and record cold/warm setup. Predeclare whether shared build/setup is charged or amortized, then apply it uniformly. The total deadline includes frontend/adapter startup, instrumentation, observation, inference, mapping, solver search, original-runtime replay, returned-input generation and target-preserving shrinking. Report each cost and peak memory; retain interrupted runs as censored outcomes.

Primary outcomes are distinct original-oracle-confirmed faulty implementations, time to first confirmation (censored at deadline), marginal faults and regressions. Count a fault once regardless of witnesses or properties. Keep real defects, validated mutants, source-authored controlled faults and curated functional cases in separate tables. Secondary outcomes include capability denominator, adaptation time, exact source branch coverage, output novelty, observation truncation, relation support/contradictions, target yield, replay acceptance, returned-seed usefulness, fallback, per-phase cost and memory. EtsIR target reach is not concrete source coverage. Analyze paired runs within case, then summarize at project/family level; do not treat 10 seeds or near-duplicate mutants as 10 independent projects. Give effect sizes and interval estimates, show no-gain cases, and state the multiple-comparison treatment before final inference. Never select the best seed as the result.

The planned same-language coverage-guided comparator is a pilot research choice: assess an executable TypeScript/fast-check tool and freeze its version/settings before held-out runs, or document a concrete incompatibility. JQF/Zest is Java related work, not a directly comparable TypeScript score. Preserve raw inputs, lineage, observations, formulas, guards, mapping, attempted targets, solver outcomes, original replay, shrink steps, phase clocks and failure classifications. A future runner must offer one frozen-plan command and a separate raw-to-table regeneration command; neither exists at this early checkpoint.
Loading