Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions usvm-ts-fast-check/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,21 @@ useful separation is preserved: declarative receiver/argument/result positions a
values, and condition interpretation is distinct from position resolution. The TypeScript mapper expresses this
with EtsIR-specific binding and mapping records and has no dependency on `usvm-jvm` or the taint-analysis module.

## USVM projection and property search

The existing projection path configures `TsMachine`'s initial state from exact mapper bindings and declared Kotlin
domains. The existing search path prepends a mapped synchronous precondition to the predicate entry point. Guard
completion is explicit in `TsState`: false terminates a rejected path, while an exception or non-boolean result
terminates an error path. Neither path can reach the predicate target.

Predicate false paths are re-solved on a cloned terminal state before target propagation, so the ordinary terminal
state is not rewritten. Predicate exceptions reach the same candidate target. Runtime non-boolean entry-point
results are property errors regardless of their TypeScript return annotation. A residual call that stops a path is
unsupported, while ordinary unsatisfiable path pruning is not an engine failure. Timeout, solver uncertainty,
interpreter failure, and candidate-input resolution failure retain separate search outcomes. Candidate extraction
reads the projected input state with the terminal model, so predicate-local array mutation does not rewrite the
reported input.

## Process supervision

`FastCheckProcessTransport` writes stdin and drains stdout and stderr concurrently. This is necessary because each
Expand Down Expand Up @@ -286,7 +301,9 @@ classifier because `tsx` depends on a native esbuild package.
- Backend integration tests execute real uncompiled TypeScript through the packaged adapter, including replay,
shrinking, explicit examples, preconditions, async predicates, and timeouts.
- Shared contract fixtures cover precondition admission, discard and errors; predicate violations and errors;
special values; aliases; mutation isolation; shrinking; and replay through observable outcomes.
special values; aliases; mutation isolation; shrinking; and replay through observable outcomes. One focused JVM
conformance test executes the same classification fixture through the real FastCheck and USVM paths, including
literal and never return annotations and replay of a pre-mutation USVM candidate.
- Coverage golden tests assert literal TypeScript statement and branch outcomes for successful and falsified runs,
cross-property isolation, scope and glob filtering, and source-map/report diagnostics.
- Mapping golden tests load stable TypeScript fixtures through the native frontend and cover predicate,
Expand All @@ -299,5 +316,5 @@ classifier because `tsx` depends on a native esbuild package.
- Discovering properties by scanning TypeScript source roots.
- Compiling user TypeScript as part of the PBT workflow.
- Reimplementing generation, replay, skip accounting, or shrinking in Kotlin.
- Constructing symbolic inputs or executing mapped properties in USVM.
- General purity analysis, arbitrary mutable-object projection, and persistent state across property invocations.
- Combining backend source coverage with future EtsIR replay coverage.
3 changes: 3 additions & 0 deletions usvm-ts-fast-check/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ dependencies {
implementation(Libs.clikt)
implementation(Libs.kotlinx_serialization_json)

testImplementation(project(":usvm-core"))
testImplementation(project(":usvm-ts"))
testImplementation(Libs.jacodb_ets)
testImplementation(Libs.logback)
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
package org.usvm.ts.pbt.usvm

import org.jacodb.ets.model.EtsScene
import org.jacodb.ets.utils.EtsIrProvider
import org.jacodb.ets.utils.loadEtsFileAutoConvert
import org.junit.jupiter.api.BeforeAll
import org.junit.jupiter.api.Test
import org.usvm.ts.pbt.backend.ProjectionCapability
import org.usvm.ts.pbt.backend.ProjectionLevel
import org.usvm.ts.pbt.backend.PropertyFailureKind
import org.usvm.ts.pbt.backend.PropertyRunConfiguration
import org.usvm.ts.pbt.backend.PropertyRunStatus
import org.usvm.ts.pbt.fastcheck.BackendErrorKind
import org.usvm.ts.pbt.fastcheck.FastCheckBackend
import org.usvm.ts.pbt.fastcheck.PbtBackendException
import org.usvm.ts.pbt.manifest.toManifest
import org.usvm.ts.pbt.mapping.PropertyEtsMapper
import org.usvm.ts.pbt.model.ArrayDomain
import org.usvm.ts.pbt.model.IntegerDomain
import org.usvm.ts.pbt.model.JsConcreteValue
import org.usvm.ts.pbt.model.PropertyDefinition
import org.usvm.ts.pbt.model.PropertyId
import org.usvm.ts.pbt.model.PropertyInput
import org.usvm.ts.pbt.model.TypeScriptEntryPoint
import org.usvm.ts.pbt.model.contains
import org.usvm.ts.pbt.testResourcePath
import org.usvm.ts.pbt.testResourcesRoot
import kotlin.test.assertEquals
import kotlin.test.assertNotNull
import kotlin.test.assertTrue

class PropertyExecutionConformanceTest {
@Test
fun `shared preconditions have the same concrete and search classification`() {
val cases = listOf(
ContractCase("truePrecondition", ContractOutcome.HOLDS),
ContractCase("falsePrecondition", ContractOutcome.PRECONDITION_REJECTED),
ContractCase(
exportName = "throwingPrecondition",
expected = ContractOutcome.PROPERTY_ERROR,
expectedBackendError = BackendErrorExpectation(
code = "entrypoint.precondition.threw",
path = "manifest.precondition",
),
),
ContractCase(
exportName = "nonBooleanPrecondition",
expected = ContractOutcome.PROPERTY_ERROR,
expectedBackendError = BackendErrorExpectation(
code = "entrypoint.result.invalid",
path = "manifest.precondition.result",
),
),
)

cases.forEach { case ->
val property = property(
predicateExport = "alwaysTrue",
preconditionExport = case.exportName,
)
val manifest = property.toManifest()
val mapping = mapper.map(manifest)

val concreteOutcome = concreteOutcome(property, case.expectedBackendError)
val searchOutcome = searchOutcome(
searcher.search(manifest, mapping, exactCapability),
)

assertEquals(case.expected, concreteOutcome, "${case.exportName}: concrete")
assertEquals(case.expected, searchOutcome, "${case.exportName}: search")
}
}

@Test
fun `shared predicates have the same concrete and search classification`() {
val cases = listOf(
ContractCase("falsePredicate", ContractOutcome.PREDICATE_VIOLATION),
ContractCase("throwingPredicate", ContractOutcome.PREDICATE_VIOLATION),
ContractCase("literalFalsePredicate", ContractOutcome.PREDICATE_VIOLATION),
ContractCase("literalTruePredicate", ContractOutcome.HOLDS),
ContractCase("neverPredicate", ContractOutcome.PREDICATE_VIOLATION),
ContractCase(
exportName = "nonBooleanPredicate",
expected = ContractOutcome.PROPERTY_ERROR,
expectedBackendError = BackendErrorExpectation(
code = "entrypoint.result.invalid",
path = "manifest.predicate.result",
),
),
)

cases.forEach { case ->
val property = property(predicateExport = case.exportName)
val manifest = property.toManifest()
val mapping = mapper.map(manifest)

val concreteOutcome = concreteOutcome(property, case.expectedBackendError)
val searchOutcome = searchOutcome(
searcher.search(manifest, mapping, exactCapability),
)

assertEquals(case.expected, concreteOutcome, "${case.exportName}: concrete")
assertEquals(case.expected, searchOutcome, "${case.exportName}: search")
}
}

@Test
fun `symbolic candidates retain inputs from before predicate mutation`() {
val domain = ArrayDomain(
element = IntegerDomain(min = 1, max = 1),
minLength = 1,
maxLength = 1,
)
val property = property(
predicateExport = "mutatesAndFails",
domain = domain,
)
val manifest = property.toManifest()
val mapping = mapper.map(manifest)

val concreteResult = backend.run(property, configuration)
val searchResult = searcher.search(manifest, mapping, exactCapability)

val expectedInput = JsConcreteValue.Array(
elements = listOf(JsConcreteValue.number(1.0)),
)
val symbolicInputs = assertNotNull(searchResult.inputs)
assertEquals(PropertyFailureKind.PROPERTY, concreteResult.failure?.kind)
assertEquals(listOf(expectedInput), concreteResult.counterexample)
assertEquals(UsvmPropertySearchStatus.VIOLATION_REACHED, searchResult.status)
assertEquals(listOf(expectedInput), symbolicInputs)
assertTrue(symbolicInputs.single() in domain)

val replay = backend.run(
property = property,
configuration = configuration.copy(
numRuns = 1,
examples = listOf(symbolicInputs),
),
)

assertEquals(PropertyFailureKind.PROPERTY, replay.failure?.kind)
assertEquals(symbolicInputs, replay.counterexample)
}

private fun concreteOutcome(
property: PropertyDefinition,
expectedBackendError: BackendErrorExpectation?,
): ContractOutcome = try {
val result = backend.run(property, configuration)

when (result.status) {
PropertyRunStatus.SUCCESS -> ContractOutcome.HOLDS
PropertyRunStatus.FAILURE -> when (result.failure?.kind) {
PropertyFailureKind.PROPERTY -> ContractOutcome.PREDICATE_VIOLATION
PropertyFailureKind.PRECONDITION_EXHAUSTED -> ContractOutcome.PRECONDITION_REJECTED
PropertyFailureKind.TIMEOUT, null -> error("Unexpected concrete result: $result")
}
}
} catch (failure: PbtBackendException) {
val expected = checkNotNull(expectedBackendError) {
"Unexpected concrete backend error: ${failure.kind}/${failure.code} at ${failure.path}"
}

assertEquals(BackendErrorKind.ENTRY_POINT, failure.kind)
assertEquals(expected.code, failure.code)
assertEquals(expected.path, failure.path)

ContractOutcome.PROPERTY_ERROR
}

private fun searchOutcome(result: UsvmPropertySearchResult): ContractOutcome = when (result.status) {
UsvmPropertySearchStatus.VIOLATION_REACHED -> ContractOutcome.PREDICATE_VIOLATION
UsvmPropertySearchStatus.NO_VIOLATION_REACHED -> ContractOutcome.HOLDS
UsvmPropertySearchStatus.PRECONDITION_REJECTED -> ContractOutcome.PRECONDITION_REJECTED
UsvmPropertySearchStatus.PROPERTY_ERROR -> ContractOutcome.PROPERTY_ERROR
else -> error("Unexpected search result: $result")
}

private fun property(
predicateExport: String,
preconditionExport: String? = null,
domain: org.usvm.ts.pbt.model.PropertyDomain = IntegerDomain(min = 0, max = 0),
) = PropertyDefinition(
id = PropertyId("contract.$predicateExport.${preconditionExport ?: "none"}"),
inputs = listOf(
PropertyInput(
name = "value",
domain = domain,
),
),
predicate = TypeScriptEntryPoint(
module = MODULE,
exportName = predicateExport,
),
precondition = preconditionExport?.let { exportName ->
TypeScriptEntryPoint(
module = MODULE,
exportName = exportName,
)
},
)

private data class ContractCase(
val exportName: String,
val expected: ContractOutcome,
val expectedBackendError: BackendErrorExpectation? = null,
)

private data class BackendErrorExpectation(
val code: String,
val path: String,
)

private enum class ContractOutcome {
HOLDS,
PRECONDITION_REJECTED,
PREDICATE_VIOLATION,
PROPERTY_ERROR,
}

companion object {
private const val MODULE = "PropertyExecutionContract.ts"

private val exactCapability = ProjectionCapability(level = ProjectionLevel.EXACT)
private val configuration = PropertyRunConfiguration(
seed = 42,
numRuns = 1,
timeoutMillis = 1_000,
)
private val fixtureDirectory = testResourcesRoot().resolve("properties/contract")
private val backend = FastCheckBackend(sourceRoots = listOf(fixtureDirectory))
private lateinit var mapper: PropertyEtsMapper
private lateinit var searcher: UsvmPropertySearcher

@JvmStatic
@BeforeAll
fun loadFixture() {
val source = testResourcePath("/properties/contract/$MODULE")
val file = loadEtsFileAutoConvert(source, provider = EtsIrProvider.TS_FRONTEND)
val scene = EtsScene(listOf(file))

mapper = PropertyEtsMapper(
scene = scene,
sourceRoots = listOf(fixtureDirectory),
)
searcher = UsvmPropertySearcher(scene)
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package org.usvm.ts.pbt.usvm

import org.junit.jupiter.api.Test
import org.usvm.ts.pbt.fastcheck.FastCheckProjectionClient
import org.usvm.ts.pbt.fastcheck.FastCheckProjectionRequest
import org.usvm.ts.pbt.model.ArrayDomain
import org.usvm.ts.pbt.model.BooleanDomain
import org.usvm.ts.pbt.model.IntegerDomain
import org.usvm.ts.pbt.model.JsNumber
import org.usvm.ts.pbt.model.NumberDomain
import org.usvm.ts.pbt.model.OptionalDomain
import org.usvm.ts.pbt.model.PropertyDomain
import org.usvm.ts.pbt.model.TupleDomain
import org.usvm.ts.pbt.model.contains
import kotlin.test.assertEquals
import kotlin.test.assertTrue

class UsvmProjectionConformanceTest {
@Test
fun `fast-check samples satisfy the same domains used by USVM constraint tests`() {
val domains = listOf<PropertyDomain>(
IntegerDomain(min = -5, max = 7),
NumberDomain(
min = JsNumber.finite(-1.5),
max = JsNumber.finite(2.5),
allowNaN = false,
),
OptionalDomain(IntegerDomain(min = 1, max = 3)),
TupleDomain(listOf(IntegerDomain(min = 2, max = 4), BooleanDomain)),
ArrayDomain(IntegerDomain(min = -1, max = 1), minLength = 1, maxLength = 3),
)
val response = FastCheckProjectionClient().sample(
FastCheckProjectionRequest(
seed = 351,
numSamples = 50,
domains = domains,
),
)

assertEquals(50, response.samples.size)
response.samples.forEach { sample ->
assertEquals(domains.size, sample.size)
sample.zip(domains).forEach { (value, domain) ->
assertTrue(value in domain, "$value is outside $domain")
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,26 @@ export async function asyncNonBooleanWhenPositivePredicate(value: number): Promi
return value > 0 ? 42 : false;
}

export function literalFalsePredicate(_value: number): false {
return false;
}

export function literalTruePredicate(_value: number): true {
return true;
}

export function neverPredicate(_value: number): never {
throw 'never predicate exploded';
}

export function catchesExpectedException(_value: number): boolean {
try {
throw 'expected';
} catch (error: unknown) {
return error === 'expected';
}
}

export function recognizesSpecialValues(
missing: undefined,
empty: null,
Expand Down
12 changes: 6 additions & 6 deletions usvm-ts-pbt/PROPERTY_EXECUTION_CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,13 +82,13 @@ was reached within that search.

## Implementation and regression points

- `fast-check-adapter/src/execute-property.ts` applies this contract to generation, explicit examples, replay, and
- `usvm-ts-fast-check/fast-check-adapter/src/execute-property.ts` applies this contract to generation, explicit examples, replay, and
shrinking through the existing fast-check invocation.
- `fast-check-adapter/src/project-domain.ts` projects the declared Kotlin domains for concrete execution.
- Downstream USVM projection and search implementations consume the same manifest and mapping artifacts and must
link to this contract when their dependent changes are integrated.
- `src/test/resources/properties/contract/PropertyExecutionContract.ts` provides concrete regression coverage;
downstream symbolic integration extends the same fixture with symbolic assertions.
- `usvm-ts-fast-check/fast-check-adapter/src/project-domain.ts` projects the declared Kotlin domains for concrete execution.
- `UsvmPropertySearcher` applies this contract to USVM domain projection and search in one execution path.
`PropertyExecutionConformanceTest` runs the same TypeScript fixture through `FastCheckBackend` and USVM.
- `usvm-ts-fast-check/src/test/resources/properties/contract/PropertyExecutionContract.ts` is the shared observable fixture for
concrete and symbolic contract regressions.

Replay remains ordinary concrete execution with the reported seed and path. It does not introduce a separate
property runner or alternate callback semantics.
Loading