[sudharshana-ni]

What does this Pull Request accomplish?

Adds ni.datastore.data.v2, a high-level Python wrapper around the V2 low-level client (ni.measurements.data.v2.client), plus V2 examples. Callers use Python objects and values instead of building protobuf requests and field masks.

DataStoreClient wraps every method of the V2 low-level client:

• Create: create_test_result, create_step, create_measurement (with limits, extension, and inline software/hardware/test adapter items, which must already be linked to the test result).
• Batch create: batch_create_test_results, batch_create_steps, batch_create_measurements. Test results can contain nested steps, and steps can contain child steps, conditions, and measurements. Each batch is atomic.
• Read: get_*, query_* (OData, optional read_fields) and read_measurement_values (value plus upper and lower limits).
• Update: update_* and batch_update_*. Update masks are built from keyword arguments; None clears a field. Unknown or read-only fields raise ValueError before any RPC.
• Conditions: add_condition (adds or replaces by name) and remove_condition.
• Metadata links: add/remove_test_result_metadata and add/remove_measurement_metadata. Following DEC-017, remove_test_result_metadata first checks the test result's measurements. If any still references an item being unlinked, it raises ValueError and nothing is removed; callers unlink the item from those measurements first.
• Delete: delete_* and batch_delete_*, with the service's force option for test results and steps.

Models (V1 style): TestResult, Step, Measurement, Condition, ErrorInformation, Outcome, Status, JsonValue, and QueryResults.

Values:

• Supported: bool, int, float, str, Scalar, Vector, lists of scalars, analog/complex/digital waveforms, Spectrum, XYData, and JSON dictionaries.
• Reads are consistent: a single value is always returned as a Scalar (with or without units), and a list as a Vector.
• When a Scalar is given without units, the measurement's units are taken from the Scalar, as Condition already does.
• Timestamps are UTC hightime.datetime, durations are hightime.timedelta, and extensions are dictionaries.

Forward-compatible fields: keywords, duration, uut_id, comparison_type, JSON values, and schema_id are included in the API. The service does not store them yet (New Resource Fields and Structured JSON features).

Documentation: docs/reference/ni-data-store-v2.md, linked from the reference index. It covers V1 differences, the DEC-017 unlink order, the status/outcome rule, the gRPC 4 MB receive limit for user-supplied channels, and waveform timestamp comparison.

Examples (V1 examples are unchanged):

• examples/overview-v2/: a standalone script project, the V2 companion of examples/overview/. Added to the check_examples CI matrix.
• examples/notebooks/v2/: notebooks for overview, sample data and queries, batch create, measurement values and limits, update and delete (including the DEC-017 unlink order), and structured extensions.
• Example channels remove gRPC's 4 MB receive limit, so large values such as long waveforms can be read.

Dependency note: The V2 client and proto packages currently come from the users/sudharshana/add-v2-proto-and-client-package branch of ni-apis-python, locked to commit 28aee05 in both poetry.lock and examples/overview-v2/poetry.lock. They will be switched to the published packages once that PR is merged and released. No generated protobuf code is copied into this repository. Relocking also updated ni-protobuf-types from 1.3.0.dev0 to 1.3.0.

Why should this Pull Request be merged?

• Gives Python users a convenient API for the V2 service, including conversion of common Python and nitypes values.
• Follows the V1 client's model and lifecycle patterns while supporting the V2 resource model: step-owned conditions, status separate from outcome, nested creation, atomic batches, and updates and deletes.
• Handles update masks, validation, value conversion, and the DEC-017 metadata unlink rule inside the wrapper.
• Adds documentation and examples without changing the V1 client or V1 examples.

What testing has been done?

Unit tests: 331 passed, 114 of them for V2. They cover model/protobuf round trips, request and update-mask construction, value conversions (including Scalar reads and units), validation, the DEC-017 unlink check, batch helpers, and client lifecycle.

Static checks: ni-python-styleguide lint, strict mypy, pyright, and black pass for the package and for examples/overview-v2. The Read the Docs build passes.

Validation against a local datastore-service in V2 mode: an independent validation script, written separately from the unit tests, ran 199 checks against a locally started service with a fresh SQLite store. 177 passed. All 22 failures are service features not built yet, rules working as designed, or the service behavior listed under known gaps; none is a wrapper defect.

Revisions tested:

• V2 client/proto: ni-measurements-data-v2-client / -proto 1.0.0.dev0, ni-apis-python commit 28aee05.
• Service: NI.Measurements.Data.V1.DataStoreService built from commit d5a85122 (product version 2027 Q1+d5a8512), started with NIDataStore_ApiMode=v2 and SQLite storage.
• Metadata: ni-measurements-metadata-v1-client 1.0.0 (V2 reuses the V1 Metadata API).
• Environment: Python 3.14.2 on Windows 11 (10.0.26100), grpcio 1.82.1, protobuf 7.36.2, nitypes 1.1.0, ni-protobuf-types 1.3.0, hightime 1.0.0, Poetry 2.5.1.

How it was run:

• The service was started from the build output with NIDataStore_ApiMode=v2, a local HTTP/2 port, and V2DataStoreSettings__* paths pointing to a fresh folder.
• DataStoreClient and MetadataStoreClient were connected with grpc.insecure_channel("[::1]:<port>").
• The script was run with poetry run python validate_v2.py. Checks that must fail locally ran against an unused port, to prove they fail before any RPC.

Scenarios covered:

• Create/read: test results, steps, and conditions. Every supported value type round-trips exactly. Timestamps are within 1 ns.
• Limits: scalar and Vector limits, and read_fields masks.
• Aliases: used as IDs. An unknown alias returns NOT_FOUND.
• Batch and nested create: atomic rollback verified for a bad step, a bad test result, and a bad nested operator.
• Query: $filter on Id, Name (eq/contains/startswith), Outcome, Status, TestResultId, StepId, OperatorId, and Conditions/any(...). $orderby, $top, $skip, and read_fields work. Invalid OData, $select, and unknown properties return INVALID_ARGUMENT.
• Update: every allowed field, one at a time; clearing with None; atomic batch updates. Local ValueError for empty or non-updatable fields.
• Conditions: add, replace, and remove; NOT_FOUND for a missing condition or step.
• Metadata links: add and remove on test results and measurements. Linking an item that isn't on the test result is rejected.
• Delete: single, batch, and per-item force. Without force, FAILED_PRECONDITION is returned and nothing is deleted. A step with child steps can't be deleted. A force list with the wrong length raises ValueError locally.
• Errors: NOT_FOUND for unknown IDs and INVALID_ARGUMENT for malformed IDs. Unsupported values, ints over 32 bits, and non-UTC datetimes fail locally. Use after close() raises RuntimeError. A stopped service returns UNAVAILABLE.
• Scale/concurrency: a 1,000,000-sample waveform round-trips exactly. 8 threads × 25 creates on one client produced 200 unique IDs, all readable.

After the review fixes, all 7 V2 notebooks (run with nbconvert) and the overview-v2 script were re-run successfully against the same service. Additional live checks:

• A 1,000,000-sample waveform reads back with the example channel settings and fails with RESOURCE_EXHAUSTED on a plain channel.
• remove_test_result_metadata raises ValueError and leaves both links in place while a measurement references the item. It removes an unused item normally, and succeeds after the item is unlinked from the measurements first.

Known service gaps (the wrapper already sends and reads these fields):

• New Resource Fields: keywords, duration, uut_id, and comparison_type are not stored yet, and are rejected on update.
• Structured JSON: JSON measurement values, limits, and condition values are rejected.
• Read-only fields are empty: created_at/updated_at/created_by/updated_by, Step.path/path_ids, Measurement.value_type.
• Query paging (DEC-070/DEC-071): next_link will be replaced by next_page_token with page_size/page_token, and the wrapper's query_* methods will be updated when it lands.
• schema_id is rejected with a generic "invalid fields" error (DEC-033 not decided).

[Copilot]

🟡 Changes recommended

The promised metadata unlink safeguard is missing, and inline UUT updates supported by the pinned API are rejected.

4 open issues

• Unlink all measurement references before test-result removal · New
• Add DEC-017 preflight before unlink RPC · New
• Allow inline uut in test-result updates · New
• Add V2 waveform type and dtype conversion coverage · New

What changed in this PR

Adds ni.datastore.data.v2, a high-level Python interface to the V2 Data Store service alongside the existing V1 API.

Changes:

• Introduces resource models, value conversion, and client methods for creation, queries, updates, metadata links, and deletion.
• Adds unit tests, API documentation, and V2 script and notebook examples.
• Adds V2 dependencies and includes the overview example in CI.

File	Description
tests/​unit/​data/​test_v2_types.py	Tests model conversion and API coverage.
tests/​unit/​data/​test_v2_data_store_client.py	Tests requests, validation, and client lifecycle.
src/​ni/​datastore/​data/​v2/​_types/​_test_result.py	Adds the test-result model.
src/​ni/​datastore/​data/​v2/​_types/​_step.py	Adds steps with nested resources.
src/​ni/​datastore/​data/​v2/​_types/​_status.py	Defines lifecycle statuses.
src/​ni/​datastore/​data/​v2/​_types/​_query_results.py	Adds query results with continuation links.
src/​ni/​datastore/​data/​v2/​_types/​_outcome.py	Defines terminal outcomes.
src/​ni/​datastore/​data/​v2/​_types/​_measurement.py	Adds measurements, values, and limits.
src/​ni/​datastore/​data/​v2/​_types/​_json_value.py	Adds structured JSON values and attributes.
src/​ni/​datastore/​data/​v2/​_types/​_error_information.py	Adds error-detail conversion.
src/​ni/​datastore/​data/​v2/​_types/​_condition.py	Adds step-owned conditions.
src/​ni/​datastore/​data/​v2/​_types/​__init__.py	Initializes the model package.
src/​ni/​datastore/​data/​v2/​_grpc_conversion.py	Converts Python values to and from protobuf.
src/​ni/​datastore/​data/​v2/​_data_store_client.py	Implements the high-level V2 client.
src/​ni/​datastore/​data/​v2/​__init__.py	Exports the public V2 API.
pyproject.toml	Adds Git-sourced V2 dependencies.
poetry.lock	Locks V2 dependencies and updates protobuf types.
examples/​overview-v2/​src/​overview.py	Demonstrates publishing and querying waveforms.
examples/​overview-v2/​src/​__init__.py	Initializes the example package.
examples/​overview-v2/​README.md	Documents setup and execution.
examples/​overview-v2/​pyproject.toml	Configures example dependencies and entry point.
examples/​overview-v2/​poetry.toml	Configures an in-project virtual environment.
examples/​notebooks/​v2/​update-delete/​update_delete.ipynb	Demonstrates updates, metadata links, and deletion.
examples/​notebooks/​v2/​structured-extensions/​structured_extensions.ipynb	Demonstrates structured extensions.
examples/​notebooks/​v2/​README.md	Documents notebook setup and contents.
examples/​notebooks/​v2/​query/​query_measurements.ipynb	Demonstrates OData queries.
examples/​notebooks/​v2/​query/​publish_sample_data.ipynb	Creates sample query data.
examples/​notebooks/​v2/​publish/​batch_create.ipynb	Demonstrates atomic and nested creation.
examples/​notebooks/​v2/​overview/​publish_measurement.ipynb	Introduces the V2 publishing workflow.
examples/​notebooks/​v2/​measurement-values/​measurement_values.ipynb	Demonstrates supported values and limits.
docs/​reference/​ni-data-store-v2.md	Documents V2 usage and V1 differences.
docs/​reference/​index.rst	Links the V2 reference page.
.github/​workflows/​check_examples.yml	Adds the V2 overview to example checks.

_{Review effort: Balanced}

[mjohanse-emr]

I still need to review the documentation and workflow changes, but I'm done with the v2 directory and tests.

[mjohanse-emr]

Ok, I think I'm done with the first round. This is a big PR and I expect some back and forth. Feel free to reply with specific questions, even before resetting my vote.

[kaviarasu-ni]

Looks like this has to be updated once after publishing the version of the other repo project packages.

[sudharshana-ni]

Both V2 dependencies point to the ni-apis-python branch for now, because the low-level client (ni/ni-apis-python#404) isn't merged or published yet. Once it's merged and ni-measurements-data-v2-client and ni-measurements-data-v2-proto are published, I'll switch both to the published versions, refresh poetry.lock, and remove the temporary-dependency note from the V2 reference doc. I'll hold this PR until that's done, so main never depends on a branch.

[rbell517]

My comments on the docs can be deferred until the later feature, and additional nice to haves like overloads can also be done in a follow up, but I'd like to see what we can do about read and update masks.

[rbell517]

The service requires the IDs to be UUIDs but we're not typing it that way on the client. Is there any reason for that?

[rbell517]

For metadata IDs we accept aliases on create, so for those we could accept string or UUID.

[sudharshana-ni]

ID inputs now accept either str or uuid.UUID. This covers the model ID fields, the method ID arguments, the update builders, and the create and update keyword fields. UUIDs are converted to their string form before sending. Metadata ID fields keep accepting registered aliases, as suggested. Reads still return str IDs, to stay consistent with the protobuf string fields, the .NET client, and MetadataStoreClient, which also use string IDs. The service remains the source of truth for validating the ID format. If reads should return uuid.UUID, that can be a follow-up together with the metadata client so both stay consistent.

[rbell517]

For the query functions we should make it simple to include the next page field from the QueryResults. And make sure we have an example that shows paging.

Some of this can be deferred to the next round once @kartheeswaran-ni 's work is done, if you want.

Karthee, I don't think we've discussed how to pick the page size when using the gRPC version of the query route if we're using a request header in the HTTP OData approach. Maybe we need a new fields on the query message? Whatever we decide, that should have some exposure in the clients.

[sudharshana-ni]

Deferring this to the next round. Today the gRPC query request has only odata_query and read_mask, and the gRPC service does not populate next_link, so there is nothing for the client to page with yet. Once the service work settles how next links and page size work over gRPC, the client will add simple next-page support and a paging example.

[kartheeswaran-ni]

@rbell517 - we can discuss about the GRPC pagination in this PR.

https://github.com/ni/datastore-service/pull/1011

Currently I'm doing self-review. I'll let you know once it's ready for your review.

[rbell517]

Only for steps and test results? Why not also for conditions and measurements?

[rbell517]

What is this type error you're ignoring?

[rbell517]

I'm surprised to see we'd need to do a cast here