Add high-level Python wrapper for the MDS V2 Data Store API

.github/workflows/check_examples.yml

@@ -15,7 +15,7 @@ jobs:
         os: [windows-latest, ubuntu-latest, macos-latest]
         # Pin to 3.14.8 to workaround AB#4147280
         python-version: [3.14.8]
-        example-name: ['overview', 'system']
+        example-name: ['overview', 'overview-v2', 'system']
     runs-on: ${{ matrix.os }}
     steps:
       - name: Check out repo
@@ -29,6 +29,8 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Set up Poetry
         uses: ni/python-actions/setup-poetry@dee640bba235ae28fdc6b7337c643bd06358ae90 # v0.9.0
+        with:
+          poetry-version: '2.5.1'
       - name: Analyze Python Project
         uses: ni/python-actions/analyze-project@dee640bba235ae28fdc6b7337c643bd06358ae90 # v0.9.0
         with:

.github/workflows/check_ni_datastore.yml

@@ -27,6 +27,8 @@ jobs:
           python-version: ${{ matrix.python-version }}
       - name: Set up Poetry
         uses: ni/python-actions/setup-poetry@dee640bba235ae28fdc6b7337c643bd06358ae90 # v0.9.0
+        with:
+          poetry-version: '2.5.1'
       - name: Analyze Python Project
         uses: ni/python-actions/analyze-project@dee640bba235ae28fdc6b7337c643bd06358ae90 # v0.9.0
       - name: Bandit security checks

.github/workflows/run_unit_tests.yml

@@ -36,5 +36,7 @@ jobs:
           python-version: ${{ inputs.python-version }}
       - name: Set up Poetry
         uses: ni/python-actions/setup-poetry@dee640bba235ae28fdc6b7337c643bd06358ae90 # v0.9.0
+        with:
+          poetry-version: '2.5.1'
       - name: Run and upload unit tests for ni.datastore
         uses: ./.github/actions/run_and_upload_unit_tests

docs/reference/index.rst

@@ -8,5 +8,6 @@ This section provides comprehensive documentation for the NI Data Store system c
    :maxdepth: 3
 
    ni-data-store
+   ni-data-store-v2
    ni-metadata-store
    using-measurement-data-services

docs/reference/ni-data-store-v2.md

@@ -0,0 +1,185 @@
+# NI Data Store V2 Python API
+
+Use the separate V2 client when you need the V2 service model. Like the V1
+client, it uses Python types (`TestResult`, `Step`, `Measurement`, `Condition`,
+`ErrorInformation`, `Outcome`, and `Status`) and constructs the gRPC request
+messages internally.
+
+```python
+from datetime import timezone
+
+from hightime import datetime
+from ni.datastore.data.v2 import (
+    Condition,
+    DataStoreClient,
+    Measurement,
+    Outcome,
+    Status,
+    Step,
+    TestResult,
+)
+
+with DataStoreClient() as client:
+    test_result = client.create_test_result(
+        TestResult("Power test", start_date_time=datetime.now(tz=timezone.utc))
+    )
+    step = client.create_step(
+        Step("Measure voltage", test_result_id=test_result.id, status=Status.RUNNING)
+    )
+    client.add_condition(
+        step.id, Condition("supply-voltage", 5.0, condition_type="Nominal", units="V")
+    )
+    measurement = client.create_measurement(
+        "voltage",
+        5.02,
+        step.id,
+        units="V",
+        outcome=Outcome.PASSED,
+    )
+    voltage = client.read_measurement_value(measurement)
+```
+
+Each create method accepts either the object or its fields as arguments, so
+`client.create_step("Measure voltage", test_result_id=test_result.id)` and
+`client.create_measurement(Measurement(name="voltage", value=5.02, step_id=step.id))`
+also work. `TestResultCreateFields` and `StepCreateFields` type the keyword fields.
+Create methods return the created object as saved by the service, including its
+generated ID. Nested resources, such as the steps of a test result, and expanded
+metadata are not included. The client also provides `get_test_result`, `get_step`,
+`query_test_results`, `query_steps`, `query_measurements`, and update methods for
+test results, steps, and measurements. These return the Python types, with
+timestamps as `hightime.datetime`, durations as `hightime.timedelta`, and
+extensions as dictionaries. Queries accept an OData expression and optional
+`read_fields`. Read fields are selected with `TestResultField`, `StepField`, or
+`MeasurementField`, for example `read_fields=[StepField.CONDITIONS]`; field names
+as strings are also accepted.
+
+Updates change only the fields named in the protobuf update mask, and the client
+builds the mask for you. Pass a `Step`, or its ID, with keyword fields, an update
+builder, or the whole object:
+
+```python
+from ni.datastore.data.v2 import Outcome, StepUpdateBuilder
+
+# Keyword fields: only these fields change. None clears a field.
+data_store_client.update_step(step, notes="Verified", outcome=Outcome.PASSED)
+
+# Update builder: each with_ call selects one field, like the .NET client builders.
+update = StepUpdateBuilder(step).with_notes("Verified").with_outcome(Outcome.PASSED)
+data_store_client.update_step(update)
+
+# Whole object: every updatable field is set from the object, so empty fields are cleared.
+step.notes = "Verified"
+data_store_client.update_step(step)
+```
+
+Keyword fields are typed by `TestResultUpdateFields`, `StepUpdateFields`, and
+`MeasurementUpdateFields`, so type checkers report unknown fields and wrong value
+types. `TestResultUpdateBuilder`, `StepUpdateBuilder`, and
+`MeasurementUpdateBuilder` follow the .NET builders: setting a relationship by ID,
+such as `with_operator_id`, replaces inline metadata set earlier, and inline
+metadata replaces an ID. A whole-object update sends Stage 2 fields (`keywords`,
+`duration`, and `uut_id`/`uut`) only when they have a value, and sends only the ID
+when a relationship has both an ID and inline metadata. Empty updates and fields
+that the service does not allow to change raise `ValueError` before an RPC is made.
+
+IDs can be passed as `str` or `uuid.UUID`, and reads return them as `str`.
+Test result, step, and measurement IDs must be UUIDs. Metadata ID fields, such as
+`operator_id` or `software_item_ids`, also accept registered aliases.
+
+`delete_test_result`, `delete_step`, and `delete_measurement` remove resources.
+They and the batch delete methods accept the objects or their IDs.
+Without `force=True`, deleting a test result with steps, or a step with
+measurements, fails with `FAILED_PRECONDITION`; with `force=True`, the owned
+children are deleted too. A step that has child steps cannot be deleted.
+
+The client wraps every method of the V2 low-level client:
+
+- `remove_condition` removes a condition from a step.
+- `add_test_result_metadata`, `remove_test_result_metadata`,
+  `add_measurement_metadata`, and `remove_measurement_metadata` link or unlink
+  software items, hardware items, and test adapters after creation. A
+  measurement can only link items that are linked to its test result, and each
+  measurement keeps its own selection. Linking an item to a test result does not
+  link it to the test result's measurements. An item cannot be unlinked from a
+  test result while a measurement of that test result still references it;
+  unlink it from those measurements first.
+- `create_measurement` accepts `upper_limit` and `lower_limit` of any supported
+  value type. The service stores the limits as given and does not compare them
+  with the value. `read_measurement_values` returns the value and both limits;
+  select fields with `MeasurementValueField`. `read_measurement_value` and
+  `read_measurement_values` accept a `Measurement` or its ID.
+- `batch_create_*`, `batch_update_*`, and `batch_delete_*` handle many test
+  results, steps, or measurements in one atomic call. Batch updates take a
+  dictionary of fields keyed by ID, or a list of builders and objects. Batch
+  deletes accept one `force` flag or one flag per ID.
+- `TestResult.steps`, `Step.children`, and `Step.measurements` create a whole
+  tree in one call. Metadata objects such as `Operator` or `SoftwareItem` can be
+  passed instead of IDs to find or create them.
+- `create_measurement` accepts `SoftwareItem`, `HardwareItem`, and
+  `TestAdapter` objects for inline find-or-create, in addition to their IDs.
+- `query_*` methods return `QueryResults`, a list with a `next_link` attribute
+  that is set when the service has more results.
+- JSON values and conditions accept a plain dictionary, or `JsonValue` when the
+  value needs attributes. Reads return a `JsonValue` only when attributes are
+  present.
+
+## V1 Convenience Differences
+
+- V1 `publish_condition` and `publish_condition_batch` do not have direct V2
+  equivalents. V2 conditions belong to a step; use `add_condition(step_id, ...)`
+  to add or replace one. Read conditions through the containing step returned
+  by `get_step` or `query_steps`; there are no standalone V2 condition get/query
+  methods in this high-level API.
+- V1 `publish_measurement_batch` is shaped for parametric sweeps. In V2, use
+  `batch_create_measurements` with `Measurement` objects; each item can target a
+  different step, and the whole batch is atomic.
+- V2 has its own `TestResult`, `Step`, `Measurement`, `Condition`, and
+  `ErrorInformation` types in `ni.datastore.data.v2`, because the V2 resource
+  fields differ from V1. They follow the V1 style: create methods take them,
+  and get/query/update methods return them instead of protobuf messages.
+- V2 create and update methods accept `hightime.datetime` timestamps and
+  `hightime.timedelta` durations, converted with the NI precision converters.
+  Timestamps must use `datetime.timezone.utc`; other time zones are rejected.
+  Round trips are exact to about 1 ns; finer precision is not preserved.
+- Test results and steps accept `keywords`, and test results accept `uut_id`
+  updates. Keywords, duration, and `uut_id` are stored once the service delivers
+  the Stage 2 New Resource Fields feature.
+- Measurement values are converted to scalar, vector, supported NI
+  waveform-family, or JSON values; `read_measurement_value` converts them back to
+  Python or nitypes values. The `MeasurementValue` type lists the values that can
+  be written, and `MeasurementReadValue` lists the values that can be read. A
+  single value is always read back as a `Scalar`,
+  with or without units, and a list of values as a `Vector`. When a `Scalar` is
+  given and `units` is not, the measurement's `units` are taken from the
+  `Scalar`. Conditions accept scalar or JSON values. A scalar condition value
+  reads back as a plain Python value, or as a `Scalar` when it has extended
+  properties other than units. JSON values
+  require the service's Stage 3 Structured JSON Storage And Query Support.
+- Software item, hardware item, and test adapter IDs on a measurement must first
+  be linked to its test result; pass them to `create_test_result`.
+- Arguments that take several strings, such as `keywords`, item IDs,
+  `read_fields`, and the IDs passed to batch deletes, raise `TypeError` when
+  given a single string. Pass a list, for example `keywords=["smoke"]`, so a
+  string is not split into its characters.
+- Discovery, channel configuration, and cleanup are delegated to the low-level
+  V2 client. Its gRPC transport exceptions pass through unchanged; invalid
+  wrapper values and update fields fail locally with `TypeError` or
+  `ValueError`. As in V1, calling a method after `close()` raises
+  `RuntimeError`.
+- V1 and V2 hosts both register the metadata store interface. When both run in
+  the same discovery cluster, `MetadataStoreClient()` cannot choose between
+  them; pass a `grpc_channel` for the intended host.
+- `status` and `outcome` are separate, but the service requires status `DONE`
+  (or no status) whenever a test result or step has an outcome other than
+  `UNSPECIFIED`. For example, `status=Status.RUNNING` with
+  `outcome=Outcome.FAILED` is rejected with `INVALID_ARGUMENT`.
+- When you pass your own `grpc_channel`, gRPC limits received messages to 4 MB
+  by default, so reading a large value such as a 1-million-sample waveform fails
+  with `RESOURCE_EXHAUSTED`. Create the channel with
+  `options=[("grpc.max_receive_message_length", -1)]` to remove the limit. The
+  default discovery path already does this.
+- Waveform values read back with the same data, but their timing start time is a
+  `nitypes.bintime.DateTime` rather than a `hightime.datetime`. As a result, `==`
+  between a published waveform and the one read back returns `False`; compare
+  the data, such as `raw_data`, instead.