Repository navigation
fix(mlops): serialize NetworkConfig and keep S3 input fields in check step requests - #6406
Open
AkshayShah03 wants to merge 1 commit into
Open
AkshayShah03 wants to merge 1 commit into
AkshayShah03 wants to merge 1 commit into
Conversation
AkshayShah03
had a problem deploying
to
manual-approval
October 9, 2026 17:51 — with 
GitHub Actions
Failure
GitHub Actions
Failure
AkshayShah03
had a problem deploying
to
manual-approval
October 9, 2026 17:51 — with
GitHub Actions
Failure
GitHub Actions
Failure
AkshayShah03
had a problem deploying
to
manual-approval
October 9, 2026 17:51 — with
GitHub Actions
Failure
GitHub Actions
Failure
… step requests The shared check-step request builder (_build_check_job_request, used by QualityCheckStep and ClarifyCheckStep) had two gaps: - It put the processor's NetworkConfig object straight into the request. CheckJobConfig.network_config is a sagemaker.core.network.NetworkConfig, so any check step with a network config (VPC, network isolation) produced arguments that are not JSON serializable, and pipeline.definition()/create()/ upsert() failed with 'Object of type NetworkConfig is not JSON serializable'. Send its request dict, as the core Processor does. - It serialized only S3Uri, LocalPath, S3DataType and S3InputMode for each input, dropping S3CompressionType and S3DataDistributionType. A DataConfig(s3_compression_type='Gzip') in a ClarifyCheckStep therefore produced a job without S3CompressionType, so the gzipped dataset reached the container still compressed. Keep both fields when they are set.
AkshayShah03
force-pushed
the
fix/check-step-network-config
branch
from
November 9, 2026 03:44
e15fb38 to
37b1784
Compare
AkshayShah03
requested a deployment
to
manual-approval
November 9, 2026 03:44 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
AkshayShah03
requested a deployment
to
manual-approval
November 9, 2026 03:44 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
AkshayShah03
requested a deployment
to
manual-approval
November 9, 2026 03:44 — with
GitHub Actions
Waiting
GitHub Actions
Waiting
AkshayShah03
changed the title
This branch is waiting to be deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue #, if available: N/A (now also covers #6421, which I'm closing in favour of this PR)
Description of changes:
Rebased onto master after #6497, which moved the
QualityCheckStep/ClarifyCheckSteprequest building into the shared_check_job_request._build_check_job_request. Both fixes now go into that helper:CheckJobConfig.network_configis asagemaker.core.network.NetworkConfig, and the helper put that object itself in the request. Any check step with a network config (VPC, network isolation) produces arguments that aren't JSON serializable, sopipeline.definition(),create()andupsert()fail withTypeError: Object of type NetworkConfig is not JSON serializable. The helper now sendsnetwork_config._to_request_dict(), as the coreProcessordoes. A plain dict is passed through unchanged.S3CompressionTypeandS3DataDistributionTypeare dropped from the processing inputs. WithDataConfig(s3_compression_type="Gzip")in aClarifyCheckStep, the job ran withoutS3CompressionType, so the gzipped dataset reached the Clarify container still compressed. Both fields are now kept when set;Noneand unassigned values are skipped.Testing:
test_check_job_request.py:NetworkConfigis serialized, and the request passesjson.dumpsand validates against botocore'sCreateProcessingJobinput shape;sagemaker-mlops/tests/unit: 983 passed (981 on master). The 28feature_processorcollection errors also happen on master.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.