Skip to content

feat(cli): --format csv, stdin streams via '-', and telecommand inspection - #7

Open
mysma-9403 wants to merge 1 commit into
mainfrom
feat/cli-csv-and-stream-enhancements
Open

mysma-9403 wants to merge 1 commit into
mainfrom
feat/cli-csv-and-stream-enhancements

Conversation

@mysma-9403

Copy link
Copy Markdown
Collaborator

Telemetry decoding often feeds analysis pipelines, spreadsheets, and stream processors, where table and JSON lines require extra parsing or do not stream into tabular consumers cleanly. xtce was also strictly file-bound, failing if piped input used the standard - convention.

Changes

  • --format csv: outputs tabular parameter values per packet in standard format:
    packet,container,parameter,eng (and an optional ,raw column with --raw).
    RFC 4180 escaping is applied when names or strings contain commas, double quotes, or newlines.
  • --no-summary: suppresses the trailing timing and packet count lines, leaving stdout clean for piping directly into downstream tools (xsv, jq, Pandas).
  • Standard input (-) support:
    • xtce decode accepts - for either the packet stream or the definition.
    • xtce info, xtce codegen and xtce bench accept - for definition files.
    • Passing - for both definition and packets in decode returns an error cleanly.
  • Telecommand inspection in xtce info --verbose: lists the telecommands section, showing each command, inheritance, backing container, and declared argument types.

Verification

  • 20 unit tests in xtce-cli (including CSV string escaping, special float formatting, and tabular output).
  • All 9 differential test cases match the Python reference implementation (cargo xtask diff).
  • cargo fmt --check and cargo clippy --workspace --all-targets -- -D warnings clean.

…ction

Telemetry decoding often feeds analysis pipelines, spreadsheets, and stream
processors, where plain text table and JSON lines require extra parsing or
do not stream into CSV consumers cleanly. xtce was also strictly file-bound,
failing if piped input used the standard '-' convention.

* `--format csv` outputs tabular parameter values per packet:
  `packet,container,parameter,eng` (and an optional `,raw` column with `--raw`).
  RFC 4180 escaping is applied when strings contain commas, double quotes,
  or newlines.
* `--no-summary` suppresses the trailing timing and packet count lines, leaving
  stdout clean for piping directly into downstream tools (`xsv`, `jq`, Pandas).
* Reading from standard input (`-`) is now supported across the CLI:
  `xtce decode` accepts `-` for either the packet stream or the definition,
  and `xtce info`, `xtce codegen` and `xtce bench` accept `-` for definition files.
* `xtce info --verbose` now lists the telecommands section, showing each
  command, inheritance, backing container, and declared argument types.

Tests added for CSV string escaping, float formatting, and tabular output.
All 9 differential cases match the reference, fmt and clippy clean.

@altanix-bot altanix-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🪶 Altanix Code Review & Security Gate

This PR introduces support for reading XTCE definitions and packet streams from standard input (stdin) across all CLI commands. It also adds a new CSV output format for packet decoding and improves the CLI reporting capabilities by including telecommands in the info command.

✨ Co wprowadza (Key Additions):

  • Added CSV output format for packet decoding with support for raw values.
  • Implemented stdin support for 'info', 'decode', 'codegen', and 'bench' commands.
  • Added telecommand reporting to the 'info' command output.
  • Added a summary toggle ('--no-summary') for the decode command.

🔄 Co zmienia / poprawia (Modifications & Fixes):

  • Refactored file loading logic into a centralized 'load_db' helper function.
  • Updated CLI argument descriptions to reflect stdin support.
  • Improved error handling and output formatting for the decode command, including redirecting non-table output to stderr.
  • Refactored 'run' function in main.rs to use dedicated runner functions for better maintainability.

Risk Score: 9/10 | Security Passed: ✅ YES

📁 Zmiany w plikach:

Zmieniony Plik Opis Zmiany
crates/xtce-cli/src/decode.rs Implements the new CSV serialization logic, summary suppression, and stdin reading for packet streams.
crates/xtce-cli/src/main.rs Contains the core logic for command routing, stdin handling, and the new centralized database loading mechanism.
crates/xtce-cli/src/info.rs Adds telecommand reporting and updates source display logic for stdin.

🧪 Wygenerowane Testy Jednostkowe & Regresyjne (AI Tester)

EDGE_CASE: DiffParser::parse (tests/test_diff_parser.rs)

Verifies that the parser does not panic on empty git diff input.

#[test]
fn test_empty_diff_handling() {
    let files = DiffParser::parse("");
    assert!(files.is_empty());
}

🛡️ Zweryfikowano autonomicznie przez Altanix — Architektoniczny Code Review & Live AppSec Gatekeeper.
✨ Chcesz chronić swoje repozytoria przed lukami CVE i wyścigami danych? Wypróbuj Altanix za darmo (14 dni) • 🚀 Odblokuj Plan Pro ($20/dev)

Comment on lines +36 to +44
fn load_stream(packets: &Path) -> std::io::Result<Vec<u8>> {
if packets.as_os_str() == "-" {
let mut bytes = Vec::new();
std::io::stdin().read_to_end(&mut bytes)?;
Ok(bytes)
} else {
std::fs::read(packets)
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading the entire packet stream into memory via read_to_end is a DATA_SCALE_RISK. If the input file or stdin stream is large, this will cause an OOM error. Consider processing the stream in chunks or using a buffered reader.

Suggested change
fn load_stream(packets: &Path) -> std::io::Result<Vec<u8>> {
if packets.as_os_str() == "-" {
let mut bytes = Vec::new();
std::io::stdin().read_to_end(&mut bytes)?;
Ok(bytes)
} else {
std::fs::read(packets)
}
}
fn load_stream(packets: &Path) -> std::io::Result<Box<dyn std::io::Read>> {
if packets.as_os_str() == "-" {
Ok(Box::new(std::io::stdin()))
} else {
Ok(Box::new(std::fs::File::open(packets)?))
}
}

@altanix-bot altanix-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🪶 Altanix Code Review & Security Gate

This PR introduces support for reading XTCE definitions and packet streams from standard input (stdin) across all CLI commands. Additionally, it adds a new CSV output format for the 'decode' command and improves the 'info' command by displaying telecommands.

✨ Key Changes & Capabilities:

  • Added CSV output format to the 'decode' command with support for raw values.
  • Added telecommand reporting to the 'info' command.
  • Implemented stdin support for 'info', 'decode', 'codegen', and 'bench' commands.

🔄 Modifications & Fixes:

  • Refactored CLI command execution into dedicated functions (run_info, run_codegen, run_bench) for better maintainability.
  • Updated 'decode' command to allow suppressing the summary output.
  • Improved error handling and input stream management for file-based operations.
  • Added unit tests for CSV formatting, including string escaping and float handling.

Risk Score: 9/10 | Security: ✅ PASSED

📁 File Changes:

Modified File Description
crates/xtce-cli/src/decode.rs Implements the new CSV output logic, including custom string escaping and float formatting, which are critical for data integrity.
crates/xtce-cli/src/main.rs Contains the main CLI entry point changes and the refactoring of command execution logic to support stdin.
crates/xtce-cli/src/info.rs Adds the new telecommand reporting feature to the info command.

🧪 Generated Unit & Regression Tests (AI Tester)

EDGE_CASE: write_csv_string (crates/xtce-cli/src/decode.rs)

Verifies that CSV string escaping correctly handles commas and double quotes to prevent malformed CSV output.

#[test]
fn test_write_csv_string_escaping() {
    let mut out = Vec::new();
    // Test escaping of quotes and commas
    super::write_csv_string(&mut out, "data, \"quoted\"").unwrap();
    assert_eq!(String::from_utf8(out).unwrap(), "\"data, \"\"quoted\"\"\"");

    let mut out = Vec::new();
    // Test no escaping needed
    super::write_csv_string(&mut out, "simple").unwrap();
    assert_eq!(String::from_utf8(out).unwrap(), "simple");
}
ERROR_HANDLING: run (crates/xtce-cli/src/decode.rs)

Ensures that the CLI correctly rejects configurations where both definition and packets are requested from stdin simultaneously.

#[test]
fn test_run_stdin_conflict_error() {
    let options = super::Options {
        definition: std::path::Path::new("-"),
        packets: std::path::Path::new("-"),
        root: None,
        skip_header_bytes: 0,
        limit: None,
        only: &[],
        format: super::Format::Csv,
        show_raw: false,
        no_summary: false,
    };
    let result = super::run(&options);
    assert!(result.is_err());
    assert_eq!(result.unwrap_err().to_string(), "cannot read both definition and packets from standard input");
}
EDGE_CASE: write_csv_float (crates/xtce-cli/src/decode.rs)

Validates that floating point special values (NaN, Infinity) are serialized into the expected CSV format.

#[test]
fn test_write_csv_float_special_values() {
    let mut out = Vec::new();
    super::write_csv_float(&mut out, f64::NEG_INFINITY).unwrap();
    assert_eq!(String::from_utf8(out).unwrap(), "-Infinity");

    let mut out = Vec::new();
    super::write_csv_float(&mut out, f64::NAN).unwrap();
    assert_eq!(String::from_utf8(out).unwrap(), "NaN");
}

🛡️ Autonomously verified by Altanix — Architectural Code Review & Live AppSec Gatekeeper.
✨ Want to protect your repositories from CVEs and data races? Try Altanix for free (14-day trial) • 🚀 Upgrade to Pro ($20/dev)

Comment on lines +36 to +44
fn load_stream(packets: &Path) -> std::io::Result<Vec<u8>> {
if packets.as_os_str() == "-" {
let mut bytes = Vec::new();
std::io::stdin().read_to_end(&mut bytes)?;
Ok(bytes)
} else {
std::fs::read(packets)
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading the entire packet stream into memory via read_to_end is a risk for large files. Consider using a buffered reader and processing packets in a streaming fashion to avoid OOM errors.

Suggested change
fn load_stream(packets: &Path) -> std::io::Result<Vec<u8>> {
if packets.as_os_str() == "-" {
let mut bytes = Vec::new();
std::io::stdin().read_to_end(&mut bytes)?;
Ok(bytes)
} else {
std::fs::read(packets)
}
}
fn load_stream(packets: &Path) -> std::io::Result<Box<dyn std::io::Read>> {
if packets.as_os_str() == "-" {
Ok(Box::new(std::io::stdin()))
} else {
Ok(Box::new(std::fs::File::open(packets)?))
}
}

@altanix-bot altanix-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🪶 Altanix Code Review & Security Gate

This PR enhances the xtce-cli tool by adding support for reading XTCE definitions and packet streams from standard input ('-'). It also introduces a CSV output format for decoded telemetry and adds support for displaying telecommand metadata in the 'info' command.

✨ Key Changes & Capabilities:

  • Added CSV output format to 'xtce decode' with support for raw/engineering values.
  • Added support for reading XTCE definitions and binary packet streams from stdin.
  • Implemented telecommand metadata reporting in 'xtce info'.
  • Added a '--no-summary' flag to suppress timing and packet count output in 'xtce decode'.

🔄 Modifications & Fixes:

  • Refactored CLI command execution into separate functions (run_info, run_codegen, run_bench) for better maintainability.
  • Updated 'xtce decode' to handle errors and non-table output formats by printing to stderr instead of stdout.
  • Improved 'xtce info' to display telecommand structures including arguments and bit-widths.
  • Standardized database loading logic to support both file paths and stdin.

Risk Score: 7/10 | Security: ✅ PASSED

📁 File Changes:

Modified File Description
crates/xtce-cli/src/decode.rs Implements the new CSV output logic, stdin reading, and modifies the core decoding loop and summary reporting.
crates/xtce-cli/src/main.rs Updates CLI argument parsing and refactors command execution logic to support stdin and new flags.
crates/xtce-cli/src/info.rs Adds telecommand reporting functionality to the info command.

🧪 Generated Unit & Regression Tests (AI Tester)

HAPPY_PATH: write_csv (crates/xtce-cli/src/decode.rs)

Verifies that the CSV formatting logic correctly concatenates fields and respects the expected CSV structure.

#[test]
fn test_write_csv_output_format() {
    use super::{write_csv, Options, Format};
    // Mocking minimal structure for DecodedPacket is complex, 
    // so we verify the logic of CSV row generation via the helper functions.
    let mut out = Vec::new();
    let container_name = "TelemetryContainer";
    let param_name = "Voltage";
    let eng_value = "12.5";
    
    // Simulate row: index,container,parameter,eng
    write!(out, "0,").unwrap();
    super::write_csv_string(&mut out, container_name).unwrap();
    write!(out, ",").unwrap();
    super::write_csv_string(&mut out, param_name).unwrap();
    write!(out, ",").unwrap();
    write!(out, "{}", eng_value).unwrap();
    
    let result = String::from_utf8(out).unwrap();
    assert_eq!(result, "0,TelemetryContainer,Voltage,12.5");
}
EDGE_CASE: write_csv_string (crates/xtce-cli/src/decode.rs)

Ensures that CSV string escaping correctly handles commas, double quotes, and newlines to prevent malformed CSV output.

#[test]
fn test_write_csv_string_escaping_edge_cases() {
    let mut out = Vec::new();
    // Test input containing all special characters that trigger quoting
    super::write_csv_string(&mut out, "data,with\"quotes\"\nnewline").unwrap();
    let result = String::from_utf8(out).unwrap();
    // Expected: "data,with""quotes""
newline"
    assert!(result.starts_with('"'));
    assert!(result.ends_with('"'));
    assert!(result.contains(""""));
}
ERROR_HANDLING: run (crates/xtce-cli/src/decode.rs)

Verifies that the CLI correctly identifies and rejects the invalid configuration where both definition and packets are sourced from stdin.

#[test]
fn test_run_stdin_conflict_error() {
    use super::{run, Options, Format};
    use std::path::Path;
    
    let options = Options {
        definition: Path::new("-"),
        packets: Path::new("-"),
        only: &[],
        format: Format::Table,
        show_raw: false,
        no_summary: false,
    };
    
    let result = run(&options);
    assert!(result.is_err());
    assert_eq!(result.unwrap_err().to_string(), "cannot read both definition and packets from standard input");
}

🛡️ Autonomously verified by Altanix — Architectural Code Review & Live AppSec Gatekeeper.
✨ Want to protect your repositories from CVEs and data races? Try Altanix for free (14-day trial) • 🚀 Upgrade to Pro ($20/dev)

Comment on lines +100 to +102
if options.definition.as_os_str() == "-" && options.packets.as_os_str() == "-" {
return Err("cannot read both definition and packets from standard input".into());
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While this logic works for standard CLI usage, consider that if a file is actually named - on the filesystem, this will incorrectly trigger the error. Given the context of xtce-cli, this is likely acceptable, but ensure that Path::new("-") is the intended way to handle stdin across all platforms.

}

let decode_start = Instant::now();
for (index, framed) in PacketIter::new(&stream, options.skip_header_bytes).enumerate() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current implementation loads the entire packet stream into memory (Vec<u8>). While this is efficient for small-to-medium files, it poses a DATA_SCALE_RISK for massive telemetry dumps. Since the current architecture relies on &[u8] slices for zero-copy parsing, this is acceptable for now, but be aware that processing multi-gigabyte files will require a memory-mapped file (memmap2) approach rather than std::fs::read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant