diff --git a/benchmarks/orf_finder/.gitignore b/benchmarks/orf_finder/.gitignore new file mode 100644 index 0000000..0b016ad --- /dev/null +++ b/benchmarks/orf_finder/.gitignore @@ -0,0 +1,5 @@ +__pycache__/ +*.py[cod] + +data/test_inputs/ +results/ diff --git a/benchmarks/orf_finder/README.md b/benchmarks/orf_finder/README.md new file mode 100644 index 0000000..a126cdb --- /dev/null +++ b/benchmarks/orf_finder/README.md @@ -0,0 +1,106 @@ +# ORFfinder benchmark + +This benchmark compares three ORF-finding paths without changing the production adapter or pipeline: + +1. **Raw Python package** — `orffinder==1.8` `getORFProteins(..., return_loci=True)` at its direct minimum-length threshold. Native loci are retained, and protein strings are preserved verbatim, including terminal `*` symbols. +2. **Corrected adapter** — benchmark-side reconstruction of `OrffinderOrfSearcher` locus discovery, length filtering, translation, and ordering. Proteins are checked against the production adapter as an ordered list; candidate coordinates come from locus dictionaries. +3. **NCBI standalone** — ORFfinder 0.4.3 CDS FASTA (outfmt 1) and protein FASTA (outfmt 0), joined by ORF identifiers and cross-checked against each other and the original input interval. + +## Test inputs and scope + +The only benchmark dataset is three test files: + +| File | RefSeqGene record | +|---|---| +| `HoxA13.fa` | `NG_008181.2` | +| `Jun.fa` | `NG_047027.2` | +| `MITF.fa` | `NG_011631.1` | + +The FASTA files themselves are **not versioned** (the repository holds code, not data). Place a local copy of the three files in a directory and pass it with `--input-dir` or the `LVERAGE_ORF_BENCHMARK_INPUTS` environment variable; `data/test_inputs/` is the git-ignored default location. Each file must contain exactly one versioned RefSeqGene FASTA record with a non-empty DNA sequence. The runner checks the accession in the original header, record identifier, DNA alphabet, uniqueness, and exact filename-to-accession assignment. `data/test_input_manifest.tsv` records each original header, accession version, sequence length, SHA-256 of the untouched input file, and SHA-256 of the upper-case sequence; whenever that manifest is present the runner verifies the files against it, so a run on different bytes fails loudly. A run repeats this provenance in `results/metadata.json` and records a checksum for the combined FASTA passed to NCBI. Inputs are consumed locally; no replacement sequences or GenBank annotations are fetched or required. + +This runner is deliberately **not a general ORF benchmark**: it accepts only these three filenames and accessions and refuses anything else. That is so the recorded numbers can only ever be reproduced against the same bytes. Generalizing it to arbitrary inputs would mean dropping the manifest and accession checks, which is a different tool. + +These are **genomic RefSeqGene sequences**, not spliced mRNA sequences. ORF scanning does not perform exon joining or splicing and cannot establish the mature transcript, the annotated protein, or whether an ORF is the named gene's protein. Results describe the scan of the genomic strings under the recorded settings only. + +`data/controls.fasta` contains synthetic **validation fixtures** for strand coordinates, stop handling, nested filtering, and terminal incomplete ORFs. The controls are run and reported separately; they are never included in test-input candidate counts or comparisons. + +## Settings and comparison semantics + +The preserved baseline uses ATG starts, standard genetic code (NCBI `-g 1`), no nested filtering, and both strands (`-strand both`). Its raw Python package and NCBI settings request 75 nt; the adapter requests loci at 74 nt, then enforces an actual 75-nt span because the package reports incomplete terminal loci one nucleotide short. The raw-package path is not threshold-corrected. NCBI has a 30-nt minimum floor. + +Equal numeric thresholds do not imply equivalent length filtering across tools. In this baseline, NCBI ORFfinder 0.4.3 at `-ml 75` returned 1,784 candidates, while the adapter's 75-nt filter returned 1,913; 129 adapter candidates had no exact NCBI interval-and-strand match. All 129 were complete 75-nt ORFs whose intervals included a terminal stop codon. A separate validation reran NCBI ORFfinder with `-ml 72` and all other baseline options unchanged. Against the adapter at 75 nt, this produced 1,913 exact interval-and-strand matches across these inputs, preserving candidate multiplicities; all paired protein strings were identical, and the longest intervals matched for all three inputs. This follow-up does not change the benchmark runner's baseline settings or default. + +Candidate coordinates use zero-based, half-open bounds on the original input with strand stored separately. NCBI CDS output includes the terminal stop in the interval while protein output excludes it; the parser checks the outputs pairwise, verifies translation, and checks the CDS sequence against the corresponding input interval. The adapter retains stop codons in genomic intervals but excludes them from protein strings. The adapter's incomplete-terminal translation uses complete codons only. + +Exact interval-and-strand matches are paired one-to-one by stable candidate ID. `overlaps.tsv` retains every positive same-strand overlap, including exact matches, and records IoU. The additional one-to-one summary reserves exact matches, then greedily pairs non-exact edges with IoU ≥ 0.5 in deterministic order; it is descriptive, not an optimal assignment. Longest ORFs are ranked by normalized nucleotide span and all tied locations and proteins are retained. + +## Findings + +For these three genomic RefSeqGene test inputs, the preserved `-ml 75` baseline comparison returned 1,913 corrected-adapter candidates and 1,784 NCBI candidates, with 1,784 exact interval-and-strand matches: + +| Input | Adapter candidates | NCBI candidates | Exact matches | +|---|---:|---:|---:| +| HoxA13 (`NG_008181.2`) | 90 | 84 | 84 | +| Jun (`NG_047027.2`) | 62 | 55 | 55 | +| MITF (`NG_011631.1`) | 1,761 | 1,645 | 1,645 | +| **Total** | **1,913** | **1,784** | **1,784** | + +Every baseline NCBI candidate had an exact adapter match; the adapter returned 129 additional candidates, all complete 75-nt ORFs with an interval-terminal stop codon. Separately, the NCBI ORFfinder 0.4.3 `-ml 72` validation matched all 1,913 adapter interval-and-strand occurrences, with identical protein strings and matching longest intervals for each input. The Counter-based comparison preserved multiplicities. Raw-package/adapter protein-string differences include terminal stop-symbol removal; their counts are not counts of incorrect proteins. These conclusions apply only to these three genomic sequences and the stated settings. ORF scanning does not perform splicing or establish the annotated protein for any named gene. + +The separate lower-threshold check can be reproduced by running NCBI ORFfinder 0.4.3 on the same combined FASTA used in the baseline run, changing only `-ml` from 75 to 72, and saving both output formats: + +```bash +ORFfinder -in /path/to/combined_input.fasta -s 0 -g 1 -ml 72 -n false -strand both -outfmt 0 -out /path/to/results/ncbi_min72_proteins.fasta +ORFfinder -in /path/to/combined_input.fasta -s 0 -g 1 -ml 72 -n false -strand both -outfmt 1 -out /path/to/results/ncbi_min72_cds.fasta +``` + +These commands describe a follow-up validation only; they do not change the benchmark runner's default `-ml 75` comparison. + +## Results and outputs + +Generated outputs are written under `results/` (override with `--results-dir`). That directory is git-ignored; the summary of the recorded run is the Findings section above. The runner writes: + +- `results/candidates.tsv`: candidate-level coordinates and proteins from the three tools. +- `results/overlaps.tsv`: all positive same-strand overlap pairs and IoUs. +- `results/matches.tsv`: exact and deterministic IoU-threshold pairs. +- `results/unmatched.tsv`: candidates left unmatched by those summaries, one row per pairwise comparison (`comparison` and `unmatched_against` columns say which), so a candidate may legitimately appear up to twice. +- `results/translation_differences.tsv`: raw-package/adapter protein comparisons only for exact same-region pairs. +- `results/comparisons.tsv`: per-input counts, tied longest calls, overlaps, matches, and protein multiplicities. +- `results/metadata.json`: tool versions, settings, commands, input provenance/checksums, and separate synthetic-control validation metadata. +- `results/raw/`: NCBI outputs, combined input FASTA, and raw Python package loci for the test inputs and controls. +- `results/report.md`: generated summary of completed test-input results and their genomic-sequence limitations. + +## Setup and execution + +Nothing in this directory refers to a particular user's environment, clone, or tool install. You need: + +1. A Python 3.11+ environment with this checkout installed, so that the adapter under test is the one that gets imported. From the repository root: `pip install -e .` (installs `orffinder==1.8` and `biopython`). If the environment already has another Lverage clone installed in editable mode, either reinstall from this checkout or prefix commands with `PYTHONPATH=src` as below. +2. The NCBI standalone ORFfinder binary (0.4.3 was used for the recorded run). It is not installed by Conda or pip; download it from NCBI. Tell the benchmark where it is with `--executable`, the `ORFFINDER_EXECUTABLE` environment variable, or by putting an `ORFfinder` command on `PATH`. +3. A local directory containing `HoxA13.fa`, `Jun.fa`, and `MITF.fa` (see above). + +Run the focused tests from the repository root: + +```bash +PYTHONPATH=src python -m unittest discover -s benchmarks/orf_finder/tests -v +bash -n benchmarks/orf_finder/run.qsub +``` + +Run the benchmark directly: + +```bash +PYTHONPATH=src python -m benchmarks.orf_finder.benchmark run \ + --executable /path/to/ORFfinder \ + --input-dir /path/to/combined_fastas +``` + +Or on a cluster that uses Grid Engine (SGE/UGE), submit `run.qsub` **from the repository root** of the checkout you want to benchmark. The job script reads its configuration from environment variables (see the header of `run.qsub`); add whatever scheduler options your site requires, such as `-P `: + +```bash +qsub \ + -v ORFFINDER_EXECUTABLE=/path/to/ORFfinder \ + -v ORF_BENCHMARK_INPUTS=/path/to/test_fastas \ + -v CONDA_ENV=my-lverage-env \ + benchmarks/orf_finder/run.qsub +``` + +The job verifies it was submitted from a repository root, verifies the executable, and runs only the three test inputs plus separate synthetic validation fixtures. On a workstation, just run the `python -m` command above instead. diff --git a/benchmarks/orf_finder/benchmark.py b/benchmarks/orf_finder/benchmark.py new file mode 100644 index 0000000..e1db23e --- /dev/null +++ b/benchmarks/orf_finder/benchmark.py @@ -0,0 +1,798 @@ +"""Reproducible, benchmark-only comparison of Python and NCBI ORFfinders.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import importlib.metadata +import itertools +import json +import os +import re +import shutil +import subprocess +import sys +from collections import Counter, defaultdict +from dataclasses import asdict, dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Iterable + +from Bio import SeqIO, __version__ as biopython_version +from Bio.Seq import Seq +from Bio.SeqRecord import SeqRecord +from orffinder import orffinder + +from lverage.orf_searcher import OrffinderOrfSearcher + +ROOT = Path(__file__).resolve().parents[2] +HERE = Path(__file__).resolve().parent +DATA = HERE / "data" +RESULTS = HERE / "results" +# The NCBI standalone binary is not installed by Conda or pip. It is located +# from, in order: --executable, the ORFFINDER_EXECUTABLE environment variable, +# then an ``ORFfinder`` command on PATH. No machine-specific default is kept +# in the repository. +EXECUTABLE_ENV_VAR = "ORFFINDER_EXECUTABLE" +EXECUTABLE_NAME = "ORFfinder" +MINIMUM_NT = 75 +GENETIC_CODE = 1 +# Test FASTA inputs are not versioned; point --input-dir (or +# LVERAGE_ORF_BENCHMARK_INPUTS) at a local copy of the three files. +TEST_INPUT_DIR = DATA / "test_inputs" +INPUT_DIR_ENV_VAR = "LVERAGE_ORF_BENCHMARK_INPUTS" +TEST_INPUT_MANIFEST = DATA / "test_input_manifest.tsv" +TEST_INPUT_FILES = ("HoxA13.fa", "Jun.fa", "MITF.fa") +TEST_INPUT_ACCESSIONS = { + "HoxA13.fa": "NG_008181.2", + "Jun.fa": "NG_047027.2", + "MITF.fa": "NG_011631.1", +} +DNA_ALPHABET = frozenset("ACGTRYSWKMBDHVN") +ACCESSION_VERSION = re.compile(r"\b(NG_\d+\.\d+)\b") + + +@dataclass(frozen=True) +class Candidate: + candidate_id: str + tool: str + record_id: str + start0: int + end0: int + strand: str + nucleotide_length: int + protein: str + terminal_stop_in_nt: bool + complete: bool + + @property + def interval(self) -> tuple[int, int, str]: + return self.start0, self.end0, self.strand + + +def sha256_text(sequence: str) -> str: + return hashlib.sha256(sequence.encode("ascii")).hexdigest() + + +def _locus_interval(locus: dict, sequence_length: int) -> tuple[int, int, str, int]: + """Convert orffinder's native locus dictionary to original-input bounds.""" + start_index = ( + locus["start"] - 1 + if locus["sense"] == "+" + else sequence_length - locus["start"] + 1 + ) + nucleotide_length = sequence_length - start_index if locus["trailing"] else locus["length"] + if nucleotide_length <= 0 or start_index < 0 or start_index + nucleotide_length > sequence_length: + raise ValueError(f"Invalid orffinder locus for input length {sequence_length}: {locus}") + if locus["sense"] == "+": + start0, end0 = start_index, start_index + nucleotide_length + else: + start0, end0 = sequence_length - start_index - nucleotide_length, sequence_length - start_index + return start0, end0, locus["sense"], nucleotide_length + + +def interval_iou(left: tuple[int, int], right: tuple[int, int]) -> float: + intersection = max(0, min(left[1], right[1]) - max(left[0], right[0])) + union = max(left[1], right[1]) - min(left[0], right[0]) + return intersection / union if union else 0.0 + + +def python_raw_candidates(record_id: str, sequence: str, + raw_outputs: dict[str, list[dict]] | None = None) -> list[Candidate]: + """Keep package-native coordinates and protein translations (including '*'). + + When ``raw_outputs`` is given, the package's native locus dictionaries for + ``record_id`` are stored in it so the caller can persist them verbatim. + """ + # This is the direct package baseline at the requested threshold. Its + # incomplete-terminal length behavior is intentionally not adapter-corrected. + loci = orffinder.getORFProteins( + SeqRecord(Seq(sequence)), + minimum_length=MINIMUM_NT, + start_codons=["ATG"], + remove_nested=False, + trim_trailing=False, + return_loci=True, + ) + if raw_outputs is not None: + raw_outputs[record_id] = [ + {key: str(value) if key == "protein" else value for key, value in locus.items()} + for locus in loci + ] + candidates = [] + for locus in loci: + start0, end0, strand, nucleotide_length = _locus_interval(locus, len(sequence)) + protein = str(locus["protein"]) + candidates.append(Candidate( + f"{record_id}:raw:{locus['index']}", "python_raw", record_id, + start0, end0, strand, nucleotide_length, protein, + bool(not locus["trailing"]), not locus["trailing"], + )) + return candidates + + +def corrected_adapter_candidates(record_id: str, sequence: str) -> list[Candidate]: + """Mirror adapter discovery/translation and verify against production output.""" + sequence = sequence.upper() + minimum_for_finder = max(1, MINIMUM_NT - 1) + loci = orffinder.getORFs( + SeqRecord(Seq(sequence)), + minimum_length=minimum_for_finder, + start_codons=["ATG"], + remove_nested=False, + trim_trailing=False, + ) + forward = sequence + reverse = str(Seq(sequence).reverse_complement()) + candidates = [] + for locus in loci: + start0, end0, strand, nucleotide_length = _locus_interval(locus, len(sequence)) + if nucleotide_length < MINIMUM_NT: + continue + strand_sequence = forward if strand == "+" else reverse + start_index = locus["start"] - 1 if strand == "+" else len(sequence) - locus["start"] + 1 + coding_length = ( + nucleotide_length - nucleotide_length % 3 + if locus["trailing"] + else nucleotide_length - 3 + ) + if coding_length <= 0: + continue + protein = str(Seq(strand_sequence[start_index:start_index + coding_length]).translate()).rstrip("*") + if not protein: + continue + candidates.append(Candidate( + f"{record_id}:adapter:{locus['index']}", "adapter_corrected", record_id, + start0, end0, strand, nucleotide_length, protein, + bool(not locus["trailing"]), not locus["trailing"], + )) + + actual = OrffinderOrfSearcher( + minimum_length=MINIMUM_NT, + start_codons=("ATG",), + remove_nested=False, + trim_trailing=False, + ).get_orfs(sequence) + reconstructed = [candidate.protein for candidate in candidates] + if reconstructed != actual: + raise AssertionError( + f"Benchmark adapter reconstruction differs for {record_id}: " + f"reconstructed={Counter(reconstructed)}, production={Counter(actual)}" + ) + return candidates + + +def _ncbi_record_id(orf_tag: str, known_ids: Iterable[str]) -> str: + matches = [record_id for record_id in known_ids if orf_tag.endswith("_" + record_id)] + if len(matches) != 1: + raise ValueError(f"Cannot uniquely associate NCBI ORF {orf_tag!r} with input records") + return matches[0] + + +_NCBI_CDS_HEADER = re.compile(r"^lcl\|([^:]+):(c?)(\d+)-(\d+)$") +_NCBI_ORF_TAG = re.compile(r"(ORF\d+_[^:\s]+):(\d+):(\d+)") + + +def parse_ncbi_outputs(cds_fasta: Path, protein_fasta: Path, records: dict[str, SeqRecord]) -> list[Candidate]: + """Parse NCBI outfmt 1 CDS FASTA and outfmt 0 protein FASTA as a checked pair.""" + known_ids = sorted(records, key=len, reverse=True) + cds_by_tag = {} + with cds_fasta.open(encoding="utf-8") as handle: + for cds_record in SeqIO.parse(handle, "fasta"): + match = _NCBI_CDS_HEADER.match(cds_record.id) + tag_match = _NCBI_ORF_TAG.search(cds_record.description) + if not match or not tag_match: + raise ValueError(f"Unrecognized NCBI CDS FASTA header: {cds_record.description}") + input_id, reverse_mark, first, second = match.groups() + record_id = input_id if input_id in records else None + if record_id is None: + raise ValueError(f"NCBI CDS header refers to unknown input record {input_id!r}") + a, b = int(first), int(second) + if reverse_mark: + start0, end0, strand = b - 1, a, "-" + else: + start0, end0, strand = a - 1, b, "+" + tag = tag_match.group(1) + if _ncbi_record_id(tag, known_ids) != record_id: + raise ValueError(f"NCBI ORF tag {tag} and CDS header disagree on record") + if tag in cds_by_tag: + raise ValueError(f"Duplicate NCBI ORF tag in CDS FASTA: {tag}") + cds_by_tag[tag] = (record_id, start0, end0, strand, str(cds_record.seq).upper()) + + protein_by_tag = {} + with protein_fasta.open(encoding="utf-8") as handle: + for protein_record in SeqIO.parse(handle, "fasta"): + match = _NCBI_ORF_TAG.search(protein_record.id) + if not match: + raise ValueError(f"Unrecognized NCBI protein FASTA header: {protein_record.description}") + tag = match.group(1) + coords = tuple(map(int, match.groups()[1:])) + if tag in protein_by_tag: + raise ValueError(f"Duplicate NCBI ORF tag in protein FASTA: {tag}") + protein_by_tag[tag] = (str(protein_record.seq), coords) + + if set(cds_by_tag) != set(protein_by_tag): + raise ValueError( + f"NCBI CDS/protein ORF identifiers differ; CDS-only={sorted(set(cds_by_tag)-set(protein_by_tag))}, " + f"protein-only={sorted(set(protein_by_tag)-set(cds_by_tag))}" + ) + + candidates = [] + for tag in sorted(cds_by_tag): + record_id, start0, end0, strand, cds_sequence = cds_by_tag[tag] + protein, encoded_coords = protein_by_tag[tag] + # NCBI protein headers encode zero-based inclusive endpoints in either + # orientation; the CDS FASTA's 1-based inclusive range is authoritative. + if strand == "+": + expected_coords = (start0, end0 - 1) + else: + expected_coords = (end0 - 1, start0) + if encoded_coords != expected_coords: + raise ValueError(f"NCBI protein/CDS coordinate encodings disagree for {tag}") + translated_cds = str(Seq(cds_sequence).translate(table=GENETIC_CODE)).rstrip("*") + if protein != translated_cds: + raise ValueError(f"NCBI protein does not match translated CDS for {tag}") + input_sequence = str(records[record_id].seq).upper() + if strand == "+": + expected_cds = input_sequence[start0:end0] + else: + expected_cds = str(Seq(input_sequence[start0:end0]).reverse_complement()) + if expected_cds != cds_sequence: + raise ValueError(f"NCBI CDS sequence does not match saved input interval for {tag}") + candidates.append(Candidate( + f"{record_id}:ncbi:{tag}", "ncbi", record_id, + start0, end0, strand, end0 - start0, protein, + bool(cds_sequence[-3:] in {"TAA", "TAG", "TGA"}), + bool(cds_sequence[-3:] in {"TAA", "TAG", "TGA"}), + )) + return candidates + + +def exact_pairs(left: list[Candidate], right: list[Candidate]) -> list[tuple[Candidate, Candidate, float]]: + """Pair exact interval/strand matches one-to-one by sorted stable IDs.""" + right_by_interval: dict[tuple[int, int, str], list[Candidate]] = defaultdict(list) + for candidate in right: + right_by_interval[candidate.interval].append(candidate) + for values in right_by_interval.values(): + values.sort(key=lambda candidate: candidate.candidate_id) + pairs = [] + for candidate in sorted(left, key=lambda item: item.candidate_id): + matches = right_by_interval.get(candidate.interval, []) + if matches: + pairs.append((candidate, matches.pop(0), 1.0)) + return pairs + + +def compare_candidates(left: list[Candidate], right: list[Candidate]) -> dict: + """Return all overlaps, exact matches, threshold pairs, and unmatched IDs.""" + exact = exact_pairs(left, right) + exact_left = {pair[0].candidate_id for pair in exact} + exact_right = {pair[1].candidate_id for pair in exact} + overlaps = [] + for a in left: + for b in right: + if a.strand != b.strand: + continue + intersection = max(0, min(a.end0, b.end0) - max(a.start0, b.start0)) + if intersection <= 0: + continue + iou = interval_iou((a.start0, a.end0), (b.start0, b.end0)) + overlaps.append((a, b, iou)) + overlaps.sort(key=lambda pair: (pair[0].candidate_id, pair[1].candidate_id)) + + # Deterministic greedy one-to-one summary: reserve exact pairs first, then + # rank remaining eligible edges by descending IoU and ascending stable IDs. + eligible = [pair for pair in overlaps if pair[2] >= 0.5 + and pair[0].interval != pair[1].interval + and pair[0].candidate_id not in exact_left + and pair[1].candidate_id not in exact_right] + eligible.sort(key=lambda pair: (-pair[2], pair[0].candidate_id, pair[1].candidate_id)) + used_left, used_right = set(exact_left), set(exact_right) + threshold_pairs = [] + for pair in eligible: + if pair[0].candidate_id in used_left or pair[1].candidate_id in used_right: + continue + threshold_pairs.append(pair) + used_left.add(pair[0].candidate_id) + used_right.add(pair[1].candidate_id) + return { + "exact": exact, + "overlaps": overlaps, + "threshold_pairs": threshold_pairs, + "unmatched_left": sorted(c.candidate_id for c in left if c.candidate_id not in used_left), + "unmatched_right": sorted(c.candidate_id for c in right if c.candidate_id not in used_right), + } + + +def load_test_inputs(input_dir: Path = TEST_INPUT_DIR) -> tuple[dict[str, SeqRecord], list[dict]]: + """Load and validate the test genomic RefSeqGene FASTAs.""" + fasta_paths = sorted( + path.name for path in input_dir.iterdir() + if path.is_file() and path.suffix.lower() in {".fa", ".fasta"} + ) + if tuple(fasta_paths) != tuple(sorted(TEST_INPUT_FILES)): + raise ValueError( + f"Expected exactly {sorted(TEST_INPUT_FILES)} in {input_dir}, got {fasta_paths}" + ) + + records: dict[str, SeqRecord] = {} + provenance = [] + for filename in TEST_INPUT_FILES: + path = input_dir / filename + with path.open(encoding="utf-8") as handle: + parsed = list(SeqIO.parse(handle, "fasta")) + if len(parsed) != 1: + raise ValueError(f"Expected exactly one FASTA record in {filename}; found {len(parsed)}") + record = parsed[0] + if not record.id or not record.description: + raise ValueError(f"Missing FASTA identifier/header in {filename}") + accession_match = ACCESSION_VERSION.search(record.description) + if accession_match is None: + raise ValueError(f"No versioned RefSeqGene accession in FASTA header: {record.description}") + accession_version = accession_match.group(1) + if accession_version != TEST_INPUT_ACCESSIONS[filename]: + raise ValueError( + f"{filename} expected RefSeqGene accession {TEST_INPUT_ACCESSIONS[filename]}, " + f"found {accession_version}" + ) + if record.id != accession_version: + raise ValueError( + f"FASTA identifier {record.id!r} does not match header accession {accession_version!r}" + ) + sequence = str(record.seq).upper() + if not sequence: + raise ValueError(f"Empty DNA sequence in {filename}") + invalid = sorted(set(sequence) - DNA_ALPHABET) + if invalid: + raise ValueError(f"Invalid DNA symbols in {filename}: {invalid}") + if record.id in records: + raise ValueError(f"Duplicate FASTA record identifier: {record.id}") + records[record.id] = record + provenance.append({ + "filename": filename, + "original_header": record.description, + "accession_version": accession_version, + "sequence_length": len(sequence), + "file_sha256": hashlib.sha256(path.read_bytes()).hexdigest(), + "sequence_sha256": sha256_text(sequence), + }) + if TEST_INPUT_MANIFEST.is_file(): + with TEST_INPUT_MANIFEST.open(encoding="utf-8", newline="") as handle: + saved_manifest = list(csv.DictReader(handle, delimiter="\t")) + normalized_manifest = [ + {**entry, "sequence_length": int(entry["sequence_length"])} + for entry in saved_manifest + ] + if normalized_manifest != provenance: + raise ValueError( + f"Test FASTA files do not match their saved provenance manifest: " + f"{TEST_INPUT_MANIFEST}" + ) + return records, provenance + + +def write_tsv(path: Path, rows: list[dict]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + if not rows: + path.write_text("", encoding="utf-8") + return + with path.open("w", newline="", encoding="utf-8") as handle: + writer = csv.DictWriter(handle, fieldnames=list(rows[0]), delimiter="\t", extrasaction="ignore") + writer.writeheader() + writer.writerows(rows) + + +def write_json(path: Path, content) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(content, indent=2) + "\n", encoding="utf-8") + + +def _run_checked(command: list[str]) -> str: + """Run a command and surface its stderr if it fails, instead of just the exit code.""" + completed = subprocess.run(command, capture_output=True, text=True) + if completed.returncode != 0: + raise RuntimeError( + f"Command failed with exit status {completed.returncode}: {' '.join(command)}\n" + f"--- stderr ---\n{completed.stderr.strip()}" + ) + return completed.stdout + + +def _run_ncbi(executable: str, input_fasta: Path, out: Path, fmt: int, + minimum_nt: int = MINIMUM_NT, nested: bool = False) -> list[str]: + command = [executable, "-in", str(input_fasta), "-s", "0", "-g", str(GENETIC_CODE), + "-ml", str(minimum_nt), "-n", str(nested).lower(), "-strand", "both", + "-outfmt", str(fmt), "-out", str(out)] + _run_checked(command) + return command + + +def run_control_probes(executable: str) -> dict: + """Run and assert synthetic-only controls, preserving every NCBI output.""" + control_fasta = DATA / "controls.fasta" + control_records = {record.id: record for record in SeqIO.parse(control_fasta, "fasta")} + raw_dir = RESULTS / "raw" + raw_dir.mkdir(parents=True, exist_ok=True) + + outputs = {} + commands = [] + for minimum_nt in (30, MINIMUM_NT): + protein_path = raw_dir / f"controls_ml{minimum_nt}_nfalse_outfmt0.fasta" + cds_path = raw_dir / f"controls_ml{minimum_nt}_nfalse_outfmt1.fasta" + commands.append(_run_ncbi(executable, control_fasta, protein_path, 0, minimum_nt)) + commands.append(_run_ncbi(executable, control_fasta, cds_path, 1, minimum_nt)) + outputs[minimum_nt] = parse_ncbi_outputs(cds_path, protein_path, control_records) + nested_path = raw_dir / "controls_ml30_ntrue_outfmt1.fasta" + commands.append(_run_ncbi(executable, control_fasta, nested_path, 1, 30, nested=True)) + + baseline30 = outputs[30] + baseline75 = outputs[MINIMUM_NT] + by_record30 = defaultdict(list) + by_record75 = defaultdict(list) + for candidate in baseline30: + by_record30[candidate.record_id].append(candidate) + for candidate in baseline75: + by_record75[candidate.record_id].append(candidate) + required = {"forward_complete", "reverse_complete", "nested_pair", "incomplete_terminal", + "incomplete_terminal_reverse"} + if not required.issubset(control_records): + raise ValueError(f"Synthetic controls missing expected records: {sorted(required-set(control_records))}") + if not any(c.strand == "+" and c.terminal_stop_in_nt for c in by_record75["forward_complete"]): + raise AssertionError("NCBI did not report the controlled forward complete ORF with its stop") + if not any(c.strand == "-" and c.terminal_stop_in_nt for c in by_record75["reverse_complete"]): + raise AssertionError("NCBI did not report the controlled reverse complete ORF with its stop") + if not any(not c.terminal_stop_in_nt and c.nucleotide_length >= MINIMUM_NT for c in by_record75["incomplete_terminal"]): + raise AssertionError("NCBI terminal-ORF behavior did not retain the long incomplete control") + if not any(c.strand == "-" and not c.terminal_stop_in_nt and c.nucleotide_length >= MINIMUM_NT + for c in by_record75["incomplete_terminal_reverse"]): + raise AssertionError("NCBI terminal-ORF behavior did not retain the reverse-strand incomplete control") + nested_fasta_count = sum( + 1 for candidate in SeqIO.parse(nested_path, "fasta") + if candidate.id.startswith("lcl|nested_pair:") + ) + nested_false_count = len(by_record30["nested_pair"]) + if nested_false_count <= nested_fasta_count: + raise AssertionError("NCBI -n true did not remove the known nested control candidate") + + # Independently check the Python package's own nested-filter switch on the + # same control sequence; this is not mixed into the human transcript output. + nested_sequence = str(control_records["nested_pair"].seq).upper() + py_unfiltered = orffinder.getORFs(SeqRecord(Seq(nested_sequence)), minimum_length=30, + start_codons=["ATG"], remove_nested=False, trim_trailing=False) + py_filtered = orffinder.getORFs(SeqRecord(Seq(nested_sequence)), minimum_length=30, + start_codons=["ATG"], remove_nested=True, trim_trailing=False) + if len(py_unfiltered) <= len(py_filtered): + raise AssertionError("Python package nested-filter control did not remove a nested candidate") + control_raw_outputs: dict[str, list[dict]] = {} + control_candidate_rows = [] + control_by_tool = {"ncbi_ml30": baseline30, "ncbi_ml75": baseline75, + "python_raw": [], "adapter_corrected": []} + for record_id, record in control_records.items(): + sequence = str(record.seq).upper() + raw_candidates = python_raw_candidates(record_id, sequence, control_raw_outputs) + adapter_candidates = corrected_adapter_candidates(record_id, sequence) + control_by_tool["python_raw"].extend(raw_candidates) + control_by_tool["adapter_corrected"].extend(adapter_candidates) + write_json(RESULTS / "raw" / "controls_python_package_loci.json", control_raw_outputs) + for tool, candidates in control_by_tool.items(): + for candidate in candidates: + row = asdict(candidate) + row["tool"] = tool + control_candidate_rows.append(row) + write_tsv(RESULTS / "control_candidates.tsv", control_candidate_rows) + control_comparison_rows = [] + for record_id in control_records: + raw = [c for c in control_by_tool["python_raw"] if c.record_id == record_id] + corrected = [c for c in control_by_tool["adapter_corrected"] if c.record_id == record_id] + ncbi = [c for c in baseline75 if c.record_id == record_id] + for left_name, left, right_name, right in ( + ("python_raw", raw, "adapter_corrected", corrected), + ("adapter_corrected", corrected, "ncbi_ml75", ncbi), + ): + comparison = compare_candidates(left, right) + control_comparison_rows.append({ + "record_id": record_id, "left_tool": left_name, "right_tool": right_name, + "left_count": len(left), "right_count": len(right), + "exact_count": len(comparison["exact"]), + "positive_overlap_pair_count": len(comparison["overlaps"]), + "iou_ge_0_5_pair_count": len(comparison["threshold_pairs"]), + "unmatched_left_count": len(comparison["unmatched_left"]), + "unmatched_right_count": len(comparison["unmatched_right"]), + }) + write_tsv(RESULTS / "control_comparisons.tsv", control_comparison_rows) + return { + "ncbi_ml30_candidate_count": len(baseline30), + "ncbi_ml75_candidate_count": len(baseline75), + "ncbi_nested_false_count": nested_false_count, + "ncbi_nested_true_count": nested_fasta_count, + "python_nested_false_count": len(py_unfiltered), + "python_nested_true_count": len(py_filtered), + "ncbi_minimum_75_incomplete_control_included": bool(by_record75["incomplete_terminal"]), + "commands": commands, + } + + +def resolve_executable(executable: str | None = None) -> str: + """Locate the NCBI ORFfinder binary without any repository-stored default.""" + candidate = executable or os.environ.get(EXECUTABLE_ENV_VAR) or shutil.which(EXECUTABLE_NAME) + if not candidate: + raise FileNotFoundError( + f"NCBI ORFfinder not found. Pass --executable, set {EXECUTABLE_ENV_VAR}, " + f"or put an {EXECUTABLE_NAME!r} command on PATH." + ) + path = Path(candidate) + if not path.is_file() or not os.access(path, os.X_OK): + raise FileNotFoundError(f"NCBI ORFfinder is not an executable file: {path}") + return str(path.resolve()) + + +def run_benchmark(executable: str, input_dir: Path = TEST_INPUT_DIR) -> None: + executable = resolve_executable(executable) + ncbi_version = _run_checked([executable, "-version-full"]).strip() + records, input_provenance = load_test_inputs(input_dir) + RESULTS.mkdir(parents=True, exist_ok=True) + raw_dir = RESULTS / "raw" + raw_dir.mkdir(parents=True, exist_ok=True) + combined_fasta = raw_dir / "test_inputs.fasta" + SeqIO.write(list(records.values()), combined_fasta, "fasta") + + control_metadata = run_control_probes(executable) + raw_package_outputs: dict[str, list[dict]] = {} + ncbi_cds_path = raw_dir / "ncbi_outfmt1_cds.fasta" + ncbi_protein_path = raw_dir / "ncbi_outfmt0_proteins.fasta" + command1 = _run_ncbi(executable, combined_fasta, ncbi_protein_path, 0) + command2 = _run_ncbi(executable, combined_fasta, ncbi_cds_path, 1) + ncbi_candidates = parse_ncbi_outputs(ncbi_cds_path, ncbi_protein_path, records) + + candidates_by_tool: dict[str, list[Candidate]] = {"python_raw": [], "adapter_corrected": [], "ncbi": ncbi_candidates} + for record_id, record in records.items(): + sequence = str(record.seq).upper() + candidates_by_tool["python_raw"].extend(python_raw_candidates(record_id, sequence, raw_package_outputs)) + candidates_by_tool["adapter_corrected"].extend(corrected_adapter_candidates(record_id, sequence)) + + write_json(raw_dir / "python_package_loci.json", raw_package_outputs) + candidate_rows = [asdict(candidate) for tool in candidates_by_tool.values() for candidate in tool] + write_tsv(RESULTS / "candidates.tsv", candidate_rows) + pair_rows, match_rows, unmatched_rows, translation_difference_rows = [], [], [], [] + tools = ("python_raw", "adapter_corrected", "ncbi") + comparisons = {} + for left_name, right_name in itertools.combinations(tools, 2): + left_groups = defaultdict(list) + right_groups = defaultdict(list) + for candidate in candidates_by_tool[left_name]: + left_groups[candidate.record_id].append(candidate) + for candidate in candidates_by_tool[right_name]: + right_groups[candidate.record_id].append(candidate) + for record_id in records: + result = compare_candidates(left_groups[record_id], right_groups[record_id]) + comparisons[left_name, right_name, record_id] = result + for a, b, iou in result["exact"]: + match_rows.append({"record_id": record_id, "left_tool": left_name, "right_tool": right_name, + "match_type": "exact", "left_candidate_id": a.candidate_id, + "right_candidate_id": b.candidate_id, "iou": iou}) + if (left_name, right_name) == ("python_raw", "adapter_corrected"): + translation_difference_rows.append({ + "record_id": record_id, + "start0": a.start0, + "end0": a.end0, + "strand": a.strand, + "raw_candidate_id": a.candidate_id, + "adapter_candidate_id": b.candidate_id, + "raw_package_protein": a.protein, + "corrected_adapter_protein": b.protein, + "same_translation": a.protein == b.protein, + }) + for a, b, iou in result["overlaps"]: + pair_rows.append({"record_id": record_id, "left_tool": left_name, "right_tool": right_name, + "left_candidate_id": a.candidate_id, "right_candidate_id": b.candidate_id, + "left_interval": f"{a.start0}:{a.end0}:{a.strand}", + "right_interval": f"{b.start0}:{b.end0}:{b.strand}", "iou": iou, + "exact_interval_and_strand": a.interval == b.interval}) + for a, b, iou in result["threshold_pairs"]: + match_rows.append({"record_id": record_id, "left_tool": left_name, "right_tool": right_name, + "match_type": "greedy_iou_ge_0.5", "left_candidate_id": a.candidate_id, + "right_candidate_id": b.candidate_id, "iou": iou}) + comparison_label = f"{left_name}_vs_{right_name}" + unmatched_rows.extend( + {"record_id": record_id, "comparison": comparison_label, "tool": left_name, + "unmatched_against": right_name, "candidate_id": candidate_id} + for candidate_id in result["unmatched_left"] + ) + unmatched_rows.extend( + {"record_id": record_id, "comparison": comparison_label, "tool": right_name, + "unmatched_against": left_name, "candidate_id": candidate_id} + for candidate_id in result["unmatched_right"] + ) + + write_tsv(RESULTS / "overlaps.tsv", pair_rows) + write_tsv(RESULTS / "matches.tsv", match_rows) + write_tsv(RESULTS / "unmatched.tsv", unmatched_rows) + write_tsv(RESULTS / "translation_differences.tsv", translation_difference_rows) + provenance_by_accession = {entry["accession_version"]: entry for entry in input_provenance} + comparison_rows = [] + for record_id in records: + input_entry = provenance_by_accession[record_id] + gene = Path(input_entry["filename"]).stem + entry = { + "record_id": record_id, + "input_filename": input_entry["filename"], + "gene_label": gene, + } + for tool in tools: + candidates = [candidate for candidate in candidates_by_tool[tool] if candidate.record_id == record_id] + longest_length = max((candidate.nucleotide_length for candidate in candidates), default=0) + longest = [candidate for candidate in candidates if candidate.nucleotide_length == longest_length] + entry[f"{tool}_count"] = len(candidates) + longest_proteins = Counter(candidate.protein for candidate in candidates) + entry[f"{tool}_protein_translation_counts"] = json.dumps(dict(sorted(longest_proteins.items()))) + entry[f"{tool}_longest_nt"] = longest_length + entry[f"{tool}_longest_candidate_ids"] = json.dumps([c.candidate_id for c in longest]) + entry[f"{tool}_longest_locations"] = json.dumps([f"{c.start0}:{c.end0}:{c.strand}" for c in longest]) + entry[f"{tool}_longest_proteins"] = json.dumps([c.protein for c in longest]) + for left_name, right_name in itertools.combinations(tools, 2): + result = comparisons[left_name, right_name, record_id] + prefix = f"{left_name}_vs_{right_name}" + entry[f"{prefix}_exact_count"] = len(result["exact"]) + entry[f"{prefix}_positive_overlap_pair_count"] = len(result["overlaps"]) + entry[f"{prefix}_iou_ge_0_5_pair_count"] = len(result["threshold_pairs"]) + entry[f"{prefix}_unmatched_left_count"] = len(result["unmatched_left"]) + entry[f"{prefix}_unmatched_right_count"] = len(result["unmatched_right"]) + raw_adapter_exact = comparisons["python_raw", "adapter_corrected", record_id]["exact"] + entry["raw_adapter_exact_translation_difference_count"] = sum( + raw.protein != corrected.protein for raw, corrected, _ in raw_adapter_exact + ) + comparison_rows.append(entry) + write_tsv(RESULTS / "comparisons.tsv", comparison_rows) + + metadata = { + "generated_utc": datetime.now(timezone.utc).isoformat(), + "python_version": sys.version, + "biopython_version": biopython_version, + "orffinder_version": importlib.metadata.version("orffinder"), + "ncbi_orffinder_executable": executable, + "ncbi_orffinder_version": ncbi_version, + "parameters": {"start_codons": ["ATG"], "ncbi_start_option": "-s 0", "genetic_code": GENETIC_CODE, + "minimum_orf_length_nt": MINIMUM_NT, "nested_filter": False, "strand": "both", + "ncbi_minimum_floor_nt": 30}, + "commands": [command1, command2], + "test_inputs": input_provenance, + "combined_input_fasta_sha256": hashlib.sha256(combined_fasta.read_bytes()).hexdigest(), + "synthetic_control_results": control_metadata, + "dataset_type": "three genomic RefSeqGene FASTA test records; no sequence download or GenBank annotation", + "biological_scope": "ORF scanning does not perform splicing and does not establish the protein annotated for the named gene", + "coordinate_system": "zero-based half-open on original input; strand separate", + "ncbi_terminal_stop_normalization": "CDS FASTA sequence/location includes stop; protein excludes translated terminal stop; normalized interval retains the terminal stop nucleotide", + "overlap_rule": "All same-strand pairs with positive intersection are emitted in overlaps.tsv, including exact interval-and-strand matches. Exact pairs are flagged and also listed in matches.tsv.", + "pairing_rule": "Reserve exact matches one-to-one in ascending stable candidate-ID order. Among remaining candidates, sort positive-overlap edges with IoU >= 0.5 by descending IoU then ascending left/right candidate IDs; greedily accept an edge if neither endpoint is already paired. Report unmatched candidates after these pairings.", + } + write_json(RESULTS / "metadata.json", metadata) + write_report(comparison_rows, input_provenance) + + +def write_report(comparisons: list[dict], input_provenance: list[dict]) -> None: + input_count = len(comparisons) + input_label = "record" if input_count == 1 else "records" + lines = [ + "# ORFfinder benchmark results", + "", + f"Completed comparison of the {input_count} genomic RefSeqGene FASTA test {input_label} using raw `orffinder`, Lverage's corrected adapter, and NCBI standalone ORFfinder. ORF scanning on genomic sequence does not perform splicing or establish the protein annotated for the named gene.", + "", + "## Results per test input", + "", + "| Input file | RefSeqGene accession.version | Raw package ORFs | Corrected adapter ORFs | NCBI ORFs | Raw/adapter exact-region translation differences | Adapter/NCBI exact matches | Adapter/NCBI positive overlaps | Same longest interval? |", + "|---|---|---:|---:|---:|---:|---:|---:|---|", + ] + by_record = {row["record_id"]: row for row in comparisons} + adapter_total = sum(int(row["adapter_corrected_count"]) for row in comparisons) + ncbi_total = sum(int(row["ncbi_count"]) for row in comparisons) + exact_total = sum(int(row["adapter_corrected_vs_ncbi_exact_count"]) for row in comparisons) + adapter_unmatched = max(0, adapter_total - exact_total) + ncbi_unmatched = max(0, ncbi_total - exact_total) + same_longest_inputs = sum( + 1 + for row in comparisons + if set(json.loads(row["adapter_corrected_longest_locations"])) & set(json.loads(row["ncbi_longest_locations"])) + ) + input_word = "input" if input_count == 1 else "inputs" + for item in input_provenance: + record_id = item["accession_version"] + row = by_record[record_id] + exact = row["adapter_corrected_vs_ncbi_exact_count"] + same_longest = bool(set(json.loads(row["adapter_corrected_longest_locations"])) & set(json.loads(row["ncbi_longest_locations"]))) + lines.append(f"| {item['filename']} | {record_id} | {row['python_raw_count']} | {row['adapter_corrected_count']} | {row['ncbi_count']} | {row['raw_adapter_exact_translation_difference_count']} | {exact} | {row['adapter_corrected_vs_ncbi_positive_overlap_pair_count']} | {'yes' if same_longest else 'no'} |") + + if input_count == 1: + if same_longest_inputs: + longest_summary = "The test input shares at least one exact longest-ORF interval and strand between the adapter and NCBI." + else: + longest_summary = "The test input does not share an exact longest-ORF interval and strand between the adapter and NCBI." + elif same_longest_inputs == input_count: + longest_summary = f"All {input_count} test inputs share at least one exact longest-ORF interval and strand between the adapter and NCBI." + else: + longest_summary = f"{same_longest_inputs} of {input_count} test inputs share at least one exact longest-ORF interval and strand between the adapter and NCBI." + + if ncbi_unmatched == 0: + unmatched_summary = ( + "Every NCBI candidate had an exact adapter match. " + f"The corrected adapter had {adapter_unmatched:,} candidates without an exact NCBI match; " + f"NCBI had {ncbi_unmatched:,} candidates without an exact adapter match." + ) + else: + unmatched_summary = ( + "Not every NCBI candidate had an exact adapter match: " + f"the corrected adapter had {adapter_unmatched:,} candidates without an exact NCBI match, " + f"and NCBI had {ncbi_unmatched:,} candidates without an exact adapter match." + ) + + lines.extend([ + "", + "## Findings", + "", + f"- Across the {input_count} test {input_word}, the corrected adapter returned {adapter_total:,} candidates and NCBI returned {ncbi_total:,}; there were {exact_total:,} exact interval-and-strand matches.", + f"- {unmatched_summary}", + f"- {longest_summary}", + "- Raw-package/adapter protein-string differences include terminal stop-symbol removal. Their counts do not indicate incorrect proteins.", + "- These findings apply only to the test genomic sequences and recorded settings. ORF scanning does not perform splicing or establish the annotated protein for the named gene.", + "", + "## Interpretation", + "", + "- Called-region differences are represented in `candidates.tsv`, `overlaps.tsv`, `matches.tsv`, and `unmatched.tsv`; exact matches and non-exact overlaps are separate.", + "- Overlap means a positive intersection on the same strand; `overlaps.tsv` includes exact interval pairs with IoU 1.0 and marks them explicitly. Exact matches are also listed separately in `matches.tsv`. The additional one-to-one summary uses deterministic greedy pairing at IoU ≥ 0.5 after exact matches are reserved.", + "- Longest ORFs are ranked by normalized nucleotide span. All tied longest locations and proteins are retained in `comparisons.tsv`; same-longest means the tied sets share at least one exact interval-and-strand location.", + "- Raw package translation differences are counted only for exact same-region raw/adapter candidate pairs in `translation_differences.tsv`; candidates found by only one discovery path remain region differences. The corrected adapter candidate reconstruction is checked against production `OrffinderOrfSearcher.get_orfs()` as an ordered protein list and `Counter`, preserving duplicate counts.", + "- Tool settings and exact commands are in `metadata.json`. NCBI outfmt 1 locations are 1-based inclusive; they are normalized to half-open intervals while retaining stop codons. Outfmt 0 supplies proteins and is cross-checked against the matching CDS record.", + "- `metadata.json` records each input filename, original FASTA header, accession version, sequence length, and file/sequence SHA-256 checksums. Synthetic controls are validation fixtures and are not part of these test-input results.", + "", + ]) + # Keep generated Markdown sections separated and normalize the file ending + # so renderers do not treat trailing blank lines as an extra paragraph. + report = "\n".join(lines).rstrip() + (RESULTS / "report.md").write_text(f"{report}\n", encoding="utf-8") + + +def main() -> None: + global RESULTS + parser = argparse.ArgumentParser(description=__doc__) + subparsers = parser.add_subparsers(dest="command", required=True) + run_parser = subparsers.add_parser("run", help="compare the three genomic RefSeqGene test FASTAs") + run_parser.add_argument( + "--executable", default=None, + help=f"NCBI ORFfinder binary (default: ${EXECUTABLE_ENV_VAR}, then {EXECUTABLE_NAME!r} on PATH)", + ) + run_parser.add_argument( + "--input-dir", type=Path, + default=Path(os.environ.get(INPUT_DIR_ENV_VAR, TEST_INPUT_DIR)), + help=f"directory holding {', '.join(TEST_INPUT_FILES)} (default: ${INPUT_DIR_ENV_VAR} or {TEST_INPUT_DIR})", + ) + run_parser.add_argument( + "--results-dir", type=Path, default=RESULTS, + help=f"where to write generated outputs (default: {RESULTS}); not versioned", + ) + args = parser.parse_args() + RESULTS = args.results_dir + run_benchmark(args.executable, args.input_dir) + + +if __name__ == "__main__": + main() diff --git a/benchmarks/orf_finder/data/controls.fasta b/benchmarks/orf_finder/data/controls.fasta new file mode 100644 index 0000000..439be05 --- /dev/null +++ b/benchmarks/orf_finder/data/controls.fasta @@ -0,0 +1,10 @@ +>forward_complete controlled forward ORF with terminal stop +ATGGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTTAA +>reverse_complete controlled reverse-strand ORF with terminal stop +TTAAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCCAT +>nested_pair controlled nested-candidate test sequence +AATATCAATGGGTCAGCGGTACGCCGGGGGTACCGTAAGGCACATAGTATTAGAATGATA +>incomplete_terminal 78-nt terminal ORF without a stop codon +ATGGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCTGCT +>incomplete_terminal_reverse reverse-strand incomplete terminal ORF without a stop codon +AGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCAGCCAT diff --git a/benchmarks/orf_finder/data/test_input_manifest.tsv b/benchmarks/orf_finder/data/test_input_manifest.tsv new file mode 100644 index 0000000..a64f5ee --- /dev/null +++ b/benchmarks/orf_finder/data/test_input_manifest.tsv @@ -0,0 +1,4 @@ +filename original_header accession_version sequence_length file_sha256 sequence_sha256 +HoxA13.fa NG_008181.2 Homo sapiens homeobox A13 (HOXA13), RefSeqGene (LRG_1349) on chromosome 7 NG_008181.2 12743 a4d037ab00e9af8d17daddc5da9b12daf400bef83428581b3f94a1270b15e9da 55df719104935ff8d6a01d3d1c1d114833e363e2051cf2569d475f75ceb6c8e0 +Jun.fa NG_047027.2 Homo sapiens Jun proto-oncogene, AP-1 transcription factor subunit (JUN), RefSeqGene on chromosome 1 NG_047027.2 10257 1271f5cec76acf8b5e996ff8bf0a13e1e58188640f5e70674554837421f3bb64 9ddf2b109fae43b10667d4eab5b2214b2c8d35d4fcca21ea61fd6b7f80d8d323 +MITF.fa NG_011631.1 Homo sapiens melanocyte inducing transcription factor (MITF), RefSeqGene (LRG_776) on chromosome 3 NG_011631.1 235855 2800eadff068caa44224fb79374879c47dff349f7b2789e130feae296710f5c7 9672268bbb58d07683e2b25a996200e7dd6fe02b0c79e2f3c1a277d391dec6fa diff --git a/benchmarks/orf_finder/run.qsub b/benchmarks/orf_finder/run.qsub new file mode 100644 index 0000000..163c107 --- /dev/null +++ b/benchmarks/orf_finder/run.qsub @@ -0,0 +1,64 @@ +#!/bin/bash -l + +# Grid Engine (SGE/UGE) job script for the ORFfinder benchmark. Nothing +# site- or user-specific is stored here; add scheduler options your cluster +# needs (for example -P ) on the qsub command line, and configure +# the job through environment variables (qsub -v NAME=value, or export before +# qsub): +# +# ORFFINDER_EXECUTABLE path to the NCBI ORFfinder binary (required unless +# an "ORFfinder" command is already on PATH) +# ORF_BENCHMARK_INPUTS directory holding HoxA13.fa, Jun.fa, MITF.fa +# (default: benchmarks/orf_finder/data/test_inputs) +# CONDA_ENV conda environment name or prefix to activate +# (default: none; the job uses whatever "python" +# resolves to) +# CONDA_MODULE environment module to load before activating +# CONDA_ENV, on clusters that provide conda that way +# (default: none) +# +# The job runs from the directory it was submitted from (-cwd), which must be +# the repository root of the checkout you want to benchmark. +#$ -N lverage_orf_benchmark +#$ -cwd +#$ -j y +#$ -l h_rt=02:00:00 +#$ -pe omp 1 + +set -euo pipefail + +REPO_ROOT="${SGE_O_WORKDIR:-$PWD}" +cd "$REPO_ROOT" +if [[ ! -f pyproject.toml || ! -f benchmarks/orf_finder/benchmark.py ]]; then + echo "Submit this job from the Lverage repository root (got: $REPO_ROOT)" >&2 + exit 2 +fi + +if [[ -n "${CONDA_ENV:-}" ]]; then + if [[ -n "${CONDA_MODULE:-}" ]]; then + module load "$CONDA_MODULE" + fi + source "$(conda info --base)/etc/profile.d/conda.sh" + conda activate "$CONDA_ENV" +fi + +# Import the checkout being benchmarked, not whatever lverage happens to be +# installed in the environment (for example another clone in editable mode). +export PYTHONPATH="$REPO_ROOT/src${PYTHONPATH:+:$PYTHONPATH}" + +ORFFINDER_EXECUTABLE="${ORFFINDER_EXECUTABLE:-$(command -v ORFfinder || true)}" +if [[ -z "$ORFFINDER_EXECUTABLE" || ! -x "$ORFFINDER_EXECUTABLE" ]]; then + echo "Set ORFFINDER_EXECUTABLE to the NCBI ORFfinder binary (got: '${ORFFINDER_EXECUTABLE}')" >&2 + exit 2 +fi +export ORFFINDER_EXECUTABLE +INPUT_DIR="${ORF_BENCHMARK_INPUTS:-$REPO_ROOT/benchmarks/orf_finder/data/test_inputs}" + +printf 'Host: %s\n' "$(hostname)" +printf 'Repository: %s\n' "$REPO_ROOT" +printf 'Python: %s\n' "$(command -v python)" +printf 'ORFfinder: %s\n' "$ORFFINDER_EXECUTABLE" +printf 'Test input FASTAs: %s\n' "$INPUT_DIR" +python -m benchmarks.orf_finder.benchmark run \ + --executable "$ORFFINDER_EXECUTABLE" \ + --input-dir "$INPUT_DIR" diff --git a/benchmarks/orf_finder/tests/test_benchmark.py b/benchmarks/orf_finder/tests/test_benchmark.py new file mode 100644 index 0000000..a0c6deb --- /dev/null +++ b/benchmarks/orf_finder/tests/test_benchmark.py @@ -0,0 +1,373 @@ +import tempfile +import unittest +from collections import Counter +from pathlib import Path +from unittest.mock import patch + +from Bio.Seq import Seq +from Bio.SeqRecord import SeqRecord + +from benchmarks.orf_finder import benchmark + + +class BenchmarkCoordinateTests(unittest.TestCase): + + def test_locus_coordinates_convert_both_strands_to_half_open(self): + forward = {"start": 1, "end": 34, "sense": "+", "length": 33, "trailing": False} + reverse = {"start": 34, "end": 1, "sense": "-", "length": 33, "trailing": False} + + self.assertEqual(benchmark._locus_interval(forward, 33), (0, 33, "+", 33)) + self.assertEqual(benchmark._locus_interval(reverse, 33), (0, 33, "-", 33)) + + def test_interval_iou_uses_half_open_bounds(self): + self.assertEqual(benchmark.interval_iou((0, 10), (5, 15)), 1 / 3) + self.assertEqual(benchmark.interval_iou((0, 10), (10, 20)), 0.0) + + def test_incomplete_terminal_locus_uses_actual_span(self): + locus = {"start": 1, "end": 75, "sense": "+", "length": 74, "trailing": True} + + self.assertEqual(benchmark._locus_interval(locus, 75), (0, 75, "+", 75)) + + def test_reverse_incomplete_terminal_locus_uses_original_input_bounds(self): + locus = {"start": 76, "end": 1, "sense": "-", "length": 74, "trailing": True} + + self.assertEqual(benchmark._locus_interval(locus, 75), (0, 75, "-", 75)) + + +class BenchmarkPairingTests(unittest.TestCase): + + @staticmethod + def candidate(candidate_id, start, end, strand="+"): + return benchmark.Candidate( + candidate_id, "test", "TX.1", start, end, strand, end - start, + "MTEST", False, False, + ) + + def test_all_positive_overlaps_are_retained_and_exact_matches_separate(self): + left = [self.candidate("a", 0, 10), self.candidate("b", 5, 15)] + right = [self.candidate("x", 0, 10), self.candidate("y", 8, 18), self.candidate("z", 1, 4, "-")] + + result = benchmark.compare_candidates(left, right) + + self.assertEqual([(a.candidate_id, b.candidate_id) for a, b, _ in result["exact"]], [("a", "x")]) + self.assertEqual( + {(a.candidate_id, b.candidate_id) for a, b, _ in result["overlaps"]}, + {("a", "x"), ("a", "y"), ("b", "x"), ("b", "y")}, + ) + self.assertEqual(sum(iou == 1.0 for _, _, iou in result["overlaps"]), 1) + self.assertEqual(result["unmatched_left"], []) + self.assertEqual(result["unmatched_right"], ["z"]) + + def test_threshold_pairing_is_deterministic_and_one_to_one(self): + left = [self.candidate("left-b", 0, 10), self.candidate("left-a", 0, 10)] + right = [self.candidate("right-b", 0, 10), self.candidate("right-a", 0, 10)] + + result = benchmark.compare_candidates(left, right) + + self.assertEqual( + [(a.candidate_id, b.candidate_id) for a, b, _ in result["exact"]], + [("left-a", "right-a"), ("left-b", "right-b")], + ) + self.assertEqual(result["threshold_pairs"], []) + + def test_greedy_iou_tie_breaks_by_stable_candidate_ids(self): + left = [self.candidate("A", 0, 10), self.candidate("B", 4, 14)] + right = [self.candidate("X", 2, 12), self.candidate("Y", 2, 12)] + + result = benchmark.compare_candidates(left, right) + + self.assertEqual( + [(a.candidate_id, b.candidate_id) for a, b, _ in result["threshold_pairs"]], + [("A", "X"), ("B", "Y")], + ) + + +class BenchmarkNcbiParserTests(unittest.TestCase): + + def parse(self, record_id, interval_header, orf_tag, cds_sequence, protein, input_sequence): + record = SeqRecord(Seq(input_sequence), id=record_id) + with tempfile.TemporaryDirectory() as temp_dir: + cds_path = Path(temp_dir) / "cds.fasta" + protein_path = Path(temp_dir) / "protein.fasta" + cds_path.write_text(f">lcl|{record_id}:{interval_header} {orf_tag}\n{cds_sequence}\n") + protein_path.write_text(f">lcl|{orf_tag}\n{protein}\n") + return benchmark.parse_ncbi_outputs(cds_path, protein_path, {record_id: record}) + + def test_plus_strand_location_stop_inclusion_and_protein_crosscheck(self): + cds = "ATG" + "GCT" * 9 + "TAA" + protein = "M" + "A" * 9 + candidates = self.parse("TX.1", "1-33", "ORF1_TX.1:0:32", cds, protein, cds) + + self.assertEqual(len(candidates), 1) + self.assertEqual(candidates[0].interval, (0, 33, "+")) + self.assertEqual(candidates[0].nucleotide_length, 33) + self.assertTrue(candidates[0].terminal_stop_in_nt) + self.assertEqual(candidates[0].protein, protein) + + def test_plus_strand_location_requires_start_to_end_endpoint_order(self): + cds = "ATG" + "GCT" * 9 + "TAA" + protein = "M" + "A" * 9 + candidates = self.parse("TX.1", "1-33", "ORF1_TX.1:0:32", cds, protein, cds) + + self.assertEqual(candidates[0].interval, (0, 33, "+")) + + def test_minus_strand_location_requires_end_to_start_endpoint_order(self): + oriented_cds = "ATG" + "GCT" * 9 + "TAA" + input_sequence = str(Seq(oriented_cds).reverse_complement()) + protein = "M" + "A" * 9 + candidates = self.parse("TX.1", "c33-1", "ORF1_TX.1:32:0", oriented_cds, protein, input_sequence) + + self.assertEqual(candidates[0].interval, (0, 33, "-")) + + def test_reversed_endpoint_order_is_rejected_on_both_strands(self): + cds = "ATG" + "GCT" * 9 + "TAA" + protein = "M" + "A" * 9 + with self.assertRaisesRegex(ValueError, "coordinate encodings disagree"): + self.parse("TX.1", "1-33", "ORF1_TX.1:32:0", cds, protein, cds) + + oriented_cds = "ATG" + "GCT" * 9 + "TAA" + input_sequence = str(Seq(oriented_cds).reverse_complement()) + with self.assertRaisesRegex(ValueError, "coordinate encodings disagree"): + self.parse("TX.1", "c33-1", "ORF1_TX.1:0:32", oriented_cds, protein, input_sequence) + + def test_mismatched_ncbi_cds_and_protein_is_rejected(self): + cds = "ATG" + "GCT" * 9 + "TAA" + with self.assertRaisesRegex(ValueError, "does not match translated CDS"): + self.parse("TX.1", "1-33", "ORF1_TX.1:0:32", cds, "M", cds) + + def test_terminal_stop_is_removed_from_protein_but_retained_in_interval(self): + cds = "ATG" + "GCT" * 9 + "TAA" + protein = "M" + "A" * 9 + candidate = self.parse("TX.1", "1-33", "ORF1_TX.1:0:32", cds, protein, cds)[0] + + self.assertEqual(candidate.interval, (0, 33, "+")) + self.assertTrue(candidate.terminal_stop_in_nt) + self.assertTrue(candidate.complete) + self.assertEqual(candidate.protein, protein) + + +class BenchmarkAdapterAssociationTests(unittest.TestCase): + + def setUp(self): + self.record_id = "SYNTHETIC.1" + self.sequence = "ATG" + "GCT" * 24 + + def test_adapter_corrects_package_one_nt_terminal_threshold_behavior(self): + corrected = benchmark.corrected_adapter_candidates(self.record_id, self.sequence) + raw = benchmark.python_raw_candidates(self.record_id, self.sequence) + + self.assertEqual(len(corrected), 1) + self.assertEqual(corrected[0].interval, (0, 75, "+")) + self.assertEqual(corrected[0].protein, "M" + "A" * 24) + self.assertEqual(raw, []) + + def test_corrected_candidate_proteins_match_production_adapter_with_duplicates(self): + orf = "ATG" + "GCT" * 23 + "TAA" + sequence = orf + orf + candidates = benchmark.corrected_adapter_candidates(self.record_id, sequence) + + proteins = [candidate.protein for candidate in candidates] + self.assertGreaterEqual(Counter(proteins).most_common(1)[0][1], 2) + + +class TestInputValidationTests(unittest.TestCase): + + @staticmethod + def write_test_files(directory, replacement=None): + accessions = { + "HoxA13.fa": "NG_008181.2", + "Jun.fa": "NG_047027.2", + "MITF.fa": "NG_011631.1", + } + for filename, accession in accessions.items(): + sequence = replacement if filename == "HoxA13.fa" and replacement is not None else "ATGAAATAG" + (directory / filename).write_text(f">{accession} test record\n{sequence}\n", encoding="ascii") + + @unittest.skipUnless( + benchmark.TEST_INPUT_DIR.is_dir(), + "test FASTA inputs are not versioned; place them under data/test_inputs to run", + ) + def test_local_inputs_are_single_versioned_dna_records_with_checksums(self): + records, provenance = benchmark.load_test_inputs() + + self.assertEqual([entry["filename"] for entry in provenance], list(benchmark.TEST_INPUT_FILES)) + self.assertEqual(len(records), 3) + for entry in provenance: + self.assertEqual(entry["sequence_length"], len(records[entry["accession_version"]].seq)) + self.assertEqual(len(entry["file_sha256"]), 64) + self.assertEqual(len(entry["sequence_sha256"]), 64) + self.assertIn(entry["accession_version"], entry["original_header"]) + + def test_dataset_validation_rejects_invalid_dna_symbols(self): + with tempfile.TemporaryDirectory() as temp_dir: + directory = Path(temp_dir) + self.write_test_files(directory, replacement="ATG-UAA") + + with self.assertRaisesRegex(ValueError, "Invalid DNA symbols"): + benchmark.load_test_inputs(directory) + + def test_external_input_directory_is_checked_against_saved_manifest(self): + with tempfile.TemporaryDirectory() as temp_dir: + directory = Path(temp_dir) + self.write_test_files(directory) + + with self.assertRaisesRegex(ValueError, "saved provenance manifest"): + benchmark.load_test_inputs(directory) + + def test_executable_resolution_requires_an_explicit_or_discoverable_binary(self): + with patch.dict(benchmark.os.environ, {benchmark.EXECUTABLE_ENV_VAR: ""}), \ + patch.object(benchmark.shutil, "which", return_value=None): + with self.assertRaisesRegex(FileNotFoundError, benchmark.EXECUTABLE_ENV_VAR): + benchmark.resolve_executable(None) + with tempfile.TemporaryDirectory() as temp_dir: + missing = Path(temp_dir) / "ORFfinder" + with self.assertRaisesRegex(FileNotFoundError, "not an executable file"): + benchmark.resolve_executable(str(missing)) + + def test_dataset_validation_rejects_multiple_records_in_one_input(self): + with tempfile.TemporaryDirectory() as temp_dir: + directory = Path(temp_dir) + self.write_test_files(directory) + path = directory / "HoxA13.fa" + path.write_text(path.read_text(encoding="ascii") + ">extra\nATGAAATAG\n", encoding="ascii") + + with self.assertRaisesRegex(ValueError, "exactly one FASTA record"): + benchmark.load_test_inputs(directory) + + @staticmethod + def report_for(rows, provenance): + with tempfile.TemporaryDirectory() as temp_dir, patch.object(benchmark, "RESULTS", Path(temp_dir)): + benchmark.write_report(rows, provenance) + return (Path(temp_dir) / "report.md").read_text(encoding="utf-8") + + def test_report_current_all_ncbi_match_case_uses_row_counts(self): + rows = [ + { + "record_id": "NG_008181.2", + "python_raw_count": 2, + "adapter_corrected_count": 3, + "ncbi_count": 3, + "raw_adapter_exact_translation_difference_count": 1, + "adapter_corrected_vs_ncbi_exact_count": 3, + "adapter_corrected_vs_ncbi_positive_overlap_pair_count": 5, + "adapter_corrected_longest_locations": '["0:99:+"]', + "ncbi_longest_locations": '["0:99:+"]', + }, + { + "record_id": "NG_047027.2", + "python_raw_count": 2, + "adapter_corrected_count": 3, + "ncbi_count": 3, + "raw_adapter_exact_translation_difference_count": 1, + "adapter_corrected_vs_ncbi_exact_count": 3, + "adapter_corrected_vs_ncbi_positive_overlap_pair_count": 4, + "adapter_corrected_longest_locations": '["10:20:+"]', + "ncbi_longest_locations": '["10:20:+"]', + }, + { + "record_id": "NG_011631.1", + "python_raw_count": 2, + "adapter_corrected_count": 3, + "ncbi_count": 3, + "raw_adapter_exact_translation_difference_count": 1, + "adapter_corrected_vs_ncbi_exact_count": 3, + "adapter_corrected_vs_ncbi_positive_overlap_pair_count": 4, + "adapter_corrected_longest_locations": '["20:30:+"]', + "ncbi_longest_locations": '["20:30:+"]', + }, + ] + provenance = [ + {"filename": "HoxA13.fa", "accession_version": "NG_008181.2"}, + {"filename": "Jun.fa", "accession_version": "NG_047027.2"}, + {"filename": "MITF.fa", "accession_version": "NG_011631.1"}, + ] + + report = self.report_for(rows, provenance) + + self.assertIn("Across the 3 test inputs, the corrected adapter returned 9 candidates and NCBI returned 9; there were 9 exact interval-and-strand matches.", report) + self.assertIn("Every NCBI candidate had an exact adapter match.", report) + self.assertIn("The corrected adapter had 0 candidates without an exact NCBI match; NCBI had 0 candidates without an exact adapter match.", report) + self.assertIn("All 3 test inputs share at least one exact longest-ORF interval and strand between the adapter and NCBI.", report) + + def test_report_marks_unmatched_candidates_on_both_sides_with_more_ncbi_candidates(self): + rows = [{ + "record_id": "NG_008181.2", + "python_raw_count": 2, + "adapter_corrected_count": 5, + "ncbi_count": 7, + "raw_adapter_exact_translation_difference_count": 1, + "adapter_corrected_vs_ncbi_exact_count": 3, + "adapter_corrected_vs_ncbi_positive_overlap_pair_count": 5, + "adapter_corrected_longest_locations": '["0:99:+"]', + "ncbi_longest_locations": '["0:99:+"]', + }] + provenance = [{"filename": "HoxA13.fa", "accession_version": "NG_008181.2"}] + + report = self.report_for(rows, provenance) + + self.assertIn("Across the 1 test input", report) + self.assertIn("Not every NCBI candidate had an exact adapter match:", report) + self.assertIn("the corrected adapter had 2 candidates without an exact NCBI match, and NCBI had 4 candidates without an exact adapter match.", report) + + def test_report_reports_longest_orf_disagreement(self): + rows = [{ + "record_id": "NG_008181.2", + "python_raw_count": 2, + "adapter_corrected_count": 3, + "ncbi_count": 3, + "raw_adapter_exact_translation_difference_count": 1, + "adapter_corrected_vs_ncbi_exact_count": 3, + "adapter_corrected_vs_ncbi_positive_overlap_pair_count": 5, + "adapter_corrected_longest_locations": '["0:99:+"]', + "ncbi_longest_locations": '["1:100:+"]', + }] + provenance = [{"filename": "HoxA13.fa", "accession_version": "NG_008181.2"}] + + report = self.report_for(rows, provenance) + + self.assertIn("The test input does not share an exact longest-ORF interval and strand between the adapter and NCBI.", report) + + def test_report_uses_single_input_pluralization_for_one_input(self): + rows = [{ + "record_id": "NG_008181.2", + "python_raw_count": 1, + "adapter_corrected_count": 4, + "ncbi_count": 2, + "raw_adapter_exact_translation_difference_count": 1, + "adapter_corrected_vs_ncbi_exact_count": 2, + "adapter_corrected_vs_ncbi_positive_overlap_pair_count": 3, + "adapter_corrected_longest_locations": '["0:99:+"]', + "ncbi_longest_locations": '["0:99:+"]', + }] + provenance = [{"filename": "HoxA13.fa", "accession_version": "NG_008181.2"}] + + report = self.report_for(rows, provenance) + + self.assertIn("Completed comparison of the 1 genomic RefSeqGene FASTA test record using raw `orffinder`, Lverage's corrected adapter, and NCBI standalone ORFfinder.", report) + self.assertIn("Across the 1 test input, the corrected adapter returned 4 candidates and NCBI returned 2; there were 2 exact interval-and-strand matches.", report) + self.assertIn("The test input shares at least one exact longest-ORF interval and strand between the adapter and NCBI.", report) + + def test_completed_report_uses_input_labels_and_clean_markdown_spacing(self): + row = { + "record_id": "NG_008181.2", + "python_raw_count": 2, + "adapter_corrected_count": 3, + "ncbi_count": 4, + "raw_adapter_exact_translation_difference_count": 1, + "adapter_corrected_vs_ncbi_exact_count": 2, + "adapter_corrected_vs_ncbi_positive_overlap_pair_count": 5, + "adapter_corrected_longest_locations": '["0:99:+"]', + "ncbi_longest_locations": '["0:99:+"]', + } + provenance = [{"filename": "HoxA13.fa", "accession_version": "NG_008181.2"}] + + report = self.report_for([row], provenance) + + self.assertIn("HoxA13.fa | NG_008181.2 | 2 | 3 | 4", report) + self.assertIn("does not perform splicing", report) + self.assertTrue(report.endswith("\n")) + self.assertFalse(report.endswith("\n\n")) + + +if __name__ == "__main__": + unittest.main() diff --git a/conda_lverage.yml b/conda_lverage.yml index 64b9eb2..cdf4962 100644 --- a/conda_lverage.yml +++ b/conda_lverage.yml @@ -34,6 +34,7 @@ dependencies: - idna==3.7 - itsdangerous==2.2.0 - msgspec==0.18.6 + - orffinder==1.8 - pytz==2024.1 - six==1.16.0 - urllib3==2.2.1 diff --git a/docs/singularity_setup.md b/docs/singularity_setup.md index 497deb7..556e2cb 100644 --- a/docs/singularity_setup.md +++ b/docs/singularity_setup.md @@ -28,4 +28,32 @@ Required data: Setup notes: - BLAST+ must be installed and available on `PATH`. - Protein databases should be created with `makeblastdb`. -- Use `-parse_seqids` so sequences can be retrieved later with `blastdbcmd`. \ No newline at end of file +- Use `-parse_seqids` so sequences can be retrieved later with `blastdbcmd`. + +## ORFfinder comparison benchmark + +- The benchmark compares Python `orffinder==1.8` with NCBI standalone ORFfinder. +- NCBI ORFfinder 0.4.3 was used for the recorded run. Download the standalone + binary from NCBI and point the benchmark at it with `ORFFINDER_EXECUTABLE` + (or put an `ORFfinder` command on `PATH`). +- The production Lverage adapter continues to use the Python package; the NCBI + executable is required only for the optional benchmark. +- The benchmark uses only the three professor-provided genomic RefSeqGene FASTA + files (HoxA13.fa, Jun.fa, MITF.fa). They are not versioned; supply a local + directory via `--input-dir` or `LVERAGE_ORF_BENCHMARK_INPUTS`. It does not + fetch replacement sequences or require GenBank records/CDS annotations. +- These are genomic sequences: ORF scanning does not splice exons and cannot + establish the named gene's mature transcript or annotated protein. Synthetic + sequences under `data/controls.fasta` are validation fixtures kept separate + from assigned-input results. +- The runner, parser/coordinate tests, SCC job script, and input + provenance/checksums are under `benchmarks/orf_finder/`. Generated outputs + are written to `benchmarks/orf_finder/results/` and are not versioned; the + summary of the recorded run is in the benchmark README. +- The standalone binary is not installed by Conda or pip. Confirm its version + and executable path before running. The benchmark job script activates the + Conda environment named in `CONDA_ENV` if that variable is set, and runs + from the repository root it was submitted from. +- SCC SGE requires an account scheduler project via `qsub -P`; this repository + does not infer that identifier from a Unix group or directory name. Set the + confirmed project at submission time as documented in the benchmark README. \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 29411bd..58ed8ff 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,4 +1,5 @@ selenium biopython +orffinder==1.8 six ete3 \ No newline at end of file