Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/documentation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,9 @@ jobs:
run: python -m unittest discover -s tests/v2 -p 'test_*.py' -v

- name: Run the retained legacy tests
run: python -m unittest discover -s tests -p 'test_jaspar_window.py' -v
# Run only the legacy file; V2 tests run separately above. Recursive
# discovery mixes the legacy dependency stubs with real V2 imports.
run: python tests/test_jaspar_window.py -v

- name: Build the documentation
run: sphinx-build -W --keep-going --nitpicky -b html docs docs/_build/html
Expand Down
28 changes: 21 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,14 +100,28 @@ pipeline.

## V2 Command-Line Interface

Install the V2 package from the repository root into your Python 3.11+
environment with `python -m pip install -e .`. This installs the Python
dependencies declared in `pyproject.toml`, including NumPy and the pinned
`orffinder` adapter dependency. The Python 3.10 Conda configuration above
is for the legacy workflow; it is not a V2 environment specification.
BLAST+, HMMER, PfamScan, and their reference databases are separate
requirements and are not installed by this command.

The V2 CLI is separate from the legacy workflow below. Run the installed
`lverage` command or use `PYTHONPATH=src python -m lverage.cli` from a source
checkout. It requires Python 3.11+, local BLAST+ and PfamScan tools/data, and a
local protein database; full motif lookup uses the online JASPAR2024 service.
See the [V2 FASTA and CLI guide](docs/user_guide.rst) for input mapping,
portable examples (including local runs stopped after alignment), output
formats, selection behavior, and status codes. See the local BLAST database
instructions above for database setup.
`lverage` command or use `PYTHONPATH=src python -m lverage.cli` from the Lverage
source checkout. It requires Python 3.11+, local BLAST+ and PfamScan tools/data,
and a local protein database; full motif inference uses the JASPAR calculations
adapted into Lverage and local JASPAR 2026 files, with no online requests.
Users need the external reference data directory and ``JASPAR2026.sqlite``,
but not the upstream checkout or a separate interpreter. HMMER/BLASTP remain
external executables. An alignment-only stop does not require JASPAR resources.
The cutoff is applied to the inference BLAST E-value; equivalence to the former
REST E-value has not been established, and identical motif results are not
claimed. See the [V2 FASTA and CLI guide](docs/user_guide.rst) for provenance,
local-data requirements, portable examples, output fields, selection behavior,
and status codes. See the local BLAST database instructions above for database
setup.

## Legacy Command-Line Workflow

Expand Down
1 change: 1 addition & 0 deletions conda_lverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ dependencies:
- idna==3.7
- itsdangerous==2.2.0
- msgspec==0.18.6
- numpy>=1.24
- pytz==2024.1
- six==1.16.0
- urllib3==2.2.1
Expand Down
35 changes: 26 additions & 9 deletions docs/api/jaspar.rst
Original file line number Diff line number Diff line change
@@ -1,20 +1,37 @@
JASPAR
======

The adapter uses the current ``jaspar.elixir.no`` REST API. Network requests
use finite timeouts, species availability is loaded lazily, and valid no-hit
responses return an empty list. Service, protocol, and malformed-response
failures are raised to the caller.
The V2 adapter implements the JASPAR profile-inference calculations within
Lverage, adapted from source revision
``ac4024148f1f140c7370d7b13a70556b4d13f7a3`` of
``amathelier/JASPAR-inference-tool``. It uses normal package imports and local
JASPAR reference data; no upstream checkout, Git history, CLI entry point, or
separate inference interpreter is required. Runtime inference makes no online
requests. Exact-version motif matrices and annotations are resolved from a local
``JASPAR2026.sqlite`` database. The algorithm applies the upstream HMMER domain
thresholds and overlap selection, BLASTP/Rost filter, insertion removal,
positional identity/BLOSUM62 scoring, and Cis-BP HSim sequence-identity or
similarity-regression decisions.

Inference uses a window of at most 2,000 ortholog residues while retaining the
complete ortholog domain. Accepted hits are sorted by inference E-value, checked
against every species listed in their motif details, and limited only after
species and threshold filtering.
The adapter filters by the upstream BLAST E-value, exact matrix species
taxonomy IDs, and then the configured hit limit.
The configured E-value cutoff applies specifically to the local inference
tool's BLAST E-value. Its equivalence to the former REST endpoint's ``evalue``
field has not been established; identical motif results between the services
are not claimed. Exact matrix versions returned by the fork are retained,
subject to species filtering, the E-value cutoff, and the hit limit.

The input is the unchanged ortholog-protein window of at most 2,000 residues
centered on the complete supporting domain. The inference BLAST E-value and DBD
identity are retained independently from Lverage BLAST metrics and Lverage's
global-alignment identity. Exact versioned matrix identifiers are required;
missing identifiers or local annotations are errors rather than an implicit
version substitution.

.. autoclass:: lverage.jaspar.JasparRecord
:members:
:show-inheritance:

.. autoclass:: lverage.jaspar.Jaspar2024MotifDB
.. autoclass:: lverage.jaspar.Jaspar2026MotifDB
:members:
:show-inheritance:
170 changes: 129 additions & 41 deletions docs/user_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,12 +19,39 @@ it does not construct adapters, inspect external tools, or access a network.

Prerequisites are the Python package dependencies, local BLAST+ executables
(``blastp`` and ``blastdbcmd``), a usable local protein database, and a local
PfamScan installation/database. JASPAR2024 is queried through its online
service, so a full run that reaches motif lookup requires network access. A
local run stopped after alignment does not contact JASPAR and can run without
network access. The local BLAST database must support complete protein
retrieval with ``blastdbcmd``. See the README's V2 local BLAST database
instructions for building one.
PfamScan installation/database. Local motif inference uses the JASPAR profile
inference calculations adapted into Lverage from
``amathelier/JASPAR-inference-tool`` and local JASPAR 2026 reference data; it
makes no online requests. Users do not need that source checkout, its Git
history, an upstream entry point, or a separate Python environment. A run
stopped after alignment does not require or inspect JASPAR data. The local BLAST
database must support complete protein retrieval with ``blastdbcmd``. See the
README's V2 local BLAST database instructions for building one.

Configure the external inference data directory and local
``JASPAR2026.sqlite`` using CLI options or the ``LVERAGE_JASPAR_DATA_DIR`` and
``LVERAGE_JASPAR_SQLITE_DB`` environment variables. The implementation records
upstream source revision ``ac4024148f1f140c7370d7b13a70556b4d13f7a3`` and
version ``2026.1.0`` as algorithm provenance; neither is a runtime checkout
requirement. Keep reference data, databases, and temporary files outside the
repository. Configure BLASTP through ``--blastp`` and HMMER through ``--hmmscan``
and ``--hmmalign`` (or their executable names on ``PATH``); those external
programs are invoked directly with argument lists. The Python inference
implementation imports Biopython and NumPy from the normal Lverage environment.
The configured data directory is a standalone data tree, not the upstream
source checkout. If reusing its downloaded reference files, stage the data
outside the checkout; Lverage does not read the upstream source tree at runtime.

The adapted routines correspond to the upstream sequence-level inference flow:
load the per-taxon Pfam/profile/UniProt and Cis-BP models; detect and align
query Pfam domains with HMMER; search the local taxon BLAST databases and apply
the upstream Rost identity curve; remove HMM alignment insertions; calculate
the upstream positional identity and BLOSUM62 features; and apply the Cis-BP
identity or similarity-regression ``HSim`` decision before returning associated
JASPAR profiles. The upstream source revision is provenance only. The external
data directory contains the precomputed models, domain mappings, and BLAST
databases generated for the JASPAR 2026 profile set; the separate SQLite file
supplies exact-version matrices and annotations.

Create a JSON species-name-to-NCBI-tax-ID map. Species names must match the
bracketed names in BLAST descriptions (case-insensitive), and tax IDs must be
Expand All @@ -41,34 +68,59 @@ Example using portable placeholder paths:

.. code-block:: console

lverage --input-dir ./inputs --blast-db ./db/proteins \
--species-map ./config/species-taxids.json --pfam-db ./db/pfam \
--output-dir ./results
lverage --input-dir ./inputs --blast-db /path/to/local-data/blast/proteins \
--species-map ./config/species-taxids.json \
--pfam-db /path/to/local-data/pfam --output-dir /path/outside/repository/results \
--jaspar-data-dir /path/to/local-data/jaspar-inference/files \
--jaspar-sqlite-db /path/to/local-data/jaspar/JASPAR2026.sqlite \
--blastp /path/to/blastp --hmmscan /path/to/hmmscan \
--hmmalign /path/to/hmmalign

For a local pre-JASPAR run that stops after domain alignment:

.. code-block:: console

lverage --input-dir ./inputs --blast-db ./db/proteins \
--species-map ./config/species-taxids.json --pfam-db ./db/pfam \
--output-dir ./alignment-results --stop-after alignment
lverage --input-dir ./inputs --blast-db /path/to/local-data/blast/proteins \
--species-map ./config/species-taxids.json \
--pfam-db /path/to/local-data/pfam \
--output-dir /path/outside/repository/alignment-results \
--stop-after alignment

Required options are ``--input-dir``, ``--blast-db``, ``--species-map``,
``--pfam-db``, and ``--output-dir``. The output directory must be new. It
contains three files: ``results.tsv`` contains typed pipeline evidence,
contains three files: ``results.csv`` contains typed pipeline evidence,
``orfs.fasta`` stores each unique candidate protein sequence once, and
``metadata.json`` records run settings and summaries. Optional executable
overrides are ``--blastp`` (``blastp``),
``--blastdbcmd`` (``blastdbcmd``), and ``--pfamscan`` (``pfam_scan.pl``).
The adapted motif inference also accepts ``--hmmscan`` and ``--hmmalign``
overrides.
``--identity-threshold`` defaults to ``0.7``. Repeat ``--valid-pfam`` to
restrict accepted domains; if it is omitted, every detected Pfam domain is
accepted. The pipeline selects at most one query ORF per gene: candidates from
the input fragments are deduplicated and sorted longest-first, then the first
candidate with an accepted domain is selected for downstream analysis.
JASPAR options default to 10 hits and an inference E-value threshold of
``1e-6``. Other component options retain their V2 defaults: ORF minimum length
JASPAR options default to 10 hits and a maximum inference-tool BLAST E-value of
``1e-6``; this cutoff applies to the BLAST E-value reported by the local
inference tool. Equivalence between that field and the former REST service's
``evalue`` field has not been established, so equivalent cutoff behavior or
identical motif results must not be assumed. The adapter retains all exact
versioned matrix IDs returned by the inference before applying the species
filter, this cutoff, and the configured hit limit. Set data paths using
``--jaspar-data-dir`` and ``--jaspar-sqlite-db``, or the corresponding
``LVERAGE_JASPAR_DATA_DIR`` and ``LVERAGE_JASPAR_SQLITE_DB`` environment
variables. Configure ``blastp``, ``hmmscan`` and ``hmmalign`` through their
executable options. The data directory must contain the locally stored JASPAR
profile-inference files, Pfam HMMs and pressed indexes, Cis-BP models, and the
per-taxon JSON and BLAST-database files used by the adapted algorithm. The
SQLite database can be the local JASPAR 2026 database distributed with
pyJASPAR; it supplies exact-version PFMs, TF classes, and species taxonomy
IDs. Other component options retain their V2 defaults: ORF minimum length
75 nucleotides; BLAST E-value ``1e-6``, at most 20 hits, and the standard
description filters.
description filters. By default, descriptions containing ``hypothetical``,
``unnamed``, ``uncharacterized``, ``unknown``, or ``partial`` are excluded
case-insensitively. An ``isoform`` label alone does not exclude a protein:
isoform-labelled proteins are retained unless another exclusion applies.

The CLI discovers ``.fa``, ``.fasta``, and ``.fna`` files nonrecursively,
case-insensitively, in filename order. The filename stem labels results and
Expand All @@ -78,41 +130,76 @@ fragments, and it cannot determine from record headers whether they really
belong to the same gene. Empty files/records and unsupported DNA symbols are
rejected before external-tool adapters are constructed.

The TSV has explicit ``Row Type``, ``Stage``, ``Status``, ``Error``, and
``Outcome Detail`` fields. ``Error`` is reserved for execution failures;
biological reasons and not-reached explanations use ``Outcome Detail``.
Row types include ``input``, ``orf``, ``query_domain``, ``ortholog``,
``ortholog_retrieval``, ``ortholog_domain``, ``domain_comparison``, ``motif``,
and ``gene_error``.
``stage_status`` rows record biological no-hits, below-threshold completion,
intentional stops, and stages not reached.
``ortholog_retrieval`` rows retain failed retrieval attempts and skipped hits,
including available native hit IDs/accessions, the attempted identifier, and a
diagnostic reason. A BLAST no-hit status is distinct from retrieval failure.
``results.tsv`` has multiple evidence row types; a row is not necessarily a
separate ORF or a final motif result. For example, an ORF may have candidate,
domain, selection, ortholog, comparison, and stage-status rows, while a motif
row is emitted only for a returned motif. Each motif has one row identifying
its input gene, query ORF, ortholog, and matching query/ortholog domains.
Multiple hits, domain occurrences, and motifs remain distinct. ORF rows retain
candidates even when no accepted domain exists, and an input row records a
gene with no ORF. ``orfs.fasta`` stores each unique discovered candidate
protein once under a deterministic full-sequence SHA-256 identifier
referenced by the TSV; gene associations and selection status remain in the
TSV. Candidate export does not run downstream analysis on unselected
candidates.
``results.csv`` is a comma-separated UTF-8 table ordered by gene, ORF candidate,
query Pfam, BLAST hit, hit Pfam, global alignment, and motif hit. Each row is a
self-contained evidence record: every applicable gene, ORF, query-Pfam,
BLAST-hit, hit-Pfam, and alignment field is repeated on each descendant result
row. Columns are named for their stage (for example, ``Query Pfam ID``,
``BLAST Accession``, and ``Global Alignment Identity Ratio``). Candidates are
listed in longest-first examination order. The ORF examination and selection
columns identify examined candidates, candidates with no accepted domain, and
the selected candidate. Unexamined candidates still show their order, ID, and
length; their examination, selection, and downstream cells remain blank. All
unique ORFs are represented.

Within the selected ORF, each query-domain occurrence has its own occurrence
number and retains its Pfam ID, name, and zero-based, end-exclusive amino-acid
coordinates. BLAST hits are grouped beneath query Pfams and retain accession,
native hit ID, description, species, E-value, identity, and query coverage.
Occurrence and hit-group numbers, BLAST descriptive fields, and supporting
alignment fields are repeated on every associated motif row. When a hit is
evaluated against multiple query-Pfam occurrences, each row identifies the
specific query/hit-domain occurrence pair. Hit Pfam occurrences remain visible
even when no query Pfam matches them. Domain comparisons are associated by the
actual query-domain and hit-domain occurrences, not by protein name. Each
computed global domain alignment has an identity ratio, threshold, and pass/fail
decision. There is one row per motif hit per supporting domain comparison;
multiple motifs and multiple comparisons therefore remain distinct rows.

Rows remain grouped contiguously for convenient reading, but their meaning does
not depend on carrying values forward from preceding rows. Do not forward-fill
blank cells: a field is blank only when the information is unavailable or its
stage was not reached. No-hit outcomes and execution errors are recorded in the
relevant stage status, outcome, and error columns. Sorting or filtering may
change presentation order, but each row retains its own gene, candidate, hit,
domain-occurrence, alignment, and motif associations.
BLAST no hits and failed sequence retrieval are distinct outcomes. An empty
Pfam result, a domain pair below threshold, no matching Pfam, and a motif
search with no returned motifs are also distinguishable.

``orfs.fasta`` stores each unique discovered candidate protein once under a
deterministic full-sequence SHA-256 identifier referenced by the table; gene
associations and selection status are retained in the grouped rows. Candidate export does not run downstream analysis on
unselected candidates.

BLAST identity is the identity ratio from the HSP with the lowest E-value and
is distinct from domain-pair identity, which is the exact-match ratio across
every column of a global amino-acid alignment of matching Pfam domains
(including gaps).
The TSV reports BLAST identity and query coverage as ratios and percentages,
The CSV reports BLAST identity and query coverage as ratios and percentages,
and records every computed domain-pair identity, including values below the
threshold. Domain start and end coordinates in exported rows are zero-based,
end-exclusive amino-acid positions. The domain-alignment identity score is
exported as a ratio and percentage, but the aligned sequences themselves are
not currently exported.

Motif rows separately record the upstream inference tool's BLAST E-value,
query/TF regions, TF name, and upstream DBD identity ratio. This upstream DBD
identity is not the Lverage global-alignment identity and is not substituted
for it. The local BLAST E-value is not established to be equivalent to the old
REST inference E-value; identical motif results between those services are not
claimed. The input sent to local inference is the same ortholog-protein window
previously sent to the REST endpoint: at most 2,000 amino acids centered on the
supporting ortholog domain, retaining the complete domain. It is not replaced
by the query protein or by a domain-only sequence. Matrix IDs must be
versioned, and each ID is resolved exactly in the configured local SQLite
database; there is no version fallback. Motif species are filtered against
exact local NCBI taxonomy IDs, not broad taxonomic groups.

Metadata schema version 2 identifies this layout with
``results_format="grouped-csv"``. It replaces the former event-row TSV schema. Fields containing commas,
quotation marks, or line breaks are quoted using standard CSV rules.

``metadata.json`` records run timestamps, settings, input paths/checksums,
sequence-ID conventions, known tool versions (unknown versions are reported as
``unknown``), and completion/failure summaries. Results and metadata are
Expand All @@ -123,7 +210,8 @@ gene and processing continues to other genes, retaining completed evidence if
a later stage fails.

Use ``--stop-after alignment`` for pre-JASPAR validation. This intentional stop
does not call JASPAR and is not a failure. Exit status ``0`` means completed or
does not call JASPAR and does not require the JASPAR options or data; it is not
a failure. Exit status ``0`` means completed or
intentionally stopped, including biological no-hits; ``2`` means preflight
validation failure; ``1`` means one or more execution failures despite
preserved partial results. Diagnostics go to standard error.
Expand Down
Loading
Loading