From 6780f85eb3ce739e5978675f27982126d11279c1 Mon Sep 17 00:00:00 2001 From: Anthony Garza Date: Wed, 14 Oct 2026 18:06:27 -0400 Subject: [PATCH 1/5] Add a script that builds the local BLAST ortholog database from taxonomy IDs A new user starts with nothing. Given the NCBI taxonomy IDs of the species they want orthologs from, scripts/make_blast_db.sh downloads each RefSeq reference proteome with the NCBI datasets CLI, combines them, builds a BLAST database with -parse_seqids (LocalBlastSearcher fetches each hit's complete protein with blastdbcmd -entry, which fails without it), verifies retrieval, and prints the species names to use as species_map keys. It can also build from a protein FASTA the user already has (-i), or extract taxa from an existing taxonomy-enabled database such as nr (-n). Rewrite the README section as the three steps a user actually follows: install the tools, run the script, point LocalBlastSearcher at the result. --- README.md | 99 ++++++++++++----------------------- scripts/make_blast_db.sh | 109 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+), 67 deletions(-) create mode 100644 scripts/make_blast_db.sh diff --git a/README.md b/README.md index c4b7410..7b67904 100644 --- a/README.md +++ b/README.md @@ -38,98 +38,63 @@ python3 -m pip install -r requirements.txt ## Local BLAST Database Setup for V2 -Lverage V2 can search a local protein database with BLAST+. BLAST+ is an -external requirement and is not installed with the Python package. Confirm that -both required programs are available: +Lverage finds orthologs by running BLAST against a local protein database that +you build once. The database holds the proteins of the species you want +orthologs from. If you study species A and want orthologs from B, C, and D, +the database contains the proteomes of B, C, and D; A is the query, not part +of the database. -```bash -blastp -version -blastdbcmd -version -``` - -The source FASTA must contain complete protein sequences. Each record must have -a unique sequence identifier immediately after `>`, followed by a description -containing the scientific name in square brackets. For example: - -```text ->NP_000001.1 Homeobox protein [Homo sapiens] -MPEPTIDESEQUENCE ->NP_000002.1 Homeobox protein [Mus musculus] -MSEQUENCE -``` +### 1. Install the tools -Build the protein database with `-parse_seqids`. This option is required because -Lverage retrieves the complete sequence of each BLAST hit with `blastdbcmd`. -The value passed to `-out` is a database prefix, not a directory or an -individual database file. See the -[NCBI BLAST+ documentation](https://www.ncbi.nlm.nih.gov/books/NBK52637/table/blast_setup_pc.T.programs_and_utilities/) -for the specific-retrieval requirement. +BLAST+ and the NCBI datasets CLI are external requirements and are not +installed with the Python package. With conda: ```bash -mkdir -p databases - -makeblastdb \ - -in proteins.fasta \ - -dbtype prot \ - -parse_seqids \ - -out databases/lverage_proteins +conda install -c conda-forge -c bioconda blast ncbi-datasets-cli unzip ``` -Do not omit `-parse_seqids`. A database without parsed sequence identifiers can -be searched by `blastp`, but Lverage will be unable to retrieve complete subject -proteins from it. +### 2. Build the database -Validate both the database and accession retrieval before using it: +Look up the NCBI taxonomy ID of each ortholog species at + and pass them to the script. This +example builds a human, mouse, and fruit fly database: ```bash -blastdbcmd -db databases/lverage_proteins -info - -blastdbcmd \ - -db databases/lverage_proteins \ - -entry NP_000001.1 \ - -outfmt '%f' +scripts/make_blast_db.sh -t 9606,10090,7227 -o databases/orthologs ``` -The second command should print the expected FASTA record. If it reports that -the database contains no accession information, rebuild the database from its -source FASTA with `-parse_seqids`. Use a new output prefix rather than -overwriting a database that may be in use. +The script downloads each species' RefSeq reference proteome, combines them, +builds the database with `-parse_seqids` (Lverage needs this to retrieve each +hit's complete protein), and checks that retrieval works. It finishes by +printing the species names found in the headers. -If a taxonomy-enabled source database such as NCBI `nr` is already installed, -a smaller FASTA can first be extracted. The following example selects human and -mouse proteins: +If you already have a protein FASTA, build from it with `-i proteins.fasta` +instead. Headers must carry a unique identifier and the species in brackets: -```bash -blastdbcmd \ - -db /path/to/nr \ - -taxids 9606,10090 \ - -target_only \ - -outfmt '%f' \ - -out human_mouse.fasta - -makeblastdb \ - -in human_mouse.fasta \ - -dbtype prot \ - -parse_seqids \ - -out databases/human_mouse_nr +```text +>NP_000001.1 Homeobox protein [Homo sapiens] +MPEPTIDESEQUENCE ``` -Taxonomic extraction requires the taxonomy files associated with the source -BLAST database. Check the resulting FASTA headers and sequence count before -building the new database. +If you have a taxonomy-enabled BLAST database such as NCBI `nr` on disk, add +`-n /path/to/nr` to extract the species from it rather than downloading. Run +`scripts/make_blast_db.sh -h` for all options. + +### 3. Point Lverage at it -Pass the database prefix and a mapping of the scientific names used in its -FASTA headers to `LocalBlastSearcher`: +Pass the database prefix and the species names printed by the script, mapped +to their taxonomy IDs, to `LocalBlastSearcher`: ```python from lverage.blast import LocalBlastSearcher ortholog_searcher = LocalBlastSearcher( - database_path="databases/human_mouse_nr", + database_path="databases/orthologs", species_map={ "Homo sapiens": 9606, "Mus musculus": 10090, + "Drosophila melanogaster": 7227, }, ) ``` diff --git a/scripts/make_blast_db.sh b/scripts/make_blast_db.sh new file mode 100644 index 0000000..62aeb07 --- /dev/null +++ b/scripts/make_blast_db.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# Build the local protein BLAST database that Lverage V2 searches for orthologs. +# +# Give it the NCBI taxonomy IDs of the species you want orthologs from. It +# downloads each species' RefSeq reference proteome from NCBI, combines them, +# builds a BLAST database with -parse_seqids (required: Lverage fetches each +# hit's complete protein with "blastdbcmd -entry"), and verifies retrieval. +# +# Usage: +# scripts/make_blast_db.sh -t 9606,10090,7227 -o databases/orthologs +# +# Other sources: +# -i FILE build from a protein FASTA you already have, instead of +# downloading. Headers must carry the species in brackets, +# e.g. ">NP_000001.1 Homeobox protein [Homo sapiens]". +# -n DB extract -t TAXIDS from an existing taxonomy-enabled BLAST +# database (such as NCBI nr) instead of downloading. +# +# Options: +# -t TAXIDS comma-separated NCBI taxonomy IDs (find them at +# https://www.ncbi.nlm.nih.gov/taxonomy) +# -o PREFIX output database prefix, e.g. databases/orthologs (required) +# -h show this help +# +# Requires BLAST+ (makeblastdb, blastdbcmd) on PATH. Downloading also requires +# the NCBI datasets CLI and unzip. With conda: +# conda install -c conda-forge -c bioconda blast ncbi-datasets-cli unzip + +set -euo pipefail + +usage() { sed -n '2,/^set -euo/p' "$0" | sed '$d' | sed 's/^# \{0,1\}//'; } +die() { echo "error: $*" >&2; exit 1; } +need() { command -v "$1" >/dev/null || die "$1 not found on PATH. $2"; } + +input=""; prefix=""; source_db=""; taxids="" +while getopts "i:o:n:t:h" opt; do + case "$opt" in + i) input="$OPTARG" ;; + o) prefix="$OPTARG" ;; + n) source_db="$OPTARG" ;; + t) taxids="$OPTARG" ;; + h) usage; exit 0 ;; + *) usage >&2; exit 2 ;; + esac +done + +[[ -n "$prefix" ]] || die "-o PREFIX is required" +need makeblastdb "Install BLAST+ first." +need blastdbcmd "Install BLAST+ first." +mkdir -p "$(dirname "$prefix")" + +if [[ -n "$input" && ( -n "$source_db" || -n "$taxids" ) ]]; then + die "use -i FILE on its own, or -t TAXIDS (optionally with -n DB)" +fi + +if [[ -n "$source_db" ]]; then + [[ -n "$taxids" ]] || die "-t TAXIDS is required with -n" + input="${prefix}.fasta" + echo "Extracting taxids $taxids from $source_db into $input" + # -target_only keeps only deflines belonging to the requested taxa, so the + # species in each header is one Lverage's species map will recognise. + blastdbcmd -db "$source_db" -taxids "$taxids" -target_only -outfmt '%f' -out "$input" + +elif [[ -n "$taxids" ]]; then + need datasets "Install the NCBI datasets CLI (conda: bioconda::ncbi-datasets-cli)." + need unzip "Install unzip." + input="${prefix}.fasta" + : > "$input" + workdir=$(mktemp -d "${TMPDIR:-/tmp}/make_blast_db.XXXXXX") + trap 'rm -rf "$workdir"' EXIT + IFS=',' read -r -a taxid_list <<< "$taxids" + for taxid in "${taxid_list[@]}"; do + taxid="${taxid//[[:space:]]/}" + [[ "$taxid" =~ ^[0-9]+$ ]] || die "taxid is not a number: '$taxid'" + echo "Downloading RefSeq reference proteome for taxid $taxid" + datasets download genome taxon "$taxid" --reference --include protein --no-progressbar \ + --filename "$workdir/$taxid.zip" \ + || die "download failed for taxid $taxid (no RefSeq reference genome, or no network?)" + unzip -q -o "$workdir/$taxid.zip" -d "$workdir/$taxid" + faa=$(find "$workdir/$taxid" -name 'protein.faa' | head -n 1) + [[ -n "$faa" ]] || die "no protein.faa in the download for taxid $taxid; its reference genome may lack annotation" + count=$(grep -c '^>' "$faa") + species=$(grep -m1 '^>' "$faa" | sed -n 's/.*\[\(.*\)\].*/\1/p') + echo " $count proteins, species in headers: ${species:-NONE}" + [[ -n "$species" ]] || die "headers for taxid $taxid have no bracketed species name; Lverage cannot use them" + cat "$faa" >> "$input" + done + echo "Combined proteome written to $input" +fi + +[[ -n "$input" ]] || die "give -t TAXIDS to download, or -i FILE to build from your own FASTA" +[[ -s "$input" ]] || die "input FASTA is missing or empty: $input" + +echo "Building $prefix" +makeblastdb -in "$input" -dbtype prot -parse_seqids -max_file_sz 3GB -out "$prefix" >/dev/null + +# Verify that accession retrieval works, using the first record in the input. +first_id=$(grep -m1 '^>' "$input" | sed 's/^>//; s/[[:space:]].*//') +[[ -n "$first_id" ]] || die "could not read a record identifier from $input" +blastdbcmd -db "$prefix" -entry "$first_id" -outfmt '%a' >/dev/null \ + || die "built $prefix but could not retrieve $first_id from it; Lverage will not be able to use this database" + +echo +blastdbcmd -db "$prefix" -info | head -2 +echo +echo "Species found in headers (use these exact names as species_map keys):" +grep '^>' "$input" | sed -n 's/.*\[\([^]]*\)\].*/\1/p' | sort | uniq -c | sort -rn | head -20 +echo +echo "Database ready. Pass database_path=\"$prefix\" to LocalBlastSearcher." From 233b14dcea59e475c1ba211a31e66a118157596a Mon Sep 17 00:00:00 2001 From: Anthony Garza Date: Wed, 14 Oct 2026 18:25:45 -0400 Subject: [PATCH 2/5] Drop the Python snippet from the BLAST database README section The README is for users; how to construct LocalBlastSearcher in code belongs in the API docs. Step 3 now just says what to keep from the script's output and why. --- README.md | 27 +++++++++------------------ 1 file changed, 9 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 7b67904..7e2cecc 100644 --- a/README.md +++ b/README.md @@ -80,24 +80,15 @@ If you have a taxonomy-enabled BLAST database such as NCBI `nr` on disk, add `-n /path/to/nr` to extract the species from it rather than downloading. Run `scripts/make_blast_db.sh -h` for all options. -### 3. Point Lverage at it - -Pass the database prefix and the species names printed by the script, mapped -to their taxonomy IDs, to `LocalBlastSearcher`: - -```python -from lverage.blast import LocalBlastSearcher - - -ortholog_searcher = LocalBlastSearcher( - database_path="databases/orthologs", - species_map={ - "Homo sapiens": 9606, - "Mus musculus": 10090, - "Drosophila melanogaster": 7227, - }, -) -``` +### 3. Keep the output for Lverage + +Note the database prefix you chose and the species names the script prints +at the end. Lverage needs both: the prefix says where to search, and the +species names, mapped to their taxonomy IDs, tell it which hits count as +orthologs. The names must match the headers exactly, including any strain +suffix such as `Saccharomyces cerevisiae S288C`. See the +[API documentation](docs/api/index.rst) for how to pass them to the V2 +pipeline. ## How to Use We warn against moving any file within the directory anywhere else as this will create errors. If you wish to access from other places, we suggest appending the directory to your PATH environment variable, creating an alias, or creating a shortcut. From efd6c87c4236c8df5bcb7c270de4ebaafd3a675a Mon Sep 17 00:00:00 2001 From: Anthony Garza Date: Thu, 15 Oct 2026 13:30:54 -0400 Subject: [PATCH 3/5] Mark make_blast_db.sh executable and print every species name The file was committed as 644 because this clone has core.filemode off, so the documented invocation failed with permission denied. Also drop the 20-line cap on the species report, since users copy species_map keys from it. --- scripts/make_blast_db.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) mode change 100644 => 100755 scripts/make_blast_db.sh diff --git a/scripts/make_blast_db.sh b/scripts/make_blast_db.sh old mode 100644 new mode 100755 index 62aeb07..272c39e --- a/scripts/make_blast_db.sh +++ b/scripts/make_blast_db.sh @@ -104,6 +104,6 @@ echo blastdbcmd -db "$prefix" -info | head -2 echo echo "Species found in headers (use these exact names as species_map keys):" -grep '^>' "$input" | sed -n 's/.*\[\([^]]*\)\].*/\1/p' | sort | uniq -c | sort -rn | head -20 +grep '^>' "$input" | sed -n 's/.*\[\([^]]*\)\].*/\1/p' | sort | uniq -c | sort -rn echo echo "Database ready. Pass database_path=\"$prefix\" to LocalBlastSearcher." From 785e70250050054a21d0a8f19884c1e52f7e374c Mon Sep 17 00:00:00 2001 From: Anthony Garza Date: Thu, 15 Oct 2026 13:58:10 -0400 Subject: [PATCH 4/5] Handle accessions shared between downloaded proteomes RefSeq reuses WP_ accessions for identical proteins across related bacteria, and makeblastdb -parse_seqids refuses duplicate ids (two Bacillus reference proteomes share 803). When a downloaded record's id was already added by an earlier species, suffix it with the taxid and keep the description; accessions are untouched otherwise. For -i input, fail early with the duplicate ids listed instead of makeblastdb's error. Also fix the final report: piping blastdbcmd -info into head raised SIGPIPE under pipefail, so the script exited before printing the species list and ready line. Use sed, which reads to the end. README: bacterial proteomes label many records with only a genus, so map every printed name. --- README.md | 4 +++- scripts/make_blast_db.sh | 21 +++++++++++++++++++-- 2 files changed, 22 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 7e2cecc..eba2b1d 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,9 @@ Note the database prefix you chose and the species names the script prints at the end. Lverage needs both: the prefix says where to search, and the species names, mapped to their taxonomy IDs, tell it which hits count as orthologs. The names must match the headers exactly, including any strain -suffix such as `Saccharomyces cerevisiae S288C`. See the +suffix such as `Saccharomyces cerevisiae S288C`. Bacterial proteomes label +many records with only a genus or group name, such as `Bacillus`, so map +every printed name you want to count, not just the species. See the [API documentation](docs/api/index.rst) for how to pass them to the V2 pipeline. diff --git a/scripts/make_blast_db.sh b/scripts/make_blast_db.sh index 272c39e..3da8254 100755 --- a/scripts/make_blast_db.sh +++ b/scripts/make_blast_db.sh @@ -83,7 +83,19 @@ elif [[ -n "$taxids" ]]; then species=$(grep -m1 '^>' "$faa" | sed -n 's/.*\[\(.*\)\].*/\1/p') echo " $count proteins, species in headers: ${species:-NONE}" [[ -n "$species" ]] || die "headers for taxid $taxid have no bracketed species name; Lverage cannot use them" - cat "$faa" >> "$input" + # RefSeq reuses WP_ accessions for identical proteins across related + # bacteria, and makeblastdb -parse_seqids refuses duplicate ids. When a + # record's id was already added by an earlier species, suffix it with + # this taxid; the description and species are kept as they are. + { grep '^>' "$input" || true; } | sed 's/^>//; s/[[:space:]].*//' > "$workdir/seen_ids" + awk -v taxid="$taxid" -v seen="$workdir/seen_ids" ' + BEGIN { while ((getline id < seen) > 0) have[id] = 1 } + /^>/ { id = substr($1, 2) + if (id in have) { $1 = ">" id "_" taxid; renamed++ } + print; next } + { print } + END { if (renamed) printf(" %d ids were already present from another species and were suffixed with _%s\n", renamed, taxid) > "/dev/stderr" } + ' "$faa" >> "$input" done echo "Combined proteome written to $input" fi @@ -91,6 +103,11 @@ fi [[ -n "$input" ]] || die "give -t TAXIDS to download, or -i FILE to build from your own FASTA" [[ -s "$input" ]] || die "input FASTA is missing or empty: $input" +dup_ids=$(grep '^>' "$input" | sed 's/^>//; s/[[:space:]].*//' | sort | uniq -d | sed -n '1,5p') +if [[ -n "$dup_ids" ]]; then + die "duplicate record identifiers in $input (makeblastdb -parse_seqids requires unique ids); first few: $(echo "$dup_ids" | tr '\n' ' ')" +fi + echo "Building $prefix" makeblastdb -in "$input" -dbtype prot -parse_seqids -max_file_sz 3GB -out "$prefix" >/dev/null @@ -101,7 +118,7 @@ blastdbcmd -db "$prefix" -entry "$first_id" -outfmt '%a' >/dev/null \ || die "built $prefix but could not retrieve $first_id from it; Lverage will not be able to use this database" echo -blastdbcmd -db "$prefix" -info | head -2 +blastdbcmd -db "$prefix" -info | sed -n '1,2p' echo echo "Species found in headers (use these exact names as species_map keys):" grep '^>' "$input" | sed -n 's/.*\[\([^]]*\)\].*/\1/p' | sort | uniq -c | sort -rn From fdbc5e752f7471171cf9a508c0bea6d20f23e211 Mon Sep 17 00:00:00 2001 From: Anthony Garza Date: Thu, 15 Oct 2026 14:15:29 -0400 Subject: [PATCH 5/5] Keep ids unique within a downloaded proteome too Track every id emitted so far, not only those from earlier species, and pick a collision-free suffix when a proteome repeats an accession or the suffixed name already exists. README: two species of the same genus share genus-level labels in RefSeq and cannot be told apart by species_map. --- README.md | 5 ++++- scripts/make_blast_db.sh | 9 +++++++-- 2 files changed, 11 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index eba2b1d..50f3d1c 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,10 @@ species names, mapped to their taxonomy IDs, tell it which hits count as orthologs. The names must match the headers exactly, including any strain suffix such as `Saccharomyces cerevisiae S288C`. Bacterial proteomes label many records with only a genus or group name, such as `Bacillus`, so map -every printed name you want to count, not just the species. See the +every printed name you want to count, not just the species. Two species of +the same genus cannot be told apart by such a label, so expect their shared +proteins to be attributed to whichever taxonomy ID you map that name to. See +the [API documentation](docs/api/index.rst) for how to pass them to the V2 pipeline. diff --git a/scripts/make_blast_db.sh b/scripts/make_blast_db.sh index 3da8254..5b4815b 100755 --- a/scripts/make_blast_db.sh +++ b/scripts/make_blast_db.sh @@ -91,10 +91,15 @@ elif [[ -n "$taxids" ]]; then awk -v taxid="$taxid" -v seen="$workdir/seen_ids" ' BEGIN { while ((getline id < seen) > 0) have[id] = 1 } /^>/ { id = substr($1, 2) - if (id in have) { $1 = ">" id "_" taxid; renamed++ } + if (id in have) { + new_id = id "_" taxid; n = 1 + while (new_id in have) new_id = id "_" taxid "_" (++n) + $1 = ">" new_id; id = new_id; renamed++ + } + have[id] = 1 print; next } { print } - END { if (renamed) printf(" %d ids were already present from another species and were suffixed with _%s\n", renamed, taxid) > "/dev/stderr" } + END { if (renamed) printf(" %d ids collided with records already added and were made unique with a _%s suffix\n", renamed, taxid) > "/dev/stderr" } ' "$faa" >> "$input" done echo "Combined proteome written to $input"