Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 41 additions & 80 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,101 +38,62 @@ python3 -m pip install -r requirements.txt

## Local BLAST Database Setup for V2

Lverage V2 can search a local protein database with BLAST+. BLAST+ is an
external requirement and is not installed with the Python package. Confirm that
both required programs are available:
Lverage finds orthologs by running BLAST against a local protein database that
you build once. The database holds the proteins of the species you want
orthologs from. If you study species A and want orthologs from B, C, and D,
the database contains the proteomes of B, C, and D; A is the query, not part
of the database.

```bash
blastp -version
blastdbcmd -version
```

The source FASTA must contain complete protein sequences. Each record must have
a unique sequence identifier immediately after `>`, followed by a description
containing the scientific name in square brackets. For example:

```text
>NP_000001.1 Homeobox protein [Homo sapiens]
MPEPTIDESEQUENCE
>NP_000002.1 Homeobox protein [Mus musculus]
MSEQUENCE
```
### 1. Install the tools

Build the protein database with `-parse_seqids`. This option is required because
Lverage retrieves the complete sequence of each BLAST hit with `blastdbcmd`.
The value passed to `-out` is a database prefix, not a directory or an
individual database file. See the
[NCBI BLAST+ documentation](https://www.ncbi.nlm.nih.gov/books/NBK52637/table/blast_setup_pc.T.programs_and_utilities/)
for the specific-retrieval requirement.
BLAST+ and the NCBI datasets CLI are external requirements and are not
installed with the Python package. With conda:

```bash
mkdir -p databases

makeblastdb \
-in proteins.fasta \
-dbtype prot \
-parse_seqids \
-out databases/lverage_proteins
conda install -c conda-forge -c bioconda blast ncbi-datasets-cli unzip
```

Do not omit `-parse_seqids`. A database without parsed sequence identifiers can
be searched by `blastp`, but Lverage will be unable to retrieve complete subject
proteins from it.
### 2. Build the database

Validate both the database and accession retrieval before using it:
Look up the NCBI taxonomy ID of each ortholog species at
<https://www.ncbi.nlm.nih.gov/taxonomy> and pass them to the script. This
example builds a human, mouse, and fruit fly database:

```bash
blastdbcmd -db databases/lverage_proteins -info

blastdbcmd \
-db databases/lverage_proteins \
-entry NP_000001.1 \
-outfmt '%f'
scripts/make_blast_db.sh -t 9606,10090,7227 -o databases/orthologs
```

The second command should print the expected FASTA record. If it reports that
the database contains no accession information, rebuild the database from its
source FASTA with `-parse_seqids`. Use a new output prefix rather than
overwriting a database that may be in use.
The script downloads each species' RefSeq reference proteome, combines them,
builds the database with `-parse_seqids` (Lverage needs this to retrieve each
hit's complete protein), and checks that retrieval works. It finishes by
printing the species names found in the headers.

If a taxonomy-enabled source database such as NCBI `nr` is already installed,
a smaller FASTA can first be extracted. The following example selects human and
mouse proteins:
If you already have a protein FASTA, build from it with `-i proteins.fasta`
instead. Headers must carry a unique identifier and the species in brackets:

```bash
blastdbcmd \
-db /path/to/nr \
-taxids 9606,10090 \
-target_only \
-outfmt '%f' \
-out human_mouse.fasta

makeblastdb \
-in human_mouse.fasta \
-dbtype prot \
-parse_seqids \
-out databases/human_mouse_nr
```text
>NP_000001.1 Homeobox protein [Homo sapiens]
MPEPTIDESEQUENCE
```

Taxonomic extraction requires the taxonomy files associated with the source
BLAST database. Check the resulting FASTA headers and sequence count before
building the new database.

Pass the database prefix and a mapping of the scientific names used in its
FASTA headers to `LocalBlastSearcher`:

```python
from lverage.blast import LocalBlastSearcher


ortholog_searcher = LocalBlastSearcher(
database_path="databases/human_mouse_nr",
species_map={
"Homo sapiens": 9606,
"Mus musculus": 10090,
},
)
```
If you have a taxonomy-enabled BLAST database such as NCBI `nr` on disk, add
`-n /path/to/nr` to extract the species from it rather than downloading. Run
`scripts/make_blast_db.sh -h` for all options.

### 3. Keep the output for Lverage

Note the database prefix you chose and the species names the script prints
at the end. Lverage needs both: the prefix says where to search, and the
species names, mapped to their taxonomy IDs, tell it which hits count as
orthologs. The names must match the headers exactly, including any strain
suffix such as `Saccharomyces cerevisiae S288C`. Bacterial proteomes label
many records with only a genus or group name, such as `Bacillus`, so map
every printed name you want to count, not just the species. Two species of
the same genus cannot be told apart by such a label, so expect their shared
proteins to be attributed to whichever taxonomy ID you map that name to. See
the
[API documentation](docs/api/index.rst) for how to pass them to the V2
pipeline.

## How to Use
We warn against moving any file within the directory anywhere else as this will create errors. If you wish to access from other places, we suggest appending the directory to your PATH environment variable, creating an alias, or creating a shortcut.
Expand Down
131 changes: 131 additions & 0 deletions scripts/make_blast_db.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
#!/usr/bin/env bash
Comment thread
AnthonyGarza124 marked this conversation as resolved.
Resolved
# Build the local protein BLAST database that Lverage V2 searches for orthologs.
#
# Give it the NCBI taxonomy IDs of the species you want orthologs from. It
# downloads each species' RefSeq reference proteome from NCBI, combines them,
# builds a BLAST database with -parse_seqids (required: Lverage fetches each
# hit's complete protein with "blastdbcmd -entry"), and verifies retrieval.
#
# Usage:
# scripts/make_blast_db.sh -t 9606,10090,7227 -o databases/orthologs
#
# Other sources:
# -i FILE build from a protein FASTA you already have, instead of
# downloading. Headers must carry the species in brackets,
# e.g. ">NP_000001.1 Homeobox protein [Homo sapiens]".
# -n DB extract -t TAXIDS from an existing taxonomy-enabled BLAST
# database (such as NCBI nr) instead of downloading.
#
# Options:
# -t TAXIDS comma-separated NCBI taxonomy IDs (find them at
# https://www.ncbi.nlm.nih.gov/taxonomy)
# -o PREFIX output database prefix, e.g. databases/orthologs (required)
# -h show this help
#
# Requires BLAST+ (makeblastdb, blastdbcmd) on PATH. Downloading also requires
# the NCBI datasets CLI and unzip. With conda:
# conda install -c conda-forge -c bioconda blast ncbi-datasets-cli unzip

set -euo pipefail

usage() { sed -n '2,/^set -euo/p' "$0" | sed '$d' | sed 's/^# \{0,1\}//'; }
die() { echo "error: $*" >&2; exit 1; }
need() { command -v "$1" >/dev/null || die "$1 not found on PATH. $2"; }

input=""; prefix=""; source_db=""; taxids=""
while getopts "i:o:n:t:h" opt; do
case "$opt" in
i) input="$OPTARG" ;;
o) prefix="$OPTARG" ;;
n) source_db="$OPTARG" ;;
t) taxids="$OPTARG" ;;
h) usage; exit 0 ;;
*) usage >&2; exit 2 ;;
esac
done

[[ -n "$prefix" ]] || die "-o PREFIX is required"
need makeblastdb "Install BLAST+ first."
need blastdbcmd "Install BLAST+ first."
mkdir -p "$(dirname "$prefix")"

if [[ -n "$input" && ( -n "$source_db" || -n "$taxids" ) ]]; then
die "use -i FILE on its own, or -t TAXIDS (optionally with -n DB)"
fi

if [[ -n "$source_db" ]]; then
[[ -n "$taxids" ]] || die "-t TAXIDS is required with -n"
input="${prefix}.fasta"
echo "Extracting taxids $taxids from $source_db into $input"
# -target_only keeps only deflines belonging to the requested taxa, so the
# species in each header is one Lverage's species map will recognise.
blastdbcmd -db "$source_db" -taxids "$taxids" -target_only -outfmt '%f' -out "$input"
Comment thread
Copilot marked this conversation as resolved.
Resolved

elif [[ -n "$taxids" ]]; then
need datasets "Install the NCBI datasets CLI (conda: bioconda::ncbi-datasets-cli)."
need unzip "Install unzip."
input="${prefix}.fasta"
: > "$input"
workdir=$(mktemp -d "${TMPDIR:-/tmp}/make_blast_db.XXXXXX")
trap 'rm -rf "$workdir"' EXIT
IFS=',' read -r -a taxid_list <<< "$taxids"
for taxid in "${taxid_list[@]}"; do
taxid="${taxid//[[:space:]]/}"
[[ "$taxid" =~ ^[0-9]+$ ]] || die "taxid is not a number: '$taxid'"
echo "Downloading RefSeq reference proteome for taxid $taxid"
datasets download genome taxon "$taxid" --reference --include protein --no-progressbar \
--filename "$workdir/$taxid.zip" \
|| die "download failed for taxid $taxid (no RefSeq reference genome, or no network?)"
unzip -q -o "$workdir/$taxid.zip" -d "$workdir/$taxid"
faa=$(find "$workdir/$taxid" -name 'protein.faa' | head -n 1)
[[ -n "$faa" ]] || die "no protein.faa in the download for taxid $taxid; its reference genome may lack annotation"
count=$(grep -c '^>' "$faa")
species=$(grep -m1 '^>' "$faa" | sed -n 's/.*\[\(.*\)\].*/\1/p')
echo " $count proteins, species in headers: ${species:-NONE}"
[[ -n "$species" ]] || die "headers for taxid $taxid have no bracketed species name; Lverage cannot use them"
# RefSeq reuses WP_ accessions for identical proteins across related
# bacteria, and makeblastdb -parse_seqids refuses duplicate ids. When a
# record's id was already added by an earlier species, suffix it with
# this taxid; the description and species are kept as they are.
{ grep '^>' "$input" || true; } | sed 's/^>//; s/[[:space:]].*//' > "$workdir/seen_ids"
awk -v taxid="$taxid" -v seen="$workdir/seen_ids" '
BEGIN { while ((getline id < seen) > 0) have[id] = 1 }
/^>/ { id = substr($1, 2)
if (id in have) {
new_id = id "_" taxid; n = 1
while (new_id in have) new_id = id "_" taxid "_" (++n)
$1 = ">" new_id; id = new_id; renamed++
}
have[id] = 1
print; next }
{ print }
END { if (renamed) printf(" %d ids collided with records already added and were made unique with a _%s suffix\n", renamed, taxid) > "/dev/stderr" }
' "$faa" >> "$input"
done
echo "Combined proteome written to $input"
fi

[[ -n "$input" ]] || die "give -t TAXIDS to download, or -i FILE to build from your own FASTA"
[[ -s "$input" ]] || die "input FASTA is missing or empty: $input"

dup_ids=$(grep '^>' "$input" | sed 's/^>//; s/[[:space:]].*//' | sort | uniq -d | sed -n '1,5p')
if [[ -n "$dup_ids" ]]; then
die "duplicate record identifiers in $input (makeblastdb -parse_seqids requires unique ids); first few: $(echo "$dup_ids" | tr '\n' ' ')"
fi

echo "Building $prefix"
makeblastdb -in "$input" -dbtype prot -parse_seqids -max_file_sz 3GB -out "$prefix" >/dev/null

# Verify that accession retrieval works, using the first record in the input.
first_id=$(grep -m1 '^>' "$input" | sed 's/^>//; s/[[:space:]].*//')
[[ -n "$first_id" ]] || die "could not read a record identifier from $input"
blastdbcmd -db "$prefix" -entry "$first_id" -outfmt '%a' >/dev/null \
|| die "built $prefix but could not retrieve $first_id from it; Lverage will not be able to use this database"

echo
blastdbcmd -db "$prefix" -info | sed -n '1,2p'
echo
echo "Species found in headers (use these exact names as species_map keys):"
grep '^>' "$input" | sed -n 's/.*\[\([^]]*\)\].*/\1/p' | sort | uniq -c | sort -rn
echo
echo "Database ready. Pass database_path=\"$prefix\" to LocalBlastSearcher."
Loading