diff --git a/README.md b/README.md index c4b7410..50f3d1c 100644 --- a/README.md +++ b/README.md @@ -38,101 +38,62 @@ python3 -m pip install -r requirements.txt ## Local BLAST Database Setup for V2 -Lverage V2 can search a local protein database with BLAST+. BLAST+ is an -external requirement and is not installed with the Python package. Confirm that -both required programs are available: +Lverage finds orthologs by running BLAST against a local protein database that +you build once. The database holds the proteins of the species you want +orthologs from. If you study species A and want orthologs from B, C, and D, +the database contains the proteomes of B, C, and D; A is the query, not part +of the database. -```bash -blastp -version -blastdbcmd -version -``` - -The source FASTA must contain complete protein sequences. Each record must have -a unique sequence identifier immediately after `>`, followed by a description -containing the scientific name in square brackets. For example: - -```text ->NP_000001.1 Homeobox protein [Homo sapiens] -MPEPTIDESEQUENCE ->NP_000002.1 Homeobox protein [Mus musculus] -MSEQUENCE -``` +### 1. Install the tools -Build the protein database with `-parse_seqids`. This option is required because -Lverage retrieves the complete sequence of each BLAST hit with `blastdbcmd`. -The value passed to `-out` is a database prefix, not a directory or an -individual database file. See the -[NCBI BLAST+ documentation](https://www.ncbi.nlm.nih.gov/books/NBK52637/table/blast_setup_pc.T.programs_and_utilities/) -for the specific-retrieval requirement. +BLAST+ and the NCBI datasets CLI are external requirements and are not +installed with the Python package. With conda: ```bash -mkdir -p databases - -makeblastdb \ - -in proteins.fasta \ - -dbtype prot \ - -parse_seqids \ - -out databases/lverage_proteins +conda install -c conda-forge -c bioconda blast ncbi-datasets-cli unzip ``` -Do not omit `-parse_seqids`. A database without parsed sequence identifiers can -be searched by `blastp`, but Lverage will be unable to retrieve complete subject -proteins from it. +### 2. Build the database -Validate both the database and accession retrieval before using it: +Look up the NCBI taxonomy ID of each ortholog species at + and pass them to the script. This +example builds a human, mouse, and fruit fly database: ```bash -blastdbcmd -db databases/lverage_proteins -info - -blastdbcmd \ - -db databases/lverage_proteins \ - -entry NP_000001.1 \ - -outfmt '%f' +scripts/make_blast_db.sh -t 9606,10090,7227 -o databases/orthologs ``` -The second command should print the expected FASTA record. If it reports that -the database contains no accession information, rebuild the database from its -source FASTA with `-parse_seqids`. Use a new output prefix rather than -overwriting a database that may be in use. +The script downloads each species' RefSeq reference proteome, combines them, +builds the database with `-parse_seqids` (Lverage needs this to retrieve each +hit's complete protein), and checks that retrieval works. It finishes by +printing the species names found in the headers. -If a taxonomy-enabled source database such as NCBI `nr` is already installed, -a smaller FASTA can first be extracted. The following example selects human and -mouse proteins: +If you already have a protein FASTA, build from it with `-i proteins.fasta` +instead. Headers must carry a unique identifier and the species in brackets: -```bash -blastdbcmd \ - -db /path/to/nr \ - -taxids 9606,10090 \ - -target_only \ - -outfmt '%f' \ - -out human_mouse.fasta - -makeblastdb \ - -in human_mouse.fasta \ - -dbtype prot \ - -parse_seqids \ - -out databases/human_mouse_nr +```text +>NP_000001.1 Homeobox protein [Homo sapiens] +MPEPTIDESEQUENCE ``` -Taxonomic extraction requires the taxonomy files associated with the source -BLAST database. Check the resulting FASTA headers and sequence count before -building the new database. - -Pass the database prefix and a mapping of the scientific names used in its -FASTA headers to `LocalBlastSearcher`: - -```python -from lverage.blast import LocalBlastSearcher - - -ortholog_searcher = LocalBlastSearcher( - database_path="databases/human_mouse_nr", - species_map={ - "Homo sapiens": 9606, - "Mus musculus": 10090, - }, -) -``` +If you have a taxonomy-enabled BLAST database such as NCBI `nr` on disk, add +`-n /path/to/nr` to extract the species from it rather than downloading. Run +`scripts/make_blast_db.sh -h` for all options. + +### 3. Keep the output for Lverage + +Note the database prefix you chose and the species names the script prints +at the end. Lverage needs both: the prefix says where to search, and the +species names, mapped to their taxonomy IDs, tell it which hits count as +orthologs. The names must match the headers exactly, including any strain +suffix such as `Saccharomyces cerevisiae S288C`. Bacterial proteomes label +many records with only a genus or group name, such as `Bacillus`, so map +every printed name you want to count, not just the species. Two species of +the same genus cannot be told apart by such a label, so expect their shared +proteins to be attributed to whichever taxonomy ID you map that name to. See +the +[API documentation](docs/api/index.rst) for how to pass them to the V2 +pipeline. ## How to Use We warn against moving any file within the directory anywhere else as this will create errors. If you wish to access from other places, we suggest appending the directory to your PATH environment variable, creating an alias, or creating a shortcut. diff --git a/scripts/make_blast_db.sh b/scripts/make_blast_db.sh new file mode 100755 index 0000000..5b4815b --- /dev/null +++ b/scripts/make_blast_db.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# Build the local protein BLAST database that Lverage V2 searches for orthologs. +# +# Give it the NCBI taxonomy IDs of the species you want orthologs from. It +# downloads each species' RefSeq reference proteome from NCBI, combines them, +# builds a BLAST database with -parse_seqids (required: Lverage fetches each +# hit's complete protein with "blastdbcmd -entry"), and verifies retrieval. +# +# Usage: +# scripts/make_blast_db.sh -t 9606,10090,7227 -o databases/orthologs +# +# Other sources: +# -i FILE build from a protein FASTA you already have, instead of +# downloading. Headers must carry the species in brackets, +# e.g. ">NP_000001.1 Homeobox protein [Homo sapiens]". +# -n DB extract -t TAXIDS from an existing taxonomy-enabled BLAST +# database (such as NCBI nr) instead of downloading. +# +# Options: +# -t TAXIDS comma-separated NCBI taxonomy IDs (find them at +# https://www.ncbi.nlm.nih.gov/taxonomy) +# -o PREFIX output database prefix, e.g. databases/orthologs (required) +# -h show this help +# +# Requires BLAST+ (makeblastdb, blastdbcmd) on PATH. Downloading also requires +# the NCBI datasets CLI and unzip. With conda: +# conda install -c conda-forge -c bioconda blast ncbi-datasets-cli unzip + +set -euo pipefail + +usage() { sed -n '2,/^set -euo/p' "$0" | sed '$d' | sed 's/^# \{0,1\}//'; } +die() { echo "error: $*" >&2; exit 1; } +need() { command -v "$1" >/dev/null || die "$1 not found on PATH. $2"; } + +input=""; prefix=""; source_db=""; taxids="" +while getopts "i:o:n:t:h" opt; do + case "$opt" in + i) input="$OPTARG" ;; + o) prefix="$OPTARG" ;; + n) source_db="$OPTARG" ;; + t) taxids="$OPTARG" ;; + h) usage; exit 0 ;; + *) usage >&2; exit 2 ;; + esac +done + +[[ -n "$prefix" ]] || die "-o PREFIX is required" +need makeblastdb "Install BLAST+ first." +need blastdbcmd "Install BLAST+ first." +mkdir -p "$(dirname "$prefix")" + +if [[ -n "$input" && ( -n "$source_db" || -n "$taxids" ) ]]; then + die "use -i FILE on its own, or -t TAXIDS (optionally with -n DB)" +fi + +if [[ -n "$source_db" ]]; then + [[ -n "$taxids" ]] || die "-t TAXIDS is required with -n" + input="${prefix}.fasta" + echo "Extracting taxids $taxids from $source_db into $input" + # -target_only keeps only deflines belonging to the requested taxa, so the + # species in each header is one Lverage's species map will recognise. + blastdbcmd -db "$source_db" -taxids "$taxids" -target_only -outfmt '%f' -out "$input" + +elif [[ -n "$taxids" ]]; then + need datasets "Install the NCBI datasets CLI (conda: bioconda::ncbi-datasets-cli)." + need unzip "Install unzip." + input="${prefix}.fasta" + : > "$input" + workdir=$(mktemp -d "${TMPDIR:-/tmp}/make_blast_db.XXXXXX") + trap 'rm -rf "$workdir"' EXIT + IFS=',' read -r -a taxid_list <<< "$taxids" + for taxid in "${taxid_list[@]}"; do + taxid="${taxid//[[:space:]]/}" + [[ "$taxid" =~ ^[0-9]+$ ]] || die "taxid is not a number: '$taxid'" + echo "Downloading RefSeq reference proteome for taxid $taxid" + datasets download genome taxon "$taxid" --reference --include protein --no-progressbar \ + --filename "$workdir/$taxid.zip" \ + || die "download failed for taxid $taxid (no RefSeq reference genome, or no network?)" + unzip -q -o "$workdir/$taxid.zip" -d "$workdir/$taxid" + faa=$(find "$workdir/$taxid" -name 'protein.faa' | head -n 1) + [[ -n "$faa" ]] || die "no protein.faa in the download for taxid $taxid; its reference genome may lack annotation" + count=$(grep -c '^>' "$faa") + species=$(grep -m1 '^>' "$faa" | sed -n 's/.*\[\(.*\)\].*/\1/p') + echo " $count proteins, species in headers: ${species:-NONE}" + [[ -n "$species" ]] || die "headers for taxid $taxid have no bracketed species name; Lverage cannot use them" + # RefSeq reuses WP_ accessions for identical proteins across related + # bacteria, and makeblastdb -parse_seqids refuses duplicate ids. When a + # record's id was already added by an earlier species, suffix it with + # this taxid; the description and species are kept as they are. + { grep '^>' "$input" || true; } | sed 's/^>//; s/[[:space:]].*//' > "$workdir/seen_ids" + awk -v taxid="$taxid" -v seen="$workdir/seen_ids" ' + BEGIN { while ((getline id < seen) > 0) have[id] = 1 } + /^>/ { id = substr($1, 2) + if (id in have) { + new_id = id "_" taxid; n = 1 + while (new_id in have) new_id = id "_" taxid "_" (++n) + $1 = ">" new_id; id = new_id; renamed++ + } + have[id] = 1 + print; next } + { print } + END { if (renamed) printf(" %d ids collided with records already added and were made unique with a _%s suffix\n", renamed, taxid) > "/dev/stderr" } + ' "$faa" >> "$input" + done + echo "Combined proteome written to $input" +fi + +[[ -n "$input" ]] || die "give -t TAXIDS to download, or -i FILE to build from your own FASTA" +[[ -s "$input" ]] || die "input FASTA is missing or empty: $input" + +dup_ids=$(grep '^>' "$input" | sed 's/^>//; s/[[:space:]].*//' | sort | uniq -d | sed -n '1,5p') +if [[ -n "$dup_ids" ]]; then + die "duplicate record identifiers in $input (makeblastdb -parse_seqids requires unique ids); first few: $(echo "$dup_ids" | tr '\n' ' ')" +fi + +echo "Building $prefix" +makeblastdb -in "$input" -dbtype prot -parse_seqids -max_file_sz 3GB -out "$prefix" >/dev/null + +# Verify that accession retrieval works, using the first record in the input. +first_id=$(grep -m1 '^>' "$input" | sed 's/^>//; s/[[:space:]].*//') +[[ -n "$first_id" ]] || die "could not read a record identifier from $input" +blastdbcmd -db "$prefix" -entry "$first_id" -outfmt '%a' >/dev/null \ + || die "built $prefix but could not retrieve $first_id from it; Lverage will not be able to use this database" + +echo +blastdbcmd -db "$prefix" -info | sed -n '1,2p' +echo +echo "Species found in headers (use these exact names as species_map keys):" +grep '^>' "$input" | sed -n 's/.*\[\([^]]*\)\].*/\1/p' | sort | uniq -c | sort -rn +echo +echo "Database ready. Pass database_path=\"$prefix\" to LocalBlastSearcher."