Local BLAST Databases Important Considerations

May  4, 2020

• disk space - choice of which databases to install is dependent on available disk space
• memory and cores - If the memory (RAM) and numbers of cores (CPUs) are not adequate, it may not be worthwhile to search some of the larger BLAST databases on your machine.
  
• network load - Database files are typically huge, often in hundreds of Mb. Downloads take up a lot of bandwith, and in some cases could take hours to download. WiFi in particular may be much slower than wired Ethernet.
• mirrors - It is best to download from the FTP mirror site that is geographically closest to you.
• backups - Because backing up large amounts of data requires a lot of space on backup devices and may generate large amounts of network traffic, it is probably unnecessary and often counterproductive to backup your local copies of the databases. If needed, you can always re-install the latest copies of the databases by FTP.  It may be necessary to modify your current backup protocols to prevent backups of your BLAST databases from overwhelming your backup resources.

Disk space

Databases are downloaded as gzip-compressed tar archive files (ie. .tar.gz files). When uncompressed, the combined files in any given database will require anywhere from 1.1 to 14 times the size of the .tar.gz archive files. For most databases, the ratio of decompressed to compressed is around 2.5. The table below gives a snapshot of sizes and decompression ratios for NCBI databases downloaded May 4, 2020.

blastdbkit.py:	LOCAL BLAST DATABASE REPORT
Source:	localhost	ftp.ncbi.nih.gov
Database Directory:	/home/psgendb/GenBank	/blast/db
DB name	uncompressed size (Mb)	compressed size (Mbytes)*	decompression ratio
nt	149731	64046	2.34
refseq_rna	43752	15815	2.77
human_genome	1566	1192	1.35
mouse_genome	1322	1025	1.33
ref_euk_rep_genomes	199793	186603	1.07
ref_prok_rep_genomes	12651	12368	1.03
ref_viroids_rep_genomes	0	30	1.00
ref_viruses_rep_genomes	82	108	1.05
patnt	17847	6709	2.67
pdbnt	14	31	14.00
16S_ribosomal_RNA	14	36	2.33
18S_fungal_sequences	1	30	1.00
28S_fungal_sequences	3	31	3.00
ITS_RefSeq_Fungi	5	32	2.50
ITS_eukaryote_sequences	37	39	4.11
LSU_eukaryote_rRNA	5	32	2.50
LSU_prokaryote_rRNA	2	31	2.00
SSU_eukaryote_rRNA	6	32	3.00
Betacoronavirus	29	38	3.63
nr	404350	98231	4.12
refseq_protein	170800	51766	3.30
swissprot	695	183	4.54
pdbaa	267	63	8.09
landmark	374	160	2.88
env_nt	79383	40140	1.98
taxdb	161	30	5.37
TOTAL:	1082890
*Compressed database files for each division of the database include the taxid database, which at this writing is about 30 Mb. Thus, although the total sizes of compressed files for ITS_eukaryote_sequences is 39 Mb, 30 Mb of that is a copy of taxdb. Each time the database files are de-archived, any existing copy of taxdb is overwritten. Consequently, there will only be one copy of taxdb after each install or update, even though many copies may have been downloaded. Decompression ratios in the table are calculated after subtracting the compressed size of taxdb (eg. 30 Mb) from the compressed size.

Memory and Cores*

Unless you have adequate numbers of CPUs and RAM, it may be pointless to install local copies of BLAST databases.

Standalone BLAST+ uses a great deal of memory, and may be unreasonably slow on machines without adequate numbers of cores (CPUs).  For most databases, you probably want a minimum of 8 cores and 16 Gb. RAM.  The good news is that it is relatively cheap to upgrade a computer to a configuration that will give you faster turnaround times than sending your jobs to NCBI.

A future version of this document will include some statistics on search times on computer systems with various configurations of CPU and RAM.

*The distinction between cores and CPUs is as follows:
A Central Processing Unit performs operations on data in RAM. Originally, the CPU had one processor. Today, the vast majority of CPUs manufactured today, even on low-end PCs,  have 2 or more cores, each of which can process information independently. The terms CPU and core, while not synonymous, are often used interchangeable. Strictly speaking, it is most correct to use the term core to refer to the number of processing units.

Network Load

A hard-wired (Ethernet) connection is best.  In most institutional settings, each computer will have a separate IP address on a switch.

Wifi may be too slow or not have adequate bandwith. Downloads will almost certainly take longer on Wifi. As well, since Wifi is a shared resource, large downloads may affect the Wifi performance for others nearby.

Mirrors

There are several FTP sites open to the general public for downloading copies of the NCBI BLAST databasese. Since these "mirrors" are kept in sync with NCBI, the primary consideration should be minimizing the amount of network load that your downloads generate, as a courtesy to others, as well as download speed. As well, since network traffic can often result in dropped connections during a download.

For all these reasons, it is usually best to download files from the FTP site geographically closest to your location.

Backups