Local BLAST Databases Important Considerations

May  1, 2016

• disk space - choice of which databases to install is dependent on available disk space
• memory and cores - If the memory (RAM) and numbers of cores (CPUs) are not adequate, it may not be worthwhile to search some of the larger BLAST databases on your machine.
  
• network load - Database files are typically huge, often in hundreds of Mb. Downloads take up a lot of bandwith, and in some cases could take hours to download. WiFi in particular may be much slower than wired Ethernet.
• mirrors - It is best to download from the FTP mirror site that is geographically closest to you.
• backups - Because backing up large amounts of data requires a lot of space on backup devices and may generate large amounts of network traffic, it is probably unnecessary and often counterproductive to backup your local copies of the databases. If needed, you can always re-install the latest copies of the databases by FTP.  It may be necessary to modify your current backup protocols to prevent backups of your BLAST databases from overwhelming your backup resources.

Disk space

Databases are downloaded as gzip-compressed tar archive files (ie. .tar.gz files). When uncompressed, the combined files in any given database will require anywhere from 1.1 to 5.3 times the size of the .tar.gz archive file. For most databases, the ratio of decompressed to compressed is around 2.5. The table below gives a snapshot of sizes and decompression ratios for NCBI databases downloaded between April 1 - 3, 2016.

When considering how much disk space you can realistically afford to use, keep in mind that you can't just fill up the remaining space on the file system with databases. Day to day us of any computer demands some breathing room in available disk space. A big consideration is that to download any database file, you need enough space to hold both the downloaded file and the uncompressed contents. As well, just about any program running on the system will generate some disk files, even if only temporarily. A good rule of thumb is that you should avoid allowing any filesystem to fill much beyond about 90% of its capacity.

Memory and Cores*

Unless you have adequate numbers of CPUs and RAM, it may be pointless to install local copies of BLAST databases.

Standalone BLAST+ uses a great deal of memory, and may be unreasonably slow on machines without adequate numbers of cores (CPUs).  For most databases, you probably want a minimum of 8 cores and 16 Gb. RAM.  The good news is that it is relatively cheap to upgrade a computer to a configuration that will give you faster turnaround times than sending your jobs to NCBI.

A future version of this document will include some statistics on search times on computer systems with various configurations of CPU and RAM.

*The distinction between cores and CPUs is as follows:
A Central Processing Unit performs operations on data in RAM. Originally, the CPU had one processor. Today, the vast majority of CPUs manufactured today, even on low-end PCs,  have 2 or more cores, each of which can process information independently. The terms CPU and core, while not synonymous, are often used interchangeable. Strictly speaking, it is most correct to use the term core to refer to the number of processing units.

Network Load

A hard-wired (Ethernet) connection is best.  In most institutional settings, each computer will have a separate IP address on a switch.

Wifi may be too slow or not have adequate bandwith. Downloads will almost certainly take longer on Wifi. As well, since Wifi is a shared resource, large downloads may affect the Wifi performance for others nearby.

Mirrors

There are several FTP sites open to the general public for downloading copies of the NCBI BLAST databasese. Since these "mirrors" are kept in sync with NCBI, the primary consideration should be minimizing the amount of network load that your downloads generate, as a courtesy to others, as well as download speed. As well, since network traffic can often result in dropped connections during a download.

For all these reasons, it is usually best to download files from the FTP site geographically closest to your location.

Backups