fetch update 25 Jan 2002 NAME fetch - retrieves database entries by name or accession number SYNOPSIS fetch {interactive mode} fetch [options] namefile [output file] {batch mode} DESCRIPTION fetch retrieves one or more entries from a database. Interactive mode: fetch prompts the user to set search parameters, using an interactive menu: ___________________________________________________________________ FETCH - Version 7 Feb 94 Please cite: Fristensky (1993) Nucl. Acids Res. 21:5997-6003 ___________________________________________________________________ Namefile: Outfile: Database: ------------------------------------------------------------------- Parameter Description Value 1) Name/Acc Name or Accession sequence to get 2) Namefile Get list of sequences from Namefile 3) WhatToGet a:annotation s:sequence b:both b 4) Database g:GenBank p:PIR g G:GenBank dataset P:PIR dataset 5) Outfile Send all output to a single file (Outfile) 6) Files f:Send each entry to a separate file f ------------------------------------------------------------- Type number of your choice or 0 to continue: After all parameters have been set, type 0 to commence the search. Messages regarding the progress of the search will be printed. (1,2) Which entries to get? If you want to get a single entry, option 1 lets you type in the name of that entry, without having to create a namefile. To get more than one entry, choose option 2, and specify the name of a file containing sequence names or accession numbers. namefile is a file containing one or more sequence names or accession numbers, each on a separate line. Names and accession numbers can even be interspersed, in upper or lowercase, and in any order. For example, the namefile prp.nam might contain ; plant pathogenesis related proteins ; (these are sample comment lines) ; note that any line containing a semicolon is ignored x06362 x05454 TOBPR1A1 ; comments can be interspersed with names. PUMPR13 tobpr1ar Options 1 & 2 are mutually exclusive. Setting one will negate the other. If option 2 is chosen, the name of the namefile will appear at the top of the menu. (3) WhatToGet Use this option to specify whether to get annotation, sequence, or both (default=both). (4) Database Use this option to select the database. (default=GenBank). G and P select user-created database subsets containing GenBank or PIR entries, respectively. It is assumed that the database has been split into .ano, .wrp and .ind files using splitdb. For example, if you had created a database subset called PR1.pir, splitdb would create PR1.ano, PR1.wrp and PR1.ind. These are the files actually read by FETCH. When prompted for the name of the database, simply type "PR1", without a file extension. (If you do type a file extension, it will be ignored). (5, 6) Where to send output By default, option 6 is set to f, and each entry will be written to a separate file, where the name of the file is the name of the entry, followed by a file extension. If a complete entry is retrieved, the file extension will indicate the type of database (GenBank: .gen; PIR: .pir, Vecbase: .vec; LiMB: .LiMB). If only annotation or sequence are retrieved, the file extensions will be .ano or .wrp, respectively. Using the default, the namefile above would create the following files: PUMPR13.gen TOBPR1A1.gen TOBPR1AR.gen TOBPR1CR.gen TOBPR1PS.gen By choosing option 5, you can specify the name of an output file for all entries to go to. This filename will appear at the top of the menu. Obviously, options 5 & 6 are mutually exclusive. Note entries retrieved are writen in alphabetical order (sorting by ASCII values), not the order in which they appeared in namefile. (Note for remote users only: -f will only work for a single name/accession supplied in 1). -f IS NOT ENABLED FOR NAMEFILES specified in 2).) Batch mode: Although it is transparent to the user, all fetch really does is call getloc, saving the user the trouble of knowing which database files to retrieve sequences from, or of having to execute getloc multiple times to retrieve sequences from different database files. Thus, the options are identical to those for getloc: -a Write annotation portions of entries only, terminated by '//'. -s Write sequence data only, in Pearson (.wrp) format. -f Write each entry to a separate file. -g GenBank (default) -e EMBL {not implemented} -p PIR (NBRF) -G GenBank_dataset -P PIR_dataset If -f is not specified, outfile must be specified. -L force execution of findkey on local host even if $XYLEM_RHOST is set. See "REMOTE EXECUTION" below PIR_dataset GenBank_dataset This can be either a file of PIR entries, a file of GenBank entries, or a XYLEM dataset created by splitdb. A file of PIR entries must have the file extension ".pir". A file of GenBank entries must have the file extension ".gen". A XYLEM dataset contains PIR entries split among three files by splitdb: annotation (.ano), sequence (.wrp) and index (.ind). These file extensions must be used! When specifying a split dataset, only the base name needs to be used. For example given a XYLEM dataset consisting of the files myset.ano, myset.wrp and myset.ind, the following two commands are equivalent: fetch -P myset something.nam something.pir fetch -P myset.ano something.nam something.pir If the original .pir file had been used, the command would have been fetch -P myset.pir something.nam something.pir The ability to work directly with .gen or .pir files is quite convenient. However, since FETCH needs to work with a split FETCH automatically splits .pir or .gen files into .ano, .wrp and .ind files, which are removed when finished. This requires extra disk space and execution time, which could be significant for large datasets. EXAMPLES Batch example: fetch -f chitinase.nam will retrieve annotation and sequence for sequences listed in chitinase.nam from GenBank, writing each entry to a separate file with the extension .gen. fetch -s -p test.nam test.wrp will retrieve sequence data only for the entries listed in test.nam, from PIR, and write all sequences to a Pearson format file (ie. readable by fasta) with the name test.wrp. fetch -G sample sample.nam new.gen fetch -G sample.ano sample.nam new.gen Assumes that a set of GenBank entries has been split by splitdb into sample.ano sample.wrp and sample.ind. The entries listed in sample.nam are written to new.gen. FILES Database files: The directories for database files are specified by the environment variables $GB (GenBank) and $PIR (PIR/NBRF) . Index files are $GB/gbacc.idx for GenBank (this file is supplied with each GenBank release), while the other databases use .ind files generated by splitdb. Split database files MUST have the following file extensions: .ano {annotation}, .wrp {sequence} and .ind {index}. Thus, when creating database files for pir1.dat with splitdb, the output files should be pir1.ano, pir1.wrp and pir1.ind. Temporary files: NAMEFILE.fetch PRELIMINARY.fetch TMP.fetch FOUND.fetch FETCHDIR {temporary directory} REMOTE EXECUTION Where the databases can not be stored locally, FETCH can call FETCH on another system and retrieve the results. To run FETCH remotely, your .cshrc file should contain the following lines: setenv XYLEM_RHOST remotehostname setenv XYLEM_USERID remoteuserid where remotehostname is the name of the host on which the databases reside (in XYLEM split format) and remoteuserid is your userid on the remote system. When run remotely, your local copy of FETCH will generate the following commands: rcp filename $XYLEM_USERID@$XYLEM_HOST:filename rsh $XYLEM_RHOST -l $XYLEM_USERID fetch ... rcp $XYLEM_USERID@$XYLEM_HOST:outputfilename outputfilename rsh $XYLEM_RHOST -l $XYLEM_USERID $RM temporary_files Because FETCH uses rsh and rcp, your home directory on both the local and remote systems must have a world-readable file called .rhosts, containing the names of trusted remote hosts and your userid on each host. Before trying to get FETCH to work remotely, make sure that you can rcp and rsh to the remote host. Obviously, remote execution of FETCH implies that FETCH must already be installed on the remote host. When FETCH runs another copy of FETCH remotely, it uses the -L option (findkey -L) to insure that the remote FETCH job executes, rather than calling yet another FETCH on another host. ---------- Remote execution on more than 1 host ----------- If more than 1 remote host is available for running FINDKEY (say, in a clustered environment where many servers mount a common filesystem) the choice of a host can be determined by the csh script choosehost, such that execution of choosehost returns the name of a remote server. To use this approach, the following script, called 'choosehost' should be in your bin directory: #!/bin/csh # choosehost - choose a host to use for a remote job. # This script rotates among servers listed in .rexhosts, # by choosing the host at the top of the list and moving # it to the bottom. #Rotate the list, putting the current host to the bottom. set HOST = `head -1 $home/.rexhosts` set JOBID = $$ tail +2 $home/.rexhosts > /tmp/.rexhosts.$JOBID echo $HOST >> /tmp/.rexhosts.$JOBID /usr/bin/mv /tmp/.rexhosts.$JOBID $home/.rexhosts # Write out the current host name echo $HOST You must also have a file in your home directory called .rexhosts, listing remote hosts, such as graucho.cc.umanitoba.ca harpo.cc.umanitoba.ca chico.cc.umanitoba.ca zeppo.cc.umanitoba.ca Each time choosehost is called, choosehost will rotate the names in the file. For example, starting with the .rexhosts as shown, it will move graucho.cc.umanitoba.ca to the bottom of the file, and write the line 'graucho.cc.umanitoba.ca' to the standard output. The next time choosehosts is run, it would write 'harpo.cc.umanitoba.ca', and so on. Depending on your local configuration, you may wish to rewrite choosehosts. All that is really necessary is that echo `choosehost` should return the name of a valid host. Once you have installed choosehost and tested it, you can get FINDKEY to use choosehost simply by setting setenv XYLEM_RHOST choosehost in your .cshrc file. --------------- Remote filesystems ----------------------- Finally, an alternative to remote execution is to remotely mount the file system containing the databases across the network. This has the advantage of simplicity, and means that the databases are available for ALL programs on your local workstation. However, it may still be advantageous to run FETCH remotely, since that will shift much of the computational load to another host. BUGS When retrieving entries directly from GenBank, FETCH uses the Accession Number index file gbacc.idx. In this case, FETCH can retrieve all entries containing a given accession number. This capability makes it possible to retrieve an entry using a secondary accession number. However if more than one entry share a secondary accession number, all of those entries will be retrieved. While this behavior might be a bit of an annoyance at times, it can also be useful because it alerts the user to the presence of other, related entries that might be of interest. SEE ALSO getloc features AUTHOR Dr. Brian Fristensky Dept. of Plant Science University of Manitoba Winnipeg, MB Canada R3T 2N2 Phone: 204-474-6085 FAX: 204-474-7528 frist@cc.umanitoba.ca REFERENCE Fristensky, B. (1993) Feature expressions: creating and manipulating sequence datasets. Nucleic Acids Research 21:5997-6003.