FINDKEY update 26 Jul 02 NAME findkey - finds database entries containg one or more keywords SYNOPSIS findkey findkey [-pvbmgrdutielnsaxzL] keywordfile [namefile findfile] findkey [-P PIR_dataset] keywordfile [namefile findfile] findkey [-G GenBank_dataset] keywordfile [namefile findfile] DESCRIPTION findkey uses the grep family of commands to find lines in database annotation files containing one or more keywords. Next, xylem_identify is called to create a .nam file, containing the names of entries containing the keywords, and a .fnd file, containing the actual lines from each entry containing hits. A PIR or GenBank dataset is either a file containing one or more GenBank or PIR entries, or the name of a XYLEM dataset created by splitdb. See FILES below for a more detailed description. INTERACTIVE USE findkey prompts the user to set search parameters, using an interactive menu: ___________________________________________________________________ FINDKEY - Version 12 Aug 94 Please cite: Fristensky (1993) Nucl. Acids Res. 21:5997-6003 ___________________________________________________________________ Keyfile: Dataset: ------------------------------------------------------------------- Parameter Description Value ------------------------------------------------------------------- 1) Keyword Keyword to find thionin 2) Keyfile Get list of keywords from Keyfile 3) WhereToLook p:PIR v:VecBase p GenBank - b:bacterial i:invertebrate m:mamalian e:expressed seq. tag g:phage l:plant r:primate n:rna d:rodent s:synthetic u:unannotated a:viral t:vertebrate x:patented z:STS G: GenBank dataset P: PIR dataset ------------------------------------------------------------- Type number of your choice or 0 to continue: 0 Searching /home/psgendb/PIR/pir1.ano... Sequence names will be written to thionin~pir.nam Lines containing keyword(s) will be written to thionin~pir.fnd Searching /home/psgendb/PIR/pir2.ano... Sequence names will be written to thionin~pir.nam Lines containing keyword(s) will be written to thionin~pir.fnd Searching /home/psgendb/PIR/pir3.ano... Sequence names will be written to thionin~pir.nam Lines containing keyword(s) will be written to thionin~pir.fnd As shown in the example above, the keyword thionin was specified as the keyword to search for. By default, option 3 is set to p, and the PIR protein database is searched. Messages describe the progress of the search. Since PIR is broken up into two divisions (new and protein) both are searched, but all output is written to thionin.pir.nam and thionin.pir.fnd OPTIONS (1,2) Which keywords to search for? If you want to search for a single keyword, option 1 lets you type the keyword, without having to create a file. To search for more than one keyword, choose option 2, and specify the name of a file containing the keywords. For example, entries containing genes for antibiotic resistance might be found using the following keyword file: ampicillin chloramphenicol kanamycin neomycin tetracycline Note: keyword searches are case insensitive. As you might expect, it takes longer to search for multiple keywords than a single keyword. Options 1 & 2 are mutually exclusive. Setting one will negate the other. If option 2 is chosen, the name of the keyword file will appear at the top of the menu. Finally, it is probably not a good idea to search GenBank entries using very short keywords consisting only of letters. This is because GenBank entries now include a /translation field containing the amino acid sequence of each protein coding sequence. Consequently, 3 or 4 letter keywords consisting of legal amino acid symbols (eg. CAP, recA) will turn up fairly often in protein translations. (3) WhereToLook Use this option to specify the database to be searched In the case of GenBank, only one division at a time may be searched. User-created database subsets containing PIR (P) or GenBank (G) entries may also be searched. User-created database subsets must be in the .ano/.wrp/.ind form created by splitdb. OUTPUT The output filenames take the following form: name_ex1.ex2 The 'name' part of the filename is either the keyword searched for, if option 1 was chosen, or the name of the keyword file,if option 2 obtains. 'ex1' indicates the database division that was searched. For PIR and VecBase, ex1 is 'pir' and 'vec', respectively. For GenBank, ex1 is as follows: bct - bacterial inv - invertebrate mam - other mamalian est - expressed sequence tag phg - phage pln - plant (includes fungi) pri - primate rna - structural RNAs rod - rodent syn - synthetic sequences sts - sequence tagged sites una - unannotated (new) sequences vrl - viral vrt - other vertebrate 'ex2' distinguishes the files containing the names of entries containing keywords (.nam) and the files containing the lines found in each entry (.fnd). The .nam file can be used directly as a namefile for fetch, getloc, or getob. COMMAND LINE USE OPTIONS p search PIR (default) P PIR dataset search dbfile, containing PIR entries b search Genbank bacterial division m search Genbank mamalian division g search Genbank phage division r search Genbank primate division d search Genbank rodent division u search Genbank unannotated division t search Genbank vertebrate division i search Genbank invertebrate division l search Genbank plant division s search Genbank synthetic division a search Genbank viral division x search Genbank patented division e search Genbank exp.seq.tag division z search GenBank STS division S search GenBank Genom. Survey division h search GenBank High Thrput. division G GenBank dataset search dbfile, containing GenBank entries L force execution of findkey on local host even if $XYLEM_RHOST is set. See "REMOTE EXECUTION" below FILES keywordfile - contains keywords to search for namefile - LOCUS names of hits are written to this file findfile - for each hit, a report listing the LOCUS name and the lines matching the keyword if written to this file. If namefile and findfile are not specified on the command line, filenames will be created as described above for interactive use. PIR_dataset GenBank_dataset This can be either a file of PIR entries, a file of GenBank entries, or a XYLEM dataset created by splitdb. A file of PIR entries must have the file extension ".pir". A file of GenBank entries must have the file extension ".gen". A XYLEM dataset contains PIR entries split among three files by splitdb: annotation (.ano), sequence (.wrp) and index (.ind). These file extensions must be used! When specifying a split dataset, only the base name needs to be used. For example given a XYLEM dataset consisting of the files myset.ano, myset.wrp and myset.ind, the following two commands are equivalent: findkey -P myset something.kw findkey -P myset.ano something.kw If the original .pir file had been used, the command would have been findkey -P myset.pir something.kw The ability to work directly with .gen or .pir files is quite convenient. However, since FINDKEY needs to work with a split FINDKEY automatically splits .pir or .gen files into .ano, .wrp and .ind files, which are removed when finished. This requires extra disk space and execution time, which could be significant for large datasets. EXAMPLES If the list of antibiotics shown above was stored in the file antibiotic.kw, and option 3 was set to 'b', then the annotation portion of the GenBank bacterial division would be searched, and all lines containing any of these keywords would be written to antibiotic~bac.fnd. The corresponding GenBank entry names would appear in antibiotic~bac.nam. The same keyword file could be used to search other database files. If VecBase was searched, the output files would be antibiotic~vec.fnd and antibiotic~vec.nam. These filename conventions make it easy to search different database divisions, and to keep track of where data came from. Command line examples: findkey thionin.kw would be equivalent to the interactive example shown above. In this case, the file thionin.kw contains the word 'thionin'. (Note that since PIR is the default, -p need not be supplied.) findkey -b antibiotic.kw drugs.nam drugs.fnd would search the GenBank bacterial division for the keywords contained in antibiotic.kw, and write the output to drugs.nam and drugs.kw. FILES Database files: The directories for database files are specified by the environment variables $GB (GenBank) $PIR (PIR/NBRF) and $VEC(Vecbase). Annotation (.ano) and index (.ind) are those generated by splitdb. Temporary files: $jobid.fnd $jobid.nam $jobid.grep where $jobid is a unique jobid generated by the shell REMOTE EXECUTION Where the databases can not be stored locally, FINDKEY can call FINDKEY on another system and retrieve the results. To run FINDKEY remotely, your .cshrc file should contain the following lines: setenv XYLEM_RHOST remotehostname setenv XYLEM_USERID remoteuserid where remotehostname is the name of the host on which the databases reside (in XYLEM split format) and remoteuserid is your userid on the remote system. When run remotely, your local copy of FINDKEY will generate the following commands: rcp filename $XYLEM_USERID@$XYLEM_HOST:filename rsh $XYLEM_RHOST -l $XYLEM_USERID findkey ... rcp $XYLEM_USERID@$XYLEM_HOST:outputfilename outputfilename rsh $XYLEM_RHOST -l $XYLEM_USERID rm temporary_files Because FINDKEY uses rsh and rcp, your home directory on both the local and remote systems must have a world-readable file called .rhosts, containing the names of trusted remote hosts and your userid on each host. Before trying to get FINDKEY to work remotely, make sure that you can rcp and rsh to the remote host. Obviously, remote execution of FINDKEY implies that FINDKEY must already be installed on the remote host. When FINDKEY runs another copy of FINDKEY remotely, it uses the -L option (findkey -L) to insure that the remote FINDKEY job executes, rather than calling yet another FINDKEY on another host. ---------- Remote execution on more than 1 host ----------- If more than 1 remote host is available for running FINDKEY (say, in a clustered environment where many servers mount a common filesystem) the choice of a host can be determined by the csh script choosehost, such that execution of choosehost returns the name of a remote server. To use this approach, the following script, called 'choosehost' should be in your bin directory: #!/bin/csh # choosehost - choose a host to use for a remote job. # This script rotates among servers listed in .rexhosts, # by choosing the host at the top of the list and moving # it to the bottom. #Rotate the list, putting the current host to the bottom. set HOST = `head -1 $home/.rexhosts` set JOBID = $$ tail +2 $home/.rexhosts > /tmp/.rexhosts.$JOBID echo $HOST >> /tmp/.rexhosts.$JOBID /usr/bin/mv /tmp/.rexhosts.$JOBID $home/.rexhosts # Write out the current host name echo $HOST You must also have a file in your home directory called .rexhosts, listing remote hosts, such as graucho.cc.umanitoba.ca harpo.cc.umanitoba.ca chico.cc.umanitoba.ca zeppo.cc.umanitoba.ca Each time choosehost is called, choosehost will rotate the names in the file. For example, starting with the .rexhosts as shown, it will move graucho.cc.umanitoba.ca to the bottom of the file, and write the line 'graucho.cc.umanitoba.ca' to the standard output. The next time choosehosts is run, it would write 'harpo.cc.umanitoba.ca', and so on. Depending on your local configuration, you may wish to rewrite choosehosts. All that is really necessary is that echo `choosehost` should return the name of a valid host. Once you have installed choosehost and tested it, you can get FINDKEY to use choosehost simply by setting setenv XYLEM_RHOST choosehost in your .cshrc file. --------------- Remote filesystems ----------------------- Finally, an alternative to remote execution is to remotely mount the file system containing the databases across the network. This has the advantage of simplicity, and means that the databases are available for ALL programs on your local workstation. However, it may still be advantageous to run XYLEM remotely, since that will shift much of the computational load to another host. BUGS At present, regular expression characters cannot be used for keyword searches. SEE ALSO grep(1V) xylem_identify splitdb AUTHOR Dr. Brian Fristensky Dept. of Plant Science University of Manitoba Winnipeg, MB Canada R3T 2N2 Phone: 204-474-6085 FAX: 204-261-5732 frist@cc.umanitoba.ca REFERENCE Fristensky, B. (1993) Feature expressions: creating and manipulating sequence datasets. Nucleic Acids Research 21:5997-6003.