Release 1.7 (26th January 2018) -------------------- * HTSlib, and so samtools, now support BAMs which include CIGARs with more than 65535 operations as per HTS-Specs 18th November (dab57f4 and 2f915a8). * samtools quickcheck will now write a warning to stderr if it finds any problems. These messages can be suppressed with a new `-q` option. * samtools markdup can now mark supplementary alignments of reads where the primary alignment is found to be a duplicate. Supplementary marking can be turned on by passing the `-S` option to markdup. When this option is enabled, all the alignment data will be written to a temporary file so that supplementary alignments that occur before a duplicated primary can be correctly marked in the final output. The location of this temporary file can be influenced using the new `-T` option. * samtools view now supports HTSlib's new multi-region iterator. This can be enabled by passing the `-M` option to view. When using this option: - The BED filter (`-L` option) will use the index to skip through the file - Reads from overlapping regions will only be output once * samtools bedcov will now ignore BED comment and header lines (#571; thanks to Daniel Baker). * samtools collate now updates the @HD SO: and GO: tags, and sort will remove a GO: tag if present. (#757; reported by Imran Haque). * Bug-fixes: - maq2sam now checks for input files that end early. (#751; patch supplied by Alexandre Rebert of the Mayhem team, via Andreas Tille from Debian.) - Fixed incorrect check when looking up header tags that could lead to a crash in samtools stats. (#208; thanks to Dave Larson.) - Fixed bug in samtools fastq `-O` option where it would fail if the OQ tag in the input file had an unexpected type. (#758; reported by Taejeong Bae) - The MD5 calculations in samtools dict and md5fa did not handle non-alphabetic characters in the same way as the CRAM MD5 function. They have now been updated to match. (#704; reported by Chris Norman). - Fix possible infinite loop in samtools targetcut. - Building bam_tview_curses should no longer fail if a curses header file cannot be found. Release 1.6 (28th September 2017) -------------------- * Added new markdup sub-command and '-m' option for fixmate. Used together, they allow duplicates to be marked and optionally removed. This fixes a number of problems with the old 'rmdup' sub-command, for example samtools issue #497. 'rmdup' is kept for backwards compatibility but 'markdup' should be used in preference. * Sort is now much better at keeping within the requested memory limit. It should also be slightly faster and need fewer temporary files when the file to be sorted does not fit in memory. (#593; thanks to Nathan Weeks.) * Sort no longer rewrites the header when merging from files. It can also now merge from memory, so fewer temporary files need to be written and it is better at sorting in parallel when everything fits in memory. * Both sort and merge now resolve ties when merging based on the position in the input file(s). This makes them fully stable for all ordering options. (Previously position sort was stable, but name and by tag sorts were not). * New --output-qname option for mpileup. * Support for building on Windows using msys2/mingw64 or cygwin has been improved. Release 1.5 [Solstice Release] (21st June 2017) -------------------- * Samtools fastq now has a -i option to create a fastq file from an index tag, and a -T option (similar to -t) to add user specified aux tags to the fastq header line. * Samtools fastq can now create compressed fastq files, by giving the output filenames an extention of .gq, .bgz, or .bgzf * Samtools sort has a -t TAG option, that allows records to be sorted by the value of the specified aux tag, then by position or name. Merge gets a similar option, allowing files sorted this way to be merged. (#675; thanks to Patrick Marks of 10xgenomics). Release 1.4.1 (8th May 2017) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * Added options to fastq to create fastq files from BC (or other) tags. * Samtools view has gained a -G option to exclude on all bits set. For example to discard reads where neither end has been mapped use "-G 12". * Samtools cat has a -b option to ease concatenation of many files. * Added misc/samtools_tab_completion for bash auto-completion of samtools sub-commands. (#560) * Samtools tview now has J and K keys for verticale movement by 20 lines. (#257) * Various compilation / portability improvements. * Fixed issue with more than 65536 CIGAR operations and SAM/CRAM files. (#667) Release 1.4 (13 March 2017) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Noteworthy changes in samtools: * Fixed Issue #345 - out-by-one error in insert-size in samtools stats * bam_split now add a @PG header to the bam file * Added mate cigar tag support to fixmate * Multi-threading is now supported for decoding BAM and CRAM (as well as the previously supported encoding). Most commands that read BAM or CRAM have gained an -@ or --threads arguments, providing a significant speed bonus. For commands that both read and write files the threads are shared between decoding and encoding tasks. * Added -a option to samtools mpileup to show all locations, including sites with zero depth; repeating the option as -aa or -a -a additionally shows reference sequences without any reads mapped to them (#496). * The mpileup text output no longer contains empty columns at zero coverage positions. Previously it would output "...0\t\t..." in some circumstances (zero coverage due to being below a minumum base quality); this has been fixed to output as "...0\t*\t*..." with placeholder '*' characters as in other zero coverage circumstances (see PR #537). * To stop it from creating too many temporary files, samtools sort will now not run unless its per-thread memory limit (-m) is set to at least 1 megabyte (#547). * The misc/plot-bamstats script now has a -l / --log-y option to change various graphs to display their Y axis log-scaled. Currently this affects the Insert Size graph (PR #589; thanks to Anton Kratz). * Fixmate will now also add and update MC (mate CIGAR) tags. Beta Release 1.3.1 (22 April 2016) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Noteworthy changes in samtools: * The sort command creates any needed temporary files alongside the final output file (similarly to the pre-1.3 behaviour), and now aborts when it detects a collision with another sort invocation's temporary files. When the -T PREFIX option specified is a directory (or when sorting to standard output), a random component is now added to temporary filenames to try to avoid collisions (#432, #523, #529, #535, PR #530). * All samtools commands now check for I/O errors more carefully, especially when writing output files (#111, #253, #470, PR #467). * Build fixes for 32-bit systems; be sure to run configure on such systems to enable large file support and access to 2GiB+ files. * The fasta/fastq/bam2fq command no longer ignores reads when the -s option is used (#532). * The fastq -O option no longer crashes on reads that do not have an OQ tag field (#517). * The merge and sort commands now handle (unusual) BAM files that have no textual @SQ headers (#548, #550). * Sorting files containing @CO headers no longer duplicates the comment headers, which previously happened on large sorts for which temporary files were needed (#563). * The rmdup and view -l commands no longer crash on @RG headers that do not have a LB field (#538). * Fixed miscellaneous issues #128, #130, #131, #489, and #514. Beta Release 1.3 (15 December 2015) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Noteworthy changes in samtools: * The obsolete "samtools sort in.bam out.prefix" usage has been removed. If you are still using -f, -o, or out.prefix, convert to use -T PREFIX and/or -o FILE instead. (#295, #349, #356, #418, PR #441; see also discussions in #171, #213.) * When writing CRAM output, samtools now defaults to writing CRAM v3.0 rather than v2.1. * The "bamshuf" command has been renamed to "collate" (hence the term bamshuf no longer appears in the documentation, though it still works on the command line for compatibility with existing scripts). * The mpileup command now outputs the unseen allele in VCF/BCF as <*> rather than X or as previously, and now has AD, ADF, ADR, INFO/AD, INFO/ADF, INFO/ADR --output-tags annotations that largely supersede the existing DV, DP4, DPR annotations. * The mpileup command now applies BAQ calculations at all base positions, regardless of which -l or -r options are used (previously with -l it was not applied to the first few tens of bases of each chromosome, leading to different mpileup results with -l vs. -r; #79, #125, #286, #407). * Samtools now has a configure script which checks your build environment and facilitates choosing which HTSlib to build against. See INSTALL for details. * Samtools's Makefile now fully supports the standard convention of allowing CC/CPPFLAGS/CFLAGS/LDFLAGS/LIBS to be overridden as needed. Previously it listened to $(LDLIBS) instead; if you were overriding that, you should now override LIBS rather than LDLIBS. * A new addreplacerg command that adds or alters @RG headers and RG:Z record tags has been added. * The rmdup command no longer immediately aborts (previously it always aborted with "bam_get_library() not yet implemented"), but remains not recommended for most use (#159, #252, #291, #393). * Merging files with millions of headers now completes in a reasonable amount of time (#337, #373, #419, #453; thanks to Nathan Weeks, Chris Smowton, Martin Pollard, Rob Davies). * Samtools index's optional index output path argument works again (#199). * Fixed calmd, targetcut, and potential mpileup segfaults when given broken alignments with POS far beyond the end of their reference sequences. * If you have source code using bam_md.c's bam_fillmd1_core(), bam_cap_mapQ(), or bam_prob_realn_core() functions, note that these now take an additional ref_len parameter. (The versions named without "_core" are unchanged.) * The tview command's colour scheme has been altered to be more suitable for users with colour blindness (#457). * Samtools depad command now handles CIGAR N operators and accepts CRAM files (#201, #404). * Samtools stats now outputs separate "N" and "other" columns in the ACGT content per cycle section (#376). * Added -a option to samtools depth to show all locations, including zero depth sites (#374). * New samtools dict command, which creates a sequence dictionary (as used by Picard) from a FASTA reference file. * Samtools stats --target-regions option works again. * Added legacy API sam.h functions sam_index_load() and samfetch() providing bam_fetch()-style iteration over either BAM or CRAM files. (In general we recommend recoding against the htslib API directly, but this addition may help existing libbam-using programs to be CRAM-enabled easily.) * Fixed legacy API's samopen() to write headers only with "wh" when writing SAM files. Plain "w" suppresses headers for SAM file output, but this was broken in 1.2. * "samtools fixmate - -" works in pipelines again; with 1.0 to 1.2, this failed with "[bam_mating] cannot determine output format". * Restored previous "samtools calmd -u" behaviour of writing compression level 0 BAM files. Samtools 1.0 to 1.2 incorrectly wrote raw non-BGZF BAM files, which cannot be read by most other tools. (Samtools commands other than calmd were unaffected by this bug.) * Restored bam_nt16_nt4_table[] to legacy API header bam.h. * Fixed bugs #269, #305, #320, #328, #346, #353, #365, #392, #410, #445, #462, #475, and #495. Beta Release 1.2 (2 February 2015) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Noteworthy changes in samtools: * Flagstat now works on SAM, BAM, or CRAM files (rather than BAM only) * Stats calculates mismatches per cycle for unclipped length * Merge can now merge SAM input files * CRAM reference files are now cached by default (see HTSlib release notes and samtools(1) man page) * Tested against Intel-optimised zlib (https://github.com/jtkukunas/zlib; see README for details) * Fixed bugs #302, #309, #318, and #327 and many other improvements and bugs fixed in HTSlib -- see the HTSlib release notes Beta Release 1.1 (19 September, 2014) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes in samtools: * Sorting files with thousands of reference contigs now completes in a reasonable amount of time (#260) * Fixmate and flagstat now consider supplementary reads * Fixmate now only adds a template cigar tag ("ct:Z") when requested via a new -c option, and never adds it repeatedly (#276) * Mpileup DPR annotation fixed (#274) * Checksum added to stats command output * Removed view -Q option Beta Release 1.0 (15 August, 2014) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ First release of HTSlib-based samtools. Numerous changes, notably support for the CRAM sequencing file format. The faidx command now reads either uncompressed or BGZF-compressed FASTA files compressed with bgzip. In previous samtools-0.1.x versions, faidx could read either uncompressed or RAZF-compressed FASTA files, but RAZF and razip are superseded by BGZF/bgzip and have been removed from samtools. Beta Release 0.1.20 (15 August, 2014) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Final release of standalone samtools. Beta Release 0.1.19 (15 March, 2013) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes in samtools and bcftools: * The latest source code and development moved to github, http://github.com/samtools/samtools * Many important bugfixes and contributions by many people. Thanks to all! * Performance improvements (multi-threading) * Important changes in calling, see - samtools mpileup -p - bcftools view -m * New annotations useful for filtering (RPB, HWE, QBD, MDV) * New tools, bamcheck and plot-bamcheck * New features in samtools tview * And much more.. For a detailed list of commits, please see http://github.com/samtools/samtools/commits/master (0.1.19: 15 March 2013, commit 96b5f2294ac0054230e88913c4983d548069ea4e) Beta Release 0.1.18 (2 September, 2011) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes in samtools: * Support the new =/X CIGAR operators (by Peter Cock). * Allow to subsample BAM while keeping the pairing intact (view -s). * Implemented variant distance bias as a new filter (by Petr Danecek). * Bugfix: huge memory usage during indexing * Bugfix: use of uninitialized variable in mpileup (rare) * Bugfix: wrong BAQ probability (rare) Notable changes in bcftools: * Support indel in the contrast caller. * Bugfix: LRT2=nan in rare cases (0.1.18: 2 September 2011, r982:295) Beta Release 0.1.17 (6 July, 2011) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ With the maturity of `mpileup' and the lack of update in the `pileup' command, the `pileup' command is now formally dropped. Most of the pileup functionality, such as outputting mapping quality and read positions, have been added `mpileup'. Since this release, `bcftools view' is able to perform contrast SNP calling (option -T) for discovering de novo and/or somatic mutations between a pair of samples or in a family trio. Potential mutations are scored by a log likelihood ratio, which is very simple in math, but should be comparable to more sophisticated methods. Note that getting the score is only the very first step. A lot more need to be done to reduce systematical errors due to mapping and reference errors and structural variations. Other notable changes in samtools: * Improved sorting order checking during indexing. * Improved region parsing. Colons in reference sequence names are parsed properly. * Fixed an issue where mpileup does not apply BAQ for the first few reads when a region is specified. * Fixed an issue where `faidx' does not work with FASTA files with long lines. * Bugfix: wrong SP genotype information in the BCF output. Other notable changes in bcftools: * Output the ML esitmate of the allele count. * Added the HWE plus F<0 filter to varFilter. For multiple samples, it effectively filters false heterozygous calls around centromeres. * For association mapping, perform both 1-degree and 2-degree test. The 2-degree test is conservative but more robust to HWE violation. (0.1.17: 6 July 2011, r973:277) Beta Release 0.1.16 (21 April, 2011) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes in samtools: * Support the new SAM/BAM type `B' in the latest SAM spec v1.4. * When the output file of `samtools merge' exists, do not overwrite it unless a new command-line option `-f' is applied. * Bugfix: BED support is not working when the input BED is not sorted. * Bugfix: some reads without coordinates but given on the reverse strand are lost in merging. Notable changes in bcftools: * Code cleanup: separated max-likelihood inference and Bayesian inference. * Test Hardy-Weinberg equilibrium with a likelihood-ratio test. * Provided another association test P-value by likelihood-ratio test. * Use Brent's method to estimate the site allele frequency when EM converges slowly. The resulting ML estimate of allele frequnecy is more accurate. * Added the `ldpair' command, which computes r^2 between SNP pairs given in an input file. Also, the `pileup' command, which has been deprecated by `mpileup' since version 0.1.10, will be dropped in the next release. The old `pileup' command is substandard and causing a lot of confusion. (0.1.16: 21 April 2011, r963:234) Beta Release 0.1.15 (10 April, 2011) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Noteable changes: * Allow to perform variant calling or to extract information in multiple regions specified by a BED file (`samtools mpileup -l', `samtools view -L' and `bcftools view -l'). * Added the `depth' command to samtools to compute the per-base depth with a simpler interface. File `bam2depth.c', which implements this command, is the recommended example on how to use the mpileup APIs. * Estimate genotype frequencies with ML; perform chi^2 based Hardy-Weinberg test using this estimate. * For `samtools view', when `-R' is specified, drop read groups in the header that are not contained in the specified file. * For `samtools flagstat', separate QC-pass and QC-fail reads. * Improved the command line help of `samtools mpileup' and `bcftools view'. * Use a global variable to control the verbose level of samtools stderr output. Nonetheless, it has not been full utilized. * Fixed an issue in association test which may report false associations, possibly due to floating point underflow. (0.1.15: 10 April 2011, r949:203) Beta release 0.1.14 (21 March, 2011) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This release implements a method for testing associations for case-control data. The method does not call genotypes but instead sums over all genotype configurations to compute a chi^2 based test statistics. It can be potentially applied to comparing a pair of samples (e.g. a tumor-normal pair), but this has not been evaluated on real data. Another new feature is to make X chromosome variant calls when female and male samples are both present. The user needs to provide a file indicating the ploidy of each sample (see also manual bcftools/bcftools.1). Other notable changes: * Added `bcftools view -F' to parse BCF files generated by samtools r921 or older which encodes PL in a different way. * Changed the behavior of `bcftools view -s'. Now when a list of samples is provided, the samples in the output will be reordered to match the ordering in the sample list. This change is mainly designed for association test. * Sped up `bcftools view -v' for target sequencing given thousands of samples. Also added a new option `view -d' to skip loci where only a few samples are covered by reads. * Dropped HWE test. This feature has never been implemented properly. An EM should be much better. To be implemented in future. * Added the `cat' command to samtools. This command concatenate BAMs with identical sequence dictionaries in an efficient way. Modified from bam_cat.c written by Chris Saunders. * Added `samtools view -1' to write BAMs at a low compression level but twice faster to create. The `sort' command generates temporary files at a low compression level as well. * Added `samtools mpileup -6' to accept "BAM" with Illumina 1.3+ quality strings (strictly speaking, such a file is not BAM). * Added `samtools mpileup -L' to skip INDEL calling in regions with excessively high coverage. Such regions dramatically slow down mpileup. * Updated `misc/export2sam.pl', provided by Chris Saunders from Illumina Inc. (0.1.14: 21 March 2011, r933:170) Beta release 0.1.13 (1 March, 2011) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The most important though largely invisible modification is the change of the order of genotypes in the PL VCF/BCF tag. This is to conform the upcoming VCF spec v4.1. The change means that 0.1.13 is not backward compatible with VCF/BCF generated by samtools older than r921 inclusive. VCF/BCF generated by the new samtools will contain a line `##fileformat=VCFv4.1' as well as the samtools version number. Single Individual Haplotyping (SIH) is added as an experimental feature. It originally aims to produce haploid consensus from fosmid pool sequencing, but also works with short-read data. For short reads, phased blocks are usually too short to be useful in many applications, but they can help to rule out part of SNPs close to INDELs or between copies of CNVs. Other notable changes in samtools: * Construct per-sample consensus to reduce the effect of nearby SNPs in INDEL calling. This reduces the power but improves specificity. * Improved sorting order checking in indexing. Now indexing is the preferred way to check if a BAM is sorted. * Added a switch `-E' to mpileup and calmd. This option uses an alternative way to apply BAQ, which increases sensistivity, especially to MNPs, at the cost of a little loss in specificity. * Added `mpileup -A' to allow to use reads in anomalous pairs in SNP calling. * Added `mpileup -m' to allow fine control of the collection of INDEL candidates. * Added `mpileup -S' to compute per-sample strand bias P-value. * Added `mpileup -G' to exclude read groups in variant calling. * Fixed segfault in indel calling related to unmapped and refskip reads. * Fixed an integer overflow in INDEL calling. This bug produces wrong INDEL genotypes for longer short INDELs, typically over 10bp. * Fixed a bug in tview on big-endian machines. * Fixed a very rare memory issue in bam_md.c * Fixed an out-of-boundary bug in mpileup when the read base is `N'. * Fixed a compiling error when the knetfile library is not used. Fixed a library compiling error due to the lack of bam_nt16_nt4_table[] table. Suppress a compiling warning related to the latest zlib. Other notable changes in bcftools: * Updated the BCF spec. * Added the `FQ' VCF INFO field, which gives the phred-scaled probability of all samples being the same (identical to the reference or all homozygous variants). Option `view -f' has been dropped. * Implementated of "vcfutils.pl vcf2fq" to generate a consensus sequence similar to "samtools.pl pileup2fq". * Make sure the GT FORMAT field is always the first FORMAT to conform the VCF spec. Drop bcf-fix.pl. * Output bcftools specific INFO and FORMAT in the VCF header. * Added `view -s' to call variants from a subset of samples. * Properly convert VCF to BCF with a user provided sequence dictionary. Nonetheless, custom fields are still unparsed and will be stored as a missing value. * Fixed a minor bug in Fisher's exact test; the results are rarely changed. (0.1.13: 1 March 2011, r926:134) Beta release 0.1.12a (2 December, 2010) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is another bug fix release: * Fixed a memory violation in mpileup, which causes segfault. Release 0.1.9 and above are affected. * Fixed a memory violation in the indel caller, which does not causes segfault, but may potentially affect deletion calls in an unexpected way. Release 0.1.10 and above are affected. * Fixed a bug in computing r-square in bcftools. Few are using this functionality and it only has minor effect. * Fixed a memory leak in bam_fetch(). * Fixed a bug in writing meta information to the BAM index for the last sequence. This bug is invisible to most users, but it is a bug anyway. * Fixed a bug in bcftools which causes false "DP4=0,0,0,0" annotations. (0.1.12: 2 December 2010, r862) Beta release 0.1.11 (21 November, 2010) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is mainly a bug fix release: * Fixed a bug in random retrieval (since 0.1.8). It occurs when reads are retrieved from a small region containing no reads. * Fixed a bug in pileup (since 0.1.9). The bug causes an assertion failure when the first CIGAR operation is a deletion. * Improved fault tolerence in remote access. One minor feature has been implemented in bcftools: * Added a reference-free variant calling mode. In this mode, a site is regarded as a variat iff the sample(s) contains two or more alleles; the meaning of the QUAL field in the VCF output is changed accordingly. Effectively, the reference allele is irrelevant to the result in the new mode, although the reference sequence has to be used in realignment when SAMtools computes genotype likelihoods. In addition, since 0.1.10, the `pileup' command has been deprecated by `mpileup' which is more powerful and more accurate. The `pileup' command will not be removed in the next few releases, but new features will not be added. (0.1.11: 21 November 2010, r851) Beta Release 0.1.10 (16 November, 2010) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This release is featured as the first major improvement to the indel caller. The method is similar to the old one implemented in the pileup command, but the details are handled more carefully both in theory and in practice. As a result, the new indel caller usually gives more accurate indel calls, though at the cost of sensitivity. The caller is implemented in the mpileup command and is invoked by default. It works with multiple samples. Other notable changes: * With the -r option, the calmd command writes the difference between the original base quality and the BAQ capped base quality at the BQ tag but does not modify the base quality. Please use -Ar to overwrite the original base quality (the 0.1.9 behavior). * Allow to set a maximum per-sample read depth to reduce memory. In 0.1.9, most of memory is wasted for the ultra high read depth in some regions (e.g. the chr1 centromere). * Optionally write per-sample read depth and per-sample strand bias P-value. * Compute equal-tail (Bayesian) credible interval of site allele frequency at the CI95 VCF annotation. * Merged the vcfutils.pl varFilter and filter4vcf for better SNP/indel filtering. (0.1.10: 16 November 2010, r829) Beta Release 0.1.9 (27 October, 2010) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This release is featured as the first major improvement to the samtools' SNP caller. It comes with a revised MAQ error model, the support of multi-sample SNP calling and the computation of base alignment quality (BAQ). The revised MAQ error model is based on the original model. It solves an issue of miscalling SNPs in repetitive regions. Althought such SNPs can usually be filtered at a later step, they mess up unfiltered calls. This is a theoretical flaw in the original model. The revised MAQ model deprecates the orginal MAQ model and the simplified SOAPsnp model. Multi-sample SNP calling is separated in two steps. The first is done by samtools mpileup and the second by a new program, bcftools, which is included in the samtools source code tree. Multi-sample SNP calling also works for single sample and has the advantage of enabling more powerful filtration. It is likely to deprecate pileup in future once a proper indel calling method is implemented. BAQ is the Phred-scaled probability of a read base being wrongly aligned. Capping base quality by BAQ has been shown to be very effective in suppressing false SNPs caused by misalignments around indels or in low-complexity regions with acceptable compromise on computation time. This strategy is highly recommended and can be used with other SNP callers as well. In addition to the three major improvements, other notable changes are: * Changes to the pileup format. A reference skip (the N CIGAR operator) is shown as '<' or '>' depending on the strand. Tview is also changed accordingly. * Accelerated pileup. The plain pileup is about 50% faster. * Regional merge. The merge command now accepts a new option to merge files in a specified region. * Fixed a bug in bgzip and razip which causes source files to be deleted even if option -c is applied. * In APIs, propogate errors to downstream callers and make samtools return non-zero values once errors occur. (0.1.9: 27 October 2010, r783) Beta Release 0.1.8 (11 July, 2010) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable functional changes: * Added the `reheader' command which replaces a BAM header with a new header. This command is much faster than replacing header by BAM->SAM->BAM conversions. * Added the `mpileup' command which computes the pileup of multiple alignments. * The `index' command now stores the number of mapped and unmapped reads in the index file. This information can be retrieved quickly by the new `idxstats' command. * By default, pileup used the SOAPsnp model for SNP calling. This avoids the floating overflow in the MAQ model which leads to spurious calls in repetitive regions, although these calls will be immediately filtered by varFilter. * The `tview' command now correctly handles CIGARs like 7I10M and 10M1P1I10M which cause assertion failure in earlier versions. * Tview accepts a region like `=10,000' where `=' stands for the current sequence name. This saves typing for long sequence names. * Added the `-d' option to `pileup' which avoids slow indel calling in ultradeep regions by subsampling reads locally. * Added the `-R' option to `view' which retrieves alignments in read groups listed in the specified file. Performance improvements: * The BAM->SAM conversion is up to twice faster, depending on the characteristic of the input. * Parsing SAM headers with a lot of reference sequences is now much faster. * The number of lseek() calls per query is reduced when the query region contains no read alignments. Bug fixes: * Fixed an issue in the indel caller that leads to miscall of indels. Note that this solution may not work well when the sequencing indel error rate is higher than the rate of SNPs. * Fixed another issue in the indel caller which may lead to incorrect genotype. * Fixed a bug in `sort' when option `-o' is applied. * Fixed a bug in `view -r'. APIs and other changes: * Added iterator interfaces to random access and pileup. The callback interfaces directly call the iterator interfaces. * The BGZF blocks holding the BAM header are indepedent of alignment BGZF blocks. Alignment records shorter than 64kB is guaranteed to be fully contained in one BGZF block. This change is fully compatible with the old version of samtools/picard. Changes in other utilities: * Updated export2sam.pl by Chris Saunders. * Improved the sam2vcf.pl script. * Added a Python version of varfilter.py by Aylwyn Scally. (0.1.8: 11 July 2010, r613) Beta Release 0.1.7 (10 November, 2009) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes: * Improved the indel caller in complex scenariors, in particular for long reads. The indel caller is now able to make reasonable indel calls from Craig Venter capillary reads. * Rewrote single-end duplicate removal with improved performance. Paired-end reads are not touched. * Duplicate removal is now library aware. Samtools remove potential PCR/optical dupliates inside a library rather than across libraries. * SAM header is now fully parsed, although this functionality is not used in merging and so on. * In samtools merge, optionally take the input file name as RG-ID and attach the RG tag to each alignment. * Added FTP support in the RAZF library. RAZF-compressed reference sequence can be retrieved remotely. * Improved network support for Win32. * Samtools sort and merge are now stable. Changes in other utilities: * Implemented sam2vcf.pl that converts the pileup format to the VCF format. * This release of samtools is known to work with the latest Bio-Samtools Perl module. (0.1.7: 10 November 2009, r510) Beta Release 0.1.6 (2 September, 2009) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes: * In tview, do not show a blank screen when no reads mapped to the corresponding region. * Implemented native HTTP support in the BGZF library. Samtools is now able to directly open a BAM file on HTTP. HTTP proxy is also supported via the "http_proxy" environmental variable. * Samtools is now compitable with the MinGW (win32) compiler and the PDCurses library. * The calmd (or fillmd) command now calculates the NM tag and replaces MD tags if they are wrong. * The view command now recognizes and optionally prints FLAG in HEXs or strings to make a SAM file more friendly to human eyes. This is a samtools-C extension, not implemented in Picard for the time being. Please type `samtools view -?' for more information. * BAM files now have an end-of-file (EOF) marker to facilitate truncation detection. A warning will be given if an on-disk BAM file does not have this marker. The warning will be seen on BAM files generated by an older version of samtools. It does NO harm. * New key bindings in tview: `r' to show read names and `s' to show reference skip (N operation) as deletions. * Fixed a bug in `samtools merge -n'. * Samtools merge now optionally copies the header of a user specified SAM file to the resultant BAM output. * Samtools pileup/tview works with a CIGAR with the first or the last operation is an indel. * Fixed a bug in bam_aux_get(). Changes in other utilies: * Fixed wrong FLAG in maq2sam. (0.1.6: 2 September 2009, r453) Beta Release 0.1.5 (7 July, 2009) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes: * Support opening a BAM alignment on FTP. Users can now use "tview" to view alignments at the NCBI ftp site. Please read manual for more information. * In library, propagate errors rather than exit or complain assertion failure. * Simplified the building system and fixed compiling errors caused by zlib<1.2.2.1. * Fixed an issue about lost header information when a SAM is imported with "view -t". * Implemented "samtool.pl varFilter" which filters both SNPs and short indels. This command replaces "indelFilter". * Implemented "samtools.pl pileup2fq" to generate FASTQ consensus from pileup output. * In pileup, cap mapping quality at 60. This helps filtering when different aligners are in use. * In pileup, allow to output variant sites only. * Made pileup generate correct calls in repetitive region. At the same time, I am considering to implement a simplified model in SOAPsnp, although this has not happened yet. * In view, added '-u' option to output BAM without compression. This option is preferred when the output is piped to other commands. * In view, added '-l' and '-r' to get the alignments for one library or read group. The "@RG" header lines are now partially parsed. * Do not include command line utilities to libbam.a. * Fixed memory leaks in pileup and bam_view1(). * Made faidx more tolerant to empty lines right before or after FASTA > lines. Changes in other utilities: * Updated novo2sam.pl by Colin Hercus, the key developer of novoalign. This release involves several modifications to the key code base which may potentially introduce new bugs even though we have tried to minimize this by testing on several examples. Please let us know if you catch bugs. (0.1.5: 7 July 2009, r373) Beta Release 0.1.4 (21 May, 2009) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes: * Added the 'rmdupse' command: removing duplicates for SE reads. * Fixed a critical bug in the indel caller: clipped alignments are not processed correctly. * Fixed a bug in the tview: gapped alignment may be incorrectly displayed. * Unified the interface to BAM and SAM I/O. This is done by implementing a wrapper on top of the old APIs and therefore old APIs are still valid. The new I/O APIs also recognize the @SQ header lines. * Generate the MD tag. * Generate "=" bases. However, the indel caller will not work when "=" bases are present. * Enhanced support of color-read display (by Nils Homer). * Implemented the GNU building system. However, currently the building system does not generate libbam.a. We will improve this later. For the time being, `make -f Makefile.generic' is preferred. * Fixed a minor bug in pileup: the first read in a chromosome may be skipped. * Fixed bugs in bam_aux.c. These bugs do not affect other components as they were not used previously. * Output the 'SM' tag from maq2sam. (0.1.4: 21 May 2009, r297) Beta Release 0.1.3 (15 April, 2009) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes in SAMtools: * SAMtools is more consistent with the specification: a) '*' in the QUAL field is allowed; b) the field separator is TAB only and SPACE is treated as a character in a field; c) empty header is allowed. * Implemented GLFv3 support in pileup. * Fixed a severe bug in fixmate: strand information is wrongly overwritten. * Fixed a bug in alignment retrieval: alignments bridging n*16384bp are not correctly retrieved sometimes. * Fixed a bug in rmdup: segfault if unmapped reads are present. * Move indel_filter.pl to samtools.pl and improved the filtering by checking the actual number of alignments containing indels. The indel pileup line is also changed a little to make this filtration easier. * Fixed a minor bug in indexing: the bin number of an unmapped read is wrongly calculated. * Added `flagstat' command to show statistics on the FLAG field. * Improved indel caller by setting the maximum window size in local realignment. Changes in other utilities: * Fixed a bug in maq2sam: a tag name is obsolete. * Improvement to wgsim: a) added support for SOLiD read simulation; b) show the number of substitutions/indels/errors in read name; c) considerable code clean up. * Various converters: improved functionality in general. * Updated the example SAM due to the previous bug in fixmate. (0.1.3: 15 April 2009, r227) Beta Release 0.1.2 (28 January, 2008) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Notable changes in SAMtools: * Implemented a Bayesian indel caller. The new caller generate scores and genotype and is potentially more accurate than Maq's indel caller. The pileup format is also changed accordingly. * Implemented rmdup command: remove potential PCR duplicates. Note that this command ONLY works for FR orientation and requires ISIZE is correctly set. * Added fixmate command: fill in mate coordinates, ISIZE and mate related flags from a name-sorted alignment. * Fixed a bug in indexing: reads bridging 16x kbp were not retrieved. * Allow to select reads shown in the pileup output with a mask. * Generate GLFv2 from pileup. * Added two more flags for flagging PCR/optical duplicates and for QC failure. * Fixed a bug in sort command: name sorting for large alignment did not work. * Allow to completely disable RAZF (using Makefile.lite) as some people have problem to compile it. * Fixed a bug in import command when there are reads without coordinates. * Fixed a bug in tview: clipping broke the alignment viewer. * Fixed a compiling error when _NO_CURSES is applied. * Fixed a bug in merge command. Changes in other utilities: * Added wgsim, a paired-end reads simulator. Wgsim was adapted from maq's reads simulator. Colin Hercus further improved it to allow longer indels. * Added wgsim_eval.pl, a script that evaluates the accuracy of alignment on reads generated by wgsim. * Added soap2sam.pl, a SOAP2->SAM converter. This converter does not work properly when multiple hits are output. * Added bowtie2sam.pl, a Bowtie->SAM converter. Only the top hit will be retained when multiple hits are present. * Fixed a bug in export2sam.pl for QC reads. * Support RG tag at MAQ->SAM converter. * Added novo2sam.pl, a NovoAlign->SAM converter. Multiple hits and indel are not properly handled, though. * Added zoom2sam.pl, a ZOOM->SAM converter. It only works with the default Illumina output. (0.1.2: 28 January 2008; r116) Beta Release 0.1.1 (22 December, 2008) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The is the first public release of samtools. For more information, please check the manual page `samtools.1' and the samtools website http://samtools.sourceforge.net