htslib 标签 - Gitee.com

htslib release 1.21:

The primary user-visible changes in this release are updates to
the annot-tsv tool and some speed improvements.  Full details of
other changes and bugs fixed are below.

Notice: this is the last SAMtools / HTSlib release where CRAM 3.0
will be the default CRAM version.  From the next we will change to
CRAM 3.1 unless the version is explicitly specified, for example
using "samtools view -O cram,version=3.0".

Updates
-------

* Extend annot-tsv with several new command line options.
    --delim permits use of other delimiters.
    --headers for selection of other header formats.
    --no-header-idx to suppress column index numbers in header.
  Also removed -h as it is now short for --headers.  Note --help
  still works. (PR #1779)

* Allow annot-tsv -a to rename annotations. (PR #1709)

* Extend annot-tsv --overlap to be able to specify the overlap
  fraction separately for source and target. (PR #1811)

* Added new APIs to facilitate low-level CRAM container
  manipulations, used by   the new "samtools cat" region
  filtering code. Functions are:
    cram_container_get_coords()
    cram_filter_container()
    cram_index_extents()
    cram_container_num2offset()
    cram_container_offset2num()
    cram_num_containers()
    cram_num_containers_between()
  Also improved cram_index_query() to cope with HTS_IDX_NOCOOR
  regions.  (PR #1771)

* Bgzip now retains file modification and access times when
  compressing and decompressing. (PR #1727, fixes #1718. 
  Requested by Gert Hulselmans.)

* Use FNV1a for string hashing in khash.  The old algorithm was
  particularly weak with base-64 style strings and lead to a large
  number of collisions.  (PR #1806.  Fixes samtools/samtools#2066,
  reported by Hans-Joachim Ruscheweyh)

* Improve the speed of the nibble2base() function on Intel (PR #1667,
  PR #1764, PR #1786, PR #1802, thanks to Ruben Vorderman) and ARM
  (PR #1795, thanks to John Marshall).

* bgzf_getline() will now warn if it encounters UTF-16 data. (PR
  #1487, thanks to John Marshall)

* Speed up bgzf_read().  While this does not reduce CPU
  significantly, it does increase the maximum parallelism
  available permitting 10-15% faster decoding. (PR #1772, PR
  #1800, Issue #1798)

* Speed up faidx by use of better isgraph methods (PR #1797) and
  whole-line reading (PR #1799, thanks to John Marshall).

* Speed up kputll() function, speeding up BAM -> SAM conversion by
  about 5% and also samtools depth.  (PR #1805)

* Added more example code, covering fasta/fastq indexing, tabix
  indexing and use of the thread pool. (PR #1666)

Build Changes
-------------

* Code warning fixes for pedantic compilers (PR #1777) and avoid some
  undefined behaviour (PR #1810, PR #1816, PR #1828).

* Windows based CI has been migrated from AppVeyor to GitHub Actions.
  (PR #1796, PR #1803, PR #1808)

* Miscellaneous minor build infrastructure and code fixes. (PR #1807,
  PR #1829, both thanks to John Marshall)

* Updated htscodecs submodule to version 1.6.1 (PR #1828)

* Fixed an awk script in the Makefile that only worked with gawk. (PR
  #1831)

Bug fixes
---------

* Fix small OSS-Fuzz reported issues with CRAM encoding and long
  CIGARS and/or illegal positions. (PR #1775, PR #1801, PR #1817)

* Fix issues with on-the-fly indexing of VCF/BCF (bcftools
  --write-index) when not using multiple threads. (PR #1837.
  Fixes samtools/bcftools#2267, reported by Giulio Genovese)

* Stricter limits on POS / MPOS / TLEN in sam_parse1().  This fixes a
  signed overflow reported by OSS-Fuzz and should help prevent other
  as-yet undetected bugs. (PR #1812)

* Check that the underlying file open worked for preload: URLs. 
  Fixes a NULL pointer dereference reported by OSS-Fuzz. (PR #1821)

* Fix an infinite loop in hts_itr_query() when given extremely large
  positions which cause integer overflow.  Also adds hts_bin_maxpos()
  and hts_idx_maxpos() functions. (PR #1774, thanks to John Marshall
  and reported by Jesus Alberto Munoz Mesa)

* Fix an out of bounds read in hts_itr_multi_next() when switching
  chromosomes.  This bug is present in releases 1.11 to 1.20. (PR
  #1788. Fixes samtools/samtools#2063, reported by acorvelo)

* Work around parsing problems with colons in CHROM names. Fixes
  samtools/bcftools#2139.  (PR #1781, John Marshall / James Bonfield)

* Correct the CPU detection for Mac OS X 10.7.  cpuid is used by
  htscodecs (see samtools/htscodecs#116), and the corresponding
  changes in htslib are PR #1785.  Reported by Ryan Carsten Schmidt.

* Make BAM zero-length intervals work the same as CRAM; permitted
  and returning overlapping records. (PR #1787.  Fixes
  samtools/samtools#2060, reported by acorvelo)

* Replace assert() with abort() in BCF synced reader.  This is not an
  ideal solution, but it gives consistent behaviour when compiling
  with or without NDEBUG.  (PR #1791, thanks to Martin Pollard)

* Fixed failure to change the write block size on compressed SAM or
  VCF files due to an internal type confusion.  (PR #1826)

* Fixed an out-of-bounds read in cram_codec_iter_next() (PR #1832)

0f8bedc

2024-09-12 23:06

下载

1.20

htslib release 1.20:

Updates
-------

* When working on named files, bgzip now sets the modified and
  access times of the output files it makes to match those of the
  corresponding input. (PR #1727, feature request #1718.  Requested
  by Gert Hulselmans)

* It's now possible to use a -o option to specify the output file
  name in bgzip. (PR #1747, feature request #1726.  Requested by
  Gert Hulselmans)

* Improved error faidx error messages. (PR #1743, thanks to
  Nick Moore)

* Faster reading of SAM array (type "B") tags.  These often turn up
  in ONT and PacBio data. (PR #1741)

* Improved validity checking of base modification tags. (PR #1749)

* mpileup overlap removal now works where one read has a deletion.
  (PR #1751, fixes samtools/samtools#1992.  Reported by Long Tian)

* The S3 plugin can now find buckets via S3 access point aliases. (PR
  #1756, thanks to Matt Pawelczyk; fixes samtools/samtools#1984. 
  Reported by Albert Li)

* Added a --threads option (and -@ short option) to tabix. (PR #1755,
  feature request #1735.  Requested by Dan Bolser)

* tabix can now index Graph Alignment Format (GAF) files. (See
  https://github.com/lh3/gfatools/blob/master/doc/rGFA.md) (PR
  #1763, thanks to Adam Novak)

Bug fixes
---------

* Security fix: Prevent possible heap overflow in cram_encode_aux()
  on bad RG:Z tags. (PR #1737)

* Security fix: Prevent attempts to call a NULL pointer if certain
  URL schemes are used in CRAM @SQ UR: tags. (PR #1757)

* Security fix: Fixed a bug where following certain AWS S3
  redirects could downgrade the connection from TLS (i.e.
  https://) to unencrypted http://. This could happen when using
  path-based URLs and AWS_DEFAULT_REGION was set to a region
  other that the one where the data was stored. (PR #1762, fixes
  #1760. Reported by andaca)

* Fixed arithmetic overflow when loading very long references for
  CRAM. (PR #1738, fixes #1738.  Reported by Shane McCarthy)

* Fixed faidx and CRAM reference look-ups on compressed fasta where
  the .fai index file was present, but the .gzi index of compressed
  offsets was not. (PR #1745, fixes #1744.  Reported by Theodore Li)

* Fixed BCF indexing on-the-fly bug which produced invalid indexes
  when using multiple compression threads. (PR #1742, fixes #1740. 
  Reported by graphenn)

* Ensure that pileup destructors are called by bam_plp_destroy(), to
  prevent memory leaks. (PR #1749, PR #1754)

* Ensure on-the-fly index timestamps are always older than the data
  file. Previously the files could be closed out of order, leading
  to warnings being printed when using the index. (PR #1753, fixes
  #1732.  Reported by Gert Hulselmans)

* To prevent data corruption when reading (strictly invalid) VCF
  files with duplicated FORMAT tags, all but the first copy of the
  data associated with the tag are now dropped with a warning. (PR
  #1752, PR #1761, fixes #1733.  Reported by anthakki)

* Fixed a bug introduced in release 1.19 (PR #1689) which broke
  variant record data if it tried to remove an over-long tag.
  (PR #1752, PR #1761)

* Changed error to warning when complaining about use of the CG tag
  in SAM or CRAM files. (PR #1758, fixes samtools/samtools#2002)

0cadce2

2024-04-15 22:09

下载

1.19.1

6a7d33a

2024-01-22 17:29

下载

1.19

htslib release 1.19:

Updates
-------

* A temporary work-around has been put in the VCF parser so that
  it is less likely to fail on rows with a large number of ALT
  alleles, where Number=G tags like PL can expand beyond the 2Gb
  limit enforced by HTSlib.  For now, where this happens the
  offending tag will be dropped so the data can be processed,
  albeit without the likelihood data.

In future work, the library will instead convert such tags into
  their local alternatives
  (see https://github.com/samtools/hts-specs/pull/434).

* New program. Adds annot-tsv which annotates regions in a
  destination file with texts from overlapping regions in a
  source file. (PR#1619)

* Change bam_parse_cigar() so that it can modify existing BAM
  records.  This makes more useful as public API.  Previously it
  could only handle partially formed BAM records. (PR#1651, fixes
  #1650. Reported by Oleksii Nikolaienko)

* Add "uncompressed" to hts_format_description() where appropriate. 
  This adds an "uncompressed" description to uncompressed files that
  would normally be compressed, such as BAM and BCF. (PR#1656, in
  relation to samtools#1884.  Thanks to John Marshall)

* Speed up to the VCF parser and writer. (PR#1644 and PR#1663)

* Add an hclen (hard clip length) SAM filter function. (PR#1660, with
  reference to samtools#813)

* Avoid really closing stdin/stdout in hclose()/hts_close()/et
  al. See discussion in PR for details. (PR#1665.  Thanks to
  John Marshall)

* Add support to handle multiple files in bgzip. (PR#1658, fixes
  #1642.  Requested by @bw2)

* Enable auto-vectorisation in CRAM 3.1 codecs.  Speeds decoding on
  some sequencing platform data. (PR#1669)

* Speed up removal of lines in large headers. (PR#1662, fixes #1460. 
  Reported by Anže Starič)

* Apply seqtk PR to improve kseq.h parsing performance.  Port of
  Fabian Klötzl's (kloetzl) lh3/seqtk#123 and
  attractivechaos/klib#173 to HTSlib. (PR#1674.  Thanks to
  John Marshall)

Build changes
-------------

* Updated htscodecs submodule to 1.6.0. (PR#1685, PR#1717, PR#1719)

* Apply the packed attribute to uint*_u types for Clang to prevent
  -fsanitize=alignment failures. (PR#1667.  Thanks to Fangrui Song)

* Fuzz testing improvements. (PR#1664)

* Add C++ casts for external headers in klist.h and kseq.h. (PR#1683.
   See also PR#1674 and PR#1682)

* Add test case compiling the public headers as C++. (PR#1682. 
  Thanks to John Marshall)

* Enable optimisation level -O3 for SAM QUAL+33 formatting. (PR#1679)

* Make compiler flag detection work with zig cc. (PR#1687)

* Fix unused value warnings when built with NDEBUG. (PR#1688)

* Remove some disused Makefile variables, fix typos and a warning. 
  Improve bam_parse_basemod() documentation. (PR#1705, Thanks to
  John Marshall)

Bug fixes
---------

* Fail bgzf_useek() when offset is above block limits. (PR#1668)

* Fix multi-threaded on-the-fly indexing problems. (PR#1672, fixes
  samtools#1861 and bcftools#1985.  Reported by Mark Ebbert
  and @lacek)

* Fix hfile_libcurl small seek bug. (PR#1676, fixes samtools#1918.
  Also may fix #1037, #1625 and samtools#1622. Reported by
  Alex Reynolds, Mark Walker, Arthur Gilly and skatragadda-nygc. Thanks
  to John Marshall)

* Fix a minor memory leak in malformed CRAM EXTERNAL blocks. [fuzz]
  (PR#1671)

* Fix a cram decode hang from block_resize(). (PR#1680. Reported by
  Sebastian Deorowicz)

* Cram fuzzing improvements.  Fixes a number of cram errors.
  (PR#1701, fixes #1691, #1692, #1693, #1696, #1697, #1698, #1699
  and #1700. Thanks to Octavio Galland for finding and reporting
  all these)

* Fix crypt4gh redirection. (PR#1675, fixes 
  grbot/crypt4gh-tutorial#2.  Reported by @hth4)

* Fix PG header linking when records make a loop. (PR#1702, fixes
  #1694.  Reported by Octavio Galland)

* Prevent issues with no-stored-sequence records in CRAM files,
  by ensuring they are accounted for properly in block size
  calculations, and by limiting the maximum query length in the
  CIGAR data.  Originally seen as an overflow by OSS-Fuzz /
  UBSAN, it turned out this could lead to excessive time and
  memory use by HTSlib, and could result in it writing out
  unreadable CRAM files. (PR#1710)

* Fix some illegal shifts and integer overflows found by OSS-Fuzz /
  UBSAN. (PR#1707, PR#1712, PR#1713)

8f72310

2023-12-12 22:02

下载

1.18

htslib release 1.18:

Updates
-------

* Using CRAM 3.1 no longer gives a warning about the specification
  being draft.  Note CRAM 3.0 is still the default output format.
  (PR#1583)

* Replaced use of sprintf with snprintf, to silence potential
  warnings from Apple's compilers and those who implement similar
  checks. (PR#1594, fixes #1586. Reported by Oleksii Nikolaienko)

* Fastq output will now generate empty records for reads with no
  sequence data (i.e. sequence is "*" in SAM format). (PR#1576,
  fixes samtools/samtools#1576.  Reported by Nils Homer)

* CRAM decoding speed-ups. (PR#1580)

* A new MN aux tag can now be used to verify that MM/ML base
  modification data has not been broken by hard clipping.
  (PR#1590, PR#1612. See also PR samtools/hts-specs#714 and issue
  samtools/hts-specs#646. Reported by Jared Simpson)

* The base modification API has been improved to make it easier for
  callers to tell unchecked bases from unmodified ones. (PR#1636,
  fixes #1550. Requested by Chris Wright)

* A new bam_mods_queryi() API has been added to return additional
  data about the i-th base modification returned by
  bam_mods_recorded(). (PR#1636, fixes #1550 and #1635. 
  Requested by Jared Simpson)

* Speed up index look-ups for whole-chromosome queries. (PR#1596)

* Mpileup now merges adjacent (mis)match CIGAR operations, so CIGARs
  using the X/= operators give the same results as if the M operator
  was used.  (PR#1607, fixes #1597.  Reported by Marcel Martin)

* It's now possible to call bcf_sr_set_regions() after adding
  readers using bcf_sr_add_reader() (previously this returned an
  error).  Doing so will discard any unread data, and reset the
  readers so they iterate over the new regions.  (PR#1624, fixes
  samtools/bcftools#1918.  Reported by Gregg Thomas)

* The synced BCF reader can now accept regions with reference names
  including colons and hyphens, by enclosing them in curly braces. 
  For example, {chr_part:1-1001}:10-20 will return bases 10 to 20
  from reference "chr_part:1-1001".  (PR#1630, fixes #1620.  Reported
  by Bren)

* Add a "samples" directory with code demonstrating usage of HTSlib
  plus a tutorial document. (PR#1589)

Build changes
-------------

* Htscodecs has been updated to 1.5.1 (PR#1654)

* Htscodecs SIMD code now works with Apple multiarch binaries.
  (PR#1587, HTSlib fix for samtools/htscodecs#76. Reported by
  John Marshall)

* Improve portability of "expr" usage in version.sh. (PR#1593, fixes
  #1592.  Reported by John Marshall)

* Improve portability to *BSD targets by ensuring _XOPEN_SOURCE
  is defined correctly and that source files properly include
  "config.h".  Perl scripts also now all use #!/usr/bin/env
  instead of assuming that it's in /usr/bin/perl. (PR#1628, fixes
  #1606. Reported by Robert Clausecker)

* Fixed NAME entry in htslib-s3-plugin man page so the whatis and
  apropos commands find it.  (PR#1634, thanks to Étienne Mollier)

* Assorted dependency tracking fixes.  (PR#1653, thanks to
  John Marshall)

Documentation updates
---------------------

* Changed Alpine build instructions as they've switched back to using
  openssl. (PR#1609)

* Recommend using -rdynamic when statically linking a libhts.a with
  plugins enabled.  (PR#1611, thanks to John Marshall.  Fixes #1600,
  reported by Jack Wimberley)

* Fixed example in docs for sam_hdr_add_line().  (PR#1618, thanks
  to kojix2)

* Improved test harness for base modifications API. (PR#1648)

Bug fixes
---------

* Fix a major bug when searching against a CRAM index where one
  container has start and end coordinates entirely contained within
  the previous container. This would occasionally miss data, and
  sometimes return much more than required.  The bug affected
  versions 1.11 to 1.17, although the change in 1.11 was bug-fixing
  multi-threaded index queries.  This bug did not affect index
  building.  There is no need to reindex your CRAM files. (PR#1574,
  PR#1640. Fixes #1569, #1639, samtools/samtools#1808,
  samtools/samtools#1819.  Reported by xuxif, Jens Reeder and
  Jared Simpson)

* Prevent CRAM blocks from becoming too big in files with short
  sequences but very long aux tags.  (PR #1613)

* Fix bug where the CRAM decoder for CONST_INT and CONST_BYTE
  codecs may incorrectly look for extra data in the CORE block.
  Note that this bug only affected the experimental CRAM v4.0
  decoder. (PR#1614)

* Fix crypt4gh redirection so it works in conjunction with non-file
  IO, such as using htsget. (PR#1577)

* Improve error checking for the VCF POS column, when facing invalid
  data. (PR#1575, replaces #1570 originally reported and fixed by
  Colin Nolan.)

* Improved error checking on VCF indexing to validate the data is
  BGZF compressed. (PR#1581)

* Fix bug where bin number calculation could overflow when making
  iterators over regions that go to the end of a chromosome.
  (PR#1595)

* Backport attractivechaos/klib#78 (by Pall Melsted) to HTSlib.
  Prevents infinite loops in kseq_read() when reading broken gzip
  files. (PR#1582, fixes #1579.  Reported by Goran Vinterhalter)

* Backport attractivechaos/klib@384277a (by innoink) to HTSlib. Fixes
  the kh_int_hash_func2() macro definition. (PR#1599, fixes #1598. 
  Reported by fanxinping)

* Remove a compilation warning on systems with newer libcurl
  releases. (PR#1572)

* Windows: Fixed BGZF EOF check for recent MinGW releases. (PR#1601,
  fixes samtools/bcftools#1901)

* Fixed bug where tabix would not return the correct regions for
  files where the column ordering is end, ..., begin instead of
  begin, ..., end. (PR#1626, fixes #1622.  Reported by
  Hiruna Samarakoon)

* sam_format_aux1() now always NUL-terminates Z/H tags. (PR#1631)

* Ensure base modification iterator is reset when no MM tag is
  present. (PR#1631, PR#1647)

* Fix segfault when attempting to write an uncompressed BAM file
  opened using hts_open(name, "wbu").  This was attempting to write
  BAM data without wrapping it in BGZF blocks, which is invalid
  according to the BAM specification.  "wbu" is now internally
  converted to "wb0" to output uncompressed data wrapped in BGZF
  blocks. (PR#1632, fixes #1617. Reported by Joyjit Daw)

* Fixed over-strict bounds check in probaln_glocal() which caused it
  to make sub-optimal alignments when the requested band width was
  greater than the query length.  (PR#1616, fixes #1605.  Reported by
  Jared Simpson)

* Fixed possible double frees when handling errors in
  bcf_hdr_add_hrec(), if particular memory allocations fail.
  (PR#1637)

* Ensure that bcf_hdr_remove() clears up all pointers to the items
  removed from dictionaries.  Failing to do this could have resulted
  in a call requesting a deleted item via bcf_hdr_get_hrec()
  returning a stale pointer. (PR#1637)

* Stop the gzip decompresser from finishing prematurely when an empty
  gzip block is followed by more data. (PR#1643, PR#1646)

d8ca374

2023-07-25 16:16

下载

1.17

htslib release 1.17:

* A new API for iterating through a BAM record's aux field. (PR#1354,
  addresses #1319.  Thanks to John Marshall)

* Text mode for bgzip. Allows bgzip to compress lines of text with
  block breaks at newlines. (PR#1493, thanks to Mike Lin for the
  initial version PR#1369)

* Make tabix support CSI indices with large positions.  Unlike SAM
  and VCF files, BED files do not set a maximum reference length
  which hindered CSI support.  This change sets an arbitrary large
  size of 100G to enable it to work. (PR#1506)

* Add a fai_line_length function.  Exposes the internal line-wrap
  length. (PR#1516)

* Check for invalid barcode tags in fastq output. (PR#1518, fixes
  samtools#1728.  Reported by Poshi)

* Warn if reference found in a CRAM file is not contained in the
  specified reference file. (PR#1517 and PR#1521, adds diagnostics
  for #1515. Reported by Wei WeiDeng)

* Add a faidx_seq_len64 function that can return sequence lengths
  longer than INT_MAX.  At the same time limit faidx_seq_len to
  INT_MAX output.  Also add a fai_adjust_region to ensure given
  ranges do not go beyond the end of the requested sequence.
  (PR#1519)

* Add a bcf_strerror function to give text descriptions of BCF
  errors. (PR#1510)

* Add CRAM SQ/M5 header checking when specifying a fasta file. 
  This is to prevent creating a CRAM that cannot be decoded again.
  (PR#1522.  In response to samtools#1748 though not a direct fix)

* Improve support for very long input lines (> 2Gbyte).  This is
  mostly useful for tabix which does not do much interpretation of
  its input. (PR#1542, a partial fix for #1539)

* Speed up load_ref_portion.  This function has been sped up by about
  7x, which speeds up low-depth CRAM decoding by about 10%. (PR#1551)

* Expand CRAM API to cope with new samtools cram_size command.
  (PR#1546)

* Merges neighbouring I and D ops into one op within pileup. This
  means 4M1D1D1D3M is reported as 4M3D3M.   Fixing this in sam.c
  means not only is samtools mpileup now looking better, but any
  tool using the mpileup API will be getting consistent results.
  (PR#1552, fixes the last remaining part of samtools#139)

* Update the API documentation for bgzf_mt as it refered to a
  previous iteration. (PR#1556, fixes #1553.  Reported by
  Raghavendra Padmanabhan)

Build changes
-------------

* Use POSIX grep in testing as egrep and fgrep are considered
  obsolete. (PR#1509, thanks to David Seifert)

* Switch to building libdefalte with cmake for Cirris CI. (PR#1511)

* Ensure strings in config_vars.h are escaped correctly. (PR#1530,
  fixes #1527. Reported by Lucas Czech)

* Easier modification of shared library permissions during install.
  (PR#1532, fixes #1525. Reported by StephDC)

* Fix build on ancient compilers.  Added -std=gnu90 to build tests
  so older C compilers will still be happy. (PR#1524, fixes #1523.
   Reported by Martin Jakt)

* Switch MacOS CI tests to an ARM-based image. (PR#1536)

* Cut down the number of embed_ref=2 tests that get run. (PR#1537)

* Add symbol versions to libhts.so.  This is to aid package
  developers. (PR#1560 addresses #1505, thanks to John Marshall.
  Reported by Stefan Bruens)

* htscodecs now updated to v1.4.0. (PR#1563)

* Cleaned up misleading system error reports in test_bgzf. (PR#1565)

Bug fixes
---------

* VCF. Fix n-squared complexity in sample line with many adjacent
  tabs [fuzz]. (PR#1503)

* Improved bcftools detection and reporting of bgzf decode
  errors. (PR#1504, thanks to Lilian Janin. PR#1529 thanks to
  Bergur Ragnarsson, fixes #1528. PR#1554)

* Prevent crash when the only FASTA entry has no sequence [fuzz].
  (PR#1507)

* Fixed typo in sam.h documentation. (PR#1512, thanks to kojix2)

* Fix buffer read-overrun in bam_plp_insertion_mod. (PR#1520)

* Fix hash keys being left behind by bcf_hdr_remove. (PR#1535, fixes
  #1533.  Reported by Giulio Genovese in #842)

* Make bcf_hdr_idinfo_exists more robust by checking id value exists.
  (PR#1544, fixes #1538.  Reported by Giulio Genovese)

* CRAM improvements. Fixed crash with multi-threaded CRAM.  Fixed a
  bug in the codec parameter learning for CRAM 3.1 name tokeniser.
  Fixed Cram compression container substitution matrix generation,
  (PR#1558, PR#1559 and PR#1562)

6143086

2023-02-21 20:24

下载

1.16

htslib release 1.16:

* Make hfile_s3 refresh AWS credentials on expiry in order to make
  HTSlib work better with AWS IAM credentials, which have a limited
  lifespan. (PR#1462 and PR#1474, addresses #344)

* Allow BAM headers between 2GB and 4GB in size once more.  This
  is not permitted in the BAM specification but was allowed in an
  earlier version of HTSlib.  There is now a warning at 2GB and a
  hard failure at 4GB. (PR#1421, fixes #1420 and samtools#1613.
  Reported by John Marshall and R C Mueller)

* Improve error message when failing to load an index. (PR#1468,
  example of the problem samtools#1637)

* Permit MM (base modification) tags containing "." and "?" suffixes.
   These define implicit vs explicit coordinates.  See the SAM tags
  specification for details. (PR#1423 and PR#1426, fixes #1418. 
  PR#1469, fixes #1466.  Reported by cjw85)

* Warn if spaces instead of tabs are detected in a VCF file to
  prevent confusion. (PR#1328, fixes bcftools#1575.  Reported by
  ketkijoshi278)

* Add an "sclen" filter expression keyword.  This is the length
  of a soft-clip, both left and right end.  It may be combined
  with qlen (qlen-sclen) to obtain the number of bases in the
  query sequence that have been aligned to the genome ie it
  provides a way to compare local-alignment vs global-alignment
  length. (PR#1441 and PR/samtools#1661, fixes #1436. Requested
  by Chang Y)

* Improve error messages for CRAM reference mismatches.  If the user
  specifies the wrong reference, the CRAM slice header MD5sum checks
  fail.  We now report the SQ line M5 string too so it is possible to
  validate against the whole chr in the ref.fa file.  The error
  message has also been improved to report the reference name instead
  of #num.  Finally, we now hint at the likely cause, which counters
  the misleading samtools supplied error of "truncated or corrupt"
  file. (PR#1427, fixes samtools#1640.  Reported by Jian-Guo Zhou)

* Expose more of the CRAM API and add new functionality to extract
  the reference from a CRAM file. (PR#1429 and PR#1442)

* Improvements to the implementation of embedded references in CRAM
  where no external reference is specified. (PR#1449, addresses some
  of the issues in #1445)

* The CRAM writer now allows alignment records with RG:Z: aux tags
  that don't have a corresponding @RG ID in the file header. 
  Previously these tags would have been silently dropped.  HTSlib
  will complain whenever it has to add one though, as such tags do
  not conform to recommended practice for the SAM, BAM and CRAM
  formats. (PR#1480, fixes #1479.  Reported by Alex Leonard)

* Set tab delimiter in man page for tabix GFF3 sort. (PR#1457. 
  Thanks to Colin Diesh)

* When using libdeflate, the 1...9 scale of BGZF compression levels
  is now remapped to the 1...12 range used by libdeflate instead of
  being passed directly.  In particular, HTSlib levels 8 and 9 now
  map to libdeflate levels 10 and 12, so it is possible to select the
  highest (but slowest) compression offered by libdeflate. (PR#1488,
  fixes #1477.  Reported by Gert Hulselmans)

* The VCF variant API has been extended so that it can return
  separate flags for INS and DEL variants as well as the existing
  INDEL one.  These flags have not been added to the old
  bcf_get_variant_types() interface as it could break existing
  users.  To access them, it is necessary to use new functions
  bcf_has_variant_type() and bcf_has_variant_types(). (PR#1467)

* The missing, but trivial, `le_to_u8()` function has been added to
  hts_endian. (PR#1494, Thanks to John Marshall)

* bcf_format_gt() now works properly on big-endian platforms.
  (PR#1495, Thanks to John Marshall)

Build changes
-------------

These are compiler, configuration and makefile based changes.

* Update htscodecs to version 1.3.0 for new SIMD code + various
  fixes. Updates the htscodecs submodule and adds changes necessary
  to make HTSlib build the new SIMD codec implementations. (PR#1438,
  PR#1489, PR#1500)

* Fix clang builds under mingw.  Under mingw, clang requires
  dllexport to be applied to both function declarations and
  function definitions. (PR#1435, PR#1497, PR#1498 fixes #1433. 
  Reported by teepean)

* Fix curl type warning with gcc 12.1 on Windows. (PR#1443)

* Detect ARM Neon support and only build appropriate SIMD object
  files. (PR#1451, fixes #1450.  Thanks to John Marshall)

* `make print-config` now reports extra CFLAGS that are needed to
  build the SIMD parts of htscodecs.  These may be of use to
  third-party build systems that don't use HTSlib's or htscodecs'
  build infrastructure. (PR#1485. Thanks to John Marshall)

* Fixed some Makefile dependency issues for the "check"/"test"
  targets and plugins.  In particular, "make check" will now build
  the "all" target, if not done already, before running the tests.
  (PR#1496)

Bug fixes
---------

* Fix bug when reading position -1 in BCF (0 in VCF), which is
  used to indicate telomeric  regions.  The BCF reader was
  incorrectly assuming the value stored in the file was unsigned,
  so a VCF->BCF->VCF round-trip would change it from 0 to
  4294967296. (PR#1476, fixes #1475 and bcftools#1753.  Reported
  by Rodrigo Martin)

* Various bugs and quirks have been fixed in the filter expression
  engine, mostly related to the handling of absent tags, and the
  is_true flag. Note that as a result of these fixes, some filter
  expressions may give different results:

- Fixed and-expressions including aux tag values which could give
    an invalid true result depending on the order of terms.

- The expression `![NM]` is now true if only `NM` does not
    exist.  In earlier versions it would also report true for
    tags like `NM:i:0` which exist but have a value of zero.

- The expression `[X1] != 0` is now false when `X1` does not exist.
     Earlier versions would return true for this comparison when the
    tag was missing.

- NULL values due to missing tags now propagate through string,
    bitwise and mathematical operations.  Logical operations always
    treat them as false. (PR#1463, fixes samtools#1670.  Reported
    by Gert Hulselmans; PR#1478, fixes samtools#1677.  Reported by
    johnsonzcode)

* Fix buffer overrun in bam_plp_insertion_mod.  Memory now grows to
  the proper size needed for base modification data. (PR#1430, fixes
  samtools#1652.  Reported by hd2326)

* Remove limit of returned size from fai_retrieve(). (PR#1446, fixes
  samtools#1660.  Reported by Shane McCarthy)

* Cap hts_getline() return value at INT_MAX.  Prevents hts_getline()
  from returning a negative number (a fail) for very long string
  length values. (PR#1448.  Thanks to John Marshall)

* Fix breakend detection and test bcf_set_variant_type(). (PR#1456,
  fixes #1455.  Thanks to Martin Pollard)

* Prevent arrays of BCF_BT_NULL values found in BCF files from
  causing bcf_fmt_array() to call exit() as the type is unsupported.
   These are now tested for and caught by bcf_record_check(), which
  returns an error code instead.  (PR#1486)

* Improved detection of fasta and fastq files that have very long
  comments following identifiers.  (PR#1491, thanks to John Marshall.
  Fixes samtools/samtools#1689, reported by cjw85)

* Fixed a SEGV triggered by giving a SAM file to `samtools import`.
  (PR#1492)

4e61c12

2022-08-18 18:55

下载

1.15.1

6811ac5

2022-04-07 23:32

下载

1.15

htslib release 1.15:

Features and Updates
--------------------

* Bgzip now has a --keep option to not remove the input file after
  compressing. (PR#1331)

* Improved file format detection so some BED files are no longer
  detected as FASTQ or FASTA. (PR#1350, thanks to John Marshall)

* Added xz (lzma), zstd and D4 formats to the file type detection
  functions.  We don't actively support reading these data types, but
  function calls and htsfile can detect them. (PR#1340, thanks to
  John Marshall)

* CRAM now also uses libdeflate for read-names if the libdeflate
  version is new enough (1.9 onwards).  Previously we used zlib for
  this due to poor performance of libdeflate.  This gives a slight
  speed up and reduction in file size. (PR#1383)

* The VCF and BCF readers will now issue a warning if contig, INFO
  or FORMAT IDs do not match the formats described in the VCFv4.3
  specification. Note that while the invalid names will mostly still
  be accepted, future updates will convert the warnings to errors
  causing files including invalid names to be rejected.  (PR#1389)

Build changes
-------------

These are compiler, configuration and makefile based changes.

* HTSlib now uses libhtscodecs release 1.2.1.

* Improved support for compiling and linking against HTSlib with
  Microsoft Visual Studio. (PR#1380, #1377, #1375.  Thanks to
  Aidan Bickford and John Marshall)

* Various internal CI improvements.

Bug fixes
---------

* Fixed CRAM index queries for HTSJDK output (PR#1388, reported by
  Chris Norman).  Note this also fixes writing CRAM writing, to match
  the specification (and HTSJDK), from version 3.1 onwards.

* Fixed CRAM index queries when required-fields settings are selected
  to ignore CIGARs (PR#1372, reported by Giulio Genovese).

* Unmapped but placed (having chr/pos) are now included in the BAM
  indices. (PR#1352, thanks to John Marshall)

* CRAM now honours the filename##idx##index nomenclature for
  specifying non-standard index locations. (PR#1360, reported by
  Michael Cariaso)

* Minor CRAM v1.0 read-group fix (PR#1349, thanks to John Marshall)

* Permit .fa and .fq file type detection as synonyms for FASTA and
  FASTQ. (PR#1386).

* Empty VCF format fields are now output ":.:" as instead of "::".
  (PR#1370)

* Repeated bcf_sr_seek calls now work. (PR#1363, reported by
  Giulio Genovese)

* Bcf_remove_allele_set now works on unpacked BCF records. (PR#1358,
  reported by Brent Pedersen).

* The hts_parse_decimal() function used to read numbers in region
  lists is now better at rejecting non-numeric values.  In particular
  it now rejects a lone 'G' instead of interpreting it as '0G', i.e.
  zero. (PR#1396, PR#1400, reported by SSSimon Yang; thanks to
  John Marshall).

* Improve support for GPU issues listed by -Wdouble-promotion.
  (PR#1365, reported by David Seisert)

* Fix example code in header file documentation. (PR#1381, Thanks to
  Aidan Bickford)

762d1b1

2022-02-21 19:42

下载

1.14

htslib release 1.14:

Features and Updates
--------------------

* Added a keep option to bgzip to leave the original file untouched. 
  This brings bgzip into line with gzip. (PR #1331, thanks to
  Alex Petty)

* "endpos" has been added to the filter language, giving the position
  of the rightmost mapped base as measured by the CIGAR string.  For
  unmapped reads it is the same as "pos". (PR #1307, thanks to
  John Marshall)

* Interfaces have been added to interpret the new base modification
  tags added to the SAMtags document in samtools/hts-specs#418.  (PR
  #1132)

* New API functions hts_flush()/sam_flush()/bcf_flush() for flushing
  output htsFile/samFile/vcfFile streams. (PR #1326, thanks to
  John Marshall)

* The synced_bcf_reader now sorts lines with symbolic alleles by END
  tag as well as POS.  (PR #1321)

* Added synced_bcf_reader options BCF_SR_REGIONS_OVERLAP and
  BCF_SR_TARGETS_OVERLAP for better control of records that start
  outside the desired region but overlap it are handled.  Fixes
  samtools/bcftools#1420 and samtools/bcftools#1421 raised by
  John Marshall.  (PR #1327)

* HTSlib will now accept long-cigar CG:B: tags made by htsjdk
  which don't quite follow the specification properly (using
  signed values instead of unsigned).  Thanks to Colin Diesh for
  reporting an example file. (PR #1317)

* The warning printed when the BGZF reader finds a file with no EOF
  block has been changed to be less alarming.  Unfortunately some
  third-party BGZF encoders don't write EOF blocks at the end of
  files.  Thanks to Keiran Raine for reporting an example file.  (PR
  #1323)

* The FASTA and FASTQ readers get an option to skip over the first
  item on the header line, and use the second as the read name.  It
  allows the original name to be restored on some of the fastq files
  served from the European Nucleotide Archive (ENA).  (PR #1325)

* HTSlib is now more strict when parsing the VCF samples line
  (beginning #CHROM).  It will only accept tabs between the
  mandatory field names and sample names must be separated with
  tabs. (PR #1328)

* HTSlib will now warn if it looks like the header has been corrupted
  by diagnostic messages from the program that made it.  This can
  happen when using `nohup`, which by default mixes stdout and stderr
  into the same stream.  (PR#1339, thanks to John Marshall)

* File format detection will now recognise signatures for XZ, Zstd
  and D4 files (note that HTSlib will not read them yet).  (PR #1340,
  thanks to John Marshall)

Build changes
-------------

These are compiler, configuration and makefile based changes.

* Some redundant tests have been removed from the test harness,
  speeding it up. (PR #1308)

* The version.sh script now works better on shallow checkouts.  (PR
  #1324)

* A check-untracked Makefile target has been added to catch untracked
  files (mostly) left by the test harness.  (PR #1324)

Bug fixes
---------

* Fixed a case where flushing the thread pool could very occasionally
  cause a deadlock.  (PR #1309)

* Fixed a bug where some CRAM files could fail to decode if the
  required_fields option was in use.  Thanks to Matt Sexton for
  reporting the issue. (PR #1314, fixes samtools/samtools#1475)

* Fixed a regression where the S3 plugin could not read public
  files unless you supplied some Amazon credentials.  Thanks to
  Chris Saunders for reporting. (PR #1332,
  fixes samtools/samtools#1491)

* Fixed a possible CRAM thread deadlock discovered by @ryancaicse.
  (PR #1330, fixes #1329)

* Some set-but-unused variables have been removed.  (PR #1334)

* Fixed a bug which prevented "flag.read2" from working in the filter
  language unless it was at the end of the expression.  Thanks to
  Vamsi Kodali for reporting the issue.  (PR #1342)

* Fixed a memory leak that could happen if CRAM fails to inflate a
  LZMA block. (PR #1340, thanks to John Marshall)

c37e041

2021-10-22 19:14

下载

1.13

htslib release 1.13:

Features and Updates--------------------

* In case a PG header line has multiple ID tags supplied by other
  applications, the header API now selects the first one encountered
  as the identifying tag and issues a warning when detecting
  subsequent ID tags. (#1256; fixed samtools/samtools#1393)

* VCF header reading function (vcf_hdr_read) no longer tries to
  download a remote index file by default. (#1266; fixes #380)

* Support reading and writing FASTQ format in the same way as SAM,
  BAM or CRAM. Records read from a FASTQ file will be treated as
  unmapped data. (#1156)

* Added GCP requester pays bucket access.  Thanks to `@indraniel`.
  (#1255)

* Made mpileup's overlap removal choose which copy to remove at
  random instead of always removing the second one.  This avoids
  strand bias in experiments where the +ve and -ve strand reads
  always appear in the same order. (#1273; fixes
  samtools/bcftools#1459)

* It is now possible to use platform specific BAQ parameters.  This
  also selects long-read parameters for read lengths bigger than 1kb,
  which helps bcftools mpileup call SNPs on PacBio CCS reads. (#1275)

* Improved bcf_remove_allele_set.  This fixes a bug that stopped
  iteration over alleles prematurely, marks removed alleles as
  'missing' and does automatic lazy unpacking. (#1288; fixes #1259)

* Improved compression metrics for unsorted CRAM files.  This
  improves the choice of codecs when handling unsorted data. (#1291)

* Linear index entries for empty intervals are now initialised
  with the file offset in the next non-empty interval instead of
  the previous one.  This may reduce the amount of data iterators
  have to discard before reaching the desired region, when the
  starting location is in a sequence gap. Thanks to `@carsonh` for
  reporting the issue. (#1286; fixes #486)

* A new hts_bin_level API function has been added, to compute the
  level of a given bin in the binning index. (#1286)

* Related to the above, a new API method, hts_idx_nseq, now returns
  the total number of contigs from an index. (#1295 and #1299)

* Added bracket handling to bcf_hdr_parse_line, for use with ##META
  lines. Thanks to Alberto Casas Ortiz. (#1240)

Build changes-------------

These are compiler, configuration and makefile based changes.

* HTSlib now uses libhtscodecs release 1.1.1.

* Added a curl/curl.h check to configure and improved INSTALL
  documentation on build options.  Thanks to Melanie Kirsche and
  John Marshall. (#1265; fixes #1261)

* Some fixes to address GCC 11.1 warnings. (#1280, #1284, #1285;
  fixes #1283)

* Supports building HTSlib in a separate directory. Thanks to John
  Marshall. (#1277; fixes #231)

* Supports building HTSlib on MinGW 32-bit environments. Thanks to
  John Marshall. (#1301)

Bug fixes---------

* Fixed hts_itr_query() et al region queries: fixed bug introduced in
  HTSlib 1.12, which led to iterators producing very few reads for
  some queries (especially for larger target regions) when unmapped
  reads were present. HTSlib 1.11 had a related problem in which
  iterators would omit a few unmapped reads that should have been
  produced; cf #1142. Thanks to Daniel Cooke for reporting the issue.
  (#1281; fixes #1279)

* Removed compressBound assertions on opening bgzf files.  Thanks to
  Gurt Hulselmans for reporting the issue. (#1258; fixed #1257)

* Duplicate sample name error message for a VCF file now only
  displays the duplicated name rather the entire same name list.
  (#1262; fixes samtools/bcftools#1451)

* Fix to make samtools cat work on CRAMs again. (#1276; fixes
  samtools/samtools#1420)

* Fix for a double memory free in SAM header creation.  Thanks to
  `@ihsineme`. (#1274)

* Prevent assert in bcf_sr_set_regions.  Thanks to Dr K D Murray.
  (#1270)

* Fixed crash in knet_open() etc stubs.  Thanks to John Marshall.
  (#1289)

* Fixed filter expression "cigar" on unmapped reads.  Stop treating
  an empty CIGAR string as an error.  Thanks to Chang Y for reporting
  the issue. (#1298, fixes samtools/samtools#1445)

* Bug fixes in the bundled copy of htscodecs:

- Fixed an uninitialized access in the name tokeniser decoder.
    (samtools/htscodecs#23)

- Fixed a bug with name tokeniser and variable number of names per
    slice, causing it to incorrectly report an error on certain valid
    inputs. (samtools/htscodecs#24)

911cb8e

2021-07-08 01:18

下载

1.12

htslib release 1.12:

Features and Updates
--------------------

* Added experimental CRAM 3.1 and 4.0 support. (#929)

These should not be used for long term data storage as the
  specification still needs to be ratified by GA4GH and may be
  subject to changes in format. (This is highly likely for 4.0). 
  However it may be tested using:

test/test_view -t ref.fa -C -o version=3.1 in.bam -p out31.cram

For smaller but slower files, try varying the compression profile
  with an additional "-o small".  Profile choices are fast, normal,
  small and archive, and can be applied to all CRAM versions.

* Added a general filtering syntax for alignment records in
  SAM/BAM/CRAM readers. (#1181, #1203)

An example to find chromosome spanning read-pairs with high mapping
  quality:  'mqual >= 30 && mrname != rname'

To find significant sized deletions: 'cigar =~ "[0-9]{2}D"' or
  'rlen - qlen > 10'.

To report duplicates that aren't part of a "proper pair":
    'flag.dup && !flag.proper_pair'

More details are in the samtools.1 man page under "FILTER
  EXPRESSIONS".

* The knet networking code has been removed.  It only supported the
  http and ftp protocols, and a better and safer alternative using
  libcurl has been available since release 1.3.  If you need access
  to ftp:// and http:// URLs, HTSlib should be built with libcurl
  support. (#1200)

* The old htslib/knetfile.h interfaces have been marked as
  deprecated.  Any code still using them should be updated to use
  hFILE instead. (#1200)

* Added an introspection API for checking some of the capabilities
  provided by HTSlib. (#1170) Thanks also to John Marshall for
  contributions. (#1222)

- `hfile_list_schemes`: returns the number of schemes found

- `hfile_list_plugins`: returns the number of plugins found

- `hfile_has_plugin`: checks if a specific plugin is available

- `hts_features`: returns a bit mask with all available features

- `hts_test_feature`: test if a feature is available

- `hts_feature_string`: return a string summary of enabled
      features

* Made performance improvements to `probaln_glocal` method, which
  speeds up mpileup BAQ calculations. (#1188)

- Caching of reused loop variables and removal of loop invariants

- Code reordering to remove instruction latency.

- Other refactoring and tidyups.

* Added a public method for constructing a BAM record from the
  component pieces. Thanks to Anders Kaplan. (#1159, #1164)

* Added two public methods, `sam_parse_cigar` and `bam_parse_cigar`,
  as part of a small CIGAR API (#1169, #1182). Thanks to
  Daniel Cameron for input. (#1147)

* HTSlib, and the included htsfile program, will now recognise
  the old RAZF compressed file format.  Note that while the
  format is detected, HTSlib is unable to read it.  It is
  recommended that RAZF files are uncompressed with `gunzip`
  before using them with HTSlib.  Thanks to John Marshall
  (#1244); and Matthew J. Oldach who reported problems with
  uncompressing some RAZF files (samtools/samtools#1387).

* The S3 plugin now has options to force the address style.  It
  will recognise the addressing_style and host_bucket entries in
  the respective aws .credentials and s3cmd .s3cfg files.  There is
  also a new HTS_S3_ADDRESS_STYLE environment variable.  Details
  are in the htslib-s3-plugin.7 man file (#1249).

Build changes
-------------

These are compiler, configuration and makefile based changes.

* Added new Makefile targets for the applications that embed HTSlib
  and want to run its test suite or clean its generated artefacts.
  (#1230, #1238)

* The CRAM codecs are now obtained via the htscodecs submodule, hence
  when cloning it is now best to use "git clone --recursive".  In an
  existing clone, you may use "git submodule update --init" to obtain
  the htscodecs submodule checkout.

* Updated CI test configuration to recurse HTSlib submodules. (#1359)

* Added Cirrus-CI integration as a replacement for Travis, which was
  phased out.  (#1175; #1212)

* Updated the Windows image used by Appveyor to 'Visual Studio 2019'.
  (#1172; fixed #1166)

* Fixed a buglet in configure.ac, exposed by the release 2.70 of
  autoconf. Thanks to John Marshall. (#1198)

* Fixed plugin linking on macOS, to prevent symbol conflict when
  linking with a static HTSlib. Thanks to John Marshall. (#1184)

* Fixed a clang++9 error in `cram_io.h`. Thanks to Pjotr Prins.
  (#1190)

* Introduced $(ALL_CPPFLAGS) to allow for more flexibility in setting
  the compiler flags. Thanks to John Marshall. (#1187)

* Added 'fall through' comments to prevent warnings issued by Clang
  on intentional fall through case statements, when building with 
  `-Wextra flag`. Thanks to John Marshall. (#1163)

* Non-configure builds now define _XOPEN_SOURCE=600 to allow them
  to work when the `gcc -std=c99` option is used.  Thanks to
  John Marshall. (#1246)

Bug fixes
---------

* Fixed VCF `#CHROM` header parsing to only separate columns at tab
  characters. Thanks to Sam Morris for reporting the issue. (#1237;
  fixed samtools/bcftools#1408)

* Fixed a crash reported in `bcf_sr_sort_set`, which expects REF to
  be present. (#1204; fixed samtools/bcftools#1361)

* Fixed a bcf synced reader bug when filtering with a region
  list, and the first record for a chromosome had the same
  position as the last record for the previous chromosome.
  (#1254; fixed samtools/bcftools#1441)

* Fixed a bug in the overlapping logic of mpileup, dealing with
  iterating over CIGAR segments. Thanks to `@wulj2` for the
  analysis. (#1202; fixed #1196)

* Fixed a tabix bug that prevented setting the correct number of
  lines to be skipped in a region file. Thanks to Jim Robinson for
  reporting it. (#1189;  fixed #1186)

* Made `bam_itr_next` an alias for `sam_itr_next`, to prevent it from
  crashing when working with htsFile pointers. Thanks to
  Torbjörn Klatt for reporting it. (#1180; fixed #1179)

* Fixed once per outgoing multi-threaded block `bgzf_idx_flush`
  assertion, to accommodate situations when a single record could
  span multiple blocks. Thanks to `@lacek`. (#1168; fixed
  samtools/samtools#1328)

* Fixed assumption of pthread_t being a non-structure, as permitted
  by POSIX. Thanks also to John Marshall and Anders Kaplan. (#1167,
  #1153, #1153)

* Fixed the minimum offset of a BAI index bin, to account for
  unmapped reads. Thanks to John Marshall for spotting the issue.
  (#1158; fixed #1142)

* Fixed the CRLF handling in `sam_parse_worker` method. Thanks to
  Anders Kaplan. (#1149; fixed #1148)

* Included unistd.h and errno.h directly in HTSlib files, as opposed
  to including them indirectly, via third party code. Thanks to
  Andrew Patterson (#1143) and John Marshall (#1145).

bd133ac

2021-03-17 17:14

下载

1.11

htslib release 1.11:

Features and Updates
--------------------

* Support added for remote reference files.  fai_path() can take a
  remote reference file and will return the corresponding index file.
   Remote indexes can be handled by refs_load_fai().  UR tags in @SQ
  lines can now be set to remote URIs. (#1017)

* Added tabix --separate-regions option, which adds header comment
  lines separating different regions' output records when multiple
  target regions are supplied on the command line. (#1108)

* Added tabix --cache option to set a BGZF block cache size.  Most
  beneficial when the -R option is used and the same blocks need to
  be re-read multiple times.  (#1053)

* Improved error checking in tabix and added a --verbosity option
  so it is possible to change the amount of logging when it runs.
  (#1040)

* A note about the maximum chromosome length usable with TBI indexes
  has been added to the tabix manual page.  Thanks to John Marshall. 
  (#1070)

* New method vcf_open_mode() changes the opening mode of a variant
  file based on its file extension. Similar to sam_open_mode().
  (#1096)

* The VCF parser has been made faster and easier to maintain. (#1057)

* bcf_record_check() has been made faster, giving a 15% speed
  increase when reading an uncompressed BCF file. (#1130)

* The VCF parser now recognises the "<NON_REF>" symbolic allele
  produced by GATK. (#1045)

* Support has been added for simultaneous reading of unindexed
  VCF/BCF files when using the synced_bcf_reader interface.  Input
  files must have the chromosomes in the same order as each other
  and be consistent with the order of sequences in the header.
  (#1089)

* The VCF and BCF readers will now attempt to fix up invalid INFO/END
  tags where the stored END value is less than POS, resulting in an
  apparently negative record length.  Such files have been generated
  by programs which used END incorrectly, and by broken lift-over
  processes that failed to update any END tags present. (#1021; fixed
  samtools/bcftools#1154)

* The htsFile interface can now detect the crypt4gh encrypted
  format (see https://samtools.github.io/hts-specs/crypt4gh.pdf).
   If HTSlib is built with external plug-in support, and the
  hfile_crypt4gh plug-in is present, the file will be passed to
  it for decryption.  The plug-in can be obtained from
  https://github.com/samtools/htslib-crypt4gh. (#1046)

* hts_srand48() now seeds the same POSIX-standard sequences of
  pseudo-random numbers regardless of platform, including on OpenBSD
  where plain srand48() produces a different cryptographically-strong
  non-deterministic sequence. Thanks to John Marshall. (#1002)

* Iterators now work with 64 bit positions. (#1018)

* Improved the speed of range queries when using BAI indexes by
  making better use of the linear index data included in the file.
  The best improvement is on low-coverage data. (#1031)

* Alignments which consume no reference bases are now considered to
  have length 1.  This would make such alignments cover 1 reference
  position in the same manner as alignments that are unmapped or
  have no CIGAR strings. These alignments can now be returned by
  iterator-based queries.  Thanks to John Marshall. (#1063; fixed
  samtools/samtools#1240, see also samtools/hts-specs#521).

* A bam_set_seqi() function to modify a single base in the BAM
  structure has been added.  This is a companion function to
  bam_seqi(). (#1022)

* Writing SAM format is around 30% faster. (#1035)

* Added sam_format_aux1() which converts a BAM aux tag to a SAM
  format string. (#1134)

* bam_aux_update_str() no longer requires NUL-terminated strings.  It
  is also now possible to create tags containing part of a longer
  string. (#1088)

* It is now possible to use external plug-ins in language bindings
  that dynamically load HTSlib.  Note that a side-effect of this
  change is that some plug-ins now link against libhts.so, which
  means that they have to be able to find the shared library when
  they are started up.  Thanks to John Marshall. (#1072)

* bgzf_close(), and therefore hts_close(), will now return non-zero
  when closing a BGZF handle on which errors have been detected. 
  (Part of #1117)

* Added a special case to the kt_fisher_exact() test for when the
  table probability is too small to be represented in a double.  This
  fixes a bug where it would, for some inputs, fail to correctly
  determine which side of the distribution the table was on resulting
  in swapped p-values being returned for the left- and right-tailed
  tests.  The two-tailed test value was not affected by this problem.
  (#1126)

* Improved error diagnostics in the CRAM decoder (#1042), BGZF
  (#1049), the VCF and BCF readers (#1059), and the SAM parser
  (#1073).

* ks_resize() now allocates 1.5 times the requested size when it
  needs to expand a kstring instead of rounding up to the next
  power of two. This has been done mainly to make the inlined
  function smaller, but it also reduces the overhead of storing
  data in kstrings at the expense of possibly needing a few more
  reallocations. (#1129)

CRAM improvements
-----------------

* Delay CRAM crc32 checks until the data actually needs to be used. 
  With other changes this leads to a 20x speed up in indexing and
  other sub-query based actions. (#988)

* CRAM now handles the transition from mapped to unmapped data in a
  better way, improving compression of the unmapped data. (#961)

* CRAM can now use libdeflate. (#961)

* Fixed bug in MD tag generation with "b" read feature codes,
  causing the numbers in the tag to be too large.  Note that
  HTSlib never uses this feature code so it is unlikely that this
  bug would be seen on real data. The problem was found when
  testing against hand-crafted CRAM files. (#1086)

* Fixed a regression where the CRAM multi-region iterator became
  much less efficient when using threads.  It now works more like
  the single iterator and does not preemptively decode the next
  container unless it will be used. (#1061)

* Set CRAM default quality in lossy quality modes. If lossy quality
  is enabled and 'B', 'q' or 'Q' features are used, CRAM starts off
  with QUAL being all 255 (as per BAM spec and "*" quality) and then
  modifies individual qualities as dictated by the specific features.

However that then produces ASCII quality " " (space, q=-1) for the
unmodified bases.  Instead ASCII quality "?" (q=30) is used, as per
HTSJDK. Quality 255 is still used for sequences with no modifications
at all. (#1094)

Build changes
-------------

These are compiler, configuration and makefile based changes.

* `make all` now also builds htslib_static.mk and
  htslib-uninstalled.pc. Thanks to John Marshall. (#1011)

* Various cppcheck-1.90 warnings have been fixed. (#995, #1011)

* HTSlib now prefers its own headers when being compiled, fixing
  build failures on machines that already had a system-installed
  HTSlib.  Thanks to John Marshall. (#1078; fixed #347)

* Define HTSLIB_EXPORT without using a helper macro to reduce the
  length of compiler diagnostics that mention exported functions. 
  Thanks to John Marshall. (#1029)

* Fix dirty default build by including latest pkg.m4 instead of using
  aclocal.m4. Thanks to Damien Zammit. (#1091)

* Struct tags have been added to htslib/*.h public typedefs.  This
  makes it possible to forward declare htsFile without including
  htslib/hts.h. Thanks to Lucas Czech and John Marshall. (#1115;
  fixed #1106)

* Fixed compiler warnings emitted by the latest gcc and clang
  releases when compiling HTSlib, along with some -Wextra warnings
  in the public include files.  Thanks to John Marshall. (#1066,
  #1063, #1083)

Bug fixes
---------

* Fixed hfile_libcurl breakage when using libcurl 7.69.1 or
  later.  Thanks to John Marshall for tracking down the exact
  libcurl change that caused the incompatibility. (#1105; fixed
  samtools/samtools#1254 and samtools/samtools#1284)

* Fixed overflows kroundup32() and kroundup_size_t() which caused
  them to return zero when rounding up values where the most
  significant bit was set.  When this happens they now return the
  highest value that can be stored (#1044).  All of the kroundup
  macro definitions have also been gathered together into a
  unified implementation (#1051).

* Fixed missing return parameter value in idx_test_and_fetch().
  Thanks to Lilian Janin. (#1014)

* Fixed crashes due to inconsistent selection between BGZF and plain
  (hFILE) interfaces when reading files. [fuzz] (#1019)

* Added and/or fixed byte swapping code for big-endian platforms. 
  Thanks to Jun Aruga, John Marshall, Michael R Crusoe and Gianfranco
  Costamagna for their help. (#1023; fixed #119 and #355)

* Fixed a problem with multi-threaded on-the-fly indexes which would
  occasionally write virtual offsets pointing at the end of a BGZF
  block. Attempting to read from such an offset caused EOF to be
  incorrectly reported.  These offsets are now handled correctly, and
  the indexer has been updated to avoid generating them.  (#1028;
  fixed samtools/samtools#1197)

* In sam_hdr_create(), free newly allocated SN strings when
  encountering an error. [fuzz] (#1034)

* Prevent double free in case of idx_test_and_fetch() failure. Thanks
  to @fanwayne for the bug report. (#1047; fixed #1033)

* In the header, link a new PG line only to valid chains. Prevents
  an explosive growth of PG lines on headers where PG lines are
  already present but not linked together correctly. (#1062; fixed
  samtools/samtools#1235)

* Also in the header, when calling sam_hdr_update_line(), update
  target arrays only when the name or length is changed. (#1007)

* Fixed buffer overflows in CRAM MD5 calculation triggered by files
  with invalid compression headers, or files with embedded references
  that were one byte too short. [fuzz] (#1024, #1068)

* Fix mpileup regression between 1.9 and 1.10 where overlap detection
  was incorrectly skipped on reads where RNEXT, PNEXT and TLEN were
  set to the "unavailable" values ("*", 0, 0 in SAM). (#1097)

* kputs() now checks for null pointer in source string. [fuzz]
  (#1087)

* Fix potential bcf_update_alleles() crash on 0 alleles. Thanks to
  John Marshall. (#994)

* Added bcf_unpack() calls to some bcf_update functions to fix a bug
  where updates made after a call to bcf_dup() could be lost. (#1032;
  fixed #1030)

* Error message typo "Number=R" instead of "Number=G" fixed in
  bcf_remove_allele_set(). Thanks to Ilya Vorontsov. (#1100)

* Fixed crashes that could occur in BCF files that use IDX= header
  annotations to create a sparse set of CHROM, FILTER or FORMAT
  indexes, and include records that use one of the missing index
  values. [fuzz] (#1092)

* Fixed potential integer overflows in the VCF parser and ensured
  that the total length of FORMAT fields cannot go over 2Gbytes.
  [fuzz] (#1044, #1104)

* Download index files atomically in idx_test_and_fetch().  This
  prevents corruption when running parallel jobs on S3 files. 
  Thanks to John Marshall. (#1112; samtools/samtools#1242).

* The pileup constructor callback is now given the copy of the
  bam1_t struct made by pileup instead of the original one passed
  to bam_plp_push().  This makes it the same as the one passed to
  the destructor and ensures that cached data, for example the
  location of an aux tag, will remain valid. (#1127)

* Fixed possible error in code_sort() on negative CRAM Huffman code
  length. (#1008)

* Fixed possible undefined shift in
  cram_byte_array_stop_decode_init(). (#1009)

* Fixed a bug where range queries to the end of a given reference
  would return incorrect results on CRAM files. (#1016; fixed
  samtools/samtools#1173)

* Fixed an integer overflow in cram_read_slice(). [fuzz] (#1026)

* Fixed a memory leak on failure in cram_decode_slice(). [fuzz]
  (#1054)

* Fixed a regression which caused cram_transcode_rg() to fail,
  resulting in a crash in "samtools cat" on CRAM files. (#1093;
  fixed samtools/samtools#1276)

* Fixed an undersized string reallocation in the threaded SAM reader
  which caused it to crash when reading SAM files with very long
  lines.  Numerous memory allocation checks have also been added.
  (#1117)

a7a90fe

2020-09-22 20:15

下载

1.11.0