draft-kucherawy-rfc8478bis-04.txt   draft-kucherawy-rfc8478bis.txt 
Network Working Group Y. Collet Network Working Group Y. Collet
Internet-Draft M. Kucherawy, Ed. Internet-Draft M. Kucherawy, Ed.
Obsoletes: 8478 (if approved) Facebook Obsoletes: 8478 (if approved) Facebook
Intended status: Informational April 23, 2020 Intended status: Informational April 23, 2020
Expires: October 25, 2020 Expires: October 25, 2020
Zstandard Compression and the application/zstd Media Type Zstandard Compression and the application/zstd Media Type
draft-kucherawy-rfc8478bis-04 draft-kucherawy-rfc8478bis-05
Abstract Abstract
Zstandard, or "zstd" (pronounced "zee standard"), is a data Zstandard, or "zstd" (pronounced "zee standard"), is a data
compression mechanism. This document describes the mechanism and compression mechanism. This document describes the mechanism and
registers a media type and content encoding to be used when registers a media type and content encoding to be used when
transporting zstd-compressed content via Multipurpose Internet Mail transporting zstd-compressed content via Multipurpose Internet Mail
Extensions (MIME). Extensions (MIME). It also registers a corresponding media type,
content encoding, and structured syntax suffix.
Despite use of the word "standard" as part of its name, readers are Despite use of the word "standard" as part of its name, readers are
advised that this document is not an Internet Standards Track advised that this document is not an Internet Standards Track
specification; it is being published for informational purposes only. specification; it is being published for informational purposes only.
This document replaces and obsoletes RFC 8478. This document replaces and obsoletes RFC 8478.
Status of this Memo Status of this Memo
This Internet-Draft is submitted in full conformance with the This Internet-Draft is submitted in full conformance with the
skipping to change at page 3, line 39 skipping to change at page 3, line 39
7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 40 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 40
7.1. The 'application/zstd' Media Type . . . . . . . . . . . . 40 7.1. The 'application/zstd' Media Type . . . . . . . . . . . . 40
7.2. Content Encoding . . . . . . . . . . . . . . . . . . . . . 41 7.2. Content Encoding . . . . . . . . . . . . . . . . . . . . . 41
7.3. Structured Syntax Suffix . . . . . . . . . . . . . . . . . 42 7.3. Structured Syntax Suffix . . . . . . . . . . . . . . . . . 42
7.4. Dictionaries . . . . . . . . . . . . . . . . . . . . . . . 42 7.4. Dictionaries . . . . . . . . . . . . . . . . . . . . . . . 42
8. Security Considerations . . . . . . . . . . . . . . . . . . . 42 8. Security Considerations . . . . . . . . . . . . . . . . . . . 42
9. Implementation Status . . . . . . . . . . . . . . . . . . . . 43 9. Implementation Status . . . . . . . . . . . . . . . . . . . . 43
10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 44 10. References . . . . . . . . . . . . . . . . . . . . . . . . . . 44
10.1. Normative References . . . . . . . . . . . . . . . . . . . 44 10.1. Normative References . . . . . . . . . . . . . . . . . . . 44
10.2. Informative References . . . . . . . . . . . . . . . . . . 44 10.2. Informative References . . . . . . . . . . . . . . . . . . 44
Appendix A. Decoding Tables for Predefined Codes . . . . . . . . 44 Appendix A. Decoding Tables for Predefined Codes . . . . . . . . 45
A.1. Literal Length Code Table . . . . . . . . . . . . . . . . 45 A.1. Literal Length Code Table . . . . . . . . . . . . . . . . 45
A.2. Match Length Code Table . . . . . . . . . . . . . . . . . 47 A.2. Match Length Code Table . . . . . . . . . . . . . . . . . 48
A.3. Offset Code Table . . . . . . . . . . . . . . . . . . . . 50 A.3. Offset Code Table . . . . . . . . . . . . . . . . . . . . 51
Appendix B. Changes Since RFC8478 . . . . . . . . . . . . . . . . 52 Appendix B. Changes Since RFC8478 . . . . . . . . . . . . . . . . 52
Appendix C. Acknowledgments . . . . . . . . . . . . . . . . . . . 52 Appendix C. Acknowledgments . . . . . . . . . . . . . . . . . . . 53
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 52 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 53
1. Introduction 1. Introduction
Zstandard, or "zstd" (pronounced "zee standard"), is a data Zstandard, or "zstd" (pronounced "zee standard"), is a data
compression mechanism, akin to gzip [RFC1952]. compression mechanism, akin to gzip [RFC1952].
Despite use of the word "standard" as part of its name, readers are Despite use of the word "standard" as part of its name, readers are
advised that this document is not an Internet Standards Track advised that this document is not an Internet Standards Track
specification; it is being published for informational purposes only. specification; it is being published for informational purposes only.
skipping to change at page 14, line 5 skipping to change at page 14, line 5
Block_Maximum_Size is constant for a given frame. This maximum is Block_Maximum_Size is constant for a given frame. This maximum is
applicable to both the decompressed size and the compressed size of applicable to both the decompressed size and the compressed size of
any block in the frame. any block in the frame.
The reasoning for this limit is that a decoder can read this The reasoning for this limit is that a decoder can read this
information at the beginning of a frame and use it to allocate information at the beginning of a frame and use it to allocate
buffers. The guarantees on the size of blocks ensure that the buffers. The guarantees on the size of blocks ensure that the
buffers will be large enough for any following block of the valid buffers will be large enough for any following block of the valid
frame. frame.
If the compressed block is larger than the uncompressed sending the
uncompressed block (i.e., a Raw_Block) is recommended instead.
3.1.1.3. Compressed Blocks 3.1.1.3. Compressed Blocks
To decompress a compressed block, the compressed size must be To decompress a compressed block, the compressed size must be
provided from the Block_Size field within Block_Header. provided from the Block_Size field within Block_Header.
A compressed block consists of two sections: a Literals Section A compressed block consists of two sections: a Literals Section
(Section 3.1.1.3.1) and a Sequences_Section (Section 3.1.1.3.2). The (Section 3.1.1.3.1) and a Sequences_Section (Section 3.1.1.3.2). The
results of the two sections are then combined to produce the results of the two sections are then combined to produce the
decompressed data in Sequence Execution (Section 3.1.1.4). decompressed data in Sequence Execution (Section 3.1.1.4).
skipping to change at page 40, line 18 skipping to change at page 40, line 18
long as the amount of data decoded from this frame is less than or long as the amount of data decoded from this frame is less than or
equal to Window_Size, sequence commands may specify offsets longer equal to Window_Size, sequence commands may specify offsets longer
than the total length of decoded output so far to reference back than the total length of decoded output so far to reference back
to the dictionary, even parts of the dictionary with offsets to the dictionary, even parts of the dictionary with offsets
larger than Window_Size. After the total output has surpassed larger than Window_Size. After the total output has surpassed
Window_Size, however, this is no longer allowed, and the Window_Size, however, this is no longer allowed, and the
dictionary is no longer accessible. dictionary is no longer accessible.
6. Use of Dictionaries 6. Use of Dictionaries
Provisioning for use of dictionaries with zstd is planned for future Provisioning for use of dictionaries with zstd is being explored.
work. To ensure compatibility with the future specification of use See, for example, [DICT-SEC]. The likely outcome will be a registry
of dictionaries with zstd payloads, especially with MIME, content of well-tested dictionaries optimized for different use cases and
identifiers for each, possibly with a private negotiation mechanism
for use of unregistered dictionaries.
To ensure compatibility with the future specification of use of
dictionaries with zstd payloads, especially with MIME, content
encoded with the media type registered here should not use a encoded with the media type registered here should not use a
dictionary. The exception to this requirement might be a private dictionary. The exception to this requirement might be a private
dictionary negotiaton that is not part of this specification. dictionary negotiaton, suggested above, which is not part of this
specification.
7. IANA Considerations 7. IANA Considerations
IANA has made three registrations, as described below. IANA has updated two previously existing registrations and made one
new registration as described below.
7.1. The 'application/zstd' Media Type 7.1. The 'application/zstd' Media Type
The 'application/zstd' media type identifies a block of data that is The 'application/zstd' media type identifies a block of data that is
compressed using zstd compression. The data is a stream of bytes as compressed using zstd compression. The data is a stream of bytes as
described in this document. IANA has added the following to the described in this document. IANA has added the following to the
"Media Types" registry: "Media Types" registry:
Type name: application Type name: application
skipping to change at page 44, line 10 skipping to change at page 44, line 15
Facebook infrastructure. Facebook infrastructure.
The reference version is optimized for speed and is highly portable. The reference version is optimized for speed and is highly portable.
It has been proven to run safely on multiple architectures (e.g., It has been proven to run safely on multiple architectures (e.g.,
x86, x64, ARM, MIPS, PowerPC, IA64) featuring 32- or 64-bit x86, x64, ARM, MIPS, PowerPC, IA64) featuring 32- or 64-bit
addressing schemes, a little- or big-endian storage scheme, a number addressing schemes, a little- or big-endian storage scheme, a number
of different operating systems (e.g., UNIX (including Linux, BSD, of different operating systems (e.g., UNIX (including Linux, BSD,
OS-X, and Solaris) and Windows), and a number of compilers (e.g., OS-X, and Solaris) and Windows), and a number of compilers (e.g.,
gcc, clang, visual, and icc). gcc, clang, visual, and icc).
A comprehensive and current list of known implementations can be
found at [ZSTD].
10. References 10. References
10.1. Normative References 10.1. Normative References
[ZSTD] "Zstandard", 2017, <http://www.zstd.net>. [ZSTD] "Zstandard", 2017, <http://www.zstd.net>.
10.2. Informative References 10.2. Informative References
[ANS] Duda, J., "Asymmetric numeral systems: entropy coding [ANS] Duda, J., "Asymmetric numeral systems: entropy coding
combining speed of Huffman coding with compression rate of combining speed of Huffman coding with compression rate of
arithmetic coding", January 2014, arithmetic coding", January 2014,
<https://arxiv.org/pdf/1311.2540>. <https://arxiv.org/pdf/1311.2540>.
[CRIME] "CRIME", June 2018, <https://en.wikipedia.org/w/ [CRIME] "CRIME", June 2018, <https://en.wikipedia.org/w/
index.php?title=CRIME&oldid=844538656>. index.php?title=CRIME&oldid=844538656>.
[DICT-SEC]
Handte, W., "Security Considerations Regarding Compression
Dictionaries", (work in
progress) draft-handte-httpbis-dict-sec.
[FSE] "FiniteStateEntropy", June 2018, [FSE] "FiniteStateEntropy", June 2018,
<https://github.com/Cyan4973/FiniteStateEntropy/>. <https://github.com/Cyan4973/FiniteStateEntropy/>.
[LZ4] "LZ4 Frame Format Description", January 2018, <https:// [LZ4] "LZ4 Frame Format Description", January 2018, <https://
github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md>. github.com/lz4/lz4/blob/master/doc/lz4_Frame_format.md>.
[RFC1952] Deutsch, P., "GZIP file format specification version 4.3", [RFC1952] Deutsch, P., "GZIP file format specification version 4.3",
RFC 1952, DOI 10.17487/RFC1952, May 1996, RFC 1952, DOI 10.17487/RFC1952, May 1996,
<https://www.rfc-editor.org/info/rfc1952>. <https://www.rfc-editor.org/info/rfc1952>.
skipping to change at page 52, line 39 skipping to change at page 53, line 10
o Clarify handling of reserved and corrupt inputs. o Clarify handling of reserved and corrupt inputs.
o Add fragment identifier considerations to the media type o Add fragment identifier considerations to the media type
registration. registration.
Appendix C. Acknowledgments Appendix C. Acknowledgments
zstd was developed by Yann Collet. zstd was developed by Yann Collet.
Felix Handte and Nick Terrell provided feedback that went into this Felix Handte and Nick Terrell provided feedback that went into this
revision. revision and RFC 8478. RFC 8478 also received contributions from
Bobo Bose-Kolanu, Kyle Nekritz, and David Schleimer.
Authors' Addresses Authors' Addresses
Yann Collet Yann Collet
Facebook Facebook
1 Hacker Way 1 Hacker Way
Menlo Park, CA 94025 Menlo Park, CA 94025
United States of America United States of America
Email: cyan@fb.com Email: cyan@fb.com
 End of changes. 12 change blocks. 
13 lines changed or deleted 33 lines changed or added

This html diff was produced by rfcdiff 1.33. The latest version is available from http://tools.ietf.org/tools/rfcdiff/