#### Compilation variables
-`zstd` scope can be altered by modifying the following compilation variables :
+`zstd` scope can be altered by modifying the following `make` variables :
- __HAVE_THREAD__ : multithreading is automatically enabled when `pthread` is detected.
- It's possible to disable multithread support, by setting HAVE_THREAD=0 .
- Example : make zstd HAVE_THREAD=0
- It's also possible to force compilation with multithread support, using HAVE_THREAD=1.
- In which case, linking stage will fail if `pthread` library cannot be found.
- This might be useful to prevent silent feature disabling.
+ It's possible to disable multithread support, by setting `HAVE_THREAD=0`.
+ Example : `make zstd HAVE_THREAD=0`
+ It's also possible to force multithread support, using `HAVE_THREAD=1`.
+ In which case, linking stage will fail if neither `pthread` nor `windows.h` library can be found.
+ This is useful to ensure this feature is not silently disabled.
+
+- __ZSTD_LEGACY_SUPPORT__ : `zstd` can decompress files compressed by older versions of `zstd`.
+ Starting v0.8.0, all versions of `zstd` produce frames compliant with the [specification](../doc/zstd_compression_format.md), and are therefore compatible.
+ But older versions (< v0.8.0) produced different, incompatible, frames.
+ By default, `zstd` supports decoding legacy formats >= v0.4.0 (`ZSTD_LEGACY_SUPPORT=4`).
+ This can be altered by modifying this compilation variable.
+ `ZSTD_LEGACY_SUPPORT=1` means "support all formats >= v0.1.0".
+ `ZSTD_LEGACY_SUPPORT=2` means "support all formats >= v0.2.0", and so on.
+ `ZSTD_LEGACY_SUPPORT=0` means _DO NOT_ support any legacy format.
+ if `ZSTD_LEGACY_SUPPORT >= 8`, it's the same as `0`, since there is no legacy format after `7`.
+ Note : `zstd` only supports decoding older formats, and cannot generate any legacy format.
- __HAVE_ZLIB__ : `zstd` can compress and decompress files in `.gz` format.
This is ordered through command `--format=gzip`.
Alternatively, symlinks named `gzip` or `gunzip` will mimic intended behavior.
`.gz` support is automatically enabled when `zlib` library is detected at build time.
- It's possible to disable `.gz` support, by setting HAVE_ZLIB=0.
- Example : make zstd HAVE_ZLIB=0
- It's also possible to force compilation with zlib support, using HAVE_ZLIB=1.
+ It's possible to disable `.gz` support, by setting `HAVE_ZLIB=0`.
+ Example : `make zstd HAVE_ZLIB=0`
+ It's also possible to force compilation with zlib support, `using HAVE_ZLIB=1`.
In which case, linking stage will fail if `zlib` library cannot be found.
- This might be useful to prevent silent feature disabling.
+ This is useful to prevent silent feature disabling.
- __HAVE_LZMA__ : `zstd` can compress and decompress files in `.xz` and `.lzma` formats.
This is ordered through commands `--format=xz` and `--format=lzma` respectively.
Alternatively, symlinks named `xz`, `unxz`, `lzma`, or `unlzma` will mimic intended behavior.
`.xz` and `.lzma` support is automatically enabled when `lzma` library is detected at build time.
- It's possible to disable `.xz` and `.lzma` support, by setting HAVE_LZMA=0 .
- Example : make zstd HAVE_LZMA=0
- It's also possible to force compilation with lzma support, using HAVE_LZMA=1.
+ It's possible to disable `.xz` and `.lzma` support, by setting `HAVE_LZMA=0` .
+ Example : `make zstd HAVE_LZMA=0`
+ It's also possible to force compilation with lzma support, using `HAVE_LZMA=1`.
In which case, linking stage will fail if `lzma` library cannot be found.
- This might be useful to prevent silent feature disabling.
+ This is useful to prevent silent feature disabling.
- __HAVE_LZ4__ : `zstd` can compress and decompress files in `.lz4` formats.
This is ordered through commands `--format=lz4`.
Alternatively, symlinks named `lz4`, or `unlz4` will mimic intended behavior.
`.lz4` support is automatically enabled when `lz4` library is detected at build time.
- It's possible to disable `.lz4` support, by setting HAVE_LZ4=0 .
- Example : make zstd HAVE_LZ4=0
- It's also possible to force compilation with lz4 support, using HAVE_LZ4=1.
+ It's possible to disable `.lz4` support, by setting `HAVE_LZ4=0` .
+ Example : `make zstd HAVE_LZ4=0`
+ It's also possible to force compilation with lz4 support, using `HAVE_LZ4=1`.
In which case, linking stage will fail if `lz4` library cannot be found.
- This might be useful to prevent silent feature disabling.
+ This is useful to prevent silent feature disabling.
-- __ZSTD_LEGACY_SUPPORT__ : `zstd` can decompress files compressed by older versions of `zstd`.
- Starting v0.8.0, all versions of `zstd` produce frames compliant with the [specification](../doc/zstd_compression_format.md), and are therefore compatible.
- But older versions (< v0.8.0) produced different, incompatible, frames.
- By default, `zstd` supports decoding legacy formats >= v0.4.0 (`ZSTD_LEGACY_SUPPORT=4`).
- This can be altered by modifying this compilation variable.
- `ZSTD_LEGACY_SUPPORT=1` means "support all formats >= v0.1.0".
- `ZSTD_LEGACY_SUPPORT=2` means "support all formats >= v0.2.0", and so on.
- `ZSTD_LEGACY_SUPPORT=0` means _DO NOT_ support any legacy format.
- if `ZSTD_LEGACY_SUPPORT >= 8`, it's the same as `0`, since there is no legacy format after `7`.
- Note : `zstd` only supports decoding older formats, and cannot generate any legacy format.
+- __BACKTRACE__ : `zstd` can display a stack backtrace when execution
+ generates a runtime exception. By default, this feature may be
+ degraded/disabled on some platforms unless additional compiler directives are
+ applied. When triaging a runtime issue, enabling this feature can provide
+ more context to determine the location of the fault.
+ Example : `make zstd BACKTRACE=1`
#### Aggregation of parameters
CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`.
+#### Symlink shortcuts
+It's possible to invoke `zstd` through a symlink.
+When the name of the symlink has a specific value, it triggers an associated behavior.
+- `zstdmt` : compress using all cores available on local system.
+- `zcat` : will decompress and output target file using any of the supported formats. `gzcat` and `zstdcat` are also equivalent.
+- `gzip` : if zlib support is enabled, will mimic `gzip` by compressing file using `.gz` format, removing source file by default (use `--keep` to preserve). If zlib is not supported, triggers an error.
+- `xz` : if lzma support is enabled, will mimic `xz` by compressing file using `.xz` format, removing source file by default (use `--keep` to preserve). If xz is not supported, triggers an error.
+- `lzma` : if lzma support is enabled, will mimic `lzma` by compressing file using `.lzma` format, removing source file by default (use `--keep` to preserve). If lzma is not supported, triggers an error.
+- `lz4` : if lz4 support is enabled, will mimic `lz4` by compressing file using `.lz4` format. If lz4 is not supported, triggers an error.
+- `unzstd` and `unlz4` will decompress any of the supported format.
+- `ungz`, `unxz` and `unlzma` will do the same, and will also remove source file by default (use `--keep` to preserve).
+
+
#### Dictionary builder in Command Line Interface
Zstd offers a training mode, which can be used to tune the algorithm for a selected
type of data, by providing it with a few samples. The result of the training is stored
FILE : a filename
with no FILE, or when FILE is - , read standard input
Arguments :
- -# : # compression level (1-19, default:3)
+ -# : # compression level (1-19, default: 3)
-d : decompression
-D file: use `file` as Dictionary
-o file: result stored into `file` (only if 1 input file)
--ultra : enable levels beyond 19, up to 22 (requires more memory)
--long : enable long distance matching (requires more memory)
--no-dictID : don't write dictID into header (dictionary compression)
---[no-]check : integrity check (default:enabled)
+--[no-]check : integrity check (default: enabled)
-r : operate recursively on directories
--format=gzip : compress files to the .gz format
--format=xz : compress files to the .xz format
--format=lzma : compress files to the .lzma format
--test : test compressed file integrity
---[no-]sparse : sparse mode (default:disabled)
+--[no-]sparse : sparse mode (default: disabled)
-M# : Set a memory usage limit for decompression
-- : All arguments after "--" are treated as files
Dictionary builder :
--train ## : create a dictionary from a training set of files
---train-cover[=k=#,d=#,steps=#] : use the cover algorithm with optional args
+--train-cover[=k=#,d=#,steps=#,split=#] : use the cover algorithm with optional args
+--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] : use the fastcover algorithm with optional args
--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)
-o file : `file` is dictionary name (default: dictionary)
---maxdict=# : limit dictionary to specified size (default : 112640)
+--maxdict=# : limit dictionary to specified size (default: 112640)
--dictID=# : force dictionary ID to specified value (default: random)
Benchmark arguments :
- -b# : benchmark file(s), using # compression level (default : 1)
+ -b# : benchmark file(s), using # compression level (default: 3)
-e# : test all compression levels from -bX to # (default: 1)
- -i# : minimum evaluation time in seconds (default : 3s)
+ -i# : minimum evaluation time in seconds (default: 3s)
-B# : cut file into independent blocks of size # (default: no block)
--priority=rt : set process priority to real-time
```
+#### Restricted usage of Environment Variables
+Using environment variables to set compression/decompression parameters has security implications. Therefore,
+we intentionally restrict its usage. Currently, only `ZSTD_CLEVEL` is supported for setting compression level.
+If the value of `ZSTD_CLEVEL` is not a valid integer, it will be ignored with a warning message.
+Note that command line options will override corresponding environment variable settings.
#### Long distance matching mode
The long distance matching mode, enabled with `--long`, is designed to improve
the compression ratio for files with long matches at a large distance (up to the
-maximum window size, `128 MiB`) while still maintaining compression speed.
+maximum window size, `128 MiB`) while still maintaining compression speed.
Enabling this mode sets the window size to `128 MiB` and thus increases the memory
usage for both the compressor and decompressor. Performance in terms of speed is
case: a tar of four versions of clang (versions `3.4.1`, `3.4.2`, `3.5.0`,
`3.5.1`) with a total size of `244889600 B`. This is an ideal use case as there
are many long distance matches within the maximum window size of `128 MiB` (each
-version is less than `128 MiB`).
+version is less than `128 MiB`).
Compression Speed vs Ratio | Decompression Speed
---------------------------|---------------------
- | 
+ | 
| Method | Compression ratio | Compression speed | Decompression speed |
|:-------|------------------:|-------------------------:|---------------------------:|
[Silesia compression corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
| Method | Compression ratio | Compression speed | Decompression speed |
-|:-------|------------------:|-------------------------:|---------------------------:|
-| `zstd -1` | `2.878` | `231.7 MB/s` | `594.4 MB/s` |
-| `zstd -1 --long` | `2.929` | `106.5 MB/s` | `517.9 MB/s` |
-| `zstd -5` | `3.274` | `77.1 MB/s` | `464.2 MB/s` |
-| `zstd -5 --long` | `3.319` | `51.7 MB/s` | `371.9 MB/s` |
-| `zstd -10` | `3.523` | `16.4 MB/s` | `489.2 MB/s` |
-| `zstd -10 --long`| `3.566` | `16.2 MB/s` | `415.7 MB/s` |
+|:-------|------------------:|------------------:|---------------------:|
+| `zstd -1` | `2.878` | `231.7 MB/s` | `594.4 MB/s` |
+| `zstd -1 --long` | `2.929` | `106.5 MB/s` | `517.9 MB/s` |
+| `zstd -5` | `3.274` | `77.1 MB/s` | `464.2 MB/s` |
+| `zstd -5 --long` | `3.319` | `51.7 MB/s` | `371.9 MB/s` |
+| `zstd -10` | `3.523` | `16.4 MB/s` | `489.2 MB/s` |
+| `zstd -10 --long`| `3.566` | `16.2 MB/s` | `415.7 MB/s` |
+#### zstdgrep
+`zstdgrep` is a utility which makes it possible to `grep` directly a `.zst` compressed file.
+It's used the same way as normal `grep`, for example :
+`zstdgrep pattern file.zst`
+`zstdgrep` is _not_ compatible with dictionary compression.
+To search into a file compressed with a dictionary,
+it's necessary to decompress it using `zstd` or `zstdcat`,
+and then pipe the result to `grep`. For example :
+`zstdcat -D dictionary -qc -- file.zst | grep pattern`
projects / ceph.git / blobdiff