ceph/src/zstd/programs/README.md

   1 Command Line Interface for Zstandard library
   2 ============================================
   3
   4 Command Line Interface (CLI) can be created using the `make` command without any additional parameters.
   5 There are however other Makefile targets that create different variations of CLI:
   6 - `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd versions
   7 - `zstd32` : Same as `zstd`, but forced to compile in 32-bits mode
   8 - `zstd_nolegacy` : Same as `zstd` except of support for decompression of legacy zstd versions
   9 - `zstd-small` : CLI optimized for minimal size; without dictionary builder, benchmark, and support for decompression of legacy zstd versions
  10 - `zstd-compress` : compressor-only version of CLI; without dictionary builder, benchmark, and support for decompression of legacy zstd versions
  11 - `zstd-decompress` : decompressor-only version of CLI; without dictionary builder, benchmark, and support for decompression of legacy zstd versions
  12
  13
  14 #### Aggregation of parameters
  15 CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`.
  16
  17
  18 #### Dictionary builder in Command Line Interface
  19 Zstd offers a training mode, which can be used to tune the algorithm for a selected
  20 type of data, by providing it with a few samples. The result of the training is stored
  21 in a file selected with the `-o` option (default name is `dictionary`),
  22 which can be loaded before compression and decompression.
  23
  24 Using a dictionary, the compression ratio achievable on small data improves dramatically.
  25 These compression gains are achieved while simultaneously providing faster compression and decompression speeds.
  26 Dictionary work if there is some correlation in a family of small data (there is no universal dictionary).
  27 Hence, deploying one dictionary per type of data will provide the greater benefits.
  28 Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm
  29 will rely more and more on previously decoded content to compress the rest of the file.
  30
  31 Usage of the dictionary builder and created dictionaries with CLI:
  32
  33 1. Create the dictionary : `zstd --train FullPathToTrainingSet/* -o dictionaryName`
  34 2. Compress with the dictionary: `zstd FILE -D dictionaryName`
  35 3. Decompress with the dictionary: `zstd --decompress FILE.zst -D dictionaryName`
  36
  37
  38
  39 #### Benchmark in Command Line Interface
  40 CLI includes in-memory compression benchmark module for zstd.
  41 The benchmark is conducted using given filenames. The files are read into memory and joined together.
  42 It makes benchmark more precise as it eliminates I/O overhead.
  43 Many filenames can be supplied as multiple parameters, parameters with wildcards or
  44 names of directories can be used as parameters with the `-r` option.
  45
  46 The benchmark measures ratio, compressed size, compression and decompression speed.
  47 One can select compression levels starting from `-b` and ending with `-e`.
  48 The `-i` parameter selects minimal time used for each of tested levels.
  49
  50
  51
  52 #### Usage of Command Line Interface
  53 The full list of options can be obtained with `-h` or `-H` parameter:
  54 ```
  55 Usage :
  56       zstd [args] [FILE(s)] [-o file]
  57
  58 FILE    : a filename
  59           with no FILE, or when FILE is - , read standard input
  60 Arguments :
  61  -#     : # compression level (1-19, default:3)
  62  -d     : decompression
  63  -D file: use `file` as Dictionary
  64  -o file: result stored into `file` (only if 1 input file)
  65  -f     : overwrite output without prompting
  66 --rm    : remove source file(s) after successful de/compression
  67  -k     : preserve source file(s) (default)
  68  -h/-H  : display help/long help and exit
  69
  70 Advanced arguments :
  71  -V     : display Version number and exit
  72  -v     : verbose mode; specify multiple times to increase log level (default:2)
  73  -q     : suppress warnings; specify twice to suppress errors too
  74  -c     : force write to standard output, even if it is the console
  75  -r     : operate recursively on directories
  76 --ultra : enable levels beyond 19, up to 22 (requires more memory)
  77 --no-dictID : don't write dictID into header (dictionary compression)
  78 --[no-]check : integrity check (default:enabled)
  79 --test  : test compressed file integrity
  80 --[no-]sparse : sparse mode (default:enabled on file, disabled on stdout)
  81
  82 Dictionary builder :
  83 --train ## : create a dictionary from a training set of files
  84  -o file : `file` is dictionary name (default: dictionary)
  85 --maxdict ## : limit dictionary to specified size (default : 112640)
  86  -s#    : dictionary selectivity level (default: 9)
  87 --dictID ## : force dictionary ID to specified value (default: random)
  88
  89 Benchmark arguments :
  90  -b#    : benchmark file(s), using # compression level (default : 1)
  91  -e#    : test all compression levels from -bX to # (default: 1)
  92  -i#    : minimum evaluation time in seconds (default : 3s)
  93  -B#    : cut file into independent blocks of size # (default: no block)
  94  ```