ceph/src/zstd/lib/README.md

   1 Zstandard library files
   2 ================================
   3
   4 The __lib__ directory is split into several sub-directories,
   5 in order to make it easier to select or exclude features.
   6
   7
   8 #### Building
   9
  10 `Makefile` script is provided, supporting [Makefile conventions](https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html#Makefile-Conventions),
  11 including commands variables, staged install, directory variables and standard targets.
  12 - `make` : generates both static and dynamic libraries
  13 - `make install` : install libraries and headers in target system directories
  14
  15 `libzstd` default scope is pretty large, including compression, decompression, dictionary builder,
  16 and support for decoding legacy formats >= v0.5.0.
  17 The scope can be reduced on demand (see paragraph _modular build_).
  18
  19
  20 #### Multithreading support
  21
  22 Multithreading is disabled by default when building with `make`.
  23 Enabling multithreading requires 2 conditions :
  24 - set build macro `ZSTD_MULTITHREAD` (`-DZSTD_MULTITHREAD` for `gcc`)
  25 - for POSIX systems : compile with pthread (`-pthread` compilation flag for `gcc`)
  26
  27 Both conditions are automatically applied when invoking `make lib-mt` target.
  28
  29 When linking a POSIX program with a multithreaded version of `libzstd`,
  30 note that it's necessary to invoke the `-pthread` flag during link stage.
  31
  32 Multithreading capabilities are exposed
  33 via the [advanced API defined in `lib/zstd.h`](https://github.com/facebook/zstd/blob/v1.4.3/lib/zstd.h#L351).
  34
  35
  36 #### API
  37
  38 Zstandard's stable API is exposed within [lib/zstd.h](zstd.h).
  39
  40
  41 #### Advanced API
  42
  43 Optional advanced features are exposed via :
  44
  45 - `lib/common/zstd_errors.h` : translates `size_t` function results
  46                                into a `ZSTD_ErrorCode`, for accurate error handling.
  47
  48 - `ZSTD_STATIC_LINKING_ONLY` : if this macro is defined _before_ including `zstd.h`,
  49                           it unlocks access to the experimental API,
  50                           exposed in the second part of `zstd.h`.
  51                           All definitions in the experimental APIs are unstable,
  52                           they may still change in the future, or even be removed.
  53                           As a consequence, experimental definitions shall ___never be used with dynamic library___ !
  54                           Only static linking is allowed.
  55
  56
  57 #### Modular build
  58
  59 It's possible to compile only a limited set of features within `libzstd`.
  60 The file structure is designed to make this selection manually achievable for any build system :
  61
  62 - Directory `lib/common` is always required, for all variants.
  63
  64 - Compression source code lies in `lib/compress`
  65
  66 - Decompression source code lies in `lib/decompress`
  67
  68 - It's possible to include only `compress` or only `decompress`, they don't depend on each other.
  69
  70 - `lib/dictBuilder` : makes it possible to generate dictionaries from a set of samples.
  71         The API is exposed in `lib/dictBuilder/zdict.h`.
  72         This module depends on both `lib/common` and `lib/compress` .
  73
  74 - `lib/legacy` : makes it possible to decompress legacy zstd formats, starting from `v0.1.0`.
  75         This module depends on `lib/common` and `lib/decompress`.
  76         To enable this feature, define `ZSTD_LEGACY_SUPPORT` during compilation.
  77         Specifying a number limits versions supported to that version onward.
  78         For example, `ZSTD_LEGACY_SUPPORT=2` means : "support legacy formats >= v0.2.0".
  79         Conversely, `ZSTD_LEGACY_SUPPORT=0` means "do __not__ support legacy formats".
  80         By default, this build macro is set as `ZSTD_LEGACY_SUPPORT=5`.
  81         Decoding supported legacy format is a transparent capability triggered within decompression functions.
  82         It's also allowed to invoke legacy API directly, exposed in `lib/legacy/zstd_legacy.h`.
  83         Each version does also provide its own set of advanced API.
  84         For example, advanced API for version `v0.4` is exposed in `lib/legacy/zstd_v04.h` .
  85
  86 - While invoking `make libzstd`, it's possible to define build macros
  87         `ZSTD_LIB_COMPRESSION, ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`,
  88         and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the
  89         corresponding features. This will also disable compilation of all
  90         dependencies (eg. `ZSTD_LIB_COMPRESSION=0` will also disable
  91         dictBuilder).
  92
  93 - There are a number of options that can help minimize the binary size of
  94   `libzstd`.
  95
  96   The first step is to select the components needed (using the above-described
  97   `ZSTD_LIB_COMPRESSION` etc.).
  98
  99   The next step is to set `ZSTD_LIB_MINIFY` to `1` when invoking `make`. This
 100   disables various optional components and changes the compilation flags to
 101   prioritize space-saving.
 102
 103   Detailed options: Zstandard's code and build environment is set up by default
 104   to optimize above all else for performance. In pursuit of this goal, Zstandard
 105   makes significant trade-offs in code size. For example, Zstandard often has
 106   more than one implementation of a particular component, with each
 107   implementation optimized for different scenarios. For example, the Huffman
 108   decoder has complementary implementations that decode the stream one symbol at
 109   a time or two symbols at a time. Zstd normally includes both (and dispatches
 110   between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1` or
 111   `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding
 112   compilation of the other. Similarly, `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT`
 113   and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG` force the compilation and use of
 114   only one or the other of two decompression implementations. The smallest
 115   binary is achieved by using `HUF_FORCE_DECOMPRESS_X1` and
 116   `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` (implied by `ZSTD_LIB_MINIFY`).
 117
 118   For squeezing the last ounce of size out, you can also define
 119   `ZSTD_NO_INLINE`, which disables inlining, and `ZSTD_STRIP_ERROR_STRINGS`,
 120   which removes the error messages that are otherwise returned by
 121   `ZSTD_getErrorName` (implied by `ZSTD_LIB_MINIFY`).
 122
 123   Finally, when integrating into your application, make sure you're doing link-
 124   time optimation and unused symbol garbage collection (via some combination of,
 125   e.g., `-flto`, `-ffat-lto-objects`, `-fuse-linker-plugin`,
 126   `-ffunction-sections`, `-fdata-sections`, `-fmerge-all-constants`,
 127   `-Wl,--gc-sections`, `-Wl,-z,norelro`, and an archiver that understands
 128   the compiler's intermediate representation, e.g., `AR=gcc-ar`). Consult your
 129   compiler's documentation.
 130
 131 - While invoking `make libzstd`, the build macro `ZSTD_LEGACY_MULTITHREADED_API=1`
 132   will expose the deprecated `ZSTDMT` API exposed by `zstdmt_compress.h` in
 133   the shared library, which is now hidden by default.
 134
 135 - The build macro `DYNAMIC_BMI2` can be set to 1 or 0 in order to generate binaries
 136   which can detect at runtime the presence of BMI2 instructions, and use them only if present.
 137   These instructions contribute to better performance, notably on the decoder side.
 138   By default, this feature is automatically enabled on detecting
 139   the right instruction set (x64) and compiler (clang or gcc >= 5).
 140   It's obviously disabled for different cpus,
 141   or when BMI2 instruction set is _required_ by the compiler command line
 142   (in this case, only the BMI2 code path is generated).
 143   Setting this macro will either force to generate the BMI2 dispatcher (1)
 144   or prevent it (0). It overrides automatic detection.
 145
 146
 147 #### Windows : using MinGW+MSYS to create DLL
 148
 149 DLL can be created using MinGW+MSYS with the `make libzstd` command.
 150 This command creates `dll\libzstd.dll` and the import library `dll\libzstd.lib`.
 151 The import library is only required with Visual C++.
 152 The header file `zstd.h` and the dynamic library `dll\libzstd.dll` are required to
 153 compile a project using gcc/MinGW.
 154 The dynamic library has to be added to linking options.
 155 It means that if a project that uses ZSTD consists of a single `test-dll.c`
 156 file it should be linked with `dll\libzstd.dll`. For example:
 157 ```
 158     gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll
 159 ```
 160 The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`.
 161
 162
 163 #### Deprecated API
 164
 165 Obsolete API on their way out are stored in directory `lib/deprecated`.
 166 At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`.
 167 These prototypes will be removed in some future version.
 168 Consider migrating code towards supported streaming API exposed in `zstd.h`.
 169
 170
 171 #### Miscellaneous
 172
 173 The other files are not source code. There are :
 174
 175  - `BUCK` : support for `buck` build system (https://buckbuild.com/)
 176  - `Makefile` : `make` script to build and install zstd library (static and dynamic)
 177  - `README.md` : this file
 178  - `dll/` : resources directory for Windows compilation
 179  - `libzstd.pc.in` : script for `pkg-config` (used in `make install`)