docs/file-formats.rst

   1 File Formats
   2 ============
   3
   4 .. _pxar-format:
   5
   6 Proxmox File Archive Format (``.pxar``)
   7 ---------------------------------------
   8
   9 .. graphviz:: pxar-format-overview.dot
  10
  11
  12 .. _data-blob-format:
  13
  14 Data Blob Format (``.blob``)
  15 ----------------------------
  16
  17 The data blob format is used to store small binary data. The magic number
  18 decides the exact format:
  19
  20 .. list-table::
  21    :widths: auto
  22
  23    * - ``[66, 171, 56, 7, 190, 131, 112, 161]``
  24      - unencrypted
  25      - uncompressed
  26    * - ``[49, 185, 88, 66, 111, 182, 163, 127]``
  27      - unencrypted
  28      - compressed
  29    * - ``[123, 103, 133, 190, 34, 45, 76, 240]``
  30      - encrypted
  31      - uncompressed
  32    * - ``[230, 89, 27, 191, 11, 191, 216, 11]``
  33      - encrypted
  34      - compressed
  35
  36 The compression algorithm used is ``zstd``. The encryption cipher is
  37 ``AES_256_GCM``.
  38
  39 Unencrypted blobs use the following format:
  40
  41 .. list-table::
  42    :widths: auto
  43
  44    * - ``MAGIC: [u8; 8]``
  45    * - ``CRC32: [u8; 4]``
  46    * - ``Data: (max 16MiB)``
  47
  48 Encrypted blobs additionally contain a 16 byte initialization vector (IV),
  49 followed by a 16 byte authenticated encryption (AE) tag, followed by the
  50 encrypted data:
  51
  52 .. list-table::
  53
  54    * - ``MAGIC: [u8; 8]``
  55    * - ``CRC32: [u8; 4]``
  56    * - ``IV: [u8; 16]``
  57    * - ``TAG: [u8; 16]``
  58    * - ``Data: (max 16MiB)``
  59
  60
  61 .. _fixed-index-format:
  62
  63 Fixed Index Format  (``.fidx``)
  64 -------------------------------
  65
  66 All numbers are stored as little-endian.
  67
  68 .. list-table::
  69
  70    * - ``MAGIC: [u8; 8]``
  71      - ``[47, 127, 65, 237, 145, 253, 15, 205]``
  72    * - ``uuid: [u8; 16]``,
  73      - Unique ID
  74    * - ``ctime: i64``,
  75      - Creation Time (epoch)
  76    * - ``index_csum: [u8; 32]``,
  77      - SHA-256 over the index (without header) ``SHA256(digest1||digest2||...)``
  78    * - ``size: u64``,
  79      - Image size
  80    * - ``chunk_size: u64``,
  81      - Chunk size
  82    * - ``reserved: [u8; 4016]``,
  83      - Overall header size is one page (4096 bytes)
  84    * - ``digest1: [u8; 32]``
  85      - First chunk digest
  86    * - ``digest2: [u8; 32]``
  87      - Second chunk digest
  88    * - ...
  89      - Next chunk digest ...
  90
  91
  92 .. _dynamic-index-format:
  93
  94 Dynamic Index Format (``.didx``)
  95 --------------------------------
  96
  97 All numbers are stored as little-endian.
  98
  99 .. list-table::
 100
 101    * - ``MAGIC: [u8; 8]``
 102      - ``[28, 145, 78, 165, 25, 186, 179, 205]``
 103    * - ``uuid: [u8; 16]``,
 104      - Unique ID
 105    * - ``ctime: i64``,
 106      - Creation Time (epoch)
 107    * - ``index_csum: [u8; 32]``,
 108      - SHA-256 over the index (without header) ``SHA256(offset1||digest1||offset2||digest2||...)``
 109    * - ``reserved: [u8; 4032]``,
 110      - Overall header size is one page (4096 bytes)
 111    * - ``offset1: u64``
 112      - End of first chunk
 113    * - ``digest1: [u8; 32]``
 114      - First chunk digest
 115    * - ``offset2: u64``
 116      - End of second chunk
 117    * - ``digest2: [u8; 32]``
 118      - Second chunk digest
 119    * - ...
 120      - Next chunk offset/digest