ceph/src/jaegertracing/opentelemetry-cpp/tools/vcpkg/docs/users/binarycaching.md

   1 # Binary Caching
   2
   3 **The latest version of this documentation is available on [GitHub](https://github.com/Microsoft/vcpkg/tree/master/docs/users/binarycaching.md).**
   4
   5 Binary caching is vcpkg's method for reusing package builds between projects and between machines. Think of it as a "package restore accelerator" that gives you the same results as though you built from source. Each build is packaged independently, so changing one library only requires rebuilding consuming libraries.
   6
   7 If your CI provider offers a native "caching" function, we recommend using both methods for the most performant results.
   8
   9 In-tool help is available via `vcpkg help binarycaching`.
  10
  11 Table of Contents
  12  - [Configuration](#configuration)
  13  - [CI Examples](#ci-examples)
  14    - [GitHub Packages](#github-packages)
  15    - [Azure DevOps Artifacts](#azure-devops-artifacts)
  16    - [Azure Blob Storage](#azure-blob-storage-experimental)
  17    - [Google Cloud Storage](#google-cloud-storage-experimental)
  18  - [NuGet Provider Configuration](#nuget-provider-configuration)
  19  - [Implementation Notes](#implementation-notes-internal-details-subject-to-change-without-notice)
  20
  21
  22 ## Configuration
  23
  24 Binary caching is configured via a combination of defaults, the environment variable `VCPKG_BINARY_SOURCES` (set to `<source>;<source>;...`), and the command line option `--binarysource=<source>`. Source options are evaluated in order of defaults, then environment, then command line. Binary caching can be completely disabled by passing `--binarysource=clear` as the last command line option.
  25
  26 By default, zip-based archives will be cached at the first valid location of:
  27
  28 **Windows**
  29 1. `%VCPKG_DEFAULT_BINARY_CACHE%`
  30 2. `%LOCALAPPDATA%\vcpkg\archives`
  31 3. `%APPDATA%\vcpkg\archives`
  32
  33 **Non-Windows**
  34 1. `$VCPKG_DEFAULT_BINARY_CACHE`
  35 2. `$XDG_CACHE_HOME/vcpkg/archives`
  36 3. `$HOME/.cache/vcpkg/archives`
  37
  38 ### Valid source strings (`<source>`)
  39
  40 | form                        | description
  41 |-----------------------------|---------------
  42 | `clear`                     | Removes all previous sources (including the default)
  43 | `default[,<rw>]`            | Adds the default file-based location
  44 | `files,<absolute path>[,<rw>]`       | Adds a custom file-based location
  45 | `nuget,<uri>[,<rw>]`        | Adds a NuGet-based source; equivalent to the `-Source` parameter of the NuGet CLI
  46 | `nugetconfig,<path>[,<rw>]` | Adds a NuGet-config-file-based source; equivalent to the `-Config` parameter of the NuGet CLI. This config should specify `defaultPushSource` for uploads.
  47 | `x-azblob,<baseuri>,<sas>[,<rw>]`    | **Experimental: will change or be removed without warning**<br> Adds an Azure Blob Storage source. Uses Shared Access Signature validation. URL should include the container path.
  48 | `interactive`               | Enables interactive credential management for NuGet (for debugging; requires `--debug` on the command line)
  49
  50 The `<rw>` optional parameter for certain sources controls whether they will be consulted for
  51 downloading binaries (`read`)(default), whether on-demand builds will be uploaded to that remote (`write`), or both (`readwrite`).
  52
  53 Additional configuration details for NuGet-based providers can be found below in [NuGet Provider Configuration](#nuget-provider-configuration).
  54
  55 ## CI Examples
  56
  57 If your CI system of choice is not listed, we welcome PRs to add them!
  58
  59 ### GitHub Packages
  60
  61 To use vcpkg with GitHub Packages, we recommend using the `NuGet` backend.
  62
  63 >**NOTE 2020-09-21**: GitHub's hosted agents come with an older, pre-installed copy of vcpkg on the path that does not support the latest binary caching. This means that direct calls to `bootstrap-vcpkg` or `vcpkg` without a path prefix may call an unintended vcpkg instance. We recommend taking the following two steps to avoid issues if you want to use your own copy of vcpkg:
  64 > 1. Run the equivalent of `rm -rf "$VCPKG_INSTALLATION_ROOT"` using `shell: 'bash'`
  65 > 2. Always call `vcpkg` and `bootstrap-vcpkg` with a path prefix, such as `./vcpkg`, `vcpkg/vcpkg`, `.\bootstrap-vcpkg.bat`, etc
  66
  67 ```yaml
  68 # actions.yaml
  69 #
  70 # In this example, vcpkg has been added as a submodule (`git submodule add https://github.com/Microsoft/vcpkg`).
  71 env:
  72   VCPKG_BINARY_SOURCES: 'clear;nuget,GitHub,readwrite'
  73
  74 matrix:
  75   os: ['windows-2019', 'ubuntu-20.04']
  76   include:
  77     - os: 'windows-2019'
  78       triplet: 'x86-windows'
  79       mono: ''
  80     - os: 'ubuntu-20.04'
  81       triplet: 'x64-linux'
  82       # To run `nuget.exe` on non-Windows platforms, we must use `mono`.
  83       mono: 'mono'
  84
  85 steps:
  86   # This step assumes `vcpkg` has been bootstrapped (run `./vcpkg/bootstrap-vcpkg`)
  87   - name: 'Setup NuGet Credentials'
  88     shell: 'bash'
  89     # Replace <OWNER> with your organization name
  90     run: >
  91       ${{ matrix.mono }} `./vcpkg/vcpkg fetch nuget | tail -n 1`
  92       sources add
  93       -source "https://nuget.pkg.github.com/<OWNER>/index.json"
  94       -storepasswordincleartext
  95       -name "GitHub"
  96       -username "<OWNER>"
  97       -password "${{ secrets.GITHUB_TOKEN }}"
  98
  99   # Omit this step if you're using manifests
 100   - name: 'vcpkg package restore'
 101     shell: 'bash'
 102     run: >
 103       ./vcpkg/vcpkg install sqlite3 cpprestsdk --triplet ${{ matrix.triplet }}
 104 ```
 105
 106 If you're using [manifests](../specifications/manifests.md), you can omit the `vcpkg package restore` step: it will be run automatically as part of your build.
 107
 108 More information about GitHub Packages' NuGet support is available on [GitHub Docs][github-nuget].
 109
 110 [github-nuget]: https://docs.github.com/en/packages/using-github-packages-with-your-projects-ecosystem/configuring-dotnet-cli-for-use-with-github-packages
 111
 112 ### Azure DevOps Artifacts
 113
 114 To use vcpkg with Azure DevOps Artifacts, we recommend using the `NuGet` backend.
 115
 116 First, you need to ensure Artifacts has been enabled on your DevOps instance; this can be done by an Administrator through `Project Settings > General > Overview > Azure DevOps Services > Artifacts`.
 117
 118 Next, you will need to create a feed for your project; see the [Azure DevOps Artifacts Documentation][devops-nuget] for more information. Your feed URL will be an `https://` link ending with `/nuget/v3/index.json`.
 119
 120 ```yaml
 121 # azure-pipelines.yaml
 122 variables:
 123 - name: VCPKG_BINARY_SOURCES
 124   value: 'clear;nuget,<FEED_URL>,readwrite'
 125 ```
 126
 127 If you are using custom agents with a non-Windows OS, you will need to install Mono to run `nuget.exe` (`apt install mono-complete`, `brew install mono`, etc).
 128
 129 More information about Azure DevOps Artifacts' NuGet support is available in the [Azure DevOps Artifacts Documentation][devops-nuget].
 130
 131 [devops-nuget]: https://docs.microsoft.com/en-us/azure/devops/artifacts/get-started-nuget?view=azure-devops
 132
 133 ### Azure Blob Storage (experimental)
 134
 135 > Note: This is an experimental feature and may change or be removed at any time
 136
 137 Vcpkg supports interfacing with Azure Blob Storage via the `x-azblob` source type.
 138
 139 ```
 140 x-azblob,<baseuri>,<sas>[,<rw>]
 141 ```
 142
 143 First, you need to create an Azure Storage Account as well as a container ([Quick Start Documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal)].
 144
 145 Next, you will need to create a Shared Access Signature, which can be done from the storage account under Settings -> Shared access signature. This SAS will need:
 146 - Allowed services: Blob
 147 - Allowed resource types: Object
 148 - Allowed permissions: Read, Create (if using `write` or `readwrite`)
 149
 150 The blob endpoint plus the container must be passed as the `<baseuri>` and the generated SAS without the `?` prefix must be passed as the `<sas>`.
 151
 152 Example:
 153 ```
 154 x-azblob,https://<storagename>.blob.core.windows.net/<containername>,sv=2019-12-12&ss=b&srt=o&sp=rcx&se=2020-12-31T06:20:36Z&st=2020-12-30T22:20:36Z&spr=https&sig=abcd,readwrite
 155 ```
 156
 157 Vcpkg will attempt to avoid revealing the SAS during normal operations, however:
 158 1. It will be printed in full if `--debug` is passed
 159 2. It will be passed as a command line parameter to subprocesses, such as `curl.exe`
 160
 161 ### Google Cloud Storage (experimental)
 162
 163 > Note: This is an experimental feature and may change or be removed at any time
 164
 165 Vcpkg supports interfacing with Google Cloud Storage (GCS) via the `x-gcs` source type.
 166
 167 ```
 168 x-gcs,<prefix>[,<rw>]
 169 ```
 170
 171 First, you need to create an Google Cloud Platform Account as well as a storage bucket ([GCS Quick Start](https://cloud.google.com/storage/docs/quickstart-gsutil)].
 172
 173 As part of this quickstart you would have configured the `gsutil` command-line tool to authenticate with Google Cloud.
 174 Vcpkg will use this command-line tool, make sure it is in your search path for executables.
 175
 176 Example 1 (using a bucket without a common prefix for the objects):
 177
 178 ```
 179 x-gcs,gs://<bucket-name>/,readwrite
 180 ```
 181
 182 Example 2 (using a bucket and a prefix for the objects):
 183
 184 ```
 185 x-gcs,gs://<bucket-name>/my-vcpkg-cache/maybe/with/many/slashes/,readwrite
 186 x-gcs,gs://<bucket-name>/my-vcpkg-cache/maybe/with`,commas/too!/,readwrite
 187 ```
 188
 189 Commas (`,`) are valid as part of a object prefix in GCS, just remember to escape them in the vcpkg configuration, as
 190 shown in the previous example. Note that GCS does not have folders (some of the GCS tools simulate folders), it is not
 191 necessary to create or otherwise manipulate the prefix used by your vcpkg cache.
 192
 193 ## NuGet Provider Configuration
 194
 195 ### Credentials
 196
 197 Many NuGet servers require additional credentials to access. The most flexible way to supply credentials is via the `nugetconfig` provider with a custom `nuget.config` file. See https://docs.microsoft.com/en-us/nuget/consume-packages/consuming-packages-authenticated-feeds for more information on authenticating via `nuget.config`.
 198
 199 However, it is still possible to authenticate against many servers using NuGet's built-in credential providers or via customizing your environment's default `nuget.config`. The default config can be extended via nuget client calls such as
 200 ```
 201 nuget sources add -Name MyRemote -Source https://... -Username $user -Password $pass
 202 ```
 203 and then passed to vcpkg via `--binarysource=nuget,MyRemote,readwrite`. You can get a path to the precise copy of NuGet used by vcpkg by running `vcpkg fetch nuget`, which will report something like:
 204 ```
 205 $ vcpkg fetch nuget
 206 /vcpkg/downloads/tools/nuget-5.5.1-linux/nuget.exe
 207 ```
 208 Non-Windows users will need to call this through mono via `mono /path/to/nuget.exe sources add ...`.
 209
 210 ##### Credential Example for Azure Dev Ops
 211 ```bash
 212 # On Linux or OSX
 213 $ mono `vcpkg fetch nuget | tail -n1` sources add \
 214   -name ADO \
 215   -Source https://pkgs.dev.azure.com/$ORG/_packaging/$FEEDNAME/nuget/v3/index.json \
 216   -Username $USERNAME \
 217   -Password $PAT
 218 $ export VCPKG_BINARY_SOURCES="nuget,ADO,readwrite"
 219 ```
 220 ```powershell
 221 # On Windows Powershell
 222 PS> & $(vcpkg fetch nuget | select -last 1) sources add `
 223   -name ADO `
 224   -Source https://pkgs.dev.azure.com/$ORG/_packaging/$FEEDNAME/nuget/v3/index.json `
 225   -Username $USERNAME `
 226   -Password $PAT
 227 PS> $env:VCPKG_BINARY_SOURCES="nuget,ADO,readwrite"
 228 ```
 229
 230 We recommend using a Personal Access Token (PAT) as the password for maximum security. You can generate a PAT in User Settings -> Personal Access Tokens or `https://dev.azure.com/$ORG/_usersSettings/tokens`.
 231
 232 #### `metadata.repository`
 233
 234 The `nuget` and `nugetconfig` source providers additionally respect certain environment variables while generating nuget packages. The `metadata.repository` field of any packages will be generated as:
 235 ```
 236     <repository type="git" url="${VCPKG_NUGET_REPOSITORY}"/>
 237 ```
 238 or
 239 ```
 240     <repository type="git"
 241                 url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
 242                 branch="${GITHUB_REF}"
 243                 commit="${GITHUB_SHA}"/>
 244 ```
 245 if the appropriate environment variables are defined and non-empty. This is specifically used to associate packages in GitHub Packages with the _building_ project and not intended to associate with the original package sources.
 246
 247 #### NuGet's cache
 248
 249 NuGet's cache is not used by default. To use it for every nuget-based source, set the [environment variable](config-environment.md) `VCPKG_USE_NUGET_CACHE` to `true` (case-insensitive) or `1`.
 250
 251 ## Implementation Notes (internal details subject to change without notice)
 252
 253 Binary caching relies on hashing everything that contributes to a particular package build. This includes:
 254
 255 - Every file in the port directory
 256 - The triplet file and name
 257 - The C++ compiler executable
 258 - The C compiler executable
 259 - The set of features selected
 260 - Every dependency's package hash (note: this is that package's input hash, not contents)
 261 - All helper scripts referenced by `portfile.cmake` (heuristic)
 262 - The version of CMake used
 263 - The contents of any environment variables listed in `VCPKG_ENV_PASSTHROUGH`
 264 - The hash of the toolchain file (builtin or `VCPKG_CHAINLOAD_TOOLCHAIN_FILE`)
 265
 266 Despite this extensive list, it is possible to defeat the cache and introduce nondeterminism. If you have additional details that you'd like to be tracked, the easiest resolution is to generate a triplet file with your additional information in a comment. That additional information will be included in the package's input set and ensure a unique universe of binaries.
 267
 268 The hashes used are stored in the package and in the current installed directory at `/share/<port>/vcpkg_abi_info.txt`.
 269
 270 The original specification for binary caching is available [here](../specifications/binarycaching.md).