Auto merge of #2857 - alexcrichton:redirect-sources, r=brson

Add support local mirrors of registries, take 2

This series of commits culminates in first class support in Cargo for local mirrors of registries. This is implemented through a number of other more generic mechanisms, and extra support was added along the way. The highlights of this PR, however, are:

New `.cargo/config` keys have been added to enable *replacing one source with another*. This functionality is intended to be used for mirrors of the main registry or otherwise one to one source correspondences. The support looks like:

```toml
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
registry = 'https://github.com/my-awesome/registry-index'
```

This configuration means that instead of using `crates-io` (e.g. `https://github.com/rust-lang/crates.io-index`), Cargo will query the `my-awesome-registry` source instead (configured to a different index here). This alternate source **must be the exact same as the crates.io index**. Cargo assumes that replacement sources are exact 1:1 mirrors in this respect, and the following support is designed around that assumption.

When generating a lock file for crate using a replacement registry, the *original registry* will be encoded into the lock file. For example in the configuration above, all lock files will still mention crates.io as the registry that packages originated from. This semantically represents how crates.io is the source of truth for all crates, and this is upheld because all replacements have a 1:1 correspondance.

Overall, this means that no matter what replacement source you're working with, you can ship your lock file to anyone else and you'll all still have verifiably reproducible builds!

With the above support for custom registries, it's now possible for a project to be downloading crates from any number of sources. One of Cargo's core goals is reproducible builds, and with all these new sources of information it may be easy for a few situations to arise:

1. A local replacement of crates.io could be corrupt
2. A local replacement of crates.io could have made subtle changes to crates

In both of these cases, Cargo would today simply give non-reproducible builds. To help assuage these concerns, Cargo will now track the sha256 checksum of all crates from registries in the lock file. Whenever a `Cargo.lock` is generated from now on it will contain a `[metadata]` section which lists the sha256 checksum of all crates in the lock file (or `<none>` if the sha256 checksum isn't known).

Cargo already checks registry checksums against what's actually downloaded, and Cargo will now verify between iterations of the lock file that checksums remain the same as well. This means that if a local replacement registry is **not** in a 1:1 correspondance with crates.io, the lock file will prevent the build from progressing until the discrepancy is resolved.

In addition to the support above, there is now a new kind of source in Cargo, a "local registry", which is intended to be a subset of the crates.io ecosystem purposed for a local build for any particular project here or there. The way to enable this looks like:

```toml
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
local-registry = 'path/to/my/local/registry'
```

This local registry is expected to have two components:

1. A directory called `index` which matches the same structure as the crates.io index. The `config.json` file is not required here.
2. Inside the registry directory are any number of `.crate` files (downloaded from crates.io). Each crate file has the name `<package>-<version>.crate`.

This local registry must currently be managed manually, but I plan on publishing and maintaining a Cargo subcommand to manage a local registry. It will have options to do things like:

1. Sync a local registry with a `Cargo.lock`
2. Add a registry package to a local registry
3. Remove a package from a local registry

In addition to local registries, Cargo also supports a "directory source" like so

```toml
[source.crates-io]
replace-with = 'my-awesome-registry'

[source.my-awesome-registry]
directory = 'path/to/some/sources'
```

A directory source is similar to a local registry above, except that all the crates are unpacked and visible as vendored source. This format is suitable for checking into source trees, like Gecko's.

Unlike local registries above we don't have a tarball to verify the crates.io checksum with, but each vendored dependency has metadata containing what it *would* have been. To further prevent modifications by accident, the metadata contains the checksum of each file which should prevent accidental local modifications and steer towards `[replace]` as the mechanism to edit dependencies if necessary.

This is quite a bit of new features! What's all this meant to do? Some example scenarios that this is envisioned to solve are:

1. Supporting mirrors for crates.io in a first class fashion. Once we have the ability to spin up your own local registry, it should be easy to locally select a new mirror.
2. Supporting round-robin mirrors, this provides an easy vector for configuration of "instead of crates.io hit the first source in this list that works"
3. Build environments where network access is not an option. Preparing a local registry ahead-of-time (from a known good lock file) will be a vector to ensure that all Rust dependencies are locally available.
   * Note this is intended to include use cases like Debian and Gecko

Even with the new goodies here, there's some more vectors through which this can be expanded:

* Support for running your own mirror of crates.io needs to be implemented to be "easy to do". There should for example be a `cargo install foo` available to have everything "Just Work".
* Replacing a source with a list of sources (attempted in round robin fashion) needs to be implemented
* Eventually this support will be extended to the `Cargo.toml` file itself. For example:
  * packages should be downloadable from multiple registries
  * replacement sources should be encodable into `Cargo.toml` (note that these replacements, unlike the ones above, would be encoded into `Cargo.lock`)
  * adding multiple mirrors to a `Cargo.toml` should be supported
* Implementing the subcommand above to manage local registries needs to happen (I will attend to this shortly)