1 //! A `Source` for registry-based packages.
3 //! # What's a Registry?
5 //! Registries are central locations where packages can be uploaded to,
6 //! discovered, and searched for. The purpose of a registry is to have a
7 //! location that serves as permanent storage for versions of a crate over time.
9 //! Compared to git sources, a registry provides many packages as well as many
10 //! versions simultaneously. Git sources can also have commits deleted through
11 //! rebasings where registries cannot have their versions deleted.
13 //! # The Index of a Registry
15 //! One of the major difficulties with a registry is that hosting so many
16 //! packages may quickly run into performance problems when dealing with
17 //! dependency graphs. It's infeasible for cargo to download the entire contents
18 //! of the registry just to resolve one package's dependencies, for example. As
19 //! a result, cargo needs some efficient method of querying what packages are
20 //! available on a registry, what versions are available, and what the
21 //! dependencies for each version is.
23 //! One method of doing so would be having the registry expose an HTTP endpoint
24 //! which can be queried with a list of packages and a response of their
25 //! dependencies and versions is returned. This is somewhat inefficient however
26 //! as we may have to hit the endpoint many times and we may have already
27 //! queried for much of the data locally already (for other packages, for
28 //! example). This also involves inventing a transport format between the
29 //! registry and Cargo itself, so this route was not taken.
31 //! Instead, Cargo communicates with registries through a git repository
32 //! referred to as the Index. The Index of a registry is essentially an easily
33 //! query-able version of the registry's database for a list of versions of a
34 //! package as well as a list of dependencies for each version.
36 //! Using git to host this index provides a number of benefits:
38 //! * The entire index can be stored efficiently locally on disk. This means
39 //! that all queries of a registry can happen locally and don't need to touch
42 //! * Updates of the index are quite efficient. Using git buys incremental
43 //! updates, compressed transmission, etc for free. The index must be updated
44 //! each time we need fresh information from a registry, but this is one
45 //! update of a git repository that probably hasn't changed a whole lot so
46 //! it shouldn't be too expensive.
48 //! Additionally, each modification to the index is just appending a line at
49 //! the end of a file (the exact format is described later). This means that
50 //! the commits for an index are quite small and easily applied/compressable.
52 //! ## The format of the Index
54 //! The index is a store for the list of versions for all packages known, so its
55 //! format on disk is optimized slightly to ensure that `ls registry` doesn't
56 //! produce a list of all packages ever known. The index also wants to ensure
57 //! that there's not a million files which may actually end up hitting
58 //! filesystem limits at some point. To this end, a few decisions were made
59 //! about the format of the registry:
61 //! 1. Each crate will have one file corresponding to it. Each version for a
62 //! crate will just be a line in this file.
63 //! 2. There will be two tiers of directories for crate names, under which
64 //! crates corresponding to those tiers will be located.
66 //! As an example, this is an example hierarchy of an index:
87 //! The root of the index contains a `config.json` file with a few entries
88 //! corresponding to the registry (see `RegistryConfig` below).
90 //! Otherwise, there are three numbered directories (1, 2, 3) for crates with
91 //! names 1, 2, and 3 characters in length. The 1/2 directories simply have the
92 //! crate files underneath them, while the 3 directory is sharded by the first
93 //! letter of the crate name.
95 //! Otherwise the top-level directory contains many two-letter directory names,
96 //! each of which has many sub-folders with two letters. At the end of all these
97 //! are the actual crate files themselves.
99 //! The purpose of this layout is to hopefully cut down on `ls` sizes as well as
100 //! efficient lookup based on the crate name itself.
104 //! Each file in the index is the history of one crate over time. Each line in
105 //! the file corresponds to one version of a crate, stored in JSON format (see
106 //! the `RegistryPackage` structure below).
108 //! As new versions are published, new lines are appended to this file. The only
109 //! modifications to this file that should happen over time are yanks of a
110 //! particular version.
112 //! # Downloading Packages
114 //! The purpose of the Index was to provide an efficient method to resolve the
115 //! dependency graph for a package. So far we only required one network
116 //! interaction to update the registry's repository (yay!). After resolution has
117 //! been performed, however we need to download the contents of packages so we
118 //! can read the full manifest and build the source code.
120 //! To accomplish this, this source's `download` method will make an HTTP
121 //! request per-package requested to download tarballs into a local cache. These
122 //! tarballs will then be unpacked into a destination folder.
124 //! Note that because versions uploaded to the registry are frozen forever that
125 //! the HTTP download and unpacking can all be skipped if the version has
126 //! already been downloaded and unpacked. This caching allows us to only
127 //! download a package when absolutely necessary.
129 //! # Filesystem Hierarchy
131 //! Overall, the `$HOME/.cargo` looks like this when talking about the registry:
134 //! # A folder under which all registry metadata is hosted (similar to
135 //! # $HOME/.cargo/git)
136 //! $HOME/.cargo/registry/
138 //! # For each registry that cargo knows about (keyed by hostname + hash)
139 //! # there is a folder which is the checked out version of the index for
140 //! # the registry in this location. Note that this is done so cargo can
141 //! # support multiple registries simultaneously
143 //! registry1-<hash>/
144 //! registry2-<hash>/
147 //! # This folder is a cache for all downloaded tarballs from a registry.
148 //! # Once downloaded and verified, a tarball never changes.
150 //! registry1-<hash>/<pkg>-<version>.crate
153 //! # Location in which all tarballs are unpacked. Each tarball is known to
154 //! # be frozen after downloading, so transitively this folder is also
155 //! # frozen once its unpacked (it's never unpacked again)
157 //! registry1-<hash>/<pkg>-<version>/...
161 use std
::borrow
::Cow
;
162 use std
::collections
::BTreeMap
;
163 use std
::collections
::HashSet
;
165 use std
::path
::{Path, PathBuf}
;
167 use flate2
::read
::GzDecoder
;
170 use serde
::Deserialize
;
173 use crate::core
::dependency
::{Dependency, Kind}
;
174 use crate::core
::source
::MaybePackage
;
175 use crate::core
::{Package, PackageId, Source, SourceId, Summary}
;
176 use crate::sources
::PathSource
;
177 use crate::util
::errors
::CargoResultExt
;
178 use crate::util
::hex
;
179 use crate::util
::to_url
::ToUrl
;
180 use crate::util
::{internal, CargoResult, Config, FileLock, Filesystem}
;
182 const INDEX_LOCK
: &str = ".cargo-index-lock";
183 const PACKAGE_SOURCE_LOCK
: &str = ".cargo-ok";
184 pub const CRATES_IO_INDEX
: &str = "https://github.com/rust-lang/crates.io-index";
185 pub const CRATES_IO_REGISTRY
: &str = "crates-io";
186 const CRATE_TEMPLATE
: &str = "{crate}";
187 const VERSION_TEMPLATE
: &str = "{version}";
189 pub struct RegistrySource
<'cfg
> {
191 src_path
: Filesystem
,
192 config
: &'cfg Config
,
194 ops
: Box
<dyn RegistryData
+ 'cfg
>,
195 index
: index
::RegistryIndex
<'cfg
>,
196 yanked_whitelist
: HashSet
<PackageId
>,
200 #[derive(Deserialize)]
201 pub struct RegistryConfig
{
202 /// Download endpoint for all crates.
204 /// The string is a template which will generate the download URL for the
205 /// tarball of a specific version of a crate. The substrings `{crate}` and
206 /// `{version}` will be replaced with the crate's name and version
209 /// For backwards compatibility, if the string does not contain `{crate}` or
210 /// `{version}`, it will be extended with `/{crate}/{version}/download` to
211 /// support registries like crates.io which were created before the
212 /// templating setup was created.
215 /// API endpoint for the registry. This is what's actually hit to perform
216 /// operations like yanks, owner modifications, publish new crates, etc.
217 /// If this is None, the registry does not support API commands.
218 pub api
: Option
<String
>,
221 #[derive(Deserialize)]
222 pub struct RegistryPackage
<'a
> {
225 deps
: Vec
<RegistryDependency
<'a
>>,
226 features
: BTreeMap
<Cow
<'a
, str>, Vec
<Cow
<'a
, str>>>,
228 yanked
: Option
<bool
>,
229 links
: Option
<Cow
<'a
, str>>,
233 fn escaped_cher_in_json() {
234 let _
: RegistryPackage
<'_
> = serde_json
::from_str(
235 r
#"{"name":"a","vers":"0.0.1","deps":[],"cksum":"bae3","features":{}}"#,
238 let _
: RegistryPackage
<'_
> = serde_json
::from_str(
239 r
#"{"name":"a","vers":"0.0.1","deps":[],"cksum":"bae3","features":{"test":["k","q"]},"links":"a-sys"}"#
242 // Now we add escaped cher all the places they can go
243 // these are not valid, but it should error later than json parsing
244 let _
: RegistryPackage
<'_
> = serde_json
::from_str(
246 "name":"This name has a escaped cher in it \n\t\" ",
251 "features": [" \n\t\" "],
253 "default_features": true,
254 "target": " \n\t\" ",
256 "registry": " \n\t\" "
259 "features":{"test \n\t\" ":["k \n\t\" ","q \n\t\" "]},
260 "links":" \n\t\" "}"#,
265 #[derive(Deserialize)]
266 #[serde(field_identifier, rename_all = "lowercase")]
277 #[derive(Deserialize)]
278 struct RegistryDependency
<'a
> {
281 features
: Vec
<Cow
<'a
, str>>,
283 default_features
: bool
,
284 target
: Option
<Cow
<'a
, str>>,
285 kind
: Option
<Cow
<'a
, str>>,
286 registry
: Option
<Cow
<'a
, str>>,
287 package
: Option
<Cow
<'a
, str>>,
288 public
: Option
<bool
>,
291 impl<'a
> RegistryDependency
<'a
> {
292 /// Converts an encoded dependency in the registry to a cargo dependency
293 pub fn into_dep(self, default: SourceId
) -> CargoResult
<Dependency
> {
294 let RegistryDependency
{
307 let id
= if let Some(registry
) = ®istry
{
308 SourceId
::for_registry(®istry
.to_url()?
)?
314 Dependency
::parse_no_deprecated(package
.as_ref().unwrap_or(&name
), Some(&req
), id
)?
;
315 if package
.is_some() {
316 dep
.set_explicit_name_in_toml(&name
);
318 let kind
= match kind
.as_ref().map(|s
| &s
[..]).unwrap_or("") {
319 "dev" => Kind
::Development
,
320 "build" => Kind
::Build
,
324 let platform
= match target
{
325 Some(target
) => Some(target
.parse()?
),
329 // All dependencies are private by default
330 let public
= public
.unwrap_or(false);
332 // Unfortunately older versions of cargo and/or the registry ended up
333 // publishing lots of entries where the features array contained the
334 // empty feature, "", inside. This confuses the resolution process much
335 // later on and these features aren't actually valid, so filter them all
337 features
.retain(|s
| !s
.is_empty());
339 // In index, "registry" is null if it is from the same index.
340 // In Cargo.toml, "registry" is None if it is from the default
341 if !id
.is_default_registry() {
342 dep
.set_registry_id(id
);
345 dep
.set_optional(optional
)
346 .set_default_features(default_features
)
347 .set_features(features
)
348 .set_platform(platform
)
356 pub trait RegistryData
{
357 fn prepare(&self) -> CargoResult
<()>;
358 fn index_path(&self) -> &Filesystem
;
363 data
: &mut dyn FnMut(&[u8]) -> CargoResult
<()>,
364 ) -> CargoResult
<()>;
365 fn config(&mut self) -> CargoResult
<Option
<RegistryConfig
>>;
366 fn update_index(&mut self) -> CargoResult
<()>;
367 fn download(&mut self, pkg
: PackageId
, checksum
: &str) -> CargoResult
<MaybeLock
>;
373 ) -> CargoResult
<FileLock
>;
375 fn is_crate_downloaded(&self, _pkg
: PackageId
) -> bool
{
382 Download { url: String, descriptor: String }
,
389 fn short_name(id
: SourceId
) -> String
{
390 let hash
= hex
::short_hash(&id
);
391 let ident
= id
.url().host_str().unwrap_or("").to_string();
392 format
!("{}-{}", ident
, hash
)
395 impl<'cfg
> RegistrySource
<'cfg
> {
398 yanked_whitelist
: &HashSet
<PackageId
>,
399 config
: &'cfg Config
,
400 ) -> RegistrySource
<'cfg
> {
401 let name
= short_name(source_id
);
402 let ops
= remote
::RemoteRegistry
::new(source_id
, config
, &name
);
416 yanked_whitelist
: &HashSet
<PackageId
>,
417 config
: &'cfg Config
,
418 ) -> RegistrySource
<'cfg
> {
419 let name
= short_name(source_id
);
420 let ops
= local
::LocalRegistry
::new(path
, config
, &name
);
433 config
: &'cfg Config
,
435 ops
: Box
<dyn RegistryData
+ 'cfg
>,
436 yanked_whitelist
: &HashSet
<PackageId
>,
438 ) -> RegistrySource
<'cfg
> {
440 src_path
: config
.registry_source_path().join(name
),
444 index
: index
::RegistryIndex
::new(source_id
, ops
.index_path(), config
, index_locked
),
445 yanked_whitelist
: yanked_whitelist
.clone(),
451 /// Decode the configuration stored within the registry.
453 /// This requires that the index has been at least checked out.
454 pub fn config(&mut self) -> CargoResult
<Option
<RegistryConfig
>> {
458 /// Unpacks a downloaded package into a location where it's ready to be
461 /// No action is taken if the source looks like it's already unpacked.
462 fn unpack_package(&self, pkg
: PackageId
, tarball
: &FileLock
) -> CargoResult
<PathBuf
> {
463 // The `.cargo-ok` file is used to track if the source is already
464 // unpacked and to lock the directory for unpacking.
466 let package_dir
= format
!("{}-{}", pkg
.name(), pkg
.version());
467 let dst
= self.src_path
.join(&package_dir
);
470 // Attempt to open a read-only copy first to avoid an exclusive write
471 // lock and also work with read-only filesystems. If the file has
472 // any data, assume the source is already unpacked.
473 if let Ok(ok
) = dst
.open_ro(PACKAGE_SOURCE_LOCK
, self.config
, &package_dir
) {
474 let meta
= ok
.file().metadata()?
;
476 let unpack_dir
= ok
.parent().to_path_buf();
477 return Ok(unpack_dir
);
481 dst
.open_rw(PACKAGE_SOURCE_LOCK
, self.config
, &package_dir
)?
483 let unpack_dir
= ok
.parent().to_path_buf();
485 // If the file has any data, assume the source is already unpacked.
486 let meta
= ok
.file().metadata()?
;
488 return Ok(unpack_dir
);
491 let gz
= GzDecoder
::new(tarball
.file());
492 let mut tar
= Archive
::new(gz
);
493 let prefix
= unpack_dir
.file_name().unwrap();
494 let parent
= unpack_dir
.parent().unwrap();
495 for entry
in tar
.entries()?
{
496 let mut entry
= entry
.chain_err(|| "failed to iterate over archive")?
;
497 let entry_path
= entry
499 .chain_err(|| "failed to read entry path")?
502 // We're going to unpack this tarball into the global source
503 // directory, but we want to make sure that it doesn't accidentally
504 // (or maliciously) overwrite source code from other crates. Cargo
505 // itself should never generate a tarball that hits this error, and
506 // crates.io should also block uploads with these sorts of tarballs,
507 // but be extra sure by adding a check here as well.
508 if !entry_path
.starts_with(prefix
) {
510 "invalid tarball downloaded, contains \
511 a file at {:?} which isn't under {:?}",
517 // Once that's verified, unpack the entry as usual.
520 .chain_err(|| format
!("failed to unpack entry at `{}`", entry_path
.display()))?
;
523 // Write to the lock file to indicate that unpacking was successful.
529 fn do_update(&mut self) -> CargoResult
<()> {
530 self.ops
.update_index()?
;
531 let path
= self.ops
.index_path();
533 index
::RegistryIndex
::new(self.source_id
, path
, self.config
, self.index_locked
);
538 fn get_pkg(&mut self, package
: PackageId
, path
: &FileLock
) -> CargoResult
<Package
> {
540 .unpack_package(package
, path
)
541 .chain_err(|| internal(format
!("failed to unpack package `{}`", package
)))?
;
542 let mut src
= PathSource
::new(&path
, self.source_id
, self.config
);
544 let mut pkg
= match src
.download(package
)?
{
545 MaybePackage
::Ready(pkg
) => pkg
,
546 MaybePackage
::Download { .. }
=> unreachable
!(),
549 // After we've loaded the package configure it's summary's `checksum`
550 // field with the checksum we know for this `PackageId`.
553 .summaries(package
.name().as_str(), &mut *self.ops
)?
;
554 let summary_with_cksum
= summaries
557 .find(|s
| s
.package_id() == package
)
558 .expect("summary not found");
559 if let Some(cksum
) = summary_with_cksum
.checksum() {
562 .set_checksum(cksum
.to_string());
569 impl<'cfg
> Source
for RegistrySource
<'cfg
> {
570 fn query(&mut self, dep
: &Dependency
, f
: &mut dyn FnMut(Summary
)) -> CargoResult
<()> {
571 // If this is a precise dependency, then it came from a lock file and in
572 // theory the registry is known to contain this version. If, however, we
573 // come back with no summaries, then our registry may need to be
574 // updated, so we fall back to performing a lazy update.
575 if dep
.source_id().precise().is_some() && !self.updated
{
576 debug
!("attempting query without update");
577 let mut called
= false;
579 .query_inner(dep
, &mut *self.ops
, &self.yanked_whitelist
, &mut |s
| {
588 debug
!("falling back to an update");
594 .query_inner(dep
, &mut *self.ops
, &self.yanked_whitelist
, &mut |s
| {
601 fn fuzzy_query(&mut self, dep
: &Dependency
, f
: &mut dyn FnMut(Summary
)) -> CargoResult
<()> {
603 .query_inner(dep
, &mut *self.ops
, &self.yanked_whitelist
, f
)
606 fn supports_checksums(&self) -> bool
{
610 fn requires_precise(&self) -> bool
{
614 fn source_id(&self) -> SourceId
{
618 fn update(&mut self) -> CargoResult
<()> {
619 // If we have an imprecise version then we don't know what we're going
620 // to look for, so we always attempt to perform an update here.
622 // If we have a precise version, then we'll update lazily during the
623 // querying phase. Note that precise in this case is only
624 // `Some("locked")` as other `Some` values indicate a `cargo update
625 // --precise` request
626 if self.source_id
.precise() != Some("locked") {
629 debug
!("skipping update due to locked registry");
634 fn download(&mut self, package
: PackageId
) -> CargoResult
<MaybePackage
> {
635 let hash
= self.index
.hash(package
, &mut *self.ops
)?
;
636 match self.ops
.download(package
, &hash
)?
{
637 MaybeLock
::Ready(file
) => self.get_pkg(package
, &file
).map(MaybePackage
::Ready
),
638 MaybeLock
::Download { url, descriptor }
=> {
639 Ok(MaybePackage
::Download { url, descriptor }
)
644 fn finish_download(&mut self, package
: PackageId
, data
: Vec
<u8>) -> CargoResult
<Package
> {
645 let hash
= self.index
.hash(package
, &mut *self.ops
)?
;
646 let file
= self.ops
.finish_download(package
, &hash
, &data
)?
;
647 self.get_pkg(package
, &file
)
650 fn fingerprint(&self, pkg
: &Package
) -> CargoResult
<String
> {
651 Ok(pkg
.package_id().version().to_string())
654 fn describe(&self) -> String
{
655 self.source_id
.display_index()
658 fn add_to_yanked_whitelist(&mut self, pkgs
: &[PackageId
]) {
659 self.yanked_whitelist
.extend(pkgs
);
662 fn is_yanked(&mut self, pkg
: PackageId
) -> CargoResult
<bool
> {
666 self.index
.is_yanked(pkg
, &mut *self.ops
)