bors [Mon, 8 Jan 2018 18:04:53 +0000 (18:04 +0000)]
Auto merge of #4919 - alexcrichton:faster-git-clone, r=matklad
Leverage local links on git checkouts
This commit updates the handling of git checkouts from the database to use
hardlinks if possible, speeding up this operation for large repositories
significantly.
As a refresher, Cargo caches git repositories in a few locations to speed up
local usage of git repositories. Cargo has a "database" folder which is a bare
checkout of any git repository Cargo has cached historically. This database
folder contains effectively a bunch of databases for remote repos that are
updated periodically.
When actually building a crate Cargo will clone this database into a different
location, the checkouts folder. Each rev we build (ever) is cached in the
checkouts folder. This means that once a checkout directory is created it's
frozen for all of time.
This latter step is what this commit is optimizing. When checking out the
database onto the local filesystem at a particular revision. Previously we were
instructing libgit2 to fall back to a "git aware" transport which was
exceedingly slow on some systems for filesystem-to-filesystem transfers. This
optimization (we just forgot to turn it on in libgit2) is a longstanding one and
should speed this up significantly!
Alex Crichton [Mon, 8 Jan 2018 17:38:40 +0000 (09:38 -0800)]
Leverage local links on git checkouts
This commit updates the handling of git checkouts from the database to use
hardlinks if possible, speeding up this operation for large repositories
significantly.
As a refresher, Cargo caches git repositories in a few locations to speed up
local usage of git repositories. Cargo has a "database" folder which is a bare
checkout of any git repository Cargo has cached historically. This database
folder contains effectively a bunch of databases for remote repos that are
updated periodically.
When actually building a crate Cargo will clone this database into a different
location, the checkouts folder. Each rev we build (ever) is cached in the
checkouts folder. This means that once a checkout directory is created it's
frozen for all of time.
This latter step is what this commit is optimizing. When checking out the
database onto the local filesystem at a particular revision. Previously we were
instructing libgit2 to fall back to a "git aware" transport which was
exceedingly slow on some systems for filesystem-to-filesystem transfers. This
optimization (we just forgot to turn it on in libgit2) is a longstanding one and
should speed this up significantly!
bors [Sat, 6 Jan 2018 18:10:30 +0000 (18:10 +0000)]
Auto merge of #4909 - zackmdavis:tidy_paren, r=matklad
unblock rust-lang/rust#46980 by cleaning up an unused pair of parentheses
While this might seem like too trivial of a stylistic nitpick to deserve
its own commit, this [is needed to unblock](https://github.com/rust-lang/rust/pull/46980#issuecomment-355565906) the PR rust-lang/rust#46980 (which makes
the unused-parens lint look at function arguments, which it previously didn't).
Zack M. Davis [Sat, 6 Jan 2018 17:58:05 +0000 (09:58 -0800)]
clean up an unused pair of parentheses
While this might seem like too trivial of a stylistic nitpick to deserve
its own commit, this is needed to unblock rust-lang/rust#46980 (which makes
the unused-parens lint look at function arguments).
bors [Fri, 5 Jan 2018 19:11:56 +0000 (19:11 +0000)]
Auto merge of #4904 - alexcrichton:clean-docs, r=alexcrichton
Delete the old docs, lift up the new
This commit deletes the old documentation now that the "official source" is the
main book. The book is now lifted up directly into `src/doc` instead of
`src/doc/book`.
The CI no longer builds documentation and has been updated to just run mdbook to
make sure there's no errors. The documentation will actually get published in
the rust-lang/rust repo
Alex Crichton [Fri, 5 Jan 2018 19:08:58 +0000 (11:08 -0800)]
Delete the old docs, lift up the new
This commit deletes the old documentation now that the "official source" is the
main book. The book is now lifted up directly into `src/doc` instead of
`src/doc/book`.
The CI no longer builds documentation and has been updated to just run mdbook to
make sure there's no errors. The documentation will actually get published in
the rust-lang/rust repo
bors [Thu, 4 Jan 2018 21:16:36 +0000 (21:16 +0000)]
Auto merge of #4898 - wking:license-pin-spdx, r=alexcrichton
src/doc/manifest: Pin 'license' to SPDX 2.1 expressions and the 2.4 list
Before this commit, the license-list URL was floating, which lead to issues when manifest authors used IDs from the list that had not yet made it into crate.io's whitelist (#4888). This commit pins both the SPDX-spec version (to avoid floating the license-expresion syntax) and the license-list version (to avoid floating the license/exception identifiers).
I've also deprecated the `/` syntax, since it was not clear whether that was conjunctive (like `AND`) or disjunctive (like `OR`). crates.io is [using license-exprs 1.3^][2], and [that version supports `WITH`, `AND`, `OR`, and the `+` suffix][3], so as far as crates.io-validation is concerned, the value can use vanilla license expressions.
It's unfortunate that there's not an easily-browsable version of the 2.4 license list up anywhere canonical (as far as I know). Starting with the 3.0 license list, there's [a Markdown page in the authoritative list-data repository][4]. I can also see about getting [earlier versions of the HTML][5] up on spdx.org somewhere.
I'm expecting the version-bump procedure will look something like:
1. SPDX cuts a new spec and/or license list release.
2. license-exprs updates to cover the change.
3. crates.io updates to depend on the new license-exprs version.
4. cargo updates the documentation to allow the new version.
5. authors read the new cargo docs and start using the new expression syntax and/or identifiers.
There's a window there where crates.io will be validating to a different version than the cargo docs recommend, so it would be good to have 4 follow 3 as closely as possible. But the SPDX maintainers have been good about providing long deprecation windows, so a bit of a gap is acceptable.
It's possible that crates.io will want to warn authors about their use of deprecated identifiers or syntax (e.g. the `/` I've deprecated here) so they can upgrade before the deprecated element is dropped (probably years after the initial deprecation). That would help limit the gap between 3 and 5 (although warnings sent before 4 might be confusing).
The parallel edits to the two manifest files are based on @carols10cents' recommendation. She [points out on IRC][6] that the goal is to drop the `src/doc/manifest.md` reference soon with the book up on `doc.rust-lang.org/cargo` today.
Fixes #4888, although it would be good to start the upgrade cycle to pull in the [newly-released SPDX license list 3.0][7] so folks can start using the new identifiers.
W. Trevor King [Thu, 4 Jan 2018 18:47:16 +0000 (10:47 -0800)]
src/doc/manifest: Pin 'license' to SPDX 2.1 expressions and the 2.4 list
Before this commit, the license-list URL was floating, which lead to
issues when manifest authors used IDs from the list that had not yet
made it into crate.io's whitelist [1]. This commit pins both the
SPDX-spec version (to avoid floating the license-expresion syntax) and
the license-list version (to avoid floating the license/exception
identifiers).
I've also deprecated the / syntax, since it was not clear whether that
was conjunctive (like AND) or disjunctive (like OR). crates.io is
using license-exprs 1.3^ [2], and that version supports WITH, AND, OR,
and the + suffix [3], so as far as crates.io-validation is concerned,
the value can use vanilla license expressions.
It's unfortunate that there's not an easily-browsable version of the
2.4 license list up anywhere canonical (as far as I know). Starting
with the 3.0 license list, there's a Markdown page in the
authoritative list-data repository [4]. I can also see about getting
earlier versions of the HTML (e.g. [5]) up on spdx.org somewhere.
I'm expecting the version-bump procedure will look something like:
1. SPDX cuts a new spec and/or license list release.
2. license-exprs updates to cover the change.
3. crates.io updates to depend on the new license-exprs version.
4. cargo updates the documentation to allow the new version.
5. authors read the new cargo docs and start using the new expression
syntax and/or identifiers.
There's a window there where crates.io will be validating to a
different version than the cargo docs recommend, so it would be good
to have 4 follow 3 as closely as possible. But the SPDX maintainers
have been good about providing long deprecation windows, so a bit of a
gap is acceptable.
It's possible that crates.io will want to warn authors about their use
of deprecated identifiers or syntax (e.g. the '/' I've deprecated
here) so they can upgrade before the deprecated element is dropped
(probably years after the initial deprecation). That would help limit
the gap between 3 and 5 (although warnings sent before 4 might be
confusing).
The parallel edits to the two manifest files are based on Carol
Nichols' recommendation. She points out on IRC that the goal is to
drop the src/doc/manifest.md reference soon with the book up on
doc.rust-lang.org/cargo today [6].
```
error: linker `cc` not found
|
= note: No such file or directory (os error 2)
error: aborting due to previous error
```
This currently breaks https://github.com/rust-lang/rust/pull/47052.
**NOTE:** I don't know if I handled the "ticks" around the linker name correctly (if cargo test is smart enough to see that the ticks are part of the omitted part). My reasoning was that in the rustc code the ticks are in the source and not part of the linkers name - so if rustc changes, this will probably break again.
bors [Fri, 22 Dec 2017 16:55:35 +0000 (16:55 +0000)]
Auto merge of #4852 - jtgeibel:cargo-install-target-dir, r=alexcrichton
Always respect `CARGO_TARGET_DIR` during `cargo install`
This aligns the behavior of crates.io and `--git` sources with that of `--path`
regarding the `CARGO_TARGET_DIR` and `CARGO_BUILD_TARGET_DIR` environment
variables. If neither environment variable is set, then a temporary directory
is still used when installing from crates.io or `--git`.
As discussed in #4725, this can be used to enable caching of artifacts between
continuous integration builds.
Justin Geibel [Fri, 22 Dec 2017 01:14:12 +0000 (20:14 -0500)]
Always respect `CARGO_TARGET_DIR` during `cargo install`
This aligns the behavior of crates.io and `--git` sources with that of `--path`
regarding the `CARGO_TARGET_DIR` and `CARGO_BUILD_TARGET_DIR` environment
variables. If neither environment variable is set, then a temporary directory
is still used when installing from crates.io or `--git`.
As discussed in #4725, this can be used to enable caching of artifacts between
continuous integration builds.
bors [Thu, 21 Dec 2017 18:57:07 +0000 (18:57 +0000)]
Auto merge of #4817 - alexcrichton:incremental-by-default, r=matklad
Enable incremental by default
This commit enables incremental compilation by default in Cargo for all
dev-related profiles (aka anything without `--release` or `bench`. A
number of new configuration options were also added to tweak how
incremental compilation is exposed and/or used:
* A `profile.dev.incremental` field is added to `Cargo.toml` to disable
it on a per-project basis (in case of bugs).
* A `build.incremental` field was added in `.cargo/config` to disable
globally (or enable if we flip this default back off).
Otherwise `CARGO_INCREMENTAL` can still be used to configure one
particular compilation. The global `build.incremental` configuration
cannot currently be used to enable it for the release profile.
Alex Crichton [Wed, 13 Dec 2017 22:08:16 +0000 (14:08 -0800)]
Enable incremental by default
This commit enables incremental compilation by default in Cargo for all
dev-related profiles (aka anything without `--release` or `bench`. A
number of new configuration options were also added to tweak how
incremental compilation is exposed and/or used:
* A `profile.dev.incremental` field is added to `Cargo.toml` to disable
it on a per-project basis (in case of bugs).
* A `build.incremental` field was added in `.cargo/config` to disable
globally (or enable if we flip this default back off).
Otherwise `CARGO_INCREMENTAL` can still be used to configure one
particular compilation. The global `build.incremental` configuration
cannot currently be used to enable it for the release profile.
bors [Wed, 20 Dec 2017 15:01:01 +0000 (15:01 +0000)]
Auto merge of #4837 - alexcrichton:failed-submodule-checkout, r=matklad
Fix updating submodules past failures
If a submodule-of-a-submodule failed to update then Cargo the next time
around wouldn't automatically retry updating the next submodule. This commit
fixes that by ensuring that if a parent git repository looks updated we still
recurse into its own submodules to ensure they're all updated.
Alex Crichton [Tue, 19 Dec 2017 15:34:14 +0000 (07:34 -0800)]
Fix updating submodules past failures
If a submodule-of-a-submodule failed to update then Cargo the next time
around wouldn't automatically retry updating the next submodule. This commit
fixes that by ensuring that if a parent git repository looks updated we still
recurse into its own submodules to ensure they're all updated.
bors [Tue, 19 Dec 2017 02:48:27 +0000 (02:48 +0000)]
Auto merge of #4795 - alexcrichton:failure, r=withoutboats
Start migration to the `failure` crate
This commit is the initial steps to migrate Cargo's error handling from the
`error-chain` crate to the `failure` crate. This is intended to be a low-cost
(in terms of diff) transition where possible so it's note "purely idiomatic
`failure` crate" just yet.
The `error-chain` dependency is dropped in this commit and Cargo now canonically
uses the `Error` type from the `failure` crate. The main last remnant of
`error-chain` is a custom local extension trait to use `chain_err` instead of
`with_context`. I'll try to follow up with a commit that renames this later but
I wanted to make sure everything worked first! (and `chain_err` is used
practically everywhere).
Some minor tweaks happened in the tests as I touched up a few error messages
here and there but overall the UI of Cargo should be exactly the same before and
after this commit.
bors [Tue, 19 Dec 2017 01:48:52 +0000 (01:48 +0000)]
Auto merge of #4836 - sfackler:dl-template, r=withoutboats
Template a registry's dl field
Previously, crate files were always downloaded from
`/{crate}/{version}/download`. However, if the backing crate store for a
custom registry is a raw file server rather than an API endpoint that
requires every file to be named `download` which is a bit weird. Now a
registry's dl URL can be templated with `{crate}` and `{version}` to
have more control over the resulting path.
For backwards compatibility, we append the default template suffix onto
the dl URL if neither of the template parameters are present for
backwards compatibility.
Alex Crichton [Fri, 8 Dec 2017 19:31:17 +0000 (11:31 -0800)]
Start migration to the `failure` crate
This commit is the initial steps to migrate Cargo's error handling from the
`error-chain` crate to the `failure` crate. This is intended to be a low-cost
(in terms of diff) transition where possible so it's note "purely idiomatic
`failure` crate" just yet.
The `error-chain` dependency is dropped in this commit and Cargo now canonically
uses the `Error` type from the `failure` crate. The main last remnant of
`error-chain` is a custom local extension trait to use `chain_err` instead of
`with_context`. I'll try to follow up with a commit that renames this later but
I wanted to make sure everything worked first! (and `chain_err` is used
practically everywhere).
Some minor tweaks happened in the tests as I touched up a few error messages
here and there but overall the UI of Cargo should be exactly the same before and
after this commit.
bors [Tue, 19 Dec 2017 00:41:01 +0000 (00:41 +0000)]
Auto merge of #4832 - andreastt:dumb_terminal_progress, r=alexcrichton
util/progress: no progress reporting in dumb terminals
cargo should not assume that all terminals have direct access to
the terminal. Dumb terminals are those that can interpret only a
limited number of control codes (CR, LF, &c.) and the escape codes
used by the progress bar breaks output in these by asserting control
over the cursor position to draw a bar.
A dumb terminal is identified by the TERM output variable being set to
"dumb". This adds a direct check for this in src/cargo/util/progress.rs
because TERM=dumb does not imply the same as the -q flag.
Steven Fackler [Mon, 18 Dec 2017 23:22:04 +0000 (15:22 -0800)]
Template a registry's dl field
Previously, crate files were always downloaded from
`/{crate}/{version}/download`. However, if the backing crate store for a
custom registry is a raw file server rather than an API endpoint that
requires every file to be named `download` which is a bit weird. Now a
registry's dl URL can be templated with `{crate}` and `{version}` to
have more control over the resulting path.
For backwards compatibility, we append the default template suffix onto
the dl URL if neither of the template parameters are present for
backwards compatibility.
bors [Mon, 18 Dec 2017 19:30:28 +0000 (19:30 +0000)]
Auto merge of #4818 - alexcrichton:rename-link-paths, r=matklad
Fix renaming a project using build scripts
This commit fixes an issue in Cargo where if a project's folder was
renamed but it also contained a build script the project could break.
Cargo would continue to use the previous `rustc-link-search` arguments
to configure env vars like `LD_LIBRARY_PATH` but the literal values from
the previous compilation would be stale as the directories would no
longer be there.
To fix this when parsing the build script output we now retain a log of
the previous output directory of a build script invocation as well as
the current output, tweaking paths as appropriate if they were contained
in the output folder.
Alex Crichton [Thu, 14 Dec 2017 04:54:01 +0000 (20:54 -0800)]
Fix renaming a project using build scripts
This commit fixes an issue in Cargo where if a project's folder was
renamed but it also contained a build script the project could break.
Cargo would continue to use the previous `rustc-link-search` arguments
to configure env vars like `LD_LIBRARY_PATH` but the literal values from
the previous compilation would be stale as the directories would no
longer be there.
To fix this when parsing the build script output we now retain a log of
the previous output directory of a build script invocation as well as
the current output, tweaking paths as appropriate if they were contained
in the output folder.
Andreas Tolfsen [Mon, 18 Dec 2017 16:01:30 +0000 (16:01 +0000)]
util/progress: no progress reporting in dumb terminals
cargo should not assume that all terminals have direct access to
the terminal. Dumb terminals are those that can interpret only a
limited number of control codes (CR, LF, &c.) and the escape codes
used by the progress bar breaks output in these by asserting control
over the cursor position to draw a bar.
A dumb terminal is identified by the TERM output variable being set to
"dumb". This adds a direct check for this in src/cargo/util/progress.rs
because TERM=dumb does not imply the same as the -q flag.
Errors returned by `to_virtual_manifest` were always ignored. As a result, when something was wrong in a virtual manifest, Cargo would unhelpfully exit with no more output than:
```
error: failed to parse manifest at `/tmp/a/Cargo.toml`
Caused by:
no `package` section found.
```
http://doc.crates.io/manifest.html#virtual-manifest defines a virtual manifest as “the `package` table is not present”, so let’s first determine if a manifest is virtual based on that criteria, and then only call one of the two methods.
Although it is not mentioned in the documentation, `[project]` seems to be in the code an alias for `[package]`. So let’s preserve that here too.
Errors returned by `to_virtual_manifest` were always ignored.
As a result, when something was wrong in a virtual manifest,
Cargo would unhelpfully exit with no more output than:
```
error: failed to parse manifest at `/tmp/a/Cargo.toml`
Caused by:
no `package` section found.
```
http://doc.crates.io/manifest.html#virtual-manifest defines
a virtual manifest as “the `package` table is not present”,
so let’s first determine if a manifest is virtual based
on that criteria, and then only call one of the two methods.
Although it is not mentioned in the documentation,
`[project]` seems to be in the code an alias for `[package]`.
So let’s preserve that here too.
bors [Thu, 14 Dec 2017 23:21:33 +0000 (23:21 +0000)]
Auto merge of #4821 - aidanhs:aphs-allow-overflow, r=alexcrichton
Remove overflow checks to eliminate rust build warnings
Although the checks are desirable, they cause warnings in the rust build (due to workspaces) which could cause needless concern. The checks aren't too important, so just disable them.
Remove overflow checks to eliminate rust build warnings
Although the checks are desirable, they cause warnings in the rust build
(due to workspaces) which could cause needless concern. The checks
aren't too important, so just disable them.
bors [Wed, 13 Dec 2017 21:22:53 +0000 (21:22 +0000)]
Auto merge of #4816 - alexcrichton:fix-workspace-is-home, r=matklad
Defer bailing out workspace root probing
This commit alters the logic to bail out on probing for `CARGO_HOME` to
do it a little later rather than early on in the loop iteration, notably
allowing members to reside in `CARGO_HOME` itself.
Alex Crichton [Wed, 13 Dec 2017 21:19:12 +0000 (13:19 -0800)]
Defer bailing out workspace root probing
This commit alters the logic to bail out on probing for `CARGO_HOME` to
do it a little later rather than early on in the loop iteration, notably
allowing members to reside in `CARGO_HOME` itself.
bors [Tue, 12 Dec 2017 20:00:10 +0000 (20:00 +0000)]
Auto merge of #4788 - alexcrichton:rename-works, r=matklad
Avoid rebuilding a project when cwd changes
This commit is targeted at solving a use case which typically comes up during CI
builds -- the `target` directory is cached between builds but the cwd of the
build changes over time. For example the following scenario can happen:
1. A project is compiled at `/projects/a`.
2. The `target` directory is cached.
3. A new build is started in `/projects/b`.
4. The previous `target` directory is restored to `/projects/b`.
5. The build start, and Cargo rebuilds everything.
The last piece of behavior is indeed unfortunate! Cargo's internal hashing
currently isn't that resilient to changing cwd and this PR aims to help improve
the situation!
The first point of too-much-hashing came up with `Target::src_path`. Each
`Target` was hashed and stored for all compilations, and the `src_path` field
was an absolute path on the filesystem to the file that needed to be compiled.
This path then changed over time when cwd changed, but otherwise everything else
remained the same!
This commit updates the handling of the `src_path` field to simply ignore it
when hashing. Instead the path we actually pass to rustc is later calculated and
then passed to the fingerprint calculation.
The next problem this fixes is that the dep info files were augmented after
creation to have the cwd of the compiler at the time to find the files at a
later date. This, unfortunately, would cause issues if the cwd itself changed.
Instead the cwd is now left out of dep-info files (they're no longer augmented)
and instead the cwd is recalculated when parsing the dep info later.
The final problem that this commit fixes is actually an existing issue in Cargo
today. Right now you can actually execute `cargo build` from anywhere in a
project and Cargo will execute the build. Unfortunately though the argument to
rustc was actually different depending on what directory you were in (the
compiler was invoked with a path relative to cwd). This path ends up being used
for metadata like debuginfo which means that different directories would cause
different artifacts to be created, but Cargo wouldn't rerun the compiler!
To fix this issue the matter of cwd is now entirely excluded from compilation
command lines. Instead rustc is unconditionally invoked with a relative path
*if* the path is underneath the workspace root, and otherwise it's invoked as an
absolute path (in which case the cwd doesn't matter).
Once all these fixes were added up it means that now we can have projects where
if you move the entire directory Cargo won't rebuild the original source!
Note that this may be a bit of a breaking change, however. This means that the
paths in error messages for cargo will no longer be unconditionally relative to
the current working directory, but rather relative to the root of the workspace
itself. Unfortunately this is moreso of a feature right now rather than a bug,
so it may be one that we just have to stomach.
Alex Crichton [Fri, 8 Dec 2017 15:43:45 +0000 (07:43 -0800)]
Change Cargo's own dep-file format
This commit alters the format of the dependency info that Cargo keeps track of
for each crate. In order to be more resilient against directory renames and such
Cargo will now postprocess the compiler's dep-info output and serialize into its
own format. This format is intended to primarily list relative paths *to the
root of the relevant package* rather than absolute or relative to some other
location. If paths aren't actually relative to the package root they're still
stored as absolute, but there's not much we can do about that!
bors [Tue, 12 Dec 2017 05:41:53 +0000 (05:41 +0000)]
Auto merge of #4806 - alexcrichton:fix-infinite-loop, r=matklad
Fix an infinite loop in error reporting
The `path_to_root` function unfortunately didn't account for cycles in the
dependency graph introduced through dev-dependencies, so if a cycle was present
then the function would infinitely loop pushing items onto a vector.
This commit fixes the infinite loop and also touches up the reporting to be a
little more consistent with the rest of Cargo
projects / cargo.git / log