bors [Fri, 2 Jun 2017 22:53:36 +0000 (22:53 +0000)]
Auto merge of #4110 - alexcrichton:jobserver, r=matklad
Add a GNU make jobserver implementation to Cargo
This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.
All the hairy details of the jobserver implementation are housed in the
`jobserver` crate on crates.io instead of Cargo. This should hopefully make it
much easier for the compiler to also share a jobserver implementation
eventually.
The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the `jobserver` crate.
Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual `make -jN` and this overrides the
jobserver in the environment, creating a new jobserver in the sub-make. If the
`-jN` argument is removed, however, then `make` will share Cargo's jobserver and
properly limit parallelism.
Alex Crichton [Tue, 30 May 2017 04:09:53 +0000 (21:09 -0700)]
Add a GNU make jobserver implementation to Cargo
This commit adds a GNU make jobserver implementation to Cargo, both as a client
of existing jobservers and also a creator of new jobservers. The jobserver is
actually just an IPC semaphore which manifests itself as a pipe with N bytes
of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol
is then if you want to run a job you read acquire the semaphore (read a byte on
Unix or wait on the semaphore on Windows) and then you release it when you're
done.
All the hairy details of the jobserver implementation are housed in the
`jobserver` crate on crates.io instead of Cargo. This should hopefully make it
much easier for the compiler to also share a jobserver implementation
eventually.
The main tricky bit here is that on Unix and Windows acquiring a jobserver token
will block the calling thread. We need to either way for a running job to exit
or to acquire a new token when we want to spawn a new job. To handle this the
current implementation spawns a helper thread that does the blocking and sends a
message back to Cargo when it receives a token. It's a little trickier with
shutting down this thread gracefully as well but more details can be found in
the `jobserver` crate.
Unfortunately crates are unlikely to see an immediate benefit of this once
implemented. Most crates are run with a manual `make -jN` and this overrides the
jobserver in the environment, creating a new jobserver in the sub-make. If the
`-jN` argument is removed, however, then `make` will share Cargo's jobserver and
properly limit parallelism.
bors [Wed, 31 May 2017 22:51:31 +0000 (22:51 +0000)]
Auto merge of #4090 - jluner:master, r=alexcrichton
Add error-chain errors.
Fixes #4209
Convert CargoResult, CargoError into an implementation provided by error-chain. The previous is_human machinery is mostly removed; now errors are displayed unless of the Internal kind, verbose mode will print all errors.
bors [Wed, 31 May 2017 21:23:36 +0000 (21:23 +0000)]
Auto merge of #4113 - alexcrichton:trim-travis, r=alexcrichton
Remove lots of dated configuration from this repo
Lots of data build stuff is still here from awhile ago when this repo was
producing Cargo binaries, but the rust-lang/rust repo is now responsible for all
these binaries and build configurations. We no longer need to produce artifacts
or have tons of cross-compiles as rust-lang/rust does all that work, instead
let's just test the likely-to-regress platforms and have rust-lang/rust take
care of the rest.
This commit:
* Deletes the old `configure` script and `Makefile`
* Rewrites `src/doc` management as a shell script
* Trims down Travis/AppVeyor configuration
Alex Crichton [Wed, 31 May 2017 19:55:47 +0000 (12:55 -0700)]
Remove lots of dated configuration from this repo
Lots of data build stuff is still here from awhile ago when this repo was
producing Cargo binaries, but the rust-lang/rust repo is now responsible for all
these binaries and build configurations. We no longer need to produce artifacts
or have tons of cross-compiles as rust-lang/rust does all that work, instead
let's just test the likely-to-regress platforms and have rust-lang/rust take
care of the rest.
This commit:
* Deletes the old `configure` script and `Makefile`
* Rewrites `src/doc` management as a shell script
* Trims down Travis/AppVeyor configuration
jluner [Wed, 31 May 2017 03:15:07 +0000 (22:15 -0500)]
Fixes review comments
Fix some formatting items.
Changes Internal error kind to preserve the original error.
Changes network retry logic to inspect full error chain for spurious
errors.
bors [Fri, 26 May 2017 14:05:10 +0000 (14:05 +0000)]
Auto merge of #4105 - birkenfeld:master, r=alexcrichton
Do not output "Blocking - waiting for lock" with -q
This is not an error, so it should not be printed unconditionally
to stderr. Since it can appear intermittently (e.g. due to editor
integration calling build every now and then) it will disturb
things that expect exact output from cargo (e.g. test suites).
Georg Brandl [Fri, 26 May 2017 08:07:29 +0000 (10:07 +0200)]
Do not output "Blocking - waiting for lock" with -q
This is not an error, so it should not be printed unconditionally
to stderr. Since it can appear intermittently (e.g. due to editor
integration calling build every now and then) it will disturb
things that expect exact output from cargo (e.g. test suites).
Aleksey Kladov [Thu, 25 May 2017 16:47:15 +0000 (19:47 +0300)]
Ignore only root target directory
We used to ignore all target directories, because it was common to
have multiple packages with different target directories in a single
repository. Now, when workspaces are here, such setups usually have a
single target, and we can .gitignore only it. It's useful because
sometimes you want to have a module named `target` in Rust.
If you use non-workspaced multi-package setup, you can create a
.gitignore with `/target/` for each package.
jluner [Thu, 25 May 2017 04:45:14 +0000 (23:45 -0500)]
Addresses review comments
* rebased
* removed `human` (deferring removing `internal` to a later PR)
* cargo_test.rs - fails on other error kinds
* unnecessary `map_err(CargoError::from)` removed
* fold NetworkError entirely into CargoError
* added justification comment for `extend_lifetime`
* various formatting goofs
The following tests are currently failing:
* `http_auth_offered`
* `custom_build_script_failed`
* `build_deps_for_the_right_arch`
* `dep_with_bad_submodule`
* `update_with_shared_deps`
* `finds_author_email`
* `finds_author_user`
* `finds_author_user_escaped`
* `finds_author_username`
* `finds_git_author`
* `exit_code`
jluner [Wed, 24 May 2017 04:35:54 +0000 (23:35 -0500)]
Add error-chain errors
Convert CargoResult, CargoError into an implementation provided by error-chain. The previous is_human machinery is mostly removed; now errors are displayed unless of the Internal kind, verbose mode will print all errors.
bors [Wed, 24 May 2017 15:46:58 +0000 (15:46 +0000)]
Auto merge of #4088 - Nemikolh:buildscript-stderr, r=alexcrichton
Write stderr output from build-scripts next to stdout output
Closes #3462.
Please let me know if you want to change the file name for the error output. I originally thought that `stdout` and `stderr` would have been nice but I'm worried that changing `output` to `stdout` would cause breakage. So I choose the more conservative route.
jluner [Wed, 24 May 2017 04:35:54 +0000 (23:35 -0500)]
Add error-chain errors
Convert CargoResult, CargoError into an implementation provided by error-chain. The previous is_human machinery is mostly removed; now errors are displayed unless of the Internal kind, verbose mode will print all errors.
bors [Thu, 18 May 2017 09:34:53 +0000 (09:34 +0000)]
Auto merge of #4030 - alexcrichton:rewrite-cargo-toml, r=matklad
Rewrite Cargo.toml when packaging crates
This commit is an implementation of rewriting TOML manifests when we publish
them to the registry. The rationale for doing this is to provide a guarantee
that downloaded tarballs from crates.io can be built with `cargo build`
(literally). This in turn eases a number of other possible consumers of crates
from crates.io
* Vendored sources can now be more easily modified/checked as cargo build should
work and they're standalone crates that suffice for `path` dependencies
* Tools like cargobomb/crater no longer need to edit the manifest and can
instead perform regression testing on the literal tarballs they download
* Other systems such as packaging Rust code may be able to take advantage of
this, but this is a less clear benefit.
Overall I'm hesitatnt about this, unfortunately. This is a silent translation
happening on *publish*, a rare operation, that's difficult to inspect before it
flies up to crates.io. I wrote a script to run this transformation over all
crates.io crates and found a surprisingly large number of discrepancies. The
transformation basically just downloaded all crates at all versions,
regenerated the manifest, and then tested if the two manifests were (in memory)
the same.
Unfortunately historical Cargo had a critical bug which I think made this
exercise not too useful. Cargo used to *not* recreate tarballs if one already
existed, which I believe led to situations such as:
1. `cargo publish`
2. Cargo generates an error about a dependency. This could be that there's a
`version` not present in a `path` dependency, there could be a `git`
dependency, etc.
3. Errors are fixed.
4. `cargo publish`
5. Publish is successful
In step 4 above historical Cargo *would not recreate the tarball*. This means
that the contents of the index (what was published) aren't guaranteed to match
with the tarball's `Cargo.toml`. When building from crates.io this is ok as the
index is the source of truth for dependency information, but it means that *any*
transformation to rewrite Cargo.toml is impossible to verify against all crates
on crates.io (due to historical bugs like these).
I strove to read as many errors as possible regardless, attempting to suss out
bugs in the implementation here. To further guard against surprises I've updated
the verification step of packaging to work "normally" in these sense that it's
not rewriting dependencies itself or changing summaries. I'm hoping that this
serves as a good last-ditch effort that what we're about to publish will indeed
build as expected when uploaded to crates.io
Overall I'm probably 70% confident in this change. I think it's necessary to
make progress, but I think there are going to be very painful bugs that arise
from this feature. I'm open to ideas to help weed out these bugs ahead of time!
I've done what I can but I fear it may not be entirely enough.
Alex Crichton [Thu, 11 May 2017 05:09:44 +0000 (22:09 -0700)]
Rewrite Cargo.toml when packaging crates
This commit is an implementation of rewriting TOML manifests when we publish
them to the registry. The rationale for doing this is to provide a guarantee
that downloaded tarballs from crates.io can be built with `cargo build`
(literally). This in turn eases a number of other possible consumers of crates
from crates.io
* Vendored sources can now be more easily modified/checked as cargo build should
work and they're standalone crates that suffice for `path` dependencies
* Tools like cargobomb/crater no longer need to edit the manifest and can
instead perform regression testing on the literal tarballs they download
* Other systems such as packaging Rust code may be able to take advantage of
this, but this is a less clear benefit.
Overall I'm hesitatnt about this, unfortunately. This is a silent translation
happening on *publish*, a rare operation, that's difficult to inspect before it
flies up to crates.io. I wrote a script to run this transformation over all
crates.io crates and found a surprisingly large number of discrepancies. The
transformation basically just downloaded all crates at all versions,
regenerated the manifest, and then tested if the two manifests were (in memory)
the same.
Unfortunately historical Cargo had a critical bug which I think made this
exercise not too useful. Cargo used to *not* recreate tarballs if one already
existed, which I believe led to situations such as:
1. `cargo publish`
2. Cargo generates an error about a dependency. This could be that there's a
`version` not present in a `path` dependency, there could be a `git`
dependency, etc.
3. Errors are fixed.
4. `cargo publish`
5. Publish is successful
In step 4 above historical Cargo *would not recreate the tarball*. This means
that the contents of the index (what was published) aren't guaranteed to match
with the tarball's `Cargo.toml`. When building from crates.io this is ok as the
index is the source of truth for dependency information, but it means that *any*
transformation to rewrite Cargo.toml is impossible to verify against all crates
on crates.io (due to historical bugs like these).
I strove to read as many errors as possible regardless, attempting to suss out
bugs in the implementation here. To further guard against surprises I've updated
the verification step of packaging to work "normally" in these sense that it's
not rewriting dependencies itself or changing summaries. I'm hoping that this
serves as a good last-ditch effort that what we're about to publish will indeed
build as expected when uploaded to crates.io
Overall I'm probably 70% confident in this change. I think it's necessary to
make progress, but I think there are going to be very painful bugs that arise
from this feature. I'm open to ideas to help weed out these bugs ahead of time!
I've done what I can but I fear it may not be entirely enough.
bors [Wed, 17 May 2017 01:33:51 +0000 (01:33 +0000)]
Auto merge of #4060 - alexcrichton:nobare, r=matklad
Don't use a bare checkout of the index
Both old and new Cargo share the same index, so be sure to maintain
compatibility by initializing a non-bare repository for the index. Note that
a nightly Cargo still won't check out the index, but when an older Cargo comes
along and tries to check it out then it'll work.
Alex Crichton [Tue, 16 May 2017 14:07:36 +0000 (07:07 -0700)]
Don't use a bare checkout of the index
Both old and new Cargo share the same index, so be sure to maintain
compatibility by initializing a non-bare repository for the index. Note that
a nightly Cargo still won't check out the index, but when an older Cargo comes
along and tries to check it out then it'll work.
bors [Tue, 16 May 2017 01:16:03 +0000 (01:16 +0000)]
Auto merge of #4051 - alexcrichton:fix-flaky-test, r=alexcrichton
Fix a flaky concurrent test with correct locking
The recent refactoring to clone the bare registry left in an accidental path
where index checkouts could clobber one another. This commit updates the logic
with proper locking and attempt ordering, attempting a full retry of the open
operation after the index locked.
Alex Crichton [Mon, 15 May 2017 19:09:16 +0000 (12:09 -0700)]
Fix a flaky concurrent test with correct locking
The recent refactoring to clone the bare registry left in an accidental path
where index checkouts could clobber one another. This commit updates the logic
with proper locking and attempt ordering, attempting a full retry of the open
operation after the index locked.
bors [Mon, 15 May 2017 18:34:40 +0000 (18:34 +0000)]
Auto merge of #3929 - Xion:master, r=alexcrichton
Allow cargo:rustc-env in build scripts
This is an attempt to address issue #2875. Basically, I'm trying to allow build scripts to produce`cargo:rustc-env=FOO=foo` lines in their output. These are then translated to `env!()`-friendly env. vars that rustc is run with.
bors [Sat, 13 May 2017 18:06:26 +0000 (18:06 +0000)]
Auto merge of #3954 - RReverser:run-with, r=alexcrichton
Add support for custom target-specific runners
When `target.$triple.runner` is specified, it will be used for any execution commands by cargo including `cargo run`, `cargo test` and `cargo bench`. The original file is passed to the runner executable as a first argument.
This allows to run tests when cross-comping Rust projects.
This is not a complete solution and might be extended in future for better ergonomics to support passing extra arguments to the runner itself or overriding runner from the command line, but it should already unlock major existing use cases.
When `target.$triple.runner` is specified, it will be used for any execution commands by cargo including `cargo run`, `cargo test` and `cargo bench`. The original file is passed to the runner executable as a first argument.
This allows to run tests when cross-comping Rust projects.
This is not a complete solution, and might be extended in future for better ergonomics to support passing extra arguments to the runner itself or overriding runner from command line, but it should already unlock major existing usecases.
bors [Thu, 11 May 2017 22:05:55 +0000 (22:05 +0000)]
Auto merge of #4026 - alexcrichton:bare-registry, r=matklad
Don't check out the crates.io index locally
This commit moves working with the crates.io index to operating on the git
object layers rather than actually literally checking out the index. This is
aimed at two different goals:
* Improving the on-disk file size of the registry
* Improving cloning times for the registry as the index doesn't need to be
checked out
The on disk size of my `registry` folder of a fresh check out of the index went
form 124M to 48M, saving a good chunk of space! The entire operation took about
0.6s less on a Unix machine (out of 4.7s total for current Cargo). On Windows,
however, the clone operation went from 11s to 6.7s, a much larger improvement!
Alex Crichton [Wed, 10 May 2017 22:00:33 +0000 (15:00 -0700)]
Don't check out the crates.io index locally
This commit moves working with the crates.io index to operating on the git
object layers rather than actually literally checking out the index. This is
aimed at two different goals:
* Improving the on-disk file size of the registry
* Improving cloning times for the registry as the index doesn't need to be
checked out
The on disk size of my `registry` folder of a fresh check out of the index went
form 124M to 48M, saving a good chunk of space! The entire operation took about
0.6s less on a Unix machine (out of 4.7s total for current Cargo). On Windows,
however, the clone operation went from 11s to 6.7s, a much larger improvement!
bors [Thu, 11 May 2017 20:24:47 +0000 (20:24 +0000)]
Auto merge of #4032 - alexcrichton:retry-500, r=matklad
Automatically retry HTTP requests returning 5xx
This commit implements auto-retry for downloading crates from crates.io whenever
a 5xx response is returned. This should help assist with automatic retries
whenever Cargo attempts to download directly from S3 but S3 returns a 500 error,
which is defined as "please retry again".
This logic may be a little eager to retry *all* 500 errors, but there's a
maximum cap on all retries regardless, so hopefully it doesn't result in too
many problems.
Alex Crichton [Thu, 11 May 2017 19:41:13 +0000 (12:41 -0700)]
Automatically retry HTTP requests returning 5xx
This commit implements auto-retry for downloading crates from crates.io whenever
a 5xx response is returned. This should help assist with automatic retries
whenever Cargo attempts to download directly from S3 but S3 returns a 500 error,
which is defined as "please retry again".
This logic may be a little eager to retry *all* 500 errors, but there's a
maximum cap on all retries regardless, so hopefully it doesn't result in too
many problems.
bors [Tue, 9 May 2017 17:15:40 +0000 (17:15 +0000)]
Auto merge of #4006 - mcgoo:cargo_test_dylib, r=alexcrichton
fix `cargo test` of dylib projects for end user runs too
Fixes running `cargo test` and `cargo test --target <target>` for dylib projects.
Moves the logic just landed in https://github.com/rust-lang/cargo/pull/3996 into cargo itself, so cargo sets the dylib path for anyone running `cargo test` or `cargo run`. Current master sets the dylib path only for `cargo test` on cargo itself.
This PR pins to rustup 1.2.0 for the purposes of testing. If https://github.com/rust-lang-nursery/rustup.rs/pull/1093 ends up working out, then this PR would only be important for non-rustup users and people doing cross testing, `cargo test --target <target>`.
Arguably https://github.com/mcgoo/cargo/blob/ed273851f8bc76f726eda4a2e2a7bb470c3718bc/src/cargo/ops/cargo_rustc/context.rs#L249-L253 should point to lib/rustlib/\<host triple\>/lib instead of sysroot/lib, because I think if the libs are different, you will never be able to compile a working plugin anyway, and for the host=target case you get the lib/rustlib/\<host triple\>/lib anyhow. Is there ever a case where the lib/rustlib/\<host triple\>/lib and sysroot/lib versions of the libs would be expected to differ?
This is not a huge deal for me one way or the other - it doesn't impact my workflow at all. I nearly dropped it when I saw @alexcrichton had made it all work in 3996, but I think it's worth doing because it removes a surprise. It certainly would have saved me a couple of days of confusion. Either way, thanks for looking it over.
bors [Tue, 9 May 2017 15:35:20 +0000 (15:35 +0000)]
Auto merge of #4010 - jonhoo:better-incremental-nightly-test, r=alexcrichton
Bring test of nightly in line with tests
#4000 passes `-Zincremental` to `rustc` only on nightly, but uses a different mechanism for detecting nightly than [cargotest does](https://github.com/rust-lang/cargo/blob/9bf9bddd9297cfb5098be6146d85be551c6d4eff/tests/cargotest/lib.rs#L37). This PR brings the two in line, which should hopefully fix the build failure observed in https://github.com/rust-lang/rust/pull/41830#issuecomment-300052969.