bors [Fri, 7 Feb 2020 15:35:03 +0000 (15:35 +0000)]
Auto merge of #7857 - ehuss:fix-build-script-dupe, r=alexcrichton
Fix BuildScriptOutput when a build script is run multiple times.
When I implemented profile build overrides, I created a scenario where a build script could run more than once for different scenarios. See the test for an example.
However, the `BuildScriptOutputs` map did not take this into consideration. This caused multiple build script runs to stomp on the output of previous runs. This is further exacerbated by the new feature resolver in #7820 where build scripts can run with different features enabled.
The solution is to make the map key unique for the Unit it is running for. Since this map must be updated on different threads, `Unit` cannot be used as a key, so I chose just using the metadata hash which is hopefully unique. Most of this patch is involved with the fussiness of getting the correct metadata hash (we want the RunCustomBuild unit's hash). I also added some checks to avoid collisions and assert assumptions.
bors [Thu, 6 Feb 2020 19:38:30 +0000 (19:38 +0000)]
Auto merge of #7874 - ehuss:update-tar, r=Eh2406
Update tar.
Updates to the latest tar. rust-lang/rust is currently on 0.4.20. The change I'm most interested in is fixing the TarError cause/source, so that Cargo reports a better error message. Compare:
```
Caused by:
failed to unpack `/…/registry/src/github.com-1ecc6299db9ec823/curl-sys-0.4.25/curl/docs/libcurl/libcurl-multi.3`
```
to the new version:
```
Caused by:
failed to unpack `curl-sys-0.4.25/curl/docs/libcurl/libcurl-multi.3` into `/…/registry/src/github.com-1ecc6299db9ec823/curl-sys-0.4.25/curl/docs/libcurl/libcurl-multi.3`
Caused by:
No space left on device (os error 28)
```
bors [Thu, 6 Feb 2020 18:57:21 +0000 (18:57 +0000)]
Auto merge of #7872 - ehuss:timings-error, r=Eh2406
Emit report on error with Ztimings.
Previously the report was not saved on error.
I'm not actually sure this is all that useful. I was using it to gather a picture of what was being built (I wasn't actually interested in the timing data). There might be better ways to accomplish what I wanted, but it's a small change, so probably doesn't hurt.
bors [Wed, 5 Feb 2020 04:16:45 +0000 (04:16 +0000)]
Auto merge of #7865 - ehuss:fix-rebuild_sub_package_then_while_package, r=alexcrichton
Fix rebuild_sub_package_then_while_package on HFS.
This test was flaky on HFS ([azure failure](https://dev.azure.com/rust-lang/cargo/_build/results?buildId=20144&view=logs&j=a5e52b91-c83f-5429-4a68-c246fc63a4f7&t=d4864165-4be3-5e34-b483-a6b05303aa68&l=2018)), resulting in this error:
```
Compiling foo v0.0.1 (/Users/runner/runners/2.164.7/work/1/s/target/cit/t750/foo)
error[E0460]: found possibly newer version of crate `b` which `a` depends on
--> src/lib.rs:1:1
|
1 | extern crate a; extern crate b; pub fn toplevel() {}
| ^^^^^^^^^^^^^^^
|
= note: perhaps that crate needs to be recompiled?
= note: the following crate versions were found:
crate `b`: /Users/runner/runners/2.164.7/work/1/s/target/cit/t750/foo/target/debug/deps/libb-98160c67a5811c37.rlib
crate `b`: /Users/runner/runners/2.164.7/work/1/s/target/cit/t750/foo/target/debug/deps/libb-98160c67a5811c37.rmeta
crate `a`: /Users/runner/runners/2.164.7/work/1/s/target/cit/t750/foo/target/debug/deps/liba-7d2b9ccd932a36e9.rmeta
```
There are two race-condition bugs here.
Race 1: The second cargo build command (`cargo build -pb`) would sometimes not build, because it thought `b` is fresh. This can happen when the first build finishes and changing `b/src/lib.rs` happen within the same second. (#5918) The test silently ignored this failure, this is not the cause of the CI failure, though.
Race 2: The first and second build commands work as expected. The third build command fails because it thinks both `a` and `b` are "fresh". However, `b` was just rebuilt, and `a` depends on it, so `a` should have been rebuilt. It thinks `a` is fresh because `a`'s mtime is equal to `b` when `b` finishes compiling within the same second that the first build finished.
The solution here is to make sure the second step happens well after the first. The underlying problem is #5918.
bors [Wed, 5 Feb 2020 02:31:09 +0000 (02:31 +0000)]
Auto merge of #7855 - ehuss:fix-required-features-renamed, r=alexcrichton
Fix required-features using renamed dependencies.
The dep_name/feat_name syntax in required-features should use the "name in toml" for dep_name, not the actual package name. Otherwise, it is impossible to actually enable the feature (because the `--features` flag expects a "name in toml").
bors [Tue, 4 Feb 2020 22:23:31 +0000 (22:23 +0000)]
Auto merge of #7860 - ehuss:fix-std-collision, r=alexcrichton
Fix build-std collisions.
`build-std` unit filenames can collide with user dependencies in some situations. In particular, `cc` as a build-dependency is likely to be exactly the same as a user's dep. This would result in the `cc` crate being built twice, but with the same filename, causing a collision.
Other dependencies typically never collide because they have the `rustc-dep-of-std` feature, but build-dependencies do not.
The solution here is to include `is_std` in the metadata hash.
bors [Fri, 31 Jan 2020 21:56:58 +0000 (21:56 +0000)]
Auto merge of #7837 - ehuss:fix-global-opt-alias, r=alexcrichton
Fix using global options before an alias.
Options before an alias were being ignored (like `cargo -v b`). The solution is to extract those global options before expanding an alias, and then merging it later.
An alternative to this is to try to avoid discarding the options during expansion, but I couldn't figure out a way to get the position of the subcommand in the argument list. Clap only provides a way to get the arguments *following* the subcommand.
I also cleaned up some of the code in `Config::configure`, which was carrying some weird baggage from previous refactorings.
bors [Fri, 31 Jan 2020 11:50:31 +0000 (11:50 +0000)]
Auto merge of #7823 - ehuss:stabilize-config-profile, r=alexcrichton
Stabilize config-profile.
This is a proposal to stabilize config-profiles. This feature was proposed in [RFC 2282](https://github.com/rust-lang/rfcs/pull/2282) and implemented in #5506. Tracking issue is rust-lang/rust#48683.
This is intended to land in 1.43 which will reach the stable channel on April 23rd.
This is a fairly straightforward extension of profiles where the exact same syntax from `Cargo.toml` can be specified in a config file. Environment variables are supported for everything except the `package` override table, where we do not support the ability to read arbitrary keys in the environment name.
bors [Thu, 30 Jan 2020 06:51:41 +0000 (06:51 +0000)]
Auto merge of #7847 - rust-lang:dependabot/cargo/pretty_env_logger-0.4, r=alexcrichton
Update pretty_env_logger requirement from 0.3 to 0.4
Updates the requirements on [pretty_env_logger](https://github.com/seanmonstar/pretty-env-logger) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/fa4e28537f153a20e655349556ec982e3657a848"><code>fa4e285</code></a> v0.4.0</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/693b5e70880ada9fcca6c10766f5c55f656980e8"><code>693b5e7</code></a> Remove chrono dependency</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/28c5ad0cbbc206b661f1e35784fc2a2d2d11a9ed"><code>28c5ad0</code></a> env_logger: 0.6.2 -> 0.7.0</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/0288e4ed4b730239c401b46b78b818fa3875171b"><code>0288e4e</code></a> fixes env goof in README.md</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/76d9fc760608eff42bbf7deb77ad2cab2cd1172e"><code>76d9fc7</code></a> Make env_logger dependency public</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/8ddffae2c556108640af9d4faf95d51f773f2623"><code>8ddffae</code></a> v0.3.1</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/04c1aa50e1f07fa2be33ea6866c05e39a073f015"><code>04c1aa5</code></a> require latest env_logger</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/ce8c2f12fb72f634f8467f2e5e4fc4a19edafa7c"><code>ce8c2f1</code></a> fix deprecated calls</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/67d2e7d68b3449229c4876bb0813bed50d65b207"><code>67d2e7d</code></a> timestamps with milliseconds</li>
<li><a href="https://github.com/seanmonstar/pretty-env-logger/commit/27eaa2bc1bb666b0888027c106d00cc0ab50d2bc"><code>27eaa2b</code></a> fix with_builder_1 example (<a href="https://github-redirect.dependabot.com/seanmonstar/pretty-env-logger/issues/25">#25</a>)</li>
<li>See full diff in <a href="https://github.com/seanmonstar/pretty-env-logger/compare/v0.3.0...v0.4.0">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
Update pretty_env_logger requirement from 0.3 to 0.4
Updates the requirements on [pretty_env_logger](https://github.com/seanmonstar/pretty-env-logger) to permit the latest version.
- [Release notes](https://github.com/seanmonstar/pretty-env-logger/releases)
- [Commits](https://github.com/seanmonstar/pretty-env-logger/compare/v0.3.0...v0.4.0)
bors [Mon, 27 Jan 2020 18:13:10 +0000 (18:13 +0000)]
Auto merge of #7768 - chrisduerr:install-workspaces-from-git, r=ehuss
Search for root manifest with ephemeral workspaces
Fixes #5495.
This seems like it's too simple to just work like this, but after trying a few different things, this was the only solution which worked reliably for me.
I've verified that no `/target` is present in the actual checkout location, the target directory used is actually the one created in `/tmp`.
I've also verified that both workspaces and "normal" packages still install through git and that a normal `cargo install --path` works too (though that doesn't use ephemeral workspaces anyways).
Kinrany [Mon, 27 Jan 2020 16:42:20 +0000 (19:42 +0300)]
Add tests
Test 1: init a library crate with a `rustfmt.toml` config file in it.
Expect the generated source files to be formatted according to the
config file.
Test 2: same as test 1, but with missing `rustfmt`. Expect `cargo init`
to ignore the absence of `rustfmt` and generate source files
with default formatting.
bors [Sun, 26 Jan 2020 01:15:51 +0000 (01:15 +0000)]
Auto merge of #7829 - Mark-Simulacrum:fix-progress-panics, r=ehuss
Store maximum queue length
Previously, the queue length was constantly decreasing as we built crates, which
meant that we were incorrectly displaying the progress bar. In debug builds,
this even led to panics (due to underflow on subtraction).
Not sure if we can add a test case for this. I have made the panic unconditional on release/debug though by explicitly checking that current is less than the maximum for the progress bar.
Mark Rousskov [Sat, 25 Jan 2020 19:20:40 +0000 (14:20 -0500)]
Store maximum queue length
Previously, the queue length was constantly decreasing as we built crates, which
meant that we were incorrectly displaying the progress bar. In debug builds,
this even led to panics (due to underflow on subtraction).
kinrany [Sat, 25 Jan 2020 02:41:22 +0000 (05:41 +0300)]
Log entry 2: first implementation
I think I found the correct place in the algorithm where rustfmt should
be called.
I'm not sure if using `std::process::Command` and converting errors into
warnings is the correct way here.
Also the warning should only be printed once. Unless
`config.shell().warn()` deduplicates warnings, right now there will be
one warning per source file.
So I should probably check that `rustfmt` exists only once and disable
formatting if it doesn't.
bors [Fri, 24 Jan 2020 18:26:23 +0000 (18:26 +0000)]
Auto merge of #7826 - eddyb:recursion-limit-diagnostic, r=Eh2406
test: allow some flexibility in check::error_from_deep_recursion's expected diagnostic.
This should unblock https://github.com/rust-lang/rust/pull/68407, by loosening the expected output pattern.
As per https://github.com/rust-lang/rust/pull/68407#issuecomment-578189644, this is the change in the diagnostic:
```diff
-recursion limit reached while expanding the macro `m`
+recursion limit reached while expanding `m!`
```
Ideally I would use something like this regex:
```
recursion limit reached while expanding (the macro `m`|`m!`)
```
but AFAIK these tests don't support regexes.
There were no tests for `cargo owner -a/-r` and `cargo yank --undo`. However, These commands in tests had empty response. So I also update response handling with reference to https://github.com/rust-lang/cargo/pull/3301/commits/7dd0f932a864d97bef5da26a0e148fca3f06d448.
bors [Wed, 22 Jan 2020 22:42:20 +0000 (22:42 +0000)]
Auto merge of #7731 - Mark-Simulacrum:chatty-jobserver, r=alexcrichton
Scalable jobserver for rustc
This refactors the job queue code for support of [per-rustc process jobservers](https://github.com/rust-lang/rust/pull/67398). Furthermore, it also cleans up the code and refactors the main loop to be more amenable to understanding (splitting into methods and such).
Assignment of tokens to either rustc "threads" or processes is dedicated to the main loop, which proceeds in a strict "least recently requested" fashion among both thread and process token requests. Specifically, we will first allocate tokens to all pending process token requests (i.e., high-level units of work), and then (in per-rustc jobserver mode) follow up by assigning any remaining tokens to rustcs, again in the order that requests came into cargo (first request served first).
It's not quite clear that that model is good (no modeling or so has been done). On the other hand this strategy should mean that long-running crates will get more thread tokens once we bottom out in terms of rustc parallelism than short-running crates, which means that crates like syn which start early on but finish pretty late should hopefully get more parallelism nicely (without any more complex heuristics).
One plausible change that may be worth exploring is making the assignment prefer earlier rustc's, globally, rather than first attempting to spawn new crates and only then increasing parallelism for old crates. syn for example frequently gets compiled in the early storm of dozens of crates so is somewhat unlikely to have parallelism, until fairly late in its compilation.
We also currently conflate under this model the rayon threads and codegen threads. Eventually inside rustc those will probably(?) also be just one thing, and the rustc side of this implementation provides no information as to what the token request is for so we can't do better here yet.
Mark Rousskov [Mon, 20 Jan 2020 17:50:04 +0000 (12:50 -0500)]
Refactor to_send_clients to use a BTreeMap
This is both a performance optimization (avoiding O(n) shifting from the
beginning), and communicates intent in a nicer way overall.
It is plausible that we will eventually want to tie this data structure to
something like the DependencyQueue, i.e., to get more information on which rustc
to give tokens to. An old rustc with a very late dependency edge is less
important than one we'll need sooner, probably.
Mark Rousskov [Thu, 16 Jan 2020 01:35:51 +0000 (20:35 -0500)]
Do not send acquired tokens to waiting rustc threads
This removes the ad-hoc token re-send in the message processing; this sort of
decision should be left up to the main loop which manages tokens.
Notably, the behavior change here is that new tokens will go solely to spawning
new rustc *processes* rather than increasing rustc internal parallelism, unless
we can't spawn new processes.
Otherwise, before this commit, we may be saturating a single rustc with tokens
rather than creating lots of rustcs that can work in parallel. In particular in
the beginning of a build, it's likely that this is worse (i.e., crates are small
and rustc internal parallelism is not at that point all that helpful) since it
severely limits the benefits of pipelining and generally makes the build
nearly serial.
Mark Rousskov [Thu, 16 Jan 2020 01:24:05 +0000 (20:24 -0500)]
Split waiting for an event out of the primary drainer
This has the slight behavior change where we won't ask for new dependencies and
so forth if no events have been received but I believe that there's no activity
that can happen if an event hasn't occurred (i.e., no state change has occurred)
so there's no need for us to actually do anything in practice.
To make sure we still record CPU usage and such sufficiently often that is also
moved into the inner "waiting for events" loop.
bors [Wed, 22 Jan 2020 00:59:36 +0000 (00:59 +0000)]
Auto merge of #7819 - ehuss:fix-replay-newlines, r=alexcrichton
Fix cache replay including extra newlines.
The compiler output cache replay was changed in #7737 to use `BufReader::read_line` instead of `str::lines`. `read_line`, unlike `lines`, includes the trailing line ending. The code is written assuming that the line endings are stripped, so make sure they are stripped here, too.
This only happens for non-JSON messages, like `RUSTC_LOG`.
bors [Tue, 21 Jan 2020 16:15:39 +0000 (16:15 +0000)]
Auto merge of #7798 - jnbr:dylib_path, r=alexcrichton
Fix wrong directories in host_libdir.
This fixes a regression from #7482 where the sysroot_target_libdir leaks into the host libdir. This can cause problems when the dynamic linker does not ignore the target libraries but tries to load them instead. This happens for example when building on x86_64-musl for aarch64-musl.
bors [Mon, 20 Jan 2020 20:33:50 +0000 (20:33 +0000)]
Auto merge of #7815 - rust-lang:dependabot/cargo/humantime-2.0.0, r=Eh2406
Update humantime requirement from 1.2.0 to 2.0.0
Updates the requirements on [humantime](https://github.com/tailhook/humantime) to permit the latest version.
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/tailhook/humantime/commit/d478f8a7878edba8a609a287776e27fc16d7009b"><code>d478f8a</code></a> Version bumped to v2.0.0</li>
<li><a href="https://github.com/tailhook/humantime/commit/49f11fdc2a59746085d2457cb46bce204dec746a"><code>49f11fd</code></a> Another improvement of the error message</li>
<li><a href="https://github.com/tailhook/humantime/commit/8a13b047ca0dc731e0c515b1db544fe1fd75bd7d"><code>8a13b04</code></a> Nicer error message for plain numeric duration</li>
<li><a href="https://github.com/tailhook/humantime/commit/edfa493e8cb8217aa5a0cc638398f60d67648c93"><code>edfa493</code></a> vagga.yaml: upgrade rust to 1.31.0</li>
<li><a href="https://github.com/tailhook/humantime/commit/8b8d748566c85a73b2f940e755dc0160c93f465a"><code>8b8d748</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/tailhook/humantime/issues/13">#13</a> from gh0st42/master</li>
<li><a href="https://github.com/tailhook/humantime/commit/da7723529fa30222b73561fad0ed7ce47d9a15a4"><code>da77235</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/tailhook/humantime/issues/12">#12</a> from koushiro/works</li>
<li><a href="https://github.com/tailhook/humantime/commit/227d7e591dd76bbfbca6a236d2e8661ddfd16d63"><code>227d7e5</code></a> Made crate unsafe free and forbid unsafe</li>
<li><a href="https://github.com/tailhook/humantime/commit/b7da4ab6ad24f68660cf92ede8dc88d7400a39fa"><code>b7da4ab</code></a> Downgrade MSRV to 1.31</li>
<li><a href="https://github.com/tailhook/humantime/commit/4a406825951039c6c3414b76c855dc38384d874d"><code>4a40682</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/tailhook/humantime/issues/11">#11</a> from koushiro/apply-rustfmt-and-clippy</li>
<li><a href="https://github.com/tailhook/humantime/commit/f00dbbae37ac281c919fbf0d56caf5900f1e0085"><code>f00dbba</code></a> Apply suggestions</li>
<li>Additional commits viewable in <a href="https://github.com/tailhook/humantime/compare/v1.2.0...v2.0.0">compare view</a></li>
</ul>
</details>
<br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language
- `@dependabot badge me` will comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot [dashboard](https://app.dependabot.com):
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
Updates the requirements on [humantime](https://github.com/tailhook/humantime) to permit the latest version.
- [Release notes](https://github.com/tailhook/humantime/releases)
- [Commits](https://github.com/tailhook/humantime/compare/v1.2.0...v2.0.0)