Alex Crichton [Fri, 19 Apr 2019 17:27:58 +0000 (10:27 -0700)]
Remove the `Key` type from `JobQueue`
This isn't actually necessary with a bit of refactoring, and in general
it makes management of `JobQueue` simpler since there's one less type to
worry about. The one main usage of `Key` vs `Unit` was that `Key` was
`Send` to be placed in `Message`, but this was replaced by just
assigning each `Unit` an ever-increasing integer, and then `Finished`
contains the integer rather than the `Key` itself which we can map once
we get back to the main thread.
Alex Crichton [Fri, 19 Apr 2019 16:47:01 +0000 (09:47 -0700)]
Correct a condition to request jobserver tokens
This looks like it was a bug ever present from the original
implementation of a GNU jobserver in #4110, but we currently
unconditionally request a token is allocated for any job we pull off our
job queue. Rather we only need to request tokens for everything but the
first job because we already have an implicit token for that job.
Alex Crichton [Fri, 19 Apr 2019 16:42:53 +0000 (09:42 -0700)]
Remove the `Vec<Job>` from `DependencyQueue`
I... don't think this needed any more! Doing some digging looks like
this was originally added in 79768eb0d. That was so early it didn't even
use a PR and was almost 5 years ago. Since then we've had a huge number
of changes to the backend and `Unit` nowadays does all the deduplication
we need, so no need to store a `Vec` here and we can just have a mapping
of `Key` to `Job` and that's it.
Auto merge of #6863 - daxpedda:patch-1, r=alexcrichton
Improved docs for `maintenance` options
I thought the lack of clear meanings for the available options could use some improvement.
Basically an upgrade of #4431.
Got the relevant text from rust-lang/crates.io#704 and the corresponding [rfc](https://github.com/rust-lang/rfcs/blob/master/text/1824-crates.io-default-ranking.md)
*Questions*
1. Should I put the descriptions outside of the code block?
2. I copied the wording including "maintainer" and "author", should I replace it with "you" or something more appropriate?
I thought the lack of clear meanings for the available options could use some improvement.
Basically an upgrade of #4431.
Got the relevant text from rust-lang/crates.io#704 and the corresponding [rfc](https://github.com/rust-lang/rfcs/blob/master/text/1824-crates.io-default-ranking.md)
*Questions*
1. Should I put the descriptions outside of the code block?
2. I copied the wording including "maintainer" and "author", should I replace it with "you" or something more appropriate?
Auto merge of #6840 - ehuss:publish-lockfile-updates, r=alexcrichton
publish-lockfile: Various updates
This is a collection of changes to the `publish-lockfile` feature. They are separated by commit (see each for more detailed commit messages), and I can separate them into separate PRs if necessary.
- Move publish-lockfile tests to a dedicated file.
- Regenerate `Cargo.lock` when creating a package.
- Ignore `Cargo.lock` in `cargo install` by default (requires `--locked` to use).
- Add warnings for yanked dependencies.
- Change default of publish-lockfile to true.
Eric Huss [Wed, 17 Apr 2019 21:06:05 +0000 (14:06 -0700)]
Include Cargo.lock in package checks.
This includes `Cargo.lock` in the git dirty check. It explicitly excludes
`Cargo.lock` as an untracked file, since it is not relevant for the dirty check;
it is only checked if it is committed to git.
This also adds `Cargo.lock` to the "did anything modify this" check during
verification. I don't see a reason to exclude it (particularly since ephemeral
workspaces do not save the lock file).
Also add "Archiving: Cargo.lock" to the verbose output.
Mike Hommey [Tue, 16 Apr 2019 23:37:46 +0000 (08:37 +0900)]
Avoid adding DYLD_FALLBACK_LIBRARY_PATH defaults when it is set
The macos dynamic linker behavior wrt DYLD_FALLBACK_LIBRARY_PATH is to
use the value it is set with, and if there is no such value (the
environment variable is either not set or set but empty), it uses a
default value of $HOME/lib:/usr/local/lib:/usr/lib.
Currently, cargo takes the value of DYLD_FALLBACK_LIBRARY_PATH, prepends
its paths to it, and then unconditionally adds
$HOME/lib:/usr/local/lib:/usr/lib, which in principle, shouldn't happen
if DYLD_FALLBACK_LIBRARY_PATH was set originally.
Eric Huss [Tue, 16 Apr 2019 18:33:07 +0000 (11:33 -0700)]
Remove `source` from compile_ws.
This was added for `cargo install` to avoid updating the source multiple times.
Now that multiple updates are guarded via `Config::updated_sources`, it is
no longer necessary.
Eric Huss [Sun, 7 Apr 2019 21:58:56 +0000 (14:58 -0700)]
publish-lockfile: Always check Cargo.lock is up-to-date.
This changes it so that `cargo package` will make sure the Cargo.lock file is
in sync with the Cargo.toml that is generated during packaging. This has several
points:
- This makes the Cargo.lock more accurately reflect what would be locked
if a user runs `cargo install` on the resulting package.
- In a workspace, this removes irrelevant packages from the lock file.
- This handles `[patch]` dependencies and dual-source dependencies (like
path/version).
- Warnings are generated for any differences in the lock file compared to the
original.
This has a significant change in how `cargo package` works. It now
unconditionally copies the package to `target/package`. Previously this was only
done during the verification step. This is necessary to run the resolver against
the new `Cargo.toml` that gets generated.
Mike Hommey [Tue, 16 Apr 2019 01:17:26 +0000 (10:17 +0900)]
Use env::var_os("HOME") rather than env::var("HOME")
Because the value is used to create a Path, there's no reason to go all
the way validating it is utf-8 (although in practice, it almost
certainly is, but it doesn't even matter).
Mike Hommey [Tue, 16 Apr 2019 00:53:48 +0000 (09:53 +0900)]
Remove /lib from DYLD_FALLBACK_LIBRARY_PATH
While the manual page for dyld says the default used when
DYLD_FALLBACK_LIBRARY_PATH is not set is
$(HOME)/lib:/usr/local/lib:/lib:/usr/lib, its code actually says
```
sLibraryFallbackPaths[] = { "$HOME/lib", "/usr/local/lib", "/usr/lib", NULL };
```
as far back as the first version of dyld released in OSX 10.4:
https://opensource.apple.com/source/dyld/dyld-43/src/dyld.cpp.auto.html
(and the manual page was wrong back then too
https://opensource.apple.com/source/dyld/dyld-43/doc/man/man1/dyld.1.auto.html)
It is better to avoid putting more paths than necessary in this variable.
I think it is also easier to modify when it is a closure. However, I don't feel strongly about it, so if anyone wants the Clippy form, I'm fine with switching things over to that.
Auto merge of #6849 - johnbartholomew:issue-2511-cargo-non-utf8-args, r=alexcrichton
Pass OsStr/OsString args through to the process spawned by cargo run.
This is intended to fix #2511, allowing non-UTF8 arguments to be passed through from cargo run to the spawned process. I was not sure whether the interface of cargo::ops needs to remain unchanged - I assume it does, so I added cargo::ops::run_os() taking &[OsString], and retained cargo::ops::run() with its current signature. If it's ok to change the internal cargo API then it may be better to just pass &[OsString] to run().
I have not tried to pass through OsStr/OsString to other places that could reasonably use them. Just one test added covering this path. It is restricted to `cfg(unix)` due to use of `std::os::unix::ffi::OsStrExt`.
This is my first pull request so I expect there will be changes needed to make this mergeable.
Auto merge of #6850 - ehuss:fix-include_overrides_gitignore, r=alexcrichton
Fix test include_overrides_gitignore.
This test was added in #4180 and was disabled in #4218. I don't think it is really feasible to get filetime into the test. The test was also using some questionable behavior of modifying contents of `src` from a `build.rs` script. I rewrote the test to directly test the original change of having `package.include` override `.gitignore`.
Auto merge of #6851 - danielcompton:patch-1, r=Eh2406
Clarify optional registry key behaviour
Changing from "and" to "but" makes the sentence clearer to read. I think it gives the reader an appropriate caution, as you do lose features if you don't provide the `api` key.
John Bartholomew [Sun, 14 Apr 2019 14:38:59 +0000 (15:38 +0100)]
Pass OsStr/OsString args through to the process spawned by cargo run.
To avoid breaking other (external) callers of ops::run(), this adds a
new function ops::run_os() taking an &[OsString], and turns ops::run()
into a wrapper (keeping its original signature) that calls run_os().
Auto merge of #6842 - alexcrichton:set-cksum, r=ehuss
Ensure Summary::checksum works for registry crates
While not actually used in Cargo itself the `Summary::checksum` method
of downloaded packages from the registry currently returns `None`. This
was accidentally regressed in #6500 and noticed when updating
`cargo-vendor` (it took a long time to get around to updating that).
The fix here is to override the `checksum` field of the summary for
packages we return from the registry, and everything else should remain
the same!
Alex Crichton [Fri, 12 Apr 2019 16:42:51 +0000 (09:42 -0700)]
Ensure Summary::checksum works for registry crates
While not actually used in Cargo itself the `Summary::checksum` method
of downloaded packages from the registry currently returns `None`. This
was accidentally regressed in #6500 and noticed when updating
`cargo-vendor` (it took a long time to get around to updating that).
The fix here is to override the `checksum` field of the summary for
packages we return from the registry, and everything else should remain
the same!
Auto merge of #6832 - alexcrichton:freshness, r=ehuss
Remove `Freshness` from `DependencyQueue`
Ever since the inception of Cargo and the advent of incremental
compilation at the crate level via Cargo, Cargo has tracked whether it
needs to recompile something at a unit level in its "dependency queue"
which manages when items are ready for execution. Over time we've fixed
lots and lots of bugs related to incremental compilation, and perhaps
one of the most impactful realizations was that the model Cargo started
with fundamentally doesn't handle interrupting Cargo halfway through and
resuming the build later.
The previous model relied upon implicitly propagating "dirtiness" based
on whether the one of the dependencies of a build was rebuilt or not.
This information is not available, however, if Cargo is interrupted and
resumed (or performs a subset of steps and then later performs more).
We've fixed this in a number of places historically but the purpose of
this commit is to put a nail in this coffin once and for all.
Implicit propagation of whether a unit is fresh or dirty is no longer
present at all. Instead Cargo should always know, irrespective of it's
in-memory state, whether a unit needs to be recompiled or not. This
commit actually turns up a few bugs in the test suite, so later commits
will be targeted at fixing this.
Note that this required a good deal of work on the `fingerprint` module
to fix some longstanding bugs (like #6780) and some serious hoops had to
be jumped through for others (like #6779). While these were fallout from
this change they weren't necessarily the primary motivation, but rather
to help make `fingerprints` a bit more straightforward in what's an
already confusing system!
Alex Crichton [Tue, 9 Apr 2019 18:37:31 +0000 (11:37 -0700)]
Purge mtime information from `Fingerprint`
This has proven to be a very unreliable piece of information to hash, so
let's not! Instead we track what files are supposed to be relative to,
and we check both mtimes when necessary.
Alex Crichton [Fri, 5 Apr 2019 19:54:50 +0000 (12:54 -0700)]
Remove `Freshness` from `DependencyQueue`
Ever since the inception of Cargo and the advent of incremental
compilation at the crate level via Cargo, Cargo has tracked whether it
needs to recompile something at a unit level in its "dependency queue"
which manages when items are ready for execution. Over time we've fixed
lots and lots of bugs related to incremental compilation, and perhaps
one of the most impactful realizations was that the model Cargo started
with fundamentally doesn't handle interrupting Cargo halfway through and
resuming the build later.
The previous model relied upon implicitly propagating "dirtiness" based
on whether the one of the dependencies of a build was rebuilt or not.
This information is not available, however, if Cargo is interrupted and
resumed (or performs a subset of steps and then later performs more).
We've fixed this in a number of places historically but the purpose of
this commit is to put a nail in this coffin once and for all.
Implicit propagation of whether a unit is fresh or dirty is no longer
present at all. Instead Cargo should always know, irrespective of it's
in-memory state, whether a unit needs to be recompiled or not. This
commit actually turns up a few bugs in the test suite, so later commits
will be targeted at fixing this.
Note that this required a good deal of work on the `fingerprint` module
to fix some longstanding bugs (like #6780) and some serious hoops had to
be jumped through for others (like #6779). While these were fallout from
this change they weren't necessarily the primary motivation, but rather
to help make `fingerprints` a bit more straightforward in what's an
already confusing system!
Auto merge of #6824 - ehuss:improve-test-rerun-hint, r=alexcrichton
Improve error message to rerun a test in a workspace.
In a non-virtual workspace, if you run `cargo test --all` and something fails, it tells you to rerun `--test foo`, but if `foo` is not the default, then it won't work without a `-p` flag. This tries to be a little more careful about when `-p` is needed in the hint.
Auto merge of #6798 - ehuss:install-upgrade, r=alexcrichton
Add install-upgrade.
This implements the feature described in #6667. Instead of failing when `cargo install` detects a package is already installed, it will upgrade if the versions don't match, or do nothing (exit 0) if it is considered "up-to-date".
Closes #6667.
Notes:
- This feature rejects ambiguous `--version` input (such as `1.0`). This is not required, but seemed like a reasonable time to make the change.
- There is a slight change to the output on stable which reports what was installed/replaced.
- Added better support for `*` in `--version` (don't show warning).
Auto merge of #6823 - spl:clarify-install-path, r=alexcrichton
Clarify docs of install without <crate>
This is an attempt to clarify the documentation of `install` without `<crate>`. I found it confusing trying to determine to which behavior “this” referred as well as whether “supported” meant “maintained” or “worked.”
Auto merge of #6818 - dtolnay:comma, r=alexcrichton
Accept trailing comma in test of impl Debug for PackageId
The standard library is planning to begin emitting trailing commas in multiline Debug representations to align with the dominant style in modern Rust code -- https://github.com/rust-lang/rust/pull/59076.
For now, change this tests to accept both with and without trailing comma. Once the trailing comma change reaches the stable channel we will be able to remove one of the cases.
David Tolnay [Wed, 3 Apr 2019 20:17:16 +0000 (13:17 -0700)]
Accept trailing comma in test of impl Debug for PackageId
The standard library is planning to begin emitting trailing commas in
multiline Debug representations to align with the dominant style in
modern Rust code.
For now, change this tests to accept both with and without trailing
comma. Once the trailing comma change reaches the stable channel we will
be able to remove one of the cases.
Auto merge of #6814 - ehuss:offline-optional-dep, r=alexcrichton
Resolve: Be less strict while offline.
When offline, the resolver was requiring everything to be downloaded, even dependencies that are not used. This changes it so that the resolver can still resolve unavailable dependencies when offline. This pushes the failure to a later stage of Cargo where it attempts to download the dependency. This makes `-Z offline` work for target-cfg or optional dependencies that are not being used.
Fixes #6014.
This changes the error message significantly for the "unavailable" case (see test diff). I personally think the new error message is clearer, although it is shorter and provides less information. The old error message seemed large and scary, and was a little hard for me to grok. However, I'd be willing to look at tweaking the error behavior if not everyone agrees.
Auto merge of #6776 - Eh2406:resolver-simplification, r=alexcrichton
Resolver: A dep is equivalent to one of the things it can resolve to.
This is a series of small changes to the resolver, each one on its own is not worth the cherne, but somehow all together we can add a new optimization rule. The result is that the test in #6283 is no longer exponencial (still a large polynomial, cubick?) and so N can be bumped from 3 to 20. This also means that we pass with all the cases reported in #6258. Resolution is NP-Hard, so we are moving the slow cases around. To reduce the chance that we will be flooded by new bad cases I run the 4 proptests overnight, and they did not find a new exponencial case.
I would recommend reviewing this commit by commit. As each change is pretty simple on its own, but the mixed diff is harder to follow. This is submitted as one big PR as that is @alexcrichton's preference.
A special thanks to @nex3, our conversation was important in convincing me that several of these changes would be needed even in an eventual PubGrub based system. And, the question "why would PubGrub not have a problem with #6283" was wat crystallized this optimization opportunity in my mind.
Auto merge of #6811 - ehuss:include-proc-macro, r=alexcrichton
Include proc-macros in `build-override`.
This adds proc-macros (and their dependencies) to the `build-override` profile setting. The motivation is that these are all "build time" dependencies, and as such should probably behave the same. See the discussion on the [tracking issue](https://github.com/rust-lang/rust/issues/48683#issuecomment-472705245). My intent is that this paves the way for stabilizing without necessarily waiting for #6577.
The only change here is the line in `with_for_host`, the rest is just renaming for clarity.
This also includes some of the testsuite changes from #6577 to make it easier to check for compiler flags.
Alex Crichton [Mon, 1 Apr 2019 17:22:23 +0000 (10:22 -0700)]
Remove no longer needed fields in `BuildConfig`
Now the fields are just all represented as `rustc_wrapper` which
internally contains all process configuration that's later passed down
to `Rustc` as necessary.
Auto merge of #6800 - ehuss:git-cli-force, r=alexcrichton
Support force-pushed repos with git-fetch-with-cli.
If a git repo had a force push, then the `git-fetch-with-cli` feature would fail to fetch an update. This adds the `--force` flag to force the fetch. There's a bit of a lengthy discussion in the [git docs](https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt-ltrefspecgt) about why this is needed when a refspec is supplied.
I also changed it to remove the `--quiet` flag and to capture the output, which is only displayed on error. I'm not sure what the reasoning for the `--quiet` flag was, but I don't see any harm since the output was previously unused. This can maybe help people debug problems, however in this situation
all you would see is ` ! [rejected] master -> master (non-fast-forward)` which isn't terribly informative.
Auto merge of #6801 - ehuss:install-flags, r=Eh2406
cargo install: Be more restrictive about cli flags.
Several flags in `cargo install` are silently ignored depending on what is used. This adds some validation so that invalid combinations are rejected. I have been sorta confused by these in the past.
- The 3 source flags (`--git`, `--path`, and `--registry`) are mutually exclusive.
- The `--git` flags (branch, tag, rev) are only valid if `--git` is specified.
- `--registry` requires a crate name (otherwise it would be ignored and treated as a path source).
- `--version` is only used when a crate name is specified.
Auto merge of #6806 - ehuss:warn-semver-metadata, r=Eh2406
Warn on version req with metadata.
Metadata in a version requirement (such as `1.0.0+1234`) is ignored. This adds a warning that it will be ignored.
On crates.io I found about 5 crates, plus a few dozen google-* crates (presumably all created by the same person) which have dependencies of this form.
See discussion at https://github.com/rust-lang/cargo/issues/6504#issuecomment-451254084. cc rust-lang/crates.io#1059 for ongoing discussion about what to do about publishing such versions.
Auto merge of #6804 - ehuss:install-path-config2, r=alexcrichton
Allow `cargo install --path P` to load config from P.
`cargo install` was changed to ignore configs except for the home directory (#6026). However, it seems like there are legitimate needs when using `--path`, so allow loading from that path, too.
Auto merge of #6803 - ehuss:doc-open-multi, r=alexcrichton
Allow `cargo doc --open` with multiple packages.
If `cargo doc --open` builds multiple packages, open the first one. This seems pretty natural to me (the first one on the command line, or the first default member if `default-members` is specified, or the root of a workspace). Rustdoc shows a list of crates in the sidebar, so if it doesn't open the one the user wants, it's a trivial matter of clicking on the crate name.
@alexcrichton specifically asked for an error [here](https://github.com/rust-lang/cargo/pull/1828#discussion-diff-39650485). However, at the time I don't think rustdoc dynamically generated the "Crates" listing in the sidebar. Alex, I wonder if your stance still holds?
Auto merge of #6791 - ehuss:unstable-help, r=Eh2406
Add some help and documentation for unstable flags.
This is intended to give a little extra information about unstable flags.
- `-Z help` tells you if you can't use it on stable.
- `-Z help` includes link to the unstable chapter.
- `--out-dir` and `--build-plan` on stable give more information about what channels are and a link to their respective tracking issues. Add links in the man pages, too.