auto merge of #612 : alexcrichton/cargo/nocapture, r=wycats
There are some competing concerns when it comes to the output of compiling
dependencies:
* Not capturing anything leads to getting drowned in unrelated output
* Capturing requires coloration be compromised because of the way windows
terminal colors are implemented.
* Path dependencies are often developed in tandem with the rest of a package,
and capturing their output is not always desired.
To address these concerns, cargo previously captured output of dependent
compilations and then re-printed it to the screen if an error occurred. This
patch modifies the behavior to as follows:
* No output is captured. This preserves any coloration rustc provides.
* All dependencies are compiled with `-Awarnings`. This should suppress any
extraneous output from the compiler and it is considered a bug otherwise if
the compiler prints a warnings when `-Awarnings` is specified.
* All *path* dependencies (`path="..."`, overrides, etc) are *not* compiled with
`-Awarnings`. The reason for this is that you are always in control of these
packages and probably want to see warnings anyway.
Alex Crichton [Sun, 21 Sep 2014 17:22:46 +0000 (10:22 -0700)]
Stop capturing output of all dependencies
There are some competing concerns when it comes to the output of compiling
dependencies:
* Not capturing anything leads to getting drowned in unrelated output
* Capturing requires coloration be compromised because of the way windows
terminal colors are implemented.
* Path dependencies are often developed in tandem with the rest of a package,
and capturing their output is not always desired.
To address these concerns, cargo previously captured output of dependent
compilations and then re-printed it to the screen if an error occurred. This
patch modifies the behavior to as follows:
* No output is captured. This preserves any coloration rustc provides.
* All dependencies are compiled with `-Awarnings`. This should suppress any
extraneous output from the compiler and it is considered a bug otherwise if
the compiler prints a warnings when `-Awarnings` is specified.
* All *path* dependencies (`path="..."`, overrides, etc) are *not* compiled with
`-Awarnings`. The reason for this is that you are always in control of these
packages and probably want to see warnings anyway.
auto merge of #604 : klutzy/cargo/win-make-install, r=alexcrichton
Since `Makefile` passes `--destdir="$$(DESTDIR)/"` to `install.sh`,
`make install` tries to install libraries to
`$CFG_DESTDIR$CFG_PREFIX/$CFG_LIBDIR_RELATIVE` which is usually
`//path/to/usr/$CFG_LIBDIR_RELATIVE`.
The POSIX spec [1] states that if path begins with `//` it is
implementation-defined.
Usual systems treat them as normal abaolute path, but cygwin and MSYS
does not! They use `//hostname/path` syntax for network drives.
This caused `make install` issue on Windows.
This patch removes `/` of destdir to solve the issue.
Since `Makefile` passes `--destdir="$$(DESTDIR)/"` to `install.sh`,
`make install` tries to install libraries to
`$CFG_DESTDIR$CFG_PREFIX/$CFG_LIBDIR_RELATIVE` which is usually
`//path/to/usr/$CFG_LIBDIR_RELATIVE`.
The POSIX spec [1] states that if path begins with `//` it is
implementation-defined.
Usual systems treat them as normal abaolute path, but cygwin and MSYS
does not! They use `//hostname/path` syntax for network drives.
This caused `make install` issue on Windows.
This patch removes `/` of destdir to solve the issue.
auto merge of #596 : alexcrichton/cargo/update-git2, r=brson
This brings in a commit which enables global template options by default as part
of initializing new git repositories, allowing a templates to be used as part of
`cargo new`.
auto merge of #599 : epdtry/cargo/profile-codegen-units, r=alexcrichton
If the profile sets `codegen-units`, the value will be passed to `rustc`'s `-C codegen-units` option. If no value is set, then no argument will be passed, so `rustc` will use its default settings.
Alex Crichton [Thu, 18 Sep 2014 15:51:00 +0000 (08:51 -0700)]
Update git2-rs
This brings in a commit which enables global template options by default as part
of initializing new git repositories, allowing a templates to be used as part of
`cargo new`.
auto merge of #586 : alexcrichton/cargo/fix-win64, r=brson
Right now the win64 snapshot builders are failing to produce a snapshot, and
I've managed to track it down to a path length issue. Windows paths have a
maximum of 260 characters, and the characters add up pretty fast for a path
like:
The normal builders aren't failing I presume because `cargo-nightly-win-64` is
longer than `cargo-win64-64` (we must be *right up* against the limit). I've
confirmed that this shortening fixes the tests on the bots.
Alex Crichton [Wed, 17 Sep 2014 15:30:35 +0000 (08:30 -0700)]
Shorten the cargo integration test directory name
Right now the win64 snapshot builders are failing to produce a snapshot, and
I've managed to track it down to a path length issue. Windows paths have a
maximum of 260 characters, and the characters add up pretty fast for a path
like:
The normal builders aren't failing I presume because `cargo-nightly-win-64` is
longer than `cargo-win64-64` (we must be *right up* against the limit). I've
confirmed that this shortening fixes the tests on the bots.
auto merge of #570 : alexcrichton/cargo/fetch, r=brson
This command is used to download all dependencies of a package ahead of time to
ensure that no more network communication will be necessary as part of a build.
Alex Crichton [Thu, 11 Sep 2014 18:50:57 +0000 (11:50 -0700)]
Implement a `cargo fetch` command
This command is used to download all dependencies of a package ahead of time to
ensure that no more network communication will be necessary as part of a build.
auto merge of #541 : alexcrichton/cargo/cargo-upload, r=brson
This PR implements the plumbing necessary for uploading packages to a registry, downloading packages, depending on those packages, etc. All APIs (upload/download) are tied to the current implementation of [the registry](https://github.com/alexcrichton/cargo-registry).
Most of the design-level details in this PR are in the first commit. The later commits are largely adding tests, polishing it off, and making it work on windows.
Dependency-wise, this picks up a dependency on `curl-rust`, albeit a small fork until the upstream linkage changes are merged. Linkage details can be found [here](https://github.com/carllerche/curl-rust/pull/16). I'll probably have to do some configuration of the bots to get this to work (or hopefully not!).
Alex Crichton [Fri, 18 Jul 2014 15:40:45 +0000 (08:40 -0700)]
Implement a registry source
# cargo upload
The cargo-upload command will take the local package and upload it to the
specified registry. The local package is uploaded as a tarball compressed with
gzip under maximum compression. Most of this is done by just delegating to
`cargo package` The host to upload to is specified, in order of priority, by a
command line `--host` flag, the `registry.host` config key, and then the default
registry. The default registry is still `example.com`
The registry itself is still a work in progress, but the general plumbing for a
command such as this would look like:
1. Ensure the local package has been compressed into an archive.
2. Fetch the relevant registry and login token from config files.
3. Ensure all dependencies for a package are listed as coming from the same
registry.
4. Upload the archive to the registry with the login token.
5. The registry will verify the package is under 2MB (configurable).
6. The registry will upload the archive to S3, calculating a checksum in the
process.
7. The registry will add an entry to the registry's index (a git repository).
The entry will include the name of the package, the version uploaded, the
checksum of the upload, and then the list of dependencies (name/version req)
8. The local `cargo upload` command will succeed.
# cargo login
Uploading requires a token from the api server, and this token follows the same
config chain for the host except that there is no fallback. To implement login,
the `cargo login` command is used. With 0 arguments, the command will request
that a site be visited for a login token, and with an argument it will set the
argument as the new login token.
The `util::config` module was modified to allow writing configuration as well as
reading it. The support is a little lacking in that comments are blown away, but
the support is there at least.
# RegistrySource
An implementation of `RegistrySource` has been created (deleting the old
`DummyRegistrySource`). This implementation only needs a URL to be constructed,
and it is assumed that the URL is running an instance of the cargo registry.
## RegistrySource::update
Currently this will unconditionally update the registry's index (a git
repository). Tuning is necessary to prevent updating the index each time (more
coming soon).
## RegistrySource::query
This is called in the resolve phase of cargo. This function is given a
dependency to query for, and the source will simply look into the index to see
if any package with the name is present. If found, the package's index file will
be loaded and parsed into a list of summaries.
The main optimization of this function is to not require the entire registry to
ever be resident in memory. Instead, only necessary packages are loaded into
memory and parsed.
## RegistrySource::download
This is also called during the resolve phase of cargo, but only when a package
has been selected to be built (actually resolved). This phase of the source will
actually download and unpack the tarball for the package.
Currently a configuration file is located in the root of a registry's index
describing the root url to download packages from.
This function is optimized for two different metrics:
1. If a tarball is downloaded, it is not downloaded again. It is assumed that
once a tarball is successfully downloaded it will never change.
2. If the unpacking destination has a `.cargo-ok` file, it is assumed that the
unpacking has already occurred and does not need to happen again.
With these in place, a rebuild should take almost no time at all.
## RegistrySource::get
This function is simply implemented in terms of a PathSource's `get` function by
creating a `PathSource` for all unpacked tarballs as part of the `download`
stage.
## Filesystem layout
There are a few new directories as part of the `.cargo` home folder:
* `.cargo/registry/index/$hostname-$hash` - This is the directory containing the
actual index of the registry. `$hostname` comes from its url, and `$hash` is
the hash of the entire url.
* `.cargo/registry/cache/$hostname-$hash/$pkg-$vers.tar.gz` - This is a
directory used to cache the downloads of packages from the registry.
* `.cargo/registry/src/$hostname-$hash/$pkg-$vers` - This is the location of the
unpacked packages. They will be compiled from this location.
# New Dependencies
Cargo has picked up a new dependency on the `curl-rust` package in order to send
HTTP requests to the registry as well as send HTTP requests to download
tarballs.
Alex Crichton [Fri, 12 Sep 2014 15:02:14 +0000 (08:02 -0700)]
Rewrite git tests to not rely on the CLI
The CLI has started to have differences on windows than on other systems, and
instead of coding against multiple CLI versions, instead just rewrite everything
to use libgit2.
auto merge of #575 : mbrubeck/cargo/sleep, r=alexcrichton
Travis is showing intermittent failures in `test_cargo_compile_path_deps::path_dep_build_cmd` and `test_cargo_compile_git_deps::git_dep_build_cmd` (added in #561 and #563).
I haven't managed to reproduce these failures locally, but I suspect the tests are timing-sensitive. This tries to fix the failure by sleeping for a second between the two operations.
Matt Brubeck [Sat, 13 Sep 2014 04:49:35 +0000 (21:49 -0700)]
Try to fix intermittent test failures
Travis is showing intermittent failures in
`test_cargo_compile_path_deps::path_dep_build_cmd` and
`test_cargo_compile_git_deps::git_dep_build_cmd` (added in #561 and #563).
I haven't managed to reproduce these failures locally, but I suspect the tests
are timing-sensitive. This tries to guarantee that the timestamps during the
two operations can't be the same.
auto merge of #563 : mbrubeck/cargo/list-files-git, r=alexcrichton
`list_files_git` assumes that the package is at the root of the git repository. This breaks when trying to list files for a package in a subdirectory of a repo, or in other cases, e.g. if the user's home directory is a git repo.
auto merge of #561 : mbrubeck/cargo/list_file_walk, r=alexcrichton
Previously this was calling `.is_file()` on a relative path. This would fail if the path was not relative to the current working directory, for example when listing files in a path dependency.
auto merge of #566 : alexcrichton/cargo/issue-565, r=brson
When cloning a remote repository, the default behavior of libgit2 is to only
update the local ref that's actually checked out, when we actually would prefer
to update all refs of the remote repo. This removes a call to clone() to a
init_bare() + fetch() which should update all refs accordingly
Matt Brubeck [Thu, 11 Sep 2014 21:09:44 +0000 (14:09 -0700)]
Fix list_files_git for paths in subdirs of a repo
list_files_git previously assumed that the package was always at the root of
the git repository. This broke when trying to list files for a package in a
subdirectory of a repo, or in other cases, e.g. if the user's home directory
is a git repo.
Matt Brubeck [Thu, 11 Sep 2014 19:00:11 +0000 (12:00 -0700)]
Fix bug in list_file_walk when cwd != path
Previously this was calling .is_file() on a relative path. This would fail if
the path was not relative to the current working directory, for example when
fingerprinting a path dependency.
Alex Crichton [Thu, 11 Sep 2014 21:30:13 +0000 (14:30 -0700)]
Don't clone remote repos, use fetch instead
When cloning a remote repository, the default behavior of libgit2 is to only
update the local ref that's actually checked out, when we actually would prefer
to update all refs of the remote repo. This removes a call to clone() to a
init_bare() + fetch() which should update all refs accordingly
auto merge of #559 : mbrubeck/cargo/make-install-fix, r=alexcrichton
This fixes an issue where `make dist` incorrectly does nothing after the source has changed and been rebuilt. It also prevents `make install` from installing outdated files from the `dist` directory.
Brian Koropoff [Thu, 11 Sep 2014 05:29:14 +0000 (22:29 -0700)]
Fix build break by upgrading git2-rs
Update git2-rs to the latest version which includes a fix for `extern crate`
syntax changes. This version changes the interface for credential callbacks,
so update the git source provider code as well.
auto merge of #550 : alexcrichton/cargo/rustdoc, r=brson
* All markdown files are now rendered with `rustdoc` into HTML
* A JS syntax highlighter, prism, is included for TOML syntax highlighting.
* A new makefile target, `make doc`, will build docs into `target/doc`
Alex Crichton [Sun, 7 Sep 2014 17:53:20 +0000 (10:53 -0700)]
Move to rustdoc instead of ruby's middleman
* All markdown files are now rendered with `rustdoc` into HTML
* A JS syntax highlighter, prism, is included for TOML syntax highlighting.
* A new makefile target, `make doc`, will build docs into `target/doc`
auto merge of #549 : alexcrichton/cargo/git-no-delete, r=brson
This primarily blows away all *submodules* as well, which sometimes can be quite
large and take some time to update. Instead, re-use an existing checkout, just
reset it to the right revision if possible.
Also, move the submodule update step to occur unconditionally to account for
corrupt submodule checkouts or interrupted downloads. This update step should be
much faster than `git submodule update` because we're using libgit2, so yay!
auto merge of #548 : alexcrichton/cargo/frobbing-git-deps, r=brson
The previous logic for recompiling any dependency was almost entirely based on
the mtimes of the relevant input files. This isn't quite what's desired for git
and registry dependencies because the mtime could be fluctuating while the files
aren't changing. For example:
1. Project A checks out git repo C at revision C1
2. Project A builds, records mtimes of C
3. Project B checks out git repo C at revision C2
4. Project B builds, records new mtimes of C
5. Project A is rebuilt, rebuilding C b/c mtimes are different
In step 5 here C should not be rebuilt because the revision didn't actually
change.
This commit alters git/registry dependencies to completely bypass the --dep-info
emitted and only rely on the git/registry source to inform what the fingerprint
is. This is the revision/version, respectively, and should be all that's
necessary to track changes to the repo and trigger a rebuild.
Alex Crichton [Wed, 10 Sep 2014 14:47:11 +0000 (07:47 -0700)]
Don't blow away checked out git repos
This primarily blows away all *submodules* as well, which sometimes can be quite
large and take some time to update. Instead, re-use an existing checkout, just
reset it to the right revision if possible.
Also, move the submodule update step to occur unconditionally to account for
corrupt submodule checkouts or interrupted downloads. This update step should be
much faster than `git submodule update` because we're using libgit2, so yay!
Alex Crichton [Wed, 10 Sep 2014 14:26:42 +0000 (07:26 -0700)]
Don't recompile git deps so frequently
The previous logic for recompiling any dependency was almost entirely based on
the mtimes of the relevant input files. This isn't quite what's desired for git
and registry dependencies because the mtime could be fluctuating while the files
aren't changing. For example:
1. Project A checks out git repo C at revision C1
2. Project A builds, records mtimes of C
3. Project B checks out git repo C at revision C2
4. Project B builds, records new mtimes of C
5. Project A is rebuilt, rebuilding C b/c mtimes are different
In step 5 here C should not be rebuilt because the revision didn't actually
change.
This commit alters git/registry dependencies to completely bypass the --dep-info
emitted and only rely on the git/registry source to inform what the fingerprint
is. This is the revision/version, respectively, and should be all that's
necessary to track changes to the repo and trigger a rebuild.
projects / cargo.git / log