This PR is extracted from https://github.com/rust-lang/cargo/pull/6864 and then also additionally adds some commits related to performance optimizations that I noticed while profiling https://github.com/rust-lang/cargo/pull/6864. Each commit is in theory standalone and should pass all the tests, as well as being descriptive about what it's doing.