Paolo Bonzini [Thu, 18 May 2023 14:50:00 +0000 (16:50 +0200)]
meson: subprojects: replace submodules with wrap files
Compared to submodules, .wrap files have several advantages:
* option parsing and downloading is delegated to meson
* the commit is stored in a text file instead of a magic entry in the
git tree object
* we could stop shipping external dependencies that are only used as a
fallback, but not break compilation on platforms that lack them.
For example it may make sense to download dtc at build time, controlled
by --enable-download, even when building from a tarball. Right now,
this patch does the opposite: make-release treats dtc like libvfio-user
(which is not stable API and therefore hasn't found its way into any
distros) and keycodemap (which is a copylib, for better or worse).
dependency() can fall back to a wrap automatically. However, this
is only possible for libraries that come with a .pc file, and this
is not very common for libfdt even though the upstream project in
principle provides it; it also removes the control that we provide with
--enable-fdt={system,internal}. Therefore, the logic to pick system
vs. internal libfdt is left untouched.
--enable-fdt=git is removed; it was already a synonym for
--enable-fdt=internal.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Tue, 30 May 2023 15:10:29 +0000 (17:10 +0200)]
git-submodule: allow partial update of .git-submodule-status
Allow a specific subdirectory to run git-submodule.sh with only a
subset of submodules, without removing the others from the
.git-submodule-status file.
This also allows scripts/git-submodule.sh to be more lenient:
validating an empty set of submodules is not a mistake.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Thu, 18 May 2023 14:11:29 +0000 (16:11 +0200)]
configure: rename --enable-pypi to --enable-download, control subprojects too
The behavior of --{enable,disable}-pypi is similar to that of
-Dwrapmode={default,nodownload} respectively. In particular,
in both cases a feature needs to be explicitly enabled for the
dependency to be downloaded.
So, use a single option to control both cases. Now, --enable-slirp
will trigger cloning and building of libslirp if the .pc file
is not found on the machine.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Thu, 25 May 2023 10:36:28 +0000 (12:36 +0200)]
configure: remove --with-git= option
The scenario for which --with-git= was introduced was to use a SOCKS proxy
such as tsocks. However, this was back in 2017 when QEMU's submodules
used the git:// protocol, and it is not as important when using the
"smart HTTP" backend; for example, neither "meson subprojects download"
nor scripts/checkpatch.pl obey the GIT environment variable.
So remove the knob, but test for the presence of git in the configure and
git-submodule.sh scripts, and suggest using --with-git-submodules=validate
+ a manual invocation of git-submodule.sh when git does not work. Hopefully
in the future the GIT environment variable will be supported by Meson.
Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Tue, 6 Jun 2023 08:14:45 +0000 (10:14 +0200)]
mkvenv: always pass locally-installed packages to pip
Let pip decide whether a new version should be installed or the current
one is okay. This ensures that the virtual environment is updated
(either upgraded or downgraded) whenever a new version of a package is
requested.
The hardest part here is figuring out if a package is installed in
the venv (which also has to be done twice to account for the presence
of either setuptools in Python <3.8, or importlib in Python >=3.8).
Suggested-by: Peter Maydell <peter.maydell@linaro.org> Cc: John Snow <jsnow@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Mon, 5 Jun 2023 15:30:38 +0000 (17:30 +0200)]
tests: Use separate virtual environment for avocado
This reverts commits eea2d141179 ("Makefile: remove $(TESTS_PYTHON)",
2023-05-26) and 9c6692db550 ("tests: Use configure-provided pyvenv for
tests", 2023-05-18).
Right now, there is a conflict between wanting a ">=" constraint when
using a distro-provided package and wanting a "==" constraint when
installing Avocado from PyPI; this would provide the best of both worlds
in terms of resiliency for both distros that have required packages and
distros that don't.
The conflict is visible also for meson, where we would like to install
the latest 0.63.x version but also accept a distro 1.1.x version.
But it is worse for avocado, for two reasons:
1) we cannot use an "==" constraint to install avocado if the venv
includes a system avocado. The distro will package plugins that have
"==" constraints on the version that is included in the distro, and, using
"pip install avocado==88.1" on a venv that includes system packages will
result in this error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'
2) we cannot use ">=" either if the venv does _not_ include a system
avocado, because that would result in the installation of v101.0 which
is the one we've just reverted.
So the idea is to encode the dependencies as an (acceptable, locked)
tuple, like this hypothetical TOML that would be committed inside
python/ and used by mkvenv.py:
Once this is implemented, it would also be possible to install avocado in
pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora
and CentOS Stream (the only distros where it's available). But until
this is implemented, keep avocado in a separate venv. There is still the
benefit of using a single python for meson custom_targets and for sphinx.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Bumping avocado to version 101 has two issues. First, there are problems
where Avocado is not logging of command lines or terminal output, and not
collecting Python logs outside the avocado namespace.
Second, the recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Requiring a too-new version of avocado causes conflicts with any avocado
plugins installed on the host:
$ make check-venv
make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
GIT ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
VENVPIP install -e /home/berrange/src/virt/qemu/python/
VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'
To avoid this issue, tests/requirements.txt should use a ">=" constraint
and the version of Avocado should be limited to what distros provide
in the system packages. Only Fedora has Avocado, and more specifically
version 92.0 (though 98.0 is also available as a module). As a first
step, this patch reverts the introduction of a too-new Avocado.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
scsi/qemu-pr-helper: Drop support for 'old' libmultipath API
Commit 1b0578f5c4 ("qemu-pr-helper: Fix build on CentOS 7") added
code to probe for 'old' libmultipath API on CentOS 7. However since
merge commit 8c345b3e6a (June 2021) we don't build/test CentOS 7 as
it felt out of our list of supported distrib versions. Therefore we
can safely remove the 'old' API check (mostly reverting commit 1b0578f5c4, except the code got converted to meson in commit 6ec0e15d95 "meson: move libmpathpersist test").
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230605174146.87440-2-philmd@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Thomas Huth [Mon, 5 Jun 2023 11:45:23 +0000 (13:45 +0200)]
meson.build: Use -Wno-undef only for SDL2 versions that need it
There is no need to disable this useful compiler warning for
all versions of the SDL. Unfortunately, various versions are
buggy (beside SDL 2.0.8, the version 2.26.0 and 2.26.1 are
broken, too, see https://github.com/libsdl-org/SDL/issues/6619 ),
but we can use a simple compiler check to see whether we need
the -Wno-undef or not.
This also enables the printing of the version number with
good versions of the SDL in the summary of the meson output
again.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20230605114523.282987-1-thuth@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Thomas Huth [Fri, 2 Jun 2023 17:18:32 +0000 (19:18 +0200)]
meson.build: Group the audio backend entries in a separate summary section
Let's make it easier for the users to spot audio-related entries
in the summary of the meson output.
Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20230602171832.533739-4-thuth@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Thomas Huth [Fri, 2 Jun 2023 17:18:31 +0000 (19:18 +0200)]
meson.build: Group the network backend entries in a separate summary section
Let's make it easier for the users to spot network-related entries
in the summary of the meson output.
Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20230602171832.533739-3-thuth@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Thomas Huth [Fri, 2 Jun 2023 17:18:30 +0000 (19:18 +0200)]
meson.build: Group the UI entries in a separate summary section
Let's make it easier for the users to spot UI-related entries in
the summary of the meson output.
Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20230602171832.533739-2-thuth@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Sat, 3 Jun 2023 21:09:32 +0000 (23:09 +0200)]
scripts: remove dead file
scripts/test-driver.py was used when "make check" was already using meson
introspection data, but it did not execute "meson test". It is dead since
commit 3d2f73ef75e ("build: use "meson test" as the test harness", 2021-12-23).
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Fri, 3 Mar 2023 13:37:51 +0000 (14:37 +0100)]
atomics: eliminate mb_read/mb_set
qatomic_mb_read and qatomic_mb_set were the very first atomic primitives
introduced for QEMU; their semantics are unclear and they provide a false
sense of safety.
The last use of qatomic_mb_read() has been removed, so delete it.
qatomic_mb_set() instead can survive as an optimized
qatomic_set()+smp_mb(), similar to Linux's smp_store_mb(), but
rename it to qatomic_set_mb() to match the order of the two
operations.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Michal Privoznik [Tue, 30 May 2023 10:31:23 +0000 (12:31 +0200)]
meson: Avoid implicit declaration of absent functions
While detecting a presence of a function via 'cc.links()'
gives desired result (i.e. detects whether function is present),
it also produces a warning on systems where the function is not
present (into meson-log.txt), e.g.:
qemu.git/build/meson-private/tmph74x3p38/testfile.c:2:34: \
warning: implicit declaration of function 'malloc_trim' [-Wimplicit-function-declaration]
And some distributions (e.g. Gentoo) parse the meson log and
consider these erroneous because it can lead to feature
misdetection (see [1]).
We can check whether given function exists via
'cc.has_function()' or whether STATX_* macros exist via
'cc.has_header_symbol()'.
1: https://wiki.gentoo.org/wiki/Modern_C_porting
Resolves: https://bugs.gentoo.org/898810 Signed-off-by: Michal Privoznik <mprivozn@redhat.com>
Message-Id: <8e02776d18595a1c575c90a189ff65f1785f76ca.1685442612.git.mprivozn@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
* tag 'pull-tcg-20230605' of https://gitlab.com/rth7680/qemu: (52 commits)
tcg/tcg-op-vec: Remove left over _link_error() definitions
accel/tcg: Unmap perf_marker
tcg: Build once for system and once for user-only
exec/poison: Do not poison CONFIG_SOFTMMU
plugins: Drop unused headers from exec/plugin-gen.h
plugins: Move plugin_insn_append to translator.c
tcg: Remove target-specific headers from tcg.[ch]
tcg: Move env defines out of NEED_CPU_H in helper-head.h
tcg: Fix PAGE/PROT confusion
accel/tcg: Tidy includes for translator.[ch]
target/arm: Add missing include of exec/exec-all.h
target/*: Add missing includes of exec/translation-block.h
target/mips: Tidy helpers for translation
target/arm: Tidy helpers for translation
accel/tcg: Move translator_fake_ldb out of line
target/ppc: Inline gen_icount_io_start()
accel/tcg: Introduce translator_io_start
accel/tcg: Move most of gen-icount.h into translator.c
include/exec: Remove CODE_GEN_AVG_BLOCK_SIZE
tcg: Spit out exec/translation-block.h
...
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
tcg/tcg-op-vec: Remove left over _link_error() definitions
In commit d56fea79f9 ("tcg: Move TCG_{LOW,HIGH} to tcg-internal.h")
we replaced the "_link_error" definitions with modern QEMU_ERROR()
attribute markup. We covered tcg-op.c but forgot to completely
clean tcg-op-vec.c. Do it now.
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230605175647.88395-3-philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reduce the header to only bswap.h and cpu_ldst.h.
Move exec/translate-all.h to translator.c.
Reduce tcg.h and tcg-op.h to tcg-op-common.h.
Remove otherwise unused headers.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Move most includes from *translate*.c to translate.h, ensuring
that we get the ordering correct. Ensure cpu.h is first.
Use disas/disas.h instead of exec/log.h.
Drop otherwise unused includes.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Move most includes from *translate*.c to translate.h, ensuring
that we get the ordering correct. Ensure cpu.h is first.
Use disas/disas.h instead of exec/log.h.
Drop otherwise unused includes.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
This is used by exactly one host in extraordinary circumstances.
This means that translator.h need not include plugin-gen.h;
translator.c already includes plugin-gen.h.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Now that gen_icount_io_start() is a simple wrapper to
translator_io_start(), inline it.
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230602095439.48102-1-philmd@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
accel/tcg: Move most of gen-icount.h into translator.c
The only usage of gen_tb_start and gen_tb_end are here.
Move the static icount_start_insn variable into a local
within translator_loop. Simplify the two subroutines
by passing in the existing local cflags variable.
Leave only the declaration of gen_io_start in gen-icount.h.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Fixes: 2ac01d6dafab ("translate-all: use a binary search tree to track TBs in TBContext") Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
exec-all: Widen TranslationBlock pc and cs_base to 64-bits
This makes TranslationBlock agnostic to the address size of the guest.
Use vaddr for pc, since that's always a virtual address.
Use uint64_t for cs_base, since usage varies between guests.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
From this remove, it's no longer clear what this is attempting
to protect. The last time a use of this define was added to
the source tree, as opposed to merely moved around, was 2008.
There have been many cleanups since that time and this is
no longer required for the build to succeed.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
tcg: Move TLB_FLAGS_MASK check out of get_alignment_bits
The replacement isn't ideal, as the raw count of bits
is not easily synced with exec/cpu-all.h, but it does
remove from tcg.h the target dependency on TARGET_PAGE_BITS_MIN
which is built into TLB_FLAGS_MASK.
Reviewed-by: Anton Johansson <anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Create helper-proto-common.h without the target specific portion.
Use that in tcg-op-common.h. Include helper-proto.h in target/arm
and target/hexagon before helper-info.c.inc; all other targets are
already correct in this regard.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Create helper-gen-common.h without the target specific portion.
Use that in tcg-op-common.h. Reorg headers in target/arm to
ensure that helper-gen.h is included before helper-info.c.inc.
All other targets are already correct in this regard.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
In preparation for compiling tcg/ only once, eliminate
the all_helpers array. Instantiate the info structs for
the generic helpers in accel/tcg/, and the structs for
the target-specific helpers in each translate.c.
Since we don't see all of the info structs at startup,
initialize at first use, using g_once_init_* to make
sure we don't race while doing so.
Reviewed-by: Anton Johansson <anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
This had been included via tcg-op-common.h via tcg-op.h,
but that is going away. In idef-parser.y, shuffle some
tcg related includes into a more logical order.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Create tcg/tcg-op-common.h, moving everything that does not concern
TARGET_LONG_BITS or TCGv. Adjust tcg/*.c to use the new header
instead of tcg-op.h, in preparation for compiling tcg/ only once.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
The symbol is always defined, even if to 0. We wanted to test for
TCG_OVERSIZED_GUEST == 0.
This fixed, the #error is reached while building arm-softmmu, because
TCG_OVERSIZED_GUEST is not true (nor supposed to be true) for arm32
guest on a 32-bit host. But that's ok, because this feature doesn't
apply to arm32. Add an #ifdef for TARGET_AARCH64.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
This makes CPUTLBEntry agnostic to the address size of the guest.
When 32-bit addresses are in effect, we can simply read the low
32 bits of the 64-bit field. Similarly when we need to update
the field for setting TLB_NOTDIRTY.
For TCG backends that could in theory be big-endian, but in
practice are not (arm, loongarch, riscv), use QEMU_BUILD_BUG_ON
to document and ensure this is not accidentally missed.
For s390x, which is always big-endian, use HOST_BIG_ENDIAN anyway,
to document the reason for the adjustment.
For sparc64 and ppc64, always perform a 64-bit load, and rely on
the following 32-bit comparison to ignore the high bits.
Rearrange mips and ppc if ladders for clarity.
Reviewed-by: Anton Johansson <anjo@rev.ng> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Removes the only use of TARGET_LONG_BITS from tcg.h, which is to be
target independent. Move the symbol to a define in tcg-op.h, which
will continue to be target dependent. Rather than complicate matters
for the use in tb_gen_code(), expand the definition there.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Merge tag 'pull-block-2023-06-05' of https://gitlab.com/hreitz/qemu into staging
Block patches
- Fix padding of unaligned vectored requests to match the host alignment
for vectors with 1023 or 1024 buffers
- Refactor and fix bugs in parallels's image check functionality
- Add an option to the qcow2 driver to retain (qcow2-level) allocations
on discard requests from the guest (while still forwarding the discard
to the lower level and marking the range as zero)
# -----BEGIN PGP SIGNATURE-----
#
# iQJGBAABCAAwFiEEy2LXoO44KeRfAE00ofpA0JgBnN8FAmR+AT4SHGhyZWl0ekBy
# ZWRoYXQuY29tAAoJEKH6QNCYAZzfnboQAKD6YrreZLoseomRfqOAoApSf6yOdcHk
# 6kfsvzwzjosomsF1Pkzm4851vX5PyDqTdeu0iViM+pxanVO1b494q1P4VcAERqMB
# iZVs68R6M0l6HV9btWFGm+ibHJf4FapdntkIdwog1ka5TIhw5oDWCVNLigjhIoRv
# sM37Bgf14kC3sFTR++0HESsyU1eUP5gJjwJbPZ2IgJBmzYay0is1z5nHA/3VUswu
# 8dKnGQDsv62EtlK7PK8cU2BhLOeNi6Wr3bAb6Wf2QLB5e0qRb7oAkqNx5/UcTznk
# a3XMC1aiWhYvM/+DaYIpQUcIPgA8xQ1KHKeD6WjbGfLgZBqseX0aGWMByUsiY8Bo
# +BPIBnUDrbiPnAKB/XLQfnzlE+s7121/JpEbB7AkZqVFRGuw8Wur4tbc2fzvy8Pw
# x/uQfv3ZPi/2Lf6u7hv/TVHubXi8jucVgx3Ubu5Jeo3901S4/KOQBQ4BQ/GYIGQX
# 38ijSROcEd0eQJ1mTKPEctouxjSZCghNSbrn9DfsL1V3VWqWNKKGCU3hM+RQ1SJT
# 688qvnyYt8QZfTsiDSHR/GfKsufG0DkoqE7c9IhSEPohecAH8Rrc3HcLut7fuwD2
# gCFQhm68CPwwRmBjPCY6Zi1RDzeOyFBSWN31T6t0yTb4OHJ/3/cSZVBJtwwkOVbx
# zwabHDNdY5Kw
# =GuoL
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 05 Jun 2023 08:37:34 AM PDT
# gpg: using RSA key CB62D7A0EE3829E45F004D34A1FA40D098019CDF
# gpg: issuer "hreitz@redhat.com"
# gpg: Good signature from "Hanna Reitz <hreitz@redhat.com>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: CB62 D7A0 EE38 29E4 5F00 4D34 A1FA 40D0 9801 9CDF
* tag 'pull-block-2023-06-05' of https://gitlab.com/hreitz/qemu:
qcow2: add discard-no-unref option
parallels: Incorrect condition in out-of-image check
parallels: Replace qemu_co_mutex_lock by WITH_QEMU_LOCK_GUARD
parallels: Move statistic collection to a separate function
parallels: Move check of leaks to a separate function
parallels: Fix statistics calculation
parallels: Move check of cluster outside image to a separate function
parallels: Move check of unclean image to a separate function
parallels: Use generic infrastructure for BAT writing in parallels_co_check()
parallels: create parallels_set_bat_entry_helper() to assign BAT value
parallels: Fix image_end_offset and data_end after out-of-image check
parallels: Fix high_off calculation in parallels_co_check()
parallels: Out of image offset in BAT leads to image inflation
iotests/iov-padding: New test
util/iov: Remove qemu_iovec_init_extended()
block: Collapse padded I/O vecs exceeding IOV_MAX
util/iov: Make qiov_slice() public
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Merge tag 'qemu-sparc-20230605' of https://github.com/mcayland/qemu into staging
qemu-sparc queue
# -----BEGIN PGP SIGNATURE-----
#
# iQFSBAABCgA8FiEEzGIauY6CIA2RXMnEW8LFb64PMh8FAmR9xHseHG1hcmsuY2F2
# ZS1heWxhbmRAaWxhbmRlLmNvLnVrAAoJEFvCxW+uDzIfFsgH/0lV2y9CphGdWwtV
# S/00CRhp5n0kv3P2m4yDvXBhBgMa8zS6m/ZHhcfuU5xOxkJAnPudvXiZucHP4TSy
# 6KFBgVi2+xBW1YJf2UEHlBG2cDnVDHtYQze6Ak3S+W5sVeZZU8n1k8k8kRwpuNDS
# io23oKXzzKMYErlwy12bEHEF7vpfcG9HktjpJxUtppkNDuPTysyKSA6BavEm3mJC
# MlIypbxqxgBvu2SJETeP/m1yLQOaswViCNkqNqUH+MgEc1ubtSCXEuXTfSHIsxLR
# 224kRPFciY1sm3NvG9yz75UYSq1JUvUC5mNIIqwjb4MZJ+YHqFb3S/6ilCYChQ8V
# IBPyvH0=
# =pxbI
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 05 Jun 2023 04:18:19 AM PDT
# gpg: using RSA key CC621AB98E82200D915CC9C45BC2C56FAE0F321F
# gpg: issuer "mark.cave-ayland@ilande.co.uk"
# gpg: Good signature from "Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: CC62 1AB9 8E82 200D 915C C9C4 5BC2 C56F AE0F 321F
* tag 'qemu-sparc-20230605' of https://github.com/mcayland/qemu:
hw/isa/i82378: Remove unused "io" attribute
hw/arm/omap: Remove unused omap_uart_attach()
hw/timer/i8254_common: Share "iobase" property via base class
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
When we for example have a sparse qcow2 image and discard: unmap is enabled,
there can be a lot of fragmentation in the image after some time. Especially on VM's
that do a lot of writes/deletes.
This causes the qcow2 image to grow even over 110% of its virtual size,
because the free gaps in the image get too small to allocate new
continuous clusters. So it allocates new space at the end of the image.
Disabling discard is not an option, as discard is needed to keep the
incremental backup size as low as possible. Without discard, the
incremental backups would become large, as qemu thinks it's just dirty
blocks but it doesn't know the blocks are unneeded.
So we need to avoid fragmentation but also 'empty' the unneeded blocks in
the image to have a small incremental backup.
In addition, we also want to send the discards further down the stack, so
the underlying blocks are still discarded.
Therefor we introduce a new qcow2 option "discard-no-unref".
When setting this option to true, discards will no longer have the qcow2
driver relinquish cluster allocations. Other than that, the request is
handled as normal: All clusters in range are marked as zero, and, if
pass-discard-request is true, it is passed further down the stack.
The only difference is that the now-zero clusters are preallocated
instead of being unallocated.
This will avoid fragmentation on the qcow2 image.
Fixes: https://gitlab.com/qemu-project/qemu/-/issues/1621 Signed-off-by: Jean-Louis Dupond <jean-louis@dupond.be>
Message-Id: <20230605084523.34134-2-jean-louis@dupond.be> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:47 +0000 (11:31 +0200)]
parallels: Incorrect condition in out-of-image check
All the offsets in the BAT must be lower than the file size.
Fix the check condition for correct check.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org>
Message-Id: <20230424093147.197643-13-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:46 +0000 (11:31 +0200)]
parallels: Replace qemu_co_mutex_lock by WITH_QEMU_LOCK_GUARD
Replace the way we use mutex in parallels_co_check() for simplier
and less error prone code.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org>
Message-Id: <20230424093147.197643-12-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:45 +0000 (11:31 +0200)]
parallels: Move statistic collection to a separate function
We will add more and more checks so we need a better code structure
in parallels_co_check. Let each check performs in a separate loop
in a separate helper.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-Id: <20230424093147.197643-11-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:44 +0000 (11:31 +0200)]
parallels: Move check of leaks to a separate function
We will add more and more checks so we need a better code structure
in parallels_co_check. Let each check performs in a separate loop
in a separate helper.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
Message-Id: <20230424093147.197643-10-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:43 +0000 (11:31 +0200)]
parallels: Fix statistics calculation
Exclude out-of-image clusters from allocated and fragmented clusters
calculation.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
Message-Id: <20230424093147.197643-9-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:42 +0000 (11:31 +0200)]
parallels: Move check of cluster outside image to a separate function
We will add more and more checks so we need a better code structure in
parallels_co_check. Let each check performs in a separate loop in a
separate helper.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org>
Message-Id: <20230424093147.197643-8-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:41 +0000 (11:31 +0200)]
parallels: Move check of unclean image to a separate function
We will add more and more checks so we need a better code structure
in parallels_co_check. Let each check performs in a separate loop
in a separate helper.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-Id: <20230424093147.197643-7-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:40 +0000 (11:31 +0200)]
parallels: Use generic infrastructure for BAT writing in parallels_co_check()
BAT is written in the context of conventional operations over the image
inside bdrv_co_flush() when it calls parallels_co_flush_to_os() callback.
Thus we should not modify BAT array directly, but call
parallels_set_bat_entry() helper and bdrv_co_flush() further on. After
that there is no need to manually write BAT and track its modification.
This makes code more generic and allows to split parallels_set_bat_entry()
for independent pieces.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org>
Message-Id: <20230424093147.197643-6-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:39 +0000 (11:31 +0200)]
parallels: create parallels_set_bat_entry_helper() to assign BAT value
This helper will be reused in next patches during parallels_co_check
rework to simplify its code.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-Id: <20230424093147.197643-5-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:38 +0000 (11:31 +0200)]
parallels: Fix image_end_offset and data_end after out-of-image check
Set data_end to the end of the last cluster inside the image. In such a
way we can be sure that corrupted offsets in the BAT can't affect on the
image size. If there are no allocated clusters set image_end_offset by
data_end.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org>
Message-Id: <20230424093147.197643-4-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:37 +0000 (11:31 +0200)]
parallels: Fix high_off calculation in parallels_co_check()
Don't let high_off be more than the file size even if we don't fix the
image.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-Id: <20230424093147.197643-3-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Alexander Ivanov [Mon, 24 Apr 2023 09:31:36 +0000 (11:31 +0200)]
parallels: Out of image offset in BAT leads to image inflation
data_end field in BDRVParallelsState is set to the biggest offset present
in BAT. If this offset is outside of the image, any further write will
create the cluster at this offset and/or the image will be truncated to
this offset on close. This is definitely not correct.
Raise an error in parallels_open() if data_end points outside the image
and it is not a check (let the check to repaire the image). Set data_end
to the end of the cluster with the last correct offset.
Signed-off-by: Alexander Ivanov <alexander.ivanov@virtuozzo.com>
Message-Id: <20230424093147.197643-2-alexander.ivanov@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Hanna Czenczek [Tue, 11 Apr 2023 17:34:18 +0000 (19:34 +0200)]
iotests/iov-padding: New test
Test that even vectored IO requests with 1024 vector elements that are
not aligned to the device's request alignment will succeed.
Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Message-Id: <20230411173418.19549-5-hreitz@redhat.com>
Hanna Czenczek [Tue, 11 Apr 2023 17:34:17 +0000 (19:34 +0200)]
util/iov: Remove qemu_iovec_init_extended()
bdrv_pad_request() was the main user of qemu_iovec_init_extended().
HEAD^ has removed that use, so we can remove qemu_iovec_init_extended()
now.
The only remaining user is qemu_iovec_init_slice(), which can easily
inline the small part it really needs.
Note that qemu_iovec_init_extended() offered a memcpy() optimization to
initialize the new I/O vector. qemu_iovec_concat_iov(), which is used
to replace its functionality, does not, but calls qemu_iovec_add() for
every single element. If we decide this optimization was important, we
will need to re-implement it in qemu_iovec_concat_iov(), which might
also benefit its pre-existing users.
Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Message-Id: <20230411173418.19549-4-hreitz@redhat.com>
Hanna Czenczek [Tue, 11 Apr 2023 17:34:16 +0000 (19:34 +0200)]
block: Collapse padded I/O vecs exceeding IOV_MAX
When processing vectored guest requests that are not aligned to the
storage request alignment, we pad them by adding head and/or tail
buffers for a read-modify-write cycle.
The guest can submit I/O vectors up to IOV_MAX (1024) in length, but
with this padding, the vector can exceed that limit. As of 4c002cef0e9abe7135d7916c51abce47f7fc1ee2 ("util/iov: make
qemu_iovec_init_extended() honest"), we refuse to pad vectors beyond the
limit, instead returning an error to the guest.
To the guest, this appears as a random I/O error. We should not return
an I/O error to the guest when it issued a perfectly valid request.
Before 4c002cef0e9abe7135d7916c51abce47f7fc1ee2, we just made the vector
longer than IOV_MAX, which generally seems to work (because the guest
assumes a smaller alignment than we really have, file-posix's
raw_co_prw() will generally see bdrv_qiov_is_aligned() return false, and
so emulate the request, so that the IOV_MAX does not matter). However,
that does not seem exactly great.
I see two ways to fix this problem:
1. We split such long requests into two requests.
2. We join some elements of the vector into new buffers to make it
shorter.
I am wary of (1), because it seems like it may have unintended side
effects.
(2) on the other hand seems relatively simple to implement, with
hopefully few side effects, so this patch does that.
To do this, the use of qemu_iovec_init_extended() in bdrv_pad_request()
is effectively replaced by the new function bdrv_create_padded_qiov(),
which not only wraps the request IOV with padding head/tail, but also
ensures that the resulting vector will not have more than IOV_MAX
elements. Putting that functionality into qemu_iovec_init_extended() is
infeasible because it requires allocating a bounce buffer; doing so
would require many more parameters (buffer alignment, how to initialize
the buffer, and out parameters like the buffer, its length, and the
original elements), which is not reasonable.
Conversely, it is not difficult to move qemu_iovec_init_extended()'s
functionality into bdrv_create_padded_qiov() by using public
qemu_iovec_* functions, so that is what this patch does.
Because bdrv_pad_request() was the only "serious" user of
qemu_iovec_init_extended(), the next patch will remove the latter
function, so the functionality is not implemented twice.
Hanna Czenczek [Tue, 11 Apr 2023 17:34:15 +0000 (19:34 +0200)]
util/iov: Make qiov_slice() public
We want to inline qemu_iovec_init_extended() in block/io.c for padding
requests, and having access to qiov_slice() is useful for this. As a
public function, it is renamed to qemu_iovec_slice().
(We will need to count the number of I/O vector elements of a slice
there, and then later process this slice. Without qiov_slice(), we
would need to call qemu_iovec_subvec_niov(), and all further
IOV-processing functions may need to skip prefixing elements to
accomodate for a qiov_offset. Because qemu_iovec_subvec_niov()
internally calls qiov_slice(), we can just have the block/io.c code call
qiov_slice() itself, thus get the number of elements, and also create an
iovec array with the superfluous prefixing elements stripped, so the
following processing functions no longer need to skip them.)
Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
Message-Id: <20230411173418.19549-2-hreitz@redhat.com>
Signed-off-by: Bernhard Beschow <shentey@gmail.com> Acked-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Message-Id: <20230523195608.125820-4-shentey@gmail.com> Signed-off-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Signed-off-by: Bernhard Beschow <shentey@gmail.com> Acked-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230523195608.125820-3-shentey@gmail.com> Signed-off-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Bernhard Beschow [Tue, 23 May 2023 19:56:06 +0000 (21:56 +0200)]
hw/timer/i8254_common: Share "iobase" property via base class
Both TYPE_KVM_I8254 and TYPE_I8254 have their own but same implementation of
the "iobase" property. The storage for the property already resides in
PITCommonState, so also move the property definition there.
Signed-off-by: Bernhard Beschow <shentey@gmail.com> Acked-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230523195608.125820-2-shentey@gmail.com> Signed-off-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Jiaxun Yang [Sun, 21 May 2023 10:23:04 +0000 (11:23 +0100)]
hw/intc/loongarch_ipi: Bring back all 4 IPI mailboxes
As per "Loongson 3A5000/3B5000 Processor Reference Manual",
Loongson 3A5000's IPI implementation have 4 mailboxes per
core.
However, in 78464f023b54 ("hw/loongarch/virt: Modify ipi as
percpu device"), the number of IPI mailboxes was reduced to
one, which mismatches actual hardware.
It won't affect LoongArch based system as LoongArch boot code
only uses the first mailbox, however MIPS based Loongson boot
code uses all 4 mailboxes.
Fixes Coverity CID: 1512452, 1512453 Fixes: 78464f023b54 ("hw/loongarch/virt: Modify ipi as percpu device") Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com> Reviewed-by: Song Gao <gaosong@loongson.cn>
Message-Id: <20230521102307.87081-2-jiaxun.yang@flygoat.com> Signed-off-by: Song Gao <gaosong@loongson.cn>
This PULL request get:
- All migration-test patches except last one (daniel)
- Documentation about live test cases (peter)
Please apply.
# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmR5yRwACgkQ9IfvGFhy
# 1yOLQQ/+NsrXEj7Bwp2PdGo+wBRkq4Gah/mc9F00pqtJc2CGNWgfgDohhZjBrhRv
# cTABsfEcIKgCYqGYwVCklJGlUMzxlJPPcMfvou5SWN59E4FBFSg4DWaBfDPCS8LW
# yjnz0JcpxJ+Ge0eqP6xpTPKQ0YGisdav/PjF8GZewBCjyrhZop062a92B2t59D8Y
# shJYKaZZU/5/4zx6KqOm9OClD/yJ+w5q6cGn89/rFE0RMSVywZ3Y1O8/LwIAEP6U
# oj88rczh3geGlsmtPIeyhA3BdnYuPonmyLz8CINFH9+y2tR9l1dN59q1uwEOIvff
# BhJvxTNmkTvsi5zeAmbp2CYmRTwhBmlanh8v2OLNj8zlt0cHYNpiYUZO9qxCHIyT
# LnNTTYhrpqAqINdm+Z8c3ymDKkTz0KECBa45hdFtNB4ZOXPDQTHVqkQRfe3CxDKz
# f/WM4TxHEzVMw/Ow1K9Kbk7/AEwIV6Ol2BSf9D+ZcU4ydmu6ENhV9G4cQ9Orlv8I
# opychxf+O/b6yhVFq7J1ufDhfn3aWQmUQC06npEgfrIV/fLrXhYfs2CXkNZs78v6
# MTMNPNBN/UasM8hx+ldsjZEHf625lO3eNWoNY1Xxog5YICnNLA+tG6n69uybew2+
# UOVyoHwX7iqaToK6bQNCS4H/PjCp3v7fzw1Nsz48Pfaklpivz/k=
# =4exy
# -----END PGP SIGNATURE-----
# gpg: Signature made Fri 02 Jun 2023 03:49:00 AM PDT
# gpg: using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [unknown]
# gpg: aka "Juan Quintela <quintela@trasno.org>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03 4B82 F487 EF18 5872 D723
* tag 'migration-20230602-pull-request' of https://gitlab.com/juan.quintela/qemu:
qtest/migration: Document live=true cases
tests/qtest: make more migration pre-copy scenarios run non-live
tests/qtest: distinguish src/dst migration VM stop/resume events
tests/qtest: capture RESUME events during migration
tests/qtest: replace wait_command() with qtest_qmp_assert_success
tests/qtest: switch to using event callbacks for STOP event
tests/qtest: get rid of some 'qtest_qmp' usage in migration test
tests/qtest: get rid of 'qmp_command' helper in migration test
tests/qtest: add support for callback to receive QMP events
tests/qtest: add various qtest_qmp_assert_success() variants
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Merge tag 'pull-nbd-2023-06-01-v2' of https://repo.or.cz/qemu/ericb into staging
nbd and misc patches for 2023-06-01
- Eric Blake: Fix iotest 104 for NBD
- Eric Blake: Improve qcow2 spec on padding bytes
- Eric Blake: Fix read-beyond-bounds bug in qemu_strtosz
# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCAAdFiEEccLMIrHEYCkn0vOqp6FrSiUnQ2oFAmR6JzEACgkQp6FrSiUn
# Q2oGwgf+PIaN8iedQo5KR08OEf9YxJXab7nL5Oh12+ZvrPOt8XoJcd585KblQ1YI
# 3bGC4CO1l4QO3xmKltVHi7hnlX+3/8WMEvh0jBQBG1AjjPCi5Y1A/gGTEJFX60Ux
# /ffEpo8+1vaHQ8srkxBMWIvpF/dYRaMXSm/CP5SNqTllTalTR46YHKL9odXTzIeN
# 0Zu9UQw/Jwp5A9/8KB+0M9SYXA6zOEmEqEyOwVESEAU2Lm7titwqdBny6GZc6DH6
# Sa2lKO0qQA/e9ya6jHm2c9ycoNCtQ/2VR8QuCd6WCf9DX8q/9RhdiJir+EK5gqp6
# 3JRUFtx783d0BwPnUDUqPawi4txFtw==
# =Cg4e
# -----END PGP SIGNATURE-----
# gpg: Signature made Fri 02 Jun 2023 10:30:25 AM PDT
# gpg: using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg: aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg: aka "[jpeg image of size 6874]" [full]
* tag 'pull-nbd-2023-06-01-v2' of https://repo.or.cz/qemu/ericb: (21 commits)
cutils: Improve qemu_strtosz handling of fractions
cutils: Improve qemu_strtod* error paths
cutils: Use parse_uint in qemu_strtosz for negative rejection
cutils: Set value in all integral qemu_strto* error paths
cutils: Set value in all qemu_strtosz* error paths
test-cutils: Add more coverage to qemu_strtosz
numa: Check for qemu_strtosz_MiB error
cutils: Allow NULL str in qemu_strtosz
test-cutils: Refactor qemu_strtosz tests for less boilerplate
test-cutils: Prepare for upcoming semantic change in qemu_strtosz
test-cutils: Add coverage of qemu_strtod
cutils: Allow NULL endptr in parse_uint()
cutils: Adjust signature of parse_uint[_full]
cutils: Document differences between parse_uint and qemu_strtou64
cutils: Fix wraparound parsing in qemu_strtoui
test-cutils: Test more integer corner cases
test-cutils: Test integral qemu_strto* value on failures
test-cutils: Use g_assert_cmpuint where appropriate
test-cutils: Avoid g_assert in unit tests
qcow2: Explicit mention of padding bytes
...
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Eric Blake [Mon, 22 May 2023 19:04:41 +0000 (14:04 -0500)]
cutils: Improve qemu_strtosz handling of fractions
We have several limitations and bugs worth fixing; they are
inter-related enough that it is not worth splitting this patch into
smaller pieces:
* ".5k" should work to specify 512, just as "0.5k" does
* "1.9999k" and "1." + "9"*50 + "k" should both produce the same
result of 2048 after rounding
* "1." + "0"*350 + "1B" should not be treated the same as "1.0B";
underflow in the fraction should not be lost
* "7.99e99" and "7.99e999" look similar, but our code was doing a
read-out-of-bounds on the latter because it was not expecting ERANGE
due to overflow. While we document that scientific notation is not
supported, and the previous patch actually fixed
qemu_strtod_finite() to no longer return ERANGE overflows, it is
easier to pre-filter than to try and determine after the fact if
strtod() consumed more than we wanted. Note that this is a
low-level semantic change (when endptr is not NULL, we can now
successfully parse with a scale of 'E' and then report trailing
junk, instead of failing outright with EINVAL); but an earlier
commit already argued that this is not a high-level semantic change
since the only caller passing in a non-NULL endptr also checks that
the tail is whitespace-only.
Fixes: https://gitlab.com/qemu-project/qemu/-/issues/1629 Fixes: cf923b78 ("utils: Improve qemu_strtosz() to have 64 bits of precision", 6.0.0) Fixes: 7625a1ed ("utils: Use fixed-point arithmetic in qemu_strtosz", 6.0.0) Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-Id: <20230522190441.64278-20-eblake@redhat.com>
[eblake: tweak function comment for accuracy]
Eric Blake [Mon, 22 May 2023 19:04:40 +0000 (14:04 -0500)]
cutils: Improve qemu_strtod* error paths
Previous patches changed all integral qemu_strto*() error paths to
guarantee that *value is never left uninitialized. Do likewise for
qemu_strtod. Also, tighten qemu_strtod_finite() to never return a
non-finite value (prior to this patch, we were rejecting "inf" with
-EINVAL and unspecified result 0.0, but failing "9e999" with -ERANGE
and HUGE_VAL - which is infinite on IEEE machines - despite our
function claiming to recognize only finite values).
Auditing callers, we have no external callers of qemu_strtod, and
among the callers of qemu_strtod_finite:
- qapi/qobject-input-visitor.c:qobject_input_type_number_keyval() and
qapi/string-input-visitor.c:parse_type_number() which reject all
errors (does not matter what we store)
- utils/cutils.c:do_strtosz() incorrectly assumes that *endptr points
to '.' on all failures (that is, it is not distinguishing between
EINVAL and ERANGE; and therefore still does the WRONG THING for
"9.9e999". The change here does not entirely fix that (a later
patch will tackle this more systematically), but at least it fixes
the read-out-of-bounds first diagnosed in
https://gitlab.com/qemu-project/qemu/-/issues/1629
- our testsuite, which we can update to match what we document
Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> CC: qemu-stable@nongnu.org
Message-Id: <20230522190441.64278-19-eblake@redhat.com>
Eric Blake [Mon, 22 May 2023 19:04:39 +0000 (14:04 -0500)]
cutils: Use parse_uint in qemu_strtosz for negative rejection
Rather than open-coding two different ways to check for an unwanted
negative sign, reuse the same code in both functions. That way, if we
decide down the road to accept "-0" instead of rejecting it, we have
fewer places to change. Also, it means we now get ERANGE instead of
EINVAL for negative values in qemu_strtosz, which is reasonable for
what it represents. This in turn changes the expected output of a
couple of iotests.
The change is not quite complete: negative fractional scaled values
can trip us up. This will be fixed in a later patch addressing other
issues with fractional scaled values.
Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-Id: <20230522190441.64278-18-eblake@redhat.com>
Eric Blake [Mon, 22 May 2023 19:04:38 +0000 (14:04 -0500)]
cutils: Set value in all integral qemu_strto* error paths
Our goal in writing qemu_strtoi() and friends is to have an interface
harder to abuse than libc's strtol(). Leaving the return value
uninitialized on some but not all error paths does not lend itself
well to this goal; and our documentation wasn't helpful on what to
expect.
Note that the previous patch changed all qemu_strtosz() EINVAL error
paths to slam value to 0 rather than stay uninitialized, even when the
EINVAL eror occurs because of trailing junk. But for the remaining
integral qemu_strto*, it's easier to return the parsed value than to
force things back to zero, in part because of how check_strtox_error
works; in part because people expect that from libc strto* (while
there is no libc strtosz to compare to), and in part because doing so
creates less churn in the testsuite.
Here, the list of affected callers is much longer ('git grep
"qemu_strto[ui]" "*.c" "**/*.c" | grep -v tests/ |wc -l' outputs 107,
although a few of those are the implementation in in cutils.c), so
touching as little as possible is the wisest course of action.
Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-Id: <20230522190441.64278-17-eblake@redhat.com>
Eric Blake [Mon, 22 May 2023 19:04:37 +0000 (14:04 -0500)]
cutils: Set value in all qemu_strtosz* error paths
Making callers determine whether or not *value was populated on error
is not nice for usability. Pre-patch, we have unit tests that check
that *result is left unchanged on most EINVAL errors and set to 0 on
many ERANGE errors. This is subtly different from libc strtoumax()
behavior which returns UINT64_MAX on ERANGE errors, as well as
different from our parse_uint() which slams to 0 on EINVAL on the
grounds that we want our functions to be harder to mis-use than
strtoumax().
Let's audit callers:
- hw/core/numa.c:parse_numa() fixed in the previous patch to check for
errors
- migration/migration-hmp-cmds.c:hmp_migrate_set_parameter(),
monitor/hmp.c:monitor_parse_arguments(),
qapi/opts-visitor.c:opts_type_size(),
qapi/qobject-input-visitor.c:qobject_input_type_size_keyval(),
qemu-img.c:cvtnum_full(), qemu-io-cmds.c:cvtnum(),
target/i386/cpu.c:x86_cpu_parse_featurestr(), and
util/qemu-option.c:parse_option_size() appear to reject all failures
(although some with distinct messages for ERANGE as opposed to
EINVAL), so it doesn't matter what is in the value parameter on
error.
- All remaining callers are in the testsuite, where we can tweak our
expectations to match our new desired behavior.
Advancing to the end of the string parsed on overflow (ERANGE), while
still returning 0, makes sense (UINT64_MAX as a size is unlikely to be
useful); likewise, our size parsing code is complex enough that it's
easier to always return 0 when endptr is NULL but trailing garbage was
found, rather than trying to return the value of the prefix actually
parsed (no current caller cared about the value of the prefix).
Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-Id: <20230522190441.64278-16-eblake@redhat.com>
projects / mirror_qemu.git / log