Peter Maydell [Thu, 16 Jul 2020 20:46:18 +0000 (21:46 +0100)]
Merge remote-tracking branch 'remotes/ehabkost/tags/x86-next-pull-request' into staging
x86 fixes for -rc1
Fixes for x86 that missed hard freeze:
* Don't trigger warnings for features set by
CPU model versions (Xiaoyao Li)
* Missing features in Icelake-Server, Skylake-Server,
Cascadelake-Server CPU models (Chenyi Qiang)
* Fix hvf x86_64 guest boot crash (Roman Bolshakov)
* remotes/ehabkost/tags/x86-next-pull-request:
i386: hvf: Explicitly set CR4 guest/host mask
target/i386: add the missing vmx features for Skylake-Server and Cascadelake-Server CPU models
target/i386: fix model number and add missing features for Icelake-Server CPU model
target/i386: add fast short REP MOV support
i386/cpu: Don't add unavailable_features to env->user_features
i368/cpu: Clear env->user_features after loading versioned CPU model
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
target/i386: fix model number and add missing features for Icelake-Server CPU model
Add the missing features(sha_ni, avx512ifma, rdpid, fsrm,
vmx-rdseed-exit, vmx-pml, vmx-eptp-switching) and change the model
number to 106 in the Icelake-Server-v4 CPU model.
Xiaoyao Li [Mon, 13 Jul 2020 17:44:36 +0000 (01:44 +0800)]
i386/cpu: Don't add unavailable_features to env->user_features
Features unavailable due to absent of their dependent features should
not be added to env->user_features. env->user_features only contains the
feature explicity specified with -feature/+feature by user.
Xiaoyao Li [Mon, 13 Jul 2020 17:44:35 +0000 (01:44 +0800)]
i368/cpu: Clear env->user_features after loading versioned CPU model
Features defined in versioned CPU model are recorded in env->user_features
since they are updated as property. It's unwated because they are not
user specified.
Simply clear env->user_features as a fix. It won't clear user specified
features because user specified features are filled to
env->user_features later in x86_cpu_expand_features().
Peter Maydell [Thu, 16 Jul 2020 13:46:47 +0000 (14:46 +0100)]
Merge remote-tracking branch 'remotes/stefanberger/tags/pull-tpm-2020-07-15-1' into staging
Merge tpm 2020/07/15 v1
# gpg: Signature made Wed 15 Jul 2020 20:16:21 BST
# gpg: using RSA key B818B9CADF9089C2D5CEC66B75AD65802A0B4211
# gpg: Good signature from "Stefan Berger <stefanb@linux.vnet.ibm.com>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: B818 B9CA DF90 89C2 D5CE C66B 75AD 6580 2A0B 4211
* remotes/stefanberger/tags/pull-tpm-2020-07-15-1:
tests: tpm: Skip over pcrUpdateCounter byte in result comparison
tpm: tpm_spapr: Exit on TPM backend failures
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Peter Maydell [Thu, 16 Jul 2020 12:12:05 +0000 (13:12 +0100)]
Merge remote-tracking branch 'remotes/jasowang/tags/net-pull-request' into staging
# gpg: Signature made Wed 15 Jul 2020 14:49:07 BST
# gpg: using RSA key EF04965B398D6211
# gpg: Good signature from "Jason Wang (Jason Wang on RedHat) <jasowang@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg: It is not certain that the signature belongs to the owner.
# Primary key fingerprint: 215D 46F4 8246 689E C77F 3562 EF04 965B 398D 6211
* remotes/jasowang/tags/net-pull-request:
ftgmac100: fix dblac write test
net: detect errors from probing vnet hdr flag for TAP devices
net: check if the file descriptor is valid before using it
qemu-options.hx: Clean up and fix typo for colo-compare
net/colo-compare.c: Expose compare "max_queue_size" to users
hw/net: Added CSO for IPv6
virtio-net: fix removal of failover device
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Fix the contition to figure whenever we need to wait for more data or
not. Simply check the mode, if we are not in DATAIN state any more we
are done already and don't need to go ASYNC.
Fixes: 7ad3d51ebb8a ("usb: add short-packet handling to usb-storage driver") Reported-by: Sai Pavan Boddu <saipava@xilinx.com> Tested-by: Paul Zimmerman <pauldzim@gmail.com> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Message-id: 20200713062712.1476-1-kraxel@redhat.com
Calling ramfb_display_update() might replace the DisplaySurface with the
boot display, which in turn will free the currently active
DisplaySurface.
So clear our DisplaySurface pinter (dpy->region.surface pointer) to (a)
avoid use-after-free and (b) force replacing the boot display with the
real display when switching back.
Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> Reviewed-by: Alex Williamson <alex.williamson@redhat.com> Acked-by: Alex Williamson <alex.williamson@redhat.com>
Message-id: 20200713124520.23266-1-kraxel@redhat.com
Stefan Berger [Tue, 7 Jul 2020 20:16:25 +0000 (16:16 -0400)]
tests: tpm: Skip over pcrUpdateCounter byte in result comparison
The TPM 2 code in libtpms was fixed to handle the PCR 'TCB group' according
to the PCClient profile. The change of the PCRs belonging to the 'TCB group'
now affects the pcrUpdateCounter in the TPM2_PCRRead() responses where its
value is now different (typically lower by '1') than what it was before. To
not fail the tests, we skip the comparison of the 14th byte, which
represents the pcrUpdateCounter.
Signed-off-by: Stefan Berger <stefanb@linux.ibm.com> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-id: 20200707201625.4177419-3-stefanb@linux.vnet.ibm.com
Stefan Berger [Tue, 7 Jul 2020 20:16:24 +0000 (16:16 -0400)]
tpm: tpm_spapr: Exit on TPM backend failures
Exit on TPM backend failures in the same way as the TPM CRB and TIS device
models do. With this change we now get an error report when the backend
did not start up properly:
error: internal error: qemu unexpectedly closed the monitor:
2020-07-07T12:49:28.333928Z qemu-system-ppc64: tpm-emulator: \
TPM result for CMD_INIT: 0x101 operation failed
Signed-off-by: Stefan Berger <stefanb@linux.ibm.com> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-id: 20200707201625.4177419-2-stefanb@linux.vnet.ibm.com
Peter Maydell [Wed, 15 Jul 2020 16:16:39 +0000 (17:16 +0100)]
Merge remote-tracking branch 'remotes/stsquad/tags/pull-misc-for-rc0-150720-3' into staging
Final fixes for 5.1-rc0
- minor documentation nit
- docker.py bootstrap fixes
- tweak containers.yml wildcards
- fix float16 nan detection
- conditional use of -Wpsabi
- fix missing iotlb data for plugins
- proper locking for helper based bb count
- drop ppc64abi32 from the plugin check-tcg test
# gpg: Signature made Wed 15 Jul 2020 11:59:08 BST
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-misc-for-rc0-150720-3:
.travis.yml: skip ppc64abi32-linux-user with plugins
plugins: expand the bb plugin to be thread safe and track per-cpu
cputlb: ensure we save the IOTLB data in case of reset
tests/plugins: don't unconditionally add -Wpsabi
fpu/softfloat: fix up float16 nan recognition
gitlab-ci/containers: Add missing wildcard where we should look for changes
docker.py: fix fetching of FROM layers
tests/docker: Remove the libssh workaround from the ubuntu 20.04 image
docs/devel: fix grammar in multi-thread-tcg
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
erik-smit [Sun, 28 Jun 2020 14:26:59 +0000 (16:26 +0200)]
ftgmac100: fix dblac write test
The test of the write of the dblac register was testing the old value
instead of the new value. This would accept the write of an invalid value
but subsequently refuse any following valid writes.
Signed-off-by: erik-smit <erik.lucas.smit@gmail.com> Reviewed-by: Cédric Le Goater <clg@kaod.org> Signed-off-by: Jason Wang <jasowang@redhat.com>
net: detect errors from probing vnet hdr flag for TAP devices
When QEMU sets up a tap based network device backend, it mostly ignores errors
reported from various ioctl() calls it makes, assuming the TAP file descriptor
is valid. This assumption can easily be violated when the user is passing in a
pre-opened file descriptor. At best, the ioctls may fail with a -EBADF, but if
the user passes in a bogus FD number that happens to clash with a FD number that
QEMU has opened internally for another reason, a wide variety of errnos may
result, as the TUNGETIFF ioctl number may map to a completely different command
on a different type of file.
By ignoring all these errors, QEMU sets up a zombie network backend that will
never pass any data. Even worse, when QEMU shuts down, or that network backend
is hot-removed, it will close this bogus file descriptor, which could belong to
another QEMU device backend.
There's no obvious guaranteed reliable way to detect that a FD genuinely is a
TAP device, as opposed to a UNIX socket, or pipe, or something else. Checking
the errno from probing vnet hdr flag though, does catch the big common cases.
ie calling TUNGETIFF will return EBADF for an invalid FD, and ENOTTY when FD is
a UNIX socket, or pipe which catches accidental collisions with FDs used for
stdio, or monitor socket.
Previously the example below where bogus fd 9 collides with the FD used for the
chardev saw:
$ ./x86_64-softmmu/qemu-system-x86_64 -netdev tap,id=hostnet0,fd=9 \
-chardev socket,id=charchannel0,path=/tmp/qga,server,nowait \
-monitor stdio -vnc :0
qemu-system-x86_64: -netdev tap,id=hostnet0,fd=9: TUNGETIFF ioctl() failed: Inappropriate ioctl for device
TUNSETOFFLOAD ioctl() failed: Bad address
QEMU 2.9.1 monitor - type 'help' for more information
(qemu) Warning: netdev hostnet0 has no peer
which gives a running QEMU with a zombie network backend.
With this change applied we get an error message and QEMU immediately exits
before carrying on and making a bigger disaster:
$ ./x86_64-softmmu/qemu-system-x86_64 -netdev tap,id=hostnet0,fd=9 \
-chardev socket,id=charchannel0,path=/tmp/qga,server,nowait \
-monitor stdio -vnc :0
qemu-system-x86_64: -netdev tap,id=hostnet0,vhost=on,fd=9: Unable to query TUNGETIFF on FD 9: Inappropriate ioctl for device
Reported-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Signed-off-by: Daniel P. Berrange <berrange@redhat.com> Tested-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-id: 20171027085548.3472-1-berrange@redhat.com
[lv: to simplify, don't check on EINVAL with TUNGETIFF as it exists since v2.6.27] Signed-off-by: Laurent Vivier <lvivier@redhat.com> Signed-off-by: Jason Wang <jasowang@redhat.com>
net: check if the file descriptor is valid before using it
qemu_set_nonblock() checks that the file descriptor can be used and, if
not, crashes QEMU. An assert() is used for that. The use of assert() is
used to detect programming error and the coredump will allow to debug
the problem.
But in the case of the tap device, this assert() can be triggered by
a misconfiguration by the user. At startup, it's not a real problem, but it
can also happen during the hot-plug of a new device, and here it's a
problem because we can crash a perfectly healthy system.
For instance:
# ip link add link virbr0 name macvtap0 type macvtap mode bridge
# ip link set macvtap0 up
# TAP=/dev/tap$(ip -o link show macvtap0 | cut -d: -f1)
# qemu-system-x86_64 -machine q35 -device pcie-root-port,id=pcie-root-port-0 -monitor stdio 9<> $TAP
(qemu) netdev_add type=tap,id=hostnet0,vhost=on,fd=9
(qemu) device_add driver=virtio-net-pci,netdev=hostnet0,id=net0,bus=pcie-root-port-0
(qemu) device_del net0
(qemu) netdev_del hostnet0
(qemu) netdev_add type=tap,id=hostnet1,vhost=on,fd=9
qemu-system-x86_64: .../util/oslib-posix.c:247: qemu_set_nonblock: Assertion `f != -1' failed.
Aborted (core dumped)
To avoid that, add a function, qemu_try_set_nonblock(), that allows to report the
problem without crashing.
In the same way, we also update the function for vhostfd in net_init_tap_one() and
for fd in net_init_socket() (both descriptors are provided by the user and can
be wrong).
Signed-off-by: Laurent Vivier <lvivier@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Jason Wang <jasowang@redhat.com>
CC: Jason Wang <jasowang@redhat.com> CC: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Laurent Vivier <lvivier@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Jason Wang <jasowang@redhat.com>
Peter Maydell [Wed, 15 Jul 2020 12:54:09 +0000 (13:54 +0100)]
Merge remote-tracking branch 'remotes/philmd-gitlab/tags/mips-next-20200714' into staging
MIPS patches for 5.1
- A pair of fixes,
- Add Huacai Chen as MIPS KVM maintainer,
- Add Jiaxun Yang as designated MIPS TCG reviewer.
CI jobs results:
. https://travis-ci.org/github/philmd/qemu/builds/708079271
. https://gitlab.com/philmd/qemu/-/pipelines/166528104
. https://cirrus-ci.com/build/6483996878045184
# gpg: Signature made Tue 14 Jul 2020 20:59:58 BST
# gpg: using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD 6BB2 E3E3 2C2C DEAD C0DE
Peter Maydell [Wed, 15 Jul 2020 12:04:27 +0000 (13:04 +0100)]
Merge remote-tracking branch 'remotes/philmd-gitlab/tags/python-next-20200714' into staging
Python patches for 5.1
- Reduce race conditions on QEMUMachine::shutdown()
1. Remove the "bare except" pattern in the existing shutdown code,
which can mask problems and make debugging difficult.
2. Ensure that post-shutdown cleanup is always performed, even when
graceful termination fails.
3. Unify cleanup paths such that no matter how the VM is terminated,
the same functions and steps are always taken to reset the object
state.
4. Rewrite shutdown() such that any error encountered when attempting
a graceful shutdown will be raised as an AbnormalShutdown exception.
The pythonic idiom is to allow the caller to decide if this is a
problem or not.
- Modify part of the python/qemu library to comply with:
. mypy --strict
. pylint
. flake8
- Script for the TCG Continuous Benchmarking project that uses
callgrind to dissect QEMU execution into three main phases:
CI jobs results:
. https://cirrus-ci.com/build/5421349961203712
. https://gitlab.com/philmd/qemu/-/pipelines/166556001
. https://travis-ci.org/github/philmd/qemu/builds/708102347
# gpg: Signature made Tue 14 Jul 2020 21:40:05 BST
# gpg: using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD 6BB2 E3E3 2C2C DEAD C0DE
* remotes/philmd-gitlab/tags/python-next-20200714:
python/qmp.py: add QMPProtocolError
python/qmp.py: add casts to JSON deserialization
python/qmp.py: Do not return None from cmd_obj
python/qmp.py: re-absorb MonitorResponseError
iotests.py: use qemu.qmp type aliases
python/qmp.py: Define common types
python/machine.py: change default wait timeout to 3 seconds
python/machine.py: re-add sigkill warning suppression
python/machine.py: split shutdown into hard and soft flavors
tests/acceptance: Don't test reboot on cubieboard
tests/acceptance: wait() instead of shutdown() where appropriate
python/machine.py: Make wait() call shutdown()
python/machine.py: Add a configurable timeout to shutdown()
python/machine.py: Prohibit multiple shutdown() calls
python/machine.py: Perform early cleanup for wait() calls, too
python/machine.py: Add _early_cleanup hook
python/machine.py: Close QMP socket in cleanup
python/machine.py: consolidate _post_shutdown()
scripts/performance: Add dissect.py script
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
0: 978 times (97.80%), avg time 0.270 (0.01 varience/0.08 deviation)
-6: 21 times (2.10%), avg time 0.336 (0.01 varience/0.12 deviation)
-11: 1 times (0.10%), avg time 0.502 (0.00 varience/0.00 deviation)
Ran command 1000 times, 978 passes
But when running with plugins we hit the failure a lot more often:
0: 91 times (91.00%), avg time 0.302 (0.04 varience/0.19 deviation)
-11: 9 times (9.00%), avg time 0.558 (0.01 varience/0.11 deviation)
Ran command 100 times, 91 passes
The crash occurs in guest code which is the same in both pass and fail
cases. However we see various messages reported on the console about
corrupted memory lists which seems to imply the guest memory allocation
is corrupted. This lines up with the seg fault being in the guest
__libc_free function. So we think this is a guest bug which is
exacerbated by various modes of translation. If anyone has access to
real hardware to soak test the test case we could prove this properly.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Acked-by: David Gibson <david@gibson.dropbear.id.au> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200714175516.5475-1-alex.bennee@linaro.org>
Alex Bennée [Mon, 13 Jul 2020 20:04:11 +0000 (21:04 +0100)]
plugins: expand the bb plugin to be thread safe and track per-cpu
While there isn't any easy way to make the inline counts thread safe
we can ensure the callback based ones are. While we are at it we can
reduce introduce a new option ("idle") to dump a report of the current
bb and insn count each time a vCPU enters the idle state.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Robert Foley <robert.foley@linaro.org> Cc: Dave Bort <dbort@dbort.com>
Message-Id: <20200713200415.26214-8-alex.bennee@linaro.org>
Alex Bennée [Mon, 13 Jul 2020 20:04:10 +0000 (21:04 +0100)]
cputlb: ensure we save the IOTLB data in case of reset
Any write to a device might cause a re-arrangement of memory
triggering a TLB flush and potential re-size of the TLB invalidating
previous entries. This would cause users of qemu_plugin_get_hwaddr()
to see the warning:
invalid use of qemu_plugin_get_hwaddr
because of the failed tlb_lookup which should always succeed. To
prevent this we save the IOTLB data in case it is later needed by a
plugin doing a lookup.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200713200415.26214-7-alex.bennee@linaro.org>
Alex Bennée [Mon, 13 Jul 2020 20:04:09 +0000 (21:04 +0100)]
tests/plugins: don't unconditionally add -Wpsabi
Not all compilers support the -Wpsabi (clang-9 in my case). To handle
this gracefully we pare back the shared build machinery so the
Makefile is relatively "standalone". We still take advantage of
config-host.mak as configure has done a bunch of probing for us but
that is it.
Fixes: bac8d222a Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200713200415.26214-6-alex.bennee@linaro.org>
Thomas Huth [Mon, 13 Jul 2020 18:22:35 +0000 (20:22 +0200)]
gitlab-ci/containers: Add missing wildcard where we should look for changes
The tests/docker/* wildcard seems to only match the files that are directly
in the tests/docker folder - but changes to the files in the directory
tests/docker/dockerfiles are currently ignored. Seems like we need a
separate entry to match the files in that folder. With this wildcard added,
the stages now get re-run successfully when something in the dockerfiles
has been changed.
Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20200713182235.30379-1-thuth@redhat.com>
Alex Bennée [Mon, 13 Jul 2020 20:04:07 +0000 (21:04 +0100)]
docker.py: fix fetching of FROM layers
This worked on a system that was already bootstrapped because the
stage 2 images already existed even if they wouldn't be used. What we
should have pulled down was the FROM line containers first because
building on gitlab doesn't have the advantage of using our build
system to build the pre-requisite bits.
We still pull the image we want to build just in case we can use the
cached data.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> Tested-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713200415.26214-4-alex.bennee@linaro.org>
Alex Bennée [Mon, 13 Jul 2020 20:04:12 +0000 (21:04 +0100)]
docs/devel: fix grammar in multi-thread-tcg
Review comment came just too late ;-)
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200713200415.26214-9-alex.bennee@linaro.org>
Peter Maydell [Wed, 15 Jul 2020 08:06:55 +0000 (09:06 +0100)]
Merge remote-tracking branch 'remotes/philmd-gitlab/tags/sdcard-CVE-2020-13253-pull-request' into staging
Fix CVE-2020-13253
By using invalidated address, guest can do out-of-bounds accesses.
These patches fix the issue by only allowing SD card image sizes
power of 2, and not switching to SEND_DATA state when the address
is invalid (out of range).
This issue was found using QEMU fuzzing mode (using --enable-fuzzing,
see docs/devel/fuzzing.txt) and reported by Alexander Bulekov.
CI jobs results:
. https://cirrus-ci.com/build/5157142548185088
. https://gitlab.com/philmd/qemu/-/pipelines/166381731
. https://travis-ci.org/github/philmd/qemu/builds/707956535
# gpg: Signature made Tue 14 Jul 2020 14:54:44 BST
# gpg: using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]
# Primary key fingerprint: FAAB E75E 1291 7221 DCFD 6BB2 E3E3 2C2C DEAD C0DE
* remotes/philmd-gitlab/tags/sdcard-CVE-2020-13253-pull-request:
hw/sd/sdcard: Do not switch to ReceivingData if address is invalid
hw/sd/sdcard: Update coding style to make checkpatch.pl happy
hw/sd/sdcard: Do not allow invalid SD card sizes
hw/sd/sdcard: Simplify realize() a bit
hw/sd/sdcard: Restrict Class 6 commands to SCSD cards
tests/acceptance/boot_linux: Expand SD card image to power of 2
tests/acceptance/boot_linux: Tag tests using a SD card with 'device:sd'
docs/orangepi: Add instructions for resizing SD image to power of two
MAINTAINERS: Cc qemu-block mailing list
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
John Snow [Fri, 10 Jul 2020 05:22:10 +0000 (01:22 -0400)]
python/qmp.py: add QMPProtocolError
In the case that we receive a reply but are unable to understand it,
use this exception name to indicate that case.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-7-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:22:09 +0000 (01:22 -0400)]
python/qmp.py: add casts to JSON deserialization
mypy and python type hints are not powerful enough to properly describe
JSON messages in Python 3.6. The best we can do, generally, is describe
them as Dict[str, Any].
Add casts to coerce this type for static analysis; but do NOT enforce
this type at runtime in any way.
Note: Python 3.8 adds a TypedDict construct which allows for the
description of more arbitrary Dictionary shapes. There is a third-party
module, "Pydantic", which is compatible with 3.6 that can be used
instead of the JSON library that parses JSON messages to fully-typed
Python objects, and may be preferable in some cases.
(That is well beyond the scope of this commit or series.)
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-6-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:22:08 +0000 (01:22 -0400)]
python/qmp.py: Do not return None from cmd_obj
This makes typing the qmp library difficult, as it necessitates wrapping
Optional[] around the type for every return type up the stack. At some
point, it becomes difficult to discern or remember why it's None instead
of the expected object.
Use the python exception system to tell us exactly why we didn't get an
object. Remove this special-cased return.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-5-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:22:07 +0000 (01:22 -0400)]
python/qmp.py: re-absorb MonitorResponseError
When I initially split this out, I considered this more of a machine
error than a QMP protocol error, but I think that's misguided.
Move this back to qmp.py and name it QMPResponseError. Convert
qmp.command() to use this exception type.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-4-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:22:06 +0000 (01:22 -0400)]
iotests.py: use qemu.qmp type aliases
iotests.py should use the type definitions from qmp.py instead of its
own.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-3-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:22:05 +0000 (01:22 -0400)]
python/qmp.py: Define common types
Define some common types that we'll need to annotate a lot of other
functions going forward.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200710052220.3306-2-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:49 +0000 (01:06 -0400)]
python/machine.py: change default wait timeout to 3 seconds
Machine.wait() does not appear to be used except in the acceptance tests,
and an infinite timeout by default in a test suite is not the most helpful.
Change it to 3 seconds, like the default shutdown timeout.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-13-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
If the user kills QEMU on purpose, we don't need to warn
them about that having happened: they know already.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-12-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:47 +0000 (01:06 -0400)]
python/machine.py: split shutdown into hard and soft flavors
This is done primarily to avoid the 'bare except' pattern, which
suppresses all exceptions during shutdown and can obscure errors.
Replace this with a pattern that isolates the different kind of shutdown
paradigms (_hard_shutdown and _soft_shutdown), and a new fallback shutdown
handler (_do_shutdown) that gracefully attempts one before the other.
This split now also ensures that no matter what happens,
_post_shutdown() is always invoked.
shutdown() changes in behavior such that if it attempts to do a graceful
shutdown and is unable to, it will now always raise an exception to
indicate this. This can be avoided by the test writer in three ways:
1. If the VM is expected to have already exited or is in the process of
exiting, wait() can be used instead of shutdown() to clean up resources
instead. This helps avoid race conditions in shutdown.
2. If a test writer is expecting graceful shutdown to fail, shutdown
should be called in a try...except block.
3. If the test writer has no interest in performing a graceful shutdown
at all, kill() can be used instead.
Handling shutdown in this way makes it much more explicit which type of
shutdown we want and allows the library to report problems with this
process.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-11-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:46 +0000 (01:06 -0400)]
tests/acceptance: Don't test reboot on cubieboard
cubieboard does not have a functioning reboot, it halts and QEMU does
not exit.
vm.shutdown() is modified in a forthcoming patch that makes it less tolerant
of race conditions on shutdown; tests should consciously decide to WAIT
or to SHUTDOWN qemu.
So long as this test is attempting to reboot, the correct choice would
be to WAIT for the VM to exit. However, since that's broken, we should
SHUTDOWN instead.
SHUTDOWN is indeed what already happens when the test performs teardown,
however, if anyone fixes cubieboard reboot in the future, this test will
develop a new race condition that might be hard to debug.
Therefore: remove the reboot test and make it obvious that the VM is
still running when the test concludes, where the test teardown will do
the right thing.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-10-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:45 +0000 (01:06 -0400)]
tests/acceptance: wait() instead of shutdown() where appropriate
When issuing 'reboot' to a VM with the no-reboot option, that VM will
exit. When then issuing a shutdown command, the cleanup may race.
Add calls to vm.wait() which will gracefully mark the VM as having
exited. Subsequent vm.shutdown() calls in generic tearDown code will not
race when called after completion of the call.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-9-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:44 +0000 (01:06 -0400)]
python/machine.py: Make wait() call shutdown()
At this point, shutdown(has_quit=True) and wait() do essentially the
same thing; they perform cleanup without actually instructing QEMU to
quit.
Define one in terms of the other.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-8-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:43 +0000 (01:06 -0400)]
python/machine.py: Add a configurable timeout to shutdown()
Three seconds is hardcoded. Use it as a default parameter instead, and use that
value for both waits that may occur in the function.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-7-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
If the VM is not launched, don't try to shut it down. As a change,
_post_shutdown now unconditionally also calls _early_cleanup in order to
offer comprehensive object cleanup in failure cases.
As a courtesy, treat it as a NOP instead of rejecting it as an
error. This is slightly nicer for acceptance tests where vm.shutdown()
is issued unconditionally in tearDown callbacks.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200710050649.32434-6-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:41 +0000 (01:06 -0400)]
python/machine.py: Perform early cleanup for wait() calls, too
This is primarily for consistency, and is a step towards wait() and
shutdown() sharing the same implementation so that the two cleanup paths
cannot diverge.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-5-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:40 +0000 (01:06 -0400)]
python/machine.py: Add _early_cleanup hook
Some parts of cleanup need to occur prior to shutdown, otherwise
shutdown might break. Move this into a suitably named method/callback.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-4-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:39 +0000 (01:06 -0400)]
python/machine.py: Close QMP socket in cleanup
It's not important to do this before waiting for the process to exit, so
it can be done during generic post-shutdown cleanup.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200710050649.32434-3-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
John Snow [Fri, 10 Jul 2020 05:06:38 +0000 (01:06 -0400)]
python/machine.py: consolidate _post_shutdown()
Move more cleanup actions into _post_shutdown. As a change, if QEMU
should so happen to be terminated during a call to wait(), that event
will now be logged.
This is not likely to occur during normative use.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200710050649.32434-2-jsnow@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Ahmed Karaman <ahmedkhaledkaraman@gmail.com> Reviewed-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200709052055.2650-2-ahmedkhaledkaraman@gmail.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Peter Maydell [Tue, 14 Jul 2020 20:21:58 +0000 (21:21 +0100)]
Merge remote-tracking branch 'remotes/mdroth/tags/qga-pull-2020-07-13-tag' into staging
qemu-ga patch queue for hard-freeze
* fix erroneously reporting stale hostname in guest-get-host-name
* fix regression where guest-shutdown asserts when called
* fix race condition with guest-fs-freeze/thaw on w32
Huacai Chen and Jiaxun Yang step in as new energy [1].
Aurelien Jarno comment [2]:
It happens that I known Huacai Chen from the time he was
upstreaming the Loongson 3 support to the kernel, I have been
testing and reviewing his patches. I also know Jiaxun Yang from
the #debian-mips IRC channel. I know that they are both very
competent and have a good knowledge of the open source world.
I therefore agree that they are good additions to maintain and/or
review the MIPS part of QEMU.
Alex Richardson [Fri, 3 Jul 2020 16:15:15 +0000 (17:15 +0100)]
target/mips: Fix ADD.S FPU instruction
After merging latest QEMU upstream into our CHERI fork,
I noticed that some of the FPU tests in our MIPS baremetal
testsuite [*] started failing.
It turns out commit 1ace099f2a accidentally changed add.s
into a subtract.
[*] https://github.com/CTSRD-CHERI/cheritest
Fixes: 1ace099f2a ("target/mips: fpu: Demacro ADD.<D|S|PS>") Signed-off-by: Alex Richardson <Alexander.Richardson@cl.cam.ac.uk> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200703161515.25966-1-Alexander.Richardson@cl.cam.ac.uk> Signed-off-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Original contributor can't remember what was his intention.
Fixes: 5a25ce9487 ("mips: Hook in more reg accesses via mttr/mftr") Buglink: https://bugs.launchpad.net/qemu/+bug/1885718 Signed-off-by: Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>
Message-Id: <20200701182559.28841-2-aleksandar.qemu.devel@gmail.com> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
* remotes/vivier2/tags/linux-user-for-5.1-pull-request:
linux-user: fix print_syscall_err() when syscall returned value is negative
linux-user: fix the errno value in print_syscall_err()
linux-user: add netlink RTM_SETLINK command
linux-user: add new netlink types
linux-user: Fix Coverity CID 1430271 / CID 1430272
linux-user: refactor ipc syscall and support of semtimedop syscall
linux-user: Use EPROTONOSUPPORT for unimplemented netlink protocols
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Peter Maydell [Tue, 14 Jul 2020 18:39:51 +0000 (19:39 +0100)]
Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging
Block layer patches:
- file-posix: Mitigate file fragmentation with extent size hints
- Tighten qemu-img rules on missing backing format
- qemu-img map: Don't limit block status request size
- Fix crash with virtio-scsi and iothreads
# gpg: Signature made Tue 14 Jul 2020 14:24:19 BST
# gpg: using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg: issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74 56FE 7F09 B272 C88F 2FD6
* remotes/kevin/tags/for-upstream:
block: Avoid stale pointer dereference in blk_get_aio_context()
qemu-img: Deprecate use of -b without -F
block: Add support to warn on backing file change without format
iotests: Specify explicit backing format where sensible
qcow2: Deprecate use of qemu-img amend to change backing file
block: Error if backing file fails during creation without -u
qcow: Tolerate backing_fmt=
vmdk: Add trivial backing_fmt support
sheepdog: Add trivial backing_fmt support
block: Finish deprecation of 'qemu-img convert -n -o'
qemu-img: Flush stdout before before potential stderr messages
file-posix: Mitigate file fragmentation with extent size hints
iotests/059: Filter out disk size with more standard filter
qemu-img map: Don't limit block status request size
iotests: Simplify _filter_img_create() a bit
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Peter Maydell [Tue, 14 Jul 2020 16:58:00 +0000 (17:58 +0100)]
Merge remote-tracking branch 'remotes/alistair/tags/pull-riscv-to-apply-20200713' into staging
This is a colection of bug fixes and small imrprovements for RISC-V.
This includes some vector extensions fixes, a PMP bug fix, OpenTitan
UART bug fix and support for OpenSBI dynamic firmware.
# gpg: Signature made Tue 14 Jul 2020 01:29:44 BST
# gpg: using RSA key F6C4AC46D4934868D3B8CE8F21E10D29DF977054
# gpg: Good signature from "Alistair Francis <alistair@alistair23.me>" [full]
# Primary key fingerprint: F6C4 AC46 D493 4868 D3B8 CE8F 21E1 0D29 DF97 7054
* remotes/alistair/tags/pull-riscv-to-apply-20200713:
target/riscv: Fix pmp NA4 implementation
tcg/riscv: Remove superfluous breaks
hw/char: Convert the Ibex UART to use the registerfields API
hw/char: Convert the Ibex UART to use the qdev Clock model
target/riscv: fix vill bit index in vtype register
target/riscv: fix return value of do_opivx_widen()
target/riscv: correct the gvec IR called in gen_vec_rsub16_i64()
target/riscv: fix rsub gvec tcg_assert_listed_vecop assertion
hw/riscv: Modify MROM size to end at 0x10000
RISC-V: Support 64 bit start address
riscv: Add opensbi firmware dynamic support
RISC-V: Copy the fdt in dram instead of ROM
riscv: Unify Qemu's reset vector code path
hw/riscv: virt: Sort the SoC memmap table entries
MAINTAINERS: Add an entry for OpenSBI firmware
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
hw/sd/sdcard: Do not switch to ReceivingData if address is invalid
Only move the state machine to ReceivingData if there is no
pending error. This avoids later OOB access while processing
commands queued.
"SD Specifications Part 1 Physical Layer Simplified Spec. v3.01"
4.3.3 Data Read
Read command is rejected if BLOCK_LEN_ERROR or ADDRESS_ERROR
occurred and no data transfer is performed.
4.3.4 Data Write
Write command is rejected if BLOCK_LEN_ERROR or ADDRESS_ERROR
occurred and no data transfer is performed.
WP_VIOLATION errors are not modified: the error bit is set, we
stay in receive-data state, wait for a stop command. All further
data transfer is ignored. See the check on sd->card_status at the
beginning of sd_read_data() and sd_write_data().
Fixes: CVE-2020-13253 Cc: qemu-stable@nongnu.org Reported-by: Alexander Bulekov <alxndr@bu.edu> Buglink: https://bugs.launchpad.net/qemu/+bug/1880822 Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Message-Id: <20200630133912.9428-6-f4bug@amsat.org>
hw/sd/sdcard: Update coding style to make checkpatch.pl happy
To make the next commit easier to review, clean this code first.
Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <20200630133912.9428-3-f4bug@amsat.org>
QEMU allows to create SD card with unrealistic sizes. This could
work, but some guests (at least Linux) consider sizes that are not
a power of 2 as a firmware bug and fix the card size to the next
power of 2.
While the possibility to use small SD card images has been seen as
a feature, it became a bug with CVE-2020-13253, where the guest is
able to do OOB read/write accesses past the image size end.
In a pair of commits we will fix CVE-2020-13253 as:
Read command is rejected if BLOCK_LEN_ERROR or ADDRESS_ERROR
occurred and no data transfer is performed.
Write command is rejected if BLOCK_LEN_ERROR or ADDRESS_ERROR
occurred and no data transfer is performed.
WP_VIOLATION errors are not modified: the error bit is set, we
stay in receive-data state, wait for a stop command. All further
data transfer is ignored. See the check on sd->card_status at the
beginning of sd_read_data() and sd_write_data().
While this is the correct behavior, in case QEMU create smaller SD
cards, guests still try to access past the image size end, and QEMU
considers this is an invalid address, thus "all further data transfer
is ignored". This is wrong and make the guest looping until
eventually timeouts.
Fix by not allowing invalid SD card sizes (suggesting the expected
size as a hint):
$ qemu-system-arm -M orangepi-pc -drive file=rootfs.ext2,if=sd,format=raw
qemu-system-arm: Invalid SD card size: 60 MiB
SD card size has to be a power of 2, e.g. 64 MiB.
You can resize disk images with 'qemu-img resize <imagefile> <new-size>'
(note that this will lose data if you make the image smaller than it currently is).
Cc: qemu-stable@nongnu.org Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200713183209.26308-8-f4bug@amsat.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Message-Id: <20200630133912.9428-18-f4bug@amsat.org>
tests/acceptance/boot_linux: Expand SD card image to power of 2
In few commits we won't allow SD card images with invalid size
(not aligned to a power of 2). Prepare the tests: add the
pow2ceil() and image_pow2ceil_expand() methods and resize the
images (expanding) of the tests using SD cards.
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200713183209.26308-5-f4bug@amsat.org>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200713183209.26308-4-f4bug@amsat.org>
docs/orangepi: Add instructions for resizing SD image to power of two
SD cards need to have a size of a power of two.
Update the Orange Pi machine documentation to include
instructions for resizing downloaded images using the
qemu-img command.
Signed-off-by: Niek Linnenbank <nieklinnenbank@gmail.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20200712183708.15450-1-nieklinnenbank@gmail.com> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
We forgot to include the qemu-block mailing list while adding
this section in commit 076a0fc32a7. Fix this.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20200630133912.9428-2-f4bug@amsat.org>
Greg Kurz [Thu, 9 Jul 2020 13:50:45 +0000 (15:50 +0200)]
block: Avoid stale pointer dereference in blk_get_aio_context()
It is possible for blk_remove_bs() to race with blk_drain_all(), causing
the latter to dereference a stale blk->root pointer:
blk_remove_bs(blk)
bdrv_root_unref_child(blk->root)
child_bs = blk->root->bs
bdrv_detach_child(blk->root)
...
g_free(blk->root) <============== blk->root becomes stale
bdrv_unref(child_bs) <============ yield at some point
A blk_drain_all() can be triggered by some guest action in the
meantime, eg. on POWER, SLOF might disable bus mastering on
a virtio-scsi-pci device:
Then, depending on one's luck, QEMU either crashes with SEGV or
hits the assertion in blk_get_aio_context().
blk->root is set by blk_insert_bs() which calls bdrv_root_attach_child()
first. The blk_remove_bs() function should rollback the changes made
by blk_insert_bs() in the opposite order (or it should be documented
somewhere why this isn't the case). Clear blk->root before calling
bdrv_root_unref_child() in blk_remove_bs().
Signed-off-by: Greg Kurz <groug@kaod.org>
Message-Id: <159430264541.389456.11925072456012783045.stgit@bahia.lan> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:54 +0000 (15:39 -0500)]
qemu-img: Deprecate use of -b without -F
Creating an image that requires format probing of the backing image is
potentially unsafe (we've had several CVEs over the years based on
probes leaking information to the guest on a subsequent boot, although
these days tools like libvirt are aware of the issue enough to prevent
the worst effects). For example, if our probing algorithm ever
changes, or if other tools like libvirt determine a different probe
result than we do, then subsequent use of that backing file under a
different format will present corrupted data to the guest.
Fortunately, the worst effects occur only when the backing image is
originally raw, and we at least prevent commit into a probed raw
backing file that would change its probed type.
Still, it is worth starting a deprecation clock so that future
qemu-img can refuse to create backing chains that would rely on
probing, to encourage clients to avoid unsafe practices. Most
warnings are intentionally emitted from bdrv_img_create() in the block
layer, but qemu-img convert uses bdrv_create() which cannot emit its
own warning without causing spurious warnings on other code paths. In
the end, all command-line image creation or backing file rewriting now
performs a check.
Furthermore, if we probe a backing file as non-raw, then it is safe to
explicitly record that result (rather than relying on future probes);
only where we probe a raw image do we care about further warnings to
the user when using such an image (for example, commits into a
probed-raw backing file are prevented), to help them improve their
tooling. But whether or not we make the probe results explicit, we
still warn the user to remind them to upgrade their workflow to supply
-F always.
iotest 114 specifically wants to create an unsafe image for later
amendment rather than defaulting to our new default of recording a
probed format, so it needs an update. While touching it, expand it to
cover all of the various warnings enabled by this patch. iotest 301
also shows a change to qcow messages.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-11-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:53 +0000 (15:39 -0500)]
block: Add support to warn on backing file change without format
For now, this is a mechanical addition; all callers pass false. But
the next patch will use it to improve 'qemu-img rebase -u' when
selecting a backing file with no format.
Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Peter Krempa <pkrempa@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com>
Message-Id: <20200706203954.341758-10-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:52 +0000 (15:39 -0500)]
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:51 +0000 (15:39 -0500)]
qcow2: Deprecate use of qemu-img amend to change backing file
The use of 'qemu-img amend' to change qcow2 backing files is not
tested very well. In particular, our implementation has a bug where
if a new backing file is provided without a format, then the prior
format is blindly reused, even if this results in data corruption, but
this is not caught by iotests.
There are also situations where amending other options needs access to
the original backing file (for example, on a downgrade to a v2 image,
knowing whether a v3 zero cluster must be allocated or may be left
unallocated depends on knowing whether the backing file already reads
as zero), but the command line does not have a nice way to tell us
both the backing file to use for opening the image as well as the
backing file to install after the operation is complete.
Even if we do allow changing the backing file, it is redundant with
the existing ability to change backing files via 'qemu-img rebase -u'.
It is time to deprecate this support (leaving the existing behavior
intact, even if it is buggy), and at a point in the future, require
the use of only 'qemu-img rebase' for adjusting backing chain
relations, saving 'qemu-img amend' for changes unrelated to the
backing chain.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-8-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:50 +0000 (15:39 -0500)]
block: Error if backing file fails during creation without -u
Back in commit 6e6e55f5 (Jul 2017, v2.10), we tweaked the code to warn
if the backing file could not be opened but the user gave a size,
unless the user also passes the -u option to bypass the open of the
backing file. As one common reason for failure to open the backing
file is when there is mismatch in the requested backing format in
relation to what the backing file actually contains, we actually want
to open the backing file and ensure that it has the right format in as
many cases as possible. iotest 301 for qcow demonstrates how
detecting explicit format mismatch is useful to prevent the creation
of an image that would probe differently than the user requested. Now
is the time to finally turn the warning an error, as promised.
Note that the original warning was added prior to our documentation of
an official deprecation policy (eb22aeca, also Jul 2017), and because
the warning didn't mention the word "deprecated", we never actually
remembered to document it as such. But the warning has been around
long enough that I don't see prolonging it another two releases.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-7-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:49 +0000 (15:39 -0500)]
qcow: Tolerate backing_fmt=
qcow has no space in the metadata to store a backing format, and there
are existing qcow images backed both by raw or by other formats
(usually qcow) images, reliant on probing to tell the difference. On
the bright side, because we probe every time, raw files are marked as
probed and we thus forbid a commit action into the backing file where
guest-controlled contents could change the result of the probe next
time around (the iotest added here proves that).
Still, allowing the user to specify the backing format during
creation, even if we can't record it, is a good thing. This patch
blindly allows any value that resolves to a known driver, even if the
user's request is a mismatch from what probing finds; then the next
patch will further enhance things to verify that the user's request
matches what we actually probe. With this and the next patch in
place, we will finally be ready to deprecate the creation of images
where a backing format was not explicitly specified by the user.
Note that this is only for QemuOpts usage; there is no change to the
QAPI to allow a format through -blockdev.
Add a new iotest 301 just for qcow, to demonstrate the latest
behavior, and to make it easier to show the improvements made in the
next patch.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-6-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:48 +0000 (15:39 -0500)]
vmdk: Add trivial backing_fmt support
vmdk already requires that if backing_file is present, that it be
another vmdk image (see vmdk_co_do_create). Meanwhile, we want to
move towards always being explicit about the backing format for other
drivers where it matters. So for convenience, make qemu-img create -F
vmdk work, while rejecting all other explicit formats (note that this
is only for QemuOpts usage; there is no change to the QAPI to allow a
format through -blockdev).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-5-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:47 +0000 (15:39 -0500)]
sheepdog: Add trivial backing_fmt support
Sheepdog already requires that if backing_file is present, that it be
another sheepdog image (see sd_co_create). Meanwhile, we want to move
towards always being explicit about the backing format for other
drivers where it matters. So for convenience, make qemu-img create -F
sheepdog work, while rejecting all other explicit formats (note that
this is only for QemuOpts usage; there is no change to the QAPI to
allow a format through -blockdev).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-4-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Eric Blake [Mon, 6 Jul 2020 20:39:45 +0000 (15:39 -0500)]
qemu-img: Flush stdout before before potential stderr messages
During 'qemu-img create ... 2>&1', if --quiet is not in force, we can
end up with buffered I/O in stdout that was produced before failure,
but which appears in output after failure. This is confusing; the fix
is to flush stdout prior to attempting anything that might produce an
error message. Several iotests demonstrate the resulting ordering
change now that the merged outputs now reflect chronology. (An even
better fix would be to avoid printf from within block.c altogether,
but that's much more invasive...)
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-2-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Kevin Wolf [Tue, 7 Jul 2020 14:23:29 +0000 (16:23 +0200)]
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Kevin Wolf [Mon, 13 Jul 2020 13:45:37 +0000 (15:45 +0200)]
iotests/059: Filter out disk size with more standard filter
The actual disk space used by an image can vary between filesystems and
depending on other settings like an extent size hint. Replace the one
call of "$QEMU_IMG info" and the associated one-off sed filter with the
more standard "_img_info" and the standard filter from common.filter.
Apart from turning "vmdk" into "IMGFMT" and changing the placeholder for
cid fields, this only removes the "disk size" line.
Kevin Wolf [Tue, 7 Jul 2020 14:46:29 +0000 (16:46 +0200)]
qemu-img map: Don't limit block status request size
Limiting each loop iteration of qemu-img map to 1 GB was arbitrary from
the beginning, though it only cut the maximum in half then because the
interface was a signed 32 bit byte count. These days, bdrv_block_status
supports a 64 bit byte count, so the arbitrary limit is even worse.
On file-posix, bdrv_block_status() eventually maps to SEEK_HOLE and
SEEK_DATA, which don't support a limit, but always do all of the work
necessary to find the start of the next hole/data. Much of this work may
be repeated if we don't use this information fully, but query with an
only slightly larger offset in the next loop iteration. Therefore, if
bdrv_block_status() is called in a loop, it should always pass the
full number of bytes that the whole loop is interested in.
This removes the arbitrary limit and speeds up 'qemu-img map'
significantly on heavily fragmented images.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707144629.51235-1-kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Max Reitz [Thu, 9 Jul 2020 11:02:05 +0000 (13:02 +0200)]
iotests: Simplify _filter_img_create() a bit
Not only is it a bit stupid to try to filter multi-line "Formatting"
output (because we only need it for a single test, which can easily be
amended to no longer need it), it is also problematic when there can be
output after a "Formatting" line that we do not want to filter as if it
were part of it.
So rename _filter_img_create to _do_filter_img_create, let it filter
only a single line, and let _filter_img_create loop over all input
lines, calling _do_filter_img_create only on those that match
/^Formatting/ (basically, what _filter_img_create_in_qmp did already).
(And fix 020 to work with that.)
Reported-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200709110205.310942-1-mreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
* remotes/ericb/tags/pull-nbd-2020-07-13:
iotests.py: filter_testfiles(): filter SOCK_DIR too
iotests.py: QemuIoInteractive: print output on failure
iotests: QemuIoInteractive: use qemu_io_args_no_fmt
hax: Fix setting of FD_CLOEXEC
nbd: Avoid off-by-one in long export name truncation
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
linux-user: fix the errno value in print_syscall_err()
errno of the target is returned as a negative value by the syscall,
not in the host errno variable.
The emulation of the target syscall can return an error while the
host doesn't set an errno value. Target errnos and host errnos can
also differ in some cases.
Fixes: c84be71f6854 ("linux-user: Extend strace support to enable argument printing after syscall execution") Cc: Filip.Bozuta@syrmia.com Signed-off-by: Laurent Vivier <laurent@vivier.eu> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Filip Bozuta <Filip.Bozuta@syrmia.com>
Message-Id: <20200708152435.706070-2-laurent@vivier.eu>
The end address calculation for NA4 mode is wrong because the address
used isn't shifted.
It doesn't watch 4 bytes but a huge range because the end address
calculation is wrong.
The solution is to use the shifted address calculated for start address
variable.
Modifications are tested on Zephyr OS userspace test suite which works
for other RISC-V boards (E31 and E34 core).
Signed-off-by: Alexandre Mergnat <amergnat@baylibre.com> Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Message-id: 20200706084550.24117-1-amergnat@baylibre.com
Message-Id: <20200706084550.24117-1-amergnat@baylibre.com>
[ Changes by AF:
- Improve the commit title and message
] Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Remove superfluous breaks, as there is a "return" before them.
Signed-off-by: Liao Pingfang <liao.pingfang@zte.com.cn> Signed-off-by: Yi Wang <wang.yi59@zte.com.cn> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Message-Id: <1594600421-22942-1-git-send-email-wang.yi59@zte.com.cn> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Frank Chang [Fri, 10 Jul 2020 10:48:18 +0000 (18:48 +0800)]
target/riscv: fix vill bit index in vtype register
vill bit is at vtype[XLEN-1].
Signed-off-by: Frank Chang <frank.chang@sifive.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200710104920.13550-5-frank.chang@sifive.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Frank Chang [Fri, 10 Jul 2020 10:48:17 +0000 (18:48 +0800)]
target/riscv: fix return value of do_opivx_widen()
do_opivx_widen() should return false if check function returns false.
Signed-off-by: Frank Chang <frank.chang@sifive.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200710104920.13550-4-frank.chang@sifive.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Frank Chang [Fri, 10 Jul 2020 10:48:16 +0000 (18:48 +0800)]
target/riscv: correct the gvec IR called in gen_vec_rsub16_i64()
Signed-off-by: Frank Chang <frank.chang@sifive.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200710104920.13550-3-frank.chang@sifive.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
gvec should provide vecop_list to avoid:
"tcg_tcg_assert_listed_vecop: code should not be reached bug" assertion.
Signed-off-by: Frank Chang <frank.chang@sifive.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20200710104920.13550-2-frank.chang@sifive.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
Bin Meng [Thu, 9 Jul 2020 10:05:43 +0000 (03:05 -0700)]
hw/riscv: Modify MROM size to end at 0x10000
At present the size of Mask ROM for sifive_u / spike / virt machines
is set to 0x11000, which ends at an unusual address. This changes the
size to 0xf000 so that it ends at 0x10000.
Signed-off-by: Bin Meng <bin.meng@windriver.com> Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <1594289144-24723-1-git-send-email-bmeng.cn@gmail.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
OpenSBI is the default firmware in Qemu and has various firmware loading
options. Currently, qemu loader uses fw_jump which has a compile time
pre-defined address where fdt & kernel image must reside. This puts a
constraint on image size of the Linux kernel depending on the fdt location
and available memory. However, fw_dynamic allows the loader to specify
the next stage location (i.e. Linux kernel/U-Boot) in memory and other
configurable boot options available in OpenSBI.
Add support for OpenSBI dynamic firmware loading support. This doesn't
break existing setup and fw_jump will continue to work as it is. Any
other firmware will continue to work without any issues as long as it
doesn't expect anything specific from loader in "a2" register.
Signed-off-by: Atish Patra <atish.patra@wdc.com> Reviewed-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Bin Meng <bin.meng@windriver.com> Tested-by: Bin Meng <bin.meng@windriver.com>
Message-Id: <20200701183949.398134-4-atish.patra@wdc.com> Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
projects / mirror_qemu.git / log