Steve Sistare [Wed, 25 Oct 2023 19:44:26 +0000 (12:44 -0700)]
cpr: relax blockdev migration blockers
Some blockdevs block migration because they do not support sharing across
hosts and/or do not support dirty bitmaps. These prohibitions do not apply
if the old and new qemu processes do not run concurrently, and if new qemu
starts on the same host as old, which is the case for cpr. Narrow the scope
of these blockers so they only apply to normal mode. They will not block
cpr modes when they are added in subsequent patches.
No functional change until a new mode is added.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-4-git-send-email-steven.sistare@oracle.com>
Steve Sistare [Wed, 25 Oct 2023 19:44:25 +0000 (12:44 -0700)]
migration: per-mode blockers
Extend the blocker interface so that a blocker can be registered for
one or more migration modes. The existing interfaces register a
blocker for all modes, and the new interfaces take a varargs list
of modes.
Internally, maintain a separate blocker list per mode. The same Error
object may be added to multiple lists. When a block is deleted, it is
removed from every list, and the Error is freed.
No functional change until a new mode is added.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-3-git-send-email-steven.sistare@oracle.com>
Steve Sistare [Wed, 25 Oct 2023 19:44:24 +0000 (12:44 -0700)]
migration: mode parameter
Create a mode migration parameter that can be used to select alternate
migration algorithms. The default mode is normal, representing the
current migration algorithm, and does not need to be explicitly set.
No functional change until a new mode is added, except that the mode is
shown by the 'info migrate' command.
Signed-off-by: Steve Sistare <steven.sistare@oracle.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-2-git-send-email-steven.sistare@oracle.com>
Add tracepoints for major downtime checkpoints on both src and dst. They
share the same tracepoint with a string showing its stage.
Besides the checkpoints in the previous patch, this patch also added
destination checkpoints.
On src, we have these checkpoints added:
- src-downtime-start: right before vm stops on src
- src-vm-stopped: after vm is fully stopped
- src-iterable-saved: after all iterables saved (END sections)
- src-non-iterable-saved: after all non-iterable saved (FULL sections)
- src-downtime-stop: migration fully completed
On dst, we have these checkpoints added:
- dst-precopy-loadvm-completes: after loadvm all done for precopy
- dst-precopy-bh-*: record BH steps to resume VM for precopy
- dst-postcopy-bh-*: record BH steps to resume VM for postcopy
On dst side, we don't have a good way to trace total time consumed by
iterable or non-iterable for now. We can mark it by 1st time receiving a
FULL / END section, but rather than that let's just rely on the other
tracepoints added for vmstates to back up the information.
With this patch, one can enable "vmstate_downtime*" tracepoints and it'll
enable all tracepoints for downtime measurements necessary.
Drop loadvm_postcopy_handle_run_bh() tracepoint alongside, because they
service the same purpose, which was only for postcopy. We then have
unified prefix for all downtime relevant tracepoints.
Co-developed-by: Joao Martins <joao.m.martins@oracle.com> Signed-off-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-6-peterx@redhat.com>
Peter Xu [Mon, 30 Oct 2023 16:33:45 +0000 (12:33 -0400)]
migration: migration_stop_vm() helper
Provide a helper for non-COLO use case of migration to stop a VM. This
prepares for adding some downtime relevant tracepoints to migration, where
they may or may not apply to COLO.
Signed-off-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-5-peterx@redhat.com>
Peter Xu [Mon, 30 Oct 2023 16:33:44 +0000 (12:33 -0400)]
migration: Add per vmstate downtime tracepoints
We have a bunch of savevm_section* tracepoints, they're good to analyze
migration stream, but not always suitable if someone would like to analyze
the migration downtime. Two major problems:
- savevm_section* tracepoints are dumping all sections, we only care
about the sections that contribute to the downtime
- They don't have an identifier to show the type of sections, so no way
to filter downtime information either easily.
We can add type into the tracepoints, but instead of doing so, this patch
kept them untouched, instead of adding a bunch of downtime specific
tracepoints, so one can enable "vmstate_downtime*" tracepoints and get a
full picture of how the downtime is distributed across iterative and
non-iterative vmstate save/load.
Note that here both save() and load() need to be traced, because both of
them may contribute to the downtime. The contribution is not a simple "add
them together", though: consider when the src is doing a save() of device1
while the dest can be load()ing for device2, so they can happen
concurrently.
Tracking both sides make sense because device load() and save() can be
imbalanced, one device can save() super fast, but load() super slow, vice
versa. We can't figure that out without tracing both.
Signed-off-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-4-peterx@redhat.com>
Peter Xu [Mon, 30 Oct 2023 16:33:42 +0000 (12:33 -0400)]
migration: Set downtime_start even for postcopy
Postcopy calculates its downtime separately. It always sets
MigrationState.downtime properly, but not MigrationState.downtime_start.
Make postcopy do the same as other modes on properly recording the
timestamp when the VM is going to be stopped. Drop the temporary variable
in postcopy_start() along the way.
Signed-off-by: Peter Xu <peterx@redhat.com> Reviewed-by: Fabiano Rosas <farosas@suse.de> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-2-peterx@redhat.com>
Juan Quintela [Fri, 20 Oct 2023 09:07:31 +0000 (11:07 +0200)]
migration: Use vmstate_register_any() for vmware_vga
I have no idea if we can have more than one vmware_vga device, so play
it safe.
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-14-quintela@redhat.com>
Juan Quintela [Fri, 20 Oct 2023 09:07:30 +0000 (11:07 +0200)]
migration: Use vmstate_register_any() for eeprom93xx
We can have more than one eeprom93xx.
For instance:
e100_nic_realize() -> eeprom93xx_new()
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-13-quintela@redhat.com>
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-12-quintela@redhat.com>
Juan Quintela [Fri, 20 Oct 2023 09:07:28 +0000 (11:07 +0200)]
migration: Improve example and documentation of vmstate_register()
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-11-quintela@redhat.com>
Peter Xu [Fri, 20 Oct 2023 09:07:27 +0000 (11:07 +0200)]
migration: Check in savevm_state_handler_insert for dups
Before finally register one SaveStateEntry, we detect for duplicated
entries. This could be helpful to notify us asap instead of get
silent migration failures which could be hard to diagnose.
For example, this patch will generate a message like this (if without
previous fixes on x2apic) as long as we wants to boot a VM instance
with "-smp 200,maxcpus=288,sockets=2,cores=72,threads=2" and QEMU will
bail out even before VM starts:
Suggested-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Signed-off-by: Peter Xu <peterx@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-10-quintela@redhat.com>
Juan Quintela [Fri, 20 Oct 2023 09:07:25 +0000 (11:07 +0200)]
migration: Hack to maintain backwards compatibility for ppc
Current code does:
- register pre_2_10_vmstate_dummy_icp with "icp/server" and instance
dependinfg on cpu number
- for newer machines, it register vmstate_icp with "icp/server" name
and instance 0
- now it unregisters "icp/server" for the 1st instance.
This is wrong at many levels:
- we shouldn't have two VMSTATEDescriptions with the same name
- In case this is the only solution that we can came with, it needs to
be:
* register pre_2_10_vmstate_dummy_icp
* unregister pre_2_10_vmstate_dummy_icp
* register real vmstate_icp
Created vmstate_replace_hack_for_ppc() with warnings left and right
that it is a hack.
CC: Cedric Le Goater <clg@kaod.org> CC: Daniel Henrique Barboza <danielhb413@gmail.com> CC: David Gibson <david@gibson.dropbear.id.au> CC: Greg Kurz <groug@kaod.org> Reviewed-by: Nicholas Piggin <npiggin@gmail.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-8-quintela@redhat.com>
Juan Quintela [Fri, 20 Oct 2023 09:07:23 +0000 (11:07 +0200)]
migration: Use VMSTATE_INSTANCE_ID_ANY for slirp
Each user network conection create a new slirp instance. We register
more than one slirp instance for number 0.
qemu-system-x86_64: -netdev user,id=hs1: savevm_state_handler_insert: Detected duplicate SaveStateEntry: id=slirp, instance_id=0x0
Broken pipe
../../../../../mnt/code/qemu/full/tests/qtest/libqtest.c:195: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
Aborted (core dumped)
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-6-quintela@redhat.com>
Juan Quintela [Fri, 20 Oct 2023 09:07:21 +0000 (11:07 +0200)]
migration: Use vmstate_register_any() for isa-ide
Otherwise qom-test fails.
ok 4 /i386/qom/x-remote
qemu-system-i386: savevm_state_handler_insert: Detected duplicate SaveStateEntry: id=isa-ide, instance_id=0x0
Broken pipe
../../../../../mnt/code/qemu/full/tests/qtest/libqtest.c:195: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
Aborted (core dumped)
$
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-4-quintela@redhat.com>
Juan Quintela [Fri, 20 Oct 2023 09:07:20 +0000 (11:07 +0200)]
migration: Use vmstate_register_any()
This are the easiest cases, where we were already using
VMSTATE_INSTANCE_ID_ANY.
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-3-quintela@redhat.com>
Juan Quintela [Fri, 20 Oct 2023 09:07:19 +0000 (11:07 +0200)]
migration: Create vmstate_register_any()
We have lots of cases where we are using an instance_id==0 when we
should be using VMSTATE_INSTANCE_ID_ANY (-1). Basically everything
that can have more than one needs to have a proper instance_id or -1
and the system will take one for it.
vmstate_register_any(): We register with -1.
Reviewed-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-2-quintela@redhat.com>
Thomas Huth [Fri, 20 Oct 2023 15:05:54 +0000 (17:05 +0200)]
hw/s390x/s390-stattrib: Don't call register_savevm_live() during instance_init()
We must not call register_savevm_live() from an instance_init() function
(since this could be called multiple times during device introspection).
Move this to the realize() function instead.
Acked-by: David Hildenbrand <david@redhat.com> Reviewed-by: Eric Farman <farman@linux.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020150554.664422-4-thuth@redhat.com>
Thomas Huth [Fri, 20 Oct 2023 15:05:53 +0000 (17:05 +0200)]
hw/s390x/s390-stattrib: Simplify handling of the "migration-enabled" property
There's no need for dedicated handlers here if they don't do anything
special.
Acked-by: David Hildenbrand <david@redhat.com> Reviewed-by: Eric Farman <farman@linux.ibm.com> Acked-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020150554.664422-3-thuth@redhat.com>
Thomas Huth [Fri, 20 Oct 2023 15:05:52 +0000 (17:05 +0200)]
hw/s390x/s390-skeys: Don't call register_savevm_live() during instance_init()
Since the instance_init() function immediately tries to set the
property to "true", the s390_skeys_set_migration_enabled() tries
to register a savevm handler during instance_init(). However,
instance_init() functions can be called multiple times, e.g. for
introspection of devices. That means multiple instances of devices
can be created during runtime (which is fine as long as they all
don't get realized, too), so the "Prevent double registration of
savevm handler" check in the s390_skeys_set_migration_enabled()
function does not work at all as expected (since there could be
more than one instance).
Thus we must not call register_savevm_live() from an instance_init()
function at all. Move this to the realize() function instead. This
way we can also get rid of the property getter and setter functions
completely, simplifying the code along the way quite a bit.
Acked-by: David Hildenbrand <david@redhat.com> Reviewed-by: Eric Farman <farman@linux.ibm.com> Acked-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020150554.664422-2-thuth@redhat.com>
Thomas Huth [Fri, 20 Oct 2023 14:55:54 +0000 (16:55 +0200)]
hw/ipmi: Don't call vmstate_register() from instance_init() functions
instance_init() can be called multiple times, e.g. during introspection
of the device. We should not install the vmstate handlers here. Do it
in the realize() function instead.
Signed-off-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Acked-by: Corey Minyard <cminyard@mvista.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020145554.662751-1-thuth@redhat.com>
Stefan Hajnoczi [Tue, 31 Oct 2023 21:58:11 +0000 (06:58 +0900)]
Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging
Block layer patches
- virtio-blk: use blk_io_plug_call() instead of notification BH
- mirror: allow switching from background to active mode
- qemu-img rebase: add compression support
- Fix locking in media change monitor commands
- Fix a few blockjob-related deadlocks when using iothread
* tag 'for-upstream' of https://repo.or.cz/qemu/kevin: (27 commits)
iotests: add test for changing mirror's copy_mode
mirror: return mirror-specific information upon query
blockjob: query driver-specific info via a new 'query' driver method
qapi/block-core: turn BlockJobInfo into a union
qapi/block-core: use JobType for BlockJobInfo's type
mirror: implement mirror_change method
block/mirror: determine copy_to_target only once
block/mirror: move dirty bitmap to filter
block/mirror: set actively_synced even after the job is ready
blockjob: introduce block-job-change QMP command
virtio-blk: remove batch notification BH
virtio: use defer_call() in virtio_irqfd_notify()
util/defer-call: move defer_call() to util/
block: rename blk_io_plug_call() API to defer_call()
blockdev: mirror: avoid potential deadlock when using iothread
block: avoid potential deadlock during bdrv_graph_wrlock() in bdrv_close()
blockjob: drop AioContext lock before calling bdrv_graph_wrlock()
iotests: Test media change with iothreads
block: Fix locking in media change monitor commands
iotests: add tests for "qemu-img rebase" with compression
...
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Stefan Hajnoczi [Tue, 31 Oct 2023 21:57:29 +0000 (06:57 +0900)]
Merge tag 'pull-halloween-omnibus-311023-2' of https://gitlab.com/stsquad/qemu into staging
Maintainer updates for testing, gitlab, gdbstub and plugins:
- add dtc package to openbsd VMs
- use -fno-stack-protector for non-stdlib tests
- split alpha and sh4 compilers into legacy image
- harmonise other compilers into debian-all-test-cross
- fix NULL check in gdb_regs
- fix memleak in semihosting
- remove unused parameter in plugin code
- fix fd leak in lockstep plugin
# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmVBCvQACgkQ+9DbCVqe
# KkR8jAgAjFC3BE6fu80zYT0Dmeu8zh20QY/wgKQebaFfGEmPL4Bqkl2D/Rx7PhQA
# EH8fR/LAH/iXAO07+LYOB6QiyMb9PWiXS52iHyE3q11mOaM8iKkkj7a59NW8DfGC
# biSrj9o3wpz9gGkJjzTCcHC8DOMbrAuE12XnmhW7uTqqkrcTMC393dSEeyL+nrP9
# lKS5XzFyn3FOT4YIL8hAC02ObKH4LpWIO3gdWeDAo56yg24fLir9a2wYSXMaxQtN
# kDf6UtL97CIIhbNi6qrUPBB13MV8MlXno3wnb9+E4Cn5sGntGSnTyh7j6XrGqYj9
# p/Vio6ye8xP1IjlavKiBM0nnozcAhw==
# =ZOMS
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 31 Oct 2023 23:11:00 JST
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* tag 'pull-halloween-omnibus-311023-2' of https://gitlab.com/stsquad/qemu:
contrib/plugins: Close file descriptor on error return
plugins: Remove an extra parameter
semihosting: fix memleak at semihosting_arg_fallback
gdbstub: Check if gdb_regs is NULL
tests/docker: upgrade debian-all-test-cross to bookworm
tests/docker: use debian-all-test-cross for sparc64
tests/docker: use debian-all-test-cross for riscv64
tests/docker: use debian-all-test-cross for mips
tests/docker: use debian-all-test-cross for mips64
tests/docker: use debian-all-test-cross for m68k
tests/docker: use debian-all-test-cross for hppa
tests/docker: use debian-all-test-cross for power
tests/docker: move sh4 to use debian-legacy-test-cross
tests/docker: use debian-legacy-test-cross for alpha
gitlab: add build-loongarch to matrix
gitlab: clean-up build-soft-softmmu job
gitlab: split alpha testing into a legacy container
tests/tcg: Add -fno-stack-protector
tests/vm/openbsd: Use the system dtc package
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* tag 'migration-20231031-pull-request' of https://gitlab.com/juan.quintela/qemu: (38 commits)
qemu-file: Make qemu_fflush() return errors
migration: Remove transferred atomic counter
migration: Use migration_transferred_bytes()
qemu-file: Simplify qemu_file_get_error()
migration: migration_rate_limit_reset() don't need the QEMUFile
migration: migration_transferred_bytes() don't need the QEMUFile
qemu-file: Remove _noflush from qemu_file_transferred_noflush()
qemu_file: Remove unused qemu_file_transferred()
migration: Use the number of transferred bytes directly
qemu_file: total_transferred is not used anymore
qemu_file: Use a stat64 for qemu_file_transferred
qemu-file: Don't increment qemu_file_transferred at qemu_file_fill_buffer
migration: Stop migration immediately in RDMA error paths
migration: Deprecate old compression method
migration: Deprecate block migration
migration: migrate 'blk' command option is deprecated.
migration: migrate 'inc' command option is deprecated.
qemu-iotests: Filter warnings about block migration being deprecated
migration: set file error on subsection loading
migration: rename vmstate_save_needed->vmstate_section_needed
...
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:31 +0000 (14:54 +0100)]
iotests: add test for changing mirror's copy_mode
One part of the test is using a throttled source to ensure that there
are no obvious issues when changing the copy_mode while there are
ongoing requests (source and target images are compared at the very
end).
The other part of the test is using a throttled target to ensure that
the change to active mode actually happened. This is done by hitting
the throttling limit, issuing a synchronous write and then immediately
verifying the target side. QSD is used, because otherwise, a
synchronous write would hang there.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-11-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:30 +0000 (14:54 +0100)]
mirror: return mirror-specific information upon query
To start out, only actively-synced is returned.
For example, this is useful for jobs that started out in background
mode and switched to active mode. Once actively-synced is true, it's
clear that the mode switch has been completed. Note that completion of
the switch might happen much earlier, e.g. if the switch happens
before the job is ready, once all background operations have finished.
It's assumed that whether the disks are actively-synced or not is more
interesting than whether the mode switch completed. That information
can still be added if required in the future.
In presence of an iothread, the actively_synced member is now shared
between the iothread and the main thread, so turn accesses to it
atomic.
Requires to adapt the output for iotest 109.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-10-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:29 +0000 (14:54 +0100)]
blockjob: query driver-specific info via a new 'query' driver method
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-9-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:28 +0000 (14:54 +0100)]
qapi/block-core: turn BlockJobInfo into a union
In preparation to additionally return job-type-specific information.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20231031135431.393137-8-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:27 +0000 (14:54 +0100)]
qapi/block-core: use JobType for BlockJobInfo's type
In preparation to turn BlockJobInfo into a union with @type as the
discriminator. That requires it to be an enum. Even without that
requirement, it's nicer to have an enum instead of a str here.
No functional change is intended.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20231031135431.393137-7-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:26 +0000 (14:54 +0100)]
mirror: implement mirror_change method
which allows switching the @copy-mode from 'background' to
'write-blocking'.
This is useful for management applications, so they can start out in
background mode to avoid limiting guest write speed and switch to
active mode when certain criteria are fulfilled.
In presence of an iothread, the copy_mode member is now shared between
the iothread and the main thread, so turn accesses to it atomic.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-6-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:25 +0000 (14:54 +0100)]
block/mirror: determine copy_to_target only once
In preparation to allow changing the copy_mode via QMP. When running
in an iothread, it could be that copy_mode is changed from the main
thread in between reading copy_mode in bdrv_mirror_top_pwritev() and
reading copy_mode in bdrv_mirror_top_do_write(), so they might end up
disagreeing about whether copy_to_target is true or false. Avoid that
scenario by determining copy_to_target only once and passing it to
bdrv_mirror_top_do_write() as an argument.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20231031135431.393137-5-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:24 +0000 (14:54 +0100)]
block/mirror: move dirty bitmap to filter
In preparation to allow switching to active mode without draining.
Initialization of the bitmap in mirror_dirty_init() still happens with
the original/backing BlockDriverState, which should be fine, because
the mirror top has the same length.
Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-4-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:23 +0000 (14:54 +0100)]
block/mirror: set actively_synced even after the job is ready
In preparation to allow switching from background to active mode. This
ensures that setting actively_synced will not be missed when the
switch happens after the job is ready.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20231031135431.393137-3-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Tue, 31 Oct 2023 13:54:22 +0000 (14:54 +0100)]
blockjob: introduce block-job-change QMP command
which will allow changing job-type-specific options after job
creation.
In the JobVerbTable, the same allow bits as for set-speed are used,
because set-speed can be considered an existing change command.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20231031135431.393137-2-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Stefan Hajnoczi [Wed, 13 Sep 2023 20:00:45 +0000 (16:00 -0400)]
virtio-blk: remove batch notification BH
There is a batching mechanism for virtio-blk Used Buffer Notifications
that is no longer needed because the previous commit added batching to
virtio_notify_irqfd().
Note that this mechanism was rarely used in practice because it is only
enabled when EVENT_IDX is not negotiated by the driver. Modern drivers
enable EVENT_IDX.
Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230913200045.1024233-5-stefanha@redhat.com> Reviewed-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Stefan Hajnoczi [Wed, 13 Sep 2023 20:00:44 +0000 (16:00 -0400)]
virtio: use defer_call() in virtio_irqfd_notify()
virtio-blk and virtio-scsi invoke virtio_irqfd_notify() to send Used
Buffer Notifications from an IOThread. This involves an eventfd
write(2) syscall. Calling this repeatedly when completing multiple I/O
requests in a row is wasteful.
Use the defer_call() API to batch together virtio_irqfd_notify() calls
made during thread pool (aio=threads), Linux AIO (aio=native), and
io_uring (aio=io_uring) completion processing.
Behavior is unchanged for emulated devices that do not use
defer_call_begin()/defer_call_end() since defer_call() immediately
invokes the callback when called outside a
defer_call_begin()/defer_call_end() region.
fio rw=randread bs=4k iodepth=64 numjobs=8 IOPS increases by ~9% with a
single IOThread and 8 vCPUs. iodepth=1 decreases by ~1% but this could
be noise. Detailed performance data and configuration specifics are
available here:
https://gitlab.com/stefanha/virt-playbooks/-/tree/blk_io_plug-irqfd
This duplicates the BH that virtio-blk uses for batching. The next
commit will remove it.
Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230913200045.1024233-4-stefanha@redhat.com> Reviewed-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Stefan Hajnoczi [Wed, 13 Sep 2023 20:00:43 +0000 (16:00 -0400)]
util/defer-call: move defer_call() to util/
The networking subsystem may wish to use defer_call(), so move the code
to util/ where it can be reused.
As a reminder of what defer_call() does:
This API defers a function call within a defer_call_begin()/defer_call_end()
section, allowing multiple calls to batch up. This is a performance
optimization that is used in the block layer to submit several I/O requests
at once instead of individually:
defer_call_begin(); <-- start of section
...
defer_call(my_func, my_obj); <-- deferred my_func(my_obj) call
defer_call(my_func, my_obj); <-- another
defer_call(my_func, my_obj); <-- another
...
defer_call_end(); <-- end of section, my_func(my_obj) is called once
Suggested-by: Ilya Maximets <i.maximets@ovn.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230913200045.1024233-3-stefanha@redhat.com> Reviewed-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Stefan Hajnoczi [Wed, 13 Sep 2023 20:00:42 +0000 (16:00 -0400)]
block: rename blk_io_plug_call() API to defer_call()
Prepare to move the blk_io_plug_call() API out of the block layer so
that other subsystems call use this deferred call mechanism. Rename it
to defer_call() but leave the code in block/plug.c.
The next commit will move the code out of the block layer.
Suggested-by: Ilya Maximets <i.maximets@ovn.org> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Paul Durrant <paul@xen.org> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230913200045.1024233-2-stefanha@redhat.com> Reviewed-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
semihosting: fix memleak at semihosting_arg_fallback
We duplicate "cmd" as strtok may modify its argument, but we forgot
to free it later. Furthermore, add_semihosting_arg doesn't take
responsibility for this memory either (it strdup's the argument).
Signed-off-by: Matheus Tavares Bernardino <quic_mathbern@quicinc.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <03d81c56bfc3d08224e4106efca5949d8894cfa5.1697801632.git.quic_mathbern@quicinc.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-18-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:29 +0000 (14:50 +0000)]
tests/docker: upgrade debian-all-test-cross to bookworm
This requires a few more tweaks than usual as:
- the default sources format has changed
- bring in python3-tomli from the repos
- split base install from cross compilers
- also include libclang-rt-dev for sanitiser builds
Acked-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-16-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:28 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for sparc64
Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-15-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:27 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for riscv64
Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-14-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:26 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for mips
Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-13-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:25 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for mips64
Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-12-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:24 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for m68k
Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-11-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:23 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for hppa
Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-10-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:22 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for power
Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-9-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:21 +0000 (14:50 +0000)]
tests/docker: move sh4 to use debian-legacy-test-cross
sh4 is another target which doesn't work with bookworm compilers. To
keep on buster move across to the debian-legacy-test-cross image and
update accordingly.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20231030135715.800164-1-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:20 +0000 (14:50 +0000)]
tests/docker: use debian-legacy-test-cross for alpha
Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-7-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:19 +0000 (14:50 +0000)]
gitlab: add build-loongarch to matrix
We have the compiler and with a few updates a container that can build
QEMU so we should at least run the check-tcg smoke tests.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-6-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:18 +0000 (14:50 +0000)]
gitlab: clean-up build-soft-softmmu job
Having dropped alpha we also now drop xtensa as we don't have the
compiler in this image. It's not all doom and gloom though as a number
of other targets have gained softmmu TCG tests so we can add them. We
will take care of the other targets with their own containers in
future commits.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-5-alex.bennee@linaro.org>
Alex Bennée [Sun, 29 Oct 2023 14:50:17 +0000 (14:50 +0000)]
gitlab: split alpha testing into a legacy container
The current bookworm compiler doesn't build the static binaries due to
bug #1054412 and it might be awhile before it gets fixed. The problem
of keeping older architecture compilers running isn't going to go away
so lets prepare the ground. Create a legacy container and move some
tests around so the others can get upgraded.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-4-alex.bennee@linaro.org>
Akihiko Odaki [Sun, 29 Oct 2023 14:50:16 +0000 (14:50 +0000)]
tests/tcg: Add -fno-stack-protector
A build of GCC 13.2 will have stack protector enabled by default if it
was configured with --enable-default-ssp option. For such a compiler,
it is necessary to explicitly disable stack protector when linking
without standard libraries.
Fiona Ebner [Thu, 19 Oct 2023 13:19:36 +0000 (15:19 +0200)]
blockdev: mirror: avoid potential deadlock when using iothread
The bdrv_getlength() function is a generated co-wrapper and uses
AIO_WAIT_WHILE() to wait for the spawned coroutine. AIO_WAIT_WHILE()
expects the lock to be acquired exactly once.
Fix a case where it may be acquired twice. This can happen when the
source node is explicitly specified as the @replaces parameter or if the
source node is a filter node.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231019131936.414246-4-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Thu, 19 Oct 2023 13:19:35 +0000 (15:19 +0200)]
block: avoid potential deadlock during bdrv_graph_wrlock() in bdrv_close()
by passing the BlockDriverState along, so the held AioContext can be
dropped before polling. See commit 31b2ddfea3 ("graph-lock: Unlock the
AioContext while polling") which introduced this functionality for
more information.
The only way to reach bdrv_close() is via bdrv_unref() and for calling
that the BlockDriverState's AioContext lock is supposed to be held.
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231019131936.414246-3-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Fiona Ebner [Thu, 19 Oct 2023 13:19:34 +0000 (15:19 +0200)]
blockjob: drop AioContext lock before calling bdrv_graph_wrlock()
Same rationale as in 31b2ddfea3 ("graph-lock: Unlock the AioContext
while polling"). Otherwise, a deadlock can happen.
The alternative would be to pass a BlockDriverState along to
bdrv_graph_wrlock(), but there is no BlockDriverState readily
available and it's also better conceptually, because the lock is held
for the job.
The function is always called with the job's AioContext lock held, via
one of the .abort, .clean, .free or .prepare job driver functions.
Thus, it's safe to drop it.
While mirror_exit_common() does hold a second AioContext lock while
calling block_job_remove_all_bdrv(), that is for the main thread's
AioContext and does not need to be dropped (bdrv_graph_wrlock(bs) also
skips dropping the lock if bdrv_get_aio_context(bs) ==
qemu_get_aio_context()).
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231019131936.414246-2-f.ebner@proxmox.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Kevin Wolf [Fri, 13 Oct 2023 15:33:02 +0000 (17:33 +0200)]
iotests: Test media change with iothreads
iotests case 118 already tests all relevant operations for media change
with multiple devices, however never with iothreads. This changes the
test so that the virtio-scsi tests run with an iothread.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-ID: <20231013153302.39234-3-kwolf@redhat.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Kevin Wolf [Fri, 13 Oct 2023 15:33:01 +0000 (17:33 +0200)]
block: Fix locking in media change monitor commands
blk_insert_bs() requires that the caller holds the AioContext lock for
the node to be inserted. Since commit c066e808e11, neglecting to do so
causes a crash when the child has to be moved to a different AioContext
to attach it to the BlockBackend.
This fixes qmp_blockdev_insert_anon_medium(), which is called for the
QMP commands 'blockdev-insert-medium' and 'blockdev-change-medium', to
correctly take the lock.
Cc: qemu-stable@nongnu.org Fixes: https://issues.redhat.com/browse/RHEL-3922 Fixes: c066e808e11a5c181b625537b6c78e0de27a4801 Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-ID: <20231013153302.39234-2-kwolf@redhat.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
iotests: add tests for "qemu-img rebase" with compression
The test cases considered so far:
314 (new test suite):
1. Check that compression mode isn't compatible with "-f raw" (raw
format doesn't support compression).
2. Check that rebasing an image onto no backing file preserves the data
and writes the copied clusters actually compressed.
3. Same as 2, but with a raw backing file (i.e. the clusters copied from the
backing are originally uncompressed -- we check they end up compressed
after being merged).
4. Remove a single delta from a backing chain, perform the same checks
as in 2.
5. Check that even when backing and overlay are initially uncompressed,
copied clusters end up compressed when rebase with compression is
performed.
271:
1. Check that when target image has subclusters, rebase with compression
will make an entire cluster containing the written subcluster
compressed.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-9-andrey.drobyshev@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qemu-img: add compression option to rebase subcommand
If we rebase an image whose backing file has compressed clusters, we
might end up wasting disk space since the copied clusters are now
uncompressed. In order to have better control over this, let's add
"--compress" option to the "qemu-img rebase" command.
Note that this option affects only the clusters which are actually being
copied from the original backing file. The clusters which were
uncompressed in the target image will remain so.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-8-andrey.drobyshev@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
iotests/{024, 271}: add testcases for qemu-img rebase
As the previous commit changes the logic of "qemu-img rebase" (it's using
write alignment now), let's add a couple more test cases which would
ensure it works correctly. In particular, the following scenarios:
024: add test case for rebase within one backing chain when the overlay
cluster size > backings cluster size;
271: add test case for rebase images that contain subclusters. Check
that no extra allocations are being made.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-7-andrey.drobyshev@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
When rebasing an image from one backing file to another, we need to
compare data from old and new backings. If the diff between that data
happens to be unaligned to the target cluster size, we might end up
doing partial writes, which would lead to copy-on-write and additional IO.
Consider the following simple case (virtual_size == cluster_size == 64K):
While doing rebase, we'll write a half of the cluster to inc2, and block
layer will have to read the 2nd half of the same cluster from the base image
inc1 while doing this write operation, although the whole cluster is already
read earlier to perform data comparison.
In order to avoid these unnecessary IO cycles, let's make sure every
write request is aligned to the overlay subcluster boundaries. Using
subcluster size is universal as for the images which don't have them
this size equals to the cluster size. so in any case we end up aligning
to the smallest unit of allocation.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Message-ID: <20230919165804.439110-6-andrey.drobyshev@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qemu-img: add chunk size parameter to compare_buffers()
Add @chsize param to the function which, if non-zero, would represent
the chunk size to be used for comparison. If it's zero, then
BDRV_SECTOR_SIZE is used as default chunk size, which is the previous
behaviour.
In particular, we're going to use this param in img_rebase() to make the
write requests aligned to a predefined alignment value.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com> Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-5-andrey.drobyshev@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qemu-img: rebase: use backing files' BlockBackend for buffer alignment
Since commit bb1c05973cf ("qemu-img: Use qemu_blockalign"), buffers for
the data read from the old and new backing files are aligned using
BlockDriverState (or BlockBackend later on) referring to the target image.
However, this isn't quite right, because buf_new is only being used for
reading from the new backing, while buf_old is being used for both reading
from the old backing and writing to the target. Let's take that into account
and use more appropriate values as alignments.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Message-ID: <20230919165804.439110-4-andrey.drobyshev@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qemu-iotests: 024: add rebasing test case for overlay_size > backing_size
Before previous commit, rebase was getting infitely stuck in case of
rebasing within the same backing chain and when overlay_size > backing_size.
Let's add this case to the rebasing test 024 to make sure it doesn't
break again.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-3-andrey.drobyshev@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
qemu-img: rebase: stop when reaching EOF of old backing file
In case when we're rebasing within one backing chain, and when target image
is larger than old backing file, bdrv_is_allocated_above() ends up setting
*pnum = 0. As a result, target offset isn't getting incremented, and we
get stuck in an infinite for loop. Let's detect this case and proceed
further down the loop body, as the offsets beyond the old backing size need
to be explicitly zeroed.
Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com> Reviewed-by: Denis V. Lunev <den@openvz.org> Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-2-andrey.drobyshev@virtuozzo.com> Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Juan Quintela [Wed, 25 Oct 2023 09:11:17 +0000 (11:11 +0200)]
qemu-file: Make qemu_fflush() return errors
This let us simplify code of this shape.
qemu_fflush(f);
int ret = qemu_file_get_error(f);
if (ret) {
return ret;
}
into:
int ret = qemu_fflush(f);
if (ret) {
return ret;
}
I updated all callers where there is any error check.
qemu_fclose() don't need to check for f->last_error because
qemu_fflush() returns it at the beggining of the function.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Reviewed-by: Fabiano Rosas <farosas@suse.de> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231025091117.6342-13-quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Juan Quintela [Wed, 25 Oct 2023 09:11:15 +0000 (11:11 +0200)]
migration: Use migration_transferred_bytes()
There are only two differnces with the old value:
- the amount of QEMUFile that hasn't yet been flushed. It can be
discussed what is more exact, the new or the old one.
- the amount of transferred bytes that we forgot to account for (the
newer is better, i.e. exact).
Notice that this two values are used to:
a - present to the user
b - calculate the rate_limit
So a few KB here and there is not going to make a difference.
Reviewed-by: Fabiano Rosas <farosas@suse.de> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231025091117.6342-11-quintela@redhat.com>
Juan Quintela [Wed, 25 Oct 2023 09:11:13 +0000 (11:11 +0200)]
migration: migration_rate_limit_reset() don't need the QEMUFile
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231025091117.6342-9-quintela@redhat.com>
Juan Quintela [Wed, 25 Oct 2023 09:11:12 +0000 (11:11 +0200)]
migration: migration_transferred_bytes() don't need the QEMUFile
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231025091117.6342-8-quintela@redhat.com>
Juan Quintela [Wed, 25 Oct 2023 09:11:09 +0000 (11:11 +0200)]
migration: Use the number of transferred bytes directly
We only use migration_transferred_bytes() to calculate the rate_limit,
for that we don't need to flush whatever is on the qemu_file buffer.
Remember that the buffer is really small (normal case is 32K if we use
iov's can be 64 * TARGET_PAGE_SIZE), so this is not relevant to
calculations.
Reviewed-by: Fabiano Rosas <farosas@suse.de> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231025091117.6342-5-quintela@redhat.com>
Juan Quintela [Wed, 25 Oct 2023 09:11:06 +0000 (11:11 +0200)]
qemu-file: Don't increment qemu_file_transferred at qemu_file_fill_buffer
We only call qemu_file_transferred_* on the sending side. Remove the
increment at qemu_file_fill_buffer() and add asserts to
qemu_file_transferred* functions.
Reviewed-by: Fabiano Rosas <farosas@suse.de> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231025091117.6342-2-quintela@redhat.com>
Juan Quintela [Wed, 18 Oct 2023 11:55:13 +0000 (13:55 +0200)]
migration: Deprecate old compression method
Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Acked-by: Peter Xu <peterx@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231018115513.2163-6-quintela@redhat.com>
Juan Quintela [Wed, 18 Oct 2023 11:55:12 +0000 (13:55 +0200)]
migration: Deprecate block migration
It is obsolete. It is better to use driver-mirror with NBD instead.
CC: Kevin Wolf <kwolf@redhat.com> CC: Eric Blake <eblake@redhat.com> CC: Stefan Hajnoczi <stefanha@redhat.com> CC: Hanna Czenczek <hreitz@redhat.com> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231018115513.2163-5-quintela@redhat.com>
Juan Quintela [Wed, 18 Oct 2023 11:55:11 +0000 (13:55 +0200)]
migration: migrate 'blk' command option is deprecated.
Use blocked-mirror with NBD instead.
Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231018115513.2163-4-quintela@redhat.com>
Juan Quintela [Wed, 18 Oct 2023 11:55:10 +0000 (13:55 +0200)]
migration: migrate 'inc' command option is deprecated.
Use blockdev-mirror with NBD instead.
Reviewed-by: Thomas Huth <thuth@redhat.com> Acked-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231018115513.2163-3-quintela@redhat.com>
Juan Quintela [Wed, 18 Oct 2023 11:55:09 +0000 (13:55 +0200)]
qemu-iotests: Filter warnings about block migration being deprecated
Create a new filter that removes the two warnings for test 183.
Reviewed-by: Hanna Czenczek <hreitz@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231018115513.2163-2-quintela@redhat.com>
Stefan Hajnoczi [Mon, 30 Oct 2023 22:07:42 +0000 (07:07 +0900)]
Merge tag 'pull-target-arm-20231027' of https://git-us.linaro.org/people/pmaydell/qemu-arm into staging
target-arm queue:
* Correct minor errors in Cortex-A710 definition
* Implement Neoverse N2 CPU model
* Refactor feature test functions out into separate header
* Fix syndrome for FGT traps on ERET
* Remove 'hw/arm/boot.h' includes from various header files
* pxa2xx: Refactoring/cleanup
* Avoid using 'first_cpu' when first ARM CPU is reachable
* misc/led: LED state is set opposite of what is expected
* hw/net/cadence_gen: clean up to use FIELD macros
* hw/net/cadence_gem: perform PHY access on write only
* hw/net/cadence_gem: enforce 32 bits variable size for CRC
* tag 'pull-target-arm-20231027' of https://git-us.linaro.org/people/pmaydell/qemu-arm: (41 commits)
hw/net/cadence_gem: enforce 32 bits variable size for CRC
hw/net/cadence_gem: perform PHY access on write only
hw/net/cadence_gem: use FIELD to describe PHYMNTNC register fields
hw/net/cadence_gem: use FIELD to describe DESCONF6 register fields
hw/net/cadence_gem: use FIELD to describe IRQ register fields
hw/net/cadence_gem: use FIELD to describe [TX|RX]STATUS register fields
hw/net/cadence_gem: use FIELD to describe DMACFG register fields
hw/net/cadence_gem: use FIELD to describe NWCFG register fields
hw/net/cadence_gem: use FIELD to describe NWCTRL register fields
hw/net/cadence_gem: use FIELD for screening registers
hw/net/cadence_gem: use REG32 macro for register definitions
misc/led: LED state is set opposite of what is expected
hw/arm: Avoid using 'first_cpu' when first ARM CPU is reachable
hw/arm/pxa2xx: Realize PXA2XX_I2C device before accessing it
hw/intc/pxa2xx: Factor pxa2xx_pic_realize() out of pxa2xx_pic_init()
hw/intc/pxa2xx: Pass CPU reference using QOM link property
hw/intc/pxa2xx: Convert to Resettable interface
hw/pcmcia/pxa2xx: Inline pxa2xx_pcmcia_init()
hw/pcmcia/pxa2xx: Do not open-code sysbus_create_simple()
hw/pcmcia/pxa2xx: Realize sysbus device before accessing it
...
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This is not ideal, since it requires all cross-compilers
to be present rather than a simple subset. But since it
is only run manually, should be good enough for now.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Requires a relatively recent binutils version in order to avoid
spurious R_LARCH_NONE relocations. The presence of these relocs
are diagnosed by our gen-vdso tool.
Tested-by: Song Gao <gaosong@loongson.cn> Reviewed-by: Song Gao <gaosong@loongson.cn> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Alistair Francis <alistair.francis@wdc.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
projects / mirror_qemu.git / log