]> git.proxmox.com Git - mirror_qemu.git/log
mirror_qemu.git
7 months agoMerge tag 'migration-20231102-pull-request' of https://gitlab.com/juan.quintela/qemu...
Stefan Hajnoczi [Fri, 3 Nov 2023 01:57:32 +0000 (09:57 +0800)]
Merge tag 'migration-20231102-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20231102)

Hi

In this pull request:

- migration reboot mode (steve)
  * I disabled the test because our CI don't like programs using so
    much shared memory.  Searching for a fix.
- test for postcopy recover (fabiano)
- MigrateAddress QAPI (het)
- better return path error handling (peter)
- traces for downtime (peter)
- vmstate_register() check for duplicates (juan)
  thomas find better solutions for s390x and ipmi.
  now also works on s390x

Please, apply.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmVDipMACgkQ9IfvGFhy
# 1yNYnQ/9E5Cywsoqljqa/9FiKBSII2qMrmkfu6JLKqePnsh5pFZiukbudYRuJCCe
# ZTDEmD0NmKRJbDx2xRU1qx/e6gKJy+gz37KP89Buuh/WwZHPboPYtxQpGvCSiH26
# J3i+1+TgaqmkLzcO35wa8tp6gneQclWeAwKgMvdb4cm2pJEhgWRKI62ccyLzxeve
# UCzFQn60t55ETyVZGnRD4YwdTQvGKH+DPlyTuJOLR3DePuvZd8EdH+ypvB4RLAy7
# 3+CuQOxmF5LRXPbpJuAeOsudbmhhHzrO/yL7ZmsiKQTthsJv+SzC1bO94jhQrawZ
# Q7GCii5KpGq0KnRTRKZRGk6XKwxcYRduXMX3R5tXuVmDmCZsjhXzziU8yEdftph8
# 5TJdk1o0Gb043EFu81mrsQYS+9yJqe6sy6m3PTJaec54cAty5ln+c17WOvpAOaSV
# +1phe05ftuVPmQ3KWhbIR/tCmavNLwEZxpVIfyaKJx04bFbtQ9gRpRyURORX4KXc
# s4WXvNirQEohxYBnP4TPvA09xBTW3V08pk/wRDwt0YDXnLiqCltOuxD8r05K8K4B
# MkCLcWj0g7he2tBkF60oz1KSIE0oTB81um9AzLIv5F2YSYLaJM5BIcoC437MR2f4
# MOR7drR1fP5GsRu/SeU5BWvhVq3IvdOxR7G2MLNRJJvl7ZtGXDc=
# =uaqL
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 02 Nov 2023 19:40:03 HKT
# gpg:                using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg:                 aka "Juan Quintela <quintela@trasno.org>" [full]
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03  4B82 F487 EF18 5872 D723

* tag 'migration-20231102-pull-request' of https://gitlab.com/juan.quintela/qemu: (40 commits)
  migration: modify test_multifd_tcp_none() to use new QAPI syntax.
  migration: Implement MigrateChannelList to hmp migration flow.
  migration: Implement MigrateChannelList to qmp migration flow.
  migration: modify migration_channels_and_uri_compatible() for new QAPI syntax
  migration: New migrate and migrate-incoming argument 'channels'
  migration: Convert the file backend to the new QAPI syntax
  migration: convert exec backend to accept MigrateAddress.
  migration: convert rdma backend to accept MigrateAddress
  migration: convert socket backend to accept MigrateAddress
  migration: convert migration 'uri' into 'MigrateAddress'
  migration: New QAPI type 'MigrateAddress'
  migration: Change ram_dirty_bitmap_reload() retval to bool
  tests/migration-test: Add a test for postcopy hangs during RECOVER
  migration: Allow network to fail even during recovery
  migration: Refactor error handling in source return path
  tests/qtest: migration: add reboot mode test
  cpr: reboot mode
  cpr: relax vhost migration blockers
  cpr: relax blockdev migration blockers
  migration: per-mode blockers
  ...

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
7 months agomigration: modify test_multifd_tcp_none() to use new QAPI syntax.
Het Gala [Mon, 23 Oct 2023 18:20:53 +0000 (15:20 -0300)]
migration: modify test_multifd_tcp_none() to use new QAPI syntax.

modify multifd tcp common test to incorporate the new QAPI
syntax defined.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-15-farosas@suse.de>

7 months agomigration: Implement MigrateChannelList to hmp migration flow.
Het Gala [Mon, 23 Oct 2023 18:20:52 +0000 (15:20 -0300)]
migration: Implement MigrateChannelList to hmp migration flow.

Integrate MigrateChannelList with all transport backends
(socket, exec and rdma) for both src and dest migration
endpoints for hmp migration.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Message-ID: <20231023182053.8711-14-farosas@suse.de>
Signed-off-by: Juan Quintela <quintela@redhat.com>
7 months agomigration: Implement MigrateChannelList to qmp migration flow.
Het Gala [Mon, 23 Oct 2023 18:20:51 +0000 (15:20 -0300)]
migration: Implement MigrateChannelList to qmp migration flow.

Integrate MigrateChannelList with all transport backends
(socket, exec and rdma) for both src and dest migration
endpoints for qmp migration.

For current series, limit the size of MigrateChannelList
to single element (single interface) as runtime check.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-13-farosas@suse.de>

7 months agomigration: modify migration_channels_and_uri_compatible() for new QAPI syntax
Het Gala [Mon, 23 Oct 2023 18:20:50 +0000 (15:20 -0300)]
migration: modify migration_channels_and_uri_compatible() for new QAPI syntax

migration_channels_and_uri_compatible() check for transport mechanism
suitable for multifd migration gets executed when the caller calls old
uri syntax. It needs it to be run when using the modern MigrateChannel
QAPI syntax too.

After URI -> 'MigrateChannel' :
migration_channels_and_uri_compatible() ->
migration_channels_and_transport_compatible() passes object as argument
and check for valid transport mechanism.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-12-farosas@suse.de>

7 months agomigration: New migrate and migrate-incoming argument 'channels'
Het Gala [Mon, 23 Oct 2023 18:20:48 +0000 (15:20 -0300)]
migration: New migrate and migrate-incoming argument 'channels'

MigrateChannelList allows to connect accross multiple interfaces.
Add MigrateChannelList struct as argument to migration QAPIs.

We plan to include multiple channels in future, to connnect
multiple interfaces. Hence, we choose 'MigrateChannelList'
as the new argument over 'MigrateChannel' to make migration
QAPIs future proof.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-10-farosas@suse.de>

7 months agomigration: Convert the file backend to the new QAPI syntax
Fabiano Rosas [Mon, 23 Oct 2023 18:20:47 +0000 (15:20 -0300)]
migration: Convert the file backend to the new QAPI syntax

Convert the file: URI to accept a FileMigrationArgs to be compatible
with the new migration QAPI.

Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-9-farosas@suse.de>

7 months agomigration: convert exec backend to accept MigrateAddress.
Het Gala [Mon, 23 Oct 2023 18:20:46 +0000 (15:20 -0300)]
migration: convert exec backend to accept MigrateAddress.

Exec transport backend for 'migrate'/'migrate-incoming' QAPIs accept
new wire protocol of MigrateAddress struct.

It is achived by parsing 'uri' string and storing migration parameters
required for exec connection into strList struct.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-8-farosas@suse.de>

7 months agomigration: convert rdma backend to accept MigrateAddress
Het Gala [Mon, 23 Oct 2023 18:20:45 +0000 (15:20 -0300)]
migration: convert rdma backend to accept MigrateAddress

RDMA based transport backend for 'migrate'/'migrate-incoming' QAPIs
accept new wire protocol of MigrateAddress struct.

It is achived by parsing 'uri' string and storing migration parameters
required for RDMA connection into well defined InetSocketAddress struct.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-7-farosas@suse.de>

7 months agomigration: convert socket backend to accept MigrateAddress
Het Gala [Mon, 23 Oct 2023 18:20:44 +0000 (15:20 -0300)]
migration: convert socket backend to accept MigrateAddress

Socket transport backend for 'migrate'/'migrate-incoming' QAPIs accept
new wire protocol of MigrateAddress struct.

It is achived by parsing 'uri' string and storing migration parameters
required for socket connection into well defined SocketAddress struct.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-6-farosas@suse.de>

7 months agomigration: convert migration 'uri' into 'MigrateAddress'
Het Gala [Mon, 23 Oct 2023 18:20:42 +0000 (15:20 -0300)]
migration: convert migration 'uri' into 'MigrateAddress'

This patch parses 'migrate' and 'migrate-incoming' QAPI's 'uri'
string containing migration connection related information
and stores them inside well defined 'MigrateAddress' struct.

Fabiano fixed for "file" transport.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-4-farosas@suse.de>
Message-ID: <20231023182053.8711-5-farosas@suse.de>

7 months agomigration: New QAPI type 'MigrateAddress'
Het Gala [Mon, 23 Oct 2023 18:20:40 +0000 (15:20 -0300)]
migration: New QAPI type 'MigrateAddress'

This patch introduces well defined MigrateAddress struct
and its related child objects.

The existing argument of 'migrate' and 'migrate-incoming' QAPI
- 'uri' is of type string. The current implementation follows
double encoding scheme for fetching migration parameters like
'uri' and this is not an ideal design.

Motive for intoducing struct level design is to prevent double
encoding of QAPI arguments, as Qemu should be able to directly
use the QAPI arguments without any level of encoding.

Note: this commit only adds the type, and actual uses comes
in later commits.

Fabiano fixed for "file" transport.

Suggested-by: Aravind Retnakaran <aravind.retnakaran@nutanix.com>
Signed-off-by: Het Gala <het.gala@nutanix.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231023182053.8711-2-farosas@suse.de>
Message-Id: <20231023182053.8711-3-farosas@suse.de>

7 months agomigration: Change ram_dirty_bitmap_reload() retval to bool
Peter Xu [Tue, 17 Oct 2023 20:26:32 +0000 (16:26 -0400)]
migration: Change ram_dirty_bitmap_reload() retval to bool

Now we have a Error** passed into the return path thread stack, which is
even clearer than an int retval.  Change ram_dirty_bitmap_reload() and the
callers to use a bool instead to replace errnos.

Suggested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231017202633.296756-5-peterx@redhat.com>

7 months agotests/migration-test: Add a test for postcopy hangs during RECOVER
Fabiano Rosas [Tue, 17 Oct 2023 20:26:31 +0000 (16:26 -0400)]
tests/migration-test: Add a test for postcopy hangs during RECOVER

To do so, create two paired sockets, but make them not providing real data.
Feed those fake sockets to src/dst QEMUs for recovery to let them go into
RECOVER stage without going out.  Test that we can always kick it out and
recover again with the right ports.

This patch is based on Fabiano's version here:

https://lore.kernel.org/r/877cowmdu0.fsf@suse.de

Signed-off-by: Fabiano Rosas <farosas@suse.de>
[peterx: write commit message, remove case 1, fix bugs, and more]
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231017202633.296756-4-peterx@redhat.com>

7 months agomigration: Allow network to fail even during recovery
Peter Xu [Tue, 17 Oct 2023 20:26:30 +0000 (16:26 -0400)]
migration: Allow network to fail even during recovery

Normally the postcopy recover phase should only exist for a super short
period, that's the duration when QEMU is trying to recover from an
interrupted postcopy migration, during which handshake will be carried out
for continuing the procedure with state changes from PAUSED -> RECOVER ->
POSTCOPY_ACTIVE again.

Here RECOVER phase should be super small, that happens right after the
admin specified a new but working network link for QEMU to reconnect to
dest QEMU.

However there can still be case where the channel is broken in this small
RECOVER window.

If it happens, with current code there's no way the src QEMU can got kicked
out of RECOVER stage. No way either to retry the recover in another channel
when established.

This patch allows the RECOVER phase to fail itself too - we're mostly
ready, just some small things missing, e.g. properly kick the main
migration thread out when sleeping on rp_sem when we found that we're at
RECOVER stage.  When this happens, it fails the RECOVER itself, and
rollback to PAUSED stage.  Then the user can retry another round of
recovery.

To make it even stronger, teach QMP command migrate-pause to explicitly
kick src/dst QEMU out when needed, so even if for some reason the migration
thread didn't got kicked out already by a failing rethrn-path thread, the
admin can also kick it out.

This will be an super, super corner case, but still try to cover that.

One can try to test this with two proxy channels for migration:

  (a) socat unix-listen:/tmp/src.sock,reuseaddr,fork tcp:localhost:10000
  (b) socat tcp-listen:10000,reuseaddr,fork unix:/tmp/dst.sock

So the migration channel will be:

                      (a)          (b)
  src -> /tmp/src.sock -> tcp:10000 -> /tmp/dst.sock -> dst

Then to make QEMU hang at RECOVER stage, one can do below:

  (1) stop the postcopy using QMP command postcopy-pause
  (2) kill the 2nd proxy (b)
  (3) try to recover the postcopy using /tmp/src.sock on src
  (4) src QEMU will go into RECOVER stage but won't be able to continue
      from there, because the channel is actually broken at (b)

Before this patch, step (4) will make src QEMU stuck in RECOVER stage,
without a way to kick the QEMU out or continue the postcopy again.  After
this patch, (4) will quickly fail qemu and bounce back to PAUSED stage.

Admin can also kick QEMU from (4) into PAUSED when needed using
migrate-pause when needed.

After bouncing back to PAUSED stage, one can recover again.

Reported-by: Xiaohui Li <xiaohli@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2111332
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231017202633.296756-3-peterx@redhat.com>

7 months agomigration: Refactor error handling in source return path
Peter Xu [Tue, 17 Oct 2023 20:26:29 +0000 (16:26 -0400)]
migration: Refactor error handling in source return path

rp_state.error was a boolean used to show error happened in return path
thread.  That's not only duplicating error reporting (migrate_set_error),
but also not good enough in that we only do error_report() and set it to
true, we never can keep a history of the exact error and show it in
query-migrate.

To make this better, a few things done:

  - Use error_setg() rather than error_report() across the whole lifecycle
    of return path thread, keeping the error in an Error*.

  - With above, no need to have mark_source_rp_bad(), remove it, alongside
    with rp_state.error itself.

  - Use migrate_set_error() to apply that captured error to the global
    migration object when error occured in this thread.

  - Do the same when detected qemufile error in source return path

We need to re-export qemu_file_get_error_obj() to do the last one.

Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231017202633.296756-2-peterx@redhat.com>

7 months agotests/qtest: migration: add reboot mode test
Steve Sistare [Wed, 25 Oct 2023 19:44:29 +0000 (12:44 -0700)]
tests/qtest: migration: add reboot mode test

[ Maintainer note:

I put the test as flaky because our CI has problems with shared
memory.  We will remove the flaky bits as soon as we get a solution.
]

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-7-git-send-email-steven.sistare@oracle.com>

7 months agoMerge tag 'm68k-pull-2023-11-02' of https://gitlab.com/huth/qemu into staging
Stefan Hajnoczi [Thu, 2 Nov 2023 10:21:21 +0000 (18:21 +0800)]
Merge tag 'm68k-pull-2023-11-02' of https://gitlab.com/huth/qemu into staging

* Avoid qemu_get_cpu() and first_cpu, and use properties instead
* Mirror next-cube bios to address 0
* Instantiate ESP SCSI controller in the next-cube machine
* Fix URL in the next-cube avocado test

# -----BEGIN PGP SIGNATURE-----
#
# iQJHBAABCAAxFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAmVDbcETHGh1dGhAdHV4
# ZmFtaWx5Lm9yZwAKCRAu2dd0/nAttY1UEACfU8+L6ySkA6X0gtHKyMCQTna+RgGw
# TCQM3eUcL+/ZhcKrBBj397ea80h+VWrniixEpB2VSnQJsHraaIWtLTiOQFPRfVF9
# 4K5Nx/yuNq/6LX+jB5rvntKBeiU97E2XFZ7MKWQRhnjyUKeu6wAXtv4mhiLpj5wN
# PwbREVb0dqI7km/RAfUPJ4cAClokTIHUiUWRcaFR646weH6SU8JIOAyaGpeOhdYY
# QtWRIUThHjuGt2jfXzgOcX+kkc63BuxOn1OOVJpx5j39PhbXeIoPDQoY+EMgEztt
# HS7LYNqh7K8H2nmcgX7pzMEEZlJbDp5R8nLsK7G2a3/s3eAFc1ssAUQV8ZRmncNp
# gup6PDlNjoAb1qr8cu8vfdDMQzlGAIlH68YckSDHSa2H+bFF5LDWMsK1CKUz04x8
# XYub9uA2lH9j34B63Jbnk1grJsB/6gCFl3CTGgljL3kMZZncvmHSUi0TliX9v8pv
# xo7CH/SlG+Xp3XG72E2O01GZ4fZqaStDySr/xCc1hySurR2Ylw7qlu5e40nh8+CT
# CPjY9eWx/4mv7sNTCC/TndawKOBBfQpV3m+mWB2gNnLIuZmdnigcY+GuQaTwpTma
# PteoEMNgqE+H0FAC88n5SL9+tiAoEQ/Xu6HuBDqwJJC1d/+KqvJrkFH6BxzyFrH4
# nSBXNb8y1+qpkA==
# =MOWk
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 02 Nov 2023 17:37:05 HKT
# gpg:                using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg:                issuer "huth@tuxfamily.org"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg:                 aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg:                 aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg:                 aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3  EAB9 2ED9 D774 FE70 2DB5

* tag 'm68k-pull-2023-11-02' of https://gitlab.com/huth/qemu:
  tests/avocado/machine_m68k_nextcube: Fix the download URL for the ROM image
  m68k: Instantiate the ESP SCSI controller for the NeXTcube machine
  hw/m68k/next-cube: Mirror BIOS to address 0
  hw/char/mcf_uart: Have mcf_uart_create() return DeviceState
  hw/m68k/virt: Do not open-code sysbus_create_simple()
  hw/m68k/next-cube: Do not open-code sysbus_create_simple()
  hw/m68k/mcf_intc: Pass CPU using QOM link property
  hw/m68k/mcf_intc: Expose MMIO region via SysBus API
  hw/m68k/mcf5206: Pass CPU using QOM link property
  hw/m68k/irqc: Pass CPU using QOM link property

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
7 months agotests/avocado/machine_m68k_nextcube: Fix the download URL for the ROM image
Thomas Huth [Wed, 1 Nov 2023 20:19:34 +0000 (21:19 +0100)]
tests/avocado/machine_m68k_nextcube: Fix the download URL for the ROM image

If Avocado has to fetch this asset, the download fails with a 403 HTTP
error. Use a different URL to fix the issue.

Message-ID: <20231101201934.27637-1-huth@tuxfamily.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agom68k: Instantiate the ESP SCSI controller for the NeXTcube machine
Thomas Huth [Sat, 30 Sep 2023 13:23:49 +0000 (15:23 +0200)]
m68k: Instantiate the ESP SCSI controller for the NeXTcube machine

The NeXTcube uses a NCR 53C90 SCSI interface for its disks, so we should
be able to use the ESP controller from QEMU here. The code here has been
basically taken from Bryce Lanham's GSoC 2011 contribution, except for
the next_scsi_init() function which has been rewritte as a replacement
for the esp_init() function (that has been removed quite a while ago).

Note that SCSI is not working yet. The ESP code likely needs some more
fixes first and there still might be some bugs left in they way we wire
it up for the NeXT-Cube machine.

Signed-off-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Message-ID: <20230930132351.30282-4-huth@tuxfamily.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agohw/m68k/next-cube: Mirror BIOS to address 0
Thomas Huth [Sat, 30 Sep 2023 13:23:47 +0000 (15:23 +0200)]
hw/m68k/next-cube: Mirror BIOS to address 0

The ROM is also available at address 0, so add a proper mirror
for this address.

Acked-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Message-ID: <20230930132351.30282-2-huth@tuxfamily.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agocpr: reboot mode
Steve Sistare [Wed, 25 Oct 2023 19:44:28 +0000 (12:44 -0700)]
cpr: reboot mode

Add the cpr-reboot migration mode.  Usage:

$ qemu-system-$arch -monitor stdio ...
QEMU 8.1.50 monitor - type 'help' for more information
(qemu) migrate_set_capability x-ignore-shared on
(qemu) migrate_set_parameter mode cpr-reboot
(qemu) migrate -d file:vm.state
(qemu) info status
VM status: paused (postmigrate)
(qemu) quit

$ qemu-system-$arch -monitor stdio -incoming defer ...
QEMU 8.1.50 monitor - type 'help' for more information
(qemu) migrate_set_capability x-ignore-shared on
(qemu) migrate_set_parameter mode cpr-reboot
(qemu) migrate_incoming file:vm.state
(qemu) info status
VM status: running

In this mode, the migrate command saves state to a file, allowing one
to quit qemu, reboot to an updated kernel, and restart an updated version
of qemu.  The caller must specify a migration URI that writes to and reads
from a file.  Unlike normal mode, the use of certain local storage options
does not block the migration, but the caller must not modify guest block
devices between the quit and restart.  To avoid saving guest RAM to the
file, the memory backend must be shared, and the @x-ignore-shared migration
capability must be set.  Guest RAM must be non-volatile across reboot, such
as by backing it with a dax device, but this is not enforced.  The restarted
qemu arguments must match those used to initially start qemu, plus the
-incoming option.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-6-git-send-email-steven.sistare@oracle.com>

7 months agocpr: relax vhost migration blockers
Steve Sistare [Wed, 25 Oct 2023 19:44:27 +0000 (12:44 -0700)]
cpr: relax vhost migration blockers

vhost blocks migration if logging is not supported to track dirty
memory, and vhost-user blocks it if the log cannot be saved to a shm fd.

vhost-vdpa blocks migration if both hosts do not support all the device's
features using a shadow VQ, for tracking requests and dirty memory.

vhost-scsi blocks migration if storage cannot be shared across hosts,
or if state cannot be migrated.

None of these conditions apply if the old and new qemu processes do
not run concurrently, and if new qemu starts on the same host as old,
which is the case for cpr.

Narrow the scope of these blockers so they only apply to normal mode.
They will not block cpr modes when they are added in subsequent patches.

No functional change until a new mode is added.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-5-git-send-email-steven.sistare@oracle.com>

7 months agocpr: relax blockdev migration blockers
Steve Sistare [Wed, 25 Oct 2023 19:44:26 +0000 (12:44 -0700)]
cpr: relax blockdev migration blockers

Some blockdevs block migration because they do not support sharing across
hosts and/or do not support dirty bitmaps.  These prohibitions do not apply
if the old and new qemu processes do not run concurrently, and if new qemu
starts on the same host as old, which is the case for cpr.  Narrow the scope
of these blockers so they only apply to normal mode.  They will not block
cpr modes when they are added in subsequent patches.

No functional change until a new mode is added.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-4-git-send-email-steven.sistare@oracle.com>

7 months agomigration: per-mode blockers
Steve Sistare [Wed, 25 Oct 2023 19:44:25 +0000 (12:44 -0700)]
migration: per-mode blockers

Extend the blocker interface so that a blocker can be registered for
one or more migration modes.  The existing interfaces register a
blocker for all modes, and the new interfaces take a varargs list
of modes.

Internally, maintain a separate blocker list per mode.  The same Error
object may be added to multiple lists.  When a block is deleted, it is
removed from every list, and the Error is freed.

No functional change until a new mode is added.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-3-git-send-email-steven.sistare@oracle.com>

7 months agomigration: mode parameter
Steve Sistare [Wed, 25 Oct 2023 19:44:24 +0000 (12:44 -0700)]
migration: mode parameter

Create a mode migration parameter that can be used to select alternate
migration algorithms.  The default mode is normal, representing the
current migration algorithm, and does not need to be explicitly set.

No functional change until a new mode is added, except that the mode is
shown by the 'info migrate' command.

Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <1698263069-406971-2-git-send-email-steven.sistare@oracle.com>

7 months agomigration: Add tracepoints for downtime checkpoints
Peter Xu [Mon, 30 Oct 2023 16:33:46 +0000 (12:33 -0400)]
migration: Add tracepoints for downtime checkpoints

This patch is inspired by Joao Martin's patch here:

https://lore.kernel.org/r/20230926161841.98464-1-joao.m.martins@oracle.com

Add tracepoints for major downtime checkpoints on both src and dst.  They
share the same tracepoint with a string showing its stage.

Besides the checkpoints in the previous patch, this patch also added
destination checkpoints.

On src, we have these checkpoints added:

  - src-downtime-start: right before vm stops on src
  - src-vm-stopped: after vm is fully stopped
  - src-iterable-saved: after all iterables saved (END sections)
  - src-non-iterable-saved: after all non-iterable saved (FULL sections)
  - src-downtime-stop: migration fully completed

On dst, we have these checkpoints added:

  - dst-precopy-loadvm-completes: after loadvm all done for precopy
  - dst-precopy-bh-*: record BH steps to resume VM for precopy
  - dst-postcopy-bh-*: record BH steps to resume VM for postcopy

On dst side, we don't have a good way to trace total time consumed by
iterable or non-iterable for now.  We can mark it by 1st time receiving a
FULL / END section, but rather than that let's just rely on the other
tracepoints added for vmstates to back up the information.

With this patch, one can enable "vmstate_downtime*" tracepoints and it'll
enable all tracepoints for downtime measurements necessary.

Drop loadvm_postcopy_handle_run_bh() tracepoint alongside, because they
service the same purpose, which was only for postcopy.  We then have
unified prefix for all downtime relevant tracepoints.

Co-developed-by: Joao Martins <joao.m.martins@oracle.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-6-peterx@redhat.com>

7 months agomigration: migration_stop_vm() helper
Peter Xu [Mon, 30 Oct 2023 16:33:45 +0000 (12:33 -0400)]
migration: migration_stop_vm() helper

Provide a helper for non-COLO use case of migration to stop a VM.  This
prepares for adding some downtime relevant tracepoints to migration, where
they may or may not apply to COLO.

Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-5-peterx@redhat.com>

7 months agomigration: Add per vmstate downtime tracepoints
Peter Xu [Mon, 30 Oct 2023 16:33:44 +0000 (12:33 -0400)]
migration: Add per vmstate downtime tracepoints

We have a bunch of savevm_section* tracepoints, they're good to analyze
migration stream, but not always suitable if someone would like to analyze
the migration downtime.  Two major problems:

  - savevm_section* tracepoints are dumping all sections, we only care
    about the sections that contribute to the downtime

  - They don't have an identifier to show the type of sections, so no way
    to filter downtime information either easily.

We can add type into the tracepoints, but instead of doing so, this patch
kept them untouched, instead of adding a bunch of downtime specific
tracepoints, so one can enable "vmstate_downtime*" tracepoints and get a
full picture of how the downtime is distributed across iterative and
non-iterative vmstate save/load.

Note that here both save() and load() need to be traced, because both of
them may contribute to the downtime.  The contribution is not a simple "add
them together", though: consider when the src is doing a save() of device1
while the dest can be load()ing for device2, so they can happen
concurrently.

Tracking both sides make sense because device load() and save() can be
imbalanced, one device can save() super fast, but load() super slow, vice
versa.  We can't figure that out without tracing both.

Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-4-peterx@redhat.com>

7 months agomigration: Add migration_downtime_start|end() helpers
Peter Xu [Mon, 30 Oct 2023 16:33:43 +0000 (12:33 -0400)]
migration: Add migration_downtime_start|end() helpers

Unify the three users on recording downtimes with the same pair of helpers.

Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-3-peterx@redhat.com>

7 months agomigration: Set downtime_start even for postcopy
Peter Xu [Mon, 30 Oct 2023 16:33:42 +0000 (12:33 -0400)]
migration: Set downtime_start even for postcopy

Postcopy calculates its downtime separately.  It always sets
MigrationState.downtime properly, but not MigrationState.downtime_start.

Make postcopy do the same as other modes on properly recording the
timestamp when the VM is going to be stopped.  Drop the temporary variable
in postcopy_start() along the way.

Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231030163346.765724-2-peterx@redhat.com>

7 months agomigration: Use vmstate_register_any() for vmware_vga
Juan Quintela [Fri, 20 Oct 2023 09:07:31 +0000 (11:07 +0200)]
migration: Use vmstate_register_any() for vmware_vga

I have no idea if we can have more than one vmware_vga device, so play
it safe.

Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-14-quintela@redhat.com>

7 months agomigration: Use vmstate_register_any() for eeprom93xx
Juan Quintela [Fri, 20 Oct 2023 09:07:30 +0000 (11:07 +0200)]
migration: Use vmstate_register_any() for eeprom93xx

We can have more than one eeprom93xx.
For instance:

e100_nic_realize() -> eeprom93xx_new()

Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-13-quintela@redhat.com>

7 months agomigration: Use vmstate_register_any() for audio
Juan Quintela [Fri, 20 Oct 2023 09:07:29 +0000 (11:07 +0200)]
migration: Use vmstate_register_any() for audio

We can have more than one audio backend.

void audio_init_audiodevs(void)
{
    AudiodevListEntry *e;

    QSIMPLEQ_FOREACH(e, &audiodevs, next) {
        audio_init(e->dev, &error_fatal);
    }
}

Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-12-quintela@redhat.com>

7 months agomigration: Improve example and documentation of vmstate_register()
Juan Quintela [Fri, 20 Oct 2023 09:07:28 +0000 (11:07 +0200)]
migration: Improve example and documentation of vmstate_register()

Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-11-quintela@redhat.com>

7 months agomigration: Check in savevm_state_handler_insert for dups
Peter Xu [Fri, 20 Oct 2023 09:07:27 +0000 (11:07 +0200)]
migration: Check in savevm_state_handler_insert for dups

Before finally register one SaveStateEntry, we detect for duplicated
entries.  This could be helpful to notify us asap instead of get
silent migration failures which could be hard to diagnose.

For example, this patch will generate a message like this (if without
previous fixes on x2apic) as long as we wants to boot a VM instance
with "-smp 200,maxcpus=288,sockets=2,cores=72,threads=2" and QEMU will
bail out even before VM starts:

savevm_state_handler_insert: Detected duplicate SaveStateEntry: id=apic, instance_id=0x0

Suggested-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-10-quintela@redhat.com>

7 months agomigration: Hack to maintain backwards compatibility for ppc
Juan Quintela [Fri, 20 Oct 2023 09:07:25 +0000 (11:07 +0200)]
migration: Hack to maintain backwards compatibility for ppc

Current code does:
- register pre_2_10_vmstate_dummy_icp with "icp/server" and instance
  dependinfg on cpu number
- for newer machines, it register vmstate_icp with "icp/server" name
  and instance 0
- now it unregisters "icp/server" for the 1st instance.

This is wrong at many levels:
- we shouldn't have two VMSTATEDescriptions with the same name
- In case this is the only solution that we can came with, it needs to
  be:
  * register pre_2_10_vmstate_dummy_icp
  * unregister pre_2_10_vmstate_dummy_icp
  * register real vmstate_icp

Created vmstate_replace_hack_for_ppc() with warnings left and right
that it is a hack.

CC: Cedric Le Goater <clg@kaod.org>
CC: Daniel Henrique Barboza <danielhb413@gmail.com>
CC: David Gibson <david@gibson.dropbear.id.au>
CC: Greg Kurz <groug@kaod.org>
Reviewed-by: Nicholas Piggin <npiggin@gmail.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-8-quintela@redhat.com>

7 months agomigration: Use VMSTATE_INSTANCE_ID_ANY for slirp
Juan Quintela [Fri, 20 Oct 2023 09:07:23 +0000 (11:07 +0200)]
migration: Use VMSTATE_INSTANCE_ID_ANY for slirp

Each user network conection create a new slirp instance.  We register
more than one slirp instance for number 0.

qemu-system-x86_64: -netdev user,id=hs1: savevm_state_handler_insert: Detected duplicate SaveStateEntry: id=slirp, instance_id=0x0
Broken pipe
../../../../../mnt/code/qemu/full/tests/qtest/libqtest.c:195: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
Aborted (core dumped)

Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-6-quintela@redhat.com>

7 months agomigration: Use vmstate_register_any() for isa-ide
Juan Quintela [Fri, 20 Oct 2023 09:07:21 +0000 (11:07 +0200)]
migration: Use vmstate_register_any() for isa-ide

Otherwise qom-test fails.

ok 4 /i386/qom/x-remote
qemu-system-i386: savevm_state_handler_insert: Detected duplicate SaveStateEntry: id=isa-ide, instance_id=0x0
Broken pipe
../../../../../mnt/code/qemu/full/tests/qtest/libqtest.c:195: kill_qemu() tried to terminate QEMU process but encountered exit status 1 (expected 0)
Aborted (core dumped)
$

Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-4-quintela@redhat.com>

7 months agomigration: Use vmstate_register_any()
Juan Quintela [Fri, 20 Oct 2023 09:07:20 +0000 (11:07 +0200)]
migration: Use vmstate_register_any()

This are the easiest cases, where we were already using
VMSTATE_INSTANCE_ID_ANY.

Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-3-quintela@redhat.com>

7 months agomigration: Create vmstate_register_any()
Juan Quintela [Fri, 20 Oct 2023 09:07:19 +0000 (11:07 +0200)]
migration: Create vmstate_register_any()

We have lots of cases where we are using an instance_id==0 when we
should be using VMSTATE_INSTANCE_ID_ANY (-1).  Basically everything
that can have more than one needs to have a proper instance_id or -1
and the system will take one for it.

vmstate_register_any(): We register with -1.

Reviewed-by: Stefan Berger <stefanb@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020090731.28701-2-quintela@redhat.com>

7 months agohw/s390x/s390-stattrib: Don't call register_savevm_live() during instance_init()
Thomas Huth [Fri, 20 Oct 2023 15:05:54 +0000 (17:05 +0200)]
hw/s390x/s390-stattrib: Don't call register_savevm_live() during instance_init()

We must not call register_savevm_live() from an instance_init() function
(since this could be called multiple times during device introspection).
Move this to the realize() function instead.

Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Eric Farman <farman@linux.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020150554.664422-4-thuth@redhat.com>

7 months agohw/s390x/s390-stattrib: Simplify handling of the "migration-enabled" property
Thomas Huth [Fri, 20 Oct 2023 15:05:53 +0000 (17:05 +0200)]
hw/s390x/s390-stattrib: Simplify handling of the "migration-enabled" property

There's no need for dedicated handlers here if they don't do anything
special.

Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Eric Farman <farman@linux.ibm.com>
Acked-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020150554.664422-3-thuth@redhat.com>

7 months agohw/s390x/s390-skeys: Don't call register_savevm_live() during instance_init()
Thomas Huth [Fri, 20 Oct 2023 15:05:52 +0000 (17:05 +0200)]
hw/s390x/s390-skeys: Don't call register_savevm_live() during instance_init()

Since the instance_init() function immediately tries to set the
property to "true", the s390_skeys_set_migration_enabled() tries
to register a savevm handler during instance_init(). However,
instance_init() functions can be called multiple times, e.g. for
introspection of devices. That means multiple instances of devices
can be created during runtime (which is fine as long as they all
don't get realized, too), so the "Prevent double registration of
savevm handler" check in the s390_skeys_set_migration_enabled()
function does not work at all as expected (since there could be
more than one instance).

Thus we must not call register_savevm_live() from an instance_init()
function at all. Move this to the realize() function instead. This
way we can also get rid of the property getter and setter functions
completely, simplifying the code along the way quite a bit.

Acked-by: David Hildenbrand <david@redhat.com>
Reviewed-by: Eric Farman <farman@linux.ibm.com>
Acked-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020150554.664422-2-thuth@redhat.com>

7 months agohw/ipmi: Don't call vmstate_register() from instance_init() functions
Thomas Huth [Fri, 20 Oct 2023 14:55:54 +0000 (16:55 +0200)]
hw/ipmi: Don't call vmstate_register() from instance_init() functions

instance_init() can be called multiple times, e.g. during introspection
of the device. We should not install the vmstate handlers here. Do it
in the realize() function instead.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Acked-by: Corey Minyard <cminyard@mvista.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Message-ID: <20231020145554.662751-1-thuth@redhat.com>

7 months agohw/char/mcf_uart: Have mcf_uart_create() return DeviceState
Philippe Mathieu-Daudé [Thu, 19 Oct 2023 10:49:29 +0000 (12:49 +0200)]
hw/char/mcf_uart: Have mcf_uart_create() return DeviceState

There is no point in having mcf_uart_init() demote the DeviceState
pointer and return a void one. Directly return the real typedef.

mcf_uart_init() do both init + realize: rename as mcf_uart_create().

Similarly, mcf_uart_mm_init() do init / realize / mmap: rename as
mcf_uart_create_mmap().

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20231019104929.16517-1-philmd@linaro.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agohw/m68k/virt: Do not open-code sysbus_create_simple()
Philippe Mathieu-Daudé [Tue, 24 Oct 2023 08:30:09 +0000 (10:30 +0200)]
hw/m68k/virt: Do not open-code sysbus_create_simple()

Mechanical change using the following coccinelle script:

  @@
  identifier dev;
  expression qom_type;
  expression addr;
  expression irq;
  @@
  -    dev = qdev_new(qom_type);
  -    sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
  -    sysbus_mmio_map(SYS_BUS_DEVICE(dev), 0, addr);
  -    sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0, irq);
  +    dev = sysbus_create_simple(qom_type, addr, irq);

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <huth@tuxfamily.org>
Message-ID: <20231024083010.12453-7-philmd@linaro.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agohw/m68k/next-cube: Do not open-code sysbus_create_simple()
Philippe Mathieu-Daudé [Tue, 24 Oct 2023 08:30:08 +0000 (10:30 +0200)]
hw/m68k/next-cube: Do not open-code sysbus_create_simple()

Mechanical change using the following coccinelle script:

  @@
  identifier dev;
  identifier sbd;
  expression qom_type;
  expression addr;
  @@
  -    dev = qdev_new(qom_type);
  -    sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), &error_fatal);
  -    sysbus_mmio_map(SYS_BUS_DEVICE(dev), 0, addr);
  +    dev = sysbus_create_simple(qom_type, addr, NULL);

then manually removing the 'dev' variable to avoid:

  error: variable 'dev' set but not used [-Werror,-Wunused-but-set-variable]

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <huth@tuxfamily.org>
Message-ID: <20231024083010.12453-6-philmd@linaro.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agohw/m68k/mcf_intc: Pass CPU using QOM link property
Philippe Mathieu-Daudé [Tue, 24 Oct 2023 08:30:07 +0000 (10:30 +0200)]
hw/m68k/mcf_intc: Pass CPU using QOM link property

QOM objects shouldn't access each other internals fields
except using the QOM API.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Thomas Huth <huth@tuxfamily.org>
Reviewed-by: Thomas Huth <huth@tuxfamily.org>
Message-ID: <20231024083010.12453-5-philmd@linaro.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agohw/m68k/mcf_intc: Expose MMIO region via SysBus API
Philippe Mathieu-Daudé [Tue, 24 Oct 2023 08:30:06 +0000 (10:30 +0200)]
hw/m68k/mcf_intc: Expose MMIO region via SysBus API

QOM objects shouldn't access each other internals fields
except using the QOM API.

Here the caller of mcf_intc_init() access the MMIO region from
the MCF_INTC state. Avoid that by exposing that region via
sysbus_init_mmio(), then get it with sysbus_mmio_get_region().

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Thomas Huth <huth@tuxfamily.org>
Reviewed-by: Thomas Huth <huth@tuxfamily.org>
Message-ID: <20231024083010.12453-4-philmd@linaro.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agohw/m68k/mcf5206: Pass CPU using QOM link property
Philippe Mathieu-Daudé [Tue, 24 Oct 2023 08:30:05 +0000 (10:30 +0200)]
hw/m68k/mcf5206: Pass CPU using QOM link property

Avoid the interrupt controller directly access the first cpu via
the qemu_get_cpu() call. Pass it as argument to mcf5206_init()
from the board code.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <huth@tuxfamily.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20231024083010.12453-3-philmd@linaro.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agohw/m68k/irqc: Pass CPU using QOM link property
Philippe Mathieu-Daudé [Tue, 24 Oct 2023 08:30:04 +0000 (10:30 +0200)]
hw/m68k/irqc: Pass CPU using QOM link property

Avoid the interrupt controller directly access the 'first_cpu'
global. Pass 'cpu' from the board code.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <huth@tuxfamily.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20231024083010.12453-2-philmd@linaro.org>
Signed-off-by: Thomas Huth <huth@tuxfamily.org>
7 months agoMerge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging
Stefan Hajnoczi [Tue, 31 Oct 2023 21:58:11 +0000 (06:58 +0900)]
Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging

Block layer patches

- virtio-blk: use blk_io_plug_call() instead of notification BH
- mirror: allow switching from background to active mode
- qemu-img rebase: add compression support
- Fix locking in media change monitor commands
- Fix a few blockjob-related deadlocks when using iothread

# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEE3D3rFZqa+V09dFb+fwmycsiPL9YFAmVBTkERHGt3b2xmQHJl
# ZGhhdC5jb20ACgkQfwmycsiPL9ZiqRAAqvsWbblmEGJ7TBKYQK3f8QshJ66RxzbC
# 4eSjKHrciWNTeeIeU8r8OvFcPPoTcPXxpcmasD2gsAxG5W5N8vkPbBkW+YT4YdDJ
# pWJXrbJ15nILC4DmnR1ARVtvxKgv9zy5LSm5bjss1K+OSYJl/nx+ILjmfVZnYDF7
# z1dP/G0JxKKm4JzAIdBE3uZS+6Q5kx/wGYlJv8EQmlH3DYfsJfy6Lthe9jfw8ijg
# lSqLoQ+D0lEd6Bk4XbkUqqBxFcYBWTfU6qPZoyIO94zCTwTG9yIjmoivxmmfwQZq
# cJUTGGZjcxpJYnvcC6P13WgcWBtcD9L2kYFVH0JyjpwcSg9cCGHMF66n9pSlyEGq
# DUikwVzbTwOotwzYQyM88v4ET+2+Qdcwn8pRbv9PllEczh0kAsUAEuxSgtz4NEcN
# bZrap/16xHFybNOKkMZcmpqxspT5NXKbDODUP0IvbSYMOYpWS983nBTxwMRpyHog
# 2TFDZu4DjNiPkI2BcYM5VOKk6diNowZFShcEKvoaOLX/n9EBhP0tjoH9VUn1800F
# myHrhF2jpIf9GhErMWB7N2W3/0aK0pqdQgbpVnd1ARDdIdYkr7G/S+50D9K80b6n
# 0q2E7br4S5bcsY0HQzBL9YARSayY+lVOssLoolCWEsYzijdBQmAvs5THajFKcism
# /idI6nlp2Vs=
# =RdxS
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 01 Nov 2023 03:58:09 JST
# gpg:                using RSA key DC3DEB159A9AF95D3D7456FE7F09B272C88F2FD6
# gpg:                issuer "kwolf@redhat.com"
# gpg: Good signature from "Kevin Wolf <kwolf@redhat.com>" [full]
# Primary key fingerprint: DC3D EB15 9A9A F95D 3D74  56FE 7F09 B272 C88F 2FD6

* tag 'for-upstream' of https://repo.or.cz/qemu/kevin: (27 commits)
  iotests: add test for changing mirror's copy_mode
  mirror: return mirror-specific information upon query
  blockjob: query driver-specific info via a new 'query' driver method
  qapi/block-core: turn BlockJobInfo into a union
  qapi/block-core: use JobType for BlockJobInfo's type
  mirror: implement mirror_change method
  block/mirror: determine copy_to_target only once
  block/mirror: move dirty bitmap to filter
  block/mirror: set actively_synced even after the job is ready
  blockjob: introduce block-job-change QMP command
  virtio-blk: remove batch notification BH
  virtio: use defer_call() in virtio_irqfd_notify()
  util/defer-call: move defer_call() to util/
  block: rename blk_io_plug_call() API to defer_call()
  blockdev: mirror: avoid potential deadlock when using iothread
  block: avoid potential deadlock during bdrv_graph_wrlock() in bdrv_close()
  blockjob: drop AioContext lock before calling bdrv_graph_wrlock()
  iotests: Test media change with iothreads
  block: Fix locking in media change monitor commands
  iotests: add tests for "qemu-img rebase" with compression
  ...

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
7 months agoMerge tag 'pull-halloween-omnibus-311023-2' of https://gitlab.com/stsquad/qemu into...
Stefan Hajnoczi [Tue, 31 Oct 2023 21:57:29 +0000 (06:57 +0900)]
Merge tag 'pull-halloween-omnibus-311023-2' of https://gitlab.com/stsquad/qemu into staging

Maintainer updates for testing, gitlab, gdbstub and plugins:

  - add dtc package to openbsd VMs
  - use -fno-stack-protector for non-stdlib tests
  - split alpha and sh4 compilers into legacy image
  - harmonise other compilers into debian-all-test-cross
  - fix NULL check in gdb_regs
  - fix memleak in semihosting
  - remove unused parameter in plugin code
  - fix fd leak in lockstep plugin

# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmVBCvQACgkQ+9DbCVqe
# KkR8jAgAjFC3BE6fu80zYT0Dmeu8zh20QY/wgKQebaFfGEmPL4Bqkl2D/Rx7PhQA
# EH8fR/LAH/iXAO07+LYOB6QiyMb9PWiXS52iHyE3q11mOaM8iKkkj7a59NW8DfGC
# biSrj9o3wpz9gGkJjzTCcHC8DOMbrAuE12XnmhW7uTqqkrcTMC393dSEeyL+nrP9
# lKS5XzFyn3FOT4YIL8hAC02ObKH4LpWIO3gdWeDAo56yg24fLir9a2wYSXMaxQtN
# kDf6UtL97CIIhbNi6qrUPBB13MV8MlXno3wnb9+E4Cn5sGntGSnTyh7j6XrGqYj9
# p/Vio6ye8xP1IjlavKiBM0nnozcAhw==
# =ZOMS
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 31 Oct 2023 23:11:00 JST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg:          There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* tag 'pull-halloween-omnibus-311023-2' of https://gitlab.com/stsquad/qemu:
  contrib/plugins: Close file descriptor on error return
  plugins: Remove an extra parameter
  semihosting: fix memleak at semihosting_arg_fallback
  gdbstub: Check if gdb_regs is NULL
  tests/docker: upgrade debian-all-test-cross to bookworm
  tests/docker: use debian-all-test-cross for sparc64
  tests/docker: use debian-all-test-cross for riscv64
  tests/docker: use debian-all-test-cross for mips
  tests/docker: use debian-all-test-cross for mips64
  tests/docker: use debian-all-test-cross for m68k
  tests/docker: use debian-all-test-cross for hppa
  tests/docker: use debian-all-test-cross for power
  tests/docker: move sh4 to use debian-legacy-test-cross
  tests/docker: use debian-legacy-test-cross for alpha
  gitlab: add build-loongarch to matrix
  gitlab: clean-up build-soft-softmmu job
  gitlab: split alpha testing into a legacy container
  tests/tcg: Add -fno-stack-protector
  tests/vm/openbsd: Use the system dtc package

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
7 months agoMerge tag 'migration-20231031-pull-request' of https://gitlab.com/juan.quintela/qemu...
Stefan Hajnoczi [Tue, 31 Oct 2023 21:56:53 +0000 (06:56 +0900)]
Merge tag 'migration-20231031-pull-request' of https://gitlab.com/juan.quintela/qemu into staging

Migration Pull request (20231031)

Hi

This is repeat of the Migration PULL for 20231020.
- I removed vmstate_register(big problems with s390x)
- I added yet more countes (juan)

CI: https://gitlab.com/juan.quintela/qemu/-/pipelines/1055797950

Please apply.

Thanks, Juan.

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCAAdFiEEGJn/jt6/WMzuA0uC9IfvGFhy1yMFAmVAwmYACgkQ9IfvGFhy
# 1yPJ9g//f8Up+5Az0DmJMWwRe+08vLa3ZRCSh5aCRJguFVfMZSVxRNuoikQ/C/Gz
# 1ePB+Q8H0NcP86FF7pifhtLU0uE9L4At4Z+vOQP1+n67p7aush050kKQxyDYIfO2
# 3tO2HkfHvC/R3S5FtqQtE1Y0/MpHdj1vgV9bNidPorA6EZ01KEEfWw3soptuD14I
# LPvXA8BG5mOvB7R55MymTAej3ZDmOUQlZotsE2KmlkOfzYoqTtApkLtW03/WH8b8
# fAYJ0ghYpesRTO1rF61n1peLMUr+/HRLqGJmhLDSEZZlB5tnUYeiLR9dRJ1/1+o2
# zNjLr6X2hnia6Kb0UibRoAcyyy8lSLp79Zt5nhDneuTSQxeYhNh6EecxAzKvd/02
# vfE/reOEkZn7KzYH/MvlD5P6XmwrT5aV9cqmyC/8BkNnipHAtJ2Av1H4ONdnahuK
# hOhLRAGE7SINtgo8jdauQNor1QAsIX19nvYk9p7ta5VAysrDSbuD+9Yq7HtUErlP
# 585z5BPGfaP2GwIXPNJNcqXwPh0InInGASqEWmYSlu8GF3Ic0KNWWrC5bwSn7tHL
# I7qaMrCHxvWGYx6cRzzp08EqCcbOQCixrPyk8g6o3SgXHrTGKthzjPG5bLe+QXpv
# P2gblC7Fo3sUo89IwVjsRMO3nU9wBfb9skE7iZM06SILO7QD3u8=
# =r1DI
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 31 Oct 2023 18:01:26 JST
# gpg:                using RSA key 1899FF8EDEBF58CCEE034B82F487EF185872D723
# gpg: Good signature from "Juan Quintela <quintela@redhat.com>" [full]
# gpg:                 aka "Juan Quintela <quintela@trasno.org>" [full]
# Primary key fingerprint: 1899 FF8E DEBF 58CC EE03  4B82 F487 EF18 5872 D723

* tag 'migration-20231031-pull-request' of https://gitlab.com/juan.quintela/qemu: (38 commits)
  qemu-file: Make qemu_fflush() return errors
  migration: Remove transferred atomic counter
  migration: Use migration_transferred_bytes()
  qemu-file: Simplify qemu_file_get_error()
  migration: migration_rate_limit_reset() don't need the QEMUFile
  migration: migration_transferred_bytes() don't need the QEMUFile
  qemu-file: Remove _noflush from qemu_file_transferred_noflush()
  qemu_file: Remove unused qemu_file_transferred()
  migration: Use the number of transferred bytes directly
  qemu_file: total_transferred is not used anymore
  qemu_file: Use a stat64 for qemu_file_transferred
  qemu-file: Don't increment qemu_file_transferred at qemu_file_fill_buffer
  migration: Stop migration immediately in RDMA error paths
  migration: Deprecate old compression method
  migration: Deprecate block migration
  migration: migrate 'blk' command option is deprecated.
  migration: migrate 'inc' command option is deprecated.
  qemu-iotests: Filter warnings about block migration being deprecated
  migration: set file error on subsection loading
  migration: rename vmstate_save_needed->vmstate_section_needed
  ...

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
7 months agoiotests: add test for changing mirror's copy_mode
Fiona Ebner [Tue, 31 Oct 2023 13:54:31 +0000 (14:54 +0100)]
iotests: add test for changing mirror's copy_mode

One part of the test is using a throttled source to ensure that there
are no obvious issues when changing the copy_mode while there are
ongoing requests (source and target images are compared at the very
end).

The other part of the test is using a throttled target to ensure that
the change to active mode actually happened. This is done by hitting
the throttling limit, issuing a synchronous write and then immediately
verifying the target side. QSD is used, because otherwise, a
synchronous write would hang there.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-11-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agomirror: return mirror-specific information upon query
Fiona Ebner [Tue, 31 Oct 2023 13:54:30 +0000 (14:54 +0100)]
mirror: return mirror-specific information upon query

To start out, only actively-synced is returned.

For example, this is useful for jobs that started out in background
mode and switched to active mode. Once actively-synced is true, it's
clear that the mode switch has been completed. Note that completion of
the switch might happen much earlier, e.g. if the switch happens
before the job is ready, once all background operations have finished.
It's assumed that whether the disks are actively-synced or not is more
interesting than whether the mode switch completed. That information
can still be added if required in the future.

In presence of an iothread, the actively_synced member is now shared
between the iothread and the main thread, so turn accesses to it
atomic.

Requires to adapt the output for iotest 109.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-10-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblockjob: query driver-specific info via a new 'query' driver method
Fiona Ebner [Tue, 31 Oct 2023 13:54:29 +0000 (14:54 +0100)]
blockjob: query driver-specific info via a new 'query' driver method

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-9-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoqapi/block-core: turn BlockJobInfo into a union
Fiona Ebner [Tue, 31 Oct 2023 13:54:28 +0000 (14:54 +0100)]
qapi/block-core: turn BlockJobInfo into a union

In preparation to additionally return job-type-specific information.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20231031135431.393137-8-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoqapi/block-core: use JobType for BlockJobInfo's type
Fiona Ebner [Tue, 31 Oct 2023 13:54:27 +0000 (14:54 +0100)]
qapi/block-core: use JobType for BlockJobInfo's type

In preparation to turn BlockJobInfo into a union with @type as the
discriminator. That requires it to be an enum. Even without that
requirement, it's nicer to have an enum instead of a str here.

No functional change is intended.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20231031135431.393137-7-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agomirror: implement mirror_change method
Fiona Ebner [Tue, 31 Oct 2023 13:54:26 +0000 (14:54 +0100)]
mirror: implement mirror_change method

which allows switching the @copy-mode from 'background' to
'write-blocking'.

This is useful for management applications, so they can start out in
background mode to avoid limiting guest write speed and switch to
active mode when certain criteria are fulfilled.

In presence of an iothread, the copy_mode member is now shared between
the iothread and the main thread, so turn accesses to it atomic.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-6-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblock/mirror: determine copy_to_target only once
Fiona Ebner [Tue, 31 Oct 2023 13:54:25 +0000 (14:54 +0100)]
block/mirror: determine copy_to_target only once

In preparation to allow changing the copy_mode via QMP. When running
in an iothread, it could be that copy_mode is changed from the main
thread in between reading copy_mode in bdrv_mirror_top_pwritev() and
reading copy_mode in bdrv_mirror_top_do_write(), so they might end up
disagreeing about whether copy_to_target is true or false. Avoid that
scenario by determining copy_to_target only once and passing it to
bdrv_mirror_top_do_write() as an argument.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20231031135431.393137-5-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblock/mirror: move dirty bitmap to filter
Fiona Ebner [Tue, 31 Oct 2023 13:54:24 +0000 (14:54 +0100)]
block/mirror: move dirty bitmap to filter

In preparation to allow switching to active mode without draining.
Initialization of the bitmap in mirror_dirty_init() still happens with
the original/backing BlockDriverState, which should be fine, because
the mirror top has the same length.

Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231031135431.393137-4-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblock/mirror: set actively_synced even after the job is ready
Fiona Ebner [Tue, 31 Oct 2023 13:54:23 +0000 (14:54 +0100)]
block/mirror: set actively_synced even after the job is ready

In preparation to allow switching from background to active mode. This
ensures that setting actively_synced will not be missed when the
switch happens after the job is ready.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20231031135431.393137-3-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblockjob: introduce block-job-change QMP command
Fiona Ebner [Tue, 31 Oct 2023 13:54:22 +0000 (14:54 +0100)]
blockjob: introduce block-job-change QMP command

which will allow changing job-type-specific options after job
creation.

In the JobVerbTable, the same allow bits as for set-speed are used,
because set-speed can be considered an existing change command.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20231031135431.393137-2-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agovirtio-blk: remove batch notification BH
Stefan Hajnoczi [Wed, 13 Sep 2023 20:00:45 +0000 (16:00 -0400)]
virtio-blk: remove batch notification BH

There is a batching mechanism for virtio-blk Used Buffer Notifications
that is no longer needed because the previous commit added batching to
virtio_notify_irqfd().

Note that this mechanism was rarely used in practice because it is only
enabled when EVENT_IDX is not negotiated by the driver. Modern drivers
enable EVENT_IDX.

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230913200045.1024233-5-stefanha@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agovirtio: use defer_call() in virtio_irqfd_notify()
Stefan Hajnoczi [Wed, 13 Sep 2023 20:00:44 +0000 (16:00 -0400)]
virtio: use defer_call() in virtio_irqfd_notify()

virtio-blk and virtio-scsi invoke virtio_irqfd_notify() to send Used
Buffer Notifications from an IOThread. This involves an eventfd
write(2) syscall. Calling this repeatedly when completing multiple I/O
requests in a row is wasteful.

Use the defer_call() API to batch together virtio_irqfd_notify() calls
made during thread pool (aio=threads), Linux AIO (aio=native), and
io_uring (aio=io_uring) completion processing.

Behavior is unchanged for emulated devices that do not use
defer_call_begin()/defer_call_end() since defer_call() immediately
invokes the callback when called outside a
defer_call_begin()/defer_call_end() region.

fio rw=randread bs=4k iodepth=64 numjobs=8 IOPS increases by ~9% with a
single IOThread and 8 vCPUs. iodepth=1 decreases by ~1% but this could
be noise. Detailed performance data and configuration specifics are
available here:
https://gitlab.com/stefanha/virt-playbooks/-/tree/blk_io_plug-irqfd

This duplicates the BH that virtio-blk uses for batching. The next
commit will remove it.

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230913200045.1024233-4-stefanha@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoutil/defer-call: move defer_call() to util/
Stefan Hajnoczi [Wed, 13 Sep 2023 20:00:43 +0000 (16:00 -0400)]
util/defer-call: move defer_call() to util/

The networking subsystem may wish to use defer_call(), so move the code
to util/ where it can be reused.

As a reminder of what defer_call() does:

This API defers a function call within a defer_call_begin()/defer_call_end()
section, allowing multiple calls to batch up. This is a performance
optimization that is used in the block layer to submit several I/O requests
at once instead of individually:

  defer_call_begin(); <-- start of section
  ...
  defer_call(my_func, my_obj); <-- deferred my_func(my_obj) call
  defer_call(my_func, my_obj); <-- another
  defer_call(my_func, my_obj); <-- another
  ...
  defer_call_end(); <-- end of section, my_func(my_obj) is called once

Suggested-by: Ilya Maximets <i.maximets@ovn.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230913200045.1024233-3-stefanha@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblock: rename blk_io_plug_call() API to defer_call()
Stefan Hajnoczi [Wed, 13 Sep 2023 20:00:42 +0000 (16:00 -0400)]
block: rename blk_io_plug_call() API to defer_call()

Prepare to move the blk_io_plug_call() API out of the block layer so
that other subsystems call use this deferred call mechanism. Rename it
to defer_call() but leave the code in block/plug.c.

The next commit will move the code out of the block layer.

Suggested-by: Ilya Maximets <i.maximets@ovn.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Paul Durrant <paul@xen.org>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230913200045.1024233-2-stefanha@redhat.com>
Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agocontrib/plugins: Close file descriptor on error return
Cong Liu [Sun, 29 Oct 2023 14:50:33 +0000 (14:50 +0000)]
contrib/plugins: Close file descriptor on error return

This patch closes the file descriptor fd on error return to avoid
resource leak.

Fixes: ec7ee95db909 ("contrib/plugins: fix coverity warning in lockstep")
Signed-off-by: Cong Liu <liucong2@kylinos.cn>
Message-Id: <20231018025225.1640122-1-liucong2@kylinos.cn>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-20-alex.bennee@linaro.org>

7 months agoplugins: Remove an extra parameter
Akihiko Odaki [Sun, 29 Oct 2023 14:50:32 +0000 (14:50 +0000)]
plugins: Remove an extra parameter

copy_call() has an unused parameter so remove it.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Message-Id: <20231019101030.128431-7-akihiko.odaki@daynix.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20231029145033.592566-19-alex.bennee@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
7 months agosemihosting: fix memleak at semihosting_arg_fallback
Matheus Tavares Bernardino [Sun, 29 Oct 2023 14:50:31 +0000 (14:50 +0000)]
semihosting: fix memleak at semihosting_arg_fallback

We duplicate "cmd" as strtok may modify its argument, but we forgot
to free it later. Furthermore, add_semihosting_arg doesn't take
responsibility for this memory either (it strdup's the argument).

Signed-off-by: Matheus Tavares Bernardino <quic_mathbern@quicinc.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <03d81c56bfc3d08224e4106efca5949d8894cfa5.1697801632.git.quic_mathbern@quicinc.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-18-alex.bennee@linaro.org>

7 months agogdbstub: Check if gdb_regs is NULL
Akihiko Odaki [Sun, 29 Oct 2023 14:50:30 +0000 (14:50 +0000)]
gdbstub: Check if gdb_regs is NULL

cpu->gdb_regs may be NULL if no coprocessor is registered.

Fixes: 73c392c26b ("gdbstub: Replace gdb_regs with an array")
Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Message-Id: <20231029145033.592566-17-alex.bennee@linaro.org>
Message-Id: <20231019101030.128431-2-akihiko.odaki@daynix.com>
Tested-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
7 months agotests/docker: upgrade debian-all-test-cross to bookworm
Alex Bennée [Sun, 29 Oct 2023 14:50:29 +0000 (14:50 +0000)]
tests/docker: upgrade debian-all-test-cross to bookworm

This requires a few more tweaks than usual as:

  - the default sources format has changed
  - bring in python3-tomli from the repos
  - split base install from cross compilers
  - also include libclang-rt-dev for sanitiser builds

Acked-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-16-alex.bennee@linaro.org>

7 months agotests/docker: use debian-all-test-cross for sparc64
Alex Bennée [Sun, 29 Oct 2023 14:50:28 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for sparc64

Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-15-alex.bennee@linaro.org>

7 months agotests/docker: use debian-all-test-cross for riscv64
Alex Bennée [Sun, 29 Oct 2023 14:50:27 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for riscv64

Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-14-alex.bennee@linaro.org>

7 months agotests/docker: use debian-all-test-cross for mips
Alex Bennée [Sun, 29 Oct 2023 14:50:26 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for mips

Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-13-alex.bennee@linaro.org>

7 months agotests/docker: use debian-all-test-cross for mips64
Alex Bennée [Sun, 29 Oct 2023 14:50:25 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for mips64

Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-12-alex.bennee@linaro.org>

7 months agotests/docker: use debian-all-test-cross for m68k
Alex Bennée [Sun, 29 Oct 2023 14:50:24 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for m68k

Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-11-alex.bennee@linaro.org>

7 months agotests/docker: use debian-all-test-cross for hppa
Alex Bennée [Sun, 29 Oct 2023 14:50:23 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for hppa

Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-10-alex.bennee@linaro.org>

7 months agotests/docker: use debian-all-test-cross for power
Alex Bennée [Sun, 29 Oct 2023 14:50:22 +0000 (14:50 +0000)]
tests/docker: use debian-all-test-cross for power

Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-9-alex.bennee@linaro.org>

7 months agotests/docker: move sh4 to use debian-legacy-test-cross
Alex Bennée [Sun, 29 Oct 2023 14:50:21 +0000 (14:50 +0000)]
tests/docker: move sh4 to use debian-legacy-test-cross

sh4 is another target which doesn't work with bookworm compilers. To
keep on buster move across to the debian-legacy-test-cross image and
update accordingly.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20231030135715.800164-1-alex.bennee@linaro.org>

7 months agotests/docker: use debian-legacy-test-cross for alpha
Alex Bennée [Sun, 29 Oct 2023 14:50:20 +0000 (14:50 +0000)]
tests/docker: use debian-legacy-test-cross for alpha

Maintaining two sets of containers for test building is silly. While
it makes sense for the QEMU cross-compile targets to have their own
fat containers built by lcitool we might as well merge the other
random debian based compilers into the same one used on gitlab.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-7-alex.bennee@linaro.org>

7 months agogitlab: add build-loongarch to matrix
Alex Bennée [Sun, 29 Oct 2023 14:50:19 +0000 (14:50 +0000)]
gitlab: add build-loongarch to matrix

We have the compiler and with a few updates a container that can build
QEMU so we should at least run the check-tcg smoke tests.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-6-alex.bennee@linaro.org>

7 months agogitlab: clean-up build-soft-softmmu job
Alex Bennée [Sun, 29 Oct 2023 14:50:18 +0000 (14:50 +0000)]
gitlab: clean-up build-soft-softmmu job

Having dropped alpha we also now drop xtensa as we don't have the
compiler in this image. It's not all doom and gloom though as a number
of other targets have gained softmmu TCG tests so we can add them. We
will take care of the other targets with their own containers in
future commits.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-5-alex.bennee@linaro.org>

7 months agogitlab: split alpha testing into a legacy container
Alex Bennée [Sun, 29 Oct 2023 14:50:17 +0000 (14:50 +0000)]
gitlab: split alpha testing into a legacy container

The current bookworm compiler doesn't build the static binaries due to
bug #1054412 and it might be awhile before it gets fixed. The problem
of keeping older architecture compilers running isn't going to go away
so lets prepare the ground. Create a legacy container and move some
tests around so the others can get upgraded.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-4-alex.bennee@linaro.org>

7 months agotests/tcg: Add -fno-stack-protector
Akihiko Odaki [Sun, 29 Oct 2023 14:50:16 +0000 (14:50 +0000)]
tests/tcg: Add -fno-stack-protector

A build of GCC 13.2 will have stack protector enabled by default if it
was configured with --enable-default-ssp option. For such a compiler,
it is necessary to explicitly disable stack protector when linking
without standard libraries.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Message-Id: <20230731091042.139159-3-akihiko.odaki@daynix.com>
[AJB: fix comment string typo]
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-3-alex.bennee@linaro.org>

7 months agotests/vm/openbsd: Use the system dtc package
Thomas Huth [Sun, 29 Oct 2023 14:50:15 +0000 (14:50 +0000)]
tests/vm/openbsd: Use the system dtc package

We can use the pre-packaged libfdt from the dtc package to avoid
that we have to compile this code each time again and again.

While we're at it, the "--python=python3" does not seemt to be
necessary anymore, so we can drop it.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20231016154049.37147-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231029145033.592566-2-alex.bennee@linaro.org>

7 months agoblockdev: mirror: avoid potential deadlock when using iothread
Fiona Ebner [Thu, 19 Oct 2023 13:19:36 +0000 (15:19 +0200)]
blockdev: mirror: avoid potential deadlock when using iothread

The bdrv_getlength() function is a generated co-wrapper and uses
AIO_WAIT_WHILE() to wait for the spawned coroutine. AIO_WAIT_WHILE()
expects the lock to be acquired exactly once.

Fix a case where it may be acquired twice. This can happen when the
source node is explicitly specified as the @replaces parameter or if the
source node is a filter node.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231019131936.414246-4-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblock: avoid potential deadlock during bdrv_graph_wrlock() in bdrv_close()
Fiona Ebner [Thu, 19 Oct 2023 13:19:35 +0000 (15:19 +0200)]
block: avoid potential deadlock during bdrv_graph_wrlock() in bdrv_close()

by passing the BlockDriverState along, so the held AioContext can be
dropped before polling. See commit 31b2ddfea3 ("graph-lock: Unlock the
AioContext while polling") which introduced this functionality for
more information.

The only way to reach bdrv_close() is via bdrv_unref() and for calling
that the BlockDriverState's AioContext lock is supposed to be held.

Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231019131936.414246-3-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblockjob: drop AioContext lock before calling bdrv_graph_wrlock()
Fiona Ebner [Thu, 19 Oct 2023 13:19:34 +0000 (15:19 +0200)]
blockjob: drop AioContext lock before calling bdrv_graph_wrlock()

Same rationale as in 31b2ddfea3 ("graph-lock: Unlock the AioContext
while polling"). Otherwise, a deadlock can happen.

The alternative would be to pass a BlockDriverState along to
bdrv_graph_wrlock(), but there is no BlockDriverState readily
available and it's also better conceptually, because the lock is held
for the job.

The function is always called with the job's AioContext lock held, via
one of the .abort, .clean, .free or .prepare job driver functions.
Thus, it's safe to drop it.

While mirror_exit_common() does hold a second AioContext lock while
calling block_job_remove_all_bdrv(), that is for the main thread's
AioContext and does not need to be dropped (bdrv_graph_wrlock(bs) also
skips dropping the lock if bdrv_get_aio_context(bs) ==
qemu_get_aio_context()).

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Fiona Ebner <f.ebner@proxmox.com>
Message-ID: <20231019131936.414246-2-f.ebner@proxmox.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoiotests: Test media change with iothreads
Kevin Wolf [Fri, 13 Oct 2023 15:33:02 +0000 (17:33 +0200)]
iotests: Test media change with iothreads

iotests case 118 already tests all relevant operations for media change
with multiple devices, however never with iothreads. This changes the
test so that the virtio-scsi tests run with an iothread.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-ID: <20231013153302.39234-3-kwolf@redhat.com>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoblock: Fix locking in media change monitor commands
Kevin Wolf [Fri, 13 Oct 2023 15:33:01 +0000 (17:33 +0200)]
block: Fix locking in media change monitor commands

blk_insert_bs() requires that the caller holds the AioContext lock for
the node to be inserted. Since commit c066e808e11, neglecting to do so
causes a crash when the child has to be moved to a different AioContext
to attach it to the BlockBackend.

This fixes qmp_blockdev_insert_anon_medium(), which is called for the
QMP commands 'blockdev-insert-medium' and 'blockdev-change-medium', to
correctly take the lock.

Cc: qemu-stable@nongnu.org
Fixes: https://issues.redhat.com/browse/RHEL-3922
Fixes: c066e808e11a5c181b625537b6c78e0de27a4801
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-ID: <20231013153302.39234-2-kwolf@redhat.com>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoiotests: add tests for "qemu-img rebase" with compression
Andrey Drobyshev [Tue, 19 Sep 2023 16:58:04 +0000 (19:58 +0300)]
iotests: add tests for "qemu-img rebase" with compression

The test cases considered so far:

314 (new test suite):

1. Check that compression mode isn't compatible with "-f raw" (raw
   format doesn't support compression).
2. Check that rebasing an image onto no backing file preserves the data
   and writes the copied clusters actually compressed.
3. Same as 2, but with a raw backing file (i.e. the clusters copied from the
   backing are originally uncompressed -- we check they end up compressed
   after being merged).
4. Remove a single delta from a backing chain, perform the same checks
   as in 2.
5. Check that even when backing and overlay are initially uncompressed,
   copied clusters end up compressed when rebase with compression is
   performed.

271:

1. Check that when target image has subclusters, rebase with compression
   will make an entire cluster containing the written subcluster
   compressed.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-9-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoqemu-img: add compression option to rebase subcommand
Andrey Drobyshev [Tue, 19 Sep 2023 16:58:03 +0000 (19:58 +0300)]
qemu-img: add compression option to rebase subcommand

If we rebase an image whose backing file has compressed clusters, we
might end up wasting disk space since the copied clusters are now
uncompressed.  In order to have better control over this, let's add
"--compress" option to the "qemu-img rebase" command.

Note that this option affects only the clusters which are actually being
copied from the original backing file.  The clusters which were
uncompressed in the target image will remain so.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-8-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoiotests/{024, 271}: add testcases for qemu-img rebase
Andrey Drobyshev [Tue, 19 Sep 2023 16:58:02 +0000 (19:58 +0300)]
iotests/{024, 271}: add testcases for qemu-img rebase

As the previous commit changes the logic of "qemu-img rebase" (it's using
write alignment now), let's add a couple more test cases which would
ensure it works correctly.  In particular, the following scenarios:

024: add test case for rebase within one backing chain when the overlay
     cluster size > backings cluster size;
271: add test case for rebase images that contain subclusters.  Check
     that no extra allocations are being made.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-7-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoqemu-img: rebase: avoid unnecessary COW operations
Andrey Drobyshev [Tue, 19 Sep 2023 16:58:01 +0000 (19:58 +0300)]
qemu-img: rebase: avoid unnecessary COW operations

When rebasing an image from one backing file to another, we need to
compare data from old and new backings.  If the diff between that data
happens to be unaligned to the target cluster size, we might end up
doing partial writes, which would lead to copy-on-write and additional IO.

Consider the following simple case (virtual_size == cluster_size == 64K):

base <-- inc1 <-- inc2

qemu-io -c "write -P 0xaa 0 32K" base.qcow2
qemu-io -c "write -P 0xcc 32K 32K" base.qcow2
qemu-io -c "write -P 0xbb 0 32K" inc1.qcow2
qemu-io -c "write -P 0xcc 32K 32K" inc1.qcow2
qemu-img rebase -f qcow2 -b base.qcow2 -F qcow2 inc2.qcow2

While doing rebase, we'll write a half of the cluster to inc2, and block
layer will have to read the 2nd half of the same cluster from the base image
inc1 while doing this write operation, although the whole cluster is already
read earlier to perform data comparison.

In order to avoid these unnecessary IO cycles, let's make sure every
write request is aligned to the overlay subcluster boundaries.  Using
subcluster size is universal as for the images which don't have them
this size equals to the cluster size. so in any case we end up aligning
to the smallest unit of allocation.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Message-ID: <20230919165804.439110-6-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoqemu-img: add chunk size parameter to compare_buffers()
Andrey Drobyshev [Tue, 19 Sep 2023 16:58:00 +0000 (19:58 +0300)]
qemu-img: add chunk size parameter to compare_buffers()

Add @chsize param to the function which, if non-zero, would represent
the chunk size to be used for comparison.  If it's zero, then
BDRV_SECTOR_SIZE is used as default chunk size, which is the previous
behaviour.

In particular, we're going to use this param in img_rebase() to make the
write requests aligned to a predefined alignment value.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-5-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoqemu-img: rebase: use backing files' BlockBackend for buffer alignment
Andrey Drobyshev [Tue, 19 Sep 2023 16:57:59 +0000 (19:57 +0300)]
qemu-img: rebase: use backing files' BlockBackend for buffer alignment

Since commit bb1c05973cf ("qemu-img: Use qemu_blockalign"), buffers for
the data read from the old and new backing files are aligned using
BlockDriverState (or BlockBackend later on) referring to the target image.
However, this isn't quite right, because buf_new is only being used for
reading from the new backing, while buf_old is being used for both reading
from the old backing and writing to the target.  Let's take that into account
and use more appropriate values as alignments.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Message-ID: <20230919165804.439110-4-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
7 months agoqemu-iotests: 024: add rebasing test case for overlay_size > backing_size
Andrey Drobyshev [Tue, 19 Sep 2023 16:57:58 +0000 (19:57 +0300)]
qemu-iotests: 024: add rebasing test case for overlay_size > backing_size

Before previous commit, rebase was getting infitely stuck in case of
rebasing within the same backing chain and when overlay_size > backing_size.
Let's add this case to the rebasing test 024 to make sure it doesn't
break again.

Signed-off-by: Andrey Drobyshev <andrey.drobyshev@virtuozzo.com>
Reviewed-by: Denis V. Lunev <den@openvz.org>
Reviewed-by: Hanna Czenczek <hreitz@redhat.com>
Message-ID: <20230919165804.439110-3-andrey.drobyshev@virtuozzo.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>