]> git.proxmox.com Git - mirror_qemu.git/log
mirror_qemu.git
3 years agoMerge remote-tracking branch 'remotes/bonzini-gitlab/tags/for-upstream' into staging
Peter Maydell [Mon, 12 Oct 2020 21:48:45 +0000 (22:48 +0100)]
Merge remote-tracking branch 'remotes/bonzini-gitlab/tags/for-upstream' into staging

* qtest documentation improvements (Eduardo, myself)
* libqtest event buffering (Maxim)
* use RCU for list of children of a bus (Maxim)
* move more files to softmmu/ (myself)
* meson.build cleanups, qemu-storage-daemon fix (Philippe)

# gpg: Signature made Mon 12 Oct 2020 16:55:19 BST
# gpg:                using RSA key F13338574B662389866C7682BFFBD25F78C7AE83
# gpg:                issuer "pbonzini@redhat.com"
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>" [full]
# gpg:                 aka "Paolo Bonzini <pbonzini@redhat.com>" [full]
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4  E2F7 7E15 100C CD36 69B1
#      Subkey fingerprint: F133 3857 4B66 2389 866C  7682 BFFB D25F 78C7 AE83

* remotes/bonzini-gitlab/tags/for-upstream: (38 commits)
  meson: identify more sections of meson.build
  scsi/scsi_bus: fix races in REPORT LUNS
  virtio-scsi: use scsi_device_get
  scsi/scsi_bus: Add scsi_device_get
  scsi/scsi-bus: scsi_device_find: don't return unrealized devices
  device-core: use atomic_set on .realized property
  scsi: switch to bus->check_address
  device-core: use RCU for list of children of a bus
  device_core: use drain_call_rcu in in qmp_device_add
  scsi/scsi_bus: switch search direction in scsi_device_find
  qdev: add "check if address free" callback for buses
  qemu-iotests, qtest: rewrite test 067 as a qtest
  qtest: check that drives are really appearing and disappearing
  qtest: switch users back to qtest_qmp_receive
  device-plug-test: use qtest_qmp to send the device_del command
  qtest: remove qtest_qmp_receive_success
  qtest: Reintroduce qtest_qmp_receive with QMP event buffering
  qtest: rename qtest_qmp_receive to qtest_qmp_receive_dict
  meson.build: Re-enable KVM support for MIPS
  build-sys: fix git version from -version
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
3 years agomeson: identify more sections of meson.build
Paolo Bonzini [Wed, 7 Oct 2020 15:01:51 +0000 (11:01 -0400)]
meson: identify more sections of meson.build

Add more headers that clarify code organization.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoscsi/scsi_bus: fix races in REPORT LUNS
Maxim Levitsky [Tue, 6 Oct 2020 12:39:04 +0000 (15:39 +0300)]
scsi/scsi_bus: fix races in REPORT LUNS

Currently scsi_target_emulate_report_luns iterates over the child device list
twice, and there is no guarantee that this list is the same in both iterations.

The reason for iterating twice is that the first iteration calculates
how much memory to allocate.  However if we use a dynamic array we can
avoid iterating twice, and therefore we avoid this race.

Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1866707
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-10-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-14-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agovirtio-scsi: use scsi_device_get
Maxim Levitsky [Tue, 6 Oct 2020 12:39:03 +0000 (15:39 +0300)]
virtio-scsi: use scsi_device_get

This will help us to avoid the scsi device disappearing
after we took a reference to it.

It doesn't by itself forbid case when we try to access
an unrealized device

Suggested-by: Stefan Hajnoczi <stefanha@gmail.com>
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-9-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-13-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoscsi/scsi_bus: Add scsi_device_get
Maxim Levitsky [Tue, 6 Oct 2020 12:39:02 +0000 (15:39 +0300)]
scsi/scsi_bus: Add scsi_device_get

Add scsi_device_get which finds the scsi device
and takes a reference to it.

Suggested-by: Stefan Hajnoczi <stefanha@gmail.com>
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20200913160259.32145-8-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-12-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoscsi/scsi-bus: scsi_device_find: don't return unrealized devices
Paolo Bonzini [Tue, 6 Oct 2020 12:39:01 +0000 (15:39 +0300)]
scsi/scsi-bus: scsi_device_find: don't return unrealized devices

The device core first places a device on the bus and then realizes it.
Make scsi_device_find avoid returing such devices to avoid
races in drivers that use an iothread (currently virtio-scsi)

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1812399

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-7-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-11-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agodevice-core: use atomic_set on .realized property
Maxim Levitsky [Tue, 6 Oct 2020 12:39:00 +0000 (15:39 +0300)]
device-core: use atomic_set on .realized property

Some code might race with placement of new devices on a bus.
We currently first place a (unrealized) device on the bus
and then realize it.

As a workaround, users that scan the child device list, can
check the realized property to see if it is safe to access such a device.
Use an atomic write here too to aid with this.

A separate discussion is what to do with devices that are unrealized:
It looks like for this case we only call the hotplug handler's unplug
callback and its up to it to unrealize the device.
An atomic operation doesn't cause harm for this code path though.

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-6-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-10-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoscsi: switch to bus->check_address
Paolo Bonzini [Tue, 6 Oct 2020 12:38:56 +0000 (15:38 +0300)]
scsi: switch to bus->check_address

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-6-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agodevice-core: use RCU for list of children of a bus
Maxim Levitsky [Tue, 6 Oct 2020 12:38:59 +0000 (15:38 +0300)]
device-core: use RCU for list of children of a bus

This fixes the race between device emulation code that tries to find
a child device to dispatch the request to (e.g a scsi disk),
and hotplug of a new device to that bus.

Note that this doesn't convert all the readers of the list
but only these that might go over that list without BQL held.

This is a very small first step to make this code thread safe.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-5-mlevitsk@redhat.com>
[Use RCU_READ_LOCK_GUARD in more places, adjust testcase now that
 the delay in DEVICE_DELETED due to RCU is more consistent. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-9-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agodevice_core: use drain_call_rcu in in qmp_device_add
Maxim Levitsky [Tue, 6 Oct 2020 12:38:58 +0000 (14:38 +0200)]
device_core: use drain_call_rcu in in qmp_device_add

Soon, a device removal might only happen on RCU callback execution.
This is okay for device-del which provides a DEVICE_DELETED event,
but not for the failure case of device-add.  To avoid changing
monitor semantics, just drain all pending RCU callbacks on error.

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Suggested-by: Stefan Hajnoczi <stefanha@gmail.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-4-mlevitsk@redhat.com>
[Don't use it in qmp_device_del. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoscsi/scsi_bus: switch search direction in scsi_device_find
Maxim Levitsky [Tue, 6 Oct 2020 12:38:57 +0000 (14:38 +0200)]
scsi/scsi_bus: switch search direction in scsi_device_find

This change will allow us to convert the bus children list to RCU,
while not changing the logic of this function

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-2-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqdev: add "check if address free" callback for buses
Paolo Bonzini [Tue, 6 Oct 2020 12:38:55 +0000 (15:38 +0300)]
qdev: add "check if address free" callback for buses

Check if an address is free on the bus before plugging in the
device.  This makes it possible to do the check without any
side effects, and to detect the problem early without having
to do it in the realize callback.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-5-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqemu-iotests, qtest: rewrite test 067 as a qtest
Paolo Bonzini [Wed, 7 Oct 2020 10:43:03 +0000 (06:43 -0400)]
qemu-iotests, qtest: rewrite test 067 as a qtest

Test 067 from qemu-iotests is executing QMP commands to hotplug
and hot-unplug disks, devices and blockdevs.  Because the power
of the text-based test harness is limited, it is actually limiting
the checks that it does, for example by skipping DEVICE_DELETED
events.

tests/qtest already has a similar test, drive_del-test.c.
We can merge them, and even reuse some of the existing code in
drive_del-test.c.  This will improve the quality of the test by
covering DEVICE_DELETED events and testing multiple architectures
(therefore covering multiple PCI hotplug mechanisms as well as s390x
virtio-ccw).

The only difference is that the new test will always use null-co:// for
the medium rather than qcow2 or raw, but this should be irrelevant for
what the test is covering.  For example there are no "qemu-img check"
runs in 067 that would check that the file is properly closed.

The new tests requires PCI hot-plug support, so drive_del-test
is moved from qemu-system-ppc to qemu-system-ppc64.

Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqtest: check that drives are really appearing and disappearing
Paolo Bonzini [Wed, 7 Oct 2020 09:50:22 +0000 (05:50 -0400)]
qtest: check that drives are really appearing and disappearing

Do not just trust the HMP commands to create and delete the drive, use
query-block to check that this is actually the case.

Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqtest: switch users back to qtest_qmp_receive
Maxim Levitsky [Tue, 6 Oct 2020 12:38:53 +0000 (15:38 +0300)]
qtest: switch users back to qtest_qmp_receive

Let test use the new functionality for buffering events.
The only remaining users of qtest_qmp_receive_dict are tests
that fuzz the QMP protocol.

Tested with 'make check-qtest'.

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20201006123904.610658-4-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agodevice-plug-test: use qtest_qmp to send the device_del command
Paolo Bonzini [Wed, 7 Oct 2020 11:37:41 +0000 (07:37 -0400)]
device-plug-test: use qtest_qmp to send the device_del command

Simplify the code now that events are buffered.  There is no need
anymore to separate sending the command and retrieving the response.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqtest: remove qtest_qmp_receive_success
Maxim Levitsky [Tue, 6 Oct 2020 12:59:32 +0000 (08:59 -0400)]
qtest: remove qtest_qmp_receive_success

The purpose of qtest_qmp_receive_success was mostly to process events
that arrived between the issueing of a command and the "return"
line from QMP.  This is now handled by the buffering of events
that libqtest performs automatically.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
3 years agoqtest: Reintroduce qtest_qmp_receive with QMP event buffering
Maxim Levitsky [Tue, 6 Oct 2020 12:38:53 +0000 (15:38 +0300)]
qtest: Reintroduce qtest_qmp_receive with QMP event buffering

The new qtest_qmp_receive buffers all the received qmp events, allowing
qtest_qmp_eventwait_ref to return them.

This is intended to solve the race in regard to ordering of qmp events
vs qmp responses, as soon as the callers start using the new interface.

In addition to that, define qtest_qmp_event_ref a function which only scans
the buffer that qtest_qmp_receive stores the events to.  This is intended
for callers that are only interested in events that were received during
the last call to the qtest_qmp_receive.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20201006123904.610658-3-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqtest: rename qtest_qmp_receive to qtest_qmp_receive_dict
Maxim Levitsky [Tue, 6 Oct 2020 12:38:52 +0000 (14:38 +0200)]
qtest: rename qtest_qmp_receive to qtest_qmp_receive_dict

In the next patch a new version of qtest_qmp_receive will be
reintroduced that will buffer received qmp events for later
consumption in qtest_qmp_eventwait_ref

No functional change intended.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agomeson.build: Re-enable KVM support for MIPS
Huacai Chen [Wed, 7 Oct 2020 08:39:28 +0000 (16:39 +0800)]
meson.build: Re-enable KVM support for MIPS

After converting from configure to meson, KVM support is lost for MIPS,
so re-enable it in meson.build.

Fixes: fdb75aeff7c212e1afaaa3a43 ("configure: remove target configuration")
Fixes: 8a19980e3fc42239aae054bc9 ("configure: move accelerator logic to meson")
Cc: aolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Huacai Chen <chenhc@lemote.com>
Message-Id: <1602059975-10115-3-git-send-email-chenhc@lemote.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agobuild-sys: fix git version from -version
Marc-André Lureau [Tue, 29 Sep 2020 14:36:54 +0000 (18:36 +0400)]
build-sys: fix git version from -version

Typo introduced with the script.

Fixes: 2c273f32d3 ("meson: generate qemu-version.h")
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reported-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Tested-by: Laszlo Ersek <lersek@redhat.com>
Message-Id: <20200929143654.518157-1-marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agodocs/devel: update instruction on how to add new unit tests
Paolo Bonzini [Tue, 6 Oct 2020 12:49:55 +0000 (08:49 -0400)]
docs/devel: update instruction on how to add new unit tests

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqtest: unify extra_qtest_srcs and extra_qtest_deps
Paolo Bonzini [Tue, 6 Oct 2020 12:44:00 +0000 (08:44 -0400)]
qtest: unify extra_qtest_srcs and extra_qtest_deps

Currently the extra sources and extra dependencies of qtests are held
in two separate dictionaries.  Use the same trick as tests/meson.build
to combine them into one.  This will make it easier to update the
documentation for unit tests and qtests.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agodocs/devel/qtest: Include libqtest API reference
Eduardo Habkost [Mon, 5 Oct 2020 20:52:28 +0000 (16:52 -0400)]
docs/devel/qtest: Include libqtest API reference

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201005205228.697463-4-ehabkost@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agodocs/devel/qtest: Include protocol spec in document
Eduardo Habkost [Mon, 5 Oct 2020 20:52:27 +0000 (16:52 -0400)]
docs/devel/qtest: Include protocol spec in document

Include the QTest Protocol doc string in docs/devel/qtest.rst,
after converting it to use Sphinx syntax.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201005205228.697463-3-ehabkost@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agodocs: Move QTest documentation to its own document
Eduardo Habkost [Mon, 5 Oct 2020 20:52:26 +0000 (16:52 -0400)]
docs: Move QTest documentation to its own document

The qtest and libqtest doc comments will be parsed to generate
API documentation, so move QTest documentation to its own
document where the API and format documentation and will be
included.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201005205228.697463-2-ehabkost@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqom: fix objects with improper parent type
Paolo Bonzini [Wed, 7 Oct 2020 16:23:56 +0000 (12:23 -0400)]
qom: fix objects with improper parent type

Some objects accidentally inherit ObjectClass instead of Object.
They compile silently but may crash after downcasting.

In this patch, we introduce a coccinelle script to find broken
declarations and fix them manually with proper base type.

Signed-off-by: Sergey Nizovtsev <snizovtsev@gmail.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoexec: split out non-softmmu-specific parts
Paolo Bonzini [Tue, 6 Oct 2020 07:05:29 +0000 (09:05 +0200)]
exec: split out non-softmmu-specific parts

Over the years, most parts of exec.c that were not specific to softmmu
have been moved to accel/tcg; what's left is mostly the low-level part
of the memory API, which includes RAMBlock and AddressSpaceDispatch.
However exec.c also hosts 4-500 lines of code for the target specific
parts of the CPU QOM object, plus a few functions for user-mode
emulation that do not have a better place (they are not TCG-specific so
accel/tcg/user-exec.c is not a good place either).

Move these parts to a new file, so that exec.c can be moved to
softmmu/physmem.c.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agosoftmmu: move more files to softmmu/
Paolo Bonzini [Tue, 6 Oct 2020 07:01:22 +0000 (09:01 +0200)]
softmmu: move more files to softmmu/

Keep most softmmu_ss files into the system-emulation-specific
directory.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agohw/nvram: Always register FW_CFG_DATA_GENERATOR_INTERFACE
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 11:19:09 +0000 (13:19 +0200)]
hw/nvram: Always register FW_CFG_DATA_GENERATOR_INTERFACE

While the FW_CFG_DATA_GENERATOR_INTERFACE is only consumed
by a device only available using system-mode (fw_cfg), it is
implemented by a crypto component (tls-cipher-suites) which
is always available when crypto is used.

Commit 69699f3055 introduced the following error in the
qemu-storage-daemon binary:

  $ echo -e \
    '{"execute": "qmp_capabilities"}\r\n{"execute": "qom-list-types"}\r\n{"execute": "quit"}\r\n' \
    | storage-daemon/qemu-storage-daemon --chardev stdio,id=qmp0  --monitor qmp0
  {"QMP": {"version": {"qemu": {"micro": 50, "minor": 1, "major": 5}, "package": ""}, "capabilities": ["oob"]}}
  {"return": {}}
  missing interface 'fw_cfg-data-generator' for object 'tls-creds'
  Aborted (core dumped)

Since QOM dependencies are resolved at runtime, this issue
could not be triggered at linktime, and we don't have test
running the qemu-storage-daemon binary.

Fix by always registering the QOM interface.

Reported-by: Kevin Wolf <kwolf@redhat.com>
Fixes: 69699f3055 ("crypto/tls-cipher-suites: Produce fw_cfg consumable blob")
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006111909.2302081-2-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoqom: Move the creation of the library to the main meson.build
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:56:02 +0000 (14:56 +0200)]
qom: Move the creation of the library to the main meson.build

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-10-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoauthz: Move the creation of the library to the main meson.build
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:56:01 +0000 (14:56 +0200)]
authz: Move the creation of the library to the main meson.build

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-9-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agocrypto: Move the creation of the library to the main meson.build
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:56:00 +0000 (14:56 +0200)]
crypto: Move the creation of the library to the main meson.build

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-8-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoio: Move the creation of the library to the main meson.build
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:55:59 +0000 (14:55 +0200)]
io: Move the creation of the library to the main meson.build

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-7-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agomigration: Move the creation of the library to the main meson.build
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:55:58 +0000 (14:55 +0200)]
migration: Move the creation of the library to the main meson.build

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-6-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agochardev: Move the creation of the library to the main meson.build
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:55:57 +0000 (14:55 +0200)]
chardev: Move the creation of the library to the main meson.build

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-5-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agohw/core: Move the creation of the library to the main meson.build
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:55:56 +0000 (14:55 +0200)]
hw/core: Move the creation of the library to the main meson.build

Be consistent creating all the libraries in the main meson.build file.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-4-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agomeson.build: Sort sourcesets alphabetically
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:55:55 +0000 (14:55 +0200)]
meson.build: Sort sourcesets alphabetically

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-3-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agomeson.build: Add comments to clarify code organization
Philippe Mathieu-Daudé [Tue, 6 Oct 2020 12:55:54 +0000 (14:55 +0200)]
meson.build: Add comments to clarify code organization

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-2-philmd@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
3 years agoMerge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20201012a' into...
Peter Maydell [Mon, 12 Oct 2020 12:12:04 +0000 (13:12 +0100)]
Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20201012a' into staging

v3 Migration+ virtiofsd pull 2020-10-12

V3
  Remove the postcopy recovery changes

Migration:
  Dirtyrate measurement API cleanup

Virtiofsd:
  Missing qemu_init_exec_dir call
  Support for setting the group on socket creation
  Stop a gcc warning
  Avoid tempdir in sandboxing

# gpg: Signature made Mon 12 Oct 2020 12:43:30 BST
# gpg:                using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A  9FA9 0516 331E BC5B FDE7

* remotes/dgilbert/tags/pull-migration-20201012a:
  migration/dirtyrate: present dirty rate only when querying the rate has completed
  migration/dirtyrate: record start_time and calc_time while at the measuring state
  virtiofsd: avoid /proc/self/fd tempdir
  virtiofsd: Call qemu_init_exec_dir
  tools/virtiofsd: add support for --socket-group
  virtiofsd: Silence gcc warning

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
3 years agomigration/dirtyrate: present dirty rate only when querying the rate has completed
Chuan Zheng [Tue, 29 Sep 2020 03:42:18 +0000 (11:42 +0800)]
migration/dirtyrate: present dirty rate only when querying the rate has completed

Make dirty_rate field optional, present dirty rate only when querying
the rate has completed.
The qmp results is shown as follow:
@unstarted:
{"return":{"status":"unstarted","start-time":0,"calc-time":0},"id":"libvirt-12"}
@measuring:
{"return":{"status":"measuring","start-time":102931,"calc-time":1},"id":"libvirt-85"}
@measured:
{"return":{"status":"measured","dirty-rate":4,"start-time":150146,"calc-time":1},"id":"libvirt-15"}

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Reviewed-by: David Edmondson <david.edmondson@oracle.com>
Message-Id: <1601350938-128320-3-git-send-email-zhengchuan@huawei.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
3 years agomigration/dirtyrate: record start_time and calc_time while at the measuring state
Chuan Zheng [Tue, 29 Sep 2020 03:42:17 +0000 (11:42 +0800)]
migration/dirtyrate: record start_time and calc_time while at the measuring state

Querying could include both the start-time and the calc-time while at the measuring
state, allowing a caller to determine when they should expect to come back looking
for a result.

Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Message-Id: <1601350938-128320-2-git-send-email-zhengchuan@huawei.com>
Reviewed-by: David Edmondson <david.edmondson@oracle.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
3 years agovirtiofsd: avoid /proc/self/fd tempdir
Stefan Hajnoczi [Tue, 6 Oct 2020 09:58:26 +0000 (10:58 +0100)]
virtiofsd: avoid /proc/self/fd tempdir

In order to prevent /proc/self/fd escapes a temporary directory is
created where /proc/self/fd is bind-mounted. This doesn't work on
read-only file systems.

Avoid the temporary directory by bind-mounting /proc/self/fd over /proc.
This does not affect other processes since we remounted / with MS_REC |
MS_SLAVE. /proc must exist and virtiofsd does not use it so it's safe to
do this.

Path traversal can be tested with the following function:

  static void test_proc_fd_escape(struct lo_data *lo)
  {
      int fd;
      int level = 0;
      ino_t last_ino = 0;

      fd = lo->proc_self_fd;
      for (;;) {
          struct stat st;

          if (fstat(fd, &st) != 0) {
              perror("fstat");
              return;
          }
          if (last_ino && st.st_ino == last_ino) {
              fprintf(stderr, "inode number unchanged, stopping\n");
              return;
          }
          last_ino = st.st_ino;

          fprintf(stderr, "Level %d dev %lu ino %lu\n", level,
                  (unsigned long)st.st_dev,
                  (unsigned long)last_ino);
          fd = openat(fd, "..", O_PATH | O_DIRECTORY | O_NOFOLLOW);
          level++;
      }
  }

Before and after this patch only Level 0 is displayed. Without
/proc/self/fd bind-mount protection it is possible to traverse parent
directories.

Fixes: 397ae982f4df4 ("virtiofsd: jail lo->proc_self_fd")
Cc: Miklos Szeredi <mszeredi@redhat.com>
Cc: Jens Freimann <jfreimann@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20201006095826.59813-1-stefanha@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Tested-by: Jens Freimann <jfreimann@redhat.com>
Reviewed-by: Jens Freimann <jfreimann@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
3 years agovirtiofsd: Call qemu_init_exec_dir
Dr. David Alan Gilbert [Fri, 2 Oct 2020 12:40:15 +0000 (13:40 +0100)]
virtiofsd: Call qemu_init_exec_dir

Since fcb4f59c879 qemu_get_local_state_pathname relies on the
init_exec_dir, and virtiofsd asserts because we never set it.
Set it.

Reported-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20201002124015.44820-1-dgilbert@redhat.com>
Tested-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
3 years agotools/virtiofsd: add support for --socket-group
Alex Bennée [Fri, 25 Sep 2020 12:51:29 +0000 (13:51 +0100)]
tools/virtiofsd: add support for --socket-group

If you like running QEMU as a normal user (very common for TCG runs)
but you have to run virtiofsd as a root user you run into connection
problems. Adding support for an optional --socket-group allows the
users to keep using the command line.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200925125147.26943-2-alex.bennee@linaro.org>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
  dgilbert: Split long line

3 years agovirtiofsd: Silence gcc warning
Dr. David Alan Gilbert [Thu, 27 Aug 2020 15:36:52 +0000 (16:36 +0100)]
virtiofsd: Silence gcc warning

Gcc worries fd might be used unset, in reality it's always set if
fi is set, and only used if fi is set so it's safe.  Initialise it to -1
just to keep gcc happy for now.

Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20200827153657.111098-2-dgilbert@redhat.com>
Reviewed-by: Ján Tomko <jtomko@redhat.com>
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
3 years agoMerge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2020-10-10' into staging
Peter Maydell [Mon, 12 Oct 2020 10:29:41 +0000 (11:29 +0100)]
Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2020-10-10' into staging

QAPI patches patches for 2020-10-10

# gpg: Signature made Sat 10 Oct 2020 10:43:14 BST
# gpg:                using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg:                issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg:                 aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867  4E5F 3870 B400 EB91 8653

* remotes/armbru/tags/pull-qapi-2020-10-10: (34 commits)
  qapi/visit.py: add type hint annotations
  qapi/visit.py: remove unused parameters from gen_visit_object
  qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
  qapi/types.py: remove one-letter variables
  qapi/types.py: add type hint annotations
  qapi/gen.py: delint with pylint
  qapi/gen.py: update write() to be more idiomatic
  qapi/gen.py: Remove unused parameter
  qapi/gen.py: add type hint annotations
  qapi/gen: Make _is_user_module() return bool
  qapi/source.py: delint with pylint
  qapi/source.py: add type hint annotations
  qapi/commands.py: add type hint annotations
  qapi/commands.py: Don't re-bind to variable of different type
  qapi/events.py: Move comments into docstrings
  qapi/events.py: add type hint annotations
  qapi: establish mypy type-checking baseline
  qapi/common.py: move build_params into gen.py
  qapi/common.py: Convert comments into docstrings, and elaborate
  qapi/common.py: add type hint annotations
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
3 years agoMerge remote-tracking branch 'remotes/stsquad/tags/pull-various-091020-1' into staging
Peter Maydell [Sun, 11 Oct 2020 13:34:07 +0000 (14:34 +0100)]
Merge remote-tracking branch 'remotes/stsquad/tags/pull-various-091020-1' into staging

Testing, gitdm and plugin fixes:

  - fix acceptance regressions in MIPS and IDE
  - speed up cirrus msys2/mingw builds
  - add genisoimage to more docker images
  - slew of gitdb updates
  - fix some windows compile issues for plugins
  - add V=1 to cirrus output
  - disable rxsim in gitlab CI

# gpg: Signature made Fri 09 Oct 2020 17:30:29 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-various-091020-1: (22 commits)
  tests/acceptance: disable machine_rx_gdbsim on GitLab
  cirrus: use V=1 when running tests on FreeBSD and macOS
  plugin: Fixes compiling errors on msys2/mingw
  plugins: Fixes a issue when dlsym failed, the handle not closed
  .mailmap: Fix more contributor entries
  contrib/gitdm: Add Yandex to the domain map
  contrib/gitdm: Add Yadro to the domain map
  contrib/gitdm: Add SUSE to the domain map
  contrib/gitdm: Add Nir Soffer to Red Hat domain
  contrib/gitdm: Add Qualcomm to the domain map
  contrib/gitdm: Add Nuvia to the domain map
  contrib/gitdm: Add Google to the domain map
  contrib/gitdm: Add ByteDance to the domain map
  contrib/gitdm: Add Baidu to the domain map
  contrib/gitdm: Add more individual contributors
  contrib/gitdm: Add more academic domains
  tests/docker: Add genisoimage to the docker file
  cirrus: msys2/mingw speed is up, add excluded target back
  cirrus: Fixing and speedup the msys2/mingw CI
  hw/ide: restore replay support of IDE
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
3 years agoqapi/visit.py: add type hint annotations
John Snow [Fri, 9 Oct 2020 16:15:58 +0000 (12:15 -0400)]
qapi/visit.py: add type hint annotations

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-37-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/visit.py: remove unused parameters from gen_visit_object
John Snow [Fri, 9 Oct 2020 16:15:57 +0000 (12:15 -0400)]
qapi/visit.py: remove unused parameters from gen_visit_object

And this fixes the pylint report for this file, so make sure we check
this in the future, too.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201009161558.107041-36-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/visit.py: assert tag_member contains a QAPISchemaEnumType
John Snow [Fri, 9 Oct 2020 16:15:56 +0000 (12:15 -0400)]
qapi/visit.py: assert tag_member contains a QAPISchemaEnumType

This is true by design, but not presently able to be expressed in the
type system. An assertion helps mypy understand our constraints.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-35-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/types.py: remove one-letter variables
John Snow [Fri, 9 Oct 2020 16:15:55 +0000 (12:15 -0400)]
qapi/types.py: remove one-letter variables

"John, if pylint told you to jump off a bridge, would you?"
Hey, if it looked like fun, I might.

Now that this file is clean, enable pylint checks on this file.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-34-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/types.py: add type hint annotations
John Snow [Fri, 9 Oct 2020 16:15:54 +0000 (12:15 -0400)]
qapi/types.py: add type hint annotations

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-33-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/gen.py: delint with pylint
John Snow [Fri, 9 Oct 2020 16:15:53 +0000 (12:15 -0400)]
qapi/gen.py: delint with pylint

'fp' and 'fd' are self-evident in context, add them to the list of OK
names.

_top and _bottom also need to stay standard methods because some users
override the method and need to use `self`. Tell pylint to shush.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-32-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/gen.py: update write() to be more idiomatic
John Snow [Fri, 9 Oct 2020 16:15:52 +0000 (12:15 -0400)]
qapi/gen.py: update write() to be more idiomatic

Make the file handling here just a tiny bit more idiomatic.
(I realize this is heavily subjective.)

Use exist_ok=True for os.makedirs and remove the exception,
use fdopen() to wrap the file descriptor in a File-like object,
and use a context manager for managing the file pointer.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201009161558.107041-31-jsnow@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/gen.py: Remove unused parameter
John Snow [Fri, 9 Oct 2020 16:15:51 +0000 (12:15 -0400)]
qapi/gen.py: Remove unused parameter

_module_dirname doesn't use the 'what' argument, so remove it.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-30-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/gen.py: add type hint annotations
John Snow [Fri, 9 Oct 2020 16:15:49 +0000 (12:15 -0400)]
qapi/gen.py: add type hint annotations

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-28-jsnow@redhat.com>
Message-Id: <20201009161558.107041-29-jsnow@redhat.com>
[mypy.ini update squashed in]
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/gen: Make _is_user_module() return bool
John Snow [Fri, 9 Oct 2020 16:15:48 +0000 (12:15 -0400)]
qapi/gen: Make _is_user_module() return bool

_is_user_module() returns thruth values.  The next commit wants it to
return bool.  Make it so.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201009161558.107041-27-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message rewritten]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/source.py: delint with pylint
John Snow [Fri, 9 Oct 2020 16:15:47 +0000 (12:15 -0400)]
qapi/source.py: delint with pylint

Shush an error and leave a hint for future cleanups when we're allowed
to use Python 3.7+.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-26-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/source.py: add type hint annotations
John Snow [Fri, 9 Oct 2020 16:15:46 +0000 (12:15 -0400)]
qapi/source.py: add type hint annotations

Annotations do not change runtime behavior.
This commit *only* adds annotations.

A note on typing of __init__: mypy requires init functions with no
parameters to document a return type of None to be considered fully
typed. In the case when there are input parameters, None may be omitted.

Since __init__ may never return any value, it is preferred to omit the
return annotation whenever possible.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-25-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/commands.py: add type hint annotations
John Snow [Fri, 9 Oct 2020 16:15:44 +0000 (12:15 -0400)]
qapi/commands.py: add type hint annotations

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-23-jsnow@redhat.com>
Message-Id: <20201009161558.107041-24-jsnow@redhat.com>
[mypy.ini update squashed in]
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/commands.py: Don't re-bind to variable of different type
John Snow [Fri, 9 Oct 2020 16:15:43 +0000 (12:15 -0400)]
qapi/commands.py: Don't re-bind to variable of different type

Mypy isn't a fan of rebinding a variable with a new data type.
It's easy enough to avoid.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201009161558.107041-22-jsnow@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/events.py: Move comments into docstrings
John Snow [Fri, 9 Oct 2020 16:15:42 +0000 (12:15 -0400)]
qapi/events.py: Move comments into docstrings

Clarify them while we're here.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-21-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/events.py: add type hint annotations
John Snow [Fri, 9 Oct 2020 16:15:41 +0000 (12:15 -0400)]
qapi/events.py: add type hint annotations

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Note: __init__ does not need its return type annotated, as it is special.
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-20-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi: establish mypy type-checking baseline
John Snow [Fri, 9 Oct 2020 16:15:40 +0000 (12:15 -0400)]
qapi: establish mypy type-checking baseline

Fix a minor typing issue, and then establish a mypy type-checking
baseline.

Like pylint, this should be run from the folder above:

 > mypy --config-file=qapi/mypy.ini qapi/

This is designed and tested for mypy 0.770 or greater.

Signed-off-by: John Snow <jsnow@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-19-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/common.py: move build_params into gen.py
John Snow [Fri, 9 Oct 2020 16:15:39 +0000 (12:15 -0400)]
qapi/common.py: move build_params into gen.py

Including it in common.py creates a circular import dependency; schema
relies on common, but common.build_params requires a type annotation
from schema. To type this properly, it needs to be moved outside the
cycle.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-18-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/common.py: Convert comments into docstrings, and elaborate
John Snow [Fri, 9 Oct 2020 16:15:38 +0000 (12:15 -0400)]
qapi/common.py: Convert comments into docstrings, and elaborate

As docstrings, they'll show up in documentation and IDE help.

The docstring style being targeted is the Sphinx documentation
style. Sphinx uses an extension of ReST with "domains". We use the
(implicit) Python domain, which supports a number of custom "info
fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists

Primarily, we use `:param X: descr`, `:return[s]: descr`, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.

(No, nothing checks or enforces this style that I am aware of. Sphinx
either chokes or succeeds, but does not enforce a standard of what is
otherwise inside the docstring. Pycharm does highlight when your param
fields are not aligned with the actual fields present. It does not
highlight missing return or exception statements. There is no existing
style guide I am aware of that covers a standard for a minimally
acceptable docstring. I am debating writing one.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-17-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/common.py: add type hint annotations
John Snow [Fri, 9 Oct 2020 16:15:37 +0000 (12:15 -0400)]
qapi/common.py: add type hint annotations

Annotations do not change runtime behavior.
This commit *only* adds annotations.

Note that build_params() cannot be fully annotated due to import
dependency issues.  The commit after next will take care of it.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-16-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/common.py: check with pylint
John Snow [Fri, 9 Oct 2020 16:15:36 +0000 (12:15 -0400)]
qapi/common.py: check with pylint

Remove qapi/common.py from the pylintrc ignore list.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20201009161558.107041-15-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/common.py: Replace one-letter 'c' variable
John Snow [Fri, 9 Oct 2020 16:15:35 +0000 (12:15 -0400)]
qapi/common.py: Replace one-letter 'c' variable

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201009161558.107041-14-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/common.py: delint with pylint
John Snow [Fri, 9 Oct 2020 16:15:34 +0000 (12:15 -0400)]
qapi/common.py: delint with pylint

At this point, that just means using a consistent strategy for constant names.
constants get UPPER_CASE and names not used externally get a leading underscore.

As a preference, while renaming constants to be UPPERCASE, move them to
the head of the file. Generally, it's nice to be able to audit the code
that runs on import in one central place.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-13-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/common.py: Add indent manager
John Snow [Fri, 9 Oct 2020 16:15:33 +0000 (12:15 -0400)]
qapi/common.py: Add indent manager

Code style tools really dislike the use of global keywords, because it
generally involves re-binding the name at runtime which can have strange
effects depending on when and how that global name is referenced in
other modules.

Make a little indent level manager instead.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-12-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi/common.py: Remove python compatibility workaround
John Snow [Fri, 9 Oct 2020 16:15:32 +0000 (12:15 -0400)]
qapi/common.py: Remove python compatibility workaround

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-11-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi: add pylintrc
John Snow [Fri, 9 Oct 2020 16:15:31 +0000 (12:15 -0400)]
qapi: add pylintrc

Using `pylint --generate-rcfile > pylintrc`, generate a skeleton
pylintrc file. Sections that are not presently relevant (by the end of
this series) are removed leaving just the empty section as a search
engine / documentation hint to future authors.

I am targeting pylint 2.6.0. In the future (and hopefully before 5.2 is
released), I aim to have gitlab CI running the specific targeted
versions of pylint, mypy, flake8, etc in a job.

2.5.x will work if you additionally pass --disable=bad-whitespace.
This warning was removed from 2.6.x, for lack of consistent support.

Right now, quite a few modules are ignored as they are known to fail as
of this commit. modules will be removed from the known-bad list
throughout this and following series as they are repaired.

Note: Normally, pylintrc would go in the folder above the module, but as
that folder is shared by many things, it is going inside the module
folder (for now). Due to a bug in pylint 2.5+, pylint does not
correctly recognize when it is being run from "inside" a package, and
must be run *outside* of the package.

Therefore, to run it, you must:

 > pylint scripts/qapi/ --rcfile=scripts/qapi/pylintrc

Signed-off-by: John Snow <jsnow@redhat.com>
Tested-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-10-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi: delint using flake8
John Snow [Fri, 9 Oct 2020 16:15:30 +0000 (12:15 -0400)]
qapi: delint using flake8

Petty style guide fixes and line length enforcement.  Not a big win, not
a big loss, but flake8 passes 100% on the qapi module, which gives us an
easy baseline to enforce hereafter.

A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-9-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi: enforce import order/styling with isort
John Snow [Fri, 9 Oct 2020 16:15:29 +0000 (12:15 -0400)]
qapi: enforce import order/styling with isort

While we're mucking around with imports, we might as well formalize the
style we use. Let's use isort to do it for us.

lines_after_imports=2: Use two lines after imports, to match PEP8's
desire to have "two lines before and after" class definitions, which are
likely to start immediately after imports.

force_sort_within_sections: Intermingles "from x" and "import x" style
statements, such that sorting is always performed strictly on the module
name itself.

force_grid_wrap=4: Four or more imports from a single module will force
the one-per-line style that's more git-friendly. This will generally
happen for 'typing' imports.

multi_line_output=3: Uses the one-per-line indented style for long
imports.

include_trailing_comma: Adds a comma to the last import in a group,
which makes git conflicts nicer to deal with, generally.

line_length: 72 is chosen to match PEP8's "docstrings and comments" line
length limit. If you have a single line import that exceeds 72
characters, your names are too long!

Suggested-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201009161558.107041-8-jsnow@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi: Remove wildcard includes
John Snow [Fri, 9 Oct 2020 16:15:28 +0000 (12:15 -0400)]
qapi: Remove wildcard includes

Wildcard includes become hard to manage when refactoring and dealing
with circular dependencies with strictly typed mypy.

flake8 also flags each one as a warning, as it is not smart enough to
know which names exist in the imported file.

Remove them and include things explicitly by name instead.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201009161558.107041-7-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi: Prefer explicit relative imports
John Snow [Fri, 9 Oct 2020 16:15:27 +0000 (12:15 -0400)]
qapi: Prefer explicit relative imports

All of the QAPI include statements are changed to be package-aware, as
explicit relative imports.

A quirk of Python packages is that the name of the package exists only
*outside* of the package. This means that to a module inside of the qapi
folder, there is inherently no such thing as the "qapi" package. The
reason these imports work is because the "qapi" package exists in the
context of the caller -- the execution shim, where sys.path includes a
directory that has a 'qapi' folder in it.

When we write "from qapi import sibling", we are NOT referencing the folder
'qapi', but rather "any package named qapi in sys.path". If you should
so happen to have a 'qapi' package in your path, it will use *that*
package.

When we write "from .sibling import foo", we always reference explicitly
our sibling module; guaranteeing consistency in *where* we are importing
these modules from.

This can be useful when working with virtual environments and packages
in development mode. In development mode, a package is installed as a
series of symlinks that forwards to your same source files. The problem
arises because code quality checkers will follow "import qapi.x" to the
"installed" version instead of the sibling file and -- even though they
are the same file -- they have different module paths, and this causes
cyclic import problems, false positive type mismatch errors, and more.

It can also be useful when dealing with hierarchical packages, e.g. if
we allow qemu.core.qmp, qemu.qapi.parser, etc.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201009161558.107041-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi: move generator entrypoint into package
John Snow [Fri, 9 Oct 2020 16:15:26 +0000 (12:15 -0400)]
qapi: move generator entrypoint into package

As part of delinting and adding type hints to the QAPI generator, it's
helpful for the entrypoint to be part of the package, only leaving a
very tiny entrypoint shim outside of the package.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[invalid_char() renamed to invalid_prefix_char()]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi-gen: Separate arg-parsing from generation
John Snow [Fri, 9 Oct 2020 16:15:25 +0000 (12:15 -0400)]
qapi-gen: Separate arg-parsing from generation

This is a minor re-work of the entrypoint script. It isolates a
generate() method from the actual command-line mechanism.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-4-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[invalid_char() renamed to invalid_prefix_char()]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoqapi: modify docstrings to be sphinx-compatible
John Snow [Fri, 9 Oct 2020 16:15:24 +0000 (12:15 -0400)]
qapi: modify docstrings to be sphinx-compatible

A precise style guide and a package-wide overhaul is forthcoming pending
further discussion and consensus. For now, merely avoid obvious errors
that cause Sphinx documentation build problems, using a style loosely
based on PEP 257 and Sphinx Autodoc. It is chosen for interoperability
with our existing Sphinx framework, and because it has loose recognition
in the Pycharm IDE.

See also:

https://www.python.org/dev/peps/pep-0257/
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20201009161558.107041-3-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agodocs: repair broken references
John Snow [Fri, 9 Oct 2020 16:15:23 +0000 (12:15 -0400)]
docs: repair broken references

In two different places, we are not making a cross-reference to some
resource correctly.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20201009161558.107041-2-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
3 years agoMerge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2020-10-08-v3' into staging
Peter Maydell [Fri, 9 Oct 2020 21:55:46 +0000 (22:55 +0100)]
Merge remote-tracking branch 'remotes/ericb/tags/pull-nbd-2020-10-08-v3' into staging

nbd patches for 2020-10-08

- silence compilation warnings
- more fixes to prevent reconnect hangs
- improve 'qemu-nbd' termination behavior
- cleaner NBD protocol compliance on string handling

# gpg: Signature made Fri 09 Oct 2020 21:05:30 BST
# gpg:                using RSA key 71C2CC22B1C4602927D2F3AAA7A16B4A2527436A
# gpg: Good signature from "Eric Blake <eblake@redhat.com>" [full]
# gpg:                 aka "Eric Blake (Free Software Programmer) <ebb9@byu.net>" [full]
# gpg:                 aka "[jpeg image of size 6874]" [full]
# Primary key fingerprint: 71C2 CC22 B1C4 6029 27D2  F3AA A7A1 6B4A 2527 436A

* remotes/ericb/tags/pull-nbd-2020-10-08-v3:
  nbd: Simplify meta-context parsing
  nbd/server: Reject embedded NUL in NBD strings
  qemu-nbd: Honor SIGINT and SIGHUP
  block/nbd: nbd_co_reconnect_loop(): don't connect if drained
  block/nbd: fix reconnect-delay
  block/nbd: correctly use qio_channel_detach_aio_context when needed
  block/nbd: fix drain dead-lock because of nbd reconnect-delay
  nbd: silence maybe-uninitialized warnings

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
3 years agonbd: Simplify meta-context parsing
Eric Blake [Wed, 30 Sep 2020 12:11:03 +0000 (07:11 -0500)]
nbd: Simplify meta-context parsing

We had a premature optimization of trying to read as little from the
wire as possible while handling NBD_OPT_SET_META_CONTEXT in phases.
But in reality, we HAVE to read the entire string from the client
before we can get to the next command, and it is easier to just read
it all at once than it is to read it in pieces.  And once we do that,
several functions end up no longer performing I/O, so they can drop
length and errp parameters, and just return a bool instead of
modifying through a pointer.

Our iotests still pass; I also checked that libnbd's testsuite (which
covers more corner cases of odd meta context requests) still passes.
There are cases where the sequence of trace messages produced differs
(for example, when no bitmap is exported, a query for "qemu:" now
produces two trace lines instead of one), but trace points are for
debug and have no effect on what the client sees.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200930121105.667049-4-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: enhance commit message]
Signed-off-by: Eric Blake <eblake@redhat.com>
3 years agonbd/server: Reject embedded NUL in NBD strings
Eric Blake [Wed, 30 Sep 2020 12:11:02 +0000 (07:11 -0500)]
nbd/server: Reject embedded NUL in NBD strings

The NBD spec is clear that any string sent from the client must not
contain embedded NUL characters.  If the client passes "a\0", we
should reject that option request rather than act on "a".

Testing this is not possible with a compliant client, but I was able
to use gdb to coerce libnbd into temporarily behaving as such a
client.

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200930121105.667049-3-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
3 years agoqemu-nbd: Honor SIGINT and SIGHUP
Eric Blake [Wed, 30 Sep 2020 12:11:01 +0000 (07:11 -0500)]
qemu-nbd: Honor SIGINT and SIGHUP

Honoring just SIGTERM on Linux is too weak; we also want to handle
other common signals, and do so even on BSD.  Why?  Because at least
'qemu-nbd -B bitmap' needs a chance to clean up the in-use bit on
bitmaps when the server is shut down via a signal.

See also: http://bugzilla.redhat.com/1883608

Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200930121105.667049-2-eblake@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
[eblake: apply comment tweak suggested by Vladimir; fix ifdef around
termsig_handler]
Signed-off-by: Eric Blake <eblake@redhat.com>
3 years agoblock/nbd: nbd_co_reconnect_loop(): don't connect if drained
Vladimir Sementsov-Ogievskiy [Thu, 3 Sep 2020 19:03:01 +0000 (22:03 +0300)]
block/nbd: nbd_co_reconnect_loop(): don't connect if drained

In a recent commit 12c75e20a269ac we've improved
nbd_co_reconnect_loop() to not make drain wait for additional sleep.
Similarly, we shouldn't try to connect, if previous sleep was
interrupted by drain begin, otherwise drain_begin will have to wait for
the whole connection attempt.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200903190301.367620-5-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
3 years agoblock/nbd: fix reconnect-delay
Vladimir Sementsov-Ogievskiy [Thu, 3 Sep 2020 19:03:00 +0000 (22:03 +0300)]
block/nbd: fix reconnect-delay

reconnect-delay has a design flaw: we handle it in the same loop where
we do connection attempt. So, reconnect-delay may be exceeded by
unpredictable time of connection attempt.

Let's instead use separate timer.

How to reproduce the bug:

1. Create an image on node1:
   qemu-img create -f qcow2 xx 100M

2. Start NBD server on node1:
   qemu-nbd xx

3. On node2 start qemu-io:

./build/qemu-io --image-opts \
driver=nbd,server.type=inet,server.host=192.168.100.5,server.port=10809,reconnect-delay=15

4. Type 'read 0 512' in qemu-io interface to check that connection
   works

Be careful: you should make steps 5-7 in a short time, less than 15
seconds.

5. Kill nbd server on node1

6. Run 'read 0 512' in qemu-io interface again, to be sure that nbd
client goes to reconnect loop.

7. On node1 run the following command

   sudo iptables -A INPUT -p tcp --dport 10809 -j DROP

This will make the connect() call of qemu-io at node2 take a long time.

And you'll see that read command in qemu-io will hang for a long time,
more than 15 seconds specified by reconnect-delay parameter. It's the
bug.

8. Don't forget to drop iptables rule on node1:

   sudo iptables -D INPUT -p tcp --dport 10809 -j DROP

Important note: Step [5] is necessary to reproduce _this_ bug. If we
miss step [5], the read command (step 6) will hang for a long time and
this commit doesn't help, because there will be not long connect() to
unreachable host, but long sendmsg() to unreachable host, which should
be fixed by enabling and adjusting keep-alive on the socket, which is a
thing for further patch set.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200903190301.367620-4-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
3 years agoblock/nbd: correctly use qio_channel_detach_aio_context when needed
Vladimir Sementsov-Ogievskiy [Thu, 3 Sep 2020 19:02:59 +0000 (22:02 +0300)]
block/nbd: correctly use qio_channel_detach_aio_context when needed

Don't use nbd_client_detach_aio_context() driver handler where we want
to finalize the connection. We should directly use
qio_channel_detach_aio_context() in such cases. Driver handler may (and
will) contain another things, unrelated to the qio channel.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200903190301.367620-3-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
3 years agoblock/nbd: fix drain dead-lock because of nbd reconnect-delay
Vladimir Sementsov-Ogievskiy [Thu, 3 Sep 2020 19:02:58 +0000 (22:02 +0300)]
block/nbd: fix drain dead-lock because of nbd reconnect-delay

We pause reconnect process during drained section. So, if we have some
requests, waiting for reconnect we should cancel them, otherwise they
deadlock the drained section.

How to reproduce:

1. Create an image:
   qemu-img create -f qcow2 xx 100M

2. Start NBD server:
   qemu-nbd xx

3. Start vm with second nbd disk on node2, like this:

  ./build/x86_64-softmmu/qemu-system-x86_64 -nodefaults -drive \
     file=/work/images/cent7.qcow2 -drive \
     driver=nbd,server.type=inet,server.host=192.168.100.5,server.port=10809,reconnect-delay=60 \
     -vnc :0 -m 2G -enable-kvm -vga std

4. Access the vm through vnc (or some other way?), and check that NBD
   drive works:

   dd if=/dev/sdb of=/dev/null bs=1M count=10

   - the command should succeed.

5. Now, kill the nbd server, and run dd in the guest again:

   dd if=/dev/sdb of=/dev/null bs=1M count=10

Now Qemu is trying to reconnect, and dd-generated requests are waiting
for the connection (they will wait up to 60 seconds (see
reconnect-delay option above) and than fail). But suddenly, vm may
totally hang in the deadlock. You may need to increase reconnect-delay
period to catch the dead-lock.

VM doesn't respond because drain dead-lock happens in cpu thread with
global mutex taken. That's not good thing by itself and is not fixed
by this commit (true way is using iothreads). Still this commit fixes
drain dead-lock itself.

Note: probably, we can instead continue to reconnect during drained
section. To achieve this, we may move negotiation to the connect thread
to make it independent of bs aio context. But expanding drained section
doesn't seem good anyway. So, let's now fix the bug the simplest way.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200903190301.367620-2-vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
3 years agonbd: silence maybe-uninitialized warnings
Christian Borntraeger [Wed, 30 Sep 2020 15:58:57 +0000 (17:58 +0200)]
nbd: silence maybe-uninitialized warnings

gcc 10 from Fedora 32 gives me:

Compiling C object libblock.fa.p/nbd_server.c.o
../nbd/server.c: In function ‘nbd_co_client_start’:
../nbd/server.c:625:14: error: ‘namelen’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  625 |         rc = nbd_negotiate_send_info(client, NBD_INFO_NAME, namelen, name,
      |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  626 |                                      errp);
      |                                      ~~~~~
../nbd/server.c:564:14: note: ‘namelen’ was declared here
  564 |     uint32_t namelen;
      |              ^~~~~~~
cc1: all warnings being treated as errors

As I cannot see how this can happen, let uns silence the warning.

Signed-off-by: Christian Borntraeger <borntraeger@de.ibm.com>
Message-Id: <20200930155859.303148-3-borntraeger@de.ibm.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Eric Blake <eblake@redhat.com>
3 years agotests/acceptance: disable machine_rx_gdbsim on GitLab
Alex Bennée [Wed, 7 Oct 2020 16:00:38 +0000 (17:00 +0100)]
tests/acceptance: disable machine_rx_gdbsim on GitLab

While I can get the ssh test to fail on my test setup this seems a lot
more stable except when on GitLab. Hopefully we can re-enable both
once the serial timing patches have been added.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201007160038.26953-23-alex.bennee@linaro.org>

3 years agocirrus: use V=1 when running tests on FreeBSD and macOS
Paolo Bonzini [Wed, 7 Oct 2020 16:00:37 +0000 (17:00 +0100)]
cirrus: use V=1 when running tests on FreeBSD and macOS

Using "V=1" makes it easier to identify hanging tests, especially
since they are run with -j1.  It is already used on Windows builds,
do the same for FreeBSD and macOS.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Ed Maste <emaste@FreeBSD.org>
Message-Id: <20201007140103.711142-1-pbonzini@redhat.com>
Message-Id: <20201007160038.26953-22-alex.bennee@linaro.org>

3 years agoplugin: Fixes compiling errors on msys2/mingw
Yonggang Luo [Wed, 7 Oct 2020 16:00:36 +0000 (17:00 +0100)]
plugin: Fixes compiling errors on msys2/mingw

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20201001163429.1348-3-luoyonggang@gmail.com>
Message-Id: <20201007160038.26953-21-alex.bennee@linaro.org>

3 years agoplugins: Fixes a issue when dlsym failed, the handle not closed
Yonggang Luo [Wed, 7 Oct 2020 16:00:35 +0000 (17:00 +0100)]
plugins: Fixes a issue when dlsym failed, the handle not closed

Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20201001163429.1348-2-luoyonggang@gmail.com>
Message-Id: <20201007160038.26953-20-alex.bennee@linaro.org>

3 years ago.mailmap: Fix more contributor entries
Philippe Mathieu-Daudé [Wed, 7 Oct 2020 16:00:34 +0000 (17:00 +0100)]
.mailmap: Fix more contributor entries

These authors have some incorrect author email field.
For each of them, there is one commit with the replaced
entry.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Stefan Berger <stefanb@linux.ibm.com>
Acked-by: Erik Smit <erik.lucas.smit@gmail.com>
Message-Id: <20201006160653.2391972-13-f4bug@amsat.org>
Message-Id: <20201007160038.26953-19-alex.bennee@linaro.org>

3 years agocontrib/gitdm: Add Yandex to the domain map
Philippe Mathieu-Daudé [Wed, 7 Oct 2020 16:00:33 +0000 (17:00 +0100)]
contrib/gitdm: Add Yandex to the domain map

There is a number of contributors from this domain,
add its own entry to the gitdm domain map.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Roman Kagan <rvkagan@yandex-team.ru>
Message-Id: <20201006160653.2391972-12-f4bug@amsat.org>
Message-Id: <20201007160038.26953-18-alex.bennee@linaro.org>

3 years agocontrib/gitdm: Add Yadro to the domain map
Philippe Mathieu-Daudé [Wed, 7 Oct 2020 16:00:32 +0000 (17:00 +0100)]
contrib/gitdm: Add Yadro to the domain map

There is a number of contributions from this domain,
add its own entry to the gitdm domain map.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Roman Bolshakov <r.bolshakov@yadro.com>
Message-Id: <20201006160653.2391972-11-f4bug@amsat.org>
Message-Id: <20201007160038.26953-17-alex.bennee@linaro.org>

3 years agocontrib/gitdm: Add SUSE to the domain map
Philippe Mathieu-Daudé [Wed, 7 Oct 2020 16:00:31 +0000 (17:00 +0100)]
contrib/gitdm: Add SUSE to the domain map

There is a number of contributors from this domain,
add its own entry to the gitdm domain map.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Bruce Rogers <brogers@suse.com>
Message-Id: <20201006160653.2391972-10-f4bug@amsat.org>
Message-Id: <20201007160038.26953-16-alex.bennee@linaro.org>

3 years agocontrib/gitdm: Add Nir Soffer to Red Hat domain
Philippe Mathieu-Daudé [Wed, 7 Oct 2020 16:00:30 +0000 (17:00 +0100)]
contrib/gitdm: Add Nir Soffer to Red Hat domain

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Nir Soffer <nirsof@gmail.com>
Message-Id: <20201006160653.2391972-9-f4bug@amsat.org>
Message-Id: <20201007160038.26953-15-alex.bennee@linaro.org>