Peter Maydell [Tue, 13 Oct 2020 11:46:26 +0000 (12:46 +0100)]
Merge remote-tracking branch 'remotes/huth-gitlab/tags/pull-request-2020-10-13' into staging
* qtest improvements (test for crash found with the fuzzer, increase
downtime in migration test, less verbose output when running w/o KVM)
* Improve handling of acceptance tests in the Gitlab-CI
* Run checkpatch.pl in the Gitlab-CI
* Improve the gitlab-pipeline-status script
* Misc patches (mark 'moxie' as deprecated, remove stale .gitignore files, ...)
* remotes/huth-gitlab/tags/pull-request-2020-10-13: (23 commits)
scripts/ci/gitlab-pipeline-status: wait for pipeline creation
scripts/ci/gitlab-pipeline-status: use more descriptive exceptions
scripts/ci/gitlab-pipeline-status: handle keyboard interrupts
scripts/ci/gitlab-pipeline-status: refactor parser creation
scripts/ci/gitlab-pipeline-status: give early feedback on running pipelines
scripts/ci/gitlab-pipeline-status: improve message regarding timeout
scripts/ci/gitlab-pipeline-status: make branch name configurable
gitlab: assign python helper files to GitLab maintainers section
gitlab: add a CI job to validate the DCO sign off
gitlab: add a CI job for running checkpatch.pl
configure: fixes indent of $meson setup
docs/system/deprecated: Mark the 'moxie' CPU as deprecated
Remove superfluous .gitignore files
MAINTAINERS: Ignore bios-tables-test in the qtest section
Add a comment in bios-tables-test.c to clarify the reason behind approach
softmmu/vl: Be less verbose about missing KVM when running the qtests
tests/migration: Allow longer timeouts
qtest: add fuzz test case
Acceptance tests: show test report on GitLab CI
Acceptance tests: do not show canceled test logs on GitLab CI
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Cleber Rosa [Fri, 4 Sep 2020 16:42:58 +0000 (12:42 -0400)]
scripts/ci/gitlab-pipeline-status: wait for pipeline creation
When called in wait mode, this script will also wait for the pipeline
to be get to a "running" state. Because many more status may be seen
until a pipeline gets to "running", and those need to be handle too.
Reference: https://docs.gitlab.com/ee/api/pipelines.html#list-project-pipelines Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200904164258.240278-8-crosa@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Cleber Rosa [Fri, 4 Sep 2020 16:42:54 +0000 (12:42 -0400)]
scripts/ci/gitlab-pipeline-status: give early feedback on running pipelines
When waiting for a pipeline to run and finish, it's better to give
early feedback, and then sleep and wait, than the other wait around.
Specially for the first iteration, it's frustrating to see nothing
while the script is sleeping.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200904164258.240278-4-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
The script has its own timeout, which is about how long the script
will wait (when called with --wait) for the pipeline to complete, and
not necessarily for the pipeline to complete.
Hopefully this new wording will be clearer.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200904164258.240278-3-crosa@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Cleber Rosa [Fri, 4 Sep 2020 16:42:52 +0000 (12:42 -0400)]
scripts/ci/gitlab-pipeline-status: make branch name configurable
With the utility function `get_local_staging_branch_commit()`, the
name of the branch is hard coded (including in the function name).
For extensibility reasons, let's make that configurable.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20200904164258.240278-2-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
gitlab: assign python helper files to GitLab maintainers section
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200918132903.1848939-4-berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
While checkpatch.pl can validate DCO sign off that job must always be
advisory only since it is expected that certain patches will fail some
code style rules.
We require the DCO sign off to be mandatory for all commits though, so
it benefits from being validated in a standalone job.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200918132903.1848939-3-berrange@redhat.com>
[thuth: Use "stage: build" to let it run earlier] Signed-off-by: Thomas Huth <thuth@redhat.com>
This job is advisory since it is expected that certain patches will fail
the style checks and checkpatch.pl provides no way to mark exceptions to
the rules.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200918132903.1848939-2-berrange@redhat.com>
[thuth: Use "stage: build" to let it run earlier] Signed-off-by: Thomas Huth <thuth@redhat.com>
Yonggang Luo [Mon, 12 Oct 2020 23:43:44 +0000 (07:43 +0800)]
configure: fixes indent of $meson setup
convert these line from tab to space
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20201012234348.1427-2-luoyonggang@gmail.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Wed, 23 Sep 2020 14:05:34 +0000 (16:05 +0200)]
docs/system/deprecated: Mark the 'moxie' CPU as deprecated
It is currently unclear whether anybody is still using the 'moxie' CPU,
and there are no images for testing available this CPU, so the code has
likely bit-rotten in the course of time. When I asked the maintainer
for information, I did not get a reply within four weeks yet (see
https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg07201.html).
The last Signed-off-by line from Anthony in our repo is from 2013,
so it seems like this code is rather unmaintained. Time to put it onto
the deprecation list to see whether somebody is still interested in this
code or whether we could remove it in a couple of releases.
Message-Id: <20200923171815.97801-1-thuth@redhat.com> Reviewed-by: Richard Henderson <richard.henderson@linaro.org> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Thu, 1 Oct 2020 04:27:17 +0000 (06:27 +0200)]
MAINTAINERS: Ignore bios-tables-test in the qtest section
I'm very often getting CC: on rather large patch series that
modify the ACPI stuff of either ARM or x86, just because the
bios-table-test is often slightly involved here. I can't say
much about ACPI, and the bios-table-test is already covered
by the ACPI section in MAINTAINERS, so I'd rather prefer to
not getting automatically CC-ed on such patch series anymore.
If people want my opinion about qtest-related changes, they
can still put me on CC manually.
Message-Id: <20201001042717.136033-1-thuth@redhat.com> Acked-by: Laszlo Ersek <lersek@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Ani Sinha [Tue, 29 Sep 2020 14:25:01 +0000 (19:55 +0530)]
Add a comment in bios-tables-test.c to clarify the reason behind approach
A comment is added in bios-tables-test.c that explains the reasoning
behind the process of updating the ACPI table blobs when new tests are added
or old tests are modified or code is committed that affect tests. The
explanation would help future contributors follow the correct process when
making code changes that affect ACPI tables.
Signed-off-by: Ani Sinha <ani@anisinha.ca> Acked-by: Igor Mammedov <imammedo@redhat.com>
Message-Id: <20200929142501.1057-1-ani@anisinha.ca> Signed-off-by: Thomas Huth <thuth@redhat.com>
Thomas Huth [Fri, 10 Jul 2020 08:50:20 +0000 (10:50 +0200)]
softmmu/vl: Be less verbose about missing KVM when running the qtests
Some of the qtests use "-accel kvm -accel tcg" to run real guest code.
This causes some error messages when kvm is not available. We do not
really care about these messages since the fallback to tcg is expected
here. So let's silence them to avoid that they spoil the output of
the tests.
Unfortunately, we can not use the qtest_enabled() wrapper in this case,
since the qtest accelerator itself is not initialized. Thus we have to
test for the qtest_chrdev variable instead.
Message-Id: <20200710085020.28222-1-thuth@redhat.com> Reviewed-by: Alexander Bulekov <alxndr@bu.edu> Signed-off-by: Thomas Huth <thuth@redhat.com>
In travis, with gcov and gprof we're seeing timeouts; hopefully fix
this by increasing the test timeouts a bit, but for xbzrle ensure it
really does get a couple of cycles through to test the cache.
I think the problem in travis is we have about 2 host CPU threads,
in the test we have at least 3:
a) The vCPU thread (100% flat out)
b) The source migration thread
c) The destination migration thread
if (b) & (c) are slow for any reason - gcov+gperf or a slow host -
then they're sharing one host CPU thread so limit the migration
bandwidth.
Tested on my laptop with:
taskset -c 0,1 ./tests/qtest/migration-test -p /x86_64/migration
Reported-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20201008160330.130431-1-dgilbert@redhat.com>
[thuth: Move the #define to the right location] Signed-off-by: Thomas Huth <thuth@redhat.com>
Li Qiang [Mon, 21 Sep 2020 16:06:05 +0000 (09:06 -0700)]
qtest: add fuzz test case
Currently the device fuzzer finds more and more issues.
For every fuzz case, we need not only the fixes but also
the corresponding test case. We can analysis the reproducer
for every case and find what happened in where and write
a beautiful test case. However the raw data of reproducer is not
friendly to analysis. It will take a very long time, even far more
than the fixes itself. So let's create a new file to hold all of
the fuzz test cases and just use the raw data to act as the test
case. This way nobody will be afraid of writing a test case for
the fuzz reproducer.
Cleber Rosa [Fri, 9 Oct 2020 20:55:13 +0000 (16:55 -0400)]
Acceptance tests: show test report on GitLab CI
Avocado will, by default, produce JUnit files. Let's ask GitLab
to present those in the web UI.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009205513.751968-4-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Cleber Rosa [Fri, 9 Oct 2020 20:55:12 +0000 (16:55 -0400)]
Acceptance tests: do not show canceled test logs on GitLab CI
Tests resulting in "CANCEL" in Avocado are usually canceled on
purpose, and are almost identical to "SKIP". The logs for canceled
tests are adding a lot of noise to the logs being shown on GitLab CI,
and causing distraction from real failures.
As a side note, this "after script" is scheduled for removal once the
feature is implemented within Avocado itself.
Reference: https://github.com/avocado-framework/avocado/issues/4266 Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009205513.751968-3-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Cleber Rosa [Fri, 9 Oct 2020 20:55:11 +0000 (16:55 -0400)]
Acceptance tests: bump pycdlib version for easier installation
On with certain versions of "pip", package installations will attempt
to create wheels. And, on environments without a "complete" Python
installation (as described in the acceptance tests requirements docs),
that will fail.
pycdlib, starting with version 1.11.0, is now being made available
as wheels, so its instalation on those constrained environments is
now possible.
Buglink: https://bugs.launchpad.net/qemu/+bug/1897783 Reported-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009205513.751968-2-crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
With 1000 runs, there is a non-negligible chance that the fuzzer can
trigger a crash. With this CI job, we care about catching build/runtime
issues in the core fuzzing code. Actual device fuzzing takes place on
oss-fuzz. For these purposes, only running one input should be
sufficient.
Signed-off-by: Alexander Bulekov <alxndr@bu.edu> Suggested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201002143524.56930-1-alxndr@bu.edu> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Darren Kenny <darren.kenny@oracle.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
tests/qtest: Replace magic value by NANOSECONDS_PER_SECOND definition
Use self-explicit NANOSECONDS_PER_SECOND definition instead
of a magic value.
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20201011194918.3219195-5-f4bug@amsat.org> Reviewed-by: Thomas Huth <thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
Peter Maydell [Mon, 12 Oct 2020 21:48:45 +0000 (22:48 +0100)]
Merge remote-tracking branch 'remotes/bonzini-gitlab/tags/for-upstream' into staging
* qtest documentation improvements (Eduardo, myself)
* libqtest event buffering (Maxim)
* use RCU for list of children of a bus (Maxim)
* move more files to softmmu/ (myself)
* meson.build cleanups, qemu-storage-daemon fix (Philippe)
* remotes/bonzini-gitlab/tags/for-upstream: (38 commits)
meson: identify more sections of meson.build
scsi/scsi_bus: fix races in REPORT LUNS
virtio-scsi: use scsi_device_get
scsi/scsi_bus: Add scsi_device_get
scsi/scsi-bus: scsi_device_find: don't return unrealized devices
device-core: use atomic_set on .realized property
scsi: switch to bus->check_address
device-core: use RCU for list of children of a bus
device_core: use drain_call_rcu in in qmp_device_add
scsi/scsi_bus: switch search direction in scsi_device_find
qdev: add "check if address free" callback for buses
qemu-iotests, qtest: rewrite test 067 as a qtest
qtest: check that drives are really appearing and disappearing
qtest: switch users back to qtest_qmp_receive
device-plug-test: use qtest_qmp to send the device_del command
qtest: remove qtest_qmp_receive_success
qtest: Reintroduce qtest_qmp_receive with QMP event buffering
qtest: rename qtest_qmp_receive to qtest_qmp_receive_dict
meson.build: Re-enable KVM support for MIPS
build-sys: fix git version from -version
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Maxim Levitsky [Tue, 6 Oct 2020 12:39:04 +0000 (15:39 +0300)]
scsi/scsi_bus: fix races in REPORT LUNS
Currently scsi_target_emulate_report_luns iterates over the child device list
twice, and there is no guarantee that this list is the same in both iterations.
The reason for iterating twice is that the first iteration calculates
how much memory to allocate. However if we use a dynamic array we can
avoid iterating twice, and therefore we avoid this race.
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1866707 Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-10-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-14-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The device core first places a device on the bus and then realizes it.
Make scsi_device_find avoid returing such devices to avoid
races in drivers that use an iothread (currently virtio-scsi)
Maxim Levitsky [Tue, 6 Oct 2020 12:39:00 +0000 (15:39 +0300)]
device-core: use atomic_set on .realized property
Some code might race with placement of new devices on a bus.
We currently first place a (unrealized) device on the bus
and then realize it.
As a workaround, users that scan the child device list, can
check the realized property to see if it is safe to access such a device.
Use an atomic write here too to aid with this.
A separate discussion is what to do with devices that are unrealized:
It looks like for this case we only call the hotplug handler's unplug
callback and its up to it to unrealize the device.
An atomic operation doesn't cause harm for this code path though.
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-6-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-10-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Tue, 6 Oct 2020 12:38:56 +0000 (15:38 +0300)]
scsi: switch to bus->check_address
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-6-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Maxim Levitsky [Tue, 6 Oct 2020 12:38:59 +0000 (15:38 +0300)]
device-core: use RCU for list of children of a bus
This fixes the race between device emulation code that tries to find
a child device to dispatch the request to (e.g a scsi disk),
and hotplug of a new device to that bus.
Note that this doesn't convert all the readers of the list
but only these that might go over that list without BQL held.
This is a very small first step to make this code thread safe.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-5-mlevitsk@redhat.com>
[Use RCU_READ_LOCK_GUARD in more places, adjust testcase now that
the delay in DEVICE_DELETED due to RCU is more consistent. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-9-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Maxim Levitsky [Tue, 6 Oct 2020 12:38:58 +0000 (14:38 +0200)]
device_core: use drain_call_rcu in in qmp_device_add
Soon, a device removal might only happen on RCU callback execution.
This is okay for device-del which provides a DEVICE_DELETED event,
but not for the failure case of device-add. To avoid changing
monitor semantics, just drain all pending RCU callbacks on error.
Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Suggested-by: Stefan Hajnoczi <stefanha@gmail.com> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200913160259.32145-4-mlevitsk@redhat.com>
[Don't use it in qmp_device_del. - Paolo] Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Tue, 6 Oct 2020 12:38:55 +0000 (15:38 +0300)]
qdev: add "check if address free" callback for buses
Check if an address is free on the bus before plugging in the
device. This makes it possible to do the check without any
side effects, and to detect the problem early without having
to do it in the realize callback.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006123904.610658-5-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Wed, 7 Oct 2020 10:43:03 +0000 (06:43 -0400)]
qemu-iotests, qtest: rewrite test 067 as a qtest
Test 067 from qemu-iotests is executing QMP commands to hotplug
and hot-unplug disks, devices and blockdevs. Because the power
of the text-based test harness is limited, it is actually limiting
the checks that it does, for example by skipping DEVICE_DELETED
events.
tests/qtest already has a similar test, drive_del-test.c.
We can merge them, and even reuse some of the existing code in
drive_del-test.c. This will improve the quality of the test by
covering DEVICE_DELETED events and testing multiple architectures
(therefore covering multiple PCI hotplug mechanisms as well as s390x
virtio-ccw).
The only difference is that the new test will always use null-co:// for
the medium rather than qcow2 or raw, but this should be irrelevant for
what the test is covering. For example there are no "qemu-img check"
runs in 067 that would check that the file is properly closed.
The new tests requires PCI hot-plug support, so drive_del-test
is moved from qemu-system-ppc to qemu-system-ppc64.
Reviewed-by: Kevin Wolf <kwolf@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Maxim Levitsky [Tue, 6 Oct 2020 12:59:32 +0000 (08:59 -0400)]
qtest: remove qtest_qmp_receive_success
The purpose of qtest_qmp_receive_success was mostly to process events
that arrived between the issueing of a command and the "return"
line from QMP. This is now handled by the buffering of events
that libqtest performs automatically.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Maxim Levitsky [Tue, 6 Oct 2020 12:38:53 +0000 (15:38 +0300)]
qtest: Reintroduce qtest_qmp_receive with QMP event buffering
The new qtest_qmp_receive buffers all the received qmp events, allowing
qtest_qmp_eventwait_ref to return them.
This is intended to solve the race in regard to ordering of qmp events
vs qmp responses, as soon as the callers start using the new interface.
In addition to that, define qtest_qmp_event_ref a function which only scans
the buffer that qtest_qmp_receive stores the events to. This is intended
for callers that are only interested in events that were received during
the last call to the qtest_qmp_receive.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20201006123904.610658-3-mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Maxim Levitsky [Tue, 6 Oct 2020 12:38:52 +0000 (14:38 +0200)]
qtest: rename qtest_qmp_receive to qtest_qmp_receive_dict
In the next patch a new version of qtest_qmp_receive will be
reintroduced that will buffer received qmp events for later
consumption in qtest_qmp_eventwait_ref
No functional change intended.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Tue, 6 Oct 2020 12:44:00 +0000 (08:44 -0400)]
qtest: unify extra_qtest_srcs and extra_qtest_deps
Currently the extra sources and extra dependencies of qtests are held
in two separate dictionaries. Use the same trick as tests/meson.build
to combine them into one. This will make it easier to update the
documentation for unit tests and qtests.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Eduardo Habkost [Mon, 5 Oct 2020 20:52:26 +0000 (16:52 -0400)]
docs: Move QTest documentation to its own document
The qtest and libqtest doc comments will be parsed to generate
API documentation, so move QTest documentation to its own
document where the API and format documentation and will be
included.
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Acked-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20201005205228.697463-2-ehabkost@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Paolo Bonzini [Tue, 6 Oct 2020 07:05:29 +0000 (09:05 +0200)]
exec: split out non-softmmu-specific parts
Over the years, most parts of exec.c that were not specific to softmmu
have been moved to accel/tcg; what's left is mostly the low-level part
of the memory API, which includes RAMBlock and AddressSpaceDispatch.
However exec.c also hosts 4-500 lines of code for the target specific
parts of the CPU QOM object, plus a few functions for user-mode
emulation that do not have a better place (they are not TCG-specific so
accel/tcg/user-exec.c is not a good place either).
Move these parts to a new file, so that exec.c can be moved to
softmmu/physmem.c.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
While the FW_CFG_DATA_GENERATOR_INTERFACE is only consumed
by a device only available using system-mode (fw_cfg), it is
implemented by a crypto component (tls-cipher-suites) which
is always available when crypto is used.
Commit 69699f3055 introduced the following error in the
qemu-storage-daemon binary:
Since QOM dependencies are resolved at runtime, this issue
could not be triggered at linktime, and we don't have test
running the qemu-storage-daemon binary.
Fix by always registering the QOM interface.
Reported-by: Kevin Wolf <kwolf@redhat.com> Fixes: 69699f3055 ("crypto/tls-cipher-suites: Produce fw_cfg consumable blob") Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com> Reviewed-by: Laszlo Ersek <lersek@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20201006111909.2302081-2-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
qom: Move the creation of the library to the main meson.build
Be consistent creating all the libraries in the main meson.build file.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-10-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
authz: Move the creation of the library to the main meson.build
Be consistent creating all the libraries in the main meson.build file.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-9-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
crypto: Move the creation of the library to the main meson.build
Be consistent creating all the libraries in the main meson.build file.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-8-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
io: Move the creation of the library to the main meson.build
Be consistent creating all the libraries in the main meson.build file.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-7-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
migration: Move the creation of the library to the main meson.build
Be consistent creating all the libraries in the main meson.build file.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-6-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
chardev: Move the creation of the library to the main meson.build
Be consistent creating all the libraries in the main meson.build file.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-5-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
hw/core: Move the creation of the library to the main meson.build
Be consistent creating all the libraries in the main meson.build file.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-4-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-3-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
meson.build: Add comments to clarify code organization
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201006125602.2311423-2-philmd@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Peter Maydell [Mon, 12 Oct 2020 12:12:04 +0000 (13:12 +0100)]
Merge remote-tracking branch 'remotes/dgilbert/tags/pull-migration-20201012a' into staging
v3 Migration+ virtiofsd pull 2020-10-12
V3
Remove the postcopy recovery changes
Migration:
Dirtyrate measurement API cleanup
Virtiofsd:
Missing qemu_init_exec_dir call
Support for setting the group on socket creation
Stop a gcc warning
Avoid tempdir in sandboxing
# gpg: Signature made Mon 12 Oct 2020 12:43:30 BST
# gpg: using RSA key 45F5C71B4A0CB7FB977A9FA90516331EBC5BFDE7
# gpg: Good signature from "Dr. David Alan Gilbert (RH2) <dgilbert@redhat.com>" [full]
# Primary key fingerprint: 45F5 C71B 4A0C B7FB 977A 9FA9 0516 331E BC5B FDE7
* remotes/dgilbert/tags/pull-migration-20201012a:
migration/dirtyrate: present dirty rate only when querying the rate has completed
migration/dirtyrate: record start_time and calc_time while at the measuring state
virtiofsd: avoid /proc/self/fd tempdir
virtiofsd: Call qemu_init_exec_dir
tools/virtiofsd: add support for --socket-group
virtiofsd: Silence gcc warning
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
migration/dirtyrate: present dirty rate only when querying the rate has completed
Make dirty_rate field optional, present dirty rate only when querying
the rate has completed.
The qmp results is shown as follow:
@unstarted:
{"return":{"status":"unstarted","start-time":0,"calc-time":0},"id":"libvirt-12"}
@measuring:
{"return":{"status":"measuring","start-time":102931,"calc-time":1},"id":"libvirt-85"}
@measured:
{"return":{"status":"measured","dirty-rate":4,"start-time":150146,"calc-time":1},"id":"libvirt-15"}
Signed-off-by: Chuan Zheng <zhengchuan@huawei.com> Reviewed-by: David Edmondson <david.edmondson@oracle.com>
Message-Id: <1601350938-128320-3-git-send-email-zhengchuan@huawei.com> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
migration/dirtyrate: record start_time and calc_time while at the measuring state
Querying could include both the start-time and the calc-time while at the measuring
state, allowing a caller to determine when they should expect to come back looking
for a result.
Signed-off-by: Chuan Zheng <zhengchuan@huawei.com>
Message-Id: <1601350938-128320-2-git-send-email-zhengchuan@huawei.com> Reviewed-by: David Edmondson <david.edmondson@oracle.com> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Stefan Hajnoczi [Tue, 6 Oct 2020 09:58:26 +0000 (10:58 +0100)]
virtiofsd: avoid /proc/self/fd tempdir
In order to prevent /proc/self/fd escapes a temporary directory is
created where /proc/self/fd is bind-mounted. This doesn't work on
read-only file systems.
Avoid the temporary directory by bind-mounting /proc/self/fd over /proc.
This does not affect other processes since we remounted / with MS_REC |
MS_SLAVE. /proc must exist and virtiofsd does not use it so it's safe to
do this.
Path traversal can be tested with the following function:
static void test_proc_fd_escape(struct lo_data *lo)
{
int fd;
int level = 0;
ino_t last_ino = 0;
fd = lo->proc_self_fd;
for (;;) {
struct stat st;
if (fstat(fd, &st) != 0) {
perror("fstat");
return;
}
if (last_ino && st.st_ino == last_ino) {
fprintf(stderr, "inode number unchanged, stopping\n");
return;
}
last_ino = st.st_ino;
Since fcb4f59c879 qemu_get_local_state_pathname relies on the
init_exec_dir, and virtiofsd asserts because we never set it.
Set it.
Reported-by: Alex Bennée <alex.bennee@linaro.org> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20201002124015.44820-1-dgilbert@redhat.com> Tested-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Alex Bennée [Fri, 25 Sep 2020 12:51:29 +0000 (13:51 +0100)]
tools/virtiofsd: add support for --socket-group
If you like running QEMU as a normal user (very common for TCG runs)
but you have to run virtiofsd as a root user you run into connection
problems. Adding support for an optional --socket-group allows the
users to keep using the command line.
Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-Id: <20200925125147.26943-2-alex.bennee@linaro.org> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
dgilbert: Split long line
Gcc worries fd might be used unset, in reality it's always set if
fi is set, and only used if fi is set so it's safe. Initialise it to -1
just to keep gcc happy for now.
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Message-Id: <20200827153657.111098-2-dgilbert@redhat.com> Reviewed-by: Ján Tomko <jtomko@redhat.com> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Peter Maydell [Mon, 12 Oct 2020 10:29:41 +0000 (11:29 +0100)]
Merge remote-tracking branch 'remotes/armbru/tags/pull-qapi-2020-10-10' into staging
QAPI patches patches for 2020-10-10
# gpg: Signature made Sat 10 Oct 2020 10:43:14 BST
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653
* remotes/armbru/tags/pull-qapi-2020-10-10: (34 commits)
qapi/visit.py: add type hint annotations
qapi/visit.py: remove unused parameters from gen_visit_object
qapi/visit.py: assert tag_member contains a QAPISchemaEnumType
qapi/types.py: remove one-letter variables
qapi/types.py: add type hint annotations
qapi/gen.py: delint with pylint
qapi/gen.py: update write() to be more idiomatic
qapi/gen.py: Remove unused parameter
qapi/gen.py: add type hint annotations
qapi/gen: Make _is_user_module() return bool
qapi/source.py: delint with pylint
qapi/source.py: add type hint annotations
qapi/commands.py: add type hint annotations
qapi/commands.py: Don't re-bind to variable of different type
qapi/events.py: Move comments into docstrings
qapi/events.py: add type hint annotations
qapi: establish mypy type-checking baseline
qapi/common.py: move build_params into gen.py
qapi/common.py: Convert comments into docstrings, and elaborate
qapi/common.py: add type hint annotations
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Peter Maydell [Sun, 11 Oct 2020 13:34:07 +0000 (14:34 +0100)]
Merge remote-tracking branch 'remotes/stsquad/tags/pull-various-091020-1' into staging
Testing, gitdm and plugin fixes:
- fix acceptance regressions in MIPS and IDE
- speed up cirrus msys2/mingw builds
- add genisoimage to more docker images
- slew of gitdb updates
- fix some windows compile issues for plugins
- add V=1 to cirrus output
- disable rxsim in gitlab CI
# gpg: Signature made Fri 09 Oct 2020 17:30:29 BST
# gpg: using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8 DF35 FBD0 DB09 5A9E 2A44
* remotes/stsquad/tags/pull-various-091020-1: (22 commits)
tests/acceptance: disable machine_rx_gdbsim on GitLab
cirrus: use V=1 when running tests on FreeBSD and macOS
plugin: Fixes compiling errors on msys2/mingw
plugins: Fixes a issue when dlsym failed, the handle not closed
.mailmap: Fix more contributor entries
contrib/gitdm: Add Yandex to the domain map
contrib/gitdm: Add Yadro to the domain map
contrib/gitdm: Add SUSE to the domain map
contrib/gitdm: Add Nir Soffer to Red Hat domain
contrib/gitdm: Add Qualcomm to the domain map
contrib/gitdm: Add Nuvia to the domain map
contrib/gitdm: Add Google to the domain map
contrib/gitdm: Add ByteDance to the domain map
contrib/gitdm: Add Baidu to the domain map
contrib/gitdm: Add more individual contributors
contrib/gitdm: Add more academic domains
tests/docker: Add genisoimage to the docker file
cirrus: msys2/mingw speed is up, add excluded target back
cirrus: Fixing and speedup the msys2/mingw CI
hw/ide: restore replay support of IDE
...
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
John Snow [Fri, 9 Oct 2020 16:15:57 +0000 (12:15 -0400)]
qapi/visit.py: remove unused parameters from gen_visit_object
And this fixes the pylint report for this file, so make sure we check
this in the future, too.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20201009161558.107041-36-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:52 +0000 (12:15 -0400)]
qapi/gen.py: update write() to be more idiomatic
Make the file handling here just a tiny bit more idiomatic.
(I realize this is heavily subjective.)
Use exist_ok=True for os.makedirs and remove the exception,
use fdopen() to wrap the file descriptor in a File-like object,
and use a context manager for managing the file pointer.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201009161558.107041-31-jsnow@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:46 +0000 (12:15 -0400)]
qapi/source.py: add type hint annotations
Annotations do not change runtime behavior.
This commit *only* adds annotations.
A note on typing of __init__: mypy requires init functions with no
parameters to document a return type of None to be considered fully
typed. In the case when there are input parameters, None may be omitted.
Since __init__ may never return any value, it is preferred to omit the
return annotation whenever possible.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-25-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:41 +0000 (12:15 -0400)]
qapi/events.py: add type hint annotations
Annotations do not change runtime behavior.
This commit *only* adds annotations.
Note: __init__ does not need its return type annotated, as it is special.
https://mypy.readthedocs.io/en/stable/class_basics.html#annotating-init-methods
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-20-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:39 +0000 (12:15 -0400)]
qapi/common.py: move build_params into gen.py
Including it in common.py creates a circular import dependency; schema
relies on common, but common.build_params requires a type annotation
from schema. To type this properly, it needs to be moved outside the
cycle.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-18-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:38 +0000 (12:15 -0400)]
qapi/common.py: Convert comments into docstrings, and elaborate
As docstrings, they'll show up in documentation and IDE help.
The docstring style being targeted is the Sphinx documentation
style. Sphinx uses an extension of ReST with "domains". We use the
(implicit) Python domain, which supports a number of custom "info
fields". Those info fields are documented here:
https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#info-field-lists
Primarily, we use `:param X: descr`, `:return[s]: descr`, and `:raise[s]
Z: when`. Everything else is the Sphinx dialect of ReST.
(No, nothing checks or enforces this style that I am aware of. Sphinx
either chokes or succeeds, but does not enforce a standard of what is
otherwise inside the docstring. Pycharm does highlight when your param
fields are not aligned with the actual fields present. It does not
highlight missing return or exception statements. There is no existing
style guide I am aware of that covers a standard for a minimally
acceptable docstring. I am debating writing one.)
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-17-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:34 +0000 (12:15 -0400)]
qapi/common.py: delint with pylint
At this point, that just means using a consistent strategy for constant names.
constants get UPPER_CASE and names not used externally get a leading underscore.
As a preference, while renaming constants to be UPPERCASE, move them to
the head of the file. Generally, it's nice to be able to audit the code
that runs on import in one central place.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-13-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:33 +0000 (12:15 -0400)]
qapi/common.py: Add indent manager
Code style tools really dislike the use of global keywords, because it
generally involves re-binding the name at runtime which can have strange
effects depending on when and how that global name is referenced in
other modules.
Make a little indent level manager instead.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-12-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:31 +0000 (12:15 -0400)]
qapi: add pylintrc
Using `pylint --generate-rcfile > pylintrc`, generate a skeleton
pylintrc file. Sections that are not presently relevant (by the end of
this series) are removed leaving just the empty section as a search
engine / documentation hint to future authors.
I am targeting pylint 2.6.0. In the future (and hopefully before 5.2 is
released), I aim to have gitlab CI running the specific targeted
versions of pylint, mypy, flake8, etc in a job.
2.5.x will work if you additionally pass --disable=bad-whitespace.
This warning was removed from 2.6.x, for lack of consistent support.
Right now, quite a few modules are ignored as they are known to fail as
of this commit. modules will be removed from the known-bad list
throughout this and following series as they are repaired.
Note: Normally, pylintrc would go in the folder above the module, but as
that folder is shared by many things, it is going inside the module
folder (for now). Due to a bug in pylint 2.5+, pylint does not
correctly recognize when it is being run from "inside" a package, and
must be run *outside* of the package.
John Snow [Fri, 9 Oct 2020 16:15:30 +0000 (12:15 -0400)]
qapi: delint using flake8
Petty style guide fixes and line length enforcement. Not a big win, not
a big loss, but flake8 passes 100% on the qapi module, which gives us an
easy baseline to enforce hereafter.
A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.
Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20201009161558.107041-9-jsnow@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
John Snow [Fri, 9 Oct 2020 16:15:29 +0000 (12:15 -0400)]
qapi: enforce import order/styling with isort
While we're mucking around with imports, we might as well formalize the
style we use. Let's use isort to do it for us.
lines_after_imports=2: Use two lines after imports, to match PEP8's
desire to have "two lines before and after" class definitions, which are
likely to start immediately after imports.
force_sort_within_sections: Intermingles "from x" and "import x" style
statements, such that sorting is always performed strictly on the module
name itself.
force_grid_wrap=4: Four or more imports from a single module will force
the one-per-line style that's more git-friendly. This will generally
happen for 'typing' imports.
multi_line_output=3: Uses the one-per-line indented style for long
imports.
include_trailing_comma: Adds a comma to the last import in a group,
which makes git conflicts nicer to deal with, generally.
line_length: 72 is chosen to match PEP8's "docstrings and comments" line
length limit. If you have a single line import that exceeds 72
characters, your names are too long!
Suggested-by: Cleber Rosa <crosa@redhat.com> Signed-off-by: John Snow <jsnow@redhat.com> Reviewed-by: Cleber Rosa <crosa@redhat.com> Tested-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20201009161558.107041-8-jsnow@redhat.com> Signed-off-by: Markus Armbruster <armbru@redhat.com>
projects / mirror_qemu.git / log