]> git.proxmox.com Git - mirror_qemu.git/commit
qapi: Speed up frontend tests
authorMarkus Armbruster <armbru@redhat.com>
Fri, 18 Oct 2019 07:43:42 +0000 (09:43 +0200)
committerMarkus Armbruster <armbru@redhat.com>
Tue, 22 Oct 2019 07:26:12 +0000 (09:26 +0200)
commitf01338cce692ac54109f09bc6c7b5567611e2d24
tree28237e1d9b112eb42cc833be9dfaa910032a510a
parent0002b557b5c8b013087fc18d75d370f11783f619
qapi: Speed up frontend tests

"make check-qapi-schema" takes around 10s user + system time for me.
With -j, it takes a bit over 3s real time.  We have worse tests.  It's
still annoying when you work on the QAPI generator.

Some 1.4s user + system time is consumed by make figuring out what to
do, measured by making a target that does nothing.  There's nothing I
can do about that right now.  But let's see what we can do about the
other 8s.

Almost 7s are spent running test-qapi.py for every test case, the rest
normalizing and diffing test-qapi.py output.  We have 190 test cases.

If I downgrade to python2, it's 4.5s, but python2 is a goner.

Hacking up test-qapi.py to exit(0) without doing anything makes it
only marginally faster.  The problem is Python startup overhead.

Our configure puts -B into $(PYTHON).  Running without -B is faster:
4.4s.

We could improve the Makefile to run test cases only when the test
case or the generator changed.  But I'm after improvement in the case
where the generator changed.

test-qapi.py is designed to be the simplest possible building block
for a shell script to do the complete job (it's actually a Makefile,
not a shell script; no real difference).  Python is just not meant for
that.  It's for bigger blocks.

Move the post-processing and diffing into test-qapi.py, and make it
capable of testing multiple schema files.  Set executable bits while
there.

Running it once per test case now takes slightly longer than 8s.  But
running it once for all of them takes under 0.2s.

Messing with the Makefile to run it only on the tests that need
retesting is clearly not worth the bother.

Expected error output changes because the new normalization strips off
$(SRCDIR)/tests/qapi-schema/ instead of just $(SRCDIR)/.

The .exit files go away, because there is no exit status to test
anymore.

Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20191018074345.24034-5-armbru@redhat.com>
364 files changed:
tests/Makefile.include
tests/qapi-schema/allow-preconfig-test.err
tests/qapi-schema/allow-preconfig-test.exit [deleted file]
tests/qapi-schema/alternate-any.err
tests/qapi-schema/alternate-any.exit [deleted file]
tests/qapi-schema/alternate-array.err
tests/qapi-schema/alternate-array.exit [deleted file]
tests/qapi-schema/alternate-base.err
tests/qapi-schema/alternate-base.exit [deleted file]
tests/qapi-schema/alternate-branch-if-invalid.err
tests/qapi-schema/alternate-branch-if-invalid.exit [deleted file]
tests/qapi-schema/alternate-clash.err
tests/qapi-schema/alternate-clash.exit [deleted file]
tests/qapi-schema/alternate-conflict-bool-string.err
tests/qapi-schema/alternate-conflict-bool-string.exit [deleted file]
tests/qapi-schema/alternate-conflict-dict.err
tests/qapi-schema/alternate-conflict-dict.exit [deleted file]
tests/qapi-schema/alternate-conflict-enum-bool.err
tests/qapi-schema/alternate-conflict-enum-bool.exit [deleted file]
tests/qapi-schema/alternate-conflict-enum-int.err
tests/qapi-schema/alternate-conflict-enum-int.exit [deleted file]
tests/qapi-schema/alternate-conflict-num-string.err
tests/qapi-schema/alternate-conflict-num-string.exit [deleted file]
tests/qapi-schema/alternate-conflict-string.err
tests/qapi-schema/alternate-conflict-string.exit [deleted file]
tests/qapi-schema/alternate-empty.err
tests/qapi-schema/alternate-empty.exit [deleted file]
tests/qapi-schema/alternate-invalid-dict.err
tests/qapi-schema/alternate-invalid-dict.exit [deleted file]
tests/qapi-schema/alternate-nested.err
tests/qapi-schema/alternate-nested.exit [deleted file]
tests/qapi-schema/alternate-unknown.err
tests/qapi-schema/alternate-unknown.exit [deleted file]
tests/qapi-schema/args-alternate.err
tests/qapi-schema/args-alternate.exit [deleted file]
tests/qapi-schema/args-any.err
tests/qapi-schema/args-any.exit [deleted file]
tests/qapi-schema/args-array-empty.err
tests/qapi-schema/args-array-empty.exit [deleted file]
tests/qapi-schema/args-array-unknown.err
tests/qapi-schema/args-array-unknown.exit [deleted file]
tests/qapi-schema/args-bad-boxed.err
tests/qapi-schema/args-bad-boxed.exit [deleted file]
tests/qapi-schema/args-boxed-anon.err
tests/qapi-schema/args-boxed-anon.exit [deleted file]
tests/qapi-schema/args-boxed-string.err
tests/qapi-schema/args-boxed-string.exit [deleted file]
tests/qapi-schema/args-int.err
tests/qapi-schema/args-int.exit [deleted file]
tests/qapi-schema/args-invalid.err
tests/qapi-schema/args-invalid.exit [deleted file]
tests/qapi-schema/args-member-array-bad.err
tests/qapi-schema/args-member-array-bad.exit [deleted file]
tests/qapi-schema/args-member-case.err
tests/qapi-schema/args-member-case.exit [deleted file]
tests/qapi-schema/args-member-unknown.err
tests/qapi-schema/args-member-unknown.exit [deleted file]
tests/qapi-schema/args-name-clash.err
tests/qapi-schema/args-name-clash.exit [deleted file]
tests/qapi-schema/args-union.err
tests/qapi-schema/args-union.exit [deleted file]
tests/qapi-schema/args-unknown.err
tests/qapi-schema/args-unknown.exit [deleted file]
tests/qapi-schema/bad-base.err
tests/qapi-schema/bad-base.exit [deleted file]
tests/qapi-schema/bad-data.err
tests/qapi-schema/bad-data.exit [deleted file]
tests/qapi-schema/bad-ident.err
tests/qapi-schema/bad-ident.exit [deleted file]
tests/qapi-schema/bad-if-empty-list.err
tests/qapi-schema/bad-if-empty-list.exit [deleted file]
tests/qapi-schema/bad-if-empty.err
tests/qapi-schema/bad-if-empty.exit [deleted file]
tests/qapi-schema/bad-if-list.err
tests/qapi-schema/bad-if-list.exit [deleted file]
tests/qapi-schema/bad-if.err
tests/qapi-schema/bad-if.exit [deleted file]
tests/qapi-schema/bad-type-bool.err
tests/qapi-schema/bad-type-bool.exit [deleted file]
tests/qapi-schema/bad-type-dict.err
tests/qapi-schema/bad-type-dict.exit [deleted file]
tests/qapi-schema/bad-type-int.err
tests/qapi-schema/bad-type-int.exit [deleted file]
tests/qapi-schema/base-cycle-direct.err
tests/qapi-schema/base-cycle-direct.exit [deleted file]
tests/qapi-schema/base-cycle-indirect.err
tests/qapi-schema/base-cycle-indirect.exit [deleted file]
tests/qapi-schema/command-int.err
tests/qapi-schema/command-int.exit [deleted file]
tests/qapi-schema/comments.exit [deleted file]
tests/qapi-schema/doc-bad-alternate-member.err
tests/qapi-schema/doc-bad-alternate-member.exit [deleted file]
tests/qapi-schema/doc-bad-command-arg.err
tests/qapi-schema/doc-bad-command-arg.exit [deleted file]
tests/qapi-schema/doc-bad-section.exit [deleted file]
tests/qapi-schema/doc-bad-symbol.err
tests/qapi-schema/doc-bad-symbol.exit [deleted file]
tests/qapi-schema/doc-bad-union-member.err
tests/qapi-schema/doc-bad-union-member.exit [deleted file]
tests/qapi-schema/doc-before-include.err
tests/qapi-schema/doc-before-include.exit [deleted file]
tests/qapi-schema/doc-before-pragma.err
tests/qapi-schema/doc-before-pragma.exit [deleted file]
tests/qapi-schema/doc-duplicated-arg.err
tests/qapi-schema/doc-duplicated-arg.exit [deleted file]
tests/qapi-schema/doc-duplicated-return.err
tests/qapi-schema/doc-duplicated-return.exit [deleted file]
tests/qapi-schema/doc-duplicated-since.err
tests/qapi-schema/doc-duplicated-since.exit [deleted file]
tests/qapi-schema/doc-empty-arg.err
tests/qapi-schema/doc-empty-arg.exit [deleted file]
tests/qapi-schema/doc-empty-section.err
tests/qapi-schema/doc-empty-section.exit [deleted file]
tests/qapi-schema/doc-empty-symbol.err
tests/qapi-schema/doc-empty-symbol.exit [deleted file]
tests/qapi-schema/doc-good.exit [deleted file]
tests/qapi-schema/doc-interleaved-section.err
tests/qapi-schema/doc-interleaved-section.exit [deleted file]
tests/qapi-schema/doc-invalid-end.err
tests/qapi-schema/doc-invalid-end.exit [deleted file]
tests/qapi-schema/doc-invalid-end2.err
tests/qapi-schema/doc-invalid-end2.exit [deleted file]
tests/qapi-schema/doc-invalid-return.err
tests/qapi-schema/doc-invalid-return.exit [deleted file]
tests/qapi-schema/doc-invalid-section.err
tests/qapi-schema/doc-invalid-section.exit [deleted file]
tests/qapi-schema/doc-invalid-start.err
tests/qapi-schema/doc-invalid-start.exit [deleted file]
tests/qapi-schema/doc-missing-colon.err
tests/qapi-schema/doc-missing-colon.exit [deleted file]
tests/qapi-schema/doc-missing-expr.err
tests/qapi-schema/doc-missing-expr.exit [deleted file]
tests/qapi-schema/doc-missing-space.err
tests/qapi-schema/doc-missing-space.exit [deleted file]
tests/qapi-schema/doc-missing.err
tests/qapi-schema/doc-missing.exit [deleted file]
tests/qapi-schema/doc-no-symbol.err
tests/qapi-schema/doc-no-symbol.exit [deleted file]
tests/qapi-schema/double-type.err
tests/qapi-schema/double-type.exit [deleted file]
tests/qapi-schema/duplicate-key.err
tests/qapi-schema/duplicate-key.exit [deleted file]
tests/qapi-schema/empty.exit [deleted file]
tests/qapi-schema/enum-bad-member.err
tests/qapi-schema/enum-bad-member.exit [deleted file]
tests/qapi-schema/enum-bad-name.err
tests/qapi-schema/enum-bad-name.exit [deleted file]
tests/qapi-schema/enum-bad-prefix.err
tests/qapi-schema/enum-bad-prefix.exit [deleted file]
tests/qapi-schema/enum-clash-member.err
tests/qapi-schema/enum-clash-member.exit [deleted file]
tests/qapi-schema/enum-dict-member-unknown.err
tests/qapi-schema/enum-dict-member-unknown.exit [deleted file]
tests/qapi-schema/enum-if-invalid.err
tests/qapi-schema/enum-if-invalid.exit [deleted file]
tests/qapi-schema/enum-int-member.err
tests/qapi-schema/enum-int-member.exit [deleted file]
tests/qapi-schema/enum-member-case.err
tests/qapi-schema/enum-member-case.exit [deleted file]
tests/qapi-schema/enum-missing-data.err
tests/qapi-schema/enum-missing-data.exit [deleted file]
tests/qapi-schema/enum-wrong-data.err
tests/qapi-schema/enum-wrong-data.exit [deleted file]
tests/qapi-schema/escape-outside-string.err
tests/qapi-schema/event-boxed-empty.err
tests/qapi-schema/event-boxed-empty.exit [deleted file]
tests/qapi-schema/event-case.exit [deleted file]
tests/qapi-schema/event-member-invalid-dict.err
tests/qapi-schema/event-member-invalid-dict.exit [deleted file]
tests/qapi-schema/event-nest-struct.err
tests/qapi-schema/event-nest-struct.exit [deleted file]
tests/qapi-schema/features-bad-type.err
tests/qapi-schema/features-bad-type.exit [deleted file]
tests/qapi-schema/features-duplicate-name.err
tests/qapi-schema/features-duplicate-name.exit [deleted file]
tests/qapi-schema/features-if-invalid.err
tests/qapi-schema/features-if-invalid.exit [deleted file]
tests/qapi-schema/features-missing-name.err
tests/qapi-schema/features-missing-name.exit [deleted file]
tests/qapi-schema/features-name-bad-type.err
tests/qapi-schema/features-name-bad-type.exit [deleted file]
tests/qapi-schema/features-no-list.err
tests/qapi-schema/features-no-list.exit [deleted file]
tests/qapi-schema/features-unknown-key.err
tests/qapi-schema/features-unknown-key.exit [deleted file]
tests/qapi-schema/flat-union-array-branch.err
tests/qapi-schema/flat-union-array-branch.exit [deleted file]
tests/qapi-schema/flat-union-bad-base.err
tests/qapi-schema/flat-union-bad-base.exit [deleted file]
tests/qapi-schema/flat-union-bad-discriminator.err
tests/qapi-schema/flat-union-bad-discriminator.exit [deleted file]
tests/qapi-schema/flat-union-base-any.err
tests/qapi-schema/flat-union-base-any.exit [deleted file]
tests/qapi-schema/flat-union-base-union.err
tests/qapi-schema/flat-union-base-union.exit [deleted file]
tests/qapi-schema/flat-union-clash-member.err
tests/qapi-schema/flat-union-clash-member.exit [deleted file]
tests/qapi-schema/flat-union-discriminator-bad-name.err
tests/qapi-schema/flat-union-discriminator-bad-name.exit [deleted file]
tests/qapi-schema/flat-union-empty.err
tests/qapi-schema/flat-union-empty.exit [deleted file]
tests/qapi-schema/flat-union-inline-invalid-dict.err
tests/qapi-schema/flat-union-inline-invalid-dict.exit [deleted file]
tests/qapi-schema/flat-union-inline.err
tests/qapi-schema/flat-union-inline.exit [deleted file]
tests/qapi-schema/flat-union-int-branch.err
tests/qapi-schema/flat-union-int-branch.exit [deleted file]
tests/qapi-schema/flat-union-invalid-branch-key.err
tests/qapi-schema/flat-union-invalid-branch-key.exit [deleted file]
tests/qapi-schema/flat-union-invalid-discriminator.err
tests/qapi-schema/flat-union-invalid-discriminator.exit [deleted file]
tests/qapi-schema/flat-union-invalid-if-discriminator.err
tests/qapi-schema/flat-union-invalid-if-discriminator.exit [deleted file]
tests/qapi-schema/flat-union-no-base.err
tests/qapi-schema/flat-union-no-base.exit [deleted file]
tests/qapi-schema/flat-union-optional-discriminator.err
tests/qapi-schema/flat-union-optional-discriminator.exit [deleted file]
tests/qapi-schema/flat-union-string-discriminator.err
tests/qapi-schema/flat-union-string-discriminator.exit [deleted file]
tests/qapi-schema/funny-char.err
tests/qapi-schema/funny-char.exit [deleted file]
tests/qapi-schema/funny-word.err
tests/qapi-schema/funny-word.exit [deleted file]
tests/qapi-schema/ident-with-escape.err
tests/qapi-schema/ident-with-escape.exit [deleted file]
tests/qapi-schema/include-before-err.err
tests/qapi-schema/include-before-err.exit [deleted file]
tests/qapi-schema/include-cycle.err
tests/qapi-schema/include-cycle.exit [deleted file]
tests/qapi-schema/include-extra-junk.err
tests/qapi-schema/include-extra-junk.exit [deleted file]
tests/qapi-schema/include-nested-err.err
tests/qapi-schema/include-nested-err.exit [deleted file]
tests/qapi-schema/include-no-file.err
tests/qapi-schema/include-no-file.exit [deleted file]
tests/qapi-schema/include-non-file.err
tests/qapi-schema/include-non-file.exit [deleted file]
tests/qapi-schema/include-repetition.exit [deleted file]
tests/qapi-schema/include-self-cycle.err
tests/qapi-schema/include-self-cycle.exit [deleted file]
tests/qapi-schema/include-simple.exit [deleted file]
tests/qapi-schema/indented-expr.exit [deleted file]
tests/qapi-schema/leading-comma-list.err
tests/qapi-schema/leading-comma-list.exit [deleted file]
tests/qapi-schema/leading-comma-object.err
tests/qapi-schema/leading-comma-object.exit [deleted file]
tests/qapi-schema/missing-colon.err
tests/qapi-schema/missing-colon.exit [deleted file]
tests/qapi-schema/missing-comma-list.err
tests/qapi-schema/missing-comma-list.exit [deleted file]
tests/qapi-schema/missing-comma-object.err
tests/qapi-schema/missing-comma-object.exit [deleted file]
tests/qapi-schema/missing-type.err
tests/qapi-schema/missing-type.exit [deleted file]
tests/qapi-schema/nested-struct-data-invalid-dict.err
tests/qapi-schema/nested-struct-data-invalid-dict.exit [deleted file]
tests/qapi-schema/nested-struct-data.err
tests/qapi-schema/nested-struct-data.exit [deleted file]
tests/qapi-schema/non-objects.err
tests/qapi-schema/non-objects.exit [deleted file]
tests/qapi-schema/oob-test.err
tests/qapi-schema/oob-test.exit [deleted file]
tests/qapi-schema/pragma-doc-required-crap.err
tests/qapi-schema/pragma-doc-required-crap.exit [deleted file]
tests/qapi-schema/pragma-extra-junk.err
tests/qapi-schema/pragma-extra-junk.exit [deleted file]
tests/qapi-schema/pragma-name-case-whitelist-crap.err
tests/qapi-schema/pragma-name-case-whitelist-crap.exit [deleted file]
tests/qapi-schema/pragma-non-dict.err
tests/qapi-schema/pragma-non-dict.exit [deleted file]
tests/qapi-schema/pragma-returns-whitelist-crap.err
tests/qapi-schema/pragma-returns-whitelist-crap.exit [deleted file]
tests/qapi-schema/pragma-unknown.err
tests/qapi-schema/pragma-unknown.exit [deleted file]
tests/qapi-schema/qapi-schema-test.exit [deleted file]
tests/qapi-schema/quoted-structural-chars.err
tests/qapi-schema/quoted-structural-chars.exit [deleted file]
tests/qapi-schema/redefined-builtin.err
tests/qapi-schema/redefined-builtin.exit [deleted file]
tests/qapi-schema/redefined-command.err
tests/qapi-schema/redefined-command.exit [deleted file]
tests/qapi-schema/redefined-event.err
tests/qapi-schema/redefined-event.exit [deleted file]
tests/qapi-schema/redefined-type.err
tests/qapi-schema/redefined-type.exit [deleted file]
tests/qapi-schema/reserved-command-q.err
tests/qapi-schema/reserved-command-q.exit [deleted file]
tests/qapi-schema/reserved-enum-q.err
tests/qapi-schema/reserved-enum-q.exit [deleted file]
tests/qapi-schema/reserved-member-has.err
tests/qapi-schema/reserved-member-has.exit [deleted file]
tests/qapi-schema/reserved-member-q.err
tests/qapi-schema/reserved-member-q.exit [deleted file]
tests/qapi-schema/reserved-member-u.err
tests/qapi-schema/reserved-member-u.exit [deleted file]
tests/qapi-schema/reserved-member-underscore.err
tests/qapi-schema/reserved-member-underscore.exit [deleted file]
tests/qapi-schema/reserved-type-kind.err
tests/qapi-schema/reserved-type-kind.exit [deleted file]
tests/qapi-schema/reserved-type-list.err
tests/qapi-schema/reserved-type-list.exit [deleted file]
tests/qapi-schema/returns-alternate.err
tests/qapi-schema/returns-alternate.exit [deleted file]
tests/qapi-schema/returns-array-bad.err
tests/qapi-schema/returns-array-bad.exit [deleted file]
tests/qapi-schema/returns-dict.err
tests/qapi-schema/returns-dict.exit [deleted file]
tests/qapi-schema/returns-unknown.err
tests/qapi-schema/returns-unknown.exit [deleted file]
tests/qapi-schema/returns-whitelist.err
tests/qapi-schema/returns-whitelist.exit [deleted file]
tests/qapi-schema/string-code-point-127.err
tests/qapi-schema/string-code-point-127.exit [deleted file]
tests/qapi-schema/string-code-point-31.err
tests/qapi-schema/string-code-point-31.exit [deleted file]
tests/qapi-schema/struct-base-clash-deep.err
tests/qapi-schema/struct-base-clash-deep.exit [deleted file]
tests/qapi-schema/struct-base-clash.err
tests/qapi-schema/struct-base-clash.exit [deleted file]
tests/qapi-schema/struct-data-invalid.err
tests/qapi-schema/struct-data-invalid.exit [deleted file]
tests/qapi-schema/struct-member-if-invalid.err
tests/qapi-schema/struct-member-if-invalid.exit [deleted file]
tests/qapi-schema/struct-member-invalid-dict.err
tests/qapi-schema/struct-member-invalid-dict.exit [deleted file]
tests/qapi-schema/struct-member-invalid.err
tests/qapi-schema/struct-member-invalid.exit [deleted file]
tests/qapi-schema/test-qapi.py [changed mode: 0644->0755]
tests/qapi-schema/trailing-comma-list.err
tests/qapi-schema/trailing-comma-list.exit [deleted file]
tests/qapi-schema/trailing-comma-object.err
tests/qapi-schema/trailing-comma-object.exit [deleted file]
tests/qapi-schema/type-bypass-bad-gen.err
tests/qapi-schema/type-bypass-bad-gen.exit [deleted file]
tests/qapi-schema/unclosed-list.err
tests/qapi-schema/unclosed-list.exit [deleted file]
tests/qapi-schema/unclosed-object.err
tests/qapi-schema/unclosed-object.exit [deleted file]
tests/qapi-schema/unclosed-string.err
tests/qapi-schema/unclosed-string.exit [deleted file]
tests/qapi-schema/union-base-empty.err
tests/qapi-schema/union-base-empty.exit [deleted file]
tests/qapi-schema/union-base-no-discriminator.err
tests/qapi-schema/union-base-no-discriminator.exit [deleted file]
tests/qapi-schema/union-branch-case.err
tests/qapi-schema/union-branch-case.exit [deleted file]
tests/qapi-schema/union-branch-if-invalid.err
tests/qapi-schema/union-branch-if-invalid.exit [deleted file]
tests/qapi-schema/union-branch-invalid-dict.err
tests/qapi-schema/union-branch-invalid-dict.exit [deleted file]
tests/qapi-schema/union-clash-branches.err
tests/qapi-schema/union-clash-branches.exit [deleted file]
tests/qapi-schema/union-empty.err
tests/qapi-schema/union-empty.exit [deleted file]
tests/qapi-schema/union-invalid-base.err
tests/qapi-schema/union-invalid-base.exit [deleted file]
tests/qapi-schema/union-optional-branch.err
tests/qapi-schema/union-optional-branch.exit [deleted file]
tests/qapi-schema/union-unknown.err
tests/qapi-schema/union-unknown.exit [deleted file]
tests/qapi-schema/unknown-escape.err
tests/qapi-schema/unknown-escape.exit [deleted file]
tests/qapi-schema/unknown-expr-key.err
tests/qapi-schema/unknown-expr-key.exit [deleted file]