src/compiler-rt/docs/TestingGuide.rst

   1 ========================================
   2 Compiler-rt Testing Infrastructure Guide
   3 ========================================
   4
   5 .. contents::
   6    :local:
   7
   8 Overview
   9 ========
  10
  11 This document is the reference manual for the compiler-rt modifications to the
  12 testing infrastructure. Documentation for the infrastructure itself can be found at
  13 :ref:`llvm_testing_guide`.
  14
  15 LLVM testing infrastructure organization
  16 ========================================
  17
  18 The compiler-rt testing infrastructure contains regression tests which are run
  19 as part of the usual ``make check-all`` and are expected to always pass -- they
  20 should be run before every commit.
  21
  22 Quick start
  23 ===========
  24
  25 The regressions tests are in the "compiler-rt" module and are normally checked
  26 out in the directory ``llvm/projects/compiler-rt/test``. Use ``make check-all``
  27 to run the regression tests after building compiler-rt.
  28
  29 REQUIRES, XFAIL, etc.
  30 ---------------------
  31
  32 Sometimes it is necessary to restrict a test to a specific target or mark it as
  33 an "expected fail" or XFAIL. This is normally achieved using ``REQUIRES:`` or
  34 ``XFAIL:`` with a substring of LLVM's default target triple. Unfortunately, the
  35 behaviour of this is somewhat quirky in compiler-rt. There are two main
  36 pitfalls to avoid.
  37
  38 The first pitfall is that these directives perform a substring match on the
  39 triple and as such ``XFAIL: mips`` affects more triples than expected. For
  40 example, ``mips-linux-gnu``, ``mipsel-linux-gnu``, ``mips64-linux-gnu``, and
  41 ``mips64el-linux-gnu`` will all match a ``XFAIL: mips`` directive. Including a
  42 trailing ``-`` such as in ``XFAIL: mips-`` can help to mitigate this quirk but
  43 even that has issues as described below.
  44
  45 The second pitfall is that the default target triple is often inappropriate for
  46 compiler-rt tests since compiler-rt tests may be compiled for multiple targets.
  47 For example, a typical build on an ``x86_64-linux-gnu`` host will often run the
  48 tests for both x86_64 and i386. In this situation ``XFAIL: x86_64`` will mark
  49 both the x86_64 and i386 tests as an expected failure while ``XFAIL: i386``
  50 will have no effect at all.
  51
  52 To remedy both pitfalls, compiler-rt tests provide a feature string which can
  53 be used to specify a single target. This string is of the form
  54 ``target-is-${arch}`` where ``${arch}}`` is one of the values from the
  55 following lines of the CMake output::
  56
  57   -- Compiler-RT supported architectures: x86_64;i386
  58   -- Builtin supported architectures: i386;x86_64
  59
  60 So for example ``XFAIL: target-is-x86_64`` will mark a test as expected to fail
  61 on x86_64 without also affecting the i386 test and ``XFAIL: target-is-i386``
  62 will mark a test as expected to fail on i386 even if the default target triple
  63 is ``x86_64-linux-gnu``. Directives that use these ``target-is-${arch}`` string
  64 require exact matches so ``XFAIL: target-is-mips``,
  65 ``XFAIL: target-is-mipsel``, ``XFAIL: target-is-mips64``, and
  66 ``XFAIL: target-is-mips64el`` all refer to different MIPS targets.