ceph/doc/dev/developer_guide/running-tests-in-cloud.rst

   1 Running Tests in the Cloud
   2 ==========================
   3
   4 In this chapter, we will explain in detail how use an OpenStack
   5 tenant as an environment for Ceph `integration testing`_.
   6
   7 Assumptions and caveat
   8 ----------------------
   9
  10 We assume that:
  11
  12 1. you are the only person using the tenant
  13 2. you have the credentials
  14 3. the tenant supports the ``nova`` and ``cinder`` APIs
  15
  16 Caveat: be aware that, as of this writing (July 2016), testing in
  17 OpenStack clouds is a new feature. Things may not work as advertised.
  18 If you run into trouble, ask for help on `IRC`_ or the `Mailing list`_, or
  19 open a bug report at the `ceph-workbench bug tracker`_.
  20
  21 .. _`ceph-workbench bug tracker`: http://ceph-workbench.dachary.org/root/ceph-workbench/issues
  22
  23 Prepare tenant
  24 --------------
  25
  26 If you have not tried to use ``ceph-workbench`` with this tenant before,
  27 proceed to the next step.
  28
  29 To start with a clean slate, login to your tenant via the Horizon dashboard
  30 and:
  31
  32 * terminate the ``teuthology`` and ``packages-repository`` instances, if any
  33 * delete the ``teuthology`` and ``teuthology-worker`` security groups, if any
  34 * delete the ``teuthology`` and ``teuthology-myself`` key pairs, if any
  35
  36 Also do the above if you ever get key-related errors ("invalid key", etc.)
  37 when trying to schedule suites.
  38
  39 Getting ceph-workbench
  40 ----------------------
  41
  42 Since testing in the cloud is done using the ``ceph-workbench ceph-qa-suite``
  43 tool, you will need to install that first. It is designed
  44 to be installed via Docker, so if you don't have Docker running on your
  45 development machine, take care of that first. You can follow `the official
  46 tutorial <https://docs.docker.com/engine/installation/>`_ to install if
  47 you have not installed yet.
  48
  49 Once Docker is up and running, install ``ceph-workbench`` by following the
  50 `Installation instructions in the ceph-workbench documentation
  51 <http://ceph-workbench.readthedocs.io/en/latest/#installation>`_.
  52
  53 Linking ceph-workbench with your OpenStack tenant
  54 -------------------------------------------------
  55
  56 Before you can trigger your first teuthology suite, you will need to link
  57 ``ceph-workbench`` with your OpenStack account.
  58
  59 First, download a ``openrc.sh`` file by clicking on the "Download OpenStack
  60 RC File" button, which can be found in the "API Access" tab of the "Access
  61 & Security" dialog of the OpenStack Horizon dashboard.
  62
  63 Second, create a ``~/.ceph-workbench`` directory, set its permissions to
  64 700, and move the ``openrc.sh`` file into it. Make sure that the filename
  65 is exactly ``~/.ceph-workbench/openrc.sh``.
  66
  67 Third, edit the file so it does not ask for your OpenStack password
  68 interactively. Comment out the relevant lines and replace them with
  69 something like::
  70
  71 .. prompt:: bash $
  72
  73     export OS_PASSWORD="aiVeth0aejee3eep8rogho3eep7Pha6ek"
  74
  75 When ``ceph-workbench ceph-qa-suite`` connects to your OpenStack tenant for
  76 the first time, it will generate two keypairs: ``teuthology-myself`` and
  77 ``teuthology``.
  78
  79 .. If this is not the first time you have tried to use
  80 .. ``ceph-workbench ceph-qa-suite`` with this tenant, make sure to delete any
  81 .. stale keypairs with these names!
  82
  83 Run the dummy suite
  84 -------------------
  85
  86 You are now ready to take your OpenStack teuthology setup for a test
  87 drive
  88
  89 .. prompt:: bash $
  90
  91    ceph-workbench ceph-qa-suite --suite dummy
  92
  93 Be forewarned that the first run of ``ceph-workbench ceph-qa-suite`` on a
  94 pristine tenant will take a long time to complete because it downloads a VM
  95 image and during this time the command may not produce any output.
  96
  97 The images are cached in OpenStack, so they are only downloaded once.
  98 Subsequent runs of the same command will complete faster.
  99
 100 Although ``dummy`` suite does not run any tests, in all other respects it
 101 behaves just like a teuthology suite and produces some of the same
 102 artifacts.
 103
 104 The last bit of output should look something like this::
 105
 106   pulpito web interface: http://149.202.168.201:8081/
 107   ssh access           : ssh -i /home/smithfarm/.ceph-workbench/teuthology-myself.pem ubuntu@149.202.168.201 # logs in /usr/share/nginx/html
 108
 109 What this means is that ``ceph-workbench ceph-qa-suite`` triggered the test
 110 suite run. It does not mean that the suite run has completed. To monitor
 111 progress of the run, check the Pulpito web interface URL periodically, or
 112 if you are impatient, ssh to the teuthology machine using the ssh command
 113 shown and do
 114
 115 .. prompt:: bash $
 116
 117    tail -f /var/log/teuthology.*
 118
 119 The `/usr/share/nginx/html` directory contains the complete logs of the
 120 test suite. If we had provided the ``--upload`` option to the
 121 ``ceph-workbench ceph-qa-suite`` command, these logs would have been
 122 uploaded to http://teuthology-logs.public.ceph.com.
 123
 124 Run a standalone test
 125 ---------------------
 126
 127 The standalone test explained in `Reading a standalone test`_ can be run
 128 with the following command
 129
 130 .. prompt:: bash $
 131
 132    ceph-workbench ceph-qa-suite --suite rados/singleton/all/admin-socket.yaml
 133
 134 This will run the suite shown on the current ``master`` branch of
 135 ``ceph/ceph.git``. You can specify a different branch with the ``--ceph``
 136 option, and even a different git repo with the ``--ceph-git-url`` option. (Run
 137 ``ceph-workbench ceph-qa-suite --help`` for an up-to-date list of available
 138 options.)
 139
 140 The first run of a suite will also take a long time, because ceph packages
 141 have to be built, first. Again, the packages so built are cached and
 142 ``ceph-workbench ceph-qa-suite`` will not build identical packages a second
 143 time.
 144
 145 Interrupt a running suite
 146 -------------------------
 147
 148 Teuthology suites take time to run. From time to time one may wish to
 149 interrupt a running suite. One obvious way to do this is::
 150
 151 .. prompt:: bash $
 152
 153    ceph-workbench ceph-qa-suite --teardown
 154
 155 This destroys all VMs created by ``ceph-workbench ceph-qa-suite`` and
 156 returns the OpenStack tenant to a "clean slate".
 157
 158 Sometimes you may wish to interrupt the running suite, but keep the logs,
 159 the teuthology VM, the packages-repository VM, etc. To do this, you can
 160 ``ssh`` to the teuthology VM (using the ``ssh access`` command reported
 161 when you triggered the suite -- see `Run the dummy suite`_) and, once
 162 there
 163
 164 .. prompt:: bash $
 165
 166    sudo /etc/init.d/teuthology restart
 167
 168 This will keep the teuthology machine, the logs and the packages-repository
 169 instance but nuke everything else.
 170
 171 Upload logs to archive server
 172 -----------------------------
 173
 174 Since the teuthology instance in OpenStack is only semi-permanent, with
 175 limited space for storing logs, ``teuthology-openstack`` provides an
 176 ``--upload`` option which, if included in the ``ceph-workbench ceph-qa-suite``
 177 command, will cause logs from all failed jobs to be uploaded to the log
 178 archive server maintained by the Ceph project. The logs will appear at the
 179 URL::
 180
 181     http://teuthology-logs.public.ceph.com/$RUN
 182
 183 where ``$RUN`` is the name of the run. It will be a string like this::
 184
 185     ubuntu-2016-07-23_16:08:12-rados-hammer-backports---basic-openstack
 186
 187 Even if you don't providing the ``--upload`` option, however, all the logs can
 188 still be found on the teuthology machine in the directory
 189 ``/usr/share/nginx/html``.
 190
 191 Provision VMs ad hoc
 192 --------------------
 193
 194 From the teuthology VM, it is possible to provision machines on an "ad hoc"
 195 basis, to use however you like. The magic incantation is::
 196
 197 .. prompt:: bash $
 198
 199     teuthology-lock --lock-many $NUMBER_OF_MACHINES \
 200         --os-type $OPERATING_SYSTEM \
 201         --os-version $OS_VERSION \
 202         --machine-type openstack \
 203         --owner $EMAIL_ADDRESS
 204
 205 The command must be issued from the ``~/teuthology`` directory. The possible
 206 values for ``OPERATING_SYSTEM`` AND ``OS_VERSION`` can be found by examining
 207 the contents of the directory ``teuthology/openstack/``. For example
 208
 209 .. prompt:: bash $
 210
 211     teuthology-lock --lock-many 1 --os-type ubuntu --os-version 16.04 \
 212         --machine-type openstack --owner foo@example.com
 213
 214 When you are finished with the machine, find it in the list of machines
 215
 216 .. prompt:: bash $
 217
 218     openstack server list
 219
 220 to determine the name or ID, and then terminate it with
 221
 222 .. prompt:: bash $
 223
 224     openstack server delete $NAME_OR_ID
 225
 226 Deploy a cluster for manual testing
 227 -----------------------------------
 228
 229 The `teuthology framework`_ and ``ceph-workbench ceph-qa-suite`` are
 230 versatile tools that automatically provision Ceph clusters in the cloud and
 231 run various tests on them in an automated fashion. This enables a single
 232 engineer, in a matter of hours, to perform thousands of tests that would
 233 keep dozens of human testers occupied for days or weeks if conducted
 234 manually.
 235
 236 However, there are times when the automated tests do not cover a particular
 237 scenario and manual testing is desired. It turns out that it is simple to
 238 adapt a test to stop and wait after the Ceph installation phase, and the
 239 engineer can then ssh into the running cluster. Simply add the following
 240 snippet in the desired place within the test YAML and schedule a run with the
 241 test::
 242
 243     tasks:
 244     - exec:
 245         client.0:
 246           - sleep 1000000000 # forever
 247
 248 (Make sure you have a ``client.0`` defined in your ``roles`` stanza or adapt
 249 accordingly.)
 250
 251 The same effect can be achieved using the ``interactive`` task::
 252
 253     tasks:
 254     - interactive
 255
 256 By following the test log, you can determine when the test cluster has entered
 257 the "sleep forever" condition. At that point, you can ssh to the teuthology
 258 machine and from there to one of the target VMs (OpenStack) or teuthology
 259 worker machines machine (Sepia) where the test cluster is running.
 260
 261 The VMs (or "instances" in OpenStack terminology) created by
 262 ``ceph-workbench ceph-qa-suite`` are named as follows:
 263
 264 ``teuthology`` - the teuthology machine
 265
 266 ``packages-repository`` - VM where packages are stored
 267
 268 ``ceph-*`` - VM where packages are built
 269
 270 ``target*`` - machines where tests are run
 271
 272 The VMs named ``target*`` are used by tests. If you are monitoring the
 273 teuthology log for a given test, the hostnames of these target machines can
 274 be found out by searching for the string ``Locked targets``::
 275
 276     2016-03-20T11:39:06.166 INFO:teuthology.task.internal:Locked targets:
 277       target149202171058.teuthology: null
 278       target149202171059.teuthology: null
 279
 280 The IP addresses of the target machines can be found by running ``openstack
 281 server list`` on the teuthology machine, but the target VM hostnames (e.g.
 282 ``target149202171058.teuthology``) are resolvable within the teuthology
 283 cluster.
 284
 285 .. _Integration testing: ../tests-integration-tests
 286 .. _IRC:  ../essentials/#irc
 287 .. _Mailing List: ../essentials/#mailing-list
 288 .. _Reading A Standalone Test: ../testing-integration-tests/#reading-a-standalone-test
 289 .. _teuthology framework: https://github.com/ceph/teuthology