1 .. _tests-integration-testing-teuthology-intro:
3 Testing - Integration Tests - Introduction
4 ==========================================
6 Ceph has two types of tests: :ref:`make check <make-check>` tests and
7 integration tests. When a test requires multiple machines, root access, or lasts
8 for a long time (for example, to simulate a realistic Ceph workload), it is
9 deemed to be an integration test. Integration tests are organized into "suites",
10 which are defined in the `ceph/qa sub-directory`_ and run with the
11 ``teuthology-suite`` command.
13 The ``teuthology-suite`` command is part of the `teuthology framework`_.
14 In the sections that follow we attempt to provide a detailed introduction
15 to that framework from the perspective of a beginning Ceph developer.
17 Teuthology consumes packages
18 ----------------------------
20 It may take some time to understand the significance of this fact, but it
21 is `very` significant. It means that automated tests can be conducted on
22 multiple platforms using the same packages (RPM, DEB) that can be
23 installed on any machine running those platforms.
25 Teuthology has a `list of platforms that it supports
26 <https://github.com/ceph/ceph/tree/master/qa/distros/supported>`_ (as of
27 September 2020 the list consisted of "RHEL/CentOS 8" and "Ubuntu 18.04"). It
28 expects to be provided pre-built Ceph packages for these platforms. Teuthology
29 deploys these platforms on machines (bare-metal or cloud-provisioned), installs
30 the packages on them, and deploys Ceph clusters on them - all as called for by
36 A number of integration tests are run on a regular basis in the `Sepia
37 lab`_ against the official Ceph repositories (on the ``master`` development
38 branch and the stable branches). Traditionally, these tests are called "the
39 nightlies" because the Ceph core developers used to live and work in
40 the same time zone and from their perspective the tests were run overnight.
42 The results of nightly test runs are published at http://pulpito.ceph.com/
43 under the user ``teuthology``. The developer nick appears in URL of the the
44 test results and in the first column of the Pulpito dashboard. The results are
45 also reported on the `ceph-qa mailing list <https://ceph.com/irc/>`_.
50 In brief: in the ``teuthology-suite`` command option ``-p <N>``, set the value of ``<N>`` to a number lower than 1000. An explanation of why follows.
52 The ``teuthology-suite`` command includes an option ``-p <N>``. This option specifies the priority of the jobs submitted to the queue. The lower the value of ``N``, the higher the priority.
54 The default value of ``N`` is ``1000``. This is the same priority value given to the nightly tests (the nightlies). Often, the volume of testing done during the nightly tests is so great that the full number of nightly tests do not get run during the time allotted for their run.
56 Set the value of ``N`` lower than ``1000``, or your tests will not have priority over the nightly tests. This means that they might never run.
58 Select your job's priority (the value of ``N``) in accordance with the following guidelines:
67 - Use this if the sky is falling and some group of tests must be run ASAP.
69 - Use this if your tests are urgent and blocking other important
72 - Use this if you are testing a particular feature/fix and running fewer
73 than about 25 jobs. This range is also used for urgent release testing.
75 - Tech Leads regularly schedule integration tests with this priority to
76 verify pull requests against master.
77 * - **100 <= N < 150**
78 - This priority is used for QE validation of point releases.
79 * - **150 <= N < 200**
80 - Use this priority for 100 jobs or fewer that test a particular feature
81 or fix. Results are available in about 24 hours.
82 * - **200 <= N < 1000**
83 - Use this priority for large test runs. Results are available in about a
86 To see how many jobs the ``teuthology-suite`` command will trigger, use the
87 ``--dry-run`` flag. If you are happy with the number of jobs returned by the
88 dry run, issue the ``teuthology-suite`` command again without ``--dry-run`` and
89 with ``-p`` and an appropriate number as an argument.
91 To skip the priority check, use ``--force-priority``. Be considerate of the needs of other developers to run tests, and use ``--force-priority`` only in emergencies.
96 The ``suites`` directory of the `ceph/qa sub-directory`_ contains all the
97 integration tests for all the Ceph components.
99 .. list-table:: **Suites**
104 * - `ceph-deploy <https://github.com/ceph/ceph/tree/master/qa/suites/ceph-deploy>`_
105 - install a Ceph cluster with ``ceph-deploy`` (`ceph-deploy man page`_)
107 * - `dummy <https://github.com/ceph/ceph/tree/master/qa/suites/dummy>`_
108 - get a machine, do nothing and return success (commonly used to verify
109 that the integration testing infrastructure works as expected)
111 * - `fs <https://github.com/ceph/ceph/tree/master/qa/suites/fs>`_
112 - test CephFS mounted using kernel and FUSE clients, also with multiple MDSs.
114 * - `krbd <https://github.com/ceph/ceph/tree/master/qa/suites/krbd>`_
115 - test the RBD kernel module
117 * - `powercycle <https://github.com/ceph/ceph/tree/master/qa/suites/powercycle>`_
118 - verify the Ceph cluster behaves when machines are powered off and on
121 * - `rados <https://github.com/ceph/ceph/tree/master/qa/suites/rados>`_
122 - run Ceph clusters including OSDs and MONs, under various conditions of
125 * - `rbd <https://github.com/ceph/ceph/tree/master/qa/suites/rbd>`_
126 - run RBD tests using actual Ceph clusters, with and without qemu
128 * - `rgw <https://github.com/ceph/ceph/tree/master/qa/suites/rgw>`_
129 - run RGW tests using actual Ceph clusters
131 * - `smoke <https://github.com/ceph/ceph/tree/master/qa/suites/smoke>`_
132 - run tests that exercise the Ceph API with an actual Ceph cluster
134 * - `teuthology <https://github.com/ceph/ceph/tree/master/qa/suites/teuthology>`_
135 - verify that teuthology can run integration tests, with and without OpenStack
137 * - `upgrade <https://github.com/ceph/ceph/tree/master/qa/suites/upgrade>`_
138 - for various versions of Ceph, verify that upgrades can happen without disrupting an ongoing workload (`Upgrade Testing`_)
143 ``teuthology-describe`` was added to the `teuthology framework`_ to facilitate
144 documentation and better understanding of integration tests.
146 Tests can be documented by embedding ``meta:`` annotations in the yaml files
147 used to define the tests. The results can be seen in the `teuthology-describe
150 Since this is a new feature, many yaml files have yet to be annotated.
151 Developers are encouraged to improve the coverage and the quality of the
154 How to run integration tests
155 ----------------------------
157 Typically, the `Sepia lab`_ is used to run integration tests. But as a new Ceph
158 developer, you will probably not have access to the `Sepia lab`_. You might
159 however be able to run some integration tests in an environment separate from
160 the `Sepia lab`_ . Ask members from the relevant team how to do this.
162 One way to run your own integration tests is to set up a teuthology cluster on
163 bare metal. Setting up a teuthology cluster on bare metal is a complex task.
165 <https://docs.ceph.com/projects/teuthology/en/latest/LAB_SETUP.html>`_ to get
166 you started if you decide that you are interested in undertaking the complex
167 task of setting up a teuthology cluster on bare metal.
169 Running integration tests on your code contributions and publishing the results
170 allows reviewers to verify that changes to the code base do not cause
171 regressions, and allows reviewers to analyze test failures when they occur.
173 Every teuthology cluster, whether bare-metal or cloud-provisioned, has a
174 so-called "teuthology machine" from which tests suites are triggered using the
175 ``teuthology-suite`` command.
177 A detailed and up-to-date description of each `teuthology-suite`_ option is
178 available by running the following command on the teuthology machine:
182 teuthology-suite --help
184 .. _teuthology-suite: https://docs.ceph.com/projects/teuthology/en/latest/commands/teuthology-suite.html
186 How integration tests are defined
187 ---------------------------------
189 Integration tests are defined by yaml files found in the ``suites``
190 subdirectory of the `ceph/qa sub-directory`_ and implemented by python
191 code found in the ``tasks`` subdirectory. Some tests ("standalone tests")
192 are defined in a single yaml file, while other tests are defined by a
193 directory tree containing yaml files that are combined, at runtime, into a
197 .. _reading-standalone-test:
199 Reading a standalone test
200 -------------------------
202 Let us first examine a standalone test, or "singleton".
204 Here is a commented example using the integration test
205 `rados/singleton/all/admin-socket.yaml
206 <https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/admin-socket.yaml>`_
223 config set filestore_dump_file /tmp/foo:
227 The ``roles`` array determines the composition of the cluster (how
228 many MONs, OSDs, etc.) on which this test is designed to run, as well
229 as how these roles will be distributed over the machines in the
230 testing cluster. In this case, there is only one element in the
231 top-level array: therefore, only one machine is allocated to the
232 test. The nested array declares that this machine shall run a MON with
233 id ``a`` (that is the ``mon.a`` in the list of roles) and two OSDs
234 (``osd.0`` and ``osd.1``).
236 The body of the test is in the ``tasks`` array: each element is
237 evaluated in order, causing the corresponding python file found in the
238 ``tasks`` subdirectory of the `teuthology repository`_ or
239 `ceph/qa sub-directory`_ to be run. "Running" in this case means calling
240 the ``task()`` function defined in that file.
242 In this case, the `install
243 <https://github.com/ceph/teuthology/blob/master/teuthology/task/install/__init__.py>`_
244 task comes first. It installs the Ceph packages on each machine (as
245 defined by the ``roles`` array). A full description of the ``install``
246 task is `found in the python file
247 <https://github.com/ceph/teuthology/blob/master/teuthology/task/install/__init__.py>`_
248 (search for "def task").
250 The ``ceph`` task, which is documented `here
251 <https://github.com/ceph/ceph/blob/master/qa/tasks/ceph.py>`__ (again,
252 search for "def task"), starts OSDs and MONs (and possibly MDSs as well)
253 as required by the ``roles`` array. In this example, it will start one MON
254 (``mon.a``) and two OSDs (``osd.0`` and ``osd.1``), all on the same
255 machine. Control moves to the next task when the Ceph cluster reaches
258 The next task is ``admin_socket`` (`source code
259 <https://github.com/ceph/ceph/blob/master/qa/tasks/admin_socket.py>`_).
260 The parameter of the ``admin_socket`` task (and any other task) is a
261 structure which is interpreted as documented in the task. In this example
262 the parameter is a set of commands to be sent to the admin socket of
263 ``osd.0``. The task verifies that each of them returns on success (i.e.
266 This test can be run with
270 teuthology-suite --machine-type smithi --suite rados/singleton/all/admin-socket.yaml fs/ext4.yaml
275 Each test has a "test description", which is similar to a directory path,
276 but not the same. In the case of a standalone test, like the one in
277 `Reading a standalone test`_, the test description is identical to the
278 relative path (starting from the ``suites/`` directory of the
279 `ceph/qa sub-directory`_) of the yaml file defining the test.
281 Much more commonly, tests are defined not by a single yaml file, but by a
282 `directory tree of yaml files`. At runtime, the tree is walked and all yaml
283 files (facets) are combined into larger yaml "programs" that define the
284 tests. A full listing of the yaml defining the test is included at the
285 beginning of every test log.
287 In these cases, the description of each test consists of the
288 subdirectory under `suites/
289 <https://github.com/ceph/ceph/tree/master/qa/suites>`_ containing the
290 yaml facets, followed by an expression in curly braces (``{}``) consisting of
291 a list of yaml facets in order of concatenation. For instance the
294 ceph-deploy/basic/{distros/centos_7.0.yaml tasks/ceph-deploy.yaml}
296 signifies the concatenation of two files:
298 * ceph-deploy/basic/distros/centos_7.0.yaml
299 * ceph-deploy/basic/tasks/ceph-deploy.yaml
301 How tests are built from directories
302 ------------------------------------
304 As noted in the previous section, most tests are not defined in a single
305 yaml file, but rather as a `combination` of files collected from a
306 directory tree within the ``suites/`` subdirectory of the `ceph/qa sub-directory`_.
308 The set of all tests defined by a given subdirectory of ``suites/`` is
309 called an "integration test suite", or a "teuthology suite".
311 Combination of yaml facets is controlled by special files (``%`` and
312 ``+``) that are placed within the directory tree and can be thought of as
313 operators. The ``%`` file is the "convolution" operator and ``+``
314 signifies concatenation.
319 The convolution operator, implemented as a (typically empty) file called ``%``,
320 tells teuthology to construct a test matrix from yaml facets found in
321 subdirectories below the directory containing the operator.
323 For example, the `ceph-deploy suite
324 <https://github.com/ceph/ceph/tree/master/qa/suites/ceph-deploy/>`_ is
325 defined by the ``suites/ceph-deploy/`` tree, which consists of the files and
326 subdirectories in the following structure
330 qa/suites/ceph-deploy
333 │ ├── centos_latest.yaml
334 │ └── ubuntu_latest.yaml
336 ├── ceph-admin-commands.yaml
337 └── rbd_import_export.yaml
339 This is interpreted as a 2x1 matrix consisting of two tests:
341 1. ceph-deploy/basic/{distros/centos_7.0.yaml tasks/ceph-deploy.yaml}
342 2. ceph-deploy/basic/{distros/ubuntu_16.04.yaml tasks/ceph-deploy.yaml}
344 i.e. the concatenation of centos_7.0.yaml and ceph-deploy.yaml and
345 the concatenation of ubuntu_16.04.yaml and ceph-deploy.yaml, respectively.
346 In human terms, this means that the task found in ``ceph-deploy.yaml`` is
347 intended to run on both CentOS 7.0 and Ubuntu 16.04.
349 Without the file percent, the ``ceph-deploy`` tree would be interpreted as
350 three standalone tests:
352 * ceph-deploy/basic/distros/centos_7.0.yaml
353 * ceph-deploy/basic/distros/ubuntu_16.04.yaml
354 * ceph-deploy/basic/tasks/ceph-deploy.yaml
356 (which would of course be wrong in this case).
358 Referring to the `ceph/qa sub-directory`_, you will notice that the
359 ``centos_7.0.yaml`` and ``ubuntu_16.04.yaml`` files in the
360 ``suites/ceph-deploy/basic/distros/`` directory are implemented as symlinks.
361 By using symlinks instead of copying, a single file can appear in multiple
362 suites. This eases the maintenance of the test framework as a whole.
364 All the tests generated from the ``suites/ceph-deploy/`` directory tree
365 (also known as the "ceph-deploy suite") can be run with
369 teuthology-suite --machine-type smithi --suite ceph-deploy
371 An individual test from the `ceph-deploy suite`_ can be run by adding the
377 --machine-type smithi \
378 --suite ceph-deploy/basic \
379 --filter 'ceph-deploy/basic/{distros/ubuntu_16.04.yaml tasks/ceph-deploy.yaml}'
381 .. note:: To run a standalone test like the one in `Reading a standalone
382 test`_, ``--suite`` alone is sufficient. If you want to run a single
383 test from a suite that is defined as a directory tree, ``--suite`` must
384 be combined with ``--filter``. This is because the ``--suite`` option
385 understands POSIX relative paths only.
390 Suites can get quite large with the combinatorial explosion of yaml
391 configurations. At the time of writing, the ``rados``` suite is more than
392 100,000 jobs. For this reason, scheduling often uses the ``--subset`` option to
393 only run a subset of the jobs (see also: :ref:`subset`). However, this applies
394 only at the top-level of the suite being run (e.g. ``fs``). That may
395 incidentally inflate the ratio of jobs for some larger sub-suites (like
396 ``fs:workload``) vs. smaller but critical suites (like ``fs:volumes``).
398 It is therefore attractive to automatically subset some sub-suites which are
399 never run fully. This is done by providing an integer divisor for the ``%``
400 convolution operator file instead of leaving it empty. That divisor
401 automatically subsets the resulting matrix. For example, if the convolution
402 file ``%`` contains ``2``, the matrix will be divided into two using the same
403 logic as the ``--subset`` mechanism.
405 Note the numerator is not specified as with the ``--subset`` option as there is
406 no meaningful way to express this when there could be several layers of
407 nesting. Instead, a random subset is selected (1 of 2 in our example). The
408 choice is based off the random seed (``--seed``) used for the scheduling.
409 Remember that seed is saved in the results so that a ``--rerun`` of failed
410 tests will still preserve the correct numerator (subset of subsets).
412 You can disable nested subsets using the ``--no-nested-subset`` argument to
413 ``teuthology-suite``.
415 Concatenation operator
416 ^^^^^^^^^^^^^^^^^^^^^^
418 For even greater flexibility in sharing yaml files between suites, the
419 special file plus (``+``) can be used to concatenate files within a
420 directory. For instance, consider the `suites/rbd/thrash
421 <https://github.com/ceph/ceph/tree/master/qa/suites/rbd/thrash>`_
433 ├── rbd_api_tests_copy_on_read.yaml
434 ├── rbd_api_tests.yaml
435 └── rbd_fsx_rate_limit.yaml
437 This creates two tests:
439 * rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml}
440 * rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests.yaml}
442 Because the ``clusters/`` subdirectory contains the special file plus
443 (``+``), all the other files in that subdirectory (``fixed-2.yaml`` and
444 ``openstack.yaml`` in this case) are concatenated together
445 and treated as a single file. Without the special file plus, they would
446 have been convolved with the files from the workloads directory to create
449 * rbd/thrash/{clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml}
450 * rbd/thrash/{clusters/openstack.yaml workloads/rbd_api_tests.yaml}
451 * rbd/thrash/{clusters/fixed-2.yaml workloads/rbd_api_tests_copy_on_read.yaml}
452 * rbd/thrash/{clusters/fixed-2.yaml workloads/rbd_api_tests.yaml}
454 The ``clusters/fixed-2.yaml`` file is shared among many suites to
455 define the following ``roles``
460 - [mon.a, mon.c, osd.0, osd.1, osd.2, client.0]
461 - [mon.b, osd.3, osd.4, osd.5, client.1]
463 The ``rbd/thrash`` suite as defined above, consisting of two tests,
468 teuthology-suite --machine-type smithi --suite rbd/thrash
470 A single test from the rbd/thrash suite can be run by adding the
476 --machine-type smithi \
478 --filter 'rbd/thrash/{clusters/fixed-2.yaml clusters/openstack.yaml workloads/rbd_api_tests_copy_on_read.yaml}'
485 Using the upgrade suite we are able to verify that upgrades from earlier releases can complete
486 successfully without disrupting any ongoing workload.
487 Each Release branch upgrade directory includes 2-x upgrade testing.
488 Meaning, we are able to test the upgrade from 2 preceding releases to the current one.
489 The upgrade sequence is done in `parallel <https://github.com/ceph/teuthology/blob/main/teuthology/task/parallel.py>`_
490 with other given workloads.
492 For instance, the upgrade test directory from the Quincy release branch is as follows:
500 It is possible to test upgrades from Octopus (2-x) or from Pacific (1-x) to Quincy (x).
501 A simple upgrade test consists the following order:
507 ├── upgrade-sequence.yaml
510 After starting the cluster with the older release we begin running the given ``workload``
511 and the ``upgrade-sequnce`` in parallel.
515 - print: "**** done start parallel"
519 - print: "**** done end parallel"
521 While the ``workload`` directory consists regular yaml files just as in any other suite,
522 the ``upgrade-sequnce`` is resposible for running the upgrade and awaitng its completion:
526 - print: "**** done start upgrade, wait"
529 - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1
530 - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; sleep 30 ; done\
532 - print: "**** done end upgrade, wait..."
535 It is also possible to upgrade in stages while running workloads in between those:
542 ├── 2-partial-upgrade
545 ├── 5-finish-upgrade.yaml
549 After starting a cluster we upgrade only 2/3 of the cluster (``2-partial-upgrade``).
550 The next stage is running thrash tests and given workload tests. Later on, continuing to upgrade the
551 rest of the cluster (``5-finish-upgrade.yaml``).
552 The last stage is requiring the updated release (``ceph require-osd-release quincy``,
553 ``ceph osd set-require-min-compat-client quincy``) and running the ``final-workload``.
555 Position Independent Linking
556 ----------------------------
558 Under the ``qa/suites`` directory are ``.qa`` symbolic links in every
559 directory. Each link is recursive by always linking to ``../.qa/``. The final
560 terminating link is in the ``qa/`` directory itself as ``qa/.qa -> .``. This
561 layout of symbolic links allows a suite to be easily copied or moved without
562 breaking a number of symbolic links. For example::
564 qa/suites/fs/upgrade/nofs/centos_latest.yaml -> .qa/distros/supported/centos_latest.yaml
566 If we copy the ``nofs`` suite somewhere else, add a parent directory above
567 ``nofs``, or move the ``centos_latest.yaml`` fragment into a sub-directory, the
568 link will not break. Compare to::
570 qa/suites/fs/upgrade/nofs/centos_latest.yaml -> ../../../../distros/supported/centos_latest.yaml
572 If the link is moved, it is very likely it will break because the number of
573 parent directories to reach the ``distros`` directory may change.
575 When adding new directories or suites, it is recommended to also remember
576 adding ``.qa`` symbolic links. A trivial find command may do this for you:
580 find qa/suites/ -type d -execdir ln -sfT ../.qa/ {}/.qa \;
583 Filtering tests by their description
584 ------------------------------------
586 When a few jobs fail and need to be run again, the ``--filter`` option
587 can be used to select tests with a matching description. For instance, if the
588 ``rados`` suite fails the `all/peer.yaml <https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/peer.yaml>`_ test, the following will only
589 run the tests that contain this file
593 teuthology-suite --machine-type smithi --suite rados --filter all/peer.yaml
595 The ``--filter-out`` option does the opposite (it matches tests that do `not`
596 contain a given string), and can be combined with the ``--filter`` option.
598 Both ``--filter`` and ``--filter-out`` take a comma-separated list of strings
599 (which means the comma character is implicitly forbidden in filenames found in
600 the `ceph/qa sub-directory`_). For instance
604 teuthology-suite --machine-type smithi --suite rados --filter all/peer.yaml,all/rest-api.yaml
606 will run tests that contain either
607 `all/peer.yaml <https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/peer.yaml>`_
609 `all/rest-api.yaml <https://github.com/ceph/ceph/blob/master/qa/suites/rados/singleton/all/rest-api.yaml>`_
611 Each string is looked up anywhere in the test description and has to
612 be an exact match: they are not regular expressions.
617 Reducing the number of tests
618 ----------------------------
620 The ``rados`` suite generates tens or even hundreds of thousands of tests out
621 of a few hundred files. This happens because teuthology constructs test
622 matrices from subdirectories wherever it encounters a file named ``%``. For
623 instance, all tests in the `rados/basic suite
624 <https://github.com/ceph/ceph/tree/master/qa/suites/rados/basic>`_ run with
625 different messenger types: ``simple``, ``async`` and ``random``, because they
626 are combined (via the special file ``%``) with the `msgr directory
627 <https://github.com/ceph/ceph/tree/master/qa/suites/rados/basic/msgr>`_
629 All integration tests are required to be run before a Ceph release is
630 published. When merely verifying whether a contribution can be merged without
631 risking a trivial regression, it is enough to run a subset. The ``--subset``
632 option can be used to reduce the number of tests that are triggered. For
637 teuthology-suite --machine-type smithi --suite rados --subset 0/4000
639 will run as few tests as possible. The tradeoff in this case is that
640 not all combinations of test variations will together,
641 but no matter how small a ratio is provided in the ``--subset``,
642 teuthology will still ensure that all files in the suite are in at
643 least one test. Understanding the actual logic that drives this
644 requires reading the teuthology source code.
646 Note: some suites are now using a **nested subset** feature that automatically
647 applies a subset to a carefully chosen set of YAML configurations. You may
648 disable this behavior (for some custom filtering, perhaps) using the
649 ``--no-nested-subset`` option.
651 The ``--limit`` option only runs the first ``N`` tests in the suite:
652 this is rarely useful, however, because there is no way to control which
655 .. _ceph/qa sub-directory: https://github.com/ceph/ceph/tree/master/qa
656 .. _Sepia Lab: https://wiki.sepia.ceph.com/doku.php
657 .. _teuthology repository: https://github.com/ceph/teuthology
658 .. _teuthology framework: https://github.com/ceph/teuthology
659 .. _teuthology-describe usecases: https://gist.github.com/jdurgin/09711d5923b583f60afc
660 .. _ceph-deploy man page: ../../../../man/8/ceph-deploy