1 Ceph Dashboard Developer Documentation
2 ======================================
4 .. contents:: Table of Contents
9 Before you can start the dashboard from within a development environment, you
10 will need to generate the frontend code and either use a compiled and running
11 Ceph cluster (e.g. started by ``vstart.sh``) or the standalone development web
14 The build process is based on `Node.js <https://nodejs.org/>`_ and requires the
15 `Node Package Manager <https://www.npmjs.com/>`_ ``npm`` to be installed.
20 * Node 8.9.0 or higher
24 During Ceph's build we create a virtualenv with ``node`` and ``npm``
25 installed, which can be used as an alternative to installing node/npm in your
28 If you want to use the node installed in the virtualenv you just need to
29 activate the virtualenv before you run any npm commands. To activate it run
30 ``. build/src/pybind/mgr/dashboard/node-env/bin/activate``.
32 Once you finish, you can simply run ``deactivate`` and exit the virtualenv.
35 If you do not have the `Angular CLI <https://github.com/angular/angular-cli>`_
36 installed globally, then you need to execute ``ng`` commands with an
37 additional ``npm run`` before it.
42 Run ``npm install`` in directory ``src/pybind/mgr/dashboard/frontend`` to
43 install the required packages locally.
45 Setting up a Development Server
46 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
48 Create the ``proxy.conf.json`` file based on ``proxy.conf.json.sample``.
50 Run ``npm start`` for a dev server.
51 Navigate to ``http://localhost:4200/``. The app will automatically
52 reload if you change any of the source files.
57 Run ``ng generate component component-name`` to generate a new
58 component. You can also use
59 ``ng generate directive|pipe|service|class|guard|interface|enum|module``.
64 Run ``npm run build`` to build the project. The build artifacts will be
65 stored in the ``dist/`` directory. Use the ``--prod`` flag for a
66 production build (``npm run build -- --prod``). Navigate to ``https://localhost:8443``.
68 Build the Code Documentation
69 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
71 Run ``npm run doc-build`` to generate code docs in the ``documentation/``
72 directory. To make them accesible locally for a web browser, run
73 ``npm run doc-serve`` and they will become available at ``http://localhost:8444``.
74 With ``npm run compodoc -- <opts>`` you may
75 `fully configure it <https://compodoc.app/guides/usage.html>`_.
77 Code linting and formatting
78 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
80 We use the following tools to lint and format the code in all our TS, SCSS and
83 - `codelyzer <http://codelyzer.com/>`_
84 - `html-linter <https://github.com/chinchiheather/html-linter>`_
85 - `Prettier <https://prettier.io/>`_
86 - `TSLint <https://palantir.github.io/tslint/>`_
88 We added 2 npm scripts to help run these tools:
90 - ``npm run lint``, will check frontend files against all linters
91 - ``npm run fix``, will try to fix all the detected linting errors
96 To write unit tests most efficient we have a small collection of tools,
97 we use within test suites.
99 Those tools can be found under
100 ``src/pybind/mgr/dashboard/frontend/src/testing/``, especially take
101 a look at ``unit-test-helper.ts``.
103 There you will be able to find:
105 ``configureTestBed`` that replaces the initial ``TestBed``
106 methods. It takes the same arguments as ``TestBed.configureTestingModule``.
107 Using it will run your tests a lot faster in development, as it doesn't
108 recreate everything from scratch on every test. To use the default behaviour
109 pass ``true`` as the second argument.
111 ``PermissionHelper`` to help determine if
112 the correct actions are shown based on the current permissions and selection
115 ``FormHelper`` which makes testing a form a lot easier
116 with a few simple methods. It allows you to set a control or multiple
117 controls, expect if a control is valid or has an error or just do both with
118 one method. Additional you can expect a template element or multiple elements
119 to be visible in the rendered template.
124 Create ``unit-test-configuration.ts`` file based on
125 ``unit-test-configuration.ts.sample`` in directory
126 ``src/pybind/mgr/dashboard/frontend/src``.
128 Run ``npm run test`` to execute the unit tests via `Jest
129 <https://facebook.github.io/jest/>`_.
131 If you get errors on all tests, it could be because `Jest
132 <https://facebook.github.io/jest/>`__ or something else was updated.
133 There are a few ways how you can try to resolve this:
135 - Remove all modules with ``rm -rf dist node_modules`` and run ``npm install``
136 again in order to reinstall them
137 - Clear the cache of jest by running ``npx jest --clearCache``
139 Running End-to-End (E2E) Tests
140 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
142 We use `Cypress <https://www.cypress.io/>`__ to run our frontend E2E tests.
147 You need to previously build the frontend.
149 In some environments, depending on your user permissions and the CYPRESS_CACHE_FOLDER,
150 you might need to run ``npm ci`` with the ``--unsafe-perm`` flag.
152 You might need to install additional packages to be able to run Cypress.
153 Please run ``npx cypress verify`` to verify it.
155 run-frontend-e2e-tests.sh
156 .........................
158 Our ``run-frontend-e2e-tests.sh`` script is the go to solution when you wish to
159 do a full scale e2e run.
160 It will verify if everything needed is installed, start a new vstart cluster
161 and run the full test suite.
163 Start all frontend E2E tests by running::
165 $ ./run-frontend-e2e-tests.sh
168 You can follow the e2e report on the terminal and you can find the screenshots
169 of failed test cases by opening the following directory::
171 src/pybind/mgr/dashboard/frontend/cypress/screenshots/
174 You can force the script to use a specific device with the ``-d`` flag::
176 $ ./run-frontend-e2e-tests.sh -d <chrome|chromium|electron|docker>
179 By default this script will stop and start a new vstart cluster.
180 If you want to run the tests outside the ceph environment, you will need to
181 manually define the dashboard url using ``-r`` and, optionally, credentials
184 $ ./run-frontend-e2e-tests.sh -r <DASHBOARD_URL> -u <E2E_LOGIN_USER> -p <E2E_LOGIN_PWD>
187 When using docker, as your device, you might need to run the script with sudo
190 Other running options
191 .....................
193 During active development, it is not recommended to run the previous script,
194 as it is not prepared for constant file changes.
195 Instead you should use one of the following commands:
197 - ``npm run e2e`` - This will run ``ng serve`` and open the Cypress Test Runner.
198 - ``npm run e2e:ci`` - This will run ``ng serve`` and run the Cypress Test Runner once.
199 - ``npx cypress run`` - This calls cypress directly and will run the Cypress Test Runner.
200 You need to have a running frontend server.
201 - ``npx cypress open`` - This calls cypress directly and will open the Cypress Test Runner.
202 You need to have a running frontend server.
204 Calling Cypress directly has the advantage that you can use any of the available
205 `flags <https://docs.cypress.io/guides/guides/command-line.html#cypress-run>`__
206 to customize your test run and you don't need to start a frontend server each time.
208 Using one of the ``open`` commands, will open a cypress application where you
209 can see all the test files you have and run each individually.
210 This is going to be run in watch mode, so if you make any changes to test files,
211 it will retrigger the test run.
212 This cannot be used inside docker, as it requires X11 environment to be able to open.
214 By default Cypress will look for the web page at ``https://localhost:4200/``.
215 If you are serving it in a different URL you will need to configure it by
216 exporting the environment variable CYPRESS_BASE_URL with the new value.
217 E.g.: ``CYPRESS_BASE_URL=https://localhost:41076/ npx cypress open``
220 .....................
222 When installing cypress via npm, a binary of the cypress app will also be
223 downloaded and stored in a cache folder.
224 This removes the need to download it every time you run ``npm ci`` or even when
225 using cypress in a separate project.
227 By default Cypress uses ~/.cache to store the binary.
228 To prevent changes to the user home directory, we have changed this folder to
229 ``/ceph/build/src/pybind/mgr/dashboard/cypress``, so when you build ceph or run
230 ``run-frontend-e2e-tests.sh`` this is the directory Cypress will use.
232 When using any other command to install or run cypress,
233 it will go back to the default directory. It is recommended that you export the
234 CYPRESS_CACHE_FOLDER environment variable with a fixed directory, so you always
235 use the same directory no matter which command you use.
238 Writing End-to-End Tests
239 ~~~~~~~~~~~~~~~~~~~~~~~~
241 The PagerHelper class
242 .....................
244 The ``PageHelper`` class is supposed to be used for general purpose code that
245 can be used on various pages or suites.
249 - ``navigateTo()`` - Navigates to a specific page and waits for it to load
250 - ``getFirstTableCell()`` - returns the first table cell. You can also pass a
251 string with the desired content and it will return the first cell that
253 - ``getTabsCount()`` - returns the amount of tabs
255 Every method that could be useful on several pages belongs there. Also, methods
256 which enhance the derived classes of the PageHelper belong there. A good
257 example for such a case is the ``restrictTo()`` decorator. It ensures that a
258 method implemented in a subclass of PageHelper is called on the correct page.
259 It will also show a developer-friendly warning if this is not the case.
261 Subclasses of PageHelper
262 ........................
267 In order to make code reusable which is specific for a particular suite, make
268 sure to put it in a derived class of the ``PageHelper``. For instance, when
269 talking about the pool suite, such methods would be ``create()``, ``exist()``
270 and ``delete()``. These methods are specific to a pool but are useful for other
273 Methods that return HTML elements which can only be found on a specific page,
274 should be either implemented in the helper methods of the subclass of PageHelper
275 or as own methods of the subclass of PageHelper.
280 In any suite, an instance of the specific ``Helper`` class should be
281 instantiated and called directly.
285 const pools = new PoolPageHelper();
287 it('should create a pool', () => {
288 pools.exist(poolName, false);
289 pools.navigateTo('create');
290 pools.create(poolName, 8);
291 pools.exist(poolName, true);
297 Please refer to the official `Cypress Core Concepts
298 <https://docs.cypress.io/guides/core-concepts/introduction-to-cypress.html#Cypress-Can-Be-Simple-Sometimes>`__
299 for a better insight on how to write and structure tests.
301 ``describe()`` vs ``it()``
302 """"""""""""""""""""""""""
304 Both ``describe()`` and ``it()`` are function blocks, meaning that any
305 executable code necessary for the test can be contained in either block.
306 However, Typescript scoping rules still apply, therefore any variables declared
307 in a ``describe`` are available to the ``it()`` blocks inside of it.
309 ``describe()`` typically are containers for tests, allowing you to break tests
310 into multiple parts. Likewise, any setup that must be made before your tests are
311 run can be initialized within the ``describe()`` block. Here is an example:
315 describe('create, edit & delete image test', () => {
316 const poolName = 'e2e_images_pool';
320 pools.navigateTo('create');
321 pools.create(poolName, 8, 'rbd');
322 pools.exist(poolName, true);
334 As shown, we can initiate the variable ``poolName`` as well as run commands
335 before our test suite begins (creating a pool). ``describe()`` block messages
336 should include what the test suite is.
338 ``it()`` blocks typically are parts of an overarching test. They contain the
339 functionality of the test suite, each performing individual roles.
344 describe('create, edit & delete image test', () => {
347 it('should create image', () => {
348 images.createImage(imageName, poolName, '1');
349 images.getFirstTableCell(imageName).should('exist');
352 it('should edit image', () => {
353 images.editImage(imageName, poolName, newImageName, '2');
354 images.getFirstTableCell(newImageName).should('exist');
360 As shown from the previous example, our ``describe()`` test suite is to create,
361 edit and delete an image. Therefore, each ``it()`` completes one of these steps,
362 one for creating, one for editing, and so on. Likewise, every ``it()`` blocks
363 message should be in lowercase and written so long as "it" can be the prefix of
364 the message. For example, ``it('edits the test image' () => ...)`` vs.
365 ``it('image edit test' () => ...)``. As shown, the first example makes
366 grammatical sense with ``it()`` as the prefix whereas the second message does
367 not. ``it()`` should describe what the individual test is doing and what it
370 Differences between Frontend Unit Tests and End-to-End (E2E) Tests / FAQ
371 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
373 General introduction about testing and E2E/unit tests
376 What are E2E/unit tests designed for?
377 .....................................
381 It requires a fully functional system and tests the interaction of all components
382 of the application (Ceph, back-end, front-end).
383 E2E tests are designed to mimic the behavior of the user when interacting with the application
384 - for example when it comes to workflows like creating/editing/deleting an item.
385 Also the tests should verify that certain items are displayed as a user would see them
386 when clicking through the UI (for example a menu entry or a pool that has been
387 created during a test and the pool and its properties should be displayed in the table).
391 Unit tests, as the name suggests, are tests for smaller units of the code.
392 Those tests are designed for testing all kinds of Angulars' components (e.g. services, pipes etc.).
393 They do not require a connection to the backend, hence those tests are independent of it.
394 The expected data of the backend is mocked in the frontend and by using this data
395 the functionality of the frontend can be tested without having to have real data from the backend.
396 As previously mentioned, data is either mocked or, in a simple case, contains a static input,
397 a function call and an expected static output.
398 More complex examples include the state of a component (attributes of the component class),
399 that define how the output changes according to the given input.
401 Which E2E/unit tests are considered to be valid?
402 ................................................
404 This is not easy to answer, but new tests that are written in the same way as already existing
405 dashboard tests should generally be considered valid.
406 Unit tests should focus on the component to be tested.
407 This is either an Angular component, directive, service, pipe, etc.
409 E2E tests should focus on testing the functionality of the whole application.
410 Approximately a third of the overall E2E tests should verify the correctness
411 of user visible elements.
413 How should an E2E/unit test look like?
414 ......................................
416 Unit tests should focus on the described purpose
417 and shouldn't try to test other things in the same `it` block.
419 E2E tests should contain a description that either verifies
420 the correctness of a user visible element or a complete process
421 like for example the creation/validation/deletion of a pool.
423 What should an E2E/unit test cover?
424 ...................................
426 E2E tests should mostly, but not exclusively, cover interaction with the backend.
427 This way the interaction with the backend is utilized to write integration tests.
429 A unit test should mostly cover critical or complex functionality
430 of a component (Angular Components, Services, Pipes, Directives, etc).
432 What should an E2E/unit test NOT cover?
433 .......................................
435 Avoid duplicate testing: do not write E2E tests for what's already
436 been covered as frontend-unit tests and vice versa.
437 It may not be possible to completely avoid an overlap.
439 Unit tests should not be used to extensively click through components and E2E tests
440 shouldn't be used to extensively test a single component of Angular.
442 Best practices/guideline
443 ........................
445 As a general guideline we try to follow the 70/20/10 approach - 70% unit tests,
446 20% integration tests and 10% end-to-end tests.
447 For further information please refer to `this document
448 <https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html>`__
449 and the included "Testing Pyramid".
454 To get more help on the Angular CLI use ``ng help`` or go check out the
456 README <https://github.com/angular/angular-cli/blob/master/README.md>`__.
458 Example of a Generator
459 ~~~~~~~~~~~~~~~~~~~~~~
463 # Create module 'Core'
464 src/app> ng generate module core -m=app --routing
466 # Create module 'Auth' under module 'Core'
467 src/app/core> ng generate module auth -m=core --routing
469 src/app> ng generate module core/auth -m=core --routing
471 # Create component 'Login' under module 'Auth'
472 src/app/core/auth> ng generate component login -m=core/auth
474 src/app> ng generate component core/auth/login -m=core/auth
476 Frontend Typescript Code Style Guide Recommendations
477 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
479 Group the imports based on its source and separate them with a blank
482 The source groups can be either from Angular, external or internal.
488 import { Component } from '@angular/core';
489 import { Router } from '@angular/router';
491 import { ToastrManager } from 'ngx-toastr';
493 import { Credentials } from '../../../shared/models/credentials.model';
494 import { HostService } from './services/host.service';
499 There are several components that can be reused on different pages.
500 This components are declared on the components module:
501 `src/pybind/mgr/dashboard/frontend/src/app/shared/components`.
506 This component should be used to provide additional information to the user.
513 Some <strong>helper</strong> html text
516 Terminology and wording
517 ~~~~~~~~~~~~~~~~~~~~~~~
519 Instead of using the Ceph component names, the approach
520 suggested is to use the logical/generic names (Block over RBD, Filesystem over
521 CephFS, Object over RGW). Nevertheless, as Ceph-Dashboard cannot completely hide
522 the Ceph internals, some Ceph-specific names might remain visible.
524 Regarding the wording for action labels and other textual elements (form titles,
525 buttons, etc.), the chosen approach is to follow `these guidelines
526 <https://www.patternfly.org/styles/terminology-and-wording/#terminology-and-wording-for-action-labels>`_.
527 As a rule of thumb, 'Create' and 'Delete' are the proper wording for most forms,
528 instead of 'Add' and 'Remove', unless some already created item is either added
529 or removed to/from a set of items (e.g.: 'Add permission' to a user vs. 'Create
532 In order to enforce the use of this wording, a service ``ActionLabelsI18n`` has
533 been created, which provides translated labels for use in UI elements.
538 Every vendor can customize the 'Ceph dashboard' to his needs. No matter if
539 logo, HTML-Template or TypeScript, every file inside the frontend folder can be
542 To replace files, open ``./frontend/angular.json`` and scroll to the section
543 ``fileReplacements`` inside the production configuration. Here you can add the
544 files you wish to brand. We recommend to place the branded version of a file in
545 the same directory as the original one and to add a ``.brand`` to the file
546 name, right in front of the file extension. A ``fileReplacement`` could for
547 example look like this:
552 "replace": "src/app/core/auth/login/login.component.html",
553 "with": "src/app/core/auth/login/login.component.brand.html"
556 To serve or build the branded user interface run:
558 $ npm run start -- --prod
562 $ npm run build -- --prod
564 Unfortunately it's currently not possible to use multiple configurations when
565 serving or building the UI at the same time. That means a configuration just
566 for the branding ``fileReplacements`` is not an option, because you want to use
567 the production configuration anyway
568 (https://github.com/angular/angular-cli/issues/10612).
569 Furthermore it's also not possible to use glob expressions for
570 ``fileReplacements``. As long as the feature hasn't been implemented, you have
571 to add the file replacements manually to the angular.json file
572 (https://github.com/angular/angular-cli/issues/12354).
574 Nevertheless you should stick to the suggested naming scheme because it makes
575 it easier for you to use glob expressions once it's supported in the future.
577 To change the variable defaults you can overwrite them in the file
578 ``./frontend/src/vendor.variables.scss``. Just reassign the variable you want
579 to change, for example ``$color-primary: teal;``
580 To overwrite or extend the default CSS, you can add your own styles in
581 ``./frontend/src/vendor.overrides.scss``.
586 How to extract messages from source code?
587 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
589 To extract the I18N messages from the templates and the TypeScript files just
590 run the following command in ``src/pybind/mgr/dashboard/frontend``::
592 $ npm run i18n:extract
594 This will extract all marked messages from the HTML templates first and then
595 add all marked strings from the TypeScript files to the translation template.
596 Since the extraction from TypeScript files is still not supported by Angular
597 itself, we are using the
598 `ngx-translator <https://github.com/ngx-translate/i18n-polyfill>`_ extractor to
599 parse the TypeScript files.
601 When the command ran successfully, it should have created or updated the file
602 ``src/locale/messages.xlf``.
604 The file isn't tracked by git, you can just use it to start with the
605 translation offline or add/update the resource files on transifex.
610 All our supported languages should be registered in both exports in
611 ``supported-languages.enum.ts`` and have a corresponding test in
612 ``language-selector.component.spec.ts``.
614 The ``SupportedLanguages`` enum will provide the list for the default language selection.
616 The ``languageBootstrapMapping`` variable will provide the
617 `language support <https://github.com/valor-software/ngx-bootstrap/tree/development/src/chronos/i18n>`_
618 for ngx-bootstrap components like the
619 `date picker <https://valor-software.com/ngx-bootstrap/#/datepicker#locales>`_.
624 To facilitate the translation process of the dashboard we are using a web tool
625 called `transifex <https://www.transifex.com/>`_.
627 If you wish to help translating to any language just go to our `transifex
628 project page <https://www.transifex.com/ceph/ceph-dashboard/>`_, join the
629 project and you can start translating immediately.
631 All translations will then be reviewed and later pushed upstream.
633 Updating translated messages
634 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
636 Any time there are new messages translated and reviewed in a specific language
637 we should update the translation file upstream.
639 To do that, check the settings in the i18n config file
640 ``src/pybind/mgr/dashboard/frontend/i18n.config.json``:: and make sure that the
641 organization is *ceph*, the project is *ceph-dashboard* and the resource is
642 the one you want to pull from and push to e.g. *Master:master*. To find a list
643 of avaiable resources visit `<https://www.transifex.com/ceph/ceph-dashboard/content/>`_.
645 After you checked the config go to the directory ``src/pybind/mgr/dashboard/frontend`` and run::
649 This command will extract all marked messages from the HTML templates and
650 TypeScript files. Once the source file has been created it will push it to
651 transifex and pull the latest translations. It will also fill all the
652 untranslated strings with the source string.
653 The tool will ask you for an api token, unless you added it by running:
657 To create a transifex api token visit `<https://www.transifex.com/user/settings/api/>`_.
659 After the command ran successfully, build the UI and check if everything is
660 working as expected. You also might want to run the frontend tests.
665 Strings need to start and end in the same line as the element:
675 <span i18n>Foo</span>
685 <span i18n>Foo bar baz.
688 Isolated interpolations should not be translated:
693 <span i18n>{{ foo }}</span>
696 <span>{{ foo }}</span>
698 Interpolations used in a sentence should be kept in the translation:
703 <span i18n>There are {{ x }} OSDs.</span>
705 Remove elements that are outside the context of the translation:
712 <span class="required"></span>
717 <ng-container i18n>Profile<ng-container>
718 <span class="required"></span>
721 Keep elements that affect the sentence:
726 <span i18n>Profile <b>foo</b> will be removed.</span>
731 The Python backend code of this module requires a number of Python modules to be
732 installed. They are listed in file ``requirements.txt``. Using `pip
733 <https://pypi.python.org/pypi/pip>`_ you may install all required dependencies
734 by issuing ``pip install -r requirements.txt`` in directory
735 ``src/pybind/mgr/dashboard``.
737 If you're using the `ceph-dev-docker development environment
738 <https://github.com/ricardoasmarques/ceph-dev-docker/>`_, simply run
739 ``./install_deps.sh`` from the toplevel directory to install them.
744 In dashboard we have two different kinds of backend tests:
746 1. Unit tests based on ``tox``
747 2. API tests based on Teuthology.
749 Unit tests based on tox
750 ~~~~~~~~~~~~~~~~~~~~~~~~
752 We included a ``tox`` configuration file that will run the unit tests under
753 Python 2 or 3, as well as linting tools to guarantee the uniformity of code.
755 You need to install ``tox`` and ``coverage`` before running it. To install the
756 packages in your system, either install it via your operating system's package
757 management tools, e.g. by running ``dnf install python-tox python-coverage`` on
760 Alternatively, you can use Python's native package installation method::
763 $ pip install coverage
765 To run the tests, run ``src/script/run_tox.sh`` in the dashboard directory (where
766 ``tox.ini`` is located)::
768 ## Run Python 2+3 tests+lint commands:
769 $ ../../../script/run_tox.sh --tox-env py27,py3,lint,check
771 ## Run Python 3 tests+lint commands:
772 $ ../../../script/run_tox.sh --tox-env py3,lint,check
774 ## Run Python 3 arbitrary command (e.g. 1 single test):
775 $ ../../../script/run_tox.sh --tox-env py3 "" tests/test_rgw_client.py::RgwClientTest::test_ssl_verify
777 You can also run tox instead of ``run_tox.sh``::
779 ## Run Python 3 tests command:
782 ## Run Python 3 arbitrary command (e.g. 1 single test):
783 $ tox -e py3 tests/test_rgw_client.py::RgwClientTest::test_ssl_verify
785 Python files can be automatically fixed and formatted according to PEP8
786 standards by using ``run_tox.sh --tox-env fix`` or ``tox -e fix``.
788 We also collect coverage information from the backend code when you run tests. You can check the
789 coverage information provided by the tox output, or by running the following
790 command after tox has finished successfully::
794 This command will create a directory ``htmlcov`` with an HTML representation of
795 the code coverage of the backend.
797 API tests based on Teuthology
798 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
800 How to run existing API tests:
801 To run the API tests against a real Ceph cluster, we leverage the Teuthology
802 framework. This has the advantage of catching bugs originated from changes in
803 the internal Ceph code.
805 Our ``run-backend-api-tests.sh`` script will start a ``vstart`` Ceph cluster
806 before running the Teuthology tests, and then it stops the cluster after the
807 tests are run. Of course this implies that you have built/compiled Ceph
810 Start all dashboard tests by running::
812 $ ./run-backend-api-tests.sh
814 Or, start one or multiple specific tests by specifying the test name::
816 $ ./run-backend-api-tests.sh tasks.mgr.dashboard.test_pool.PoolTest
818 Or, ``source`` the script and run the tests manually::
820 $ source run-backend-api-tests.sh
821 $ run_teuthology_tests [tests]...
824 How to write your own tests:
825 There are two possible ways to write your own API tests:
827 The first is by extending one of the existing test classes in the
828 ``qa/tasks/mgr/dashboard`` directory.
830 The second way is by adding your own API test module if you're creating a new
831 controller for example. To do so you'll just need to add the file containing
832 your new test class to the ``qa/tasks/mgr/dashboard`` directory and implement
835 .. note:: Don't forget to add the path of the newly created module to
836 ``modules`` section in ``qa/suites/rados/mgr/tasks/dashboard.yaml``.
838 Short example: Let's assume you created a new controller called
839 ``my_new_controller.py`` and the related test module
840 ``test_my_new_controller.py``. You'll need to add
841 ``tasks.mgr.dashboard.test_my_new_controller`` to the ``modules`` section in
842 the ``dashboard.yaml`` file.
844 Also, if you're removing test modules please keep in mind to remove the
845 related section. Otherwise the Teuthology test run will fail.
847 Please run your API tests on your dev environment (as explained above)
848 before submitting a pull request. Also make sure that a full QA run in
849 Teuthology/sepia lab (based on your changes) has completed successfully
850 before it gets merged. You don't need to schedule the QA run yourself, just
851 add the 'needs-qa' label to your pull request as soon as you think it's ready
852 for merging (e.g. make check was successful, the pull request is approved and
853 all comments have been addressed). One of the developers who has access to
854 Teuthology/the sepia lab will take care of it and report the result back to
858 How to add a new controller?
859 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
861 A controller is a Python class that extends from the ``BaseController`` class
862 and is decorated with either the ``@Controller``, ``@ApiController`` or
863 ``@UiApiController`` decorators. The Python class must be stored inside a Python
864 file located under the ``controllers`` directory. The Dashboard module will
865 automatically load your new controller upon start.
867 ``@ApiController`` and ``@UiApiController`` are both specializations of the
868 ``@Controller`` decorator.
870 The ``@ApiController`` should be used for controllers that provide an API-like
871 REST interface and the ``@UiApiController`` should be used for endpoints consumed
872 by the UI but that are not part of the 'public' API. For any other kinds of
873 controllers the ``@Controller`` decorator should be used.
875 A controller has a URL prefix path associated that is specified in the
876 controller decorator, and all endpoints exposed by the controller will share
877 the same URL prefix path.
879 A controller's endpoint is exposed by implementing a method on the controller
880 class decorated with the ``@Endpoint`` decorator.
882 For example create a file ``ping.py`` under ``controllers`` directory with the
885 .. code-block:: python
887 from ..tools import Controller, ApiController, UiApiController, BaseController, Endpoint
890 class Ping(BaseController):
893 return {'msg': "Hello"}
895 @ApiController('/ping')
896 class ApiPing(BaseController):
899 return {'msg': "Hello"}
901 @UiApiController('/ping')
902 class UiApiPing(BaseController):
905 return {'msg': "Hello"}
907 The ``hello`` endpoint of the ``Ping`` controller can be reached by the
908 following URL: https://mgr_hostname:8443/ping/hello using HTTP GET requests.
909 As you can see the controller URL path ``/ping`` is concatenated to the
910 method name ``hello`` to generate the endpoint's URL.
912 In the case of the ``ApiPing`` controller, the ``hello`` endpoint can be
913 reached by the following URL: https://mgr_hostname:8443/api/ping/hello using a
915 The API controller URL path ``/ping`` is prefixed by the ``/api`` path and then
916 concatenated to the method name ``hello`` to generate the endpoint's URL.
917 Internally, the ``@ApiController`` is actually calling the ``@Controller``
918 decorator by passing an additional decorator parameter called ``base_url``::
920 @ApiController('/ping') <=> @Controller('/ping', base_url="/api")
922 ``UiApiPing`` works in a similar way than the ``ApiPing``, but the URL will be
923 prefixed by ``/ui-api``: https://mgr_hostname:8443/ui-api/ping/hello. ``UiApiPing`` is
924 also a ``@Controller`` extension::
926 @UiApiController('/ping') <=> @Controller('/ping', base_url="/ui-api")
928 The ``@Endpoint`` decorator also supports many parameters to customize the
931 * ``method="GET"``: the HTTP method allowed to access this endpoint.
932 * ``path="/<method_name>"``: the URL path of the endpoint, excluding the
933 controller URL path prefix.
934 * ``path_params=[]``: list of method parameter names that correspond to URL
935 path parameters. Can only be used when ``method in ['POST', 'PUT']``.
936 * ``query_params=[]``: list of method parameter names that correspond to URL
938 * ``json_response=True``: indicates if the endpoint response should be
939 serialized in JSON format.
940 * ``proxy=False``: indicates if the endpoint should be used as a proxy.
942 An endpoint method may have parameters declared. Depending on the HTTP method
943 defined for the endpoint the method parameters might be considered either
944 path parameters, query parameters, or body parameters.
946 For ``GET`` and ``DELETE`` methods, the method's non-optional parameters are
947 considered path parameters by default. Optional parameters are considered
948 query parameters. By specifying the ``query_parameters`` in the endpoint
949 decorator it is possible to make a non-optional parameter to be a query
952 For ``POST`` and ``PUT`` methods, all method parameters are considered
953 body parameters by default. To override this default, one can use the
954 ``path_params`` and ``query_params`` to specify which method parameters are
955 path and query parameters respectively.
956 Body parameters are decoded from the request body, either from a form format, or
957 from a dictionary in JSON format.
959 Let's use an example to better understand the possible ways to customize an
962 .. code-block:: python
964 from ..tools import Controller, BaseController, Endpoint
967 class Ping(BaseController):
969 # URL: /ping/{key}?opt1=...&opt2=...
970 @Endpoint(path="/", query_params=['opt1'])
971 def index(self, key, opt1, opt2=None):
974 # URL: /ping/{key}?opt1=...&opt2=...
975 @Endpoint(query_params=['opt1'])
976 def __call__(self, key, opt1, opt2=None):
979 # URL: /ping/post/{key1}/{key2}
980 @Endpoint('POST', path_params=['key1', 'key2'])
981 def post(self, key1, key2, data1, data2=None):
985 In the above example we see how the ``path`` option can be used to override the
986 generated endpoint URL in order to not use the method's name in the URL. In the
987 ``index`` method we set the ``path`` to ``"/"`` to generate an endpoint that is
988 accessible by the root URL of the controller.
990 An alternative approach to generate an endpoint that is accessible through just
991 the controller's path URL is by using the ``__call__`` method, as we show in
994 From the third method you can see that the path parameters are collected from
995 the URL by parsing the list of values separated by slashes ``/`` that come
996 after the URL path ``/ping`` for ``index`` method case, and ``/ping/post`` for
997 the ``post`` method case.
999 Defining path parameters in endpoints's URLs using python methods's parameters
1000 is very easy but it is still a bit strict with respect to the position of these
1001 parameters in the URL structure.
1002 Sometimes we may want to explicitly define a URL scheme that
1003 contains path parameters mixed with static parts of the URL.
1004 Our controller infrastructure also supports the declaration of URL paths with
1005 explicit path parameters at both the controller level and method level.
1007 Consider the following example:
1009 .. code-block:: python
1011 from ..tools import Controller, BaseController, Endpoint
1013 @Controller('/ping/{node}/stats')
1014 class Ping(BaseController):
1016 # URL: /ping/{node}/stats/{date}/latency?unit=...
1017 @Endpoint(path="/{date}/latency")
1018 def latency(self, node, date, unit="ms"):
1021 In this example we explicitly declare a path parameter ``{node}`` in the
1022 controller URL path, and a path parameter ``{date}`` in the ``latency``
1023 method. The endpoint for the ``latency`` method is then accessible through
1024 the URL: https://mgr_hostname:8443/ping/{node}/stats/{date}/latency .
1026 For a full set of examples on how to use the ``@Endpoint``
1027 decorator please check the unit test file: ``tests/test_controllers.py``.
1028 There you will find many examples of how to customize endpoint methods.
1031 Implementing Proxy Controller
1032 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1034 Sometimes you might need to relay some requests from the Dashboard frontend
1035 directly to an external service.
1036 For that purpose we provide a decorator called ``@Proxy``.
1037 (As a concrete example, check the ``controllers/rgw.py`` file where we
1038 implemented an RGW Admin Ops proxy.)
1041 The ``@Proxy`` decorator is a wrapper of the ``@Endpoint`` decorator that
1042 already customizes the endpoint for working as a proxy.
1043 A proxy endpoint works by capturing the URL path that follows the controller
1044 URL prefix path, and does not do any decoding of the request body.
1048 .. code-block:: python
1050 from ..tools import Controller, BaseController, Proxy
1052 @Controller('/foo/proxy')
1053 class FooServiceProxy(BaseController):
1056 def proxy(self, path, **params):
1058 if requested URL is "/foo/proxy/access/service?opt=1"
1059 then path is "access/service" and params is {'opt': '1'}
1063 How does the RESTController work?
1064 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1066 We also provide a simple mechanism to create REST based controllers using the
1067 ``RESTController`` class. Any class which inherits from ``RESTController`` will,
1068 by default, return JSON.
1070 The ``RESTController`` is basically an additional abstraction layer which eases
1071 and unifies the work with collections. A collection is just an array of objects
1072 with a specific type. ``RESTController`` enables some default mappings of
1073 request types and given parameters to specific method names. This may sound
1074 complicated at first, but it's fairly easy. Lets have look at the following
1077 .. code-block:: python
1080 from ..tools import ApiController, RESTController
1082 @ApiController('ping')
1083 class Ping(RESTController):
1085 return {"msg": "Hello"}
1088 return self.objects[id]
1090 In this case, the ``list`` method is automatically used for all requests to
1091 ``api/ping`` where no additional argument is given and where the request type
1092 is ``GET``. If the request is given an additional argument, the ID in our
1093 case, it won't map to ``list`` anymore but to ``get`` and return the element
1094 with the given ID (assuming that ``self.objects`` has been filled before). The
1095 same applies to other request types:
1097 +--------------+------------+----------------+-------------+
1098 | Request type | Arguments | Method | Status Code |
1099 +==============+============+================+=============+
1100 | GET | No | list | 200 |
1101 +--------------+------------+----------------+-------------+
1102 | PUT | No | bulk_set | 200 |
1103 +--------------+------------+----------------+-------------+
1104 | POST | No | create | 201 |
1105 +--------------+------------+----------------+-------------+
1106 | DELETE | No | bulk_delete | 204 |
1107 +--------------+------------+----------------+-------------+
1108 | GET | Yes | get | 200 |
1109 +--------------+------------+----------------+-------------+
1110 | PUT | Yes | set | 200 |
1111 +--------------+------------+----------------+-------------+
1112 | DELETE | Yes | delete | 204 |
1113 +--------------+------------+----------------+-------------+
1115 How to use a custom API endpoint in a RESTController?
1116 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1118 If you don't have any access restriction you can use ``@Endpoint``. If you
1119 have set a permission scope to restrict access to your endpoints,
1120 ``@Endpoint`` will fail, as it doesn't know which permission property should be
1121 used. To use a custom endpoint inside a restricted ``RESTController`` use
1122 ``@RESTController.Collection`` instead. You can also choose
1123 ``@RESTController.Resource`` if you have set a ``RESOURCE_ID`` in your
1124 ``RESTController`` class.
1126 .. code-block:: python
1129 from ..tools import ApiController, RESTController
1131 @ApiController('ping', Scope.Ping)
1132 class Ping(RESTController):
1133 RESOURCE_ID = 'ping'
1135 @RESTController.Resource('GET')
1136 def some_get_endpoint(self):
1137 return {"msg": "Hello"}
1139 @RESTController.Collection('POST')
1140 def some_post_endpoint(self, **data):
1141 return {"msg": data}
1143 Both decorators also support four parameters to customize the
1146 * ``method="GET"``: the HTTP method allowed to access this endpoint.
1147 * ``path="/<method_name>"``: the URL path of the endpoint, excluding the
1148 controller URL path prefix.
1149 * ``status=200``: set the HTTP status response code
1150 * ``query_params=[]``: list of method parameter names that correspond to URL
1153 How to restrict access to a controller?
1154 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1156 All controllers require authentication by default.
1157 If you require that the controller can be accessed without authentication,
1158 then you can add the parameter ``secure=False`` to the controller decorator.
1162 .. code-block:: python
1165 from . import ApiController, RESTController
1168 @ApiController('ping', secure=False)
1169 class Ping(RESTController):
1171 return {"msg": "Hello"}
1173 How to create a dedicated UI endpoint which uses the 'public' API?
1174 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1176 Sometimes we want to combine multiple calls into one single call
1177 to save bandwidth or for other performance reasons.
1178 In order to achieve that, we first have to create an ``@UiApiController`` which
1179 is used for endpoints consumed by the UI but that are not part of the
1180 'public' API. Let the ui class inherit from the REST controller class.
1181 Now you can use all methods from the api controller.
1185 .. code-block:: python
1188 from . import UiApiController, ApiController, RESTController
1191 @ApiController('ping', secure=False) # /api/ping
1192 class Ping(RESTController):
1196 def _list(self): # To not get in conflict with the JSON wrapper
1200 @UiApiController('ping', secure=False) # /ui-api/ping
1203 return self._list() + [4, 5, 6]
1205 How to access the manager module instance from a controller?
1206 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1208 We provide the manager module instance as a global variable that can be
1209 imported in any module.
1213 .. code-block:: python
1218 from ..tools import ApiController, RESTController
1220 logger = logging.getLogger(__name__)
1222 @ApiController('servers')
1223 class Servers(RESTController):
1225 logger.debug('Listing available servers')
1226 return {'servers': mgr.list_servers()}
1229 How to write a unit test for a controller?
1230 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1232 We provide a test helper class called ``ControllerTestCase`` to easily create
1233 unit tests for your controller.
1235 If we want to write a unit test for the above ``Ping`` controller, create a
1236 ``test_ping.py`` file under the ``tests`` directory with the following code:
1238 .. code-block:: python
1240 from .helper import ControllerTestCase
1241 from .controllers.ping import Ping
1244 class PingTest(ControllerTestCase):
1246 def setup_test(cls):
1247 Ping._cp_config['tools.authenticate.on'] = False
1248 cls.setup_controllers([Ping])
1250 def test_ping(self):
1251 self._get("/api/ping")
1252 self.assertStatus(200)
1253 self.assertJsonBody({'msg': 'Hello'})
1255 The ``ControllerTestCase`` class starts by initializing a CherryPy webserver.
1256 Then it will call the ``setup_test()`` class method where we can explicitly
1257 load the controllers that we want to test. In the above example we are only
1258 loading the ``Ping`` controller. We can also disable authentication of a
1259 controller at this stage, as depicted in the example.
1262 How to listen for manager notifications in a controller?
1263 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1265 The manager notifies the modules of several types of cluster events, such
1266 as cluster logging event, etc...
1268 Each module has a "global" handler function called ``notify`` that the manager
1269 calls to notify the module. But this handler function must not block or spend
1270 too much time processing the event notification.
1271 For this reason we provide a notification queue that controllers can register
1272 themselves with to receive cluster notifications.
1274 The example below represents a controller that implements a very simple live
1277 .. code-block:: python
1279 from __future__ import absolute_import
1285 from ..tools import ApiController, BaseController, NotificationQueue
1288 @ApiController('livelog')
1289 class LiveLog(BaseController):
1290 log_buffer = collections.deque(maxlen=1000)
1293 super(LiveLog, self).__init__()
1294 NotificationQueue.register(self.log, 'clog')
1296 def log(self, log_struct):
1297 self.log_buffer.appendleft(log_struct)
1301 ret = '<html><meta http-equiv="refresh" content="2" /><body>'
1302 for l in self.log_buffer:
1303 ret += "{}<br>".format(l)
1304 ret += "</body></html>"
1307 As you can see above, the ``NotificationQueue`` class provides a register
1308 method that receives the function as its first argument, and receives the
1309 "notification type" as the second argument.
1310 You can omit the second argument of the ``register`` method, and in that case
1311 you are registering to listen all notifications of any type.
1313 Here is an list of notification types (these might change in the future) that
1316 * ``clog``: cluster log notifications
1317 * ``command``: notification when a command issued by ``MgrModule.send_command``
1319 * ``perf_schema_update``: perf counters schema update
1320 * ``mon_map``: monitor map update
1321 * ``fs_map``: cephfs map update
1322 * ``osd_map``: OSD map update
1323 * ``service_map``: services (RGW, RBD-Mirror, etc.) map update
1324 * ``mon_status``: monitor status regular update
1325 * ``health``: health status regular update
1326 * ``pg_summary``: regular update of PG status information
1329 How to write a unit test when a controller accesses a Ceph module?
1330 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1332 Consider the following example that implements a controller that retrieves the
1333 list of RBD images of the ``rbd`` pool:
1335 .. code-block:: python
1339 from ..tools import ApiController, RESTController
1342 @ApiController('rbdimages')
1343 class RbdImages(RESTController):
1345 self.ioctx = mgr.rados.open_ioctx('rbd')
1346 self.rbd = rbd.RBD()
1349 return [{'name': n} for n in self.rbd.list(self.ioctx)]
1351 In the example above, we want to mock the return value of the ``rbd.list``
1352 function, so that we can test the JSON response of the controller.
1354 The unit test code will look like the following:
1356 .. code-block:: python
1359 from .helper import ControllerTestCase
1362 class RbdImagesTest(ControllerTestCase):
1363 @mock.patch('rbd.RBD.list')
1364 def test_list(self, rbd_list_mock):
1365 rbd_list_mock.return_value = ['img1', 'img2']
1366 self._get('/api/rbdimages')
1367 self.assertJsonBody([{'name': 'img1'}, {'name': 'img2'}])
1371 How to add a new configuration setting?
1372 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1374 If you need to store some configuration setting for a new feature, we already
1375 provide an easy mechanism for you to specify/use the new config setting.
1377 For instance, if you want to add a new configuration setting to hold the
1378 email address of the dashboard admin, just add a setting name as a class
1379 attribute to the ``Options`` class in the ``settings.py`` file::
1382 class Options(object):
1385 ADMIN_EMAIL_ADDRESS = ('admin@admin.com', str)
1387 The value of the class attribute is a pair composed by the default value for that
1388 setting, and the python type of the value.
1390 By declaring the ``ADMIN_EMAIL_ADDRESS`` class attribute, when you restart the
1391 dashboard module, you will automatically gain two additional CLI commands to
1392 get and set that setting::
1394 $ ceph dashboard get-admin-email-address
1395 $ ceph dashboard set-admin-email-address <value>
1397 To access, or modify the config setting value from your Python code, either
1398 inside a controller or anywhere else, you just need to import the ``Settings``
1399 class and access it like this:
1401 .. code-block:: python
1403 from settings import Settings
1406 tmp_var = Settings.ADMIN_EMAIL_ADDRESS
1409 Settings.ADMIN_EMAIL_ADDRESS = 'myemail@admin.com'
1411 The settings management implementation will make sure that if you change a
1412 setting value from the Python code you will see that change when accessing
1413 that setting from the CLI and vice-versa.
1416 How to run a controller read-write operation asynchronously?
1417 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1419 Some controllers might need to execute operations that alter the state of the
1420 Ceph cluster. These operations might take some time to execute and to maintain
1421 a good user experience in the Web UI, we need to run those operations
1422 asynchronously and return immediately to frontend some information that the
1423 operations are running in the background.
1425 To help in the development of the above scenario we added the support for
1426 asynchronous tasks. To trigger the execution of an asynchronous task we must
1427 use the following class method of the ``TaskManager`` class::
1429 from ..tools import TaskManager
1431 TaskManager.run(name, metadata, func, args, kwargs)
1433 * ``name`` is a string that can be used to group tasks. For instance
1434 for RBD image creation tasks we could specify ``"rbd/create"`` as the
1435 name, or similarly ``"rbd/remove"`` for RBD image removal tasks.
1437 * ``metadata`` is a dictionary where we can store key-value pairs that
1438 characterize the task. For instance, when creating a task for creating
1439 RBD images we can specify the metadata argument as
1440 ``{'pool_name': "rbd", image_name': "test-img"}``.
1442 * ``func`` is the python function that implements the operation code, which
1443 will be executed asynchronously.
1445 * ``args`` and ``kwargs`` are the positional and named arguments that will be
1446 passed to ``func`` when the task manager starts its execution.
1448 The ``TaskManager.run`` method triggers the asynchronous execution of function
1449 ``func`` and returns a ``Task`` object.
1450 The ``Task`` provides the public method ``Task.wait(timeout)``, which can be
1451 used to wait for the task to complete up to a timeout defined in seconds and
1452 provided as an argument. If no argument is provided the ``wait`` method
1453 blocks until the task is finished.
1455 The ``Task.wait`` is very useful for tasks that usually are fast to execute but
1456 that sometimes may take a long time to run.
1457 The return value of the ``Task.wait`` method is a pair ``(state, value)``
1458 where ``state`` is a string with following possible values:
1460 * ``VALUE_DONE = "done"``
1461 * ``VALUE_EXECUTING = "executing"``
1463 The ``value`` will store the result of the execution of function ``func`` if
1464 ``state == VALUE_DONE``. If ``state == VALUE_EXECUTING`` then
1467 The pair ``(name, metadata)`` should unequivocally identify the task being
1468 run, which means that if you try to trigger a new task that matches the same
1469 ``(name, metadata)`` pair of the currently running task, then the new task
1470 is not created and you get the task object of the current running task.
1472 For instance, consider the following example:
1474 .. code-block:: python
1476 task1 = TaskManager.run("dummy/task", {'attr': 2}, func)
1477 task2 = TaskManager.run("dummy/task", {'attr': 2}, func)
1479 If the second call to ``TaskManager.run`` executes while the first task is
1480 still executing then it will return the same task object:
1481 ``assert task1 == task2``.
1484 How to get the list of executing and finished asynchronous tasks?
1485 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1487 The list of executing and finished tasks is included in the ``Summary``
1488 controller, which is already polled every 5 seconds by the dashboard frontend.
1489 But we also provide a dedicated controller to get the same list of executing
1492 The ``Task`` controller exposes the ``/api/task`` endpoint that returns the
1493 list of executing and finished tasks. This endpoint accepts the ``name``
1494 parameter that accepts a glob expression as its value.
1495 For instance, an HTTP GET request of the URL ``/api/task?name=rbd/*``
1496 will return all executing and finished tasks which name starts with ``rbd/``.
1498 To prevent the finished tasks list from growing unbounded, we will always
1499 maintain the 10 most recent finished tasks, and the remaining older finished
1500 tasks will be removed when reaching a TTL of 1 minute. The TTL is calculated
1501 using the timestamp when the task finished its execution. After a minute, when
1502 the finished task information is retrieved, either by the summary controller or
1503 by the task controller, it is automatically deleted from the list and it will
1504 not be included in further task queries.
1506 Each executing task is represented by the following dictionary::
1509 'name': "name", # str
1510 'metadata': { }, # dict
1511 'begin_time': "2018-03-14T15:31:38.423605Z", # str (ISO 8601 format)
1512 'progress': 0 # int (percentage)
1515 Each finished task is represented by the following dictionary::
1518 'name': "name", # str
1519 'metadata': { }, # dict
1520 'begin_time': "2018-03-14T15:31:38.423605Z", # str (ISO 8601 format)
1521 'end_time': "2018-03-14T15:31:39.423605Z", # str (ISO 8601 format)
1522 'duration': 0.0, # float
1523 'progress': 0 # int (percentage)
1524 'success': True, # bool
1525 'ret_value': None, # object, populated only if 'success' == True
1526 'exception': None, # str, populated only if 'success' == False
1530 How to use asynchronous APIs with asynchronous tasks?
1531 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1533 The ``TaskManager.run`` method as described in a previous section, is well
1534 suited for calling blocking functions, as it runs the function inside a newly
1535 created thread. But sometimes we want to call some function of an API that is
1536 already asynchronous by nature.
1538 For these cases we want to avoid creating a new thread for just running a
1539 non-blocking function, and want to leverage the asynchronous nature of the
1540 function. The ``TaskManager.run`` is already prepared to be used with
1541 non-blocking functions by passing an object of the type ``TaskExecutor`` as an
1542 additional parameter called ``executor``. The full method signature of
1543 ``TaskManager.run``::
1545 TaskManager.run(name, metadata, func, args=None, kwargs=None, executor=None)
1548 The ``TaskExecutor`` class is responsible for code that executes a given task
1549 function, and defines three methods that can be overridden by
1552 def init(self, task)
1554 def finish(self, ret_value, exception)
1556 The ``init`` method is called before the running the task function, and
1557 receives the task object (of class ``Task``).
1559 The ``start`` method runs the task function. The default implementation is to
1560 run the task function in the current thread context.
1562 The ``finish`` method should be called when the task function finishes with
1563 either the ``ret_value`` populated with the result of the execution, or with
1564 an exception object in the case that execution raised an exception.
1566 To leverage the asynchronous nature of a non-blocking function, the developer
1567 should implement a custom executor by creating a subclass of the
1568 ``TaskExecutor`` class, and provide an instance of the custom executor class
1569 as the ``executor`` parameter of the ``TaskManager.run``.
1571 To better understand the expressive power of executors, we write a full example
1572 of use a custom executor to execute the ``MgrModule.send_command`` asynchronous
1575 .. code-block:: python
1578 from mgr_module import CommandResult
1580 from ..tools import ApiController, RESTController, NotificationQueue, \
1581 TaskManager, TaskExecutor
1584 class SendCommandExecutor(TaskExecutor):
1586 super(SendCommandExecutor, self).__init__()
1590 def init(self, task):
1591 super(SendCommandExecutor, self).init(task)
1593 # we need to listen for 'command' events to know when the command
1595 NotificationQueue.register(self._handler, 'command')
1597 # store the CommandResult object to retrieve the results
1598 self.result = self.task.fn_args[0]
1599 if len(self.task.fn_args) > 4:
1600 # the user specified a tag for the command, so let's use it
1601 self.tag = self.task.fn_args[4]
1603 # let's generate a unique tag for the command
1604 self.tag = 'send_command_{}'.format(id(self))
1605 self.task.fn_args.append(self.tag)
1607 def _handler(self, data):
1608 if data == self.tag:
1609 # the command has finished, notifying the task with the result
1610 self.finish(self.result.wait(), None)
1611 # deregister listener to avoid memory leaks
1612 NotificationQueue.deregister(self._handler, 'command')
1615 @ApiController('test')
1616 class Test(RESTController):
1618 def _run_task(self, osd_id):
1619 task = TaskManager.run("test/task", {}, mgr.send_command,
1620 [CommandResult(''), 'osd', osd_id,
1621 json.dumps({'prefix': 'perf histogram dump'})],
1622 executor=SendCommandExecutor())
1623 return task.wait(1.0)
1625 def get(self, osd_id):
1626 status, value = self._run_task(osd_id)
1627 return {'status': status, 'value': value}
1630 The above ``SendCommandExecutor`` executor class can be used for any call to
1631 ``MgrModule.send_command``. This means that we should need just one custom
1632 executor class implementation for each non-blocking API that we use in our
1635 The default executor, used when no executor object is passed to
1636 ``TaskManager.run``, is the ``ThreadedExecutor``. You can check its
1637 implementation in the ``tools.py`` file.
1640 How to update the execution progress of an asynchronous task?
1641 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1643 The asynchronous tasks infrastructure provides support for updating the
1644 execution progress of an executing task.
1645 The progress can be updated from within the code the task is executing, which
1646 usually is the place where we have the progress information available.
1648 To update the progress from within the task code, the ``TaskManager`` class
1649 provides a method to retrieve the current task object::
1651 TaskManager.current_task()
1653 The above method is only available when using the default executor
1654 ``ThreadedExecutor`` for executing the task.
1655 The ``current_task()`` method returns the current ``Task`` object. The
1656 ``Task`` object provides two public methods to update the execution progress
1657 value: the ``set_progress(percentage)``, and the ``inc_progress(delta)``
1660 The ``set_progress`` method receives as argument an integer value representing
1661 the absolute percentage that we want to set to the task.
1663 The ``inc_progress`` method receives as argument an integer value representing
1664 the delta we want to increment to the current execution progress percentage.
1666 Take the following example of a controller that triggers a new task and
1667 updates its progress:
1669 .. code-block:: python
1671 from __future__ import absolute_import
1675 from ..tools import TaskManager, ApiController, BaseController
1678 @ApiController('dummy_task')
1679 class DummyTask(BaseController):
1681 top = random.randrange(100)
1682 for i in range(top):
1683 TaskManager.current_task().set_progress(i*100/top)
1684 # or TaskManager.current_task().inc_progress(100/top)
1689 @cherrypy.tools.json_out()
1691 task = TaskManager.run("dummy/task", {}, self._dummy)
1692 return task.wait(5) # wait for five seconds
1695 How to deal with asynchronous tasks in the front-end?
1696 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1698 All executing and most recently finished asynchronous tasks are displayed on
1699 "Background-Tasks" and if finished on "Recent-Notifications" in the menu bar.
1700 For each task a operation name for three states (running, success and failure),
1701 a function that tells who is involved and error descriptions, if any, have to
1702 be provided. This can be achieved by appending
1703 ``TaskManagerMessageService.messages``. This has to be done to achieve
1704 consistency among all tasks and states.
1707 Ensures consistency among all tasks. It consists of three verbs for each
1708 different state f.e.
1709 ``{running: 'Creating', failure: 'create', success: 'Created'}``.
1711 #. Put running operations in present participle f.e. ``'Updating'``.
1712 #. Failed messages always start with ``'Failed to '`` and should be continued
1713 with the operation in present tense f.e. ``'update'``.
1714 #. Put successful operations in past tense f.e. ``'Updated'``.
1717 Ensures consistency among all messages of a task, it resembles who's
1718 involved by the operation. It's a function that returns a string which
1719 takes the metadata from the task to return f.e.
1720 ``"RBD 'somePool/someImage'"``.
1722 Both combined create the following messages:
1724 * Failure => ``"Failed to create RBD 'somePool/someImage'"``
1725 * Running => ``"Creating RBD 'somePool/someImage'"``
1726 * Success => ``"Created RBD 'somePool/someImage'"``
1728 For automatic task handling use ``TaskWrapperService.wrapTaskAroundCall``.
1730 If for some reason ``wrapTaskAroundCall`` is not working for you,
1731 you have to subscribe to your asynchronous task manually through
1732 ``TaskManagerService.subscribe``, and provide it with a callback,
1733 in case of a success to notify the user. A notification can
1734 be triggered with ``NotificationService.notifyTask``. It will use
1735 ``TaskManagerMessageService.messages`` to display a message based on the state
1738 Notifications of API errors are handled by ``ApiInterceptorService``.
1742 .. code-block:: javascript
1744 export class TaskManagerMessageService {
1747 // Messages for task 'rbd/create'
1748 'rbd/create': new TaskManagerMessage(
1750 ['create', 'Creating', 'Created'],
1752 (metadata) => `RBD '${metadata.pool_name}/${metadata.image_name}'`,
1754 // Error code and description
1755 '17': `Name is already used by RBD '${metadata.pool_name}/${
1756 metadata.image_name}'.`
1764 export class RBDFormComponent {
1767 const request = this.createRequest();
1768 // Subscribes to 'call' with submitted 'task' and handles notifications
1769 return this.taskWrapper.wrapTaskAroundCall({
1770 task: new FinishedTask('rbd/create', {
1771 pool_name: request.pool_name,
1772 image_name: request.name
1774 call: this.rbdService.create(request)
1781 REST API documentation
1782 ~~~~~~~~~~~~~~~~~~~~~~
1783 There is an automatically generated Swagger UI page for documentation of the REST
1784 API endpoints.However, by default it is not very detailed. There are two
1785 decorators that can be used to add more information:
1787 * ``@EndpointDoc()`` for documentation of endpoints. It has four optional arguments
1788 (explained below): ``description``, ``group``, ``parameters`` and
1790 * ``@ControllerDoc()`` for documentation of controller or group associated with
1791 the endpoints. It only takes the two first arguments: ``description`` and
1795 ``description``: A a string with a short (1-2 sentences) description of the object.
1798 ``group``: By default, an endpoint is grouped together with other endpoints
1799 within the same controller class. ``group`` is a string that can be used to
1800 assign an endpoint or all endpoints in a class to another controller or a
1801 conceived group name.
1804 ``parameters``: A dict used to describe path, query or request body parameters.
1805 By default, all parameters for an endpoint are listed on the Swagger UI page,
1806 including information of whether the parameter is optional/required and default
1807 values. However, there will be no description of the parameter and the parameter
1808 type will only be displayed in some cases.
1809 When adding information, each parameters should be described as in the example
1810 below. Note that the parameter type should be expressed as a built-in python
1811 type and not as a string. Allowed values are ``str``, ``int``, ``bool``, ``float``.
1813 .. code-block:: python
1815 @EndpointDoc(parameters={'my_string': (str, 'Description of my_string')})
1816 def method(my_string): pass
1818 For body parameters, more complex cases are possible. If the parameter is a
1819 dictionary, the type should be replaced with a ``dict`` containing its nested
1820 parameters. When describing nested parameters, the same format as other
1821 parameters is used. However, all nested parameters are set as required by default.
1822 If the nested parameter is optional this must be specified as for ``item2`` in
1823 the example below. If a nested parameters is set to optional, it is also
1824 possible to specify the default value (this will not be provided automatically
1825 for nested parameters).
1827 .. code-block:: python
1829 @EndpointDoc(parameters={
1831 'item1': (str, 'Description of item1'),
1832 'item2': (str, 'Description of item2', True), # item2 is optional
1833 'item3': (str, 'Description of item3', True, 'foo'), # item3 is optional with 'foo' as default value
1834 }, 'Description of my_dictionary')})
1835 def method(my_dictionary): pass
1837 If the parameter is a ``list`` of primitive types, the type should be
1838 surrounded with square brackets.
1840 .. code-block:: python
1842 @EndpointDoc(parameters={'my_list': ([int], 'Description of my_list')})
1843 def method(my_list): pass
1845 If the parameter is a ``list`` with nested parameters, the nested parameters
1846 should be placed in a dictionary and surrounded with square brackets.
1848 .. code-block:: python
1850 @EndpointDoc(parameters={
1852 'list_item': (str, 'Description of list_item'),
1853 'list_item2': (str, 'Description of list_item2')
1854 }], 'Description of my_list')})
1855 def method(my_list): pass
1858 ``responses``: A dict used for describing responses. Rules for describing
1859 responses are the same as for request body parameters, with one difference:
1860 responses also needs to be assigned to the related response code as in the
1863 .. code-block:: python
1865 @EndpointDoc(responses={
1866 '400':{'my_response': (str, 'Description of my_response')}})
1870 Error Handling in Python
1871 ~~~~~~~~~~~~~~~~~~~~~~~~
1873 Good error handling is a key requirement in creating a good user experience
1874 and providing a good API.
1876 Dashboard code should not duplicate C++ code. Thus, if error handling in C++
1877 is sufficient to provide good feedback, a new wrapper to catch these errors
1878 is not necessary. On the other hand, input validation is the best place to
1879 catch errors and generate the best error messages. If required, generate
1880 errors as soon as possible.
1882 The backend provides few standard ways of returning errors.
1884 First, there is a generic Internal Server Error::
1888 "version": <cherrypy version, e.g. 13.1.0>,
1889 "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.",
1893 For errors generated by the backend, we provide a standard error
1898 "detail": str(e), # E.g. "[errno -42] <some error message>"
1899 "component": "rbd", # this can be null to represent a global error code
1900 "code": "3", # Or a error name, e.g. "code": "some_error_key"
1904 In case, the API Endpoints uses @ViewCache to temporarily cache results,
1905 the error looks like so::
1909 "detail": str(e), # E.g. "[errno -42] <some error message>"
1910 "component": "rbd", # this can be null to represent a global error code
1911 "code": "3", # Or a error name, e.g. "code": "some_error_key"
1912 'status': 3, # Indicating the @ViewCache error status
1915 In case, the API Endpoints uses a task the error looks like so::
1919 "detail": str(e), # E.g. "[errno -42] <some error message>"
1920 "component": "rbd", # this can be null to represent a global error code
1921 "code": "3", # Or a error name, e.g. "code": "some_error_key"
1922 "task": { # Information about the task itself
1929 Our WebUI should show errors generated by the API to the user. Especially
1930 field-related errors in wizards and dialogs or show non-intrusive notifications.
1932 Handling exceptions in Python should be an exception. In general, we
1933 should have few exception handlers in our project. Per default, propagate
1934 errors to the API, as it will take care of all exceptions anyway. In general,
1935 log the exception by adding ``logger.exception()`` with a description to the
1938 We need to distinguish between user errors from internal errors and
1939 programming errors. Using different exception types will ease the
1940 task for the API layer and for the user interface:
1942 Standard Python errors, like ``SystemError``, ``ValueError`` or ``KeyError``
1943 will end up as internal server errors in the API.
1945 In general, do not ``return`` error responses in the REST API. They will be
1946 returned by the error handler. Instead, raise the appropriate exception.
1951 New functionality can be provided by means of a plug-in architecture. Among the
1952 benefits this approach brings in, loosely coupled development is one of the most
1953 notable. As the Ceph Dashboard grows in feature richness, its code-base becomes
1954 more and more complex. The hook-based nature of a plug-in architecture allows to
1955 extend functionality in a controlled manner, and isolate the scope of the
1958 Ceph Dashboard relies on `Pluggy <https://pluggy.readthedocs.io>`_ to provide
1959 for plug-ing support. On top of pluggy, an interface-based approach has been
1960 implemented, with some safety checks (method override and abstract method
1963 In order to create a new plugin, the following steps are required:
1965 #. Add a new file under ``src/pybind/mgr/dashboard/plugins``.
1966 #. Import the ``PLUGIN_MANAGER`` instance and the ``Interfaces``.
1967 #. Create a class extending the desired interfaces. The plug-in library will
1968 check if all the methods of the interfaces have been properly overridden.
1969 #. Register the plugin in the ``PLUGIN_MANAGER`` instance.
1970 #. Import the plug-in from within the Ceph Dashboard ``module.py`` (currently no
1971 dynamic loading is implemented).
1973 The available Mixins (helpers) are:
1975 - ``CanMgr``: provides the plug-in with access to the ``mgr`` instance under ``self.mgr``.
1977 The available Interfaces are:
1979 - ``Initializable``: requires overriding ``init()`` hook. This method is run at
1980 the very beginning of the dashboard module, right after all imports have been
1982 - ``Setupable``: requires overriding ``setup()`` hook. This method is run in the
1983 Ceph Dashboard ``serve()`` method, right after CherryPy has been configured,
1984 but before it is started. It's a placeholder for the plug-in initialization
1986 - ``HasOptions``: requires overriding ``get_options()`` hook by returning a list
1987 of ``Options()``. The options returned here are added to the
1989 - ``HasCommands``: requires overriding ``register_commands()`` hook by defining
1990 the commands the plug-in can handle and decorating them with ``@CLICommand``.
1991 The commands can be optionally returned, so that they can be invoked
1992 externally (which makes unit testing easier).
1993 - ``HasControllers``: requires overriding ``get_controllers()`` hook by defining
1994 and returning the controllers as usual.
1995 - ``FilterRequest.BeforeHandler``: requires overriding
1996 ``filter_request_before_handler()`` hook. This method receives a
1997 ``cherrypy.request`` object for processing. A usual implementation of this
1998 method will allow some requests to pass or will raise a ``cherrypy.HTTPError``
1999 based on the ``request`` metadata and other conditions.
2001 New interfaces and hooks should be added as soon as they are required to
2002 implement new functionality. The above list only comprises the hooks needed for
2003 the existing plugins.
2005 A sample plugin implementation would look like this:
2007 .. code-block:: python
2009 # src/pybind/mgr/dashboard/plugins/mute.py
2011 from . import PLUGIN_MANAGER as PM
2012 from . import interfaces as I
2014 from mgr_module import CLICommand, Option
2018 class Mute(I.CanMgr, I.Setupable, I.HasOptions, I.HasCommands,
2019 I.FilterRequest.BeforeHandler, I.HasControllers):
2021 def get_options(self):
2022 return [Option('mute', default=False, type='bool')]
2026 self.mute = self.mgr.get_module_option('mute')
2029 def register_commands(self):
2030 @CLICommand("dashboard mute")
2033 self.mgr.set_module_option('mute', True)
2037 def filter_request_before_handler(self, request):
2039 raise cherrypy.HTTPError(500, "I'm muted :-x")
2042 def get_controllers(self):
2043 from ..controllers import ApiController, RESTController
2045 @ApiController('/mute')
2046 class MuteController(RESTController):
2050 return [MuteController]
2053 Additionally, a helper for creating plugins ``SimplePlugin`` is provided. It
2054 facilitates the basic tasks (Options, Commands, and common Mixins). The previous
2055 plugin could be rewritten like this:
2057 .. code-block:: python
2059 from . import PLUGIN_MANAGER as PM
2060 from . import interfaces as I
2061 from .plugin import SimplePlugin as SP
2066 class Mute(SP, I.Setupable, I.FilterRequest.BeforeHandler, I.HasControllers):
2068 SP.Option('mute', default=False, type='bool')
2072 self.set_option('mute', True)
2077 SP.Command("dashboard mute", handler=shut_up)
2082 self.mute = self.get_option('mute')
2085 def filter_request_before_handler(self, request):
2087 raise cherrypy.HTTPError(500, "I'm muted :-x")
2090 def get_controllers(self):
2091 from ..controllers import ApiController, RESTController
2093 @ApiController('/mute')
2094 class MuteController(RESTController):
2098 return [MuteController]