ceph/src/rocksdb/tools/advisor/README.md

   1 # Rocksdb Tuning Advisor
   2
   3 ## Motivation
   4
   5 The performance of Rocksdb is contingent on its tuning. However,
   6 because of the complexity of its underlying technology and a large number of
   7 configurable parameters, a good configuration is sometimes hard to obtain. The aim of
   8 the python command-line tool, Rocksdb Advisor, is to automate the process of
   9 suggesting improvements in the configuration based on advice from Rocksdb
  10 experts.
  11
  12 ## Overview
  13
  14 Experts share their wisdom as rules comprising of conditions and suggestions in the INI format (refer
  15 [rules.ini](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rules.ini)).
  16 Users provide the Rocksdb configuration that they want to improve upon (as the
  17 familiar Rocksdb OPTIONS file —
  18 [example](https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini))
  19 and the path of the file which contains Rocksdb logs and statistics.
  20 The [Advisor](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rule_parser_example.py)
  21 creates appropriate DataSource objects (for Rocksdb
  22 [logs](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_log_parser.py),
  23 [options](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_options_parser.py),
  24 [statistics](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_stats_fetcher.py) etc.)
  25 and provides them to the [Rules Engine](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rule_parser.py).
  26 The Rules uses rules from experts to parse data-sources and trigger appropriate rules.
  27 The Advisor's output gives information about which rules were triggered,
  28 why they were triggered and what each of them suggests. Each suggestion
  29 provided by a triggered rule advises some action on a Rocksdb
  30 configuration option, for example, increase CFOptions.write_buffer_size,
  31 set bloom_bits to 2 etc.
  32
  33 ## Usage
  34
  35 ### Prerequisites
  36 The tool needs the following to run:
  37 * python3
  38
  39 ### Running the tool
  40 An example command to run the tool:
  41
  42 ```shell
  43 cd rocksdb/tools/advisor
  44 python3 -m advisor.rule_parser_example --rules_spec=advisor/rules.ini --rocksdb_options=test/input_files/OPTIONS-000005 --log_files_path_prefix=test/input_files/LOG-0 --stats_dump_period_sec=20
  45 ```
  46
  47 ### Command-line arguments
  48
  49 Most important amongst all the input that the Advisor needs, are the rules
  50 spec and starting Rocksdb configuration. The configuration is provided as the
  51 familiar Rocksdb Options file (refer [example](https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini)).
  52 The Rules spec is written in the INI format (more details in
  53 [rules.ini](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rules.ini)).
  54
  55 In brief, a Rule is made of conditions and is triggered when all its
  56 constituent conditions are triggered. When triggered, a Rule suggests changes
  57 (increase/decrease/set to a suggested value) to certain Rocksdb options that
  58 aim to improve Rocksdb performance. Every Condition has a 'source' i.e.
  59 the data source that would be checked for triggering that condition.
  60 For example, a log Condition (with 'source=LOG') is triggered if a particular
  61 'regex' is found in the Rocksdb LOG files. As of now the Rules Engine
  62 supports 3 types of Conditions (and consequently data-sources):
  63 LOG, OPTIONS, TIME_SERIES. The TIME_SERIES data can be sourced from the
  64 Rocksdb [statistics](https://github.com/facebook/rocksdb/blob/master/include/rocksdb/statistics.h)
  65 or [perf context](https://github.com/facebook/rocksdb/blob/master/include/rocksdb/perf_context.h).
  66
  67 For more information about the remaining command-line arguments, run:
  68
  69 ```shell
  70 cd rocksdb/tools/advisor
  71 python3 -m advisor.rule_parser_example --help
  72 ```
  73
  74 ### Sample output
  75
  76 Here, a Rocksdb log-based rule has been triggered:
  77
  78 ```shell
  79 Rule: stall-too-many-memtables
  80 LogCondition: stall-too-many-memtables regex: Stopping writes because we have \d+ immutable memtables \(waiting for flush\), max_write_buffer_number is set to \d+
  81 Suggestion: inc-bg-flush option : DBOptions.max_background_flushes action : increase suggested_values : ['2']
  82 Suggestion: inc-write-buffer option : CFOptions.max_write_buffer_number action : increase
  83 scope: col_fam:
  84 {'default'}
  85 ```
  86
  87 ## Running the tests
  88
  89 Tests for the code have been added to the
  90 [test/](https://github.com/facebook/rocksdb/tree/master/tools/advisor/test)
  91 directory. For example, to run the unit tests for db_log_parser.py:
  92
  93 ```shell
  94 cd rocksdb/tools/advisor
  95 python3 -m unittest -v test.test_db_log_parser
  96 ```