ceph/doc/dev/mon-bootstrap.rst

   1 ===================
   2  Monitor bootstrap
   3 ===================
   4
   5 Terminology:
   6
   7 * ``cluster``: a set of monitors
   8 * ``quorum``: an active set of monitors consisting of a majority of the cluster
   9
  10 In order to initialize a new monitor, it must always be fed:
  11
  12 #. a logical name
  13 #. secret keys
  14 #. a cluster fsid (uuid)
  15
  16 In addition, a monitor needs to know two things:
  17
  18 #. what address to bind to
  19 #. who its peers are (if any)
  20
  21 There are a range of ways to do both.
  22
  23 Logical id
  24 ==========
  25
  26 The logical id should be unique across the cluster.  It will be
  27 appended to ``mon.`` to logically describe the monitor in the Ceph
  28 cluster.  For example, if the logical id is ``foo``, the monitor's
  29 name will be ``mon.foo``.
  30
  31 For most users, there is no more than one monitor per host, which
  32 makes the short hostname logical choice.
  33
  34 Secret keys
  35 ===========
  36
  37 The ``mon.`` secret key is stored a ``keyring`` file in the ``mon data`` directory.  It can be generated
  38 with a command like::
  39
  40         ceph-authtool --create-keyring /path/to/keyring --gen-key -n mon.
  41
  42 When creating a new monitor cluster, the keyring should also contain a ``client.admin`` key that can be used
  43 to administer the system::
  44
  45         ceph-authtool /path/to/keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
  46
  47 The resulting keyring is fed to ``ceph-mon --mkfs`` with the ``--keyring <keyring>`` command-line argument.
  48
  49 Cluster fsid
  50 ============
  51
  52 The cluster fsid is a normal uuid, like that generated by the ``uuidgen`` command.  It
  53 can be provided to the monitor in two ways:
  54
  55 #. via the ``--fsid <uuid>`` command-line argument (or config file option)
  56 #. via a monmap provided to the new monitor via the ``--monmap <path>`` command-line argument.
  57
  58 Monitor address
  59 ===============
  60
  61 The monitor address can be provided in several ways.
  62
  63 #. via the ``--public-addr <ip[:port]>`` command-line option (or config file option)
  64 #. via the ``--public-network <cidr>`` command-line option (or config file option)
  65 #. via the monmap provided via ``--monmap <path>``, if it includes a monitor with our name
  66 #. via the bootstrap monmap (provided via ``--inject-monmap <path>`` or generated from ``--mon-host <list>``) if it includes a monitor with no name (``noname-<something>``) and an address configured on the local host.
  67
  68 Peers
  69 =====
  70
  71 The monitor peers are provided in several ways:
  72
  73 #. via the initial monmap, provided via ``--monmap <filename>``
  74 #. via the bootstrap monmap generated from ``--mon-host <list>``
  75 #. via the bootstrap monmap generated from ``[mon.*]`` sections with the deprecated ``mon addr`` options in the config file (note that this method is *not* recommended and does not support binding to both v1 and v2 protocol addresses)
  76 #. dynamically via the admin socket
  77
  78 However, these methods are not completely interchangeable because of
  79 the complexity of creating a new monitor cluster without danger of
  80 races.
  81
  82 Cluster creation
  83 ================
  84
  85 There are three basic approaches to creating a cluster:
  86
  87 #. Create a new cluster by specifying the monitor names and addresses ahead of time.
  88 #. Create a new cluster by specifying the monitor names ahead of time, and dynamically setting the addresses as ``ceph-mon`` daemons configure themselves.
  89 #. Create a new cluster by specifying the monitor addresses ahead of time.
  90
  91
  92 Names and addresses
  93 -------------------
  94
  95 Generate a monmap using ``monmaptool`` with the names and addresses of the initial
  96 monitors.  The generated monmap will also include a cluster fsid.  Feed that monmap
  97 to each monitor daemon::
  98
  99         ceph-mon --mkfs -i <name> --monmap <initial_monmap> --keyring <initial_keyring>
 100
 101 When the daemons start, they will know exactly who they and their peers are.
 102
 103
 104 Addresses only
 105 --------------
 106
 107 The initial monitor addresses can be specified with the ``mon host`` configuration value,
 108 either via a config file or the command-line argument.  This method has the advantage that
 109 a single global config file for the cluster can have a line like::
 110
 111      mon host = a.foo.com, b.foo.com, c.foo.com
 112
 113 and will also serve to inform any ceph clients or daemons who the monitors are.
 114
 115 The ``ceph-mon`` daemons will need to be fed the initial keyring and cluster fsid to
 116 initialize themselves:
 117
 118      ceph-mon --mkfs -i <name> --fsid <uuid> --keyring <initial_keyring>
 119
 120 When the daemons first start up, they will share their names with each other and form a
 121 new cluster.
 122
 123 Names only
 124 ----------
 125
 126 In dynamic "cloud" environments, the cluster creator may not (yet)
 127 know what the addresses of the monitors are going to be.  Instead,
 128 they may want machines to configure and start themselves in parallel
 129 and, as they come up, form a new cluster on their own.  The problem is
 130 that the monitor cluster relies on strict majorities to keep itself
 131 consistent, and in order to "create" a new cluster, it needs to know
 132 what the *initial* set of monitors will be.
 133
 134 This can be done with the ``mon initial members`` config option, which
 135 should list the ids of the initial monitors that are allowed to create
 136 the cluster::
 137
 138      mon initial members = foo, bar, baz
 139
 140 The monitors can then be initialized by providing the other pieces of
 141 information (they keyring, cluster fsid, and a way of determining
 142 their own address).  For example::
 143
 144      ceph-mon --mkfs -i <name> --mon-initial-hosts 'foo,bar,baz' --keyring <initial_keyring> --public-addr <ip>
 145
 146 When these daemons are started, they will know their own address, but
 147 not their peers.  They can learn those addresses via the admin socket::
 148
 149      ceph daemon mon.<id> add_bootstrap_peer_hint <peer ip>
 150
 151 Once they learn enough of their peers from the initial member set,
 152 they will be able to create the cluster.
 153
 154
 155 Cluster expansion
 156 =================
 157
 158 Cluster expansion is slightly less demanding than creation, because
 159 the creation of the initial quorum is not an issue and there is no
 160 worry about creating separately independent clusters.
 161
 162 New nodes can be forced to join an existing cluster in two ways:
 163
 164 #. by providing no initial monitor peers addresses, and feeding them dynamically.
 165 #. by specifying the ``mon initial members`` config option to prevent the new nodes from forming a new, independent cluster, and feeding some existing monitors via any available method.
 166
 167 Initially peerless expansion
 168 ----------------------------
 169
 170 Create a new monitor and give it no peer addresses other than its own.  For
 171 example::
 172
 173      ceph-mon --mkfs -i <myid> --fsid <fsid> --keyring <mon secret key> --public-addr <ip>
 174
 175 Once the daemon starts, you can give it one or more peer addresses (preferably a bare IP address with no port; the mon will set the addr types and ports for you) to join with::
 176
 177      ceph daemon mon.<id> add_bootstrap_peer_hint <peer ip>
 178
 179 Alternatively, you can explicitly specify the addrvec_t with::
 180
 181      ceph daemon mon.<id> add_bootstrap_peer_hintv <peer addrvec>
 182
 183 For example,::
 184
 185      ceph daemon mon.new add_bootstrap_peer_hintv v2:1.2.3.4:3300,v1:1.2.3.4:6789
 186
 187 This monitor will never participate in cluster creation; it can only
 188 join an existing cluster.
 189
 190 Note that the address(es) specified should match exactly the addresses
 191 the new monitor is binding too.  If, for example, the new mon binds to
 192 only a v2 address but a v2 and v1 address are provided, there is some
 193 possibility of confusion in the mons.
 194
 195 Expanding with initial members
 196 ------------------------------
 197
 198 You can feed the new monitor some peer addresses initially and avoid badness by also
 199 setting ``mon initial members``.  For example::
 200
 201      ceph-mon --mkfs -i <myid> --fsid <fsid> --keyring <mon secret key> --public-addr <ip> --mon-host foo,bar,baz
 202
 203 When the daemon is started, ``mon initial members`` must be set via the command line or config file::
 204
 205      ceph-mon -i <myid> --mon-initial-members foo,bar,baz
 206
 207 to prevent any risk of split-brain.
 208
 209
 210
 211
 212