X-Git-Url: https://git.proxmox.com/?a=blobdiff_plain;f=ceph%2Fdoc%2Fceph-volume%2Flvm%2Fprepare.rst;h=ae6aac414cdda6567874d82fe3faabb2e8ff14d8;hb=39ae355f72b1d71f2212a99f2bd9f6c1e0d35528;hp=78a7c09107699d16008eb1e50f5c84974e4b6ffe;hpb=92f5a8d42d07f9929ae4fa7e01342fe8d96808a8;p=ceph.git diff --git a/ceph/doc/ceph-volume/lvm/prepare.rst b/ceph/doc/ceph-volume/lvm/prepare.rst index 78a7c0910..ae6aac414 100644 --- a/ceph/doc/ceph-volume/lvm/prepare.rst +++ b/ceph/doc/ceph-volume/lvm/prepare.rst @@ -2,26 +2,22 @@ ``prepare`` =========== -This subcommand allows a :term:`filestore` or :term:`bluestore` setup. It is -recommended to pre-provision a logical volume before using it with -``ceph-volume lvm``. +Before you run ``ceph-volume lvm prepare``, we recommend that you provision a +logical volume. Then you can run ``prepare`` on that logical volume. -Logical volumes are not altered except for adding extra metadata. +``prepare`` adds metadata to logical volumes but does not alter them in any +other way. -.. note:: This is part of a two step process to deploy an OSD. If looking for - a single-call way, please see :ref:`ceph-volume-lvm-create` +.. note:: This is part of a two-step process to deploy an OSD. If you prefer + to deploy an OSD by using only one command, see :ref:`ceph-volume-lvm-create`. -To help identify volumes, the process of preparing a volume (or volumes) to -work with Ceph, the tool will assign a few pieces of metadata information using -:term:`LVM tags`. - -:term:`LVM tags` makes volumes easy to discover later, and help identify them as -part of a Ceph system, and what role they have (journal, filestore, bluestore, -etc...) - -Although initially :term:`filestore` is supported (and supported by default) -the back end can be specified with: +``prepare`` uses :term:`LVM tags` to assign several pieces of metadata to a +logical volume. Volumes tagged in this way are easier to identify and easier to +use with Ceph. :term:`LVM tags` identify logical volumes by the role that they +play in the Ceph cluster (for example: BlueStore data or BlueStore WAL+DB). +:term:`BlueStore` is the default backend. Ceph permits changing +the backend, which can be done by using the following flags and arguments: * :ref:`--filestore ` * :ref:`--bluestore ` @@ -30,50 +26,58 @@ the back end can be specified with: ``bluestore`` ------------- -The :term:`bluestore` objectstore is the default for new OSDs. It offers a bit -more flexibility for devices compared to :term:`filestore`. -Bluestore supports the following configurations: - -* A block device, a block.wal, and a block.db device -* A block device and a block.wal device -* A block device and a block.db device -* A single block device - -The bluestore subcommand accepts physical block devices, partitions on -physical block devices or logical volumes as arguments for the various device parameters -If a physical device is provided, a logical volume will be created. A volume group will -either be created or reused it its name begins with ``ceph``. -This allows a simpler approach at using LVM but at the cost of flexibility: -there are no options or configurations to change how the LV is created. +:term:`Bluestore` is the default backend for new OSDs. It +offers more flexibility for devices than :term:`filestore` does. Bluestore +supports the following configurations: + +* a block device, a block.wal device, and a block.db device +* a block device and a block.wal device +* a block device and a block.db device +* a single block device + +The ``bluestore`` subcommand accepts physical block devices, partitions on physical +block devices, or logical volumes as arguments for the various device +parameters. If a physical block device is provided, a logical volume will be +created. If the provided volume group's name begins with `ceph`, it will be +created if it does not yet exist and it will be clobbered and reused if it +already exists. This allows for a simpler approach to using LVM but at the +cost of flexibility: no option or configuration can be used to change how the +logical volume is created. The ``block`` is specified with the ``--data`` flag, and in its simplest use -case it looks like:: +case it looks like: + +.. prompt:: bash # ceph-volume lvm prepare --bluestore --data vg/lv -A raw device can be specified in the same way:: +A raw device can be specified in the same way: + +.. prompt:: bash # ceph-volume lvm prepare --bluestore --data /path/to/device -For enabling :ref:`encryption `, the ``--dmcrypt`` flag is required:: +For enabling :ref:`encryption `, the ``--dmcrypt`` flag is required: + +.. prompt:: bash # ceph-volume lvm prepare --bluestore --dmcrypt --data vg/lv -If a ``block.db`` or a ``block.wal`` is needed (they are optional for -bluestore) they can be specified with ``--block.db`` and ``--block.wal`` -accordingly. These can be a physical device, a partition or -a logical volume. +If a ``block.db`` device or a ``block.wal`` device is needed, it can be +specified with ``--block.db`` or ``--block.wal``. These can be physical +devices, partitions, or logical volumes. ``block.db`` and ``block.wal`` are +optional for bluestore. -For both ``block.db`` and ``block.wal`` partitions aren't made logical volumes -because they can be used as-is. +For both ``block.db`` and ``block.wal``, partitions can be used as-is, and +therefore are not made into logical volumes. -While creating the OSD directory, the process will use a ``tmpfs`` mount to -place all the files needed for the OSD. These files are initially created by -``ceph-osd --mkfs`` and are fully ephemeral. +While creating the OSD directory, the process uses a ``tmpfs`` mount to hold +the files needed for the OSD. These files are created by ``ceph-osd --mkfs`` +and are ephemeral. -A symlink is always created for the ``block`` device, and optionally for -``block.db`` and ``block.wal``. For a cluster with a default name, and an OSD -id of 0, the directory could look like:: +A symlink is created for the ``block`` device, and is optional for ``block.db`` +and ``block.wal``. For a cluster with a default name and an OSD ID of 0, the +directory looks like this:: # ls -l /var/lib/ceph/osd/ceph-0 lrwxrwxrwx. 1 ceph ceph 93 Oct 20 13:05 block -> /dev/ceph-be2b6fbd-bcf2-4c51-b35d-a35a162a02f0/osd-block-25cf0a05-2bc6-44ef-9137-79d65bd7ad62 @@ -86,11 +90,11 @@ id of 0, the directory could look like:: -rw-------. 1 ceph ceph 10 Oct 20 13:05 type -rw-------. 1 ceph ceph 2 Oct 20 13:05 whoami -In the above case, a device was used for ``block`` so ``ceph-volume`` create -a volume group and a logical volume using the following convention: +In the above case, a device was used for ``block``, so ``ceph-volume`` created +a volume group and a logical volume using the following conventions: -* volume group name: ``ceph-{cluster fsid}`` or if the vg exists already - ``ceph-{random uuid}`` +* volume group name: ``ceph-{cluster fsid}`` (or if the volume group already + exists: ``ceph-{random uuid}``) * logical volume name: ``osd-block-{osd_fsid}`` @@ -99,78 +103,100 @@ a volume group and a logical volume using the following convention: ``filestore`` ------------- -This is the OSD backend that allows preparation of logical volumes for -a :term:`filestore` objectstore OSD. +``Filestore`` is the OSD backend that prepares logical volumes for a +:term:`filestore`-backed object-store OSD. + + +``Filestore`` uses a logical volume to store OSD data and it uses +physical devices, partitions, or logical volumes to store the journal. If a +physical device is used to create a filestore backend, a logical volume will be +created on that physical device. If the provided volume group's name begins +with `ceph`, it will be created if it does not yet exist and it will be +clobbered and reused if it already exists. No special preparation is needed for +these volumes, but be sure to meet the minimum size requirements for OSD data and +for the journal. + +Use the following command to create a basic filestore OSD: -It can use a logical volume for the OSD data and a physical device, a partition -or logical volume for the journal. A physical device will have a logical volume -created on it. A volume group will either be created or reused it its name begins -with ``ceph``. No special preparation is needed for these volumes other than -following the minimum size requirements for data and journal. +.. prompt:: bash # -The CLI call looks like this of a basic standalone filestore OSD:: + ceph-volume lvm prepare --filestore --data - ceph-volume lvm prepare --filestore --data +Use this command to deploy filestore with an external journal: -To deploy file store with an external journal:: +.. prompt:: bash # - ceph-volume lvm prepare --filestore --data --journal + ceph-volume lvm prepare --filestore --data --journal -For enabling :ref:`encryption `, the ``--dmcrypt`` flag is required:: +Use this command to enable :ref:`encryption `, and note that the ``--dmcrypt`` flag is required: - ceph-volume lvm prepare --filestore --dmcrypt --data --journal +.. prompt:: bash # -Both the journal and data block device can take three forms: + ceph-volume lvm prepare --filestore --dmcrypt --data --journal + +The data block device and the journal can each take one of three forms: * a physical block device * a partition on a physical block device * a logical volume -When using logical volumes the value *must* be of the format -``volume_group/logical_volume``. Since logical volume names -are not enforced for uniqueness, this prevents accidentally -choosing the wrong volume. +If you use a logical volume to deploy filestore, the value that you pass in the +command *must* be of the format ``volume_group/logical_volume_name``. Since logical +volume names are not enforced for uniqueness, using this format is an important +safeguard against accidentally choosing the wrong volume (and clobbering its data). + +If you use a partition to deploy filestore, the partition *must* contain a +``PARTUUID`` that can be discovered by ``blkid``. This ensures that the +partition can be identified correctly regardless of the device's name (or path). -When using a partition, it *must* contain a ``PARTUUID``, that can be -discovered by ``blkid``. THis ensure it can later be identified correctly -regardless of the device name (or path). +For example, to use a logical volume for OSD data and a partition +(``/dev/sdc1``) for the journal, run a command of this form: -For example: passing a logical volume for data and a partition ``/dev/sdc1`` for -the journal:: +.. prompt:: bash # - ceph-volume lvm prepare --filestore --data volume_group/lv_name --journal /dev/sdc1 + ceph-volume lvm prepare --filestore --data volume_group/logical_volume_name --journal /dev/sdc1 -Passing a bare device for data and a logical volume ias the journal:: +Or, to use a bare device for data and a logical volume for the journal: - ceph-volume lvm prepare --filestore --data /dev/sdc --journal volume_group/journal_lv +.. prompt:: bash # -A generated uuid is used to ask the cluster for a new OSD. These two pieces are -crucial for identifying an OSD and will later be used throughout the -:ref:`ceph-volume-lvm-activate` process. + ceph-volume lvm prepare --filestore --data /dev/sdc --journal volume_group/journal_lv + +A generated UUID is used when asking the cluster for a new OSD. These two +pieces of information (the OSD ID and the OSD UUID) are necessary for +identifying a given OSD and will later be used throughout the +:ref:`activation` process. The OSD data directory is created using the following convention:: /var/lib/ceph/osd/- -At this point the data volume is mounted at this location, and the journal -volume is linked:: +To link the journal volume to the mounted data volume, use this command: + +.. prompt:: bash # - ln -s /path/to/journal /var/lib/ceph/osd/-/journal + ln -s /path/to/journal /var/lib/ceph/osd/-/journal -The monmap is fetched using the bootstrap key from the OSD:: +To fetch the monmap by using the bootstrap key from the OSD, use this command: - /usr/bin/ceph --cluster ceph --name client.bootstrap-osd - --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring - mon getmap -o /var/lib/ceph/osd/-/activate.monmap +.. prompt:: bash # -``ceph-osd`` will be called to populate the OSD directory, that is already -mounted, re-using all the pieces of information from the initial steps:: + /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring + /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o + /var/lib/ceph/osd/-/activate.monmap + +To populate the OSD directory (which has already been mounted), use this ``ceph-osd`` command: +.. prompt:: bash # + + ceph-osd --cluster ceph --mkfs --mkkey -i \ --monmap + /var/lib/ceph/osd/-/activate.monmap --osd-data \ + /var/lib/ceph/osd/- --osd-journal + /var/lib/ceph/osd/-/journal \ --osd-uuid + --keyring /var/lib/ceph/osd/-/keyring \ --setuser ceph + --setgroup ceph + +All of the information from the previous steps is used in the above command. - ceph-osd --cluster ceph --mkfs --mkkey -i \ - --monmap /var/lib/ceph/osd/-/activate.monmap --osd-data \ - /var/lib/ceph/osd/- --osd-journal /var/lib/ceph/osd/-/journal \ - --osd-uuid --keyring /var/lib/ceph/osd/-/keyring \ - --setuser ceph --setgroup ceph .. _ceph-volume-lvm-partitions: @@ -254,28 +280,20 @@ work for both bluestore and filestore OSDs:: ``multipath`` support --------------------- -Devices that come from ``multipath`` are not supported as-is. The tool will -refuse to consume a raw multipath device and will report a message like:: - - --> RuntimeError: Cannot use device (/dev/mapper/). A vg/lv path or an existing device is needed +``multipath`` devices are supported if ``lvm`` is configured properly. -The reason for not supporting multipath is that depending on the type of the -multipath setup, if using an active/passive array as the underlying physical -devices, filters are required in ``lvm.conf`` to exclude the disks that are part of -those underlying devices. +**Leave it to LVM** -It is unfeasible for ceph-volume to understand what type of configuration is -needed for LVM to be able to work in various different multipath scenarios. The -functionality to create the LV for you is merely a (naive) convenience, -anything that involves different settings or configuration must be provided by -a config management system which can then provide VGs and LVs for ceph-volume -to consume. +Most Linux distributions should ship their LVM2 package with +``multipath_component_detection = 1`` in the default configuration. With this +setting ``LVM`` ignores any device that is a multipath component and +``ceph-volume`` will accordingly not touch these devices. -This situation will only arise when trying to use the ceph-volume functionality -that creates a volume group and logical volume from a device. If a multipath -device is already a logical volume it *should* work, given that the LVM -configuration is done correctly to avoid issues. +**Using filters** +Should this setting be unavailable, a correct ``filter`` expression must be +provided in ``lvm.conf``. ``ceph-volume`` must not be able to use both the +multipath device and its multipath components. Storing metadata ----------------