1 ==========================
2 Introduction to librados
3 ==========================
5 The :term:`Ceph Storage Cluster` provides the basic storage service that allows
6 :term:`Ceph` to uniquely deliver **object, block, and file storage** in one
7 unified system. However, you are not limited to using the RESTful, block, or
8 POSIX interfaces. Based upon :abbr:`RADOS (Reliable Autonomic Distributed Object
9 Store)`, the ``librados`` API enables you to create your own interface to the
12 The ``librados`` API enables you to interact with the two types of daemons in
13 the Ceph Storage Cluster:
15 - The :term:`Ceph Monitor`, which maintains a master copy of the cluster map.
16 - The :term:`Ceph OSD Daemon` (OSD), which stores data as objects on a storage node.
19 +---------------------------------+
20 | Ceph Storage Cluster Protocol |
22 +---------------------------------+
23 +---------------+ +---------------+
25 +---------------+ +---------------+
27 This guide provides a high-level introduction to using ``librados``.
28 Refer to :doc:`../../architecture` for additional details of the Ceph
29 Storage Cluster. To use the API, you need a running Ceph Storage Cluster.
30 See `Installation (Quick)`_ for details.
33 Step 1: Getting librados
34 ========================
36 Your client application must bind with ``librados`` to connect to the Ceph
37 Storage Cluster. You must install ``librados`` and any required packages to
38 write applications that use ``librados``. The ``librados`` API is written in
39 C++, with additional bindings for C, Python, Java and PHP.
42 Getting librados for C/C++
43 --------------------------
45 To install ``librados`` development support files for C/C++ on Debian/Ubuntu
46 distributions, execute the following::
48 sudo apt-get install librados-dev
50 To install ``librados`` development support files for C/C++ on RHEL/CentOS
51 distributions, execute the following::
53 sudo yum install librados2-devel
55 Once you install ``librados`` for developers, you can find the required
56 headers for C/C++ under ``/usr/include/rados``. ::
61 Getting librados for Python
62 ---------------------------
64 The ``rados`` module provides ``librados`` support to Python
65 applications. The ``librados-dev`` package for Debian/Ubuntu
66 and the ``librados2-devel`` package for RHEL/CentOS will install the
67 ``python-rados`` package for you. You may install ``python-rados``
70 To install ``librados`` development support files for Python on Debian/Ubuntu
71 distributions, execute the following::
73 sudo apt-get install python-rados
75 To install ``librados`` development support files for Python on RHEL/CentOS
76 distributions, execute the following::
78 sudo yum install python-rados
80 You can find the module under ``/usr/share/pyshared`` on Debian systems,
81 or under ``/usr/lib/python*/site-packages`` on CentOS/RHEL systems.
84 Getting librados for Java
85 -------------------------
87 To install ``librados`` for Java, you need to execute the following procedure:
89 #. Install ``jna.jar``. For Debian/Ubuntu, execute::
91 sudo apt-get install libjna-java
93 For CentOS/RHEL, execute::
97 The JAR files are located in ``/usr/share/java``.
99 #. Clone the ``rados-java`` repository::
101 git clone --recursive https://github.com/ceph/rados-java.git
103 #. Build the ``rados-java`` repository::
108 The JAR file is located under ``rados-java/target``.
110 #. Copy the JAR for RADOS to a common location (e.g., ``/usr/share/java``) and
111 ensure that it and the JNA JAR are in your JVM's classpath. For example::
113 sudo cp target/rados-0.1.3.jar /usr/share/java/rados-0.1.3.jar
114 sudo ln -s /usr/share/java/jna-3.2.7.jar /usr/lib/jvm/default-java/jre/lib/ext/jna-3.2.7.jar
115 sudo ln -s /usr/share/java/rados-0.1.3.jar /usr/lib/jvm/default-java/jre/lib/ext/rados-0.1.3.jar
117 To build the documentation, execute the following::
122 Getting librados for PHP
123 -------------------------
125 To install the ``librados`` extension for PHP, you need to execute the following procedure:
127 #. Install php-dev. For Debian/Ubuntu, execute::
129 sudo apt-get install php5-dev build-essential
131 For CentOS/RHEL, execute::
133 sudo yum install php-devel
135 #. Clone the ``phprados`` repository::
137 git clone https://github.com/ceph/phprados.git
139 #. Build ``phprados``::
147 #. Enable ``phprados`` in php.ini by adding::
152 Step 2: Configuring a Cluster Handle
153 ====================================
155 A :term:`Ceph Client`, via ``librados``, interacts directly with OSDs to store
156 and retrieve data. To interact with OSDs, the client app must invoke
157 ``librados`` and connect to a Ceph Monitor. Once connected, ``librados``
158 retrieves the :term:`Cluster Map` from the Ceph Monitor. When the client app
159 wants to read or write data, it creates an I/O context and binds to a
160 :term:`Pool`. The pool has an associated :term:`CRUSH rule` that defines how it
161 will place data in the storage cluster. Via the I/O context, the client
162 provides the object name to ``librados``, which takes the object name
163 and the cluster map (i.e., the topology of the cluster) and `computes`_ the
164 placement group and `OSD`_ for locating the data. Then the client application
165 can read or write data. The client app doesn't need to learn about the topology
166 of the cluster directly.
169 +--------+ Retrieves +---------------+
170 | Client |------------>| Cluster Map |
171 +--------+ +---------------+
179 +--------+ +---------------+
180 | Pool |---------->| CRUSH Rule |
181 +--------+ Selects +---------------+
184 The Ceph Storage Cluster handle encapsulates the client configuration, including:
186 - The `user ID`_ for ``rados_create()`` or user name for ``rados_create2()``
188 - The :term:`cephx` authentication key
189 - The monitor ID and IP address
193 Thus, the first steps in using the cluster from your app are to 1) create
194 a cluster handle that your app will use to connect to the storage cluster,
195 and then 2) use that handle to connect. To connect to the cluster, the
196 app must supply a monitor address, a username and an authentication key
197 (cephx is enabled by default).
199 .. tip:: Talking to different Ceph Storage Clusters – or to the same cluster
200 with different users – requires different cluster handles.
202 RADOS provides a number of ways for you to set the required values. For
203 the monitor and encryption key settings, an easy way to handle them is to ensure
204 that your Ceph configuration file contains a ``keyring`` path to a keyring file
205 and at least one monitor address (e.g., ``mon host``). For example::
208 mon host = 192.168.1.1
209 keyring = /etc/ceph/ceph.client.admin.keyring
211 Once you create the handle, you can read a Ceph configuration file to configure
212 the handle. You can also pass arguments to your app and parse them with the
213 function for parsing command line arguments (e.g., ``rados_conf_parse_argv()``),
214 or parse Ceph environment variables (e.g., ``rados_conf_parse_env()``). Some
215 wrappers may not implement convenience methods, so you may need to implement
216 these capabilities. The following diagram provides a high-level flow for the
221 +---------+ +---------+
222 | Client | | Monitor |
223 +---------+ +---------+
241 Once connected, your app can invoke functions that affect the whole cluster
242 with only the cluster handle. For example, once you have a cluster
245 - Get cluster statistics
246 - Use Pool Operation (exists, create, list, delete)
247 - Get and set the configuration
250 One of the powerful features of Ceph is the ability to bind to different pools.
251 Each pool may have a different number of placement groups, object replicas and
252 replication strategies. For example, a pool could be set up as a "hot" pool that
253 uses SSDs for frequently used objects or a "cold" pool that uses erasure coding.
255 The main difference in the various ``librados`` bindings is between C and
256 the object-oriented bindings for C++, Java and Python. The object-oriented
257 bindings use objects to represent cluster handles, IO Contexts, iterators,
264 For C, creating a simple cluster handle using the ``admin`` user, configuring
265 it and connecting to the cluster might look something like this:
272 #include <rados/librados.h>
274 int main (int argc, const char **argv)
277 /* Declare the cluster handle and required arguments. */
279 char cluster_name[] = "ceph";
280 char user_name[] = "client.admin";
283 /* Initialize the cluster handle with the "ceph" cluster name and the "client.admin" user */
285 err = rados_create2(&cluster, cluster_name, user_name, flags);
288 fprintf(stderr, "%s: Couldn't create the cluster handle! %s\n", argv[0], strerror(-err));
291 printf("\nCreated a cluster handle.\n");
295 /* Read a Ceph configuration file to configure the cluster handle. */
296 err = rados_conf_read_file(cluster, "/etc/ceph/ceph.conf");
298 fprintf(stderr, "%s: cannot read config file: %s\n", argv[0], strerror(-err));
301 printf("\nRead the config file.\n");
304 /* Read command line arguments */
305 err = rados_conf_parse_argv(cluster, argc, argv);
307 fprintf(stderr, "%s: cannot parse command line arguments: %s\n", argv[0], strerror(-err));
310 printf("\nRead the command line arguments.\n");
313 /* Connect to the cluster */
314 err = rados_connect(cluster);
316 fprintf(stderr, "%s: cannot connect to cluster: %s\n", argv[0], strerror(-err));
319 printf("\nConnected to the cluster.\n");
324 Compile your client and link to ``librados`` using ``-lrados``. For example::
326 gcc ceph-client.c -lrados -o ceph-client
332 The Ceph project provides a C++ example in the ``ceph/examples/librados``
333 directory. For C++, a simple cluster handle using the ``admin`` user requires
334 you to initialize a ``librados::Rados`` cluster handle object:
340 #include <rados/librados.hpp>
342 int main(int argc, const char **argv)
347 /* Declare the cluster handle and required variables. */
348 librados::Rados cluster;
349 char cluster_name[] = "ceph";
350 char user_name[] = "client.admin";
353 /* Initialize the cluster handle with the "ceph" cluster name and "client.admin" user */
355 ret = cluster.init2(user_name, cluster_name, flags);
357 std::cerr << "Couldn't initialize the cluster handle! error " << ret << std::endl;
360 std::cout << "Created a cluster handle." << std::endl;
364 /* Read a Ceph configuration file to configure the cluster handle. */
366 ret = cluster.conf_read_file("/etc/ceph/ceph.conf");
368 std::cerr << "Couldn't read the Ceph configuration file! error " << ret << std::endl;
371 std::cout << "Read the Ceph configuration file." << std::endl;
375 /* Read command line arguments */
377 ret = cluster.conf_parse_argv(argc, argv);
379 std::cerr << "Couldn't parse command line options! error " << ret << std::endl;
382 std::cout << "Parsed command line options." << std::endl;
386 /* Connect to the cluster */
388 ret = cluster.connect();
390 std::cerr << "Couldn't connect to cluster! error " << ret << std::endl;
393 std::cout << "Connected to the cluster." << std::endl;
401 Compile the source; then, link ``librados`` using ``-lrados``.
404 g++ -g -c ceph-client.cc -o ceph-client.o
405 g++ -g ceph-client.o -lrados -o ceph-client
412 Python uses the ``admin`` id and the ``ceph`` cluster name by default, and
413 will read the standard ``ceph.conf`` file if the conffile parameter is
414 set to the empty string. The Python binding converts C++ errors
418 .. code-block:: python
423 cluster = rados.Rados(conffile='')
424 except TypeError as e:
425 print('Argument validation error: {}'.format(e))
428 print("Created cluster handle.")
432 except Exception as e:
433 print("connection error: {}".format(e))
436 print("Connected to the cluster.")
439 Execute the example to verify that it connects to your cluster. ::
441 python ceph-client.py
447 Java requires you to specify the user ID (``admin``) or user name
448 (``client.admin``), and uses the ``ceph`` cluster name by default . The Java
449 binding converts C++-based errors into exceptions.
453 import com.ceph.rados.Rados;
454 import com.ceph.rados.RadosException;
458 public class CephClient {
459 public static void main (String args[]){
462 Rados cluster = new Rados("admin");
463 System.out.println("Created cluster handle.");
465 File f = new File("/etc/ceph/ceph.conf");
466 cluster.confReadFile(f);
467 System.out.println("Read the configuration file.");
470 System.out.println("Connected to the cluster.");
472 } catch (RadosException e) {
473 System.out.println(e.getMessage() + ": " + e.getReturnValue());
479 Compile the source; then, run it. If you have copied the JAR to
480 ``/usr/share/java`` and sym linked from your ``ext`` directory, you won't need
481 to specify the classpath. For example::
483 javac CephClient.java
490 With the RADOS extension enabled in PHP you can start creating a new cluster handle very easily:
497 rados_conf_read_file($r, '/etc/ceph/ceph.conf');
498 if (!rados_connect($r)) {
499 echo "Failed to connect to Ceph cluster";
501 echo "Successfully connected to Ceph cluster";
505 Save this as rados.php and run the code::
510 Step 3: Creating an I/O Context
511 ===============================
513 Once your app has a cluster handle and a connection to a Ceph Storage Cluster,
514 you may create an I/O Context and begin reading and writing data. An I/O Context
515 binds the connection to a specific pool. The user must have appropriate
516 `CAPS`_ permissions to access the specified pool. For example, a user with read
517 access but not write access will only be able to read data. I/O Context
518 functionality includes:
520 - Write/read data and extended attributes
521 - List and iterate over objects and extended attributes
522 - Snapshot pools, list snapshots, etc.
526 +---------+ +---------+ +---------+
527 | Client | | Monitor | | OSD |
528 +---------+ +---------+ +---------+
535 |---------------+-------------->|
538 |<--------------+---------------|
541 |---------------+-------------->|
544 |<--------------+---------------|
547 |---------------+-------------->|
550 |<--------------+---------------|
553 |---------------+-------------->|
556 |<--------------+---------------|
560 RADOS enables you to interact both synchronously and asynchronously. Once your
561 app has an I/O Context, read/write operations only require you to know the
562 object/xattr name. The CRUSH algorithm encapsulated in ``librados`` uses the
563 cluster map to identify the appropriate OSD. OSD daemons handle the replication,
564 as described in `Smart Daemons Enable Hyperscale`_. The ``librados`` library also
565 maps objects to placement groups, as described in `Calculating PG IDs`_.
567 The following examples use the default ``data`` pool. However, you may also
568 use the API to list pools, ensure they exist, or create and delete pools. For
569 the write operations, the examples illustrate how to use synchronous mode. For
570 the read operations, the examples illustrate how to use asynchronous mode.
572 .. important:: Use caution when deleting pools with this API. If you delete
573 a pool, the pool and ALL DATA in the pool will be lost.
585 #include <rados/librados.h>
587 int main (int argc, const char **argv)
590 * Continued from previous C example, where cluster handle and
591 * connection are established. First declare an I/O Context.
595 char *poolname = "data";
597 err = rados_ioctx_create(cluster, poolname, &io);
599 fprintf(stderr, "%s: cannot open rados pool %s: %s\n", argv[0], poolname, strerror(-err));
600 rados_shutdown(cluster);
603 printf("\nCreated I/O context.\n");
606 /* Write data to the cluster synchronously. */
607 err = rados_write(io, "hw", "Hello World!", 12, 0);
609 fprintf(stderr, "%s: Cannot write object \"hw\" to pool %s: %s\n", argv[0], poolname, strerror(-err));
610 rados_ioctx_destroy(io);
611 rados_shutdown(cluster);
614 printf("\nWrote \"Hello World\" to object \"hw\".\n");
617 char xattr[] = "en_US";
618 err = rados_setxattr(io, "hw", "lang", xattr, 5);
620 fprintf(stderr, "%s: Cannot write xattr to pool %s: %s\n", argv[0], poolname, strerror(-err));
621 rados_ioctx_destroy(io);
622 rados_shutdown(cluster);
625 printf("\nWrote \"en_US\" to xattr \"lang\" for object \"hw\".\n");
629 * Read data from the cluster asynchronously.
630 * First, set up asynchronous I/O completion.
632 rados_completion_t comp;
633 err = rados_aio_create_completion(NULL, NULL, NULL, &comp);
635 fprintf(stderr, "%s: Could not create aio completion: %s\n", argv[0], strerror(-err));
636 rados_ioctx_destroy(io);
637 rados_shutdown(cluster);
640 printf("\nCreated AIO completion.\n");
643 /* Next, read data using rados_aio_read. */
645 err = rados_aio_read(io, "hw", comp, read_res, 12, 0);
647 fprintf(stderr, "%s: Cannot read object. %s %s\n", argv[0], poolname, strerror(-err));
648 rados_ioctx_destroy(io);
649 rados_shutdown(cluster);
652 printf("\nRead object \"hw\". The contents are:\n %s \n", read_res);
655 /* Wait for the operation to complete */
656 rados_aio_wait_for_complete(comp);
658 /* Release the asynchronous I/O complete handle to avoid memory leaks. */
659 rados_aio_release(comp);
663 err = rados_getxattr(io, "hw", "lang", xattr_res, 5);
665 fprintf(stderr, "%s: Cannot read xattr. %s %s\n", argv[0], poolname, strerror(-err));
666 rados_ioctx_destroy(io);
667 rados_shutdown(cluster);
670 printf("\nRead xattr \"lang\" for object \"hw\". The contents are:\n %s \n", xattr_res);
673 err = rados_rmxattr(io, "hw", "lang");
675 fprintf(stderr, "%s: Cannot remove xattr. %s %s\n", argv[0], poolname, strerror(-err));
676 rados_ioctx_destroy(io);
677 rados_shutdown(cluster);
680 printf("\nRemoved xattr \"lang\" for object \"hw\".\n");
683 err = rados_remove(io, "hw");
685 fprintf(stderr, "%s: Cannot remove object. %s %s\n", argv[0], poolname, strerror(-err));
686 rados_ioctx_destroy(io);
687 rados_shutdown(cluster);
690 printf("\nRemoved object \"hw\".\n");
705 #include <rados/librados.hpp>
707 int main(int argc, const char **argv)
710 /* Continued from previous C++ example, where cluster handle and
711 * connection are established. First declare an I/O Context.
714 librados::IoCtx io_ctx;
715 const char *pool_name = "data";
718 ret = cluster.ioctx_create(pool_name, io_ctx);
720 std::cerr << "Couldn't set up ioctx! error " << ret << std::endl;
723 std::cout << "Created an ioctx for the pool." << std::endl;
728 /* Write an object synchronously. */
730 librados::bufferlist bl;
731 bl.append("Hello World!");
732 ret = io_ctx.write_full("hw", bl);
734 std::cerr << "Couldn't write object! error " << ret << std::endl;
737 std::cout << "Wrote new object 'hw' " << std::endl;
743 * Add an xattr to the object.
746 librados::bufferlist lang_bl;
747 lang_bl.append("en_US");
748 ret = io_ctx.setxattr("hw", "lang", lang_bl);
750 std::cerr << "failed to set xattr version entry! error "
754 std::cout << "Set the xattr 'lang' on our object!" << std::endl;
760 * Read the object back asynchronously.
763 librados::bufferlist read_buf;
764 int read_len = 4194304;
766 //Create I/O Completion.
767 librados::AioCompletion *read_completion = librados::Rados::aio_create_completion();
770 ret = io_ctx.aio_read("hw", read_completion, &read_buf, read_len, 0);
772 std::cerr << "Couldn't start read object! error " << ret << std::endl;
776 // Wait for the request to complete, and check that it succeeded.
777 read_completion->wait_for_complete();
778 ret = read_completion->get_return_value();
780 std::cerr << "Couldn't read object! error " << ret << std::endl;
783 std::cout << "Read object hw asynchronously with contents.\n"
784 << read_buf.c_str() << std::endl;
793 librados::bufferlist lang_res;
794 ret = io_ctx.getxattr("hw", "lang", lang_res);
796 std::cerr << "failed to get xattr version entry! error "
800 std::cout << "Got the xattr 'lang' from object hw!"
801 << lang_res.c_str() << std::endl;
810 ret = io_ctx.rmxattr("hw", "lang");
812 std::cerr << "Failed to remove xattr! error "
816 std::cout << "Removed the xattr 'lang' from our object!" << std::endl;
824 ret = io_ctx.remove("hw");
826 std::cerr << "Couldn't remove object! error " << ret << std::endl;
829 std::cout << "Removed object 'hw'." << std::endl;
839 .. code-block:: python
841 print("\n\nI/O Context and Object Operations")
842 print("=================================")
844 print("\nCreating a context for the 'data' pool")
845 if not cluster.pool_exists('data'):
846 raise RuntimeError('No data pool exists')
847 ioctx = cluster.open_ioctx('data')
849 print("\nWriting object 'hw' with contents 'Hello World!' to pool 'data'.")
850 ioctx.write("hw", b"Hello World!")
851 print("Writing XATTR 'lang' with value 'en_US' to object 'hw'")
852 ioctx.set_xattr("hw", "lang", b"en_US")
855 print("\nWriting object 'bm' with contents 'Bonjour tout le monde!' to pool
857 ioctx.write("bm", b"Bonjour tout le monde!")
858 print("Writing XATTR 'lang' with value 'fr_FR' to object 'bm'")
859 ioctx.set_xattr("bm", "lang", b"fr_FR")
861 print("\nContents of object 'hw'\n------------------------")
862 print(ioctx.read("hw"))
864 print("\n\nGetting XATTR 'lang' from object 'hw'")
865 print(ioctx.get_xattr("hw", "lang"))
867 print("\nContents of object 'bm'\n------------------------")
868 print(ioctx.read("bm"))
870 print("\n\nGetting XATTR 'lang' from object 'bm'")
871 print(ioctx.get_xattr("bm", "lang"))
874 print("\nRemoving object 'hw'")
875 ioctx.remove_object("hw")
877 print("Removing object 'bm'")
878 ioctx.remove_object("bm")
886 import com.ceph.rados.Rados;
887 import com.ceph.rados.RadosException;
890 import com.ceph.rados.IoCTX;
892 public class CephClient {
893 public static void main (String args[]){
896 Rados cluster = new Rados("admin");
897 System.out.println("Created cluster handle.");
899 File f = new File("/etc/ceph/ceph.conf");
900 cluster.confReadFile(f);
901 System.out.println("Read the configuration file.");
904 System.out.println("Connected to the cluster.");
906 IoCTX io = cluster.ioCtxCreate("data");
908 String oidone = "hw";
909 String contentone = "Hello World!";
910 io.write(oidone, contentone);
912 String oidtwo = "bm";
913 String contenttwo = "Bonjour tout le monde!";
914 io.write(oidtwo, contenttwo);
916 String[] objects = io.listObjects();
917 for (String object: objects)
918 System.out.println(object);
923 cluster.ioCtxDestroy(io);
925 } catch (RadosException e) {
926 System.out.println(e.getMessage() + ": " + e.getReturnValue());
939 $io = rados_ioctx_create($r, "mypool");
940 rados_write_full($io, "oidOne", "mycontents");
941 rados_remove("oidOne");
942 rados_ioctx_destroy($io);
945 Step 4: Closing Sessions
946 ========================
948 Once your app finishes with the I/O Context and cluster handle, the app should
949 close the connection and shutdown the handle. For asynchronous I/O, the app
950 should also ensure that pending asynchronous operations have completed.
958 rados_ioctx_destroy(io);
959 rados_shutdown(cluster);
976 cluster.ioCtxDestroy(io);
983 .. code-block:: python
985 print("\nClosing the connection.")
988 print("Shutting down the handle.")
1000 .. _user ID: ../../operations/user-management#command-line-usage
1001 .. _CAPS: ../../operations/user-management#authorization-capabilities
1002 .. _Installation (Quick): ../../../start
1003 .. _Smart Daemons Enable Hyperscale: ../../../architecture#smart-daemons-enable-hyperscale
1004 .. _Calculating PG IDs: ../../../architecture#calculating-pg-ids
1005 .. _computes: ../../../architecture#calculating-pg-ids
1006 .. _OSD: ../../../architecture#mapping-pgs-to-osds