]>
Commit | Line | Data |
---|---|---|
7c673cae FG |
1 | =================== |
2 | Librados (Python) | |
3 | =================== | |
4 | ||
5 | The ``rados`` module is a thin Python wrapper for ``librados``. | |
6 | ||
7 | Installation | |
8 | ============ | |
9 | ||
10 | To install Python libraries for Ceph, see `Getting librados for Python`_. | |
11 | ||
12 | ||
13 | Getting Started | |
14 | =============== | |
15 | ||
16 | You can create your own Ceph client using Python. The following tutorial will | |
17 | show you how to import the Ceph Python module, connect to a Ceph cluster, and | |
11fdf7f2 | 18 | perform object operations as a ``client.admin`` user. |
7c673cae | 19 | |
11fdf7f2 | 20 | .. note:: To use the Ceph Python bindings, you must have access to a |
7c673cae FG |
21 | running Ceph cluster. To set one up quickly, see `Getting Started`_. |
22 | ||
23 | First, create a Python source file for your Ceph client. :: | |
24 | :linenos: | |
11fdf7f2 | 25 | |
7c673cae FG |
26 | sudo vim client.py |
27 | ||
28 | ||
29 | Import the Module | |
30 | ----------------- | |
31 | ||
32 | To use the ``rados`` module, import it into your source file. | |
33 | ||
34 | .. code-block:: python | |
35 | :linenos: | |
36 | ||
37 | import rados | |
38 | ||
39 | ||
40 | Configure a Cluster Handle | |
41 | -------------------------- | |
42 | ||
43 | Before connecting to the Ceph Storage Cluster, create a cluster handle. By | |
44 | default, the cluster handle assumes a cluster named ``ceph`` (i.e., the default | |
45 | for deployment tools, and our Getting Started guides too), and a | |
46 | ``client.admin`` user name. You may change these defaults to suit your needs. | |
47 | ||
48 | To connect to the Ceph Storage Cluster, your application needs to know where to | |
49 | find the Ceph Monitor. Provide this information to your application by | |
50 | specifying the path to your Ceph configuration file, which contains the location | |
51 | of the initial Ceph monitors. | |
52 | ||
53 | .. code-block:: python | |
54 | :linenos: | |
55 | ||
56 | import rados, sys | |
11fdf7f2 | 57 | |
7c673cae FG |
58 | #Create Handle Examples. |
59 | cluster = rados.Rados(conffile='ceph.conf') | |
60 | cluster = rados.Rados(conffile=sys.argv[1]) | |
61 | cluster = rados.Rados(conffile = 'ceph.conf', conf = dict (keyring = '/path/to/keyring')) | |
62 | ||
63 | Ensure that the ``conffile`` argument provides the path and file name of your | |
64 | Ceph configuration file. You may use the ``sys`` module to avoid hard-coding the | |
11fdf7f2 | 65 | Ceph configuration path and file name. |
7c673cae FG |
66 | |
67 | Your Python client also requires a client keyring. For this example, we use the | |
68 | ``client.admin`` key by default. If you would like to specify the keyring when | |
69 | creating the cluster handle, you may use the ``conf`` argument. Alternatively, | |
11fdf7f2 TL |
70 | you may specify the keyring path in your Ceph configuration file. For example, |
71 | you may add something like the following line to you Ceph configuration file:: | |
7c673cae FG |
72 | |
73 | keyring = /path/to/ceph.client.admin.keyring | |
74 | ||
75 | For additional details on modifying your configuration via Python, see `Configuration`_. | |
76 | ||
77 | ||
78 | Connect to the Cluster | |
79 | ---------------------- | |
80 | ||
11fdf7f2 | 81 | Once you have a cluster handle configured, you may connect to the cluster. |
7c673cae FG |
82 | With a connection to the cluster, you may execute methods that return |
83 | information about the cluster. | |
84 | ||
85 | .. code-block:: python | |
86 | :linenos: | |
87 | :emphasize-lines: 7 | |
88 | ||
89 | import rados, sys | |
11fdf7f2 | 90 | |
7c673cae FG |
91 | cluster = rados.Rados(conffile='ceph.conf') |
92 | print "\nlibrados version: " + str(cluster.version()) | |
11fdf7f2 TL |
93 | print "Will attempt to connect to: " + str(cluster.conf_get('mon initial members')) |
94 | ||
7c673cae FG |
95 | cluster.connect() |
96 | print "\nCluster ID: " + cluster.get_fsid() | |
97 | ||
98 | print "\n\nCluster Statistics" | |
99 | print "==================" | |
100 | cluster_stats = cluster.get_cluster_stats() | |
101 | ||
102 | for key, value in cluster_stats.iteritems(): | |
103 | print key, value | |
104 | ||
105 | ||
106 | By default, Ceph authentication is ``on``. Your application will need to know | |
107 | the location of the keyring. The ``python-ceph`` module doesn't have the default | |
108 | location, so you need to specify the keyring path. The easiest way to specify | |
109 | the keyring is to add it to the Ceph configuration file. The following Ceph | |
110 | configuration file example uses the ``client.admin`` keyring you generated with | |
111 | ``ceph-deploy``. | |
112 | ||
113 | .. code-block:: ini | |
114 | :linenos: | |
11fdf7f2 | 115 | |
7c673cae | 116 | [global] |
11fdf7f2 | 117 | # ... elided configuration |
7c673cae FG |
118 | keyring=/path/to/keyring/ceph.client.admin.keyring |
119 | ||
120 | ||
121 | Manage Pools | |
122 | ------------ | |
123 | ||
124 | When connected to the cluster, the ``Rados`` API allows you to manage pools. You | |
125 | can list pools, check for the existence of a pool, create a pool and delete a | |
11fdf7f2 | 126 | pool. |
7c673cae FG |
127 | |
128 | .. code-block:: python | |
129 | :linenos: | |
130 | :emphasize-lines: 6, 13, 18, 25 | |
131 | ||
132 | print "\n\nPool Operations" | |
133 | print "===============" | |
134 | ||
135 | print "\nAvailable Pools" | |
136 | print "----------------" | |
137 | pools = cluster.list_pools() | |
138 | ||
139 | for pool in pools: | |
140 | print pool | |
141 | ||
142 | print "\nCreate 'test' Pool" | |
143 | print "------------------" | |
144 | cluster.create_pool('test') | |
145 | ||
146 | print "\nPool named 'test' exists: " + str(cluster.pool_exists('test')) | |
147 | print "\nVerify 'test' Pool Exists" | |
148 | print "-------------------------" | |
149 | pools = cluster.list_pools() | |
150 | ||
151 | for pool in pools: | |
152 | print pool | |
153 | ||
154 | print "\nDelete 'test' Pool" | |
155 | print "------------------" | |
156 | cluster.delete_pool('test') | |
157 | print "\nPool named 'test' exists: " + str(cluster.pool_exists('test')) | |
158 | ||
159 | ||
160 | ||
161 | Input/Output Context | |
162 | -------------------- | |
163 | ||
164 | Reading from and writing to the Ceph Storage Cluster requires an input/output | |
11fdf7f2 TL |
165 | context (ioctx). You can create an ioctx with the ``open_ioctx()`` or |
166 | ``open_ioctx2()`` method of the ``Rados`` class. The ``ioctx_name`` parameter | |
167 | is the name of the pool and ``pool_id`` is the ID of the pool you wish to use. | |
7c673cae FG |
168 | |
169 | .. code-block:: python | |
170 | :linenos: | |
171 | ||
172 | ioctx = cluster.open_ioctx('data') | |
173 | ||
174 | ||
11fdf7f2 TL |
175 | or |
176 | ||
177 | .. code-block:: python | |
178 | :linenos: | |
179 | ||
180 | ioctx = cluster.open_ioctx2(pool_id) | |
181 | ||
182 | ||
7c673cae FG |
183 | Once you have an I/O context, you can read/write objects, extended attributes, |
184 | and perform a number of other operations. After you complete operations, ensure | |
11fdf7f2 | 185 | that you close the connection. For example: |
7c673cae FG |
186 | |
187 | .. code-block:: python | |
188 | :linenos: | |
189 | ||
190 | print "\nClosing the connection." | |
191 | ioctx.close() | |
192 | ||
193 | ||
194 | Writing, Reading and Removing Objects | |
195 | ------------------------------------- | |
196 | ||
197 | Once you create an I/O context, you can write objects to the cluster. If you | |
198 | write to an object that doesn't exist, Ceph creates it. If you write to an | |
199 | object that exists, Ceph overwrites it (except when you specify a range, and | |
200 | then it only overwrites the range). You may read objects (and object ranges) | |
11fdf7f2 | 201 | from the cluster. You may also remove objects from the cluster. For example: |
7c673cae FG |
202 | |
203 | .. code-block:: python | |
204 | :linenos: | |
205 | :emphasize-lines: 2, 5, 8 | |
11fdf7f2 | 206 | |
7c673cae FG |
207 | print "\nWriting object 'hw' with contents 'Hello World!' to pool 'data'." |
208 | ioctx.write_full("hw", "Hello World!") | |
209 | ||
210 | print "\n\nContents of object 'hw'\n------------------------\n" | |
211 | print ioctx.read("hw") | |
11fdf7f2 | 212 | |
7c673cae FG |
213 | print "\nRemoving object 'hw'" |
214 | ioctx.remove_object("hw") | |
215 | ||
216 | ||
217 | Writing and Reading XATTRS | |
218 | -------------------------- | |
219 | ||
220 | Once you create an object, you can write extended attributes (XATTRs) to | |
11fdf7f2 | 221 | the object and read XATTRs from the object. For example: |
7c673cae FG |
222 | |
223 | .. code-block:: python | |
224 | :linenos: | |
225 | :emphasize-lines: 2, 5 | |
226 | ||
227 | print "\n\nWriting XATTR 'lang' with value 'en_US' to object 'hw'" | |
228 | ioctx.set_xattr("hw", "lang", "en_US") | |
229 | ||
230 | print "\n\nGetting XATTR 'lang' from object 'hw'\n" | |
231 | print ioctx.get_xattr("hw", "lang") | |
232 | ||
233 | ||
234 | Listing Objects | |
235 | --------------- | |
236 | ||
11fdf7f2 | 237 | If you want to examine the list of objects in a pool, you may |
7c673cae FG |
238 | retrieve the list of objects and iterate over them with the object iterator. |
239 | For example: | |
240 | ||
241 | .. code-block:: python | |
242 | :linenos: | |
243 | :emphasize-lines: 1, 6, 7 | |
244 | ||
245 | object_iterator = ioctx.list_objects() | |
246 | ||
11fdf7f2 TL |
247 | while True : |
248 | ||
249 | try : | |
7c673cae FG |
250 | rados_object = object_iterator.next() |
251 | print "Object contents = " + rados_object.read() | |
11fdf7f2 | 252 | |
7c673cae FG |
253 | except StopIteration : |
254 | break | |
255 | ||
256 | The ``Object`` class provides a file-like interface to an object, allowing | |
257 | you to read and write content and extended attributes. Object operations using | |
258 | the I/O context provide additional functionality and asynchronous capabilities. | |
259 | ||
260 | ||
261 | Cluster Handle API | |
262 | ================== | |
263 | ||
264 | The ``Rados`` class provides an interface into the Ceph Storage Daemon. | |
265 | ||
266 | ||
267 | Configuration | |
268 | ------------- | |
269 | ||
270 | The ``Rados`` class provides methods for getting and setting configuration | |
11fdf7f2 | 271 | values, reading the Ceph configuration file, and parsing arguments. You |
7c673cae FG |
272 | do not need to be connected to the Ceph Storage Cluster to invoke the following |
273 | methods. See `Storage Cluster Configuration`_ for details on settings. | |
274 | ||
275 | .. currentmodule:: rados | |
276 | .. automethod:: Rados.conf_get(option) | |
277 | .. automethod:: Rados.conf_set(option, val) | |
278 | .. automethod:: Rados.conf_read_file(path=None) | |
279 | .. automethod:: Rados.conf_parse_argv(args) | |
11fdf7f2 | 280 | .. automethod:: Rados.version() |
7c673cae FG |
281 | |
282 | ||
283 | Connection Management | |
284 | --------------------- | |
285 | ||
286 | Once you configure your cluster handle, you may connect to the cluster, check | |
287 | the cluster ``fsid``, retrieve cluster statistics, and disconnect (shutdown) | |
288 | from the cluster. You may also assert that the cluster handle is in a particular | |
289 | state (e.g., "configuring", "connecting", etc.). | |
290 | ||
7c673cae FG |
291 | .. automethod:: Rados.connect(timeout=0) |
292 | .. automethod:: Rados.shutdown() | |
293 | .. automethod:: Rados.get_fsid() | |
294 | .. automethod:: Rados.get_cluster_stats() | |
11fdf7f2 TL |
295 | |
296 | .. documented manually because it raises warnings because of *args usage in the | |
297 | .. signature | |
298 | ||
299 | .. py:class:: Rados | |
300 | ||
301 | .. py:method:: require_state(*args) | |
302 | ||
303 | Checks if the Rados object is in a special state | |
304 | ||
305 | :param args: Any number of states to check as separate arguments | |
306 | :raises: :class:`RadosStateError` | |
7c673cae FG |
307 | |
308 | ||
309 | Pool Operations | |
310 | --------------- | |
311 | ||
312 | To use pool operation methods, you must connect to the Ceph Storage Cluster | |
313 | first. You may list the available pools, create a pool, check to see if a pool | |
314 | exists, and delete a pool. | |
315 | ||
316 | .. automethod:: Rados.list_pools() | |
11fdf7f2 | 317 | .. automethod:: Rados.create_pool(pool_name, crush_rule=None) |
7c673cae FG |
318 | .. automethod:: Rados.pool_exists() |
319 | .. automethod:: Rados.delete_pool(pool_name) | |
320 | ||
321 | ||
322 | ||
323 | Input/Output Context API | |
324 | ======================== | |
325 | ||
326 | To write data to and read data from the Ceph Object Store, you must create | |
11fdf7f2 TL |
327 | an Input/Output context (ioctx). The `Rados` class provides `open_ioctx()` |
328 | and `open_ioctx2()` methods. The remaining ``ioctx`` operations involve | |
329 | invoking methods of the `Ioctx` and other classes. | |
7c673cae FG |
330 | |
331 | .. automethod:: Rados.open_ioctx(ioctx_name) | |
332 | .. automethod:: Ioctx.require_ioctx_open() | |
333 | .. automethod:: Ioctx.get_stats() | |
7c673cae FG |
334 | .. automethod:: Ioctx.get_last_version() |
335 | .. automethod:: Ioctx.close() | |
336 | ||
337 | ||
338 | .. Pool Snapshots | |
339 | .. -------------- | |
340 | ||
341 | .. The Ceph Storage Cluster allows you to make a snapshot of a pool's state. | |
11fdf7f2 | 342 | .. Whereas, basic pool operations only require a connection to the cluster, |
7c673cae FG |
343 | .. snapshots require an I/O context. |
344 | ||
345 | .. Ioctx.create_snap(self, snap_name) | |
346 | .. Ioctx.list_snaps(self) | |
347 | .. SnapIterator.next(self) | |
348 | .. Snap.get_timestamp(self) | |
349 | .. Ioctx.lookup_snap(self, snap_name) | |
350 | .. Ioctx.remove_snap(self, snap_name) | |
351 | ||
352 | .. not published. This doesn't seem ready yet. | |
353 | ||
354 | Object Operations | |
355 | ----------------- | |
356 | ||
357 | The Ceph Storage Cluster stores data as objects. You can read and write objects | |
358 | synchronously or asynchronously. You can read and write from offsets. An object | |
359 | has a name (or key) and data. | |
360 | ||
361 | ||
362 | .. automethod:: Ioctx.aio_write(object_name, to_write, offset=0, oncomplete=None, onsafe=None) | |
363 | .. automethod:: Ioctx.aio_write_full(object_name, to_write, oncomplete=None, onsafe=None) | |
364 | .. automethod:: Ioctx.aio_append(object_name, to_append, oncomplete=None, onsafe=None) | |
365 | .. automethod:: Ioctx.write(key, data, offset=0) | |
366 | .. automethod:: Ioctx.write_full(key, data) | |
367 | .. automethod:: Ioctx.aio_flush() | |
368 | .. automethod:: Ioctx.set_locator_key(loc_key) | |
369 | .. automethod:: Ioctx.aio_read(object_name, length, offset, oncomplete) | |
370 | .. automethod:: Ioctx.read(key, length=8192, offset=0) | |
371 | .. automethod:: Ioctx.stat(key) | |
372 | .. automethod:: Ioctx.trunc(key, size) | |
373 | .. automethod:: Ioctx.remove_object(key) | |
374 | ||
375 | ||
376 | Object Extended Attributes | |
377 | -------------------------- | |
378 | ||
379 | You may set extended attributes (XATTRs) on an object. You can retrieve a list | |
380 | of objects or XATTRs and iterate over them. | |
381 | ||
382 | .. automethod:: Ioctx.set_xattr(key, xattr_name, xattr_value) | |
383 | .. automethod:: Ioctx.get_xattrs(oid) | |
11fdf7f2 | 384 | .. automethod:: XattrIterator.__next__() |
7c673cae FG |
385 | .. automethod:: Ioctx.get_xattr(key, xattr_name) |
386 | .. automethod:: Ioctx.rm_xattr(key, xattr_name) | |
387 | ||
388 | ||
389 | ||
390 | Object Interface | |
391 | ================ | |
392 | ||
393 | From an I/O context, you can retrieve a list of objects from a pool and iterate | |
394 | over them. The object interface provide makes each object look like a file, and | |
395 | you may perform synchronous operations on the objects. For asynchronous | |
396 | operations, you should use the I/O context methods. | |
397 | ||
398 | .. automethod:: Ioctx.list_objects() | |
11fdf7f2 | 399 | .. automethod:: ObjectIterator.__next__() |
7c673cae FG |
400 | .. automethod:: Object.read(length = 1024*1024) |
401 | .. automethod:: Object.write(string_to_write) | |
402 | .. automethod:: Object.get_xattrs() | |
403 | .. automethod:: Object.get_xattr(xattr_name) | |
404 | .. automethod:: Object.set_xattr(xattr_name, xattr_value) | |
405 | .. automethod:: Object.rm_xattr(xattr_name) | |
406 | .. automethod:: Object.stat() | |
407 | .. automethod:: Object.remove() | |
408 | ||
409 | ||
410 | ||
411 | ||
412 | .. _Getting Started: ../../../start | |
413 | .. _Storage Cluster Configuration: ../../configuration | |
414 | .. _Getting librados for Python: ../librados-intro#getting-librados-for-python |