import quincy 17.2.0

[ceph.git] / ceph / src / arrow / docs / source / cpp / arrays.rst

.. Licensed to the Apache Software Foundation (ASF) under one
.. or more contributor license agreements.  See the NOTICE file
.. distributed with this work for additional information
.. regarding copyright ownership.  The ASF licenses this file
.. to you under the Apache License, Version 2.0 (the
.. "License"); you may not use this file except in compliance
.. with the License.  You may obtain a copy of the License at

..   http://www.apache.org/licenses/LICENSE-2.0

.. Unless required by applicable law or agreed to in writing,
.. software distributed under the License is distributed on an
.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
.. KIND, either express or implied.  See the License for the
.. specific language governing permissions and limitations
.. under the License.

.. default-domain:: cpp
.. highlight:: cpp

======
Arrays
======

.. seealso::
   :doc:`Array API reference <api/array>`

The central type in Arrow is the class :class:`arrow::Array`.   An array
represents a known-length sequence of values all having the same type.
Internally, those values are represented by one or several buffers, the
number and meaning of which depend on the array's data type, as documented
in :ref:`the Arrow data layout specification <format_layout>`.

Those buffers consist of the value data itself and an optional bitmap buffer
that indicates which array entries are null values.  The bitmap buffer
can be entirely omitted if the array is known to have zero null values.

There are concrete subclasses of :class:`arrow::Array` for each data type,
that help you access individual values of the array.

Building an array
=================

Available strategies
--------------------

As Arrow objects are immutable, they cannot be populated directly like for
example a ``std::vector``.  Instead, several strategies can be used:

* if the data already exists in memory with the right layout, you can wrap
  said memory inside :class:`arrow::Buffer` instances and then construct
  a :class:`arrow::ArrowData` describing the array;

  .. seealso:: :ref:`cpp_memory_management`

* otherwise, the :class:`arrow::ArrayBuilder` base class and its concrete
  subclasses help building up array data incrementally, without having to
  deal with details of the Arrow format yourself.

Using ArrayBuilder and its subclasses
-------------------------------------

To build an ``Int64`` Arrow array, we can use the :class:`arrow::Int64Builder`
class. In the following example, we build an array of the range 1 to 8 where
the element that should hold the value 4 is nulled::

   arrow::Int64Builder builder;
   builder.Append(1);
   builder.Append(2);
   builder.Append(3);
   builder.AppendNull();
   builder.Append(5);
   builder.Append(6);
   builder.Append(7);
   builder.Append(8);

   auto maybe_array = builder.Finish();
   if (!maybe_array.ok()) {
      // ... do something on array building failure
   }
   std::shared_ptr<arrow::Array> array = *maybe_array;

The resulting Array (which can be casted to the concrete :class:`arrow::Int64Array`
subclass if you want to access its values) then consists of two
:class:`arrow::Buffer`\s.
The first buffer holds the null bitmap, which consists here of a single byte with
the bits ``1|1|1|1|0|1|1|1``. As we use  `least-significant bit (LSB) numbering`_.
this indicates that the fourth entry in the array is null. The second
buffer is simply an ``int64_t`` array containing all the above values.
As the fourth entry is null, the value at that position in the buffer is
undefined.

Here is how you could access the concrete array's contents::

   // Cast the Array to its actual type to access its data
   auto int64_array = std::static_pointer_cast<arrow::Int64Array>(array);

   // Get the pointer to the null bitmap.
   const uint8_t* null_bitmap = int64_array->null_bitmap_data();

   // Get the pointer to the actual data
   const int64_t* data = int64_array->raw_values();

   // Alternatively, given an array index, query its null bit and value directly
   int64_t index = 2;
   if (!int64_array->IsNull(index)) {
      int64_t value = int64_array->Value(index);
   }

.. note::
   :class:`arrow::Int64Array` (respectively :class:`arrow::Int64Builder`) is
   just a ``typedef``, provided for convenience, of ``arrow::NumericArray<Int64Type>``
   (respectively ``arrow::NumericBuilder<Int64Type>``).

.. _least-significant bit (LSB) numbering: https://en.wikipedia.org/wiki/Bit_numbering

Performance
-----------

While it is possible to build an array value-by-value as in the example above,
to attain highest performance it is recommended to use the bulk appending
methods (usually named ``AppendValues``) in the concrete :class:`arrow::ArrayBuilder`
subclasses.

If you know the number of elements in advance, it is also recommended to
presize the working area by calling the :func:`~arrow::ArrayBuilder::Resize`
or :func:`~arrow::ArrayBuilder::Reserve` methods.

Here is how one could rewrite the above example to take advantage of those
APIs::

   arrow::Int64Builder builder;
   // Make place for 8 values in total
   builder.Reserve(8);
   // Bulk append the given values (with a null in 4th place as indicated by the
   // validity vector)
   std::vector<bool> validity = {true, true, true, false, true, true, true, true};
   std::vector<int64_t> values = {1, 2, 3, 0, 5, 6, 7, 8};
   builder.AppendValues(values, validity);

   auto maybe_array = builder.Finish();

If you still must append values one by one, some concrete builder subclasses
have methods marked "Unsafe" that assume the working area has been correctly
presized, and offer higher performance in exchange::

   arrow::Int64Builder builder;
   // Make place for 8 values in total
   builder.Reserve(8);
   builder.UnsafeAppend(1);
   builder.UnsafeAppend(2);
   builder.UnsafeAppend(3);
   builder.UnsafeAppendNull();
   builder.UnsafeAppend(5);
   builder.UnsafeAppend(6);
   builder.UnsafeAppend(7);
   builder.UnsafeAppend(8);

   auto maybe_array = builder.Finish();

Size Limitations and Recommendations
====================================

Some array types are structurally limited to 32-bit sizes.  This is the case
for list arrays (which can hold up to 2^31 elements), string arrays and binary
arrays (which can hold up to 2GB of binary data), at least.  Some other array
types can hold up to 2^63 elements in the C++ implementation, but other Arrow
implementations can have a 32-bit size limitation for those array types as well.

For these reasons, it is recommended that huge data be chunked in subsets of
more reasonable size.

Chunked Arrays
==============

A :class:`arrow::ChunkedArray` is, like an array, a logical sequence of values;
but unlike a simple array, a chunked array does not require the entire sequence
to be physically contiguous in memory.  Also, the constituents of a chunked array
need not have the same size, but they must all have the same data type.

A chunked array is constructed by aggregating any number of arrays.  Here we'll
build a chunked array with the same logical values as in the example above,
but in two separate chunks::

   std::vector<std::shared_ptr<arrow::Array>> chunks;
   std::shared_ptr<arrow::Array> array;

   // Build first chunk
   arrow::Int64Builder builder;
   builder.Append(1);
   builder.Append(2);
   builder.Append(3);
   if (!builder.Finish(&array).ok()) {
      // ... do something on array building failure
   }
   chunks.push_back(std::move(array));

   // Build second chunk
   builder.Reset();
   builder.AppendNull();
   builder.Append(5);
   builder.Append(6);
   builder.Append(7);
   builder.Append(8);
   if (!builder.Finish(&array).ok()) {
      // ... do something on array building failure
   }
   chunks.push_back(std::move(array));

   auto chunked_array = std::make_shared<arrow::ChunkedArray>(std::move(chunks));

   assert(chunked_array->num_chunks() == 2);
   // Logical length in number of values
   assert(chunked_array->length() == 8);
   assert(chunked_array->null_count() == 1);

Slicing
=======

Like for physical memory buffers, it is possible to make zero-copy slices
of arrays and chunked arrays, to obtain an array or chunked array referring
to some logical subsequence of the data.  This is done by calling the
:func:`arrow::Array::Slice` and :func:`arrow::ChunkedArray::Slice` methods,
respectively.