--- /dev/null
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_integration_testing:
+
+Integration Testing
+===================
+
+Our strategy for integration testing between Arrow implementations is:
+
+* Test datasets are specified in a custom human-readable, JSON-based format
+ designed exclusively for Arrow's integration tests
+* Each implementation provides a testing executable capable of converting
+ between the JSON and the binary Arrow file representation
+* The test executable is also capable of validating the contents of a binary
+ file against a corresponding JSON file
+
+Running integration tests
+-------------------------
+
+The integration test data generator and runner are implemented inside
+the :ref:`Archery <archery>` utility.
+
+The integration tests are run using the ``archery integration`` command.
+
+.. code-block:: shell
+
+ archery integration --help
+
+In order to run integration tests, you'll first need to build each component
+you want to include. See the respective developer docs for C++, Java, etc.
+for instructions on building those.
+
+Some languages may require additional build options to enable integration
+testing. For C++, for example, you need to add ``-DARROW_BUILD_INTEGRATION=ON``
+to your cmake command.
+
+Depending on which components you have built, you can enable and add them to
+the archery test run. For example, if you only have the C++ project built, run:
+
+.. code-block:: shell
+
+ archery integration --with-cpp=1
+
+
+For Java, it may look like:
+
+.. code-block:: shell
+
+ VERSION=0.11.0-SNAPSHOT
+ export ARROW_JAVA_INTEGRATION_JAR=$JAVA_DIR/tools/target/arrow-tools-$VERSION-jar-with-dependencies.jar
+ archery integration --with-cpp=1 --with-java=1
+
+To run all tests, including Flight integration tests, do:
+
+.. code-block:: shell
+
+ archery integration --with-all --run-flight
+
+Note that we run these tests in continuous integration, and the CI job uses
+docker-compose. You may also run the docker-compose job locally, or at least
+refer to it if you have questions about how to build other languages or enable
+certain tests.
+
+See :ref:`docker-builds` for more information about the project's
+``docker-compose`` configuration.
+
+JSON test data format
+---------------------
+
+A JSON representation of Arrow columnar data is provided for
+cross-language integration testing purposes.
+This representation is `not canonical <https://lists.apache.org/thread.html/6947fb7666a0f9cc27d9677d2dad0fb5990f9063b7cf3d80af5e270f%40%3Cdev.arrow.apache.org%3E>`_
+but it provides a human-readable way of verifying language implementations.
+
+See `here <https://github.com/apache/arrow/tree/master/docs/source/format/integration_json_examples>`_
+for some examples of this JSON data.
+
+.. can we check in more examples, e.g. from the generated_*.json test files?
+
+The high level structure of a JSON integration test files is as follows:
+
+**Data file** ::
+
+ {
+ "schema": /*Schema*/,
+ "batches": [ /*RecordBatch*/ ],
+ "dictionaries": [ /*DictionaryBatch*/ ],
+ }
+
+All files contain ``schema`` and ``batches``, while ``dictionaries`` is only
+present if there are dictionary type fields in the schema.
+
+**Schema** ::
+
+ {
+ "fields" : [
+ /* Field */
+ ],
+ "metadata" : /* Metadata */
+ }
+
+**Field** ::
+
+ {
+ "name" : "name_of_the_field",
+ "nullable" : /* boolean */,
+ "type" : /* Type */,
+ "children" : [ /* Field */ ],
+ "dictionary": {
+ "id": /* integer */,
+ "indexType": /* Type */,
+ "isOrdered": /* boolean */
+ },
+ "metadata" : /* Metadata */
+ }
+
+The ``dictionary`` attribute is present if and only if the ``Field`` corresponds to a
+dictionary type, and its ``id`` maps onto a column in the ``DictionaryBatch``. In this
+case the ``type`` attribute describes the value type of the dictionary.
+
+For primitive types, ``children`` is an empty array.
+
+**Metadata** ::
+
+ null |
+ [ {
+ "key": /* string */,
+ "value": /* string */
+ } ]
+
+A key-value mapping of custom metadata. It may be omitted or null, in which case it is
+considered equivalent to ``[]`` (no metadata). Duplicated keys are not forbidden here.
+
+**Type**: ::
+
+ {
+ "name" : "null|struct|list|largelist|fixedsizelist|union|int|floatingpoint|utf8|largeutf8|binary|largebinary|fixedsizebinary|bool|decimal|date|time|timestamp|interval|duration|map"
+ }
+
+A ``Type`` will have other fields as defined in
+`Schema.fbs <https://github.com/apache/arrow/tree/master/format/Schema.fbs>`_
+depending on its name.
+
+Int: ::
+
+ {
+ "name" : "int",
+ "bitWidth" : /* integer */,
+ "isSigned" : /* boolean */
+ }
+
+FloatingPoint: ::
+
+ {
+ "name" : "floatingpoint",
+ "precision" : "HALF|SINGLE|DOUBLE"
+ }
+
+FixedSizeBinary: ::
+
+ {
+ "name" : "fixedsizebinary",
+ "byteWidth" : /* byte width */
+ }
+
+Decimal: ::
+
+ {
+ "name" : "decimal",
+ "precision" : /* integer */,
+ "scale" : /* integer */
+ }
+
+Timestamp: ::
+
+ {
+ "name" : "timestamp",
+ "unit" : "$TIME_UNIT",
+ "timezone": "$timezone"
+ }
+
+``$TIME_UNIT`` is one of ``"SECOND|MILLISECOND|MICROSECOND|NANOSECOND"``
+
+"timezone" is an optional string.
+
+Duration: ::
+
+ {
+ "name" : "duration",
+ "unit" : "$TIME_UNIT"
+ }
+
+Date: ::
+
+ {
+ "name" : "date",
+ "unit" : "DAY|MILLISECOND"
+ }
+
+Time: ::
+
+ {
+ "name" : "time",
+ "unit" : "$TIME_UNIT",
+ "bitWidth": /* integer: 32 or 64 */
+ }
+
+Interval: ::
+
+ {
+ "name" : "interval",
+ "unit" : "YEAR_MONTH|DAY_TIME"
+ }
+
+Union: ::
+
+ {
+ "name" : "union",
+ "mode" : "SPARSE|DENSE",
+ "typeIds" : [ /* integer */ ]
+ }
+
+The ``typeIds`` field in ``Union`` are the codes used to denote which member of
+the union is active in each array slot. Note that in general these discriminants are not identical
+to the index of the corresponding child array.
+
+List: ::
+
+ {
+ "name": "list"
+ }
+
+The type that the list is a "list of" will be included in the ``Field``'s
+"children" member, as a single ``Field`` there. For example, for a list of
+``int32``, ::
+
+ {
+ "name": "list_nullable",
+ "type": {
+ "name": "list"
+ },
+ "nullable": true,
+ "children": [
+ {
+ "name": "item",
+ "type": {
+ "name": "int",
+ "isSigned": true,
+ "bitWidth": 32
+ },
+ "nullable": true,
+ "children": []
+ }
+ ]
+ }
+
+FixedSizeList: ::
+
+ {
+ "name": "fixedsizelist",
+ "listSize": /* integer */
+ }
+
+This type likewise comes with a length-1 "children" array.
+
+Struct: ::
+
+ {
+ "name": "struct"
+ }
+
+The ``Field``'s "children" contains an array of ``Fields`` with meaningful
+names and types.
+
+Map: ::
+
+ {
+ "name": "map",
+ "keysSorted": /* boolean */
+ }
+
+The ``Field``'s "children" contains a single ``struct`` field, which itself
+contains 2 children, named "key" and "value".
+
+Null: ::
+
+ {
+ "name": "null"
+ }
+
+Extension types are, as in the IPC format, represented as their underlying
+storage type plus some dedicated field metadata to reconstruct the extension
+type. For example, assuming a "uuid" extension type backed by a
+FixedSizeBinary(16) storage, here is how a "uuid" field would be represented::
+
+ {
+ "name" : "name_of_the_field",
+ "nullable" : /* boolean */,
+ "type" : {
+ "name" : "fixedsizebinary",
+ "byteWidth" : 16
+ },
+ "children" : [],
+ "metadata" : [
+ {"key": "ARROW:extension:name", "value": "uuid"},
+ {"key": "ARROW:extension:metadata", "value": "uuid-serialized"}
+ ]
+ }
+
+**RecordBatch**::
+
+ {
+ "count": /* integer number of rows */,
+ "columns": [ /* FieldData */ ]
+ }
+
+**DictionaryBatch**::
+
+ {
+ "id": /* integer */,
+ "data": [ /* RecordBatch */ ]
+ }
+
+**FieldData**::
+
+ {
+ "name": "field_name",
+ "count" "field_length",
+ "$BUFFER_TYPE": /* BufferData */
+ ...
+ "$BUFFER_TYPE": /* BufferData */
+ "children": [ /* FieldData */ ]
+ }
+
+The "name" member of a ``Field`` in the ``Schema`` corresponds to the "name"
+of a ``FieldData`` contained in the "columns" of a ``RecordBatch``.
+For nested types (list, struct, etc.), ``Field``'s "children" each have a
+"name" that corresponds to the "name" of a ``FieldData`` inside the
+"children" of that ``FieldData``.
+For ``FieldData`` inside of a ``DictionaryBatch``, the "name" field does not
+correspond to anything.
+
+Here ``$BUFFER_TYPE`` is one of ``VALIDITY``, ``OFFSET`` (for
+variable-length types, such as strings and lists), ``TYPE_ID`` (for unions),
+or ``DATA``.
+
+``BufferData`` is encoded based on the type of buffer:
+
+* ``VALIDITY``: a JSON array of 1 (valid) and 0 (null). Data for non-nullable
+ ``Field`` still has a ``VALIDITY`` array, even though all values are 1.
+* ``OFFSET``: a JSON array of integers for 32-bit offsets or
+ string-formatted integers for 64-bit offsets
+* ``TYPE_ID``: a JSON array of integers
+* ``DATA``: a JSON array of encoded values
+
+The value encoding for ``DATA`` is different depending on the logical
+type:
+
+* For boolean type: an array of 1 (true) and 0 (false).
+* For integer-based types (including timestamps): an array of JSON numbers.
+* For 64-bit integers: an array of integers formatted as JSON strings,
+ so as to avoid loss of precision.
+* For floating point types: an array of JSON numbers. Values are limited
+ to 3 decimal places to avoid loss of precision.
+* For binary types, an array of uppercase hex-encoded strings, so as
+ to represent arbitrary binary data.
+* For UTF-8 string types, an array of JSON strings.
+
+For "list" and "largelist" types, ``BufferData`` has ``VALIDITY`` and
+``OFFSET``, and the rest of the data is inside "children". These child
+``FieldData`` contain all of the same attributes as non-child data, so in
+the example of a list of ``int32``, the child data has ``VALIDITY`` and
+``DATA``.
+
+For "fixedsizelist", there is no ``OFFSET`` member because the offsets are
+implied by the field's "listSize".
+
+Note that the "count" for these child data may not match the parent "count".
+For example, if a ``RecordBatch`` has 7 rows and contains a ``FixedSizeList``
+of ``listSize`` 4, then the data inside the "children" of that ``FieldData``
+will have count 28.
+
+For "null" type, ``BufferData`` does not contain any buffers.
projects / ceph.git / blobdiff