ceph/src/arrow/r/vignettes/arrow.Rmd

   1 ---
   2 title: "Using the Arrow C++ Library in R"
   3 description: "This document describes the low-level interface to the Apache Arrow C++ library in R and reviews the patterns and conventions of the R package."
   4 output: rmarkdown::html_vignette
   5 vignette: >
   6   %\VignetteIndexEntry{Using the Arrow C++ Library in R}
   7   %\VignetteEngine{knitr::rmarkdown}
   8   %\VignetteEncoding{UTF-8}
   9 ---
  10
  11 The Apache Arrow C++ library provides rich, powerful features for working with columnar data. The `arrow` R package provides both a low-level interface to the C++ library and some higher-level, R-flavored tools for working with it. This vignette provides an overview of how the pieces fit together, and it describes the conventions that the classes and methods follow in R.
  12
  13 # Features
  14
  15 ## Multi-file datasets
  16
  17 The `arrow` package lets you work efficiently with large, multi-file datasets
  18 using `dplyr` methods. See `vignette("dataset", package = "arrow")` for an overview.
  19
  20 ## Reading and writing files
  21
  22 `arrow` provides some simple functions for using the Arrow C++ library to read and write files.
  23 These functions are designed to drop into your normal R workflow
  24 without requiring any knowledge of the Arrow C++ library
  25 and use naming conventions and arguments that follow popular R packages, particularly `readr`.
  26 The readers return `data.frame`s
  27 (or if you use the `tibble` package, they will act like `tbl_df`s),
  28 and the writers take `data.frame`s.
  29
  30 Importantly, `arrow` provides basic read and write support for the [Apache
  31 Parquet](https://parquet.apache.org/) columnar data file format.
  32
  33 ```r
  34 library(arrow)
  35 df <- read_parquet("path/to/file.parquet")
  36 ```
  37
  38 Just as you can read, you can write Parquet files:
  39
  40 ```r
  41 write_parquet(df, "path/to/different_file.parquet")
  42 ```
  43
  44 The `arrow` package also includes a faster and more robust implementation of the
  45 [Feather](https://github.com/wesm/feather) file format, providing `read_feather()` and
  46 `write_feather()`. This implementation depends
  47 on the same underlying C++ library as the Python version does,
  48 resulting in more reliable and consistent behavior across the two languages, as
  49 well as [improved performance](https://wesmckinney.com/blog/feather-arrow-future/).
  50 `arrow` also by default writes the Feather V2 format,
  51 which supports a wider range of data types, as well as compression.
  52
  53 For CSV and line-delimited JSON, there are `read_csv_arrow()` and `read_json_arrow()`, respectively.
  54 While `read_csv_arrow()` currently has fewer parsing options for dealing with
  55 every CSV format variation in the wild, for the files it can read, it is
  56 often significantly faster than other R CSV readers, such as
  57 `base::read.csv`, `readr::read_csv`, and `data.table::fread`.
  58
  59 ## Working with Arrow data in Python
  60
  61 Using [`reticulate`](https://rstudio.github.io/reticulate/), `arrow` lets you
  62 share data between R and Python (`pyarrow`) efficiently, enabling you to take
  63 advantage of the vibrant ecosystem of Python packages that build on top of
  64 Apache Arrow. See `vignette("python", package = "arrow")` for details.
  65
  66 ## Access to Arrow messages, buffers, and streams
  67
  68 The `arrow` package also provides many lower-level bindings to the C++ library, which enable you
  69 to access and manipulate Arrow objects. You can use these to build connectors
  70 to other applications and services that use Arrow. One example is Spark: the
  71 [`sparklyr`](https://spark.rstudio.com/) package has support for using Arrow to
  72 move data to and from Spark, yielding [significant performance
  73 gains](http://arrow.apache.org/blog/2019/01/25/r-spark-improvements/).
  74
  75 # Object hierarchy
  76
  77 ## Metadata objects
  78
  79 Arrow defines the following classes for representing metadata:
  80
  81 | Class      | Description                                        | How to create an instance        |
  82 | ---------- | -------------------------------------------------- | -------------------------------- |
  83 | `DataType` | attribute controlling how values are represented   | functions in `help("data-type")` |
  84 | `Field`    | a character string name and a `DataType`           | `field(name, type)`              |
  85 | `Schema`   | list of `Field`s                                   | `schema(...)`                    |
  86
  87 ## Data objects
  88
  89 Arrow defines the following classes for representing zero-dimensional (scalar),
  90 one-dimensional (array/vector-like), and two-dimensional (tabular/data
  91 frame-like) data:
  92
  93 | Dim | Class          | Description                               | How to create an instance                                                                             |
  94 | --- | -------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------|
  95 | 0   | `Scalar`       | single value and its `DataType`           | `Scalar$create(value, type)`                                                                          |
  96 | 1   | `Array`        | vector of values and its `DataType`       | `Array$create(vector, type)`                                                                          |
  97 | 1   | `ChunkedArray` | vectors of values and their `DataType`    | `ChunkedArray$create(..., type)` or alias `chunked_array(..., type)`                                  |
  98 | 2   | `RecordBatch`  | list of `Array`s with a `Schema`          | `RecordBatch$create(...)` or alias `record_batch(...)`                                                |
  99 | 2   | `Table`        | list of `ChunkedArray` with a `Schema`    | `Table$create(...)`, alias `arrow_table(...)`, or `arrow::read_*(file, as_data_frame = FALSE)`        |
 100 | 2   | `Dataset`      | list of `Table`s  with the same `Schema`  | `Dataset$create(sources, schema)` or alias `open_dataset(sources, schema)`                            |
 101
 102 Each of these is defined as an `R6` class in the `arrow` R package and
 103 corresponds to a class of the same name in the Arrow C++ library. The `arrow`
 104 package provides a variety of `R6` and S3 methods for interacting with instances
 105 of these classes.
 106
 107 For convenience, the `arrow` package also defines several synthetic classes that
 108 do not exist in the C++ library, including:
 109
 110 * `ArrowDatum`: inherited by `Scalar`, `Array`, and `ChunkedArray`
 111 * `ArrowTabular`: inherited by `RecordBatch` and `Table`
 112 * `ArrowObject`: inherited by all Arrow objects
 113
 114 # Internals
 115
 116 ## Mapping of R <--> Arrow types
 117
 118 Arrow has a rich data type system that includes direct parallels with R's data types and much more.
 119
 120 In the tables, entries with a `-` are not currently implemented.
 121
 122 ### R to Arrow
 123
 124 | R type                   | Arrow type |
 125 |--------------------------|------------|
 126 | logical                  | boolean    |
 127 | integer                  | int32      |
 128 | double ("numeric")       | float64^1^ |
 129 | character                | utf8^2^    |
 130 | factor                   | dictionary |
 131 | raw                      | uint8      |
 132 | Date                     | date32     |
 133 | POSIXct                  | timestamp  |
 134 | POSIXlt                  | struct     |
 135 | data.frame               | struct     |
 136 | list^3^                  | list       |
 137 | bit64::integer64         | int64      |
 138 | difftime                 | time32     |
 139 | vctrs::vctrs_unspecified | null       |
 140
 141
 142
 143 ^1^: `float64` and `double` are the same concept and data type in Arrow C++;
 144 however, only `float64()` is used in arrow as the function `double()` already
 145 exists in base R
 146
 147 ^2^: If the character vector exceeds 2GB of strings, it will be converted to a
 148 `large_utf8` Arrow type
 149
 150 ^3^: Only lists where all elements are the same type are able to be translated
 151 to Arrow list type (which is a "list of" some type).
 152
 153
 154 ### Arrow to R
 155
 156 | Arrow type        | R type                       |
 157 |-------------------|------------------------------|
 158 | boolean           | logical                      |
 159 | int8              | integer                      |
 160 | int16             | integer                      |
 161 | int32             | integer                      |
 162 | int64             | integer^1^                   |
 163 | uint8             | integer                      |
 164 | uint16            | integer                      |
 165 | uint32            | integer^1^                   |
 166 | uint64            | integer^1^                   |
 167 | float16           | -^2^                         |
 168 | float32           | double                       |
 169 | float64           | double                       |
 170 | utf8              | character                    |
 171 | large_utf8        | character                    |
 172 | binary            | arrow_binary ^3^             |
 173 | large_binary      | arrow_large_binary ^3^       |
 174 | fixed_size_binary | arrow_fixed_size_binary ^3^  |
 175 | date32            | Date                         |
 176 | date64            | POSIXct                      |
 177 | time32            | hms::difftime                |
 178 | time64            | hms::difftime                |
 179 | timestamp         | POSIXct                      |
 180 | duration          | -^2^                         |
 181 | decimal           | double                       |
 182 | dictionary        | factor^4^                    |
 183 | list              | arrow_list ^5^               |
 184 | large_list        | arrow_large_list ^5^         |
 185 | fixed_size_list   | arrow_fixed_size_list ^5^    |
 186 | struct            | data.frame                   |
 187 | null              | vctrs::vctrs_unspecified     |
 188 | map               | -^2^                         |
 189 | union             | -^2^                         |
 190
 191 ^1^: These integer types may contain values that exceed the range of R's
 192 `integer` type (32-bit signed integer). When they do, `uint32` and `uint64` are
 193 converted to `double` ("numeric") and `int64` is converted to
 194 `bit64::integer64`. This conversion can be disabled (so that `int64` always
 195 yields a `bit64::integer64` vector) by setting `options(arrow.int64_downcast = FALSE)`.
 196
 197 ^2^: Some Arrow data types do not currently have an R equivalent and will raise an error
 198 if cast to or mapped to via a schema.
 199
 200 ^3^: `arrow*_binary` classes are implemented as lists of raw vectors.
 201
 202 ^4^: Due to the limitation of R factors, Arrow `dictionary` values are coerced
 203 to string when translated to R if they are not already strings.
 204
 205 ^5^: `arrow*_list` classes are implemented as subclasses of `vctrs_list_of`
 206 with a `ptype` attribute set to what an empty Array of the value type converts to.
 207
 208
 209 ### R object attributes
 210
 211 Arrow supports custom key-value metadata attached to Schemas. When we convert a `data.frame` to an Arrow Table or RecordBatch, the package stores any `attributes()` attached to the columns of the `data.frame` in the Arrow object's Schema. These attributes are stored under the "r" key; you can assign additional string metadata under any other key you wish, like `x$metadata$new_key <- "new value"`.
 212
 213 This metadata is preserved when writing the table to Feather or Parquet, and when reading those files into R, or when calling `as.data.frame()` on a Table/RecordBatch, the column attributes are restored to the columns of the resulting `data.frame`. This means that custom data types, including `haven::labelled`, `vctrs` annotations, and others, are preserved when doing a round-trip through Arrow.
 214
 215 Note that the `attributes()` stored in `$metadata$r` are only understood by R. If you write a `data.frame` with `haven` columns to a Feather file and read that in Pandas, the `haven` metadata won't be recognized there. (Similarly, Pandas writes its own custom metadata, which the R package does not consume.) You are free, however, to define custom metadata conventions for your application and assign any (string) values you want to other metadata keys. For more details, see the documentation for `schema()`.
 216
 217 ## Class structure and package conventions
 218
 219 C++ is an object-oriented language, so the core logic of the Arrow library is encapsulated in classes and methods. In the R package, these classes are implemented as `R6` reference classes, most of which are exported from the namespace.
 220
 221 In order to match the C++ naming conventions, the `R6` classes are in TitleCase, e.g. `RecordBatch`. This makes it easy to look up the relevant C++ implementations in the [code](https://github.com/apache/arrow/tree/master/cpp) or [documentation](https://arrow.apache.org/docs/cpp/). To simplify things in R, the C++ library namespaces are generally dropped or flattened; that is, where the C++ library has `arrow::io::FileOutputStream`, it is just `FileOutputStream` in the R package. One exception is for the file readers, where the namespace is necessary to disambiguate. So `arrow::csv::TableReader` becomes `CsvTableReader`, and `arrow::json::TableReader` becomes `JsonTableReader`.
 222
 223 Some of these classes are not meant to be instantiated directly; they may be base classes or other kinds of helpers. For those that you should be able to create, use the `$create()` method to instantiate an object. For example, `rb <- RecordBatch$create(int = 1:10, dbl = as.numeric(1:10))` will create a `RecordBatch`. Many of these factory methods that an R user might most often encounter also have a `snake_case` alias, in order to be more familiar for contemporary R users. So `record_batch(int = 1:10, dbl = as.numeric(1:10))` would do the same as `RecordBatch$create()` above.
 224
 225 The typical user of the `arrow` R package may never deal directly with the `R6` objects. We provide more R-friendly wrapper functions as a higher-level interface to the C++ library. An R user can call `read_parquet()` without knowing or caring that they're instantiating a `ParquetFileReader` object and calling the `$ReadFile()` method on it. The classes are there and available to the advanced programmer who wants fine-grained control over how the C++ library is used.