,
OBiBa group [cph]| Title: | S3 Resource Resolver |
|---|---|
| Description: | A S3 resource is provided by Amazon Web Services S3 or a S3-compatible object store (such as Minio). The resource can be a tidy file to be downloaded from the object store, or a data lake (such as Delta Lake) Parquet file to be read by Apache Spark. |
| Authors: | Yannick Marcon [aut, cre] ,
OBiBa group [cph] |
| Maintainer: | Yannick Marcon <[email protected]> |
| License: | LGPL (>= 2.1) |
| Version: | 1.2.0 |
| Built: | 2025-02-17 06:16:51 UTC |
| Source: | https://github.com/obiba/s3.resourcer |
AWS S3 file resource getter
AWS S3 file resource getter
A R6 object of class S3FileResourceGetter
Access a file that is in the Amazon Web Services S3 file store or in a HTTP S3 compatible file store such as Minio. For AWS S3 the host name is the bucket name. For Minio, the url will include http or https base protocol. Credentials may apply.
resourcer::FileResourceGetter -> S3FileResourceGetter
new()
Creates a new S3FileResourceGetter instance.
S3FileResourceGetter$new()
A S3FileResourceGetter object.
isFor()
Check that the provided resource has a URL that locates a file accessible through "s3" protocol or "s3+http" or "s3+https" protocol (i.e. using Minio implementation of the AWS S3 file store API over HTTP).
S3FileResourceGetter$isFor(resource)
resourceThe resource object to validate.
A logical.
downloadFile()
Download the file from the remote address in a temporary location. Applies authentication if credentials are provided in the resource.
S3FileResourceGetter$downloadFile(resource, ...)
resourceA valid resource object.
...Unused additional parameters.
The "resource.file" object.
clone()
The objects of this class are cloneable with this method.
S3FileResourceGetter$clone(deep = FALSE)
deepWhether to make a deep clone.
Apache Spark DBI resource connector for S3
Apache Spark DBI resource connector for S3
A R6 object of class SparkResourceConnector
Makes a Apache Spark connection object, that is also a DBI connection object, from a S3 resource description.
resourcer::DBIResourceConnector -> resourcer::SparkResourceConnector -> S3SparkResourceConnector
new()
Create a SparkResourceConnector instance.
S3SparkResourceConnector$new()
A SparkResourceConnector object.
isFor()
Check if the provided resource applies to a Apache Spark server. The resource URL scheme must be one of "s3+spark", "s3+spark+http" or "s3+spark+https".
S3SparkResourceConnector$isFor(resource)
resourceThe resource object to validate.
A logical.
createDBIConnection()
Creates a DBI connection object from a Apache Spark resource.
S3SparkResourceConnector$createDBIConnection(resource)
resourceA valid resource object.
A DBI connection object.
getTableName()
Get the SQL table name from the resource URL.
S3SparkResourceConnector$getTableName(resource)
resourceA valid resource object.
The SQL table name.
readDBTable()
Read a table as a vanilla tibble using DBI connection object.
S3SparkResourceConnector$readDBTable(conn, resource)
connA DBI connection object.
resourceA valid resource object.
readDBTibble()
Read a table as a SQL tibble using DBI connection object.
S3SparkResourceConnector$readDBTibble(conn, resource)
connA DBI connection object.
resourceA valid resource object.
closeDBIConnection()
Close the DBI connection to Apache Spark.
S3SparkResourceConnector$closeDBIConnection(conn)
connA DBI connection object.
clone()
The objects of this class are cloneable with this method.
S3SparkResourceConnector$clone(deep = FALSE)
deepWhether to make a deep clone.