Skip to content

Commit 627a5eb

Browse files
thomas-pfeifferThomas Pfeiffer (ext.)
thomas-pfeiffer
and
Thomas Pfeiffer (ext.)
authored
Documentation s3.profile-name and client.profile-name (#3085)
Closes #3066 # Rationale for this change This PR addes a small remark in the documentation to `s3.profile-name` and `client.profile-name`, since they are currently only supported when using `FsspecFileIO`. Support for AWS profile for `PyArrowFileIO` is not yet implemented, see #3065 ## Are these changes tested? n/a, purely changes to documentation. ## Are there any user-facing changes? Yes, minor change log adjustment needed compared to #2948. - `client.profile-name`: Sets the AWS profile for both the catalog client and FileIO (unified configuration). (only supported by `FsspecFileIO` currently) - `s3.profile-name`: Sets the AWS profile specifically for S3 FileIO. (only supported by `FsspecFileIO` currently) <!-- In the case of user-facing changes, please add the changelog label. --> --------- Co-authored-by: Thomas Pfeiffer (ext.) <thomas.tp.pfeiffer@partner.bmwgroup.com>
1 parent 6c5e45c commit 627a5eb

File tree

1 file changed

+27
-27
lines changed

1 file changed

+27
-27
lines changed

‎mkdocs/docs/configuration.md‎

Lines changed: 27 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -110,25 +110,25 @@ For the FileIO there are several configuration options available:
110110
<!-- markdown-link-check-disable -->
111111

112112
| Key | Example | Description |
113-
|-----------------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
114-
| s3.endpoint | <https://10.0.19.25/> | Configure an alternative endpoint of the S3 service for the FileIO to access. This could be used to use S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. |
115-
| s3.access-key-id | admin | Configure the static access key id used to access the FileIO. |
116-
| s3.secret-access-key | password | Configure the static secret access key used to access the FileIO. |
117-
| s3.session-token | AQoDYXdzEJr... | Configure the static session token used to access the FileIO. |
118-
| s3.profile-name | default | Configure the AWS profile used to access the S3 FileIO. |
119-
| s3.role-session-name | session | An optional identifier for the assumed role session. |
120-
| s3.role-arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role. |
121-
| s3.signer | bearer | Configure the signature version of the FileIO. |
122-
| s3.signer.uri | <http://my.signer:8080/s3> | Configure the remote signing uri if it differs from the catalog uri. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. |
123-
| s3.signer.endpoint | v1/main/s3-sign | Configure the remote signing endpoint. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. (default : v1/aws/s3/sign). |
124-
| s3.region | us-west-2 | Configure the default region used to initialize an `S3FileSystem`. `PyArrowFileIO` attempts to automatically tries to resolve the region if this isn't set (only supported for AWS S3 Buckets). |
125-
| s3.resolve-region | False | Only supported for `PyArrowFileIO`, when enabled, it will always try to resolve the location of the bucket (only supported for AWS S3 Buckets). |
126-
| s3.proxy-uri | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO. |
127-
| s3.connect-timeout | 60.0 | Configure socket connection timeout, in seconds. |
128-
| s3.request-timeout | 60.0 | Configure socket read timeouts on Windows and macOS, in seconds. |
113+
|-----------------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
114+
| s3.endpoint | <https://10.0.19.25/> | Configure an alternative endpoint of the S3 service for the FileIO to access. This could be used to use S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. |
115+
| s3.access-key-id | admin | Configure the static access key id used to access the FileIO. |
116+
| s3.secret-access-key | password | Configure the static secret access key used to access the FileIO. |
117+
| s3.session-token | AQoDYXdzEJr... | Configure the static session token used to access the FileIO. |
118+
| s3.profile-name | default | Configure the AWS profile used to access the S3 FileIO (only supported by `FsspecFileIO` currently). |
119+
| s3.role-session-name | session | An optional identifier for the assumed role session. |
120+
| s3.role-arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role. |
121+
| s3.signer | bearer | Configure the signature version of the FileIO. |
122+
| s3.signer.uri | <http://my.signer:8080/s3> | Configure the remote signing uri if it differs from the catalog uri. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. |
123+
| s3.signer.endpoint | v1/main/s3-sign | Configure the remote signing endpoint. Remote signing is only implemented for `FsspecFileIO`. The final request is sent to `<s3.signer.uri>/<s3.signer.endpoint>`. (default : v1/aws/s3/sign). |
124+
| s3.region | us-west-2 | Configure the default region used to initialize an `S3FileSystem`. `PyArrowFileIO` attempts to automatically tries to resolve the region if this isn't set (only supported for AWS S3 Buckets). |
125+
| s3.resolve-region | False | Only supported for `PyArrowFileIO`, when enabled, it will always try to resolve the location of the bucket (only supported for AWS S3 Buckets). |
126+
| s3.proxy-uri | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO. |
127+
| s3.connect-timeout | 60.0 | Configure socket connection timeout, in seconds. |
128+
| s3.request-timeout | 60.0 | Configure socket read timeouts on Windows and macOS, in seconds. |
129129
| s3.force-virtual-addressing | False | Whether to use virtual addressing of buckets. If true, then virtual addressing is always enabled. If false, then virtual addressing is only enabled if endpoint_override is empty. This can be used for non-AWS backends that only support virtual hosted-style access. |
130-
| s3.retry-strategy-impl | None | Ability to set a custom S3 retry strategy. A full path to a class needs to be given that extends the [S3RetryStrategy](https://github.com/apache/arrow/blob/639201bfa412db26ce45e73851432018af6c945e/python/pyarrow/_s3fs.pyx#L110) base class. |
131-
| s3.anonymous | True | Configure whether to use anonymous connection. If False (default), uses key/secret if configured or boto's credential resolver. |
130+
| s3.retry-strategy-impl | None | Ability to set a custom S3 retry strategy. A full path to a class needs to be given that extends the [S3RetryStrategy](https://github.com/apache/arrow/blob/639201bfa412db26ce45e73851432018af6c945e/python/pyarrow/_s3fs.pyx#L110) base class. |
131+
| s3.anonymous | True | Configure whether to use anonymous connection. If False (default), uses key/secret if configured or boto's credential resolver. |
132132

133133
<!-- markdown-link-check-enable-->
134134

@@ -839,15 +839,15 @@ catalog:
839839

840840
configures the AWS credentials for both Glue Catalog and S3 FileIO.
841841

842-
| Key | Example | Description |
843-
| ------------------------ | -------------- | ------------------------------------------------------------------------------------------------------ |
844-
| client.region | us-east-1 | Set the region of both the Glue/DynamoDB Catalog and the S3 FileIO |
845-
| client.access-key-id | admin | Configure the static access key id used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
846-
| client.secret-access-key | password | Configure the static secret access key used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
847-
| client.session-token | AQoDYXdzEJr... | Configure the static session token used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
848-
| client.profile-name | default | Configure the AWS profile used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
849-
| client.role-session-name | session | An optional identifier for the assumed role session. |
850-
| client.role-arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role. |
842+
| Key | Example | Description |
843+
| ------------------------ | -------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
844+
| client.region | us-east-1 | Set the region of both the Glue/DynamoDB Catalog and the S3 FileIO |
845+
| client.access-key-id | admin | Configure the static access key id used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
846+
| client.secret-access-key | password | Configure the static secret access key used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
847+
| client.session-token | AQoDYXdzEJr... | Configure the static session token used to access both the Glue/DynamoDB Catalog and the S3 FileIO |
848+
| client.profile-name | default | Configure the AWS profile used to access both the Glue/DynamoDB Catalog and the S3 FileIO (only supported by `FsspecFileIO` currently) |
849+
| client.role-session-name | session | An optional identifier for the assumed role session. |
850+
| client.role-arn | arn:aws:... | AWS Role ARN. If provided instead of access_key and secret_key, temporary credentials will be fetched by assuming this role. |
851851

852852
<!-- prettier-ignore-start -->
853853

0 commit comments

Comments
 (0)