Skip to content

Commit

Permalink
Merge pull request #852 from terascope/update-readme-s3-setup
Browse files Browse the repository at this point in the history
Add S3 connector setup instructions to README.md
  • Loading branch information
godber authored Oct 16, 2023
2 parents f36b064 + 63470dc commit 0a41158
Showing 1 changed file with 77 additions and 0 deletions.
77 changes: 77 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,83 @@ This bundle includes the following processors:
- [`s3_sender_api`](./docs/s3_sender_api.md)
- [`s3_reader_api`](./docs/s3_reader_api.md)

## Releases

You can find a list of releases, changes, and pre-built asset bundles [here](https://github.com/terascope/file-assets/releases).

## Getting Started

This asset bundle requires a running Teraslice cluster, you can find the documentation [here](https://github.com/terascope/teraslice/blob/master/README.md).

```bash
# Step 1: make sure you have teraslice-cli installed
yarn global add teraslice-cli

# Step 2:
teraslice-cli assets deploy clusterAlias terascope/file-assets
```

## Connectors
### S3 Connector

**Configuration:**

The S3 connector configuration, in your Teraslice configuration file, includes the following parameters:

| Configuration | Description | Type | Notes |
| --------- | -------- | ------ | ------ |
| endpoint | Target S3 HTTP endpoint, must be URL | String | optional, defaults to `http://127.0.0.1:80` |
| accessKeyId | S3 access key ID | String | required |
| secretAccessKey | S3 secret access key | String | required |
| region | AWS Region where bucket is located | String | optional, defaults to `us-east-1` |
| maxRetries | Maximum retry attempts | Number | optional, defaults to `3` |
| maxRedirects | Maximum redirects allowed | Number | optional, defaults to `10` |
| sslEnabled | Flag to enable/disable SSL communication | Boolean | optional, defaults to `true` |
| certLocation | Location of ssl cert | String | Must be provided if `sslEnabled` is true |
| forcePathStyle | Whether to force path style URLs for S3 objects | Boolean | optional, defaults to `false` |
| bucketEndpoint | Whether to use the bucket name as the endpoint for this request | Boolean | optional, defaults to `false` |

**Terafoundation S3 configuration example:**

```yaml
terafoundation:
connectors:
s3:
default:
endpoint: "http://localhost:9000"
accessKeyId: "yourId"
secretAccessKey: "yourPassword"
forcePathStyle: true
sslEnabled: false
```
## Development
### Tests
Run the file-assets tests
**Requirements:**
- `minio` - A running instance of minio. See this [Quickstart Guide](https://hub.docker.com/r/minio/minio).

```bash
yarn test
```

### Build

Build a compiled asset bundle to deploy to a teraslice cluster.

**Install Teraslice CLI**

```bash
yarn global add teraslice-cli
```

```bash
teraslice-cli assets build
```

## Contributing

Expand Down

0 comments on commit 0a41158

Please sign in to comment.