diff --git a/media/tidb-cloud/serverless-external-storage/azure-storage-account-overview.png b/media/tidb-cloud/serverless-external-storage/azure-storage-account-overview.png new file mode 100644 index 0000000000000..a12f85adee548 Binary files /dev/null and b/media/tidb-cloud/serverless-external-storage/azure-storage-account-overview.png differ diff --git a/tidb-cloud/serverless-export.md b/tidb-cloud/serverless-export.md index 16f728f490b3c..ba2a0e504310f 100644 --- a/tidb-cloud/serverless-export.md +++ b/tidb-cloud/serverless-export.md @@ -42,10 +42,10 @@ Exporting data to a local file has the following limitations: To export data to Amazon S3, you need to provide the following information: -- URI: `s3:///` +- URI: `s3:////` - One of the following access credentials: - [An access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html): make sure the access key has the `s3:PutObject` and `s3:ListBucket` permissions. - - [A role ARN](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html): make sure the role ARN has the `s3:PutObject` and `s3:ListBucket` permissions. + - [A role ARN](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference-arns.html): make sure the role ARN (Amazon Resource Name) has the `s3:PutObject` and `s3:ListBucket` permissions. For more information, see [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). @@ -53,28 +53,20 @@ For more information, see [Configure External Storage Access for TiDB Cloud Serv To export data to Google Cloud Storage, you need to provide the following information: -- URI: `gs:///` +- URI: `gs:////` - Access credential: a **base64 encoded** [service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) for your bucket. Make sure the service account key has the `storage.objects.create` permission. For more information, see [Configure External Storage Access for TiDB Serverless](/tidb-cloud/serverless-external-storage.md#configure-gcs-access). -> **Note:** -> -> Currently, you can only export to Google Cloud Storage using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md). - ### Azure Blob Storage To export data to Azure Blob Storage, you need to provide the following information: -- URI: `azure://.blob.core.windows.net//` +- URI: `azure://.blob.core.windows.net///` or `https://.blob.core.windows.net///` - Access credential: a [shared access signature (SAS) token](https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview) for your Azure Blob Storage container. Make sure the SAS token has the `Read` and `Write` permissions on the `Container` and `Object` resources. For more information, see [Configure External Storage Access for TiDB Serverless](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access). -> **Note:** -> -> Currently, you can only export to Azure Blob Storage using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md). - ## Export options ### Data filtering @@ -92,7 +84,7 @@ You can export data in the following formats: - `separator`: specify the character used to separate fields in the exported data. The default separator is `,`. - `header`: specify whether to include a header row in the exported data. The default value is `true`. - `null-value`: specify the string that represents a NULL value in the exported data. The default value is `\N`. -- `Parquet`: export data in Parquet format. Currently, it is only supported in TiDB Cloud CLI. +- `Parquet`: export data in Parquet format. The schema and data are exported according to the following naming conventions: @@ -181,7 +173,7 @@ When exporting data to the Parquet format, the data conversion between TiDB Clou - **Task Name**: enter a name for the export task. The default value is `SNAPSHOT_{snapshot_time}`. - **Exported Data**: choose the databases and tables you want to export. - - **Data Format**: choose **SQL File** or **CSV**. + - **Data Format**: choose **SQL**, **CSV**, or **Parquet**. - **Compression**: choose **Gzip**, **Snappy**, **Zstd**, or **None**. > **Tip:** @@ -232,12 +224,12 @@ When exporting data to the Parquet format, the data conversion between TiDB Clou - **Task Name**: enter a name for the export task. The default value is `SNAPSHOT_{snapshot_time}`. - **Exported Data**: choose the databases and tables you want to export. - - **Data Format**: choose **SQL File** or **CSV**. + - **Data Format**: choose **SQL**, **CSV**, or **Parquet**. - **Compression**: choose **Gzip**, **Snappy**, **Zstd**, or **None**. - **Folder URI**: enter the URI of the Amazon S3 with the `s3:////` format. - - **Bucket Access**: choose one of the following access credentials and then fill in the credential information. If you do not have such information, see [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). - - **AWS Role ARN**: enter the role ARN that has the `s3:PutObject` and `s3:ListBucket` permissions to access the bucket. - - **AWS Access Key**: enter the access key ID and access key secret that have the `s3:PutObject` and `s3:ListBucket` permissions to access the bucket. + - **Bucket Access**: choose one of the following access credentials and then fill in the credential information: + - **AWS Role ARN**: enter the role ARN that has the permission to access the bucket. It is recommended to create the role ARN with AWS CloudFormation. For more information, see [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/serverless-external-storage.md#configure-amazon-s3-access). + - **AWS Access Key**: enter the access key ID and access key secret that have the permission to access the bucket. 4. Click **Export**. @@ -246,38 +238,95 @@ When exporting data to the Parquet format, the data conversion between TiDB Clou
```shell -ticloud serverless export create -c --s3.uri --s3.access-key-id --s3.secret-access-key --filter "database.table" +ticloud serverless export create -c --target-type S3 --s3.uri --s3.access-key-id --s3.secret-access-key --filter "database.table" + +ticloud serverless export create -c --target-type S3 --s3.uri --s3.role-arn --filter "database.table" ``` -- `s3.uri`: the Amazon S3 URI with the `s3:///` format. +- `s3.uri`: the Amazon S3 URI with the `s3:////` format. - `s3.access-key-id`: the access key ID of the user who has the permission to access the bucket. - `s3.secret-access-key`: the access key secret of the user who has the permission to access the bucket. +- `s3.role-arn`: the role ARN that has the permission to access the bucket.
### Export data to Google Cloud Storage -Currently, you can only export data to Google Cloud Storage using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md). + +
+ +1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. + + > **Tip:** + > + > If you have multiple projects, you can click in the lower-left corner and switch to another project. + +2. Click the name of your target cluster to go to its overview page, and then click **Import** in the left navigation pane. + +3. On the **Import** page, click **Export Data to** in the upper-right corner, and then choose **Google Cloud Storage** from the drop-down list. Fill in the following parameters: + + - **Task Name**: enter a name for the export task. The default value is `SNAPSHOT_{snapshot_time}`. + - **Exported Data**: choose the databases and tables you want to export. + - **Data Format**: choose **SQL**, **CSV**, or **Parquet**. + - **Compression**: choose **Gzip**, **Snappy**, **Zstd**, or **None**. + - **Folder URI**: enter the URI of the Google Cloud Storage with the `gs:////` format. + - **Bucket Access**: upload the Google Cloud credentials file that has permission to access the bucket. + +4. Click **Export**. + +
+ +
```shell -ticloud serverless export create -c --gcs.uri --gcs.service-account-key --filter "database.table" +ticloud serverless export create -c --target-type GCS --gcs.uri --gcs.service-account-key --filter "database.table" ``` -- `gcs.uri`: the URI of the Google Cloud Storage bucket in the `gs:///` format. +- `gcs.uri`: the URI of the Google Cloud Storage bucket in the `gs:////` format. - `gcs.service-account-key`: the base64 encoded service account key. +
+ + ### Export data to Azure Blob Storage -Currently, you can only export data to Azure Blob Storage using [TiDB Cloud CLI](/tidb-cloud/cli-reference.md). + +
+ +1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. + + > **Tip:** + > + > If you have multiple projects, you can click in the lower-left corner and switch to another project. + +2. Click the name of your target cluster to go to its overview page, and then click **Import** in the left navigation pane. + +3. On the **Import** page, click **Export Data to** in the upper-right corner, and then choose **Azure Blob Storage** from the drop-down list. Fill in the following parameters: + + - **Task Name**: enter a name for the export task. The default value is `SNAPSHOT_{snapshot_time}`. + - **Exported Data**: choose the databases and tables you want to export. + - **Data Format**: choose **SQL**, **CSV**, or **Parquet**. + - **Compression**: choose **Gzip**, **Snappy**, **Zstd**, or **None**. + - **Folder URI**: enter the URI of Azure Blob Storage with the `azure://.blob.core.windows.net///` format. + - **SAS Token**: enter the SAS token that has the permission to access the container. It is recommended to create a SAS token with the [Azure ARM template](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/). For more information, see [Configure External Storage Access for TiDB Cloud Serverless](/tidb-cloud/serverless-external-storage.md#configure-azure-blob-storage-access). + +4. Click **Export**. + +
+ +
```shell -ticloud serverless export create -c --azblob.uri --azblob.sas-token --filter "database.table" +ticloud serverless export create -c --target-type AZURE_BLOB --azblob.uri --azblob.sas-token --filter "database.table" ``` -- `azblob.uri`: the URI of the Azure Blob Storage in the `azure://.blob.core.windows.net//` format. +- `azblob.uri`: the URI of the Azure Blob Storage in the `(azure|https)://.blob.core.windows.net///` format. - `azblob.sas-token`: the account SAS token of the Azure Blob Storage. +
+ + ### Cancel an export task To cancel an ongoing export task, take the following steps: diff --git a/tidb-cloud/serverless-external-storage.md b/tidb-cloud/serverless-external-storage.md index aafdf53005dbe..de7e3f4e0501c 100644 --- a/tidb-cloud/serverless-external-storage.md +++ b/tidb-cloud/serverless-external-storage.md @@ -199,14 +199,49 @@ Take the following steps to configure a service account key: ![service-account-key](/media/tidb-cloud/serverless-external-storage/gcs-service-account-key.png) -3. Choose the default `JSON` key type, and then click the **CREATE** button to download the service account key. +3. Choose the default `JSON` key type, and then click **CREATE** to download the Google Cloud credentials file. The file contains the service account key that you need to use when configuring the GCS access for the TiDB Cloud Serverless cluster. ## Configure Azure Blob Storage access -To allow TiDB Serverless to access your Azure Blob container, you need to configure the Azure Blob access for the container. You can use a service SAS token to configure the container access: +To allow TiDB Serverless to access your Azure Blob container, you need to create a service SAS token for the container. -1. On the [Azure Storage account](https://portal.azure.com/#browse/Microsoft.Storage%2FStorageAccounts) page, click your storage account to which the container belongs. +You can create a SAS token either using an [Azure ARM template](https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/overview) (recommended) or manual configuration. + +To create a SAS token using an Azure ARM template, take the following steps: + +1. Open the **Import** page for your target cluster. + 1. Log in to the [TiDB Cloud console](https://tidbcloud.com/) and navigate to the [**Clusters**](https://tidbcloud.com/console/clusters) page of your project. + + 2. Click the name of your target cluster to go to its overview page, and then click **Import** in the left navigation pane. + +2. Open the **Generate New SAS Token via ARM Template Deployment** dialog. + + 1. Click **Export data to...** > **Azure Blob Storage**. If your cluster has neither imported nor exported any data before, click **Click here to export data to...** > **Azure Blob Storage** at the bottom of the page. + + 2. Scroll down to the **Azure Blob Storage Settings** area, and then click **Click here to create a new one with Azure ARM template** under the SAS Token field. + +3. Create a SAS token with the Azure ARM template. + + 1. In the **Generate New SAS Token via ARM Template Deployment** dialog, click **Click to open the Azure Portal with the pre-configured ARM template**. + + 2. After logging in to Azure, you will be redirected to the Azure **Custom deployment** page. + + 3. Fill in the **Resource group** and **Storage Account Name** in the **Custom deployment** page. You can get all the information from the storage account overview page where the container is located. + + ![azure-storage-account-overview](/media/tidb-cloud/serverless-external-storage/azure-storage-account-overview.png) + + 4. Click **Review + create** or **Next** to review the deployment. Click **Create** to start the deployment. + + 5. After it completes, you will be redirected to the deployment overview page. Navigate to the **Outputs** section to get the SAS token. + +If you have any trouble creating a SAS token with the Azure ARM template, take the following steps to create one manually: + +
+Click here to see details + +1. On the [Azure Storage account](https://portal.azure.com/#browse/Microsoft.Storage%2FStorageAccounts) page, click your storage account to which the container belongs. + 2. On your **Storage account** page, click the **Security+network**, and then click **Shared access signature**. ![sas-position](/media/tidb-cloud/serverless-external-storage/azure-sas-position.png) @@ -222,3 +257,5 @@ To allow TiDB Serverless to access your Azure Blob container, you need to config ![sas-create](/media/tidb-cloud/serverless-external-storage/azure-sas-create.png) 4. Click **Generate SAS and connection string** to generate the SAS token. + +
\ No newline at end of file