Skip to content

Commit

Permalink
Export command fix (#237)
Browse files Browse the repository at this point in the history
## What is the purpose of the change

<!-- (For example: This pull request makes task deployment go through
the blob server, rather than through RPC. That way we avoid
re-transferring them on each deployment (during recovery).) -->

## Brief change log

<!-- *(for example:)*
- *The TaskInfo is stored in the blob store on job creation time as a
persistent artifact*
- *Deployments RPC transmits only the blob storage reference*
- *TaskManagers retrieve the TaskInfo from the blob cache* -->
  • Loading branch information
shiyuhang0 authored Sep 19, 2024
1 parent 620913f commit 364c131
Show file tree
Hide file tree
Showing 11 changed files with 56 additions and 52 deletions.
4 changes: 2 additions & 2 deletions docs/generate_doc/ticloud_serverless_create.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ ticloud serverless create [flags]
### Options

```
--disable-public-endpoint Whether the public endpoint is disabled. (optional)
--disable-public-endpoint Whether the public endpoint is disabled.
-n, --display-name string Display name of the cluster to de created.
--encryption Whether Enhanced Encryption at Rest is enabled. (optional)
--encryption Whether Enhanced Encryption at Rest is enabled.
-h, --help help for create
-p, --project-id string The ID of the project, in which the cluster will be created. (default: "default project")
-r, --region string The name of cloud region. You can use "ticloud serverless region" to see all regions.
Expand Down
5 changes: 3 additions & 2 deletions docs/generate_doc/ticloud_serverless_export_create.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ ticloud serverless export create [flags]
Export all data with S3 type in non-interactive mode:
$ ticloud serverless export create -c <cluster-id> --target-type S3 --s3.uri <s3-uri> --s3.access-key-id <access-key-id> --s3.secret-access-key <secret-access-key>
Export all data and customize csv format in non-interactive mode:
Export all data and customize CSV format in non-interactive mode:
$ ticloud serverless export create -c <cluster-id> --file-type CSV --csv.separator ";" --csv.delimiter "\"" --csv.null-value 'NaN' --csv.skip-header
Export test.t1 and test.t2 in non-interactive mode:
Expand All @@ -39,7 +39,8 @@ ticloud serverless export create [flags]
--csv.null-value string Representation of null values in CSV files. (default "\\N")
--csv.separator string Separator of each value in CSV files. (default ",")
--csv.skip-header Export CSV files of the tables without header.
--file-type string The export file type. One of ["CSV" "SQL" "PARQUET"]. (default "CSV")
--display-name string The display name of the export. (default: SNAPSHOT_<snapshot_time>)
--file-type string The export file type. One of ["SQL" "CSV" "PARQUET"]. (default "CSV")
--filter strings Specify the exported table(s) with table filter patterns. See https://docs.pingcap.com/tidb/stable/table-filter to learn table filter.
--force Create without confirmation. You need to confirm when you want to export the whole cluster in non-interactive mode.
--gcs.service-account-key string The base64 encoded service account key of GCS.
Expand Down
23 changes: 16 additions & 7 deletions docs/generate_doc/ticloud_serverless_import_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,15 @@ ticloud serverless import start [flags]
Start a local import task with custom CSV format:
$ ticloud serverless import start --local.file-path <file-path> --cluster-id <cluster-id> --file-type CSV --local.target-database <target-database> --local.target-table <target-table> --csv.separator \" --csv.delimiter \' --csv.backslash-escape=false --csv.trim-last-separator=true
Start an S3 import task in non-interactive mode:
$ ticloud serverless import start --source-type S3 --s3.uri <s3-uri> --cluster-id <cluster-id> --file-type <file-type> --s3.role-arn <role-arn>
Start a GCS import task in non-interactive mode:
$ ticloud serverless import start --source-type GCS --gcs.uri <gcs-uri> --cluster-id <cluster-id> --file-type <file-type> --gcs.service-account-key <service-account-key>
Start an Azure Blob import task in non-interactive mode:
$ ticloud serverless import start --source-type AZURE_BLOB --azblob.uri <azure-blob-uri> --cluster-id <cluster-id> --file-type <file-type> --azblob.sas-token <sas-token>
```

### Options
Expand All @@ -36,19 +45,19 @@ ticloud serverless import start [flags]
--csv.separator string The field separator of CSV file. (default ",")
--csv.skip-header Specifies whether the CSV file contains a header line.
--csv.trim-last-separator Specifies whether to treat separator as the line terminator and trim all trailing separators in the CSV file.
--file-type string The import file type, one of ["CSV"].
--file-type string The import file type, one of ["CSV" "SQL" "AURORA_SNAPSHOT" "PARQUET"].
--gcs.service-account-key string The base64 encoded service account key of GCS.
--gcs.uri string The GCS folder URI for import.
--gcs.uri string The GCS URI in gcs://<bucket>/<path> format. Required when source type is GCS.
-h, --help help for start
--local.concurrency int The concurrency for uploading file. (default 5)
--local.file-path string The local file path to import.
--local.target-database string Target database to which import data.
--local.target-table string Target table to which import data.
--s3.access-key-id string The access key ID for S3.
--s3.role-arn string The role ARN for S3.
--s3.secret-access-key string The secret access key for S3.
--s3.uri string The S3 folder URI for import.
--source-type string The import source type, one of ["S3" "LOCAL" "GCS" "AZURE_BLOB"]. (default "LOCAL")
--s3.access-key-id string The access key ID of the S3. You only need to set one of the s3.role-arn and [s3.access-key-id, s3.secret-access-key].
--s3.role-arn string The role arn of the S3. You only need to set one of the s3.role-arn and [s3.access-key-id, s3.secret-access-key].
--s3.secret-access-key string The secret access key of the S3. You only need to set one of the s3.role-arn and [s3.access-key-id, s3.secret-access-key].
--s3.uri string The S3 URI in s3://<bucket>/<path> format. Required when source type is S3.
--source-type string The import source type, one of ["LOCAL" "S3" "GCS" "AZURE_BLOB"]. (default "LOCAL")
```

### Options inherited from parent commands
Expand Down
4 changes: 2 additions & 2 deletions internal/cli/serverless/create.go
Original file line number Diff line number Diff line change
Expand Up @@ -306,8 +306,8 @@ func CreateCmd(h *internal.Helper) *cobra.Command {
createCmd.Flags().StringP(flag.Region, flag.RegionShort, "", "The name of cloud region. You can use \"ticloud serverless region\" to see all regions.")
createCmd.Flags().StringP(flag.ProjectID, flag.ProjectIDShort, "", "The ID of the project, in which the cluster will be created. (default: \"default project\")")
createCmd.Flags().Int32(flag.SpendingLimitMonthly, 0, "Maximum monthly spending limit in USD cents. (optional)")
createCmd.Flags().Bool(flag.Encryption, false, "Whether Enhanced Encryption at Rest is enabled. (optional)")
createCmd.Flags().Bool(flag.PublicEndpointDisabled, false, "Whether the public endpoint is disabled. (optional)")
createCmd.Flags().Bool(flag.Encryption, false, "Whether Enhanced Encryption at Rest is enabled.")
createCmd.Flags().Bool(flag.PublicEndpointDisabled, false, "Whether the public endpoint is disabled.")
return createCmd
}

Expand Down
19 changes: 19 additions & 0 deletions internal/cli/serverless/export/create.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ func (c CreateOpts) NonInteractiveFlags() []string {
flag.AzureBlobURI,
flag.AzureBlobSASToken,
flag.ParquetCompression,
flag.DisplayName,
}
}

Expand Down Expand Up @@ -154,6 +155,7 @@ func CreateCmd(h *internal.Helper) *cobra.Command {
var gcsURI, gcsServiceAccountKey string
// azure
var azBlobURI, azBlobSasToken string
var displayName string

if opts.interactive {
if !h.IOStreams.CanPrompt {
Expand All @@ -171,6 +173,15 @@ func CreateCmd(h *internal.Helper) *cobra.Command {
}
clusterId = cluster.ID

// display name
fmt.Fprintln(h.IOStreams.Out, color.HiGreenString("Input the display name (optional):"))
inputs := []string{flag.DisplayName}
textInput, err := ui.InitialInputModel(inputs, inputDescription)
if err != nil {
return err
}
displayName = textInput.Inputs[0].Value()

// target
targetType, err = GetSelectedTargetType()
if err != nil {
Expand Down Expand Up @@ -369,6 +380,10 @@ func CreateCmd(h *internal.Helper) *cobra.Command {
} else {
// non-interactive mode, get values from flags
var err error
displayName, err = cmd.Flags().GetString(flag.DisplayName)
if err != nil {
return errors.Trace(err)
}
clusterId, err = cmd.Flags().GetString(flag.ClusterID)
if err != nil {
return errors.Trace(err)
Expand Down Expand Up @@ -544,6 +559,9 @@ func CreateCmd(h *internal.Helper) *cobra.Command {
Type: &targetType,
},
}
if displayName != "" {
params.DisplayName = &displayName
}
// add target
switch targetType {
case export.EXPORTTARGETTYPEENUM_S3:
Expand Down Expand Up @@ -642,6 +660,7 @@ func CreateCmd(h *internal.Helper) *cobra.Command {
createCmd.Flags().String(flag.AzureBlobURI, "", "The Azure Blob URI in azure://<account>.blob.core.windows.net/<container>/<path> format. Required when target type is AZURE_BLOB.")
createCmd.Flags().String(flag.AzureBlobSASToken, "", "The SAS token of Azure Blob.")
createCmd.Flags().String(flag.ParquetCompression, "ZSTD", fmt.Sprintf("The parquet compression algorithm. One of %q.", export.AllowedExportParquetCompressionTypeEnumEnumValues))
createCmd.Flags().String(flag.DisplayName, "", "The display name of the export. (default: SNAPSHOT_<snapshot_time>)")

createCmd.MarkFlagsMutuallyExclusive(flag.TableFilter, flag.SQL)
createCmd.MarkFlagsMutuallyExclusive(flag.TableWhere, flag.SQL)
Expand Down
1 change: 1 addition & 0 deletions internal/cli/serverless/export/ui.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ var inputDescription = map[string]string{
flag.CSVDelimiter: "Input the CSV delimiter: delimiter of string type variables in CSV files, skip to use default value (\"). If you want to set empty string, please use non-interactive mode",
flag.CSVNullValue: "Input the CSV null value: representation of null values in CSV files, skip to use default value (\\N). If you want to set empty string, please use non-interactive mode",
flag.CSVSkipHeader: "Input the CSV skip header: export CSV files of the tables without header. Type `true` to skip header, others will not skip header",
flag.DisplayName: "Input the name of export. You can skip and use the default name SNAPSHOT_<snapshot_time> by pressing Enter",
}

func GetSelectedParquetCompression() (export.ExportParquetCompressionTypeEnum, error) {
Expand Down
13 changes: 9 additions & 4 deletions internal/cli/serverless/update.go
Original file line number Diff line number Diff line change
Expand Up @@ -138,14 +138,14 @@ func UpdateCmd(h *internal.Helper) *cobra.Command {
return err
}

if fieldName == string(PublicEndpointDisabledHumanReadable) {
if fieldName == PublicEndpointDisabledHumanReadable {
publicEndpointDisabled, err = cloud.GetSelectedBool("Disable the public endpoint of the cluster?")
if err != nil {
return err
}
} else {
// variables for input
inputModel, err := GetUpdateClusterInput()
inputModel, err := GetUpdateClusterInput(fieldName)
if err != nil {
return err
}
Expand Down Expand Up @@ -223,14 +223,19 @@ func UpdateCmd(h *internal.Helper) *cobra.Command {
return updateCmd
}

func GetUpdateClusterInput() (tea.Model, error) {
func GetUpdateClusterInput(fieldName string) (tea.Model, error) {
m := ui.TextInputModel{
Inputs: make([]textinput.Model, 1),
}
t := textinput.New()
t.Cursor.Style = config.CursorStyle
t.CharLimit = 64
t.Placeholder = "New value"
switch fieldName {
case string(Labels):
t.Placeholder = "update labels, e.g. {\"label1\":\"value1\",\"label2\":\"value2\"}"
default:
t.Placeholder = "new value"
}
t.Focus()
t.PromptStyle = config.FocusedStyle
t.TextStyle = config.FocusedStyle
Expand Down
13 changes: 1 addition & 12 deletions pkg/tidbcloud/v1beta1/serverless/export.swagger.json
Original file line number Diff line number Diff line change
Expand Up @@ -221,14 +221,6 @@
"in": "path",
"required": true,
"type": "string"
},
{
"name": "body",
"in": "body",
"required": true,
"schema": {
"$ref": "#/definitions/ExportService.CancelExportBody"
}
}
],
"tags": [
Expand Down Expand Up @@ -760,9 +752,6 @@
],
"description": " - ROLE_ARN: The access method is role arn.\n - ACCESS_KEY: The access method is access key."
},
"ExportService.CancelExportBody": {
"type": "object"
},
"ExportService.CreateExportBody": {
"type": "object",
"properties": {
Expand Down Expand Up @@ -799,7 +788,7 @@
"items": {
"type": "string"
},
"description": "Required. The name of the files to be downloaded."
"description": "Required. The name of the files to download."
}
},
"required": [
Expand Down
11 changes: 1 addition & 10 deletions pkg/tidbcloud/v1beta1/serverless/export/api/openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -163,12 +163,6 @@ paths:
required: true
schema:
type: string
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/ExportService.CancelExportBody'
required: true
responses:
"200":
content:
Expand All @@ -185,7 +179,6 @@ paths:
summary: Cancel a specific export job.
tags:
- ExportService
x-codegen-request-body-name: body
/v1beta1/clusters/{clusterId}/exports/{exportId}:download:
post:
deprecated: true
Expand Down Expand Up @@ -633,8 +626,6 @@ components:
- ROLE_ARN
- ACCESS_KEY
type: string
ExportService.CancelExportBody:
type: object
ExportService.CreateExportBody:
properties:
exportOptions:
Expand All @@ -656,7 +647,7 @@ components:
ExportService.DownloadExportFilesBody:
properties:
fileNames:
description: Required. The name of the files to be downloaded.
description: Required. The name of the files to download.
items:
type: string
type: array
Expand Down
13 changes: 1 addition & 12 deletions pkg/tidbcloud/v1beta1/serverless/export/api_export_service.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

0 comments on commit 364c131

Please sign in to comment.