Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

*: replace dumpling with tidb/dumpling #2263

Open
wants to merge 28 commits into
base: master
Choose a base branch
from
Open

*: replace dumpling with tidb/dumpling #2263

Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
10ab568
support lightning
Little-Wallace Sep 24, 2021
df036b6
fix bug
Little-Wallace Sep 24, 2021
44474f5
Update dm/config/subtask.go
Little-Wallace Sep 24, 2021
789997c
fix repeat
Little-Wallace Sep 24, 2021
490247c
address comment
Little-Wallace Sep 24, 2021
a836389
add config
Little-Wallace Sep 24, 2021
c2edae1
change test to tidb backend
Little-Wallace Sep 26, 2021
b950088
add some comment
Little-Wallace Sep 26, 2021
17e9fcd
address comment
Little-Wallace Sep 27, 2021
f9a73a5
fix fmt
Little-Wallace Sep 27, 2021
8026c1e
fix comment
Little-Wallace Sep 27, 2021
9dd6693
fix test
Little-Wallace Sep 28, 2021
b4d9bdd
Merge branch 'master' into lightning
Little-Wallace Sep 28, 2021
0bbaeb4
fix config test
Little-Wallace Sep 28, 2021
c59ae59
Merge branch 'lightning' of github.com:Little-Wallace/dm into lightning
Little-Wallace Sep 28, 2021
6b0dc8c
fix fmt
Little-Wallace Sep 28, 2021
ac64b6f
fix db name
Little-Wallace Sep 28, 2021
6fb6334
add sql mode
Little-Wallace Sep 28, 2021
9fed14c
fix meta
Little-Wallace Sep 28, 2021
4aed643
remove timestamp
Little-Wallace Sep 29, 2021
0062d81
Merge branch 'master' into lightning
Little-Wallace Oct 11, 2021
df256a3
fix tablename quotes
Little-Wallace Oct 11, 2021
99d9487
support time zone
Little-Wallace Oct 11, 2021
c921c46
remove checkpoint
Little-Wallace Oct 13, 2021
557241d
replace dumpling
Little-Wallace Oct 26, 2021
0ac6da0
Merge branch 'master' into checkpoint
Little-Wallace Oct 26, 2021
ad92b60
revert lightning
Little-Wallace Oct 26, 2021
858bff7
Merge branch 'master' into checkpoint
ti-chi-bot Oct 26, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -540,7 +540,7 @@ All notable changes to this project will be documented in this file.
- Remove the restriction of "the next shard DDL statement cannot be executed unless the current shard DDL operation is completely finished in shard merge scene for binlog replication" [#177](https://github.com/pingcap/dm/pull/177)
- Support retry task on the `invalid connection` error for binlog replication [#66](https://github.com/pingcap/dm/pull/66)
- Support generating `-schema-create.sql` files automatically for full migration [#186](https://github.com/pingcap/dm/pull/186)
- Use [TiDB SQL Parser](https://github.com/pingcap/parser) to parse and restore DDL statements in binlog query events, and remove the `use db` statement when replicating to downstream [#54](https://github.com/pingcap/dm/pull/54)
- Use [TiDB SQL Parser](https://github.com/pingcap/tidb/parser) to parse and restore DDL statements in binlog query events, and remove the `use db` statement when replicating to downstream [#54](https://github.com/pingcap/dm/pull/54)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Use [TiDB SQL Parser](https://github.com/pingcap/tidb/parser) to parse and restore DDL statements in binlog query events, and remove the `use db` statement when replicating to downstream [#54](https://github.com/pingcap/dm/pull/54)
- Use [TiDB SQL Parser](https://github.com/pingcap/tidb/tree/master/parser) to parse and restore DDL statements in binlog query events, and remove the `use db` statement when replicating to downstream [#54](https://github.com/pingcap/dm/pull/54)

- Support migrating tables with generated column for full migration and binlog replication [#42](https://github.com/pingcap/dm/pull/42) [#60](https://github.com/pingcap/dm/pull/60)
- Support appending the task name as a suffix to the dumped data directory (`dir` in the task configuration file) for full migration [#100](https://github.com/pingcap/dm/pull/100)
- Support resuming tasks automatically after the DM-worker process restarted [#88](https://github.com/pingcap/dm/pull/88) [#116](https://github.com/pingcap/dm/pull/116)
Expand Down
2 changes: 1 addition & 1 deletion dumpling/dumpling.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ import (
"strings"
"time"

"github.com/pingcap/dumpling/v4/export"
"github.com/pingcap/errors"
"github.com/pingcap/failpoint"
filter "github.com/pingcap/tidb-tools/pkg/table-filter"
"github.com/pingcap/tidb/dumpling/export"
"github.com/prometheus/client_golang/prometheus"
"go.uber.org/atomic"
"go.uber.org/zap"
Expand Down
2 changes: 1 addition & 1 deletion dumpling/dumpling_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ import (
"testing"
"time"

"github.com/pingcap/dumpling/v4/export"
"github.com/pingcap/failpoint"
"github.com/pingcap/tidb-tools/pkg/filter"
"github.com/pingcap/tidb/dumpling/export"

"github.com/pingcap/dm/dm/config"
"github.com/pingcap/dm/dm/pb"
Expand Down
2 changes: 1 addition & 1 deletion dumpling/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
package dumpling

import (
"github.com/pingcap/dumpling/v4/export"
"github.com/pingcap/failpoint"
"github.com/pingcap/tidb/dumpling/export"
"github.com/prometheus/client_golang/prometheus"
"go.uber.org/zap"

Expand Down
2 changes: 1 addition & 1 deletion dumpling/util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ import (
"fmt"
"strings"

"github.com/pingcap/dumpling/v4/export"
filter "github.com/pingcap/tidb-tools/pkg/table-filter"
"github.com/pingcap/tidb/dumpling/export"
"github.com/spf13/pflag"

dutils "github.com/pingcap/dm/pkg/dumpling"
Expand Down
5 changes: 2 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,12 @@ require (
github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d
github.com/labstack/echo/v4 v4.4.0
github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712
github.com/pingcap/dumpling v0.0.0-20211025035249-dc2cee7df4a1
github.com/pingcap/errors v0.11.5-0.20210513014640-40f9a1999b3b
github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd
github.com/pingcap/log v0.0.0-20210906054005-afc726e70354
github.com/pingcap/tidb v1.1.0-beta.0.20211025024448-36e694bfc536
github.com/pingcap/tidb v1.1.0-beta.0.20211026030648-c497d5c06348
github.com/pingcap/tidb-tools v5.2.2-0.20211019062242-37a8bef2fa17+incompatible
github.com/pingcap/tidb/parser v0.0.0-20211025024448-36e694bfc536
github.com/pingcap/tidb/parser v0.0.0-20211026030648-c497d5c06348
github.com/pkg/errors v0.9.1
github.com/prometheus/client_golang v1.5.1
github.com/rakyll/statik v0.1.6
Expand Down
13 changes: 5 additions & 8 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -608,8 +608,6 @@ github.com/pingcap/check v0.0.0-20191107115940-caf2b9e6ccf4/go.mod h1:PYMCGwN0JH
github.com/pingcap/check v0.0.0-20191216031241-8a5a85928f12/go.mod h1:PYMCGwN0JHjoqGr3HrZoD+b8Tgx8bKnArhSq8YVzUMc=
github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712 h1:R8gStypOBmpnHEx1qi//SaqxJVI4inOqljg/Aj5/390=
github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712/go.mod h1:PYMCGwN0JHjoqGr3HrZoD+b8Tgx8bKnArhSq8YVzUMc=
github.com/pingcap/dumpling v0.0.0-20211025035249-dc2cee7df4a1 h1:ivchAmico+rkqbdboAX3KOidlGJEp/x1IH697U0/cZk=
github.com/pingcap/dumpling v0.0.0-20211025035249-dc2cee7df4a1/go.mod h1:o0a2u0xBlyY0zqFxd/NxAPKNKTHLIDXehymMnVbJYOE=
github.com/pingcap/errcode v0.3.0 h1:IF6LC/4+b1KNwrMlr2rBTUrojFPMexXBcDWZSpNwxjg=
github.com/pingcap/errcode v0.3.0/go.mod h1:4b2X8xSqxIroj/IZ9MX/VGZhAwc11wB9wRIzHvz6SeM=
github.com/pingcap/errors v0.11.0/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8=
Expand Down Expand Up @@ -650,17 +648,17 @@ github.com/pingcap/sysutil v0.0.0-20210315073920-cc0985d983a3/go.mod h1:tckvA041
github.com/pingcap/sysutil v0.0.0-20210730114356-fcd8a63f68c5 h1:7rvAtZe/ZUzOKzgriNPQoBNvleJXBk4z7L3Z47+tS98=
github.com/pingcap/sysutil v0.0.0-20210730114356-fcd8a63f68c5/go.mod h1:XsOaV712rUk63aOEKYP9PhXTIE3FMNHmC2r1wX5wElY=
github.com/pingcap/tidb v1.1.0-beta.0.20211023132847-efa94595c071/go.mod h1:Ci7ABF58a4jn6YtaHi7655jP409edqC2JxWWFRqOubg=
github.com/pingcap/tidb v1.1.0-beta.0.20211025024448-36e694bfc536 h1:J7+2HNfbM19h1x3MScwWiq8dti6zWUtAhPgx+4i/9Bk=
github.com/pingcap/tidb v1.1.0-beta.0.20211025024448-36e694bfc536/go.mod h1:nAzxGAGr6oO6muhwcjwI4EgT23lRZB7bqgaYqiFLVhQ=
github.com/pingcap/tidb v1.1.0-beta.0.20211026030648-c497d5c06348 h1:DVcIszrlW/71/VA1Vpu0Sl6kY1oJiGRRHEpP+LqNrw0=
github.com/pingcap/tidb v1.1.0-beta.0.20211026030648-c497d5c06348/go.mod h1:0x9V4RDM9rVr1FduK7jZLG3E6gp4ggqB896y1l2HFUI=
github.com/pingcap/tidb-dashboard v0.0.0-20210312062513-eef5d6404638/go.mod h1:OzFN8H0EDMMqeulPhPMw2i2JaiZWOKFQ7zdRPhENNgo=
github.com/pingcap/tidb-dashboard v0.0.0-20210716172320-2226872e3296/go.mod h1:OCXbZTBTIMRcIt0jFsuCakZP+goYRv6IjawKbwLS2TQ=
github.com/pingcap/tidb-tools v5.0.3+incompatible/go.mod h1:XGdcy9+yqlDSEMTpOXnwf3hiTeqrV6MN/u1se9N8yIM=
github.com/pingcap/tidb-tools v5.2.2-0.20211019062242-37a8bef2fa17+incompatible h1:c7+izmker91NkjkZ6FgTlmD4k1A5FLOAq+li6Ki2/GY=
github.com/pingcap/tidb-tools v5.2.2-0.20211019062242-37a8bef2fa17+incompatible/go.mod h1:XGdcy9+yqlDSEMTpOXnwf3hiTeqrV6MN/u1se9N8yIM=
github.com/pingcap/tidb/parser v0.0.0-20211011031125-9b13dc409c5e/go.mod h1:e1MGCA9Sg3T8jid8PKAEq5eYVuMMCq4n8gJ+Kqp4Plg=
github.com/pingcap/tidb/parser v0.0.0-20211023132847-efa94595c071/go.mod h1:e1MGCA9Sg3T8jid8PKAEq5eYVuMMCq4n8gJ+Kqp4Plg=
github.com/pingcap/tidb/parser v0.0.0-20211025024448-36e694bfc536 h1:pwacrgkhtURkMdwZfq1Qz4NzYxs+ktLHRVojZ8nI7nY=
github.com/pingcap/tidb/parser v0.0.0-20211025024448-36e694bfc536/go.mod h1:e1MGCA9Sg3T8jid8PKAEq5eYVuMMCq4n8gJ+Kqp4Plg=
github.com/pingcap/tidb/parser v0.0.0-20211026030648-c497d5c06348 h1:Dt7Lsp9i6pMF/eKp/wMG2lO9ZsTM/gLSwRfeLUI3SDk=
github.com/pingcap/tidb/parser v0.0.0-20211026030648-c497d5c06348/go.mod h1:e1MGCA9Sg3T8jid8PKAEq5eYVuMMCq4n8gJ+Kqp4Plg=
github.com/pingcap/tipb v0.0.0-20211008080435-3fd327dfce0e h1:fZY5T65QWiPc9noQJ1UkdwejZyBZjNfxzSyTcBjKrEU=
github.com/pingcap/tipb v0.0.0-20211008080435-3fd327dfce0e/go.mod h1:A7mrd7WHBl1o63LE2bIBGEJMTNWXqhgmYiOvMLxozfs=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
Expand Down Expand Up @@ -852,9 +850,8 @@ github.com/xeipuuv/gojsonschema v1.2.0/go.mod h1:anYRn/JVcOK2ZgGU+IjEV4nwlhoK5sQ
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2 h1:eY9dn8+vbi4tKz5Qo6v2eYzo7kUS51QINcR5jNpbZS8=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
github.com/xitongsys/parquet-go v1.5.1/go.mod h1:xUxwM8ELydxh4edHGegYq1pA8NnMKDx0K/GyB0o2bww=
github.com/xitongsys/parquet-go v1.5.5-0.20201110004701-b09c49d6d457 h1:tBbuFCtyJNKT+BFAv6qjvTFpVdy97IYNaBwGUXifIUs=
github.com/xitongsys/parquet-go v1.5.5-0.20201110004701-b09c49d6d457/go.mod h1:pheqtXeHQFzxJk45lRQ0UIGIivKnLXvialZSFWs81A8=
github.com/xitongsys/parquet-go v1.6.0 h1:j6YrTVZdQx5yywJLIOklZcKVsCoSD1tqOVRXyTBFSjs=
github.com/xitongsys/parquet-go v1.6.0/go.mod h1:pheqtXeHQFzxJk45lRQ0UIGIivKnLXvialZSFWs81A8=
github.com/xitongsys/parquet-go-source v0.0.0-20190524061010-2b72cbee77d5/go.mod h1:xxCx7Wpym/3QCo6JhujJX51dzSXrwmb0oH6FQb39SEA=
github.com/xitongsys/parquet-go-source v0.0.0-20200817004010-026bad9b25d0 h1:a742S4V5A15F93smuVxA60LQWsrCnN8bKeWDBARU1/k=
github.com/xitongsys/parquet-go-source v0.0.0-20200817004010-026bad9b25d0/go.mod h1:HYhIKsdns7xz80OgkbgJYrtQY7FjHWHKH6cvN7+czGE=
Expand Down
2 changes: 1 addition & 1 deletion pkg/parser/common.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ import (

const (
// SingleRenameTableNameNum stands for number of TableNames in a single table renaming. it's 2 after
// https://github.com/pingcap/parser/pull/1021
// https://github.com/pingcap/tidb/parser/pull/1021
Copy link
Contributor

@sleepymole sleepymole Oct 26, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line should be reverted.

SingleRenameTableNameNum = 2
)

Expand Down
6 changes: 3 additions & 3 deletions syncer/util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,10 @@ package syncer
import (
"fmt"

dcontext "github.com/pingcap/dumpling/v4/context"
"github.com/pingcap/dumpling/v4/export"
dlog "github.com/pingcap/dumpling/v4/log"
"github.com/pingcap/tidb-tools/pkg/filter"
dcontext "github.com/pingcap/tidb/dumpling/context"
"github.com/pingcap/tidb/dumpling/export"
dlog "github.com/pingcap/tidb/dumpling/log"
"github.com/pingcap/tidb/parser/ast"
"go.uber.org/zap"

Expand Down