loader, syncer: support create/drop view #1310

lance6716 · 2020-11-27T06:43:31Z

What problem does this PR solve?

What is changed and how it works?

support dumpling's view file in load unit, support sync view as well.

sync of view didn't need a sharding sync lock, that is to say we could directly execute CREATE VIEW IF NOT EXISTS / DROP VIEW IF EXISTS to downstream when any shard send it

Check List

Tests

Unit test
Integration test

Code changes

Side effects

Related changes

Need to cherry-pick to the release branch
Need to be included in the release note

lance6716 · 2020-11-28T12:40:06Z

waiting pingcap/tidb-tools#405 (but I think it's OK to merge this PR before tidb-tools. In days before v2.0.1 release will update dependencies as well)

csuzhangxc · 2020-11-30T06:15:06Z

loader/loader.go

@@ -1272,8 +1420,18 @@ func (l *Loader) restoreData(ctx context.Context) error {
 				dispatchMap[fmt.Sprintf("%s_%s_%s", db, table, file)] = j
 			}
 		}
+
+		for view := range views {


Is it possible to create the view in a different database with the table?

I guess user may create that type of view. And dumpling will generate a "table file" for each view at view's database, I guess this logic is fine?

done in d168716

lichunzhu · 2020-11-30T06:19:20Z

loader/loader.go

+				if err != nil {
+					return terror.WithScope(err, terror.ScopeInternal)
+				}
+			}


It sames that loader will ignore all SET sqls here. Is this by design?

forgot 😂

oh, query will be added in L1222. We just didn't rename them

oh, yes. My fault. Maybe you can add a comment before L1191 🤣

lichunzhu · 2020-11-30T06:27:00Z

loader/loader.go

+		for view := range views {
+			viewFile := fmt.Sprintf("%s/%s.%s-schema-view.sql", l.cfg.Dir, db, view)
+			l.logger.Info("start to create view", zap.String("view file", viewFile))
+			err := l.restoreView(ctx, dbConn, viewFile, db, view)
+			if err != nil {
+				return err
+			}
+			l.logger.Info("finish to create view", zap.String("view file", viewFile))
+		}


What if some views needed tables in other databases are not created yet? I think we should create views when all tables in all databases have been created correctly.

done in d168716

lance6716 · 2020-12-30T10:43:56Z

updated code and PR description, PTAL @lichunzhu @GMHDBJD @csuzhangxc

go.mod

This reverts commit a91adcb.

This reverts commit 5f55bf3.

lichunzhu

LGTM

lance6716 · 2021-01-06T08:31:45Z

PTAL @GMHDBJD

only LGT1

GMHDBJD · 2021-01-07T02:15:50Z

syncer/syncer.go

@@ -1868,7 +1872,18 @@ func (s *Syncer) handleQueryEvent(ev *replication.QueryEvent, ec eventContext) e
 		return err
 	}

-	if s.cfg.ShardMode == "" {
+	// VIEW statements are not needed sharding synchronized, so pretend this is no-shard-mode


What's the behavior when view statement in resharding sync? such as ddl1 view ddl2. How about add a test for it.

56ba826

in pessimistic mode, DDL lock will block sync of VIEW. in other cases VIEW is executed directly in downstream

lance6716 · 2021-01-07T04:56:27Z

data race recorded in #1351

[2021-01-07T04:24:29.483Z] WARNING: DATA RACE
[2021-01-07T04:24:29.483Z] Write at 0x00c0002da4f0 by goroutine 194:
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/pkg/gtid.(*MySQLGTIDSet).Set()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/pkg/gtid/gtid.go:105 +0xb5
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/relay.(*Relay).handleEvents()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/relay/relay.go:474 +0x1217
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/relay.(*Relay).process()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/relay/relay.go:257 +0x742
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/relay.(*Relay).Process()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/relay/relay.go:173 +0xb1
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/dm/worker.(*realRelayHolder).run()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/relay.go:141 +0x1e8
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/dm/worker.(*realRelayHolder).Start.func1()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/relay.go:117 +0x8a
[2021-01-07T04:24:29.483Z] 
[2021-01-07T04:24:29.483Z] Previous read at 0x00c0002da4f0 by goroutine 196:
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/pkg/gtid.(*MySQLGTIDSet).Clone()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/pkg/gtid/gtid.go:159 +0x8b
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/relay.(*LocalMeta).GTID()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/relay/meta.go:352 +0x101
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/relay.(*Relay).Status()
[2021-01-07T04:24:29.483Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/relay/relay.go:763 +0x213
[2021-01-07T04:24:29.483Z]   github.com/pingcap/dm/dm/worker.(*realRelayHolder).Status()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/relay.go:162 +0x1a3
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*realTaskStatusChecker).checkRelayStatus()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/task_checker.go:305 +0x156
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*realTaskStatusChecker).check()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/task_checker.go:410 +0xfa
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*realTaskStatusChecker).run()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/task_checker.go:216 +0x2ba
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*realTaskStatusChecker).Start.func1()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/task_checker.go:177 +0x8a
[2021-01-07T04:24:29.484Z] 
[2021-01-07T04:24:29.484Z] Goroutine 194 (running) created at:
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*realRelayHolder).Start()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/relay.go:115 +0x85
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*Worker).Start()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/worker.go:126 +0x772
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*Server).startWorker.func1()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/server.go:591 +0x59
[2021-01-07T04:24:29.484Z] 
[2021-01-07T04:24:29.484Z] Goroutine 196 (running) created at:
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*realTaskStatusChecker).Start()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/task_checker.go:175 +0x85
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*Worker).Start()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/worker.go:134 +0x6c8
[2021-01-07T04:24:29.484Z]   github.com/pingcap/dm/dm/worker.(*Server).startWorker.func1()
[2021-01-07T04:24:29.484Z]       /home/jenkins/agent/workspace/dm_ghpr_test/go/src/github.com/pingcap/dm/dm/worker/server.go:591 +0x59
[2021-01-07T04:24:29.484Z] ==================

GMHDBJD · 2021-01-07T05:29:41Z

tests/shardddl3/run.sh

+    run_sql_source1 "create view ${shardddl1}.v1 as select * from ${shardddl1}.${tb1};"
+    sleep 1
+    run_sql_tidb "show create view ${shardddl}.v"
+    check_contains "View: v"


Seems view is created here in pessimistic?

yes. VIEW is pretend to sync in none ShardMode, try to immediately execute to downstream.

But the table may not be created?

create table <-- not sync create view <-- sync create table create view

ti-chi-bot · 2021-03-24T16:15:57Z

@lance6716: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lance6716 · 2021-08-17T01:37:36Z

/hold

lance6716 added 6 commits November 27, 2020 10:50

save work

3fafe7b

add test

246e4bb

add ut

a6878f8

Merge branch 'master' of https://github.com/pingcap/dm into support-view

7412f64

fix CI

ba1e0b2

change script

8089a8e

lance6716 added the status/DNM Do not merge, test is failing or blocked by another PR label Nov 27, 2020

This comment has been minimized.

Sign in to view

lance6716 and others added 5 commits November 27, 2020 15:01

fix CI

e431dda

Merge branch 'master' into support-view

0823ba2

Merge branch 'master' of https://github.com/pingcap/dm into support-view

fbcd1fc

support "as select xxx" in view

7d6fa37

remove debug

89934cc

This comment has been minimized.

Sign in to view

lance6716 added 2 commits November 28, 2020 20:13

remove TODO

b7f7438

try fix CI

8a54a3d

lance6716 mentioned this pull request Nov 28, 2020

binlog-filter: add VIEW related type pingcap/tidb-tools#405

Merged

add check and remove unused code

0b14140

lance6716 added this to the v2.0.1 milestone Nov 28, 2020

csuzhangxc reviewed Nov 30, 2020

View reviewed changes

lichunzhu reviewed Nov 30, 2020

View reviewed changes

lance6716 added the status/WIP This PR is still work in progress label Nov 30, 2020

lance6716 added 2 commits December 1, 2020 10:23

address comment

d168716

Merge branch 'master' into support-view

4d94b05

lance6716 added 2 commits December 30, 2020 11:50

Merge branch 'master' of https://github.com/pingcap/dm into support-view

1fa1a7e

VIEW didn't need sync group

a9fdc88

lichunzhu reviewed Dec 30, 2020

View reviewed changes

go.mod Show resolved Hide resolved

lance6716 added 2 commits December 31, 2020 09:43

update dumpling

5f55bf3

register / remove metrics of dumpling

a91adcb

lance6716 force-pushed the support-view branch from 1f24a99 to a91adcb Compare December 31, 2020 03:45

lance6716 added 2 commits December 31, 2020 12:25

Revert "register / remove metrics of dumpling"

ff9d825

This reverts commit a91adcb.

Revert "update dumpling"

690b7e7

This reverts commit 5f55bf3.

lichunzhu previously approved these changes Jan 4, 2021

View reviewed changes

lance6716 added the status/LGT1 One reviewer already commented LGTM label Jan 4, 2021

GMHDBJD reviewed Jan 7, 2021

View reviewed changes

test shard DDL mixed with VIEW

56ba826

improve wait time

1cd0dc4

GMHDBJD reviewed Jan 7, 2021

View reviewed changes

lance6716 added status/DNM Do not merge, test is failing or blocked by another PR and removed status/PTAL This PR is ready for review. Add this label back after committing new changes labels Jan 11, 2021

ti-chi-bot added the needs-rebase label Mar 24, 2021

lance6716 added status/Stale and removed status/DNM Do not merge, test is failing or blocked by another PR labels Aug 2, 2021

ti-chi-bot added the do-not-merge/hold label Aug 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

loader, syncer: support create/drop view #1310

loader, syncer: support create/drop view #1310

lance6716 commented Nov 27, 2020 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

lance6716 commented Nov 28, 2020 •

edited

Loading

csuzhangxc Nov 30, 2020

lance6716 Nov 30, 2020

lance6716 Dec 1, 2020

lichunzhu Nov 30, 2020

lance6716 Nov 30, 2020

lance6716 Nov 30, 2020

lichunzhu Dec 1, 2020

lichunzhu Nov 30, 2020 •

edited

Loading

lance6716 Dec 1, 2020

lance6716 commented Dec 30, 2020

lichunzhu left a comment

lance6716 commented Jan 6, 2021

GMHDBJD Jan 7, 2021 •

edited

Loading

lance6716 Jan 7, 2021

lance6716 commented Jan 7, 2021 •

edited

Loading

GMHDBJD Jan 7, 2021

lance6716 Jan 7, 2021

GMHDBJD Jan 7, 2021

ti-chi-bot commented Mar 24, 2021

lance6716 commented Aug 17, 2021

loader, syncer: support create/drop view #1310

Are you sure you want to change the base?

loader, syncer: support create/drop view #1310

Conversation

lance6716 commented Nov 27, 2020 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

This comment has been minimized.

This comment has been minimized.

lance6716 commented Nov 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lichunzhu Nov 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lance6716 commented Dec 30, 2020

lichunzhu left a comment

Choose a reason for hiding this comment

lance6716 commented Jan 6, 2021

GMHDBJD Jan 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lance6716 commented Jan 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ti-chi-bot commented Mar 24, 2021

lance6716 commented Aug 17, 2021

lance6716 commented Nov 27, 2020 •

edited

Loading

lance6716 commented Nov 28, 2020 •

edited

Loading

lichunzhu Nov 30, 2020 •

edited

Loading

GMHDBJD Jan 7, 2021 •

edited

Loading

lance6716 commented Jan 7, 2021 •

edited

Loading