00-arch-en.md

Gravity's Architecture

Gravity can be deployed as a single process, and can also be deployed on Kubernetes as a cluster.

Gravity Single Process architecture

Gravity is designed with micro-kernel/plugin architecture. The message process pipeline is around the core.Msg data structure.

Each plugin has its own configuration options.

Gravity consists of the following types of plugins:

• Input

  It adapts various data sources. For example, it adapts MySQL binlog files and generates core.Msg.

• Filter

  It mutates the data flow generated by Input, like filtering some data, renaming some columns and encrypting columns.

• Scheduler

  It schedules the data flow generated by Input and writes the data to the target by Output. It defines the data consistency feature the current system supports ( The default scheduler supports modifying the data in the same row in order by default).

• Output

  It writes data to the target, like Kafka and MySQL. During this process, the routing rules defined by Router are used.

• Matcher

  It matches the data generated by Input. Filter and Router use Matcher to match data.

You can develop plugins of the above types for your specific requirements.

The core.Msg definition is as follows:

type DDLMsg struct {
	Statement string
}

type DMLMsg struct {
	Operation DMLOp
	Data      map[string]interface{}
	Old       map[string]interface{}
	Pks       map[string]interface{}
	PkColumns []string
}

type Msg struct {
	Type      MsgType
	Host      string
	Database  string
	Table     string
	Timestamp time.Time

	DdlMsg *DDLMsg
	DmlMsg *DMLMsg
	...
}

Gravity Cluster architecture

Gravity Cluster supports cluster deployment on Kubernetes natively. Please see here.

It provides Rest API to create the data synchronization task to report the task status. It manages each DRC task in the Web interface (Gravity Admin).