8000 GitHub - isold23/DTS: DTS基于MySQL Binlog实时数据同步系统
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

isold23/DTS

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

53 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DTS

    DTS基于mysql binlog的实时数据传输系统,支持数据获取、数据加工、数据发布。类似腾讯云和阿里云的DTS服务。 单进程单线程架构设计,在大流量的广告系统中,可以跑满千兆网卡, 由于在万兆网卡下测试的是历史数据, 瓶颈就归到了MySQL SSD磁盘的IO读写上。
    MySQL主从复制有两种方式:
    1、index+offset
        代码目前支持这种模式, offset模式在故障恢复时有一定概率出现数据丢失的情况。
    2、GTID(MySQL5.6+支持)
        代码升级后支持, GTID在MySQL出现异常,切换到其他数据库,可以进行精确的数据对齐,这个是offset方式不具备的。

MySQL主从复制原理

image image

MySQL并行复制

image

MySQL Replication Format

image

MySQL Two-Phase Commit Protocol(2PC) && Group Commit

image

MySQL命令

image

MySQL Replication Protocol

image

MySQL Binlog Event

image image

MySQL Switch Master During Failover

image

编译

编译工具cmake

git clone https://gitlab.kitware.com/cmake/cmake.git
./configure
make
make install  

依赖库

protobuf

git clone -b v3.20.0 https://github.com/protocolbuffers/protobuf.git
cd protobuf
git submodule update --init --recursive
./autogen.sh
./configure
make -j$(nproc) # $(nproc) ensures it uses all cores for compilation
make check
sudo make install
sudo ldconfig # refresh shared library cache.

[root@VM-0-11-centos protobuf]# protoc --version  
libprotoc 3.20.0-rc2  

glog

git clone https://github.com/google/glog.git 
mkdir build 
cd build 
cmake .. 
make
make install  

MySQL

MySQL install

MySQL5.7.20 下载

MySQL config

修改MySQL配置文件/etc/my.cnf

[mysqld]  
binlog_format=ROW  

MySQL账号权限

MySQL账号获取复制权限

grant replication slave on *.* to 'dbuser'@'%';
#查看binlog index&offset  
show master status;  

cmake编译

cd ./src 
mkdir build  
cd ./build 
cmake .. 
make 

运行

./dts -f ./dts.xml

Star History

Star History Chart

Contributors

About

DTS基于MySQL Binlog实时数据同步系统

Resources

License

Stars

Watchers

Forks

Packages

No packages published
0