Skip to content

Commit

Permalink
init archer-2.0
Browse files Browse the repository at this point in the history
  • Loading branch information
jly8866 committed Apr 15, 2018
1 parent 95f5ddc commit f087f63
Show file tree
Hide file tree
Showing 89 changed files with 8,991 additions and 1,283 deletions.
66 changes: 45 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ linux : 64位linux操作系统均可
很多时候DBA并不知道SQL的业务含义,所以人工审核最好由其他研发工程师或研发经理来审核. 这是archer的设计理念.
* 回滚数据展示<br/>
* 在线查询<br/>
查询权限控制,基于inception解析查询语句,查询权限限制到表级
查询权限控制,基于inception解析查询语句,查询权限支持限制到表级<br/>
查询权限申请、审核和管理,支持审核流程配置<br/>
查询结果集限制、查询结果导出、表结构展示、多结果级展示<br/>
查询结果集限制、查询结果导出、表结构展示、多结果集展示<br/>
* 动态脱敏<br/>
基于inception解析查询语句,配合脱敏字段配置、脱敏规则(正则表达式)实现动态脱敏<br/>
* 主库集群配置
* 主库集群配置<br/>
* 用户权限配置<br/>
工程师角色(engineer)与审核角色(review_man):工程师可以发起SQL上线,在通过了inception自动审核之后,需要由人工审核点击确认才能执行SQL.<br/>
还有一个特殊的超级管理员即可以上线、审核,又可以登录admin界面进行管理.
Expand All @@ -34,7 +34,7 @@ linux : 64位linux操作系统均可

### 设计规范:
* 合理的数据库设计和规范很有必要,尤其是MySQL数据库,内核没有oracle、db2、SQL Server等数据库这么强大,需要合理设计,扬长避短。互联网业界有成熟的MySQL设计规范,特此撰写如下。请读者在公司上线使用archer系统之前由专业DBA给所有后端开发人员培训一下此规范,做到知其然且知其所以然。<br/>
下载链接: https://github.com/jly8866/archer/raw/master/docs/mysql_db_design_guide.docx
下载链接: https://github.com/jly8866/archer/master/src/docs/mysql_db_design_guide.docx

### 主要配置文件:
* archer/archer/settings.py<br/>
Expand All @@ -50,16 +50,8 @@ cd Python-3.4.1 <br/>
./configure --prefix=/path/to/python3 && make && make install
或者rpm、yum、binary等其他安装方式
3. 安装所需相关模块:<br/>
(1)django:<br/>
tar -xzvf Django-1.8.17 && cd Django-1.8.17 && python3 setup.py install<br/>
或者pip3 install Django==1.8.17<br/>
(2)Crypto:<br/>
pip3 install Crypto<br/>
pip3 install pycrypto<br/>
(3)其他模块:<br/>
pip3 install -r requirements.txt<br/>
4. 给python3安装MySQLdb模块:<br/>
pip3 install pymysql<br/>
记得确保settings.py里有如下两行:<br/>
import pymysql<br/>
pymysql.install_as_MySQLdb()<br/>
Expand All @@ -82,13 +74,13 @@ python3 manage.py migrate<br/>
cd archer && python3 manage.py createsuperuser<br/>
8. 启动,有两种方式:<br/>
(1)用django内置runserver启动服务,需要修改debug.sh里的ip和port<br/>
cd archer && bash debug.sh<br/>
cd archer && bash debug.sh<br/>
(2)用gunicorn启动服务,可以使用pip3 install gunicorn安装并用startup.sh启动,但需要配合nginx处理静态资源. (nginx安装这里不做示范)<br/>
* gunicorn的安装配置示例:
* pip3 install gunicorn
* cat startup.sh
* ![image](https://github.com/jly8866/archer/raw/master/screenshots/startup.png)<br/>
* nginx配置示例:
* 静态资源地址请指定setting.py里面的STATIC_ROOT配置项地址,一般为/archer/static
* cat nginx.conf
* ![image](https://github.com/jly8866/archer/raw/master/screenshots/nginx.png)<br/>
9. 创建archer系统登录用户:<br/>
Expand All @@ -102,8 +94,7 @@ cd archer && python3 manage.py createsuperuser<br/>
这一步是为了进行sql在线查询,所用到的用户名密码、端口等,建议账号仅开放SELECT权限。<br/>
12. 配置查询权限审核人:<br/>
使用浏览器访问http://X.X.X.X:port/admin/sql/workflowauditsetting/ ,点击右侧Add 工作流配置<br/>
这一步是为了添加查询权限审核人,单人审核格式为:user1,多人审核格式为:user1,user2,请正确配置。<br/>

这一步是为了添加查询权限审核人,单人审核格式为:user1,多级审核格式为:user1,user2,请正确配置。<br/>
13. 正式访问:<br/>
以上步骤完毕,就可以使用步骤9创建的用户登录archer系统啦, 首页地址 http://X.X.X.X:port/<br/>
<br/>
Expand All @@ -115,6 +106,23 @@ cd archer && python3 manage.py createsuperuser<br/>
3. centos需要执行yum install openldap-devel<br/>
4. settings中以AUTH_LDAP开头的配置,需要根据自己的ldap对应修改<br/>

### 集成SQLAdvisor
1. 安装SQLAdvisor,[项目地址](https://github.com/Meituan-Dianping/SQLAdvisor)
2. 修改配置文件SQLADVISOR为程序路径,路径需要完整,如'/opt/SQLAdvisor/sqladvisor/sqladvisor'

### 慢日志管理
1. 安装percona-toolkit(版本>3.0),以centos为例
yum -y install http://www.percona.com/downloads/percona-release/redhat/0.1-3/percona-release-0.1-3.noarch.rpm
yum -y install percona-toolkit.x86_64
2. 使用src/script/mysql_slow_query_review.sql创建慢日志收集表
3. 将src/script/analysis_slow_query.sh部署到各个监控机器,注意按照说明修改配置信息
4. 如果有阿里云rds实例,可以在后台数据管理模块添加关联关系,可直接拉取rds慢日志

### 集成阿里云rds管理
1. 修改配置文件ENABLE_ALIYUN=True
2. 访问http://X.X.X.X:port/admin/sql/aliyunaccesskey/, 添加aliyun账号的accesskey信息,重新启动服务
3. 访问http://X.X.X.X:port/admin/sql/aliyunrdsconfig/,添加实例id信息即可实现rds进程管理、慢日志管理

### admin后台加固,防暴力破解
* 1.patch目录下,名称为:django_1.8.17_admin_secure_archer.patch
* 2.使用命令:patch python/site-packages/django/contrib/auth/views.py django_1.8.17_admin_secure_archer.patch
Expand All @@ -136,12 +144,28 @@ cd archer && python3 manage.py createsuperuser<br/>
![image](https://github.com/jly8866/archer/raw/master/screenshots/waitingforme.png)<br/>
5. 用户登录页:<br/>
![image](https://github.com/jly8866/archer/raw/master/screenshots/login.png)<br/>
6. 用户、集群、工单管理:<br/>
![image](https://github.com/jly8866/archer/raw/master/screenshots/adminsqlusers.png)<br/>
7. 工单统计图表:<br/>
![image](https://github.com/jly8866/archer/raw/master/screenshots/charts.png)<br/><br/>
8.pt-osc进度条,以及中止pt-osc进程按钮:<br/>
6. 工单统计图表:<br/>
![image](https://github.com/jly8866/archer/raw/master/screenshots/charts.png)<br/>
7. pt-osc进度条,以及中止pt-osc进程按钮:<br/>
![image](https://raw.githubusercontent.com/johnliu2008/archer/master/screenshots/osc_progress.png)<br/>
8. SQL在线查询、自动补全:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/query.png)<br/>
9. 动态脱敏:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/datamasking.png)<br/>
10. SQL在线查询日志:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/querylog.png)<br/>
11. SQL在线查询权限申请:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/applyforprivileges.png)<br/>
12. SQL慢查日志统计:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/slowquery.png)<br/>
13. SQL慢查日志明细:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/slowquerylog.png)<br/>
14. 阿里云RDS进程管理、表空间查询:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/process.png)<br/>
15. SQLAdvisor:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/sqladvisor.png)<br/>
15. 后台数据管理:<br/>
![image](https://github.com/hhyo/archer/blob/master/src/screenshots/admin.png)<br/>

### 联系方式:
QQ群:524233225
Expand Down
26 changes: 23 additions & 3 deletions archer/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'django.middleware.security.SecurityMiddleware',
'sql.check_login_middleware.CheckLoginMiddleware',
'sql.exception_logging_middleware.ExceptionLoggingMiddleware',
)

ROOT_URLCONF = 'archer.urls'
Expand Down Expand Up @@ -208,6 +209,16 @@
'handlers': ['default'],
'level': 'DEBUG',
},
'django.db': { # 打印SQL语句到console,方便开发
'handlers': ['console'],
'level': 'DEBUG',
'propagate': True,
},
'django.request': { # 打印SQL语句到console,方便开发
'handlers': ['console'],
'level': 'DEBUG',
'propagate': True,
},
'django_auth_ldap': { # django_auth_ldap模块相关日志打印到console
'handlers': ['ldap'],
'level': 'DEBUG',
Expand All @@ -229,8 +240,17 @@
#on是开,会首先用正则表达式匹配sqlContent,如果匹配到高危DDL操作,则判断为“自动审核不通过”;off是关,直接将所有的SQL语句提交给inception,对于上述高危DDL操作,只备份元数据
CRITICAL_DDL_ON_OFF='off'

# 在线查询当inception语法树打印失败时的控制,on是允许继续执行,off是禁止执行返回错误,允许执行会导致解析失败的查询表权限验证和脱敏功能失效
CRITICAL_QUERY_ON_OFF = 'on'
# 在线查询当inception语法树打印失败时的控制,on是开启校验,失败不允许继续执行并返回错误,off是关闭校验,继续执行,允许执行会导致解析失败的查询表权限验证和脱敏功能失效
CHECK_QUERY_ON_OFF = True

# 是否开启动态脱敏查询,由于采取遍历处理结果集的方式,会影响部分查询效率
CRITICAL_DATA_MASKING_ON_OFF = 'on'
DATA_MASKING_ON_OFF = True

# 管理员在线查询的结果集限制
ADMIN_QUERY_LIMIT = 5000

# sqladvisor的路径配置
SQLADVISOR = ''

# 是否开启AliYun相关功能
ENABLE_ALIYUN = False
2 changes: 1 addition & 1 deletion debug.sh
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
#!/bin/bash

python3 manage.py runserver 0.0.0.0:9123
python3 manage.py runserver 0.0.0.0:9123 --insecure
19 changes: 17 additions & 2 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,19 @@
certifi==2017.7.27.1
chardet==3.0.4
crypto==1.4.1
Django==1.8.17
django-auth-ldap==1.3.0
django-admin-bootstrapped==2.5.7
idna==2.6
Naked==0.1.31
pycrypto==2.6.1
PyMySQL==0.7.11
PyYAML==3.12
requests==2.18.4
shellescape==3.4.1
simplejson==3.13.2
urllib3==1.22
django-admin-bootstrapped==2.5.7
gunicorn==19.7.1
django-auth-ldap==1.3.0
aliyun-python-sdk-core==2.3.5
aliyun-python-sdk-core-v3==2.5.3
aliyun-python-sdk-rds==2.1.1
31 changes: 21 additions & 10 deletions sql/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@
from django.contrib.auth.forms import UserCreationForm, UserChangeForm

# Register your models here.
from .models import users, master_config, slave_config, workflow, WorkflowAudit, WorkflowAuditSetting, DataMaskingColumns, DataMaskingRules
from .models import users, master_config, slave_config, workflow, WorkflowAudit, WorkflowAuditSetting, DataMaskingColumns, DataMaskingRules, AliyunAccessKey, AliyunRdsConfig


class master_configAdmin(admin.ModelAdmin):
list_display = ('id', 'cluster_name', 'master_host', 'master_port', 'master_user', 'master_password', 'create_time', 'update_time')
list_display = ('id', 'cluster_name', 'master_host', 'master_port', 'master_user', 'create_time')
search_fields = ['id', 'cluster_name', 'master_host', 'master_port', 'master_user', 'master_password', 'create_time', 'update_time']

class workflowAdmin(admin.ModelAdmin):
list_display = ('id','workflow_name', 'engineer', 'review_man', 'create_time', 'finish_time', 'status', 'is_backup', 'review_content', 'cluster_name', 'reviewok_time', 'sql_content', 'execute_result')
list_display = ('id', 'workflow_name', 'cluster_name', 'engineer', 'create_time', 'status', 'is_backup')
search_fields = ['id','workflow_name', 'engineer', 'review_man', 'sql_content']

#创建用户表单重新定义,继承自UserCreationForm
Expand All @@ -24,12 +24,12 @@ def __init__(self, *args, **kwargs):
self.fields['role'].required = True

#编辑用户表单重新定义,继承自UserChangeForm
class usersChangeForm(UserChangeForm):
class usersChangeForm(UserChangeForm):
def __init__(self, *args, **kwargs):
super(usersChangeForm, self).__init__(*args, **kwargs)
self.fields['email'].required = True
self.fields['display'].required = True
self.fields['role'].required = True
self.fields['display'].required = True
self.fields['role'].required = True

class usersAdmin(UserAdmin):
def __init__(self, *args, **kwargs):
Expand All @@ -40,7 +40,7 @@ def __init__(self, *args, **kwargs):
self.add_form = usersCreationForm
#以上的属性都可以在django源码的UserAdmin类中找到,我们做以覆盖

def changelist_view(self, request, extra_context=None):
def changelist_view(self, request, extra_context=None):
#这个方法在源码的admin/options.py文件的ModelAdmin这个类中定义,我们要重新定义它,以达到不同权限的用户,返回的表单内容不同
if request.user.is_superuser:
#此字段定义UserChangeForm表单中的具体显示内容,并可以分类显示
Expand All @@ -66,7 +66,7 @@ def changelist_view(self, request, extra_context=None):
@admin.register(slave_config)
class WorkflowAuditAdmin(admin.ModelAdmin):
list_display = (
'cluster_id', 'cluster_name', 'slave_host', 'slave_port', 'slave_user', 'create_time', 'update_time')
'cluster_name', 'slave_host', 'slave_port', 'slave_user', 'create_time', 'update_time')
search_fields = ['id', 'cluster_name', 'slave_host', 'slave_port', 'slave_user', 'slave_password', ]

# 工作流列表
Expand All @@ -85,10 +85,21 @@ class WorkflowAuditSettingAdmin(admin.ModelAdmin):
@admin.register(DataMaskingColumns)
class DataMaskingColumnsAdmin(admin.ModelAdmin):
list_display = (
'column_id', 'rule_type', 'active', 'cluster_id', 'cluster_name', 'table_schema', 'table_name', 'column_name', 'create_time',)
'column_id', 'rule_type', 'active', 'cluster_name', 'table_schema', 'table_name', 'column_name', 'create_time',)

# 脱敏规则页面定义
@admin.register(DataMaskingRules)
class DataMaskingRulesAdmin(admin.ModelAdmin):
list_display = (
'rule_type', 'rule_regex', 'rule_desc', 'sys_time',)
'rule_type', 'rule_regex', 'hide_group', 'rule_desc', 'sys_time',)

# 阿里云的认证信息
@admin.register(AliyunAccessKey)
class AliyunAccessKeyAdmin(admin.ModelAdmin):
list_display = ('ak', 'secret', 'is_enable', 'remark',)

# 阿里云集群配置信息
@admin.register(AliyunRdsConfig)
class AliyunRdsConfigAdmin(admin.ModelAdmin):
list_display = ('cluster_name', 'rds_dbinstanceid',)

83 changes: 83 additions & 0 deletions sql/aliyun_api.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# -*- coding: UTF-8 -*-
import datetime
from aliyunsdkcore import client
from aliyunsdkrds.request.v20140815 import DescribeSlowLogsRequest, DescribeSlowLogRecordsRequest, \
RequestServiceOfCloudDBARequest
import simplejson as json
from .models import AliyunAccessKey
from .aes_decryptor import Prpcrypt


class Aliyun(object):
def __init__(self):
prpCryptor = Prpcrypt()
auth = AliyunAccessKey.objects.filter(is_enable=1)
try:
ak = prpCryptor.decrypt(auth[0].ak)
secret = prpCryptor.decrypt(auth[0].secret)
except Exception:
ak = ''
secret = ''
self.clt = client.AcsClient(
ak=ak,
secret=secret)

def request_api(self, request, *values):
if values:
for value in values:
for k, v in value.items():
request.add_query_param(k, v)
request.set_accept_format('json')
result = self.clt.do_action_with_exception(request)
return json.dumps(json.loads(result.decode('utf-8')), indent=4, sort_keys=False, ensure_ascii=False)

# 阿里云2017-12-10T16:00:00Z时间加上8小时时区显示
def aliyun_time_format(self, str_time):
if 'T' in str_time:
Ymd = str_time.split('T')[0]
HMS = str_time.split('T')[1].split('Z')[0]
str_time = '%s %s' % (Ymd, HMS)
time = datetime.datetime.strptime(str_time, "%Y-%m-%d %H:%M:%S")
format_time = time + datetime.timedelta(hours=8)
elif 'Z' in str_time:
Ymd = str_time.split('Z')[0]
format_time = '%s' % Ymd
else:
format_time = str_time
return format_time

def DescribeSlowLogs(self, DBInstanceId, StartTime, EndTime, **kwargs):
'''获取集群慢日志列表
DBName,SortKey、PageSize、PageNumber'''
request = DescribeSlowLogsRequest.DescribeSlowLogsRequest()
values = {"action_name": "DescribeSlowLogs", "DBInstanceId": DBInstanceId,
"StartTime": StartTime, "EndTime": EndTime, "SortKey": "TotalExecutionCounts"}
values = dict(values, **kwargs)
result = self.request_api(request, values)
return result

def DescribeSlowLogRecords(self, DBInstanceId, StartTime, EndTime, **kwargs):
'''查看慢日志明细
SQLId,DBName、PageSize、PageNumber'''
request = DescribeSlowLogRecordsRequest.DescribeSlowLogRecordsRequest()
values = {"action_name": "DescribeSlowLogRecords", "DBInstanceId": DBInstanceId,
"StartTime": StartTime, "EndTime": EndTime}
values = dict(values, **kwargs)
result = self.request_api(request, values)
return result

def RequestServiceOfCloudDBA(self, DBInstanceId, ServiceRequestType, ServiceRequestParam, **kwargs):
'''
获取统计信息:'GetTimedMonData',{"Language":"zh","KeyGroup":"mem_cpu_usage","KeyName":"","StartTime":"2018-01-15T04:03:26Z","EndTime":"2018-01-15T05:03:26Z"}
mem_cpu_usage、iops_usage、detailed_disk_space
获取process信息:'ShowProcessList',{"Language":"zh","Command":"Query"} -- Not Sleep , All
终止进程:'ConfirmKillSessionRequest',{"Language":"zh","SQLRequestID":75865,"SQLStatement":"kill 34022786;"}
获取表空间信息:'GetSpaceStatForTables',{"Language": "zh", "OrderType": "Data"}
获取资源利用信息:'GetResourceUsage',{"Language":"zh"}
'''
request = RequestServiceOfCloudDBARequest.RequestServiceOfCloudDBARequest()
values = {"action_name": "RequestServiceOfCloudDBA", "DBInstanceId": DBInstanceId,
"ServiceRequestType": ServiceRequestType, "ServiceRequestParam": ServiceRequestParam}
values = dict(values, **kwargs)
result = self.request_api(request, values)
return result
Loading

0 comments on commit f087f63

Please sign in to comment.