Skip to content

Commit c1e8f85

Browse files
notJacksonnotJackson
committed
优化前后端部分bug，docker镜像制作中
1 parent b85ba2d commit c1e8f85

File tree

396 files changed

+1289
-4371
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

396 files changed

+1289
-4371
lines changed

‎.DS_Store

-4 KB
Binary file not shown.

‎README.md

+15-101
Original file line numberDiff line numberDiff line change
@@ -1,111 +1,25 @@
1-
## 柯南流量回放平台
2-
<p align="left">
3-
<a href="https://github.com/1042970366/">
4-
<img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License" />
5-
</a>
6-
<a href="https://java.org/">
7-
<img src="https://img.shields.io/badge/java-1.8.1-green" alt="Vue2.0">
8-
</a>
9-
<a href="https://vuejs.org/">
10-
<img src="https://img.shields.io/badge/vue.js-2.0-green" alt="Vue2.0">
11-
</a>
12-
<a href="https://github.com/1042970366/">
13-
<img src="https://img.shields.io/badge/author-TALconan-blueviolet" alt="Author">
14-
</a>
15-
<a href="https://github.com/1042970366/">
16-
<img src="https://img.shields.io/badge/🚀-open--in--browser-blueviolet" alt="Live Demo">
17-
</a>
18-
</p>
1+
# conan
192

20-
经过在线教育业务中的持续打磨与迭代，柯南平台终于开源，旨在为行业内更多的的质效保障团队提供更专业更稳定的质效保障方案。随着业务与技术架构的不断变化，服务端的质量保障工作变得越来越复杂。近几年流量回放的方案在行业内落地生根，但大部分以工具为主并且使用成本与二次开发生成本较高，柯南平台应运而生。
3+
柯南流量回放整体服务
4+
http://conan.xesv5.com
215

6+
## 一、 背景
227

8+
柯南是一站式的开源流量回放平台，基于线上流量回放能力助力开发和测试团队快速拥有相关能力，加速高质量软件的交付。
239

24-
## 目标
25-
基于线上真实用户流量的录制回放能力与结果校验能力，为冒烟测试，集成回归测试，线上验证与线上巡检提供解决方案。
10+
- 流量采集: ES日志；
11+
- 回放协议: http协议；
12+
- 接入成本: 无需安装采集，对服务性能无影响；
13+
- 支持场景: 目前支持读接口。
2614

15+
## 二、项目介绍
2716

28-
## 核心功能
29-
**流量采集**
17+
### 1. 工程文件目录
3018

31-
基于ES日志源的流量录制采集，平台化配置接入，降低使用成本，并且提供详细的流量采集数据。
19+
柯南项目整体采用的父子结构
3220

21+
conan-common存放的是公共资源
3322

34-
**流量回放**
23+
conan-admin提供的是与前端数据交互的接口
3524

36-
分布式的后端架构，为流量回放提升执行效率，支持服务鉴权配置，基于http协议的回放符合真实业务场景。
37-
38-
**结果校验**
39-
40-
流量回放的常规校验方式基本上是以流量结果的DIFF为主，但大量的流量噪声（时间戳，自增数据...）一直影响结果的准确性，柯南平台在回放中基于配置的jsonSchema做第一层校验，再结合自研的降噪比对服务进行流量DIFF的第二层校验，从而保障了结果校验的准确性，大大提升了流量回放结果的可信度。
41-
42-
43-
## 平台优势与应用场景
44-
**优势**
45-
- 解决传统自动化覆盖率低，维护成本高的问题
46-
- 多规则的流量结果断言校验
47-
- 多规则的流量结果比对支持
48-
- 流量数据可用于自动化测试与性能测试
49-
- 交互简单，配置化接入
50-
- 开源共建，持续优化
51-
52-
**应用场景**
53-
- 提测质量卡点
54-
- CI/CD流水线质量卡点
55-
- 服务线上监控巡检
56-
57-
58-
**平台业务架构**
59-
![后端业务架构.png](http://ttc-tal.oss-cn-beijing.aliyuncs.com/1605259627/%E5%90%8E%E7%AB%AF%E4%B8%9A%E5%8A%A1%E6%9E%B6%E6%9E%84.png)
60-
<center>
61-
业务架构
62-
</center>
63-
64-
**平台技术架构**
65-
![后端技术架构.png](http://ttc-tal.oss-cn-beijing.aliyuncs.com/1605255935/%E5%90%8E%E7%AB%AF%E6%8A%80%E6%9C%AF%E6%9E%B6%E6%9E%84.png)
66-
<center>
67-
服务端架构
68-
</center>
69-
70-
<br>
71-
72-
**平台能力及功能**
73-
![柯南能力图.png](http://ttc-tal.oss-cn-beijing.aliyuncs.com/1605261800/%E6%9F%AF%E5%8D%97%E8%83%BD%E5%8A%9B%E5%9B%BE.png)
74-
75-
**使用须知**
76-
- 流量采集: ES日志；
77-
- 回放协议: http协议；
78-
- 具体环境可参考开源详细技术文档
79-
80-
## 写在最后
81-
质效的提升也许不能单单通过一个平台，技术与人的结合才能带来更大的突破。善于利用技术创新才能从容的面对越来越频繁的需求，越来越复杂的业务，柯南平台的技术方案产出于学而思网校的大班业务并且逐步通用化，平台现已开源，希望更多优秀的人或团队参与进来，为质效保障工作提供更多的解决方案。
82-
83-
**详细使用文档**
84-
https://dengkunnanmayun.gitee.io/conan-docs/#/use/README
85-
86-
**更多介绍**
87-
https://mp.weixin.qq.com/s/1Cvi5kkqfF9y1rBi97qLwg
88-
89-
</br>
90-
</br>
91-
92-
**项目负责人**-李宁
93-
94-
<img src="http://ttc-tal.oss-cn-beijing.aliyuncs.com/1606904630/image.png" width="100" height="100" align="middle" />
95-
</br>
96-
</br>
97-
98-
**项目成员**-刘劲松 胡耀国 邓坤楠 纪莹
99-
100-
<img src="http://ttc-tal.oss-cn-beijing.aliyuncs.com/1605698754/image.png" width="100" height="100" align="middle" />
101-
<img src="http://ttc-tal.oss-cn-beijing.aliyuncs.com/1605702320/image.png" width="100" height="100" align="middle" />
102-
<img src="http://ttc-tal.oss-cn-beijing.aliyuncs.com/1605702371/image.png" width="100" height="100" align="middle" />
103-
<img src="http://ttc-tal.oss-cn-beijing.aliyuncs.com/1605698855/image.png" width="100" height="100" align="middle" />
104-
</br>
105-
</br>
106-
107-
**柯南官方QQ群**
108-
109-
<img src="http://ttc-tal.oss-cn-beijing.aliyuncs.com/1614485571/image.png" width="150" height="230" align="middle" />
110-
111-
<br>
25+
conan-agent提供的是比较费时的服务包括录制回放和比对

‎bin/run.bat

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ cd ../conan-admin/target
88

99
set JAVA_OPTS=-Xms256m -Xmx1024m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m
1010

11-
java -jar %JAVA_OPTS% conan-admin.jar
11+
java -jar %JAVA_OPTS% ruoyi-admin.jar
1212

1313
cd bin
1414

‎conan-admin/README.md

+8-1
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,17 @@
33
TBA
44

55
#技术文档
6-
https://dengkunnanmayun.gitee.io/conan-docs/#/use/README
6+
7+
* [原型图](https://org.modao.cc/app/4e0d2060afbbbbfcc6ccbaebf7c1ace6#screen=s8A340F089B1557914187952)
8+
* [流程图]()
9+
* [架构图](http://wiki.xesv5.com/pages/viewpage.action?pageId=18573890)
10+
* [数据库](http://wiki.xesv5.com/pages/viewpage.action?pageId=18573883)
11+
* [接口文档]()
712

813
#本地开发
914

15+
*[代码规范](http://wiki.xesv5.com/pages/viewpage.action?pageId=17701300)*
16+
1017
* 访问地址：http://localhost:8080/api/1.0
1118
* 接口文档及调试：http://localhost:8080/docs.html
1219
* Druid监控：http://localhost:8080/druid (conan/conan)

‎conan-admin/shell/run.bat

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ cd ../conan-admin/target
88

99
set JAVA_OPTS=-Xms256m -Xmx1024m -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=512m
1010

11-
java -jar %JAVA_OPTS% conan-admin.jar
11+
java -jar %JAVA_OPTS% ruoyi-admin.jar
1212

1313
cd bin
1414

‎conan-admin/src/main/java/com/tal/wangxiao/conan/admin/controller/ReplayController.java

+32-22
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import com.tal.wangxiao.conan.admin.service.ReplayService;
44
import com.tal.wangxiao.conan.common.api.ApiResponse;
5+
import com.tal.wangxiao.conan.common.api.ResponseCode;
56
import com.tal.wangxiao.conan.common.controller.ConanBaseController;
67
import com.tal.wangxiao.conan.common.model.Result;
78
import com.tal.wangxiao.conan.common.model.vo.ReplayDetailVO;
@@ -47,30 +48,39 @@ public class ReplayController extends ConanBaseController {
4748
})
4849
@PreAuthorize("@ss.hasPermi('admin:execution:replay')")
4950
@PostMapping("")
50-
public ApiResponse<Object> startReplay(@RequestParam(value = "task_execution_id") Integer taskExecutionId,@RequestParam(value = "replay_env") String replayEnv,@RequestParam(value = "replay_type") Integer replayType) {
51+
public ApiResponse<Object> startReplay(@RequestParam(value = "task_execution_id") Integer taskExecutionId, @RequestParam(value = "replay_env") String replayEnv, @RequestParam(value = "replay_type") Integer replayType) {
5152
log.info("ReplayController#startReplay");
52-
Result<Object> result = replayService.startReplay(taskExecutionId,replayEnv,replayType);
53+
Result<Object> result = replayService.startReplay(taskExecutionId, replayEnv, replayType);
5354
return new ApiResponse<>(result);
5455
}
5556

5657
@ApiOperation(value = "根据任务执行ID查询流量回放记录列表", notes = "根据任务执行ID查询流量回放记录列表")
5758
@PreAuthorize("@ss.hasPermi('admin:execution:replay:list')")
5859
@ApiImplicitParam(name = "task_execution_id", value = "任务执行ID", dataType = "int", paramType = "query")
5960
@GetMapping(value = "")
60-
public ApiResponse<List<ReplayVO>> findReplayListByTaskExecutionId(@RequestParam(value = "task_execution_id") Integer taskExecutionId){
61-
log.info("ReplayController#findReplayListByTaskExecutionId:"+taskExecutionId);
61+
public ApiResponse<List<ReplayVO>> findReplayListByTaskExecutionId(@RequestParam(value = "task_execution_id") Integer taskExecutionId) {
62+
log.info("ReplayController#findReplayListByTaskExecutionId:" + taskExecutionId);
6263
Result<Object> result = replayService.findReplaysByTaskExecutionId(taskExecutionId);
63-
return new ApiResponse(result);
64+
return new ApiResponse(result);
65+
}
66+
67+
@ApiOperation(value = "根据回放ID获取最新比对ID", notes = "根据回放ID获取最新比对ID")
68+
@ApiImplicitParam(name = "replayId", value = "replayId", dataType = "int", paramType = "query")
69+
@GetMapping(value = "/getDiffIdByReplayId")
70+
public ApiResponse<Integer> findDiffIdByReplayId(@RequestParam(value = "replayId") Integer replayId) {
71+
log.info("replayId:" + replayId);
72+
Integer diffId = replayService.findDiffIdByReplayId(replayId);
73+
return new ApiResponse(new Result<>(ResponseCode.SUCCESS, diffId));
6474
}
6575

6676
@ApiOperation(value = "查询回放执行详情", notes = "根据回放ID查询回放执行详情")
6777
@Cacheable(value = "replay-info")
6878
@GetMapping(value = "detail")
6979
@PreAuthorize("@ss.hasPermi('admin:replay:detail')")
70-
public ApiResponse<List<ReplayDetailVO>> findDetailByReplayId(@RequestParam(value = "replay_id") Integer replayId){
80+
public ApiResponse<List<ReplayDetailVO>> findDetailByReplayId(@RequestParam(value = "replay_id") Integer replayId) {
7181
log.info("ReplayController#findDetailByReplayId, replay_id = {}", replayId);
7282
Result<Object> result = replayService.findDetailByReplayId(replayId);
73-
return new ApiResponse(result);
83+
return new ApiResponse(result);
7484
}
7585

7686
@ApiOperation(value = "单接口的回放数据", notes = "根据ApiID查询单接口的流量回放数据")
@@ -79,49 +89,49 @@ public ApiResponse<List<ReplayDetailVO>> findDetailByReplayId(@RequestParam(val
7989
@ApiImplicitParam(name = "api_id", value = "接口Id", dataType = "int", paramType = "query")
8090
})
8191
@GetMapping(value = "oneApiDetail")
82-
public ApiResponse<Object> findOneApiDetailByReplayIdAndApiId(@RequestParam(value = "replay_id") Integer replayId,@RequestParam(value = "api_id") Integer apiId) {
92+
public ApiResponse<Object> findOneApiDetailByReplayIdAndApiId(@RequestParam(value = "replay_id") Integer replayId, @RequestParam(value = "api_id") Integer apiId) {
8393
log.info("ReplayController#findOneApiDetailByReplayIdAndApiId");
84-
Result<Object> result = replayService.findOneApiDetailByReplayIdAndApiId(replayId,apiId);
85-
return new ApiResponse<>(result);
94+
Result<Object> result = replayService.findOneApiDetailByReplayIdAndApiId(replayId, apiId);
95+
return new ApiResponse<>(result);
8696
}
8797

8898
@ApiOperation(value = "根据回放ID查询回放日志", notes = "根据回放ID查询回放日志")
8999
@ApiImplicitParam(name = "replay_id", value = "回放Id", dataType = "int", paramType = "query")
90100
@GetMapping(value = "/log")
91101
@PreAuthorize("@ss.hasPermi('admin:replay:log')")
92-
public ApiResponse<String> findReplayLog(@RequestParam(value = "replay_id") Integer replayId){
93-
log.info("ReplayController#findReplayLog:"+replayId);
102+
public ApiResponse<String> findReplayLog(@RequestParam(value = "replay_id") Integer replayId) {
103+
log.info("ReplayController#findReplayLog:" + replayId);
94104
Result<String> result = replayService.findLogByReplayId(replayId);
95-
return new ApiResponse<>(result);
105+
return new ApiResponse<>(result);
96106
}
97107

98108
@ApiOperation(value = "根据回放ID查询回放进度", notes = "根据回放ID查询回放进度")
99109
@ApiImplicitParam(name = "replay_id", value = "回放Id", dataType = "int", paramType = "query")
100110
@GetMapping(value = "/progress")
101-
public ApiResponse<Object> findReplayProgress(@RequestParam(value = "replay_id") Integer replayId){
102-
log.info("ReplayController#findReplayProgress:"+replayId);
111+
public ApiResponse<Object> findReplayProgress(@RequestParam(value = "replay_id") Integer replayId) {
112+
log.info("ReplayController#findReplayProgress:" + replayId);
103113
Result<Object> result = replayService.findReplayProgress(replayId);
104-
return new ApiResponse<>(result);
114+
return new ApiResponse<>(result);
105115
}
106116

107117
@ApiOperation(value = "设置某次回放记录作为比对基准", notes = "设置某次回放记录作为比对基准")
108118
@ApiImplicitParams({
109119
@ApiImplicitParam(name = "replayId", value = "回放ID", dataType = "int", paramType = "query")})
110120
@GetMapping(value = "/setBaseLine")
111121
public ApiResponse<String> setReplayAsBaseLine(
112-
@RequestParam(value = "replayId") Integer replayId){
122+
@RequestParam(value = "replayId") Integer replayId) {
113123
log.info("ReplayController#setReplayAsBaseLine:replay_id=" + replayId);
114124
Result<String> result = replayService.setBaseLine(replayId);
115-
return new ApiResponse<>(result);
125+
return new ApiResponse<>(result);
116126
}
117127

118128
@ApiOperation(value = "根据回放ID写入回放进度（上线后清除）", notes = "根据回放ID写入回放进度")
119129
@ApiImplicitParam(name = "replay_id", value = "回放Id", dataType = "int", paramType = "query")
120130
@GetMapping(value = "/setProgress")
121-
public ApiResponse<Object> setReplayProgress(@RequestParam(value = "replay_id") Integer replayId,@RequestParam(value = "progress") String progress ){
122-
log.info("ReplayController#setRecordProgress:"+replayId);
123-
redisTemplateTool.setReplayProgress(replayId,progress);
124-
return new ApiResponse<>(200,"ok");
131+
public ApiResponse<Object> setReplayProgress(@RequestParam(value = "replay_id") Integer replayId, @RequestParam(value = "progress") String progress) {
132+
log.info("ReplayController#setRecordProgress:" + replayId);
133+
redisTemplateTool.setReplayProgress(replayId, progress);
134+
return new ApiResponse<>(200, "ok");
125135
}
126136

127137
}

‎conan-admin/src/main/java/com/tal/wangxiao/conan/admin/service/ReplayService.java

+7
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,13 @@ public interface ReplayService {
1717
*/
1818
Result<Object> findReplaysByTaskExecutionId(Integer taskExecutionId);
1919

20+
21+
/**
22+
* 根据任务执行ID获取流量回放记录列表
23+
* @param replayId replayId
24+
* @return
25+
*/
26+
Integer findDiffIdByReplayId(Integer replayId);
2027
/**
2128
* 查询流量回放详情
2229
* @param replayId 回放ID

‎conan-admin/src/main/java/com/tal/wangxiao/conan/admin/service/impl/DiffServiceImpl.java

+39-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package com.tal.wangxiao.conan.admin.service.impl;
22

3+
import com.alibaba.fastjson.JSON;
34
import com.tal.wangxiao.conan.admin.cache.AdminCache;
45
import com.tal.wangxiao.conan.admin.service.DiffService;
56
import com.tal.wangxiao.conan.common.api.ApiResponse;
@@ -75,11 +76,18 @@ public class DiffServiceImpl implements DiffService {
7576

7677

7778
@Resource
78-
private RedisTemplate<Object,Object> redisTemplateLog;
79+
private RedisTemplate<Object, Object> redisTemplateLog;
80+
81+
/* @Resource
82+
private StringRedisTemplate stringRedisTemplate;*/
83+
7984

8085
@Resource
8186
private StringRedisTemplate stringRedisTemplate;
8287

88+
@Resource
89+
private RedisTemplate<String, String> redisTemplate;
90+
8391
@Resource
8492
ReplayRepository replayRepository;
8593

@@ -208,7 +216,7 @@ public ApiResponse getDiffDetail( Integer apiId, Integer diffId) {
208216
String baseData = "";
209217
DiffResultInRedis diffResultInRedis = null;
210218
String requestBody = "";
211-
int baseDataReplayId = 1;
219+
int baseDataReplayId = diff.getBaseReplayId();
212220
List<String> apiRequestIdSet = diffMapper.getRecordRequestIdApiByTaskexcuteId(taskExcutionId, apiId);
213221
ApiInfo apiInfo = apiMapper.selectApiById(apiId);
214222
Optional<Record> recordOptional = recordRepository.findFirstByTaskExecutionId(taskExcutionId);
@@ -225,8 +233,11 @@ public ApiResponse getDiffDetail( Integer apiId, Integer diffId) {
225233
boolean flag = false;
226234
DiffApiLogInfo apiLogInfo = new DiffApiLogInfo();
227235
try {
228-
compareData = stringRedisTemplate.opsForValue().get(requestId + "-" + recordId + "-" + replayId + "-" + apiId)+"";
229-
baseData = stringRedisTemplate.opsForValue().get(requestId + "-" + recordId + "-" + baseDataReplayId + "-" + apiId)+"";
236+
compareData = redisTemplate.opsForValue().get(requestId + "-" + recordId + "-" + replayId + "-" + apiId)+"";
237+
compareData = splitStrDelete(compareData);
238+
239+
baseData = redisTemplate.opsForValue().get(requestId + "-" + recordId + "-" + baseDataReplayId + "-" + apiId)+"";
240+
baseData = splitStrDelete(baseData);
230241
} catch (Exception e) {
231242
return new ApiResponse(new Result(ResponseCode.INVALID_REDIS_KEY, requestId + "-" + recordId + "-" + baseDataReplayId + "-" + apiId));
232243
}
@@ -251,7 +262,8 @@ public ApiResponse getDiffDetail( Integer apiId, Integer diffId) {
251262
}
252263
//获取body信息
253264
try {
254-
requestBody = stringRedisTemplate.opsForValue().get(requestId + "-" + recordId + "-" + replayId + "-" + apiId + "-body")+"";
265+
requestBody = redisTemplate.opsForValue().get(requestId + "-" + recordId + "-" + replayId + "-" + apiId + "-body")+"";
266+
requestBody = splitStrDelete(requestBody);
255267
} catch (Exception e) {
256268
log.error("获取body信息失败，无效的redis key");
257269
}
@@ -287,6 +299,28 @@ public ApiResponse getDiffDetail( Integer apiId, Integer diffId) {
287299
return ApiResponse.success("获取diff 详细信息成功", apiDiffDetailInfo);//.success("", resultMap);
288300
}
289301

302+
private String splitStrDelete(String str) {
303+
/* if(StringHandlerUtils.isNull(str)) {
304+
return "";
305+
}
306+
if(str.length()<2) {
307+
return str;
308+
}
309+
if("\"\"".equals(str.substring(0,2))){
310+
str = str.substring(1,str.length());
311+
}
312+
if("\"\"".equals(str.substring(str.length()-3,str.length() -1))){
313+
str = str.substring(0,str.length()-2);
314+
}*/
315+
try {
316+
JSON json = JSON.parseObject(str);
317+
return json.toJSONString();
318+
} catch (Exception e) {
319+
}
320+
321+
return str;
322+
}
323+
290324
@Override
291325
public Result<String> findDiffLog( Integer diffId) {
292326
String res = stringRedisTemplate.opsForValue().get("diffLog="+diffId);

0 commit comments

Comments
 (0)