diff --git a/LICENSE b/LICENSE index 261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64..48d4f578282c0a3136ac7045a888248e030fb918 100644 --- a/LICENSE +++ b/LICENSE @@ -186,7 +186,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright [2020] [JD.com, Inc.] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/README.md b/README.md index c557dae9e84251d2f2fa25e445fe20a3aa438107..833510e29bc4757927546baad5099bc47ff52f30 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,50 @@ # hotkey +![输入图片说明](https://images.gitee.com/uploads/images/2020/0616/105737_e5b876cd_303698.png "redis热key探测及缓存到JVM (1).png") -正在京东APP后台灰度了几千台机器，等618海量并发检验后再推广使用，暂时不要下载使用它。 +对任意突发性的无法预先感知的热点数据，包括并不限于热点数据（如突发大量请求同一个商品）、热用户（如恶意爬虫刷子）、热接口（突发海量请求同一个接口）等，进行毫秒级精准探测到。然后对这些热数据、热用户等，推送到所有服务端JVM内存中，以大幅减轻对后端数据存储层的冲击，并可以由使用者决定如何分配、使用这些热key（譬如对热商品做本地缓存、对热用户进行拒绝访问、对热接口进行熔断或返回默认值）。这些热数据在整个服务端集群内保持一致性，并且业务隔离，worker端性能强悍。 -该框架历经多次压测，8核单机worker端每秒可接收处理16万个key探测任务，16核单机至少每秒20万，实际压测达到30万以上，CPU平稳支撑，框架无异常。测试详情可去我[CSDN博客](https://blog.csdn.net/tianyaleixiaowu)查看。 +京东APP后台热数据探测框架，历经多次高压压测和2020年京东618、双11大促考验。 + +在上线运行的这段时间内，每天探测的key数量数十亿计，精准捕获了大量爬虫、刷子用户，另准确探测大量热门商品并毫秒级推送到各个服务端内存，大幅降低了热数据对数据层的查询压力，提升了应用性能。 + +在大促期间，hotkey的worker集群秒级吞吐量达到1500万级别，由hotkey探测出的热key进而产生的本地缓存占应用总访问量的50%以上，使得大部分请求进行的是本地查询，减轻了redis层一半以上负担。 + +该框架历经多次压测，性能指标主要有两个： + +1 探测性能：8核单机worker端每秒可接收处理16万个key探测任务，16核单机至少每秒平稳处理30万以上，实际压测达到37万，CPU平稳支撑，框架无异常。 + +2 推送性能：在高并发写入的同时，对外推送目前性能约平稳推送每秒10-12万次，譬如有1千台server，一台worker上每秒产生了100个热key，那么这1秒会平稳推送100 * 1000 = 10万次，10万次推送会明确在1s内全部送达。如果是写入少，推送多，以纯推送来计数的话，该框架每秒可稳定对外推送40-60万次平稳，80万次极限可撑几秒。 + +每秒单机吞吐量（写入+对外推送）目前在70万左右稳定。 + +在真实业务场景中，可用1：1000的比例，即1台worker支撑1000台业务服务端的key探测任务，即可带来极大的数据存储资源节省（如对redis集群的扩充）。测试详情可去我[CSDN博客](https://blog.csdn.net/tianyaleixiaowu)查看。 + +#### 架架架架架架架架架架架构构构构构构构构构构构设设设设设设设设设设计计计计计计计计计计计计 + **见京东零售公众号** +https://mp.weixin.qq.com/s/xOzEj5HtCeh_ezHDPHw6Jw + + **！！！！！！！！！！！！！！！！重要！！！！！！！！！！！！！！！不然看不懂框架原理** + +#### 参与开发 +该框架的开发得到了来自于： + +京东数科-生活服务研发组-李云峰的大力支持，完成了整个dashboard控制台的研发。 + +京东零售-平台业务中心-APP后台架构师 杜晓宇的大力支持，参与并帮助解决了诸多技术难题。 + +京东零售-平台业务中心-PC购物车业务组 任启恒，进行了多次极限压测及功能验证，为框架的健壮性提供了更高的标准。 + +京东零售-技术与数据中心-订单研发组 姜坤坤，进行了漫长和覆盖更加全面的功能验证，协助进行了bug发现和bugfix。 + +京东零售-平台业务中心-业务测试部 郭世儒，配合进行了多次军演模拟压力测试演练，协助验证框架在模拟真实流量时的表现。 + +京东数科-消费金融研发部-账务服务 王小飞，对框架提出了重要的功能新增建议，以及新增功能的协助验证。 + +外部开发者，来自于[lowbMan](https://gitee.com/lowbMan)提供了关于修改proto序列化方式的建议，以及其他有相当价值的优化建议。 + +另有京东集团多个部门提供的意见、建议，和对框架的见解、优化方案。 + +在此一起感谢。 #### 介绍 对任意突发性的无法预先感知的热点请求，包括并不限于热点数据（如突发大量请求同一个商品）、热用户（如爬虫、刷子）、热接口（突发海量请求同一个接口）等，进行毫秒级精准探测到。 @@ -23,64 +65,202 @@ 5 接口、用户维度限流 -6 单机接口、用户维度限流限流 +6 单机接口、用户维度限流 7 集群用户维度限流 8 集群接口维度限流 -#### 尚未完成 -控制台功能缺失如下： +该开源项目战略意义重大，经历百万级并发，参与京东开源中间件项目建设，一直在等你。 + +#### worker端强悍的性能表现 +每10秒打印一行，totalDealCount代表处理过的key总量，可以看到每10秒处理量在270万-310万之间，对应每秒30万左右QPS。 + +仅需要很少的机器，即可完成海量key的实时探测计算推送任务。比扩容redis集群规模成本低太多。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152336_78597937_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152249_4ac01178_303698.png "屏幕截图.png") -1.rule的定时保存，当etcd更换时需要能一键导入原有配置 -2.对/jd/count/cartsoa/ 目录下信息进行展示，代表的是每个worker连接的client数量（已完成） +采用protobuf序列化后性能进一步得到提升。在秒级36万以上时，能稳定在CPU 60%，压测持续时长超过5小时，未见任何异常。30万时，压测时长超过数日，未见任何异常。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/104833_6837a091_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/104930_c522bd91_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/105009_132a878b_303698.png "屏幕截图.png") -3.对/jd/caffeineSize/ 目录进行展示，里面是每个worker内caffeine缓存的数量 +### 界面效果 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/163805_0aa68d4b_303698.png "屏幕截图.png") +### 加微信入群讨论问题，1群已满，请加2群 +![输入图片说明](https://images.gitee.com/uploads/images/2020/1019/094556_bd1f7057_303698.png "12.png") -4.筛选功能，对记录表里做筛选，按时间范围筛选key出现次数大于xx次的数据 -5.导出功能，将key排重后导出的功能，按时间范围筛选 +#### 常见问题 + **1 worker挂了怎么办** -6.还有很多 +client根据worker的数量对key进行hash后分发，同一个key一定会被发往同一个worker。譬如4台，挂了一台，key就自动hash到另外3台。那么这个过程中，就会丢失最多一个探测周期内的所有发来的key，譬如2秒10次算热，那么就可能全部被rehash，丢失这2秒的数据。 +它的影响是什么呢？我要不要去存下来所有发来的key呢？很多人都会问的问题。 -该开源项目战略意义重大，要经历百万级并发，参与京东开源中间件项目建设，一直在等你。 +首先挂机，那是极其罕见的事件，即便挂了，对于特别热的key，完全不影响，hash丢几秒，不影响它继续瞬间变热。对于不热的key，它挂不挂，它也热不了。对于那些将热未热的，可能会这次让它热不起来，但没有什么影响，业务服务完全可以吃下这个热key。而加上一堆别的组件如存储、worker间通信传输key等，它的复杂度，性能都会影响很大。 -#### worker端强悍的性能表现 -每10秒打印一行，totalDealCount代表处理过的key总量，可以看到每10秒处理量在270万-310万之间，对应每秒30万左右QPS。 +所以它挂了对系统没有任何影响 -仅需要很少的机器，即可完成海量key的实时探测计算推送任务。比扩容redis集群规模成本低太多了。 -![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152336_78597937_303698.png "屏幕截图.png") -![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152249_4ac01178_303698.png "屏幕截图.png") +**2 为什么全部要worker汇总计算，而不是客户端自己计算** + +首先，客户端是会本地累加的，在固定的上报周期内，如500ms内，本地就是在累加，每500ms批量上报给worker一次。如果上报频率很高，如10ms一次，那么大概率本地同一个key是没有累加。 + +有人会说，把这个间隔拉长，譬如本地计算3秒后，本地判定热key，再上报给其他机器。那么这种场景首先对于京东是不可行的，哪怕1秒都不行。譬如一个用户刷子，它在非常频繁地刷接口，一秒刷了500次，而正常用户一秒最多点5次，它已经是非常严重的刷子了。但我们本地还是判断不出来它是不是刷子。为什么？机器多。 + +随便一个app小组都有数千台机器，一秒500次请求，一个机器连1次都平均不到，大部分是0次，本地如何判断它是刷子呢？总不能访问我一次就算它刷吧。 +然后抢购场景，有些秒杀商品，1-2秒就没了，流量就停了，你本地计算了3秒，才去上报，那活动已经结束了，你的热key再推送已经没价值了。我们就要在活动即将开始之前的可能在10ms内，就要该商品被推送到所有client的jvm里去，根本等不了1秒。 + +**3 为什么是worker推送，而不是worker发送热key到etcd，客户端直接监听etcd获取热key** + +（1） worker和client是长连接，产生热key后，直接推送过去，链路短，耗时少。如果是发到etcd，客户端再通过etcd获取，多了一层中转，耗时明显增加。 + +（2） etcd性能不够，存在单点风险。譬如我有5000台client，每秒产生100个热key，那么每秒就对应50万次推送。我用2台worker即可轻松完成，随着worker的横向扩展，每秒的推送上限线性增加。但无论是etcd、redis等等任何组件，都不可能做到1秒50万次拉取或推送，会瞬间cpu爆满卡死。因为worker是各自隔离的，而etcd是单点的。实际情况下，也不止5000台client，每秒也不止100个热key，只有当前的架构能支撑。 + +**4 为什么是etcd，不是zookeeper之类的** + +etcd里面具备一个过期删除的功能，你可以设置一个key几秒过期，etcd会自动删除它，删除时还会给所有监听的client回调，这个功能在框架里是在用的，别的配置中心没有这个功能。 + +etcd的性能和稳定性、低负载等各项指标非常优异，完全满足我们的需求。而zk在很多暴涨流量前和高负载下，并不是那么稳定，性能也差的远。 #### 安装教程 -1. xxxx -2. xxxx -3. xxxx +1. 安装etcd + + 在etcd下载页面下载对应操作系统的etcd，https://github.com/etcd-io/etcd/releases 使用3.4.x以上。相关搭建细节，及常见问题会发布到CSDN博客内。 + +2. 启动worker（集群） + 下载并编译好代码，将worker打包为jar，启动即可。如： + + **` java -jar $JAVA_OPTS worker-0.0.1-SNAPSHOT.jar --etcd.server=${etcdServer}`** + + worker可供配置项如下： +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/164514_c57d740a_303698.png "屏幕截图.png") + etcdServer为etcd集群的地址，用逗号分隔 + + JAVA_OPTS是配置的JVM相关，可根据实际情况配置 + + threadCount为处理key的线程数，不指定时由程序来计算。 + + workerPath代表该worker为哪个应用提供计算服务，譬如不同的应用appName需要用不同的worker进行隔离，以避免资源竞争。 + +3. 启动控制台 + + 下载并编译好dashboard项目，创建数据库并导入resource下db.sql文件。 配置一下application.yml里的数据库相关和etcdServer地址。 + + 启动dashboard项目，访问ip:8081，即可看到界面。 + + 其中节点信息里，即是当前已启动的worker列表。 + + 规则配置就是为各app设置规则的地方，初次使用时需要先添加APP。在用户管理菜单中，添加一个新用户，设置他的APP名字，如sample。之后新添加的这个用户就可以登录dashboard给自己的APP设置规则了，登录密码默认123456。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/175255_e1b05b4c_303698.png "屏幕截图.png") + + 如图就是一组规则，譬如其中as__开头的热key的规则就是interval-2秒内出现了threshold-10次就认为它是热key，它就会被推送到jvm内存中，并缓存60秒，prefix-true代表前缀匹配。那么在应用中，就可以把一组key，都用as__开头，用来探测。 + +4. client端接入使用 + + 引入client的pom依赖。 + + 在应用启动的地方初始化HotKey，譬如 + +``` +@PostConstruct + +public void initHotkey() { + + ClientStarter.Builder builder = new ClientStarter.Builder(); + ClientStarter starter = builder.setAppName("appName").setEtcdServer("http://1.8.8.4:2379,http://1.1.4.4:2379,http://1.1.1.1:2379").build(); + starter.startPipeline(); +} +``` +其中还可以setCaffeineSize(int size)设置本地缓存最大数量，默认5万，setPushPeriod(Long period)设置批量推送key的间隔时间，默认500ms，该值越小，上报热key越频繁，相应越及时，建议根据实际情况调整，如单机每秒qps10个，那么0.5秒上报一次即可，否则是空跑。该值最小为1，即1ms上报一次。 + +注意： + +如果原有项目里使用了guava，需要升级guava为以下版本，否则过低的guava版本可能发生jar包冲突。或者删除自己项目里的guava的maven依赖，guava升级不会影响原有任何逻辑。 + +``` + + com.google.guava + guava + 28.2-jre + compile + +``` + +有时可能项目里没有直接依赖guava，但是引入的某个pom里引了guava，也需要将guava排除掉。 + + + +使用： + +主要有如下4个方法可供使用 + +boolean JdHotKeyStore.isHotKey(String key) + +Object JdHotKeyStore.get(String key) + +void JdHotKeyStore.smartSet(String key, Object value) + +Object JdHotKeyStore.getValue(String key) + + + +1 boolean isHotKey(String key) ，该方法会返回该key是否是热key，如果是返回true，如果不是返回false，并且会将key上报到探测集群进行数量计算。该方法通常用于判断只需要判断key是否热、不需要缓存value的场景，如刷子用户、接口访问频率等。 + +2 Object get(String key)，该方法返回该key本地缓存的value值，可用于判断是热key后，再去获取本地缓存的value值，通常用于redis热key缓存 + +3 void smartSet(String key, Object value)，方法给热key赋值value，如果是热key，该方法才会赋值，非热key，什么也不做 + +4 Object getValue(String key)，该方法是一个整合方法，相当于isHotKey和get两个方法的整合，该方法直接返回本地缓存的value。 +如果是热key，则存在两种情况，1是返回value，2是返回null。返回null是因为尚未给它set真正的value，返回非null说明已经调用过set方法了，本地缓存value有值了。 +如果不是热key，则返回null，并且将key上报到探测集群进行数量探测。 + + +最佳实践： + +1 判断用户是否是刷子 + + if (JdHotKeyStore.isHotKey(“pin__” + thePin)) { + //限流他，do your job + } +2 判断商品id是否是热点 + + + Object skuInfo = JdHotKeyStore.getValue("skuId__" + skuId); + if(skuInfo == null) { + JdHotKeyStore.smartSet("skuId__" + skuId, theSkuInfo); + } else { + //使用缓存好的value即可 + } + + 或者这样： + + + + if (JdHotKeyStore.isHotKey(key)) { + //注意是get，不是getValue。getValue会获取并上报，get是纯粹的本地获取 + Object skuInfo = JdHotKeyStore.get("skuId__" + skuId); + if(skuInfo == null) { + JdHotKeyStore.smartSet("skuId__" + skuId, theSkuInfo); + } else { + //使用缓存好的value即可 + } + + } + +### 测试环境 -#### 使用说明 +我司为方便大家快速接入试用、查看hotkey效果，提供了1台公网etcd机器（16c16g）、2台worker机器（8c12g），供快速接入测试。 -1. xxxx -2. xxxx -3. xxxx -#### 参与贡献 +etcd地址为： **http://open-etcd.jd.com:2000** ，可以在下载项目后，在sample项目、dashboard项目里yml文件修改etcd连接地址为该地址，然后进入界面控制台，即可看到当前连接的worker，之后可以在sample项目里，进行hotkey测试。 -1. Fork 本仓库 -2. 新建 Feat_xxx 分支 -3. 提交代码 -4. 新建 Pull Request +控制台公网测试地址：http://hotkey.tianyalei.com:9001/ 账号admin，密码123456 -#### 码云特技 -1. 使用 Readme\_XXX.md 来支持不同的语言，例如 Readme\_en.md, Readme\_zh.md -2. 码云官方博客 [blog.gitee.com](https://blog.gitee.com) -3. 你可以 [https://gitee.com/explore](https://gitee.com/explore) 这个地址来了解码云上的优秀开源项目 -4. [GVP](https://gitee.com/gvp) 全称是码云最有价值开源项目，是码云综合评定出的优秀开源项目 -5. 码云官方提供的使用手册 [https://gitee.com/help](https://gitee.com/help) -6. 码云封面人物是一档用来展示码云会员风采的栏目 [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/) +测试前，注意在控制台新建自己的app规则。 diff --git a/client/pom.xml b/client/pom.xml index 18ef75f4da6be3c177b566549f42a86cbc57fd6d..12f5bc4df993894cdbbc0a64fdc0a5d98d8362d0 100644 --- a/client/pom.xml +++ b/client/pom.xml @@ -5,12 +5,12 @@ hotkey com.jd.platform.hotkey - 0.0.2-SNAPSHOT + 0.0.4-SNAPSHOT 4.0.0 jar - 0.0.2-SNAPSHOT + 0.0.4-SNAPSHOT hotkey-client diff --git a/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java b/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java index 1102d8a19acecf4db3955dd2a5204aa7004f83a1..27474e7bc6c8348ed5ec9e8d28409b043b8d6ece 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java @@ -1,6 +1,5 @@ package com.jd.platform.hotkey.client; -import com.jd.platform.hotkey.client.cache.LocalCache; import com.jd.platform.hotkey.client.callback.ReceiveNewKeySubscribe; import com.jd.platform.hotkey.client.core.eventbus.EventBusCenter; import com.jd.platform.hotkey.client.core.key.PushSchedulerStarter; @@ -40,9 +39,8 @@ public class ClientStarter { public static class Builder { private String appName; private String etcdServer; - private LocalCache localCache; private Long pushPeriod; - private int caffeineSize = 50000; + private int caffeineSize = 200000; public Builder() { } @@ -65,11 +63,6 @@ public class ClientStarter { return this; } - public Builder setLocalCache(LocalCache localCache) { - this.localCache = localCache; - return this; - } - public Builder setPushPeriod(Long pushPeriod) { this.pushPeriod = pushPeriod; return this; diff --git a/client/src/main/java/com/jd/platform/hotkey/client/Context.java b/client/src/main/java/com/jd/platform/hotkey/client/Context.java index f0455a21a20d2c5e146374ed9841c8b12496fea1..fea41067c1047bd1538f6794fc71e03502c47d96 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/Context.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/Context.java @@ -12,5 +12,5 @@ public class Context { */ public static boolean NEED_RECONNECT = true; - public static int CAFFEINE_SIZE = 50000; + public static int CAFFEINE_SIZE; } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/callback/AbsReceiveNewKey.java b/client/src/main/java/com/jd/platform/hotkey/client/callback/AbsReceiveNewKey.java deleted file mode 100644 index e3013be6ca248a686b7a8d880321ee2b4b4e644f..0000000000000000000000000000000000000000 --- a/client/src/main/java/com/jd/platform/hotkey/client/callback/AbsReceiveNewKey.java +++ /dev/null @@ -1,65 +0,0 @@ -//package com.jd.platform.hotkey.client.callback; -// -//import com.jd.platform.hotkey.client.log.JdLogger; -//import com.jd.platform.hotkey.common.model.HotKeyModel; -//import com.jd.platform.hotkey.common.model.typeenum.KeyType; -// -///** -// * @author wuweifeng wrote on 2020-02-24 -// * @version 1.0 -// */ -//public abstract class AbsReceiveNewKey implements ReceiveNewKeyListener { -// -// -// @Override -// public void newKey(HotKeyModel hotKeyModel) { -// long now = System.currentTimeMillis(); -// //如果key到达时已经过去5秒了，记录一下。手工删除key时，没有CreateTime -// if (hotKeyModel.getCreateTime() != 0 && Math.abs(now - hotKeyModel.getCreateTime()) > 1000) { -// JdLogger.warn(getClass(), "the key comes too late : " + hotKeyModel.getKey() + " now " + -// + now + " keyCreateAt " + hotKeyModel.getCreateTime()); -// } -// if (hotKeyModel.isRemove()) { -// deleteKey(hotKeyModel.getKey(), hotKeyModel.getKeyType(), hotKeyModel.getCreateTime()); -// } else { -// //已经是热key了，又推过来同样的热key，做个日志记录，并刷新一下 -// if (JdHotKeyStore.isHot(hotKeyModel.getKey())) { -// JdLogger.warn(getClass(), "receive repeat hot key ：" + hotKeyModel.getKey() + " at " + now); -// -// //可能存在瞬间的该value过期的情况，所以要判空 -//// ValueModel valueModel = JdHotKeyStore.getValueSimple(hotKeyModel.getKey()); -//// if (valueModel != null) { -//// valueModel.setCreateTime(System.currentTimeMillis()); -//// } else { -//// //else大概率走不到 -//// valueModel = ValueModel.defaultValue(hotKeyModel.getKey()); -//// } -// //已经是热key了，重新发过来了一样的，就将value初始化,客户端需要重新reset the value -// //why my idea is so kakaka 怎么办，用baidu还卡不卡呢还是卡啊范围欧尼潍坊 -// ValueModel valueModel = ValueModel.defaultValue(hotKeyModel.getKey()); -// if (valueModel != null) { -// valueModel.setValue(valueModel); -// //刷新过期时间 -// JdHotKeyStore.setValueDirectly(hotKeyModel.getKey(), valueModel); -// } -// } else { -// //不是重复热key时，新建热key -// addKey(hotKeyModel.getKey(), hotKeyModel.getKeyType(), hotKeyModel.getCreateTime()); -// } -// } -// -// } -// -// abstract void addKey(String key, KeyType keyType, long createTime); -// -// abstract void deleteKey(String key, KeyType keyType, long createTime); -// -// protected void addNewKey(HotKeyModel hotKeyModel) { -// ValueModel valueModel = ValueModel.defaultValue(hotKeyModel.getKey()); -// if (valueModel != null) { -// valueModel.setValue(valueModel); -// //刷新过期时间 -// JdHotKeyStore.setValueDirectly(hotKeyModel.getKey(), valueModel); -// } -// } -//} diff --git a/client/src/main/java/com/jd/platform/hotkey/client/callback/JdHotKeyStore.java b/client/src/main/java/com/jd/platform/hotkey/client/callback/JdHotKeyStore.java index 17e4fc29caebc3b8332783cb5012a4c774c3f9f4..356fafaab9626a3298dd100d37b64e6f830b3b35 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/callback/JdHotKeyStore.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/callback/JdHotKeyStore.java @@ -83,6 +83,17 @@ public class JdHotKeyStore { } } + /** + * 强制给value赋值 + */ + public static void forceSet(String key, Object value) { + ValueModel valueModel = ValueModel.defaultValue(key); + if (valueModel != null) { + valueModel.setValue(value); + } + setValueDirectly(key, valueModel); + } + /** * 获取value，如果value不存在则发往netty */ diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/HotKeyPusher.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/HotKeyPusher.java index 50231ebb78836303066496bb24370a81bb1e537d..6ef8d2578725e2b1c7d36d326b98287aa7ca66b2 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/HotKeyPusher.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/HotKeyPusher.java @@ -8,6 +8,8 @@ import com.jd.platform.hotkey.common.model.typeenum.KeyType; import com.jd.platform.hotkey.common.tool.Constant; import com.jd.platform.hotkey.common.tool.HotKeyPathTool; +import java.util.concurrent.atomic.LongAdder; + /** * 客户端上传热key的入口调用 * @@ -26,10 +28,14 @@ public class HotKeyPusher { if (key == null) { return; } + + LongAdder adderCnt = new LongAdder(); + adderCnt.add(count); + HotKeyModel hotKeyModel = new HotKeyModel(); hotKeyModel.setAppName(Context.APP_NAME); hotKeyModel.setKeyType(keyType); - hotKeyModel.setCount(count); + hotKeyModel.setCount(adderCnt); hotKeyModel.setRemove(remove); hotKeyModel.setKey(key); diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyRemover.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyRemover.java deleted file mode 100644 index 564c832d55ad7dfd52b037099c8dadc8e2ae3407..0000000000000000000000000000000000000000 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyRemover.java +++ /dev/null @@ -1,21 +0,0 @@ -package com.jd.platform.hotkey.client.core.key; - -import com.jd.platform.hotkey.client.etcd.EtcdConfigFactory; -import com.jd.platform.hotkey.client.log.JdLogger; - -/** - * 删除某个key - * @author wuweifeng - * @version 1.0 - * @date 2020-07-16 - */ -public class KeyRemover { - - public static void remove(String key) { - try { - EtcdConfigFactory.configCenter().delete(key); - } catch (Exception e) { - JdLogger.error(KeyRemover.class, "remove key error"); - } - } -} diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java index d5afa72900cffa6c9c37570a83eafd85171b5554..f6210f9a4e692d7965cbdffd369427a9e8b2c32b 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java @@ -6,9 +6,7 @@ import com.jd.platform.hotkey.client.log.JdLogger; import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.common.model.HotKeyMsg; import com.jd.platform.hotkey.common.model.KeyCountModel; -import com.jd.platform.hotkey.common.model.MsgBuilder; import com.jd.platform.hotkey.common.model.typeenum.MessageType; -import com.jd.platform.hotkey.common.tool.FastJsonUtils; import io.netty.channel.Channel; import java.net.InetSocketAddress; @@ -44,7 +42,9 @@ public class NettyKeyPusher implements IKeyPusher { for (Channel channel : map.keySet()) { try { List batch = map.get(channel); - channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.REQUEST_NEW_KEY, FastJsonUtils.convertObjectToJSON(batch)))); + HotKeyMsg hotKeyMsg = new HotKeyMsg(MessageType.REQUEST_NEW_KEY, Context.APP_NAME); + hotKeyMsg.setHotKeyModels(batch); + channel.writeAndFlush(hotKeyMsg).sync(); } catch (Exception e) { try { InetSocketAddress insocket = (InetSocketAddress) channel.remoteAddress(); @@ -77,8 +77,9 @@ public class NettyKeyPusher implements IKeyPusher { for (Channel channel : map.keySet()) { try { List batch = map.get(channel); - channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(Context.APP_NAME, - MessageType.REQUEST_HIT_COUNT, FastJsonUtils.convertObjectToJSON(batch)))); + HotKeyMsg hotKeyMsg = new HotKeyMsg(MessageType.REQUEST_HIT_COUNT, Context.APP_NAME); + hotKeyMsg.setKeyCountModels(batch); + channel.writeAndFlush(hotKeyMsg).sync(); } catch (Exception e) { try { InetSocketAddress insocket = (InetSocketAddress) channel.remoteAddress(); diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnCountCollector.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnCountCollector.java index 329ec7c64c2c7430cbea8cc489622b3010697990..d2274916ce3c660a84d627767d2e902b35919b18 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnCountCollector.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnCountCollector.java @@ -11,8 +11,9 @@ import java.util.Date; import java.util.List; import java.util.Map; import java.util.concurrent.ConcurrentHashMap; -import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; +import java.util.concurrent.atomic.LongAdder; +import java.util.stream.Collectors; /** * 热点数量统计 @@ -36,6 +37,8 @@ public class TurnCountCollector implements IKeyCollector lockAndGetResult() { //自增后，对应的map就会停止被写入，等待被读取 @@ -56,14 +59,45 @@ public class TurnCountCollector implements IKeyCollector get(ConcurrentHashMap map) { - List list = new ArrayList<>(); + //根据待转换并上报的统计数据的数据量选择是否启用并行参数转换 + if (map.size()>DATA_CONVERT_SWITCH_THRESHOLD){ + return parallelConvert(map); + }else { + return syncConvert(map); + } + } + + /** + * 在数据量足够大的情况下 并行转换可以拥有比串行for循环更好的性能 + * @param map 统计数据 + * @return 待上报数据 + */ + private List parallelConvert(ConcurrentHashMap map) { + return map.entrySet().parallelStream().map(entry->{ + String key = entry.getKey(); + HitCount hitCount = entry.getValue(); + KeyCountModel keyCountModel = new KeyCountModel(); + keyCountModel.setTotalHitCount((int)hitCount.totalHitCount.sum()); + keyCountModel.setRuleKey(key); + keyCountModel.setHotHitCount((int)hitCount.hotHitCount.sum()); + return keyCountModel; + }).collect(Collectors.toList()); + } + + /** + * 在数据量不大的情况下,使用同步for循环进行数据转换性能也不错 + * @param map 统计数据 + * @return 待上报数据 + */ + private List syncConvert(ConcurrentHashMap map) { + List list = new ArrayList<>(map.size()); for (Map.Entry entry : map.entrySet()) { String key = entry.getKey(); HitCount hitCount = entry.getValue(); KeyCountModel keyCountModel = new KeyCountModel(); - keyCountModel.setTotalHitCount(hitCount.totalHitCount.get()); + keyCountModel.setTotalHitCount((int)hitCount.totalHitCount.sum()); keyCountModel.setRuleKey(key); - keyCountModel.setHotHitCount(hitCount.hotHitCount.get()); + keyCountModel.setHotHitCount((int)hitCount.hotHitCount.sum()); list.add(keyCountModel); } return list; @@ -97,9 +131,9 @@ public class TurnCountCollector implements IKeyCollector new HitCount()); if (isHot) { - hitCount.hotHitCount.incrementAndGet(); + hitCount.hotHitCount.increment(); } - hitCount.totalHitCount.incrementAndGet(); + hitCount.totalHitCount.increment(); } private String nowTime() { @@ -109,7 +143,7 @@ public class TurnCountCollector implements IKeyCollector //不存在时返回null并将key-value放入，已有相同key时，返回该key对应的value，并且不覆盖 HotKeyModel model = map0.putIfAbsent(key, hotKeyModel); if (model != null) { - model.setCount(model.getCount() + hotKeyModel.getCount()); + model.add(hotKeyModel.getCount()); } } else { HotKeyModel model = map1.putIfAbsent(key, hotKeyModel); if (model != null) { - model.setCount(model.getCount() + hotKeyModel.getCount()); + model.add(hotKeyModel.getCount()); } } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/worker/WorkerInfoHolder.java b/client/src/main/java/com/jd/platform/hotkey/client/core/worker/WorkerInfoHolder.java index 72471a9c49c9073af28b7a03b71937f7e07e40cb..c12182f2c5a68235d91cacaae6447c829a26f0ee 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/worker/WorkerInfoHolder.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/worker/WorkerInfoHolder.java @@ -57,10 +57,11 @@ public class WorkerInfoHolder { } public static Channel chooseChannel(String key) { - if (StrUtil.isEmpty(key) || WORKER_HOLDER.size() == 0) { + int size = WORKER_HOLDER.size(); + if (StrUtil.isEmpty(key) || size == 0) { return null; } - int index = Math.abs(key.hashCode() % WORKER_HOLDER.size()); + int index = Math.abs(key.hashCode() % size); return WORKER_HOLDER.get(index).channel; } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java b/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java index c129fb155b99d17303e2d2bcad97056cdc992c52..ebf49a59716f1360fa64182ff94fc531c931ce55 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java @@ -38,8 +38,6 @@ public class EtcdStarter { fetchRule(); - fetchExistHotKey(); - // startWatchWorker(); startWatchRule(); @@ -49,7 +47,7 @@ public class EtcdStarter { } /** - * 启动后先拉取已存在的热key + * 启动后先拉取已存在的热key（来自于手工添加的目录） */ private void fetchExistHotKey() { JdLogger.info(getClass(), "--- begin fetch exist hotKey from etcd ----"); @@ -57,7 +55,6 @@ public class EtcdStarter { try { //获取所有热key List handKeyValues = configCenter.getPrefix(ConfigConstant.hotKeyPath + Context.APP_NAME); - List workerKeyValues = configCenter.getPrefix(ConfigConstant.hotKeyRecordPath + Context.APP_NAME); for (KeyValue keyValue : handKeyValues) { String key = keyValue.getKey().toStringUtf8().replace(ConfigConstant.hotKeyPath + Context.APP_NAME + "/", ""); @@ -66,13 +63,6 @@ public class EtcdStarter { model.setKey(key); EventBusCenter.getInstance().post(new ReceiveNewKeyEvent(model)); } - for (KeyValue keyValue : workerKeyValues) { - String key = keyValue.getKey().toStringUtf8().replace(ConfigConstant.hotKeyRecordPath + Context.APP_NAME + "/", ""); - HotKeyModel model = new HotKeyModel(); - model.setRemove(false); - model.setKey(key); - EventBusCenter.getInstance().post(new ReceiveNewKeyEvent(model)); - } } catch (StatusRuntimeException ex) { //etcd连不上 JdLogger.error(getClass(), "etcd connected fail. Check the etcd address!!!"); @@ -243,6 +233,9 @@ public class EtcdStarter { JdLogger.info(getClass(), "trying to connect to etcd and fetch rule info"); boolean success = fetchRuleFromEtcd(); if (success) { + //拉取已存在的热key + fetchExistHotKey(); + scheduledExecutorService.shutdown(); } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java b/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java index d2a36a41c7819ad537e1f8852a2cfe7906573da0..6caa7b2fea9d929c6c57db6f155497cfb14e3114 100755 --- a/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java @@ -2,8 +2,8 @@ package com.jd.platform.hotkey.client.netty; import com.jd.platform.hotkey.client.core.worker.WorkerInfoHolder; import com.jd.platform.hotkey.client.log.JdLogger; -import com.jd.platform.hotkey.common.coder.Codec; -import com.jd.platform.hotkey.common.coder.NettyCodec; +import com.jd.platform.hotkey.common.coder.MsgDecoder; +import com.jd.platform.hotkey.common.coder.MsgEncoder; import com.jd.platform.hotkey.common.tool.Constant; import io.netty.bootstrap.Bootstrap; import io.netty.buffer.ByteBuf; @@ -13,7 +13,6 @@ import io.netty.channel.nio.NioEventLoopGroup; import io.netty.channel.socket.SocketChannel; import io.netty.channel.socket.nio.NioSocketChannel; import io.netty.handler.codec.DelimiterBasedFrameDecoder; -import io.netty.handler.codec.string.StringDecoder; import io.netty.handler.timeout.IdleStateHandler; import java.util.List; @@ -28,9 +27,6 @@ public class NettyClient { private Bootstrap bootstrap; - private Codec codec = new NettyCodec(); - - public static NettyClient getInstance() { return nettyClient; } @@ -56,9 +52,8 @@ public class NettyClient { ByteBuf delimiter = Unpooled.copiedBuffer(Constant.DELIMITER.getBytes()); ch.pipeline() .addLast(new DelimiterBasedFrameDecoder(Constant.MAX_LENGTH, delimiter)) -// .addLast(codec.newEncoder()) -// .addLast(codec.newDecoder()) - .addLast(new StringDecoder()) + .addLast(new MsgDecoder()) + .addLast(new MsgEncoder()) //10秒没消息时，就发心跳包过去 .addLast(new IdleStateHandler(0, 0, 30)) .addLast(nettyClientHandler); diff --git a/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClientHandler.java b/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClientHandler.java index aa78dcdeabe0a854f2037d3970dbb3e15fa9562e..67100172bdbc28b64579b47a3e742ac51c2039a1 100755 --- a/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClientHandler.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClientHandler.java @@ -1,5 +1,6 @@ package com.jd.platform.hotkey.client.netty; +import cn.hutool.core.collection.CollectionUtil; import com.jd.platform.hotkey.client.Context; import com.jd.platform.hotkey.client.callback.ReceiveNewKeyEvent; import com.jd.platform.hotkey.client.core.eventbus.EventBusCenter; @@ -7,10 +8,7 @@ import com.jd.platform.hotkey.client.log.JdLogger; import com.jd.platform.hotkey.client.netty.event.ChannelInactiveEvent; import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.common.model.HotKeyMsg; -import com.jd.platform.hotkey.common.model.MsgBuilder; import com.jd.platform.hotkey.common.model.typeenum.MessageType; -import com.jd.platform.hotkey.common.tool.Constant; -import com.jd.platform.hotkey.common.tool.FastJsonUtils; import io.netty.channel.Channel; import io.netty.channel.ChannelHandler; import io.netty.channel.ChannelHandlerContext; @@ -22,7 +20,7 @@ import io.netty.handler.timeout.IdleStateEvent; * @author wuweifeng wrote on 2019-11-05. */ @ChannelHandler.Sharable -public class NettyClientHandler extends SimpleChannelInboundHandler { +public class NettyClientHandler extends SimpleChannelInboundHandler { @Override public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception { @@ -31,7 +29,7 @@ public class NettyClientHandler extends SimpleChannelInboundHandler { if (idleStateEvent.state() == IdleState.ALL_IDLE) { //向服务端发送消息 - ctx.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.PING, Constant.PING))); + ctx.writeAndFlush(new HotKeyMsg(MessageType.PING, Context.APP_NAME)); } } @@ -41,7 +39,7 @@ public class NettyClientHandler extends SimpleChannelInboundHandler { @Override public void channelActive(ChannelHandlerContext ctx) { JdLogger.info(getClass(), "channelActive:" + ctx.name()); - ctx.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.APP_NAME, Context.APP_NAME))); + ctx.writeAndFlush(new HotKeyMsg(MessageType.APP_NAME, Context.APP_NAME)); } @Override @@ -57,16 +55,19 @@ public class NettyClientHandler extends SimpleChannelInboundHandler { } @Override - protected void channelRead0(ChannelHandlerContext channelHandlerContext, String message) { - HotKeyMsg msg = FastJsonUtils.toBean(message, HotKeyMsg.class); + protected void channelRead0(ChannelHandlerContext channelHandlerContext, HotKeyMsg msg) { if (MessageType.PONG == msg.getMessageType()) { JdLogger.info(getClass(), "heart beat"); return; } if (MessageType.RESPONSE_NEW_KEY == msg.getMessageType()) { JdLogger.info(getClass(), "receive new key : " + msg); - HotKeyModel model = FastJsonUtils.toBean(msg.getBody(), HotKeyModel.class); - EventBusCenter.getInstance().post(new ReceiveNewKeyEvent(model)); + if (CollectionUtil.isEmpty(msg.getHotKeyModels())) { + return; + } + for (HotKeyModel model : msg.getHotKeyModels()) { + EventBusCenter.getInstance().post(new ReceiveNewKeyEvent(model)); + } } } diff --git a/common/pom.xml b/common/pom.xml index 59a7d76b70114ca8342f9c9bb936b31764caf036..ecfc89efebcf6734cfc909f173e53223c15a74ea 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -5,7 +5,7 @@ hotkey com.jd.platform.hotkey - 0.0.2-SNAPSHOT + 0.0.4-SNAPSHOT 4.0.0 @@ -15,15 +15,27 @@ 1.8 4.1.42.Final - 3.4.2 1.1.7.3 1.2.70 - 0.0.14 + 0.0.16 2.8.0 5.1.0 + 1.7.4 + + io.protostuff + protostuff-core + ${protostuff.version} + + + + io.protostuff + protostuff-runtime + ${protostuff.version} + + cn.hutool hutool-all @@ -40,13 +52,6 @@ netty-all ${netty4.version} - - - com.lmax - disruptor - ${disruptor.version} - - com.alibaba fastjson diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java deleted file mode 100755 index c61805b9d34c84981d0b2c90891e305ae3b2bd61..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.jd.platform.hotkey.common.coder; - -import io.netty.channel.ChannelHandler; - -/** - * @author wuweifeng wrote on 2019-12-11 - * @version 1.0 - */ -public interface Codec { - - ChannelHandler newEncoder(); - - ChannelHandler newDecoder(); -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java deleted file mode 100644 index 24cb0ed25000966a1dc925fdacca90f5af8d0e1a..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java +++ /dev/null @@ -1,53 +0,0 @@ -package com.jd.platform.hotkey.common.coder; - -import com.jd.platform.hotkey.common.model.HotKeyMsg; -import com.jd.platform.hotkey.common.model.typeenum.MessageType; -import com.jd.platform.hotkey.common.tool.Constant; -import io.netty.buffer.ByteBuf; -import io.netty.channel.ChannelHandlerContext; -import io.netty.handler.codec.ByteToMessageDecoder; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.nio.charset.Charset; -import java.util.List; - -/** - * @author wuweifeng wrote on 2020-01-06 - * @version 1.0 - */ -public class MessageDecoder extends ByteToMessageDecoder { - private Logger logger = LoggerFactory.getLogger(getClass()); - - @Override - protected void decode(ChannelHandlerContext ctx, ByteBuf byteBuf, List out) { - if (byteBuf.readableBytes() < 10 || byteBuf.readableBytes() > Constant.MAX_LENGTH) { - logger.warn("数据包不正确，当前包大小为：" + byteBuf.readableBytes()); - return; - } - - byteBuf.markReaderIndex(); - - HotKeyMsg message = new HotKeyMsg(); - - int magicNumber = byteBuf.readInt(); - if (Constant.MAGIC_NUMBER != magicNumber) { - logger.warn("MAGIC_NUMBER不正确:" + magicNumber); - return; - } - - message.setMagicNumber(byteBuf.readInt()); // 读取魔数 - - MessageType messageType = MessageType.get(byteBuf.readByte()); - if (messageType == null) { - logger.error("messageType is null , byteBuf readByte = " + byteBuf.readByte()); - return; - } - message.setMessageType(messageType); // 读取当前的消息类型 - - int bodyLength = byteBuf.readInt(); // 读取消息体长度和数据 - CharSequence body = byteBuf.readCharSequence(bodyLength, Charset.defaultCharset()); - message.setBody(body.toString()); - out.add(message); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java deleted file mode 100644 index acf21624c6323f2c616d132c67701a12a4b802e2..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java +++ /dev/null @@ -1,47 +0,0 @@ -package com.jd.platform.hotkey.common.coder; - -import com.jd.platform.hotkey.common.model.HotKeyMsg; -import com.jd.platform.hotkey.common.model.typeenum.MessageType; -import com.jd.platform.hotkey.common.tool.Constant; -import io.netty.buffer.ByteBuf; -import io.netty.channel.ChannelHandlerContext; -import io.netty.handler.codec.MessageToByteEncoder; - -import java.nio.charset.Charset; - -/** - * @author wuweifeng wrote on 2020-01-06 - * @version 1.0 - */ -public class MessageEncoder extends MessageToByteEncoder { - - @Override - protected void encode(ChannelHandlerContext ctx, HotKeyMsg message, ByteBuf out) { - MessageType messageType = message.getMessageType(); - //非法类型 - if (MessageType.get(messageType.getType()) == null) { - return; - } - // 这里会判断消息类型是不是EMPTY类型，如果是EMPTY类型，则表示当前消息不需要写入到管道中 - if (messageType == MessageType.EMPTY) { - return; - } - //4个byte - out.writeInt(Constant.MAGIC_NUMBER); // 写入当前的魔数 - //1个byte - out.writeByte(message.getMessageType().getType()); // 写入当前消息的类型 - - if (null == message.getBody()) { - out.writeInt(0); // 如果消息体为空，则写入0，表示消息体长度为0 - } else { - System.out.println(message.getBody().length()); - out.writeInt(message.getBody().length()); - out.writeCharSequence(message.getBody(), Charset.defaultCharset()); - } - out.writeBytes(Constant.DELIMITER.getBytes()); - } - - public static void main(String[] args) { - System.out.println(Constant.MAGIC_NUMBER + ""); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/MsgDecoder.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/MsgDecoder.java new file mode 100644 index 0000000000000000000000000000000000000000..492c5e7892e24fa85c3d5fc3e500baf894537972 --- /dev/null +++ b/common/src/main/java/com/jd/platform/hotkey/common/coder/MsgDecoder.java @@ -0,0 +1,30 @@ +package com.jd.platform.hotkey.common.coder; + +import com.jd.platform.hotkey.common.model.HotKeyMsg; +import com.jd.platform.hotkey.common.tool.ProtostuffUtils; +import io.netty.buffer.ByteBuf; +import io.netty.channel.ChannelHandlerContext; +import io.netty.handler.codec.ByteToMessageDecoder; + +import java.util.List; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-07-29 + */ +public class MsgDecoder extends ByteToMessageDecoder { + @Override + protected void decode(ChannelHandlerContext channelHandlerContext, ByteBuf in, List list) { + try { + + byte[] body = new byte[in.readableBytes()]; //传输正常 + in.readBytes(body); + + list.add(ProtostuffUtils.deserialize(body, HotKeyMsg.class)); + + } catch (Exception e) { + e.printStackTrace(); + } + } +} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/MsgEncoder.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/MsgEncoder.java new file mode 100644 index 0000000000000000000000000000000000000000..6cdeca3dd882e80ede4cb7749bdb975a09bd6060 --- /dev/null +++ b/common/src/main/java/com/jd/platform/hotkey/common/coder/MsgEncoder.java @@ -0,0 +1,30 @@ +package com.jd.platform.hotkey.common.coder; + +import com.jd.platform.hotkey.common.model.HotKeyMsg; +import com.jd.platform.hotkey.common.tool.Constant; +import com.jd.platform.hotkey.common.tool.ProtostuffUtils; +import io.netty.buffer.ByteBuf; +import io.netty.channel.ChannelHandlerContext; +import io.netty.handler.codec.MessageToByteEncoder; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-07-30 + */ +public class MsgEncoder extends MessageToByteEncoder { + + @Override + public void encode(ChannelHandlerContext ctx, Object in, ByteBuf out) { + if (in instanceof HotKeyMsg) { + byte[] bytes = ProtostuffUtils.serialize(in); + byte[] delimiter = Constant.DELIMITER.getBytes(); + + byte[] total = new byte[bytes.length + delimiter.length]; + System.arraycopy(bytes, 0, total, 0, bytes.length); + System.arraycopy(delimiter, 0, total, bytes.length, delimiter.length); + + out.writeBytes(total); + } + } +} \ No newline at end of file diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java deleted file mode 100755 index 1aeac45f5948d3595b367b811b666dcfb01870ab..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.jd.platform.hotkey.common.coder; - -import io.netty.channel.ChannelHandler; - -/** - * @author wuweifeng wrote on 2019-12-11 - * @version 1.0 - */ -public class NettyCodec implements Codec { - - @Override - public ChannelHandler newEncoder() { - return new MessageEncoder(); - } - - @Override - public ChannelHandler newDecoder() { - return new MessageDecoder(); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java index e88389d8c5bef4cab53b5fe815ed57d029e516f7..009a1075e5b0eb51b10ef6ff3550f6883939c464 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java @@ -13,6 +13,10 @@ public interface ConfigConstant { * 所有的workers，存这里 */ String workersPath = "/jd/workers/"; + /** + * dashboard的ip存这里 + */ + String dashboardPath = "/jd/dashboard/"; /** * 该app所有的workers地址的path。需要手工分配，默认每个app都用所有的worker */ @@ -54,4 +58,24 @@ public interface ConfigConstant { * 存放客户端hotKey访问次数和总访问次数的path */ String keyHitCountPath = "/jd/keyHitCount/"; + /** + * 是否开启日志 + */ + String logToggle = "/jd/logOn"; + + /** + * 清理历史数据的配置的path + * time unit : day + */ + String clearCfgPath = "/jd/clearCfg/"; + + /** + * app配置 + */ + String appCfgPath = "/jd/appCfg/"; + + /** + * 控制台启动的netty端口 + */ + int dashboardPort = 11112; } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java index 06e85e957794fa9b0559ac81836480bdbd997f56..fb64d9e28b9a3217e1824f93b762f147427fd205 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java @@ -3,7 +3,6 @@ package com.jd.platform.hotkey.common.configcenter.etcd; import cn.hutool.core.collection.CollectionUtil; import com.google.protobuf.ByteString; import com.ibm.etcd.api.KeyValue; -import com.ibm.etcd.api.Kv; import com.ibm.etcd.api.LeaseGrantResponse; import com.ibm.etcd.api.RangeResponse; import com.ibm.etcd.client.KvStoreClient; diff --git a/common/src/main/java/com/jd/platform/hotkey/common/convert/LongAdderDelegate.java b/common/src/main/java/com/jd/platform/hotkey/common/convert/LongAdderDelegate.java new file mode 100644 index 0000000000000000000000000000000000000000..7e0d9eb579af4a3761c5df865e00c9cb4cae9a36 --- /dev/null +++ b/common/src/main/java/com/jd/platform/hotkey/common/convert/LongAdderDelegate.java @@ -0,0 +1,45 @@ +package com.jd.platform.hotkey.common.convert; + +import io.protostuff.Input; +import io.protostuff.Output; +import io.protostuff.Pipe; +import io.protostuff.WireFormat; +import io.protostuff.runtime.Delegate; + +import java.io.IOException; +import java.util.concurrent.atomic.LongAdder; + +/** + * @author qiujw wrote on 2021-06-29 + * @version 1.0 + */ +public class LongAdderDelegate implements Delegate { + + @Override + public WireFormat.FieldType getFieldType() { + return WireFormat.FieldType.INT64; + } + + @Override + public java.util.concurrent.atomic.LongAdder readFrom(Input input) throws IOException { + java.util.concurrent.atomic.LongAdder cnt = new java.util.concurrent.atomic.LongAdder(); + cnt.add(input.readInt64()); + return cnt; + } + + @Override + public void writeTo(Output output, int number, LongAdder longAdder, boolean repeated) throws IOException { + output.writeInt64(number, longAdder.sum(), repeated); + } + + @Override + public void transfer(Pipe pipe, Input input, Output output, int number, + boolean repeated) throws IOException { + output.writeInt64(number, input.readInt64(), repeated); + } + + @Override + public Class typeClass() { + return java.util.concurrent.atomic.LongAdder.class; + } +} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/convert/LongAdderSerializer.java b/common/src/main/java/com/jd/platform/hotkey/common/convert/LongAdderSerializer.java new file mode 100644 index 0000000000000000000000000000000000000000..c22685af5e902c8cbb4eb71c8b54644206da8b64 --- /dev/null +++ b/common/src/main/java/com/jd/platform/hotkey/common/convert/LongAdderSerializer.java @@ -0,0 +1,28 @@ +package com.jd.platform.hotkey.common.convert; + +import com.alibaba.fastjson.serializer.JSONSerializer; +import com.alibaba.fastjson.serializer.ObjectSerializer; +import com.alibaba.fastjson.serializer.SerializeWriter; + +import java.io.IOException; +import java.lang.reflect.Type; +import java.util.concurrent.atomic.LongAdder; + +/** + * @author qiujw wrote on 2021-06-29 + * @version 1.0 + */ +public class LongAdderSerializer implements ObjectSerializer { + + @Override + public void write(JSONSerializer serializer, Object object, Object fieldName, Type fieldType, int features) throws IOException { + SerializeWriter out = serializer.out; + if (object instanceof Long) { + out.writeFieldValue('{', "value", ((Long) object)); + out.write('}'); + }else if (object instanceof LongAdder) { + out.writeFieldValue('{', "value", ((LongAdder) object).longValue()); + out.write('}'); + } + } +} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/BaseModel.java b/common/src/main/java/com/jd/platform/hotkey/common/model/BaseModel.java index de3f62ee166600c3b868e379a9911377a62b68a6..44f6ccb553381f81fa4f4a9af805ccf250a02be2 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/BaseModel.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/BaseModel.java @@ -1,7 +1,11 @@ package com.jd.platform.hotkey.common.model; +import com.alibaba.fastjson.annotation.JSONField; +import com.jd.platform.hotkey.common.convert.LongAdderSerializer; import com.jd.platform.hotkey.common.tool.IdGenerater; +import java.util.concurrent.atomic.LongAdder; + /** * 热key的定义 * @author wuweifeng wrote on 2019-12-05 @@ -19,8 +23,10 @@ public class BaseModel { private String key; /** * 该key出现的数量，如果一次一发那就是1，累积多次发那就是count + * 使用 LongAdder 解决 多线程计数不准确的问题 */ - private int count; + @JSONField(serializeUsing = LongAdderSerializer.class) + private LongAdder count; @Override public String toString() { @@ -32,14 +38,30 @@ public class BaseModel { '}'; } - public int getCount() { - return count; + /** + * 获取计数总数 + * @return 总数 + */ + public long getCount() { + return count.sum(); } - public void setCount(int count) { + /** + * 设置计数 + * @param count 计数 LongAdder 对象 + */ + public void setCount(LongAdder count) { this.count = count; } + /** + * 计数自增指定数量 + * @param count 指定数量 + */ + public void add(long count){ + this.count.add(count); + } + public String getId() { return id; } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java index 0451fce64a1dc414d41a410de91ebd1be7038e9e..11de0ccaf76915b7dd59f58f6d4120850912d459 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java @@ -23,11 +23,7 @@ public class HotKeyModel extends BaseModel { @Override public String toString() { - return "HotKeyModel{" + - "appName='" + appName + '\'' + - ", keyType=" + keyType + - ", remove=" + remove + - '}'; + return "appName:" + appName + "-key=" + getKey(); } public boolean isRemove() { @@ -53,4 +49,5 @@ public class HotKeyModel extends BaseModel { public void setKeyType(KeyType keyType) { this.keyType = keyType; } + } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyMsg.java b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyMsg.java index a1caf1a8c9688cbc9fe71aa4b9eab812a4137313..f8ee4e0b34eaea6413730e0671f94d88a45b61ac 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyMsg.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyMsg.java @@ -2,8 +2,11 @@ package com.jd.platform.hotkey.common.model; import com.jd.platform.hotkey.common.model.typeenum.MessageType; +import java.util.List; + /** * netty通信消息 + * * @author wuweifeng wrote on 2020-01-06 * @version 1.0 */ @@ -16,14 +19,17 @@ public class HotKeyMsg { private String body; - public HotKeyMsg(MessageType messageType, String body) { - this(null, messageType, body); + private List hotKeyModels; + + private List keyCountModels; + + public HotKeyMsg(MessageType messageType) { + this(messageType, null); } - public HotKeyMsg(String appName, MessageType messageType, String body) { + public HotKeyMsg(MessageType messageType, String appName) { this.appName = appName; this.messageType = messageType; - this.body = body; } public HotKeyMsg() { @@ -33,11 +39,30 @@ public class HotKeyMsg { public String toString() { return "HotKeyMsg{" + "magicNumber=" + magicNumber + + ", appName='" + appName + '\'' + ", messageType=" + messageType + ", body='" + body + '\'' + + ", hotKeyModels=" + hotKeyModels + + ", keyCountModels=" + keyCountModels + '}'; } + public List getHotKeyModels() { + return hotKeyModels; + } + + public void setHotKeyModels(List hotKeyModels) { + this.hotKeyModels = hotKeyModels; + } + + public List getKeyCountModels() { + return keyCountModels; + } + + public void setKeyCountModels(List keyCountModels) { + this.keyCountModels = keyCountModels; + } + public String getAppName() { return appName; } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java b/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java index cd3f98393d6cd6407d02afb85d7fc50b8f84cc99..5b07624630cae927ac724a738977f67c41110d78 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java @@ -8,7 +8,8 @@ public enum MessageType { APP_NAME((byte) 1), REQUEST_NEW_KEY((byte) 2), RESPONSE_NEW_KEY((byte) 3), - REQUEST_HIT_COUNT((byte) 7), + REQUEST_HIT_COUNT((byte) 7), //命中率 + REQUEST_HOT_KEY((byte) 8), //热key，worker->dashboard PING((byte) 4), PONG((byte) 5), EMPTY((byte) 6); diff --git a/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java b/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java index 5c747540ad3452fa161d7d7d591c98f99611e29c..1d3ed6ba4de181e79dcc93c48e52a79c7e4b6002 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java @@ -30,7 +30,7 @@ public class Constant { */ public static String DEFAULT_DELETE_VALUE = "#[DELETE]#"; - //单次包最大2M - public static int MAX_LENGTH = 2 * 1024 * 1024; + //单次包最大4M + public static int MAX_LENGTH = 4 * 1024 * 1024; } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/tool/ProtostuffUtils.java b/common/src/main/java/com/jd/platform/hotkey/common/tool/ProtostuffUtils.java new file mode 100644 index 0000000000000000000000000000000000000000..0ddbd639ad8e8a7e752ba8185d086c525f21c5ba --- /dev/null +++ b/common/src/main/java/com/jd/platform/hotkey/common/tool/ProtostuffUtils.java @@ -0,0 +1,84 @@ +package com.jd.platform.hotkey.common.tool; + +import com.jd.platform.hotkey.common.convert.LongAdderDelegate; +import io.protostuff.*; +import io.protostuff.runtime.DefaultIdStrategy; +import io.protostuff.runtime.RuntimeEnv; +import io.protostuff.runtime.RuntimeSchema; + +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.ConcurrentHashMap; + +/** + * + * @author wuweifeng10 + * @date 2020/7/18 + **/ +public class ProtostuffUtils { + + private final static DefaultIdStrategy idStrategy = ((DefaultIdStrategy) RuntimeEnv.ID_STRATEGY); + + /** + * 避免每次序列化都重新申请Buffer空间 + * 这句话在实际生产上没有意义，耗时减少的极小，但高并发下，如果还用这个buffer，会报异常说buffer还没清空，就又被使用了 + */ +// private static LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE); + /** + * 缓存Schema + */ + private static Map, Schema> schemaCache = new ConcurrentHashMap<>(); + + static { + idStrategy.registerDelegate(new LongAdderDelegate()); + } + + /** + * 序列化方法，把指定对象序列化成字节数组 + * + * @param obj + * @param + * @return + */ + @SuppressWarnings("unchecked") + public static byte[] serialize(T obj) { + Class clazz = (Class) obj.getClass(); + Schema schema = getSchema(clazz); + LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE); + byte[] data; + try { + data = ProtobufIOUtil.toByteArray(obj, schema, buffer); +// data = ProtostuffIOUtil.toByteArray(obj, schema, buffer); + } finally { + buffer.clear(); + } + + return data; + } + + /** + * 反序列化方法，将字节数组反序列化成指定Class类型 + */ + public static T deserialize(byte[] data, Class clazz) { + Schema schema = getSchema(clazz); + T obj = schema.newMessage(); + ProtobufIOUtil.mergeFrom(data, obj, schema); +// ProtostuffIOUtil.mergeFrom(data, obj, schema); + return obj; + } + + @SuppressWarnings("unchecked") + private static Schema getSchema(Class clazz) { + Schema schema = (Schema) schemaCache.get(clazz); + if (Objects.isNull(schema)) { + //这个schema通过RuntimeSchema进行懒创建并缓存 + //所以可以一直调用RuntimeSchema.getSchema(),这个方法是线程安全的 + schema = RuntimeSchema.getSchema(clazz, idStrategy); + if (Objects.nonNull(schema)) { + schemaCache.put(clazz, schema); + } + } + + return schema; + } +} \ No newline at end of file diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java b/common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java similarity index 95% rename from worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java rename to common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java index cdc1299c2f33bf3ef3b6944271ae8434184a35c2..54a7e6d94675ff056c632719e43988c2fe00c873 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.worker.netty.flush; +package com.jd.platform.hotkey.common.tool.flush; import io.netty.buffer.ByteBuf; import io.netty.channel.ChannelHandlerContext; @@ -31,4 +31,5 @@ public class FlushUtil { } } } + } diff --git a/dashboard/pom.xml b/dashboard/pom.xml index 30eacfa44eebc69b66ed08db05272acf602260ec..9675a2f742a6f2e1766776ab6b6a00539a914eba 100644 --- a/dashboard/pom.xml +++ b/dashboard/pom.xml @@ -57,7 +57,7 @@ com.jd.platform.hotkey common - 0.0.2-SNAPSHOT + 0.0.4-SNAPSHOT compile @@ -81,11 +81,6 @@ poi-ooxml 4.0.1 - - com.jd.common - sso-uim-spring - 1.2.0-SNAPSHOT - diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java index 2b361c66ba6119558b49e80da745a2b5d80fb2d7..e1431c9f8af5502860afcdf8aaf57db1ce1b12ae 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java @@ -1,35 +1,16 @@ package com.jd.platform.hotkey.dashboard; -import com.alibaba.fastjson.JSON; -import com.jd.platform.hotkey.common.configcenter.ConfigConstant; -import com.jd.platform.hotkey.common.configcenter.IConfigCenter; -import com.jd.platform.hotkey.common.tool.IpUtils; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; -import com.jd.platform.hotkey.dashboard.mapper.RulesMapper; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.boot.CommandLineRunner; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.scheduling.annotation.EnableAsync; import org.springframework.scheduling.annotation.EnableScheduling; -import javax.annotation.Resource; -import java.util.HashMap; -import java.util.Map; - @EnableAsync @EnableScheduling @SpringBootApplication -public class DashboardApplication implements CommandLineRunner { - - - private Logger logger = LoggerFactory.getLogger(getClass()); - - @Resource - private KeyTimelyMapper timelyMapper; +public class DashboardApplication{ public static void main(String[] args) { try { @@ -37,13 +18,6 @@ public class DashboardApplication implements CommandLineRunner { }catch (Exception e){ e.printStackTrace(); } - - } - - @Override - public void run(String... args) { - int row = timelyMapper.clear(); - logger.info("clear db timely hotKey, effect row : {}",row); } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/AppCfgController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/AppCfgController.java new file mode 100644 index 0000000000000000000000000000000000000000..10fc82425030ee7c5f50cecb0790431024912bab --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/AppCfgController.java @@ -0,0 +1,58 @@ +package com.jd.platform.hotkey.dashboard.biz.controller; + +import com.jd.platform.hotkey.dashboard.common.base.BaseController; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; +import com.jd.platform.hotkey.dashboard.biz.service.AppCfgService; +import org.springframework.stereotype.Controller; +import org.springframework.ui.ModelMap; +import org.springframework.web.bind.annotation.*; + +import javax.annotation.Resource; + + +@Controller +@RequestMapping("/appCfg") +public class AppCfgController extends BaseController { + + + private String prefix = "admin/appcfg"; + + + @Resource + private AppCfgService appCfgService; + + @GetMapping("/view") + public String view(ModelMap modelMap){ + modelMap.put("title", Constant.APP_CFG_VIEW); + return prefix + "/list"; + } + + + @PostMapping("/list") + @ResponseBody + public Page list(PageReq page, String app){ + return appCfgService.pageAppCfgVo(page, app); + } + + + @GetMapping("/edit/{app}") + public String edit(@PathVariable("app") String app, ModelMap modelMap){ + modelMap.put("appCfg", appCfgService.selectAppCfgVo(app)); + return prefix + "/edit"; + } + + @PostMapping("/save") + @ResponseBody + public Result save(AppCfgVo cfg){ + cfg.setModifier(userName()); + appCfgService.saveAppCfgVo(cfg); + return Result.success(); + } + + +} + diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ChangLogController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ChangLogController.java similarity index 90% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ChangLogController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ChangLogController.java index e9a1788f647c237fe29db9224359473c98ca85cf..1e18dc2bd7a4dfb2254c17e94db1ea513da8b5c1 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ChangLogController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ChangLogController.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.controller; +package com.jd.platform.hotkey.dashboard.biz.controller; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.*; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.model.ChangeLog; -import com.jd.platform.hotkey.dashboard.service.ChangeLogService; +import com.jd.platform.hotkey.dashboard.biz.service.ChangeLogService; import org.springframework.stereotype.Controller; import org.springframework.ui.ModelMap; import org.springframework.web.bind.annotation.*; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ClearController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ClearController.java new file mode 100644 index 0000000000000000000000000000000000000000..e6ce6b48943e3533f01c154e1de0de0035145296 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ClearController.java @@ -0,0 +1,62 @@ +package com.jd.platform.hotkey.dashboard.biz.controller; + +import com.github.pagehelper.PageInfo; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; +import com.jd.platform.hotkey.dashboard.biz.service.ClearService; +import org.springframework.stereotype.Controller; +import org.springframework.ui.ModelMap; +import org.springframework.web.bind.annotation.*; + +import javax.annotation.Resource; + +/** + * @ProjectName: hotkey + * @ClassName: ClearController + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:48 + */ +@Controller +@RequestMapping("/clear") +public class ClearController { + + + private String prefix = "admin/clear"; + + + @Resource + private ClearService clearService; + + @GetMapping("/view") + public String view(ModelMap modelMap){ + modelMap.put("title", Constant.CLEAR_VIEW); + return prefix + "/list"; + } + + + @PostMapping("/list") + @ResponseBody + public Page list(PageReq page, String app){ + PageInfo info = clearService.pageClearCfg(page, app); + return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); + } + + + @GetMapping("/edit/{app}") + public String edit(@PathVariable("app") String app, ModelMap modelMap){ + modelMap.put("clearCfg", clearService.selectClearCfg(app)); + return prefix + "/edit"; + } + + @PostMapping("/save") + @ResponseBody + public Result save(ClearCfgVo cfg){ + int b = clearService.saveClearCfg(cfg); + return b == 0 ? Result.fail():Result.success(); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/KeyController.java similarity index 77% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/KeyController.java index 0b287a83d09533e685d4149a7172e394b5149d38..767387f62bc599517d72dc28e389177d68c24e91 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/KeyController.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.controller; +package com.jd.platform.hotkey.dashboard.biz.controller; import com.github.pagehelper.PageInfo; import com.github.pagehelper.util.StringUtil; @@ -14,7 +14,7 @@ import com.jd.platform.hotkey.dashboard.common.domain.vo.HotKeyLineChartVo; import com.jd.platform.hotkey.dashboard.model.KeyRecord; import com.jd.platform.hotkey.dashboard.model.KeyTimely; import com.jd.platform.hotkey.dashboard.model.Statistics; -import com.jd.platform.hotkey.dashboard.service.KeyService; +import com.jd.platform.hotkey.dashboard.biz.service.KeyService; import com.jd.platform.hotkey.dashboard.util.DateUtil; import com.jd.platform.hotkey.dashboard.util.ExcelUtil; import org.springframework.stereotype.Controller; @@ -34,7 +34,7 @@ import java.util.List; @Controller @RequestMapping("/key") public class KeyController extends BaseController { - + private String prefix = "admin/key"; @Resource @@ -79,8 +79,7 @@ public class KeyController extends BaseController { @PostMapping("/listTimely") @ResponseBody public Page listTimely(PageReq page, SearchReq searchReq){ - PageInfo info = keyService.pageKeyTimely(page, searchReq); - return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); + return keyService.pageKeyTimely(page, searchReq); } @@ -123,13 +122,6 @@ public class KeyController extends BaseController { } - @GetMapping("/edit/{id}") - public String edit(@PathVariable("id") Long id, ModelMap modelMap){ - modelMap.put("key", keyService.selectByPk(id)); - return prefix + "/edit"; - } - - @PostMapping("/edit") @ResponseBody public Result editSave(KeyTimely key) { @@ -158,6 +150,28 @@ public class KeyController extends BaseController { } + @RequestMapping(value = "/exportList", method = RequestMethod.GET) + @ResponseBody + public void exportList(HttpServletResponse resp,String startTime,String endTime,String app,String key){ + SearchReq req = new SearchReq(); + if(StringUtil.isNotEmpty(startTime)){ + req.setStartTime(DateUtil.strToDate(startTime)); + } + if(StringUtil.isNotEmpty(endTime)){ + req.setEndTime(DateUtil.strToDate(endTime)); + } + req.setApp(app); + req.setKey(key); + List records = keyService.listKeyRecord(req); + if(records.size() > ExcelUtil.MAX_ROW){ + records = records.subList(0,ExcelUtil.MAX_ROW); + } + List> rows = transformList(records); + ExcelDataDto data = new ExcelDataDto("keyRecord.xlsx", Constant.RECORD_HEAD,rows); + ExcelUtil.exportExcel(resp,data); + } + + private List> transform(List records){ List> rows = new ArrayList<>(); @@ -170,5 +184,17 @@ public class KeyController extends BaseController { } return rows; } + + private List> transformList(List records){ + List> rows = new ArrayList<>(); + for (KeyRecord record : records) { + List list = new ArrayList<>(); + list.add(record.getKey()); + list.add(record.getAppName()); + list.add(DateUtil.dateToStr(record.getCreateTime())); + rows.add(list); + } + return rows; + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/RuleController.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/RuleController.java index e1869c9fb3d75e981d28c0b561cc30e05f509b67..ce8600d017063ce93da023ddfc6287ab6923f9ed 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/RuleController.java @@ -1,9 +1,8 @@ -package com.jd.platform.hotkey.dashboard.controller; +package com.jd.platform.hotkey.dashboard.biz.controller; import cn.hutool.json.JSONArray; import cn.hutool.json.JSONObject; import cn.hutool.json.JSONTokener; -import com.alibaba.fastjson.JSON; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.Constant; @@ -14,9 +13,8 @@ import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; -import com.jd.platform.hotkey.dashboard.model.Rule; import com.jd.platform.hotkey.dashboard.model.Rules; -import com.jd.platform.hotkey.dashboard.service.RuleService; +import com.jd.platform.hotkey.dashboard.biz.service.RuleService; import org.springframework.stereotype.Controller; import org.springframework.ui.ModelMap; import org.springframework.web.bind.annotation.*; @@ -31,7 +29,7 @@ import java.util.List; @Controller @RequestMapping("/rule") public class RuleController extends BaseController { - + @Resource private RuleService ruleService; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/UserController.java similarity index 96% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/UserController.java index bc571ba53252bac31d25651c221387a27ddf0e94..8711eb48e200261f5f60bfd9b7bfb37ac4926696 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/UserController.java @@ -1,17 +1,15 @@ -package com.jd.platform.hotkey.dashboard.controller; - -import java.util.*; +package com.jd.platform.hotkey.dashboard.biz.controller; import com.github.pagehelper.PageInfo; import com.github.pagehelper.util.StringUtil; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.Page; -import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.model.User; -import com.jd.platform.hotkey.dashboard.service.UserService; import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import io.jsonwebtoken.Claims; @@ -23,6 +21,8 @@ import javax.annotation.Resource; import javax.servlet.http.Cookie; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; +import java.util.HashMap; +import java.util.Map; @Controller diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/WorkerController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/WorkerController.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/WorkerController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/WorkerController.java index 4e9471ea855ee90872a3a9b0b228581031665b8f..aedd57cf2edc0826a1114b6a77ebcaf1c66c4de5 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/WorkerController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/WorkerController.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.controller; +package com.jd.platform.hotkey.dashboard.biz.controller; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.*; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.model.Worker; -import com.jd.platform.hotkey.dashboard.service.WorkerService; +import com.jd.platform.hotkey.dashboard.biz.service.WorkerService; import org.springframework.stereotype.Controller; import org.springframework.ui.ModelMap; import org.springframework.web.bind.annotation.*; @@ -16,7 +16,7 @@ import javax.annotation.Resource; @Controller @RequestMapping("/worker") public class WorkerController extends BaseController { - + private String prefix = "admin/worker"; @Resource @@ -63,7 +63,7 @@ public class WorkerController extends BaseController { modelMap.put("worker", workerService.selectByKey(key.replace("_","/"))); return prefix + "/edit"; } - + @PostMapping("/edit") @ResponseBody diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.java similarity index 87% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.java index c36f87178df5b364ef9f0a21f0af5db1d6c13901..a1ceb2002f7bfce4405f24a86fa62cc17c6d2376 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.ChangeLog; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/KeyRecordMapper.java similarity index 78% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/KeyRecordMapper.java index 3ef46b31a58494eabc0daadf386cc759c1053c7b..e8714aa5c3973b6b020f87135df3f44d82d7052e 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/KeyRecordMapper.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; -import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.KeyRecord; import com.jd.platform.hotkey.dashboard.model.Statistics; import org.apache.ibatis.annotations.Mapper; +import java.util.Date; import java.util.List; /** @@ -25,4 +25,9 @@ public interface KeyRecordMapper { List maxHotKey(SearchReq req); List statisticsByRule(SearchReq req); + + int clearExpireData(String app, Date expireDate); + + int countKeyRecord(SearchReq req); + } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.java similarity index 83% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.java index 1c6835c005554ea218f09c723f1176e71e85ab1f..3c1c894bf824b6a13a393b2b1faa299831c0455c 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.model.Rules; import org.apache.ibatis.annotations.Mapper; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.java similarity index 85% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.java index dabdfc584a4c4379ac1e31ba9de79704748c5b9b..a657342fbd71d592a6fbe4a11e12a6aa2869a2bf 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.java @@ -1,9 +1,10 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.Statistics; import org.apache.ibatis.annotations.Mapper; +import java.util.Date; import java.util.List; /** @@ -42,7 +43,6 @@ public interface StatisticsMapper { /** * 清理 - * @param type type */ - void clear(int type); + int clearExpireData(String app, Date expireDate); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.java similarity index 79% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.java index 4685e76b4760726c153941015c2cfb274aaff8a6..99882f74f552c5f3991b9ed1ef51490238c09339 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; -import com.jd.platform.hotkey.dashboard.model.Statistics; import com.jd.platform.hotkey.dashboard.model.Summary; import org.apache.ibatis.annotations.Mapper; +import java.util.Date; import java.util.List; /** @@ -22,4 +22,6 @@ public interface SummaryMapper { int saveOrUpdate(Summary records); List listRuleHitCount(SearchReq req); + + int clearExpireData(String app, Date expireDate); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.java similarity index 85% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.java index c8d72ee2c2e06a13df2ba81d712eee9b8fb1cd40..30f0be8d918fe5ee29d0aa70ca90d958d67a5a06 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.User; @@ -25,4 +25,6 @@ public interface UserMapper { List selectHkUserList(User user); User selectByUserName(String userName); + + List listErpByApp(String app); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.java similarity index 91% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.java index 501316b563788ef2f7183134fb93a66247df2c9f..29befc2469071ac3f4936273fdf83f7f0152baf0 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.Worker; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/AppCfgService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/AppCfgService.java new file mode 100644 index 0000000000000000000000000000000000000000..fa814b8838405662ebda8c970de17e0b25782f3c --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/AppCfgService.java @@ -0,0 +1,21 @@ +package com.jd.platform.hotkey.dashboard.biz.service; + +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; + +/** + * @ProjectName: hotkey + * @ClassName: ClearService + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:51 + */ +public interface AppCfgService { + + Page pageAppCfgVo(PageReq page, String app); + + AppCfgVo selectAppCfgVo(String app); + + void saveAppCfgVo(AppCfgVo cfg); + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ChangeLogService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ChangeLogService.java similarity index 91% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ChangeLogService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ChangeLogService.java index 498f4e4d87743cfe567cba63c5a23872e747ceb3..cfed2493c00b8c824ecb1ef2b19933ea81cecc85 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ChangeLogService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ChangeLogService.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ClearService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ClearService.java new file mode 100644 index 0000000000000000000000000000000000000000..383fa9cb53f71bbeb0397ae0dddbbceff1448875 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ClearService.java @@ -0,0 +1,23 @@ +package com.jd.platform.hotkey.dashboard.biz.service; + +import com.github.pagehelper.PageInfo; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; + +/** + * @ProjectName: hotkey + * @ClassName: ClearService + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:51 + */ +public interface ClearService { + + + PageInfo pageClearCfg(PageReq page, String app); + + ClearCfgVo selectClearCfg(String app); + + int saveClearCfg(ClearCfgVo cfg); + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/KeyService.java similarity index 83% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/KeyService.java index 3dfcf1892068913c4404faed50bf1d62031b101c..23127970d9310ecadcc44e1d4606c3536ce13453 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/KeyService.java @@ -1,6 +1,7 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; +import com.jd.platform.hotkey.dashboard.common.domain.Page; import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; @@ -21,6 +22,8 @@ import java.util.List; public interface KeyService { + List listKeyRecord(SearchReq param); + PageInfo pageKeyRecord(PageReq page, SearchReq param); int insertKeyByUser(KeyTimely keyTimely); @@ -29,11 +32,7 @@ public interface KeyService { int delKeyByUser(KeyTimely keyTimely); - KeyTimely selectByKey(String key); - - KeyTimely selectByPk(Long key); - - PageInfo pageKeyTimely(PageReq page, SearchReq param); + Page pageKeyTimely(PageReq page, SearchReq param); PageInfo pageMaxHot(PageReq page, SearchReq param); diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/RuleService.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/RuleService.java index 5580df1b67ec483f8a65351d61f1f0d0b69a5edb..1a55b995504aaece992bada0989c6eb03c98ba2f 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/RuleService.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/UserService.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/UserService.java index 71f57ea705fefcbe37dfdd62b940d6bb303ae83b..b4bb8687fefae6a52dfe927484df3daf564d77ca 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/UserService.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/WorkerService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/WorkerService.java similarity index 93% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/WorkerService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/WorkerService.java index 4995172fded57cb93d723aa153f928243d2903ef..398dae8de872cc504a81eb2aaba0a4b34a7ae143 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/WorkerService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/WorkerService.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/AppCfgServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/AppCfgServiceImpl.java new file mode 100644 index 0000000000000000000000000000000000000000..8f59b91534b3ec642bb8338e0d9f6bb63d603cfe --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/AppCfgServiceImpl.java @@ -0,0 +1,121 @@ +package com.jd.platform.hotkey.dashboard.biz.service.impl; + +import com.alibaba.fastjson.JSON; +import com.github.pagehelper.util.StringUtil; +import com.google.protobuf.ByteString; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.dashboard.biz.mapper.UserMapper; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; +import com.jd.platform.hotkey.dashboard.biz.service.AppCfgService; +import com.jd.platform.hotkey.dashboard.util.PageUtil; +import org.springframework.stereotype.Service; +import org.springframework.util.StringUtils; + +import javax.annotation.Resource; +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +/** + * @ProjectName: hotkey + * @ClassName: AppCfgServiceImpl + * @Author: liyunfeng31 + * @Date: 2020/9/2 9:57 + */ +@Service +public class AppCfgServiceImpl implements AppCfgService { + + + @Resource + private IConfigCenter configCenter; + + @Resource + private UserMapper userMapper; + + + @Override + public Page pageAppCfgVo(PageReq page, String app) { + + List apps = userMapper.listApp(); + + List keyValues = getKVList(apps); + + List cfgVos = new ArrayList<>(); + for (KeyValue kv : keyValues) { + String key = kv.getKey().toStringUtf8(); + String k = key.replace(ConfigConstant.appCfgPath, ""); + //如果某个配置所属的app已经被删了，则删除该app的配置 + if (!apps.contains(k)) { + configCenter.delete(key); + continue; + } + //取到配置信息 + String v = kv.getValue().toStringUtf8(); + //如果为空，则赋值初始化 + if (StringUtil.isEmpty(v)) { + v = JSON.toJSONString(new AppCfgVo(k)); + configCenter.put(key, v); + } + + AppCfgVo vo = JSON.parseObject(v, AppCfgVo.class); + vo.setVersion(kv.getModRevision()); + //如果是管理员，则添加所有 + if (StringUtils.isEmpty(app)) { + cfgVos.add(vo); + } else { + //添加自己的app信息 + if (k.equals(app)) { + cfgVos.add(vo); + } + } + } + return PageUtil.pagination(cfgVos, page.getPageSize(), page.getPageNum() - 1); + } + + @Override + public AppCfgVo selectAppCfgVo(String app) { + KeyValue kv = configCenter.getKv(ConfigConstant.appCfgPath + app); + if (kv == null || kv.getValue() == null) { + AppCfgVo ap = new AppCfgVo(app); + configCenter.put(ConfigConstant.appCfgPath + app, JSON.toJSONString(ap)); + return ap; + } + String v = kv.getValue().toStringUtf8(); + AppCfgVo cfg = JSON.parseObject(v, AppCfgVo.class); + cfg.setVersion(kv.getModRevision()); + return cfg; + } + + /** + * todo 多节点问题 待完善 + */ + @Override + public void saveAppCfgVo(AppCfgVo cfg) { + configCenter.put(ConfigConstant.appCfgPath + cfg.getApp(), JSON.toJSONString(cfg)); + } + + + /** + * 比较和补充 + * @param apps all db apps + */ + private List getKVList(List apps){ + + List keyValues = new ArrayList<>(configCenter.getPrefix(ConfigConstant.appCfgPath)); + List etcdApps = keyValues.stream() + .map(x -> x.getKey().toStringUtf8() + .replace(ConfigConstant.appCfgPath, "")) + .collect(Collectors.toList()); + List tempList = new ArrayList<>(apps); + tempList.removeAll(etcdApps); + for (String diffApp : tempList) { + KeyValue kv = KeyValue.newBuilder().setKey(ByteString.copyFromUtf8(ConfigConstant.appCfgPath + diffApp)).build(); + keyValues.add(kv); + } + return keyValues; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ChangeLogServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ChangeLogServiceImpl.java similarity index 84% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ChangeLogServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ChangeLogServiceImpl.java index 50e11afd895c5630218c7a30a3170de80eff43d5..5fa90a5911b79045122559a36507cfd4d600f13e 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ChangeLogServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ChangeLogServiceImpl.java @@ -1,12 +1,12 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import com.github.pagehelper.PageHelper; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.ChangeLogMapper; import com.jd.platform.hotkey.dashboard.model.ChangeLog; -import com.jd.platform.hotkey.dashboard.service.ChangeLogService; +import com.jd.platform.hotkey.dashboard.biz.service.ChangeLogService; import org.springframework.stereotype.Service; import javax.annotation.Resource; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ClearServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ClearServiceImpl.java new file mode 100644 index 0000000000000000000000000000000000000000..462e5c8f801575760172e4ea61864b4d72f83be7 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ClearServiceImpl.java @@ -0,0 +1,85 @@ +package com.jd.platform.hotkey.dashboard.biz.service.impl; + +import com.github.pagehelper.PageInfo; +import com.github.pagehelper.util.StringUtil; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; +import com.jd.platform.hotkey.dashboard.biz.service.ClearService; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; +import org.apache.commons.collections4.CollectionUtils; +import org.springframework.stereotype.Service; +import org.springframework.util.StringUtils; +import javax.annotation.Resource; +import java.util.ArrayList; +import java.util.List; + +/** + * @ProjectName: hotkey + * @ClassName: ClearServiceImpl + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:57 + */ +@Service +public class ClearServiceImpl implements ClearService { + + + @Resource + private IConfigCenter configCenter; + + @Resource + private UserService userService; + + + @Override + public PageInfo pageClearCfg(PageReq page, String app) { + List keyValues = configCenter.getPrefix(ConfigConstant.clearCfgPath); + if(CollectionUtils.isEmpty(keyValues)){ + List apps = userService.listApp(); + for (String ap : apps) { + configCenter.put(ConfigConstant.clearCfgPath + ap, Constant.THIRTY_DAY); + } + keyValues = configCenter.getPrefix(ConfigConstant.clearCfgPath); + } + List cfgVos = new ArrayList<>(); + for (KeyValue kv : keyValues) { + String v = kv.getValue().toStringUtf8(); + String key = kv.getKey().toStringUtf8(); + if(StringUtil.isEmpty(v)){ + configCenter.put(key, Constant.THIRTY_DAY); + continue; + } + long version = kv.getModRevision(); + String k = key.replace(ConfigConstant.clearCfgPath,""); + if(StringUtils.isEmpty(app)){ + cfgVos.add(new ClearCfgVo(k, v, version)); + }else{ + if(k.equals(app)){ + cfgVos.add(new ClearCfgVo(k, v, version)); + } + } + } + return new PageInfo<>(cfgVos); + } + + @Override + public ClearCfgVo selectClearCfg(String app) { + KeyValue kv = configCenter.getKv(ConfigConstant.clearCfgPath + app); + if(kv == null || kv.getValue() == null){ + configCenter.put(ConfigConstant.clearCfgPath + app, Constant.THIRTY_DAY); + return new ClearCfgVo(app, Constant.THIRTY_DAY, 0L); + } + String v = kv.getValue().toStringUtf8(); + return new ClearCfgVo(app,v,kv.getModRevision()); + } + + @Override + public int saveClearCfg(ClearCfgVo cfg) { + configCenter.put(ConfigConstant.clearCfgPath + cfg.getApp(), cfg.getTtl()); + return 1; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/KeyServiceImpl.java similarity index 44% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/KeyServiceImpl.java index 88a4263a3090e97301543dcd8d049fdf155996c7..d8a0555c80eb39d774b19973d0969aecdef83da6 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/KeyServiceImpl.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import cn.hutool.core.date.SystemClock; @@ -8,24 +8,29 @@ import com.github.pagehelper.PageInfo; import com.ibm.etcd.api.Event; import com.jd.platform.hotkey.common.configcenter.ConfigConstant; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.Page; import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HotKeyLineChartVo; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; -import com.jd.platform.hotkey.dashboard.mapper.KeyRecordMapper; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; -import com.jd.platform.hotkey.dashboard.mapper.StatisticsMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.ChangeLogMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.KeyRecordMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.StatisticsMapper; import com.jd.platform.hotkey.dashboard.model.ChangeLog; import com.jd.platform.hotkey.dashboard.model.KeyRecord; import com.jd.platform.hotkey.dashboard.model.KeyTimely; import com.jd.platform.hotkey.dashboard.model.Statistics; -import com.jd.platform.hotkey.dashboard.service.KeyService; -import com.jd.platform.hotkey.dashboard.service.RuleService; +import com.jd.platform.hotkey.dashboard.netty.HotKeyReceiver; +import com.jd.platform.hotkey.dashboard.biz.service.KeyService; +import com.jd.platform.hotkey.dashboard.biz.service.RuleService; import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.DateUtil; +import com.jd.platform.hotkey.dashboard.util.PageUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.springframework.stereotype.Service; import javax.annotation.Resource; @@ -50,16 +55,17 @@ public class KeyServiceImpl implements KeyService { @Resource private KeyRecordMapper recordMapper; @Resource - private KeyTimelyMapper keyTimelyMapper; - @Resource private StatisticsMapper statisticsMapper; @Resource private RuleService ruleService; @Resource private ChangeLogMapper logMapper; + private Logger logger = LoggerFactory.getLogger(getClass()); + /** * 折线图 + * * @param req req * @return vo */ @@ -68,52 +74,48 @@ public class KeyServiceImpl implements KeyService { int type = req.getType(); String appReq = req.getApp(); // admin 全查 - if(StrUtil.isNotEmpty(appReq)){ + if (StrUtil.isNotEmpty(appReq)) { app = appReq; } req.setApp(null); LocalDateTime now = LocalDateTime.now(); req.setEndTime(req.getEndTime() == null ? DateUtil.ldtToDate(now) : req.getEndTime()); List rules = ruleService.listRules(null); - if(type == 4){ - LocalDateTime st = req.getStartTime() == null? now.minusMinutes(31) : DateUtil.dateToLdt(req.getStartTime()); + if (type == 4) { + LocalDateTime st = req.getStartTime() == null ? now.minusMinutes(31) : DateUtil.dateToLdt(req.getStartTime()); req.setStartTime(DateUtil.ldtToDate(st)); LocalDateTime et = DateUtil.dateToLdt(req.getEndTime()); - boolean longTime = Duration.between(st,et).toHours()>2; + boolean longTime = Duration.between(st, et).toHours() > 2; req.setType(longTime ? 6 : 5); List list = statisticsMapper.listOrderByTime(req); - return CommonUtil.processData(st,et,list,!longTime,rules,app); + return CommonUtil.processData(st, et, list, !longTime, rules, app); } - if(type == 5){ + if (type == 5) { LocalDateTime startTime = now.minusMinutes(31); req.setStartTime(DateUtil.ldtToDate(startTime)); List list = statisticsMapper.listOrderByTime(req); - return CommonUtil.processData(startTime,now,list,true,rules,app); - }else if(type == 6){ + return CommonUtil.processData(startTime, now, list, true, rules, app); + } else if (type == 6) { LocalDateTime startTime2 = now.minusHours(25); req.setStartTime(DateUtil.ldtToDate(startTime2)); List list2 = statisticsMapper.listOrderByTime(req); - return CommonUtil.processData(startTime2,now,list2,false,rules,app); - }else{ + return CommonUtil.processData(startTime2, now, list2, false, rules, app); + } else { LocalDateTime startTime3 = now.minusDays(7).minusHours(1); req.setStartTime(DateUtil.ldtToDate(startTime3)); req.setType(6); List list3 = statisticsMapper.listOrderByTime(req); - return CommonUtil.processData(startTime3,now,list3,false,rules,app); + return CommonUtil.processData(startTime3, now, list3, false, rules, app); } } @Override - public PageInfo pageKeyTimely(PageReq page, SearchReq param) { - PageHelper.startPage(page.getPageNum(), page.getPageSize()); - List listKey = keyTimelyMapper.listKeyTimely(param); - for (KeyTimely timely : listKey) { - timely.setKey(CommonUtil.keyName(timely.getKey())); - timely.setRuleDesc(RuleUtil.ruleDesc(timely.getAppName() + "/" + timely.getKey())); - } - return new PageInfo<>(listKey); + public Page pageKeyTimely(PageReq page, SearchReq param) { + List keyTimelies = HotKeyReceiver.list(param); + return PageUtil.pagination(keyTimelies, page.getPageSize(), page.getPageNum()-1); + } @Override @@ -143,15 +145,20 @@ public class KeyServiceImpl implements KeyService { Map keyDateMap = keyDateMap(statistics, hours); // 获取时间x轴 List list = new ArrayList<>(); - for (int i = hours; i >0 ; i--) { - LocalDateTime time = LocalDateTime.now().minusHours(i-1); + for (int i = hours; i > 0; i--) { + LocalDateTime time = LocalDateTime.now().minusHours(i - 1); int hour = time.getHour(); - list.add(hour+"时"); + list.add(hour + "时"); } - return new HotKeyLineChartVo(list,keyDateMap); + return new HotKeyLineChartVo(list, keyDateMap); } + @Override + public List listKeyRecord(SearchReq param) { + return recordMapper.listKeyRecord(param); + } + @Override public PageInfo pageKeyRecord(PageReq page, SearchReq param) { PageHelper.startPage(page.getPageNum(), page.getPageSize()); @@ -168,6 +175,13 @@ public class KeyServiceImpl implements KeyService { public int insertKeyByUser(KeyTimely key) { configCenter.putAndGrant(ConfigConstant.hotKeyPath + key.getAppName() + "/" + key.getKey(), System.currentTimeMillis() + "", key.getDuration()); + + //写入本地缓存，实时热key信息 + HotKeyModel hotKeyModel = new HotKeyModel(); + hotKeyModel.setCreateTime(System.currentTimeMillis()); + hotKeyModel.setAppName(key.getAppName()); + hotKeyModel.setKey(key.getKey()); + HotKeyReceiver.writeToLocalCaffeine(hotKeyModel); return logMapper.insertSelective(new ChangeLog(key.getAppName(), Constant.HOTKEY_CHANGE, "", key.getKey(), key.getUpdater(), SystemClock.now() + "")); } @@ -185,56 +199,46 @@ public class KeyServiceImpl implements KeyService { String[] arr = keyTimely.getKey().split("/"); //删除client监听目录的key String etcdKey = ConfigConstant.hotKeyPath + arr[0] + "/" + arr[1]; + + //删除caffeine里的实时key + HotKeyReceiver.delete(arr[0] + "/" + arr[1]); + if (configCenter.get(etcdKey) == null) { //如果手工目录也就是client监听的目录里没有该key，那么就往里面放一个，然后再删掉它，这样client才能监听到删除事件 configCenter.putAndGrant(etcdKey, com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE, 1); } configCenter.delete(etcdKey); - //也删除Record目录下的该key，因为不确定要删的key到底在哪 - String recordKey = ConfigConstant.hotKeyRecordPath + arr[0] + "/" + arr[1]; - configCenter.delete(recordKey); - - KeyRecord keyRecord = new KeyRecord(arr[1], "", arr[0], 0L, Constant.HAND, + KeyRecord keyRecord = new KeyRecord(arr[1], "", arr[0], 0, Constant.HAND, Event.EventType.DELETE_VALUE, UUID.randomUUID().toString(), new Date()); recordMapper.insertSelective(keyRecord); - return logMapper.insertSelective(new ChangeLog(arr[0], Constant.HOTKEY_CHANGE, arr[1],"", keyTimely.getUpdater(), SystemClock.now() + "")); + return logMapper.insertSelective(new ChangeLog(keyTimely.getKey(), Constant.HOTKEY_CHANGE, keyTimely.getKey(), "", keyTimely.getUpdater(), SystemClock.now() + "")); } - @Override - public KeyTimely selectByKey(String key) { - return keyTimelyMapper.selectByKey(key); - } - @Override - public KeyTimely selectByPk(Long id) { - return keyTimelyMapper.selectByPrimaryKey(id); - } - - - - - private Map keyDateMap(List statistics, int hours){ + private Map keyDateMap(List statistics, int hours) { Map map = new HashMap<>(10); Map> listMap = statistics.stream().collect(Collectors.groupingBy(Statistics::getKeyName)); for (Map.Entry> m : listMap.entrySet()) { int start = DateUtil.preHoursInt(5); - map.put(m.getKey(),new int[hours]); + map.put(m.getKey(), new int[hours]); int[] data = map.get(m.getKey()); int tmp = 0; for (int i = 0; i < hours; i++) { Statistics st; try { st = m.getValue().get(tmp); - if(String.valueOf(start).endsWith("24")){ start = start + 77; } - if(start != st.getHours()){ + if (String.valueOf(start).endsWith("24")) { + start = start + 77; + } + if (start != st.getHours()) { data[i] = 0; - }else{ - tmp ++; + } else { + tmp++; data[i] = st.getCount(); } start++; - }catch (Exception e){ + } catch (Exception e) { data[i] = 0; } } @@ -243,9 +247,8 @@ public class KeyServiceImpl implements KeyService { } - private void checkParam(SearchReq req) { - if(req.getStartTime() == null || req.getEndTime() == null){ + if (req.getStartTime() == null || req.getEndTime() == null) { req.setStartTime(DateUtil.preTime(5)); req.setEndTime(new Date()); } @@ -255,173 +258,6 @@ public class KeyServiceImpl implements KeyService { }*/ } - - private static List statisticsList(){ - Random rd = new Random(); - List records = new ArrayList<>(); - // statisticsMapper.clear(1); - for (int i = 0; i < 35; i++) { - if(i== 2||i== 3||i== 4||i== 25||i== 12||i== 13||i== 14||i== 30||i==32||i==33||i== 34){ - continue; - } - // 每分钟小时 统计一次record 表 结果记录到统计表 - LocalDateTime now = LocalDateTime.now().minusMinutes(35-i); - Date nowTime = DateUtil.ldtToDate(now); - int day = DateUtil.nowDay(now); - int hour = DateUtil.nowHour(now); - int minus = DateUtil.nowMinus(now); - - Statistics s1 = new Statistics(); - s1.setKeyName("k21"); - s1.setApp("key21APP"); - s1.setCreateTime(nowTime); - s1.setDays(day); - s1.setHours(hour); - s1.setCount(rd.nextInt(800)); - s1.setBizType(1); - s1.setMinutes(minus); - s1.setUuid(1 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s1); - - Statistics s11 = new Statistics(); - s11.setKeyName("k22"); - s11.setApp("key22APP"); - s11.setBizType(1); - s11.setCreateTime(nowTime); - s11.setDays(day); - s11.setMinutes(minus); - s11.setCount(rd.nextInt(600)); - s11.setHours(hour); - s11.setUuid(2 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s11); - } - - if(1==1){ - return records; - } - // statisticsMapper.batchInsert(records); - List list = new ArrayList<>(); - for (int i = 0; i < 30 ; i++) { - if(i == 12 || i ==13){ - }else{ - Statistics st = new Statistics(); - st.setApp("k21"); - st.setKeyName("k21"); - st.setCount(rd.nextInt(100)); - st.setBizType(1); - // LocalDateTime ldf = DateUtil.strToLdt("2006082355"); - // System.out.println(ldf.toString()+" - "+DateUtil.strToLdt("2006082355")); - int time = DateUtil.reviseTime(DateUtil.strToLdt("2006082355",DateUtil.PATTERN_MINUS), i, 1); - // System.out.println(time); - st.setMinutes(time); - // list.add(st); - } - } - List list2 = new ArrayList<>(); - for (int i = 0; i < 30 ; i++) { - if(i == 0 || i == 1 || i == 18 || i == 19 || i == 25){ - - }else{ - Statistics st2 = new Statistics(); - st2.setApp("k22"); - st2.setKeyName("k22"); - st2.setCount(rd.nextInt(100)); - st2.setBizType(1); - st2.setMinutes(DateUtil.reviseTime(DateUtil.strToLdt("2006082355", DateUtil.PATTERN_MINUS),i,1)); - list2.add(st2); - } - } - // list.addAll(list2); - return list; - - } - - private List statisticsList1(int type){ - Random rd = new Random(); - List records = new ArrayList<>(); - statisticsMapper.clear(6); - - if(type == 3){ - int temp = 1; - for (int j = 0; j < 9; j++) { - for (int i = 0; i < 24; i++) { - // 每分钟小时 统计一次record 表 结果记录到统计表 - LocalDateTime now = LocalDateTime.now().minusHours(9 * 24 - temp); - - Date nowTime = DateUtil.ldtToDate(now); - int day = DateUtil.nowDay(now); - int hour = DateUtil.nowHour(now); - Statistics s1 = new Statistics(); - s1.setKeyName("k21"); - s1.setApp("key21APP"); - s1.setCreateTime(nowTime); - s1.setDays(day); - s1.setHours(hour); - s1.setCount(rd.nextInt(200)); - s1.setBizType(6); - s1.setUuid(1 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s1); - - Statistics s11 = new Statistics(); - s11.setKeyName("k22"); - s11.setApp("key22APP"); - s11.setBizType(6); - s11.setCreateTime(nowTime); - s11.setDays(day); - s11.setCount(rd.nextInt(500)); - s11.setHours(hour); - s11.setUuid(2 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s11); - - Statistics s33 = new Statistics(); - s33.setKeyName("k33"); - s33.setApp("key33APP"); - s33.setBizType(6); - s33.setCreateTime(nowTime); - s33.setDays(day); - s33.setCount(rd.nextInt(300)); - s33.setHours(hour); - s33.setUuid(3 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s33); - temp++; - } - } - - - }else{ - for (int i = 0; i < 30; i++) { - // 每分钟小时 统计一次record 表 结果记录到统计表 - LocalDateTime now = LocalDateTime.now().minusHours(30-i); - Date nowTime = DateUtil.ldtToDate(now); - int day = DateUtil.nowDay(now); - int hour = DateUtil.nowHour(now); - Statistics s1 = new Statistics(); - s1.setKeyName("key21"); - s1.setApp("key21APP"); - s1.setCreateTime(nowTime); - s1.setDays(day); - s1.setHours(hour); - s1.setCount(rd.nextInt(800)); - s1.setBizType(2); - s1.setUuid(1 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s1); - - Statistics s11 = new Statistics(); - s11.setKeyName("key22"); - s11.setApp("key22APP"); - s11.setBizType(2); - s11.setCreateTime(nowTime); - s11.setDays(day); - s11.setCount(rd.nextInt(600)); - s11.setHours(hour); - s11.setUuid(2 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s11); - } - } - statisticsMapper.batchInsert(records); - return records; - } - } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/RuleServiceImpl.java similarity index 73% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/RuleServiceImpl.java index 9dd493e79c3fce3177b9565f5a78f70e18ada0ef..23b225c19c4eb663f028c5d99242c6b4530674c3 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/RuleServiceImpl.java @@ -1,5 +1,6 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; +import cn.hutool.core.date.SystemClock; import cn.hutool.core.lang.UUID; import com.alibaba.fastjson.JSON; import com.github.pagehelper.PageHelper; @@ -8,14 +9,16 @@ import com.github.pagehelper.util.StringUtil; import com.ibm.etcd.api.KeyValue; import com.jd.platform.hotkey.common.configcenter.ConfigConstant; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.dashboard.biz.mapper.ChangeLogMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.RulesMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.SummaryMapper; +import com.jd.platform.hotkey.dashboard.biz.service.RuleService; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; -import com.jd.platform.hotkey.dashboard.mapper.RulesMapper; -import com.jd.platform.hotkey.dashboard.mapper.SummaryMapper; -import com.jd.platform.hotkey.dashboard.model.*; -import com.jd.platform.hotkey.dashboard.service.RuleService; +import com.jd.platform.hotkey.dashboard.model.ChangeLog; +import com.jd.platform.hotkey.dashboard.model.Rule; +import com.jd.platform.hotkey.dashboard.model.Rules; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import org.springframework.util.StringUtils; @@ -52,13 +55,13 @@ public class RuleServiceImpl implements RuleService { @Override public Rules selectRules(String app) { KeyValue kv = configCenter.getKv(ConfigConstant.rulePath + app); - if(kv == null || kv.getValue() == null){ + if (kv == null || kv.getValue() == null) { return new Rules(); } String k = kv.getKey().toStringUtf8(); String v = kv.getValue().toStringUtf8(); - List rule = JSON.parseArray(v,Rule.class); - return new Rules(app,v); + List rule = JSON.parseArray(v, Rule.class); + return new Rules(app, v); } @Transactional(rollbackFor = Exception.class) @@ -67,7 +70,7 @@ public class RuleServiceImpl implements RuleService { String app = rules.getApp(); Rules oldRules = rulesMapper.select(app); String from = JSON.toJSONString(oldRules); - configCenter.put(ConfigConstant.rulePath+app, rules.getRules()); + configCenter.put(ConfigConstant.rulePath + app, rules.getRules()); String to = JSON.toJSONString(rules); logMapper.insertSelective(new ChangeLog(app, 1, from, to, rules.getUpdateUser(), app, UUID.fastUUID().toString(true))); @@ -78,7 +81,7 @@ public class RuleServiceImpl implements RuleService { @Override public Integer add(Rules rules) { String app = rules.getApp(); - configCenter.put(ConfigConstant.rulePath+app,rules.getRules()); + configCenter.put(ConfigConstant.rulePath + app, rules.getRules()); String to = JSON.toJSONString(rules); logMapper.insertSelective(new ChangeLog(app, 1, "", to, rules.getUpdateUser(), app, UUID.fastUUID().toString(true))); @@ -103,15 +106,15 @@ public class RuleServiceImpl implements RuleService { List rules = new ArrayList<>(); for (KeyValue kv : keyValues) { String v = kv.getValue().toStringUtf8(); - if(StringUtil.isEmpty(v)){ + if (StringUtil.isEmpty(v)) { continue; } String key = kv.getKey().toStringUtf8(); - String k = key.replace(ConfigConstant.rulePath,""); - if(StringUtils.isEmpty(appName)){ + String k = key.replace(ConfigConstant.rulePath, ""); + if (StringUtils.isEmpty(appName)) { rules.add(new Rules(k, v)); - }else{ - if(k.equals(appName)){ + } else { + if (k.equals(appName)) { rules.add(new Rules(k, v)); } } @@ -122,9 +125,16 @@ public class RuleServiceImpl implements RuleService { @Override public int save(Rules rules) { String app = rules.getApp(); - String from = ""; + + KeyValue kv = configCenter.getKv(ConfigConstant.rulePath + app); + String from = null; + if (kv != null) { + from = kv.getValue().toStringUtf8(); + } String to = JSON.toJSONString(rules); configCenter.put(ConfigConstant.rulePath + app, rules.getRules()); + + logMapper.insertSelective(new ChangeLog(app, 1, from, to, rules.getUpdateUser(), app, SystemClock.nowDate())); return 1; } @@ -134,14 +144,14 @@ public class RuleServiceImpl implements RuleService { List rules = new ArrayList<>(); for (KeyValue kv : keyValues) { String v = kv.getValue().toStringUtf8(); - if(StringUtil.isEmpty(v)){ + if (StringUtil.isEmpty(v)) { continue; } String key = kv.getKey().toStringUtf8(); - String appKey = key.replace(ConfigConstant.rulePath,""); + String appKey = key.replace(ConfigConstant.rulePath, ""); List rs = JSON.parseArray(v, Rule.class); for (Rule r : rs) { - rules.add(appKey+"-"+r.getKey()); + rules.add(appKey + "-" + r.getKey()); } } return rules; @@ -149,11 +159,10 @@ public class RuleServiceImpl implements RuleService { @Override public PageInfo pageRuleHitCount(PageReq pageReq, SearchReq req, String ownApp) { - PageHelper.startPage(pageReq.getPageNum(),pageReq.getPageSize()); + PageHelper.startPage(pageReq.getPageNum(), pageReq.getPageSize()); List hitCountVos = summaryMapper.listRuleHitCount(req); - return new PageInfo<>(hitCountVos); + return new PageInfo<>(hitCountVos); } - } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/UserServiceImpl.java similarity index 95% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/UserServiceImpl.java index 5cb83e5d0c0a983ba0a78bc40fdf273c15300c2b..f5da4a8a7218f5c4b17afd85154c9586003a9a25 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/UserServiceImpl.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import com.github.pagehelper.PageHelper; import com.github.pagehelper.PageInfo; @@ -8,9 +8,9 @@ import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; -import com.jd.platform.hotkey.dashboard.mapper.UserMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.UserMapper; import com.jd.platform.hotkey.dashboard.model.User; -import com.jd.platform.hotkey.dashboard.service.UserService; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Service; @@ -34,7 +34,7 @@ import java.util.Set; @Service public class UserServiceImpl implements UserService { - @Value("${erp.defaultPwd}") + @Value("${erp.defaultPwd:123}") private String defaultPwd; @Resource private UserMapper userMapper; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/WorkerServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/WorkerServiceImpl.java similarity index 95% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/WorkerServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/WorkerServiceImpl.java index 4c133ee5732e2e41299decdf4924081784a07498..d6270a1a3ccf66eb00a4da3cff9d6ec31ba1fdf9 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/WorkerServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/WorkerServiceImpl.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import cn.hutool.core.date.SystemClock; import com.alibaba.fastjson.JSON; @@ -10,11 +10,11 @@ import com.jd.platform.hotkey.common.tool.FastJsonUtils; import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; -import com.jd.platform.hotkey.dashboard.mapper.WorkerMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.ChangeLogMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.WorkerMapper; import com.jd.platform.hotkey.dashboard.model.ChangeLog; import com.jd.platform.hotkey.dashboard.model.Worker; -import com.jd.platform.hotkey.dashboard.service.WorkerService; +import com.jd.platform.hotkey.dashboard.biz.service.WorkerService; import org.springframework.dao.DuplicateKeyException; import org.springframework.stereotype.Service; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java new file mode 100644 index 0000000000000000000000000000000000000000..63612e8d8c12c59307c216d598e32ced13d02c49 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java @@ -0,0 +1,30 @@ +package com.jd.platform.hotkey.dashboard.cache; + +import com.github.benmanes.caffeine.cache.Cache; +import com.github.benmanes.caffeine.cache.Caffeine; + +import java.util.concurrent.TimeUnit; + +/** + * @author wuweifeng wrote on 2019-12-12 + * @version 1.0 + */ +public class CaffeineBuilder { + + public static Cache cache(int duration) { + return cache(128, 1000000, duration); + } + + + /** + * 构建所有来的要缓存的key getCache + */ + public static Cache cache(int minSize, int maxSize, int expireSeconds) { + return Caffeine.newBuilder() + .initialCapacity(minSize)//初始大小 + .maximumSize(maxSize)//最大数量 + .expireAfterWrite(expireSeconds, TimeUnit.SECONDS)//过期时间 + .build(); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java index 7aa346fa58d47c702cff13531c7caea4af3e24d9..e4065ece95d63cee5566b71428aeb4600471b2ec 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java @@ -8,8 +8,7 @@ import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; -import com.jd.platform.hotkey.dashboard.model.User; -import com.jd.platform.hotkey.dashboard.service.UserService; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import io.jsonwebtoken.Claims; import org.springframework.beans.propertyeditors.CustomDateEditor; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java index 963892110d934ec2163949947443fd9271f29bd5..a0524722bacde21d7a04a049ded819969d298e08 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java @@ -25,7 +25,7 @@ public class MyExceptionHandler { @ResponseBody public Result bizExceptionHandler(BizException e, HttpServletResponse resp){ resp.setStatus(e.getCode()); - logger.info("业务异常：",e); + logger.error("业务异常：",e); return Result.error(e.getCode(),e.getMsg()); } @@ -33,8 +33,9 @@ public class MyExceptionHandler { @ExceptionHandler(value =Exception.class) @ResponseBody - public Result exceptionHandler(Exception e){ - logger.info("未知异常：",e); + public Result exceptionHandler(Exception e,HttpServletResponse resp){ + logger.error("未知异常：",e); + resp.setStatus(500); return Result.error(ResultEnum.BIZ_ERROR); } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java index b11fa73687d71cc9a6a13b968b09885622ef0cf8..6847429927bb00b8552f33ff4ab51b2a276c0475 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java @@ -35,6 +35,10 @@ public class Constant { public static final String WORKER_VIEW = "节点信息"; + public static final String CLEAR_VIEW = "数据清理"; + + public static final String APP_CFG_VIEW = "应用配置"; + public static final int MAX_DAY_RANGE = 3; public static final int VERSION = 1; @@ -55,8 +59,21 @@ public class Constant { public static final String INFO = "info"; + public static final String THIRTY_DAY = "30"; + + public static final List HEAD = new ArrayList<>(); static { HEAD.add("热点key"); HEAD.add("次数"); HEAD.add("所属APP");} + public static final List RECORD_HEAD = new ArrayList<>(); + + static { RECORD_HEAD.add("热点key"); RECORD_HEAD.add("所属APP"); RECORD_HEAD.add(" 时间");} + + + public static final int WARN_INIT_MIN = -1; + + + public static final int WARN_INIT_MAX = 5000; + } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java deleted file mode 100644 index 04d4a8325c0eac341df8e743c7d6b5098c951544..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java +++ /dev/null @@ -1,95 +0,0 @@ -package com.jd.platform.hotkey.dashboard.common.domain; - -import com.ibm.etcd.api.Event; - -import java.io.Serializable; -import java.util.Date; - -/** - * 包装类，用于保存事件最准确的时间 - */ -public class EventWrapper implements Serializable { - - private String key; - - private String value; - - private Event.EventType eventType; - - private Date date; - - private Long ttl; - - private long version; - - private String uuid; - - @Override - public String toString() { - return "EventWrapper{" + - "key='" + key + '\'' + - ", value='" + value + '\'' + - ", eventType=" + eventType + - ", date=" + date + - ", ttl=" + ttl + - ", version=" + version + - ", uuid='" + uuid + '\'' + - '}'; - } - - public String getUuid() { - return uuid; - } - - public void setUuid(String uuid) { - this.uuid = uuid; - } - - public long getVersion() { - return version; - } - - public void setVersion(long version) { - this.version = version; - } - - public String getKey() { - return key; - } - - public void setKey(String key) { - this.key = key; - } - - public String getValue() { - return value; - } - - public void setValue(String value) { - this.value = value; - } - - public Event.EventType getEventType() { - return eventType; - } - - public void setEventType(Event.EventType eventType) { - this.eventType = eventType; - } - - public void setDate(Date date) { - this.date = date; - } - - public void setTtl(Long ttl) { - this.ttl = ttl; - } - - public Date getDate() { - return date; - } - - public Long getTtl() { - return ttl; - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java new file mode 100644 index 0000000000000000000000000000000000000000..3585f73c25e12beb3ad6622ca515ad7aac53ef10 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java @@ -0,0 +1,27 @@ +package com.jd.platform.hotkey.dashboard.common.domain; + +import java.util.Date; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-09-02 + */ +public interface IRecord { + /** + * appName + "/" + key + */ + String appNameKey(); + + /** + * 手工添加的是时间戳13位，worker传过来的是uuid + */ + String value(); + + /** + * 0插入，1删除 + */ + int type(); + + Date createTime(); +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java index 5f64c7575f762d7585ac458c3c3cc9b75ee8b94f..d10c3a5169fb88777149563c4a2cb1ace803c823 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java @@ -7,9 +7,13 @@ import java.util.List; public class Page implements Serializable { private static final long serialVersionUID = 1L; - + /** + * 当前第几页 + */ private Integer page; - + /** + * 总共多少条 + */ private int total; private List rows; @@ -20,6 +24,7 @@ public class Page implements Serializable { this.rows = rows; } + public Integer getPage() { return page; } @@ -43,4 +48,13 @@ public class Page implements Serializable { public void setRows(List rows) { this.rows = rows; } + + @Override + public String toString() { + return "Page{" + + "page=" + page + + ", total=" + total + + ", rows=" + rows + + '}'; + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/PushMsgWrapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/PushMsgWrapper.java new file mode 100644 index 0000000000000000000000000000000000000000..9b6b78872301e11b81c7d6e7471183125c2f979d --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/PushMsgWrapper.java @@ -0,0 +1,62 @@ +package com.jd.platform.hotkey.dashboard.common.domain; + +import cn.hutool.core.date.SystemClock; + +import java.io.Serializable; + +/** + * 报警消息包装类，用于保存事件最准确的时间 + */ +public class PushMsgWrapper implements Serializable { + + private String app; + + private Long date; + + /** + * -1小于最小阈值; 1超过最大阈值 + */ + private Integer count; + + private String msg; + + public PushMsgWrapper(String app, int count) { + this.app = app; + this.count = count; + this.date = SystemClock.now(); + } + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public Integer getCount() { + return count; + } + + public void setCount(Integer count) { + this.count = count; + } + + public Long getDate() { + return date; + } + + public void setDate(Long date) { + this.date = date; + } + + public String getMsg() { + return msg; + } + + public void setMsg(String msg) { + this.msg = msg; + } +} + + diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/AppCfgVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/AppCfgVo.java new file mode 100644 index 0000000000000000000000000000000000000000..4809303516bf7318c6af1a10989c3317e6a1a53e --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/AppCfgVo.java @@ -0,0 +1,131 @@ +package com.jd.platform.hotkey.dashboard.common.domain.vo; + + +import com.jd.platform.hotkey.dashboard.common.domain.Constant; + +import java.io.Serializable; + +/** + * @ProjectName: hotkey + * @ClassName: AppCfgVo + * @Author: liyunfeng31 + * @Date: 2020/9/2 10:29 + */ +public class AppCfgVo implements Serializable { + + private String app; + + /** + * 数据保存时长 + */ + private Integer dataTtl; + + /** + * 警报周期 + */ + private Integer warnPeriod; + + /** + * 警报阈值最小值 + */ + private Integer warnMin; + + /** + * 警报阈值最大值 + */ + private Integer warnMax; + + /** + * 警报开关 1开启 0关闭 + */ + private Integer warn; + + /** + * 版本 + */ + private Long version; + + /* + * 最后修改人 + */ + private String modifier; + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public Integer getDataTtl() { + return dataTtl; + } + + public void setDataTtl(Integer dataTtl) { + this.dataTtl = dataTtl; + } + + public Integer getWarnPeriod() { + return warnPeriod; + } + + public void setWarnPeriod(Integer warnPeriod) { + this.warnPeriod = warnPeriod; + } + + public Integer getWarnMin() { + return warnMin; + } + + public void setWarnMin(Integer warnMin) { + this.warnMin = warnMin; + } + + public Integer getWarnMax() { + return warnMax; + } + + public void setWarnMax(Integer warnMax) { + this.warnMax = warnMax; + } + + public Integer getWarn() { + return warn; + } + + public void setWarn(Integer warn) { + this.warn = warn; + } + + public Long getVersion() { + return version; + } + + public void setVersion(Long version) { + this.version = version; + } + + public String getModifier() { + return modifier; + } + + public void setModifier(String modifier) { + this.modifier = modifier; + } + + public AppCfgVo() { + } + + public AppCfgVo(String app) { + this.app = app; + this.dataTtl = 30; + this.warnPeriod = 60; + this.warnMin = Constant.WARN_INIT_MIN; + this.warnMax = Constant.WARN_INIT_MAX; + this.version = 0L; + this.warn = 1; + this.modifier = "SYSTEM"; + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java new file mode 100644 index 0000000000000000000000000000000000000000..65df755e0262be8a4772369de663ff3e843b16f9 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java @@ -0,0 +1,49 @@ +package com.jd.platform.hotkey.dashboard.common.domain.vo; + +import java.io.Serializable; + +/** + * @ProjectName: hotkey + * @ClassName: ClearCfgVo + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:54 + */ +public class ClearCfgVo implements Serializable { + + private String app; + + private String ttl; + + private Long version; + + public ClearCfgVo(String app, String ttl, Long version) { + this.app = app; + this.ttl = ttl; + this.version = version; + } + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public String getTtl() { + return ttl; + } + + public void setTtl(String ttl) { + this.ttl = ttl; + } + + public Long getVersion() { + return version; + } + + public void setVersion(Long version) { + this.version = version; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java index 3392fc2494c96339b84052b0935dc1fcb7ef5102..71171a6089c8d18d114c849fc0aecb2195c7be3e 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java @@ -1,5 +1,6 @@ package com.jd.platform.hotkey.dashboard.common.domain.vo; +import java.io.Serializable; import java.util.List; import java.util.Map; import java.util.Set; @@ -7,7 +8,7 @@ import java.util.Set; /** * @author liyunfeng31 */ -public class HotKeyLineChartVo { +public class HotKeyLineChartVo implements Serializable { private List xAxis; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java index 273ce59998d5c60dc2f4c77fae0ef1749d8dc535..3179c5ff818b12cc90f7119aff589a3d635c6dde 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java @@ -1,5 +1,6 @@ package com.jd.platform.hotkey.dashboard.common.domain.vo; +import java.io.Serializable; import java.util.List; import java.util.Map; import java.util.Set; @@ -7,7 +8,7 @@ import java.util.Set; /** * @author liyunfeng31 */ -public class LineChartVo { +public class LineChartVo implements Serializable { private Set legend; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/RuleLineChartVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/RuleLineChartVo.java index 31ba5305e879aa21bdaa2c339fb639a9f8360584..8f6fbc50509ed4015ba8b281cbbce222161121ed 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/RuleLineChartVo.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/RuleLineChartVo.java @@ -1,9 +1,11 @@ package com.jd.platform.hotkey.dashboard.common.domain.vo; +import java.io.Serializable; + /** * @author liyunfeng31 */ -public class RuleLineChartVo { +public class RuleLineChartVo implements Serializable { private String rule; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java index b5b1a1ee09f5439000c2af7727ff82cf67100ea7..00bc8ee9bfe618b3c2dec383784f2017078ab505 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java @@ -1,31 +1,35 @@ package com.jd.platform.hotkey.dashboard.common.monitor; +import cn.hutool.core.collection.CollectionUtil; import com.alibaba.fastjson.JSON; -import com.ibm.etcd.api.Event; +import com.google.common.collect.Queues; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.dashboard.biz.mapper.KeyRecordMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.StatisticsMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.SummaryMapper; import com.jd.platform.hotkey.dashboard.common.domain.Constant; -import com.jd.platform.hotkey.dashboard.common.domain.EventWrapper; +import com.jd.platform.hotkey.dashboard.common.domain.IRecord; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.mapper.KeyRecordMapper; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; -import com.jd.platform.hotkey.dashboard.mapper.StatisticsMapper; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; import com.jd.platform.hotkey.dashboard.model.KeyRecord; -import com.jd.platform.hotkey.dashboard.model.KeyTimely; import com.jd.platform.hotkey.dashboard.model.Statistics; +import com.jd.platform.hotkey.dashboard.netty.HotKeyReceiver; import com.jd.platform.hotkey.dashboard.util.DateUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; -import com.jd.platform.hotkey.dashboard.util.TwoTuple; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; +import javax.annotation.PostConstruct; import javax.annotation.Resource; import java.time.LocalDateTime; -import java.util.Date; -import java.util.List; -import java.util.concurrent.BlockingQueue; -import java.util.concurrent.LinkedBlockingQueue; +import java.util.*; +import java.util.concurrent.*; @Component public class DataHandler { @@ -34,22 +38,30 @@ public class DataHandler { @Resource private KeyRecordMapper keyRecordMapper; - @Resource - private KeyTimelyMapper keyTimelyMapper; + @Resource private StatisticsMapper statisticsMapper; + @Resource + private SummaryMapper summaryMapper; + + @Resource + private IConfigCenter configCenter; + + private static final Integer CACHE_SIZE = 10000; + + /** * 队列 */ - private BlockingQueue queue = new LinkedBlockingQueue<>(); + private BlockingQueue queue = new LinkedBlockingQueue<>(); /** * 入队 */ - public void offer(EventWrapper eventWrapper) { + public void offer(IRecord record) { try { - queue.put(eventWrapper); + queue.put(record); } catch (InterruptedException e) { e.printStackTrace(); } @@ -57,59 +69,100 @@ public class DataHandler { public void insertRecords() { while (true) { - TwoTuple twoTuple; try { - twoTuple = handHotKey(queue.take()); - if (twoTuple == null) { + List records = new ArrayList<>(); + Queues.drain(queue, records, 1000, 1, TimeUnit.SECONDS); + if (CollectionUtil.isEmpty(records)) { + continue; + } + List keyRecordList = new ArrayList<>(records.size()); + for (IRecord iRecord : records) { + KeyRecord keyRecord = handHotKey(iRecord); + if (keyRecord != null) { + keyRecordList.add(keyRecord); + } + } + if(CollectionUtil.isEmpty(keyRecordList)){ continue; } + keyRecordMapper.batchInsert(keyRecordList); + } catch (Exception e) { - e.printStackTrace(); - log.error("handHotKey error ," + e.getCause()); - continue; + log.error("batch insert error:{}", e.getMessage(), e); +// e.printStackTrace(); } - KeyRecord keyRecord = twoTuple.getSecond(); - KeyTimely keyTimely = twoTuple.getFirst(); - - if (keyTimely.getUuid() == null) { - keyTimelyMapper.deleteByKeyAndApp(keyTimely.getKey(), keyTimely.getAppName()); - } else { - try { - keyTimelyMapper.saveOrUpdate(keyTimely); - } catch (Exception e) { - log.info("insert timely error",e); + } + + } + + /** + * 读取netty发来的热key，进行初步处理，并写入本地caffeine + */ + public void dealHotKey() { + while (true) { + try { + HotKeyModel model = HotKeyReceiver.take(); + //将该key放入实时热key本地缓存中 + if (model != null) { + //将key放到队列里，供入库时分批调用 + putRecord(model.getAppName(), model.getKey(), model.getCreateTime()); + //获取发来的这个热key，存入本地caffeine，设置过期时间 + HotKeyReceiver.writeToLocalCaffeine(model); } + } catch (Exception e) { + e.printStackTrace(); } + } + } - if (keyRecord != null) { - //插入记录 - keyRecordMapper.insertSelective(keyRecord); - } + /** + * 将待入库的热key放到队列 + */ + private void putRecord(String app, String key, long createTime) { + try { + queue.put(new IRecord() { + @Override + public String appNameKey() { + return app + "/" + key; + } - } + @Override + public String value() { + return UUID.randomUUID().toString(); + } + @Override + public int type() { + return 0; + } + + @Override + public Date createTime() { + return new Date(createTime); + } + }); + } catch (InterruptedException e) { + e.printStackTrace(); + } } /** * 处理热点key和记录 */ - private TwoTuple handHotKey(EventWrapper eventWrapper) { - Date date = eventWrapper.getDate(); - long ttl = eventWrapper.getTtl(); - Event.EventType eventType = eventWrapper.getEventType(); - String appKey = eventWrapper.getKey(); - String value = eventWrapper.getValue(); + private KeyRecord handHotKey(IRecord record) { + Date date = record.createTime(); + String appKey = record.appNameKey(); + String value = record.value(); //appName+"/"+"key" String[] arr = appKey.split("/"); String appName = arr[0]; String key = arr[1]; - String uuid = eventWrapper.getUuid(); - int type = eventType.getNumber(); + String uuid = UUID.randomUUID().toString(); + int type = record.type(); //组建成对象，供累计后批量插入、删除 - TwoTuple timelyKeyRecordTwoTuple = new TwoTuple<>(); - if (eventType.equals(Event.EventType.PUT)) { + if (type == 0) { //如果是客户端删除时发出的put指令 if (com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE.equals(value)) { log.info("client remove key event : " + appKey); @@ -117,17 +170,14 @@ public class DataHandler { } //手工添加的是时间戳13位，worker传过来的是uuid String source = value.length() == 13 ? Constant.HAND : Constant.SYSTEM; - timelyKeyRecordTwoTuple.setFirst(KeyTimely.aKeyTimely().key(key).val(value).appName(appName).duration(ttl).uuid(appKey).createTime(date).build()); String rule = RuleUtil.rule(appKey); - KeyRecord keyRecord = new KeyRecord(key, rule, appName, ttl, source, type, uuid, date); + KeyRecord keyRecord = new KeyRecord(key, rule, appName, 1, source, type, uuid, date); keyRecord.setRule(rule); - timelyKeyRecordTwoTuple.setSecond(keyRecord); - return timelyKeyRecordTwoTuple; - } else if (eventType.equals(Event.EventType.DELETE)) { - timelyKeyRecordTwoTuple.setFirst(KeyTimely.aKeyTimely().key(key).appName(appName).build()); - return timelyKeyRecordTwoTuple; + return keyRecord; + } else { + //是删除 + return null; } - return timelyKeyRecordTwoTuple; } @@ -210,4 +260,89 @@ public class DataHandler { } + + /** + * 每天根据app的配置清理过期数据 + */ + @Scheduled(cron = "0 0 1 * * ?") + public void clearExpireData() { + try { + LocalDateTime now = LocalDateTime.now(); + List keyValues = configCenter.getPrefix(ConfigConstant.clearCfgPath); + for (KeyValue kv : keyValues) { + String key = kv.getKey().toStringUtf8(); + String ttl = kv.getValue().toStringUtf8(); + String app = key.replace(ConfigConstant.clearCfgPath, ""); + Date expireDate = DateUtil.ldtToDate(now.minusDays(Integer.parseInt(ttl))); + summaryMapper.clearExpireData(app, expireDate); + keyRecordMapper.clearExpireData(app, expireDate); + statisticsMapper.clearExpireData(app, expireDate); + } + } catch (Exception e) { + e.printStackTrace(); + } + + } + + /** + * 每10秒检测一次热点记录 用于监控报警 + */ + //TODO WANG +// @PostConstruct + public void scanRecordForMonitor() { + ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); + //开启拉取etcd的worker信息，如果拉取失败，则定时继续拉取 + scheduledExecutorService.scheduleAtFixedRate(() -> { + try { + SearchReq req = new SearchReq(); + Date date = new Date(); + req.setEndTime(date); + List keyValues = configCenter.getPrefix(ConfigConstant.appCfgPath); + for (KeyValue kv : keyValues) { + queryRecordAndCheck(kv, req, date); + } + } catch (Exception e) { + e.printStackTrace(); + } + }, 2, 10, TimeUnit.SECONDS); + + } + + /** + * 查询条数 比对配置 发送报警 + */ + private void queryRecordAndCheck(KeyValue kv, SearchReq req, Date date) { + String val = kv.getValue().toStringUtf8(); + AppCfgVo cfg = JSON.parseObject(val, AppCfgVo.class); + if (cfg.getWarn() != 1) { + return; + } + + req.setApp(cfg.getApp()); + req.setStartTime(new Date(date.getTime() - cfg.getWarnPeriod() * 1000)); + + int count = keyRecordMapper.countKeyRecord(req); + //抽样2%打印 + if (count > 0 && Math.abs(new Random().nextInt()) % 50 == 0) { + log.info("应用app:{}, 记录count:{}, 统计时间Period:{}", cfg.getApp(), count, cfg.getWarnPeriod()); + } + int type = 0; + if (count >= cfg.getWarnMax()) { + type = 1; + } else if (count <= cfg.getWarnMin()) { + type = 2; + } + + if (type > 0) { + String str = type == 1 ? "高于最大" : "低于最小"; + int threshold = type == 1 ? cfg.getWarnMax() : cfg.getWarnMin(); + String time = LocalDateTime.now().toString().replace("T", " "); + String content = String.format("【警报】 应用：【%s】 热点记录在%d秒内累计: %d, %s阈值: %d \n【时间】：%s", cfg.getApp(), cfg.getWarnPeriod(), count, str, threshold, time); + //TODO WANG + System.out.println(content); +// pushHandler.pushMsg(cfg.getApp(), date, content); + } + } + + } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java deleted file mode 100644 index 5d4239437810b0a8535acb233bb2d116adcc77ba..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java +++ /dev/null @@ -1,21 +0,0 @@ -package com.jd.platform.hotkey.dashboard.controller; - -/** - * @author wuweifeng - * @version 1.0 - * @date 2020-07-16 - */ -//@RestController -//public class TestController { -// @Resource -// private IConfigCenter configCenter; -// -// @RequestMapping("delete") -// public Object delete(String key) { -//// configCenter.putAndGrant(ConfigConstant.hotKeyRecordPath + "sample" + "/" + key, com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE, 1); -// configCenter.putAndGrant(ConfigConstant.hotKeyRecordPath + "bbb" + "/" + key, "", 60); -//// configCenter.delete(ConfigConstant.hotKeyPath + "sample" + "/" + key); -// -// return "1"; -// } -//} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpConfiguration.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpConfiguration.java deleted file mode 100644 index 6b12a85d17330d3175b0efe00331b8d757156f31..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpConfiguration.java +++ /dev/null @@ -1,24 +0,0 @@ -package com.jd.platform.hotkey.dashboard.erp; - - -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; - -@Configuration -public class ErpConfiguration { - - @Autowired - ErpProperties erpProperties; - - @Bean - public ErpUimInterceptor erpUimInterceptor() { - ErpUimInterceptor springSSOInterceptor = new ErpUimInterceptor(); - springSSOInterceptor.setSsoAppKey(erpProperties.getSsoAppKey()); - springSSOInterceptor.setSsoAppToken(erpProperties.getSsoAppToken()); - springSSOInterceptor.setLoginUrl(erpProperties.getLoginUrl()); - springSSOInterceptor.setExcludePath(erpProperties.getExcludePath()); - springSSOInterceptor.setSsoAppUrl(erpProperties.getSsoAppUrl()); - return springSSOInterceptor; - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpProperties.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpProperties.java deleted file mode 100644 index 1d4cf23e1875976e8761598f1e1017408ce758ef..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpProperties.java +++ /dev/null @@ -1,75 +0,0 @@ -package com.jd.platform.hotkey.dashboard.erp; - -import com.jd.platform.hotkey.dashboard.autoconfigure.AbstractProperties; -import org.springframework.boot.context.properties.ConfigurationProperties; -import org.springframework.stereotype.Component; - -/** - * User: fuxueliang - * Date: 16/8/17 - * Email: fuxueliang@jd.com - */ -@Component -@ConfigurationProperties(prefix = "erp") -public class ErpProperties extends AbstractProperties { - - private String excludePath; - - private String loginUrl; - - private String ssoAppUrl; - - private String ssoAppKey; - - private String ssoAppToken; - - private Boolean enabled; - - public String getExcludePath() { - return excludePath; - } - - public void setExcludePath(String excludePath) { - this.excludePath = excludePath; - } - - public String getLoginUrl() { - return loginUrl; - } - - public void setLoginUrl(String loginUrl) { - this.loginUrl = loginUrl; - } - - public String getSsoAppUrl() { - return ssoAppUrl; - } - - public void setSsoAppUrl(String ssoAppUrl) { - this.ssoAppUrl = ssoAppUrl; - } - - public String getSsoAppKey() { - return ssoAppKey; - } - - public void setSsoAppKey(String ssoAppKey) { - this.ssoAppKey = ssoAppKey; - } - - public String getSsoAppToken() { - return ssoAppToken; - } - - public void setSsoAppToken(String ssoAppToken) { - this.ssoAppToken = ssoAppToken; - } - - public Boolean getEnabled() { - return enabled; - } - - public void setEnabled(Boolean enabled) { - this.enabled = enabled; - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpUimInterceptor.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpUimInterceptor.java deleted file mode 100644 index beabf4e29cf2a1b453357ad18d5b0cc05d95f892..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpUimInterceptor.java +++ /dev/null @@ -1,56 +0,0 @@ -package com.jd.platform.hotkey.dashboard.erp; - -import com.jd.common.springmvc.interceptor.SpringSSOInterceptor; -import com.jd.common.web.LoginContext; -import com.jd.platform.hotkey.dashboard.model.User; -import com.jd.platform.hotkey.dashboard.service.UserService; -import org.springframework.beans.factory.annotation.Autowired; - -import javax.servlet.http.Cookie; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - - -public class ErpUimInterceptor extends SpringSSOInterceptor { - - @Autowired - private UserService userService; - - public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception { - LoginContext loginContext = LoginContext.getLoginContext(); - if (null == loginContext) { - super.preHandle(request, response, handler); - } - loginContext = LoginContext.getLoginContext(); - if (null == loginContext) { - return true; - } - /*Cookie cookie = new Cookie("erp", loginContext.getNick()); - cookie.setMaxAge(3600*24*7); - cookie.setPath("/"); - response.addCookie(cookie);*/ - String pin = loginContext.getPin(); - String nickName = loginContext.getNick(); - String orgName = loginContext.getOrgName(); - String orgId = loginContext.getOrgId(); - String mobile = loginContext.getMobile(); - String email = loginContext.getEmail(); - User user = new User(); - user.setAppName("test"); - user.setUserName(pin); - user.setNickName(nickName); - user.setPhone(mobile); - user.setRole("APPUSER"); - Cookie cookie = userService.loginErpUser(user); - Cookie[] cookies = request.getCookies(); - if(cookies != null){ - for(Cookie tempCookie : cookies){ - if("token".equals(tempCookie.getName()) && tempCookie.getValue() != null){ - return true; - } - } - } - response.addCookie(cookie); - return true; - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/EtcdConfig.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdConfig.java similarity index 93% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/EtcdConfig.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdConfig.java index 0f585b61aaf5f396ce92a41f63d01788aa333930..edb0708dadf8cae2230d480e237d936ce6387cfe 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/EtcdConfig.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdConfig.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.common.config; +package com.jd.platform.hotkey.dashboard.etcd; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; import com.jd.platform.hotkey.common.configcenter.etcd.JdEtcdBuilder; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java similarity index 66% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java index 92fa472d0c6b42101ebf2bea6528682feec762af..887616270dcf947bd9ba7d3f6fe979f0787ecd6e 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.common.monitor; +package com.jd.platform.hotkey.dashboard.etcd; import cn.hutool.core.util.StrUtil; import com.ibm.etcd.api.Event; @@ -6,27 +6,28 @@ import com.ibm.etcd.api.KeyValue; import com.ibm.etcd.client.kv.KvClient; import com.jd.platform.hotkey.common.configcenter.ConfigConstant; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.common.rule.KeyRule; import com.jd.platform.hotkey.common.tool.FastJsonUtils; import com.jd.platform.hotkey.dashboard.common.domain.Constant; -import com.jd.platform.hotkey.dashboard.common.domain.EventWrapper; -import com.jd.platform.hotkey.dashboard.mapper.SummaryMapper; +import com.jd.platform.hotkey.dashboard.common.domain.IRecord; +import com.jd.platform.hotkey.dashboard.common.monitor.DataHandler; +import com.jd.platform.hotkey.dashboard.biz.mapper.SummaryMapper; import com.jd.platform.hotkey.dashboard.model.Worker; -import com.jd.platform.hotkey.dashboard.service.WorkerService; +import com.jd.platform.hotkey.dashboard.netty.HotKeyReceiver; +import com.jd.platform.hotkey.dashboard.biz.service.WorkerService; import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; import io.grpc.StatusRuntimeException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; -import org.springframework.util.StringUtils; import javax.annotation.PostConstruct; +import javax.annotation.PreDestroy; import javax.annotation.Resource; -import java.util.ArrayList; -import java.util.Date; -import java.util.List; -import java.util.Map; +import java.util.*; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; @@ -55,7 +56,7 @@ public class EtcdMonitor { private DataHandler dataHandler; - public static final ExecutorService threadPoolExecutor = Executors.newCachedThreadPool(); + private final ExecutorService threadPoolExecutor = Executors.newFixedThreadPool(16); /** * 监听新来的热key，该key的产生是来自于手工在控制台添加 @@ -65,48 +66,36 @@ public class EtcdMonitor { KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.hotKeyPath); while (watchIterator.hasNext()) { Event event = event(watchIterator); - EventWrapper eventWrapper = build(event); + KeyValue kv = event.getKv(); + String v = kv.getValue().toStringUtf8(); + Event.EventType eventType = event.getType(); String appKey = event.getKv().getKey().toStringUtf8().replace(ConfigConstant.hotKeyPath, ""); - eventWrapper.setKey(appKey); - dataHandler.offer(eventWrapper); - } - }); - } + dataHandler.offer(new IRecord() { + @Override + public String appNameKey() { + return appKey; + } - /** - * 监听新来的热key，该key的产生是来自于worker集群推送过来的 - */ - public void watchHotKeyRecord() { - threadPoolExecutor.submit(() -> { - KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.hotKeyRecordPath); - while (watchIterator.hasNext()) { - Event event = event(watchIterator); - EventWrapper eventWrapper = build(event); + @Override + public String value() { + return v; + } - String appKey = event.getKv().getKey().toStringUtf8().replace(ConfigConstant.hotKeyRecordPath, ""); - eventWrapper.setKey(appKey); + @Override + public int type() { + return eventType.getNumber(); + } - dataHandler.offer(eventWrapper); + @Override + public Date createTime() { + return new Date(); + } + }); } }); } - private EventWrapper build(Event event) { - KeyValue kv = event.getKv(); - long ttl = configCenter.timeToLive(kv.getLease()); - String v = kv.getValue().toStringUtf8(); - Event.EventType eventType = event.getType(); - EventWrapper eventWrapper = new EventWrapper(); - eventWrapper.setValue(v); - eventWrapper.setDate(new Date()); - eventWrapper.setTtl(ttl); - eventWrapper.setVersion(kv.getVersion()); - eventWrapper.setEventType(eventType); - eventWrapper.setUuid(v); - return eventWrapper; - } - @PostConstruct public void startWatch() { @@ -116,26 +105,83 @@ public class EtcdMonitor { //规则拉取完毕后才能去开始入库 insertRecords(); - //开始监听热key产生 - watchHotKeyRecord(); + //开始入库 + dealHotKey(); + //监听手工创建的key watchHandOperationHotKey(); //监听rule变化 watchRule(); - watchWorkers(); +// watchWorkers(); //观察热key访问次数和总访问次数，并做统计 watchHitCount(); } + /** + * 每秒去清理一次caffeine + */ + @Scheduled(fixedRate = 1000) + public void cleanCaffeine() { + try { + HotKeyReceiver.cleanUpCaffeine(); + } catch (Exception e) { + e.printStackTrace(); + } + } + private void insertRecords() { threadPoolExecutor.submit(() -> { dataHandler.insertRecords(); }); } + + /** + * 开始消费各worker发来的热key + */ + private void dealHotKey() { + threadPoolExecutor.submit(() -> { + while (true) { + try { + //获取发来的这个热key，存入本地caffeine，设置过期时间 + HotKeyModel model = HotKeyReceiver.take(); + + //将该key放入实时热key本地缓存中 + HotKeyReceiver.writeToLocalCaffeine(model); + + dataHandler.offer(new IRecord() { + @Override + public String appNameKey() { + return model.getAppName() + "/" + model.getKey(); + } + + @Override + public String value() { + return UUID.randomUUID().toString(); + } + + @Override + public int type() { + return 0; + } + + @Override + public Date createTime() { + return new Date(); + } + }); + + } catch (Exception e) { + e.printStackTrace(); + } + } + + }); + } + /** * 异步监听rule规则变化 */ @@ -149,7 +195,7 @@ public class EtcdMonitor { Event event = event(watchIterator); try { - log.info("---------watch rule change---------"); + log.info("---------watch rule change---------"+event.getType().toString()); //全量拉取rule信息 fetchRuleFromEtcd(); } catch (Exception e) { @@ -166,8 +212,6 @@ public class EtcdMonitor { private void fetchRuleFromEtcd() { RuleUtil.init(); try { - List ruleList = new ArrayList<>(); - //从etcd获取rule List keyValues = configCenter.getPrefix(ConfigConstant.rulePath); @@ -217,7 +261,6 @@ public class EtcdMonitor { } - /** * 监听热key访问次数和总访问次数 */ @@ -252,5 +295,13 @@ public class EtcdMonitor { return watchIterator.next().getEvents().get(0); } + @PreDestroy + public void close() { + try { + threadPoolExecutor.shutdown(); + } catch (Exception e) { + log.error("EtcdMonitor threadPoolExecutor shutdown exception:{}", e.getMessage(), e); + } + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java new file mode 100644 index 0000000000000000000000000000000000000000..d0a44c5690e9fa212317bbc86055bbb3e36994a0 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java @@ -0,0 +1,63 @@ +package com.jd.platform.hotkey.dashboard.etcd; + +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.tool.IpUtils; +import com.jd.platform.hotkey.dashboard.netty.NodesServer; +import org.springframework.stereotype.Component; + +import javax.annotation.PostConstruct; +import javax.annotation.Resource; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; + +/** + * dashboard也注册到etcd，供各个worker连接 + * + * @author wuweifeng + * @version 1.0 + * @date 2020-08-28 + */ +@Component +public class EtcdRegister { + @Resource + private IConfigCenter configCenter; + @Resource + private NodesServer nodesServer; + + /** + * 每隔一会去check一下，自己还在不在etcd里 + */ + @PostConstruct + public void makeSureSelfOn() { + new Thread(() -> nodesServer.startNettyServer(ConfigConstant.dashboardPort)).start(); + + //开启上传worker信息 + ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); + scheduledExecutorService.scheduleAtFixedRate(() -> { + + try { + uploadSelfInfo(); + } catch (Exception e) { + //do nothing + } + + }, 3, 30, TimeUnit.SECONDS); + } + + private void uploadSelfInfo() { + configCenter.putAndGrant(buildKey(), buildValue(), 32); + } + + private String buildKey() { + String hostName = IpUtils.getHostName(); + return ConfigConstant.dashboardPath + hostName; + } + + private String buildValue() { + String ip = IpUtils.getIp(); + return ip + ":" + ConfigConstant.dashboardPort; + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java index 173a46598708858ad5b59aab24f3a855ca6f7ac8..1f74e2ab87d0c1153111d53a5901a88afb023f02 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java @@ -4,7 +4,6 @@ package com.jd.platform.hotkey.dashboard.interceptor; import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; -import com.jd.platform.hotkey.dashboard.service.UserService; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import io.jsonwebtoken.Claims; import org.springframework.http.HttpMethod; @@ -29,9 +28,8 @@ public class JwtInterceptor extends HandlerInterceptorAdapter{ } String header = request.getHeader("x-requested-with"); if(!StringUtils.isEmpty(header) && "XMLHttpRequest".endsWith(header) && request.getMethod().equals(Constant.POST)){ - final String authHeader = JwtTokenUtil.getAuthHeader(request); - if (StringUtils.isEmpty(authHeader) - || !authHeader.startsWith(JwtTokenUtil.TOKEN_PREFIX)) { + String authHeader = request.getHeader(JwtTokenUtil.AUTH_HEADER_KEY); + if (StringUtils.isEmpty(authHeader) || !authHeader.startsWith(JwtTokenUtil.TOKEN_PREFIX)) { throw new BizException(ResultEnum.NO_LOGIN); } final String token = authHeader.substring(2); diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/MyWebAppConfigurer.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/MyWebAppConfigurer.java index 529e41683a9683b98454a852c53cc84b923dbe76..7360a638195fffb47203f3c5b9909c863e00de07 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/MyWebAppConfigurer.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/MyWebAppConfigurer.java @@ -1,22 +1,18 @@ package com.jd.platform.hotkey.dashboard.interceptor; -import com.jd.platform.hotkey.dashboard.erp.ErpUimInterceptor; -import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Configuration; import org.springframework.core.Ordered; import org.springframework.web.servlet.config.annotation.*; + @Configuration public class MyWebAppConfigurer implements WebMvcConfigurer { - @Autowired - ErpUimInterceptor erpUimInterceptor; @Override public void addInterceptors(InterceptorRegistry registry) { //拦截路径可自行配置多个 可用 ，分隔开 InterceptorRegistration registration = registry.addInterceptor(new JwtInterceptor()).addPathPatterns("/**"); - registry.addInterceptor(erpUimInterceptor); - registration.excludePathPatterns("/user/login","/user/getUserName","/error","/static/**","/main/**"); + registration.excludePathPatterns("/user/login","/error","/static/**","/main/**"); } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java deleted file mode 100644 index 3423da34835955cdee38ce5bf8dc47973e4dc682..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java +++ /dev/null @@ -1,31 +0,0 @@ -package com.jd.platform.hotkey.dashboard.mapper; - -import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.model.KeyTimely; -import org.apache.ibatis.annotations.Mapper; - -import java.util.List; - -@Mapper -public interface KeyTimelyMapper { - - int clear(); - - int deleteByKeyAndApp(String key,String appName); - - int insertSelective(KeyTimely key); - - KeyTimely selectByPrimaryKey(Long id); - - KeyTimely selectByKey(String key); - - int updateByKey(KeyTimely key); - - List listKeyTimely(SearchReq param); - - int batchInsert(List list); - - void batchDeleted(List deleteList); - - void saveOrUpdate(KeyTimely keyTimely); -} \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java index 97f0b4b6ff0e6d2fe8b6e2f8a742497f01535786..e8fdf0f06dbc9524a516944c0845a5c139328dda 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java @@ -16,7 +16,7 @@ public class KeyRecord implements Serializable { /** * 缓存时间 */ - private Long duration; + private int duration; /** * 来源： SYSTEM 系统探测；USERNAME创建人 @@ -43,7 +43,7 @@ public class KeyRecord implements Serializable { public KeyRecord() { } - public KeyRecord(String key,String val, String appName, Long duration, + public KeyRecord(String key,String val, String appName, int duration, String source, Integer type,String uuid, Date createTime) { this.key = key; this.val = val; @@ -103,11 +103,11 @@ public class KeyRecord implements Serializable { this.val = val; } - public Long getDuration() { + public int getDuration() { return duration; } - public void setDuration(Long duration) { + public void setDuration(int duration) { this.duration = duration; } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java index 6efd7c5d9b6f9eebe3a88dd9e55718f326feaf9c..67b9d0ac997647e7f8a05db6cc9915a3c286d332 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java @@ -22,7 +22,6 @@ public class KeyTimely implements Serializable { private Date createTime; - private String uuid; /** * 该rule的描述 */ @@ -44,12 +43,25 @@ public class KeyTimely implements Serializable { public KeyTimely(String key, String val, String appName, Long duration, String uuid, Date createTime) { this.key = key; this.val = val; - this.uuid = uuid; this.appName = appName; this.duration = duration; this.createTime = createTime; } + @Override + public String toString() { + return "KeyTimely{" + + "id=" + id + + ", key='" + key + '\'' + + ", appName='" + appName + '\'' + + ", val='" + val + '\'' + + ", duration=" + duration + + ", createTime=" + createTime + + ", ruleDesc='" + ruleDesc + '\'' + + ", updater='" + updater + '\'' + + '}'; + } + public String getRuleDesc() { return ruleDesc; } @@ -106,14 +118,6 @@ public class KeyTimely implements Serializable { this.createTime = createTime; } - public String getUuid() { - return uuid; - } - - public void setUuid(String uuid) { - this.uuid = uuid; - } - public String getUpdater() { return updater; } @@ -186,7 +190,6 @@ public class KeyTimely implements Serializable { keyTimely.setVal(val); keyTimely.setDuration(duration); keyTimely.setCreateTime(createTime); - keyTimely.setUuid(uuid); keyTimely.setRuleDesc(ruleDesc); keyTimely.setUpdater(updater); return keyTimely; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java new file mode 100644 index 0000000000000000000000000000000000000000..5f5c5f2f825d64be6e2a90e871471956c1ee2f8f --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java @@ -0,0 +1,175 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import cn.hutool.core.util.StrUtil; +import com.github.benmanes.caffeine.cache.Cache; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.rule.KeyRule; +import com.jd.platform.hotkey.dashboard.cache.CaffeineBuilder; +import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; +import com.jd.platform.hotkey.dashboard.model.KeyTimely; +import com.jd.platform.hotkey.dashboard.util.RuleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.stream.Collectors; + +/** + * 此处存储所有发来的热key，统一处理入库 + * + * @author wuweifeng + * @version 1.0 + * @date 2020-08-31 + */ +public class HotKeyReceiver { + + /** + * 发来的热key集中营 + */ + private static LinkedBlockingQueue hotKeyStoreQueue = new LinkedBlockingQueue<>(); + /** + * 存储实时hotkey，供界面查询实时热key + */ + private static Map> aliveKeyStore = new ConcurrentHashMap<>(); + + private static Logger logger = LoggerFactory.getLogger("HotKeyReceiver"); + + /** + * netty收到的先存这里 + */ + public static void push(HotKeyModel model) { + hotKeyStoreQueue.offer(model); + } + + public static HotKeyModel take() { + try { + return hotKeyStoreQueue.take(); + } catch (InterruptedException e) { + e.printStackTrace(); + return null; + } + } + + /** + * 将热key存入本地缓存，设置过期时间 + */ + public static void writeToLocalCaffeine(HotKeyModel hotKeyModel) { + String appNameKey = hotKeyModel.getAppName() + "/" + hotKeyModel.getKey(); + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + logger.error("rule is null, hotkeyModel " + hotKeyModel.getAppName() + "-" + hotKeyModel.getKey()); + return; + } + Cache cache = aliveKeyStore.computeIfAbsent(keyRule.getDuration(), s -> CaffeineBuilder.cache(keyRule.getDuration())); + cache.put(appNameKey, hotKeyModel); + } + + /** + * 展示当前所有的实时热key + */ + public static List list(SearchReq searchReq) { + List timelyList = new ArrayList<>(); + + long now = System.currentTimeMillis(); + + for (Integer duration : aliveKeyStore.keySet()) { + Cache cache = aliveKeyStore.get(duration); + ConcurrentMap concurrentHashMap = cache.asMap(); + for (Map.Entry entry : concurrentHashMap.entrySet()) { + KeyTimely keyTimely = parse((HotKeyModel) entry.getValue(), now); + if (keyTimely == null) { + continue; + } + timelyList.add(keyTimely); + } + } + + if (searchReq != null) { + if (StrUtil.isNotEmpty(searchReq.getApp())) { + timelyList = timelyList.parallelStream().filter(keyTimely -> searchReq.getApp().equals(keyTimely.getAppName())).collect(Collectors.toList()); + } + if (StrUtil.isNotEmpty(searchReq.getKey())) { + timelyList = timelyList.parallelStream().filter(keyTimely -> keyTimely.getKey().startsWith(searchReq.getKey())).collect(Collectors.toList()); + } + + } + + timelyList.sort(Comparator.comparing(KeyTimely::getCreateTime).reversed()); + + return timelyList; + } + + /** + * 将hotkeyModel变成前端需要的对象 + */ + private static KeyTimely parse(HotKeyModel hotKeyModel, long now) { + String appNameKey = hotKeyModel.getAppName() + "/" + hotKeyModel.getKey(); + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + return null; + } + long remainTime = keyRule.getDuration() * 1000 - (now - hotKeyModel.getCreateTime()); + return KeyTimely.aKeyTimely() + .key(hotKeyModel.getKey()) + .val(UUID.randomUUID().toString()) + .appName(hotKeyModel.getAppName()) + .duration(remainTime / 1000) + .ruleDesc(RuleUtil.ruleDesc(appNameKey)) + .createTime(new Date(hotKeyModel.getCreateTime())).build(); + } + +// public static void main(String[] args) { +// List keyTimelyList = new ArrayList<>(); +// for (int i = 0; i < 24; i++) { +// KeyTimely keyTimely = new KeyTimely(); +// keyTimely.setKey(i + ""); +// keyTimely.setCreateTime(new Date()); +// try { +// Thread.sleep(10); +// } catch (InterruptedException e) { +// e.printStackTrace(); +// } +// keyTimelyList.add(keyTimely); +// } +// keyTimelyList.sort(Comparator.comparing(KeyTimely::getCreateTime).reversed()); +// +// Page page =PageUtil.pagination(keyTimelyList, 10, 3); +// System.out.println(page.getTotal()); +// System.out.println(page.getPage()); +// System.out.println(page.getRows()); +// } + + + + + + /** + * 删除实时热key + */ + public static boolean delete(String appNameKey) { + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + return false; + } + Cache cache = aliveKeyStore.get(keyRule.getDuration()); + if (cache == null) { + return false; + } + cache.invalidate(appNameKey); + return true; + } + + /** + * 定时清理caffeine + */ + public static void cleanUpCaffeine() { + for (Integer duration : aliveKeyStore.keySet()) { + Cache cache = aliveKeyStore.get(duration); + cache.cleanUp(); + } + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java new file mode 100644 index 0000000000000000000000000000000000000000..a3dc7eeb5e78b7d6e1364e0b2aca50ddd1ab809e --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java @@ -0,0 +1,78 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import com.jd.platform.hotkey.common.tool.Constant; +import io.netty.bootstrap.ServerBootstrap; +import io.netty.buffer.ByteBuf; +import io.netty.buffer.Unpooled; +import io.netty.channel.*; +import io.netty.channel.nio.NioEventLoopGroup; +import io.netty.channel.socket.nio.NioServerSocketChannel; +import io.netty.handler.codec.DelimiterBasedFrameDecoder; +import io.netty.handler.codec.string.StringDecoder; +import io.netty.handler.logging.LogLevel; +import io.netty.handler.logging.LoggingHandler; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import java.util.concurrent.TimeUnit; + +/** + * 该server用于给各个worker实例连接用。 + * + * @author wuweifeng wrote on 2019-11-05. + */ +@Component +public class NodesServer { + private Logger logger = LoggerFactory.getLogger(getClass()); + + public void startNettyServer(int port) { + //boss单线程 + EventLoopGroup bossGroup = new NioEventLoopGroup(1); + EventLoopGroup workerGroup = new NioEventLoopGroup(4); + try { + ServerBootstrap bootstrap = new ServerBootstrap(); + bootstrap.group(bossGroup, workerGroup) + .channel(NioServerSocketChannel.class) + .handler(new LoggingHandler(LogLevel.INFO)) + .option(ChannelOption.SO_BACKLOG, 1024) + //保持长连接 + .childOption(ChannelOption.SO_KEEPALIVE, true) + //出来网络io事件，如记录日志、对消息编解码等 + .childHandler(new ChildChannelHandler()); + //绑定端口，同步等待成功 + ChannelFuture future = bootstrap.bind(port).sync(); + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + bossGroup.shutdownGracefully (1000, 3000, TimeUnit.MILLISECONDS); + workerGroup.shutdownGracefully (1000, 3000, TimeUnit.MILLISECONDS); + })); + //等待服务器监听端口关闭 + future.channel().closeFuture().sync(); + } catch (Exception e) { + e.printStackTrace(); + logger.error("dashboard netty server start failure"); + } finally { + //优雅退出，释放线程池资源 + bossGroup.shutdownGracefully(); + workerGroup.shutdownGracefully(); + } + } + + /** + * handler类 + */ + private class ChildChannelHandler extends ChannelInitializer { + + @Override + protected void initChannel(Channel ch) { + NodesServerHandler serverHandler = new NodesServerHandler(); + + ByteBuf delimiter = Unpooled.copiedBuffer(Constant.DELIMITER.getBytes()); + ch.pipeline() + .addLast(new DelimiterBasedFrameDecoder(Constant.MAX_LENGTH, delimiter)) + .addLast(new StringDecoder()) + .addLast(serverHandler); + } + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java new file mode 100644 index 0000000000000000000000000000000000000000..9650c2fd9ef5b517eb0769e75aae5fe2fce7d810 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java @@ -0,0 +1,66 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.model.HotKeyMsg; +import com.jd.platform.hotkey.common.model.MsgBuilder; +import com.jd.platform.hotkey.common.model.typeenum.MessageType; +import com.jd.platform.hotkey.common.tool.FastJsonUtils; +import com.jd.platform.hotkey.common.tool.flush.FlushUtil; +import io.netty.channel.ChannelHandlerContext; +import io.netty.channel.SimpleChannelInboundHandler; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.util.StringUtils; + +import java.util.List; + +import static com.jd.platform.hotkey.common.tool.Constant.PONG; + +/** + * 这里处理所有netty事件。 + * + * @author wuweifeng wrote on 2019-11-05. + */ +public class NodesServerHandler extends SimpleChannelInboundHandler { + + private Logger logger = LoggerFactory.getLogger(getClass()); + + @Override + protected void channelRead0(ChannelHandlerContext ctx, String message) { + if (StringUtils.isEmpty(message)) { + return; + } + try { + HotKeyMsg msg = FastJsonUtils.toBean(message, HotKeyMsg.class); + if (MessageType.PING == msg.getMessageType()) { + String hotMsg = FastJsonUtils.convertObjectToJSON(new HotKeyMsg(MessageType.PONG, PONG)); + FlushUtil.flush(ctx, MsgBuilder.buildByteBuf(hotMsg)); + } else if (MessageType.REQUEST_HOT_KEY == msg.getMessageType()) { + List list = FastJsonUtils.toList(msg.getBody(), HotKeyModel.class); + for (HotKeyModel hotKeyModel : list) { + HotKeyReceiver.push(hotKeyModel); + } + } + } catch (Exception e) { + e.printStackTrace(); + } + + } + + @Override + public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception { + logger.error("some thing is error , " + cause.getMessage()); + } + + @Override + public void channelActive(ChannelHandlerContext ctx) throws Exception { + super.channelActive(ctx); + } + + @Override + public void channelInactive(ChannelHandlerContext ctx) throws Exception { + ctx.close(); + super.channelInactive(ctx); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java index 21488acba6910f019869940523bdfda0eae2b0a3..350c3b1a5bb5491e2a60f1d13fea6b369c985694 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java @@ -132,10 +132,16 @@ public class CommonUtil { * @return map */ private static Map> listGroup(List list) { + // return list.stream().collect(Collectors.groupingBy(Statistics::getRule)); + return list.stream().collect(Collectors.groupingBy(Statistics::getKeyName)); + + /* + // master code if (Constant.VERSION != 1) { return list.stream().collect(Collectors.groupingBy(Statistics::getRule)); } return list.stream().collect(Collectors.groupingBy(Statistics::getKeyName)); + */ } /** diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/DateUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/DateUtil.java index 8232567181b3a5381d52c6bb894f9d21d6daf8a2..ce7a1fffee2f6a44a8b91a26e628093283d93d72 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/DateUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/DateUtil.java @@ -40,6 +40,11 @@ public class DateUtil { return null; } + public static String dateToStr(Date date){ + SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); + return format.format(date); + } + public static LocalDateTime strToLdt(String str, String pattern){ return LocalDateTime.parse(str, DateTimeFormatter.ofPattern(pattern)); } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/ExcelUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/ExcelUtil.java index 7641f0e14dad9b90f47dcaea08b08564fc837eec..fa134564ab581a43a6601c74810ee82eb6f363be 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/ExcelUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/ExcelUtil.java @@ -13,6 +13,9 @@ import org.slf4j.LoggerFactory; public class ExcelUtil { + + public static final int MAX_ROW = 60000; + private static Logger log = LoggerFactory.getLogger(ExcelUtil.class); public static void exportExcel(HttpServletResponse response, ExcelDataDto data) { diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java new file mode 100644 index 0000000000000000000000000000000000000000..cd4a9872edd3f4e64af16860b707aabcf41e2ba5 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java @@ -0,0 +1,36 @@ +package com.jd.platform.hotkey.dashboard.util; + +import com.google.common.collect.Lists; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import org.springframework.util.CollectionUtils; + +import java.util.ArrayList; +import java.util.List; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-09-01 + */ +public class PageUtil { + /** + * 通用分页工具类 + * + * @param data + * @param pageSize + * @param pageNum + * @param + * @return + */ + public static Page pagination(final List data, final int pageSize, final int pageNum) { + if (CollectionUtils.isEmpty(data)) { + return new Page<>(1, 0, new ArrayList<>()); + } + List> lists = Lists.partition(data, pageSize); + int localPageNum = pageNum; + if (localPageNum >= lists.size()) { + localPageNum = lists.size() - 1; + } + return new Page<>(localPageNum, data.size(), lists.get(localPageNum)); + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java index 40d82ff98dbd20e3ffc8994900e679c40abb5a50..d9352dc4595b47c048b4503fb7b51edf71670ea8 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java @@ -33,7 +33,7 @@ public class RuleUtil { } /** - * 根据APP的key，获取该key对应的rule + * 根据APP的key，获取该key对应的rule.如 cartpc-pu__ */ public static String rule(String key) { try { @@ -62,7 +62,7 @@ public class RuleUtil { return ""; } - private static KeyRule findByKey(String appNameKey) { + public static KeyRule findByKey(String appNameKey) { synchronized (RULE_MAP) { if (StrUtil.isEmpty(appNameKey)) { return null; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/TwoTuple.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/TwoTuple.java deleted file mode 100644 index 0fdd807c0e54263ae45439b6c53aa33a25a7b232..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/TwoTuple.java +++ /dev/null @@ -1,58 +0,0 @@ -package com.jd.platform.hotkey.dashboard.util; - -import java.io.Serializable; - -/** - * TwoTupleTwoTupleTwoTuple - * @date 2020-02-24 - * @author wuweifeng wrote on 2020-02-24 - * @version 1.0 - */ -public class TwoTuple implements Serializable { - private static final long serialVersionUID = 1L; - /** - * first - */ - private K first; - /** - * second - */ - private V second; - - /** - * TwoTuple - */ - public TwoTuple() { - } - - /** - * getFirst - * @return first - */ - public K getFirst() { - return this.first; - } - - public void setFirst(K first) { - this.first = first; - } - - /** - * second - * @return second - */ - public V getSecond() { - return this.second; - } - - public void setSecond(V second) { - this.second = second; - } - - @Override - public String toString() { - StringBuilder strBuilder = new StringBuilder(); - strBuilder.append("[first = ").append(this.first).append(",").append("second = ").append("]"); - return strBuilder.toString(); - } -} diff --git a/dashboard/src/main/resources/application.yml b/dashboard/src/main/resources/application.yml index 1678e74c5fcc2b262ffd71306c084db0ae54475b..987fbbf059bac0bac0cc6a512239564505a1e025 100644 --- a/dashboard/src/main/resources/application.yml +++ b/dashboard/src/main/resources/application.yml @@ -1,5 +1,5 @@ server : - port : 8080 + port : 8081 servlet : context-path : / spring : @@ -27,26 +27,25 @@ spring : time-zone: GMT+8 date-format: yyyy-MM-dd HH:mm:ss datasource: - username: root - password: password - url: jdbc:mysql://localhost:3306/hotkey?useUnicode=true&characterEncoding=utf-8&useSSL=true&serverTimezone=UTC + username: ${MYSQL_USER:root} + password: ${MYSQL_PASS:root} + url: jdbc:mysql://${MYSQL_HOST:localhost}:3306/hotkey_db?useUnicode=true&characterEncoding=utf-8&useSSL=true&serverTimezone=UTC&useTimezone=true&serverTimezone=GMT driver-class-name: com.mysql.cj.jdbc.Driver + hikari: + max-lifetime: 120000 + idle-timeout: 60000 + connection-timeout: 30000 + maximum-pool-size: 32 + minimum-idle: 10 pagehelper: helperDialect: mysql reasonable: true supportMethodsArguments: true params: count=countSql -mybatis: - configuration: - log-impl: org.apache.ibatis.logging.stdout.StdOutImpl +#mybatis: +# configuration: +# log-impl: org.apache.ibatis.logging.stdout.StdOutImpl etcd: server: ${etcdServer:http://127.0.0.1:2379} -erp: - enabled: false - ssoAppKey: fangzhou - ssoAppToken: 3f48d7ade61940f2ba22079ebc273fb8 - loginUrl: https://ssa.jd.com/sso/login - excludePath: /login,/assets,/bootstarp,/common,/htmlformat,/images,/jn,/jquery,/user/login,login - ssoAppUrl: http://ssa.jd.com - defaultPwd: 123456 + diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.xml similarity index 97% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.xml index 90b3b5d29876ba5ffe387e54d46d20eb57958649..9654fee8a750fbe2a2d2c3508b819097477e3ff1 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.xml @@ -1,6 +1,6 @@ - + @@ -16,7 +16,7 @@ id, biz_id, biz_type, from_str, to_str, app_name,update_user, create_time,uuid + + + + INSERT ignore INTO hk_key_record( @@ -183,20 +197,31 @@ + + delete from hk_key_record + where app_name = #{app} + + limit 10000 + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.xml similarity index 95% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.xml index 505fa979cb3f8328cd63d67853eee6f0eb22f062..568633131087c9635f0f86075ee39ba03ed4424e 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.xml @@ -1,6 +1,6 @@ - + diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.xml similarity index 93% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.xml index 8829d120b9683c00e1195445cda20d2570a7b53a..c409f64a38cecc04069c8c262dd86b03040bddaa 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.xml @@ -1,6 +1,6 @@ - + @@ -74,8 +74,9 @@ - + where app = #{app} + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.xml similarity index 90% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.xml index 60f41a28ea0779691f0780acb55eafd5bfcfd15c..6cae25391df38ce3dd70518a1d9f34df8e8a6036 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.xml @@ -1,6 +1,6 @@ - + @@ -51,4 +51,12 @@ + + delete from hk_summary + where app = #{app} + + limit 10000 + + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.xml similarity index 96% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.xml index bacdd874ef3ccfbfafa31276edfab19a894c7bb1..afe87f1f97804d6c49aa9b131bef4f98d8833838 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.xml @@ -1,6 +1,6 @@ - + @@ -16,7 +16,7 @@ id, nick_name, user_name, pwd, phone, role, app_name, create_time, state + + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.xml similarity index 97% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.xml index 5be300c9ba347f3a1fec0bbd5daec95ec09a3826..09e72b2f9c1967f8f3b8c8eff539efa31797f931 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.xml @@ -1,6 +1,6 @@ - + @@ -16,7 +16,7 @@ - select - - from hk_key_timely - where id = #{id,jdbcType=BIGINT} - - - - delete from hk_key_timely - where key_name = #{key} and app_name = #{appName} - - - - - delete from hk_key_timely - where (app_name,key_name) in - - (#{item.appName},#{item.key}) - - - - - delete from hk_key_timely where 1 = 1 - - - - - INSERT ignore INTO - hk_key_timely( - key_name, - val, - app_name, - uuid, - duration, - create_time) - VALUES - - (#{k.key}, - #{k.val}, - #{k.appName}, - #{k.uuid}, - #{k.duration}, - #{k.createTime}) - - - - - insert into hk_key_timely - - - id, - - - key_name, - - - val, - - - uuid, - - - app_name, - - - duration, - - - create_time, - - - - - #{id,jdbcType=BIGINT}, - - - #{key,jdbcType=VARCHAR}, - - - #{val,jdbcType=VARCHAR}, - - - #{uuid,jdbcType=VARCHAR}, - - - #{appName,jdbcType=VARCHAR}, - - - #{duration,jdbcType=BIGINT}, - - - #{createTime,jdbcType=TIMESTAMP}, - - - - - - update hk_key_timely - - - key_name = #{key,jdbcType=VARCHAR}, - - - val = #{val,jdbcType=VARCHAR}, - - - uuid = #{uuid,jdbcType=VARCHAR}, - - - app_name = #{appName,jdbcType=VARCHAR}, - - - duration = #{duration,jdbcType=BIGINT}, - - - create_time = #{createTime,jdbcType=TIMESTAMP}, - - - where key_name = #{key} - - - - - - - - insert into hk_key_timely - - - key_name, - - - val, - - - uuid, - - - app_name, - - - duration, - - - create_time, - - - - - #{key,jdbcType=VARCHAR}, - - - #{val,jdbcType=VARCHAR}, - - - #{uuid,jdbcType=VARCHAR}, - - - #{appName,jdbcType=VARCHAR}, - - - #{duration,jdbcType=BIGINT}, - - - #{createTime,jdbcType=TIMESTAMP}, - - - on duplicate key update - create_time = now() - - - \ No newline at end of file diff --git a/dashboard/src/main/resources/db.sql b/dashboard/src/main/resources/db.sql index 06e6b0c4d8442092708ef8ae15666fd545131932..7497750b4cd707f29eed5a1b364cc65c62c4d9aa 100644 --- a/dashboard/src/main/resources/db.sql +++ b/dashboard/src/main/resources/db.sql @@ -1,3 +1,8 @@ +-- !!! 注意设置sql model 否则可能sql报错 ！！！ +-- 查询你的sql_model参数：select @@global.sql_mode; 发现ONLY_FULL_GROUP_BY 则会导致报错 +-- 解决方式：set @@global.sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION' +-- 详情查阅：https://www.cnblogs.com/hjhsblogs/p/11079356.html + DROP TABLE IF EXISTS `hk_change_log`; CREATE TABLE `hk_change_log` ( `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键', @@ -31,6 +36,11 @@ CREATE TABLE `hk_user` ( ) ENGINE = InnoDB AUTO_INCREMENT = 2 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Compact; +-- pwd: 123456 +INSERT INTO `hk_user` VALUES (2, 'admin', 'admin', 'e10adc3949ba59abbe56e057f20f883e', '1888888', 'ADMIN', '', '2020-07-28 14:01:03', 1); + + + DROP TABLE IF EXISTS `hk_key_record`; CREATE TABLE `hk_key_record` ( `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', @@ -115,3 +125,19 @@ CREATE TABLE `hk_summary` ( ) ENGINE = InnoDB AUTO_INCREMENT = 18 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin COMMENT = '汇总表' ROW_FORMAT = Compact; + +/* 请确认以下SQL符合您的变更需求，务必确认无误后再提交执行 */ + +CREATE TABLE `biz_access_token` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `token` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '' COMMENT 'token', + `flag` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '' COMMENT 'flag', + `CREATED_BY` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL COMMENT '创建人', + `CREATED_TIME` datetime NULL ON UPDATE CURRENT_TIMESTAMP COMMENT '创建时间', + `UPDATED_BY` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL COMMENT '修改人', + `UPDATED_TIME` datetime NULL ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间', + PRIMARY KEY (`id`) +) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin COMMENT = '咚咚token表' ROW_FORMAT = Compact; + + + diff --git a/dashboard/src/main/resources/logback-spring.xml b/dashboard/src/main/resources/logback-spring.xml index f97e83e65348a9ba35ad545f5dfc813a328375ac..14649ecaf12491c1d2ef9679d53f3ac7b69b03fd 100644 --- a/dashboard/src/main/resources/logback-spring.xml +++ b/dashboard/src/main/resources/logback-spring.xml @@ -6,7 +6,7 @@ debug：当此属性设置为true时，将打印出logback内部日志信息， --> - + diff --git a/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js b/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js index 05eb56eb7f5ac72a9d28e8518840d0971a752788..13c97e45ebeae4d237457aab13b410054cafa62d 100644 --- a/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js +++ b/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js @@ -40,20 +40,13 @@ ajaxOptions:{ headers: {"Authorization": getCookie("token")} }, - exportOptions:{ - ignoreColumn: [0,1], //忽略某一列的索引 - fileName: '报表导出', //文件名称设置 - worksheetName: 'sheet1', //表格工作区名称 - tableName: '报表导出', - excelstyles: ['background-color', 'color', 'font-size', 'font-weight'] - }, onLoadSuccess: function(){ //加载成功时执行 var datas = $('#dataTable').bootstrapTable('getData'); console.info("加载成功"); }, onLoadError: function(status){ //加载失败时执行 let token = getCookie("token"); - if(status === 500 && ( token == "undefined" || token =="")){ + if(status === 1000 && ( token == "undefined" || token =="")){ top.location.href = '/user/login'; } console.info("加载数据失败"); @@ -70,7 +63,15 @@ if(app!==undefined && app != null && app!==""){ search.app = app; } - return search; + let time1 = $("#startTime").val(); + if(time1!==undefined && time1 != null && time1!==""){ + search.startTime = time1; + } + let time2 = $("#endTime").val(); + if(time2!==undefined && time2 != null && time2!==""){ + search.endTime = time2; + } + return search; }, search:function(my){//查询条件 diff --git a/dashboard/src/main/resources/static/admin/common/js/appcfg-edit.js b/dashboard/src/main/resources/static/admin/common/js/appcfg-edit.js new file mode 100644 index 0000000000000000000000000000000000000000..f2ba469069467bb157001e6f0ddedffa5ab7cf0a --- /dev/null +++ b/dashboard/src/main/resources/static/admin/common/js/appcfg-edit.js @@ -0,0 +1,25 @@ +$("#form-edit").validate({ + submitHandler:function(form){ + edit(); + } +}); + +function edit() { + let dataFormJson=$("#form-edit").serialize(); + $.ajax({ + cache : true, + type : "POST", + url : "/appCfg/save", + data : dataFormJson, + headers: { + "Authorization":getCookie("token") + }, + async : false, + error : function(XMLHttpRequest){ + $.modal.alertError(XMLHttpRequest.responseJSON.msg); + }, + success : function(data) { + $.operate.saveSuccess(data); + } + }); +} diff --git a/dashboard/src/main/resources/static/admin/common/js/clearcfg-edit.js b/dashboard/src/main/resources/static/admin/common/js/clearcfg-edit.js new file mode 100644 index 0000000000000000000000000000000000000000..0d8de56a7dfb159200148fa225691269d0e574bc --- /dev/null +++ b/dashboard/src/main/resources/static/admin/common/js/clearcfg-edit.js @@ -0,0 +1,25 @@ +$("#form-edit").validate({ + submitHandler:function(form){ + edit(); + } +}); + +function edit() { + let dataFormJson=$("#form-edit").serialize(); + $.ajax({ + cache : true, + type : "POST", + url : "/clear/save", + data : dataFormJson, + headers: { + "Authorization":getCookie("token") + }, + async : false, + error : function(XMLHttpRequest){ + $.modal.alertError(XMLHttpRequest.responseJSON.msg); + }, + success : function(data) { + $.operate.saveSuccess(data); + } + }); +} diff --git a/dashboard/src/main/resources/static/admin/common/js/key-add.js b/dashboard/src/main/resources/static/admin/common/js/key-add.js index f9aadfe88cb89de64f2685ca03c88044b5657b0e..4cf1263e15a1e2c9de9a6ab2b3ea62e3a79a70f6 100644 --- a/dashboard/src/main/resources/static/admin/common/js/key-add.js +++ b/dashboard/src/main/resources/static/admin/common/js/key-add.js @@ -14,6 +14,10 @@ $.ajax({ async : false, error : function(XMLHttpRequest){ $.modal.alertError(XMLHttpRequest.responseJSON.msg); + let token = getCookie("token"); + if(XMLHttpRequest.status == 1000 && ( token == "undefined" || token =="")){ + top.location.href = '/user/login'; + } }, success : function(data) { console.log(data) diff --git a/dashboard/src/main/resources/static/admin/common/js/key-edit.js b/dashboard/src/main/resources/static/admin/common/js/key-edit.js index 3579fbf37223db09153ed26c139135368014933e..edcb793e970316d7a11fef0fd8724c826f024609 100644 --- a/dashboard/src/main/resources/static/admin/common/js/key-edit.js +++ b/dashboard/src/main/resources/static/admin/common/js/key-edit.js @@ -18,6 +18,10 @@ function edit() { async : false, error : function(XMLHttpRequest){ $.modal.alertError(XMLHttpRequest.responseJSON.msg); + let token = getCookie("token"); + if(XMLHttpRequest.status == 1000 && ( token == "undefined" || token =="")){ + top.location.href = '/user/login'; + } }, success : function(data) { $.operate.saveSuccess(data); diff --git a/dashboard/src/main/resources/static/admin/common/js/rule-add.js b/dashboard/src/main/resources/static/admin/common/js/rule-add.js index fc87f3532dec01774c9258cffa0b64725af400e3..dc07324c8eac13ca9cbe2f89c315a0b7e9e072e2 100644 --- a/dashboard/src/main/resources/static/admin/common/js/rule-add.js +++ b/dashboard/src/main/resources/static/admin/common/js/rule-add.js @@ -17,6 +17,10 @@ function add() { async : false, error : function(XMLHttpRequest){ $.modal.alertError(XMLHttpRequest.responseJSON.msg); + let token = getCookie("token"); + if(XMLHttpRequest.status == 1000 && ( token == "undefined" || token =="")){ + top.location.href = '/user/login'; + } }, success : function(data) { $.operate.saveSuccess(data); diff --git a/dashboard/src/main/resources/static/admin/common/js/rule-edit.js b/dashboard/src/main/resources/static/admin/common/js/rule-edit.js index 46a1115d5e1f3b2fe8c20d669dea599ba40bcc3e..1ad15905a740c0bc7c591d42f2cc03db4177da1a 100644 --- a/dashboard/src/main/resources/static/admin/common/js/rule-edit.js +++ b/dashboard/src/main/resources/static/admin/common/js/rule-edit.js @@ -16,9 +16,11 @@ function edit() { "Authorization":getCookie("token") }, async : false, - error : function(response,status,xhr) { - - $.modal.alertError("系统错误"); + error : function(XMLHttpRequest) { + let token = getCookie("token"); + if(XMLHttpRequest.status == 1000 && ( token == "undefined" || token =="")){ + top.location.href = '/user/login'; + } }, success : function(data) { $.operate.saveSuccess(data); diff --git a/dashboard/src/main/resources/static/admin/common/js/user-add.js b/dashboard/src/main/resources/static/admin/common/js/user-add.js index 9f5103fa1120fd8e617e2538e17f35da818427fe..7c2b6feb3624eb35fc884ede79439aa7fb0d3d57 100644 --- a/dashboard/src/main/resources/static/admin/common/js/user-add.js +++ b/dashboard/src/main/resources/static/admin/common/js/user-add.js @@ -32,6 +32,10 @@ function add() { async : false, error : function(XMLHttpRequest){ $.modal.alertError(XMLHttpRequest.responseJSON.msg); + let token = getCookie("token"); + if(XMLHttpRequest.status == 1000 && ( token == "undefined" || token =="")){ + top.location.href = '/user/login'; + } }, success : function(data) { $.operate.saveSuccess(data); diff --git a/dashboard/src/main/resources/static/admin/common/js/user-edit.js b/dashboard/src/main/resources/static/admin/common/js/user-edit.js index b147d59648708720233f406ef94729bdb2687c8b..3b95013cc84d00a053a58a918ff4c9232da4691f 100644 --- a/dashboard/src/main/resources/static/admin/common/js/user-edit.js +++ b/dashboard/src/main/resources/static/admin/common/js/user-edit.js @@ -18,6 +18,10 @@ function edit() { async : false, error : function(XMLHttpRequest){ $.modal.alertError(XMLHttpRequest.responseJSON.msg); + let token = getCookie("token"); + if(XMLHttpRequest.status == 1000 && ( token == "undefined" || token =="")){ + top.location.href = '/user/login'; + } }, success : function(data) { $.operate.saveSuccess(data); diff --git a/dashboard/src/main/resources/templates/admin/appcfg/edit.html b/dashboard/src/main/resources/templates/admin/appcfg/edit.html new file mode 100644 index 0000000000000000000000000000000000000000..d636f2927aa6f4156d95e91c8a9d1e92f0ea3a76 --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/appcfg/edit.html @@ -0,0 +1,95 @@ + + + + + + + + + + + + + + + + + +
+
+
+
+
+
+
+
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+
+ + +
+
+
+
+
+
+
+
+
+ + +
+ +
+ + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/appcfg/list.html b/dashboard/src/main/resources/templates/admin/appcfg/list.html new file mode 100644 index 0000000000000000000000000000000000000000..c26b982ce7e3b8c69b3c2b7f8e456bdb50ab149b --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/appcfg/list.html @@ -0,0 +1,167 @@ + + + + + + + + + + + + + + + +
+
+ +
+ +
+
+ +
+
+
搜索条件
+
+ +
+
+
+
+ + +
+ +
+
+
+
+ + +
+
+
表单
+
+ + + +
+
+ + + +
+
+ +
+ +
+
+
+
+ +
+ +
+
+
+ + + +
+ +
+ + + + + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/changeLog/list.html b/dashboard/src/main/resources/templates/admin/changeLog/list.html index a6fbf8dec7ad1e188a58550a8bc03b2a912f605f..d6ed97aaea1e10b9a00fb023cc82834083b8bf82 100644 --- a/dashboard/src/main/resources/templates/admin/changeLog/list.html +++ b/dashboard/src/main/resources/templates/admin/changeLog/list.html @@ -142,6 +142,11 @@ var options = { } } }, + { field: 'createTime', title: '创建时间', + formatter:function (val,row,index) { + return changeDateFormat(val); + } + }, { field: 'from', title: '初始值', diff --git a/dashboard/src/main/resources/templates/admin/clear/edit.html b/dashboard/src/main/resources/templates/admin/clear/edit.html new file mode 100644 index 0000000000000000000000000000000000000000..dd2b4d8bf9fab5d5bd40c3060420f9a7b6c68f33 --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/clear/edit.html @@ -0,0 +1,69 @@ + + + + + + + + + + + + + + + + + +
+
+
+
+
+
+
+
+
+
+ +
+ +
+
+
+ +
+ +
+
+ +
+
+ + +
+
+
+
+
+
+
+
+
+ + +
+ +
+ + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/clear/list.html b/dashboard/src/main/resources/templates/admin/clear/list.html new file mode 100644 index 0000000000000000000000000000000000000000..7087f7929f522c6402334f7b213f6b481ec11d4e --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/clear/list.html @@ -0,0 +1,157 @@ + + + + + + + + + + + + + + + +
+
+ +
+ +
+
+ +
+
+
搜索条件
+
+ +
+
+
+
+ + +
+ +
+
+
+
+ + +
+
+
表单
+
+ + + +
+
+ + + +
+
+ +
+ +
+
+
+
+ +
+ +
+
+
+ + + +
+ +
+ + + + + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/common/html/footer.html b/dashboard/src/main/resources/templates/admin/common/html/footer.html index 662d4f30764f30e25475563e15ea89bfeb0405c9..703a3594a77952ccb19a881d1cf944cfee009c22 100644 --- a/dashboard/src/main/resources/templates/admin/common/html/footer.html +++ b/dashboard/src/main/resources/templates/admin/common/html/footer.html @@ -10,8 +10,8 @@ - 2020 京东零售-平台业务中心-平台业务研发部-基础业务研发部 版权所有 | 使用支持：武伟峰 | 咚咚：85134158 -
接入文档：https://cf.jd.com/pages/viewpage.action?pageId=277756998 + diff --git a/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html b/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html index 29baffe1bc32bf1bf3941635d9002821c4a7e5e2..39d52f15c094795634021bab6eb0fa78ecdb5bbd 100644 --- a/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html +++ b/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html @@ -28,7 +28,7 @@ - +
  • @@ -52,7 +52,7 @@ - + 热点记录 变更记录 + + + 数据清理配置 + diff --git a/dashboard/src/main/resources/templates/admin/common/html/title.html b/dashboard/src/main/resources/templates/admin/common/html/title.html index 08cf7c852be0a4c250e6364ec7390222649a51e8..aa2fcafcb48c7a172780444f97aa42d28e7e9143 100644 --- a/dashboard/src/main/resources/templates/admin/common/html/title.html +++ b/dashboard/src/main/resources/templates/admin/common/html/title.html @@ -36,8 +36,8 @@
    Widget