diff --git a/LICENSE b/LICENSE index 261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64..48d4f578282c0a3136ac7045a888248e030fb918 100644 --- a/LICENSE +++ b/LICENSE @@ -186,7 +186,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright [2020] [JD.com, Inc.] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/README.md b/README.md index fb0af04859f2e1f453784dd2336cbe4e06d5edd8..a13d86ac3f38c379bf45bcf0f4e6df0df2a62f17 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,44 @@ # hotkey +![输入图片说明](https://images.gitee.com/uploads/images/2020/0616/105737_e5b876cd_303698.png "redis热key探测及缓存到JVM (1).png") -正在京东APP后台灰度内测中，小幅度压测过，3台worker每秒接收16万个key，持续10几个小时，大概处理了60多亿个key，平稳无异常，cpu15%无大波动。 +对任意突发性的无法预先感知的热点数据，包括并不限于热点数据（如突发大量请求同一个商品）、热用户（如恶意爬虫刷子）、热接口（突发海量请求同一个接口）等，进行毫秒级精准探测到。然后对这些热数据、热用户等，推送到所有服务端JVM内存中，以大幅减轻对后端数据存储层的冲击，并可以由使用者决定如何分配、使用这些热key（譬如对热商品做本地缓存、对热用户进行拒绝访问、对热接口进行熔断或返回默认值）。这些热数据在整个服务端集群内保持一致性，并且业务隔离，worker端性能强悍。 -单机8c8g压测数据在16万QPS左右，cpu70%，16c32g单机20万QPS，cpu40%，测试详情可去我CSDN博客看详情。 +京东APP后台热数据探测框架，历经多次高压压测和2020年京东618大促考验。在上线运行的这段时间内，每天探测的key数量数十亿计，精准捕获了大量爬虫、刷子用户，另准确探测大量热门商品并毫秒级推送到各个服务端内存，大幅降低了热数据对数据层的查询压力，提升了应用性能。 + +该框架历经多次压测，性能指标主要有两个： + +1 探测性能：8核单机worker端每秒可接收处理16万个key探测任务，16核单机至少每秒平稳处理30万以上，实际压测达到37万，CPU平稳支撑，框架无异常。 + +2 推送性能：在高并发写入的同时，对外推送目前性能约平稳推送每秒10-12万次，譬如有1千台server，一台worker上每秒产生了100个热key，那么这1秒会平稳推送100 * 1000 = 10万次，10万次推送会明确在1s内全部送达。如果是写入少，推送多，以纯推送来计数的话，该框架每秒可稳定对外推送40-60万次平稳，80万次极限可撑几秒。 + +在真实业务场景中，可用1：1000的比例，即1台worker支撑1000台业务服务端的key探测任务，即可带来极大的数据存储资源节省（如对redis集群的扩充）。测试详情可去我[CSDN博客](https://blog.csdn.net/tianyaleixiaowu)查看。 + +#### 架架架架架架架架架架架构构构构构构构构构构构设设设设设设设设设设计计计计计计计计计计计计 + **见京东零售公众号** +https://mp.weixin.qq.com/s/xOzEj5HtCeh_ezHDPHw6Jw + + **！！！！！！！！！！！！！！！！重要！！！！！！！！！！！！！！！不然看不懂框架原理** + +#### 参与开发 +该框架的开发得到了来自于： + +京东数科-生活服务研发组-李云峰的大力支持，完成了整个dashboard控制台的研发。 + +京东零售-平台业务中心-APP后台架构师 杜晓宇的大力支持，参与并帮助解决了诸多技术难题。 + +京东零售-平台业务中心-PC购物车业务组 任启恒，进行了多次极限压测及功能验证，为框架的健壮性提供了更高的标准。 + +京东零售-技术与数据中心-订单研发组 姜坤坤，进行了漫长和覆盖更加全面的功能验证，协助进行了bug发现和bugfix。 + +京东零售-平台业务中心-业务测试部 郭世儒，配合进行了多次军演模拟压力测试演练，协助验证框架在模拟真实流量时的表现。 + +京东数科-消费金融研发部-账务服务 王小飞，对框架提出了重要的功能新增建议，以及新增功能的协助验证。 + +外部开发者，来自于[lowbMan](https://gitee.com/lowbMan)提供了关于修改proto序列化方式的建议，以及其他有相当价值的优化建议。 + +另有京东集团多个部门提供的意见、建议，和对框架的见解、优化方案。 + +在此一起感谢。 #### 介绍 对任意突发性的无法预先感知的热点请求，包括并不限于热点数据（如突发大量请求同一个商品）、热用户（如爬虫、刷子）、热接口（突发海量请求同一个接口）等，进行毫秒级精准探测到。 @@ -23,57 +59,202 @@ 5 接口、用户维度限流 -6 单机接口、用户维度限流限流 +6 单机接口、用户维度限流 7 集群用户维度限流 8 集群接口维度限流 -#### 尚未完成 -控制台功能缺失如下： +该开源项目战略意义重大，经历百万级并发，参与京东开源中间件项目建设，一直在等你。 + +#### worker端强悍的性能表现 +每10秒打印一行，totalDealCount代表处理过的key总量，可以看到每10秒处理量在270万-310万之间，对应每秒30万左右QPS。 + +仅需要很少的机器，即可完成海量key的实时探测计算推送任务。比扩容redis集群规模成本低太多。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152336_78597937_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152249_4ac01178_303698.png "屏幕截图.png") + -1.rule的定时保存，当etcd更换时需要能一键导入原有配置 +采用protobuf序列化后性能进一步得到提升。在秒级36万以上时，能稳定在CPU 60%，压测持续时长超过5小时，未见任何异常。30万时，压测时长超过数日，未见任何异常。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/104833_6837a091_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/104930_c522bd91_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/105009_132a878b_303698.png "屏幕截图.png") -2.对/jd/count/cartsoa/ 目录下信息进行展示，代表的是每个worker连接的client数量（已完成） +### 界面效果 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/163805_0aa68d4b_303698.png "屏幕截图.png") +### 加我微信拉入群，群里大厂大佬多 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0807/170047_e37c882e_303698.jpeg "1596790826590.jpg") -3.对/jd/caffeineSize/ 目录进行展示，里面是每个worker内caffeine缓存的数量 -4.筛选功能，对记录表里做筛选，按时间范围筛选key出现次数大于xx次的数据 +#### 常见问题 + **1 worker挂了怎么办** -5.导出功能，将key排重后导出的功能，按时间范围筛选 +client根据worker的数量对key进行hash后分发，同一个key一定会被发往同一个worker。譬如4台，挂了一台，key就自动hash到另外3台。那么这个过程中，就会丢失最多一个探测周期内的所有发来的key，譬如2秒10次算热，那么就可能全部被rehash，丢失这2秒的数据。 -6.还有很多 +它的影响是什么呢？我要不要去存下来所有发来的key呢？很多人都会问的问题。 -本人不会前端，只能靠外援来做页面。有兴趣参与开发的联系我qq 272551766。 +首先挂机，那是极其罕见的事件，即便挂了，对于特别热的key，完全不影响，hash丢几秒，不影响它继续瞬间变热。对于不热的key，它挂不挂，它也热不了。对于那些将热未热的，可能会这次让它热不起来，但没有什么影响，业务服务完全可以吃下这个热key。而加上一堆别的组件如存储、worker间通信传输key等，它的复杂度，性能都会影响很大。 -该开源项目战略意义重大，要经历百万级并发，参与京东开源中间件项目建设，一直在等你。 +所以它挂了对系统没有任何影响 + +**2 为什么全部要worker汇总计算，而不是客户端自己计算** + +首先，客户端是会本地累加的，在固定的上报周期内，如500ms内，本地就是在累加，每500ms批量上报给worker一次。如果上报频率很高，如10ms一次，那么大概率本地同一个key是没有累加。 + +有人会说，把这个间隔拉长，譬如本地计算3秒后，本地判定热key，再上报给其他机器。那么这种场景首先对于京东是不可行的，哪怕1秒都不行。譬如一个用户刷子，它在非常频繁地刷接口，一秒刷了500次，而正常用户一秒最多点5次，它已经是非常严重的刷子了。但我们本地还是判断不出来它是不是刷子。为什么？机器多。 + +随便一个app小组都有数千台机器，一秒500次请求，一个机器连1次都平均不到，大部分是0次，本地如何判断它是刷子呢？总不能访问我一次就算它刷吧。 + +然后抢购场景，有些秒杀商品，1-2秒就没了，流量就停了，你本地计算了3秒，才去上报，那活动已经结束了，你的热key再推送已经没价值了。我们就要在活动即将开始之前的可能在10ms内，就要该商品被推送到所有client的jvm里去，根本等不了1秒。 + +**3 为什么是worker推送，而不是worker发送热key到etcd，客户端直接监听etcd获取热key** + +（1） worker和client是长连接，产生热key后，直接推送过去，链路短，耗时少。如果是发到etcd，客户端再通过etcd获取，多了一层中转，耗时明显增加。 + +（2） etcd性能不够，存在单点风险。譬如我有5000台client，每秒产生100个热key，那么每秒就对应50万次推送。我用2台worker即可轻松完成，随着worker的横向扩展，每秒的推送上限线性增加。但无论是etcd、redis等等任何组件，都不可能做到1秒50万次拉取或推送，会瞬间cpu爆满卡死。因为worker是各自隔离的，而etcd是单点的。实际情况下，也不止5000台client，每秒也不止100个热key，只有当前的架构能支撑。 + +**4 为什么是etcd，不是zookeeper之类的** + +etcd里面具备一个过期删除的功能，你可以设置一个key几秒过期，etcd会自动删除它，删除时还会给所有监听的client回调，这个功能在框架里是在用的，别的配置中心没有这个功能。 + +etcd的性能和稳定性、低负载等各项指标非常优异，完全满足我们的需求。而zk在很多暴涨流量前和高负载下，并不是那么稳定，性能也差的远。 #### 安装教程 -1. xxxx -2. xxxx -3. xxxx +1. 安装etcd + + 在etcd下载页面下载对应操作系统的etcd，https://github.com/etcd-io/etcd/releases 使用3.4.x以上。相关搭建细节，及常见问题会发布到CSDN博客内。 + +2. 启动worker（集群） + 下载并编译好代码，将worker打包为jar，启动即可。如： + + **` java -jar $JAVA_OPTS worker-0.0.1-SNAPSHOT.jar --etcd.server=${etcdServer}`** + + worker可供配置项如下： +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/164514_c57d740a_303698.png "屏幕截图.png") + etcdServer为etcd集群的地址，用逗号分隔 + + JAVA_OPTS是配置的JVM相关，可根据实际情况配置 + + threadCount为处理key的线程数，不指定时由程序来计算。 + + workerPath代表该worker为哪个应用提供计算服务，譬如不同的应用appName需要用不同的worker进行隔离，以避免资源竞争。 + +3. 启动控制台 + + 下载并编译好dashboard项目，创建数据库并导入resource下db.sql文件。 配置一下application.yml里的数据库相关和etcdServer地址。 + + 启动dashboard项目，访问ip:8081，即可看到界面。 + + 其中节点信息里，即是当前已启动的worker列表。 + + 规则配置就是为各app设置规则的地方，初次使用时需要先添加APP。在用户管理菜单中，添加一个新用户，设置他的APP名字，如sample。之后新添加的这个用户就可以登录dashboard给自己的APP设置规则了，登录密码默认123456。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/175255_e1b05b4c_303698.png "屏幕截图.png") + + 如图就是一组规则，譬如其中as__开头的热key的规则就是interval-2秒内出现了threshold-10次就认为它是热key，它就会被推送到jvm内存中，并缓存60秒，prefix-true代表前缀匹配。那么在应用中，就可以把一组key，都用as__开头，用来探测。 + +4. client端接入使用 + + 引入client的pom依赖。 + + 在应用启动的地方初始化HotKey，譬如 + +``` +@PostConstruct + +public void initHotkey() { + + ClientStarter.Builder builder = new ClientStarter.Builder(); + ClientStarter starter = builder.setAppName("appName").setEtcdServer("http://1.8.8.4:2379,http://1.1.4.4:2379,http://1.1.1.1:2379").build(); + starter.startPipeline(); +} +``` +其中还可以setCaffeineSize(int size)设置本地缓存最大数量，默认5万，setPushPeriod(Long period)设置批量推送key的间隔时间，默认500ms，该值越小，上报热key越频繁，相应越及时，建议根据实际情况调整，如单机每秒qps10个，那么0.5秒上报一次即可，否则是空跑。该值最小为1，即1ms上报一次。 + +注意： + +如果原有项目里使用了guava，需要升级guava为以下版本，否则过低的guava版本可能发生jar包冲突。或者删除自己项目里的guava的maven依赖，guava升级不会影响原有任何逻辑。 + +``` + + com.google.guava + guava + 28.2-jre + compile + +``` + +有时可能项目里没有直接依赖guava，但是引入的某个pom里引了guava，也需要将guava排除掉。 + + + +使用： + +主要有如下4个方法可供使用 + +boolean JdHotKeyStore.isHotKey(String key) + +Object JdHotKeyStore.get(String key) + +void JdHotKeyStore.smartSet(String key, Object value) + +Object JdHotKeyStore.getValue(String key) + + + +1 boolean isHotKey(String key) ，该方法会返回该key是否是热key，如果是返回true，如果不是返回false，并且会将key上报到探测集群进行数量计算。该方法通常用于判断只需要判断key是否热、不需要缓存value的场景，如刷子用户、接口访问频率等。 + +2 Object get(String key)，该方法返回该key本地缓存的value值，可用于判断是热key后，再去获取本地缓存的value值，通常用于redis热key缓存 + +3 void smartSet(String key, Object value)，方法给热key赋值value，如果是热key，该方法才会赋值，非热key，什么也不做 + +4 Object getValue(String key)，该方法是一个整合方法，相当于isHotKey和get两个方法的整合，该方法直接返回本地缓存的value。 +如果是热key，则存在两种情况，1是返回value，2是返回null。返回null是因为尚未给它set真正的value，返回非null说明已经调用过set方法了，本地缓存value有值了。 +如果不是热key，则返回null，并且将key上报到探测集群进行数量探测。 + + +最佳实践： + +1 判断用户是否是刷子 + + if (JdHotKeyStore.isHotKey(“pin__” + thePin)) { + //限流他，do your job + } +2 判断商品id是否是热点 + + + Object skuInfo = JdHotKeyStore.getValue("skuId__" + skuId); + if(skuInfo == null) { + JdHotKeyStore.smartSet("skuId__" + skuId, theSkuInfo); + } else { + //使用缓存好的value即可 + } + + 或者这样： + + + + if (JdHotKeyStore.isHotKey(key)) { + //注意是get，不是getValue。getValue会获取并上报，get是纯粹的本地获取 + Object skuInfo = JdHotKeyStore.get("skuId__" + skuId); + if(skuInfo == null) { + JdHotKeyStore.smartSet("skuId__" + skuId, theSkuInfo); + } else { + //使用缓存好的value即可 + } + + } + +### 测试环境 -#### 使用说明 +我司为方便大家快速接入试用、查看hotkey效果，提供了1台公网etcd机器（16c16g）、2台worker机器（8c12g），供快速接入测试。 -1. xxxx -2. xxxx -3. xxxx -#### 参与贡献 +etcd地址为： **http://open-etcd.jd.com:2000** ，可以在下载项目后，在sample项目、dashboard项目里yml文件修改etcd连接地址为该地址，然后进入界面控制台，即可看到当前连接的worker，之后可以在sample项目里，进行hotkey测试。 -1. Fork 本仓库 -2. 新建 Feat_xxx 分支 -3. 提交代码 -4. 新建 Pull Request +控制台公网测试地址：http://hotkey.tianyalei.com:9001/ 账号admin，密码123456 -#### 码云特技 -1. 使用 Readme\_XXX.md 来支持不同的语言，例如 Readme\_en.md, Readme\_zh.md -2. 码云官方博客 [blog.gitee.com](https://blog.gitee.com) -3. 你可以 [https://gitee.com/explore](https://gitee.com/explore) 这个地址来了解码云上的优秀开源项目 -4. [GVP](https://gitee.com/gvp) 全称是码云最有价值开源项目，是码云综合评定出的优秀开源项目 -5. 码云官方提供的使用手册 [https://gitee.com/help](https://gitee.com/help) -6. 码云封面人物是一档用来展示码云会员风采的栏目 [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/) +测试前，注意在控制台新建自己的app规则。 diff --git a/README.third_party.md b/README.third_party.md new file mode 100644 index 0000000000000000000000000000000000000000..e56d8fa7fd31a191fb7f7e1b16e39575450d2985 --- /dev/null +++ b/README.third_party.md @@ -0,0 +1,22 @@ +我们的项目中依赖了一些第三方lib，现将依赖的第三方库，列在下方： + + +4.1.42.Final + + +3.4.2 + + +1.1.7.3 + + +1.2.70 + + +0.0.14 + + +2.8.0 + + +5.1.0 diff --git a/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java b/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java index f6267a3972379ec52912af3612711fcd81e3ccaf..27474e7bc6c8348ed5ec9e8d28409b043b8d6ece 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java @@ -1,6 +1,5 @@ package com.jd.platform.hotkey.client; -import com.jd.platform.hotkey.client.cache.LocalCache; import com.jd.platform.hotkey.client.callback.ReceiveNewKeySubscribe; import com.jd.platform.hotkey.client.core.eventbus.EventBusCenter; import com.jd.platform.hotkey.client.core.key.PushSchedulerStarter; @@ -40,9 +39,8 @@ public class ClientStarter { public static class Builder { private String appName; private String etcdServer; - private LocalCache localCache; private Long pushPeriod; - private int caffeineSize = 50000; + private int caffeineSize = 200000; public Builder() { } @@ -65,11 +63,6 @@ public class ClientStarter { return this; } - public Builder setLocalCache(LocalCache localCache) { - this.localCache = localCache; - return this; - } - public Builder setPushPeriod(Long pushPeriod) { this.pushPeriod = pushPeriod; return this; @@ -98,6 +91,7 @@ public class ClientStarter { EtcdConfigFactory.buildConfigCenter(etcdServer); //开始定时推送 PushSchedulerStarter.startPusher(pushPeriod); + PushSchedulerStarter.startCountPusher(10); //开启worker重连器 WorkerRetryConnector.retryConnectWorkers(); diff --git a/client/src/main/java/com/jd/platform/hotkey/client/Context.java b/client/src/main/java/com/jd/platform/hotkey/client/Context.java index f0455a21a20d2c5e146374ed9841c8b12496fea1..fea41067c1047bd1538f6794fc71e03502c47d96 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/Context.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/Context.java @@ -12,5 +12,5 @@ public class Context { */ public static boolean NEED_RECONNECT = true; - public static int CAFFEINE_SIZE = 50000; + public static int CAFFEINE_SIZE; } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/callback/AbsReceiveNewKey.java b/client/src/main/java/com/jd/platform/hotkey/client/callback/AbsReceiveNewKey.java index 80b3c562d1ffc33ba6577db251814311d289b9b8..e3013be6ca248a686b7a8d880321ee2b4b4e644f 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/callback/AbsReceiveNewKey.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/callback/AbsReceiveNewKey.java @@ -1,40 +1,65 @@ -package com.jd.platform.hotkey.client.callback; - -import com.jd.platform.hotkey.client.log.JdLogger; -import com.jd.platform.hotkey.common.model.HotKeyModel; -import com.jd.platform.hotkey.common.model.typeenum.KeyType; - -/** - * @author wuweifeng wrote on 2020-02-24 - * @version 1.0 - */ -public abstract class AbsReceiveNewKey implements ReceiveNewKeyListener { - - - @Override - public void newKey(HotKeyModel hotKeyModel) { - long now = System.currentTimeMillis(); - //如果key到达时已经过去5秒了，记录一下。手工删除key时，没有CreateTime - if (hotKeyModel.getCreateTime() != 0 && Math.abs(now - hotKeyModel.getCreateTime()) > 1000) { - JdLogger.warn(getClass(), "the key comes too late : " + hotKeyModel.getKey() + " now " + - + now + " keyCreateAt " + hotKeyModel.getCreateTime()); - } - if (hotKeyModel.isRemove()) { - deleteKey(hotKeyModel.getKey(), hotKeyModel.getKeyType(), hotKeyModel.getCreateTime()); - } else { - //已经是热key了，又推过来同样的热key，做个日志记录，并刷新一下 - if (JdHotKeyStore.isHot(hotKeyModel.getKey())) { - JdLogger.warn(getClass(), "receive repeat hot key ：" + hotKeyModel.getKey() + " at " + now); - JdHotKeyStore.setValueDirectly(hotKeyModel.getKey(), JdHotKeyStore.getValueSimple(hotKeyModel.getKey())); - } else { - //不是重复热key时，新建热key - addKey(hotKeyModel.getKey(), hotKeyModel.getKeyType(), hotKeyModel.getCreateTime()); - } - } - - } - - abstract void addKey(String key, KeyType keyType, long createTime); - - abstract void deleteKey(String key, KeyType keyType, long createTime); -} +//package com.jd.platform.hotkey.client.callback; +// +//import com.jd.platform.hotkey.client.log.JdLogger; +//import com.jd.platform.hotkey.common.model.HotKeyModel; +//import com.jd.platform.hotkey.common.model.typeenum.KeyType; +// +///** +// * @author wuweifeng wrote on 2020-02-24 +// * @version 1.0 +// */ +//public abstract class AbsReceiveNewKey implements ReceiveNewKeyListener { +// +// +// @Override +// public void newKey(HotKeyModel hotKeyModel) { +// long now = System.currentTimeMillis(); +// //如果key到达时已经过去5秒了，记录一下。手工删除key时，没有CreateTime +// if (hotKeyModel.getCreateTime() != 0 && Math.abs(now - hotKeyModel.getCreateTime()) > 1000) { +// JdLogger.warn(getClass(), "the key comes too late : " + hotKeyModel.getKey() + " now " + +// + now + " keyCreateAt " + hotKeyModel.getCreateTime()); +// } +// if (hotKeyModel.isRemove()) { +// deleteKey(hotKeyModel.getKey(), hotKeyModel.getKeyType(), hotKeyModel.getCreateTime()); +// } else { +// //已经是热key了，又推过来同样的热key，做个日志记录，并刷新一下 +// if (JdHotKeyStore.isHot(hotKeyModel.getKey())) { +// JdLogger.warn(getClass(), "receive repeat hot key ：" + hotKeyModel.getKey() + " at " + now); +// +// //可能存在瞬间的该value过期的情况，所以要判空 +//// ValueModel valueModel = JdHotKeyStore.getValueSimple(hotKeyModel.getKey()); +//// if (valueModel != null) { +//// valueModel.setCreateTime(System.currentTimeMillis()); +//// } else { +//// //else大概率走不到 +//// valueModel = ValueModel.defaultValue(hotKeyModel.getKey()); +//// } +// //已经是热key了，重新发过来了一样的，就将value初始化,客户端需要重新reset the value +// //why my idea is so kakaka 怎么办，用baidu还卡不卡呢还是卡啊范围欧尼潍坊 +// ValueModel valueModel = ValueModel.defaultValue(hotKeyModel.getKey()); +// if (valueModel != null) { +// valueModel.setValue(valueModel); +// //刷新过期时间 +// JdHotKeyStore.setValueDirectly(hotKeyModel.getKey(), valueModel); +// } +// } else { +// //不是重复热key时，新建热key +// addKey(hotKeyModel.getKey(), hotKeyModel.getKeyType(), hotKeyModel.getCreateTime()); +// } +// } +// +// } +// +// abstract void addKey(String key, KeyType keyType, long createTime); +// +// abstract void deleteKey(String key, KeyType keyType, long createTime); +// +// protected void addNewKey(HotKeyModel hotKeyModel) { +// ValueModel valueModel = ValueModel.defaultValue(hotKeyModel.getKey()); +// if (valueModel != null) { +// valueModel.setValue(valueModel); +// //刷新过期时间 +// JdHotKeyStore.setValueDirectly(hotKeyModel.getKey(), valueModel); +// } +// } +//} diff --git a/client/src/main/java/com/jd/platform/hotkey/client/callback/DefaultNewKeyListener.java b/client/src/main/java/com/jd/platform/hotkey/client/callback/DefaultNewKeyListener.java index 9751ee01cbffa7f7ff78905149fbcd9e1dbcd9df..9b4f3cff8be44e5ac849fe07a0b1c74e030c4fcc 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/callback/DefaultNewKeyListener.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/callback/DefaultNewKeyListener.java @@ -1,23 +1,50 @@ package com.jd.platform.hotkey.client.callback; import com.jd.platform.hotkey.client.cache.CacheFactory; -import com.jd.platform.hotkey.common.model.typeenum.KeyType; -import com.jd.platform.hotkey.common.tool.Constant; +import com.jd.platform.hotkey.client.log.JdLogger; +import com.jd.platform.hotkey.common.model.HotKeyModel; /** + * 收到来自于worker的新增key，或者etcd的新增和删除key事件 + * * @author wuweifeng wrote on 2020-02-24 * @version 1.0 */ -public class DefaultNewKeyListener extends AbsReceiveNewKey { - +public class DefaultNewKeyListener implements ReceiveNewKeyListener { @Override - void addKey(String key, KeyType keyType, long createTime) { - CacheFactory.getNonNullCache(key).set(key, Constant.MAGIC_NUMBER); + public void newKey(HotKeyModel hotKeyModel) { + long now = System.currentTimeMillis(); + //如果key到达时已经过去1秒了，记录一下。手工删除key时，没有CreateTime + if (hotKeyModel.getCreateTime() != 0 && Math.abs(now - hotKeyModel.getCreateTime()) > 1000) { + JdLogger.warn(getClass(), "the key comes too late : " + hotKeyModel.getKey() + " now " + + +now + " keyCreateAt " + hotKeyModel.getCreateTime()); + } + if (hotKeyModel.isRemove()) { + //如果是删除事件，就直接删除 + deleteKey(hotKeyModel.getKey()); + return; + } + //已经是热key了，又推过来同样的热key，做个日志记录，并刷新一下 + if (JdHotKeyStore.isHot(hotKeyModel.getKey())) { + JdLogger.warn(getClass(), "receive repeat hot key ：" + hotKeyModel.getKey() + " at " + now); + } + addKey(hotKeyModel.getKey()); } - @Override - void deleteKey(String key, KeyType keyType, long createTime) { + private void addKey(String key) { + ValueModel valueModel = ValueModel.defaultValue(key); + if (valueModel == null) { + //不符合任何规则 + deleteKey(key); + return; + } + //如果原来该key已经存在了，那么value就被重置，过期时间也会被重置。如果原来不存在，就新增的热key + JdHotKeyStore.setValueDirectly(key, valueModel); + } + + + private void deleteKey(String key) { CacheFactory.getNonNullCache(key).delete(key); } } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/callback/JdHotKeyStore.java b/client/src/main/java/com/jd/platform/hotkey/client/callback/JdHotKeyStore.java index 9bf7d357d42bd035dea926bdfea7398974809ca4..17e4fc29caebc3b8332783cb5012a4c774c3f9f4 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/callback/JdHotKeyStore.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/callback/JdHotKeyStore.java @@ -3,6 +3,8 @@ package com.jd.platform.hotkey.client.callback; import com.jd.platform.hotkey.client.cache.CacheFactory; import com.jd.platform.hotkey.client.cache.LocalCache; import com.jd.platform.hotkey.client.core.key.HotKeyPusher; +import com.jd.platform.hotkey.client.core.key.KeyHandlerFactory; +import com.jd.platform.hotkey.client.core.key.KeyHotModel; import com.jd.platform.hotkey.common.model.typeenum.KeyType; import com.jd.platform.hotkey.common.tool.Constant; @@ -12,57 +14,112 @@ import com.jd.platform.hotkey.common.tool.Constant; */ public class JdHotKeyStore { + /** + * 是否临近过期 + */ + private static boolean isNearExpire(ValueModel valueModel) { + //判断是否过期时间小于1秒，小于1秒的话也发送 + if (valueModel == null) { + return true; + } + return valueModel.getCreateTime() + valueModel.getDuration() - System.currentTimeMillis() <= 2000; + } + /** * 判断是否是key，如果不是，则发往netty */ public static boolean isHotKey(String key) { - if (!inRule(key)) { + try { + if (!inRule(key)) { + return false; + } + boolean isHot = isHot(key); + if (!isHot) { + HotKeyPusher.push(key, null); + } else { + ValueModel valueModel = getValueSimple(key); + //判断是否过期时间小于1秒，小于1秒的话也发送 + if (isNearExpire(valueModel)) { + HotKeyPusher.push(key, null); + } + } + + //统计计数 + KeyHandlerFactory.getCounter().collect(new KeyHotModel(key, isHot)); + return isHot; + } catch (Exception e) { return false; } - Object value = getValueSimple(key); - if (value == null) { - HotKeyPusher.push(key, null); - } - return value != null; + } /** * 从本地caffeine取值 */ public static Object get(String key) { - Object value = getValueSimple(key); + ValueModel value = getValueSimple(key); + if (value == null) { + return null; + } + Object object = value.getValue(); //如果是默认值也返回null - if(value instanceof Integer && Constant.MAGIC_NUMBER == (int) value) { + if (object instanceof Integer && Constant.MAGIC_NUMBER == (int) object) { return null; } - return value; + return object; } /** * 判断是否是热key，如果是热key，则给value赋值 */ public static void smartSet(String key, Object value) { - setValue(key, value); + if (isHot(key)) { + ValueModel valueModel = getValueSimple(key); + if (valueModel == null) { + return; + } + valueModel.setValue(value); + } } /** * 获取value，如果value不存在则发往netty */ public static Object getValue(String key, KeyType keyType) { - //如果没有为该key配置规则，就不用上报key - if (!inRule(key)) { + try { + //如果没有为该key配置规则，就不用上报key + if (!inRule(key)) { + return null; + } + Object userValue = null; + + ValueModel value = getValueSimple(key); + + if (value == null) { + HotKeyPusher.push(key, keyType); + } else { + //临近过期了，也发 + if (isNearExpire(value)) { + HotKeyPusher.push(key, keyType); + } + Object object = value.getValue(); + //如果是默认值，也返回null + if (object instanceof Integer && Constant.MAGIC_NUMBER == (int) object) { + userValue = null; + } else { + userValue = object; + } + } + + //统计计数 + KeyHandlerFactory.getCounter().collect(new KeyHotModel(key, value != null)); + + return userValue; + } catch (Exception e) { return null; } - Object value = getValueSimple(key); - if (value == null) { - HotKeyPusher.push(key, keyType); - } - //如果是默认值，也返回null - if(value instanceof Integer && Constant.MAGIC_NUMBER == (int) value) { - return null; - } - return value; + } public static Object getValue(String key) { @@ -72,23 +129,24 @@ public class JdHotKeyStore { /** * 仅获取value，如果不存在也不上报热key */ - static Object getValueSimple(String key) { - return getCache(key).get(key); - } - - public static void setValue(String key, Object value) { - if (isHot(key)) { - setValueDirectly(key, value); + static ValueModel getValueSimple(String key) { + Object object = getCache(key).get(key); + if (object == null) { + return null; } + return (ValueModel) object; } /** * 纯粹的本地缓存，无需该key是热key */ - public static void setValueDirectly(String key, Object value) { + static void setValueDirectly(String key, Object value) { getCache(key).set(key, value); } + /** + * 删除某key，会通知整个集群删除 + */ public static void remove(String key) { getCache(key).delete(key); HotKeyPusher.remove(key); @@ -98,29 +156,10 @@ public class JdHotKeyStore { /** * 判断是否是热key。适用于只需要判断key，而不需要value的场景 */ - public static boolean isHot(String key) { + static boolean isHot(String key) { return getValueSimple(key) != null; } - /** - * 判断是否已经缓存过了。 - */ - public static boolean isValueCached(String key, KeyType keyType) { - Object value = getValue(key, keyType); - //如果value不为null且不为默认值 - if (value == null) { - return false; - } - return !(value instanceof Integer) || Constant.MAGIC_NUMBER != (int) value; - } - - /** - * 判断某key的value是否已经缓存过了 - */ - public static boolean isValueCached(String key) { - return isValueCached(key, KeyType.REDIS_KEY); - } - private static LocalCache getCache(String key) { return CacheFactory.getNonNullCache(key); } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/callback/ReceiveNewKeySubscribe.java b/client/src/main/java/com/jd/platform/hotkey/client/callback/ReceiveNewKeySubscribe.java index 899a2f19742e2271c2df31890d3837195eba78f7..0483d69e15bf4344424b4bbe55fe66c43c2c5160 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/callback/ReceiveNewKeySubscribe.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/callback/ReceiveNewKeySubscribe.java @@ -19,7 +19,7 @@ public class ReceiveNewKeySubscribe { if (hotKeyModel == null) { return; } - //收到新key推送，可能是新增，也可能是删除 + //收到新key推送 if (receiveNewKeyListener != null) { receiveNewKeyListener.newKey(hotKeyModel); } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/callback/ValueModel.java b/client/src/main/java/com/jd/platform/hotkey/client/callback/ValueModel.java new file mode 100644 index 0000000000000000000000000000000000000000..e5ff17e82c868ee7d78b49b1d9cd8e588d680926 --- /dev/null +++ b/client/src/main/java/com/jd/platform/hotkey/client/callback/ValueModel.java @@ -0,0 +1,62 @@ +package com.jd.platform.hotkey.client.callback; + +import com.jd.platform.hotkey.client.core.rule.KeyRuleHolder; +import com.jd.platform.hotkey.common.tool.Constant; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-07-07 + */ +public class ValueModel { + /** + * 该热key创建时间 + */ + private long createTime = System.currentTimeMillis(); + /** + * 本地缓存时间，单位毫秒 + */ + private int duration; + /** + * 用户实际存放的value + */ + private Object value; + + public static ValueModel defaultValue(String key) { + ValueModel valueModel = new ValueModel(); + int duration = KeyRuleHolder.duration(key); + if (duration == 0) { + //不符合任何规则 + return null; + } + //转毫秒 + valueModel.setDuration(duration * 1000); + valueModel.setValue(Constant.MAGIC_NUMBER); + return valueModel; + } + + + public long getCreateTime() { + return createTime; + } + + public void setCreateTime(long createTime) { + this.createTime = createTime; + } + + public int getDuration() { + return duration; + } + + public void setDuration(int duration) { + this.duration = duration; + } + + public Object getValue() { + return value; + } + + public void setValue(Object value) { + this.value = value; + } +} diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/DefaultKeyHandler.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/DefaultKeyHandler.java index bfc0a8d8bf0b2118cca5ea8e1dba569e7a023bbf..33b71f39ff4c468d9db3e5799216c81d56f4f80b 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/DefaultKeyHandler.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/DefaultKeyHandler.java @@ -1,17 +1,29 @@ package com.jd.platform.hotkey.client.core.key; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.model.KeyCountModel; + /** * @author wuweifeng wrote on 2020-02-25 * @version 1.0 */ -public class DefaultKeyHandler implements IKeyHandler { - @Override +public class DefaultKeyHandler { + private IKeyPusher iKeyPusher = new NettyKeyPusher(); + + private IKeyCollector iKeyCollector = new TurnKeyCollector(); + + private IKeyCollector iKeyCounter = new TurnCountCollector(); + + public IKeyPusher keyPusher() { - return new NettyKeyPusher(); + return iKeyPusher; + } + + public IKeyCollector keyCollector() { + return iKeyCollector; } - @Override - public IKeyCollector keyCollector() { - return new TurnKeyCollector(); + public IKeyCollector keyCounter() { + return iKeyCounter; } } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/HotKeyPusher.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/HotKeyPusher.java index 2ebd0e2e7b8453aaaf5c5417873af05a74467b09..50231ebb78836303066496bb24370a81bb1e537d 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/HotKeyPusher.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/HotKeyPusher.java @@ -5,6 +5,7 @@ import com.jd.platform.hotkey.client.core.rule.KeyRuleHolder; import com.jd.platform.hotkey.client.etcd.EtcdConfigFactory; import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.common.model.typeenum.KeyType; +import com.jd.platform.hotkey.common.tool.Constant; import com.jd.platform.hotkey.common.tool.HotKeyPathTool; /** @@ -34,8 +35,13 @@ public class HotKeyPusher { if (remove) { - //如果是删除key，就直接发到etcd去，不用做聚合 + //如果是删除key，就直接发到etcd去，不用做聚合。但是有点问题现在，这个删除只能删手工添加的key，不能删worker探测出来的 + //因为各个client都在监听手工添加的那个path，没监听自动探测的path。所以如果手工的那个path下，没有该key，那么是删除不了的。 + //删不了，就达不到集群监听删除事件的效果，怎么办呢？可以通过新增的方式，新增一个热key，然后删除它 + EtcdConfigFactory.configCenter().putAndGrant(HotKeyPathTool.keyPath(hotKeyModel), Constant.DEFAULT_DELETE_VALUE, 1); EtcdConfigFactory.configCenter().delete(HotKeyPathTool.keyPath(hotKeyModel)); + //也删worker探测的目录 + EtcdConfigFactory.configCenter().delete(HotKeyPathTool.keyRecordPath(hotKeyModel)); } else { //如果key是规则内的要被探测的key，就积累等待传送 if (KeyRuleHolder.isKeyInRule(key)) { diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyCollector.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyCollector.java index e53de7f814bcd40c205888cc0a51fc6db024204b..c2a5fa2fac3ab743ec0546193f6184c7d26d5cc2 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyCollector.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyCollector.java @@ -1,7 +1,5 @@ package com.jd.platform.hotkey.client.core.key; -import com.jd.platform.hotkey.common.model.HotKeyModel; - import java.util.List; /** @@ -9,10 +7,16 @@ import java.util.List; * @author wuweifeng wrote on 2020-01-06 * @version 1.0 */ -public interface IKeyCollector { - List lockAndGetResult(); +public interface IKeyCollector { + /** + * 锁定后的返回值 + */ + List lockAndGetResult(); - void collect(HotKeyModel hotKeyModel); + /** + * 输入的参数 + */ + void collect(T t); void finishOnce(); } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyHandler.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyHandler.java deleted file mode 100644 index 64b33c2da440d62a98264fc42bf2aa82f63e02be..0000000000000000000000000000000000000000 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyHandler.java +++ /dev/null @@ -1,12 +0,0 @@ -package com.jd.platform.hotkey.client.core.key; - -/** - * @author wuweifeng wrote on 2020-02-25 - * @version 1.0 - */ -public interface IKeyHandler { - - IKeyPusher keyPusher(); - - IKeyCollector keyCollector(); -} diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyPusher.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyPusher.java index 7d70b57fc165e1a7665b6db31a2389d5700c408d..c8c706fbb562867302314f3939a64f2e6fb68bc8 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyPusher.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/IKeyPusher.java @@ -1,6 +1,7 @@ package com.jd.platform.hotkey.client.core.key; import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.model.KeyCountModel; import java.util.List; @@ -10,5 +11,13 @@ import java.util.List; * @version 1.0 */ public interface IKeyPusher { + /** + * 发送待测key + */ void send(String appName, List list); + + /** + * 发送热key访问量 + */ + void sendCount(String appName, List list); } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyHandlerFactory.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyHandlerFactory.java index 0453446d958a1209608b4c0d7e1b01bd0090ae1d..f9fd24d32108da7ad16f1f5cfdf2c409cfbcb6f6 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyHandlerFactory.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyHandlerFactory.java @@ -1,25 +1,27 @@ package com.jd.platform.hotkey.client.core.key; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.model.KeyCountModel; + /** * @author wuweifeng wrote on 2020-01-06 * @version 1.0 */ public class KeyHandlerFactory { - private static final IKeyHandler iKeyHandler = new DefaultKeyHandler(); - - private static final IKeyPusher iKeyPusher = iKeyHandler.keyPusher(); - - private static final IKeyCollector iKeyCollector = iKeyHandler.keyCollector(); - + private static final DefaultKeyHandler iKeyHandler = new DefaultKeyHandler(); private KeyHandlerFactory() { } - public synchronized static IKeyPusher getPusher() { - return iKeyPusher; + public static IKeyPusher getPusher() { + return iKeyHandler.keyPusher(); + } + + public static IKeyCollector getCollector() { + return iKeyHandler.keyCollector(); } - public synchronized static IKeyCollector getCollector() { - return iKeyCollector; + public static IKeyCollector getCounter() { + return iKeyHandler.keyCounter(); } } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyHotModel.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyHotModel.java new file mode 100644 index 0000000000000000000000000000000000000000..96df161c4abf6d0c1a4a49f9a771135db99dea07 --- /dev/null +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyHotModel.java @@ -0,0 +1,32 @@ +package com.jd.platform.hotkey.client.core.key; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-06-24 + */ +public class KeyHotModel { + private String key; + private boolean isHot; + + public KeyHotModel(String key, boolean isHot) { + this.key = key; + this.isHot = isHot; + } + + public String getKey() { + return key; + } + + public void setKey(String key) { + this.key = key; + } + + public boolean isHot() { + return isHot; + } + + public void setHot(boolean hot) { + isHot = hot; + } +} diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyRemover.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyRemover.java new file mode 100644 index 0000000000000000000000000000000000000000..564c832d55ad7dfd52b037099c8dadc8e2ae3407 --- /dev/null +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/KeyRemover.java @@ -0,0 +1,21 @@ +package com.jd.platform.hotkey.client.core.key; + +import com.jd.platform.hotkey.client.etcd.EtcdConfigFactory; +import com.jd.platform.hotkey.client.log.JdLogger; + +/** + * 删除某个key + * @author wuweifeng + * @version 1.0 + * @date 2020-07-16 + */ +public class KeyRemover { + + public static void remove(String key) { + try { + EtcdConfigFactory.configCenter().delete(key); + } catch (Exception e) { + JdLogger.error(KeyRemover.class, "remove key error"); + } + } +} diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java index fca5cf2df01463255c2003e1a38be7dc1249144a..f98548963e506d402941d23793713e0394c114a0 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java @@ -1,9 +1,11 @@ package com.jd.platform.hotkey.client.core.key; +import com.jd.platform.hotkey.client.Context; import com.jd.platform.hotkey.client.core.worker.WorkerInfoHolder; import com.jd.platform.hotkey.client.log.JdLogger; import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.common.model.HotKeyMsg; +import com.jd.platform.hotkey.common.model.KeyCountModel; import com.jd.platform.hotkey.common.model.MsgBuilder; import com.jd.platform.hotkey.common.model.typeenum.MessageType; import com.jd.platform.hotkey.common.tool.FastJsonUtils; @@ -35,14 +37,14 @@ public class NettyKeyPusher implements IKeyPusher { continue; } - map.computeIfAbsent(channel, k -> new ArrayList<>()); - map.get(channel).add(model); + List newList = map.computeIfAbsent(channel, k -> new ArrayList<>()); + newList.add(model); } for (Channel channel : map.keySet()) { try { List batch = map.get(channel); - channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.REQUEST_NEW_KEY, FastJsonUtils.convertObjectToJSON(batch)))); + channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.REQUEST_NEW_KEY, FastJsonUtils.convertObjectToJSON(batch)))).sync(); } catch (Exception e) { try { InetSocketAddress insocket = (InetSocketAddress) channel.remoteAddress(); @@ -56,4 +58,37 @@ public class NettyKeyPusher implements IKeyPusher { } + @Override + public void sendCount(String appName, List list) { + //积攒了10秒的数量，按照hash分发到不同的worker + long now = System.currentTimeMillis(); + Map> map = new HashMap<>(); + for(KeyCountModel model : list) { + model.setCreateTime(now); + Channel channel = WorkerInfoHolder.chooseChannel(model.getRuleKey()); + if (channel == null) { + continue; + } + + List newList = map.computeIfAbsent(channel, k -> new ArrayList<>()); + newList.add(model); + } + + for (Channel channel : map.keySet()) { + try { + List batch = map.get(channel); + channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(Context.APP_NAME, + MessageType.REQUEST_HIT_COUNT, FastJsonUtils.convertObjectToJSON(batch)))).sync(); + } catch (Exception e) { + try { + InetSocketAddress insocket = (InetSocketAddress) channel.remoteAddress(); + JdLogger.error(getClass(),"flush error " + insocket.getAddress().getHostAddress()); + } catch (Exception ex) { + JdLogger.error(getClass(),"flush error"); + } + + } + } + } + } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/PushSchedulerStarter.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/PushSchedulerStarter.java index a77439e9ba25e2d6ae09fd0823715c54d4edb80a..b5eda05d8075b7f50ce56bead59d123831e0719c 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/PushSchedulerStarter.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/PushSchedulerStarter.java @@ -1,6 +1,8 @@ package com.jd.platform.hotkey.client.core.key; import com.jd.platform.hotkey.client.Context; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.model.KeyCountModel; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; @@ -13,16 +15,34 @@ import java.util.concurrent.TimeUnit; */ public class PushSchedulerStarter { + /** + * 每0.5秒推送一次待测key + */ public static void startPusher(Long period) { if (period == null || period <= 0) { period = 500L; } ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); scheduledExecutorService.scheduleAtFixedRate(() -> { - IKeyCollector collectHK = KeyHandlerFactory.getCollector(); + IKeyCollector collectHK = KeyHandlerFactory.getCollector(); KeyHandlerFactory.getPusher().send(Context.APP_NAME, collectHK.lockAndGetResult()); collectHK.finishOnce(); },0, period, TimeUnit.MILLISECONDS); } + /** + * 每10秒推送一次数量统计 + */ + public static void startCountPusher(Integer period) { + if (period == null || period <= 0) { + period = 10; + } + ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); + scheduledExecutorService.scheduleAtFixedRate(() -> { + IKeyCollector collectHK = KeyHandlerFactory.getCounter(); + KeyHandlerFactory.getPusher().sendCount(Context.APP_NAME, collectHK.lockAndGetResult()); + collectHK.finishOnce(); + },0, period, TimeUnit.SECONDS); + } + } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnCountCollector.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnCountCollector.java new file mode 100644 index 0000000000000000000000000000000000000000..329ec7c64c2c7430cbea8cc489622b3010697990 --- /dev/null +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnCountCollector.java @@ -0,0 +1,115 @@ +package com.jd.platform.hotkey.client.core.key; + +import cn.hutool.core.util.StrUtil; +import com.jd.platform.hotkey.client.core.rule.KeyRuleHolder; +import com.jd.platform.hotkey.common.model.KeyCountModel; +import com.jd.platform.hotkey.common.tool.Constant; + +import java.text.SimpleDateFormat; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; + +/** + * 热点数量统计 + * @author wuweifeng + * @version 1.0 + * @date 2020-06-24 + */ +public class TurnCountCollector implements IKeyCollector { + /** + * 存储格式为： + * pin_20200624091024 -> 10 + * pin_20200624091025 -> 20 + *

+ * sku_20200624091024 -> 123 + * sku_20200624091025 -> 142 + */ + private ConcurrentHashMap HIT_MAP_0 = new ConcurrentHashMap<>(512); + private ConcurrentHashMap HIT_MAP_1 = new ConcurrentHashMap<>(512); + + private static final String FORMAT = "yyyy-MM-dd HH:mm:ss"; + + private AtomicLong atomicLong = new AtomicLong(0); + + @Override + public List lockAndGetResult() { + //自增后，对应的map就会停止被写入，等待被读取 + atomicLong.addAndGet(1); + + List list; + if (atomicLong.get() % 2 == 0) { + list = get(HIT_MAP_1); + HIT_MAP_1.clear(); + } else { + list = get(HIT_MAP_0); + HIT_MAP_0.clear(); + } + return list; + } + + /** + * 每10秒上报一次最近10秒的数据，并且删掉相应的key + */ + private List get(ConcurrentHashMap map) { + List list = new ArrayList<>(); + for (Map.Entry entry : map.entrySet()) { + String key = entry.getKey(); + HitCount hitCount = entry.getValue(); + KeyCountModel keyCountModel = new KeyCountModel(); + keyCountModel.setTotalHitCount(hitCount.totalHitCount.get()); + keyCountModel.setRuleKey(key); + keyCountModel.setHotHitCount(hitCount.hotHitCount.get()); + list.add(keyCountModel); + } + return list; + } + + @Override + public void collect(KeyHotModel keyHotModel) { + if (atomicLong.get() % 2 == 0) { + put(keyHotModel.getKey(), keyHotModel.isHot(), HIT_MAP_0); + } else { + put(keyHotModel.getKey(), keyHotModel.isHot(), HIT_MAP_1); + } + } + + @Override + public void finishOnce() { + + } + + public void put(String key, boolean isHot, ConcurrentHashMap map) { + //如key是pin_的前缀，则存储pin_ + String rule = KeyRuleHolder.rule(key); + //不在规则内的不处理 + if (StrUtil.isEmpty(rule)) { + return; + } + String nowTime = nowTime(); + + //rule + 分隔符 + 2020-10-23 21:11:22 + String mapKey = rule + Constant.COUNT_DELIMITER + nowTime; + //该方法线程安全 + HitCount hitCount = map.computeIfAbsent(mapKey, v -> new HitCount()); + if (isHot) { + hitCount.hotHitCount.incrementAndGet(); + } + hitCount.totalHitCount.incrementAndGet(); + } + + private String nowTime() { + Date nowTime = new Date(System.currentTimeMillis()); + SimpleDateFormat sdFormatter = new SimpleDateFormat(FORMAT); + return sdFormatter.format(nowTime); + } + + private class HitCount { + private AtomicInteger hotHitCount = new AtomicInteger(); + private AtomicInteger totalHitCount = new AtomicInteger(); + } +} diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnKeyCollector.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnKeyCollector.java index 87e903165ef9a44f02aec3609dcc3fb8ddcf6ce7..353938cea40f86313ec54ae2cfdcb75db38d61d4 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnKeyCollector.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/TurnKeyCollector.java @@ -16,7 +16,7 @@ import java.util.concurrent.atomic.AtomicLong; * @author wuweifeng wrote on 2020-01-06 * @version 1.0 */ -public class TurnKeyCollector implements IKeyCollector { +public class TurnKeyCollector implements IKeyCollector { private ConcurrentHashMap map0 = new ConcurrentHashMap<>(); private ConcurrentHashMap map1 = new ConcurrentHashMap<>(); @@ -49,21 +49,15 @@ public class TurnKeyCollector implements IKeyCollector { return; } if (atomicLong.get() % 2 == 0) { - HotKeyModel model = map0.get(key); - if (model == null) { - map0.put(key, hotKeyModel); - } else { + //不存在时返回null并将key-value放入，已有相同key时，返回该key对应的value，并且不覆盖 + HotKeyModel model = map0.putIfAbsent(key, hotKeyModel); + if (model != null) { model.setCount(model.getCount() + hotKeyModel.getCount()); - map0.put(key, model); } - } else { - HotKeyModel model = map1.get(key); - if (model == null) { - map1.put(key, hotKeyModel); - } else { + HotKeyModel model = map1.putIfAbsent(key, hotKeyModel); + if (model != null) { model.setCount(model.getCount() + hotKeyModel.getCount()); - map1.put(key, model); } } @@ -73,4 +67,5 @@ public class TurnKeyCollector implements IKeyCollector { public void finishOnce() { } + } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/rule/KeyRuleHolder.java b/client/src/main/java/com/jd/platform/hotkey/client/core/rule/KeyRuleHolder.java index 79dd5048d75954d185f8b686ed454e3cddbe0ef0..37eea6417cf1573761b2f976235b2bd5f788d308 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/rule/KeyRuleHolder.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/rule/KeyRuleHolder.java @@ -75,30 +75,56 @@ public class KeyRuleHolder { if (StrUtil.isEmpty(key)) { return null; } - synchronized (KEY_RULES) { - LocalCache prefix = null; - LocalCache common = null; - - //遍历该app的所有rule，找到与key匹配的rule。优先全匹配->prefix匹配-> * 通配 - //这一段虽然看起来比较奇怪，但是没毛病，不要乱改 - for (KeyRule keyRule : KEY_RULES) { - if (key.equals(keyRule.getKey())) { - return RULE_CACHE_MAP.get(keyRule.getDuration()); - } - if ((keyRule.isPrefix() && key.startsWith(keyRule.getKey()))) { - prefix = RULE_CACHE_MAP.get(keyRule.getDuration()); - } - if ("*".equals(keyRule.getKey())) { - common = RULE_CACHE_MAP.get(keyRule.getDuration()); - } - } + KeyRule keyRule = findRule(key); + if (keyRule == null) { + return null; + } + return RULE_CACHE_MAP.get(keyRule.getDuration()); - if (prefix != null) { - return prefix; - } - return common; + } + + /** + * 判断该key命中了哪个rule + */ + public static String rule(String key) { + KeyRule keyRule = findRule(key); + if (keyRule != null) { + return keyRule.getKey(); } + return ""; + } + + /** + * 获取该key应该缓存多久 + */ + public static int duration(String key) { + KeyRule keyRule = findRule(key); + if (keyRule != null) { + return keyRule.getDuration(); + } + return 0; + } + //遍历该app的所有rule，找到与key匹配的rule。优先全匹配->prefix匹配-> * 通配 + //这一段虽然看起来比较奇怪，但是没毛病，不要乱改 + private static KeyRule findRule(String key) { + KeyRule prefix = null; + KeyRule common = null; + for (KeyRule keyRule : KEY_RULES) { + if (key.equals(keyRule.getKey())) { + return keyRule; + } + if ((keyRule.isPrefix() && key.startsWith(keyRule.getKey()))) { + prefix = keyRule; + } + if ("*".equals(keyRule.getKey())) { + common = keyRule; + } + } + if (prefix != null) { + return prefix; + } + return common; } /** @@ -121,7 +147,7 @@ public class KeyRuleHolder { @Subscribe public void ruleChange(KeyRuleInfoChangeEvent event) { - JdLogger.info(getClass(),"new rules info is :" + event.getKeyRules()); + JdLogger.info(getClass(), "new rules info is :" + event.getKeyRules()); List ruleList = event.getKeyRules(); if (ruleList == null) { return; diff --git a/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java b/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java index ee7d07bb42efbb59d24452ec3286a35de92f87a1..216160a3b7e0d47f9e70db646018f9066ca87f4d 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java @@ -16,16 +16,14 @@ import com.jd.platform.hotkey.common.configcenter.ConfigConstant; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.common.rule.KeyRule; +import com.jd.platform.hotkey.common.tool.Constant; import com.jd.platform.hotkey.common.tool.FastJsonUtils; import io.grpc.StatusRuntimeException; import io.netty.util.internal.StringUtil; import java.util.ArrayList; import java.util.List; -import java.util.concurrent.CompletableFuture; -import java.util.concurrent.Executors; -import java.util.concurrent.ScheduledExecutorService; -import java.util.concurrent.TimeUnit; +import java.util.concurrent.*; /** * etcd连接管理器 @@ -40,8 +38,6 @@ public class EtcdStarter { fetchRule(); - fetchExistHotKey(); - // startWatchWorker(); startWatchRule(); @@ -133,8 +129,9 @@ public class EtcdStarter { private void fetchWorkerInfo() { ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); //开启拉取etcd的worker信息，如果拉取失败，则定时继续拉取 - scheduledExecutorService.scheduleAtFixedRate(() -> { JdLogger.info(getClass(), "trying to connect to etcd and fetch worker info"); - fetch(); + scheduledExecutorService.scheduleAtFixedRate(() -> { + JdLogger.info(getClass(), "trying to connect to etcd and fetch worker info"); + fetch(); }, 0, 30, TimeUnit.SECONDS); } @@ -183,10 +180,11 @@ public class EtcdStarter { } /** - * 异步开始监听热key变化信息 + * 异步开始监听热key变化信息，该目录里只有手工添加的key信息 */ private void startWatchHotKey() { - CompletableFuture.runAsync(() -> { + ExecutorService executorService = Executors.newSingleThreadExecutor(); + executorService.submit(() -> { JdLogger.info(getClass(), "--- begin watch hotKey change ----"); IConfigCenter configCenter = EtcdConfigFactory.configCenter(); try { @@ -208,17 +206,19 @@ public class EtcdStarter { model.setKey(key); EventBusCenter.getInstance().post(new ReceiveNewKeyEvent(model)); } else { - JdLogger.info(getClass(), "etcd receive new key : " + key); HotKeyModel model = new HotKeyModel(); model.setRemove(false); - //value是uuid的，就是worker推到etcd推送过来的。value是时间戳的，就是手工创建的 - //该功能已摒弃，client不再监听worker发往etcd的key了，只监听etcd手工的和client发到etcd的 - if (keyValue.getValue().toStringUtf8().length() != 13) { - model.setCreateTime(System.currentTimeMillis()); - } else { - model.setCreateTime(Long.valueOf(keyValue.getValue().toStringUtf8())); + String value = keyValue.getValue().toStringUtf8(); + //新增热key + JdLogger.info(getClass(), "etcd receive new key : " + key + " --value:" + value); + //如果这是一个删除指令，就什么也不干 + if (Constant.DEFAULT_DELETE_VALUE.equals(value)) { + continue; } + //手工创建的value是时间戳 + model.setCreateTime(Long.valueOf(keyValue.getValue().toStringUtf8())); + model.setKey(key); EventBusCenter.getInstance().post(new ReceiveNewKeyEvent(model)); } @@ -241,6 +241,9 @@ public class EtcdStarter { JdLogger.info(getClass(), "trying to connect to etcd and fetch rule info"); boolean success = fetchRuleFromEtcd(); if (success) { + //拉取已存在的热key + fetchExistHotKey(); + scheduledExecutorService.shutdown(); } @@ -278,7 +281,8 @@ public class EtcdStarter { * 异步监听rule规则变化 */ private void startWatchRule() { - CompletableFuture.runAsync(() -> { + ExecutorService executorService = Executors.newSingleThreadExecutor(); + executorService.submit(() -> { JdLogger.info(getClass(), "--- begin watch rule change ----"); try { IConfigCenter configCenter = EtcdConfigFactory.configCenter(); diff --git a/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java b/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java index d2a36a41c7819ad537e1f8852a2cfe7906573da0..d8ba8927c491ae75823e2511187d024e5767fb94 100755 --- a/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java @@ -2,8 +2,6 @@ package com.jd.platform.hotkey.client.netty; import com.jd.platform.hotkey.client.core.worker.WorkerInfoHolder; import com.jd.platform.hotkey.client.log.JdLogger; -import com.jd.platform.hotkey.common.coder.Codec; -import com.jd.platform.hotkey.common.coder.NettyCodec; import com.jd.platform.hotkey.common.tool.Constant; import io.netty.bootstrap.Bootstrap; import io.netty.buffer.ByteBuf; @@ -28,9 +26,6 @@ public class NettyClient { private Bootstrap bootstrap; - private Codec codec = new NettyCodec(); - - public static NettyClient getInstance() { return nettyClient; } @@ -56,8 +51,6 @@ public class NettyClient { ByteBuf delimiter = Unpooled.copiedBuffer(Constant.DELIMITER.getBytes()); ch.pipeline() .addLast(new DelimiterBasedFrameDecoder(Constant.MAX_LENGTH, delimiter)) -// .addLast(codec.newEncoder()) -// .addLast(codec.newDecoder()) .addLast(new StringDecoder()) //10秒没消息时，就发心跳包过去 .addLast(new IdleStateHandler(0, 0, 30)) diff --git a/common/pom.xml b/common/pom.xml index 4c855aa083318eeaab2b024e9002c4e144e3db5a..e11fe7a0b39801872527ad1e513e4d79699a6a5c 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -14,11 +14,10 @@ 1.8 - 4.1.24.Final - 3.4.2 + 4.1.42.Final 1.1.7.3 1.2.70 - 0.0.14 + 0.0.16 2.8.0 5.1.0 @@ -40,13 +39,6 @@ netty-all ${netty4.version} - - - com.lmax - disruptor - ${disruptor.version} - - com.alibaba fastjson diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java deleted file mode 100755 index c61805b9d34c84981d0b2c90891e305ae3b2bd61..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.jd.platform.hotkey.common.coder; - -import io.netty.channel.ChannelHandler; - -/** - * @author wuweifeng wrote on 2019-12-11 - * @version 1.0 - */ -public interface Codec { - - ChannelHandler newEncoder(); - - ChannelHandler newDecoder(); -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java deleted file mode 100644 index 24cb0ed25000966a1dc925fdacca90f5af8d0e1a..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java +++ /dev/null @@ -1,53 +0,0 @@ -package com.jd.platform.hotkey.common.coder; - -import com.jd.platform.hotkey.common.model.HotKeyMsg; -import com.jd.platform.hotkey.common.model.typeenum.MessageType; -import com.jd.platform.hotkey.common.tool.Constant; -import io.netty.buffer.ByteBuf; -import io.netty.channel.ChannelHandlerContext; -import io.netty.handler.codec.ByteToMessageDecoder; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.nio.charset.Charset; -import java.util.List; - -/** - * @author wuweifeng wrote on 2020-01-06 - * @version 1.0 - */ -public class MessageDecoder extends ByteToMessageDecoder { - private Logger logger = LoggerFactory.getLogger(getClass()); - - @Override - protected void decode(ChannelHandlerContext ctx, ByteBuf byteBuf, List out) { - if (byteBuf.readableBytes() < 10 || byteBuf.readableBytes() > Constant.MAX_LENGTH) { - logger.warn("数据包不正确，当前包大小为：" + byteBuf.readableBytes()); - return; - } - - byteBuf.markReaderIndex(); - - HotKeyMsg message = new HotKeyMsg(); - - int magicNumber = byteBuf.readInt(); - if (Constant.MAGIC_NUMBER != magicNumber) { - logger.warn("MAGIC_NUMBER不正确:" + magicNumber); - return; - } - - message.setMagicNumber(byteBuf.readInt()); // 读取魔数 - - MessageType messageType = MessageType.get(byteBuf.readByte()); - if (messageType == null) { - logger.error("messageType is null , byteBuf readByte = " + byteBuf.readByte()); - return; - } - message.setMessageType(messageType); // 读取当前的消息类型 - - int bodyLength = byteBuf.readInt(); // 读取消息体长度和数据 - CharSequence body = byteBuf.readCharSequence(bodyLength, Charset.defaultCharset()); - message.setBody(body.toString()); - out.add(message); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java deleted file mode 100644 index acf21624c6323f2c616d132c67701a12a4b802e2..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java +++ /dev/null @@ -1,47 +0,0 @@ -package com.jd.platform.hotkey.common.coder; - -import com.jd.platform.hotkey.common.model.HotKeyMsg; -import com.jd.platform.hotkey.common.model.typeenum.MessageType; -import com.jd.platform.hotkey.common.tool.Constant; -import io.netty.buffer.ByteBuf; -import io.netty.channel.ChannelHandlerContext; -import io.netty.handler.codec.MessageToByteEncoder; - -import java.nio.charset.Charset; - -/** - * @author wuweifeng wrote on 2020-01-06 - * @version 1.0 - */ -public class MessageEncoder extends MessageToByteEncoder { - - @Override - protected void encode(ChannelHandlerContext ctx, HotKeyMsg message, ByteBuf out) { - MessageType messageType = message.getMessageType(); - //非法类型 - if (MessageType.get(messageType.getType()) == null) { - return; - } - // 这里会判断消息类型是不是EMPTY类型，如果是EMPTY类型，则表示当前消息不需要写入到管道中 - if (messageType == MessageType.EMPTY) { - return; - } - //4个byte - out.writeInt(Constant.MAGIC_NUMBER); // 写入当前的魔数 - //1个byte - out.writeByte(message.getMessageType().getType()); // 写入当前消息的类型 - - if (null == message.getBody()) { - out.writeInt(0); // 如果消息体为空，则写入0，表示消息体长度为0 - } else { - System.out.println(message.getBody().length()); - out.writeInt(message.getBody().length()); - out.writeCharSequence(message.getBody(), Charset.defaultCharset()); - } - out.writeBytes(Constant.DELIMITER.getBytes()); - } - - public static void main(String[] args) { - System.out.println(Constant.MAGIC_NUMBER + ""); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java deleted file mode 100755 index 1aeac45f5948d3595b367b811b666dcfb01870ab..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.jd.platform.hotkey.common.coder; - -import io.netty.channel.ChannelHandler; - -/** - * @author wuweifeng wrote on 2019-12-11 - * @version 1.0 - */ -public class NettyCodec implements Codec { - - @Override - public ChannelHandler newEncoder() { - return new MessageEncoder(); - } - - @Override - public ChannelHandler newDecoder() { - return new MessageDecoder(); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java index 9ac0b379741870404ddb8a6151734d657567d694..3d22e8cbecc2b70ee02aad78657b214ac6e269cd 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java @@ -13,6 +13,10 @@ public interface ConfigConstant { * 所有的workers，存这里 */ String workersPath = "/jd/workers/"; + /** + * dashboard的ip存这里 + */ + String dashboardPath = "/jd/dashboard/"; /** * 该app所有的workers地址的path。需要手工分配，默认每个app都用所有的worker */ @@ -49,4 +53,25 @@ public interface ConfigConstant { * bufferPool直接内存 */ String bufferPoolPath = "/jd/bufferPool/"; + + /** + * 存放客户端hotKey访问次数和总访问次数的path + */ + String keyHitCountPath = "/jd/keyHitCount/"; + + /** + * 清理历史数据的配置的path + * time unit : day + */ + String clearCfgPath = "/jd/clearCfg/"; + + /** + * app配置 + */ + String appCfgPath = "/jd/appCfg/"; + + /** + * 控制台启动的netty端口 + */ + int dashboardPort = 11112; } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java index b6ddd584550d7e8ec5aea622e41e8144a45a6a5a..fb64d9e28b9a3217e1824f93b762f147427fd205 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java @@ -3,7 +3,6 @@ package com.jd.platform.hotkey.common.configcenter.etcd; import cn.hutool.core.collection.CollectionUtil; import com.google.protobuf.ByteString; import com.ibm.etcd.api.KeyValue; -import com.ibm.etcd.api.Kv; import com.ibm.etcd.api.LeaseGrantResponse; import com.ibm.etcd.api.RangeResponse; import com.ibm.etcd.client.KvStoreClient; @@ -54,6 +53,14 @@ public class JdEtcdClient implements IConfigCenter { this.kvClient = kvClient; } + public LockClient getLockClient() { + return lockClient; + } + + public void setLockClient(LockClient lockClient) { + this.lockClient = lockClient; + } + @Override public void put(String key, String value) { kvClient.put(ByteString.copyFromUtf8(key), ByteString.copyFromUtf8(value)).sync(); diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java index 0451fce64a1dc414d41a410de91ebd1be7038e9e..11de0ccaf76915b7dd59f58f6d4120850912d459 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java @@ -23,11 +23,7 @@ public class HotKeyModel extends BaseModel { @Override public String toString() { - return "HotKeyModel{" + - "appName='" + appName + '\'' + - ", keyType=" + keyType + - ", remove=" + remove + - '}'; + return "appName:" + appName + "-key=" + getKey(); } public boolean isRemove() { @@ -53,4 +49,5 @@ public class HotKeyModel extends BaseModel { public void setKeyType(KeyType keyType) { this.keyType = keyType; } + } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyMsg.java b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyMsg.java index 4e197f3b7ab00b2f6a2c94786a004f1284228cd8..a1caf1a8c9688cbc9fe71aa4b9eab812a4137313 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyMsg.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyMsg.java @@ -10,11 +10,18 @@ import com.jd.platform.hotkey.common.model.typeenum.MessageType; public class HotKeyMsg { private int magicNumber; + private String appName; + private MessageType messageType; private String body; public HotKeyMsg(MessageType messageType, String body) { + this(null, messageType, body); + } + + public HotKeyMsg(String appName, MessageType messageType, String body) { + this.appName = appName; this.messageType = messageType; this.body = body; } @@ -31,6 +38,14 @@ public class HotKeyMsg { '}'; } + public String getAppName() { + return appName; + } + + public void setAppName(String appName) { + this.appName = appName; + } + public int getMagicNumber() { return magicNumber; } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/KeyCountModel.java b/common/src/main/java/com/jd/platform/hotkey/common/model/KeyCountModel.java new file mode 100644 index 0000000000000000000000000000000000000000..09f3563b9a5d797c78524a31a834359f474faa20 --- /dev/null +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/KeyCountModel.java @@ -0,0 +1,67 @@ +package com.jd.platform.hotkey.common.model; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-06-24 + */ +public class KeyCountModel { + /** + * 对应的规则名，如 pin_2020-08-09 11:32:43 + */ + private String ruleKey; + /** + * 总访问次数 + */ + private int totalHitCount; + /** + * 热后访问次数 + */ + private int hotHitCount; + /** + * 发送时的时间 + */ + private long createTime; + + @Override + public String toString() { + return "KeyCountModel{" + + "ruleKey='" + ruleKey + '\'' + + ", totalHitCount=" + totalHitCount + + ", hotHitCount=" + hotHitCount + + ", createTime=" + createTime + + '}'; + } + + public long getCreateTime() { + return createTime; + } + + public void setCreateTime(long createTime) { + this.createTime = createTime; + } + + public String getRuleKey() { + return ruleKey; + } + + public void setRuleKey(String ruleKey) { + this.ruleKey = ruleKey; + } + + public int getTotalHitCount() { + return totalHitCount; + } + + public void setTotalHitCount(int totalHitCount) { + this.totalHitCount = totalHitCount; + } + + public int getHotHitCount() { + return hotHitCount; + } + + public void setHotHitCount(int hotHitCount) { + this.hotHitCount = hotHitCount; + } +} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java b/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java index 57daa2b6feedf10b054433e0c088cc9fc31142ac..5b07624630cae927ac724a738977f67c41110d78 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java @@ -8,6 +8,8 @@ public enum MessageType { APP_NAME((byte) 1), REQUEST_NEW_KEY((byte) 2), RESPONSE_NEW_KEY((byte) 3), + REQUEST_HIT_COUNT((byte) 7), //命中率 + REQUEST_HOT_KEY((byte) 8), //热key，worker->dashboard PING((byte) 4), PONG((byte) 5), EMPTY((byte) 6); diff --git a/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java b/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java index 9e0cc85ed7455f4a54d993d5f36b6066009e50b6..5c747540ad3452fa161d7d7d591c98f99611e29c 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java @@ -14,7 +14,21 @@ public class Constant { public static int Default_Threads = 4; public static int MAGIC_NUMBER = 0x12fcf76; + /** + * netty的分隔符 + */ public static String DELIMITER = "$(* *)$"; + /** + * 数量统计时，rule+时间 组成key用的分隔符 + */ + public static String COUNT_DELIMITER = "#**#"; + + public static String BAK_DELIMITER = "#\\*\\*#"; + + /** + * 当客户端要删除某个key时，就往etcd里赋值这个value，设置1秒过期，就算删除了 + */ + public static String DEFAULT_DELETE_VALUE = "#[DELETE]#"; //单次包最大2M public static int MAX_LENGTH = 2 * 1024 * 1024; diff --git a/common/src/main/java/com/jd/platform/hotkey/common/tool/TwoTuple.java b/common/src/main/java/com/jd/platform/hotkey/common/tool/TwoTuple.java new file mode 100644 index 0000000000000000000000000000000000000000..9aa4051a391ab194046e3f33d492643c23581895 --- /dev/null +++ b/common/src/main/java/com/jd/platform/hotkey/common/tool/TwoTuple.java @@ -0,0 +1,23 @@ +package com.jd.platform.hotkey.common.tool; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-06-24 + */ +public class TwoTuple { + + public final A first; + + public final B second; + + public TwoTuple(A a, B b){ + first = a; + second = b; + } + + public String toString(){ + return "(" + first + ", " + second + ")"; + } + +} \ No newline at end of file diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java b/common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java similarity index 63% rename from worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java rename to common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java index cdc1299c2f33bf3ef3b6944271ae8434184a35c2..009eb471cea2e441b308f6030c0e171405998692 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.worker.netty.flush; +package com.jd.platform.hotkey.common.tool.flush; import io.netty.buffer.ByteBuf; import io.netty.channel.ChannelHandlerContext; @@ -16,19 +16,19 @@ public class FlushUtil { * 往channel里输出消息 */ public static void flush(ChannelHandlerContext channelHandlerContext, ByteBuf byteBuf) { - if (channelHandlerContext.channel().isWritable()) { - channelHandlerContext.channel().writeAndFlush(byteBuf).addListener(future -> { - if (!future.isSuccess()) { - logger.warn("flush error " + future.cause().getMessage()); - } - }); - } else { +// if (channelHandlerContext.channel().isWritable()) { +// channelHandlerContext.channel().writeAndFlush(byteBuf).addListener(future -> { +// if (!future.isSuccess()) { +// logger.warn("flush error " + future.cause().getMessage()); +// } +// }); +// } else { try { //同步发送 channelHandlerContext.channel().writeAndFlush(byteBuf).sync(); } catch (InterruptedException e) { logger.error("flush error " + e.getMessage()); } - } +// } } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java index 84e91c12fd1a514b48d170627f3d4671f1ec4eca..3070702bf1a01db93361dbedfabe9ac90740d259 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java @@ -1,9 +1,6 @@ package com.jd.platform.hotkey.dashboard; -import com.jd.platform.hotkey.common.configcenter.IConfigCenter; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; -import com.jd.platform.hotkey.dashboard.mapper.RulesMapper; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.boot.CommandLineRunner; @@ -12,24 +9,14 @@ import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.scheduling.annotation.EnableAsync; import org.springframework.scheduling.annotation.EnableScheduling; -import javax.annotation.Resource; - @EnableAsync @EnableScheduling @SpringBootApplication public class DashboardApplication implements CommandLineRunner { - private Logger logger = LoggerFactory.getLogger(getClass()); - @Resource - private IConfigCenter configCenter; - @Resource - private KeyTimelyMapper timelyMapper; - @Resource - private RulesMapper rulesMapper; - public static void main(String[] args) { try { @@ -37,13 +24,11 @@ public class DashboardApplication implements CommandLineRunner { }catch (Exception e){ e.printStackTrace(); } - } + @Override - public void run(String... args) { - int row = timelyMapper.clear(); - logger.info("clear db timely hotKey, effect row : {}"); + public void run(String... args) { } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/autoconfigure/AbstractProperties.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/autoconfigure/AbstractProperties.java new file mode 100644 index 0000000000000000000000000000000000000000..23500671ec54bd510735fa47188979f7ab3b47a7 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/autoconfigure/AbstractProperties.java @@ -0,0 +1,18 @@ +package com.jd.platform.hotkey.dashboard.autoconfigure; + +/** + * User: fuxueliang + * Date: 16/8/25 + * Email: fuxueliang@jd.com + */ +public abstract class AbstractProperties { + private boolean enabled = true; + + public boolean isEnabled() { + return enabled; + } + + public void setEnabled(boolean enabled) { + this.enabled = enabled; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java new file mode 100644 index 0000000000000000000000000000000000000000..63612e8d8c12c59307c216d598e32ced13d02c49 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java @@ -0,0 +1,30 @@ +package com.jd.platform.hotkey.dashboard.cache; + +import com.github.benmanes.caffeine.cache.Cache; +import com.github.benmanes.caffeine.cache.Caffeine; + +import java.util.concurrent.TimeUnit; + +/** + * @author wuweifeng wrote on 2019-12-12 + * @version 1.0 + */ +public class CaffeineBuilder { + + public static Cache cache(int duration) { + return cache(128, 1000000, duration); + } + + + /** + * 构建所有来的要缓存的key getCache + */ + public static Cache cache(int minSize, int maxSize, int expireSeconds) { + return Caffeine.newBuilder() + .initialCapacity(minSize)//初始大小 + .maximumSize(maxSize)//最大数量 + .expireAfterWrite(expireSeconds, TimeUnit.SECONDS)//过期时间 + .build(); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java index edf7053a7aaafb2cc4c1a6bfa01ba644a025e6f5..7aa346fa58d47c702cff13531c7caea4af3e24d9 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java @@ -4,8 +4,14 @@ import java.text.SimpleDateFormat; import java.util.Date; import com.alibaba.fastjson.JSON; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; +import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; +import com.jd.platform.hotkey.dashboard.common.ex.BizException; +import com.jd.platform.hotkey.dashboard.model.User; +import com.jd.platform.hotkey.dashboard.service.UserService; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; +import io.jsonwebtoken.Claims; import org.springframework.beans.propertyeditors.CustomDateEditor; import org.springframework.web.bind.WebDataBinder; @@ -19,6 +25,8 @@ public class BaseController { @Resource protected HttpServletRequest request; + @Resource + protected UserService userService; @InitBinder public void initBinder(WebDataBinder binder) { @@ -27,9 +35,21 @@ public class BaseController { binder.registerCustomEditor(Date.class, new CustomDateEditor(dateFormat, true)); } + public void checkApp(String app){ + String authHeader = JwtTokenUtil.getAuthHeader(request); + assert authHeader != null; + Claims claims = JwtTokenUtil.claims(authHeader.substring(2)); + String role = claims.get("role",String.class); + if(!role.equals(Constant.ADMIN)){ + if(!ownApp().equals(app)){ + throw new BizException(ResultEnum.NO_PERMISSION); + } + } + } + public String userName(){ - final String authHeader = request.getHeader(JwtTokenUtil.AUTH_HEADER_KEY); + final String authHeader = JwtTokenUtil.getAuthHeader(request); final String token = authHeader.substring(2); return JwtTokenUtil.getUsername(token); } @@ -37,18 +57,19 @@ public class BaseController { public SearchReq param(String text){ - String authHeader = request.getHeader(JwtTokenUtil.AUTH_HEADER_KEY); + String authHeader = JwtTokenUtil.getAuthHeader(request); SearchReq dto = JSON.parseObject(text, SearchReq.class); if(dto == null){ dto = new SearchReq(); } - dto.setAppName(JwtTokenUtil.getAppName(authHeader.substring(2))); + // dto.setAppName(JwtTokenUtil.getAppName(authHeader.substring(2))); return dto; } - public SearchReq param2(SearchReq dto){ - String authHeader = request.getHeader(JwtTokenUtil.AUTH_HEADER_KEY); - if(dto == null){ dto = new SearchReq(); } - dto.setAppName(JwtTokenUtil.getAppName(authHeader.substring(2))); - return dto; + + public String ownApp(){ + String authHeader = JwtTokenUtil.getAuthHeader(request); + assert authHeader != null; + Claims claims = JwtTokenUtil.claims(authHeader.substring(2)); + return userService.selectByUserName(claims.getSubject()).getAppName(); } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java new file mode 100644 index 0000000000000000000000000000000000000000..963892110d934ec2163949947443fd9271f29bd5 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java @@ -0,0 +1,40 @@ +package com.jd.platform.hotkey.dashboard.common.config; + +import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; +import com.jd.platform.hotkey.dashboard.common.ex.BizException; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.web.bind.annotation.ControllerAdvice; +import org.springframework.web.bind.annotation.ExceptionHandler; +import org.springframework.web.bind.annotation.ResponseBody; + +import javax.servlet.http.HttpServletResponse; + + + +/** + * @author liyunfeng31 + */ +@ControllerAdvice +public class MyExceptionHandler { + + private Logger logger = LoggerFactory.getLogger(getClass()); + + @ExceptionHandler(value = BizException.class) + @ResponseBody + public Result bizExceptionHandler(BizException e, HttpServletResponse resp){ + resp.setStatus(e.getCode()); + logger.info("业务异常：",e); + return Result.error(e.getCode(),e.getMsg()); + + } + + + @ExceptionHandler(value =Exception.class) + @ResponseBody + public Result exceptionHandler(Exception e){ + logger.info("未知异常：",e); + return Result.error(ResultEnum.BIZ_ERROR); + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java index 8ff0af7f319bbea79e5dcb50f7c09383304d1387..952aa2a3ac358f33f592012058fafe35b4d51c37 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java @@ -1,7 +1,6 @@ package com.jd.platform.hotkey.dashboard.common.domain; import java.util.ArrayList; -import java.util.Collections; import java.util.List; public class Constant { @@ -20,6 +19,9 @@ public class Constant { public static final int WORKER_CHANGE = 2; + public static final int HOTKEY_CHANGE = 3; + + public static final String MONITOR_VIEW = "秒级监控"; public static final String CHANG_LOG_VIEW = "变更记录"; @@ -33,10 +35,32 @@ public class Constant { public static final String WORKER_VIEW = "节点信息"; + public static final String CLEAR_VIEW = "数据清理"; + + public static final int MAX_DAY_RANGE = 3; public static final int VERSION = 1; + public static final String ADMIN = "ADMIN"; + + public static final String APP_UER = "APPUER"; + + public static final String POST = "POST"; + + + public static final String VIEW = "view"; + + + public static final String LIST = "list"; + + + public static final String INFO = "info"; + + + public static final String THIRTY_DAY = "30"; + + public static final List HEAD = new ArrayList<>(); static { HEAD.add("热点key"); HEAD.add("次数"); HEAD.add("所属APP");} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java deleted file mode 100644 index 04d4a8325c0eac341df8e743c7d6b5098c951544..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java +++ /dev/null @@ -1,95 +0,0 @@ -package com.jd.platform.hotkey.dashboard.common.domain; - -import com.ibm.etcd.api.Event; - -import java.io.Serializable; -import java.util.Date; - -/** - * 包装类，用于保存事件最准确的时间 - */ -public class EventWrapper implements Serializable { - - private String key; - - private String value; - - private Event.EventType eventType; - - private Date date; - - private Long ttl; - - private long version; - - private String uuid; - - @Override - public String toString() { - return "EventWrapper{" + - "key='" + key + '\'' + - ", value='" + value + '\'' + - ", eventType=" + eventType + - ", date=" + date + - ", ttl=" + ttl + - ", version=" + version + - ", uuid='" + uuid + '\'' + - '}'; - } - - public String getUuid() { - return uuid; - } - - public void setUuid(String uuid) { - this.uuid = uuid; - } - - public long getVersion() { - return version; - } - - public void setVersion(long version) { - this.version = version; - } - - public String getKey() { - return key; - } - - public void setKey(String key) { - this.key = key; - } - - public String getValue() { - return value; - } - - public void setValue(String value) { - this.value = value; - } - - public Event.EventType getEventType() { - return eventType; - } - - public void setEventType(Event.EventType eventType) { - this.eventType = eventType; - } - - public void setDate(Date date) { - this.date = date; - } - - public void setTtl(Long ttl) { - this.ttl = ttl; - } - - public Date getDate() { - return date; - } - - public Long getTtl() { - return ttl; - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java new file mode 100644 index 0000000000000000000000000000000000000000..3585f73c25e12beb3ad6622ca515ad7aac53ef10 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java @@ -0,0 +1,27 @@ +package com.jd.platform.hotkey.dashboard.common.domain; + +import java.util.Date; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-09-02 + */ +public interface IRecord { + /** + * appName + "/" + key + */ + String appNameKey(); + + /** + * 手工添加的是时间戳13位，worker传过来的是uuid + */ + String value(); + + /** + * 0插入，1删除 + */ + int type(); + + Date createTime(); +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java index 5f64c7575f762d7585ac458c3c3cc9b75ee8b94f..d10c3a5169fb88777149563c4a2cb1ace803c823 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java @@ -7,9 +7,13 @@ import java.util.List; public class Page implements Serializable { private static final long serialVersionUID = 1L; - + /** + * 当前第几页 + */ private Integer page; - + /** + * 总共多少条 + */ private int total; private List rows; @@ -20,6 +24,7 @@ public class Page implements Serializable { this.rows = rows; } + public Integer getPage() { return page; } @@ -43,4 +48,13 @@ public class Page implements Serializable { public void setRows(List rows) { this.rows = rows; } + + @Override + public String toString() { + return "Page{" + + "page=" + page + + ", total=" + total + + ", rows=" + rows + + '}'; + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Result.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Result.java index ea1df6b35ce70e72050ef65b572552103b3998f4..70db062c4cd757c4b5515698e1bfabe573ca9adb 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Result.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Result.java @@ -30,7 +30,7 @@ public class Result implements Serializable { public Result(ResultEnum resultEnum) { this.code = resultEnum.getCode(); - this.msg = resultEnum.getMessage(); + this.msg = resultEnum.getMsg(); } public static Result success(){ @@ -45,6 +45,10 @@ public class Result implements Serializable { return new Result<>(ResultEnum.NO_CHANGE); } + public static Result error(int code, String msg){ + return new Result<>(code,msg); + } + public static Result error(){ return new Result<>(ResultEnum.BIZ_ERROR); } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/ExcelData.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/dto/ExcelDataDto.java similarity index 82% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/ExcelData.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/dto/ExcelDataDto.java index be933c6eacdc84148bbc96eddf6a3a1814754a42..8f6594e1864045e0b4504c267e8fe6fbe897300a 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/ExcelData.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/dto/ExcelDataDto.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.common.domain; +package com.jd.platform.hotkey.dashboard.common.domain.dto; import java.util.List; /** * @author liyunfeng31 */ -public class ExcelData { +public class ExcelDataDto { /** * 文件名 */ @@ -19,7 +19,7 @@ public class ExcelData { */ private List> rows; - public ExcelData(String fileName, List head, List> rows) { + public ExcelDataDto(String fileName, List head, List> rows) { this.fileName = fileName; this.head = head; this.rows = rows; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/dto/KeyCountDto.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/dto/KeyCountDto.java deleted file mode 100644 index 976af9c9c078e81606eb3941424d25320634b25f..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/dto/KeyCountDto.java +++ /dev/null @@ -1,39 +0,0 @@ -package com.jd.platform.hotkey.dashboard.common.domain.dto; - -import java.io.Serializable; - -/** - * @author liyunfeng31 - */ -public class KeyCountDto implements Serializable { - - private String k; - - private Integer count; - - private String app; - - public String getK() { - return k; - } - - public void setK(String k) { - this.k = k; - } - - public Integer getCount() { - return count; - } - - public void setCount(Integer count) { - this.count = count; - } - - public String getApp() { - return app; - } - - public void setApp(String app) { - this.app = app; - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/req/ChartReq.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/req/ChartReq.java index 6b0b4f1b410a353bb743556aa9c10a118cf25919..5154e437a3172a36ace6d37410fd050b66f28444 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/req/ChartReq.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/req/ChartReq.java @@ -28,8 +28,8 @@ public class ChartReq implements Serializable { } public ChartReq(LocalDateTime st, LocalDateTime et, Integer limit) { - this.startTime = DateUtil.localDateTimeToDate(st); - this.endTime = DateUtil.localDateTimeToDate(et); + this.startTime = DateUtil.ldtToDate(st); + this.endTime = DateUtil.ldtToDate(et); this.limit = limit; } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/req/SearchReq.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/req/SearchReq.java index 4e7a462bb6250cf691e4796fe8545a7c60825e52..a63d9878b6545c93661505583d1a30186bc5b367 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/req/SearchReq.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/req/SearchReq.java @@ -26,7 +26,7 @@ public class SearchReq implements Serializable { private Integer type; - private String appName; + private String app; private String key; @@ -65,12 +65,12 @@ public class SearchReq implements Serializable { this.type = type; } - public String getAppName() { - return appName; + public String getApp() { + return app; } - public void setAppName(String appName) { - this.appName = appName; + public void setApp(String app) { + this.app = app; } public String getKey() { @@ -81,9 +81,8 @@ public class SearchReq implements Serializable { this.key = key; } - public SearchReq(LocalDateTime st) { - this.startTime = DateUtil.localDateTimeToDate(st); + this.startTime = DateUtil.ldtToDate(st); this.endTime = new Date(); } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java new file mode 100644 index 0000000000000000000000000000000000000000..14b872b46de2cdc0e1b6e9a16cab8efcba6b7c62 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java @@ -0,0 +1,47 @@ +package com.jd.platform.hotkey.dashboard.common.domain.vo; + +/** + * @ProjectName: hotkey + * @ClassName: ClearCfgVo + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:54 + */ +public class ClearCfgVo { + + private String app; + + private String ttl; + + private Long version; + + public ClearCfgVo(String app, String ttl, Long version) { + this.app = app; + this.ttl = ttl; + this.version = version; + } + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public String getTtl() { + return ttl; + } + + public void setTtl(String ttl) { + this.ttl = ttl; + } + + public Long getVersion() { + return version; + } + + public void setVersion(Long version) { + this.version = version; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HitCountVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HitCountVo.java new file mode 100644 index 0000000000000000000000000000000000000000..b9ee681e34dbd2c871dba79bfdc8346f87adc07f --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HitCountVo.java @@ -0,0 +1,200 @@ +package com.jd.platform.hotkey.dashboard.common.domain.vo; + + +import java.io.Serializable; +import java.math.BigDecimal; +import java.util.Date; + + +/** + * @author liyunfeng31 + */ +public class HitCountVo implements Serializable { + + private String rule; + + private String app; + + private Integer hitCount; + + private Integer totalCount; + + private BigDecimal ratio; + + private Integer days; + + private Integer hours; + + private Integer minutes; + + private Integer seconds; + + private Date createTime; + + public String getRule() { + return rule; + } + + public void setRule(String rule) { + this.rule = rule; + } + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public Integer getHitCount() { + return hitCount; + } + + public void setHitCount(Integer hitCount) { + this.hitCount = hitCount; + } + + public Integer getTotalCount() { + return totalCount; + } + + public void setTotalCount(Integer totalCount) { + this.totalCount = totalCount; + } + + public BigDecimal getRatio() { + return ratio; + } + + public void setRatio(BigDecimal ratio) { + this.ratio = ratio; + } + + public Integer getDays() { + return days; + } + + public void setDays(Integer days) { + this.days = days; + } + + public Integer getHours() { + return hours; + } + + public void setHours(Integer hours) { + this.hours = hours; + } + + public Integer getMinutes() { + return minutes; + } + + public void setMinutes(Integer minutes) { + this.minutes = minutes; + } + + public Integer getSeconds() { + return seconds; + } + + public void setSeconds(Integer seconds) { + this.seconds = seconds; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + + public static HitCountVoBuilder aHitCountVo() { + return new HitCountVoBuilder(); + } + + + public static final class HitCountVoBuilder { + private String rule; + private String app; + private Integer hitCount; + private Integer totalCount; + private BigDecimal ratio; + private Integer days; + private Integer hours; + private Integer minutes; + private Integer seconds; + private Date createTime; + + private HitCountVoBuilder() { + } + + public HitCountVoBuilder rule(String rule) { + this.rule = rule; + return this; + } + + public HitCountVoBuilder app(String app) { + this.app = app; + return this; + } + + public HitCountVoBuilder hitCount(Integer hitCount) { + this.hitCount = hitCount; + return this; + } + + public HitCountVoBuilder totalCount(Integer totalCount) { + this.totalCount = totalCount; + return this; + } + + public HitCountVoBuilder ratio(BigDecimal ratio) { + this.ratio = ratio; + return this; + } + + public HitCountVoBuilder days(Integer days) { + this.days = days; + return this; + } + + public HitCountVoBuilder hours(Integer hours) { + this.hours = hours; + return this; + } + + public HitCountVoBuilder minutes(Integer minutes) { + this.minutes = minutes; + return this; + } + + public HitCountVoBuilder seconds(Integer seconds) { + this.seconds = seconds; + return this; + } + + public HitCountVoBuilder createTime(Date createTime) { + this.createTime = createTime; + return this; + } + + public HitCountVo build() { + HitCountVo hitCountVo = new HitCountVo(); + hitCountVo.setRule(rule); + hitCountVo.setApp(app); + hitCountVo.setHitCount(hitCount); + hitCountVo.setTotalCount(totalCount); + hitCountVo.setRatio(ratio); + hitCountVo.setDays(days); + hitCountVo.setHours(hours); + hitCountVo.setMinutes(minutes); + hitCountVo.setSeconds(seconds); + hitCountVo.setCreateTime(createTime); + return hitCountVo; + } + } +} \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java index ce4945e8e5e1ec6ab5d02b7d0ab17e96af58a818..3392fc2494c96339b84052b0935dc1fcb7ef5102 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java @@ -2,6 +2,7 @@ package com.jd.platform.hotkey.dashboard.common.domain.vo; import java.util.List; import java.util.Map; +import java.util.Set; /** * @author liyunfeng31 @@ -12,6 +13,9 @@ public class HotKeyLineChartVo { private Map series; + public HotKeyLineChartVo() { + } + public HotKeyLineChartVo(List xAxis, Map series) { this.xAxis = xAxis; this.series = series; @@ -24,4 +28,35 @@ public class HotKeyLineChartVo { public Map getSeries() { return series; } + + + private Set xAxis2; + + private Map> series2; + + private Set legend; + + public Set getxAxis2() { + return xAxis2; + } + + public void setxAxis2(Set xAxis2) { + this.xAxis2 = xAxis2; + } + + public Map> getSeries2() { + return series2; + } + + public void setSeries2(Map> series2) { + this.series2 = series2; + } + + public Set getLegend() { + return legend; + } + + public void setLegend(Set legend) { + this.legend = legend; + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java new file mode 100644 index 0000000000000000000000000000000000000000..273ce59998d5c60dc2f4c77fae0ef1749d8dc535 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java @@ -0,0 +1,52 @@ +package com.jd.platform.hotkey.dashboard.common.domain.vo; + +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * @author liyunfeng31 + */ +public class LineChartVo { + + private Set legend; + + private Set xAxis; + + private Map> series; + + public LineChartVo(Set xAxis, Map> series) { + this.xAxis = xAxis; + this.series = series; + } + + public LineChartVo(Set legend, Set xAxis, Map> series) { + this.legend = legend; + this.xAxis = xAxis; + this.series = series; + } + + public Set getxAxis() { + return xAxis; + } + + public void setxAxis(Set xAxis) { + this.xAxis = xAxis; + } + + public Map> getSeries() { + return series; + } + + public void setSeries(Map> series) { + this.series = series; + } + + public Set getLegend() { + return legend; + } + + public void setLegend(Set legend) { + this.legend = legend; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/eunm/ResultEnum.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/eunm/ResultEnum.java index 133746f0a0a56df919a1866f452c9300eb3f129d..9aab5ed481f4ababb0001a8f2759886a69eceeb5 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/eunm/ResultEnum.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/eunm/ResultEnum.java @@ -24,16 +24,18 @@ public enum ResultEnum { CONFLICT_ERROR(1007, "用户名/手机号已存在"), - TIME_RANGE_LARGE(1008, "查询时间过大"); + ILLEGAL_JSON_ARR(1008, "非法的json数组"), + + TIME_RANGE_LARGE(1022, "查询时间过大"); private int code; - private String message; + private String msg; - ResultEnum(int code, String message) { + ResultEnum(int code, String msg) { this.code = code; - this.message = message; + this.msg = msg; } public int getCode() { @@ -44,11 +46,11 @@ public enum ResultEnum { this.code = code; } - public String getMessage() { - return message; + public String getMsg() { + return msg; } - public void setMessage(String message) { - this.message = message; + public void setMsg(String msg) { + this.msg = msg; } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/ex/BizException.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/ex/BizException.java index cd1c1e23feb3319db9d7f997ebac3ba3c8eec428..6453a684334fb820e1acd3d41f669b346dd745c5 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/ex/BizException.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/ex/BizException.java @@ -15,7 +15,7 @@ public class BizException extends RuntimeException { public BizException(ResultEnum result) { this.code = result.getCode(); - this.msg = result.getMessage(); + this.msg = result.getMsg(); } public int getCode() { diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java index 5ac7073b3981cbbefef5f841d76cb4c0dd34bbfa..bd3a5e0e834bebea4ba5f6d5f43d32cb0002caf8 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java @@ -1,20 +1,22 @@ package com.jd.platform.hotkey.dashboard.common.monitor; -import com.ibm.etcd.api.Event; +import cn.hutool.core.collection.CollectionUtil; +import com.alibaba.fastjson.JSON; +import com.google.common.collect.Queues; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; import com.jd.platform.hotkey.dashboard.common.domain.Constant; -import com.jd.platform.hotkey.dashboard.common.domain.EventWrapper; -import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; +import com.jd.platform.hotkey.dashboard.common.domain.IRecord; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.mapper.KeyRecordMapper; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; import com.jd.platform.hotkey.dashboard.mapper.StatisticsMapper; +import com.jd.platform.hotkey.dashboard.mapper.SummaryMapper; import com.jd.platform.hotkey.dashboard.model.KeyRecord; -import com.jd.platform.hotkey.dashboard.model.KeyTimely; import com.jd.platform.hotkey.dashboard.model.Statistics; import com.jd.platform.hotkey.dashboard.util.DateUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; -import com.jd.platform.hotkey.dashboard.util.TwoTuple; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.scheduling.annotation.Scheduled; @@ -22,11 +24,13 @@ import org.springframework.stereotype.Component; import javax.annotation.Resource; import java.time.LocalDateTime; +import java.util.ArrayList; import java.util.Date; import java.util.List; +import java.util.UUID; import java.util.concurrent.BlockingQueue; -import java.util.concurrent.CompletableFuture; import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeUnit; @Component public class DataHandler { @@ -36,63 +40,52 @@ public class DataHandler { @Resource private KeyRecordMapper keyRecordMapper; @Resource - private KeyTimelyMapper keyTimelyMapper; - @Resource private StatisticsMapper statisticsMapper; - private volatile boolean hasBegin = false; + @Resource + private SummaryMapper summaryMapper; + + @Resource + private IConfigCenter configCenter; /** * 队列 */ - private BlockingQueue queue = new LinkedBlockingQueue<>(); + private BlockingQueue queue = new LinkedBlockingQueue<>(); /** * 入队 */ - public void offer(EventWrapper eventWrapper) { + public void offer(IRecord record) { try { - queue.put(eventWrapper); + queue.put(record); } catch (InterruptedException e) { e.printStackTrace(); } } public void insertRecords() { - if (hasBegin) { - return; - } - hasBegin = true; - CompletableFuture.runAsync(() -> { - while (true) { - TwoTuple twoTuple; - try { - twoTuple = handHotKey(queue.take()); - } catch (Exception e) { - e.printStackTrace(); - log.error("handHotKey error ," + e.getMessage()); + while (true) { + try { + List records = new ArrayList<>(); + Queues.drain(queue, records, 1000, 1, TimeUnit.SECONDS); + if (CollectionUtil.isEmpty(records)) { continue; } - KeyRecord keyRecord = twoTuple.getSecond(); - KeyTimely keyTimely = twoTuple.getFirst(); - - if (keyTimely.getUuid() == null) { - keyTimelyMapper.deleteByKeyAndApp(keyTimely.getKey(), keyTimely.getAppName()); - } else { - try { - keyTimelyMapper.insertSelective(keyTimely); - } catch (Exception e) { - log.info("insert timely error"); + List keyRecordList = new ArrayList<>(); + for (IRecord iRecord : records) { + KeyRecord keyRecord = handHotKey(iRecord); + if (keyRecord != null) { + keyRecordList.add(keyRecord); } } - if (keyRecord != null) { - //插入记录 - keyRecordMapper.insertSelective(keyRecord); - } + keyRecordMapper.batchInsert(keyRecordList); + } catch (InterruptedException e) { + e.printStackTrace(); } - }); + } } @@ -100,59 +93,75 @@ public class DataHandler { /** * 处理热点key和记录 */ - private TwoTuple handHotKey(EventWrapper eventWrapper) { - Date date = eventWrapper.getDate(); - long ttl = eventWrapper.getTtl(); - Event.EventType eventType = eventWrapper.getEventType(); - String appKey = eventWrapper.getKey(); - String v = eventWrapper.getValue(); + private KeyRecord handHotKey(IRecord record) { + Date date = record.createTime(); + String appKey = record.appNameKey(); + String value = record.value(); //appName+"/"+"key" String[] arr = appKey.split("/"); - String uuid = eventWrapper.getUuid(); - int type = eventType.getNumber(); + String appName = arr[0]; + String key = arr[1]; + String uuid = UUID.randomUUID().toString(); + int type = record.type(); //组建成对象，供累计后批量插入、删除 - TwoTuple timelyKeyRecordTwoTuple = new TwoTuple<>(); - if (eventType.equals(Event.EventType.PUT)) { + if (type == 0) { + //如果是客户端删除时发出的put指令 + if (com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE.equals(value)) { + log.info("client remove key event : " + appKey); + return null; + } //手工添加的是时间戳13位，worker传过来的是uuid - String source = v.length() == 13 ? Constant.HAND : Constant.SYSTEM; - timelyKeyRecordTwoTuple.setFirst(new KeyTimely(arr[1], v, arr[0], ttl, uuid, date)); - KeyRecord keyRecord = new KeyRecord(arr[1], v, arr[0], ttl, source, type, uuid, date); - keyRecord.setRule(RuleUtil.rule(appKey)); - timelyKeyRecordTwoTuple.setSecond(keyRecord); - return timelyKeyRecordTwoTuple; - } else if (eventType.equals(Event.EventType.DELETE)) { - timelyKeyRecordTwoTuple.setFirst(new KeyTimely(arr[1], null, arr[0], 0L, null, null)); - //删除事件就不记录了 -// timelyKeyRecordTwoTuple.setSecond(new KeyRecord(arr[1], v, arr[0], 0L, Constant.SYSTEM, type, uuid, date)); - return timelyKeyRecordTwoTuple; + String source = value.length() == 13 ? Constant.HAND : Constant.SYSTEM; + String rule = RuleUtil.rule(appKey); + KeyRecord keyRecord = new KeyRecord(key, rule, appName, 1, source, type, uuid, date); + keyRecord.setRule(rule); + return keyRecord; + } else { + //是删除 + return null; } - return timelyKeyRecordTwoTuple; } + /** + * 每小时 统计一次record 表 结果记录到统计表 + */ @Scheduled(cron = "0 0 * * * ?") public void offlineStatistics() { try { - // 每小时 统计一次record 表 结果记录到统计表 LocalDateTime now = LocalDateTime.now(); - Date nowTime = DateUtil.localDateTimeToDate(now); + Date nowTime = DateUtil.ldtToDate(now); int day = DateUtil.nowDay(now); int hour = DateUtil.nowHour(now); - - List records = keyRecordMapper.maxHotKey(new SearchReq(now.minusHours(1))); - if (records.size() == 0) { - return; + SearchReq preHour = new SearchReq(now.minusHours(1)); + List records = keyRecordMapper.maxHotKey(preHour); + if (records.size() != 0) { + records.forEach(x -> { + x.setBizType(1); + x.setCreateTime(nowTime); + x.setDays(day); + x.setHours(hour); + x.setUuid(1 + "_" + x.getKeyName() + "_" + hour); + }); + } + log.info("每小时统计最热点,时间：{}, 行数：{}", now.toString(), records.size()); + List statistics = keyRecordMapper.statisticsByRule(preHour); + if (statistics.size() != 0) { + statistics.forEach(x -> { + x.setBizType(6); + x.setRule(x.getRule()); + x.setCreateTime(nowTime); + x.setDays(day); + x.setHours(hour); + x.setUuid(6 + "_" + x.getKeyName() + "_" + hour); + }); + log.info("每小时统计规则,时间：{}, data list：{}", now.toString(), JSON.toJSONString(statistics)); + records.addAll(statistics); + } + if (records.size() > 0) { + int row = statisticsMapper.batchInsert(records); } - records.forEach(x -> { - x.setBizType(1); - x.setCreateTime(nowTime); - x.setDays(day); - x.setHours(hour); - x.setUuid(1 + "_" + x.getKeyName() + "_" + hour); - }); - int row = statisticsMapper.batchInsert(records); - log.info("定时统计热点记录时间：{}, 影响行数：{}", now.toString(), row); } catch (Exception e) { e.printStackTrace(); } @@ -160,32 +169,63 @@ public class DataHandler { } - //@Scheduled(cron = "0 0 * * * ?") + /** + * 每分钟统计一次record 表 结果记录到统计表 + */ + @Scheduled(cron = "0 */1 * * * ?") public void offlineStatisticsRule() { try { - // 每分钟小时 统计一次record 表 结果记录到统计表 LocalDateTime now = LocalDateTime.now(); - Date nowTime = DateUtil.localDateTimeToDate(now); + Date nowTime = DateUtil.ldtToDate(now); int day = DateUtil.nowDay(now); int hour = DateUtil.nowHour(now); + int minus = DateUtil.nowMinus(now); - List records = keyRecordMapper.statisticsByRule(null); + List records = keyRecordMapper.statisticsByRule(new SearchReq(now.minusMinutes(1))); if (records.size() == 0) { return; } records.forEach(x -> { - x.setBizType(1); + x.setBizType(5); + x.setRule(x.getRule()); x.setCreateTime(nowTime); x.setDays(day); x.setHours(hour); - x.setUuid(1 + "_" + x.getKeyName() + "_" + hour); + x.setMinutes(minus); + // 骚操作 临时解决没有rule字段的问题 + x.setUuid(5 + "_" + x.getKeyName() + "_" + minus); }); int row = statisticsMapper.batchInsert(records); - log.info("定时统计热点记录时间：{}, 影响行数：{}", now.toString(), row); +// log.info("每分钟统计规则，时间：{}, 影响行数：{}, data list:{}", now.toString(), row, JSON.toJSONString(records)); + } catch (Exception e) { + e.printStackTrace(); + } + + } + + + /** + * 每天根据app的配置清理过期数据 + */ + @Scheduled(cron = "0 0 1 * * ?") + public void clearExpireData() { + try { + LocalDateTime now = LocalDateTime.now(); + List keyValues = configCenter.getPrefix(ConfigConstant.clearCfgPath); + for (KeyValue kv : keyValues) { + String key = kv.getKey().toStringUtf8(); + String ttl = kv.getValue().toStringUtf8(); + String app = key.replace(ConfigConstant.clearCfgPath, ""); + Date expireDate = DateUtil.ldtToDate(now.minusDays(Integer.parseInt(ttl))); + summaryMapper.clearExpireData(app, expireDate); + keyRecordMapper.clearExpireData(app, expireDate); + statisticsMapper.clearExpireData(app, expireDate); + } } catch (Exception e) { e.printStackTrace(); } } + } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/Test.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/Test.java deleted file mode 100644 index 017f42cf8517ca0c111a06d58624292b99179072..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/Test.java +++ /dev/null @@ -1,54 +0,0 @@ -package com.jd.platform.hotkey.dashboard.common.monitor; - -import java.util.ArrayList; -import java.util.List; -import java.util.Objects; -import java.util.concurrent.ConcurrentLinkedQueue; -import java.util.concurrent.Executor; -import java.util.concurrent.Executors; -import java.util.stream.Collectors; - -/** - * @author wuweifeng - * @version 1.0 - * @date 2020-05-18 - */ -public class Test { - public static void main(String[] args) throws InterruptedException { - ConcurrentLinkedQueue queue = new ConcurrentLinkedQueue<>(); - for (int i = 0; i < 19000; i++) { - queue.offer(i + ""); - } - Executor executor = Executors.newFixedThreadPool(4); - while (true) { - List keyRecords = new ArrayList<>(10000); - for (int i = 0; i < 10000; i++) { - if (!queue.isEmpty()) { - keyRecords.add(queue.poll()); - } else { - keyRecords.add(null); - } - } - - for (int i = 0; i < 10; i++) { - List tempRecords = keyRecords.subList(1000 * i, 1000 * (i + 1)); - executor.execute(() -> batch(tempRecords)); - } - - Thread.sleep(1000); - } - - } - - private static void batch(List strings) { - List records = strings.stream().filter(Objects::nonNull).collect(Collectors.toList()); - if (records.size() > 0) { - try { - Thread.sleep(1000); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println(records.size()); - } - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ClearController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ClearController.java new file mode 100644 index 0000000000000000000000000000000000000000..0ce81f03fe3b3a45d5912efae38bd66663c154fd --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ClearController.java @@ -0,0 +1,62 @@ +package com.jd.platform.hotkey.dashboard.controller; + +import com.github.pagehelper.PageInfo; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; +import com.jd.platform.hotkey.dashboard.service.ClearService; +import org.springframework.stereotype.Controller; +import org.springframework.ui.ModelMap; +import org.springframework.web.bind.annotation.*; + +import javax.annotation.Resource; + +/** + * @ProjectName: hotkey + * @ClassName: ClearController + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:48 + */ +@Controller +@RequestMapping("/clear") +public class ClearController { + + + private String prefix = "admin/clear"; + + + @Resource + private ClearService clearService; + + @GetMapping("/view") + public String view(ModelMap modelMap){ + modelMap.put("title", Constant.CLEAR_VIEW); + return prefix + "/list"; + } + + + @PostMapping("/list") + @ResponseBody + public Page list(PageReq page, String app){ + PageInfo info = clearService.pageClearCfg(page, app); + return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); + } + + + @GetMapping("/edit/{app}") + public String edit(@PathVariable("app") String app, ModelMap modelMap){ + modelMap.put("clearCfg", clearService.selectClearCfg(app)); + return prefix + "/edit"; + } + + @PostMapping("/save") + @ResponseBody + public Result save(ClearCfgVo cfg){ + int b = clearService.saveClearCfg(cfg); + return b == 0 ? Result.fail():Result.success(); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java index b61ce04d6c6d286ea9160c0430d64d8aa2e3a201..f3f362dda3825117c07310fb3c1f92ddceceaf7a 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java @@ -4,7 +4,7 @@ import com.github.pagehelper.PageInfo; import com.github.pagehelper.util.StringUtil; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.Constant; -import com.jd.platform.hotkey.dashboard.common.domain.ExcelData; +import com.jd.platform.hotkey.dashboard.common.domain.dto.ExcelDataDto; import com.jd.platform.hotkey.dashboard.common.domain.Page; import com.jd.platform.hotkey.dashboard.common.domain.Result; import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; @@ -24,8 +24,6 @@ import org.springframework.web.bind.annotation.*; import javax.annotation.Resource; import javax.servlet.http.HttpServletResponse; import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; import java.util.List; @@ -46,7 +44,7 @@ public class KeyController extends BaseController { @PostMapping("/ruleLineChart") @ResponseBody public HotKeyLineChartVo ruleLineChart(SearchReq req){ - return keyService.ruleLineChart(req); + return keyService.ruleLineChart(req,ownApp()); } @@ -57,15 +55,6 @@ public class KeyController extends BaseController { } - //@PostMapping("/qps") - @GetMapping("/qps") - @ResponseBody - public HotKeyLineChartVo qpsLineChart(ChartReq ChartReq) { - return keyService.getQpsLineChart(ChartReq); - } - - - @GetMapping("/view") public String view(ModelMap modelMap){ modelMap.put("title", Constant.KEY_RECORD_VIEW); @@ -76,7 +65,7 @@ public class KeyController extends BaseController { @PostMapping("/list") @ResponseBody public Page list(PageReq page, SearchReq searchReq){ - PageInfo info = keyService.pageKeyRecord(page, param2(searchReq)); + PageInfo info = keyService.pageKeyRecord(page, searchReq); return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); } @@ -90,8 +79,7 @@ public class KeyController extends BaseController { @PostMapping("/listTimely") @ResponseBody public Page listTimely(PageReq page, SearchReq searchReq){ - PageInfo info = keyService.pageKeyTimely(page, param2(searchReq)); - return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); + return keyService.pageKeyTimely(page, searchReq); } @@ -118,8 +106,8 @@ public class KeyController extends BaseController { @PostMapping("/add") @ResponseBody public Result add(KeyTimely key){ - key.setType(0); - key.setSource(userName()); + checkApp(key.getAppName()); + key.setUpdater(userName()); int b = keyService.insertKeyByUser(key); return b == 0 ? Result.fail():Result.success(); } @@ -127,29 +115,25 @@ public class KeyController extends BaseController { @PostMapping("/remove") @ResponseBody public Result remove(String key){ + String[] arr = key.split("/"); + checkApp(arr[0]); int b = keyService.delKeyByUser(new KeyTimely(key,userName())); return b == 0 ? Result.fail():Result.success(); } - @GetMapping("/edit/{id}") - public String edit(@PathVariable("id") Long id, ModelMap modelMap){ - modelMap.put("key", keyService.selectByPk(id)); - return prefix + "/edit"; - } - - @PostMapping("/edit") @ResponseBody - public Result editSave(KeyTimely keyTimely) { - return Result.success(keyService.updateKeyByUser(keyTimely)); + public Result editSave(KeyTimely key) { + checkApp(key.getAppName()); + return Result.success(keyService.updateKeyByUser(key)); } @RequestMapping(value = "/export", method = RequestMethod.GET) @ResponseBody - public void export(HttpServletResponse resp,String startTime,String endTime,String appName,String key){ + public void export(HttpServletResponse resp,String startTime,String endTime,String app,String key){ SearchReq req = new SearchReq(); if(StringUtil.isNotEmpty(startTime)){ req.setStartTime(DateUtil.strToDate(startTime)); @@ -157,11 +141,11 @@ public class KeyController extends BaseController { if(StringUtil.isNotEmpty(endTime)){ req.setEndTime(DateUtil.strToDate(endTime)); } - req.setAppName(appName); + req.setApp(app); req.setKey(key); List records = keyService.listMaxHot(req); List> rows = transform(records); - ExcelData data = new ExcelData("hotKey.xlsx", Constant.HEAD,rows); + ExcelDataDto data = new ExcelDataDto("hotKey.xlsx", Constant.HEAD,rows); ExcelUtil.exportExcel(resp,data); } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java index ba80e85989939fa9c6b0e63a957dbb48d55b8bca..e1869c9fb3d75e981d28c0b561cc30e05f509b67 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java @@ -1,12 +1,20 @@ package com.jd.platform.hotkey.dashboard.controller; +import cn.hutool.json.JSONArray; +import cn.hutool.json.JSONObject; +import cn.hutool.json.JSONTokener; +import com.alibaba.fastjson.JSON; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.Page; -import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.Result; -import com.jd.platform.hotkey.dashboard.model.KeyRule; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; +import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; +import com.jd.platform.hotkey.dashboard.common.ex.BizException; +import com.jd.platform.hotkey.dashboard.model.Rule; import com.jd.platform.hotkey.dashboard.model.Rules; import com.jd.platform.hotkey.dashboard.service.RuleService; import org.springframework.stereotype.Controller; @@ -14,8 +22,12 @@ import org.springframework.ui.ModelMap; import org.springframework.web.bind.annotation.*; import javax.annotation.Resource; +import java.util.List; +/** + * @author liyunfeng31 + */ @Controller @RequestMapping("/rule") public class RuleController extends BaseController { @@ -26,16 +38,10 @@ public class RuleController extends BaseController { - @GetMapping("/view2") - public String view2(ModelMap modelMap){ - modelMap.put("title", Constant.RULE_CONFIG_VIEW); - return "admin/rule/view"; - } - @GetMapping("/viewDetail") - public String view3(ModelMap modelMap){ + public String viewDetail(ModelMap modelMap){ modelMap.put("title", Constant.RULE_CONFIG_VIEW); - return "admin/rule/jn"; + return "admin/rule/json"; } @PostMapping("/getRule") @@ -45,35 +51,23 @@ public class RuleController extends BaseController { } - @PostMapping("/add") - @ResponseBody - public Result add(Rules rule){ - rule.setUpdateUser(userName()); - int b = ruleService.add(rule); - return b == 0 ? Result.fail():Result.success(); - } - - - @PostMapping("/update") - @ResponseBody - public Result update(Rules rules){ - rules.setUpdateUser(userName()); - int b = ruleService.updateRule(rules); - return b == 0 ? Result.fail():Result.success(); - } @PostMapping("/save") @ResponseBody public Result save(Rules rules){ + checkApp(rules.getApp()); + checkRule(rules.getRules()); rules.setUpdateUser(userName()); int b = ruleService.save(rules); return b == 0 ? Result.fail():Result.success(); } + @PostMapping("/remove") @ResponseBody public Result remove(String key){ + checkApp(key); int b = ruleService.delRule(key, userName()); return b == 0 ? Result.fail():Result.success(); } @@ -87,9 +81,9 @@ public class RuleController extends BaseController { @PostMapping("/list") @ResponseBody - public Page list2(PageReq page, String searchText){ + public Page list(PageReq page, String app){ page.setPageSize(30); - PageInfo info = ruleService.pageKeyRule(page, param(searchText)); + PageInfo info = ruleService.pageKeyRule(page, app); return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); } @@ -107,5 +101,47 @@ public class RuleController extends BaseController { return "admin/rule/view"; } + + @PostMapping("/listRules") + @ResponseBody + public List rules(){ + return ruleService.listRules(null); + } + + + @GetMapping("/viewHitCount") + public String viewHitCount(ModelMap modelMap){ + modelMap.put("title", Constant.MONITOR_VIEW); + return "admin/rule/listhitcount"; + } + + @PostMapping("/listHitCount") + @ResponseBody + public Page pageRuleHitCount(PageReq page, SearchReq req){ + PageInfo info = ruleService.pageRuleHitCount(page, req, ownApp()); + return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); + } + + + /** + * 校验是否合法 + * @param rules rules + * @return boolean + */ + private void checkRule(String rules) { + try { + Object json = new JSONTokener(rules).nextValue(); + if (json instanceof JSONObject) { + // JSONObject jsonObject = (JSONObject) json; + throw new BizException(ResultEnum.ILLEGAL_JSON_ARR); + } else if (json instanceof JSONArray) { + // JSONArray jsonArray = (JSONArray) json; + } + }catch(Exception e){ + throw new BizException(ResultEnum.ILLEGAL_JSON_ARR); + } + } + + } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java new file mode 100644 index 0000000000000000000000000000000000000000..5d4239437810b0a8535acb233bb2d116adcc77ba --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java @@ -0,0 +1,21 @@ +package com.jd.platform.hotkey.dashboard.controller; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-07-16 + */ +//@RestController +//public class TestController { +// @Resource +// private IConfigCenter configCenter; +// +// @RequestMapping("delete") +// public Object delete(String key) { +//// configCenter.putAndGrant(ConfigConstant.hotKeyRecordPath + "sample" + "/" + key, com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE, 1); +// configCenter.putAndGrant(ConfigConstant.hotKeyRecordPath + "bbb" + "/" + key, "", 60); +//// configCenter.delete(ConfigConstant.hotKeyPath + "sample" + "/" + key); +// +// return "1"; +// } +//} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java index cc7f6e58a5de2da2558470a5ff48603ccf3e3b59..67053667f47241e683edc25ad4a8c32648dee025 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java @@ -1,22 +1,18 @@ package com.jd.platform.hotkey.dashboard.controller; -import java.util.Collections; -import java.util.HashMap; -import java.util.List; -import java.util.Map; - import com.github.pagehelper.PageInfo; import com.github.pagehelper.util.StringUtil; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.Page; -import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.model.User; import com.jd.platform.hotkey.dashboard.service.UserService; import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; +import io.jsonwebtoken.Claims; import org.springframework.stereotype.Controller; import org.springframework.ui.ModelMap; import org.springframework.web.bind.annotation.*; @@ -25,6 +21,8 @@ import javax.annotation.Resource; import javax.servlet.http.Cookie; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; +import java.util.HashMap; +import java.util.Map; @Controller @@ -44,11 +42,10 @@ public class UserController extends BaseController { @ResponseBody public Result login(User param, HttpServletResponse response) { User user = userService.findByNameAndPwd(param); - if(user == null) return Result.error(ResultEnum.PWD_ERROR); - String token = JwtTokenUtil.createJWT(user.getId(), user.getUserName(), user.getRole(), user.getAppName()); + if(user == null) { return Result.error(ResultEnum.PWD_ERROR); } + String token = JwtTokenUtil.createJWT(user.getId(), user.getUserName(), user.getRole(), user.getAppName(), user.getNickName()); Cookie cookie = new Cookie("token", JwtTokenUtil.TOKEN_PREFIX + token); cookie.setMaxAge(3600*24*7); - cookie.setDomain("localhost"); cookie.setPath("/"); response.addCookie(cookie); Map map = new HashMap<>(2); @@ -77,21 +74,28 @@ public class UserController extends BaseController { @ResponseBody @PostMapping("/info") public User info(HttpServletRequest request){ - String authHeader = request.getHeader(JwtTokenUtil.AUTH_HEADER_KEY); - User userPower = JwtTokenUtil.userPower(authHeader.substring(2)); - String appName = userPower.getAppName(); - String role = userPower.getRole(); - if(role.equals("ADMIN")){ - List apps = userService.listApp(); - return new User(role,apps); - } - return new User(role, Collections.singletonList(appName)); + String authHeader = JwtTokenUtil.getAuthHeader(request); + assert authHeader != null; + Claims claims = JwtTokenUtil.claims(authHeader.substring(2)); + String role = claims.get("role",String.class); + String appName = userService.selectByUserName(claims.getSubject()).getAppName(); + return new User(role, userService.listApp(),appName); } @GetMapping("/LoginOut") - public String LoginOut(){ - return "redirect:/admin/user/login"; + public String LoginOut(HttpServletRequest request, HttpServletResponse response){ + Cookie[] cookies = request.getCookies(); + for(Cookie cookie : cookies){ + if("token".equals(cookie.getName())){ + Cookie tempCookie = new Cookie(cookie.getName(), null); + tempCookie.setPath("/");//路径要相同 + tempCookie.setMaxAge(0);//生命周期设置为0 + response.addCookie(tempCookie); + break; + } + } + return "redirect:/user/login"; } @@ -177,5 +181,10 @@ public class UserController extends BaseController { return "redirect:/error/500"; } + @PostMapping("getUserName") + @ResponseBody + public String getUserName(HttpServletRequest request, HttpServletResponse response){ + return userName(); + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java similarity index 50% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java index 26e5d39d218b05bb13caf4a60a4154fe58c523b9..234e447485a97da513765e9fa53122629284aca5 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java @@ -1,33 +1,34 @@ -package com.jd.platform.hotkey.dashboard.common.monitor; +package com.jd.platform.hotkey.dashboard.etcd; import cn.hutool.core.util.StrUtil; import com.ibm.etcd.api.Event; import com.ibm.etcd.api.KeyValue; import com.ibm.etcd.client.kv.KvClient; -import com.ibm.etcd.client.kv.WatchUpdate; import com.jd.platform.hotkey.common.configcenter.ConfigConstant; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.common.rule.KeyRule; import com.jd.platform.hotkey.common.tool.FastJsonUtils; import com.jd.platform.hotkey.dashboard.common.domain.Constant; -import com.jd.platform.hotkey.dashboard.common.domain.EventWrapper; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; -import com.jd.platform.hotkey.dashboard.mapper.ReceiveCountMapper; -import com.jd.platform.hotkey.dashboard.model.ReceiveCount; +import com.jd.platform.hotkey.dashboard.common.domain.IRecord; +import com.jd.platform.hotkey.dashboard.common.monitor.DataHandler; +import com.jd.platform.hotkey.dashboard.mapper.SummaryMapper; import com.jd.platform.hotkey.dashboard.model.Worker; +import com.jd.platform.hotkey.dashboard.netty.HotKeyReceiver; import com.jd.platform.hotkey.dashboard.service.WorkerService; +import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; import io.grpc.StatusRuntimeException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; import javax.annotation.PostConstruct; import javax.annotation.Resource; -import java.util.ArrayList; -import java.util.Date; -import java.util.List; -import java.util.concurrent.CompletableFuture; +import java.util.*; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; /** * @ProjectName: hotkey @@ -43,105 +44,171 @@ public class EtcdMonitor { @Resource private IConfigCenter configCenter; - @Resource - private ChangeLogMapper logMapper; + @Resource private WorkerService workerService; @Resource - private DataHandler dataHandler; + private SummaryMapper summaryMapper; @Resource - private ReceiveCountMapper receiveCountMapper; - -// @PostConstruct -// public void init() { -// CompletableFuture.runAsync(() -> { -// try { -// Thread.sleep(100); -// } catch (InterruptedException e) { -// e.printStackTrace(); -// } -// for (int i = 0; i < 20000; i++) { -// configCenter.put(ConfigConstant.hotKeyRecordPath + "abc/" + i, UUID.randomUUID().toString()); -// } -// }); -// } -// -// /** -// * 每隔60秒同步一下rule到本地db -// */ -// @Scheduled(fixedRate = 60000) -// public void syncRuleToDb() { -// List keyValues = configCenter.getPrefix(ConfigConstant.rulePath); -// logger.info("get rule from ETCD, rules: {}", keyValues.size()); -// for (KeyValue kv : keyValues) { -// String val = kv.getValue().toStringUtf8(); -// if(StringUtil.isEmpty(val)) continue; -// String key = kv.getKey().toStringUtf8(); -// rulesMapper.update(new Rules(key, val)); -// } -// } + private DataHandler dataHandler; + + + public static final ExecutorService threadPoolExecutor = Executors.newFixedThreadPool(16); /** * 监听新来的热key，该key的产生是来自于手工在控制台添加 */ - @PostConstruct public void watchHandOperationHotKey() { - CompletableFuture.runAsync(() -> { + threadPoolExecutor.submit(() -> { KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.hotKeyPath); while (watchIterator.hasNext()) { Event event = event(watchIterator); - EventWrapper eventWrapper = build(event); + KeyValue kv = event.getKv(); + String v = kv.getValue().toStringUtf8(); + Event.EventType eventType = event.getType(); String appKey = event.getKv().getKey().toStringUtf8().replace(ConfigConstant.hotKeyPath, ""); - eventWrapper.setKey(appKey); - dataHandler.offer(eventWrapper); + dataHandler.offer(new IRecord() { + @Override + public String appNameKey() { + return appKey; + } + + @Override + public String value() { + return v; + } + + @Override + public int type() { + return eventType.getNumber(); + } + + @Override + public Date createTime() { + return new Date(); + } + }); } }); } + + @PostConstruct + public void startWatch() { + //拉取rules + fetchRuleFromEtcd(); + + //规则拉取完毕后才能去开始入库 + insertRecords(); + + //开始入库 + dealHotkey(); + + //监听手工创建的key + watchHandOperationHotKey(); + + //监听rule变化 + watchRule(); + +// watchWorkers(); + + //观察热key访问次数和总访问次数，并做统计 + watchHitCount(); + } + /** - * 监听新来的热key，该key的产生是来自于worker集群推送过来的 + * 每秒去清理一次caffeine */ - @PostConstruct - public void watchHotKeyRecord() { - CompletableFuture.runAsync(() -> { - KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.hotKeyRecordPath); - while (watchIterator.hasNext()) { - Event event = event(watchIterator); - EventWrapper eventWrapper = build(event); + @Scheduled(fixedRate = 1000) + public void cleanCaffeine() { + try { + HotKeyReceiver.cleanUpCaffeine(); + } catch (Exception e) { + e.printStackTrace(); + } + } + + private void insertRecords() { + threadPoolExecutor.submit(() -> { + dataHandler.insertRecords(); + }); + } - String appKey = event.getKv().getKey().toStringUtf8().replace(ConfigConstant.hotKeyRecordPath, ""); - eventWrapper.setKey(appKey); - dataHandler.offer(eventWrapper); + /** + * 开始消费各worker发来的热key + */ + private void dealHotkey() { + threadPoolExecutor.submit(() -> { + while (true) { + try { + //获取发来的这个热key，存入本地caffeine，设置过期时间 + HotKeyModel model = HotKeyReceiver.take(); + + //将该key放入实时热key本地缓存中 + HotKeyReceiver.put(model); + + dataHandler.offer(new IRecord() { + @Override + public String appNameKey() { + return model.getAppName() + "/" + model.getKey(); + } + + @Override + public String value() { + return UUID.randomUUID().toString(); + } + + @Override + public int type() { + return 0; + } + + @Override + public Date createTime() { + return new Date(); + } + }); + + } catch (Exception e) { + e.printStackTrace(); + } } + }); } - private EventWrapper build(Event event) { - KeyValue kv = event.getKv(); - long ttl = configCenter.timeToLive(kv.getLease()); - String v = kv.getValue().toStringUtf8(); - Event.EventType eventType = event.getType(); - EventWrapper eventWrapper = new EventWrapper(); - eventWrapper.setValue(v); - eventWrapper.setDate(new Date()); - eventWrapper.setTtl(ttl); - eventWrapper.setVersion(kv.getVersion()); - eventWrapper.setEventType(eventType); - eventWrapper.setUuid(v); - return eventWrapper; - } + /** + * 异步监听rule规则变化 + */ + private void watchRule() { + threadPoolExecutor.submit(() -> { + KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.rulePath); + //如果有新事件，即rule的变更，就重新拉取所有的信息 + while (watchIterator.hasNext()) { + //这句必须写，next会让他卡住，除非真的有新rule变更 + Event event = event(watchIterator); + try { + log.info("---------watch rule change---------"); + //全量拉取rule信息 + fetchRuleFromEtcd(); + } catch (Exception e) { + log.error("watch rule err"); + } + } + + }); + } /** * 启动后从etcd拉取所有rule */ - @PostConstruct - public void fetchRuleFromEtcd() { + private void fetchRuleFromEtcd() { RuleUtil.init(); try { List ruleList = new ArrayList<>(); @@ -165,8 +232,6 @@ public class EtcdMonitor { } - //规则拉取完毕后才能去开始入库 - dataHandler.insertRecords(); } catch (StatusRuntimeException ex) { //etcd连不上 log.error("etcd connected fail. Check the etcd address!!!"); @@ -174,35 +239,8 @@ public class EtcdMonitor { } - /** - * 异步监听rule规则变化 - */ - @PostConstruct - public void startWatchRule() { - CompletableFuture.runAsync(() -> { - try { - KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.rulePath); - //如果有新事件，即rule的变更，就重新拉取所有的信息 - while (watchIterator.hasNext()) { - //这句必须写，next会让他卡住，除非真的有新rule变更 - WatchUpdate watchUpdate = watchIterator.next(); - List eventList = watchUpdate.getEvents(); - - //全量拉取rule信息 - fetchRuleFromEtcd(); - } - } catch (Exception e) { - log.error("watch rule err"); - } - - }); - } - - - - @PostConstruct - public void watchWorkers() { - CompletableFuture.runAsync(() -> { + private void watchWorkers() { + threadPoolExecutor.submit(() -> { KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.workersPath); while (watchIterator.hasNext()) { Event event = event(watchIterator); @@ -224,24 +262,31 @@ public class EtcdMonitor { } - - //@PostConstruct - public void watchReceiveKeyCount() { - CompletableFuture.runAsync(() -> { - KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.totalReceiveKeyCount); + /** + * 监听热key访问次数和总访问次数 + */ + private void watchHitCount() { + threadPoolExecutor.submit(() -> { + KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.keyHitCountPath); while (watchIterator.hasNext()) { Event event = event(watchIterator); KeyValue kv = event.getKv(); Event.EventType eventType = event.getType(); + if (Event.EventType.DELETE.equals(eventType)) { + continue; + } String k = kv.getKey().toStringUtf8(); String v = kv.getValue().toStringUtf8(); - long version = kv.getModRevision(); - String uuid = k + Constant.JOIN + version; - if (eventType.equals(Event.EventType.PUT)) { - receiveCountMapper.insert(new ReceiveCount(k, Long.parseLong(v), uuid)); - } else if (eventType.equals(Event.EventType.DELETE)) { - receiveCountMapper.insert(new ReceiveCount(k, 0L, uuid)); + + try { + Map map = FastJsonUtils.stringToCollect(v); + for (String key : map.keySet()) { + int row = summaryMapper.saveOrUpdate(CommonUtil.buildSummary(key, map)); + } + } catch (Exception e) { + e.printStackTrace(); } + } }); } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java new file mode 100644 index 0000000000000000000000000000000000000000..d0a44c5690e9fa212317bbc86055bbb3e36994a0 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java @@ -0,0 +1,63 @@ +package com.jd.platform.hotkey.dashboard.etcd; + +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.tool.IpUtils; +import com.jd.platform.hotkey.dashboard.netty.NodesServer; +import org.springframework.stereotype.Component; + +import javax.annotation.PostConstruct; +import javax.annotation.Resource; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; + +/** + * dashboard也注册到etcd，供各个worker连接 + * + * @author wuweifeng + * @version 1.0 + * @date 2020-08-28 + */ +@Component +public class EtcdRegister { + @Resource + private IConfigCenter configCenter; + @Resource + private NodesServer nodesServer; + + /** + * 每隔一会去check一下，自己还在不在etcd里 + */ + @PostConstruct + public void makeSureSelfOn() { + new Thread(() -> nodesServer.startNettyServer(ConfigConstant.dashboardPort)).start(); + + //开启上传worker信息 + ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); + scheduledExecutorService.scheduleAtFixedRate(() -> { + + try { + uploadSelfInfo(); + } catch (Exception e) { + //do nothing + } + + }, 3, 30, TimeUnit.SECONDS); + } + + private void uploadSelfInfo() { + configCenter.putAndGrant(buildKey(), buildValue(), 32); + } + + private String buildKey() { + String hostName = IpUtils.getHostName(); + return ConfigConstant.dashboardPath + hostName; + } + + private String buildValue() { + String ip = IpUtils.getIp(); + return ip + ":" + ConfigConstant.dashboardPort; + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java index 9f3fe55380d32613f75920a5d83190df936d02ef..02bc53e79934459c16291fd31c3dd19ce59e8a2f 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java @@ -1,17 +1,24 @@ package com.jd.platform.hotkey.dashboard.interceptor; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; +import com.jd.platform.hotkey.dashboard.service.UserService; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import io.jsonwebtoken.Claims; import org.springframework.http.HttpMethod; import org.springframework.util.StringUtils; import org.springframework.web.servlet.handler.HandlerInterceptorAdapter; + +import javax.annotation.Resource; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; +/** + * @author liyunfeng31 + */ public class JwtInterceptor extends HandlerInterceptorAdapter{ @Override @@ -20,38 +27,23 @@ public class JwtInterceptor extends HandlerInterceptorAdapter{ response.setStatus(HttpServletResponse.SC_OK); return true; } - //判断是否为ajax请求，默认不是 - boolean isAjaxRequest = false; - if(!StringUtils.isEmpty(request.getHeader("x-requested-with")) - && request.getHeader("x-requested-with").equals("XMLHttpRequest") - && request.getMethod().equals("POST")){ - isAjaxRequest = true; - } - String url = request.getRequestURI(); - if(isAjaxRequest){ - final String authHeader = request.getHeader(JwtTokenUtil.AUTH_HEADER_KEY); - if (StringUtils.isEmpty(authHeader) - || !authHeader.startsWith(JwtTokenUtil.TOKEN_PREFIX)) { - // response.sendRedirect("login"); + String header = request.getHeader("x-requested-with"); + if(!StringUtils.isEmpty(header) && "XMLHttpRequest".endsWith(header) && request.getMethod().equals(Constant.POST)){ + String authHeader = request.getHeader(JwtTokenUtil.AUTH_HEADER_KEY); + if (StringUtils.isEmpty(authHeader) || !authHeader.startsWith(JwtTokenUtil.TOKEN_PREFIX)) { throw new BizException(ResultEnum.NO_LOGIN); - // response.setStatus(HttpServletResponse.SC_FORBIDDEN); - // return false; - // throw new RuntimeException("NO_LOGIN"); - } + } final String token = authHeader.substring(2); Claims claims = JwtTokenUtil.parseJWT(token); String role = claims.get("role", String.class); - if(role.equals("ADMIN") || role.equals("APPADMIN")){ + if(role.equals(Constant.ADMIN)){ return true; } - // appUser只读 - if(url.contains("view")||url.contains("list")){ + String url = request.getRequestURI(); + if(url.contains(Constant.VIEW)||url.contains(Constant.LIST)||url.contains(Constant.INFO)){ return true; } - throw new BizException(ResultEnum.NO_PERMISSION); - // throw new RuntimeException("NO_PERMISSION"); - // response.setStatus(HttpServletResponse.SC_FORBIDDEN); - // return false; + } return true; } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/MyWebAppConfigurer.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/MyWebAppConfigurer.java index 135b277baa26695642c9a179ee5d1f701eca7f63..7360a638195fffb47203f3c5b9909c863e00de07 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/MyWebAppConfigurer.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/MyWebAppConfigurer.java @@ -29,13 +29,13 @@ public class MyWebAppConfigurer implements WebMvcConfigurer { @Override public void addCorsMappings(CorsRegistry registry){ } - + /** 静态资源处理 **/ @Override - public void addResourceHandlers(ResourceHandlerRegistry registry) { - registry.addResourceHandler("/static/**").addResourceLocations("classpath:/static/"); + public void addResourceHandlers(ResourceHandlerRegistry registry) { + registry.addResourceHandler("/static/**").addResourceLocations("classpath:/static/"); registry.addResourceHandler("/templates/**").addResourceLocations("classpath:/templates/"); - } + } /** 默认静态资源处理器 **/ @Override @@ -44,6 +44,6 @@ public class MyWebAppConfigurer implements WebMvcConfigurer { } - - + + } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java index 3ef46b31a58494eabc0daadf386cc759c1053c7b..e135ff6dd47281864ce9d86a5b7f8043ff7e68c1 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java @@ -1,11 +1,11 @@ package com.jd.platform.hotkey.dashboard.mapper; -import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.KeyRecord; import com.jd.platform.hotkey.dashboard.model.Statistics; import org.apache.ibatis.annotations.Mapper; +import java.util.Date; import java.util.List; /** @@ -25,4 +25,6 @@ public interface KeyRecordMapper { List maxHotKey(SearchReq req); List statisticsByRule(SearchReq req); + + int clearExpireData(String app, Date expireDate); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java deleted file mode 100644 index 826352ae2b02069cc13491179407788c1ca22324..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java +++ /dev/null @@ -1,29 +0,0 @@ -package com.jd.platform.hotkey.dashboard.mapper; - -import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.model.KeyTimely; -import org.apache.ibatis.annotations.Mapper; - -import java.util.List; - -@Mapper -public interface KeyTimelyMapper { - - int clear(); - - int deleteByKeyAndApp(String key,String appName); - - int insertSelective(KeyTimely key); - - KeyTimely selectByPrimaryKey(Long id); - - KeyTimely selectByKey(String key); - - int updateByKey(KeyTimely key); - - List listKeyTimely(SearchReq param); - - int batchInsert(List list); - - void batchDeleted(List deleteList); -} \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ReceiveCountMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ReceiveCountMapper.java deleted file mode 100644 index 6593ce04e46a8d2607daf4731848cb09e60ed3da..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ReceiveCountMapper.java +++ /dev/null @@ -1,31 +0,0 @@ -package com.jd.platform.hotkey.dashboard.mapper; - - -import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; -import com.jd.platform.hotkey.dashboard.model.ReceiveCount; -import org.apache.ibatis.annotations.Mapper; - -import java.util.List; - - -/** - * @author liyunfeng31 - */ -@Mapper -public interface ReceiveCountMapper { - - /** - * 插入最新记录 - * @param record record - * @return row - */ - int insert(ReceiveCount record); - - /** - * 查询统计 - * @param chartReq req - * @return list - */ - List list(ChartReq chartReq); - -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java index 3c2ee086a6393fd22c9248242194b67a2514df5f..daf848e4d7e45528e34a8befd512397221885c5b 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java @@ -1,10 +1,10 @@ package com.jd.platform.hotkey.dashboard.mapper; -import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.Statistics; import org.apache.ibatis.annotations.Mapper; +import java.util.Date; import java.util.List; /** @@ -18,7 +18,14 @@ public interface StatisticsMapper { * 查看 * @return list */ - List listStatistics(ChartReq chartReq); + List listStatistics(SearchReq req); + + + /** + * 查看 + * @return list + */ + List listOrderByTime(SearchReq req); /** * records @@ -33,4 +40,9 @@ public interface StatisticsMapper { * @return list */ List sumStatistics(SearchReq req); + + /** + * 清理 + */ + int clearExpireData(String app, Date expireDate); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRuleMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java similarity index 31% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRuleMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java index e9f41699457d3a650ef21d131f16a798fb8bb156..0305c4c277f760ced757e2a8ef3b6e2cdb2de030 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRuleMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java @@ -1,23 +1,27 @@ package com.jd.platform.hotkey.dashboard.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.model.KeyRule; +import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; +import com.jd.platform.hotkey.dashboard.model.Summary; import org.apache.ibatis.annotations.Mapper; -import org.apache.ibatis.annotations.Options; +import java.util.Date; import java.util.List; +/** + * @author liyunfeng31 + */ @Mapper -public interface KeyRuleMapper { +public interface SummaryMapper { - @Options(useGeneratedKeys = true) - int insertSelective(KeyRule record); + /** + * records + * @param records records + * @return row + */ + int saveOrUpdate(Summary records); - KeyRule selectByKey(String key); + List listRuleHitCount(SearchReq req); - int updateByKey(KeyRule record); - - List listRule(SearchReq param); - - void batchInsert(List rules); + int clearExpireData(String app, Date expireDate); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java index 14d4b6c66583ca3dad130b6cc270a98275984a71..c8d72ee2c2e06a13df2ba81d712eee9b8fb1cd40 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java @@ -10,7 +10,7 @@ import java.util.List; public interface UserMapper { int deleteByPrimaryKey(Integer id); - int insertSelective(User record); + int insertSelective(User user); User selectByPrimaryKey(Integer id); @@ -22,4 +22,7 @@ public interface UserMapper { List listApp(); + List selectHkUserList(User user); + + User selectByUserName(String userName); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java index 97f0b4b6ff0e6d2fe8b6e2f8a742497f01535786..e8fdf0f06dbc9524a516944c0845a5c139328dda 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java @@ -16,7 +16,7 @@ public class KeyRecord implements Serializable { /** * 缓存时间 */ - private Long duration; + private int duration; /** * 来源： SYSTEM 系统探测；USERNAME创建人 @@ -43,7 +43,7 @@ public class KeyRecord implements Serializable { public KeyRecord() { } - public KeyRecord(String key,String val, String appName, Long duration, + public KeyRecord(String key,String val, String appName, int duration, String source, Integer type,String uuid, Date createTime) { this.key = key; this.val = val; @@ -103,11 +103,11 @@ public class KeyRecord implements Serializable { this.val = val; } - public Long getDuration() { + public int getDuration() { return duration; } - public void setDuration(Long duration) { + public void setDuration(int duration) { this.duration = duration; } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRule.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRule.java deleted file mode 100644 index 5231a3805faf5782d210f34139eaf473fa4622ec..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRule.java +++ /dev/null @@ -1,136 +0,0 @@ -package com.jd.platform.hotkey.dashboard.model; - -import java.util.Date; - -public class KeyRule { - private Integer id; - - /** - * key的前缀，也可以完全和key相同。为"*"时代表通配符 - */ - private String key; - /** - * 是否是前缀，true是前缀 - */ - private Boolean prefix; - /** - * 间隔时间（秒） - */ - private Integer interval; - /** - * 累计数量 - */ - private Integer threshold; - /** - * 变热key后，本地、etcd缓存它多久。单位（秒），默认60 - */ - private Integer duration; - - private String appName; - - private Integer state; - - private String updateUser; - - private Date updateTime; - - private Integer version; - - public KeyRule() { - } - - public KeyRule(String key, Integer state,String updateUser) { - this.key = key; - this.state = state; - this.updateUser = updateUser; - } - - public Integer getId() { - return id; - } - - public void setId(Integer id) { - this.id = id; - } - - public String getKey() { - return key; - } - - public void setKey(String key) { - this.key = key; - } - - public Boolean getPrefix() { - return prefix; - } - - public void setPrefix(Boolean prefix) { - this.prefix = prefix; - } - - public Integer getInterval() { - return interval; - } - - public void setInterval(Integer interval) { - this.interval = interval; - } - - public Integer getThreshold() { - return threshold; - } - - public void setThreshold(Integer threshold) { - this.threshold = threshold; - } - - public Integer getDuration() { - return duration; - } - - public void setDuration(Integer duration) { - this.duration = duration; - } - - public String getAppName() { - return appName; - } - - public void setAppName(String appName) { - this.appName = appName == null ? null : appName.trim(); - } - - public Integer getState() { - return state; - } - - public void setState(Integer state) { - this.state = state; - } - - public String getUpdateUser() { - return updateUser; - } - - public void setUpdateUser(String updateUser) { - this.updateUser = updateUser == null ? null : updateUser.trim(); - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } - - public Integer getVersion() { - return version; - } - - public void setVersion(Integer version) { - this.version = version; - } - -} \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java index 1975e21baa68814a4ff345f07600fc7d95f7e75e..67b9d0ac997647e7f8a05db6cc9915a3c286d332 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java @@ -20,42 +20,48 @@ public class KeyTimely implements Serializable { */ private Long duration; - /** - * 来源： SYSTEM 系统探测；USERNAME创建人 - */ - private String source; - - /** - * 事件类型： 0 PUT； 1 删除 - */ - private Integer type; - private Date createTime; - private String uuid; /** * 该rule的描述 */ private transient String ruleDesc; + + private String updater; + + public KeyTimely() { } - public KeyTimely(String key, String source) { + public KeyTimely(String key, String updater) { this.key = key; - this.source = source; + this.updater = updater; } public KeyTimely(String key, String val, String appName, Long duration, String uuid, Date createTime) { this.key = key; this.val = val; - this.uuid = uuid; this.appName = appName; this.duration = duration; this.createTime = createTime; } + @Override + public String toString() { + return "KeyTimely{" + + "id=" + id + + ", key='" + key + '\'' + + ", appName='" + appName + '\'' + + ", val='" + val + '\'' + + ", duration=" + duration + + ", createTime=" + createTime + + ", ruleDesc='" + ruleDesc + '\'' + + ", updater='" + updater + '\'' + + '}'; + } + public String getRuleDesc() { return ruleDesc; } @@ -104,22 +110,6 @@ public class KeyTimely implements Serializable { this.duration = duration; } - public String getSource() { - return source; - } - - public void setSource(String source) { - this.source = source; - } - - public Integer getType() { - return type; - } - - public void setType(Integer type) { - this.type = type; - } - public Date getCreateTime() { return createTime; } @@ -128,11 +118,81 @@ public class KeyTimely implements Serializable { this.createTime = createTime; } - public String getUuid() { - return uuid; - } - - public void setUuid(String uuid) { - this.uuid = uuid; + public String getUpdater() { + return updater; + } + + public void setUpdater(String updater) { + this.updater = updater; + } + + public static Builder aKeyTimely() { + return new Builder(); + } + + public static final class Builder { + private String key; + private String appName; + private String val; + private Long duration; + private Date createTime; + private String uuid; + private transient String ruleDesc; + private String updater; + + private Builder() { + } + + public Builder key(String key) { + this.key = key; + return this; + } + + public Builder appName(String appName) { + this.appName = appName; + return this; + } + + public Builder val(String val) { + this.val = val; + return this; + } + + public Builder duration(Long duration) { + this.duration = duration; + return this; + } + + public Builder createTime(Date createTime) { + this.createTime = createTime; + return this; + } + + public Builder uuid(String uuid) { + this.uuid = uuid; + return this; + } + + public Builder ruleDesc(String ruleDesc) { + this.ruleDesc = ruleDesc; + return this; + } + + public Builder updater(String updater) { + this.updater = updater; + return this; + } + + public KeyTimely build() { + KeyTimely keyTimely = new KeyTimely(); + keyTimely.setKey(key); + keyTimely.setAppName(appName); + keyTimely.setVal(val); + keyTimely.setDuration(duration); + keyTimely.setCreateTime(createTime); + keyTimely.setRuleDesc(ruleDesc); + keyTimely.setUpdater(updater); + return keyTimely; + } } } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/ReceiveCount.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/ReceiveCount.java deleted file mode 100644 index 7376eaa21d010894d285444a00593e08c10a239b..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/ReceiveCount.java +++ /dev/null @@ -1,98 +0,0 @@ -package com.jd.platform.hotkey.dashboard.model; - -import com.jd.platform.hotkey.dashboard.util.DateUtil; - -import java.io.Serializable; -import java.time.LocalDateTime; -import java.util.Date; - -/** - * @author liyunfeng31 - */ -public class ReceiveCount implements Serializable { - - private Long id; - - private String workerName; - - private Long receiveCount; - - private String uuid; - - private Date updateTime; - - private Integer hour; - - private Integer minutes; - - - public ReceiveCount() { - } - - public ReceiveCount(String workerName, Long receiveCount,String uuid) { - this.workerName = workerName; - this.receiveCount = receiveCount; - this.uuid = uuid; - this.updateTime = new Date(); - LocalDateTime now = LocalDateTime.now(); - this.hour = DateUtil.nowHour(now); - this.minutes = DateUtil.nowMinus(now); - } - - - public Long getId() { - return id; - } - - public void setId(Long id) { - this.id = id; - } - - public String getWorkerName() { - return workerName; - } - - public void setWorkerName(String workerName) { - this.workerName = workerName; - } - - public Long getReceiveCount() { - return receiveCount; - } - - public void setReceiveCount(Long receiveCount) { - this.receiveCount = receiveCount; - } - - public String getUuid() { - return uuid; - } - - public void setUuid(String uuid) { - this.uuid = uuid; - } - - public Date getUpdateTime() { - return updateTime; - } - - public void setUpdateTime(Date updateTime) { - this.updateTime = updateTime; - } - - public Integer getHour() { - return hour; - } - - public void setHour(Integer hour) { - this.hour = hour; - } - - public Integer getMinutes() { - return minutes; - } - - public void setMinutes(Integer minutes) { - this.minutes = minutes; - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/Summary.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/Summary.java new file mode 100644 index 0000000000000000000000000000000000000000..ddf90a6020fdfbe125d16f97655505a47cb49401 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/Summary.java @@ -0,0 +1,269 @@ +package com.jd.platform.hotkey.dashboard.model; + +import java.io.Serializable; +import java.math.BigDecimal; +import java.util.Date; + + +/** + * @author liyunfeng31 + */ +public class Summary implements Serializable { + + private Integer id; + + private String indexName; + + private String rule; + + private String app; + + private Integer val1; + + private Integer val2; + + private BigDecimal val3; + + private Integer days; + + private Integer hours; + + private Integer minutes; + + private Integer seconds; + + private Integer bizType; + + private String uuid; + + private Date createTime; + + public Summary() { + } + + public Integer getId() { + return id; + } + + public void setId(Integer id) { + this.id = id; + } + + public String getIndexName() { + return indexName; + } + + public void setIndexName(String indexName) { + this.indexName = indexName; + } + + public String getRule() { + return rule; + } + + public void setRule(String rule) { + this.rule = rule; + } + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public Integer getVal1() { + return val1; + } + + public void setVal1(Integer val1) { + this.val1 = val1; + } + + public Integer getVal2() { + return val2; + } + + public void setVal2(Integer val2) { + this.val2 = val2; + } + + public BigDecimal getVal3() { + return val3; + } + + public void setVal3(BigDecimal val3) { + this.val3 = val3; + } + + public Integer getDays() { + return days; + } + + public void setDays(Integer days) { + this.days = days; + } + + public Integer getHours() { + return hours; + } + + public void setHours(Integer hours) { + this.hours = hours; + } + + public Integer getMinutes() { + return minutes; + } + + public void setMinutes(Integer minutes) { + this.minutes = minutes; + } + + public Integer getSeconds() { + return seconds; + } + + public void setSeconds(Integer seconds) { + this.seconds = seconds; + } + + public Integer getBizType() { + return bizType; + } + + public void setBizType(Integer bizType) { + this.bizType = bizType; + } + + public String getUuid() { + return uuid; + } + + public void setUuid(String uuid) { + this.uuid = uuid; + } + + public Date getCreateTime() { + return createTime; + } + + public void setCreateTime(Date createTime) { + this.createTime = createTime; + } + + + public static SummaryBuilder aSummary() { + return new SummaryBuilder(); + } + + public static final class SummaryBuilder { + private Integer id; + private String indexName; + private String rule; + private String app; + private Integer val1; + private Integer val2; + private BigDecimal val3; + private Integer days; + private Integer hours; + private Integer minutes; + private Integer seconds; + private Integer bizType; + private String uuid; + private Date createTime; + + private SummaryBuilder() { + } + + public SummaryBuilder id(Integer id) { + this.id = id; + return this; + } + + public SummaryBuilder indexName(String indexName) { + this.indexName = indexName; + return this; + } + + public SummaryBuilder rule(String rule) { + this.rule = rule; + return this; + } + + public SummaryBuilder app(String app) { + this.app = app; + return this; + } + + public SummaryBuilder val1(Integer val1) { + this.val1 = val1; + return this; + } + + public SummaryBuilder val2(Integer val2) { + this.val2 = val2; + return this; + } + + public SummaryBuilder val3(BigDecimal val3) { + this.val3 = val3; + return this; + } + + public SummaryBuilder days(Integer days) { + this.days = days; + return this; + } + + public SummaryBuilder hours(Integer hours) { + this.hours = hours; + return this; + } + + public SummaryBuilder minutes(Integer minutes) { + this.minutes = minutes; + return this; + } + + public SummaryBuilder seconds(Integer seconds) { + this.seconds = seconds; + return this; + } + + public SummaryBuilder bizType(Integer bizType) { + this.bizType = bizType; + return this; + } + + public SummaryBuilder uuid(String uuid) { + this.uuid = uuid; + return this; + } + + public SummaryBuilder createTime(Date createTime) { + this.createTime = createTime; + return this; + } + + public Summary build() { + Summary summary = new Summary(); + summary.setId(id); + summary.setIndexName(indexName); + summary.setRule(rule); + summary.setApp(app); + summary.setVal1(val1); + summary.setVal2(val2); + summary.setVal3(val3); + summary.setDays(days); + summary.setHours(hours); + summary.setMinutes(minutes); + summary.setSeconds(seconds); + summary.setBizType(bizType); + summary.setUuid(uuid); + summary.setCreateTime(createTime); + return summary; + } + } +} \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/User.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/User.java index 16440361ba641850fa48e614a2499bb903976ab9..79ed8cc12a8aec92b969f53cd546bfe2eee8d456 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/User.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/User.java @@ -38,8 +38,13 @@ public class User { this.appNames = appNames; } - public User(Integer id, String userName, String role, String appName) { - this.id = id; + public User(String role, List appNames, String app) { + this.role = role; + this.appNames = appNames; + this.appName = app; + } + + public User(String userName, String role, String appName) { this.userName = userName; this.role = role; this.appName = appName; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java new file mode 100644 index 0000000000000000000000000000000000000000..88057c9fabc8548effa88aefa0724dcb33701abb --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java @@ -0,0 +1,175 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import cn.hutool.core.util.StrUtil; +import com.github.benmanes.caffeine.cache.Cache; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.rule.KeyRule; +import com.jd.platform.hotkey.dashboard.cache.CaffeineBuilder; +import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; +import com.jd.platform.hotkey.dashboard.model.KeyTimely; +import com.jd.platform.hotkey.dashboard.util.RuleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.stream.Collectors; + +/** + * 此处存储所有发来的热key，统一处理入库 + * + * @author wuweifeng + * @version 1.0 + * @date 2020-08-31 + */ +public class HotKeyReceiver { + + /** + * 发来的热key集中营 + */ + private static LinkedBlockingQueue hotKeyStoreQueue = new LinkedBlockingQueue<>(); + /** + * 存储实时hotkey，供界面查询实时热key + */ + private static Map> aliveKeyStore = new ConcurrentHashMap<>(); + + private static Logger logger = LoggerFactory.getLogger("HotKeyReceiver"); + + /** + * netty收到的先存这里 + */ + public static void push(HotKeyModel model) { + hotKeyStoreQueue.offer(model); + } + + public static HotKeyModel take() { + try { + return hotKeyStoreQueue.take(); + } catch (InterruptedException e) { + e.printStackTrace(); + return null; + } + } + + /** + * 将热key存入本地缓存，设置过期时间 + */ + public static void put(HotKeyModel hotKeyModel) { + String appNameKey = hotKeyModel.getAppName() + "/" + hotKeyModel.getKey(); + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + logger.error("rule is null, hotkeyModel " + hotKeyModel.getAppName() + "-" + hotKeyModel.getKey()); + return; + } + Cache cache = aliveKeyStore.computeIfAbsent(keyRule.getDuration(), s -> CaffeineBuilder.cache(keyRule.getDuration())); + cache.put(appNameKey, hotKeyModel); + } + + /** + * 展示当前所有的实时热key + */ + public static List list(SearchReq searchReq) { + List timelyList = new ArrayList<>(); + + long now = System.currentTimeMillis(); + + for (Integer duration : aliveKeyStore.keySet()) { + Cache cache = aliveKeyStore.get(duration); + ConcurrentMap concurrentHashMap = cache.asMap(); + for (Map.Entry entry : concurrentHashMap.entrySet()) { + KeyTimely keyTimely = parse((HotKeyModel) entry.getValue(), now); + if (keyTimely == null) { + continue; + } + timelyList.add(keyTimely); + } + } + + if (searchReq != null) { + if (StrUtil.isNotEmpty(searchReq.getApp())) { + timelyList = timelyList.parallelStream().filter(keyTimely -> searchReq.getApp().equals(keyTimely.getAppName())).collect(Collectors.toList()); + } + if (StrUtil.isNotEmpty(searchReq.getKey())) { + timelyList = timelyList.parallelStream().filter(keyTimely -> keyTimely.getKey().startsWith(searchReq.getKey())).collect(Collectors.toList()); + } + + } + + timelyList.sort(Comparator.comparing(KeyTimely::getCreateTime).reversed()); + + return timelyList; + } + + /** + * 将hotkeyModel变成前端需要的对象 + */ + private static KeyTimely parse(HotKeyModel hotKeyModel, long now) { + String appNameKey = hotKeyModel.getAppName() + "/" + hotKeyModel.getKey(); + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + return null; + } + long remainTime = keyRule.getDuration() * 1000 - (now - hotKeyModel.getCreateTime()); + return KeyTimely.aKeyTimely() + .key(hotKeyModel.getKey()) + .val(UUID.randomUUID().toString()) + .appName(hotKeyModel.getAppName()) + .duration(remainTime / 1000) + .ruleDesc(RuleUtil.ruleDesc(appNameKey)) + .createTime(new Date(hotKeyModel.getCreateTime())).build(); + } + +// public static void main(String[] args) { +// List keyTimelyList = new ArrayList<>(); +// for (int i = 0; i < 24; i++) { +// KeyTimely keyTimely = new KeyTimely(); +// keyTimely.setKey(i + ""); +// keyTimely.setCreateTime(new Date()); +// try { +// Thread.sleep(10); +// } catch (InterruptedException e) { +// e.printStackTrace(); +// } +// keyTimelyList.add(keyTimely); +// } +// keyTimelyList.sort(Comparator.comparing(KeyTimely::getCreateTime).reversed()); +// +// Page page =PageUtil.pagination(keyTimelyList, 10, 3); +// System.out.println(page.getTotal()); +// System.out.println(page.getPage()); +// System.out.println(page.getRows()); +// } + + + + + + /** + * 删除实时热key + */ + public static boolean delete(String appNameKey) { + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + return false; + } + Cache cache = aliveKeyStore.get(keyRule.getDuration()); + if (cache == null) { + return false; + } + cache.invalidate(appNameKey); + return true; + } + + /** + * 定时清理caffeine + */ + public static void cleanUpCaffeine() { + for (Integer duration : aliveKeyStore.keySet()) { + Cache cache = aliveKeyStore.get(duration); + cache.cleanUp(); + } + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java new file mode 100644 index 0000000000000000000000000000000000000000..a3dc7eeb5e78b7d6e1364e0b2aca50ddd1ab809e --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java @@ -0,0 +1,78 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import com.jd.platform.hotkey.common.tool.Constant; +import io.netty.bootstrap.ServerBootstrap; +import io.netty.buffer.ByteBuf; +import io.netty.buffer.Unpooled; +import io.netty.channel.*; +import io.netty.channel.nio.NioEventLoopGroup; +import io.netty.channel.socket.nio.NioServerSocketChannel; +import io.netty.handler.codec.DelimiterBasedFrameDecoder; +import io.netty.handler.codec.string.StringDecoder; +import io.netty.handler.logging.LogLevel; +import io.netty.handler.logging.LoggingHandler; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import java.util.concurrent.TimeUnit; + +/** + * 该server用于给各个worker实例连接用。 + * + * @author wuweifeng wrote on 2019-11-05. + */ +@Component +public class NodesServer { + private Logger logger = LoggerFactory.getLogger(getClass()); + + public void startNettyServer(int port) { + //boss单线程 + EventLoopGroup bossGroup = new NioEventLoopGroup(1); + EventLoopGroup workerGroup = new NioEventLoopGroup(4); + try { + ServerBootstrap bootstrap = new ServerBootstrap(); + bootstrap.group(bossGroup, workerGroup) + .channel(NioServerSocketChannel.class) + .handler(new LoggingHandler(LogLevel.INFO)) + .option(ChannelOption.SO_BACKLOG, 1024) + //保持长连接 + .childOption(ChannelOption.SO_KEEPALIVE, true) + //出来网络io事件，如记录日志、对消息编解码等 + .childHandler(new ChildChannelHandler()); + //绑定端口，同步等待成功 + ChannelFuture future = bootstrap.bind(port).sync(); + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + bossGroup.shutdownGracefully (1000, 3000, TimeUnit.MILLISECONDS); + workerGroup.shutdownGracefully (1000, 3000, TimeUnit.MILLISECONDS); + })); + //等待服务器监听端口关闭 + future.channel().closeFuture().sync(); + } catch (Exception e) { + e.printStackTrace(); + logger.error("dashboard netty server start failure"); + } finally { + //优雅退出，释放线程池资源 + bossGroup.shutdownGracefully(); + workerGroup.shutdownGracefully(); + } + } + + /** + * handler类 + */ + private class ChildChannelHandler extends ChannelInitializer { + + @Override + protected void initChannel(Channel ch) { + NodesServerHandler serverHandler = new NodesServerHandler(); + + ByteBuf delimiter = Unpooled.copiedBuffer(Constant.DELIMITER.getBytes()); + ch.pipeline() + .addLast(new DelimiterBasedFrameDecoder(Constant.MAX_LENGTH, delimiter)) + .addLast(new StringDecoder()) + .addLast(serverHandler); + } + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java new file mode 100644 index 0000000000000000000000000000000000000000..9650c2fd9ef5b517eb0769e75aae5fe2fce7d810 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java @@ -0,0 +1,66 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.model.HotKeyMsg; +import com.jd.platform.hotkey.common.model.MsgBuilder; +import com.jd.platform.hotkey.common.model.typeenum.MessageType; +import com.jd.platform.hotkey.common.tool.FastJsonUtils; +import com.jd.platform.hotkey.common.tool.flush.FlushUtil; +import io.netty.channel.ChannelHandlerContext; +import io.netty.channel.SimpleChannelInboundHandler; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.util.StringUtils; + +import java.util.List; + +import static com.jd.platform.hotkey.common.tool.Constant.PONG; + +/** + * 这里处理所有netty事件。 + * + * @author wuweifeng wrote on 2019-11-05. + */ +public class NodesServerHandler extends SimpleChannelInboundHandler { + + private Logger logger = LoggerFactory.getLogger(getClass()); + + @Override + protected void channelRead0(ChannelHandlerContext ctx, String message) { + if (StringUtils.isEmpty(message)) { + return; + } + try { + HotKeyMsg msg = FastJsonUtils.toBean(message, HotKeyMsg.class); + if (MessageType.PING == msg.getMessageType()) { + String hotMsg = FastJsonUtils.convertObjectToJSON(new HotKeyMsg(MessageType.PONG, PONG)); + FlushUtil.flush(ctx, MsgBuilder.buildByteBuf(hotMsg)); + } else if (MessageType.REQUEST_HOT_KEY == msg.getMessageType()) { + List list = FastJsonUtils.toList(msg.getBody(), HotKeyModel.class); + for (HotKeyModel hotKeyModel : list) { + HotKeyReceiver.push(hotKeyModel); + } + } + } catch (Exception e) { + e.printStackTrace(); + } + + } + + @Override + public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception { + logger.error("some thing is error , " + cause.getMessage()); + } + + @Override + public void channelActive(ChannelHandlerContext ctx) throws Exception { + super.channelActive(ctx); + } + + @Override + public void channelInactive(ChannelHandlerContext ctx) throws Exception { + ctx.close(); + super.channelInactive(ctx); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ClearService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ClearService.java new file mode 100644 index 0000000000000000000000000000000000000000..5266ef348bccc0374fdd2c4f670962b94f353dd8 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ClearService.java @@ -0,0 +1,23 @@ +package com.jd.platform.hotkey.dashboard.service; + +import com.github.pagehelper.PageInfo; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; + +/** + * @ProjectName: hotkey + * @ClassName: ClearService + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:51 + */ +public interface ClearService { + + + PageInfo pageClearCfg(PageReq page, String app); + + ClearCfgVo selectClearCfg(String app); + + int saveClearCfg(ClearCfgVo cfg); + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java index d4a06c427db3f5b56cb8183d98a10141dd87e640..558e814e46d03af4e769728cb5bf5631ee4cb367 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java @@ -1,7 +1,7 @@ package com.jd.platform.hotkey.dashboard.service; import com.github.pagehelper.PageInfo; -import com.jd.platform.hotkey.dashboard.common.domain.dto.KeyCountDto; +import com.jd.platform.hotkey.dashboard.common.domain.Page; import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; @@ -30,11 +30,7 @@ public interface KeyService { int delKeyByUser(KeyTimely keyTimely); - KeyTimely selectByKey(String key); - - KeyTimely selectByPk(Long key); - - PageInfo pageKeyTimely(PageReq page, SearchReq param); + Page pageKeyTimely(PageReq page, SearchReq param); PageInfo pageMaxHot(PageReq page, SearchReq param); @@ -42,7 +38,5 @@ public interface KeyService { HotKeyLineChartVo getLineChart(ChartReq chartReq); - HotKeyLineChartVo getQpsLineChart(ChartReq chartReq); - - HotKeyLineChartVo ruleLineChart(SearchReq req); + HotKeyLineChartVo ruleLineChart(SearchReq req, String app); } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java index 8c6ca1086441fa28bd45f3f8a9330e4b98141cdd..5580df1b67ec483f8a65351d61f1f0d0b69a5edb 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java @@ -3,9 +3,11 @@ package com.jd.platform.hotkey.dashboard.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.model.KeyRule; +import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; import com.jd.platform.hotkey.dashboard.model.Rules; +import java.util.List; + /** * @ProjectName: hotkey * @ClassName: RuleService @@ -24,7 +26,11 @@ public interface RuleService { int delRule(String key, String updater); - PageInfo pageKeyRule(PageReq page, SearchReq param); + PageInfo pageKeyRule(PageReq page, String appName); int save(Rules rules); + + List listRules(String app); + + PageInfo pageRuleHitCount(PageReq pageReq, SearchReq req, String ownApp); } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java index 8d374b4c08587358d4506b2244e68fbbdcda4121..71f57ea705fefcbe37dfdd62b940d6bb303ae83b 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java @@ -5,6 +5,7 @@ import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.User; +import javax.servlet.http.Cookie; import java.util.List; /** @@ -22,10 +23,16 @@ public interface UserService { int insertUser(User user); + int insertUserByErp(User user); + + Cookie loginErpUser(User user); + int deleteByPrimaryKey(int id); User selectByPrimaryKey(int id); + User selectByUserName(String userName); + int updateUser(User user); List listApp(); diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ClearServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ClearServiceImpl.java new file mode 100644 index 0000000000000000000000000000000000000000..8227a955ecefbbc9d5ec2d89b6ff414fcc280eaf --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ClearServiceImpl.java @@ -0,0 +1,85 @@ +package com.jd.platform.hotkey.dashboard.service.impl; + +import com.github.pagehelper.PageInfo; +import com.github.pagehelper.util.StringUtil; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; +import com.jd.platform.hotkey.dashboard.service.ClearService; +import com.jd.platform.hotkey.dashboard.service.UserService; +import org.apache.commons.collections4.CollectionUtils; +import org.springframework.stereotype.Service; +import org.springframework.util.StringUtils; +import javax.annotation.Resource; +import java.util.ArrayList; +import java.util.List; + +/** + * @ProjectName: hotkey + * @ClassName: ClearServiceImpl + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:57 + */ +@Service +public class ClearServiceImpl implements ClearService { + + + @Resource + private IConfigCenter configCenter; + + @Resource + private UserService userService; + + + @Override + public PageInfo pageClearCfg(PageReq page, String app) { + List keyValues = configCenter.getPrefix(ConfigConstant.clearCfgPath); + if(CollectionUtils.isEmpty(keyValues)){ + List apps = userService.listApp(); + for (String ap : apps) { + configCenter.put(ConfigConstant.clearCfgPath + ap, Constant.THIRTY_DAY); + } + keyValues = configCenter.getPrefix(ConfigConstant.clearCfgPath); + } + List cfgVos = new ArrayList<>(); + for (KeyValue kv : keyValues) { + String v = kv.getValue().toStringUtf8(); + String key = kv.getKey().toStringUtf8(); + if(StringUtil.isEmpty(v)){ + configCenter.put(key, Constant.THIRTY_DAY); + continue; + } + long version = kv.getModRevision(); + String k = key.replace(ConfigConstant.clearCfgPath,""); + if(StringUtils.isEmpty(app)){ + cfgVos.add(new ClearCfgVo(k, v, version)); + }else{ + if(k.equals(app)){ + cfgVos.add(new ClearCfgVo(k, v, version)); + } + } + } + return new PageInfo<>(cfgVos); + } + + @Override + public ClearCfgVo selectClearCfg(String app) { + KeyValue kv = configCenter.getKv(ConfigConstant.clearCfgPath + app); + if(kv == null || kv.getValue() == null){ + configCenter.put(ConfigConstant.clearCfgPath + app, Constant.THIRTY_DAY); + return new ClearCfgVo(app, Constant.THIRTY_DAY, 0L); + } + String v = kv.getValue().toStringUtf8(); + return new ClearCfgVo(app,v,kv.getModRevision()); + } + + @Override + public int saveClearCfg(ClearCfgVo cfg) { + configCenter.put(ConfigConstant.clearCfgPath + cfg.getApp(), cfg.getTtl()); + return 1; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java index 92011807e8e45d00c1300f6a6914be77a6670f3f..09dbd32d36a61264e8d7743a9e03557f862e0dfa 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java @@ -1,33 +1,40 @@ package com.jd.platform.hotkey.dashboard.service.impl; +import cn.hutool.core.date.SystemClock; +import cn.hutool.core.util.StrUtil; import com.github.pagehelper.PageHelper; import com.github.pagehelper.PageInfo; import com.ibm.etcd.api.Event; import com.jd.platform.hotkey.common.configcenter.ConfigConstant; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.Page; import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HotKeyLineChartVo; -import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; -import com.jd.platform.hotkey.dashboard.common.ex.BizException; +import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; import com.jd.platform.hotkey.dashboard.mapper.KeyRecordMapper; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; -import com.jd.platform.hotkey.dashboard.mapper.ReceiveCountMapper; import com.jd.platform.hotkey.dashboard.mapper.StatisticsMapper; +import com.jd.platform.hotkey.dashboard.model.ChangeLog; import com.jd.platform.hotkey.dashboard.model.KeyRecord; import com.jd.platform.hotkey.dashboard.model.KeyTimely; -import com.jd.platform.hotkey.dashboard.model.ReceiveCount; import com.jd.platform.hotkey.dashboard.model.Statistics; +import com.jd.platform.hotkey.dashboard.netty.HotKeyReceiver; import com.jd.platform.hotkey.dashboard.service.KeyService; +import com.jd.platform.hotkey.dashboard.service.RuleService; import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.DateUtil; +import com.jd.platform.hotkey.dashboard.util.PageUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.springframework.stereotype.Service; import javax.annotation.Resource; +import java.time.Duration; import java.time.LocalDateTime; import java.util.*; import java.util.stream.Collectors; @@ -48,100 +55,67 @@ public class KeyServiceImpl implements KeyService { @Resource private KeyRecordMapper recordMapper; @Resource - private KeyTimelyMapper keyTimelyMapper; + private StatisticsMapper statisticsMapper; @Resource - private ReceiveCountMapper countMapper; + private RuleService ruleService; @Resource - private StatisticsMapper statisticsMapper; - - - public HotKeyLineChartVo ruleLineChart2(SearchReq req) { - int type = req.getType(); - if(req.getEndTime() == null){ - req.setEndTime(new Date()); - } - switch (type){ - case 1: - req.setStartTime(DateUtil.preMinus(30)); - List list = statisticsList(); - System.out.println("30 min"); - break; - case 2: - req.setStartTime(DateUtil.preDays(1)); - System.out.println("24 hours"); - break; - case 3: - req.setStartTime(DateUtil.preDays(7)); - System.out.println("7 days"); - break; - default: - System.out.println("============="); - } - return null; - } + private ChangeLogMapper logMapper; + private Logger logger = LoggerFactory.getLogger(getClass()); + /** + * 折线图 + * + * @param req req + * @return vo + */ @Override - public HotKeyLineChartVo ruleLineChart(SearchReq req) { + public HotKeyLineChartVo ruleLineChart(SearchReq req, String app) { int type = req.getType(); - if(req.getEndTime() == null){ - req.setEndTime(new Date()); + String appReq = req.getApp(); + // admin 全查 + if (StrUtil.isNotEmpty(appReq)) { + app = appReq; + } + req.setApp(null); + LocalDateTime now = LocalDateTime.now(); + req.setEndTime(req.getEndTime() == null ? DateUtil.ldtToDate(now) : req.getEndTime()); + List rules = ruleService.listRules(null); + if (type == 4) { + LocalDateTime st = req.getStartTime() == null ? now.minusMinutes(31) : DateUtil.dateToLdt(req.getStartTime()); + req.setStartTime(DateUtil.ldtToDate(st)); + LocalDateTime et = DateUtil.dateToLdt(req.getEndTime()); + boolean longTime = Duration.between(st, et).toHours() > 2; + req.setType(longTime ? 6 : 5); + List list = statisticsMapper.listOrderByTime(req); + return CommonUtil.processData(st, et, list, !longTime, rules, app); } - switch (type){ - case 1: - req.setStartTime(DateUtil.preMinus(30)); - List list = statisticsList(); - Map map = new HashMap<>(10); - Map> listMap = list.stream().collect(Collectors.groupingBy(Statistics::getKeyName)); - for (Map.Entry> m : listMap.entrySet()) { - int start = 1; - map.put(m.getKey(),new int[30]); - int[] data = map.get(m.getKey()); - int tmp = 0; - for (int i = 0; i < 30; i++) { - Statistics st; - try { - st = m.getValue().get(tmp); - if(String.valueOf(start).endsWith("24")){ start = start + 77; } - if(start != st.getHours()){ - data[i] = 0; - }else{ - tmp ++; - data[i] = st.getCount(); - } - start++; - }catch (Exception e){ - data[i] = 0; - } - } - } - System.out.println("30 min"); - break; - case 2: - req.setStartTime(DateUtil.preDays(1)); - System.out.println("24 hours"); - break; - case 3: - req.setStartTime(DateUtil.preDays(7)); - System.out.println("7 days"); - break; - default: - System.out.println("============="); + if (type == 5) { + LocalDateTime startTime = now.minusMinutes(31); + req.setStartTime(DateUtil.ldtToDate(startTime)); + List list = statisticsMapper.listOrderByTime(req); + return CommonUtil.processData(startTime, now, list, true, rules, app); + } else if (type == 6) { + LocalDateTime startTime2 = now.minusHours(25); + req.setStartTime(DateUtil.ldtToDate(startTime2)); + List list2 = statisticsMapper.listOrderByTime(req); + return CommonUtil.processData(startTime2, now, list2, false, rules, app); + } else { + LocalDateTime startTime3 = now.minusDays(7).minusHours(1); + req.setStartTime(DateUtil.ldtToDate(startTime3)); + req.setType(6); + List list3 = statisticsMapper.listOrderByTime(req); + return CommonUtil.processData(startTime3, now, list3, false, rules, app); } - return null; } @Override - public PageInfo pageKeyTimely(PageReq page, SearchReq param) { - PageHelper.startPage(page.getPageNum(), page.getPageSize()); - List listKey = keyTimelyMapper.listKeyTimely(param); - for (KeyTimely timely : listKey) { - timely.setKey(CommonUtil.keyName(timely.getKey())); - timely.setRuleDesc(RuleUtil.ruleDesc(timely.getAppName() + "/" + timely.getKey())); - } - return new PageInfo<>(listKey); + public Page pageKeyTimely(PageReq page, SearchReq param) { + List keyTimelies = HotKeyReceiver.list(param); + return PageUtil.pagination(keyTimelies, page.getPageSize(), page.getPageNum()-1); + } @Override @@ -163,57 +137,20 @@ public class KeyServiceImpl implements KeyService { public HotKeyLineChartVo getLineChart(ChartReq chartReq) { int hours = 6; // 默认查询6小时内的数据 - if(chartReq.getStartTime() == null || chartReq.getEndTime() == null){ - chartReq.setStartTime(DateUtil.preTime(hours)); - chartReq.setEndTime(new Date()); - } - - List statistics = statisticsMapper.listStatistics(chartReq); + SearchReq req = new SearchReq(); + req.setStartTime(DateUtil.preTime(hours)); + req.setEndTime(new Date()); + List statistics = statisticsMapper.listStatistics(req); // 获取data Y轴 Map keyDateMap = keyDateMap(statistics, hours); // 获取时间x轴 List list = new ArrayList<>(); - for (int i = hours; i >0 ; i--) { - LocalDateTime time = LocalDateTime.now().minusHours(i-1); + for (int i = hours; i > 0; i--) { + LocalDateTime time = LocalDateTime.now().minusHours(i - 1); int hour = time.getHour(); - list.add(hour+"时"); - } - return new HotKeyLineChartVo(list,keyDateMap); - } - - - @Override - public HotKeyLineChartVo getQpsLineChart(ChartReq chartReq) { - if(chartReq.getStartTime() == null || chartReq.getEndTime() == null){ - /* chartReq.setStartTime(DateUtil.preTime()); - chartReq.setEndTime(new Date());*/ + list.add(hour + "时"); } - List countList = countMapper.list(chartReq); - Map map = new HashMap<>(10); - Set minutes = new HashSet<>(); - List> workerList = new ArrayList<>(); - countList.stream().collect(Collectors.groupingBy(ReceiveCount::getWorkerName,Collectors.toList())) - .forEach((name,data)-> workerList.add(data)); - for (List cts : workerList) { - int size = cts.size(); - for (int i = 0; i < size; i++) { - ReceiveCount dto = cts.get(i); - String k = dto.getWorkerName(); - Long v = dto.getReceiveCount(); - Integer ms = dto.getMinutes(); - minutes.add(ms.toString()); - if(map.get(k) == null){ - int [] data = new int[size]; - data[i] = v.intValue(); - map.put(k, data); - }else{ - int [] data = map.get(k); - data[i] = v.intValue(); - map.put(k, data); - } - } - } - return new HotKeyLineChartVo(new ArrayList<>(minutes),map); + return new HotKeyLineChartVo(list, keyDateMap); } @@ -233,7 +170,15 @@ public class KeyServiceImpl implements KeyService { public int insertKeyByUser(KeyTimely key) { configCenter.putAndGrant(ConfigConstant.hotKeyPath + key.getAppName() + "/" + key.getKey(), System.currentTimeMillis() + "", key.getDuration()); - return 1; + + //写入本地缓存，实时热key信息 + HotKeyModel hotKeyModel = new HotKeyModel(); + hotKeyModel.setCreateTime(System.currentTimeMillis()); + hotKeyModel.setAppName(key.getAppName()); + hotKeyModel.setKey(key.getKey()); + HotKeyReceiver.put(hotKeyModel); + return logMapper.insertSelective(new ChangeLog(key.getAppName(), Constant.HOTKEY_CHANGE, "", + key.getKey(), key.getUpdater(), SystemClock.now() + "")); } @Override @@ -248,54 +193,47 @@ public class KeyServiceImpl implements KeyService { //app + "_" + key String[] arr = keyTimely.getKey().split("/"); //删除client监听目录的key - String ectdKey = ConfigConstant.hotKeyPath + arr[0] + "/" + arr[1]; - configCenter.delete(ectdKey); - //也删除Record目录下的该key，因为不确定要删的key到底在哪 - String recordKey = ConfigConstant.hotKeyRecordPath + arr[0] + "/" + arr[1]; - configCenter.delete(recordKey); - - KeyRecord keyRecord = new KeyRecord(arr[1], "", arr[0], 0L, Constant.HAND, - Event.EventType.DELETE_VALUE, UUID.randomUUID().toString(), new Date()); + String etcdKey = ConfigConstant.hotKeyPath + arr[0] + "/" + arr[1]; - recordMapper.insertSelective(keyRecord); - - return 1; - } + //删除caffeine里的实时key + HotKeyReceiver.delete(arr[0] + "/" + arr[1]); - @Override - public KeyTimely selectByKey(String key) { - return keyTimelyMapper.selectByKey(key); - } + if (configCenter.get(etcdKey) == null) { + //如果手工目录也就是client监听的目录里没有该key，那么就往里面放一个，然后再删掉它，这样client才能监听到删除事件 + configCenter.putAndGrant(etcdKey, com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE, 1); + } + configCenter.delete(etcdKey); - @Override - public KeyTimely selectByPk(Long id) { - return keyTimelyMapper.selectByPrimaryKey(id); + KeyRecord keyRecord = new KeyRecord(arr[1], "", arr[0], 0, Constant.HAND, + Event.EventType.DELETE_VALUE, UUID.randomUUID().toString(), new Date()); + recordMapper.insertSelective(keyRecord); + return logMapper.insertSelective(new ChangeLog(keyTimely.getKey(), Constant.HOTKEY_CHANGE, keyTimely.getKey(), "", keyTimely.getUpdater(), SystemClock.now() + "")); } - - - private Map keyDateMap(List statistics, int hours){ + private Map keyDateMap(List statistics, int hours) { Map map = new HashMap<>(10); Map> listMap = statistics.stream().collect(Collectors.groupingBy(Statistics::getKeyName)); for (Map.Entry> m : listMap.entrySet()) { - int start = DateUtil.preHours(LocalDateTime.now(),5); - map.put(m.getKey(),new int[hours]); + int start = DateUtil.preHoursInt(5); + map.put(m.getKey(), new int[hours]); int[] data = map.get(m.getKey()); int tmp = 0; for (int i = 0; i < hours; i++) { Statistics st; try { st = m.getValue().get(tmp); - if(String.valueOf(start).endsWith("24")){ start = start + 77; } - if(start != st.getHours()){ + if (String.valueOf(start).endsWith("24")) { + start = start + 77; + } + if (start != st.getHours()) { data[i] = 0; - }else{ - tmp ++; + } else { + tmp++; data[i] = st.getCount(); } start++; - }catch (Exception e){ + } catch (Exception e) { data[i] = 0; } } @@ -304,9 +242,8 @@ public class KeyServiceImpl implements KeyService { } - private void checkParam(SearchReq req) { - if(req.getStartTime() == null || req.getEndTime() == null){ + if (req.getStartTime() == null || req.getEndTime() == null) { req.setStartTime(DateUtil.preTime(5)); req.setEndTime(new Date()); } @@ -316,51 +253,6 @@ public class KeyServiceImpl implements KeyService { }*/ } - - private List statisticsList(){ - Random rd = new Random(); - List list = new ArrayList<>(); - for (int i = 0; i < 30 ; i++) { - Statistics st = new Statistics(); - st.setApp("rule1"); - st.setKeyName("key1"); - st.setCount(rd.nextInt(100)); - st.setBizType(1); - st.setMinutes(2006052140+i); - if(String.valueOf(st.getMinutes()).endsWith("60")){ - st.setMinutes(st.getMinutes()+1); - } - list.add(st); - } - List list2 = new ArrayList<>(); - for (int i = 0; i < 30 ; i++) { - Statistics st2 = new Statistics(); - st2.setApp("rule2"); - st2.setKeyName("key2"); - st2.setCount(rd.nextInt(100)); - st2.setBizType(1); - st2.setMinutes(2006052140+i); - list2.add(st2); - } - list.addAll(list2); - return list; - } - - private List statisticsList1(){ - Random rd = new Random(); - List list = new ArrayList<>(); - for (int i = 0; i < 24 ; i++) { - Statistics st = new Statistics(); - st.setApp("rule1"); - st.setKeyName("key1"); - st.setCount(rd.nextInt(100)); - st.setBizType(1); - st.setHours(20060500+i); - } - return list; - } - - } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java index 6d5a4fdba419d0d72d03a58b8710e7b0e7641163..9dd493e79c3fce3177b9565f5a78f70e18ada0ef 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java @@ -2,6 +2,7 @@ package com.jd.platform.hotkey.dashboard.service.impl; import cn.hutool.core.lang.UUID; import com.alibaba.fastjson.JSON; +import com.github.pagehelper.PageHelper; import com.github.pagehelper.PageInfo; import com.github.pagehelper.util.StringUtil; import com.ibm.etcd.api.KeyValue; @@ -9,14 +10,15 @@ import com.jd.platform.hotkey.common.configcenter.ConfigConstant; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; import com.jd.platform.hotkey.dashboard.mapper.RulesMapper; -import com.jd.platform.hotkey.dashboard.model.ChangeLog; -import com.jd.platform.hotkey.dashboard.model.Rule; -import com.jd.platform.hotkey.dashboard.model.Rules; +import com.jd.platform.hotkey.dashboard.mapper.SummaryMapper; +import com.jd.platform.hotkey.dashboard.model.*; import com.jd.platform.hotkey.dashboard.service.RuleService; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; +import org.springframework.util.StringUtils; import javax.annotation.Resource; import java.util.ArrayList; @@ -43,6 +45,8 @@ public class RuleServiceImpl implements RuleService { @Resource private ChangeLogMapper logMapper; + @Resource + private SummaryMapper summaryMapper; @Override @@ -94,20 +98,23 @@ public class RuleServiceImpl implements RuleService { } @Override - public PageInfo pageKeyRule(PageReq page, SearchReq param) { - String app = param.getAppName(); - String prefix = StringUtil.isEmpty(app) ? ConfigConstant.rulePath : ConfigConstant.rulePath + app; - List keyValues = configCenter.getPrefix(prefix); + public PageInfo pageKeyRule(PageReq page, String appName) { + List keyValues = configCenter.getPrefix(ConfigConstant.rulePath); List rules = new ArrayList<>(); for (KeyValue kv : keyValues) { - String v = kv.getValue().toStringUtf8(); if(StringUtil.isEmpty(v)){ continue; } String key = kv.getKey().toStringUtf8(); String k = key.replace(ConfigConstant.rulePath,""); - rules.add(new Rules(k, v)); + if(StringUtils.isEmpty(appName)){ + rules.add(new Rules(k, v)); + }else{ + if(k.equals(appName)){ + rules.add(new Rules(k, v)); + } + } } return new PageInfo<>(rules); } @@ -116,17 +123,37 @@ public class RuleServiceImpl implements RuleService { public int save(Rules rules) { String app = rules.getApp(); String from = ""; -// Rules oldRules = rulesMapper.select(app); -// if(oldRules == null){ -// rulesMapper.insert(rules); -// }else{ -// from = JSON.toJSONString(oldRules); -// rulesMapper.update(rules); -// } String to = JSON.toJSONString(rules); -// logMapper.insertSelective(new ChangeLog(app, 1, from, to, -// rules.getUpdateUser(), app, SystemClock.nowDate())); configCenter.put(ConfigConstant.rulePath + app, rules.getRules()); return 1; } + + @Override + public List listRules(String app) { + List keyValues = configCenter.getPrefix(ConfigConstant.rulePath); + List rules = new ArrayList<>(); + for (KeyValue kv : keyValues) { + String v = kv.getValue().toStringUtf8(); + if(StringUtil.isEmpty(v)){ + continue; + } + String key = kv.getKey().toStringUtf8(); + String appKey = key.replace(ConfigConstant.rulePath,""); + List rs = JSON.parseArray(v, Rule.class); + for (Rule r : rs) { + rules.add(appKey+"-"+r.getKey()); + } + } + return rules; + } + + @Override + public PageInfo pageRuleHitCount(PageReq pageReq, SearchReq req, String ownApp) { + PageHelper.startPage(pageReq.getPageNum(),pageReq.getPageSize()); + List hitCountVos = summaryMapper.listRuleHitCount(req); + return new PageInfo<>(hitCountVos); + } + + + } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java index b77f6956dc5ed56d5068ab2bccb9358d298539fc..c03ea2ed1d6f3bda40edf455614169eab86c4c9e 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java @@ -11,11 +11,14 @@ import com.jd.platform.hotkey.dashboard.common.ex.BizException; import com.jd.platform.hotkey.dashboard.mapper.UserMapper; import com.jd.platform.hotkey.dashboard.model.User; import com.jd.platform.hotkey.dashboard.service.UserService; +import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; +import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Service; import org.springframework.util.DigestUtils; import org.springframework.util.StringUtils; import javax.annotation.Resource; +import javax.servlet.http.Cookie; import java.util.Date; import java.util.HashSet; import java.util.List; @@ -31,6 +34,8 @@ import java.util.Set; @Service public class UserServiceImpl implements UserService { + @Value("${erp.defaultPwd:123}") + private String defaultPwd; @Resource private UserMapper userMapper; @Resource @@ -63,6 +68,47 @@ public class UserServiceImpl implements UserService { return userMapper.insertSelective(user); } + @Override + public int insertUserByErp(User user) { + User userParam = new User(); + userParam.setUserName(user.getUserName()); + List users = userMapper.selectHkUserList(userParam); + if(users.size() == 0){ + user.setCreateTime(new Date()); + user.setPwd(DigestUtils.md5DigestAsHex(defaultPwd.getBytes())); + int ret = userMapper.insertSelective(user); + System.out.println(user.getId()); + return ret; + } + return 0; + } + + @Override + public Cookie loginErpUser(User user){ + User userParam = new User(); + userParam.setUserName(user.getUserName()); + List users = userMapper.selectHkUserList(userParam); + if(users.size() == 0){ + user.setCreateTime(new Date()); + user.setPwd(DigestUtils.md5DigestAsHex(defaultPwd.getBytes())); + userMapper.insertSelective(user); + String token = JwtTokenUtil.createJWT(user.getId(), user.getUserName(), "", user.getAppName(), user.getNickName()); + Cookie cookie = new Cookie("token", JwtTokenUtil.TOKEN_PREFIX + token); + cookie.setMaxAge(3600*24*7); + //cookie.setDomain("localhost"); + cookie.setPath("/"); + return cookie; + }else{ + user =users.get(0); + String token = JwtTokenUtil.createJWT(user.getId(), user.getUserName(), "", user.getAppName(), user.getNickName()); + Cookie cookie = new Cookie("token", JwtTokenUtil.TOKEN_PREFIX + token); + cookie.setMaxAge(3600*24*7); + //cookie.setDomain("localhost"); + cookie.setPath("/"); + return cookie; + } + } + @Override public int deleteByPrimaryKey(int id) { return userMapper.deleteByPrimaryKey(id); @@ -73,6 +119,11 @@ public class UserServiceImpl implements UserService { return userMapper.selectByPrimaryKey(id); } + @Override + public User selectByUserName(String userName) { + return userMapper.selectByUserName(userName); + } + @Override public int updateUser(User user) { if(!StringUtils.isEmpty(user.getPwd())){ diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java index 42c466777a82a4456eddbc8a8be326b6985a4d4c..21488acba6910f019869940523bdfda0eae2b0a3 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java @@ -1,66 +1,245 @@ package com.jd.platform.hotkey.dashboard.util; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.vo.HotKeyLineChartVo; +import com.jd.platform.hotkey.dashboard.model.Statistics; +import com.jd.platform.hotkey.dashboard.model.Summary; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.util.StringUtils; + import java.io.UnsupportedEncodingException; -import java.util.Base64; +import java.math.BigDecimal; +import java.time.Duration; +import java.time.LocalDateTime; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArraySet; +import java.util.stream.Collectors; public class CommonUtil { - /** - * 获取父级Key - * @param key key - * @return string - */ - public static String parentK(String key){ - if(key.endsWith("/")){ - key = key.substring(0,key.length()-1); - } - int index = key.lastIndexOf("/"); - return key.substring(0,index+1); - } - - /** - * 获取AppName - * @param k k - * @return str - */ - public static String appName(String k){ - String[] arr = k.split("/"); - for (int i = 0; i < arr.length ; i++) { - if(i == 3){ - return arr[i]; - } - } - return null; - } - - - - public static String keyName(String k){ - int index = k.lastIndexOf("/"); - return k.substring(index+1); - } - - - - public static String encoder(String text) { - try { - return Base64.getEncoder().encodeToString(text.getBytes("utf-8")); - } catch (UnsupportedEncodingException e) { - e.printStackTrace(); - } - return ""; - } - - - public static String decoder(String text) { - byte[] bytes=Base64.getDecoder().decode(text); - try { - return new String(bytes,"utf-8"); - } catch (UnsupportedEncodingException e) { - e.printStackTrace(); - } - return ""; - } + private static Logger log = LoggerFactory.getLogger(CommonUtil.class); + + + /** + * 获取父级Key + * + * @param key key + * @return string + */ + public static String parentK(String key) { + if (key.endsWith("/")) { + key = key.substring(0, key.length() - 1); + } + int index = key.lastIndexOf("/"); + return key.substring(0, index + 1); + } + + /** + * 获取AppName + * + * @param k k + * @return str + */ + public static String appName(String k) { + String[] arr = k.split("/"); + for (int i = 0; i < arr.length; i++) { + if (i == 3) { + return arr[i]; + } + } + return null; + } + + + public static String keyName(String k) { + int index = k.lastIndexOf("/"); + return k.substring(index + 1); + } + + + public static String encoder(String text) { + try { + return Base64.getEncoder().encodeToString(text.getBytes("utf-8")); + } catch (UnsupportedEncodingException e) { + e.printStackTrace(); + } + return ""; + } + + + public static String decoder(String text) { + byte[] bytes = Base64.getDecoder().decode(text); + try { + return new String(bytes, "utf-8"); + } catch (UnsupportedEncodingException e) { + e.printStackTrace(); + } + return ""; + } + + + /** + * 拼装数据 + * + * @param list list-data + * @param startTime 开始时间 + * @param size 格子数 + * @param type 类型 1分钟 2小时 + * @return vo + */ + public static HotKeyLineChartVo assembleData(List list, LocalDateTime startTime, int size, int type) { + Set set = new TreeSet<>(); + boolean isHour = type == 1; + String suffix = isHour ? "60" : "24"; + String pattern = isHour ? DateUtil.PATTERN_MINUS : DateUtil.PATTERN_HOUR; + Map map = new HashMap<>(10); + Map> listMap = listGroup(list); +// log.info("按照rule分组以后的listMap--> {}", JSON.toJSONString(listMap)); + for (Map.Entry> m : listMap.entrySet()) { + int start = DateUtil.reviseTime(startTime, 0, type); + map.put(m.getKey(), new int[size]); + int[] data = map.get(m.getKey()); + int tmp = 0; + for (int i = 0; i < size; i++) { + if (String.valueOf(start).endsWith(suffix)) { + LocalDateTime tmpTime = DateUtil.strToLdt((start - 1) + "", pattern); + start = DateUtil.reviseTime(tmpTime, 1, type); + } +// log.info("start--> {}, tmp---> {} ", start, tmp); + set.add(DateUtil.strToLdt(start + "", pattern).toString().replace("T", " ")); + Statistics st = m.getValue().get(tmp); + int val = isHour ? st.getMinutes() : st.getHours(); + if (start != val) { + data[i] = 0; + } else { + tmp++; + data[i] = st.getCount(); + } + start++; + } + } + return new HotKeyLineChartVo(new ArrayList<>(set), map); + } + + + /** + * 分组 + * + * @param list list + * @return map + */ + private static Map> listGroup(List list) { + if (Constant.VERSION != 1) { + return list.stream().collect(Collectors.groupingBy(Statistics::getRule)); + } + return list.stream().collect(Collectors.groupingBy(Statistics::getKeyName)); + } + + /** + * 分组 + * + * @param list list + * @return map + */ + private static Map> listGroupByTime(List list, boolean isMinute) { + if (isMinute) { + return list.stream().collect(Collectors.groupingBy(Statistics::getMinutes)); + } + return list.stream().collect(Collectors.groupingBy(Statistics::getHours)); + } + + + /** + * 处理数据 + * + * @param st 开始时间 + * @param et 结束时间 + * @param list 数据 + * @param isMinute 类型 + * @return vo + */ + public static HotKeyLineChartVo processData(LocalDateTime st, LocalDateTime et, List list, + boolean isMinute, List rules, String app) { + Set xAxisSet = new TreeSet<>(); + Duration duration = Duration.between(st, et); + long passTime = isMinute ? duration.toMinutes() : duration.toHours(); + Map timeCountMap = new TreeMap<>(); + String pattern = isMinute ? DateUtil.PATTERN_MINUS : DateUtil.PATTERN_HOUR; + for (int i = 1; i < passTime; i++) { + int time = DateUtil.reviseTime(st, i, isMinute ? 1 : 2); + xAxisSet.add(DateUtil.formatTime(time, pattern)); + timeCountMap.put(time, null); + } + Map> ruleStatsMap = listGroup(list); + Map> ruleDataMap = new ConcurrentHashMap<>(ruleStatsMap.size()); + ruleStatsMap.forEach((rule, statistics) -> { + Map> timeStatsMap = listGroupByTime(statistics, isMinute); + timeCountMap.forEach((k, v) -> { + if (timeStatsMap.get(k) == null) { + timeCountMap.put(k, 0); + } else { + timeCountMap.put(k, timeStatsMap.get(k).get(0).getCount()); + } + }); + ruleDataMap.put(rule, new ArrayList<>(timeCountMap.values())); + }); + HotKeyLineChartVo vo = new HotKeyLineChartVo(); + vo.setxAxis2(xAxisSet); + if (!StringUtils.isEmpty(app)) { + ruleDataMap.forEach((rule, data) -> { + if (!rule.startsWith(app)) { + ruleDataMap.remove(rule); + } + }); + } + vo.setSeries2(ruleDataMap); + Set ruleSet = ruleDataMap.keySet(); + Set etcdRuleSet = new HashSet<>(rules); + Set legend = new CopyOnWriteArraySet<>(etcdRuleSet); + legend.retainAll(ruleSet); + if (!StringUtils.isEmpty(app)) { + legend.forEach(x -> { + if (!x.startsWith(app)) { + legend.remove(x); + } + }); + } + vo.setLegend(legend); + return vo; + } + + + /** + * build存入对象 + * + * @param key key是 appName + #**# + pin__#**#2020-10-23 21:11:22 + * @param map value是"67-1937" 前面是热key访问量，后面是总访问量 + * @return Summary + */ + public static Summary buildSummary(String key, Map map) { + String[] args = key.split(com.jd.platform.hotkey.common.tool.Constant.BAK_DELIMITER); + String app = args[0]; + String rule = args[1]; + String hitTime = args[2]; + Date time = DateUtil.strToDate(hitTime); + assert time != null; + LocalDateTime ldt = DateUtil.dateToLdt(time); + int day = DateUtil.nowDay(ldt); + int hour = DateUtil.nowHour(ldt); + int minus = DateUtil.nowMinus(ldt); + long seconds = time.getTime() / 1000; + String[] counts = map.get(key).split("-"); + int hitCount = Integer.parseInt(counts[0]); + int totalCount = Integer.parseInt(counts[1]); + String uuid = app + "-" + rule + hitTime; + + return Summary.aSummary().indexName(rule).rule(rule).app(app) + .val1(totalCount).val2(hitCount).val3(BigDecimal.ZERO) + .days(day).hours(hour).minutes(minus).seconds((int) seconds) + .bizType(1).uuid(uuid).createTime(new Date()).build(); + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/DateUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/DateUtil.java index 8864fe61e0d9dd0164a9dc51858bbb99d430aed9..8232567181b3a5381d52c6bb894f9d21d6daf8a2 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/DateUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/DateUtil.java @@ -12,14 +12,24 @@ import java.util.Date; */ public class DateUtil { + public static final String PATTERN_SECONDS="yyMMddHHmmss"; - private static SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); + public static final String PATTERN_MINUS="yyMMddHHmm"; + + public static final String PATTERN_HOUR="yyMMddHH"; + + public static final String PATTERN_DAY="yyMMdd"; + + private static final DateTimeFormatter FORMAT_SECONDS = DateTimeFormatter.ofPattern(PATTERN_SECONDS); + + private static final DateTimeFormatter FORMAT_MINUS = DateTimeFormatter.ofPattern(PATTERN_MINUS); + + private static final DateTimeFormatter FORMAT_HOUR = DateTimeFormatter.ofPattern(PATTERN_HOUR); - public static final DateTimeFormatter TIME_FORMAT1 = DateTimeFormatter.ofPattern("yyMMddHHmm"); + private static final DateTimeFormatter FORMAT_DAY = DateTimeFormatter.ofPattern(PATTERN_DAY); - public static final DateTimeFormatter TIME_FORMAT2 = DateTimeFormatter.ofPattern("yyMMddHH"); + private static SimpleDateFormat simpleDateFormat = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss"); - public static final DateTimeFormatter TIME_FORMAT3 = DateTimeFormatter.ofPattern("yyMMddHH"); public static Date strToDate(String str){ try { @@ -30,37 +40,73 @@ public class DateUtil { return null; } + public static LocalDateTime strToLdt(String str, String pattern){ + return LocalDateTime.parse(str, DateTimeFormatter.ofPattern(pattern)); + } + + + public static int reviseTime(LocalDateTime time, int diff, int type){ + switch (type){ + case 0: + return Integer.parseInt(FORMAT_SECONDS.format(time.plusSeconds(diff))); + case 1: + return Integer.parseInt(FORMAT_MINUS.format(time.plusMinutes(diff))); + case 2: + return Integer.parseInt(FORMAT_HOUR.format(time.plusHours(diff))); + case 3: + return Integer.parseInt(FORMAT_DAY.format(time.plusDays(diff))); + default: + } + return 0; + } - public static Date localDateTimeToDate(LocalDateTime localDateTime){ + + + + public static Date ldtToDate(LocalDateTime localDateTime){ return Date.from(localDateTime.atZone(ZoneId.systemDefault()).toInstant()); } + public static LocalDateTime dateToLdt(Date date){ + return LocalDateTime.ofInstant( date.toInstant(), ZoneId.systemDefault()); + } + + public static int nowMinus(LocalDateTime now){ - return Integer.parseInt(now.format(TIME_FORMAT1)) ; + return Integer.parseInt(now.format(FORMAT_MINUS)) ; } public static int nowHour(LocalDateTime now){ - return Integer.parseInt(now.format(TIME_FORMAT2)); + return Integer.parseInt(now.format(FORMAT_HOUR)); } - public static int nowDay(LocalDateTime now){ return Integer.parseInt(now.format(TIME_FORMAT3));} + public static int nowDay(LocalDateTime now){ return Integer.parseInt(now.format(FORMAT_DAY));} - public static int preHours(LocalDateTime now, int hours){ - return Integer.parseInt(now.minusHours(hours).format(TIME_FORMAT2)); + public static int preHoursInt(int hours){ + return Integer.parseInt(LocalDateTime.now().minusHours(hours).format(FORMAT_HOUR)); + } + + public static Date preHours(int hours){ + return ldtToDate(LocalDateTime.now().minusHours(hours)); } public static Date preTime(int hours){ - return localDateTimeToDate(LocalDateTime.now().minusHours(hours)); + return ldtToDate(LocalDateTime.now().minusHours(hours)); } public static Date preMinus(int minus){ - return localDateTimeToDate(LocalDateTime.now().minusMinutes(minus)); + return ldtToDate(LocalDateTime.now().minusMinutes(minus)); } public static Date preDays(int days){ - return localDateTimeToDate(LocalDateTime.now().minusDays(days)); + return ldtToDate(LocalDateTime.now().minusDays(days)); } + + public static String formatTime(int time,String pattern){ + return strToLdt(time+"", pattern).toString().replace("T", " "); + } + } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/ExcelUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/ExcelUtil.java index 1c6370aaebe7d394a77a1737b94a2807f0a6ebe4..7641f0e14dad9b90f47dcaea08b08564fc837eec 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/ExcelUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/ExcelUtil.java @@ -4,7 +4,7 @@ import javax.servlet.http.HttpServletResponse; import java.io.OutputStream; import java.util.List; -import com.jd.platform.hotkey.dashboard.common.domain.ExcelData; +import com.jd.platform.hotkey.dashboard.common.domain.dto.ExcelDataDto; import org.apache.poi.hssf.usermodel.HSSFDataFormat; import org.apache.poi.xssf.usermodel.*; import org.slf4j.Logger; @@ -15,7 +15,7 @@ public class ExcelUtil { private static Logger log = LoggerFactory.getLogger(ExcelUtil.class); - public static void exportExcel(HttpServletResponse response, ExcelData data) { + public static void exportExcel(HttpServletResponse response, ExcelDataDto data) { log.info("导出解析开始，fileName:{}",data.getFileName()); try { //实例化HSSFWorkbook diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/JwtTokenUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/JwtTokenUtil.java index 57c111eabe8da46304fd26d9e677e623bedcfaba..9e931e18383bfe5f89643af7dbfaeb4630b4acf4 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/JwtTokenUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/JwtTokenUtil.java @@ -1,6 +1,5 @@ package com.jd.platform.hotkey.dashboard.util; -import com.jd.platform.hotkey.dashboard.model.User; import io.jsonwebtoken.*; import org.apache.logging.log4j.util.Base64Util; import org.apache.tomcat.util.codec.binary.Base64; @@ -8,9 +7,10 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import javax.crypto.spec.SecretKeySpec; +import javax.servlet.http.Cookie; +import javax.servlet.http.HttpServletRequest; import javax.xml.bind.DatatypeConverter; import java.security.Key; -import java.text.SimpleDateFormat; import java.util.Date; @@ -47,7 +47,7 @@ public class JwtTokenUtil { * @param role * @return */ - public static String createJWT(Integer userId, String username, String role, String appName) { + public static String createJWT(Integer userId, String username, String role, String appName, String nickName) { try { // 使用HS256加密算法 SignatureAlgorithm signatureAlgorithm = SignatureAlgorithm.HS256; @@ -62,7 +62,7 @@ public class JwtTokenUtil { JwtBuilder builder = Jwts.builder().setHeaderParam("type", "JWT") .claim("userId", encryId) .claim("role", role) - .claim("appName", appName) + .claim("nickName", nickName) .setSubject(username) // 代表这个JWT的主体，即它的所有人 // .setIssuer(audience.getClientId()) // 代表这个JWT的签发主体； .setIssuedAt(new Date()) // 是一个时间戳，代表这个JWT的签发时间； @@ -73,9 +73,6 @@ public class JwtTokenUtil { if (TTLMillis >= 0) { long expMillis = nowMillis + TTLMillis; Date exp = new Date(expMillis); - System.out.println("当前时间： "+new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(new Date())); - - System.out.println("过期时间： "+new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(exp)); builder.setExpiration(exp); // 是一个时间戳，代表这个JWT的过期时间； // .setNotBefore(now); // 是一个时间戳，代表这个JWT生效的开始时间，意味着在这个时间之前验证JWT是会失败的 } @@ -89,16 +86,6 @@ public class JwtTokenUtil { } - /** - * 从token中获取appName - * @param token - * @return - */ - public static String getAppName(String token){ - Claims re = parseJWT(token); - return re.get("appName", String.class); - } - /** * 从token中获取role @@ -131,11 +118,8 @@ public class JwtTokenUtil { } - public static User userPower(String token){ - Claims claims = parseJWT(token); - String role = claims.get("role",String.class); - String appName = claims.get("appName",String.class); - return new User(role,appName); + public static Claims claims(String token){ + return parseJWT(token); } /** @@ -147,6 +131,14 @@ public class JwtTokenUtil { return parseJWT(token).getExpiration().before(new Date()); } - + public static String getAuthHeader(HttpServletRequest request){ + Cookie[] cookies = request.getCookies(); + for(Cookie cookie : cookies){ + if("token".equals(cookie.getName())){ + return cookie.getValue(); + } + } + return null; + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java new file mode 100644 index 0000000000000000000000000000000000000000..cd4a9872edd3f4e64af16860b707aabcf41e2ba5 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java @@ -0,0 +1,36 @@ +package com.jd.platform.hotkey.dashboard.util; + +import com.google.common.collect.Lists; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import org.springframework.util.CollectionUtils; + +import java.util.ArrayList; +import java.util.List; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-09-01 + */ +public class PageUtil { + /** + * 通用分页工具类 + * + * @param data + * @param pageSize + * @param pageNum + * @param + * @return + */ + public static Page pagination(final List data, final int pageSize, final int pageNum) { + if (CollectionUtils.isEmpty(data)) { + return new Page<>(1, 0, new ArrayList<>()); + } + List> lists = Lists.partition(data, pageSize); + int localPageNum = pageNum; + if (localPageNum >= lists.size()) { + localPageNum = lists.size() - 1; + } + return new Page<>(localPageNum, data.size(), lists.get(localPageNum)); + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java index 01bf47ccf68fed0103ea54f5ed6eb62a1e330009..d9352dc4595b47c048b4503fb7b51edf71670ea8 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java @@ -1,7 +1,10 @@ package com.jd.platform.hotkey.dashboard.util; import cn.hutool.core.util.StrUtil; +import com.alibaba.fastjson.JSON; import com.jd.platform.hotkey.common.rule.KeyRule; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import java.util.List; import java.util.concurrent.ConcurrentHashMap; @@ -14,6 +17,8 @@ import java.util.concurrent.ConcurrentHashMap; public class RuleUtil { private static final ConcurrentHashMap> RULE_MAP = new ConcurrentHashMap<>(); + private static Logger logger = LoggerFactory.getLogger("RuleUtil"); + public static void init() { synchronized (RULE_MAP) { RULE_MAP.clear(); @@ -22,21 +27,28 @@ public class RuleUtil { public static void put(String appName, List list) { synchronized (RULE_MAP) { + logger.info("更新了appName:{} rule:{}",appName, JSON.toJSONString(list)); RULE_MAP.put(appName, list); } } /** - * 根据APP的key，获取该key对应的rule + * 根据APP的key，获取该key对应的rule.如 cartpc-pu__ */ public static String rule(String key) { - KeyRule keyRule = findByKey(key); - if (keyRule != null) { - String[] appKey = key.split("/"); - String appName = appKey[0]; - return appName + "-" + keyRule.getKey(); + try { + KeyRule keyRule = findByKey(key); + if (keyRule != null) { + String[] appKey = key.split("/"); + String appName = appKey[0]; + return appName + "-" + keyRule.getKey(); + } else { + logger.info("rule is null，key is " + key); + } + }catch (Exception e){ + logger.error("findByKey error",e); } - return null; + return ""; } /** @@ -47,10 +59,10 @@ public class RuleUtil { if (keyRule != null) { return keyRule.getDesc(); } - return null; + return ""; } - private static KeyRule findByKey(String appNameKey) { + public static KeyRule findByKey(String appNameKey) { synchronized (RULE_MAP) { if (StrUtil.isEmpty(appNameKey)) { return null; @@ -60,6 +72,10 @@ public class RuleUtil { String realKey = appKey[1]; KeyRule prefix = null; KeyRule common = null; + + if (RULE_MAP.get(appName) == null) { + return null; + } //遍历该app的所有rule，找到与key匹配的rule。优先全匹配->prefix匹配-> * 通配 //这一段虽然看起来比较奇怪，但是没毛病，不要乱改 for (KeyRule keyRule : RULE_MAP.get(appName)) { diff --git a/dashboard/src/main/resources/application.yml b/dashboard/src/main/resources/application.yml index 9e852a21c0551d4b4fd342804a1baad77bb00ce6..fd37556ed5eadb7f619ab20747f881a098232de4 100644 --- a/dashboard/src/main/resources/application.yml +++ b/dashboard/src/main/resources/application.yml @@ -1,5 +1,5 @@ server : - port : 8080 + port : 8081 servlet : context-path : / spring : @@ -27,18 +27,24 @@ spring : time-zone: GMT+8 date-format: yyyy-MM-dd HH:mm:ss datasource: - username: root - password: JRTEST - url: jdbc:mysql://172.24.7.182:3306/test1?useUnicode=true&characterEncoding=utf-8&useSSL=true&serverTimezone=UTC + username: ${MYSQL_USER:root} + password: ${MYSQL_PASS:JRTEST} + url: jdbc:mysql://${MYSQL_HOST:localhost}:3306/luck?useUnicode=true&characterEncoding=utf-8&useSSL=true&serverTimezone=UTC&useTimezone=true&serverTimezone=GMT%2B8 driver-class-name: com.mysql.cj.jdbc.Driver + hikari: + max-lifetime: 120000 + idle-timeout: 60000 + connection-timeout: 30000 + maximum-pool-size: 32 + minimum-idle: 10 pagehelper: helperDialect: mysql reasonable: true supportMethodsArguments: true params: count=countSql -mybatis: - configuration: - log-impl: org.apache.ibatis.logging.stdout.StdOutImpl +#mybatis: +# configuration: +# log-impl: org.apache.ibatis.logging.stdout.StdOutImpl etcd: server: ${etcdServer:http://127.0.0.1:2379} diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml index f3111db31151e839ce2756947ecc06f58cd6eb30..90b3b5d29876ba5ffe387e54d46d20eb57958649 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml @@ -3,7 +3,7 @@ - + @@ -13,7 +13,7 @@ - id, biz_key, biz_type, from_str, to_str, app_name,update_user, create_time,uuid + id, biz_id, biz_type, from_str, to_str, app_name,update_user, create_time,uuid diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.xml index cba785218a71b723a6c0dfd2aa74062aac356538..94037b7c2025532f3e3525c332ae28821d08610b 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.xml @@ -49,9 +49,9 @@ duration, - + type, @@ -81,9 +81,9 @@ #{duration,jdbcType=BIGINT}, - + #{type,jdbcType=INTEGER}, @@ -126,8 +126,8 @@ from hk_key_record where 1=1 - - and app_name = #{appName} + + and app_name = #{app} and key_name LIKE CONCAT(CONCAT('%', #{key}), '%') @@ -164,13 +164,12 @@ - + + delete from hk_key_record + where app_name = #{app} + + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyRuleMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyRuleMapper.xml deleted file mode 100644 index 78761c6f582fd6a558b15ff2b08da90518818820..0000000000000000000000000000000000000000 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyRuleMapper.xml +++ /dev/null @@ -1,189 +0,0 @@ - - - - - - - - - - - - - - - - - - id, key_name, prefix, intervals, threshold, duration, app_name, state, update_user, update_time, - version - - - - - - - - insert ignore into hk_key_rule - - - id, - - - key_name, - - - prefix, - - - intervals, - - - threshold, - - - duration, - - - app_name, - - - state, - - - update_user, - - - update_time, - - - version, - - - - - #{id,jdbcType=INTEGER}, - - - #{key,jdbcType=VARCHAR}, - - - #{prefix,jdbcType=INTEGER}, - - - #{interval,jdbcType=INTEGER}, - - - #{threshold,jdbcType=INTEGER}, - - - #{duration,jdbcType=INTEGER}, - - - #{appName,jdbcType=VARCHAR}, - - - #{state,jdbcType=INTEGER}, - - - #{updateUser,jdbcType=VARCHAR}, - - - #{updateTime,jdbcType=TIMESTAMP}, - - - #{version,jdbcType=INTEGER}, - - - - - - update hk_key_rule - - - key_name = #{key,jdbcType=VARCHAR}, - - - prefix = #{prefix,jdbcType=INTEGER}, - - - intervals = #{interval,jdbcType=INTEGER}, - - - threshold = #{threshold,jdbcType=INTEGER}, - - - duration = #{duration,jdbcType=INTEGER}, - - - app_name = #{appName,jdbcType=VARCHAR}, - - - state = #{state,jdbcType=INTEGER}, - - - update_user = #{updateUser,jdbcType=VARCHAR}, - - - update_time = #{updateTime,jdbcType=TIMESTAMP}, - - - version = #{version,jdbcType=INTEGER}, - - - where key_name = #{key} - - - - - - - - INSERT ignore INTO - hk_key_rule( - key_name, - prefix, - intervals, - threshold, - duration, - app_name, - state, - update_user, - update_time) - VALUES - - (#{k.key}, - #{k.prefix}, - #{k.interval}, - #{k.threshold}, - #{k.duration}, - #{k.appName}, - #{k.state}, - #{k.updateUser}, - #{k.updateTime}) - - - - \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.xml deleted file mode 100644 index 977c0d9dd6234a5644f1e595fdeb325eea749e6b..0000000000000000000000000000000000000000 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.xml +++ /dev/null @@ -1,158 +0,0 @@ - - - - - - - - - - - - - - id, key_name, val, uuid, app_name, duration, create_time - - - - - delete from hk_key_timely - where key_name = #{key} and app_name = #{appName} - - - - - delete from hk_key_timely - where (app_name,key_name) in - - (#{item.appName},#{item.key}) - - - - - delete from hk_key_timely where 1 = 1 - - - - - INSERT ignore INTO - hk_key_timely( - key_name, - val, - app_name, - uuid, - duration, - create_time) - VALUES - - (#{k.key}, - #{k.val}, - #{k.appName}, - #{k.uuid}, - #{k.duration}, - #{k.createTime}) - - - - - insert into hk_key_timely - - - id, - - - key_name, - - - val, - - - uuid, - - - app_name, - - - duration, - - - create_time, - - - - - #{id,jdbcType=BIGINT}, - - - #{key,jdbcType=VARCHAR}, - - - #{val,jdbcType=VARCHAR}, - - - #{uuid,jdbcType=VARCHAR}, - - - #{appName,jdbcType=VARCHAR}, - - - #{duration,jdbcType=BIGINT}, - - - #{createTime,jdbcType=TIMESTAMP}, - - - - - - update hk_key_timely - - - key_name = #{key,jdbcType=VARCHAR}, - - - val = #{val,jdbcType=VARCHAR}, - - - uuid = #{uuid,jdbcType=VARCHAR}, - - - app_name = #{appName,jdbcType=VARCHAR}, - - - duration = #{duration,jdbcType=BIGINT}, - - - create_time = #{createTime,jdbcType=TIMESTAMP}, - - - where key_name = #{key} - - - - \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ReceiveCountMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ReceiveCountMapper.xml deleted file mode 100644 index 1bc638b64c969685571e933b1a2f129ad0dd8d8c..0000000000000000000000000000000000000000 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ReceiveCountMapper.xml +++ /dev/null @@ -1,50 +0,0 @@ - - - - - - - - - - - - - - id, receive_count, worker_name, uuid,create_time,hours,minutes - - - - delete from hk_receive_count - where id = #{id,jdbcType=BIGINT} - - - insert into hk_receive_count (id, worker_name, receive_count, - hours, minutes, uuid, create_time) - values (#{id,jdbcType=BIGINT}, #{workerName,jdbcType=VARCHAR}, #{receiveCount,jdbcType=BIGINT}, - #{hours,jdbcType=BIGINT}, #{minutes,jdbcType=BIGINT}, #{uuid,jdbcType=VARCHAR}, - #{createTime,jdbcType=TIMESTAMP}) - - - - - - - - - \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml index 14354721c35e58f0cbcb553c8971649af2f48d90..ac7f464fd2e61fd81f4c5706b0f4c3108561d37b 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml @@ -5,6 +5,8 @@ + + @@ -13,10 +15,28 @@ + + - + select id, key_name, count, app,rule, days, hours, minutes, biz_type, uuid, create_time + from hk_statistics + where 1=1 + + and biz_type = #{type} + + + = #{startTime} and create_time < #{endTime}]]> + + order by id @@ -27,8 +47,8 @@ and key_name LIKE CONCAT(CONCAT('%', #{key}), '%') - - and app = #{appName} + + and app = #{app} = #{startTime} and create_time < #{endTime}]]> @@ -40,12 +60,12 @@ INSERT ignore INTO hk_statistics( - key_name,count,app, + key_name,count,app,rule, days,hours, minutes, biz_type,create_time,uuid) VALUES - ( #{k.keyName},#{k.count},#{k.app}, + (#{k.keyName},#{k.count},#{k.app},#{k.rule}, #{k.days}, #{k.hours},#{k.minutes}, #{k.bizType},#{k.createTime},#{k.uuid}) @@ -53,4 +73,10 @@ + + + delete from hk_statistics + where app = #{app} + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml new file mode 100644 index 0000000000000000000000000000000000000000..bd04545b5f34b11a91e30888a524718363ca60ff --- /dev/null +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml @@ -0,0 +1,61 @@ + + + + + + + + + + + + + + + + + + + + + + insert into hk_summary ( + index_name,rule,app,index_val1,index_val2,index_val3,days,hours, minutes,seconds,biz_type,create_time,uuid + ) + values ( + #{indexName},#{rule},#{app}, + #{val1},#{val2},#{val3}, + #{days}, #{hours},#{minutes},#{seconds}, + #{bizType},#{createTime},#{uuid}) + on duplicate key update + index_val1 = index_val1+ #{val1}, + index_val2 = index_val2+ #{val2}, + index_val3 = index_val3+ #{val3} + + + + + + + + + delete from hk_summary + where app = #{app} + + + + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml index 87b2328b5565fe5c9a3b3e3ae1fea474463b31bc..bacdd874ef3ccfbfafa31276edfab19a894c7bb1 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml @@ -21,11 +21,38 @@ from hk_user where id = #{id,jdbcType=INTEGER} + + + + + + + + delete from hk_user where id = #{id,jdbcType=INTEGER} - + insert into hk_user @@ -123,8 +150,8 @@ from hk_user where 1=1 - - and app_name = #{appName} + + and app_name = #{app} order by id desc diff --git a/dashboard/src/main/resources/db.sql b/dashboard/src/main/resources/db.sql index b294852d899710664156ccb24ee2e87682c31659..fbbdc51b779970cbae599afbb28902bf955ecf1a 100644 --- a/dashboard/src/main/resources/db.sql +++ b/dashboard/src/main/resources/db.sql @@ -1,7 +1,12 @@ +-- !!! 注意设置sql model 否则可能sql报错 ！！！ +-- 查询你的sql_model参数：select @@global.sql_mode; 发现ONLY_FULL_GROUP_BY 则会导致报错 +-- 解决方式：set @@global.sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION' +-- 详情查阅：https://www.cnblogs.com/hjhsblogs/p/11079356.html + DROP TABLE IF EXISTS `hk_change_log`; CREATE TABLE `hk_change_log` ( `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键', - `biz_key` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT '业务key', + `biz_id` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT '业务key', `biz_type` int(11) NOT NULL COMMENT '业务类型：1规则变更；2worker变更', `from_str` varchar(1024) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '原始值', `to_str` varchar(1024) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '目标值', @@ -30,28 +35,10 @@ CREATE TABLE `hk_user` ( UNIQUE INDEX `uniq_userName`(`user_name`) USING BTREE COMMENT '账号唯一索引' ) ENGINE = InnoDB AUTO_INCREMENT = 2 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Compact; --- ---------------------------- --- Records of hk_user --- ---------------------------- -INSERT INTO `hk_user` VALUES (1, 'lyfa', 'lyf', '202cb962ac59075b964b07152d234b70', '', 'ADMIN', '', '2020-04-24 02:28:33', 1); - +-- pwd: 123456 +INSERT INTO `hk_user` VALUES (2, 'admin', 'admin', 'e10adc3949ba59abbe56e057f20f883e', '1888888', 'ADMIN', '', '2020-07-28 14:01:03', 1); -DROP TABLE IF EXISTS `hk_key_rule`; -CREATE TABLE `hk_key_rule` ( - `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键', - `key_name` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'key', - `prefix` int(11) NOT NULL COMMENT '是否前缀：1是；0否', - `intervals` int(11) NOT NULL COMMENT '间隔时间（秒）', - `threshold` int(11) NOT NULL COMMENT '阈值', - `duration` int(11) NOT NULL DEFAULT 60 COMMENT '缓存时间', - `app_name` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '所属appName', - `state` int(11) NOT NULL COMMENT '状态：1可用；-1删除', - `update_user` varchar(32) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT '修改人', - `update_time` datetime(0) NOT NULL ON UPDATE CURRENT_TIMESTAMP(0) COMMENT '修改时间', - `version` int(11) NOT NULL DEFAULT 0 COMMENT '数据版本', - PRIMARY KEY (`id`) USING BTREE -) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Compact; DROP TABLE IF EXISTS `hk_key_record`; @@ -65,28 +52,12 @@ CREATE TABLE `hk_key_record` ( `type` int(11) NOT NULL DEFAULT 1 COMMENT '记录类型：1put；2del; -1unkonw', `create_time` datetime(0) NOT NULL ON UPDATE CURRENT_TIMESTAMP(0) COMMENT '创建时间', `uuid` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '防重ID', + `rule` varchar(64) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '''' COMMENT '规则', PRIMARY KEY (`id`) USING BTREE, UNIQUE INDEX `uniq_key`(`uuid`) USING BTREE COMMENT '唯一索引' ) ENGINE = InnoDB AUTO_INCREMENT = 3 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Compact; -DROP TABLE IF EXISTS `hk_change_log`; -CREATE TABLE `hk_change_log` ( - `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键', - `biz_key` varchar(128) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '业务ID', - `biz_type` int(11) NOT NULL COMMENT '业务类型：1规则变更；2worker变更', - `from_str` varchar(1024) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '原始值', - `to_str` varchar(1024) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '目标值', - `app_name` varchar(32) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '数据所属APP', - `update_user` varchar(32) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '修改人', - `create_time` datetime(0) NOT NULL ON UPDATE CURRENT_TIMESTAMP(0) COMMENT '创建时间', - `uuid` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '防重ID', - PRIMARY KEY (`id`) USING BTREE, - UNIQUE INDEX `uniq_key`(`uuid`) USING BTREE COMMENT '唯一索引' -) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Compact; - - - DROP TABLE IF EXISTS `hk_key_timely`; CREATE TABLE `hk_key_timely` ( `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', @@ -100,26 +71,6 @@ CREATE TABLE `hk_key_timely` ( UNIQUE INDEX `uniq_key`(`uuid`) USING BTREE COMMENT '唯一索引' ) ENGINE = InnoDB AUTO_INCREMENT = 8 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Compact; --- ---------------------------- --- Records of hk_key_timely --- ---------------------------- -INSERT INTO `hk_key_timely` VALUES (1, 'testk1', 'testv1', '/jd/hotkeys/test/', 'test', 60000, 1212121); ---INSERT INTO `hk_key_timely` VALUES (2, 'testk2', 'testv2', '/jd/hotkeys/test/', 'test', 10000, 123443345); - - -DROP TABLE IF EXISTS `hk_app_info`; -CREATE TABLE `hk_app_info` ( - `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', - `app_name` varchar(60) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '应用名称', - `principal_name` varchar(20) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '负责人', - `principal_phone` varchar(11) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '负责人手机号', - `app_desc` varchar(64) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT '应用描述', - `create_time` datetime NOT NULL ON UPDATE CURRENT_TIMESTAMP(0) COMMENT '创建/接入时间', - PRIMARY KEY (`id`) USING BTREE, - UNIQUE INDEX `uniq_key`(`app_name`) USING BTREE COMMENT '唯一索引' -) ENGINE = InnoDB AUTO_INCREMENT = 8 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Compact; - - CREATE TABLE `hk_statistics` ( `id` int(11) NOT NULL AUTO_INCREMENT, @@ -138,21 +89,6 @@ CREATE TABLE `hk_statistics` ( ) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Compact; -CREATE TABLE `hk_receive_count` ( - `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键ID', - `worker_name` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL COMMENT 'key名称', - `receive_count` int(11) NOT NULL COMMENT '接收数', - `hours` int(20) NOT NULL COMMENT '小时数', - `minutes` bigint(20) NOT NULL COMMENT '分钟数', - `seconds` bigint(20) NOT NULL COMMENT '秒数', - `uuid` varchar(150) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL COMMENT '防重ID', - `create_time` datetime(0) NOT NULL COMMENT '修改时间', - PRIMARY KEY (`id`) USING BTREE, - UNIQUE INDEX `uniq_uuid`(`uuid`) USING BTREE COMMENT '防重ID' -) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Compact; - - - CREATE TABLE `hk_rules` ( `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键', `rules` varchar(5000) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL COMMENT '规则JSON', @@ -165,3 +101,27 @@ CREATE TABLE `hk_rules` ( ) ENGINE = InnoDB AUTO_INCREMENT = 15 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin ROW_FORMAT = Compact; + +CREATE TABLE `hk_summary` ( + `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键', + `index_name` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '' COMMENT '指标名称', + `rule` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '' COMMENT '规则', + `app` varchar(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '' COMMENT 'app', + `index_val1` int(11) NOT NULL DEFAULT 0 COMMENT '指标值1', + `index_val2` int(11) NOT NULL DEFAULT 0 COMMENT '指标值2', + `index_val3` decimal(10, 2) NOT NULL DEFAULT 0.00 COMMENT '指标值3', + `days` int(11) NOT NULL DEFAULT 0 COMMENT '天数', + `hours` int(11) NOT NULL DEFAULT 0 COMMENT '小时数', + `minutes` int(11) NOT NULL DEFAULT 0 COMMENT '分钟数', + `seconds` int(11) NOT NULL DEFAULT 0 COMMENT '秒数', + `biz_type` tinyint(2) NOT NULL DEFAULT 0 COMMENT '类型', + `uuid` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '' COMMENT '防重ID', + `create_time` datetime(0) NOT NULL COMMENT '创建时间', + `update_time` datetime(0) NULL DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP(0) COMMENT '修改时间', + PRIMARY KEY (`id`) USING BTREE, + UNIQUE INDEX `uniq_uuid`(`uuid`) USING BTREE COMMENT '防重索引', + INDEX `idx_apprule`(`app`, `rule`) USING BTREE COMMENT '查询索引', + INDEX `ix_ct`(`create_time`) USING BTREE COMMENT '时间索引' +) ENGINE = InnoDB AUTO_INCREMENT = 18 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin COMMENT = '汇总表' ROW_FORMAT = Compact; + + diff --git a/dashboard/src/main/resources/logback-spring.xml b/dashboard/src/main/resources/logback-spring.xml index f97e83e65348a9ba35ad545f5dfc813a328375ac..9e1b14837b90a6f37add61dae0d4768bce25ea62 100644 --- a/dashboard/src/main/resources/logback-spring.xml +++ b/dashboard/src/main/resources/logback-spring.xml @@ -6,7 +6,7 @@ debug：当此属性设置为true时，将打印出logback内部日志信息， --> - + diff --git a/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js b/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js index 9ced40527989b1c670edada15bea488faf52acda..3abf2a411d70da95972e28f5e289cf80cf2367a2 100644 --- a/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js +++ b/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js @@ -4,7 +4,6 @@ _option: {}, oTableInit:function(parms){//初始化表单 $.table._option=parms; - console.log("option"+JSON.stringify($.table._option)); var oTableInit = {}; //初始化Table oTableInit.Init = function () { @@ -41,20 +40,13 @@ ajaxOptions:{ headers: {"Authorization": getCookie("token")} }, - exportOptions:{ - ignoreColumn: [0,1], //忽略某一列的索引 - fileName: '报表导出', //文件名称设置 - worksheetName: 'sheet1', //表格工作区名称 - tableName: '报表导出', - excelstyles: ['background-color', 'color', 'font-size', 'font-weight'] - }, onLoadSuccess: function(){ //加载成功时执行 var datas = $('#dataTable').bootstrapTable('getData'); console.info("加载成功"); }, onLoadError: function(status){ //加载失败时执行 - let token = getCookie("token"); - if(status === 500 && ( token == "undefined" || token =="")){ + var token = getCookie("token"); + if(status === 1000 && ( token == "undefined" || token =="")){ top.location.href = '/user/login'; } console.info("加载数据失败"); @@ -67,8 +59,19 @@ var search = {}; search.pageSize= params.pageSize; search.pageNum=params.pageNumber; - search.searchText=params.searchText; - return search; + var app = $("#apps").val(); + if(app!==undefined && app != null && app!==""){ + search.app = app; + } + var time1 = $("#startTime").val(); + if(time1!==undefined && time1 != null && time1!==""){ + search.startTime = time1; + } + var time2 = $("#endTime").val(); + if(time2!==undefined && time2 != null && time2!==""){ + search.endTime = time2; + } + return search; }, search:function(my){//查询条件 @@ -78,7 +81,6 @@ search[field.name] = field.value; }); var params = $("#dataTable").bootstrapTable("getOptions"); - console.log(params.searchText); params.queryParams=function(params){ search.pageSize= params.pageSize; search.pageNum=params.pageNumber; @@ -301,7 +303,7 @@ title: title, content: url, success:function(dom){ - let $iframeDom=$(dom[0]).find("iframe").eq(0).contents(); + var $iframeDom=$(dom[0]).find("iframe").eq(0).contents(); $iframeDom.find("#test").html("我是从父级传来的值哟……") } }) @@ -493,23 +495,20 @@ modal_status = { function getCookie(cname){ - let token = window.localStorage.getItem('token'); - let time = window.localStorage.getItem('time'); - console.log("token---> "+token); - console.log("存入的时间---> "+time+" 当前的时间---> "+Date.now()); + var token = window.localStorage.getItem('token'); if(token != null && token !==""){ - let time = window.localStorage.getItem('time'); + var time = window.localStorage.getItem('time'); if(Date.now()-time>7*24*360000){ localStorage.removeItem(time); return ""; } return token; } - let name = cname + "="; - let ca = document.cookie.split(';'); - for(let i=0; i { // only changing strings diff --git a/dashboard/src/main/resources/templates/admin/changeLog/list.html b/dashboard/src/main/resources/templates/admin/changeLog/list.html index 8235b8528936da52e8ac43d9a6b224c8745c18f6..d6ed97aaea1e10b9a00fb023cc82834083b8bf82 100644 --- a/dashboard/src/main/resources/templates/admin/changeLog/list.html +++ b/dashboard/src/main/resources/templates/admin/changeLog/list.html @@ -126,7 +126,7 @@ var options = { title: '序号' }, { - field: 'bizKey', + field: 'key', title: '业务Key' }, { @@ -135,10 +135,18 @@ var options = { formatter: function (value) { if(value == 1){ return "规则变更"; + }else if(value == 2){ + return "worker变更"; + }else if(value == 3){ + return "热点变更"; } - return "worker变更"; } }, + { field: 'createTime', title: '创建时间', + formatter:function (val,row,index) { + return changeDateFormat(val); + } + }, { field: 'from', title: '初始值', diff --git a/dashboard/src/main/resources/templates/admin/clear/edit.html b/dashboard/src/main/resources/templates/admin/clear/edit.html new file mode 100644 index 0000000000000000000000000000000000000000..4ced25c9ab226999e767425212a89ae017ef7fba --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/clear/edit.html @@ -0,0 +1,69 @@ + + + + + + + + + + + + + + + + + +
+
+
+
+
+
+
+
+
+
+ +
+ +
+
+
+ +
+ +
+
+ +
+
+ + +
+
+
+
+
+
+
+
+
+ + +
+ +
+ + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/clear/list.html b/dashboard/src/main/resources/templates/admin/clear/list.html new file mode 100644 index 0000000000000000000000000000000000000000..ceb03f10382deb7e137a4bc5da197a75525fb55d --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/clear/list.html @@ -0,0 +1,157 @@ + + + + + + + + + + + + + + + +
+
+ +
+ +
+
+ +
+
+
搜索条件
+
+ +
+
+
+
+ + +
+ +
+
+
+
+ + +
+
+
表单
+
+ + + +
+
+ + + +
+
+ +
+ +
+
+
+
+ +
+ +
+
+
+ + + +
+ +
+ + + + + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/common/html/footer.html b/dashboard/src/main/resources/templates/admin/common/html/footer.html index fbffa9ac5d307c788c8d91e5367206db12b2f897..703a3594a77952ccb19a881d1cf944cfee009c22 100644 --- a/dashboard/src/main/resources/templates/admin/common/html/footer.html +++ b/dashboard/src/main/resources/templates/admin/common/html/footer.html @@ -1,14 +1,21 @@
- -
diff --git a/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html b/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html index fa5ae9b820d10e21866eaebde9658fbe95275e5e..668341da8da7c624320b607cf617eeeb0432bc43 100644 --- a/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html +++ b/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html @@ -2,9 +2,10 @@
- + @@ -21,13 +22,19 @@
@@ -34,33 +33,33 @@ + diff --git a/dashboard/src/main/resources/templates/admin/key/add.html b/dashboard/src/main/resources/templates/admin/key/add.html index 0ffc3deb05a03ca688c3201410f742638eabf229..44b97561723fed33c9672c13e394aa950772774c 100644 --- a/dashboard/src/main/resources/templates/admin/key/add.html +++ b/dashboard/src/main/resources/templates/admin/key/add.html @@ -32,7 +32,7 @@
- +