diff --git a/LICENSE b/LICENSE index 261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64..48d4f578282c0a3136ac7045a888248e030fb918 100644 --- a/LICENSE +++ b/LICENSE @@ -186,7 +186,7 @@ same "printed page" as the copyright notice for easier identification within third-party archives. - Copyright [yyyy] [name of copyright owner] + Copyright [2020] [JD.com, Inc.] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/README.md b/README.md index c557dae9e84251d2f2fa25e445fe20a3aa438107..2d7a8587ddd89c6199d268f562c13857a1f8b745 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,44 @@ # hotkey +![输入图片说明](https://images.gitee.com/uploads/images/2020/0616/105737_e5b876cd_303698.png "redis热key探测及缓存到JVM (1).png") -正在京东APP后台灰度了几千台机器，等618海量并发检验后再推广使用，暂时不要下载使用它。 +对任意突发性的无法预先感知的热点数据，包括并不限于热点数据（如突发大量请求同一个商品）、热用户（如恶意爬虫刷子）、热接口（突发海量请求同一个接口）等，进行毫秒级精准探测到。然后对这些热数据、热用户等，推送到所有服务端JVM内存中，以大幅减轻对后端数据存储层的冲击，并可以由使用者决定如何分配、使用这些热key（譬如对热商品做本地缓存、对热用户进行拒绝访问、对热接口进行熔断或返回默认值）。这些热数据在整个服务端集群内保持一致性，并且业务隔离，worker端性能强悍。 -该框架历经多次压测，8核单机worker端每秒可接收处理16万个key探测任务，16核单机至少每秒20万，实际压测达到30万以上，CPU平稳支撑，框架无异常。测试详情可去我[CSDN博客](https://blog.csdn.net/tianyaleixiaowu)查看。 +京东APP后台热数据探测框架，历经多次高压压测和2020年京东618大促考验。在上线运行的这段时间内，每天探测的key数量数十亿计，精准捕获了大量爬虫、刷子用户，另准确探测大量热门商品并毫秒级推送到各个服务端内存，大幅降低了热数据对数据层的查询压力，提升了应用性能。 + +该框架历经多次压测，性能指标主要有两个： + +1 探测性能：8核单机worker端每秒可接收处理16万个key探测任务，16核单机至少每秒平稳处理30万以上，实际压测达到37万，CPU平稳支撑，框架无异常。 + +2 推送性能：在高并发写入的同时，对外推送目前性能约平稳推送每秒10-12万次，譬如有1千台server，一台worker上每秒产生了100个热key，那么这1秒会平稳推送100 * 1000 = 10万次，10万次推送会明确在1s内全部送达。如果是写入少，推送多，以纯推送来计数的话，该框架每秒可稳定对外推送40-60万次平稳，80次极限可撑几秒。 + +在真实业务场景中，可用1：1000的比例，即1台worker支撑1000台业务服务端的key探测任务，即可带来极大的数据存储资源节省（如对redis集群的扩充）。测试详情可去我[CSDN博客](https://blog.csdn.net/tianyaleixiaowu)查看。 + +#### 架架架架架架架架架架架构构构构构构构构构构构设设设设设设设设设设计计计计计计计计计计计计 + **见京东零售公众号** +https://mp.weixin.qq.com/s/xOzEj5HtCeh_ezHDPHw6Jw + + **！！！！！！！！！！！！！！！！重要！！！！！！！！！！！！！！！不然看不懂框架原理** + +#### 参与开发 +该框架的开发得到了来自于： + +京东数科-生活服务研发组-李云峰的大力支持，完成了整个dashboard控制台的研发。 + +京东零售-平台业务中心-APP后台架构师 杜晓宇的大力支持，参与并帮助解决了诸多技术难题。 + +京东零售-平台业务中心-PC购物车业务组 任启恒，进行了多次极限压测及功能验证，为框架的健壮性提供了更高的标准。 + +京东零售-技术与数据中心-订单研发组 姜坤坤，进行了漫长和覆盖更加全面的功能验证，协助进行了bug发现和bugfix。 + +京东零售-平台业务中心-业务测试部 郭世儒，配合进行了多次军演模拟压力测试演练，协助验证框架在模拟真实流量时的表现。 + +京东数科-消费金融研发部-账务服务 王小飞，对框架提出了重要的功能新增建议，以及新增功能的协助验证。 + +外部开发者，来自于[lowbMan](https://gitee.com/lowbMan)提供了关于修改proto序列化方式的建议，以及其他有相当价值的优化建议。 + +另有京东集团多个部门提供的意见、建议，和对框架的见解、优化方案。 + +在此一起感谢。 #### 介绍 对任意突发性的无法预先感知的热点请求，包括并不限于热点数据（如突发大量请求同一个商品）、热用户（如爬虫、刷子）、热接口（突发海量请求同一个接口）等，进行毫秒级精准探测到。 @@ -23,64 +59,169 @@ 5 接口、用户维度限流 -6 单机接口、用户维度限流限流 +6 单机接口、用户维度限流 7 集群用户维度限流 8 集群接口维度限流 -#### 尚未完成 -控制台功能缺失如下： +该开源项目战略意义重大，经历百万级并发，参与京东开源中间件项目建设，一直在等你。 + +#### worker端强悍的性能表现 +每10秒打印一行，totalDealCount代表处理过的key总量，可以看到每10秒处理量在270万-310万之间，对应每秒30万左右QPS。 -1.rule的定时保存，当etcd更换时需要能一键导入原有配置 +仅需要很少的机器，即可完成海量key的实时探测计算推送任务。比扩容redis集群规模成本低太多。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152336_78597937_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152249_4ac01178_303698.png "屏幕截图.png") -2.对/jd/count/cartsoa/ 目录下信息进行展示，代表的是每个worker连接的client数量（已完成） -3.对/jd/caffeineSize/ 目录进行展示，里面是每个worker内caffeine缓存的数量 +采用protobuf序列化后性能进一步得到提升。在秒级36万以上时，能稳定在CPU 60%，压测持续时长超过5小时，未见任何异常。30万时，压测时长超过数日，未见任何异常。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/104833_6837a091_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/104930_c522bd91_303698.png "屏幕截图.png") +![输入图片说明](https://images.gitee.com/uploads/images/2020/0817/105009_132a878b_303698.png "屏幕截图.png") -4.筛选功能，对记录表里做筛选，按时间范围筛选key出现次数大于xx次的数据 +### 界面效果 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/163805_0aa68d4b_303698.png "屏幕截图.png") +### 加我微信拉入群，群里大厂大佬多 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0807/170047_e37c882e_303698.jpeg "1596790826590.jpg") -5.导出功能，将key排重后导出的功能，按时间范围筛选 -6.还有很多 +#### 安装教程 +1. 安装etcd -该开源项目战略意义重大，要经历百万级并发，参与京东开源中间件项目建设，一直在等你。 + 在etcd下载页面下载对应操作系统的etcd，https://github.com/etcd-io/etcd/releases 使用3.4.x以上。相关搭建细节，及常见问题会发布到CSDN博客内。 -#### worker端强悍的性能表现 -每10秒打印一行，totalDealCount代表处理过的key总量，可以看到每10秒处理量在270万-310万之间，对应每秒30万左右QPS。 +2. 启动worker（集群） + 下载并编译好代码，将worker打包为jar，启动即可。如： -仅需要很少的机器，即可完成海量key的实时探测计算推送任务。比扩容redis集群规模成本低太多了。 -![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152336_78597937_303698.png "屏幕截图.png") -![输入图片说明](https://images.gitee.com/uploads/images/2020/0611/152249_4ac01178_303698.png "屏幕截图.png") + **` java -jar $JAVA_OPTS worker-0.0.1-SNAPSHOT.jar --etcd.server=${etcdServer}`** + worker可供配置项如下： +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/164514_c57d740a_303698.png "屏幕截图.png") + etcdServer为etcd集群的地址，用逗号分隔 -#### 安装教程 + JAVA_OPTS是配置的JVM相关，可根据实际情况配置 + + threadCount为处理key的线程数，不指定时由程序来计算。 + + workerPath代表该worker为哪个应用提供计算服务，譬如不同的应用appName需要用不同的worker进行隔离，以避免资源竞争。 + +3. 启动控制台 + + 下载并编译好dashboard项目，创建数据库并导入resource下db.sql文件。 配置一下application.yml里的数据库相关和etcdServer地址。 + + 启动dashboard项目，访问ip:8081，即可看到界面。 + + 其中节点信息里，即是当前已启动的worker列表。 + + 规则配置就是为各app设置规则的地方，初次使用时需要先添加APP。在用户管理菜单中，添加一个新用户，设置他的APP名字，如sample。之后新添加的这个用户就可以登录dashboard给自己的APP设置规则了，登录密码默认123456。 +![输入图片说明](https://images.gitee.com/uploads/images/2020/0622/175255_e1b05b4c_303698.png "屏幕截图.png") + + 如图就是一组规则，譬如其中as__开头的热key的规则就是interval-2秒内出现了threshold-10次就认为它是热key，它就会被推送到jvm内存中，并缓存60秒，prefix-true代表前缀匹配。那么在应用中，就可以把一组key，都用as__开头，用来探测。 + +4. client端接入使用 + + 引入client的pom依赖。 + + 在应用启动的地方初始化HotKey，譬如 + +``` +@PostConstruct + +public void initHotkey() { + + ClientStarter.Builder builder = new ClientStarter.Builder(); + ClientStarter starter = builder.setAppName("appName").setEtcdServer("http://1.8.8.4:2379,http://1.1.4.4:2379,http://1.1.1.1:2379").build(); + starter.startPipeline(); +} +``` +其中还可以setCaffeineSize(int size)设置本地缓存最大数量，默认5万，setPushPeriod(Long period)设置批量推送key的间隔时间，默认500ms，该值越小，上报热key越频繁，相应越及时，建议根据实际情况调整，如单机每秒qps10个，那么0.5秒上报一次即可，否则是空跑。该值最小为1，即1ms上报一次。 + +注意： + +如果原有项目里使用了guava，需要升级guava为以下版本，否则过低的guava版本可能发生jar包冲突。或者删除自己项目里的guava的maven依赖，guava升级不会影响原有任何逻辑。 + +``` + + com.google.guava + guava + 28.2-jre + compile + +``` + +有时可能项目里没有直接依赖guava，但是引入的某个pom里引了guava，也需要将guava排除掉。 + + + +使用： + +主要有如下4个方法可供使用 + +boolean JdHotKeyStore.isHotKey(String key) + +Object JdHotKeyStore.get(String key) + +void JdHotKeyStore.smartSet(String key, Object value) + +Object JdHotKeyStore.getValue(String key) + + + +1 boolean isHotKey(String key) ，该方法会返回该key是否是热key，如果是返回true，如果不是返回false，并且会将key上报到探测集群进行数量计算。该方法通常用于判断只需要判断key是否热、不需要缓存value的场景，如刷子用户、接口访问频率等。 + +2 Object get(String key)，该方法返回该key本地缓存的value值，可用于判断是热key后，再去获取本地缓存的value值，通常用于redis热key缓存 + +3 void smartSet(String key, Object value)，方法给热key赋值value，如果是热key，该方法才会赋值，非热key，什么也不做 + +4 Object getValue(String key)，该方法是一个整合方法，相当于isHotKey和get两个方法的整合，该方法直接返回本地缓存的value。 +如果是热key，则存在两种情况，1是返回value，2是返回null。返回null是因为尚未给它set真正的value，返回非null说明已经调用过set方法了，本地缓存value有值了。 +如果不是热key，则返回null，并且将key上报到探测集群进行数量探测。 + + +最佳实践： + +1 判断用户是否是刷子 + + if (JdHotKeyStore.isHotKey(“pin__” + thePin)) { + //限流他，do your job + } +2 判断商品id是否是热点 + + + Object skuInfo = JdHotKeyStore.getValue("skuId__" + skuId); + if(skuInfo == null) { + JdHotKeyStore.smartSet("skuId__" + skuId, theSkuInfo); + } else { + //使用缓存好的value即可 + } + + 或者这样： + + + + if (JdHotKeyStore.isHotKey(key)) { + //注意是get，不是getValue。getValue会获取并上报，get是纯粹的本地获取 + Object skuInfo = JdHotKeyStore.get("skuId__" + skuId); + if(skuInfo == null) { + JdHotKeyStore.smartSet("skuId__" + skuId, theSkuInfo); + } else { + //使用缓存好的value即可 + } + + } -1. xxxx -2. xxxx -3. xxxx +### 测试环境 -#### 使用说明 +我司为方便大家快速接入试用、查看hotkey效果，提供了1台公网etcd机器（16c16g）、2台worker机器（8c12g），供快速接入测试。 -1. xxxx -2. xxxx -3. xxxx -#### 参与贡献 +etcd地址为： **http://open-etcd.jd.com:2000** ，可以在下载项目后，在sample项目、dashboard项目里yml文件修改etcd连接地址为该地址，然后进入界面控制台，即可看到当前连接的worker，之后可以在sample项目里，进行hotkey测试。 -1. Fork 本仓库 -2. 新建 Feat_xxx 分支 -3. 提交代码 -4. 新建 Pull Request +控制台公网测试地址：http://hotkey.tianyalei.com:9001/ 账号admin，密码123456 -#### 码云特技 -1. 使用 Readme\_XXX.md 来支持不同的语言，例如 Readme\_en.md, Readme\_zh.md -2. 码云官方博客 [blog.gitee.com](https://blog.gitee.com) -3. 你可以 [https://gitee.com/explore](https://gitee.com/explore) 这个地址来了解码云上的优秀开源项目 -4. [GVP](https://gitee.com/gvp) 全称是码云最有价值开源项目，是码云综合评定出的优秀开源项目 -5. 码云官方提供的使用手册 [https://gitee.com/help](https://gitee.com/help) -6. 码云封面人物是一档用来展示码云会员风采的栏目 [https://gitee.com/gitee-stars/](https://gitee.com/gitee-stars/) +测试前，注意在控制台新建自己的app规则。 diff --git a/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java b/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java index 1102d8a19acecf4db3955dd2a5204aa7004f83a1..27474e7bc6c8348ed5ec9e8d28409b043b8d6ece 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/ClientStarter.java @@ -1,6 +1,5 @@ package com.jd.platform.hotkey.client; -import com.jd.platform.hotkey.client.cache.LocalCache; import com.jd.platform.hotkey.client.callback.ReceiveNewKeySubscribe; import com.jd.platform.hotkey.client.core.eventbus.EventBusCenter; import com.jd.platform.hotkey.client.core.key.PushSchedulerStarter; @@ -40,9 +39,8 @@ public class ClientStarter { public static class Builder { private String appName; private String etcdServer; - private LocalCache localCache; private Long pushPeriod; - private int caffeineSize = 50000; + private int caffeineSize = 200000; public Builder() { } @@ -65,11 +63,6 @@ public class ClientStarter { return this; } - public Builder setLocalCache(LocalCache localCache) { - this.localCache = localCache; - return this; - } - public Builder setPushPeriod(Long pushPeriod) { this.pushPeriod = pushPeriod; return this; diff --git a/client/src/main/java/com/jd/platform/hotkey/client/Context.java b/client/src/main/java/com/jd/platform/hotkey/client/Context.java index f0455a21a20d2c5e146374ed9841c8b12496fea1..fea41067c1047bd1538f6794fc71e03502c47d96 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/Context.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/Context.java @@ -12,5 +12,5 @@ public class Context { */ public static boolean NEED_RECONNECT = true; - public static int CAFFEINE_SIZE = 50000; + public static int CAFFEINE_SIZE; } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java b/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java index d5afa72900cffa6c9c37570a83eafd85171b5554..f98548963e506d402941d23793713e0394c114a0 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/key/NettyKeyPusher.java @@ -44,7 +44,7 @@ public class NettyKeyPusher implements IKeyPusher { for (Channel channel : map.keySet()) { try { List batch = map.get(channel); - channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.REQUEST_NEW_KEY, FastJsonUtils.convertObjectToJSON(batch)))); + channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.REQUEST_NEW_KEY, FastJsonUtils.convertObjectToJSON(batch)))).sync(); } catch (Exception e) { try { InetSocketAddress insocket = (InetSocketAddress) channel.remoteAddress(); @@ -78,7 +78,7 @@ public class NettyKeyPusher implements IKeyPusher { try { List batch = map.get(channel); channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(Context.APP_NAME, - MessageType.REQUEST_HIT_COUNT, FastJsonUtils.convertObjectToJSON(batch)))); + MessageType.REQUEST_HIT_COUNT, FastJsonUtils.convertObjectToJSON(batch)))).sync(); } catch (Exception e) { try { InetSocketAddress insocket = (InetSocketAddress) channel.remoteAddress(); diff --git a/client/src/main/java/com/jd/platform/hotkey/client/core/worker/WorkerInfoHolder.java b/client/src/main/java/com/jd/platform/hotkey/client/core/worker/WorkerInfoHolder.java index 72471a9c49c9073af28b7a03b71937f7e07e40cb..c12182f2c5a68235d91cacaae6447c829a26f0ee 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/core/worker/WorkerInfoHolder.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/core/worker/WorkerInfoHolder.java @@ -57,10 +57,11 @@ public class WorkerInfoHolder { } public static Channel chooseChannel(String key) { - if (StrUtil.isEmpty(key) || WORKER_HOLDER.size() == 0) { + int size = WORKER_HOLDER.size(); + if (StrUtil.isEmpty(key) || size == 0) { return null; } - int index = Math.abs(key.hashCode() % WORKER_HOLDER.size()); + int index = Math.abs(key.hashCode() % size); return WORKER_HOLDER.get(index).channel; } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java b/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java index c129fb155b99d17303e2d2bcad97056cdc992c52..216160a3b7e0d47f9e70db646018f9066ca87f4d 100644 --- a/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/etcd/EtcdStarter.java @@ -38,8 +38,6 @@ public class EtcdStarter { fetchRule(); - fetchExistHotKey(); - // startWatchWorker(); startWatchRule(); @@ -243,6 +241,9 @@ public class EtcdStarter { JdLogger.info(getClass(), "trying to connect to etcd and fetch rule info"); boolean success = fetchRuleFromEtcd(); if (success) { + //拉取已存在的热key + fetchExistHotKey(); + scheduledExecutorService.shutdown(); } diff --git a/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java b/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java index d2a36a41c7819ad537e1f8852a2cfe7906573da0..d8ba8927c491ae75823e2511187d024e5767fb94 100755 --- a/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java +++ b/client/src/main/java/com/jd/platform/hotkey/client/netty/NettyClient.java @@ -2,8 +2,6 @@ package com.jd.platform.hotkey.client.netty; import com.jd.platform.hotkey.client.core.worker.WorkerInfoHolder; import com.jd.platform.hotkey.client.log.JdLogger; -import com.jd.platform.hotkey.common.coder.Codec; -import com.jd.platform.hotkey.common.coder.NettyCodec; import com.jd.platform.hotkey.common.tool.Constant; import io.netty.bootstrap.Bootstrap; import io.netty.buffer.ByteBuf; @@ -28,9 +26,6 @@ public class NettyClient { private Bootstrap bootstrap; - private Codec codec = new NettyCodec(); - - public static NettyClient getInstance() { return nettyClient; } @@ -56,8 +51,6 @@ public class NettyClient { ByteBuf delimiter = Unpooled.copiedBuffer(Constant.DELIMITER.getBytes()); ch.pipeline() .addLast(new DelimiterBasedFrameDecoder(Constant.MAX_LENGTH, delimiter)) -// .addLast(codec.newEncoder()) -// .addLast(codec.newDecoder()) .addLast(new StringDecoder()) //10秒没消息时，就发心跳包过去 .addLast(new IdleStateHandler(0, 0, 30)) diff --git a/common/pom.xml b/common/pom.xml index 59a7d76b70114ca8342f9c9bb936b31764caf036..e11fe7a0b39801872527ad1e513e4d79699a6a5c 100644 --- a/common/pom.xml +++ b/common/pom.xml @@ -15,10 +15,9 @@ 1.8 4.1.42.Final - 3.4.2 1.1.7.3 1.2.70 - 0.0.14 + 0.0.16 2.8.0 5.1.0 @@ -40,13 +39,6 @@ netty-all ${netty4.version} - - - com.lmax - disruptor - ${disruptor.version} - - com.alibaba fastjson diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java deleted file mode 100755 index c61805b9d34c84981d0b2c90891e305ae3b2bd61..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/Codec.java +++ /dev/null @@ -1,30 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.jd.platform.hotkey.common.coder; - -import io.netty.channel.ChannelHandler; - -/** - * @author wuweifeng wrote on 2019-12-11 - * @version 1.0 - */ -public interface Codec { - - ChannelHandler newEncoder(); - - ChannelHandler newDecoder(); -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java deleted file mode 100644 index 24cb0ed25000966a1dc925fdacca90f5af8d0e1a..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageDecoder.java +++ /dev/null @@ -1,53 +0,0 @@ -package com.jd.platform.hotkey.common.coder; - -import com.jd.platform.hotkey.common.model.HotKeyMsg; -import com.jd.platform.hotkey.common.model.typeenum.MessageType; -import com.jd.platform.hotkey.common.tool.Constant; -import io.netty.buffer.ByteBuf; -import io.netty.channel.ChannelHandlerContext; -import io.netty.handler.codec.ByteToMessageDecoder; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.nio.charset.Charset; -import java.util.List; - -/** - * @author wuweifeng wrote on 2020-01-06 - * @version 1.0 - */ -public class MessageDecoder extends ByteToMessageDecoder { - private Logger logger = LoggerFactory.getLogger(getClass()); - - @Override - protected void decode(ChannelHandlerContext ctx, ByteBuf byteBuf, List out) { - if (byteBuf.readableBytes() < 10 || byteBuf.readableBytes() > Constant.MAX_LENGTH) { - logger.warn("数据包不正确，当前包大小为：" + byteBuf.readableBytes()); - return; - } - - byteBuf.markReaderIndex(); - - HotKeyMsg message = new HotKeyMsg(); - - int magicNumber = byteBuf.readInt(); - if (Constant.MAGIC_NUMBER != magicNumber) { - logger.warn("MAGIC_NUMBER不正确:" + magicNumber); - return; - } - - message.setMagicNumber(byteBuf.readInt()); // 读取魔数 - - MessageType messageType = MessageType.get(byteBuf.readByte()); - if (messageType == null) { - logger.error("messageType is null , byteBuf readByte = " + byteBuf.readByte()); - return; - } - message.setMessageType(messageType); // 读取当前的消息类型 - - int bodyLength = byteBuf.readInt(); // 读取消息体长度和数据 - CharSequence body = byteBuf.readCharSequence(bodyLength, Charset.defaultCharset()); - message.setBody(body.toString()); - out.add(message); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java deleted file mode 100644 index acf21624c6323f2c616d132c67701a12a4b802e2..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/MessageEncoder.java +++ /dev/null @@ -1,47 +0,0 @@ -package com.jd.platform.hotkey.common.coder; - -import com.jd.platform.hotkey.common.model.HotKeyMsg; -import com.jd.platform.hotkey.common.model.typeenum.MessageType; -import com.jd.platform.hotkey.common.tool.Constant; -import io.netty.buffer.ByteBuf; -import io.netty.channel.ChannelHandlerContext; -import io.netty.handler.codec.MessageToByteEncoder; - -import java.nio.charset.Charset; - -/** - * @author wuweifeng wrote on 2020-01-06 - * @version 1.0 - */ -public class MessageEncoder extends MessageToByteEncoder { - - @Override - protected void encode(ChannelHandlerContext ctx, HotKeyMsg message, ByteBuf out) { - MessageType messageType = message.getMessageType(); - //非法类型 - if (MessageType.get(messageType.getType()) == null) { - return; - } - // 这里会判断消息类型是不是EMPTY类型，如果是EMPTY类型，则表示当前消息不需要写入到管道中 - if (messageType == MessageType.EMPTY) { - return; - } - //4个byte - out.writeInt(Constant.MAGIC_NUMBER); // 写入当前的魔数 - //1个byte - out.writeByte(message.getMessageType().getType()); // 写入当前消息的类型 - - if (null == message.getBody()) { - out.writeInt(0); // 如果消息体为空，则写入0，表示消息体长度为0 - } else { - System.out.println(message.getBody().length()); - out.writeInt(message.getBody().length()); - out.writeCharSequence(message.getBody(), Charset.defaultCharset()); - } - out.writeBytes(Constant.DELIMITER.getBytes()); - } - - public static void main(String[] args) { - System.out.println(Constant.MAGIC_NUMBER + ""); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java b/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java deleted file mode 100755 index 1aeac45f5948d3595b367b811b666dcfb01870ab..0000000000000000000000000000000000000000 --- a/common/src/main/java/com/jd/platform/hotkey/common/coder/NettyCodec.java +++ /dev/null @@ -1,36 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one or more - * contributor license agreements. See the NOTICE file distributed with - * this work for additional information regarding copyright ownership. - * The ASF licenses this file to You under the Apache License, Version 2.0 - * (the "License"); you may not use this file except in compliance with - * the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package com.jd.platform.hotkey.common.coder; - -import io.netty.channel.ChannelHandler; - -/** - * @author wuweifeng wrote on 2019-12-11 - * @version 1.0 - */ -public class NettyCodec implements Codec { - - @Override - public ChannelHandler newEncoder() { - return new MessageEncoder(); - } - - @Override - public ChannelHandler newDecoder() { - return new MessageDecoder(); - } -} diff --git a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java index e88389d8c5bef4cab53b5fe815ed57d029e516f7..009a1075e5b0eb51b10ef6ff3550f6883939c464 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/ConfigConstant.java @@ -13,6 +13,10 @@ public interface ConfigConstant { * 所有的workers，存这里 */ String workersPath = "/jd/workers/"; + /** + * dashboard的ip存这里 + */ + String dashboardPath = "/jd/dashboard/"; /** * 该app所有的workers地址的path。需要手工分配，默认每个app都用所有的worker */ @@ -54,4 +58,24 @@ public interface ConfigConstant { * 存放客户端hotKey访问次数和总访问次数的path */ String keyHitCountPath = "/jd/keyHitCount/"; + /** + * 是否开启日志 + */ + String logToggle = "/jd/logOn"; + + /** + * 清理历史数据的配置的path + * time unit : day + */ + String clearCfgPath = "/jd/clearCfg/"; + + /** + * app配置 + */ + String appCfgPath = "/jd/appCfg/"; + + /** + * 控制台启动的netty端口 + */ + int dashboardPort = 11112; } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java index 06e85e957794fa9b0559ac81836480bdbd997f56..fb64d9e28b9a3217e1824f93b762f147427fd205 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/configcenter/etcd/JdEtcdClient.java @@ -3,7 +3,6 @@ package com.jd.platform.hotkey.common.configcenter.etcd; import cn.hutool.core.collection.CollectionUtil; import com.google.protobuf.ByteString; import com.ibm.etcd.api.KeyValue; -import com.ibm.etcd.api.Kv; import com.ibm.etcd.api.LeaseGrantResponse; import com.ibm.etcd.api.RangeResponse; import com.ibm.etcd.client.KvStoreClient; diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java index 0451fce64a1dc414d41a410de91ebd1be7038e9e..11de0ccaf76915b7dd59f58f6d4120850912d459 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/HotKeyModel.java @@ -23,11 +23,7 @@ public class HotKeyModel extends BaseModel { @Override public String toString() { - return "HotKeyModel{" + - "appName='" + appName + '\'' + - ", keyType=" + keyType + - ", remove=" + remove + - '}'; + return "appName:" + appName + "-key=" + getKey(); } public boolean isRemove() { @@ -53,4 +49,5 @@ public class HotKeyModel extends BaseModel { public void setKeyType(KeyType keyType) { this.keyType = keyType; } + } diff --git a/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java b/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java index cd3f98393d6cd6407d02afb85d7fc50b8f84cc99..5b07624630cae927ac724a738977f67c41110d78 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/model/typeenum/MessageType.java @@ -8,7 +8,8 @@ public enum MessageType { APP_NAME((byte) 1), REQUEST_NEW_KEY((byte) 2), RESPONSE_NEW_KEY((byte) 3), - REQUEST_HIT_COUNT((byte) 7), + REQUEST_HIT_COUNT((byte) 7), //命中率 + REQUEST_HOT_KEY((byte) 8), //热key，worker->dashboard PING((byte) 4), PONG((byte) 5), EMPTY((byte) 6); diff --git a/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java b/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java index 5c747540ad3452fa161d7d7d591c98f99611e29c..81a40ec026ac7aeb96c81b1f907b5984b69124e3 100644 --- a/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/tool/Constant.java @@ -31,6 +31,6 @@ public class Constant { public static String DEFAULT_DELETE_VALUE = "#[DELETE]#"; //单次包最大2M - public static int MAX_LENGTH = 2 * 1024 * 1024; + public static int MAX_LENGTH = 4 * 1024 * 1024; } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java b/common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java similarity index 63% rename from worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java rename to common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java index cdc1299c2f33bf3ef3b6944271ae8434184a35c2..009eb471cea2e441b308f6030c0e171405998692 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/flush/FlushUtil.java +++ b/common/src/main/java/com/jd/platform/hotkey/common/tool/flush/FlushUtil.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.worker.netty.flush; +package com.jd.platform.hotkey.common.tool.flush; import io.netty.buffer.ByteBuf; import io.netty.channel.ChannelHandlerContext; @@ -16,19 +16,19 @@ public class FlushUtil { * 往channel里输出消息 */ public static void flush(ChannelHandlerContext channelHandlerContext, ByteBuf byteBuf) { - if (channelHandlerContext.channel().isWritable()) { - channelHandlerContext.channel().writeAndFlush(byteBuf).addListener(future -> { - if (!future.isSuccess()) { - logger.warn("flush error " + future.cause().getMessage()); - } - }); - } else { +// if (channelHandlerContext.channel().isWritable()) { +// channelHandlerContext.channel().writeAndFlush(byteBuf).addListener(future -> { +// if (!future.isSuccess()) { +// logger.warn("flush error " + future.cause().getMessage()); +// } +// }); +// } else { try { //同步发送 channelHandlerContext.channel().writeAndFlush(byteBuf).sync(); } catch (InterruptedException e) { logger.error("flush error " + e.getMessage()); } - } +// } } } diff --git a/dashboard/pom.xml b/dashboard/pom.xml index 30eacfa44eebc69b66ed08db05272acf602260ec..de493cd9567d1ccce7bf1f99a3e661b0fdb94abe 100644 --- a/dashboard/pom.xml +++ b/dashboard/pom.xml @@ -86,6 +86,16 @@ sso-uim-spring 1.2.0-SNAPSHOT + + com.jd.dd + dd-open-gw-api + 4.1.0-SNAPSHOT + + + com.github.rholder + guava-retrying + 2.0.0 + diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java index 2b361c66ba6119558b49e80da745a2b5d80fb2d7..e1431c9f8af5502860afcdf8aaf57db1ce1b12ae 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/DashboardApplication.java @@ -1,35 +1,16 @@ package com.jd.platform.hotkey.dashboard; -import com.alibaba.fastjson.JSON; -import com.jd.platform.hotkey.common.configcenter.ConfigConstant; -import com.jd.platform.hotkey.common.configcenter.IConfigCenter; -import com.jd.platform.hotkey.common.tool.IpUtils; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; -import com.jd.platform.hotkey.dashboard.mapper.RulesMapper; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; -import org.springframework.boot.CommandLineRunner; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.scheduling.annotation.EnableAsync; import org.springframework.scheduling.annotation.EnableScheduling; -import javax.annotation.Resource; -import java.util.HashMap; -import java.util.Map; - @EnableAsync @EnableScheduling @SpringBootApplication -public class DashboardApplication implements CommandLineRunner { - - - private Logger logger = LoggerFactory.getLogger(getClass()); - - @Resource - private KeyTimelyMapper timelyMapper; +public class DashboardApplication{ public static void main(String[] args) { try { @@ -37,13 +18,6 @@ public class DashboardApplication implements CommandLineRunner { }catch (Exception e){ e.printStackTrace(); } - - } - - @Override - public void run(String... args) { - int row = timelyMapper.clear(); - logger.info("clear db timely hotKey, effect row : {}",row); } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/AppCfgController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/AppCfgController.java new file mode 100644 index 0000000000000000000000000000000000000000..10fc82425030ee7c5f50cecb0790431024912bab --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/AppCfgController.java @@ -0,0 +1,58 @@ +package com.jd.platform.hotkey.dashboard.biz.controller; + +import com.jd.platform.hotkey.dashboard.common.base.BaseController; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; +import com.jd.platform.hotkey.dashboard.biz.service.AppCfgService; +import org.springframework.stereotype.Controller; +import org.springframework.ui.ModelMap; +import org.springframework.web.bind.annotation.*; + +import javax.annotation.Resource; + + +@Controller +@RequestMapping("/appCfg") +public class AppCfgController extends BaseController { + + + private String prefix = "admin/appcfg"; + + + @Resource + private AppCfgService appCfgService; + + @GetMapping("/view") + public String view(ModelMap modelMap){ + modelMap.put("title", Constant.APP_CFG_VIEW); + return prefix + "/list"; + } + + + @PostMapping("/list") + @ResponseBody + public Page list(PageReq page, String app){ + return appCfgService.pageAppCfgVo(page, app); + } + + + @GetMapping("/edit/{app}") + public String edit(@PathVariable("app") String app, ModelMap modelMap){ + modelMap.put("appCfg", appCfgService.selectAppCfgVo(app)); + return prefix + "/edit"; + } + + @PostMapping("/save") + @ResponseBody + public Result save(AppCfgVo cfg){ + cfg.setModifier(userName()); + appCfgService.saveAppCfgVo(cfg); + return Result.success(); + } + + +} + diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ChangLogController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ChangLogController.java similarity index 90% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ChangLogController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ChangLogController.java index e9a1788f647c237fe29db9224359473c98ca85cf..1e18dc2bd7a4dfb2254c17e94db1ea513da8b5c1 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/ChangLogController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ChangLogController.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.controller; +package com.jd.platform.hotkey.dashboard.biz.controller; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.*; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.model.ChangeLog; -import com.jd.platform.hotkey.dashboard.service.ChangeLogService; +import com.jd.platform.hotkey.dashboard.biz.service.ChangeLogService; import org.springframework.stereotype.Controller; import org.springframework.ui.ModelMap; import org.springframework.web.bind.annotation.*; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ClearController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ClearController.java new file mode 100644 index 0000000000000000000000000000000000000000..e6ce6b48943e3533f01c154e1de0de0035145296 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/ClearController.java @@ -0,0 +1,62 @@ +package com.jd.platform.hotkey.dashboard.biz.controller; + +import com.github.pagehelper.PageInfo; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; +import com.jd.platform.hotkey.dashboard.biz.service.ClearService; +import org.springframework.stereotype.Controller; +import org.springframework.ui.ModelMap; +import org.springframework.web.bind.annotation.*; + +import javax.annotation.Resource; + +/** + * @ProjectName: hotkey + * @ClassName: ClearController + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:48 + */ +@Controller +@RequestMapping("/clear") +public class ClearController { + + + private String prefix = "admin/clear"; + + + @Resource + private ClearService clearService; + + @GetMapping("/view") + public String view(ModelMap modelMap){ + modelMap.put("title", Constant.CLEAR_VIEW); + return prefix + "/list"; + } + + + @PostMapping("/list") + @ResponseBody + public Page list(PageReq page, String app){ + PageInfo info = clearService.pageClearCfg(page, app); + return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); + } + + + @GetMapping("/edit/{app}") + public String edit(@PathVariable("app") String app, ModelMap modelMap){ + modelMap.put("clearCfg", clearService.selectClearCfg(app)); + return prefix + "/edit"; + } + + @PostMapping("/save") + @ResponseBody + public Result save(ClearCfgVo cfg){ + int b = clearService.saveClearCfg(cfg); + return b == 0 ? Result.fail():Result.success(); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/KeyController.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/KeyController.java index 0b287a83d09533e685d4149a7172e394b5149d38..281c26a4cf54c8b4ad13b74a6eaf82f2c04c05da 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/KeyController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/KeyController.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.controller; +package com.jd.platform.hotkey.dashboard.biz.controller; import com.github.pagehelper.PageInfo; import com.github.pagehelper.util.StringUtil; @@ -14,7 +14,7 @@ import com.jd.platform.hotkey.dashboard.common.domain.vo.HotKeyLineChartVo; import com.jd.platform.hotkey.dashboard.model.KeyRecord; import com.jd.platform.hotkey.dashboard.model.KeyTimely; import com.jd.platform.hotkey.dashboard.model.Statistics; -import com.jd.platform.hotkey.dashboard.service.KeyService; +import com.jd.platform.hotkey.dashboard.biz.service.KeyService; import com.jd.platform.hotkey.dashboard.util.DateUtil; import com.jd.platform.hotkey.dashboard.util.ExcelUtil; import org.springframework.stereotype.Controller; @@ -34,7 +34,7 @@ import java.util.List; @Controller @RequestMapping("/key") public class KeyController extends BaseController { - + private String prefix = "admin/key"; @Resource @@ -79,8 +79,7 @@ public class KeyController extends BaseController { @PostMapping("/listTimely") @ResponseBody public Page listTimely(PageReq page, SearchReq searchReq){ - PageInfo info = keyService.pageKeyTimely(page, searchReq); - return new Page<>(info.getPageNum(),(int)info.getTotal(),info.getList()); + return keyService.pageKeyTimely(page, searchReq); } @@ -123,12 +122,12 @@ public class KeyController extends BaseController { } - @GetMapping("/edit/{id}") + /*@GetMapping("/edit/{id}") public String edit(@PathVariable("id") Long id, ModelMap modelMap){ modelMap.put("key", keyService.selectByPk(id)); return prefix + "/edit"; - } - + }*/ + @PostMapping("/edit") @ResponseBody diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/RuleController.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/RuleController.java index e1869c9fb3d75e981d28c0b561cc30e05f509b67..ce8600d017063ce93da023ddfc6287ab6923f9ed 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/RuleController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/RuleController.java @@ -1,9 +1,8 @@ -package com.jd.platform.hotkey.dashboard.controller; +package com.jd.platform.hotkey.dashboard.biz.controller; import cn.hutool.json.JSONArray; import cn.hutool.json.JSONObject; import cn.hutool.json.JSONTokener; -import com.alibaba.fastjson.JSON; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.Constant; @@ -14,9 +13,8 @@ import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; -import com.jd.platform.hotkey.dashboard.model.Rule; import com.jd.platform.hotkey.dashboard.model.Rules; -import com.jd.platform.hotkey.dashboard.service.RuleService; +import com.jd.platform.hotkey.dashboard.biz.service.RuleService; import org.springframework.stereotype.Controller; import org.springframework.ui.ModelMap; import org.springframework.web.bind.annotation.*; @@ -31,7 +29,7 @@ import java.util.List; @Controller @RequestMapping("/rule") public class RuleController extends BaseController { - + @Resource private RuleService ruleService; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/UserController.java similarity index 96% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/UserController.java index bc571ba53252bac31d25651c221387a27ddf0e94..8711eb48e200261f5f60bfd9b7bfb37ac4926696 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/UserController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/UserController.java @@ -1,17 +1,15 @@ -package com.jd.platform.hotkey.dashboard.controller; - -import java.util.*; +package com.jd.platform.hotkey.dashboard.biz.controller; import com.github.pagehelper.PageInfo; import com.github.pagehelper.util.StringUtil; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.Page; -import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.Result; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.model.User; -import com.jd.platform.hotkey.dashboard.service.UserService; import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import io.jsonwebtoken.Claims; @@ -23,6 +21,8 @@ import javax.annotation.Resource; import javax.servlet.http.Cookie; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; +import java.util.HashMap; +import java.util.Map; @Controller diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/WorkerController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/WorkerController.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/WorkerController.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/WorkerController.java index 4e9471ea855ee90872a3a9b0b228581031665b8f..aedd57cf2edc0826a1114b6a77ebcaf1c66c4de5 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/WorkerController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/controller/WorkerController.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.controller; +package com.jd.platform.hotkey.dashboard.biz.controller; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.base.BaseController; import com.jd.platform.hotkey.dashboard.common.domain.*; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.model.Worker; -import com.jd.platform.hotkey.dashboard.service.WorkerService; +import com.jd.platform.hotkey.dashboard.biz.service.WorkerService; import org.springframework.stereotype.Controller; import org.springframework.ui.ModelMap; import org.springframework.web.bind.annotation.*; @@ -16,7 +16,7 @@ import javax.annotation.Resource; @Controller @RequestMapping("/worker") public class WorkerController extends BaseController { - + private String prefix = "admin/worker"; @Resource @@ -63,7 +63,7 @@ public class WorkerController extends BaseController { modelMap.put("worker", workerService.selectByKey(key.replace("_","/"))); return prefix + "/edit"; } - + @PostMapping("/edit") @ResponseBody diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/BizAccessTokenMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/BizAccessTokenMapper.java new file mode 100644 index 0000000000000000000000000000000000000000..2d4db7f0f17365587355974b70ea90791d5b86f1 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/BizAccessTokenMapper.java @@ -0,0 +1,62 @@ +package com.jd.platform.hotkey.dashboard.biz.mapper; + +import com.jd.platform.hotkey.dashboard.model.BizAccessToken; +import org.apache.ibatis.annotations.Mapper; + +import java.util.List; + +/** + * 咚咚消息tokenMapper接口 + * + * @date 2020-05-21 + */ +@Mapper +public interface BizAccessTokenMapper { + /** + * 查询咚咚消息token + * + * @param id 咚咚消息tokenID + * @return 咚咚消息token + */ + public BizAccessToken selectBizAccessTokenById(Long id); + + /** + * 查询咚咚消息token列表 + * + * @param bizAccessToken 咚咚消息token + * @return 咚咚消息token集合 + */ + public List selectBizAccessTokenList(BizAccessToken bizAccessToken); + + /** + * 新增咚咚消息token + * + * @param bizAccessToken 咚咚消息token + * @return 结果 + */ + public int insertBizAccessToken(BizAccessToken bizAccessToken); + + /** + * 修改咚咚消息token + * + * @param bizAccessToken 咚咚消息token + * @return 结果 + */ + public int updateBizAccessToken(BizAccessToken bizAccessToken); + + /** + * 删除咚咚消息token + * + * @param id 咚咚消息tokenID + * @return 结果 + */ + public int deleteBizAccessTokenById(Long id); + + /** + * 批量删除咚咚消息token + * + * @param ids 需要删除的数据ID + * @return 结果 + */ + public int deleteBizAccessTokenByIds(String[] ids); +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.java similarity index 87% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.java index c36f87178df5b364ef9f0a21f0af5db1d6c13901..a1ceb2002f7bfce4405f24a86fa62cc17c6d2376 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.ChangeLog; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/KeyRecordMapper.java similarity index 82% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/KeyRecordMapper.java index 3ef46b31a58494eabc0daadf386cc759c1053c7b..dde2c254b591a76bf75bda7d2a0c37c690e49c1b 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyRecordMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/KeyRecordMapper.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; -import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.KeyRecord; import com.jd.platform.hotkey.dashboard.model.Statistics; import org.apache.ibatis.annotations.Mapper; +import java.util.Date; import java.util.List; /** @@ -25,4 +25,6 @@ public interface KeyRecordMapper { List maxHotKey(SearchReq req); List statisticsByRule(SearchReq req); + + int clearExpireData(String app, Date expireDate); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.java similarity index 83% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.java index 1c6835c005554ea218f09c723f1176e71e85ab1f..3c1c894bf824b6a13a393b2b1faa299831c0455c 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.model.Rules; import org.apache.ibatis.annotations.Mapper; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.java similarity index 85% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.java index dabdfc584a4c4379ac1e31ba9de79704748c5b9b..a657342fbd71d592a6fbe4a11e12a6aa2869a2bf 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.java @@ -1,9 +1,10 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.Statistics; import org.apache.ibatis.annotations.Mapper; +import java.util.Date; import java.util.List; /** @@ -42,7 +43,6 @@ public interface StatisticsMapper { /** * 清理 - * @param type type */ - void clear(int type); + int clearExpireData(String app, Date expireDate); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.java similarity index 79% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.java index 4685e76b4760726c153941015c2cfb274aaff8a6..99882f74f552c5f3991b9ed1ef51490238c09339 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.java @@ -1,11 +1,11 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; -import com.jd.platform.hotkey.dashboard.model.Statistics; import com.jd.platform.hotkey.dashboard.model.Summary; import org.apache.ibatis.annotations.Mapper; +import java.util.Date; import java.util.List; /** @@ -22,4 +22,6 @@ public interface SummaryMapper { int saveOrUpdate(Summary records); List listRuleHitCount(SearchReq req); + + int clearExpireData(String app, Date expireDate); } \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.java similarity index 91% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.java index c8d72ee2c2e06a13df2ba81d712eee9b8fb1cd40..257a27d30120bdad0592a0cf9ae50c8e65e9a4f3 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/UserMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.User; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.java similarity index 91% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.java index 501316b563788ef2f7183134fb93a66247df2c9f..29befc2469071ac3f4936273fdf83f7f0152baf0 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/WorkerMapper.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/mapper/WorkerMapper.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.mapper; +package com.jd.platform.hotkey.dashboard.biz.mapper; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.model.Worker; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/AppCfgService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/AppCfgService.java new file mode 100644 index 0000000000000000000000000000000000000000..fa814b8838405662ebda8c970de17e0b25782f3c --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/AppCfgService.java @@ -0,0 +1,21 @@ +package com.jd.platform.hotkey.dashboard.biz.service; + +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; + +/** + * @ProjectName: hotkey + * @ClassName: ClearService + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:51 + */ +public interface AppCfgService { + + Page pageAppCfgVo(PageReq page, String app); + + AppCfgVo selectAppCfgVo(String app); + + void saveAppCfgVo(AppCfgVo cfg); + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ChangeLogService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ChangeLogService.java similarity index 91% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ChangeLogService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ChangeLogService.java index 498f4e4d87743cfe567cba63c5a23872e747ceb3..cfed2493c00b8c824ecb1ef2b19933ea81cecc85 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/ChangeLogService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ChangeLogService.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ClearService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ClearService.java new file mode 100644 index 0000000000000000000000000000000000000000..383fa9cb53f71bbeb0397ae0dddbbceff1448875 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/ClearService.java @@ -0,0 +1,23 @@ +package com.jd.platform.hotkey.dashboard.biz.service; + +import com.github.pagehelper.PageInfo; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; + +/** + * @ProjectName: hotkey + * @ClassName: ClearService + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:51 + */ +public interface ClearService { + + + PageInfo pageClearCfg(PageReq page, String app); + + ClearCfgVo selectClearCfg(String app); + + int saveClearCfg(ClearCfgVo cfg); + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/IBizAccessTokenService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/IBizAccessTokenService.java new file mode 100644 index 0000000000000000000000000000000000000000..d5c90a6ea82aae4aea4e4db6555555d15c660736 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/IBizAccessTokenService.java @@ -0,0 +1,55 @@ +package com.jd.platform.hotkey.dashboard.biz.service; + + +import com.jd.platform.hotkey.dashboard.model.BizAccessToken; + +import java.util.List; + +/** + * 咚咚消息tokenService接口 + * + * @author ruoyi + * @date 2020-05-21 + */ +public interface IBizAccessTokenService +{ + /** + * 查询咚咚消息token + * + * @param id 咚咚消息tokenID + * @return 咚咚消息token + */ + public BizAccessToken selectBizAccessTokenById(Long id); + + /** + * 查询咚咚消息token列表 + * + * @param bizAccessToken 咚咚消息token + * @return 咚咚消息token集合 + */ + public List selectBizAccessTokenList(BizAccessToken bizAccessToken); + + /** + * 新增咚咚消息token + * + * @param bizAccessToken 咚咚消息token + * @return 结果 + */ + public int insertBizAccessToken(BizAccessToken bizAccessToken); + + /** + * 修改咚咚消息token + * + * @param bizAccessToken 咚咚消息token + * @return 结果 + */ + public int updateBizAccessToken(BizAccessToken bizAccessToken); + + /** + * 删除咚咚消息token信息 + * + * @param id 咚咚消息tokenID + * @return 结果 + */ + public int deleteBizAccessTokenById(Long id); +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/KeyService.java similarity index 85% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/KeyService.java index 3dfcf1892068913c4404faed50bf1d62031b101c..d8908f95e71b65fdb432012fd3c570fcc5fa1703 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/KeyService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/KeyService.java @@ -1,6 +1,7 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; +import com.jd.platform.hotkey.dashboard.common.domain.Page; import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; @@ -29,11 +30,7 @@ public interface KeyService { int delKeyByUser(KeyTimely keyTimely); - KeyTimely selectByKey(String key); - - KeyTimely selectByPk(Long key); - - PageInfo pageKeyTimely(PageReq page, SearchReq param); + Page pageKeyTimely(PageReq page, SearchReq param); PageInfo pageMaxHot(PageReq page, SearchReq param); diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/RuleService.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/RuleService.java index 5580df1b67ec483f8a65351d61f1f0d0b69a5edb..0f74758a0c737fbe93cc8368767d006e6fbecc04 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/RuleService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/RuleService.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; @@ -34,3 +34,4 @@ public interface RuleService { PageInfo pageRuleHitCount(PageReq pageReq, SearchReq req, String ownApp); } + diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/UserService.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/UserService.java index 71f57ea705fefcbe37dfdd62b940d6bb303ae83b..b4bb8687fefae6a52dfe927484df3daf564d77ca 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/UserService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/UserService.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/WorkerService.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/WorkerService.java similarity index 93% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/WorkerService.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/WorkerService.java index 4995172fded57cb93d723aa153f928243d2903ef..398dae8de872cc504a81eb2aaba0a4b34a7ae143 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/WorkerService.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/WorkerService.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service; +package com.jd.platform.hotkey.dashboard.biz.service; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/AppCfgServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/AppCfgServiceImpl.java new file mode 100644 index 0000000000000000000000000000000000000000..82ba42ac4388c32f003593ded165673b52c4dcfd --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/AppCfgServiceImpl.java @@ -0,0 +1,82 @@ +package com.jd.platform.hotkey.dashboard.biz.service.impl; + +import com.alibaba.fastjson.JSON; +import com.github.pagehelper.util.StringUtil; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; +import com.jd.platform.hotkey.dashboard.biz.service.AppCfgService; +import com.jd.platform.hotkey.dashboard.common.monitor.PushHandler; +import com.jd.platform.hotkey.dashboard.common.monitor.SlidingWindow; +import com.jd.platform.hotkey.dashboard.util.PageUtil; +import org.springframework.stereotype.Service; +import org.springframework.util.StringUtils; + +import javax.annotation.Resource; +import java.util.ArrayList; +import java.util.List; + +/** + * @ProjectName: hotkey + * @ClassName: AppCfgServiceImpl + * @Author: liyunfeng31 + * @Date: 2020/9/2 9:57 + */ +@Service +public class AppCfgServiceImpl implements AppCfgService { + + + @Resource + private IConfigCenter configCenter; + + + @Override + public Page pageAppCfgVo(PageReq page, String app) { + List keyValues = configCenter.getPrefix(ConfigConstant.appCfgPath); + List cfgVos = new ArrayList<>(); + for (KeyValue kv : keyValues) { + String v = kv.getValue().toStringUtf8(); + String key = kv.getKey().toStringUtf8(); + if(StringUtil.isEmpty(v)){ + configCenter.put(key, JSON.toJSONString(new AppCfgVo(key))); + continue; + } + AppCfgVo vo = JSON.parseObject(v, AppCfgVo.class); + vo.setVersion(kv.getModRevision()); + String k = key.replace(ConfigConstant.appCfgPath,""); + if(StringUtils.isEmpty(app)){ + cfgVos.add(vo); + }else{ + if(k.equals(app)){ cfgVos.add(vo); } + } + } + return PageUtil.pagination(cfgVos,page.getPageSize(),page.getPageNum()-1); + } + + @Override + public AppCfgVo selectAppCfgVo(String app) { + KeyValue kv = configCenter.getKv(ConfigConstant.appCfgPath + app); + if(kv == null || kv.getValue() == null){ + AppCfgVo ap = new AppCfgVo(app); + configCenter.put(ConfigConstant.appCfgPath + app, JSON.toJSONString(ap)); + return ap; + } + String v = kv.getValue().toStringUtf8(); + AppCfgVo cfg = JSON.parseObject(v, AppCfgVo.class); + cfg.setVersion(kv.getModRevision()); + return cfg; + } + + /** + * todo 多节点问题 待完善 + */ + @Override + public void saveAppCfgVo(AppCfgVo cfg) { + cfg.setWindow(new SlidingWindow(cfg.getWarnPeriod()/60,cfg.getWarnMin(),cfg.getWarnMax())); + PushHandler.appCfgMap.put(cfg.getApp(), cfg); + configCenter.put(ConfigConstant.appCfgPath + cfg.getApp(), JSON.toJSONString(cfg)); + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/BizAccessTokenServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/BizAccessTokenServiceImpl.java new file mode 100644 index 0000000000000000000000000000000000000000..2cfd0031206a21c755179ba471a6040a14f6f55a --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/BizAccessTokenServiceImpl.java @@ -0,0 +1,80 @@ +package com.jd.platform.hotkey.dashboard.biz.service.impl; + +import com.jd.platform.hotkey.dashboard.biz.mapper.BizAccessTokenMapper; +import com.jd.platform.hotkey.dashboard.biz.service.IBizAccessTokenService; +import com.jd.platform.hotkey.dashboard.model.BizAccessToken; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; + +import javax.annotation.Resource; +import java.util.List; + +/** + * 咚咚消息tokenService业务层处理 + * + * @author ruoyi + * @date 2020-05-21 + */ +@Service +public class BizAccessTokenServiceImpl implements IBizAccessTokenService { + + + @Resource + private BizAccessTokenMapper bizAccessTokenMapper; + + /** + * 查询咚咚消息token + * + * @param id 咚咚消息tokenID + * @return 咚咚消息token + */ + @Override + public BizAccessToken selectBizAccessTokenById(Long id){ + return bizAccessTokenMapper.selectBizAccessTokenById(id); + } + + /** + * 查询咚咚消息token列表 + * + * @param bizAccessToken 咚咚消息token + * @return 咚咚消息token + */ + @Override + public List selectBizAccessTokenList(BizAccessToken bizAccessToken){ + return bizAccessTokenMapper.selectBizAccessTokenList(bizAccessToken); + } + + /** + * 新增咚咚消息token + * + * @param bizAccessToken 咚咚消息token + * @return 结果 + */ + @Override + public int insertBizAccessToken(BizAccessToken bizAccessToken){ + return bizAccessTokenMapper.insertBizAccessToken(bizAccessToken); + } + + /** + * 修改咚咚消息token + * + * @param bizAccessToken 咚咚消息token + * @return 结果 + */ + @Override + public int updateBizAccessToken(BizAccessToken bizAccessToken){ + return bizAccessTokenMapper.updateBizAccessToken(bizAccessToken); + } + + /** + * 删除咚咚消息token信息 + * + * @param id 咚咚消息tokenID + * @return 结果 + */ + @Override + public int deleteBizAccessTokenById(Long id) + { + return bizAccessTokenMapper.deleteBizAccessTokenById(id); + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ChangeLogServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ChangeLogServiceImpl.java similarity index 84% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ChangeLogServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ChangeLogServiceImpl.java index 50e11afd895c5630218c7a30a3170de80eff43d5..5fa90a5911b79045122559a36507cfd4d600f13e 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/ChangeLogServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ChangeLogServiceImpl.java @@ -1,12 +1,12 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import com.github.pagehelper.PageHelper; import com.github.pagehelper.PageInfo; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.ChangeLogMapper; import com.jd.platform.hotkey.dashboard.model.ChangeLog; -import com.jd.platform.hotkey.dashboard.service.ChangeLogService; +import com.jd.platform.hotkey.dashboard.biz.service.ChangeLogService; import org.springframework.stereotype.Service; import javax.annotation.Resource; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ClearServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ClearServiceImpl.java new file mode 100644 index 0000000000000000000000000000000000000000..462e5c8f801575760172e4ea61864b4d72f83be7 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/ClearServiceImpl.java @@ -0,0 +1,85 @@ +package com.jd.platform.hotkey.dashboard.biz.service.impl; + +import com.github.pagehelper.PageInfo; +import com.github.pagehelper.util.StringUtil; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; +import com.jd.platform.hotkey.dashboard.common.domain.vo.ClearCfgVo; +import com.jd.platform.hotkey.dashboard.biz.service.ClearService; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; +import org.apache.commons.collections4.CollectionUtils; +import org.springframework.stereotype.Service; +import org.springframework.util.StringUtils; +import javax.annotation.Resource; +import java.util.ArrayList; +import java.util.List; + +/** + * @ProjectName: hotkey + * @ClassName: ClearServiceImpl + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:57 + */ +@Service +public class ClearServiceImpl implements ClearService { + + + @Resource + private IConfigCenter configCenter; + + @Resource + private UserService userService; + + + @Override + public PageInfo pageClearCfg(PageReq page, String app) { + List keyValues = configCenter.getPrefix(ConfigConstant.clearCfgPath); + if(CollectionUtils.isEmpty(keyValues)){ + List apps = userService.listApp(); + for (String ap : apps) { + configCenter.put(ConfigConstant.clearCfgPath + ap, Constant.THIRTY_DAY); + } + keyValues = configCenter.getPrefix(ConfigConstant.clearCfgPath); + } + List cfgVos = new ArrayList<>(); + for (KeyValue kv : keyValues) { + String v = kv.getValue().toStringUtf8(); + String key = kv.getKey().toStringUtf8(); + if(StringUtil.isEmpty(v)){ + configCenter.put(key, Constant.THIRTY_DAY); + continue; + } + long version = kv.getModRevision(); + String k = key.replace(ConfigConstant.clearCfgPath,""); + if(StringUtils.isEmpty(app)){ + cfgVos.add(new ClearCfgVo(k, v, version)); + }else{ + if(k.equals(app)){ + cfgVos.add(new ClearCfgVo(k, v, version)); + } + } + } + return new PageInfo<>(cfgVos); + } + + @Override + public ClearCfgVo selectClearCfg(String app) { + KeyValue kv = configCenter.getKv(ConfigConstant.clearCfgPath + app); + if(kv == null || kv.getValue() == null){ + configCenter.put(ConfigConstant.clearCfgPath + app, Constant.THIRTY_DAY); + return new ClearCfgVo(app, Constant.THIRTY_DAY, 0L); + } + String v = kv.getValue().toStringUtf8(); + return new ClearCfgVo(app,v,kv.getModRevision()); + } + + @Override + public int saveClearCfg(ClearCfgVo cfg) { + configCenter.put(ConfigConstant.clearCfgPath + cfg.getApp(), cfg.getTtl()); + return 1; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/KeyServiceImpl.java similarity index 44% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/KeyServiceImpl.java index 88a4263a3090e97301543dcd8d049fdf155996c7..42ebd19fc4c756a60b3bd610a42fb53be036e197 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/KeyServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/KeyServiceImpl.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import cn.hutool.core.date.SystemClock; @@ -8,24 +8,29 @@ import com.github.pagehelper.PageInfo; import com.ibm.etcd.api.Event; import com.jd.platform.hotkey.common.configcenter.ConfigConstant; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.dashboard.common.domain.Constant; +import com.jd.platform.hotkey.dashboard.common.domain.Page; import com.jd.platform.hotkey.dashboard.common.domain.req.ChartReq; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HotKeyLineChartVo; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; -import com.jd.platform.hotkey.dashboard.mapper.KeyRecordMapper; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; -import com.jd.platform.hotkey.dashboard.mapper.StatisticsMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.ChangeLogMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.KeyRecordMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.StatisticsMapper; import com.jd.platform.hotkey.dashboard.model.ChangeLog; import com.jd.platform.hotkey.dashboard.model.KeyRecord; import com.jd.platform.hotkey.dashboard.model.KeyTimely; import com.jd.platform.hotkey.dashboard.model.Statistics; -import com.jd.platform.hotkey.dashboard.service.KeyService; -import com.jd.platform.hotkey.dashboard.service.RuleService; +import com.jd.platform.hotkey.dashboard.netty.HotKeyReceiver; +import com.jd.platform.hotkey.dashboard.biz.service.KeyService; +import com.jd.platform.hotkey.dashboard.biz.service.RuleService; import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.DateUtil; +import com.jd.platform.hotkey.dashboard.util.PageUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.springframework.stereotype.Service; import javax.annotation.Resource; @@ -50,16 +55,17 @@ public class KeyServiceImpl implements KeyService { @Resource private KeyRecordMapper recordMapper; @Resource - private KeyTimelyMapper keyTimelyMapper; - @Resource private StatisticsMapper statisticsMapper; @Resource private RuleService ruleService; @Resource private ChangeLogMapper logMapper; + private Logger logger = LoggerFactory.getLogger(getClass()); + /** * 折线图 + * * @param req req * @return vo */ @@ -68,52 +74,48 @@ public class KeyServiceImpl implements KeyService { int type = req.getType(); String appReq = req.getApp(); // admin 全查 - if(StrUtil.isNotEmpty(appReq)){ + if (StrUtil.isNotEmpty(appReq)) { app = appReq; } req.setApp(null); LocalDateTime now = LocalDateTime.now(); req.setEndTime(req.getEndTime() == null ? DateUtil.ldtToDate(now) : req.getEndTime()); List rules = ruleService.listRules(null); - if(type == 4){ - LocalDateTime st = req.getStartTime() == null? now.minusMinutes(31) : DateUtil.dateToLdt(req.getStartTime()); + if (type == 4) { + LocalDateTime st = req.getStartTime() == null ? now.minusMinutes(31) : DateUtil.dateToLdt(req.getStartTime()); req.setStartTime(DateUtil.ldtToDate(st)); LocalDateTime et = DateUtil.dateToLdt(req.getEndTime()); - boolean longTime = Duration.between(st,et).toHours()>2; + boolean longTime = Duration.between(st, et).toHours() > 2; req.setType(longTime ? 6 : 5); List list = statisticsMapper.listOrderByTime(req); - return CommonUtil.processData(st,et,list,!longTime,rules,app); + return CommonUtil.processData(st, et, list, !longTime, rules, app); } - if(type == 5){ + if (type == 5) { LocalDateTime startTime = now.minusMinutes(31); req.setStartTime(DateUtil.ldtToDate(startTime)); List list = statisticsMapper.listOrderByTime(req); - return CommonUtil.processData(startTime,now,list,true,rules,app); - }else if(type == 6){ + return CommonUtil.processData(startTime, now, list, true, rules, app); + } else if (type == 6) { LocalDateTime startTime2 = now.minusHours(25); req.setStartTime(DateUtil.ldtToDate(startTime2)); List list2 = statisticsMapper.listOrderByTime(req); - return CommonUtil.processData(startTime2,now,list2,false,rules,app); - }else{ + return CommonUtil.processData(startTime2, now, list2, false, rules, app); + } else { LocalDateTime startTime3 = now.minusDays(7).minusHours(1); req.setStartTime(DateUtil.ldtToDate(startTime3)); req.setType(6); List list3 = statisticsMapper.listOrderByTime(req); - return CommonUtil.processData(startTime3,now,list3,false,rules,app); + return CommonUtil.processData(startTime3, now, list3, false, rules, app); } } @Override - public PageInfo pageKeyTimely(PageReq page, SearchReq param) { - PageHelper.startPage(page.getPageNum(), page.getPageSize()); - List listKey = keyTimelyMapper.listKeyTimely(param); - for (KeyTimely timely : listKey) { - timely.setKey(CommonUtil.keyName(timely.getKey())); - timely.setRuleDesc(RuleUtil.ruleDesc(timely.getAppName() + "/" + timely.getKey())); - } - return new PageInfo<>(listKey); + public Page pageKeyTimely(PageReq page, SearchReq param) { + List keyTimelies = HotKeyReceiver.list(param); + return PageUtil.pagination(keyTimelies, page.getPageSize(), page.getPageNum()-1); + } @Override @@ -143,12 +145,12 @@ public class KeyServiceImpl implements KeyService { Map keyDateMap = keyDateMap(statistics, hours); // 获取时间x轴 List list = new ArrayList<>(); - for (int i = hours; i >0 ; i--) { - LocalDateTime time = LocalDateTime.now().minusHours(i-1); + for (int i = hours; i > 0; i--) { + LocalDateTime time = LocalDateTime.now().minusHours(i - 1); int hour = time.getHour(); - list.add(hour+"时"); + list.add(hour + "时"); } - return new HotKeyLineChartVo(list,keyDateMap); + return new HotKeyLineChartVo(list, keyDateMap); } @@ -168,6 +170,13 @@ public class KeyServiceImpl implements KeyService { public int insertKeyByUser(KeyTimely key) { configCenter.putAndGrant(ConfigConstant.hotKeyPath + key.getAppName() + "/" + key.getKey(), System.currentTimeMillis() + "", key.getDuration()); + + //写入本地缓存，实时热key信息 + HotKeyModel hotKeyModel = new HotKeyModel(); + hotKeyModel.setCreateTime(System.currentTimeMillis()); + hotKeyModel.setAppName(key.getAppName()); + hotKeyModel.setKey(key.getKey()); + HotKeyReceiver.put(hotKeyModel); return logMapper.insertSelective(new ChangeLog(key.getAppName(), Constant.HOTKEY_CHANGE, "", key.getKey(), key.getUpdater(), SystemClock.now() + "")); } @@ -185,56 +194,46 @@ public class KeyServiceImpl implements KeyService { String[] arr = keyTimely.getKey().split("/"); //删除client监听目录的key String etcdKey = ConfigConstant.hotKeyPath + arr[0] + "/" + arr[1]; + + //删除caffeine里的实时key + HotKeyReceiver.delete(arr[0] + "/" + arr[1]); + if (configCenter.get(etcdKey) == null) { //如果手工目录也就是client监听的目录里没有该key，那么就往里面放一个，然后再删掉它，这样client才能监听到删除事件 configCenter.putAndGrant(etcdKey, com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE, 1); } configCenter.delete(etcdKey); - //也删除Record目录下的该key，因为不确定要删的key到底在哪 - String recordKey = ConfigConstant.hotKeyRecordPath + arr[0] + "/" + arr[1]; - configCenter.delete(recordKey); - - KeyRecord keyRecord = new KeyRecord(arr[1], "", arr[0], 0L, Constant.HAND, + KeyRecord keyRecord = new KeyRecord(arr[1], "", arr[0], 0, Constant.HAND, Event.EventType.DELETE_VALUE, UUID.randomUUID().toString(), new Date()); recordMapper.insertSelective(keyRecord); - return logMapper.insertSelective(new ChangeLog(arr[0], Constant.HOTKEY_CHANGE, arr[1],"", keyTimely.getUpdater(), SystemClock.now() + "")); + return logMapper.insertSelective(new ChangeLog(keyTimely.getKey(), Constant.HOTKEY_CHANGE, keyTimely.getKey(), "", keyTimely.getUpdater(), SystemClock.now() + "")); } - @Override - public KeyTimely selectByKey(String key) { - return keyTimelyMapper.selectByKey(key); - } - - @Override - public KeyTimely selectByPk(Long id) { - return keyTimelyMapper.selectByPrimaryKey(id); - } - - - - private Map keyDateMap(List statistics, int hours){ + private Map keyDateMap(List statistics, int hours) { Map map = new HashMap<>(10); Map> listMap = statistics.stream().collect(Collectors.groupingBy(Statistics::getKeyName)); for (Map.Entry> m : listMap.entrySet()) { int start = DateUtil.preHoursInt(5); - map.put(m.getKey(),new int[hours]); + map.put(m.getKey(), new int[hours]); int[] data = map.get(m.getKey()); int tmp = 0; for (int i = 0; i < hours; i++) { Statistics st; try { st = m.getValue().get(tmp); - if(String.valueOf(start).endsWith("24")){ start = start + 77; } - if(start != st.getHours()){ + if (String.valueOf(start).endsWith("24")) { + start = start + 77; + } + if (start != st.getHours()) { data[i] = 0; - }else{ - tmp ++; + } else { + tmp++; data[i] = st.getCount(); } start++; - }catch (Exception e){ + } catch (Exception e) { data[i] = 0; } } @@ -243,9 +242,8 @@ public class KeyServiceImpl implements KeyService { } - private void checkParam(SearchReq req) { - if(req.getStartTime() == null || req.getEndTime() == null){ + if (req.getStartTime() == null || req.getEndTime() == null) { req.setStartTime(DateUtil.preTime(5)); req.setEndTime(new Date()); } @@ -255,173 +253,6 @@ public class KeyServiceImpl implements KeyService { }*/ } - - private static List statisticsList(){ - Random rd = new Random(); - List records = new ArrayList<>(); - // statisticsMapper.clear(1); - for (int i = 0; i < 35; i++) { - if(i== 2||i== 3||i== 4||i== 25||i== 12||i== 13||i== 14||i== 30||i==32||i==33||i== 34){ - continue; - } - // 每分钟小时 统计一次record 表 结果记录到统计表 - LocalDateTime now = LocalDateTime.now().minusMinutes(35-i); - Date nowTime = DateUtil.ldtToDate(now); - int day = DateUtil.nowDay(now); - int hour = DateUtil.nowHour(now); - int minus = DateUtil.nowMinus(now); - - Statistics s1 = new Statistics(); - s1.setKeyName("k21"); - s1.setApp("key21APP"); - s1.setCreateTime(nowTime); - s1.setDays(day); - s1.setHours(hour); - s1.setCount(rd.nextInt(800)); - s1.setBizType(1); - s1.setMinutes(minus); - s1.setUuid(1 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s1); - - Statistics s11 = new Statistics(); - s11.setKeyName("k22"); - s11.setApp("key22APP"); - s11.setBizType(1); - s11.setCreateTime(nowTime); - s11.setDays(day); - s11.setMinutes(minus); - s11.setCount(rd.nextInt(600)); - s11.setHours(hour); - s11.setUuid(2 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s11); - } - - if(1==1){ - return records; - } - // statisticsMapper.batchInsert(records); - List list = new ArrayList<>(); - for (int i = 0; i < 30 ; i++) { - if(i == 12 || i ==13){ - }else{ - Statistics st = new Statistics(); - st.setApp("k21"); - st.setKeyName("k21"); - st.setCount(rd.nextInt(100)); - st.setBizType(1); - // LocalDateTime ldf = DateUtil.strToLdt("2006082355"); - // System.out.println(ldf.toString()+" - "+DateUtil.strToLdt("2006082355")); - int time = DateUtil.reviseTime(DateUtil.strToLdt("2006082355",DateUtil.PATTERN_MINUS), i, 1); - // System.out.println(time); - st.setMinutes(time); - // list.add(st); - } - } - List list2 = new ArrayList<>(); - for (int i = 0; i < 30 ; i++) { - if(i == 0 || i == 1 || i == 18 || i == 19 || i == 25){ - - }else{ - Statistics st2 = new Statistics(); - st2.setApp("k22"); - st2.setKeyName("k22"); - st2.setCount(rd.nextInt(100)); - st2.setBizType(1); - st2.setMinutes(DateUtil.reviseTime(DateUtil.strToLdt("2006082355", DateUtil.PATTERN_MINUS),i,1)); - list2.add(st2); - } - } - // list.addAll(list2); - return list; - - } - - private List statisticsList1(int type){ - Random rd = new Random(); - List records = new ArrayList<>(); - statisticsMapper.clear(6); - - if(type == 3){ - int temp = 1; - for (int j = 0; j < 9; j++) { - for (int i = 0; i < 24; i++) { - // 每分钟小时 统计一次record 表 结果记录到统计表 - LocalDateTime now = LocalDateTime.now().minusHours(9 * 24 - temp); - - Date nowTime = DateUtil.ldtToDate(now); - int day = DateUtil.nowDay(now); - int hour = DateUtil.nowHour(now); - Statistics s1 = new Statistics(); - s1.setKeyName("k21"); - s1.setApp("key21APP"); - s1.setCreateTime(nowTime); - s1.setDays(day); - s1.setHours(hour); - s1.setCount(rd.nextInt(200)); - s1.setBizType(6); - s1.setUuid(1 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s1); - - Statistics s11 = new Statistics(); - s11.setKeyName("k22"); - s11.setApp("key22APP"); - s11.setBizType(6); - s11.setCreateTime(nowTime); - s11.setDays(day); - s11.setCount(rd.nextInt(500)); - s11.setHours(hour); - s11.setUuid(2 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s11); - - Statistics s33 = new Statistics(); - s33.setKeyName("k33"); - s33.setApp("key33APP"); - s33.setBizType(6); - s33.setCreateTime(nowTime); - s33.setDays(day); - s33.setCount(rd.nextInt(300)); - s33.setHours(hour); - s33.setUuid(3 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s33); - temp++; - } - } - - - }else{ - for (int i = 0; i < 30; i++) { - // 每分钟小时 统计一次record 表 结果记录到统计表 - LocalDateTime now = LocalDateTime.now().minusHours(30-i); - Date nowTime = DateUtil.ldtToDate(now); - int day = DateUtil.nowDay(now); - int hour = DateUtil.nowHour(now); - Statistics s1 = new Statistics(); - s1.setKeyName("key21"); - s1.setApp("key21APP"); - s1.setCreateTime(nowTime); - s1.setDays(day); - s1.setHours(hour); - s1.setCount(rd.nextInt(800)); - s1.setBizType(2); - s1.setUuid(1 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s1); - - Statistics s11 = new Statistics(); - s11.setKeyName("key22"); - s11.setApp("key22APP"); - s11.setBizType(2); - s11.setCreateTime(nowTime); - s11.setDays(day); - s11.setCount(rd.nextInt(600)); - s11.setHours(hour); - s11.setUuid(2 + "_" + s1.getKeyName() + "_" + hour+ UUID.randomUUID().toString()); - records.add(s11); - } - } - statisticsMapper.batchInsert(records); - return records; - } - } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/RuleServiceImpl.java similarity index 94% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/RuleServiceImpl.java index 9dd493e79c3fce3177b9565f5a78f70e18ada0ef..26f38ecf3a5afd3a38def171cbee72ac477ac29a 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/RuleServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/RuleServiceImpl.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import cn.hutool.core.lang.UUID; import com.alibaba.fastjson.JSON; @@ -11,11 +11,11 @@ import com.jd.platform.hotkey.common.configcenter.IConfigCenter; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.domain.vo.HitCountVo; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; -import com.jd.platform.hotkey.dashboard.mapper.RulesMapper; -import com.jd.platform.hotkey.dashboard.mapper.SummaryMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.ChangeLogMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.RulesMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.SummaryMapper; import com.jd.platform.hotkey.dashboard.model.*; -import com.jd.platform.hotkey.dashboard.service.RuleService; +import com.jd.platform.hotkey.dashboard.biz.service.RuleService; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; import org.springframework.util.StringUtils; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/UserServiceImpl.java similarity index 95% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/UserServiceImpl.java index 5cb83e5d0c0a983ba0a78bc40fdf273c15300c2b..f5da4a8a7218f5c4b17afd85154c9586003a9a25 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/UserServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/UserServiceImpl.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import com.github.pagehelper.PageHelper; import com.github.pagehelper.PageInfo; @@ -8,9 +8,9 @@ import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; -import com.jd.platform.hotkey.dashboard.mapper.UserMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.UserMapper; import com.jd.platform.hotkey.dashboard.model.User; -import com.jd.platform.hotkey.dashboard.service.UserService; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Service; @@ -34,7 +34,7 @@ import java.util.Set; @Service public class UserServiceImpl implements UserService { - @Value("${erp.defaultPwd}") + @Value("${erp.defaultPwd:123}") private String defaultPwd; @Resource private UserMapper userMapper; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/WorkerServiceImpl.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/WorkerServiceImpl.java similarity index 95% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/WorkerServiceImpl.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/WorkerServiceImpl.java index 4c133ee5732e2e41299decdf4924081784a07498..d6270a1a3ccf66eb00a4da3cff9d6ec31ba1fdf9 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/service/impl/WorkerServiceImpl.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/biz/service/impl/WorkerServiceImpl.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.service.impl; +package com.jd.platform.hotkey.dashboard.biz.service.impl; import cn.hutool.core.date.SystemClock; import com.alibaba.fastjson.JSON; @@ -10,11 +10,11 @@ import com.jd.platform.hotkey.common.tool.FastJsonUtils; import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.req.PageReq; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.mapper.ChangeLogMapper; -import com.jd.platform.hotkey.dashboard.mapper.WorkerMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.ChangeLogMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.WorkerMapper; import com.jd.platform.hotkey.dashboard.model.ChangeLog; import com.jd.platform.hotkey.dashboard.model.Worker; -import com.jd.platform.hotkey.dashboard.service.WorkerService; +import com.jd.platform.hotkey.dashboard.biz.service.WorkerService; import org.springframework.dao.DuplicateKeyException; import org.springframework.stereotype.Service; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java new file mode 100644 index 0000000000000000000000000000000000000000..63612e8d8c12c59307c216d598e32ced13d02c49 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/cache/CaffeineBuilder.java @@ -0,0 +1,30 @@ +package com.jd.platform.hotkey.dashboard.cache; + +import com.github.benmanes.caffeine.cache.Cache; +import com.github.benmanes.caffeine.cache.Caffeine; + +import java.util.concurrent.TimeUnit; + +/** + * @author wuweifeng wrote on 2019-12-12 + * @version 1.0 + */ +public class CaffeineBuilder { + + public static Cache cache(int duration) { + return cache(128, 1000000, duration); + } + + + /** + * 构建所有来的要缓存的key getCache + */ + public static Cache cache(int minSize, int maxSize, int expireSeconds) { + return Caffeine.newBuilder() + .initialCapacity(minSize)//初始大小 + .maximumSize(maxSize)//最大数量 + .expireAfterWrite(expireSeconds, TimeUnit.SECONDS)//过期时间 + .build(); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java index 7aa346fa58d47c702cff13531c7caea4af3e24d9..e4065ece95d63cee5566b71428aeb4600471b2ec 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/base/BaseController.java @@ -8,8 +8,7 @@ import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; -import com.jd.platform.hotkey.dashboard.model.User; -import com.jd.platform.hotkey.dashboard.service.UserService; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import io.jsonwebtoken.Claims; import org.springframework.beans.propertyeditors.CustomDateEditor; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java index 963892110d934ec2163949947443fd9271f29bd5..401f52ca99f1819b64bb6e2c565164d98dfbb7dc 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/MyExceptionHandler.java @@ -25,7 +25,7 @@ public class MyExceptionHandler { @ResponseBody public Result bizExceptionHandler(BizException e, HttpServletResponse resp){ resp.setStatus(e.getCode()); - logger.info("业务异常：",e); + logger.error("业务异常：",e); return Result.error(e.getCode(),e.getMsg()); } @@ -34,7 +34,7 @@ public class MyExceptionHandler { @ExceptionHandler(value =Exception.class) @ResponseBody public Result exceptionHandler(Exception e){ - logger.info("未知异常：",e); + logger.error("未知异常：",e); return Result.error(ResultEnum.BIZ_ERROR); } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java index b11fa73687d71cc9a6a13b968b09885622ef0cf8..d47ca2ebbd4da3675d7e426f6faaaced1866320d 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Constant.java @@ -35,6 +35,10 @@ public class Constant { public static final String WORKER_VIEW = "节点信息"; + public static final String CLEAR_VIEW = "数据清理"; + + public static final String APP_CFG_VIEW = "应用配置"; + public static final int MAX_DAY_RANGE = 3; public static final int VERSION = 1; @@ -55,6 +59,9 @@ public class Constant { public static final String INFO = "info"; + public static final String THIRTY_DAY = "30"; + + public static final List HEAD = new ArrayList<>(); static { HEAD.add("热点key"); HEAD.add("次数"); HEAD.add("所属APP");} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java deleted file mode 100644 index 04d4a8325c0eac341df8e743c7d6b5098c951544..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/EventWrapper.java +++ /dev/null @@ -1,95 +0,0 @@ -package com.jd.platform.hotkey.dashboard.common.domain; - -import com.ibm.etcd.api.Event; - -import java.io.Serializable; -import java.util.Date; - -/** - * 包装类，用于保存事件最准确的时间 - */ -public class EventWrapper implements Serializable { - - private String key; - - private String value; - - private Event.EventType eventType; - - private Date date; - - private Long ttl; - - private long version; - - private String uuid; - - @Override - public String toString() { - return "EventWrapper{" + - "key='" + key + '\'' + - ", value='" + value + '\'' + - ", eventType=" + eventType + - ", date=" + date + - ", ttl=" + ttl + - ", version=" + version + - ", uuid='" + uuid + '\'' + - '}'; - } - - public String getUuid() { - return uuid; - } - - public void setUuid(String uuid) { - this.uuid = uuid; - } - - public long getVersion() { - return version; - } - - public void setVersion(long version) { - this.version = version; - } - - public String getKey() { - return key; - } - - public void setKey(String key) { - this.key = key; - } - - public String getValue() { - return value; - } - - public void setValue(String value) { - this.value = value; - } - - public Event.EventType getEventType() { - return eventType; - } - - public void setEventType(Event.EventType eventType) { - this.eventType = eventType; - } - - public void setDate(Date date) { - this.date = date; - } - - public void setTtl(Long ttl) { - this.ttl = ttl; - } - - public Date getDate() { - return date; - } - - public Long getTtl() { - return ttl; - } -} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java new file mode 100644 index 0000000000000000000000000000000000000000..3585f73c25e12beb3ad6622ca515ad7aac53ef10 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/IRecord.java @@ -0,0 +1,27 @@ +package com.jd.platform.hotkey.dashboard.common.domain; + +import java.util.Date; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-09-02 + */ +public interface IRecord { + /** + * appName + "/" + key + */ + String appNameKey(); + + /** + * 手工添加的是时间戳13位，worker传过来的是uuid + */ + String value(); + + /** + * 0插入，1删除 + */ + int type(); + + Date createTime(); +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java index 5f64c7575f762d7585ac458c3c3cc9b75ee8b94f..d10c3a5169fb88777149563c4a2cb1ace803c823 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/Page.java @@ -7,9 +7,13 @@ import java.util.List; public class Page implements Serializable { private static final long serialVersionUID = 1L; - + /** + * 当前第几页 + */ private Integer page; - + /** + * 总共多少条 + */ private int total; private List rows; @@ -20,6 +24,7 @@ public class Page implements Serializable { this.rows = rows; } + public Integer getPage() { return page; } @@ -43,4 +48,13 @@ public class Page implements Serializable { public void setRows(List rows) { this.rows = rows; } + + @Override + public String toString() { + return "Page{" + + "page=" + page + + ", total=" + total + + ", rows=" + rows + + '}'; + } } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/PushMsgWrapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/PushMsgWrapper.java new file mode 100644 index 0000000000000000000000000000000000000000..9b6b78872301e11b81c7d6e7471183125c2f979d --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/PushMsgWrapper.java @@ -0,0 +1,62 @@ +package com.jd.platform.hotkey.dashboard.common.domain; + +import cn.hutool.core.date.SystemClock; + +import java.io.Serializable; + +/** + * 报警消息包装类，用于保存事件最准确的时间 + */ +public class PushMsgWrapper implements Serializable { + + private String app; + + private Long date; + + /** + * -1小于最小阈值; 1超过最大阈值 + */ + private Integer count; + + private String msg; + + public PushMsgWrapper(String app, int count) { + this.app = app; + this.count = count; + this.date = SystemClock.now(); + } + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public Integer getCount() { + return count; + } + + public void setCount(Integer count) { + this.count = count; + } + + public Long getDate() { + return date; + } + + public void setDate(Long date) { + this.date = date; + } + + public String getMsg() { + return msg; + } + + public void setMsg(String msg) { + this.msg = msg; + } +} + + diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/AppCfgVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/AppCfgVo.java new file mode 100644 index 0000000000000000000000000000000000000000..1efe3224e28dfd494b18b3d9321556f227ec7c1c --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/AppCfgVo.java @@ -0,0 +1,145 @@ +package com.jd.platform.hotkey.dashboard.common.domain.vo; + +import com.alibaba.fastjson.JSON; +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.jd.platform.hotkey.dashboard.common.monitor.SlidingWindow; + +import java.beans.Transient; +import java.io.Serializable; + +/** + * @ProjectName: hotkey + * @ClassName: AppCfgVo + * @Author: liyunfeng31 + * @Date: 2020/9/2 10:29 + */ +public class AppCfgVo implements Serializable { + + private String app; + + /** + * 数据保存时长 + */ + private Integer dataTtl; + + /** + * 警报周期 + */ + private Integer warnPeriod; + + /** + * 警报阈值最小值 + */ + private Integer warnMin; + + /** + * 警报阈值最大值 + */ + private Integer warnMax; + + /** + * 警报开关 1开启 0关闭 + */ + private Integer warn; + + /** + * 版本 + */ + private Long version; + + /* + * 最后修改人 + */ + private String modifier; + + + private SlidingWindow window; + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public Integer getDataTtl() { + return dataTtl; + } + + public void setDataTtl(Integer dataTtl) { + this.dataTtl = dataTtl; + } + + public Integer getWarnPeriod() { + return warnPeriod; + } + + public void setWarnPeriod(Integer warnPeriod) { + this.warnPeriod = warnPeriod; + } + + public Integer getWarnMin() { + return warnMin; + } + + public void setWarnMin(Integer warnMin) { + this.warnMin = warnMin; + } + + public Integer getWarnMax() { + return warnMax; + } + + public void setWarnMax(Integer warnMax) { + this.warnMax = warnMax; + } + + public Integer getWarn() { + return warn; + } + + public void setWarn(Integer warn) { + this.warn = warn; + } + + public Long getVersion() { + return version; + } + + public void setVersion(Long version) { + this.version = version; + } + + public String getModifier() { + return modifier; + } + + public void setModifier(String modifier) { + this.modifier = modifier; + } + + public SlidingWindow getWindow() { + return window; + } + + public void setWindow(SlidingWindow window) { + this.window = window; + } + + public AppCfgVo() { + } + + public AppCfgVo(String app) { + this.app = app; + this.dataTtl = 30; + this.warnPeriod = 10*60; + this.warnMin = -1; + this.warnMax = 1000; + this.version = 0L; + this.warn = 1; + this.modifier = "SYSTEM"; + this.window = new SlidingWindow(warnPeriod/60,warnMin,warnMax); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java new file mode 100644 index 0000000000000000000000000000000000000000..65df755e0262be8a4772369de663ff3e843b16f9 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/ClearCfgVo.java @@ -0,0 +1,49 @@ +package com.jd.platform.hotkey.dashboard.common.domain.vo; + +import java.io.Serializable; + +/** + * @ProjectName: hotkey + * @ClassName: ClearCfgVo + * @Description: TODO(一句话描述该类的功能) + * @Author: liyunfeng31 + * @Date: 2020/8/3 9:54 + */ +public class ClearCfgVo implements Serializable { + + private String app; + + private String ttl; + + private Long version; + + public ClearCfgVo(String app, String ttl, Long version) { + this.app = app; + this.ttl = ttl; + this.version = version; + } + + public String getApp() { + return app; + } + + public void setApp(String app) { + this.app = app; + } + + public String getTtl() { + return ttl; + } + + public void setTtl(String ttl) { + this.ttl = ttl; + } + + public Long getVersion() { + return version; + } + + public void setVersion(Long version) { + this.version = version; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java index 3392fc2494c96339b84052b0935dc1fcb7ef5102..71171a6089c8d18d114c849fc0aecb2195c7be3e 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/HotKeyLineChartVo.java @@ -1,5 +1,6 @@ package com.jd.platform.hotkey.dashboard.common.domain.vo; +import java.io.Serializable; import java.util.List; import java.util.Map; import java.util.Set; @@ -7,7 +8,7 @@ import java.util.Set; /** * @author liyunfeng31 */ -public class HotKeyLineChartVo { +public class HotKeyLineChartVo implements Serializable { private List xAxis; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java index 273ce59998d5c60dc2f4c77fae0ef1749d8dc535..3179c5ff818b12cc90f7119aff589a3d635c6dde 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/LineChartVo.java @@ -1,5 +1,6 @@ package com.jd.platform.hotkey.dashboard.common.domain.vo; +import java.io.Serializable; import java.util.List; import java.util.Map; import java.util.Set; @@ -7,7 +8,7 @@ import java.util.Set; /** * @author liyunfeng31 */ -public class LineChartVo { +public class LineChartVo implements Serializable { private Set legend; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/RuleLineChartVo.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/RuleLineChartVo.java index 31ba5305e879aa21bdaa2c339fb639a9f8360584..8f6fbc50509ed4015ba8b281cbbce222161121ed 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/RuleLineChartVo.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/domain/vo/RuleLineChartVo.java @@ -1,9 +1,11 @@ package com.jd.platform.hotkey.dashboard.common.domain.vo; +import java.io.Serializable; + /** * @author liyunfeng31 */ -public class RuleLineChartVo { +public class RuleLineChartVo implements Serializable { private String rule; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java index b5b1a1ee09f5439000c2af7727ff82cf67100ea7..8216cddfd54218625a5a581ac67b52b4a7d0a6c7 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/DataHandler.java @@ -1,20 +1,25 @@ package com.jd.platform.hotkey.dashboard.common.monitor; +import cn.hutool.core.collection.CollectionUtil; import com.alibaba.fastjson.JSON; -import com.ibm.etcd.api.Event; +import com.google.common.collect.Queues; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.dashboard.biz.mapper.KeyRecordMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.StatisticsMapper; +import com.jd.platform.hotkey.dashboard.biz.mapper.SummaryMapper; import com.jd.platform.hotkey.dashboard.common.domain.Constant; -import com.jd.platform.hotkey.dashboard.common.domain.EventWrapper; +import com.jd.platform.hotkey.dashboard.common.domain.IRecord; import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.mapper.KeyRecordMapper; -import com.jd.platform.hotkey.dashboard.mapper.KeyTimelyMapper; -import com.jd.platform.hotkey.dashboard.mapper.StatisticsMapper; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; import com.jd.platform.hotkey.dashboard.model.KeyRecord; -import com.jd.platform.hotkey.dashboard.model.KeyTimely; import com.jd.platform.hotkey.dashboard.model.Statistics; +import com.jd.platform.hotkey.dashboard.netty.HotKeyReceiver; import com.jd.platform.hotkey.dashboard.util.DateUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; -import com.jd.platform.hotkey.dashboard.util.TwoTuple; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.scheduling.annotation.Scheduled; @@ -22,10 +27,13 @@ import org.springframework.stereotype.Component; import javax.annotation.Resource; import java.time.LocalDateTime; +import java.util.ArrayList; import java.util.Date; import java.util.List; +import java.util.UUID; import java.util.concurrent.BlockingQueue; import java.util.concurrent.LinkedBlockingQueue; +import java.util.concurrent.TimeUnit; @Component public class DataHandler { @@ -34,22 +42,35 @@ public class DataHandler { @Resource private KeyRecordMapper keyRecordMapper; - @Resource - private KeyTimelyMapper keyTimelyMapper; + @Resource private StatisticsMapper statisticsMapper; + @Resource + private SummaryMapper summaryMapper; + + @Resource + private IConfigCenter configCenter; + + @Resource + private PushHandler pushHandler; + + + private static final Integer CACHE_SIZE = 10000; + + /** * 队列 */ - private BlockingQueue queue = new LinkedBlockingQueue<>(); + private BlockingQueue RECORD_QUEUE = new LinkedBlockingQueue<>(); + /** * 入队 */ - public void offer(EventWrapper eventWrapper) { + public void offer(IRecord record) { try { - queue.put(eventWrapper); + RECORD_QUEUE.put(record); } catch (InterruptedException e) { e.printStackTrace(); } @@ -57,35 +78,25 @@ public class DataHandler { public void insertRecords() { while (true) { - TwoTuple twoTuple; try { - twoTuple = handHotKey(queue.take()); - if (twoTuple == null) { + List records = new ArrayList<>(); + Queues.drain(RECORD_QUEUE, records, CACHE_SIZE, 1, TimeUnit.SECONDS); + if (CollectionUtil.isEmpty(records)) { continue; } - } catch (Exception e) { - e.printStackTrace(); - log.error("handHotKey error ," + e.getCause()); - continue; - } - KeyRecord keyRecord = twoTuple.getSecond(); - KeyTimely keyTimely = twoTuple.getFirst(); - - if (keyTimely.getUuid() == null) { - keyTimelyMapper.deleteByKeyAndApp(keyTimely.getKey(), keyTimely.getAppName()); - } else { - try { - keyTimelyMapper.saveOrUpdate(keyTimely); - } catch (Exception e) { - log.info("insert timely error",e); + List keyRecordList = new ArrayList<>(); + for (IRecord iRecord : records) { + KeyRecord keyRecord = handHotKey(iRecord); + if (keyRecord != null) { + keyRecordList.add(keyRecord); + } } - } - if (keyRecord != null) { - //插入记录 - keyRecordMapper.insertSelective(keyRecord); - } + keyRecordMapper.batchInsert(keyRecordList); + } catch (InterruptedException e) { + e.printStackTrace(); + } } } @@ -94,22 +105,19 @@ public class DataHandler { /** * 处理热点key和记录 */ - private TwoTuple handHotKey(EventWrapper eventWrapper) { - Date date = eventWrapper.getDate(); - long ttl = eventWrapper.getTtl(); - Event.EventType eventType = eventWrapper.getEventType(); - String appKey = eventWrapper.getKey(); - String value = eventWrapper.getValue(); + private KeyRecord handHotKey(IRecord record) { + Date date = record.createTime(); + String appKey = record.appNameKey(); + String value = record.value(); //appName+"/"+"key" String[] arr = appKey.split("/"); String appName = arr[0]; String key = arr[1]; - String uuid = eventWrapper.getUuid(); - int type = eventType.getNumber(); + String uuid = UUID.randomUUID().toString(); + int type = record.type(); //组建成对象，供累计后批量插入、删除 - TwoTuple timelyKeyRecordTwoTuple = new TwoTuple<>(); - if (eventType.equals(Event.EventType.PUT)) { + if (type == 0) { //如果是客户端删除时发出的put指令 if (com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE.equals(value)) { log.info("client remove key event : " + appKey); @@ -117,17 +125,14 @@ public class DataHandler { } //手工添加的是时间戳13位，worker传过来的是uuid String source = value.length() == 13 ? Constant.HAND : Constant.SYSTEM; - timelyKeyRecordTwoTuple.setFirst(KeyTimely.aKeyTimely().key(key).val(value).appName(appName).duration(ttl).uuid(appKey).createTime(date).build()); String rule = RuleUtil.rule(appKey); - KeyRecord keyRecord = new KeyRecord(key, rule, appName, ttl, source, type, uuid, date); + KeyRecord keyRecord = new KeyRecord(key, rule, appName, 1, source, type, uuid, date); keyRecord.setRule(rule); - timelyKeyRecordTwoTuple.setSecond(keyRecord); - return timelyKeyRecordTwoTuple; - } else if (eventType.equals(Event.EventType.DELETE)) { - timelyKeyRecordTwoTuple.setFirst(KeyTimely.aKeyTimely().key(key).appName(appName).build()); - return timelyKeyRecordTwoTuple; + return keyRecord; + } else { + //是删除 + return null; } - return timelyKeyRecordTwoTuple; } @@ -210,4 +215,62 @@ public class DataHandler { } + + + /** + * 每天根据app的配置清理过期数据 + */ + @Scheduled(cron = "0 0 1 * * ?") + public void clearExpireData() { + try { + LocalDateTime now = LocalDateTime.now(); + List keyValues = configCenter.getPrefix(ConfigConstant.appCfgPath); + for (KeyValue kv : keyValues) { + String val = kv.getValue().toStringUtf8(); + AppCfgVo cfg = JSON.parseObject(val, AppCfgVo.class); + String app = cfg.getApp(); + Date expireDate = DateUtil.ldtToDate(now.minusDays(cfg.getDataTtl())); + summaryMapper.clearExpireData(app, expireDate); + keyRecordMapper.clearExpireData(app, expireDate); + statisticsMapper.clearExpireData(app, expireDate); + } + } catch (Exception e) { + e.printStackTrace(); + } + + } + + + public void dealHotKey() { + while (true) { + try { + //获取发来的这个热key，存入本地caffeine，设置过期时间 + HotKeyModel model = HotKeyReceiver.take(); + //将该key放入实时热key本地缓存中 + if(model != null){ + HotKeyReceiver.put(model); + putRecord(model.getAppName(), model.getKey()); + } + } catch (Exception e) { + e.printStackTrace(); + } + } + } + + + public void putRecord(String app, String key) throws InterruptedException { + this.offer(new IRecord() { + @Override + public String appNameKey() { return app + "/" + key; } + @Override + public String value() { return UUID.randomUUID().toString(); } + @Override + public int type() { return 0; } + @Override + public Date createTime() { return new Date(); } + }); + // 监控和推送 + pushHandler.monitorAndPush(app); + } + } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/PushHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/PushHandler.java new file mode 100644 index 0000000000000000000000000000000000000000..91135bb71fce8048582999cc2190f22d7b2d6696 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/PushHandler.java @@ -0,0 +1,163 @@ +package com.jd.platform.hotkey.dashboard.common.monitor; + +import com.alibaba.fastjson.JSON; +import com.ibm.etcd.api.KeyValue; +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; +import com.jd.platform.hotkey.dashboard.common.domain.PushMsgWrapper; +import com.jd.platform.hotkey.dashboard.common.domain.vo.AppCfgVo; +import com.jd.platform.hotkey.dashboard.warn.dongdong.DongDongApiManager; +import org.apache.commons.collections4.CollectionUtils; +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import javax.annotation.Resource; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.LinkedBlockingQueue; + +/** + * @ProjectName: hotkey + * @ClassName: PushHandler + * @Description: 处理推送 + * @Author: liyunfeng31 + * @Date: 2020/9/3 10:47 + */ + +@Component +public class PushHandler { + + private Logger logger = LoggerFactory.getLogger(getClass()); + + + /** + * 报警标题 + */ + private static final String TITLE = "hotKey异常提醒"; + + /** + * 拦截重复报警间隔 10分钟 + */ + private static final long INTERVAL = 10*60*1000L; + + /** + * app-config map + */ + public static Map appCfgMap = new ConcurrentHashMap<>(); + + /** + * app-time 用于app存储报警时间 做拦截 + */ + private static Map appIntervalMap = new ConcurrentHashMap<>(); + + /** + * 超过阈值的热点记录次数 放到队列里 + */ + private static final BlockingQueue MSG_QUEUE = new LinkedBlockingQueue<>(); + + + @Resource + private IConfigCenter configCenter; + + @Resource + private UserService userService; + + @Resource + private DongDongApiManager apiManager; + + /** + * 监控和入队 + */ + public void monitorAndPush(String app) throws InterruptedException { + AppCfgVo cfg = PushHandler.appCfgMap.get(app); + if(cfg.getWarn().equals(1)){ + SlidingWindow wd = cfg.getWindow(); + int count = wd.addCount(1); + if(count == 0){ return; } + MSG_QUEUE.put(new PushMsgWrapper(app, count)); + } + } + + + /** + * 启动线程处理警报消息队列 + */ + public void pushWarnMsg() { + // 初始化MAP到内存 + initAppCfgMap(); + logger.info("initAppCfgMap success AppCfgMap:{}",JSON.toJSONString(appCfgMap)); + + while (true){ + try { + PushMsgWrapper msgWrapper = MSG_QUEUE.take(); + String warnApp = msgWrapper.getApp(); + Long msgTime = msgWrapper.getDate(); + boolean send = check(warnApp, msgTime); + if(send){ + Integer ct = msgWrapper.getCount(); + String content = ct == -1 ? warnApp+"热点记录频率低于最小阈值，请注意！" : warnApp+"热点记录频率高于最大阈值，请注意！"; + apiManager.push(TITLE,content); + } + } catch (InterruptedException e) { + logger.info("pushWarnMsg thread error , msg :{}", e.getMessage()); + } + } + } + + + /** + * 防止重复发送警报 + * @param warnApp app + * @param msgTime time + * @return result + */ + private synchronized boolean check(String warnApp, Long msgTime){ + Long maxTime = appIntervalMap.get(warnApp); + if(maxTime == null){ + appIntervalMap.put(warnApp,msgTime+INTERVAL); + return true; + }else{ + if(msgTime > maxTime){ + appIntervalMap.put(warnApp,msgTime+INTERVAL); + return true; + } + } + return false; + } + + + /** + * 初始化cfgMap和滑动窗口 + */ + private void initAppCfgMap() { + /** + * 为了加入最小值 + */ + configCenter.delete(ConfigConstant.appCfgPath); + List keyValues = configCenter.getPrefix(ConfigConstant.appCfgPath); + if(CollectionUtils.isEmpty(keyValues) || keyValues.size()==1){ + List apps = userService.listApp(); + for (String ap : apps) { + AppCfgVo cfg = new AppCfgVo(ap); + appCfgMap.put(ap,cfg); + configCenter.put(ConfigConstant.appCfgPath + ap, JSON.toJSONString(cfg)); + } + return; + } + + for (KeyValue keyValue : keyValues) { + String val = keyValue.getValue().toStringUtf8(); + if(StringUtils.isNotEmpty(val)){ + AppCfgVo cfg = JSON.parseObject(val, AppCfgVo.class); + appCfgMap.put(cfg.getApp(),cfg); + } + } + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/SlidingWindow.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/SlidingWindow.java new file mode 100644 index 0000000000000000000000000000000000000000..9dcbcec0cf95398892ff08404a0903a1dfe83641 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/SlidingWindow.java @@ -0,0 +1,135 @@ +package com.jd.platform.hotkey.dashboard.common.monitor; + + +import cn.hutool.core.date.SystemClock; + +import java.util.concurrent.BrokenBarrierException; +import java.util.concurrent.CyclicBarrier; +import java.util.concurrent.atomic.AtomicInteger; + +/** + * 滑动窗口 + */ +public class SlidingWindow { + /** + * 循环队列，就是装多个窗口用，该数量是windowSize的2倍 + */ + private AtomicInteger[] timeSlices; + /** + * 队列的总长度 + */ + private int timeSliceSize; + /** + * 每个时间片的时长，以毫秒为单位 + */ + private int timeMillisPerSlice; + /** + * 共有多少个时间片（即窗口长度） + */ + private int windowSize; + /** + * 在一个完整窗口期内允许通过的最大阈值 + */ + private int max; + + /** + * 在一个完整窗口期内最小阈值 + */ + private int min; + /** + * 该滑窗的起始创建时间，也就是第一个数据 + */ + private long beginTimestamp; + /** + * 最后一个数据的时间戳 + */ + private long lastAddTimestamp; + + + public SlidingWindow(int windowSize, int min, int max) { + this.timeMillisPerSlice = 60 * 1000; + this.windowSize = windowSize; + this.max = max; + this.min = min; + // 保证存储在至少两个window + this.timeSliceSize = windowSize * 2; + reset(); + } + + /** + * 初始化 + */ + private void reset() { + beginTimestamp = SystemClock.now(); + //窗口个数 + AtomicInteger[] localTimeSlices = new AtomicInteger[timeSliceSize]; + for (int i = 0; i < timeSliceSize; i++) { + localTimeSlices[i] = new AtomicInteger(0); + } + timeSlices = localTimeSlices; + } + + + private int locationIndex() { + long now = SystemClock.now(); + if (now - lastAddTimestamp > timeMillisPerSlice * windowSize) { reset(); } + int index = (int) (((now - beginTimestamp) / timeMillisPerSlice) % timeSliceSize); + return Math.max(index, 0); + } + + + public synchronized int addCount(int count) { + int index = locationIndex(); + clearFromIndex(index); + int sum = 0; + // 在当前时间片里继续+1 + sum += timeSlices[index].addAndGet(count); + //加上前面几个时间片 + for (int i = 1; i < windowSize; i++) { + sum += timeSlices[(index - i + timeSliceSize) % timeSliceSize].get(); + } + lastAddTimestamp = SystemClock.now(); + if(sum >= max){ + return sum; + }else if(sum < min){ + return -1; + }else{ + return 0; + } + } + + private void clearFromIndex(int index) { + for (int i = 1; i <= windowSize; i++) { + int j = index + i; + if (j >= windowSize * 2) { + j -= windowSize * 2; + } + timeSlices[j].set(0); + } + } + + + public int getWindowSize() { + return windowSize; + } + + public void setWindowSize(int windowSize) { + this.windowSize = windowSize; + } + + public int getMax() { + return max; + } + + public void setMax(int max) { + this.max = max; + } + + public int getMin() { + return min; + } + + public void setMin(int min) { + this.min = min; + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java deleted file mode 100644 index 5d4239437810b0a8535acb233bb2d116adcc77ba..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/controller/TestController.java +++ /dev/null @@ -1,21 +0,0 @@ -package com.jd.platform.hotkey.dashboard.controller; - -/** - * @author wuweifeng - * @version 1.0 - * @date 2020-07-16 - */ -//@RestController -//public class TestController { -// @Resource -// private IConfigCenter configCenter; -// -// @RequestMapping("delete") -// public Object delete(String key) { -//// configCenter.putAndGrant(ConfigConstant.hotKeyRecordPath + "sample" + "/" + key, com.jd.platform.hotkey.common.tool.Constant.DEFAULT_DELETE_VALUE, 1); -// configCenter.putAndGrant(ConfigConstant.hotKeyRecordPath + "bbb" + "/" + key, "", 60); -//// configCenter.delete(ConfigConstant.hotKeyPath + "sample" + "/" + key); -// -// return "1"; -// } -//} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpUimInterceptor.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpUimInterceptor.java index beabf4e29cf2a1b453357ad18d5b0cc05d95f892..2bd7653a9a10d93f0f7c60cb4f3dec27467d692a 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpUimInterceptor.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/erp/ErpUimInterceptor.java @@ -3,7 +3,7 @@ package com.jd.platform.hotkey.dashboard.erp; import com.jd.common.springmvc.interceptor.SpringSSOInterceptor; import com.jd.common.web.LoginContext; import com.jd.platform.hotkey.dashboard.model.User; -import com.jd.platform.hotkey.dashboard.service.UserService; +import com.jd.platform.hotkey.dashboard.biz.service.UserService; import org.springframework.beans.factory.annotation.Autowired; import javax.servlet.http.Cookie; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/EtcdConfig.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdConfig.java similarity index 93% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/EtcdConfig.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdConfig.java index 0f585b61aaf5f396ce92a41f63d01788aa333930..edb0708dadf8cae2230d480e237d936ce6387cfe 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/config/EtcdConfig.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdConfig.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.common.config; +package com.jd.platform.hotkey.dashboard.etcd; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; import com.jd.platform.hotkey.common.configcenter.etcd.JdEtcdBuilder; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java similarity index 77% rename from dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java rename to dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java index 92fa472d0c6b42101ebf2bea6528682feec762af..6714da9f6f504e2491abe0b43f2fff5b352e466f 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/common/monitor/EtcdMonitor.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdMonitor.java @@ -1,4 +1,4 @@ -package com.jd.platform.hotkey.dashboard.common.monitor; +package com.jd.platform.hotkey.dashboard.etcd; import cn.hutool.core.util.StrUtil; import com.ibm.etcd.api.Event; @@ -9,24 +9,24 @@ import com.jd.platform.hotkey.common.configcenter.IConfigCenter; import com.jd.platform.hotkey.common.rule.KeyRule; import com.jd.platform.hotkey.common.tool.FastJsonUtils; import com.jd.platform.hotkey.dashboard.common.domain.Constant; -import com.jd.platform.hotkey.dashboard.common.domain.EventWrapper; -import com.jd.platform.hotkey.dashboard.mapper.SummaryMapper; +import com.jd.platform.hotkey.dashboard.common.domain.IRecord; +import com.jd.platform.hotkey.dashboard.common.monitor.DataHandler; +import com.jd.platform.hotkey.dashboard.biz.mapper.SummaryMapper; +import com.jd.platform.hotkey.dashboard.common.monitor.PushHandler; import com.jd.platform.hotkey.dashboard.model.Worker; -import com.jd.platform.hotkey.dashboard.service.WorkerService; +import com.jd.platform.hotkey.dashboard.netty.HotKeyReceiver; +import com.jd.platform.hotkey.dashboard.biz.service.WorkerService; import com.jd.platform.hotkey.dashboard.util.CommonUtil; import com.jd.platform.hotkey.dashboard.util.RuleUtil; import io.grpc.StatusRuntimeException; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; -import org.springframework.util.StringUtils; import javax.annotation.PostConstruct; import javax.annotation.Resource; -import java.util.ArrayList; -import java.util.Date; -import java.util.List; -import java.util.Map; +import java.util.*; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; @@ -54,8 +54,11 @@ public class EtcdMonitor { @Resource private DataHandler dataHandler; + @Resource + private PushHandler pushHandler; + - public static final ExecutorService threadPoolExecutor = Executors.newCachedThreadPool(); + public static final ExecutorService threadPoolExecutor = Executors.newFixedThreadPool(16); /** * 监听新来的热key，该key的产生是来自于手工在控制台添加 @@ -65,48 +68,36 @@ public class EtcdMonitor { KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.hotKeyPath); while (watchIterator.hasNext()) { Event event = event(watchIterator); - EventWrapper eventWrapper = build(event); + KeyValue kv = event.getKv(); + String v = kv.getValue().toStringUtf8(); + Event.EventType eventType = event.getType(); String appKey = event.getKv().getKey().toStringUtf8().replace(ConfigConstant.hotKeyPath, ""); - eventWrapper.setKey(appKey); - dataHandler.offer(eventWrapper); - } - }); - } + dataHandler.offer(new IRecord() { + @Override + public String appNameKey() { + return appKey; + } - /** - * 监听新来的热key，该key的产生是来自于worker集群推送过来的 - */ - public void watchHotKeyRecord() { - threadPoolExecutor.submit(() -> { - KvClient.WatchIterator watchIterator = configCenter.watchPrefix(ConfigConstant.hotKeyRecordPath); - while (watchIterator.hasNext()) { - Event event = event(watchIterator); - EventWrapper eventWrapper = build(event); + @Override + public String value() { + return v; + } - String appKey = event.getKv().getKey().toStringUtf8().replace(ConfigConstant.hotKeyRecordPath, ""); - eventWrapper.setKey(appKey); + @Override + public int type() { + return eventType.getNumber(); + } - dataHandler.offer(eventWrapper); + @Override + public Date createTime() { + return new Date(); + } + }); } }); } - private EventWrapper build(Event event) { - KeyValue kv = event.getKv(); - long ttl = configCenter.timeToLive(kv.getLease()); - String v = kv.getValue().toStringUtf8(); - Event.EventType eventType = event.getType(); - EventWrapper eventWrapper = new EventWrapper(); - eventWrapper.setValue(v); - eventWrapper.setDate(new Date()); - eventWrapper.setTtl(ttl); - eventWrapper.setVersion(kv.getVersion()); - eventWrapper.setEventType(eventType); - eventWrapper.setUuid(v); - return eventWrapper; - } - @PostConstruct public void startWatch() { @@ -116,26 +107,53 @@ public class EtcdMonitor { //规则拉取完毕后才能去开始入库 insertRecords(); - //开始监听热key产生 - watchHotKeyRecord(); + //开始入库 + dealHotKey(); + //监听手工创建的key watchHandOperationHotKey(); //监听rule变化 watchRule(); - watchWorkers(); +// watchWorkers(); //观察热key访问次数和总访问次数，并做统计 watchHitCount(); } + /** + * 每秒去清理一次caffeine + */ + @Scheduled(fixedRate = 1000) + public void cleanCaffeine() { + try { + HotKeyReceiver.cleanUpCaffeine(); + } catch (Exception e) { + e.printStackTrace(); + } + } + private void insertRecords() { threadPoolExecutor.submit(() -> { dataHandler.insertRecords(); }); } + + /** + * 开始消费各worker发来的热key + */ + private void dealHotKey() { + threadPoolExecutor.submit(() -> { + dataHandler.dealHotKey(); + }); + threadPoolExecutor.submit(() -> { + pushHandler.pushWarnMsg(); + }); + + } + /** * 异步监听rule规则变化 */ @@ -149,7 +167,7 @@ public class EtcdMonitor { Event event = event(watchIterator); try { - log.info("---------watch rule change---------"); + log.info("---------watch rule change---------"+event.getType().toString()); //全量拉取rule信息 fetchRuleFromEtcd(); } catch (Exception e) { @@ -217,7 +235,6 @@ public class EtcdMonitor { } - /** * 监听热key访问次数和总访问次数 */ @@ -252,5 +269,4 @@ public class EtcdMonitor { return watchIterator.next().getEvents().get(0); } - } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java new file mode 100644 index 0000000000000000000000000000000000000000..d0a44c5690e9fa212317bbc86055bbb3e36994a0 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/etcd/EtcdRegister.java @@ -0,0 +1,63 @@ +package com.jd.platform.hotkey.dashboard.etcd; + +import com.jd.platform.hotkey.common.configcenter.ConfigConstant; +import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import com.jd.platform.hotkey.common.tool.IpUtils; +import com.jd.platform.hotkey.dashboard.netty.NodesServer; +import org.springframework.stereotype.Component; + +import javax.annotation.PostConstruct; +import javax.annotation.Resource; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; + +/** + * dashboard也注册到etcd，供各个worker连接 + * + * @author wuweifeng + * @version 1.0 + * @date 2020-08-28 + */ +@Component +public class EtcdRegister { + @Resource + private IConfigCenter configCenter; + @Resource + private NodesServer nodesServer; + + /** + * 每隔一会去check一下，自己还在不在etcd里 + */ + @PostConstruct + public void makeSureSelfOn() { + new Thread(() -> nodesServer.startNettyServer(ConfigConstant.dashboardPort)).start(); + + //开启上传worker信息 + ScheduledExecutorService scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(); + scheduledExecutorService.scheduleAtFixedRate(() -> { + + try { + uploadSelfInfo(); + } catch (Exception e) { + //do nothing + } + + }, 3, 30, TimeUnit.SECONDS); + } + + private void uploadSelfInfo() { + configCenter.putAndGrant(buildKey(), buildValue(), 32); + } + + private String buildKey() { + String hostName = IpUtils.getHostName(); + return ConfigConstant.dashboardPath + hostName; + } + + private String buildValue() { + String ip = IpUtils.getIp(); + return ip + ":" + ConfigConstant.dashboardPort; + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java index 173a46598708858ad5b59aab24f3a855ca6f7ac8..8fcd977c843c9e6ed6b99723a0fbb72f8ae4a0b6 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/interceptor/JwtInterceptor.java @@ -4,14 +4,12 @@ package com.jd.platform.hotkey.dashboard.interceptor; import com.jd.platform.hotkey.dashboard.common.domain.Constant; import com.jd.platform.hotkey.dashboard.common.eunm.ResultEnum; import com.jd.platform.hotkey.dashboard.common.ex.BizException; -import com.jd.platform.hotkey.dashboard.service.UserService; import com.jd.platform.hotkey.dashboard.util.JwtTokenUtil; import io.jsonwebtoken.Claims; import org.springframework.http.HttpMethod; import org.springframework.util.StringUtils; import org.springframework.web.servlet.handler.HandlerInterceptorAdapter; -import javax.annotation.Resource; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java deleted file mode 100644 index 3423da34835955cdee38ce5bf8dc47973e4dc682..0000000000000000000000000000000000000000 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.java +++ /dev/null @@ -1,31 +0,0 @@ -package com.jd.platform.hotkey.dashboard.mapper; - -import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; -import com.jd.platform.hotkey.dashboard.model.KeyTimely; -import org.apache.ibatis.annotations.Mapper; - -import java.util.List; - -@Mapper -public interface KeyTimelyMapper { - - int clear(); - - int deleteByKeyAndApp(String key,String appName); - - int insertSelective(KeyTimely key); - - KeyTimely selectByPrimaryKey(Long id); - - KeyTimely selectByKey(String key); - - int updateByKey(KeyTimely key); - - List listKeyTimely(SearchReq param); - - int batchInsert(List list); - - void batchDeleted(List deleteList); - - void saveOrUpdate(KeyTimely keyTimely); -} \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/BizAccessToken.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/BizAccessToken.java new file mode 100644 index 0000000000000000000000000000000000000000..98c0c4d6034883bcde493d39c65400d65a1ca6e3 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/BizAccessToken.java @@ -0,0 +1,116 @@ +package com.jd.platform.hotkey.dashboard.model; + +import org.apache.commons.lang3.builder.ToStringBuilder; +import org.apache.commons.lang3.builder.ToStringStyle; + +import java.io.Serializable; +import java.util.Date; + +/** + * 咚咚消息token对象 biz_access_token + * @author ruoyi + * @date 2020-05-21 + */ +public class BizAccessToken implements Serializable { + private static final long serialVersionUID = 1L; + + public String APP = "2"; + + /** ID */ + private Long id; + + /** Token */ + private String token; + + /** flag */ + private String flag; + + /** 创建人 */ + private String createdBy; + + /** 创建时间 */ + private Date createdTime; + + /** 更新人 */ + private String updatedBy; + + /** 更新时间 */ + private Date updatedTime; + + public void setId(Long id) + { + this.id = id; + } + + public Long getId() + { + return id; + } + public void setToken(String token) + { + this.token = token; + } + + public String getToken() + { + return token; + } + public void setFlag(String flag) + { + this.flag = flag; + } + + public String getFlag() + { + return flag; + } + public void setCreatedBy(String createdBy) + { + this.createdBy = createdBy; + } + + public String getCreatedBy() + { + return createdBy; + } + public void setCreatedTime(Date createdTime) + { + this.createdTime = createdTime; + } + + public Date getCreatedTime() + { + return createdTime; + } + public void setUpdatedBy(String updatedBy) + { + this.updatedBy = updatedBy; + } + + public String getUpdatedBy() + { + return updatedBy; + } + public void setUpdatedTime(Date updatedTime) + { + this.updatedTime = updatedTime; + } + + public Date getUpdatedTime() + { + return updatedTime; + } + + @Override + public String toString() { + return new ToStringBuilder(this, ToStringStyle.MULTI_LINE_STYLE) + .append("id", getId()) + .append("token", getToken()) + .append("flag", getFlag()) + .append("createdBy", getCreatedBy()) + .append("createdTime", getCreatedTime()) + .append("updatedBy", getUpdatedBy()) + .append("updatedTime", getUpdatedTime()) + .toString(); + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java index 97f0b4b6ff0e6d2fe8b6e2f8a742497f01535786..e8fdf0f06dbc9524a516944c0845a5c139328dda 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyRecord.java @@ -16,7 +16,7 @@ public class KeyRecord implements Serializable { /** * 缓存时间 */ - private Long duration; + private int duration; /** * 来源： SYSTEM 系统探测；USERNAME创建人 @@ -43,7 +43,7 @@ public class KeyRecord implements Serializable { public KeyRecord() { } - public KeyRecord(String key,String val, String appName, Long duration, + public KeyRecord(String key,String val, String appName, int duration, String source, Integer type,String uuid, Date createTime) { this.key = key; this.val = val; @@ -103,11 +103,11 @@ public class KeyRecord implements Serializable { this.val = val; } - public Long getDuration() { + public int getDuration() { return duration; } - public void setDuration(Long duration) { + public void setDuration(int duration) { this.duration = duration; } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java index 6efd7c5d9b6f9eebe3a88dd9e55718f326feaf9c..67b9d0ac997647e7f8a05db6cc9915a3c286d332 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/model/KeyTimely.java @@ -22,7 +22,6 @@ public class KeyTimely implements Serializable { private Date createTime; - private String uuid; /** * 该rule的描述 */ @@ -44,12 +43,25 @@ public class KeyTimely implements Serializable { public KeyTimely(String key, String val, String appName, Long duration, String uuid, Date createTime) { this.key = key; this.val = val; - this.uuid = uuid; this.appName = appName; this.duration = duration; this.createTime = createTime; } + @Override + public String toString() { + return "KeyTimely{" + + "id=" + id + + ", key='" + key + '\'' + + ", appName='" + appName + '\'' + + ", val='" + val + '\'' + + ", duration=" + duration + + ", createTime=" + createTime + + ", ruleDesc='" + ruleDesc + '\'' + + ", updater='" + updater + '\'' + + '}'; + } + public String getRuleDesc() { return ruleDesc; } @@ -106,14 +118,6 @@ public class KeyTimely implements Serializable { this.createTime = createTime; } - public String getUuid() { - return uuid; - } - - public void setUuid(String uuid) { - this.uuid = uuid; - } - public String getUpdater() { return updater; } @@ -186,7 +190,6 @@ public class KeyTimely implements Serializable { keyTimely.setVal(val); keyTimely.setDuration(duration); keyTimely.setCreateTime(createTime); - keyTimely.setUuid(uuid); keyTimely.setRuleDesc(ruleDesc); keyTimely.setUpdater(updater); return keyTimely; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java new file mode 100644 index 0000000000000000000000000000000000000000..88057c9fabc8548effa88aefa0724dcb33701abb --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/HotKeyReceiver.java @@ -0,0 +1,175 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import cn.hutool.core.util.StrUtil; +import com.github.benmanes.caffeine.cache.Cache; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.rule.KeyRule; +import com.jd.platform.hotkey.dashboard.cache.CaffeineBuilder; +import com.jd.platform.hotkey.dashboard.common.domain.req.SearchReq; +import com.jd.platform.hotkey.dashboard.model.KeyTimely; +import com.jd.platform.hotkey.dashboard.util.RuleUtil; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.ConcurrentMap; +import java.util.concurrent.LinkedBlockingQueue; +import java.util.stream.Collectors; + +/** + * 此处存储所有发来的热key，统一处理入库 + * + * @author wuweifeng + * @version 1.0 + * @date 2020-08-31 + */ +public class HotKeyReceiver { + + /** + * 发来的热key集中营 + */ + private static LinkedBlockingQueue hotKeyStoreQueue = new LinkedBlockingQueue<>(); + /** + * 存储实时hotkey，供界面查询实时热key + */ + private static Map> aliveKeyStore = new ConcurrentHashMap<>(); + + private static Logger logger = LoggerFactory.getLogger("HotKeyReceiver"); + + /** + * netty收到的先存这里 + */ + public static void push(HotKeyModel model) { + hotKeyStoreQueue.offer(model); + } + + public static HotKeyModel take() { + try { + return hotKeyStoreQueue.take(); + } catch (InterruptedException e) { + e.printStackTrace(); + return null; + } + } + + /** + * 将热key存入本地缓存，设置过期时间 + */ + public static void put(HotKeyModel hotKeyModel) { + String appNameKey = hotKeyModel.getAppName() + "/" + hotKeyModel.getKey(); + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + logger.error("rule is null, hotkeyModel " + hotKeyModel.getAppName() + "-" + hotKeyModel.getKey()); + return; + } + Cache cache = aliveKeyStore.computeIfAbsent(keyRule.getDuration(), s -> CaffeineBuilder.cache(keyRule.getDuration())); + cache.put(appNameKey, hotKeyModel); + } + + /** + * 展示当前所有的实时热key + */ + public static List list(SearchReq searchReq) { + List timelyList = new ArrayList<>(); + + long now = System.currentTimeMillis(); + + for (Integer duration : aliveKeyStore.keySet()) { + Cache cache = aliveKeyStore.get(duration); + ConcurrentMap concurrentHashMap = cache.asMap(); + for (Map.Entry entry : concurrentHashMap.entrySet()) { + KeyTimely keyTimely = parse((HotKeyModel) entry.getValue(), now); + if (keyTimely == null) { + continue; + } + timelyList.add(keyTimely); + } + } + + if (searchReq != null) { + if (StrUtil.isNotEmpty(searchReq.getApp())) { + timelyList = timelyList.parallelStream().filter(keyTimely -> searchReq.getApp().equals(keyTimely.getAppName())).collect(Collectors.toList()); + } + if (StrUtil.isNotEmpty(searchReq.getKey())) { + timelyList = timelyList.parallelStream().filter(keyTimely -> keyTimely.getKey().startsWith(searchReq.getKey())).collect(Collectors.toList()); + } + + } + + timelyList.sort(Comparator.comparing(KeyTimely::getCreateTime).reversed()); + + return timelyList; + } + + /** + * 将hotkeyModel变成前端需要的对象 + */ + private static KeyTimely parse(HotKeyModel hotKeyModel, long now) { + String appNameKey = hotKeyModel.getAppName() + "/" + hotKeyModel.getKey(); + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + return null; + } + long remainTime = keyRule.getDuration() * 1000 - (now - hotKeyModel.getCreateTime()); + return KeyTimely.aKeyTimely() + .key(hotKeyModel.getKey()) + .val(UUID.randomUUID().toString()) + .appName(hotKeyModel.getAppName()) + .duration(remainTime / 1000) + .ruleDesc(RuleUtil.ruleDesc(appNameKey)) + .createTime(new Date(hotKeyModel.getCreateTime())).build(); + } + +// public static void main(String[] args) { +// List keyTimelyList = new ArrayList<>(); +// for (int i = 0; i < 24; i++) { +// KeyTimely keyTimely = new KeyTimely(); +// keyTimely.setKey(i + ""); +// keyTimely.setCreateTime(new Date()); +// try { +// Thread.sleep(10); +// } catch (InterruptedException e) { +// e.printStackTrace(); +// } +// keyTimelyList.add(keyTimely); +// } +// keyTimelyList.sort(Comparator.comparing(KeyTimely::getCreateTime).reversed()); +// +// Page page =PageUtil.pagination(keyTimelyList, 10, 3); +// System.out.println(page.getTotal()); +// System.out.println(page.getPage()); +// System.out.println(page.getRows()); +// } + + + + + + /** + * 删除实时热key + */ + public static boolean delete(String appNameKey) { + KeyRule keyRule = RuleUtil.findByKey(appNameKey); + if (keyRule == null) { + return false; + } + Cache cache = aliveKeyStore.get(keyRule.getDuration()); + if (cache == null) { + return false; + } + cache.invalidate(appNameKey); + return true; + } + + /** + * 定时清理caffeine + */ + public static void cleanUpCaffeine() { + for (Integer duration : aliveKeyStore.keySet()) { + Cache cache = aliveKeyStore.get(duration); + cache.cleanUp(); + } + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java new file mode 100755 index 0000000000000000000000000000000000000000..a3dc7eeb5e78b7d6e1364e0b2aca50ddd1ab809e --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServer.java @@ -0,0 +1,78 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import com.jd.platform.hotkey.common.tool.Constant; +import io.netty.bootstrap.ServerBootstrap; +import io.netty.buffer.ByteBuf; +import io.netty.buffer.Unpooled; +import io.netty.channel.*; +import io.netty.channel.nio.NioEventLoopGroup; +import io.netty.channel.socket.nio.NioServerSocketChannel; +import io.netty.handler.codec.DelimiterBasedFrameDecoder; +import io.netty.handler.codec.string.StringDecoder; +import io.netty.handler.logging.LogLevel; +import io.netty.handler.logging.LoggingHandler; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import java.util.concurrent.TimeUnit; + +/** + * 该server用于给各个worker实例连接用。 + * + * @author wuweifeng wrote on 2019-11-05. + */ +@Component +public class NodesServer { + private Logger logger = LoggerFactory.getLogger(getClass()); + + public void startNettyServer(int port) { + //boss单线程 + EventLoopGroup bossGroup = new NioEventLoopGroup(1); + EventLoopGroup workerGroup = new NioEventLoopGroup(4); + try { + ServerBootstrap bootstrap = new ServerBootstrap(); + bootstrap.group(bossGroup, workerGroup) + .channel(NioServerSocketChannel.class) + .handler(new LoggingHandler(LogLevel.INFO)) + .option(ChannelOption.SO_BACKLOG, 1024) + //保持长连接 + .childOption(ChannelOption.SO_KEEPALIVE, true) + //出来网络io事件，如记录日志、对消息编解码等 + .childHandler(new ChildChannelHandler()); + //绑定端口，同步等待成功 + ChannelFuture future = bootstrap.bind(port).sync(); + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + bossGroup.shutdownGracefully (1000, 3000, TimeUnit.MILLISECONDS); + workerGroup.shutdownGracefully (1000, 3000, TimeUnit.MILLISECONDS); + })); + //等待服务器监听端口关闭 + future.channel().closeFuture().sync(); + } catch (Exception e) { + e.printStackTrace(); + logger.error("dashboard netty server start failure"); + } finally { + //优雅退出，释放线程池资源 + bossGroup.shutdownGracefully(); + workerGroup.shutdownGracefully(); + } + } + + /** + * handler类 + */ + private class ChildChannelHandler extends ChannelInitializer { + + @Override + protected void initChannel(Channel ch) { + NodesServerHandler serverHandler = new NodesServerHandler(); + + ByteBuf delimiter = Unpooled.copiedBuffer(Constant.DELIMITER.getBytes()); + ch.pipeline() + .addLast(new DelimiterBasedFrameDecoder(Constant.MAX_LENGTH, delimiter)) + .addLast(new StringDecoder()) + .addLast(serverHandler); + } + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java new file mode 100755 index 0000000000000000000000000000000000000000..9650c2fd9ef5b517eb0769e75aae5fe2fce7d810 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/netty/NodesServerHandler.java @@ -0,0 +1,66 @@ +package com.jd.platform.hotkey.dashboard.netty; + +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.model.HotKeyMsg; +import com.jd.platform.hotkey.common.model.MsgBuilder; +import com.jd.platform.hotkey.common.model.typeenum.MessageType; +import com.jd.platform.hotkey.common.tool.FastJsonUtils; +import com.jd.platform.hotkey.common.tool.flush.FlushUtil; +import io.netty.channel.ChannelHandlerContext; +import io.netty.channel.SimpleChannelInboundHandler; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.util.StringUtils; + +import java.util.List; + +import static com.jd.platform.hotkey.common.tool.Constant.PONG; + +/** + * 这里处理所有netty事件。 + * + * @author wuweifeng wrote on 2019-11-05. + */ +public class NodesServerHandler extends SimpleChannelInboundHandler { + + private Logger logger = LoggerFactory.getLogger(getClass()); + + @Override + protected void channelRead0(ChannelHandlerContext ctx, String message) { + if (StringUtils.isEmpty(message)) { + return; + } + try { + HotKeyMsg msg = FastJsonUtils.toBean(message, HotKeyMsg.class); + if (MessageType.PING == msg.getMessageType()) { + String hotMsg = FastJsonUtils.convertObjectToJSON(new HotKeyMsg(MessageType.PONG, PONG)); + FlushUtil.flush(ctx, MsgBuilder.buildByteBuf(hotMsg)); + } else if (MessageType.REQUEST_HOT_KEY == msg.getMessageType()) { + List list = FastJsonUtils.toList(msg.getBody(), HotKeyModel.class); + for (HotKeyModel hotKeyModel : list) { + HotKeyReceiver.push(hotKeyModel); + } + } + } catch (Exception e) { + e.printStackTrace(); + } + + } + + @Override + public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception { + logger.error("some thing is error , " + cause.getMessage()); + } + + @Override + public void channelActive(ChannelHandlerContext ctx) throws Exception { + super.channelActive(ctx); + } + + @Override + public void channelInactive(ChannelHandlerContext ctx) throws Exception { + ctx.close(); + super.channelInactive(ctx); + } + +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java index 21488acba6910f019869940523bdfda0eae2b0a3..4ec40fc0280e581f7b95a63461020b8e909d96ba 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/CommonUtil.java @@ -54,10 +54,6 @@ public class CommonUtil { } - public static String keyName(String k) { - int index = k.lastIndexOf("/"); - return k.substring(index + 1); - } public static String encoder(String text) { @@ -81,49 +77,6 @@ public class CommonUtil { } - /** - * 拼装数据 - * - * @param list list-data - * @param startTime 开始时间 - * @param size 格子数 - * @param type 类型 1分钟 2小时 - * @return vo - */ - public static HotKeyLineChartVo assembleData(List list, LocalDateTime startTime, int size, int type) { - Set set = new TreeSet<>(); - boolean isHour = type == 1; - String suffix = isHour ? "60" : "24"; - String pattern = isHour ? DateUtil.PATTERN_MINUS : DateUtil.PATTERN_HOUR; - Map map = new HashMap<>(10); - Map> listMap = listGroup(list); -// log.info("按照rule分组以后的listMap--> {}", JSON.toJSONString(listMap)); - for (Map.Entry> m : listMap.entrySet()) { - int start = DateUtil.reviseTime(startTime, 0, type); - map.put(m.getKey(), new int[size]); - int[] data = map.get(m.getKey()); - int tmp = 0; - for (int i = 0; i < size; i++) { - if (String.valueOf(start).endsWith(suffix)) { - LocalDateTime tmpTime = DateUtil.strToLdt((start - 1) + "", pattern); - start = DateUtil.reviseTime(tmpTime, 1, type); - } -// log.info("start--> {}, tmp---> {} ", start, tmp); - set.add(DateUtil.strToLdt(start + "", pattern).toString().replace("T", " ")); - Statistics st = m.getValue().get(tmp); - int val = isHour ? st.getMinutes() : st.getHours(); - if (start != val) { - data[i] = 0; - } else { - tmp++; - data[i] = st.getCount(); - } - start++; - } - } - return new HotKeyLineChartVo(new ArrayList<>(set), map); - } - /** * 分组 @@ -132,9 +85,7 @@ public class CommonUtil { * @return map */ private static Map> listGroup(List list) { - if (Constant.VERSION != 1) { - return list.stream().collect(Collectors.groupingBy(Statistics::getRule)); - } + // return list.stream().collect(Collectors.groupingBy(Statistics::getRule)); return list.stream().collect(Collectors.groupingBy(Statistics::getKeyName)); } diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java new file mode 100644 index 0000000000000000000000000000000000000000..cd4a9872edd3f4e64af16860b707aabcf41e2ba5 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/PageUtil.java @@ -0,0 +1,36 @@ +package com.jd.platform.hotkey.dashboard.util; + +import com.google.common.collect.Lists; +import com.jd.platform.hotkey.dashboard.common.domain.Page; +import org.springframework.util.CollectionUtils; + +import java.util.ArrayList; +import java.util.List; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-09-01 + */ +public class PageUtil { + /** + * 通用分页工具类 + * + * @param data + * @param pageSize + * @param pageNum + * @param + * @return + */ + public static Page pagination(final List data, final int pageSize, final int pageNum) { + if (CollectionUtils.isEmpty(data)) { + return new Page<>(1, 0, new ArrayList<>()); + } + List> lists = Lists.partition(data, pageSize); + int localPageNum = pageNum; + if (localPageNum >= lists.size()) { + localPageNum = lists.size() - 1; + } + return new Page<>(localPageNum, data.size(), lists.get(localPageNum)); + } +} diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java index 40d82ff98dbd20e3ffc8994900e679c40abb5a50..d9352dc4595b47c048b4503fb7b51edf71670ea8 100644 --- a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/util/RuleUtil.java @@ -33,7 +33,7 @@ public class RuleUtil { } /** - * 根据APP的key，获取该key对应的rule + * 根据APP的key，获取该key对应的rule.如 cartpc-pu__ */ public static String rule(String key) { try { @@ -62,7 +62,7 @@ public class RuleUtil { return ""; } - private static KeyRule findByKey(String appNameKey) { + public static KeyRule findByKey(String appNameKey) { synchronized (RULE_MAP) { if (StrUtil.isEmpty(appNameKey)) { return null; diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/AccessTokenTask.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/AccessTokenTask.java new file mode 100644 index 0000000000000000000000000000000000000000..6b40e8b217b7d321a6fd5097c5d89ffccfc337a0 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/AccessTokenTask.java @@ -0,0 +1,31 @@ +package com.jd.platform.hotkey.dashboard.warn.dongdong; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.scheduling.annotation.Scheduled; +import org.springframework.stereotype.Component; + +import javax.annotation.Resource; + +/** + * @author liyunfeng31 + */ +@Component("accessTokenTask") +public class AccessTokenTask { + + @Resource + private DongDongApiManager apiManager; + + + private Logger log = LoggerFactory.getLogger(getClass()); + + + @Scheduled(cron = "0 0 0/1 ? * *") + public void updateAccessToken(){ + try { + apiManager.refreshAccessSignature(); + }catch (Exception e){ + log.error("refreshAccessSignature error:",e); + } + } +} \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/DongDongApiManager.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/DongDongApiManager.java new file mode 100644 index 0000000000000000000000000000000000000000..ad7ffbca4cc221ff67eeff0999a69fb1356876c2 --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/DongDongApiManager.java @@ -0,0 +1,134 @@ +package com.jd.platform.hotkey.dashboard.warn.dongdong; + +import com.github.rholder.retry.*; +import com.jd.platform.hotkey.dashboard.biz.service.IBizAccessTokenService; +import com.jd.platform.hotkey.dashboard.model.BizAccessToken; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.stereotype.Service; +import org.thymeleaf.util.StringUtils; + +import javax.annotation.Resource; +import java.util.Date; +import java.util.List; +import java.util.concurrent.Callable; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.TimeUnit; + +/** + * @author lihongliang32 + * @date 2020/5/20 5:53 下午 + */ +@Service +public class DongDongApiManager { + + @Value("${dongdong.aspId:110.200.0000002.266}") + private String aspId; + + @Value("${dongdong.secret:e2b05aa43959ddb6d37bd056b0243bc9}") + private String secret; + + @Value("${dongdong.noticeId:~hotkey}") + private String noticeId; + + @Resource + private DongDongUtil dongUtil; + + + private Logger log = LoggerFactory.getLogger(getClass()); + + + private static final String VERSION="4.3"; + + @Autowired + private IBizAccessTokenService accessTokenService; + + public boolean refreshAccessSignature() { + Callable callable = () -> { + try { + String accessToken = dongUtil.grant(); + List tokens = accessTokenService.selectBizAccessTokenList(new BizAccessToken()); + if(tokens.size()>0){ + BizAccessToken token = tokens.get(0); + token.setToken(accessToken); + token.setUpdatedBy("system"); + token.setUpdatedTime(new Date()); + log.info("updateBizAccessToken={}", accessToken); + accessTokenService.updateBizAccessToken(token); + }else{ + BizAccessToken token=new BizAccessToken(); + token.setToken(accessToken); + token.setCreatedBy("system"); + token.setCreatedTime(new Date()); + log.info("insertBizAccessToken={}", accessToken); + accessTokenService.insertBizAccessToken(token); + } + return true; + }catch (Exception e){ + return false; + } + }; + boolean result = false; + Retryer retryer = RetryerBuilder.newBuilder() + .retryIfRuntimeException() + .withWaitStrategy(WaitStrategies.fibonacciWait(100, 2, TimeUnit.MINUTES)) + .withStopStrategy(StopStrategies.stopAfterAttempt(5)) + .build(); + try { + result = retryer.call(callable); + } catch (RetryException | ExecutionException e) { + log.error("DongdongOpenAPIManagerImpl::refreshAccessSignature Exception!", e); + } + return result; + } + + + public boolean push(String title, String content) { + List tokens = accessTokenService.selectBizAccessTokenList(new BizAccessToken()); + if(tokens.size() == 0){ + log.error("没有查询到有效的token！"); + return false; + } + String accessToken = tokens.get(0).getToken(); + if(StringUtils.isEmpty(accessToken)){ + return false; + } + try { + return dongUtil.push(title,content,accessToken); + }catch (Exception e){ + return false; + } + } + + + + public String getAspId() { + return aspId; + } + + public void setAspId(String aspId) { + this.aspId = aspId; + } + + public String getSecret() { + return secret; + } + + public void setSecret(String secret) { + this.secret = secret; + } + + public String getNoticeId() { + return noticeId; + } + + public void setNoticeId(String noticeId) { + this.noticeId = noticeId; + } + + public static String getVERSION() { + return VERSION; + } +} \ No newline at end of file diff --git a/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/DongDongUtil.java b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/DongDongUtil.java new file mode 100644 index 0000000000000000000000000000000000000000..a7569237d0746ce3883a22a27ef8d45409fd012b --- /dev/null +++ b/dashboard/src/main/java/com/jd/platform/hotkey/dashboard/warn/dongdong/DongDongUtil.java @@ -0,0 +1,114 @@ +package com.jd.platform.hotkey.dashboard.warn.dongdong; + +import cn.hutool.json.JSONObject; +import com.alibaba.fastjson.JSON; +import com.jd.dd.open.gw.api.domain.AccessSignatureResult; +import com.jd.dd.open.gw.api.domain.MessagePushResult; +import org.apache.http.NameValuePair; +import org.apache.http.client.entity.UrlEncodedFormEntity; +import org.apache.http.client.methods.CloseableHttpResponse; +import org.apache.http.client.methods.HttpPost; +import org.apache.http.impl.client.CloseableHttpClient; +import org.apache.http.impl.client.HttpClients; +import org.apache.http.message.BasicNameValuePair; +import org.apache.http.util.EntityUtils; +import org.springframework.beans.factory.annotation.Value; +import org.springframework.stereotype.Component; + +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; + +/** + * @author liyunfeng31 + */ +@Component +public class DongDongUtil { + + @Value("${dongdong.aspId:110.200.0000002.266}") + private String aspId; + + @Value("${dongdong.secret:e2b05aa43959ddb6d37bd056b0243bc9}") + private String secret; + + @Value("${dongdong.noticeId:~hotkey}") + private String noticeId; + + + private final String VERSION="4.3"; + + + public String grant() throws Exception{ + String dogUrl = "http://dog-ee.jd.com/grant"; + List params = new ArrayList<>(); + params.add(new BasicNameValuePair("aspid", aspId)); + params.add(new BasicNameValuePair("secret", secret)); + params.add(new BasicNameValuePair("version", VERSION)); + + CloseableHttpClient httpclient = HttpClients.createDefault(); + HttpPost httpPost = new HttpPost(dogUrl); + //传参 + httpPost.setEntity(new UrlEncodedFormEntity(params)); + + CloseableHttpResponse response = httpclient.execute(httpPost); + String res = EntityUtils.toString(response.getEntity()); + + //返回结果对象AccessSignatureResult + AccessSignatureResult result = JSON.parseObject(res, AccessSignatureResult.class); + if (result.getCode() == 230031) { + //请求成功 + return result.getAccessToken(); + } + else { + //失败操作 + return String.valueOf(result.getCode()); + } + + } + + public boolean push(String title,String content,String accessToken) throws Exception{ + String dogUrl = "http://dog-ee.jd.com/push"; + List params = new ArrayList<>(); + params.add(new BasicNameValuePair("accessid", UUID.randomUUID().toString())); + params.add(new BasicNameValuePair("accessToken", accessToken)); + params.add(new BasicNameValuePair("aspid", aspId));//需要申请 + params.add(new BasicNameValuePair("version", VERSION)); + params.add(new BasicNameValuePair("timestamp", String.valueOf(System.currentTimeMillis()))); + + //消息相关参数 + JSONObject json = new JSONObject(); + json.put("type", "notice_message"); + json.put("ver", VERSION); + json.put("title", title); + json.put("content", content); + json.put("noticeId", noticeId); + json.put("toTerminal", 7); + json.put("sla", 0); + json.put("app", "ee"); + //接收方 + List pins = new ArrayList<>(); + pins.add("wuweifeng10"); + pins.add("liyunfeng31"); + pins.add("erp"); + json.put("tos", pins); + + JSONObject extend = new JSONObject(); + extend.put("url", "http://hotkey.jd.com"); + extend.put("pic", "https://img14.360buyimg.com/imagetools/jfs/t1/144156/7/8194/7263/5f5b34baEdfd90a49/a15144ab447bfbf2.png"); + json.put("extend", extend); + params.add(new BasicNameValuePair("jsonMsg",JSON.toJSONString(json))); + + CloseableHttpClient httpclient = HttpClients.createDefault(); + HttpPost httpPost = new HttpPost(dogUrl); + //传参 + httpPost.setEntity(new UrlEncodedFormEntity(params)); + httpPost.setEntity(new UrlEncodedFormEntity(params,"UTF-8")); + CloseableHttpResponse response = httpclient.execute(httpPost); + String res = EntityUtils.toString(response.getEntity()); + System.out.println(""+ JSON.toJSONString(res)); + //返回结果对象AccessSignatureResult + MessagePushResult result = JSON.parseObject(res, MessagePushResult.class); + return result.getCode() == 230070; + } + +} diff --git a/dashboard/src/main/resources/application.yml b/dashboard/src/main/resources/application.yml index 1678e74c5fcc2b262ffd71306c084db0ae54475b..40115d84f7bfc96088d9f033ae57d74ac25c1d5f 100644 --- a/dashboard/src/main/resources/application.yml +++ b/dashboard/src/main/resources/application.yml @@ -1,5 +1,5 @@ server : - port : 8080 + port : 8081 servlet : context-path : / spring : @@ -27,9 +27,9 @@ spring : time-zone: GMT+8 date-format: yyyy-MM-dd HH:mm:ss datasource: - username: root - password: password - url: jdbc:mysql://localhost:3306/hotkey?useUnicode=true&characterEncoding=utf-8&useSSL=true&serverTimezone=UTC + username: zxc + password: zxc4851225 + url: jdbc:mysql://rm-bp1f55o24g3im17w8yo.mysql.rds.aliyuncs.com:3306/luck?useUnicode=true&characterEncoding=utf-8&useSSL=true&serverTimezone=UTC&useTimezone=true&serverTimezone=GMT driver-class-name: com.mysql.cj.jdbc.Driver pagehelper: helperDialect: mysql @@ -50,3 +50,12 @@ erp: excludePath: /login,/assets,/bootstarp,/common,/htmlformat,/images,/jn,/jquery,/user/login,login ssoAppUrl: http://ssa.jd.com defaultPwd: 123456 + +jsf: + dongdong: + alias: ee00 + token: timline.jd.com + timeout: 5000 + aspId: 110.200.0000002.266 + secret: e2b05aa43959ddb6d37bd056b0243bc9 + noticeId: ~hotkey diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/BizAccessTokenMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/BizAccessTokenMapper.xml new file mode 100644 index 0000000000000000000000000000000000000000..07bf818e5a3cff1480e332675c7b2cf2f11ed4d5 --- /dev/null +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/BizAccessTokenMapper.xml @@ -0,0 +1,73 @@ + + + + + + + + + + + + + + + + + select id, token, flag, CREATED_BY, CREATED_TIME, UPDATED_BY, UPDATED_TIME from biz_access_token + + + + + + + + insert into biz_access_token + + token, + flag, + CREATED_BY, + CREATED_TIME, + UPDATED_BY, + UPDATED_TIME, + + + #{token}, + #{flag}, + #{createdBy}, + #{createdTime}, + #{updatedBy}, + #{updatedTime}, + + + + + update biz_access_token + + token = #{token}, + flag = #{flag}, + CREATED_BY = #{createdBy}, + CREATED_TIME = #{createdTime}, + UPDATED_BY = #{updatedBy}, + UPDATED_TIME = #{updatedTime}, + + where id = #{id} + + + + delete from biz_access_token where id = #{id} + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.xml similarity index 97% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.xml index 90b3b5d29876ba5ffe387e54d46d20eb57958649..9654fee8a750fbe2a2d2c3508b819097477e3ff1 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/ChangeLogMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/ChangeLogMapper.xml @@ -1,6 +1,6 @@ - + @@ -16,7 +16,7 @@ id, biz_id, biz_type, from_str, to_str, app_name,update_user, create_time,uuid + + delete from hk_key_record + where app_name = #{app} + + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.xml similarity index 95% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.xml index 505fa979cb3f8328cd63d67853eee6f0eb22f062..568633131087c9635f0f86075ee39ba03ed4424e 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/RulesMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/RulesMapper.xml @@ -1,6 +1,6 @@ - + diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.xml similarity index 93% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.xml index 8829d120b9683c00e1195445cda20d2570a7b53a..c409f64a38cecc04069c8c262dd86b03040bddaa 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/StatisticsMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/StatisticsMapper.xml @@ -1,6 +1,6 @@ - + @@ -74,8 +74,9 @@ - + where app = #{app} + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.xml similarity index 91% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.xml index 60f41a28ea0779691f0780acb55eafd5bfcfd15c..de74e5040cd6bd7f11b009effa7a8d816193b161 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/SummaryMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/SummaryMapper.xml @@ -1,6 +1,6 @@ - + @@ -51,4 +51,11 @@ + + delete from hk_summary + where app = #{app} + + + + \ No newline at end of file diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.xml similarity index 98% rename from dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml rename to dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.xml index bacdd874ef3ccfbfafa31276edfab19a894c7bb1..5ce765d88eb67cbdcf95971e977e87b1b4d45b74 100644 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/UserMapper.xml +++ b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/biz/mapper/UserMapper.xml @@ -1,6 +1,6 @@ - + @@ -16,7 +16,7 @@ id, nick_name, user_name, pwd, phone, role, app_name, create_time, state - select + select from hk_worker where id = #{id,jdbcType=INTEGER} diff --git a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.xml b/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.xml deleted file mode 100644 index 04704cf38f60d6daae9f23ec4c03ed385c63881f..0000000000000000000000000000000000000000 --- a/dashboard/src/main/resources/com/jd/platform/hotkey/dashboard/mapper/KeyTimelyMapper.xml +++ /dev/null @@ -1,207 +0,0 @@ - - - - - - - - - - - - - - id, key_name, val, uuid, app_name, duration, create_time - - - - - delete from hk_key_timely - where key_name = #{key} and app_name = #{appName} - - - - - delete from hk_key_timely - where (app_name,key_name) in - - (#{item.appName},#{item.key}) - - - - - delete from hk_key_timely where 1 = 1 - - - - - INSERT ignore INTO - hk_key_timely( - key_name, - val, - app_name, - uuid, - duration, - create_time) - VALUES - - (#{k.key}, - #{k.val}, - #{k.appName}, - #{k.uuid}, - #{k.duration}, - #{k.createTime}) - - - - - insert into hk_key_timely - - - id, - - - key_name, - - - val, - - - uuid, - - - app_name, - - - duration, - - - create_time, - - - - - #{id,jdbcType=BIGINT}, - - - #{key,jdbcType=VARCHAR}, - - - #{val,jdbcType=VARCHAR}, - - - #{uuid,jdbcType=VARCHAR}, - - - #{appName,jdbcType=VARCHAR}, - - - #{duration,jdbcType=BIGINT}, - - - #{createTime,jdbcType=TIMESTAMP}, - - - - - - update hk_key_timely - - - key_name = #{key,jdbcType=VARCHAR}, - - - val = #{val,jdbcType=VARCHAR}, - - - uuid = #{uuid,jdbcType=VARCHAR}, - - - app_name = #{appName,jdbcType=VARCHAR}, - - - duration = #{duration,jdbcType=BIGINT}, - - - create_time = #{createTime,jdbcType=TIMESTAMP}, - - - where key_name = #{key} - - - - - - - - insert into hk_key_timely - - - key_name, - - - val, - - - uuid, - - - app_name, - - - duration, - - - create_time, - - - - - #{key,jdbcType=VARCHAR}, - - - #{val,jdbcType=VARCHAR}, - - - #{uuid,jdbcType=VARCHAR}, - - - #{appName,jdbcType=VARCHAR}, - - - #{duration,jdbcType=BIGINT}, - - - #{createTime,jdbcType=TIMESTAMP}, - - - on duplicate key update - create_time = now() - - - \ No newline at end of file diff --git a/dashboard/src/main/resources/db.sql b/dashboard/src/main/resources/db.sql index 06e6b0c4d8442092708ef8ae15666fd545131932..7497750b4cd707f29eed5a1b364cc65c62c4d9aa 100644 --- a/dashboard/src/main/resources/db.sql +++ b/dashboard/src/main/resources/db.sql @@ -1,3 +1,8 @@ +-- !!! 注意设置sql model 否则可能sql报错 ！！！ +-- 查询你的sql_model参数：select @@global.sql_mode; 发现ONLY_FULL_GROUP_BY 则会导致报错 +-- 解决方式：set @@global.sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION' +-- 详情查阅：https://www.cnblogs.com/hjhsblogs/p/11079356.html + DROP TABLE IF EXISTS `hk_change_log`; CREATE TABLE `hk_change_log` ( `id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键', @@ -31,6 +36,11 @@ CREATE TABLE `hk_user` ( ) ENGINE = InnoDB AUTO_INCREMENT = 2 CHARACTER SET = utf8 COLLATE = utf8_bin ROW_FORMAT = Compact; +-- pwd: 123456 +INSERT INTO `hk_user` VALUES (2, 'admin', 'admin', 'e10adc3949ba59abbe56e057f20f883e', '1888888', 'ADMIN', '', '2020-07-28 14:01:03', 1); + + + DROP TABLE IF EXISTS `hk_key_record`; CREATE TABLE `hk_key_record` ( `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', @@ -115,3 +125,19 @@ CREATE TABLE `hk_summary` ( ) ENGINE = InnoDB AUTO_INCREMENT = 18 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin COMMENT = '汇总表' ROW_FORMAT = Compact; + +/* 请确认以下SQL符合您的变更需求，务必确认无误后再提交执行 */ + +CREATE TABLE `biz_access_token` ( + `id` bigint(20) NOT NULL AUTO_INCREMENT COMMENT '主键', + `token` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '' COMMENT 'token', + `flag` varchar(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NOT NULL DEFAULT '' COMMENT 'flag', + `CREATED_BY` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL COMMENT '创建人', + `CREATED_TIME` datetime NULL ON UPDATE CURRENT_TIMESTAMP COMMENT '创建时间', + `UPDATED_BY` varchar(128) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_ai_ci NULL COMMENT '修改人', + `UPDATED_TIME` datetime NULL ON UPDATE CURRENT_TIMESTAMP COMMENT '修改时间', + PRIMARY KEY (`id`) +) ENGINE = InnoDB AUTO_INCREMENT = 1 CHARACTER SET = utf8mb4 COLLATE = utf8mb4_bin COMMENT = '咚咚token表' ROW_FORMAT = Compact; + + + diff --git a/dashboard/src/main/resources/jsf-consumer.xml b/dashboard/src/main/resources/jsf-consumer.xml new file mode 100644 index 0000000000000000000000000000000000000000..acf8a7b39d76fb9a9d21c7f18ebd5fd5ec01d96f --- /dev/null +++ b/dashboard/src/main/resources/jsf-consumer.xml @@ -0,0 +1,13 @@ + + + + + + + \ No newline at end of file diff --git a/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js b/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js index 05eb56eb7f5ac72a9d28e8518840d0971a752788..13c97e45ebeae4d237457aab13b410054cafa62d 100644 --- a/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js +++ b/dashboard/src/main/resources/static/admin/assets/js/bootstrap/js/base_list.js @@ -40,20 +40,13 @@ ajaxOptions:{ headers: {"Authorization": getCookie("token")} }, - exportOptions:{ - ignoreColumn: [0,1], //忽略某一列的索引 - fileName: '报表导出', //文件名称设置 - worksheetName: 'sheet1', //表格工作区名称 - tableName: '报表导出', - excelstyles: ['background-color', 'color', 'font-size', 'font-weight'] - }, onLoadSuccess: function(){ //加载成功时执行 var datas = $('#dataTable').bootstrapTable('getData'); console.info("加载成功"); }, onLoadError: function(status){ //加载失败时执行 let token = getCookie("token"); - if(status === 500 && ( token == "undefined" || token =="")){ + if(status === 1000 && ( token == "undefined" || token =="")){ top.location.href = '/user/login'; } console.info("加载数据失败"); @@ -70,7 +63,15 @@ if(app!==undefined && app != null && app!==""){ search.app = app; } - return search; + let time1 = $("#startTime").val(); + if(time1!==undefined && time1 != null && time1!==""){ + search.startTime = time1; + } + let time2 = $("#endTime").val(); + if(time2!==undefined && time2 != null && time2!==""){ + search.endTime = time2; + } + return search; }, search:function(my){//查询条件 diff --git a/dashboard/src/main/resources/static/admin/common/js/appcfg-edit.js b/dashboard/src/main/resources/static/admin/common/js/appcfg-edit.js new file mode 100644 index 0000000000000000000000000000000000000000..f2ba469069467bb157001e6f0ddedffa5ab7cf0a --- /dev/null +++ b/dashboard/src/main/resources/static/admin/common/js/appcfg-edit.js @@ -0,0 +1,25 @@ +$("#form-edit").validate({ + submitHandler:function(form){ + edit(); + } +}); + +function edit() { + let dataFormJson=$("#form-edit").serialize(); + $.ajax({ + cache : true, + type : "POST", + url : "/appCfg/save", + data : dataFormJson, + headers: { + "Authorization":getCookie("token") + }, + async : false, + error : function(XMLHttpRequest){ + $.modal.alertError(XMLHttpRequest.responseJSON.msg); + }, + success : function(data) { + $.operate.saveSuccess(data); + } + }); +} diff --git a/dashboard/src/main/resources/static/admin/common/js/clearcfg-edit.js b/dashboard/src/main/resources/static/admin/common/js/clearcfg-edit.js new file mode 100644 index 0000000000000000000000000000000000000000..0d8de56a7dfb159200148fa225691269d0e574bc --- /dev/null +++ b/dashboard/src/main/resources/static/admin/common/js/clearcfg-edit.js @@ -0,0 +1,25 @@ +$("#form-edit").validate({ + submitHandler:function(form){ + edit(); + } +}); + +function edit() { + let dataFormJson=$("#form-edit").serialize(); + $.ajax({ + cache : true, + type : "POST", + url : "/clear/save", + data : dataFormJson, + headers: { + "Authorization":getCookie("token") + }, + async : false, + error : function(XMLHttpRequest){ + $.modal.alertError(XMLHttpRequest.responseJSON.msg); + }, + success : function(data) { + $.operate.saveSuccess(data); + } + }); +} diff --git a/dashboard/src/main/resources/templates/admin/appcfg/edit.html b/dashboard/src/main/resources/templates/admin/appcfg/edit.html new file mode 100644 index 0000000000000000000000000000000000000000..d636f2927aa6f4156d95e91c8a9d1e92f0ea3a76 --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/appcfg/edit.html @@ -0,0 +1,95 @@ + + + + + + + + + + + + + + + + + +
+
+
+
+
+
+
+
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+ +
+ +
+
+
+
+ + +
+
+
+
+
+
+
+
+
+ + +
+ +
+ + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/appcfg/list.html b/dashboard/src/main/resources/templates/admin/appcfg/list.html new file mode 100644 index 0000000000000000000000000000000000000000..c26b982ce7e3b8c69b3c2b7f8e456bdb50ab149b --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/appcfg/list.html @@ -0,0 +1,167 @@ + + + + + + + + + + + + + + + +
+
+ +
+ +
+
+ +
+
+
搜索条件
+
+ +
+
+
+
+ + +
+ +
+
+
+
+ + +
+
+
表单
+
+ + + +
+
+ + + +
+
+ +
+ +
+
+
+
+ +
+ +
+
+
+ + + +
+ +
+ + + + + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/changeLog/list.html b/dashboard/src/main/resources/templates/admin/changeLog/list.html index a6fbf8dec7ad1e188a58550a8bc03b2a912f605f..d6ed97aaea1e10b9a00fb023cc82834083b8bf82 100644 --- a/dashboard/src/main/resources/templates/admin/changeLog/list.html +++ b/dashboard/src/main/resources/templates/admin/changeLog/list.html @@ -142,6 +142,11 @@ var options = { } } }, + { field: 'createTime', title: '创建时间', + formatter:function (val,row,index) { + return changeDateFormat(val); + } + }, { field: 'from', title: '初始值', diff --git a/dashboard/src/main/resources/templates/admin/clear/edit.html b/dashboard/src/main/resources/templates/admin/clear/edit.html new file mode 100644 index 0000000000000000000000000000000000000000..dd2b4d8bf9fab5d5bd40c3060420f9a7b6c68f33 --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/clear/edit.html @@ -0,0 +1,69 @@ + + + + + + + + + + + + + + + + + +
+
+
+
+
+
+
+
+
+
+ +
+ +
+
+
+ +
+ +
+
+ +
+
+ + +
+
+
+
+
+
+
+
+
+ + +
+ +
+ + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/clear/list.html b/dashboard/src/main/resources/templates/admin/clear/list.html new file mode 100644 index 0000000000000000000000000000000000000000..7087f7929f522c6402334f7b213f6b481ec11d4e --- /dev/null +++ b/dashboard/src/main/resources/templates/admin/clear/list.html @@ -0,0 +1,157 @@ + + + + + + + + + + + + + + + +
+
+ +
+ +
+
+ +
+
+
搜索条件
+
+ +
+
+
+
+ + +
+ +
+
+
+
+ + +
+
+
表单
+
+ + + +
+
+ + + +
+
+ +
+ +
+
+
+
+ +
+ +
+
+
+ + + +
+ +
+ + + + + + + + + + + + + + diff --git a/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html b/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html index 29baffe1bc32bf1bf3941635d9002821c4a7e5e2..4764768f9438a1efc4c93a1842c419ca16f74269 100644 --- a/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html +++ b/dashboard/src/main/resources/templates/admin/common/html/leftMenu.html @@ -28,7 +28,7 @@ - +
  • @@ -52,7 +52,7 @@ - + 热点记录 变更记录 + + + 应用配置 + diff --git a/dashboard/src/main/resources/templates/admin/key/list.html b/dashboard/src/main/resources/templates/admin/key/list.html index 38c596fed6ed5bc2470fce25acc6f2ecd51d2db3..6f755c8df6d08f32cb3166cbab2f6b9043d4a861 100644 --- a/dashboard/src/main/resources/templates/admin/key/list.html +++ b/dashboard/src/main/resources/templates/admin/key/list.html @@ -163,7 +163,7 @@ var options = { { field: 'key', title: 'key'}, { field: 'ruleDesc', title: '类型'}, { field: 'appName', title: '所属APP'}, - { field: 'duration', title: '缓存时间'}, + { field: 'duration', title: '剩余缓存时间'}, { field: 'source', title: '来源'}, { field: 'type', title: '操作类型', formatter:function (val) { @@ -174,8 +174,27 @@ var options = { { field: 'createTime', title: '创建时间' }] }; -$(function(){ + + $('.datetimepicker').datetimepicker({ + language: 'zh-CN', + CustomFormat: 'yyyy-mm-dd HH:ii:ss', + weekStart: 1, + todayBtn: 1, //显示当天按钮，点击则选择当天当天时间 + autoclose: 1, //选完时间自动关闭 + todayHighlight: 1, //当天时间高亮 + startView: 2, //从月视图开始，选天 + minView: 0, //提供选择分钟的视图 + forceParse: 0, + minuteStep: 1 //用于构建小时视图。就是最小的视图是每1分钟可选一次。是以分钟为单位的 + }); + let now = new Date(); + $("#datetimepicker2").datetimepicker("setDate", now); + let pre5hours = now.setHours(now.getHours() - 5); + $("#datetimepicker1").datetimepicker("setDate", new Date(pre5hours)); + + +$(function(){ $.ajax({ cache : true, type : "POST", @@ -206,24 +225,9 @@ $(function(){ } }); - var oTab=$.table.oTableInit(options); oTab.Init(); - $('.datetimepicker').datetimepicker({ - language: 'zh-CN', - CustomFormat: 'yyyy-mm-dd HH:ii:ss', - weekStart: 1, - todayBtn: 1, //显示当天按钮，点击则选择当天当天时间 - autoclose: 1, //选完时间自动关闭 - todayHighlight: 1, //当天时间高亮 - startView: 2, //从月视图开始，选天 - minView: 0, //提供选择分钟的视图 - forceParse: 0, - minuteStep: 1 //用于构建小时视图。就是最小的视图是每1分钟可选一次。是以分钟为单位的 -}); - - }) function exportHot() { diff --git a/dashboard/src/main/resources/templates/admin/key/listmaxhot.html b/dashboard/src/main/resources/templates/admin/key/listmaxhot.html index 366b156a8ca8bd646e94e74f308ef163b1ea545b..5d7fc2251485a4af1e0b933ee08845ab7daccd02 100644 --- a/dashboard/src/main/resources/templates/admin/key/listmaxhot.html +++ b/dashboard/src/main/resources/templates/admin/key/listmaxhot.html @@ -35,14 +35,14 @@
    - + - + diff --git a/dashboard/src/main/resources/templates/admin/key/listtimely.html b/dashboard/src/main/resources/templates/admin/key/listtimely.html index 37e85bef8eda8ee3b8dfd3aa5edfa372c1b4421a..545caa74a1bbee799ec926de7eb55fff2c0e7090 100644 --- a/dashboard/src/main/resources/templates/admin/key/listtimely.html +++ b/dashboard/src/main/resources/templates/admin/key/listtimely.html @@ -162,7 +162,7 @@ var options = { { field: 'key', title: 'key'}, { field: 'ruleDesc', title: '类型'}, { field: 'appName', title: '所属APP'}, - { field: 'duration', title: '缓存时间'}, + { field: 'duration', title: '剩余缓存时间'}, { field: 'createTime', title: '创建时间', formatter:function (val,row,index) { return changeDateFormat(val); diff --git a/dashboard/src/main/resources/templates/admin/main.html b/dashboard/src/main/resources/templates/admin/main.html index ee7b7fe35996f62693af50d34bbb3722ea41c2ba..75e714a6205403d0d24cdf26402f720e924d5e36 100644 --- a/dashboard/src/main/resources/templates/admin/main.html +++ b/dashboard/src/main/resources/templates/admin/main.html @@ -179,7 +179,7 @@ }, error: function(result){ //加载失败时执行 let token = getCookie("token"); - if(result.status == 500 && ( token == "undefined" || token =="")){ + if(result.status == 1000 && ( token == "undefined" || token =="")){ top.location.href = '/user/login'; } console.info("加载数据失败"); diff --git a/dashboard/src/main/resources/templates/admin/rule/listhitcount.html b/dashboard/src/main/resources/templates/admin/rule/listhitcount.html index 7e501fa6e9f07aaa848e4a2e382c51ba0dcd501f..f1be6e38782e3cc2fbf2f58d7d2feac48053b3e1 100644 --- a/dashboard/src/main/resources/templates/admin/rule/listhitcount.html +++ b/dashboard/src/main/resources/templates/admin/rule/listhitcount.html @@ -46,7 +46,7 @@
    - + @@ -55,7 +55,7 @@
    - + @@ -134,6 +134,27 @@ } }] }; + + + + $('.datetimepicker').datetimepicker({ + language: 'zh-CN', + CustomFormat: 'yyyy-mm-dd HH:ii:ss', + weekStart: 1, + todayBtn: 1, //显示当天按钮，点击则选择当天当天时间 + autoclose: 1, //选完时间自动关闭 + todayHighlight: 1, //当天时间高亮 + startView: 2, //从月视图开始，选天 + minView: 0, //提供选择分钟的视图 + forceParse: 0, + minuteStep: 1 //用于构建小时视图。就是最小的视图是每1分钟可选一次。是以分钟为单位的 + }); + let now = new Date(); + $("#datetimepicker2").datetimepicker("setDate", now); + let pre5hours = now.setHours(now.getHours() - 5); + $("#datetimepicker1").datetimepicker("setDate", new Date(pre5hours)); + + $(function(){ $.ajax({ cache : true, @@ -164,26 +185,12 @@ } } }); - var oTab=$.table.oTableInit(options); + + let oTab=$.table.oTableInit(options); oTab.Init(); }) - - $('.datetimepicker').datetimepicker({ - language: 'zh-CN', - CustomFormat: 'yyyy-mm-dd HH:ii:ss', - weekStart: 1, - todayBtn: 1, //显示当天按钮，点击则选择当天当天时间 - autoclose: 1, //选完时间自动关闭 - todayHighlight: 1, //当天时间高亮 - startView: 2, //从月视图开始，选天 - minView: 0, //提供选择分钟的视图 - forceParse: 0, - minuteStep: 1 //用于构建小时视图。就是最小的视图是每1分钟可选一次。是以分钟为单位的 - }); - - diff --git a/sample/src/main/java/com/jd/platform/sample/Starter.java b/sample/src/main/java/com/jd/platform/sample/Starter.java index d63e3a0e1c459b5d5f953120c7d5dc6354f044fb..43baaaed628b1f9abeb20e8d44228d0881a5c816 100644 --- a/sample/src/main/java/com/jd/platform/sample/Starter.java +++ b/sample/src/main/java/com/jd/platform/sample/Starter.java @@ -2,7 +2,10 @@ package com.jd.platform.sample; import com.ibm.etcd.api.KeyValue; import com.jd.platform.hotkey.client.ClientStarter; +import com.jd.platform.hotkey.client.callback.JdHotKeyStore; import com.jd.platform.hotkey.common.configcenter.IConfigCenter; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; @@ -21,21 +24,52 @@ public class Starter { @Value("${spring.application.name}") private String appName; + @Value("${total.count}") + private int count; + @Value("${total.sleep}") + private int sleep; + @Value("${total.period}") + private long period; + @Resource private IConfigCenter iConfigCenter; - + private Logger logger = LoggerFactory.getLogger(getClass()); @PostConstruct public void init() { ClientStarter.Builder builder = new ClientStarter.Builder(); - ClientStarter starter = builder.setAppName(appName).setEtcdServer(etcd).build(); + ClientStarter starter = builder.setAppName(appName).setEtcdServer(etcd).setPushPeriod(period).build(); starter.startPipeline(); List list = iConfigCenter.getPrefix("/jd/workers/sample/host"); for (KeyValue keyValue : list) { System.out.println(keyValue.getKey() + keyValue.getValue().toStringUtf8()); } + + try { + Thread.sleep(3000); + } catch (InterruptedException e) { + e.printStackTrace(); + } + + new Thread(() -> { + long i1 = 0; + while (true) { + if (JdHotKeyStore.getValue(i1 + "") == null) { + i1++; + if (i1 % count == 0) { + try { + Thread.sleep(sleep); + logger.info("number: " + i1); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } + } + } + + }).start(); } } diff --git a/sample/src/main/java/com/jd/platform/sample/controller/TestController.java b/sample/src/main/java/com/jd/platform/sample/controller/TestController.java index bf643fb15318e1e8801050ea56e80fdc4d080535..806382291d5dcc013a4681aa59d8300f8189207e 100644 --- a/sample/src/main/java/com/jd/platform/sample/controller/TestController.java +++ b/sample/src/main/java/com/jd/platform/sample/controller/TestController.java @@ -76,8 +76,8 @@ public class TestController { // } @RequestMapping("") - public Object a() { - if (JdHotKeyStore.isHotKey("pin_tianyalei")) { + public Object a(String s) { + if (JdHotKeyStore.isHotKey("ecitemcenter_itemV2_" + s)) { logger.error("isHot"); } else { logger.error("noHot"); diff --git a/sample/src/main/resources/application.yml b/sample/src/main/resources/application.yml index 41fc686c89203651a719a44d41d278da0f9af4d2..a8e2a8c0fdb15fff4b066dcf8fc4e21f03c2c68f 100644 --- a/sample/src/main/resources/application.yml +++ b/sample/src/main/resources/application.yml @@ -1,11 +1,15 @@ #etcd的地址，如有多个用逗号分隔 etcd: -# server: ${etcdServer:https://127.0.0.1:2379} -# server: http://10.170.161.91:2379 - server: http://open-etcd.jd.com:2000 + server: ${etcdServer:https://127.0.0.1:2379} +# server: ${etcdServer:http://10.170.161.91:2379｝ +# server: http://open-etcd.jd.com:2000 +total: + count: ${count:40000} + sleep: ${sleep:1000} + period: ${period:10} spring: application: - name: sample + name: ${name:sample} ###############################---redis---############################## # redis: # host: ${REDIS_HOST:127.0.0.1} diff --git a/sample/src/test/java/com/jd/platform/sample/SampleApplicationTests.java b/sample/src/test/java/com/jd/platform/sample/SampleApplicationTests.java index 741fdca2dd6f9264b38287e55fab5cb0f36c57bf..1f7e12b48a71c56e93868849e05656ee55b25c7e 100644 --- a/sample/src/test/java/com/jd/platform/sample/SampleApplicationTests.java +++ b/sample/src/test/java/com/jd/platform/sample/SampleApplicationTests.java @@ -1,13 +1,20 @@ package com.jd.platform.sample; import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; import org.springframework.boot.test.context.SpringBootTest; @SpringBootTest class SampleApplicationTests { - + private Logger logger = LoggerFactory.getLogger(getClass()); @Test void contextLoads() { +// if (JdHotKeyStore.isHotKey("VENDER_ID_2111")) { +// logger.error("isHot"); +// } else { +// logger.error("noHot"); +// } } } diff --git a/worker/pom.xml b/worker/pom.xml index 5f234d9048657fefd24996fcded45ce50697e200..0d64df64d214f019181e772c8222d0ebbcef50be 100644 --- a/worker/pom.xml +++ b/worker/pom.xml @@ -19,18 +19,6 @@ - - io.protostuff - protostuff-core - 1.7.2 - - - - io.protostuff - protostuff-runtime - 1.7.2 - - com.jd.platform.hotkey common diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/cache/CaffeineCacheHolder.java b/worker/src/main/java/com/jd/platform/hotkey/worker/cache/CaffeineCacheHolder.java index ec8a185fc0f87d0e1390b19c45b44bc6438b9a31..d3801e8773891f1fdf49ead88b32bcc4f524d65c 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/cache/CaffeineCacheHolder.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/cache/CaffeineCacheHolder.java @@ -42,7 +42,6 @@ public class CaffeineCacheHolder { public static void clearCacheByAppName(String appName) { if(CACHE_MAP.get(appName) != null) { CACHE_MAP.get(appName).invalidateAll(); - CACHE_MAP.put(appName, null); } } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/cache/Test.java b/worker/src/main/java/com/jd/platform/hotkey/worker/cache/Test.java deleted file mode 100644 index bf2afa497606ebf2a062f149e92a29c33406197d..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/cache/Test.java +++ /dev/null @@ -1,64 +0,0 @@ -package com.jd.platform.hotkey.worker.cache; - -import com.github.benmanes.caffeine.cache.Cache; -import com.github.benmanes.caffeine.cache.Caffeine; -import org.springframework.stereotype.Component; - -import javax.annotation.Resource; -import java.util.HashMap; -import java.util.Map; -import java.util.concurrent.TimeUnit; - -/** - * @author wuweifeng - * @version 1.0 - * @date 2020-05-04 - */ -@Component -public class Test { - @Resource(name = "hotKeyCache") - private Cache hotCache; - - public static void main(String[] args) { - Map map = new HashMap(); -// map.put("a", "b"); -// System.out.println(map); - Cache cache = Caffeine.newBuilder() - .initialCapacity(1024)//初始大小 - .maximumSize(5000000)//最大数量 - .expireAfterWrite(5, TimeUnit.MINUTES)//过期时间 - .softValues() - .build(); - - cache.put("yuan2012555", 1); -// System.out.println(cache.getIfPresent("yuan2012555")); - - - System.out.println("lwymail163".hashCode() %4); - System.out.println("272551766_m".hashCode() %4); - System.out.println("hanxu123".hashCode() %4); - System.out.println("abc123ab".hashCode() %4); - - System.out.println("hanxu123hanxu".hashCode() %4); - System.out.println(Math.abs("lwy163mail".hashCode() %4)); - - } - -// @PostConstruct -// public void aa() throws InterruptedException { -// Executor executor = Executors.newCachedThreadPool(); -// Cache cache = Caffeine.newBuilder() -// .executor(executor) -// .initialCapacity(1024)//初始大小 -// .maximumSize(5000000)//最大数量 -// .expireAfterWrite(5, TimeUnit.MINUTES)//过期时间 -// .softValues() -// .build(); -// long i = 0; -// while (true) { -// cache.put(UUID.randomUUID().toString(), UUID.randomUUID().toString()); -//// hotCache.put("i" + i, i); -// i++; -// } -// } -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/config/CodecConfig.java b/worker/src/main/java/com/jd/platform/hotkey/worker/config/CodecConfig.java deleted file mode 100644 index 0d62c9097c400d330d513b854f43c32888848a73..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/config/CodecConfig.java +++ /dev/null @@ -1,20 +0,0 @@ -package com.jd.platform.hotkey.worker.config; - -import com.jd.platform.hotkey.common.coder.Codec; -import com.jd.platform.hotkey.common.coder.NettyCodec; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; - -/** - * netty的编解码器 - * @author wuweifeng wrote on 2020-02-21 - * @version 1.0 - */ -@Configuration -public class CodecConfig { - - @Bean - public Codec codec() { - return new NettyCodec(); - } -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/config/DisruptorConfig.java b/worker/src/main/java/com/jd/platform/hotkey/worker/config/DisruptorConfig.java deleted file mode 100644 index 0a92f8fd2058db1d3408836f4bd383931bfc42fd..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/config/DisruptorConfig.java +++ /dev/null @@ -1,33 +0,0 @@ -package com.jd.platform.hotkey.worker.config; - -import com.jd.platform.hotkey.worker.keylistener.IKeyListener; -import org.springframework.beans.factory.annotation.Value; -import org.springframework.context.annotation.Configuration; - -import javax.annotation.Resource; - -/** - * @author wuweifeng wrote on 2019-12-11 - * @version 1.0 - */ -@Configuration -public class DisruptorConfig { - @Resource - private IKeyListener iKeyListener; - - @Value("${open.timeout}") - private boolean openTimeOut; - @Value("${thread.count}") - private int threadCount; - -// @Bean -// public MessageProducer messageProducer() { -// InitConstant.openTimeOut = openTimeOut; -// //将实际值赋给static变量 -// InitConstant.threadCount = threadCount; -// -// return ProducerFactory.createHotKeyProducer(iKeyListener); -// } - - -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/AbsEventConsumer.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/AbsEventConsumer.java deleted file mode 100755 index 47839ae89ee541b85cd0c1337d32343f50633b94..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/AbsEventConsumer.java +++ /dev/null @@ -1,54 +0,0 @@ -//package com.jd.platform.hotkey.worker.disruptor; -// -//import cn.hutool.core.date.SystemClock; -//import com.jd.platform.hotkey.common.model.BaseModel; -//import com.jd.platform.hotkey.common.tool.Constant; -//import com.jd.platform.hotkey.worker.tool.InitConstant; -//import com.lmax.disruptor.EventHandler; -// -//import java.util.concurrent.atomic.LongAdder; -//import java.util.zip.CRC32; -// -///**各个消费者重复消费 -// * @author wuweifeng wrote on 2019-08-21. -// */ -//public abstract class AbsEventConsumer implements EventHandler { -// -// private int hashIndex; -// -// public static final LongAdder totalDealCount = new LongAdder(); -// //过期的 -// public static final LongAdder expireTotalCount = new LongAdder(); -// -// public AbsEventConsumer(int hashIndex) { -// this.hashIndex = hashIndex; -// } -// -// @Override -// public void onEvent(T t, long l, boolean b) { -// //每个消费者，只处理特定的key。保证相同的key，一定被同一个线程处理 -// BaseModel model = t.getModel(); -// if (model == null || model.getKey() == null) { -// return; -// } -// if (Math.abs(hash(model.getKey())) % Constant.Default_Threads == hashIndex) { -// //5秒前的过时消息就不处理了 -// if (SystemClock.now() - model.getCreateTime() > InitConstant.timeOut) { -// expireTotalCount.increment(); -// return; -// } -// onNewEvent(t); -// -// //处理完毕，将数量加1 -// totalDealCount.increment(); -// } -// } -// -// protected abstract void onNewEvent(T t); -// -// private long hash(String key) { -// CRC32 crc = new CRC32(); -// crc.update(key.getBytes()); -// return crc.getValue(); -// } -//} \ No newline at end of file diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/AbsWorkConsumer.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/AbsWorkConsumer.java deleted file mode 100755 index bca2e70f7677f1b5e20fd71b9468e54a71724450..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/AbsWorkConsumer.java +++ /dev/null @@ -1,54 +0,0 @@ -package com.jd.platform.hotkey.worker.disruptor; - -import cn.hutool.core.date.SystemClock; -import com.jd.platform.hotkey.common.model.BaseModel; -import com.jd.platform.hotkey.worker.tool.InitConstant; -import com.lmax.disruptor.WorkHandler; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.util.concurrent.atomic.LongAdder; - -/** - * 各个消费者不重复消费 - * - * @author wuweifeng wrote on 2019-08-21. - */ -public abstract class AbsWorkConsumer implements WorkHandler { - - private int hashIndex; - - public static final LongAdder totalDealCount = new LongAdder(); - //过期的 - public static final LongAdder expireTotalCount = new LongAdder(); - - private Logger logger = LoggerFactory.getLogger(getClass()); - - public AbsWorkConsumer(int hashIndex) { - this.hashIndex = hashIndex; - } - - @Override - public void onEvent(T t) { - BaseModel model = t.getModel(); - if (model == null || model.getKey() == null) { - return; - } - //5秒前的过时消息就不处理了 - if (SystemClock.now() - model.getCreateTime() > InitConstant.timeOut) { - expireTotalCount.increment(); - logger.warn("this key timeout : " + model.getKey() + " createAt : " + model.getCreateTime() + - " event at " + SystemClock.now()); - if (InitConstant.openTimeOut) { - return; - } - } - onNewEvent(t); - - //处理完毕，将数量加1 - totalDealCount.increment(); - } - - protected abstract void onNewEvent(T t); - -} \ No newline at end of file diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/BaseEvent.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/BaseEvent.java deleted file mode 100644 index e8483fe9a9680010221007477f23b8da765c8646..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/BaseEvent.java +++ /dev/null @@ -1,26 +0,0 @@ -package com.jd.platform.hotkey.worker.disruptor; - -import com.jd.platform.hotkey.common.model.BaseModel; - -/** - * @author wuweifeng wrote on 2019-12-10 - * @version 1.0 - */ -public class BaseEvent { - private T model; - - public BaseEvent(T model) { - this.model = model; - } - - public BaseEvent() { - } - - public T getModel() { - return model; - } - - public void setModel(T model) { - this.model = model; - } -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/DisruptorBuilder.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/DisruptorBuilder.java deleted file mode 100644 index 64dc4c075389e0e618b0a7319c9cad63057c0431..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/DisruptorBuilder.java +++ /dev/null @@ -1,65 +0,0 @@ -package com.jd.platform.hotkey.worker.disruptor; - -import com.lmax.disruptor.BlockingWaitStrategy; -import com.lmax.disruptor.EventFactory; -import com.lmax.disruptor.EventHandler; -import com.lmax.disruptor.WorkHandler; -import com.lmax.disruptor.dsl.Disruptor; -import com.lmax.disruptor.dsl.ProducerType; - -import java.util.concurrent.Executors; - -/** - * 每个worker重复消费生产者消息 - * - * @author wuweifeng wrote on 2019-12-10 - * @version 1.0 - */ -public class DisruptorBuilder { - private int bufferSize; - private EventHandler[] eventHandlers; - private WorkHandler[] workHandlers; - private EventFactory eventFactory; - - public Disruptor build() { - Disruptor disruptor = new Disruptor<>(eventFactory, bufferSize, Executors.defaultThreadFactory(), - ProducerType.SINGLE, new BlockingWaitStrategy()); - - if (eventHandlers != null) { - disruptor.handleEventsWith(eventHandlers); - } - if (workHandlers != null) { - disruptor.handleEventsWithWorkerPool(workHandlers); - } - - disruptor.start(); - return disruptor; - } - - /** - * 每个worker会重复消费 - */ - public DisruptorBuilder setEventHandlers(EventHandler... eventHandlers) { - this.eventHandlers = eventHandlers; - return this; - } - - /** - * 每个worker不会重复消费 - */ - public DisruptorBuilder setWorkerHandlers(WorkHandler... workHandlers) { - this.workHandlers = workHandlers; - return this; - } - - public DisruptorBuilder setEventFactory(EventFactory eventFactory) { - this.eventFactory = eventFactory; - return this; - } - - public DisruptorBuilder setBufferSize(int bufferSize) { - this.bufferSize = bufferSize; - return this; - } - -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/MessageProducer.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/MessageProducer.java deleted file mode 100755 index 63d142145f208956fa6ce354f511abfcb5008a54..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/MessageProducer.java +++ /dev/null @@ -1,8 +0,0 @@ -package com.jd.platform.hotkey.worker.disruptor; - -/** - * @author wuweifeng wrote on 2018/4/20. - */ -public interface MessageProducer { - void publish(T t); -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/ProducerFactory.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/ProducerFactory.java deleted file mode 100755 index 8854627d4a8c49c645fa5f315fe878e2b0214e00..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/ProducerFactory.java +++ /dev/null @@ -1,50 +0,0 @@ -package com.jd.platform.hotkey.worker.disruptor; - -import com.jd.platform.hotkey.worker.disruptor.hotkey.HotKeyEvent; -import com.jd.platform.hotkey.worker.disruptor.hotkey.HotKeyEventConsumer; -import com.jd.platform.hotkey.worker.disruptor.hotkey.HotKeyEventProducer; -import com.jd.platform.hotkey.worker.keylistener.IKeyListener; -import com.jd.platform.hotkey.worker.tool.CpuNum; -import com.jd.platform.hotkey.worker.tool.InitConstant; -import com.lmax.disruptor.dsl.Disruptor; - -/** - * Disruptor创建producer的地方 - * - * @author wuweifeng wrote on 2019-11-5. - */ -public class ProducerFactory { - - /** - * 创建热key的producer - * - * @return HotKeyEventProducer - */ - public static MessageProducer createHotKeyProducer(IKeyListener iKeyListener) { - int threadCount = CpuNum.workerCount(); - //如果手工指定了线程数，就用手工指定的 - if (InitConstant.threadCount != 0) { - threadCount = InitConstant.threadCount; - } else { - if (threadCount >= 8) { - threadCount = threadCount / 2; - } - } - - HotKeyEventConsumer[] array = new HotKeyEventConsumer[threadCount]; - for (int i = 0; i < threadCount; i++) { - array[i] = new HotKeyEventConsumer(i); - array[i].setKeyListener(iKeyListener); - } - DisruptorBuilder builder = new DisruptorBuilder<>(); - Disruptor disruptor = builder - .setBufferSize(InitConstant.bufferSize * 1024 * 1024) - .setEventFactory(HotKeyEvent::new) -// .setEventHandlers(array) //重复消费的 - .setWorkerHandlers(array) //不重复消费的 - .build(); - - return new HotKeyEventProducer(disruptor); - } - -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEvent.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEvent.java deleted file mode 100755 index 8ec41c3e1ab1dd804f0cc547f66758caa4e4274a..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEvent.java +++ /dev/null @@ -1,18 +0,0 @@ -package com.jd.platform.hotkey.worker.disruptor.hotkey; - -import com.jd.platform.hotkey.common.model.HotKeyModel; -import com.jd.platform.hotkey.worker.disruptor.BaseEvent; - -/** - * @author wuweifeng wrote on 2019-08-21. - */ -public class HotKeyEvent extends BaseEvent { - - public HotKeyEvent(HotKeyModel hotKeyModel) { - super(hotKeyModel); - } - - public HotKeyEvent() { - - } -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEventConsumer.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEventConsumer.java deleted file mode 100755 index ca248f587e4281e24bd1a0e7a2b243eb9ee608d8..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEventConsumer.java +++ /dev/null @@ -1,41 +0,0 @@ -package com.jd.platform.hotkey.worker.disruptor.hotkey; - -import com.jd.platform.hotkey.common.model.HotKeyModel; -import com.jd.platform.hotkey.worker.disruptor.AbsWorkConsumer; -import com.jd.platform.hotkey.worker.keylistener.IKeyListener; -import com.jd.platform.hotkey.worker.keylistener.KeyEventOriginal; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * 热key的消费者 - * @author wuweifeng wrote on 2019-12-12. - */ -public class HotKeyEventConsumer extends AbsWorkConsumer { - private Logger logger = LoggerFactory.getLogger(getClass()); - - private IKeyListener iKeyListener; - - public HotKeyEventConsumer(int hashIndex) { - super(hashIndex); - } - - @Override - protected void onNewEvent(HotKeyEvent hotKeyEvent) { - HotKeyModel model = hotKeyEvent.getModel(); - if (iKeyListener == null) { - logger.warn("new key is coming, but no consumer deal this key!"); - return; - } - - if (model.isRemove()) { - iKeyListener.removeKey(model, KeyEventOriginal.CLIENT); - } else { - iKeyListener.newKey(model, KeyEventOriginal.CLIENT); - } - } - - public void setKeyListener(IKeyListener iKeyListener) { - this.iKeyListener = iKeyListener; - } -} \ No newline at end of file diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEventProducer.java b/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEventProducer.java deleted file mode 100755 index 42dae5f65e899ad147bac1374b35832a227b883d..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/disruptor/hotkey/HotKeyEventProducer.java +++ /dev/null @@ -1,32 +0,0 @@ -package com.jd.platform.hotkey.worker.disruptor.hotkey; - -import com.jd.platform.hotkey.worker.disruptor.MessageProducer; -import com.lmax.disruptor.RingBuffer; -import com.lmax.disruptor.dsl.Disruptor; - -/** - * 所有节点发来的消息，都进入这里，然后publish出去，供消费者消费 - * - * @author wuweifeng wrote on 2019/11/6. - */ -public class HotKeyEventProducer implements MessageProducer { - - private Disruptor disruptor; - - public HotKeyEventProducer(Disruptor disruptor) { - this.disruptor = disruptor; - } - - @Override - public void publish(HotKeyEvent hotKeyEvent) { - RingBuffer ringBuffer = disruptor.getRingBuffer(); - long sequence = ringBuffer.next(); - try { - HotKeyEvent event = ringBuffer.get(sequence); - event.setModel(hotKeyEvent.getModel()); - } finally { - ringBuffer.publish(sequence); - } - } - -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/keydispatcher/DispatcherConfig.java b/worker/src/main/java/com/jd/platform/hotkey/worker/keydispatcher/DispatcherConfig.java index 3d620b1aec8161607d44953ec5cf5b8634047b72..7e3094ad9aa770e27e27a3d3829dc10ae615a374 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/keydispatcher/DispatcherConfig.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/keydispatcher/DispatcherConfig.java @@ -39,8 +39,12 @@ public class DispatcherConfig { public Consumer consumer() { int nowCount = CpuNum.workerCount(); //将实际值赋给static变量 - if(threadCount != 0) { + if (threadCount != 0) { nowCount = threadCount; + } else { + if (nowCount >= 8) { + nowCount = nowCount / 2; + } } List consumerList = new ArrayList<>(); diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/keydispatcher/KeyProducer.java b/worker/src/main/java/com/jd/platform/hotkey/worker/keydispatcher/KeyProducer.java index fc86ecc93d67543fec8446bd4b21968aa8130c0d..2f3320b3ead48beba0f69434f5cff977baffe345 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/keydispatcher/KeyProducer.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/keydispatcher/KeyProducer.java @@ -1,6 +1,5 @@ package com.jd.platform.hotkey.worker.keydispatcher; -import cn.hutool.core.date.SystemClock; import com.jd.platform.hotkey.common.model.HotKeyModel; import com.jd.platform.hotkey.worker.tool.InitConstant; import org.springframework.stereotype.Component; @@ -17,12 +16,12 @@ import static com.jd.platform.hotkey.worker.tool.InitConstant.totalOfferCount; @Component public class KeyProducer { - public void push(HotKeyModel model) { + public void push(HotKeyModel model, long now) { if (model == null || model.getKey() == null) { return; } //5秒前的过时消息就不处理了 - if (SystemClock.now() - model.getCreateTime() > InitConstant.timeOut) { + if (now - model.getCreateTime() > InitConstant.timeOut) { expireTotalCount.increment(); return; } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/keylistener/KeyListener.java b/worker/src/main/java/com/jd/platform/hotkey/worker/keylistener/KeyListener.java index e941eb6d22dfd4660c6a5ce39e62576d3c62fc5b..f9443940c480098e35fafaa002daef2ecbcb8d5a 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/keylistener/KeyListener.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/keylistener/KeyListener.java @@ -9,6 +9,7 @@ import com.jd.platform.hotkey.common.rule.KeyRule; import com.jd.platform.hotkey.worker.cache.CaffeineCacheHolder; import com.jd.platform.hotkey.worker.netty.pusher.IPusher; import com.jd.platform.hotkey.worker.rule.KeyRuleHolder; +import com.jd.platform.hotkey.worker.starters.EtcdStarter; import com.jd.platform.hotkey.worker.tool.SlidingWindow; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -75,7 +76,12 @@ public class KeyListener implements IKeyListener { //开启推送 hotKeyModel.setCreateTime(SystemClock.now()); - logger.info(NEW_KEY_EVENT + hotKeyModel.getKey()); + + //当开关打开时，打印日志。大促时关闭日志，就不打印了 + if (EtcdStarter.LOGGER_ON) { + logger.info(NEW_KEY_EVENT + hotKeyModel.getKey()); + } + //分别推送到各client和etcd for (IPusher pusher : iPushers) { pusher.push(hotKeyModel); diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/model/AppInfo.java b/worker/src/main/java/com/jd/platform/hotkey/worker/model/AppInfo.java index 4e106a3e8cec797ffbc4656ffba7b0201c6226a2..599edde417dff3fa4c1c182e77d11e2777a18842 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/model/AppInfo.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/model/AppInfo.java @@ -1,9 +1,9 @@ package com.jd.platform.hotkey.worker.model; import io.netty.channel.ChannelHandlerContext; - -import java.util.Map; -import java.util.concurrent.ConcurrentHashMap; +import io.netty.channel.group.ChannelGroup; +import io.netty.channel.group.DefaultChannelGroup; +import io.netty.util.concurrent.GlobalEventExecutor; /** * @author wuweifeng wrote on 2019-12-05 @@ -15,31 +15,33 @@ public class AppInfo { */ private String appName; /** - * 客户端ip 和 channel的映射关系 + * 某app的全部channel */ - private Map map = new ConcurrentHashMap<>(); - - @Override - public String toString() { - return "AppInfo{" + - "appName='" + appName + '\'' + - ", map=" + map + - '}'; + private ChannelGroup channelGroup; + + public AppInfo(String appName) { + this.appName = appName; + channelGroup = new DefaultChannelGroup(appName, GlobalEventExecutor.INSTANCE); } - public String getAppName() { - return appName; + public void groupPush(Object object) { + channelGroup.writeAndFlush(object); } - public void setAppName(String appName) { - this.appName = appName; + public void add(ChannelHandlerContext ctx) { + channelGroup.add(ctx.channel()); } - public Map getMap() { - return map; + public void remove(ChannelHandlerContext ctx) { + channelGroup.remove(ctx.channel()); } - public void setMap(Map map) { - this.map = map; + public String getAppName() { + return appName; } + + public int size() { + return channelGroup.size(); + } + } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/model/ClientInfo.java b/worker/src/main/java/com/jd/platform/hotkey/worker/model/ClientInfo.java deleted file mode 100644 index 081652b6f23bf9673448cdea32a71d2a38680f7c..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/model/ClientInfo.java +++ /dev/null @@ -1,62 +0,0 @@ -package com.jd.platform.hotkey.worker.model; - -import io.netty.channel.ChannelHandlerContext; - -/** - * @author wuweifeng wrote on 2019-12-05 - * @version 1.0 - */ -public class ClientInfo { - /** - * 应用名 - */ - private String appName; - /** - * 连接的id - */ - private String channelId; - - private ChannelHandlerContext channelHandlerContext; - - public ClientInfo() { - } - - public ClientInfo(String appName, String channelId, ChannelHandlerContext channelHandlerContext) { - this.appName = appName; - this.channelId = channelId; - this.channelHandlerContext = channelHandlerContext; - } - - @Override - public String toString() { - return "ClientInfo{" + - "appName='" + appName + '\'' + - ", channelId='" + channelId + '\'' + - ", channelHandlerContext=" + channelHandlerContext + - '}'; - } - - public ChannelHandlerContext getChannelHandlerContext() { - return channelHandlerContext; - } - - public void setChannelHandlerContext(ChannelHandlerContext channelHandlerContext) { - this.channelHandlerContext = channelHandlerContext; - } - - public String getAppName() { - return appName; - } - - public void setAppName(String appName) { - this.appName = appName; - } - - public String getChannelId() { - return channelId; - } - - public void setChannelId(String channelId) { - this.channelId = channelId; - } -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/mq/IMqMessageReceiver.java b/worker/src/main/java/com/jd/platform/hotkey/worker/mq/IMqMessageReceiver.java deleted file mode 100644 index 155a476806fbb2d768fe471f8ace7c0b82e78afc..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/mq/IMqMessageReceiver.java +++ /dev/null @@ -1,9 +0,0 @@ -package com.jd.platform.hotkey.worker.mq; - -/** - * @author wuweifeng wrote on 2019-12-12 - * @version 1.0 - */ -public interface IMqMessageReceiver { - void receive(String msg); -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/client/ClientChangeListener.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/client/ClientChangeListener.java index daca0c9bf809a033a1f2f4d61037d09a3975e1ff..f2362b7c4214494b89a7c78a5f6b8f0fda2276fc 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/client/ClientChangeListener.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/client/ClientChangeListener.java @@ -1,13 +1,12 @@ package com.jd.platform.hotkey.worker.netty.client; +import com.jd.platform.hotkey.common.tool.NettyIpUtil; import com.jd.platform.hotkey.worker.model.AppInfo; import com.jd.platform.hotkey.worker.netty.holder.ClientInfoHolder; import io.netty.channel.ChannelHandlerContext; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import java.util.Map; - /** * 对客户端的管理，新来、断线的管理 * @@ -20,7 +19,7 @@ public class ClientChangeListener implements IClientChangeListener { private static final String NEW_CLIENT = "监听到事件"; private static final String NEW_CLIENT_JOIN = "new client join"; - private static final String CLIENT_LOSE = "client removed"; + private static final String CLIENT_LOSE = "client removed "; /** * 客户端新增 @@ -33,29 +32,24 @@ public class ClientChangeListener implements IClientChangeListener { for (AppInfo appInfo : ClientInfoHolder.apps) { if (appName.equals(appInfo.getAppName())) { appExist = true; - appInfo.getMap().put(ip, ctx); + appInfo.add(ctx); break; } } if (!appExist) { - AppInfo appInfo = new AppInfo(); - appInfo.setAppName(appName); + AppInfo appInfo = new AppInfo(appName); ClientInfoHolder.apps.add(appInfo); - appInfo.getMap().put(ip, ctx); + appInfo.add(ctx); } logger.info(NEW_CLIENT_JOIN); } @Override - public synchronized void loseClient(String ip) { + public synchronized void loseClient(ChannelHandlerContext ctx) { for (AppInfo appInfo : ClientInfoHolder.apps) { - Map map = appInfo.getMap(); - if (map.containsKey(ip)) { - map.remove(ip); - break; - } + appInfo.remove(ctx); } - logger.info(CLIENT_LOSE); + logger.info(CLIENT_LOSE + NettyIpUtil.clientIp(ctx)); } } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/client/IClientChangeListener.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/client/IClientChangeListener.java index a6656b8912d85bd7feb5d5324cc8cbc0e9939394..b97825b575bd86d6f3b28b99b7f1d3f88632478c 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/client/IClientChangeListener.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/client/IClientChangeListener.java @@ -15,5 +15,5 @@ public interface IClientChangeListener { /** * 客户端掉线 */ - void loseClient(String channelId); + void loseClient(ChannelHandlerContext ctx); } \ No newline at end of file diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/DashboardHolder.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/DashboardHolder.java new file mode 100644 index 0000000000000000000000000000000000000000..6e0d57e0bd0696e7a56edcb91da86567302cc74e --- /dev/null +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/DashboardHolder.java @@ -0,0 +1,26 @@ +package com.jd.platform.hotkey.worker.netty.dashboard; + +import com.jd.platform.hotkey.common.model.HotKeyMsg; +import com.jd.platform.hotkey.common.model.MsgBuilder; +import com.jd.platform.hotkey.common.model.typeenum.MessageType; +import io.netty.channel.Channel; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-08-31 + */ +public class DashboardHolder { + /** + * 是否连上了dashboard + */ + public static boolean hasConnected = false; + /** + * channel + */ + public static Channel channel = null; + + public static void flushToDashboard(String message) { + channel.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.REQUEST_HOT_KEY, message))); + } +} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/NettyClient.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/NettyClient.java new file mode 100755 index 0000000000000000000000000000000000000000..5209c355e841c5910a4f622a4401017f38ad31ec --- /dev/null +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/NettyClient.java @@ -0,0 +1,86 @@ +package com.jd.platform.hotkey.worker.netty.dashboard; + +import com.jd.platform.hotkey.common.tool.Constant; +import io.netty.bootstrap.Bootstrap; +import io.netty.buffer.ByteBuf; +import io.netty.buffer.Unpooled; +import io.netty.channel.*; +import io.netty.channel.nio.NioEventLoopGroup; +import io.netty.channel.socket.SocketChannel; +import io.netty.channel.socket.nio.NioSocketChannel; +import io.netty.handler.codec.DelimiterBasedFrameDecoder; +import io.netty.handler.codec.string.StringDecoder; +import io.netty.handler.timeout.IdleStateHandler; + +/** + * netty连接器 + * + * @author wuweifeng wrote on 2019-11-05. + */ +public class NettyClient { + private static final NettyClient nettyClient = new NettyClient(); + + private Bootstrap bootstrap; + + + public static NettyClient getInstance() { + return nettyClient; + } + + private NettyClient() { + if (bootstrap == null) { + bootstrap = initBootstrap(); + } + } + + private Bootstrap initBootstrap() { + //少线程 + EventLoopGroup group = new NioEventLoopGroup(2); + + Bootstrap bootstrap = new Bootstrap(); + NettyClientHandler nettyClientHandler = new NettyClientHandler(); + bootstrap.group(group).channel(NioSocketChannel.class) + .option(ChannelOption.SO_KEEPALIVE, true) + .option(ChannelOption.TCP_NODELAY, true) + .handler(new ChannelInitializer() { + @Override + protected void initChannel(SocketChannel ch) { + ByteBuf delimiter = Unpooled.copiedBuffer(Constant.DELIMITER.getBytes()); + ch.pipeline() + .addLast(new DelimiterBasedFrameDecoder(Constant.MAX_LENGTH, delimiter)) + .addLast(new StringDecoder()) + //10秒没消息时，就发心跳包过去 + .addLast(new IdleStateHandler(0, 0, 30)) + .addLast(nettyClientHandler); + } + }); + return bootstrap; + } + + public synchronized void connect(String address) { + if (DashboardHolder.hasConnected) { + return; + } + String[] ss = address.split(":"); + try { + ChannelFuture channelFuture = bootstrap.connect(ss[0], Integer.parseInt(ss[1])).sync(); + DashboardHolder.channel = channelFuture.channel(); + DashboardHolder.hasConnected = true; + } catch (Exception e) { + DashboardHolder.hasConnected = false; + DashboardHolder.channel = null; + e.printStackTrace(); + } + + //这一步就阻塞了 +// channelFuture.channel().closeFuture().sync(); + //当server断开后才会走下面的 +// System.out.println("server is down"); + } + + public synchronized void disConnect() { + DashboardHolder.channel = null; + DashboardHolder.hasConnected = false; + } + +} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/NettyClientHandler.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/NettyClientHandler.java new file mode 100755 index 0000000000000000000000000000000000000000..5fdbe3a6b96f33968d98b55a65544e8aabd5a7f4 --- /dev/null +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/dashboard/NettyClientHandler.java @@ -0,0 +1,58 @@ +package com.jd.platform.hotkey.worker.netty.dashboard; + +import com.jd.platform.hotkey.common.model.HotKeyMsg; +import com.jd.platform.hotkey.common.model.MsgBuilder; +import com.jd.platform.hotkey.common.model.typeenum.MessageType; +import com.jd.platform.hotkey.common.tool.Constant; +import com.jd.platform.hotkey.common.tool.FastJsonUtils; +import io.netty.channel.ChannelHandler; +import io.netty.channel.ChannelHandlerContext; +import io.netty.channel.SimpleChannelInboundHandler; +import io.netty.handler.timeout.IdleState; +import io.netty.handler.timeout.IdleStateEvent; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * @author wuweifeng wrote on 2019-11-05. + */ +@ChannelHandler.Sharable +public class NettyClientHandler extends SimpleChannelInboundHandler { + + private Logger logger = LoggerFactory.getLogger(getClass()); + + @Override + public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception { + if (evt instanceof IdleStateEvent) { + IdleStateEvent idleStateEvent = (IdleStateEvent) evt; + + if (idleStateEvent.state() == IdleState.ALL_IDLE) { + //向服务端发送消息 + ctx.writeAndFlush(MsgBuilder.buildByteBuf(new HotKeyMsg(MessageType.PING, Constant.PING))); + } + } + + super.userEventTriggered(ctx, evt); + } + + @Override + public void channelActive(ChannelHandlerContext ctx) { + logger.info("channelActive:" + ctx.name()); + } + + @Override + public void channelInactive(ChannelHandlerContext ctx) throws Exception { + super.channelInactive(ctx); + NettyClient.getInstance().disConnect(); + } + + @Override + protected void channelRead0(ChannelHandlerContext channelHandlerContext, String message) { + HotKeyMsg msg = FastJsonUtils.toBean(message, HotKeyMsg.class); + if (MessageType.PONG == msg.getMessageType()) { + logger.info("heart beat"); + } + + } + +} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/HeartBeatFilter.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/HeartBeatFilter.java index a013d536fce4ddae36b81ef61791dcdf5f94adc0..7ef2cb83fd3d28a25ccc76962bb871fb297eb72c 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/HeartBeatFilter.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/HeartBeatFilter.java @@ -4,7 +4,7 @@ import com.jd.platform.hotkey.common.model.HotKeyMsg; import com.jd.platform.hotkey.common.model.MsgBuilder; import com.jd.platform.hotkey.common.model.typeenum.MessageType; import com.jd.platform.hotkey.common.tool.FastJsonUtils; -import com.jd.platform.hotkey.worker.netty.flush.FlushUtil; +import com.jd.platform.hotkey.common.tool.flush.FlushUtil; import io.netty.channel.ChannelHandlerContext; import org.springframework.core.annotation.Order; import org.springframework.stereotype.Component; diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/HotKeyFilter.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/HotKeyFilter.java index 080a9d982e70a1e2c600e419f4280abdbe2c149e..2640fd74da91e0193dd3420c6339c642bfb8eb09 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/HotKeyFilter.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/HotKeyFilter.java @@ -7,7 +7,6 @@ import com.jd.platform.hotkey.common.model.typeenum.MessageType; import com.jd.platform.hotkey.common.tool.FastJsonUtils; import com.jd.platform.hotkey.common.tool.NettyIpUtil; import com.jd.platform.hotkey.worker.keydispatcher.KeyProducer; -import com.jd.platform.hotkey.worker.mq.IMqMessageReceiver; import com.jd.platform.hotkey.worker.netty.holder.WhiteListHolder; import io.netty.channel.ChannelHandlerContext; import org.slf4j.Logger; @@ -27,7 +26,7 @@ import java.util.concurrent.atomic.AtomicLong; */ @Component @Order(3) -public class HotKeyFilter implements INettyMsgFilter, IMqMessageReceiver { +public class HotKeyFilter implements INettyMsgFilter { @Resource private KeyProducer keyProducer; @@ -48,33 +47,20 @@ public class HotKeyFilter implements INettyMsgFilter, IMqMessageReceiver { return true; } - @Override - public void receive(String msg) { - publishMsg(msg, null); - } - private void publishMsg(String message, ChannelHandlerContext ctx) { - //这个是给测试用的，实际走的是下面那个 - if (message.startsWith("{")) { - HotKeyModel model = FastJsonUtils.toBean(message, HotKeyModel.class); - if (WhiteListHolder.contains(model.getKey())) { - return; - } - keyProducer.push(model); - return; - } //老版的用的单个HotKeyModel，新版用的数组 List models = FastJsonUtils.toList(message, HotKeyModel.class); + long now = SystemClock.now(); for (HotKeyModel model : models) { //白名单key不处理 if (WhiteListHolder.contains(model.getKey())) { continue; } - long timeOut = SystemClock.now() - model.getCreateTime(); + long timeOut = now - model.getCreateTime(); if (timeOut > 1000) { logger.info("key timeout " + timeOut + ", from ip : " + NettyIpUtil.clientIp(ctx)); } - keyProducer.push(model); + keyProducer.push(model, now); } } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/KeyCounterFilter.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/KeyCounterFilter.java index d8eae4834dc10ad3c404a1afa971dae5662dc823..424a9fca9f2ff0243383aefa2f96c5bc847572d9 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/KeyCounterFilter.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/filter/KeyCounterFilter.java @@ -1,5 +1,6 @@ package com.jd.platform.hotkey.worker.netty.filter; +import cn.hutool.core.collection.CollectionUtil; import cn.hutool.core.date.SystemClock; import cn.hutool.core.util.StrUtil; import com.jd.platform.hotkey.common.model.HotKeyMsg; @@ -8,7 +9,6 @@ import com.jd.platform.hotkey.common.model.typeenum.MessageType; import com.jd.platform.hotkey.common.tool.FastJsonUtils; import com.jd.platform.hotkey.common.tool.NettyIpUtil; import com.jd.platform.hotkey.worker.counter.KeyCountItem; -import com.jd.platform.hotkey.worker.mq.IMqMessageReceiver; import com.jd.platform.hotkey.worker.tool.InitConstant; import io.netty.channel.ChannelHandlerContext; import org.slf4j.Logger; @@ -28,7 +28,7 @@ import static com.jd.platform.hotkey.worker.counter.CounterConfig.COUNTER_QUEUE; */ @Component @Order(4) -public class KeyCounterFilter implements INettyMsgFilter, IMqMessageReceiver { +public class KeyCounterFilter implements INettyMsgFilter { private Logger logger = LoggerFactory.getLogger(getClass()); /** @@ -52,14 +52,12 @@ public class KeyCounterFilter implements INettyMsgFilter, IMqMessageReceiver { return true; } - @Override - public void receive(String msg) { - publishMsg("", msg, null); - } private void publishMsg(String appName, String message, ChannelHandlerContext ctx) { - //老版的用的单个HotKeyModel，新版用的数组 List models = FastJsonUtils.toList(message, KeyCountModel.class); + if (CollectionUtil.isEmpty(models)) { + return; + } long timeOut = SystemClock.now() - models.get(0).getCreateTime(); //超时5秒以上的就不处理了，因为client是每10秒发送一次，所以最迟15秒以后的就不处理了 if (timeOut > InitConstant.timeOut + 10000) { diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/AppServerPusher.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/AppServerPusher.java index db89b85af07c30b3fc23ffffdc4238c820adca44..99058cf09a56b50623b6471a43ae1abaca5116f6 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/AppServerPusher.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/AppServerPusher.java @@ -6,14 +6,10 @@ import com.jd.platform.hotkey.common.model.MsgBuilder; import com.jd.platform.hotkey.common.model.typeenum.MessageType; import com.jd.platform.hotkey.common.tool.FastJsonUtils; import com.jd.platform.hotkey.worker.model.AppInfo; -import com.jd.platform.hotkey.worker.netty.flush.FlushUtil; import com.jd.platform.hotkey.worker.netty.holder.ClientInfoHolder; import io.netty.buffer.ByteBuf; -import io.netty.channel.ChannelHandlerContext; import org.springframework.stereotype.Component; -import java.util.Map; - /** * 推送到各客户端服务器 * @author wuweifeng wrote on 2020-02-24 @@ -29,21 +25,14 @@ public class AppServerPusher implements IPusher { public void push(HotKeyModel model) { for (AppInfo appInfo : ClientInfoHolder.apps) { if (model.getAppName().equals(appInfo.getAppName())) { - Map map = appInfo.getMap(); HotKeyMsg hotKeyMsg = new HotKeyMsg(MessageType.RESPONSE_NEW_KEY, FastJsonUtils.convertObjectToJSON(model)); String hotMsg = FastJsonUtils.convertObjectToJSON(hotKeyMsg); -// for (ChannelHandlerContext channel : map.values()) { -// ByteBuf byteBuf = MsgBuilder.buildByteBuf(hotMsg); -// FlushUtil.flush(channel, byteBuf); -// } + ByteBuf byteBuf = MsgBuilder.buildByteBuf(hotMsg); - //并行发送 - map.values().parallelStream().forEach(channel -> { - ByteBuf byteBuf = MsgBuilder.buildByteBuf(hotMsg); - FlushUtil.flush(channel, byteBuf); - }); + //整个app全部发送 + appInfo.groupPush(byteBuf); return; } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/BatchToEtcdScheduler.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/BatchToEtcdScheduler.java deleted file mode 100644 index fe958d573ec6c2eb2973c79daa7b3d7f38ba394a..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/BatchToEtcdScheduler.java +++ /dev/null @@ -1,45 +0,0 @@ -package com.jd.platform.hotkey.worker.netty.pusher; - -import com.jd.platform.hotkey.common.configcenter.ConfigConstant; -import com.jd.platform.hotkey.common.configcenter.IConfigCenter; -import com.jd.platform.hotkey.common.tool.FastJsonUtils; -import org.springframework.stereotype.Component; - -import javax.annotation.Resource; -import java.util.Set; -import java.util.UUID; - -/** - * @author wuweifeng - * @version 1.0 - * @date 2020-05-26 - */ -@Component -@Deprecated -public class BatchToEtcdScheduler { - @Resource - private KeyCollector keyCollector; - @Resource - private IConfigCenter iConfigCenter; - - /** - * 每隔0.5秒上传一下已探测出的热key发往etcd供入库 - */ -// @Scheduled(fixedRate = 500) - public void uploadClientCount() { - Set set = keyCollector.lockAndGetResult(); - if (set.size() == 0) { - return; - } - - //worker将热key推送到该地址，供dashboard监听入库做记录 - String hotKeyRecordPath = ConfigConstant.hotKeyRecordPath + UUID.randomUUID().toString(); - try { - //推送到etcd，供dashboard监听入库 - iConfigCenter.putAndGrant(hotKeyRecordPath, FastJsonUtils.convertObjectToJSON(set), 10); - } catch (Exception e) { - e.printStackTrace(); - } - - } -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/EtcdPusher.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/DashboardPusher.java similarity index 31% rename from worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/EtcdPusher.java rename to worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/DashboardPusher.java index 627d3eb83098182e3790b11754ab935f386c11f5..aa420cdbaf4363867912523e857c2bc7ff44bd93 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/EtcdPusher.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/DashboardPusher.java @@ -1,38 +1,24 @@ package com.jd.platform.hotkey.worker.netty.pusher; -import com.jd.platform.hotkey.common.configcenter.IConfigCenter; import com.jd.platform.hotkey.common.model.HotKeyModel; -import com.jd.platform.hotkey.common.tool.HotKeyPathTool; -import com.jd.platform.hotkey.worker.rule.KeyRuleHolder; +import com.jd.platform.hotkey.worker.netty.pusher.store.HotkeyTempStore; import org.springframework.stereotype.Component; -import javax.annotation.Resource; -import java.util.UUID; - - /** - * @author wuweifeng wrote on 2020-02-24 + * 将热key推送到dashboard供入库 + * @author wuweifeng * @version 1.0 + * @date 2020-08-31 */ @Component -public class EtcdPusher implements IPusher { - @Resource - private IConfigCenter iConfigCenter; - - +public class DashboardPusher implements IPusher { @Override public void push(HotKeyModel model) { - //推送到etcd，供dashboard监听入库 - iConfigCenter.putAndGrant(HotKeyPathTool.keyRecordPath(model), UUID.randomUUID().toString(), - KeyRuleHolder.getRuleByAppAndKey(model).getDuration()); + HotkeyTempStore.push(model); } @Override - @Deprecated public void remove(HotKeyModel model) { - //推送etcd删除 - iConfigCenter.delete(HotKeyPathTool.keyPath(model)); - } - + } } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/KeyCollector.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/KeyCollector.java deleted file mode 100644 index 1dad84245d1dbbcf3007063c69cbcb1d4e93c140..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/KeyCollector.java +++ /dev/null @@ -1,53 +0,0 @@ -package com.jd.platform.hotkey.worker.netty.pusher; - -import cn.hutool.core.collection.ConcurrentHashSet; -import cn.hutool.core.util.StrUtil; -import com.jd.platform.hotkey.common.model.HotKeyModel; -import org.springframework.stereotype.Component; - -import java.util.HashSet; -import java.util.Set; -import java.util.concurrent.atomic.AtomicLong; - -import static com.jd.platform.hotkey.common.tool.KeyRecordKeyTool.key; - -/** - * @author wuweifeng - * @version 1.0 - * @date 2020-05-26 - */ -@Component -@Deprecated -public class KeyCollector { - private Set keySet0 = new ConcurrentHashSet<>(); - private Set keySet1 = new ConcurrentHashSet<>(); - private AtomicLong atomicLong = new AtomicLong(0); - - public Set lockAndGetResult() { - //自增后，对应的map就会停止被写入，等待被读取 - atomicLong.addAndGet(1); - Set set; - if (atomicLong.get() % 2 == 0) { - set = new HashSet<>(keySet1); - keySet1.clear(); - } else { - set = new HashSet<>(keySet0); - keySet0.clear(); - } - return set; - } - - public void collect(HotKeyModel hotKeyModel) { - String key = hotKeyModel.getKey(); - if (StrUtil.isEmpty(key)) { - return; - } - if (atomicLong.get() % 2 == 0) { - keySet0.add(key(hotKeyModel)); - } else { - keySet1.add(key(hotKeyModel)); - } - - } - -} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/KeyUploader.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/KeyUploader.java new file mode 100644 index 0000000000000000000000000000000000000000..8ee6e11a4ada6f5e5830cd7a6db531383bbac18c --- /dev/null +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/KeyUploader.java @@ -0,0 +1,46 @@ +package com.jd.platform.hotkey.worker.netty.pusher; + +import cn.hutool.core.collection.CollectionUtil; +import com.google.common.collect.Queues; +import com.jd.platform.hotkey.common.model.HotKeyModel; +import com.jd.platform.hotkey.common.tool.FastJsonUtils; +import com.jd.platform.hotkey.worker.netty.dashboard.DashboardHolder; +import com.jd.platform.hotkey.worker.netty.pusher.store.HotkeyTempStore; +import com.jd.platform.hotkey.worker.tool.AsyncPool; +import org.springframework.stereotype.Component; + +import javax.annotation.PostConstruct; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.TimeUnit; + +/** + * 定时批量往dashboard发送热key，供入库 + * @author wuweifeng + * @version 1.0 + * @date 2020-08-31 + */ +@Component +public class KeyUploader { + + @PostConstruct + public void uploadToDashboard() { + AsyncPool.asyncDo(() -> { + while (true) { + try { + //要么key达到1千个，要么达到1秒，就汇总上报给etcd一次 + List tempModels = new ArrayList<>(); + Queues.drain(HotkeyTempStore.getQueue(), tempModels, 1000, 1, TimeUnit.SECONDS); + if (CollectionUtil.isEmpty(tempModels)) { + continue; + } + + //将热key推到dashboard + DashboardHolder.flushToDashboard(FastJsonUtils.convertObjectToJSON(tempModels)); + } catch (Exception e) { + e.printStackTrace(); + } + } + }); + } +} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/store/HotkeyTempStore.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/store/HotkeyTempStore.java new file mode 100644 index 0000000000000000000000000000000000000000..c93d18da36c21cc4d7ea7eeda95124f317b5ab11 --- /dev/null +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/pusher/store/HotkeyTempStore.java @@ -0,0 +1,27 @@ +package com.jd.platform.hotkey.worker.netty.pusher.store; + +import com.jd.platform.hotkey.common.model.HotKeyModel; + +import java.util.concurrent.LinkedBlockingQueue; + +/** + * 已热待上报入库的热key集中营 + * @author wuweifeng + * @version 1.0 + * @date 2020-08-31 + */ +public class HotkeyTempStore { + /** + * 热key集中营 + */ + private static LinkedBlockingQueue hotKeyStoreQueue = new LinkedBlockingQueue<>(); + + public static void push(HotKeyModel model) { + hotKeyStoreQueue.offer(model); + } + + public static LinkedBlockingQueue getQueue() { + return hotKeyStoreQueue; + } + +} diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/server/NodesServer.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/server/NodesServer.java index 6ea9b4fbde668072e1c03feab62d5afdb55fe21f..0762742489ef13857be57824a2dafafe205269e0 100755 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/server/NodesServer.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/server/NodesServer.java @@ -1,6 +1,5 @@ package com.jd.platform.hotkey.worker.netty.server; -import com.jd.platform.hotkey.common.coder.Codec; import com.jd.platform.hotkey.common.tool.Constant; import com.jd.platform.hotkey.worker.netty.client.IClientChangeListener; import com.jd.platform.hotkey.worker.netty.filter.INettyMsgFilter; @@ -17,6 +16,7 @@ import io.netty.handler.logging.LogLevel; import io.netty.handler.logging.LoggingHandler; import java.util.List; +import java.util.concurrent.TimeUnit; /** * 该server用于给各个微服务实例连接用。 @@ -26,7 +26,6 @@ import java.util.List; public class NodesServer { private IClientChangeListener clientChangeListener; private List messageFilters; - private Codec codec; public void startNettyServer(int port) throws Exception { //boss单线程 @@ -44,9 +43,14 @@ public class NodesServer { .childHandler(new ChildChannelHandler()); //绑定端口，同步等待成功 ChannelFuture future = bootstrap.bind(port).sync(); + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + bossGroup.shutdownGracefully (1000, 3000, TimeUnit.MILLISECONDS); + workerGroup.shutdownGracefully (1000, 3000, TimeUnit.MILLISECONDS); + })); //等待服务器监听端口关闭 future.channel().closeFuture().sync(); } catch (Exception e) { + e.printStackTrace(); //do nothing System.out.println("netty stop"); } finally { @@ -70,8 +74,6 @@ public class NodesServer { ByteBuf delimiter = Unpooled.copiedBuffer(Constant.DELIMITER.getBytes()); ch.pipeline() .addLast(new DelimiterBasedFrameDecoder(Constant.MAX_LENGTH, delimiter)) -// .addLast(codec.newEncoder()) -// .addLast(codec.newDecoder()) .addLast(new StringDecoder()) .addLast(serverHandler); } @@ -85,7 +87,4 @@ public class NodesServer { this.messageFilters = messageFilters; } - public void setCodec(Codec codec) { - this.codec = codec; - } } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/server/NodesServerHandler.java b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/server/NodesServerHandler.java index 64f2a2b5acd7594381127edf73f2e800e31d6301..8aa16f576974c8d643c1855b1fe9764769720445 100755 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/netty/server/NodesServerHandler.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/netty/server/NodesServerHandler.java @@ -2,7 +2,6 @@ package com.jd.platform.hotkey.worker.netty.server; import com.jd.platform.hotkey.common.model.HotKeyMsg; import com.jd.platform.hotkey.common.tool.FastJsonUtils; -import com.jd.platform.hotkey.common.tool.NettyIpUtil; import com.jd.platform.hotkey.worker.netty.client.IClientChangeListener; import com.jd.platform.hotkey.worker.netty.filter.INettyMsgFilter; import io.netty.channel.ChannelHandlerContext; @@ -47,7 +46,8 @@ public class NodesServerHandler extends SimpleChannelInboundHandler { } @Override - public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception { + public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) { + cause.printStackTrace(); logger.error("some thing is error , " + cause.getMessage()); } @@ -59,7 +59,7 @@ public class NodesServerHandler extends SimpleChannelInboundHandler { @Override public void channelInactive(ChannelHandlerContext ctx) throws Exception { if (clientEventListener != null) { - clientEventListener.loseClient(NettyIpUtil.clientIp(ctx)); + clientEventListener.loseClient(ctx); } ctx.close(); super.channelInactive(ctx); diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/starters/EtcdStarter.java b/worker/src/main/java/com/jd/platform/hotkey/worker/starters/EtcdStarter.java index 9ce9cc9db97f33190de8ecacf0ed7e06a53ec642..152af113ae337b37988eda2609984ec8f6eef946 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/starters/EtcdStarter.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/starters/EtcdStarter.java @@ -1,5 +1,6 @@ package com.jd.platform.hotkey.worker.starters; +import cn.hutool.core.collection.CollectionUtil; import cn.hutool.core.util.StrUtil; import com.ibm.etcd.api.Event; import com.ibm.etcd.api.KeyValue; @@ -13,6 +14,7 @@ import com.jd.platform.hotkey.common.tool.IpUtils; import com.jd.platform.hotkey.worker.cache.CaffeineCacheHolder; import com.jd.platform.hotkey.worker.model.AppInfo; import com.jd.platform.hotkey.worker.model.TotalCount; +import com.jd.platform.hotkey.worker.netty.dashboard.NettyClient; import com.jd.platform.hotkey.worker.netty.filter.HotKeyFilter; import com.jd.platform.hotkey.worker.netty.holder.ClientInfoHolder; import com.jd.platform.hotkey.worker.netty.holder.WhiteListHolder; @@ -20,7 +22,6 @@ import com.jd.platform.hotkey.worker.rule.KeyRuleHolder; import com.jd.platform.hotkey.worker.tool.AsyncPool; import com.jd.platform.hotkey.worker.tool.InitConstant; import io.grpc.StatusRuntimeException; -import io.netty.channel.ChannelHandlerContext; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Value; @@ -31,7 +32,6 @@ import javax.annotation.PostConstruct; import javax.annotation.PreDestroy; import javax.annotation.Resource; import java.util.List; -import java.util.Map; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; @@ -46,6 +46,11 @@ import static com.jd.platform.hotkey.worker.tool.InitConstant.*; */ @Component public class EtcdStarter { + /** + * 是否开启日志 + */ + public static boolean LOGGER_ON = true; + private Logger logger = LoggerFactory.getLogger(getClass()); @Resource @@ -100,6 +105,31 @@ public class EtcdStarter { return !DEFAULT_PATH.equals(workerPath); } + @PostConstruct + public void watchLog() { + AsyncPool.asyncDo(() -> { + try { + String loggerOn = configCenter.get(ConfigConstant.logToggle); + LOGGER_ON = "true".equals(loggerOn) || "1".equals(loggerOn); + } catch (StatusRuntimeException ex) { + logger.error(ETCD_DOWN); + } + + KvClient.WatchIterator watchIterator = configCenter.watch(ConfigConstant.logToggle); + while (watchIterator.hasNext()) { + WatchUpdate watchUpdate = watchIterator.next(); + List eventList = watchUpdate.getEvents(); + + KeyValue keyValue = eventList.get(0).getKv(); + logger.info("log toggle changed : " + keyValue); + + String value = keyValue.getValue().toStringUtf8(); + LOGGER_ON = "true".equals(value) || "1".equals(value); + } + }); + + } + /** * 启动回调监听器，监听rule变化 */ @@ -196,8 +226,7 @@ public class EtcdStarter { String ip = IpUtils.getIp(); for (AppInfo appInfo : ClientInfoHolder.apps) { String appName = appInfo.getAppName(); - Map map = appInfo.getMap(); - int count = map.values().size(); + int count = appInfo.size(); //即便是full gc也不能超过3秒 configCenter.putAndGrant(ConfigConstant.clientCountPath + appName + "/" + ip, count + "", 13); } @@ -220,6 +249,29 @@ public class EtcdStarter { } } + /** + * 每隔30秒去获取一下dashboard的地址 + */ + @Scheduled(fixedRate = 30000) + public void fetchDashboardIp() { + try { + //获取DashboardIp + List keyValues = configCenter.getPrefix(ConfigConstant.dashboardPath); + + //是空，给个警告 + if (CollectionUtil.isEmpty(keyValues)) { + logger.warn("very important warn !!! Dashboard ip is null!!!"); + return; + } + + String dashboardIp = keyValues.get(0).getValue().toStringUtf8(); + NettyClient.getInstance().connect(dashboardIp); + + } catch (Exception e) { + e.printStackTrace(); + } + } + /** * 校验一下receive的key数量，如果一段时间没变，考虑网络问题，就将worker注册自己到etcd的心跳给断掉30秒，让各client重连一下自己 */ diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/starters/NodesServerStarter.java b/worker/src/main/java/com/jd/platform/hotkey/worker/starters/NodesServerStarter.java index 0b9c52270342f7916fa345a5b65dda49e7c2fa8a..5759f5e7bfe95bef1dea576f03d080f62fb96e90 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/starters/NodesServerStarter.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/starters/NodesServerStarter.java @@ -1,6 +1,5 @@ package com.jd.platform.hotkey.worker.starters; -import com.jd.platform.hotkey.common.coder.Codec; import com.jd.platform.hotkey.worker.netty.client.IClientChangeListener; import com.jd.platform.hotkey.worker.netty.filter.INettyMsgFilter; import com.jd.platform.hotkey.worker.netty.server.NodesServer; @@ -29,8 +28,6 @@ public class NodesServerStarter { private IClientChangeListener iClientChangeListener; @Resource private List messageFilters; - @Resource - private Codec codec; @PostConstruct public void start() { @@ -40,7 +37,6 @@ public class NodesServerStarter { NodesServer nodesServer = new NodesServer(); nodesServer.setClientChangeListener(iClientChangeListener); nodesServer.setMessageFilters(messageFilters); - nodesServer.setCodec(codec); try { nodesServer.startNettyServer(port); } catch (Exception e) { diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/tool/CpuNum.java b/worker/src/main/java/com/jd/platform/hotkey/worker/tool/CpuNum.java index acaa70e79562d790ade6c04df094da31a7e86ec3..55bd8424aab7e4b63ed4901e65d7c52b8126925f 100644 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/tool/CpuNum.java +++ b/worker/src/main/java/com/jd/platform/hotkey/worker/tool/CpuNum.java @@ -13,17 +13,13 @@ public class CpuNum { public static int workerCount() { //取cpu核数，新版jdk在docker里取的就是真实分配的，老版jdk取的是宿主机的，可能特别大，如32核 int count = Runtime.getRuntime().availableProcessors(); -// if (isNewerVersion()) { -// if (count >= 4) { -// count = count / 2; -// } -// } else { -// if (count >= 8) { -// count = 4; -// } -// } - if (count >= 4) { + if (isNewerVersion()) { + return count; + } else { count = count / 2; + if (count == 0) { + count = 1; + } } return count; } diff --git a/worker/src/main/java/com/jd/platform/hotkey/worker/tool/ProtostuffUtils.java b/worker/src/main/java/com/jd/platform/hotkey/worker/tool/ProtostuffUtils.java deleted file mode 100644 index 0cb88c5eb341ef2273074e425e019b32582b0731..0000000000000000000000000000000000000000 --- a/worker/src/main/java/com/jd/platform/hotkey/worker/tool/ProtostuffUtils.java +++ /dev/null @@ -1,77 +0,0 @@ -package com.jd.platform.hotkey.worker.tool; - -import io.protostuff.LinkedBuffer; -import io.protostuff.ProtostuffIOUtil; -import io.protostuff.Schema; -import io.protostuff.runtime.RuntimeSchema; - -import java.util.Map; -import java.util.Objects; -import java.util.concurrent.ConcurrentHashMap; - -/** - * - * @author 周志刚 - * @date 2019/6/18 - **/ -public class ProtostuffUtils { - /** - * 避免每次序列化都重新申请Buffer空间 - */ - private static LinkedBuffer buffer = LinkedBuffer.allocate(LinkedBuffer.DEFAULT_BUFFER_SIZE); - /** - * 缓存Schema - */ - private static Map, Schema> schemaCache = new ConcurrentHashMap<>(); - - /** - * 序列化方法，把指定对象序列化成字节数组 - * - * @param obj - * @param - * @return - */ - @SuppressWarnings("unchecked") - public static byte[] serialize(T obj) { - Class clazz = (Class) obj.getClass(); - Schema schema = getSchema(clazz); - byte[] data; - try { - data = ProtostuffIOUtil.toByteArray(obj, schema, buffer); - } finally { - buffer.clear(); - } - - return data; - } - - /** - * 反序列化方法，将字节数组反序列化成指定Class类型 - * - * @param data - * @param clazz - * @param - * @return - */ - public static T deserialize(byte[] data, Class clazz) { - Schema schema = getSchema(clazz); - T obj = schema.newMessage(); - ProtostuffIOUtil.mergeFrom(data, obj, schema); - return obj; - } - - @SuppressWarnings("unchecked") - private static Schema getSchema(Class clazz) { - Schema schema = (Schema) schemaCache.get(clazz); - if (Objects.isNull(schema)) { - //这个schema通过RuntimeSchema进行懒创建并缓存 - //所以可以一直调用RuntimeSchema.getSchema(),这个方法是线程安全的 - schema = RuntimeSchema.getSchema(clazz); - if (Objects.nonNull(schema)) { - schemaCache.put(clazz, schema); - } - } - - return schema; - } -} \ No newline at end of file diff --git a/worker/src/main/resources/application.yml b/worker/src/main/resources/application.yml index 3d3360039581f7de029c3152141a9f6f41909b32..1708974c96cc6de8f8b0175e31f875fea0fa4248 100644 --- a/worker/src/main/resources/application.yml +++ b/worker/src/main/resources/application.yml @@ -5,7 +5,6 @@ netty: local: address: ${localAddress:} #有些获取到的ip不能用，需要手工配worker的地址 open: - timeout: ${openTimeOut:true} monitor: ${openMonitor:false} #开启持续无key发送监控，如果持续1分钟没发来key，就断开和etcd的连接，之后重建和客户端连接 thread: count: ${threadCount:0} diff --git a/worker/src/test/java/Test.java b/worker/src/test/java/Test.java index 92a993c18e863ce59b68cb3d19046c0406cf497e..c7e69406705ca8ee7914a42e6d1d29fed3572237 100644 --- a/worker/src/test/java/Test.java +++ b/worker/src/test/java/Test.java @@ -1,8 +1,3 @@ -import com.jd.platform.hotkey.common.model.HotKeyModel; -import com.jd.platform.hotkey.common.model.HotKeyMsg; -import com.jd.platform.hotkey.common.tool.FastJsonUtils; -import com.jd.platform.hotkey.worker.tool.ProtostuffUtils; - /** * @author wuweifeng * @version 1.0 @@ -10,28 +5,35 @@ import com.jd.platform.hotkey.worker.tool.ProtostuffUtils; */ public class Test { public static void main(String[] args) { - HotKeyMsg hotKeyMsg = new HotKeyMsg(); - hotKeyMsg.setAppName("cartsoa"); - HotKeyModel hotKeyModel = new HotKeyModel(); - hotKeyModel.setCount(1); - hotKeyModel.setKey("pin_xx"); - hotKeyModel.setAppName("cartsoa"); - hotKeyMsg.setBody(FastJsonUtils.convertObjectToJSON(hotKeyModel)); - - byte[] serialize = ProtostuffUtils.serialize(hotKeyMsg); - String msg = FastJsonUtils.convertObjectToJSON(hotKeyMsg); - - long time1 = System.currentTimeMillis(); - for (int i = 0; i < 300000; i++) { - HotKeyMsg hhh = ProtostuffUtils.deserialize(serialize, HotKeyMsg.class); - } - System.out.println(System.currentTimeMillis() - time1); +// HotKeyMsg hotKeyMsg = new HotKeyMsg(); +// hotKeyMsg.setAppName("cartsoafewwfwfwfw"); +// HotKeyModel hotKeyModel = new HotKeyModel(); +// hotKeyModel.setCount(1); +// hotKeyModel.setKey("pin_xwefwfaregegax"); +// hotKeyModel.setAppName("cartsoa"); +// hotKeyMsg.setBody(FastJsonUtils.convertObjectToJSON(hotKeyModel)); +// - long time = System.currentTimeMillis(); - for (int i = 0; i < 300000; i++) { - HotKeyMsg hhh = FastJsonUtils.toBean(msg, HotKeyMsg.class); - } - System.out.println(System.currentTimeMillis() - time); +// for (int i = 0; i < 10000; i++) { +// +// } +// String s = ConfigConstant.hotKeyRecordPath + hotKeyModel.getAppName() + "/" + hotKeyModel.getKey(); +// double m = ObjectSizeCalculator.getObjectSize(s); +// System.out.println(m * 10000 / 1024); +// byte[] serialize = ProtostuffUtils.serialize(hotKeyMsg); +// String msg = FastJsonUtils.convertObjectToJSON(hotKeyMsg); +// +// long time1 = System.currentTimeMillis(); +// for (int i = 0; i < 300000; i++) { +// HotKeyMsg hhh = ProtostuffUtils.deserialize(serialize, HotKeyMsg.class); +// } +// System.out.println(System.currentTimeMillis() - time1); +// +// long time = System.currentTimeMillis(); +// for (int i = 0; i < 300000; i++) { +// HotKeyMsg hhh = FastJsonUtils.toBean(msg, HotKeyMsg.class); +// } +// System.out.println(System.currentTimeMillis() - time); } diff --git a/worker/src/test/java/TestBatch.java b/worker/src/test/java/TestBatch.java new file mode 100644 index 0000000000000000000000000000000000000000..c62653fd98e1719a153148871d2ac874f2ef6af8 --- /dev/null +++ b/worker/src/test/java/TestBatch.java @@ -0,0 +1,66 @@ +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.concurrent.BlockingQueue; +import java.util.concurrent.LinkedBlockingQueue; + +/** + * @author wuweifeng + * @version 1.0 + * @date 2020-08-27 + */ +public class TestBatch { + public static BlockingQueue QUEUE = new LinkedBlockingQueue<>(2000000); + private static Logger logger = LoggerFactory.getLogger("tttt"); + private static int i; + + public static void main(String[] args) throws InterruptedException { + +// for (int j = 0; j < 10; j++) { +// QUEUE.offer("" + j); +// } +// List list = new ArrayList<>(); +// Queues.drain(QUEUE, list, 2, 1, TimeUnit.SECONDS); +// System.out.println(QUEUE); +// +// new Thread(new Runnable() { +// @Override +// public void run() { +// while (true) { +// try { +// Thread.sleep(100); +// } catch (InterruptedException e) { +// e.printStackTrace(); +// } +// QUEUE.offer("" + i++); +// } +// } +// }).start(); +// new Thread(new Runnable() { +// @Override +// public void run() { +// while (true) { +// try { +// Thread.sleep(1000); +// } catch (InterruptedException e) { +// e.printStackTrace(); +// } +// System.out.println(QUEUE); +// } +// } +// }).start(); +// +// while (true) { +// List list = new ArrayList<>(); +// try { +// System.out.println("size " + QUEUE.size()); +// Queues.drain(QUEUE, list, 2, 1, TimeUnit.SECONDS); +// Thread.sleep(100); +// } catch (InterruptedException e) { +// e.printStackTrace(); +// } +// logger.info(list.toString()); +// } + + } +}