# bigdata-application **Repository Path**: yangwanghui/bigdata-application ## Basic Information - **Project Name**: bigdata-application - **Description**: 现已在移动行业部署的大数据应用项目【iop、智能短信、旅游应用、一经逍遥、实时驻留时长等】，技术栈：SparkStreaming、StructStreaming、Kafka、Hbase、Flume等 - **Primary Language**: Unknown - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 2 - **Created**: 2020-05-27 - **Last Updated**: 2020-12-19 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README ## 目录结构 ``` bigdata-application: ├─bigdata-topic-integration `(位置信令汇总流任务)` ├─bigdata-utils `(工具包)` ├─bin `(module启动脚本)` ├─carefree-application `(一经逍遥)` ├─conf `(配置文件)` ├─doc `(文档)` ├─file-sink `(flume自定义sink到本地磁盘)` ├─iop-export-hbase `(将hbase表数据按指定要求导出到hdfs)` ├─iop-rule-parser `(iop活动解析web服务)` ├─iop-streaming `(iop实时事件处理平台)` ├─tour-application `(旅游应用)` ├─tour-repair `(旅游应用修复程序)` ├─user-stay-duration `(实时位置信令数据)` ``` --- ## 信令汇总流任务 **工程名：bigdata-topic-integration** > 该流任务将MME、2G、3G位置信令的Kafka Topic【O_DPI_LTE_MME,O_DPI_MC_LOCATIONUPDATE_2G,O_DPI_MC_LOCATIONUPDATE_3G】按需求字段取出并汇总到Topic【YZ_APP_TD_234G_DPI_DATA】 ###### 启动脚本 ``` bin/bigdata-topic-integration.sh bin/auto_start_integration_topic.sh ``` --- ## 智能短信 **工程名：intelligence-sms** ###### 启动脚本 ``` bin/IntelligentSMSApplicationAutoNoStayDuration.sh bin/IntelligentSMSApplicationAutoStayDuration.sh bin/IntelligentSMSApplicationWithNo.sh bin/IntelligentSMSApplicationWithStayDuration.sh bin/auto_start_IntelligentSMSApplicationAutoNoStayDuration.sh bin/auto_start_IntelligentSMSApplicationAutoStayDuration.sh bin/auto_start_IntelligentSMSApplicationWithNo.sh bin/auto_start_IntelligentSMSApplicationWithStayDuration.sh ``` --- ## 旅游应用 **工程名：tour-application** ###### 启动脚本 ``` bin/nm_tour.sh bin/auto_start_TourScenicAreaUsrInfoFromIntegrationTopic.sh ``` --- ## 旅游应用修复程序 **工程名：tour-repair** ###### 启动脚本 ``` 30 23 * * * /data/iop/projects/bin/nm_tour_repair.sh > /data/iop/logs/nm_tour_repair.log 2>&1 ``` --- ## iop实时事件处理平台 **工程名：iop-streaming** ###### 启动脚本 ``` bin/SendWelcomeToAPPUser.sh ##访问APP事件 bin/SendWelcomeToChangePhone.sh ##换机事件 bin/SendWelcomeToSelectPhone.sh ##位置事件 bin/SendWelcomeToSwitchPhone.sh ##开关机事件 bin/auto_start_SendWelcomeToSelectPhone.sh #自动启动脚本 ``` --- ## flume自定义sink到本地磁盘 **工程名：file-sink** ###### 启动脚本 ``` sh /data/iop/app/apache-flume-1.9.0-bin/bin/flume-ng agent \ --conf /data/iop/app/apache-flume-1.9.0-bin/conf/ \ --conf-file /data/iop/projects/conf/dhx-kafka-file-select-phone.conf \ --name dhx-select-phone-all \ -Xmx40000m -Xms4096m -Xmn256m -Xss256k -XX:+UseParallelGC -XX:ParallelGCThreads=20 \ -Dflume.root.logger=INFO,LOGFILE \ -Dflume.monitoring.type=http \ -Dflume.monitoring.port=60012 & ``` ###### 配置文件 ``` dhx-select-phone-all.sources = kafkaSource dhx-select-phone-all.channels = memoryChannel dhx-select-phone-all.sinks = fileSink dhx-select-phone-all.sources.kafkaSource.batchSize = 500000 dhx-select-phone-all.sources.kafkaSource.batchDurationMillis = 500000 dhx-select-phone-all.sources.kafkaSource.channels = memoryChannel dhx-select-phone-all.sinks.fileSink.channel = memoryChannel dhx-select-phone-all.sources.kafkaSource.type = org.apache.flume.source.kafka.KafkaSource dhx-select-phone-all.sources.kafkaSource.kafka.bootstrap.servers = ss-b02-m12-c05-r5300-4:6667,ss-b02-m12-c05-r5300-1:6667,ss-b02-m12-c05-r5300-5:6667,ss-b02-m12-c05-r5300-2:6667,ss-b02-m12-c05-r5300-6:6667,ss-b02-m12-c05-r5300-9:6667,ss-b02-m12-c04-r5300-10:6667,ss-b02-m12-c05-r5300-3:6667,ss-b02-m12-c05-r5300-7:6667,ss-b02-m12-c05-r5300-8:6667,ss-b02-m12-c05-r5300-10:6667,ss-b02-m12-c06-r5300-1:6667,ss-b02-m12-c06-r5300-2:6667,ss-b02-m12-c06-r5300-3:6667,ss-b02-m12-c06-r5300-5:6667,ss-b02-m12-c06-r5300-4:6667,ss-b02-m12-c06-r5300-6:6667,ss-b02-m12-c06-r5300-7:6667,ss-b02-m12-c06-r5300-9:6667,ss-b02-m12-c06-r5300-8:6667,ss-b02-m12-c06-r5300-10:6667,ss-b02-m12-c07-r5300-1:6667,ss-b02-m12-c07-r5300-2:6667,ss-b02-m12-c07-r5300-3:6667,ss-b02-m12-c07-r5300-4:6667,ss-b02-m12-c07-r5300-7:6667,ss-b02-m12-c07-r5300-6:6667,ss-b02-m12-c07-r5300-9:6667,ss-b02-m12-c07-r5300-8:6667,ss-b02-m12-c07-r5300-5:6667 dhx-select-phone-all.sources.kafkaSource.topic=YZ_TD_YUNMAS_ALL dhx-select-phone-all.sources.kafkaSource.groupId =dhx-select-phone-all-flume dhx-select-phone-all.sources.kafkaSource.kafka.consumer.timeout.ms = 100 dhx-select-phone-all.channels.memoryChannel.type = memory dhx-select-phone-all.channels.memoryChannel.keep-alive=60 dhx-select-phone-all.channels.memoryChannel.capacity=5000000 dhx-select-phone-all.channels.memoryChannel.transactionCapacity=5000000 dhx-select-phone-all.channels.memoryChannel.byteCapacityBufferPercentage = 20 dhx-select-phone-all.channels.memoryChannel.byteCapacity = 536870912 dhx-select-phone-all.sinks.fileSink.type = com.teradata.bigdata.flume.FileSink dhx-select-phone-all.sinks.fileSink.file.path = /data/iop/projects/data/receive/%{dayStr} dhx-select-phone-all.sinks.fileSink.file.filePrefix = dangjian_ dhx-select-phone-all.sinks.fileSink.file.extension = .AVL dhx-select-phone-all.sinks.fileSink.file.rollInterval = 20 dhx-select-phone-all.sinks.fileSink.file.txnEventMax = 100000 dhx-select-phone-all.sinks.fileSink.file.maxOpenFiles = 20 ``` --- ## 一经逍遥 **工程名：carefree-application** ###### 启动脚本 ``` ```