| 周宇泽,司鹏搏,张延华,李萌,杨睿哲.基于时间序列模型的Kafka系统智能化管理方法[J].高技术通讯(中文),2023,33(10):1047~1059 |
| 基于时间序列模型的Kafka系统智能化管理方法 |
| Intelligent management method of Kafka system based on time series model |
| |
| DOI：10. 3772/ j. issn. 1002-0470. 2023. 10. 005 |
| 中文关键词: 区块链， Kafka， 长短期记忆网络(LSTM)， 时间序列模型 |
| 英文关键词: blockchain, Kafka, long short term memory (LSTM), time series model |
| 基金项目: |
| 作者 | 单位 | | 周宇泽 | （北京工业大学信息学部北京 100124） | | 司鹏搏 | | | 张延华 | | | 李萌 | | | 杨睿哲 | |
|
| 摘要点击次数: 576 |
| 全文下载次数: 465 |
| 中文摘要: |
| 区块链是分布式的数据存储系统，共识算法为区块链实现安全存储数据提供支撑和保障。Kafka作为共识算法中的一种，其高吞吐速率、低时延的特点受到青睐。但使用Kafka算法的系统接受大量交易时，易产生数据倾斜，即分布式系统的多节点结构中，大量数据集中在少数节点，导致系统资源被占用、性能下降。为解决上述问题，本文提出基于时间序列模型长短期记忆网络(LSTM)的智能优化方法。通过学习过往生产者接收到的交易量，预测下一时刻面临的交易量，动态调整生产者节点数量，减少数据集中在少数节点的情况。实验结果显示，本文方法可以将Kafka系统时延降低2~3倍，吞吐速率提升2~3倍，与优化前相比系统效率提升52.62%，比2种传统优化方法分别提升近3%和40%，能耗仅小幅提升，系统使用情况保持更加合理。 |
| 英文摘要: |
| Blockchain is a distributed data storage system. Consensus algorithm can support and guarantee blockchain to achieve safe data storage. Kafka, as one of the consensus algorithms, is favored for its high throughput rate and low delay. However, when a system using the Kafka algorithm accepts a large number of transactions, it is prone to data skew, that is, in the multi-node structure of a distributed system, a large amount of data is concentrated on a few nodes, which leads to the occupation of system resources and performance degradation. To solve these problems, an intelligent optimization method based on long short term memory (LSTM) is proposed. By learning the trading volume received by producers in the past, the trading volume faced at the next moment can be predicted, and the number of producer nodes can be dynamically adjusted to reduce the data concentration in a few nodes. The experimental results show that the proposed method can reduce the delay of Kafka system by 2-3 times, increase the throughput rate by 2-3 times, and improve the system efficiency by 52.62% compared with the original Kafka system, which is nearly 3% and 40% higher than the two traditional optimization methods, respectively. The energy consumption is only slightly improved, and the system usage remains more reasonable. |
|
查看全文
查看/发表评论 下载PDF阅读器 |
| 关闭 |
|
|
|
