基于Gossip协议的高效集群数据同步方案

张宏海; 崔斌豪; 李一鸣; 田丰; 贾永强; 肖傲三

doi:10.13700/j.bh.1001-5965.2024.0750

基于Gossip协议的高效集群数据同步方案

doi: 10.13700/j.bh.1001-5965.2024.0750

张宏海^{1, 2, ,},
崔斌豪^{1, 2},
李一鸣^{1, 2},
田丰^{1, 2},
贾永强^{1, 2},
肖傲三^{1, 2}

1.
中国民航信息网络股份有限公司研发中心，北京 101318
2.
中国民航信息网络股份有限公司北京市民航大数据工程技术研究中心，北京 101318

基金项目:

国家自然科学基金(U2033203)

详细信息

通讯作者:
E-mail：hhzhang@travelsky.com.cn

中图分类号: TP393
计量
- 文章访问数: 134
- HTML全文浏览量: 22
- PDF下载量: 8
- 被引次数: 0
出版历程
- 收稿日期: 2024-10-16
- 录用日期: 2024-11-08
- 网络出版日期: 2024-11-19
- 整期出版日期: 2025-05-31

An efficient cluster data synchronization scheme based on the Gossip protocol

ZHANG Honghai^{1, 2
, ,},
CUI Binhao^{1, 2},
LI Yiming^{1, 2},
TIAN Feng^{1, 2},
JIA Yongqiang^{1, 2},
XIAO Aosan^{1, 2}

1.
Research and Development Center，TravelSky Technology Limited，Beijing 101318，China
2.
Beijing Engineering Technology Research Center of Civil Aviation Big Data，TravelSky Technology Limited，Beijing 101318，China

Funds:

National Natural Science Foundation of China (U2033203)

More Information

Corresponding author: E-mail：hhzhang@travelsky.com.cn

摘要

摘要:
随着民航客票运价搜索系统业务的快速发展，系统集群内部网络流量规模不断增长。为了在高负载网络流量场景下，实现集群内部数据同步，提出基于Gossip协议的集群数据同步方案。所提方案从网络协议的传输层和应用层着手设计，在传输层使用用户数据报协议（UDP）来减少集群中节点间的连接数量和交互次数，从而实现数据在网络传输过程中的低流量、低耗时。在应用层使用Gossip传播协议来实现数据的最终一致性，保证数据传输的可靠性。通过传输层UDP传输协议和应用层Gossip传播协议相结合，保证了集群监控过程中数据同步的高效性和可靠性。
- Gossip协议 /
- 数据同步 /
- 一致性算法 /
- 高性能集群 /
- 去中心化
Abstract:
With the rapid development of the civil aviation ticket fare search system business, the internal network traffic scale of the search system cluster continues to grow. Under high-load network traffic, a cluster data synchronization scheme based on the Gossip protocol was developed. This solution is designed from the transport layer and application layer of the network protocol and uses the user datagram protocol (UDP) transmission protocol in the transport layer to reduce the number of connections and interactions between cluster nodes in order to achieve low traffic and low data consumption during network transmission. At the application layer, the Gossip propagation protocol is used to achieve eventual consistency of data and ensure the reliability of data transmission. In the cluster monitoring process, the effectiveness and dependability of data synchronization are guaranteed by the combination of the Gossip propagation protocol at the application layer and the UDP transmission protocol at the transport layer.
- gossip protocol /
- data synchronization /
- consistency algorithm /
- high-performance cluster /
- decentralization

HTML全文

图 1 Gossip协议的通信方式

Figure 1. Communication method of Gossip protocol

下载: 全尺寸图片幻灯片

图 2 系统组成逻辑

Figure 2. System composition logic

下载: 全尺寸图片幻灯片

图 3 自研传播协议时序图

Figure 3. Timing diagram of self-developed propagation protocol

下载: 全尺寸图片幻灯片

图 4 网络负载率对比实验

Figure 4. Comparison experiment of network load rate

下载: 全尺寸图片幻灯片

图 5 业务服务耗时对比实验

Figure 5. Comparison experiment of business service time consumption

下载: 全尺寸图片幻灯片

表 1 验证环境配置

Table 1. Verify environment configuration

参数	配置
编程语言	Java
操作系统	Linux Redhat v6.5
主机类型	X86服务器
CPU核数	16Core
内存	512 GB
硬盘	3T SSD
集群网络	万兆局域网
Node数量	100台

下载: 导出CSV

表 2 Gossip协议关键评估指标

Table 2. Gossip Protocol Key Performance Metrics

轮次	一致概率/%	收敛周期/轮	容错率/%
1 000	98.8	3.2	99.8
2 000	99.9	2.9	100
3 000	100	2.8	99.9
4 000	99.9	3.1	98.7
5 000	99.6	2.2	99.3
6 000	99.7	2.4	98.9
7 000	100	2.5	98.8
8 000	100	3.0	100
9 000	99.9	2.7	99.9
10 000	100	2.6	100

下载: 导出CSV

参考文献(19)

[1]	张宏海, 彭明田, 刘开胜. 基于价格导向的民航运价搜索方案设计[J]. 中国民航大学学报, 2020, 38(3): 40-44. doi: 10.3969/j.issn.1674-5590.2020.03.008 ZHANG H H, PENG M T, LIU K S. Design of price-based airfare search method[J]. Journal of Civil Aviation University of China, 2020, 38(3): 40-44(in Chinese). doi: 10.3969/j.issn.1674-5590.2020.03.008
[2]	王成全. 基于UDP的消息即时传输系统的研究与设计[J]. 信息技术, 2008(7): 162-164. doi: 10.3969/j.issn.1009-2552.2008.07.050 WANG C Q. Study and design of real-time message transmission system based on UDP[J]. Information Technology, 2008(7): 162-164(in Chinese). doi: 10.3969/j.issn.1009-2552.2008.07.050
[3]	刘锦江, 范洪博, 高志伟. 基于Gossip网络协议的前置消息验证[J]. 信息技术, 2022(7): 109-113. LIU J J, FAN H B, GAO Z W. Gossip network protocol-based pre-message authentication[J]. JInformation Technology, 2022(7): 109-113(in Chinese).
[4]	BAKER B, SHOSTAK R. Gossips and telephones[J]. Discrete Mathematics, 1972, 2(3): 191-193. doi: 10.1016/0012-365X(72)90001-5
[5]	HEDETNIEMI S M, HEDETNIEMI S T, LIESTMAN A L. A survey of gossiping and broadcasting in communication networks[J]. Networks, 1988, 18(4): 319-349. doi: 10.1002/net.3230180406
[6]	BOYD S, GHOSH A, PRABHAKAR B, et al. Analysis and optimization of randomized gossip algorithms[C]//Proceedings of the 43rd IEEE Conference on Decision and Control. Piscataway: IEEE Press, 2004: 5310-5315.
[7]	WEATHERSPOON H, MIRANDA H, IWANICKI K, et al. Gossiping over storage systems is practical[J]. ACM SIGOPS Operating Systems Review, 2007: 41(5): 75-81.
[8]	DEMERS A, GREENE D, HOUSER C, et al. Epidemic algorithms for replicated database maintenance[J]. Acm Sigops Operating Systems Review, 1988, 22(1): 8-32. doi: 10.1145/43921.43922
[9]	MARK J, ALBERTO M, OZALP B. T-Man: Gossip-based fast overlay topology construction[J]. Computer Networks, 2009, 53(13): 2257-2258. doi: 10.1016/j.comnet.2009.04.006
[10]	NAOHIRO H, MAKOTO T. Design of the notification system for failure detectors[J]. International Journal of High Performance Computing and Networking, 2009, 6(1): 25-34. doi: 10.1504/IJHPCN.2009.026289
[11]	BOYD S, GHOSH A, PRABHAKAR B, et al. Randomized Gossip algorithms[J]. IEEE Transactions on Information Theory, 2006, 52(6): 2508-2530. doi: 10.1109/TIT.2006.874516
[12]	GIUSEPPE D, DENIZ H, MADAN J, et al. Dynamo: amazon's highly available key-value store[J]. Symposium on Operating Systems Principles, 2007, 41(6): 205-220.
[13]	李明, 贾智平. 基于TCP/IP的分布式监控系统的研究与开发[J]. 计算机工程与应用, 2002, 38(16): 150-153. doi: 10.3321/j.issn:1002-8331.2002.16.052 LI M, JIA Z P. Research and development of distributed monitoring and control system based on TCP/IP protocol[J]. Computer Engineering and Applications, 2002, 38(16): 150-153(in Chinese). doi: 10.3321/j.issn:1002-8331.2002.16.052
[14]	王继刚, 顾国昌, 徐立峰, 等. 可靠UDP数据传输协议的研究与设计[J]. 计算机工程与应用, 2006, 42(15): 113-116. doi: 10.3321/j.issn:1002-8331.2006.15.034 WANG J G, GU G C, XUN L F, et al. Research and design of reliable data transfer based on UDP[J]. Computer Engineering and Applications, 2006, 42(15): 113-116(in Chinese). doi: 10.3321/j.issn:1002-8331.2006.15.034
[15]	刘德辉, 尹刚, 王怀民, 等. 分布环境下的Gossip算法综述[J]. 计算机科学, 2010, 37(11): 24-28. doi: 10.3969/j.issn.1002-137X.2010.11.005 LIU D H, YIN G, WANG H M, et al. Overview of Gossip algorithm in distribute system[J]. Computer Science, 2010, 37(11): 24-28(in Chinese). doi: 10.3969/j.issn.1002-137X.2010.11.005
[16]	张国印, 李军, 王向辉, 等. 一种基于移动P2P改进的Gossip算法[J]. 计算机科学, 2013, 40(9): 103-105. doi: 10.3969/j.issn.1002-137X.2013.09.021 ZHANG G Y, LI J, WANG X H, et al. Improved Gossip algorithm based on mobile P2P networks[J]. Computer Science, 2013, 40(9): 103-105(in Chinese). doi: 10.3969/j.issn.1002-137X.2013.09.021
[17]	李燚, 顾乃杰, 黄增士, 等. Redis集群可靠性的研究与优化[J]. 计算机工程, 2018, 44(5): 40-46. LI Y, GU N J, HUANG Z S, et al. Research and optimization of redis cluster reliability[J]. Computer Engineering, 2018, 44(5): 40-46(in Chinese).
[18]	王峰, 李立新, 曹景源, 等. 发布/订阅系统中的缓存副本一致性研究[J]. 计算机应用, 2016, 36(6): 1510-1514. doi: 10.11772/j.issn.1001-9081.2016.06.1510 WANG F, LI L X, CAO J Y, et al. Research on replication consistency of cache in publish/subscribe systems[J]. Journal of Computer Applications, 2016, 36(6): 1510-1514(in Chinese). doi: 10.11772/j.issn.1001-9081.2016.06.1510
[19]	杨永凯, 彭明田, 王炜东. 集群内高效可靠的数据文件分发方案的设计[J]. 计算机技术与发展, 2019, 29(11): 163-167. doi: 10.3969/j.issn.1673-629X.2019.11.033 YANG Y K, PENG M T, WANG W D. Design of effcient and reliable data file distribution solution in cluster[J]. Journal of Computer Applications, 2019, 29(11): 163-167(in Chinese). doi: 10.3969/j.issn.1673-629X.2019.11.033