异构环境下 Spark 动态资源调度策略研究

中国民航大学学报 ›› 2021, Vol. 39 ›› Issue (6): 14-19.

异构环境下 Spark 动态资源调度策略研究

吴仁彪，刘备，贾云飞

（中国民航大学天津市智能信号与图像处理重点实验室，天津 300300）

收稿日期:2021-03-03 修回日期:2021-03-03 接受日期:2021-01-22 出版日期:2021-12-01 发布日期:2022-06-05
作者简介:吴仁彪（1966—），男，湖北武汉人，教授，博士，研究方向为自适应信号处理，现代信号谱分析及其在雷达、卫星导航、空管中的应用.

Research of Spark dynamic resource scheduling strategy in heterogeneous environment

WU Renbiao, LIU Bei, JIA Yunfei

（Intelligent Signal and Image Processing Key Lab of Tianjin, Civil Aviation University of China, Tianjin 300300, China）

Received:2021-03-03 Revised:2021-03-03 Accepted:2021-01-22 Online:2021-12-01 Published:2022-06-05

摘要/Abstract

摘要： 针对 Spark 未考虑异构集群节点间的性能差异，导致多作业场景下节点负载不均衡、作业执行效率低的问题，提出一种基于异构集群节点负载的 Spark 动态资源调度策略。根据节点静态负载和运行时的动态负载信息建立异构集群节点负载评价指标，获取集群运行时节点的实时负载；改进 Spark 默认资源管理器 Standalone 在资源分配时的节点排序规则；最终实现基于异构集群节点负载的动态资源调度策略。实验结果表明，与默认调度策略相比，该策略评价指标适用性更好，有效缓解了集群的负载失衡问题，提高了作业执行效率。

关键词: Spark, 异构集群, 负载均衡, 资源调度, Standalone

Abstract: Aiming at the problem that the Spark does not consider the performance differences of nodes under heterogeneous cluster, which leads to unbalanced node load in multi-job scenarios and reduces job execution efficiency, a Spark dynamic resource scheduling strategy based on the load of heterogeneous cluster nodes is proposed. First, according to the static and dynamic load information of the node at runtime, the heterogeneous cluster nodes load evaluation index is established to obtain the real -time load of the node during cluster operation. Then, the resource manager Standalone of Spark is modified to improve the sequencing strategy. Finally, the dynamic resource scheduling strategy based on heterogeneous cluster node load is implemented. The experimental results show that compared with the default scheduling strategy，the proposed strategy effectively mitigate the load unbalanceness among heterogeneous cluster nodes and improves the job execution efficiency.

Key words: Spark, heterogeneous cluster, load balancing, resource scheduling, Standalone

中图分类号:

TP302

吴仁彪, 刘备, 贾云飞. 异构环境下 Spark 动态资源调度策略研究[J]. 中国民航大学学报, 2021, 39(6): 14-19.

WU Renbiao, LIU Bei, JIA Yunfei. Research of Spark dynamic resource scheduling strategy in heterogeneous environment[J]. Journal of Civil Aviation University of China, 2021, 39(6): 14-19.

参考文献 14

[1]	MAVRIDIS I, KARATZA H. Performance evaluation of cloud-based log file analysis with Apache Hadoop and Apache Spark[J]. Journal of Systems and Software, 2017, 125: 133-151.
[2]	杨志伟, 郑烇, 王嵩, 等. 异构 Spark 集群下自适应任务调度策略[J]. 计算机工程, 2016, 42(1): 31-35.
[3]	CHENG D Z, ZHOU X B, XU Y G, et al. Deadline-aware MapReduce job scheduling with dynamic resource availability[J]. IEEE Transactions on Parallel and Distributed Systems, 2019, 30(4): 814-826.
[4]	冯兴杰, 贺阳. 基于节点性能的 Hadoop 作业调度算法改进[J]. 计算机应用与软件, 2017, 34(5): 223-228.
[5]	胡亚红, 盛夏, 毛家发. 资源不均衡 Spark 环境任务调度优化算法研究[J]. 计算机工程与科学, 2020, 42(2): 203-209.
[6]	KEC ，O D, SUBASI A, KEVRIC J. Cloud computing-based parallel genetic algorithm for gene selection in cancer classification[J]. Neural Computing and Applications, 2018, 30(5): 1601-1610．
[7]	赖向阳, 宫秀军, 韩来明. 一种 MapReduce 架构下基于遗传算法的 K-Medoids 聚类[J]. 计算机科学, 2017, 44(3): 23-26.[8] MASDARI M, SALEHI F, JALALI M, et al. A survey of PSO-based scheduling algorithms in cloud computing[J]. Journal of Network and Systems Management, 2017, 25(1): 122-158.
[9]	YUAN H, LI C B, DU M K. Optimal virtual machine resources scheduling based on improved particle swarm optimization in cloud computing [J]. Journal of Software, 2014, 9(3): 705-708.
[10]	KUSHWAH V S, GOYAL S K. A basic simulation of ACO algorithm under cloud computing for fault tolerant[C]//Proceedings of the International Conference on Data Engineering and Communication Technology(ICDECT 2016). Singapore: Springer, 2017: 465-472.
[11]	RAGMANI A, ELOMRI A, ABGHOUR N, et al. A performed load balancing algorithm for public cloud computing using ant colony optimization[C]//2016 2nd International Conference on Cloud Computing Technologies and Applications(CloudTech), May 24-26, 2016, Marrakech, Morocco. IEEE, 2016: 221-228.
[12]	YADAV R, ZHANG W Z, KAIWARTYA O, et al. Adaptive energyaware algorithms for minimizing energy consumption and SLA violation in cloud computing[J]. IEEE Access, 2018, 6: 55923-55936.
[13]	GROZEV N, BUYYA R. Inter-Cloud architectures and application brokering: taxonomy and survey[J]. Software: Practice and Experience, 2014, 44(3): 369-390
[14]	CHAMBERS B, ZAHARIA M. Spark 权威指南[M]. 张岩峰，王方京，陈晶晶, 译. 北京: 中国电力出版社, 2020