Journal of Civil Aviation University of China ›› 2021, Vol. 39 ›› Issue (6): 14-19.

• Civil Aviation • Previous Articles     Next Articles

Research of Spark dynamic resource scheduling strategy in heterogeneous environment

WU Renbiao, LIU Bei, JIA Yunfei   

  1. (Intelligent Signal and Image Processing Key Lab of Tianjin, Civil Aviation University of China, Tianjin 300300, China) 
  • Received:2021-03-03 Revised:2021-03-03 Accepted:2021-01-22 Online:2021-12-01 Published:2022-06-05

Abstract: Aiming at the problem that the Spark does not consider the performance differences of nodes under heterogeneous cluster, which leads to unbalanced node load in multi-job scenarios and reduces job execution efficiency, a Spark dynamic resource scheduling strategy based on the load of heterogeneous cluster nodes is proposed. First, according to the static and dynamic load information of the node at runtime, the heterogeneous cluster nodes load evaluation index is established to obtain the real -time load of the node during cluster operation. Then, the resource manager Standalone of Spark is modified to improve the sequencing strategy. Finally, the dynamic resource scheduling strategy based on heterogeneous cluster node load is implemented. The experimental results show that compared with the default scheduling strategy,the proposed strategy effectively mitigate the load unbalanceness among heterogeneous cluster nodes and improves the job execution efficiency.

Key words: Spark, heterogeneous cluster, load balancing, resource scheduling, Standalone

CLC Number: