Deduplication fixed-length block method based on characteristics of civil aviation data

Abstract

Abstract: Aiming at the problem of data duplication in the storage of civil aviation data during disaster recovery and back-up. A deduplication fixed-length block method based on the characteristics of civil aviation data is proposed. According to the consistency of civil aviation data types, this method combines the advantages of fixed-length block and variable-length block, and designs a data structure of the block strategy index table. A block strategy is provided for the same type of data, and time is saved to find the boundary of the data block during block division, the simulated data deduplication rate during backup increases to 97.8%~99.3%, which is 11.8%~12.5% higher than that of the fixed-length block method, and 2.5%~3.0% higher than that of the variable length block method; at the same time, a new block strategy is established for new data types to facilitate subsequent data stream matching and improve hit accuracy.

Key words: civil aviation data, disaster recovery backup, data deduplication, type consistency, block strategy, simulated deduplication rate

CLC Number:

DING Jianli, LI Hui, CAO Weidong. Deduplication fixed-length block method based on characteristics of civil aviation data[J]. Journal of Civil Aviation University of China, 2022, 40(4): 32-37.

References 17

[1]	王龙翔, 董凯, 王鹏博, 等. R-dedup: 一种重复数据删除指纹计算的优化方法[J]. 西安交通大学学报, 2021, 55(1): 43-51.
[2]	何柯文, 张佳辰, 刘晓光, 等. 新型存储设备上重复数据删除指纹查找优化[J]. 计算机研究与发展, 2020, 57(2): 269-280.
[3]	LILLIBRIDGE M, ESHGHI K, BHAGWAT D, et al Sparse indexing: large scale , inline deduplication using sampling and locality [C]//7th USENIX Conference on File and Storage Technologies, San Francisco , CA, USA, February 24-27, 2009: 111-123.
[4]	DEBNATH B, SENGUPTA S, LI J. ChunkStash: speeding up inline storage deduplication using flash memory[C]//2010 USENIX Annual Technical Conference,June 23-25, 2010, Boston, MA, USA. CA: USENIX Assocation, 2010.
[5]	CAO Z, WEN H, WU F, et al ALACC: accelerating restore performance of data deduplication systems using adaptive look-ahead window assist－ ed chunk caching[C]//16th USENIX Conference on File and Storage Technologies (FAST′18), February 12-15, 2018, Oakland, CA, USA. CA: USENIX Assocation, 2018: 309-324.
[6]	廖海生. 基于重复数据删除技术的数据容灾系统的研究[D]. 广州: 华南理工大学, 2009.
[7]	张甲燃. 重复数据删除技术研究[D]. 济南: 山东大学, 2013.
[8]	韩英杰. 数据容灾系统重复数据删除关键技术研究 [D]. 长春: 长春理工大学, 2018.
[9]	阎芳. 重复数据删除系统数据组织研究[D]. 北京: 北京理工大学, 2015.
[10]	WANG G H, ZHAO Y L, LI T X, et al A method for organizing metada－ ta of storage nodes with data de-duplication[J]. Journal of Communica－ tional Information Systems, 2014, 10(9): 3845-3854.
[11]	贺秦禄, 边根庆, 邵必林, 等. 移动闪存的重复数据删除技术[J]. 西安电子科技大学学报, 2020, 47(1): 128-134.
[12]	XIA W, JIANG H, FENG D, et al Silo: a similarity -locality based near -exact deduplication scheme with low ram overhead and high throughput[C]//2011 USENIX Annual Technical Conference, June 15, 2011, Portland, OR , USA. CA: USENIX Assocation, 2011: 26-28.
[13]	DANG Q L, XIE Y, LI D H, et al Updatable block-level deduplication of encrypted data with efficient auditing in cloud storage[J]. The Journal of China Universities of Posts and Telecommunications, 2019(3): 56-72.
[14]	BHAGWAT D, ESHGHI K, LONG D D E, et al Extreme Binning: scal－ able, parallel deduplication for chunk-based file backup[C]//2009 IEEE International Symposium on Modeling, Analysis & Simulation of Com－ puter and Telecommunication Systems, September 21-23, 2009, Lon－ don, UK. IEEE, 2009.
[15]	WEN X, ZHOU Y K, JIANG H, et al FastCDC: a fast and efficient con－ tent-defined chunking approach for data deduplication[C]//2016 USENIX Annual Technical Conferenc, June 22-24, 2016, Denver, CO, USA. CA: USENIX Assocation, 2016.
[16]	DENG M Z, CHEN W, XIAO N, et al GLE-dedup: a globally-locally even deduplication by request-aware placement for better read perfor－ mance[J]. International Journal of Parallel Programming, 2017, 45(4): 946-964.
[17]	WU F H, CHEN J, DONG Y, et al A holistic energy-efficient approach for a processor -memory system [J]. Tsinghua Science and Technology, 2019, 24(4): 468-483.