This project implements a data deduplication algorithm [1] and uses it with CCNx FLIC Manifests.
It is pretty alpha code and still in experimental stages.
[1] Xia, Wen, Xiangyu Zou, Hong Jiang, Yukun Zhou, Chuanyi Liu, Dan Feng, Yu Hua, Yuchong Hu, and Yucheng Zhang. "The design of fast content-defined chunking for data deduplication based storage systems." IEEE Transactions on Parallel and Distributed Systems 31, no. 9 (2020): 2017-2031. https://ranger.uta.edu/~jiang/publication/Journals/2020/2020-IEEE-TPDS(Wen%20Xia).pdf
Make sure you have python3-dev installed, which is needed by spookyhash.
poetry run deduper