feat(backup): Improve dedup algorithm to work with old backup #1077
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Dedup in photo backup means that we do not upload pictures if they already exists in the Cozy. To identify if two pictures are identical, we compare name and creation date. For new backup, we compare with our own creation date added in the io.cozy.files metadata. It just works. But in photo uploaded by old backup, we compare with the creation date that has been taken from EXIF and can where the timezone could have been badly managed. So if we compare stricly date, it may not work. So here we compare only part of the date in dedup mode by ignoring the "hour" field : - its almost impossible to have a false identity by just ignoring the "hour" field => OK - we can miss some identity => but we accept that our dedup is not 100% accurate
Crash--
approved these changes
Dec 15, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dedup in photo backup means that we do not upload pictures if they already exists in the Cozy.
To identify if two pictures are identical, we compare name and creation date.
For new backup, we compare with our own creation date added in the io.cozy.files metadata. It just works.
But in photo uploaded by old backup, we compare with the creation date that has been taken from EXIF and can where the timezone could have been badly managed. So if we compare stricly date, it may not work.
So here we compare only part of the date in dedup mode by ignoring the "hour" field :