Closed
Description
When running clickhouse-backup server --watch with backups_to_keep_local: -1 there are duplicate attempts to delete the local backup and the second one fails due to the first one successfully removing the local backup.
Logs:
2025-05-30 02:22:40.671 INF pkg/backup/upload.go:256 > done backup=2025053002-shard0-full duration=14.038s object_disk_size=0B operation=upload upload_size=71.15MiB version=2.6.18
2025-05-30 02:22:40.869 INF pkg/storage/general.go:163 > list_duration=180.019109
2025-05-30 02:22:40.869 INF pkg/backup/delete.go:157 > remove '/var/lib/clickhouse/backup/2025053002-shard0-full'
2025-05-30 02:22:41.013 INF pkg/backup/delete.go:166 > done backup=2025053002-shard0-full duration=334ms location=local operation=delete
2025-05-30 02:22:41.023 ERR pkg/server/metrics/metrics.go:246 > metrics.ExecuteWithMetrics(delete) return error: '2025053002-shard0-full' is not found on local storage
2025-05-30 02:22:41.024 ERR pkg/server/server.go:618 > Watch error: too many errors create_remote: 0, delete local: 1, during watch full_interval: 168h, abort watchingThe first delete is successful here as part of the upload process but then it tries to delete it again here which fails
I think you could alter the first delete to:
if err = b.RemoveOldBackupsLocal(ctx, !deleteSource, nil); err != nil {
return fmt.Errorf("can't remove old local backups: %v", err)
}This would keep the most recent backup from the during the upload step which would then be deleted in the Watch function. Alternatively, you could stop the delete backup from throwing an error if the backup doesn't exist locally
Metadata
Metadata
Assignees
Labels
No labels