Description
1. daily backup:
problem:
Since the new deployment, our automated daily backups for solr, mongo, and db have been consistently failing (at least for xces, for codes only worked for db). The cron logs show the following error for each backup:
/etc/cron.daily/backup_xces-db:
/usr/local/bin/daily-backup: 2: set: Illegal option -o pipefail
/etc/cron.daily/backup_xces-mongo:
/usr/local/bin/daily-backup: 2: set: Illegal option -o pipefail
/etc/cron.daily/backup_xces-solr:
/usr/local/bin/daily-backup: 2: set: Illegal option -o pipefailthe cron file (generated via here pressumably?), would go like:
[root@xces9 cron.daily]# cat backup_xces-db
#!/bin/sh
# Run daily backup for xces-db
MAILTO=xxx
export PATH="/sbin:/bin:/usr/sbin:/usr/bin"
/bin/docker exec --user=YYY xces-db /usr/local/bin/daily-backup -s /data/db -b /backup 1> /dev/nullwhil the backup script run inside the container would have the followign header:
#!/usr/bin/env bash
set -o nounset -o pipefail -o errexit
...test:
When I execute the cron scripts interactively (or line by line) the backup is done correctly. Could it be that the root interactive shell being bash passes a ritcher environment to docker exec while the cron env does not?
proposed solution:
Manually changing the line from
/bin/docker exec --user=XX xces-db /usr/local/bin/daily-backup -s /data/db -b /backup 1> /dev/null
to
/bin/docker exec --user=XX xces-db /usr/bin/env bash /usr/local/bin/daily-backup -s /data/db -b /backup 1> /dev/null
or
/bin/docker exec --user=XX xces-db bash /usr/local/bin/daily-backup -s /data/db -b /backup 1> /dev/null
(partially) solved the problem (see below)
I need to see how is this stablished in the deployment files
2. backup folders
problem:
- while the mysql db backup end up in .e.g
/var/lib/docker/volumes/xces-db_backup/_data/backup-20250610_081313.tar.gzfolderpath - the one for mongo ends up in
/var/lib/docker/overlay2/610a6e442a32b0c31a3c6026b734200531acc940f23ef2a27bfcbc389d4880e7/diff/backup/backup-20250610_084735.tar.gz
/var/lib/docker/overlay2/610a6e442a32b0c31a3c6026b734200531acc940f23ef2a27bfcbc389d4880e7/merged/backup/backup-20250610_084735.tar.gz
- the one for solr in :
/var/lib/docker/overlay2/ef684a48d73089aec898bf3f33b497b93220fb2accf1791195e393e68fd64e8f/diff/backup/backup-20250610_085429.tar.gz
/var/lib/docker/overlay2/ef684a48d73089aec898bf3f33b497b93220fb2accf1791195e393e68fd64e8f/merged/backup/backup-20250610_085429.tar.gz
Looking at the compose files e.g. /opt/freva/xces/compose_services/xces-db-compose.yml I am unable to understand how the docker container is binding the folders to the outside at e.g. /var/lib/docker/volumes/xces-db_backup/_data/:
version: '3'
services:
xces-db:
user: 200447:1076
image: ghcr.io/freva-clint/freva-mysql:9.3.0
hostname: xces-db
environment:
- ROOT_PW=XXX
- HOST=XXX
- NUM_BACKUPS=7
- PROJECT=XX
- MYSQL_USER=XXXX
- MYSQL_PASSWORD=XX
- MYSQL_DATABASE=XXX
- MYSQL_ROOT_PASSWORD=XXXX
volumes:
- data:/data/db:z
- logs:/data/logs:z
- backup:/backup:z
container_name: xces-db
tty: true
ports:
- X
volumes:
data:
backup:
logs:proposed solution:
I will need to check how these backup scripts are created