Backup and Maintenance

Daily operations: backups, log management, service upgrades, and troubleshooting.

Database Backup

Manual Backup

# Dump all SlaunchX databases
docker exec slaunchx-mysql-test mysqldump -uroot -p<root-password> \
  --databases slaunchx slaunchx_tron_wallet slaunchx_solana_wallet \
  --single-transaction --quick --routines --triggers \
  > /home/backups/mysql-$(date +%Y%m%d-%H%M%S).sql

# Verify backup integrity
tail -1 /home/backups/mysql-*.sql
# Expected: "-- Dump completed on ..."

Automated Daily Backup (Cron)

cat > /opt/slaunchx/scripts/backup-mysql.sh << 'EOF'
#!/bin/bash
BACKUP_DIR=/home/backups/mysql
RETENTION_DAYS=14
TIMESTAMP=$(date +%Y%m%d-%H%M%S)

mkdir -p $BACKUP_DIR

docker exec slaunchx-mysql-test mysqldump -uroot -p<root-password> \
  --databases slaunchx slaunchx_tron_wallet slaunchx_solana_wallet \
  --single-transaction --quick --routines --triggers \
  > "$BACKUP_DIR/backup-$TIMESTAMP.sql"

# Verify
if [ $? -eq 0 ] && [ -s "$BACKUP_DIR/backup-$TIMESTAMP.sql" ]; then
  sha256sum "$BACKUP_DIR/backup-$TIMESTAMP.sql" > "$BACKUP_DIR/backup-$TIMESTAMP.sha256"
  echo "Backup successful: backup-$TIMESTAMP.sql"
else
  echo "ERROR: Backup failed!" >&2
  exit 1
fi

# Cleanup old backups
find $BACKUP_DIR -name "backup-*.sql" -mtime +$RETENTION_DAYS -delete
find $BACKUP_DIR -name "backup-*.sha256" -mtime +$RETENTION_DAYS -delete
EOF

chmod +x /opt/slaunchx/scripts/backup-mysql.sh

# Schedule: daily at 03:40 UTC
crontab -e
# Add: 40 3 * * * /opt/slaunchx/scripts/backup-mysql.sh >> /var/log/slaunchx-backup.log 2>&1

Restore from Backup

docker exec -i slaunchx-mysql-test mysql -uroot -p<root-password> < /home/backups/mysql/backup-YYYYMMDD-HHMMSS.sql

Redis Backup

Redis with AOF persistence (--appendonly yes) recovers automatically on restart. For explicit backups:

# Trigger RDB snapshot
docker exec slaunchx-redis-test redis-cli -a <password> BGSAVE

# Copy the dump file
docker cp slaunchx-redis-test:/data/dump.rdb /home/backups/redis-$(date +%Y%m%d).rdb

MinIO Backup

# Mirror all buckets
docker exec slaunchx-minio-test mc mirror local/slaunchx /backup/minio/

Log Management

Docker Log Rotation

All containers are started with --log-opt max-size=50m --log-opt max-file=3, which limits each container to 150 MB of logs (3 × 50 MB files, auto-rotated).

Viewing Logs

# Recent logs
docker logs --tail 200 slaunchx-app-prometheus-test

# Logs since a timestamp
docker logs --since "2026-03-23T10:00:00" slaunchx-app-prometheus-test

# Follow real-time
docker logs -f slaunchx-app-prometheus-test

# Filter errors
docker logs slaunchx-app-prometheus-test 2>&1 | grep -iE "error|exception|fatal" | tail -30

Log Locations

Docker stores logs at:

{docker-data-root}/containers/{container-id}/{container-id}-json.log

Find the data root with docker info | grep "Docker Root Dir".

Service Upgrade

Standard Upgrade Procedure

MODULE=app-prometheus
ENV=test
PORT=18020
IMAGE=localhost:5000/slaunchx/$MODULE:dev-latest

# 1. Pull new image
docker pull $IMAGE

# 2. Stop and remove old container
docker stop slaunchx-$MODULE-$ENV
docker rm slaunchx-$MODULE-$ENV

# 3. Start new container (same command as initial deploy)
docker run -d \
  --name slaunchx-$MODULE-$ENV \
  --network slaunchx-intra \
  -p 0.0.0.0:$PORT:$PORT \
  --env-file /opt/slaunchx/config/$MODULE/$ENV.env \
  -e SPRING_PROFILES_ACTIVE=$ENV \
  --log-opt max-size=50m --log-opt max-file=3 \
  --restart unless-stopped \
  $IMAGE

# 4. Verify health
sleep 15  # wait for Spring Boot startup
curl -s http://localhost:$PORT/prometheus/actuator/health

Using the Deploy Script

# Upgrade a single module (handles stop/rm/pull/run)
ci/local/deploy.sh test app-prometheus

# Upgrade all modules
ci/local/deploy.sh test

Database Migration on Upgrade

Flyway runs automatically on startup. If new migration files are included in the new image:

1. Flyway detects them and applies in order
2. If migration fails, the container exits — check logs
3. Do NOT manually patch the database to work around a failed migration

# Check Flyway history
docker exec -i slaunchx-mysql-test mysql -uslaunchx -p<password> slaunchx \
  -e "SELECT version, description, success FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 10;"

Registry Cleanup

Old Docker images accumulate in the private registry. Clean periodically:

# Run garbage collection
docker exec slaunchx-registry bin/registry garbage-collect \
  /etc/docker/registry/config.yml --delete-untagged

# Check registry disk usage
du -sh /home/registry/data/

Troubleshooting

Container Won't Start

# Check exit code and logs
docker inspect slaunchx-app-prometheus-test --format '{{.State.ExitCode}}'
docker logs --tail 50 slaunchx-app-prometheus-test

Exit Code	Meaning
0	Normal shutdown
1	Application error (check logs)
137	OOM killed (increase -Xmx or host RAM)
143	SIGTERM (normal docker stop)

MySQL Connection Issues

# Test from inside the app container
docker exec -it slaunchx-app-prometheus-test bash
# Then: curl -s slaunchx-mysql-test:3306 || echo "Cannot reach MySQL"

# Test credentials
docker exec -it slaunchx-mysql-test mysql -uslaunchx -p<password> -e "SELECT 1"

Redis Connection Issues

docker exec -it slaunchx-redis-test redis-cli -a <password> INFO server | head -5

RabbitMQ Queue Buildup

# Check queue depths
curl -s -u slaunchx:<password> http://localhost:18672/api/queues | \
  python3 -c "import sys,json; [print(f'{q[\"name\"]}: {q[\"messages\"]}') for q in json.load(sys.stdin)]"

Disk Space

# Docker disk usage
docker system df

# Reclaim unused resources (careful in production)
docker system prune --volumes  # removes stopped containers, unused images/volumes