One of my LinkedIn connections, Karan, recently asked me an important question:
“When using Docker to build and deploy an application running on Golang and MySQL, what are the best practices for handling database changes in follow-up deployments? Specifically, when deploying version 2 of the app, how can we ensure that existing data is preserved and only incremental changes are applied to the database?”
In this blog post, we’ll walk through best practices for handling database schema changes in Dockerized Go applications. By following these steps, you can ensure a secure and efficient deployment process.
1. Choose a database migration tool
To manage incremental changes to your database schema, using a migration tool is a best practice. These tools help you apply and track changes to your database in a structured manner. Here are a few popular choices:
- Go-Migrate: A Go-specific tool ideal for Golang applications.
- Flyway: A versatile tool that supports various databases.
- Liquibase: Another robust option for managing database versions.
For this guide, we’ll use Go-Migrate as it integrates well with Go applications.
2. Create migration scripts
Migration scripts define how to update your database schema from one version to the next. Here’s how to create and manage these scripts:
- Create a directory for migrations:
mkdir migrations2. Write migration files:
For example, to add a users table, create the following migration files:
001_add_users_table.up.sql:
CREATE TABLE users ( id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(100), email VARCHAR(100) UNIQUE );001_add_users_table.down.sql:
DROP TABLE users;The .up.sql file defines the changes to apply, while the .down.sql file allows you to roll back those changes if needed.
3. Integrate migration into your Docker workflow
To ensure your database schema is updated before your application starts, follow these steps:
A. Add migration tool to your Docker image
Modify your Dockerfile to install Go-Migrate:
# Install Go-Migrate RUN wget https://github.com/golang-migrate/migrate/releases/download/v3.9.2/migrate.linux-amd64.tar.gz && \ tar -xzf migrate.linux-amd64.tar.gz && \ mv migrate /usr/local/bin/B. Include migration files
Add your migration scripts to the Docker image:
COPY migrations /migrationsC. Run migrations in Docker entry point
Update your entry point script to apply migrations before starting the application:
docker-entrypoint.sh:
#!/bin/sh echo "Running database migrations..." migrate -path=/migrations -database="${DATABASE_URL}" up echo "Starting the application..." exec "$@"Ensure your Dockerfile uses this entry point:
COPY docker-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["docker-entrypoint.sh"]
CMD ["your-app-binary"]4. Implement backup and rollback strategies
A. Backup your database regularly
Automate database backups using tools or scripts suited to your MySQL setup to prevent data loss. To automate database backups, you can use a cron job or a scheduled task depending on your environment. Here’s a basic example using a cron job in a Docker container:
Dockerfile for Backup Container:
FROM mysql:latest
COPY backup.sh /usr/local/bin/backup.sh
RUN chmod +x /usr/local/bin/backup.sh
# Install cron and create cron job
RUN apt-get update && apt-get install -y cron
COPY my-cron-file /etc/cron.d/my-cron-file
RUN chmod 0644 /etc/cron.d/my-cron-file
RUN crontab /etc/cron.d/my-cron-file
CMD ["cron", "-f"]backup.sh:
#!/bin/sh # Variables BACKUP_DIR="/backups" DATE=$(date +'%Y%m%d%H%M') BACKUP_FILE="$BACKUP_DIR/db_backup_$DATE.sql" # Create a backup mysqldump -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASSWORD $MYSQL_DATABASE > $BACKUP_FILE # Optional: Remove backups older than 7 days find $BACKUP_DIR -type f -mtime +7 -exec rm {} \;my-cron-file:
0 2 * * * /usr/local/bin/backup.shThis configuration schedules a daily backup at 2 AM. Adjust the timing and backup retention as needed.
B. Prepare rollback strategies
Ensure that each migration script is reversible by including .down.sql files. Test these rollback scripts thoroughly in a staging environment to verify that you can recover from any issues that arise.
5. Secure and test your migrations
A. Use environment variables
docker-compose.yml:
version: '3.8' services: app: image: my-golang-app environment: DATABASE_URL: mysql://user:password@mysql/dbname depends_on: - mysql mysql: image: mysql:latest environment: MYSQL_ROOT_PASSWORD: rootpassword MYSQL_DATABASE: dbname MYSQL_USER: user MYSQL_PASSWORD: password migration: image: my-golang-app entrypoint: ["/migrations/migrate.sh"] environment: DATABASE_URL: mysql://user:password@mysql/dbname depends_on: - mysqlNote: To better secure sensitive information like database URLs, avoid using plain environment variables in production. Instead, opt for Secrets Management solutions, Encrypted Configuration Files, HSMs, and restrict access.
B. Validate migrations
Always test your migrations in a staging environment that mirrors production. This helps you verify that data is preserved and the application operates correctly with the updated schema.
6. Monitor and test post-deployment
A. Monitor logs
After deploying the new version of your application, closely monitor logs to ensure migrations have applied correctly and the application starts without issues. For Docker, you might use a logging driver or a monitoring tool like Prometheus. Here’s an example of configuring Docker to use the json-file logging driver:
docker-compose.yml:
services:
app:
image: my-golang-app
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"B. Run tests
Execute functional and integration tests to confirm that the new version of your application works seamlessly with the updated database schema. Here’s a simple command you might use if running tests manually:
Test Command:
docker-compose run app go test ./...Ensure that your tests are comprehensive and cover critical functionality, particularly around new database changes.
Update : Persisting Database Data Across Docker Containers
“Is it necessary to use volumes to persist database data across Docker images or deployments?”
The answer is yes. Docker volumes ensure data persists outside the container’s lifecycle, preventing data loss during restarts or redeployments. For databases, without a mechanism to store data externally, you risk losing all your data between deployments. Docker volumes provide a solution by allowing data to persist outside of the container’s lifecycle, on the host machine or a remote location. Using volumes is a best practice to maintain database integrity and continuity. Here’s a quick setup:
Step 1: Define a Volume in docker-compose.yml
To persist database data, the first step is to define a volume in your docker-compose.yml file. Here’s an example configuration:
version:Step 2: Declare the Volume
Next, declare the volume in the same docker-compose.yml file under the volumes section:
volumes:
db_data:This setup ensures that PostgreSQL stores its data in the db_data volume, allowing it to persist even if the container is recreated. now, each time you recreate the database container, Docker will store the data in the db_data volume, ensuring that no data is lost between deployments.
Step 3: Test the Persistence
You can test persistence by stopping, removing, and recreating the container. The data will still be available, thanks to the volume. To confirm that the volume is working as expected, follow these steps:
- Run your Docker services:
docker-compose up -d2. Check the running containers:
docker ps3. Stop and remove the database container:
docker-compose stop db docker-compose rm db4. Restart the container:
docker-compose up -d db5. Verify data is still there: Log into the PostgreSQL database inside the container and confirm the data hasn’t been lost. You can use a tool like psql to check this:
docker exec -it <db_container_id> psql -U user -d exampledbConclusion
Managing database schema changes in Dockerized applications involves careful planning and execution. By using migration tools, integrating migrations into your Docker workflow, and implementing robust backup and rollback strategies, you can ensure a smooth and secure deployment process. Follow these best practices to maintain application stability and data integrity as your application evolves.
Happy coding and deploying! 😉