Streamlining Database Schema Changes with Liquibase

In the ever-evolving landscape of software development, managing and evolving database schemas efficiently and effectively is of utmost importance. To achieve this, organizations look forward to reliable database schema management tools like Liquibase.

Note: A database schema is a blueprint that defines how a database is structured, including 
its tables, columns, and relationships.

Tools like Liquibase are essential for maintaining database integrity, scalability, and agility, enabling organizations to quickly adapt to new requirements without compromising performance.

In this blog, we’ll explore Liquibase in detail, including its benefits and practical use with AWS in CI/CD pipelines. You’ll also gain insight into the best practices and standards for schema management particularly tailored to Liquibase.

So, let’s make your database journey easier!

What is Liquibase?

Launched in 2006, Liquibase is an open-source database schema management tool that was designed to simplify the tracking of database changes, particularly in agile software development settings. It offers developers a platform-independent solution for tracking, managing, and implementing database changes, with a key feature being its support for database version control.

At its core, Liquibase operates based on the concept of database changesets, which represent discrete units of modification to the database schema. These changesets are defined using XML, YAML, or SQL formats, allowing developers to express database changes in a declarative manner. It then applies these changesets in a controlled and sequential fashion, ensuring consistency and integrity across different environments.

Did you know?

Liquibase requires several components to run a command efficiently. Such as:

1. Liquibase.properties file: Liquibase offers the liquibase.properties file to streamline database management. They act as a centralized repository for database connection details and other static properties. By utilizing this file, users can avoid manual entry of properties via the command line, thereby saving time and minimizing errors. Notably, properties specified in the liquibase.properties file take precedence over those entered at the command prompt, making it a crucial element in Liquibase operations.

2. Changelog file: Liquibase simplifies database management by utilizing a text-based changelog file to chronologically record every database change. This file acts as a comprehensive audit trail, enabling Liquibase to review and execute pending changes. Changelog files can be stored and versioned using any preferred source control tool.

Within the changelog file, each discrete change is encapsulated in a Changeset. To modify your database, you can add a new change to the changeset file and reference its path in the changelog file. Changelog files support various formats, including SQL, XML, YAML, and JSON, offering flexibility for different project requirements.

3. Changeset file: This is like a step-by-step guide for making changes to your database. These changesets are kept in a Changelog, which is like a master list of all the changes. Each changeset has a unique ID and author (author: id), so you can easily track who made each change.

Changesets can do different things, such as creating new tables or changing existing ones. This detailed approach helps you manage and track changes to your database more effectively.

By now, you might be wondering how Liquibase can help you, right? Well, let’s get you some taste of its benefits.

Advantages of Liquibase

Here is the list of benefits you will enjoy with Liquibase on your side:

• Database Version Control: Liquibase excels in providing version control for database changes, crucial for tracking and managing modifications over time.
• Database Rollback: It supports effective rollback capabilities, acting as a safety net in case of errors during deployment within CI/CD pipelines.
• Cross-Platform Compatibility: It’s database-agnostic nature ensures versatility across various database management systems, making it adaptable to different environments in CI/CD workflows.
• Declarative Database Changes: Developers can articulate database changes in a declarative format, enhancing readability and comprehension compared to imperative scripts.
• Collaboration and Teamwork: It fosters collaboration among development teams, enabling concurrent work on database changes while minimizing conflicts in the context of CI/CD.
• CI/CD Integration: By seamlessly integrating into CI/CD pipelines, it streamlines database deployment processes. This integration guarantees consistent and reliable schema changes across environments.

Fascinating, isn’t it?

Now, let’s delve deeper and explore how Liquibase operates.

How does Liquibase work?

Here’s how Liquibase works:

1. Liquibase first locates the liquibase.properties file, which contains all the details it needs for performing changes to a database. This file should include basic mandatory attributes such as the changelog file, search-path, driver, URL, username, and password.

2. Context, label, and precondition can be added to changesets to help precisely control when a database change is made and to which database environment it is to be deployed.

3. For every successful change made to database (db) a record is inserted into DATABASECHANGELOG, DATABASECHANGELOGLOCK tables. To avoid applying existing changes, Liquibase checks the checksum in the DATABASECHANGELOGLOCK table.

4. The checksum (MD5 checksum) servers as unique identifier for every changeset applied to the datatbase. It is calculated using the ID and Author specified in changesets.

5. By default, Liquibase creates two tables in database: DATABASECHANGELOG and DATABASECHANGELOGLOCK respectively.

6. These tables act as audit logs, containing details of all changes made to the database.

Got it? Great!

Understanding practical concepts is always enjoyable. So, let me guide you through the process of applying schema changes to databases using Liquibase in simple steps, up next.

Applying Schema Changes to Databases with Liquibase

Follow these essential steps to apply schema changes to databases using Liquibase:

Step 1: Set Up Liquibase

• Follow the official manual to download and set up Liquibase.
• Once everything is set up, check the Liquibase version by running command “liquibase –version” from the directory where Liquibase is set up.
• If the Liquibase path is set up in the environment variable path, then the above command can run from any location (this step is not necessary).

Step 2. Set Up liquibase.properties

• Next, set up the liquibase.propertiesfile, which should include the following parameters: river, searchpath, changelog file path, database configurations like URL, username, password, etc.
• Check out this code snippet for reference:

#Enter the path for your changelog file.

changeLogFile=master.yaml

# Paths to search in

searchPath = changelog/, changelog/dev/, changelog/prod/

# Postgres

driver=org.postgresql.Driver

# Postgres details

liquibase.command.url=jdbc:postgresql://localhost:5432/postgres

liquibase.command.username: XXXXXXXXXX

liquibase.command.password: XXXXXXXXXX

Step 3. Create Changelog file

• Create a file named changelog.xml to define the changesets. This file will contain one or more changesets, each representing a set of database changes.
• For example, hangesets can be divided based upon env (QA, DEV, PROD) too.
• Changelog file format can be XML, JSON, YAML.
• Check out the code snippet below for reference:

databaseChangeLog:

- include:

file: dev/DimTables.sql

- include:

file: prod/FactTables.sql

Step 4. Create Changeset file

• Create one or more changeset files, each containing a set of changes you want to apply to the database. These changesets should be referenced in the changelog file.
• If only one changeset file is to be maintained, it can be directly used instead of the changelog file.
• Changeset file format can be SQL, JSON, XML, YAML.

Refer to the code snippets below: 

Step 5. Driver

• The Liquibase setup file comes included with support for databases by default.
• So, choose appropriate drivers in liquibase.properties file depending upon your database type.

Step 6. Database configurations

• To make any database level changes, Liquibase requires – database URL, username, or password.
• These details can be mentioned in liquibase.properties file.

Step 7. Run Liquibase

Use “liquibase update” command to make changes to database.

That’s it!

By following these steps, you will successfully apply database schema changes using Liquibase.

Bonus: Now, let’s get a glimpse of a specific use case where Liquibase is integrated into a Continuous Integration/Continuous Deployment (CI/CD) pipeline that is hosted on the Amazon Web Services (AWS) cloud platform.

CI/CD implementation of Liquibase with AWS Cloud

CI/CD, short for Continuous Integration and Continuous Delivery, is an integral part of DevOps, which brings together development and operations teams. It streamlines the process of getting new code from a commit into production by automating various tasks that previously required manual intervention.

This includes the build, testing (including integration, unit, and regression tests), deployment phases, and infrastructure provisioning. By implementing a CI/CD pipeline, development teams can efficiently make code changes that are automatically tested and seamlessly delivered and deployed.

Take a look at the architecture:

To break it down, here are the services used in the architecture:

• Bitbucket for version control
• AWS services like Codeuild, ECR, CodePipeline, Redshift, and S3(for creating CI/CD pipeline)

Referring to the above diagram and these steps, you can easily understand the workflow:

1. Clone Repository: Developer clones the Bitbucket repository to their local machine.

2. SQL Script Modification: The developer adds or modifies existing SQL scripts containing database schema changes.

3. Commit and Push: Changes are committed to a Bitbucket branch, triggering a push command.

4. CI/CD Trigger: The push/commit command triggers the AWS CI/CD process.

5. AWS CodeBuild: CodeBuild initiates, pulling the Liquibase Docker image from the ECR repository.

6. File Fetching: All necessary Liquibase files are fetched from the Bitbucket repository.

7. Docker Command Execution: Using a Docker command, Liquibase is started, applying changes sequentially to the database.

8. Environment Replication: Changes can be replicated to other environments, facilitating direct movement from Dev to Prod.

9. CI/CD Logging: After every successful CI/CD build, a log is maintained in an S3 bucket.

You’re all set now!

Note: Liquibase may serve as an advantage but note that you are aware of its few limitations like – difficult learning curve, limited IDE support, overhead performance, and complex migration.

Despite these considerations, Liquibase’s strengths position it as an asset for teams seeking efficient and organized control over their database schema evolution within the broader context of CI/CD workflows.

If you wish to enhance your database management and build robust software products in 2024, feel free to reach out to us at Nitor Infotech.