Preventing migration conflicts in Django during development

Django has the best combination of an ORM and Migration system that I've encountered so far. Over the past few years, I've rarely had to use Raw SQL thanks to Django's robust features. Most of the migrations are automatically generated, streamlining the development process. The systematic numbering and dependency tracking of migrations contribute to consistency, which is beneficial when multiple developers are involved and when deploying to various target environments.

However, this system can sometimes pose challenges during development. One of the issues I'll focus on in this article involves migration conflicts. These typically occur when merging a branch that lags several commits behind the target branch. This situation can arise quickly when multiple merge requests are open and can be merged without any code-level conflicts.

Extend the pipeline for a migration check job

This is one of the first jobs that should be added to the pipeline of a Django project. It's useful for detecting changes in models that haven't been reflected in migrations, and also for catching migration conflicts.

Below is a configuration that can be added in GitLab CI/CD:

stages:
  - test

check_migrations:
  stage: test
  script:
  - python manage.py makemigrations --check
  - python manage.py migrate

The solution above works well most of the times, however, you will start to encounter migration conflicts, particularly when multiple merge requests are open from different developers and features get merged.

Update 29 Mai 2023: Thanks to Gordon Wrigley for pointing out the django-linear-migrations package created by Adam Johnson. I've evaluated this for one of my projects and I'm considering adding it as the primary solution, given that it relies solely on git.

Solution 1: Django Linear Migrations

This solution utilizes django-linear-migrations, which must be installed:

pip install django-linear-migrations

Once installed, it should be added to Django's INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    "django_linear_migrations",
    ...
]

After adding the package, you should run the create_max_migration_files command:

python manage.py create_max_migration_files

This command will create a max_migration.txt file inside the migrations folder of each app. This file contains a string with the name of the latest migration.

Subsequent runs of python manage.py makemigrations will now automatically update the max_migration.txt file. If different features require changes to the models of the same app, it will cause a conflict in the max_migration.txt file before a merge can be completed.

An example pull request can be found in SnyPy Rest API

Solution 2: Merge pipelines and merge trains

This solution is specific to GitLab. However, it should be possible to adapt some of these principles for use with other services.

To utilize this feature, the first step is to extend the pipeline rules to match the merge_request_event:

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - when: always

Please note that enabling the merge_request_event for the entire workflow could potentially double the amount of time used for pipelines during merge requests. Therefore, it's worth considering the addition of this rule only to necessary jobs as a strategy to manage cost or performance issues with your pipeline.

check_migrations:
  stage: test
  script:
  - python manage.py makemigrations --check
  - python manage.py migrate
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
    - when: always

Additionally, a project admin needs to enable these features within the merge request settings of the project:

The workflow rules will not only trigger all pipelines by default (on push, tag creation, and so on), but will also create new pipelines for merge requests.

1. Merge Pipeline - This is a pipeline that runs against the merge result with the target branch at the time the merge request is created. This pipeline also starts when the commits in the merge request change.
2. Merge Train - When a merge request is ready to be merged, it's added to a queue. A pipeline then runs against the merge result before the merge is applied. If the queue contains multiple merge requests, the pipelines will run against merge results generated in the order they were added. If a pipeline fails, the merge request is removed from the queue, and a new pipeline starts against the remaining merge requests. For more details, check out the workflow docs."

The check_migrations job will now run against the merge results and will fail if a migration conflict is detected.

Conclusion

In this article, I presented two solutions for preventing migration conflicts in Django during development. The first solution involves using the django-linear-migrations package, which creates a max_migration.txt file to track the latest migration and detect any conflicts. The second solution utilises GitLab's merge pipelines and merge trains, running a pipeline against the merge result to catch migration conflicts before they are merged.

By implementing these strategies, developers can ensure their migrations remain consistent and up-to-date, thereby reducing the risk of conflicts and errors during deployment."