MongoDB migrations in Python with DBLift

MongoDB has no schema to migrate, so teams tell themselves they do not need migrations. Then the second index goes in by hand on production, a backfill script runs twice, and nobody can say which environment has which shape. The collections are schemaless; the application's expectations of them are not.

DBLift treats a MongoDB change the way it treats a SQL change: a versioned file, applied once, recorded in a history collection, previewed before it runs. The difference is the file. MongoDB has no DDL, so the file is Python, and it drives pymongo directly.

Install and point at a database

pip install "dblift[mongodb]"

The database name is required, because a MongoDB URL does not have to carry one. Put both in a file next to your migrations folder.

database:
  type: mongodb
  url: "mongodb://localhost:27017/shop"
  database: shop
migrations:
  directories:
    - "./migrations"

• Atlas works with the same two keys: a mongodb+srv:// URL, and the database name.
• Credentials belong in the environment, not in the file. The URL can reference them, or DBLift reads DBLIFT_DB_PASSWORD.
• There is no schema key. Collections are schemaless, so DBLift has nothing to name.

A migration is a Python file with one function

The name carries the version and the description, exactly as it does for SQL: V1__create_orders_collection.py. The file defines one function, and DBLift hands it a context whose db attribute is a pymongo Database.

from dblift.api import MigrationContext


def migrate(context: MigrationContext) -> None:
    if context.dry_run:
        context.log.info("[dry-run] would create orders with a unique index on order_number")
        return
    orders = context.db.create_collection("orders")
    orders.create_index("order_number", unique=True)
    orders.create_index([("customer_id", 1), ("created_at", -1)])

The dry-run guard is yours to write. On SQL engines, DBLift prints the statements instead of running them. On MongoDB it cannot see inside a pymongo call, so context.dry_run is how a preview stays a preview. Check it before every write. A migration that forgets the guard will run for real under --dry-run.

The rest of the context is what you would expect: raw_client is the MongoClient if you need another database, log writes into DBLift's output, and placeholders holds your configured values without substituting them anywhere.

Preview, then apply

Validation reads the files and the history collection and says whether they agree. On an empty database that is a short conversation.

$ dblift validate
Migration validation passed

$ dblift migrate --dry-run
Found 2 pending migration(s)
DRY RUN: Would execute the following migrations:
  - V1__create_orders_collection.py
  - V2__orders_status_backfill.py

Apply up to a version, or apply everything pending. Here the collection goes in first.

$ dblift migrate --target-version 1
Found 1 pending migration(s)
Migration lock acquired successfully
Migration V1__create_orders_collection.py executed successfully in 13ms
Successfully applied migration V1__create_orders_collection.py

The lock is a single document in a dblift_migration_lock collection, taken with an upsert, so two deploys racing each other do not both run the migration. The history document is written after the function returns without raising.

Backfill a field

Most MongoDB migrations are not structural. They add a field to documents that predate it, rename a key, or move a value between collections. That is application logic, and it belongs in a versioned file for the same reason a SQL UPDATE does.

from dblift.api import MigrationContext


def migrate(context: MigrationContext) -> None:
    orders = context.db["orders"]
    missing = orders.count_documents({"status": {"$exists": False}})
    context.log.info(f"{missing} orders without a status")
    if context.dry_run:
        return
    result = orders.update_many(
        {"status": {"$exists": False}},
        {"$set": {"status": "open"}},
    )
    context.log.info(f"backfilled {result.modified_count} orders")

Two things about this file are deliberate. The count runs before the guard, so a dry run reports how much work is waiting. And the filter is $exists: false, so running the file twice changes nothing the second time. Write every MongoDB migration that way, for a reason the next section explains.

$ dblift migrate
Found 1 pending migration(s)
Migration lock acquired successfully
2 orders without a status
backfilled 2 orders
Migration V2__orders_status_backfill.py executed successfully in 3ms
Successfully applied migration V2__orders_status_backfill.py

Afterwards, info reads the history collection and the folder together.

$ dblift info
│ Category   │ Version │ Description              │ Type   │ State   │ Undoable │
│ Versioned  │ 1       │ create_orders_collection │ Python │ Success │    No    │
│ Versioned  │ 2       │ orders_status_backfill   │ Python │ Success │   Yes    │

No transactions, so no half-applied safety net. DBLift runs SQL migrations inside a transaction where the engine allows it. A MongoDB migration that raises half-way leaves the operations before the raise in place. Keep each file to one change, make its writes idempotent, and rehearse against a copy before production. This is the rule the whole post is built around, and the next section shows what it looks like when it is broken.

When a migration fails half-way

Here is a file that writes to one document and then raises, standing in for a network error or a typo on the second statement.

from dblift.api import MigrationContext


def migrate(context: MigrationContext) -> None:
    if context.dry_run:
        return
    orders = context.db["orders"]
    orders.update_many({"order_number": "A-1001"}, {"$set": {"archived": False}})
    raise RuntimeError("simulated failure after the first write")

$ dblift migrate
Found 1 pending migration(s)
Migration lock acquired successfully
ERROR: Python migration failed V3__orders_archive_flag.py: RuntimeError: simulated failure after the first write

$ dblift info
│ Versioned  │ 3       │ orders_archive_flag      │ Python │ Failed  │    No    │

$ dblift validate
ERROR: Found 1 failed migration(s): V3__orders_archive_flag.py (version: 3)
ERROR: Run 'repair' command to update the status in the history table.

Three things are true at this point, and the tool tells you all three. The first order now carries the flag and the second does not; on PostgreSQL the transaction would have taken that write back, and here nothing does. The history holds a document for version 3 with success set to false. And every later command refuses to proceed until you say what happened.

$ dblift repair
Repairing 1 migration history issue(s).
Removed failed migration entry: V3__orders_archive_flag.py - migration can now be retried

Repair forgets the failed attempt; it does not undo the write. That is why the backfill earlier filtered on $exists: a file written that way can be fixed and run again without touching the documents it already handled. Write the retry into the migration, because MongoDB will not do it for you.

Roll it back

DBLift does not invent a reversal. A rollback is a second file with the same version and a U prefix, and you decide what reversing a backfill means. Here it means removing the field from the documents the forward file touched.

from dblift.api import MigrationContext


def migrate(context: MigrationContext) -> None:
    if context.dry_run:
        return
    context.db["orders"].update_many({"status": "open"}, {"$unset": {"status": ""}})

The Undoable column above comes from this file existing. Version 1 has none, which is honest: dropping a collection is not a rollback anyone should get by default. Preview the undo the same way as the forward run.

$ dblift undo --dry-run
Found 1 migration(s) to undo
DRY RUN: Would undo the following migrations:
  - V2__orders_status_backfill.py

What the history collection holds

Everything above is recorded in a collection named dblift_schema_history, one document per applied file. It is the same shape DBLift writes into a table on PostgreSQL, so tooling that reads one can read the other.

{
  "installed_rank": 2,
  "version": "2",
  "description": "orders_status_backfill",
  "type": "PYTHON",
  "script": "V2__orders_status_backfill.py",
  "checksum": -411458392,
  "installed_by": "cyrille",
  "execution_time": 3,
  "success": true
}

The checksum is what turns an edited migration into a validation failure. Change a file after it has been applied and validate says so, on MongoDB exactly as on SQL.

CI, Atlas, and the rest of the stack

• In GitHub Actions, the official Action installs the package and runs one command. Pass extras: mongodb so pymongo is present.
• Against Atlas, the URL carries TLS, replica set and auth source options; DBLift passes it to pymongo untouched.
• The Python client works the same way: DBLiftClient reads the same YAML and exposes validate(), migrate(), info() and undo().
• Azure Cosmos DB follows the same model with a different db object, and relational engines accept .py files next to .sql ones when a change needs Python.

Where this fits

If your MongoDB changes are already scripts in a folder that someone runs by hand, this is those scripts with a version, a history, a lock, and a preview. If you have never migrated MongoDB at all, start with the index you keep meaning to add.

• Python migrations, the full contract
• MongoDB engine reference
• dblift/dblift on GitHub