Safer iterations with sandbox environments and migration scripts

TLDR; Today, we’re happy to announce the general availability of a new, safer way to manage and maintain content structure once your content has been published. DatoCMS’s new sandbox environments and migration scripts have been built following long-established best practices, and ensure quick turnaround times and flexibility for developers — without interrupting the editorial workflow.

No matter what your DevOps process looks like and the various tools you employ, a recommended practice for any significant software development project is multiple environments. Using them ensures that your software is rigorously tested before it is deployed and made available to users.

Today we’re excited to announce the release of a set of new features allowing you to:

Build new features in isolation;
Test new features against production data;
Automate the migration process;
Instantly recover from bad deploys;
Have changes happen instantaneously.

This is a huge milestone, as finally allows you to safely iterate and test new features without putting their live website at risk.

Introducing environments

The whole idea revolves around a new concept we have introduced, which is that every DatoCMS project can now have different environments. Each is identified by a name (i.e., production ) and stores completely different models, records, uploads, plugins, locales, and timezone settings:

Primary vs. sandbox environments

Every project starts with a single environment, called primary environment, which is meant to be used for the regular editorial workflow. Additionally, multiple sandbox environments can be created by developers to safely test/experiment with new changes in the schema/content.

Sandbox environments start out as exact copies of one of the existing environments (i.e., the primary one). The process of creating a new sandbox starting off from an existing environment is called fork.

Switching environment from UI

Once there's at least one sandbox environment, developers are able to switch environments using a new top bar pane. Sandbox environments are, of course, completely invisible for regular content editors, so they’ll continue their editorial workflow on the primary environment as usual:

Fetching content from sandbox environments via API

Of course all of our APIs and integrations now offer a way to point to a sandbox environment instead of the primary one. Here are some reference links:

If you need assistance on this, just reach out to our support.

Working safely on a new feature

A developer that needs to work on a new feature should avoid changing models/records directly from the UI, and instead use our updated CLI to create a migration script:

$ dato new migration 'create article model'
Created migrations/1591173668_createArticleModel.js

What is a migration script? Just some JS code that uses our JS client to perform some changes to schema or content. For example, this will add a new model to our project:

module.exports = async (client) => {
  const articleModel = await client.itemTypes.create({
    name: 'Article',
    apiKey: 'article',
  });
}

As you’ll see, migration scripts are an essential part the workflow as they make the process reproducible and let developers run the exact same steps during the final deploy to production.

To safely run the migration in a new sandbox environment and verify that it works, we can run the dato migrate command:

$ dato migrate --destination=dev-article-model

The primary environment will be forked into a new dev-article-model sandbox, and any pending migration will be run. The CLI keeps track of which migrations have already run in a specific environment adding a special schema_migration model to the project, so developers can safely run the dato migrate command multiple times with no consequences:

$ dato migrate --source=dev-article-model --inPlace
Migrations will be run in sandbox env `dev-article-model`... done!

To learn more, visit the Write and test migration scripts guide.

Instantly deploy changes to production by promoting environments

Once everything is ready to be shipped in production, the first thing to do is turn on Maintenance mode, so that during the deployment process, no one can write new data in the primary environment:

You can also turn on maintenance mode using our CLI:

$ dato maintenance on
Maintenance mode activated: the primary environment is now read-only

DatoCMS will even warn you if other collaborators are currently working on content, so you can decide to postpone the deployment and/or contact the editors:

$ dato maintenance on
Cannot activate maintenance mode as some users are currently editing records!
To proceed anyway, please use the --force flag

With maintenance mode turned on, the next step is to call the dato migrate command to make a new complete copy of the latest schema/content present in the primary environment, and run the migrations on it:

$ dato migrate --destination=prod-article-model

You can then test that everything works as expected by deploying a preview branch of your website pointing to the new sandbox environment (in this case, prod-article-model).

Notice that, so far, the production website is still live, with no interruptions of any kind.

When QA and/or automated tests confirm that everything is ready, you can simply promote the sandbox environment to be the new primary. The action of promoting an environment to primary is instant: a few milliseconds later, content editors will see the interface refresh, and your production website will start fetching content from the new primary environment. The old primary environment will become a sandbox, ready to be promoted again as an instant rollback in case of errors you might find later on:

Once finished, turn off Maintenance mode to allow content editors to get back to their regular work on the new primary.

To learn more, visit the Apply migrations to primary environment guide.

How do I get this feature?

We think everyone should benefit of this major improvement, so every plan comes with the ability to create up to 5 different sandbox environments. Bear in mind that when you create a new environment you will actually generate exact copies of all the records, and those count towards your plan records usage.

The nice thing is that, thanks to the change in pricing we released two weeks ago, the upgrade/downgrade of your costs are automatic, so for example if you create a sandbox environment for just two days and you happen to exceed the number of records included in your plan, you will pay just for the exact minutes of usage of the extra records.

If your project is still under one of our previous per-project plans, the first sandbox environment is included and each additional sandbox environment will cost €39/month (with prorated billing). You’ll also need to upgrade/downgrade your plan manually. If you want to switch to the new per-account pricing, feel free to contact us.

Wrapping up

We’ve developed sandbox environments for modern engineering teams: using migration scripts you can ensure rapid and safe iterative development, and make it easy to integrate content management with your continuous integration and continuous delivery (CI/CD) pipelines.

Ready to get started? Then head over to our documentation on managing multiple environments!

The DatoCMS Blog