Development Seed

Anyone who has developed a relatively complex Drupal site will be able to vouch that, while launching a new site is relatively simple, continuous integration becomes very tricky once the site has actually gone live. This problem becomes especially pronounced when there is more than one developer on a project, and when the project is being built using proper development methodology.

The following example will illustrate this problem more clearly, as well as explain the use case of the Context / Features / Spaces module stack that we are writing here at Development Seed.

Building a site

This is a story of how Zak, Sara, and Ben are about to get screwed on a site build. But don’t worry, they find the Kool-Aid at the end. It’s called features.

Zak (a web developer) and Sara (a designer) are building a community site for their favorite indie artist (let’s call him Ben). Their original project plan dictates that the site will have a blog, a forum, and an image gallery. Zak has experience developing websites in Drupal, so they choose to use Drupal as the platform for their site. They set up a development server to allow them to collaborate and decide on subversion as a mechanism to store their changes.

To build their site, they:

1. Assemble a collection of contributed modules, namely views, cck, and some other lesser known ones.
2. Design and create a custom theme for the site.
3. Create a number of views and content types to handle the site content using the Drupal user interface.

Steps 1 and 2 are subsequently checked into the subversion repository, and once they are happy with their work, they check out the code and publish a dump of the database onto the staging server. They then give Ben access to the site to sign off on their work.

Ben finds a number of bugs, which Zak and Sara then fix, and republish on the staging site.

After repeating this cycle a few times, Ben is happy with his new site and within a few days they proceed to check out the code, publish the staging database onto the live server, and open the site to the public.

Adding and modifying functionality

The site has now been live for a few weeks and has been a modest success. Ben contacts Zak and Sara to make some changes to this site.

The new requirements are:

• Add a tour calendar to show tour dates
• Create a new section for information about albums
• Add new functionality to the image gallery that
  • Allows all users to upload pictures
  • Is able to be associated with specific concerts

As the site is now live and information is constantly being added to it by users, to make these changes they:

1. Retrieve the latest copy of the live database and publish it on the development server.
2. Add some contributed modules, namely node references and data api
3. Add a custom node type to handle the album information.
4. Modify the custom content types to add the additional fields.
5. Add some new views to handle the calendar.
6. Do some custom themeing to handle the new page elements.

Steps 2 and 6 are the only steps that are checked into the subversion repository, as they are the only changes that happen on a code level.

Once they are happy with their changes, they check out the code and copy the database to the staging server. Ben is not very pleased, however, as the staging site is completely out of synch with the live database. The staging server’s role is to be a representation of the live site with the new functionality, but because the live data is constantly evolving and steps 3, 4, and 5 occur only in the database, it’s unable to fill this role.

Zak is forced to take the latest backup of the live database, publish it on the development site, and manually reproduce all the changes they’ve made. Once that’s been completed, he refreshes the instance on the staging server. After this has been done, Ben is still unhappy with some of the changes, and they have to follow this entire process a number of times to get sign off.

Once Ben is happy with the changes, they take the live site offline and make a copy of it, reproduce the changes for (hopefully) the final time on the development server, republish the changes to the staging server, do a final test, and finally replace the live site. They needed to take the live site offline to stop the constant influx of data into the database so they have a stable target to hit for release.

Zak, Sara, and Ben have inadvertently come across one of the least discussed yet most widely occurring issues facing Drupal developers today.

How to solve the development -> staging -> production workflow problem

Building a Drupal site is easier than it has ever been. As Drupal becomes more powerful, it becomes much simpler to configure Drupal to solve just about any problem you might come across with very little custom code involved. The flip side to this is that more and more configuration has ended up being stored in the database. Because the database is not ideal for version control, especially once a site has gone live, it means that updating this configuration becomes a very complex problem.

The ideal development workflow involves communication in both directions. Content needs to be migrated upstream to staging and development servers, and configuration needs to be migrated downstream to staging and production.

The solution to this problem is to capture configuration in code. You may have noticed in the example above that Sara was not directly impacted by the problem because her theme changes are inherently stored in the subversion repository.

Code has the following benefits:

• It can be easily stored in version control
• It can be debugged
• It can be “diffed”
• It has reproducible results
• It has separation of content and configuration

So what does the context module have to do with it?

The database currently being used to store configuration is almost entirely context free and therefore devoid of any meaning. You are essentially throwing important structural components of your site into a big black box with no easy way to get that information out again.

To solve this problem one of the approaches Zak can use is to enable the context module on his development environment and break down the functionality into separate sections of the site. In this example, he would likely have gallery, forum, and blog contexts. Context is the tool that you use to provide structure to the information that is stored in the database. Above and beyond being a very useful and powerful tool for designing sites, it also provides a “map” of configuration to specific functionality.

Now that Zak has some contexts, what does features allow him to do?

The features module allows you to take any of the contexts you have defined and generate a standard Drupal module containing the related entities (views, node types, etc). The module that features generates is a standard Drupal module that leverages the existing functionality that Drupal uses to manage versioning, dependencies, and some other really murky problems relating to staging and deployment.

After exporting these contexts, Zak will now have fully functional and versioned Drupal modules for each of the contexts on his site that he can easily install and update on new or existing sites, provided the module dependencies are met. Zak can checks these features into his subversion repository so that when he rolls the latest code onto the staging server, using the latest live database, the configuration will be updated alongside the code. This saves Zak a lot of time and allows him to manage a site through it’s entire lifetime far more easily, while also providing much better service to his clients.

If you are interested in learning more about using features and context to help with your development workflow, be sure to check out Alex Barth’s BOF session at Drupalcon Paris about this very subject.