I was working on several client’s projects and something seems to have crystalized in my brain today. This crystallization created a new question that I posted on stackoverflow: How to deploy: database, source and binary changes in 1 patch?

Amazingly, 6 hours went by and I only got 20 views and 0 answers. You know what this says to me? To me this says that people do not have an easy answer to this question. I know for myself, i usually look for questions that I have quick answers to, I think this is not one of those questions.

Well, this actually was a test to see if something like this already exists, because I think I have an answer to this question. I outlined it in my conversation with Trevor today. I put it to your for consideration, I would love to get feedback so please tell me what you think.

A transcript of this conversation is in this post:

Taras Mankovski

I have multiple developers checking out projects from repository, the repo contains db dump of the project, they import the db, make changes to code, dump db, commit changes and db back to repo, problem is that sql dump site is pretty much useless in terms of figuring out what has changed in db

diff, is a well established mechanism for comparing text files. It works pefectly. I can at any point, check what’s changed between revision of a project. This is especially nice with git because I can choose arbitrary commits and see their changes.

This brings us back to db. DB sql dump is pretty much useless in determining what has changed in the DB because of the nature of the file. It’s long it’s wordy, it plain sucks to read.

So here is the solution:

I think it would be possible to represent the database as a directory with files in it. Every table in the database would be a directory. Every record a text file that contains in it the data from the record, represented as JSON.

What this means is that it would be extremely easy to commit the database into the repository and see what has changed.

We can use existing established mechanisms to determine the changes in the database.

The information included in the file system can be quiet extensive. For example, every directory can contain a schema file, that represents the schema of the table as json file. Again, it would be very easy to compare schema changes from revision to revision by comparing the schema files.

Trevor Twining
I think it’s a really interesting solution to the problem, just trying to work out how implementation details would work with Drupal’s architecture and how it might support or conflict with other partial solutions to the problem.

There’s been a push in drupal lately to allow more config to be exported as code for just that reason, but a lot of it is happening at the object level. There’s a growing number of modules that are using OOP principles in their code even though Drupal is procedural, so they’re basically providing var exports for some of those objects some of that work is for reusability of code as well, so that configurations can be saved and implemented in different projects

Taras Mankovski
Ok, so there is more. The title of this wave is Unified Patching. The problem with what we have right now is that “It’s very difficult to deploy changes to the server.”

The problem actually is that deployment is easy, if you’re doing it wrong. Deployment is hard if you’re doing it right.

What do I mean by that?
It’s easy to make changes to client’s site by working directly on client site. That in many many many ways is very wrong. If you break something you’re essentially screwed, it looks bad and it’s just extremely unprofessional. But it’s so damn easy.

The hard part is to do it right – do all development offline, test the changes, show the changes to the client on demo server, stage the changes and then finally apply them to client’s site without impacting the live site. This is hard.

I’m finding that deployment takes as much time as development. This is what I want to change. This is where Unified Patching will come in.

I’m breaking up project changes into 4 categories.

• Database Structure
• Database Data
• Source Code
• Binary Files

Actually, when working with CMS, Database Data can be broken up into 2 categories: Content Changes and Structural Changes. In Drupal, this would be defining the content types and the actual content data. We will ignore this point for now.

I think it would be possible to systematically apply all 4 of these changes using a unified tool, that would apply them in a reliable and predictable manner.

What’s I’m talking about is unix ‘patch’ utitility on steroids.

I have an idea of how this could be done in a very simple and reliable way.

Ok, so I think it would be possible to use diff to generate Unified Patches.

Diff is a very simple format that can be generated from any VCS.

Git’s diff includes information about the changes that happened between version with source code, binary files and permissions. It has no capacity to handle database changes.

I think if we were to use the solution outlined above, then we could create a Unified Patch generator that would parse the diff to determine changes in the 4 data types that were outlined above.

For example:

Let’s assume that we have all of the database dumpted as files onto the files system. We are going to run DIFF between 2 revisions.

1. Parse the diff to determine if any of the schema files have changed in the /db directory have changed. If changed then generate ALTER statements that correspond to the changes made to the schema.
2. Parse the diff to determine if any files in the /db directory have changed, if they have then create a corresponding REPLACE or DELETE/INSERT statement for each changed file
3. If binary files were added/removed or modified, then include the modifed files a temprorary location
4. Include standard patch for source code or text files

Now that we have all of the data types processed, put them together into 1 directory and tar.gz the directory.

Now you have an Unified Patch.

1. Upload the patch to the server.
2. run upatch ourpatch.tar.gz

upatch will perform the following actions.

1. dry-run database schema changes (possibly inside of a trasaction)
2. dry-run database content import
3. dry-run binary file move
4. dry-run patch

I don’t know how to do step 1 and 2. Step 3 could be fairily simple, we just need to know if have permissions to overwrite these files.

If dry-run succeds, then it’s fairily safe to perform live update.

Trevor Twining
I don’t know if you could do a dry run, but you could make changes in a test version (makes a dump of current live db and changes that instead of live)

transactional support might allow you to do the rollback, but you’d need to be able to do several operations before the commit, shouldn’t be a problem, but dummy database might be more useful because you can connect to it and actually see if anything is messed up, which you couldn’t do with a transactional approach.

Taras Mankovski
We really just want to know if the query succeeded. The other approach would be use to test, like you said, but actual make it a staging environment where you can push the changes to and test if the site is working before you apply the same change to the live site

Trevor Twining
Here’s a question though, what about the cases where the database is storing some serialized version of a code-object for persistance? Not looking for an immediate answer, but it’s an important question

Taras Mankovski
Is the data stored as php serialized object?

Trevor Twining
a portion, yes, and it’s how code is shared/exported, basically via a var_export call. code for views can then be moved out of the database and stored as modules

Taras Mankovski
ok, let’s take a few example:
Multiple developers are working on same site
Developer A checked out changes from repo, made changes in the view and commited them.
At the same time, Developer B also made changes to the db and try to commit them. Now, for Developer B, his commit would fail because his dbfiles are out of date. He needs to perform merge. I do not know how you would merge 2 view exports or if it’s at all possible, but atleast the developers know that there is a conflict in the views right away.

That’s it for today, not bad for 1 days, work.

Ok, what do you guys think about this?

Well, I’m making it official. I registered socialphotography.org, created Social Photography Facebook Group and socialpics twitter account.

The concept is very simple. I take portrait of people at parties, on their way to parties and generally just having fun. It’s usually a very social experience. I really enjoy the social aspect of the pictures, that’s why for me it’s – “Social first, Photography second.”

I see you later guys!

First, is a Professor from the University of Toronto, who is starting to do research on collaboration tools and why people collaborate.

Second, a techie from Vacouver who is interested in a collaboration system for a new Vacouver HUB that his groups is working on.

I’m really excited about this development, because Josh and I have been beating the collaboration drum for 3 years now and it looks like people are starting to respond.

Now can I control and meter access to the Datastore API using 3scale.

Next, I’m moving on to RabbitMQ and Celery, the goal is to be able to create tasks that are executed by offline worker servers.

Do not copy these example verbatim, I’m only showing them as examples.
Each of this methods is defined inside of a class

Imports

from nose.tools import assert_equals, assert_true, assert_false
from restclient import GET, POST, PUT, DELETE
from lxml import etree, objectify
from StringIO import StringIO

Verifying XML against XML Schema

    def verify_against_schema(self, xml, schema):
        '''
        Check if data is valid for provided schema

        xml data to verify
        schema is url to the xml schema
        '''
        schema = etree.XMLSchema(etree.parse(StringIO(schema)))
        parser = etree.XMLParser(schema=schema)
        return etree.parse(StringIO(xml), parser=parser)

    def test_review_schema(self):
        'Vacation :: Verifying review schema'

        xml = '''
        
            4
            4
            Wedding
            4
            34r3423f3rf32f3f
            334323443
            300
            true
            2009-10-07
            Family
            4
        
        '''

        verified = self.verify_against_schema(xml, GET(self.url('/api/rest/metadata/Review')))
        assert_true(verified)

Creating an item

    def create_review(self, resort, full=False):
        'Create review for testing purposes'

        xml = '''
        
            4
            4
            Wedding
            4
            %s
            334323443
            300
            1
            2009-10-07
            Family
            4
        
        '''%resort

        if full:
            url = '/api/rest/Review?type=full'
        else:
            url = '/api/rest/Review'     

        return POST(self.url(url), body=xml, async=False, resp=True)

Updating an item

   def test_updating_review(self):
        'Vacation :: Updating review'

        response, resort = self.create_resort()
        response, review_id = self.create_review(resort)

        xml = '''
        
            5
            5
            Wedding
            5
            %s
            334323443
            520
            true
            2009-10-07
            Family
            4
        
        '''%resort
        response, content = PUT(self.url('/api/rest/Review/%s?type=full'%review_id), body=xml, resp=True, async=False )

        expected = '''
        
            %s
            5
            5
            Wedding
            5
            %s
            334323443
            520
            true
            2009-10-07
            Family
            4
        
        '''%(review_id,resort)

        expected = objectify.fromstring(expected)
        expected = etree.tostring(expected)

        result = objectify.fromstring(content)
        result = etree.tostring(result)

        assert_equals(expected, result)

I hope you find these helpful

1. Add ability to POST body to restclient – created patch and contributed it to the project
2. Write a bunch of tests

Done, now I can move on to integrating 3scale.net into the API Server.

Last week, I got distracted trying to eliminate errors that dev_server.py was generating because it was trying to include packages from site-packages that it did not require for the project. I attempted to resolve this issue by creating a buildout for GAE projects that I could use in combination with virtualenv –no-site-packages configuration. This did not work because GAE was attempting to load packages from system’s python instead of looking in virtualenv directory. I’m not going to bother trying to resolve this anymore, I’ll just have to live with the errors.

Ok, on I go.

• Public User Interface
• Collection
• Processing
• Storing

Public User Interface

Running on Cherokee Web Server. Cherokee is a high efficiency web server that has low requirements and can handle very high loads. This element can be served sufficiently well by an inexpensive virtualized web server running in the cloud.

Collection

Collection element of the system will be performed by scrapy spiders which will get their tasks from the Task Server and store the result in the Beecoop Data Store.

Processing

Processing is done by workers that, like Collection, get their tasks from the Task Server and either store their data in Beecoop Data Store or PUI database.

Both Processing and Collection require a lot of RAM and CPU. We’re going to attempt to use commodity hardware to fulfill these functions. Core Networks has inexpensive hardware that we can try to use. If this fails then we can use Amazon EC2.

Storing

The collection element will capture a lot of different kinds of data. In our testing, 1 week of execution, we collected over 500,000 units of data. To handle this volume of information we’re going to utilize Google Data Store, because it will give us a huge database that we will not have to scale ourselves.

Visual Representation of our Infrastructure

There are several factors that I need to consider:

• I’m going to be working on several different projects at the same time.
• Each project is going to be using different python modules.
• Some projects will run on apache, others on cherokee.
• Cherokee does not have installer for Mac OS X, so I can not replicate our production environment locally.
• I would like to start working in TDD
• Last and not least, I have to make it easy for our sysadmin to deploy these projects.

Here are technologies that I’ve found so far and how I would like to combine them together.

Virtualenv with –no-site-packages switch allows you to create virtual environments that are seperate from your system. This forces you to be conscious of what packages your application is using. This is particularly useful when you’re doing development of applications that require different versions of python. We’re going to stick to python2.5 for the near future so this is not a major concern for us.

Buildout is the tool that i think we’re actually going to be using. It makes it easy for sysadmin to deploy the applications, while maintaining the benefit of forcing us to be conscious of what packages particular application requires. Buildout makes virtualenv unnecessary for us because it simulates virtualenv’s functionality by including the paths for all necessary packages in sys.path of every project.

Paste Script allows to create packages that we can include inside of our applications. It makes it easy to integrate multiple modules that are under development into our applications while keeping things clean and modular.