Deployment brain dump

Print Friendly, PDF & Email

Diversion…  continuous deployment, where changes are deployed in smaller cycles, so you get fewer changes (or even single changes) often, rather than lots of changes in bulk infrequently. I like the idea of making the window between change smaller so there’s the fewer possible causes when debugging a problem. One should be able to simply prevent the build process from running again and resolve the problem. When fixed, start it up again.

brain dump.. starter for 10, as I’m thinking about applying this to how bcfg2 gets pushed* of an example of a weekly cycle which might work for us:

  • Tuesday = Make new repo clones from a fresh copy of PROD_STABLE in dev branch
  • -> Friday = loop writing code, running test builds in dev, when OK tag as TESTING_UNSTABLE
  • Nightly = Run auto-build of TESTING_UNSTABLE in test env, email a show stopper to everyone if there’s a build failure.
  • Friday = freeze development, tag as TESTING_STABLE
  • Weekend = automated build against tag ^
  • Monday = People QA build results and if OK tag as PROD_STABLE
  • Tuesday = Push deployment via CI/fabric/bcfg2 whatever
  • Start again. If the TESTING_STABLE didn’t get tagged as PROD_STABLE then use TESTING_STABLE to create the dev branches.

You get a day off developing on a Monday but contribute to testing. If you want to work on a Monday make sure you make tests faster! If you want a wider window delay the tag to PROD_STABLE by a week. Basically, QA are testing last weeks build whilst you’re working on next weeks.

Or the CI way.

Everyone works directly on DEV_UNSTABLE and deploys changes immediately triggered by a commit hook via CI to dev. (Changes are forward and backward compatible so that nothing should in theory break :-) but you write a test for each change so if the build fails if the code doesn’t satisfy the test.) You do this 100s of times a day. If a automated build succeeds, it tags it TEST_STABLE. Then once a day someone manually pushes the big red button to tag it PROD_STABLE, to run the build again and deploy to prod.

Or CD, same as above but we deploy straight to prod if the test build succeeds, no big red button and we have changes in PROD in as much time as it takes to run the tests.

Perhaps we could do this in stages. Start with a softly softly 2 week schedule and move it to a week when things look like they’re working. Later we pull this down to a day when automation is smoother and we have more tests that need less manual QA. Then when we’ve been doing it for a while and feel confident we’ve got the process locked right down we do CD! =D

I think the main thing to remember with CD is that yes you can break prod, but you can fix it really quickly! A few people said things along these lines at LISA. It’s generally less expensive on time/resource to get into a situation where you can do frequent and rapid deployments, rather than lengthy design/test/deploy stages with lots of changes. You can never make sure you’ve tested for everything so it’s going to break. If it breaks in prod you’re going to need to do the same dev/test/deploy again really quick to fix it. So why not do that every time from the start?

* bcfg2 is now being test build via ci.ilrt. I’m merging changes into TESTING_STABLE which (commit hook) triggers a build which runs various tests. If it succeeds then it tags as PROD_STABLE (via merge yuck). Not so many stages but I am essentially inserting a testing between what is now one step (writing direct to PROD_STABLE). In the future I want to get automatic pushes to servers using fabric and later using mcollective. This can all be done by jenkins. =]