Edd Dumbill's Weblog: 'programming' articles

Social area networking please

2008-06-23T11:37:18Z

Tom Morris mentioned the gitjour project on the semantic web IRC channel today and it set me thinking.

Gitjour enables collaboration on a local network by tying together Bonjour (aka Zeroconf) and the git distributed source control system. It lets a developer publish the source direct from their own machine, without having to set up a public mirror. The advantages are great for camp-style hackathons.

However, I rarely get to such hurrahs and if I do, prefer to spend time talking to others and mining their brains. What I do have though is an extended, continuous "camp" that exists among a subset of my IM contacts, Twitter friends, and so on.

What I want is a version of Bonjour that works over a virtual network established from an ad-hoc list of friends and groups selectable from a social networking tool.

I know, this is starting to sound like Groove or any other number of peer-based collaborative tools. The point is I don't want to join any walled garden and get "monetized", I want to use existing Bonjour-aware tools, just among an ad-hoc group of people.

The hard bit as ever is firewall traversal, but this has been solved more than a few times now. It seems we've got the tools, we just need some enterprising developer to glue it all together.

Join the conversation about this post

We're all ops people now

2008-06-16T12:34:09Z

Ten years ago, most of us wouldn't have dreamt we'd be managing terabits of storage, tens of megabits of bandwidth, arrays of network-distributed services. The height of a programmer's worry would likely be choice of UI toolkit or finding the right way to indent code, and the height of consumer concern deciding which room to put the new computer in.

Now the problems associated with managing large networks are becoming real for everyone, right down to the consumer level. Stupendously large amounts of computing resource are available at an instant.

Your household probably has more than a terabyte of storage already. Issues such as single sign-on are going to hit home over the next year, as networked computing and entertainment devices profilerate. Features such as Apple's Time Machine will be increasingly vital — software that makes traditionally gnarly sysadmin tasks consumer-friendly. The rebranding of .Mac into "Mobile Me" is also a step in this direction.

The impact on developers

As software developers, we also have to cope with the effects of this resource-richness. For small sums of money we can get access to large computing clusters, geographically redundant hosting services. Our programs have left the desktop and found their new home on the web. System administration issues loom large upon us, security concerns lurk auspiciously in the corners of our minds.

Although the cost of infrastructure has dropped radically, other costs remain high and are going to stay that way. System administrators are not only grumpy, they demand high wages. Commercial software license fees spiral out of control: traditional per-CPU licensing models make little sense when you can quickly bring up tens of machines. The cost in power is already troubling large companies, and there's no reason to suspect the problems won't ripple down.

Help is at hand from a variety of technologies. If they don't yet make massive resource management trivial, they at least make it possible. Some of these also inhabit the weird territory of being both the source of a problem and a solution at the same time: virtualization, for example.

Distributed revision control systems

Distributed revision control is a technology whose time has finally come in popular circles, thanks in part to Linus Torvald's Git system. DRCS has several important impacts on today's developer:

Branching and, importantly, merging become much cheaper, allowing agile and flexible iterations of development.
Loosely connected and geographically diverse development becomes much easier. Even within a single organization it is not uncommon to find teams spanning countries, time zones. Complex multi-site VPN setups aren't necessary when a few SSH keys can do the job.
Revision control becomes packaging on the cheap. Like it or not, the mere tagging of a source tree has now become a valid option for releasing software.

All these trends lower the barrier to entry, increase collaboration and agility of development. You can the value of this as more software tools become free. Selling such tools is rapidly becoming a thing of the past, the advantages of sharing enable the developers at the sharp end to get their jobs done quicker.

However, such increased agility and, well, messiness leave other problems to solve, which the next two technologies address.

Virtualization

Hardware-as-a-service, infrastructure-as-a-service, call it what you will. The ability to create what we used to call entire machines, pick them up and move them around the network is revolutionary, and it's something that will have a real impact on regular developers. The benefits are at several levels.

Agile infrastructure — a ready supply of new machines makes it a lot easier and cheaper to test different scenarios, architectures, and to separate concerns. If things go wrong, throw away the image and start over. It's all about cutting the administration load.
A packaging solution — the new macroeconomics of software distribution mean that distributing entire machine images which communicate exclusively via the network is now a feasible way to distribute your software. We must adjust to the notion of distributing appliances, not code. We may mourn the lost crafts of creating RPMs or installers, but let's face it, it's now a waste of time.
New business models — your application can now be delivered as a black-box appliance, circumventing compatibility issues, or as a service, with virtualization part of the solution to scaling.

Configuration management

Computing is a zero-sum game, and despite our increased ability to create and distribute software, problems still exist. We just pushed them to the next level.

In good part, this next level is the problem of configuration management. We now have networks and clusters of (virtual) machines, software so agile we need six decimal places to describe its revision levels, and network and authentication paths that are starting to tangle. How do we manage that?

One thing developers crave is repeatability. That's why we love our makefiles, autoconf, Ant, rake and so on. It's the one time even the most imperative-minded programmer writes declarative code. We like to say "let the world be like this."

Our new sprawling world lacks this feature, and the best of our old toolkits — .debs, RPMs — address things only at the level of packages in a single environment.

So developers must look to the world of operations, a territory we probably thought we needn't enter. In this world the new "make" is called Puppet. You write recipes to describe how things ought to be, and Puppet will make it so.

I've been spending some time digging into Puppet, and feel excited by the confidence it's giving me. Now my applications exceed single source trees, and single machines, it gives me the means to tie the whole together. This article was going to be solely about Puppet, but that will have to wait now for another time.

It's likely you'll have played with virtual machines and distributed revision control, but have you tried Puppet yet? Give it a spin, and let your mind wander over the benefits for your organization and development approaches.

Conclusions

For developers and users alike, our world is changing. Hardware, connectivity and increasingly software is becoming cheap or free. The solidity of the old things we put value on — real things you can touch like disks — is eroding.

What really matters is our data, our creations, and their communication. If they don't quite yet exist in a universal "cloud" yet, they're certainly getting frisky.

As vendors provide solutions for consumers to manage their new domestic infrastructure, developers must look to network-aware toolkits and operations techniques to manage and get the best from their emergent infrastructures.

Also on this topic:

In search of agile infrastructure for web applications (June 2006)

Join the conversation about this post

Collecting minimal user information

2007-06-22T16:42:49Z

Web applications are in constant competition for a user's attention. Unlike shrink-wrap software, there's often no captive audience, and web apps must gently guide and woo users to help them to the best experience of the software.

This is particularly true when considering the problem of collecting personal details from a user. Collect too little information, and your app might not be able to function effectively. Collect too much information, and you raise the barrier for participation with your application. Users may either not be bothered, be alarmed at the apparent intrusion into their privacy, or fill out bogus information.

One size doesn't fit all

I originally stumbled on the issue in the development of Expectnation. The amount of information a conference speaker needs to give the organizers is significantly more than that of, say, a reviewer. An organizer will often want postal and phone details for a speaker, which are largely irrelevant to the review process. All conferences and events rely on the willingess of participants, so the path to participation needs to be as frictionless as possible.

Thus, a one-size-fits-all personal details form didn't make sense for us. At best, we could have simply made more fields optional, but that would result in missing information that we really did need.

Another solution would be to provide a dedicated personal details form per user type, e.g. different forms for speakers and reviewers. This is a non-starter, however, as users can take on many roles within a conference.

So, we had a problem on our hands: different levels of interaction with our application requires different levels of personal information.

We also had a second problem, which is that not all events and organizers want the same level of information. For example, a user group event is unlikely to need the same amount of information from speakers as an academic conference.

Thus we have two axes on which the amount of information required varies: by role, and by event.

Our solution: progressive disclosure

We concluded that we wanted to adhere to two principles in the collection of user information for Expectnation:

collect as little information from users as possible, and
allow organizers to determine the level of information they wish to collect.

The first objective was achieved by constructing a matrix of information items against user role, with the possible values of {required, optional, unused} for each item. We then achieved the second objective by allowing this matrix to be varied over each event.

We've named this concept "progressive disclosure."

What this means in practice is that a user is only ever asked for the information required to fulfil the roles they have within Expectnation. If they take on another role, they are asked for the additional information at that point, and not before.

Implementation concerns

Extract from disclosure level editing UI

It would be untrue to say that progressive disclosure is simple to implement.

A user in our system will have disclosure requirements that are the union of all of their roles in all of the events they attend. Within Ruby on Rails, this means the conventional validation logic is useless to us, as it only deals with static requirements. Secondly, disclosure requirements are expensive to compute, so we need to cache them, with the inevitable addition of issues such as cache invalidation.

Beyond the data model issues, the user interface issues are, if anything, more troublesome. From an end-user's point of view, the difficulty arises when we need extra information from them, and presenting this in a way that makes sense.

From the event organizer's point of view, we have the challenge of presenting a controlling user interface for rather a complex concept. Organizers must be able to change the level of required information either globally, or per-event.

We opted for a grid presentation, an extract from which is shown in the screenshot in this article. Over time we'll be able to see how effective this interface is, and what we can do to improve it.

Finally, we have chosen sensible defaults for all disclosure levels, as the best result would be that organizers have little need to reconfigure the settings in the first place.

Making it look easy

Because Expectnation is a general application for organizing events, rather than targetting one specific one, we've had to construct a framework for minimizing user information collection, as opposed to simple implementation of known scenarios.

At the end of the day we have added a fairly complicated subsystem and an extra degree of complexity to our codebase. However, the result from the user's point of view is increased simplicity and a streamlined experience.

Every time you embark on something that wide-reaching and complex, you have to ask yourself whether the feature is really worth it. In this case, though, the answer is a resounding "yes". The user is the most important factor in our software, and making their life easier in an unobtrusive manner is a priority.

Expectnation blog: Only ask users for the information you need

Join the conversation about this post

Why Bazaar rocks, and the highs and lows of its use with Rails

2007-06-20T11:38:45Z

While Subversion is for many the source code revision system of choice, I've never stopped long with it myself. Why? Because it's mostly a fixed-up CVS, and doesn't fix the really gnarly problem with revision control: merging.

Mark Shuttleworth has written an insightful little article on the key aspects of merging source code. Some of the points he makes underpin my choice of revision control system.

For several years now I've been a very contented user of Bazaar, in its first arch-based incarnation, and now in its latter form. The key reason has been ease of merging.

While most articles praising Bazaar are reports of open source development, I've also found it very handy for small in-house teams working on web applications.

There are very often multiple arcs of development going on at once, and the ease of merging makes it easy to ensure that long-running development arcs don't get ridiculously far from the main trunk of development. Bazaar's merging also makes it easier for lead developers to police the merging of the code base.

Using Bazaar with Rails and other tools

Working mostly with Rails, one repeatedly runs up against the assumption that Subversion is the golden path for revision control. This is perhaps a little telling about the level of participation most Rails-based projects have reached.

The tools story for a Bazaar user is both good and bad.

The good

Deployment: the Capistrano deployment system has support for Bazaar. I've written previously about the technique I use to deploy my Rails apps.
Collaboration: Trac is a wonderful integrated bug database, wiki and source code browser. Although it's a little hard to find, the trac-bzr integration works very well, and happily moves Trac out of the Subversion-based assumption that you can only have one repository root per project.

The bad

Continuous integration: with Expectnation, we've had to roll our own continuous integration system ('system' being rather a grand word for a shell script on a cronjob...) Most of the CI systems currently popular seem to assume Subversion.
Editor/IDE integration: some early work exists, but it's early days. I've not yet found anything making me want to leave the command line.

Join the conversation about this post

Really really getting the testing religion

2006-11-23T13:03:44Z

Writing tests is virtuous. It's the wholemeal bread of programming. Do it, and you'll be so much fitter. Except it tastes like cardboard and gets in the way of consuming the yummy treacle sponge of programming.

But now I've got the testing religion. I view the lack of tests with the disapprobation of a reformed smoker tutting and huffing at the merest suggestion of a Marlboro Light.

How did this happen and how can I be sure I won't backslide?

The moment of transition was the first time that a test saved me from releasing software with a bad bug in it. The theoretical knowledge of virtue was translated into the palpitating relief of not disgracing myself in public.

After that happened a few more times, I found myself desperately uncomfortable with the idea of releasing anything that didn't have decent test coverage. And so a virtuous habit is formed.

It's not just driven by fear. With the right tools to hand, writing tests can be a lot of fun. You don't always need to be "in the zone". If you've got a spare half hour here and there, you can use it to increase the coverage of your tests.

Getting better at testing is one of the ways I've become a much more proficient Rails developer over the last year of work on Expectnation. In case it helps anybody else, I thought I'd share some of the tools and techniques that are part of making Rails testing fun and effective.

Continuous testing

Whenever a checkin is made to our source repository, tests are run and the results emailed to everybody working on the project. As with the "you broke the build" disincentive, you need to be kept honest by frequent running of the tests.

There are a variety of tools around to help you do continuous integration with Rails, most of which have some dependence on you using Subversion for source control. In the event, I just created my own script to do what we required.

Coverage reports

One of the big mental hurdles with testing is knowing how many tests you should aim for. You can't prove a negative: even if your tests are all passing it still doesn't tell you much about the tests you haven't written that might fail!

We use rcov as part of the continuous integration process, and post the results to our project web site. The pretty red and green bars give you some idea of when you're starting to get adequate test coverage, and the line-by-line breakdown is invaluable.

Tainting strings

One of the easiest errors to make in Rails templates is to miss the call to h(), to escape any HTML in an output string. The only real way to catch these errors is through automation. The wonderful safe_erb plugin helps you do just that. By using Ruby's tainting mechanism for strings, it throws an exception if you render any external data unescaped.

One of the beneficial side effects of safe_erb is that you then become very particular about ensuring your functional tests properly cover every action in your application. Tests such as get 'index', which before seemed mundane and redundant, are now essential, as you want to test for tainted strings.

In addition to the above techniques, there are a lot of fun shiny gadgets to help you write tests, such as ZenTest, but the three things above are the ones I've found to be of significant value.

I ought to conclude with words of praise for Rails. It goes out of its way to make testing an important part of the development lifecycle, and the availability of the of unit, functional and integration test functionality makes it easy to get started. Every time I teach somebody about Rails' testing features, they end up finding it very enjoyable.

Don't forget, testing can be fun!

Join the conversation about this post

In search of agile infrastructure for web applications

2006-06-17T22:57:35Z

Many advances have been made in agile software development. Frameworks such as Ruby on Rails embody agile principles by making software easy to write, easy to test, and above all, easy to change.

If only we could say the same for the infrastructure on which we develop and deploy web applications. Not every application can be a Rails one, especially where it's been up and running for years. The systems we deploy on often rely on configuration files scattered all over the filesystem, and can be complex results of years of change.

By extension, the systems we develop on often don't exactly mirror where we deploy. If we're lucky and careful, then we have a staging server which mirrors the live environment, but that's not too much of an advantage. Among other things, we need ways to evolve and refactor the live environment, and have our development environments easily track that.

I'd like to call such an environment "agile infrastructure". Infrastructure that doesn't hamper developers, and allows live configurations to change and evolve. Infrastructure that allows new things to be tried with minimum cost, and can provide the best information possible to help future planning.

Tools enable techniques, of course. It's a lot easier to be agile if you're programming in Rails rather than plain old PHP. So what are the tools enabling agile infrastructure?

Techniques I've found useful

While there's a long way to go, I'd like to describe some of the techniques I've been using to help create a flexible environment for developing a sizeable PHP/MySQL application.

Operating system

An operating system must be easy to bring into a known state with minimum interference. For this, I've found Ubuntu or Debian to be an ideal choice. Two of the main reasons include the constancy of stable releases and the fact that it's rare you need to stray outside the distribution's servers to get the software you need.

Deployment

Deploying an application shouldn't break the cleanliness of your OS build. I prefer to package deployed applications into .deb files. This enables me to build on several advantages of the package management system:

configuration file handling: deployed machines can keep individual config in /etc, which is preserved over software update.
no random state: you know for sure exactly the state of the deployed application, no random files left hanging around.
natural workflow: packages take a little bit of effort to build and deploy, which means it focuses developers on getting them right. If things do go wrong, it's very easy to roll back one version.

Source control

The case for source control, thankfully, doesn't need making these days. Yet not every source control system is created equal. Developers shouldn't be constrained by source control, it should be cheap and easy to try new ideas and merge these in later.

Not only cheap branching, but also easy merging, is required to keep development agile while still retaining the benefits of source control. Many people flock to Subversion these days, predominantly because it fixes some of CVS' more egregious misfeatures. However, merging can still be pretty difficult in Subversion.

I prefer to use a system allowing easy merging, such as svk or bazaar. Because so many of us use laptops these days, disconnected operation is also a huge boon. Bazaar-NG offers both repository-oriented and completely decentralized operation, giving the best of both worlds. It also wins the competition for least setup overhead.

Virtualization

Previously a high-powered technology, cheap virtualization is now with us. Many identical virtual machines can be quickly created to enable testing and experimentation. Deployment to virtual machines can enable hardware independence for applications, reducing another big headache.

Large organisations have it within their reach to bring new hardware online easily, and have invested much in management systems for it. Virtualization brings this in reach of small and one-man development teams.

Desktop-based virtualization

Systems such as VMWare and Parallels untether developers from their desks. I use the OS X desktop daily for its productivity software, but an Ubuntu virtual machine for a large proportion of development work. A simple keypress lets me flick between the two. The advantages of multiple desktops to hand for web development are well known. (In fact, thanks to WINE and IES4Linux I can run Internet Explorer cheaply too from the same virtual machine).

Xen virtualization

After the initial novelty wears off, one of the most tedious tasks in the world is bringing new machines online. Combined with the tendency development environments have to get crufty as new ideas and tools are tried out, this makes for increasing disparity between the development and deployment environments.

Wouldn't it be nice just to take a clean machine off the shelf, check out the source, and pick up again? Using Xen for Linux, this cheap virtualization is possible.

Sadly, Xen isn't yet a packaged part of Ubuntu, but getting it up and running on Ubuntu Dapper isn't too hard. Warning: here's where things get pretty deep. Skip this bit if you're not interested in the nitty-gritty.

I followed these steps:

Followed instructions in the Ubuntu wiki to get the first domain (vm) up and running
Installed the xen-tools package, which provides wrappers for easily creating virtual machines
Adjusted some of the xen-tools configuration (/etc/xen-tools/hook.d/20-setup-apt) to set up for Ubuntu rather than Debian
Once the virtual machine was made, adjusted its config to allow for the kernel initrd and configure the network bridge I was using

Getting over the DNS mountain

If you want it to be easy to bring new virtual machines up and down, you need to adopt a different approach to managing your DNS. You need to know how to predictably connect to a virtual machine. Yet it's going to get very boring and error prone to manage the IP address space manually in your DNS, or even to map the MAC addresses of your virtual machines in your DHCP configuration.

Help is at hand in the form of mDNS (also known as Rendezvous / Bonjour / ZeroConf). In the host machine, and in each virtual machine, I ensured the avahi-daemon and avahi-utils packages were installed, providing mDNS services. (You should also check that mdns is present in the hosts entry of /etc/nsswitch.conf).

What all this does is allow the machines (virtual or otherwise) on the network to resolve names using mDNS. So if I create 3 VMs, alpha, beta and gamma, I can access these right away using the hostnames alpha.local, beta.local and gamma.local.

Hey presto! Easily make machines appear and disappear without any need for sysadmin involvement. All the extra steps to add avahi installation can be easily scripted with xen-tools, to reduce the number of commands needed to birth a new machine to one.

With this ease of creating new machines, it's much more straightforward to experiment with server topology and conduct experiments on server environments.

Conclusion

Agile development is necessarily constrained by the infrastructure on which it is conducted. The infrastructure itself is constrained by the tools and platforms of choice.

By appropriate tool choice we can reduce the commitment and overhead of infrastructure. This enables developers to get on with the job they do best, and makes experimentation and evolution of infrastructure much simpler.

When programming, it's dangerous to fall in love with your code, as the best solution may often involve throwing half of it away. The same hazard exists with infrastructure, with bizarre and awkward effects propagating back into code and operations. Get yourself a low commitment, agile environment, and you'll be able to keep up with the pace.

Join the conversation about this post

Firebug: neutralizes nasty niffs

2006-01-22T18:01:31Z

Sometimes, coding for the web is like having a co-worker with a body odor problem. Everyone's far too polite to say anything, so you all put up with it and nobody tells the poor smelly soul. Meanwhile, they sense something's wrong but don't know what.

Web browsers are so insanely great at making sense of tag soup, that if you code only for the end-result--a reasonable strategy, adopted by most when starting out--you can get into all sorts of tangles. Misery ensues: you're stuck with days of fiddling in the dark. You know something's up, but it's hard to see what's going wrong.

In that sense, browsers are lousy at one half of Postel's law. They get the "liberal in what you accept" bit right, but they don't help the developer one bit in implementing "be conservative in what you do." Using three or four different validators for HTML, CSS, etc. often helps a lot, but the state of the browser as an integrated development platform leaves something to be desired.

Firebug is a Firefox extension that goes a long way to providing integrated web application debugging. It is a full-featured error console that shows your errors in Javascript, CSS (a godsend!) and XML. Not only that, but it does full logging for XMLHttpRequest calls, so it's a powerful tool for debugging AJAX applications as well. Firebug also manages to bundle in a DOM inspector, and a command line for evaluating Javascript.

With these features, Firebug should let me throw out about four separate extensions I already had installed that do a similar thing. The final star turn in Firebug's act is a Javascript call to log information to its console, the equivalent of the all important debugging printf() call in C. I hope this means this will make it easier to deploy things such as unit test frameworks for AJAX applications.

Although its author Joe Hewitt cautions us to remember that Firebug is still an alpha release, I'd still recommend rushing to install it. If you're developing anything for the web browser this year, it's the most important tool you'll find.

While it won't rid you of the problems of cross-browser compatibility, Firebug will speed you in uncovering the smelly bits of your web application.

Join the conversation about this post

Why Rails?

2006-01-21T22:50:53Z

When I moved this blog from being Zope-backed to using Rails several people got in touch to ask why. This is a brief explanation of why I've moved.

Firstly I should point out I've had a long and mostly happy experience with Zope. I got into Zope originally because I'd spent some time in constructing web frameworks from scratch (we're talking 1995-1999 here) and found that not only does a framework need good templating, it needs reusable components within that.

Zope seemed like a good fit, so I spent some time getting up to speed and was happy there. However, there were some things that didn't make me feel completely comfortable. Zope offered a very stark choice between developing through the web based interface (DTML methods, etc.) and writing raw Python. To do really useful stuff, you had to get into the Python side of things, but also learn a lot about Zope guts that wasn't really well documented.

I spent more time than I probably should have had to reading Zope's own source code in order to figure out how things worked. Fine, for my own silly little projects, but not practical in general. It was hard to get the overall view of a project, and hard to test. The learning curve was a little too steep, I suppose.

So, once I'd got to a certain point I stopped, and looked around for a platform that gave me more code visibility and testability. Loyal readers will know I've looked at ASP.NET and Python, Quixote and Cheetah. ASP.NET I rejected because I didn't view Mono's web application server as mature enough at the time, and I also had reservations about ASP.NET's non-free baggage. Python, Quixote and Cheetah looked a lot more friendly.

What I wrote about Quixote and Cheetah at the time was "this combination isn't adequately presented and described anywhere." These were good technologies, but not integrated as a framework. I started work on integrating them, and made some interesting progress. I was correct that Python lacked a solid modern framework, and the time was certainly right for it, as we can now see by the emergence of Django and TurboGears.

However, at the same time I started looking at Rails and found out that it did pretty much what I wanted. To be honest, I'd avoided it for a while as a friend once did me the disservice of calling Ruby a cross between Python and Perl. "Why ruin Python?", I thought. And it takes me a while to get over the hype barrier before I conclude that there's a "there" there.

Now I've been using Rails for a while, I find its attractions have changed for me. The thrill of not having to write basic functionality over and over again has given way to more select pleasures. Writing in Ruby itself is a rewarding experience, as its idioms become plainer over time.

Sure, you can write Java or Perl in any language, and that's where I started off. But the subtle elegance of the Ruby idiom is a slowly appreciated and highly satisfying flavour. For developers who consider themselves as poets and artists too, I can't think of a better language.

Join the conversation about this post

Rails, Bazaar and SwitchTower

2006-01-19T12:14:49Z

I have just submitted an enhancement to SwitchTower to let you use Bazaar as a source control system.

SwitchTower is a deployment framework for Ruby on Rails applications, allowing fine-grained control over deployment of Rails applications to one or many servers.

My contributed module lets people who use Bazaar join in the fun. Until such time as my patch is approved, save it in the app/switchtower/scm directory of your Rails app. The module assumes that the Rails application is in the root of your tree.

In your SwitchTower deploy.rb file, add:

set :repository, "you@example.com--archive/whatever--trunk--1.0"
set :scm, :baz
set :checkout, "export"

The "export" setting can be omitted if you really want baz to check out all the {arch} directories full of version control junk onto your live servers.

Note that the repository follows a specific version. If you branch your deployed version, you'll need to change the repository to the new version.

Join the conversation about this post

Planning programming projects

2005-06-11T11:47:12Z

This small article from John Cowan, Flow, Stuckness, and Interruptions, is one of the most useful and pithy explanations I've ever seen of the difficulty of managing programming projects.

My recent return to managing a team of people has highlighted for me the difficulties of the arbitrary deadline approach to project management. Unfortunately, it's also the default management approach applied by a lot of people, because the concept is easy to grasp.

The arbitrary deadline method is troublesome because of the difficulty of estimation. As John's post elaborates, you can never foresee all of the problems you'll meet along the way. The distressing inevitability of 90% of the effort being required by 2% of the deliverable is frequently inexplicable to developers themselves. Never mind the managers remote from the development!

I found this part rather interesting:

A relatively successful model for programming management, however, is based on the notion of trouble-ticket tracking. Since projects are not for the most part self-generated but rather arise out of requests from editorial, sales, or customers directly, each of them can be treated as a trouble ticket at whatever scale, and then the question becomes not "Is this project past its highly arbitrary deadline?" but rather "How many projects being starved of the time and attention they need?"

Of course, to some people the answer will always be "all of them."

Blogs, books and forums are full of sage advice and new methods to solve problems like this one. They're great, until you put them into your own situation, and find that it's not just about technique. (Hurrah for Basecamp, which pushes communication as a primary tool of project management.)

Every little thing can help though. Best practice for me tends to be a mongrel creation, containing many contributions from the 2% of residual wisdom that sticks with me from every new good idea.

A note on flow and interruptions

Danny O'Brien's Webolodeon is a gentle way of forcing web browsing behaviour into useful patterns. Self-interruption is as big a problem as external interruption when it comes to staying in the "flow". Now, all we need is one of Danny's gadgets fixing to the RSS readers as well.

Join the conversation about this post