In a recent article I explained some of the mechanics behind putting together the schedule for XTech 2007. We've just added new visualization features to Expectnation that make proposal choice easier, and I thought I'd show some examples from the XTech 2007 review voting.
Sparklines, devised by Edward Tufte, are thumbnail sketches of data that succinctly convey patterns. When Nat Torkington used sparklines to show reviewer voting patterns for OSCON program committee, it seemed a helpful addition to make to Expectnation.
The profile of each reviewer is interesting — are they a harsh marker, or liberal with the top grades? — but sparklines really come into their own when drawn per proposal. Here are some small screenshots from this year's XTech voting, anonymized of course.
Reviewers are pretty unanimous about this proposal, as 4.0 is the top grade in our scoring system. The lack of variance suggests that I ought to ensure that all the assigned reviewers put in their scores — this could be just one person voting.
These next two are the pretty typical distribution for highly graded proposals: mostly top marks and a few average ones.
Some of the more interesting trends show up when the scoring isn't biased to either the top or bottom ends of the scale.
The reviewers are unanimous about this paper's soundness. It's not made anyone go "wow", but there's nothing to grumble about either.
This proposal clearly polarises opinion, and is such stands a better chance than the unanimous 3.0 above. Perhaps the subject matter or approach is controversial or timely.
Another proposal that divides the reviewers' opinions. It's also worth me checking here that I don't either have a pathologically strict reviewer, or at the other end, a reviewer with a wild passion for the cause this paper advocates.
The quality of submissions is usually so good that I don't give low graded proposals much attention, but the sparklines could alert me to potential oversights.
This proposal seems to polarise opinion between "rubbish" and "ok", so it's interesting to me to check out the subject matter and see if I'm missing something with potential.
These proposals were not received at all well.
In summary, I'm delighted to find a solid practical application for sparklines. My thanks to Nat Torkington for the inspiration.
