Category Archives: stats

Turn up

I saw this chart recently on Twitter. (Despite the attributions given, I haven’t been able to find a better copy or an accompanying article – if anyone knows more…)

As you can see, it shows changing levels of turnout at thirteen General Elections – 1964, 1966, 1970, 1974 (x2), 1979, 1983, 1987, 1992, 1997, 2001, 2005, 2010 and 2015 – broken down by age group. There are a couple of things I don’t like about it. Firstly, under-21s didn’t get the vote until 1970; the first two data points on that salmon-pink line aren’t 18-24s at all but (presumably) 21-24s. For internal consistency, we need to start in 1970. Secondly, at the other end of the chart there’s an uptick – or, in the case of 18-24s, a downtick – apparently referring to an election in 2016. Obviously, there wasn’t one. The figures may refer to the EU referendum, but if so they aren’t really comparing like with like; in any case they don’t appear to be correct (36% turnout in the 18-24 age range?).

So, if it were up to me, I’d start the series no earlier than 1970 and end it in 2015. But there’s a bigger problem, caused by those lines. The decision to plot a line against a continuous time-based X-axis, rather than represent the different elections as discrete events, shows how turnout can change when there are two elections close together (as there were in 1974). Apart from that, it doesn’t really gain you anything – and it creates a false impression that we’re looking at continuous change over time, i.e. that the turnout figures for 2005 and 2010 allow us to read off what the turnout would have been in 2007. So I’d go for clustered columns. Also, we are interested in how different groups have changed over time – it’s just not continuous change over time. So, rather than plot the values themselves (most of which cluster together, making for a cluttered chart), I’d plot the change for each group. A bit like this:

What you’re seeing there is the change in turnout for each group – and for the whole population (pale blue bar) – relative to 1970. (Zeroes are invisible – see 1979 and 1983.) Straight away you can see that it’s a chart of two halves: turnout in almost all groups grows or holds steady from February 1974 to 1992. Then turnout falls for under-45s in 1997, and falls across the board in 2001. The next three elections see some of those losses clawed back, but with further losses among under-25s in 2005 and 2015. By 2015 over-65 turnout is back to its 1970 level and 55-64 turnout is slightly up, but overall turnout is still down 12%.

You can see the election-to-election trends more clearly on this second chart. Percentage changes here are against the previous election.

This shows just what a landmark election 1997 was – in a bad way: turnout was down nearly 10% overall, and 15% or more among under-35s. Then look at 2001: turnout is down over 10%, with >20% declines in the younger age groups, relative to 1997. Then, in 2010, we see a huge rally of the two youngest age groups – up by a third and a fifth respectively – followed by a slump for those groups in 2015. (And I’m sure Nick Clegg is very sorry.)

What all this tells us is that there’s nothing constant or ‘given’ about young people not turning out to vote; in all six of the elections from 1974 to 1992, the 25-34 turnout showed a bigger increase from its 1970 level than over-65s’ did from its, and the same was true of the 18-24s in four out of the six. It also seems to show that something happened to British politics in the mid-90s that made it a lot less interesting to people – what could that be, eh? And it suggests that, if people in general are disengaging from electoral politics, young people in particular will really disengage. (Interesting to see that the one group where turnout actually increased between 1992 and 1997 is 55-64s. New Labour: the triumph of Dad Rock?)

In short: if Labour were running a managerial, trust-me-I-know-what-I’m-doing, we’re-in-charge-now, let’s-not-be-hasty, listen-to-your-father type of campaign – which is to say, the type that won them three elections in a row – I would be really worried now, about turnout in general and about young people’s turnout in particular. Needless to say, that’s not the campaign they’re running – and on that basis I’m not sure that past trends tell us anything at all. Except, perhaps, that there are an awful lot of relatively young non-voters out there, and a lot of them have not voted in the past for good reasons. Let’s see if we can persuade them otherwise this time. I think it could really make a difference.

Taking back control

So here we are, approaching day 9 of what was surely meant to be a 24-hour coup. Stuck as we all variously are, discussions among Labour people have gone over the same ground rather a lot during the week. Two themes that keep recurring are the role of the party’s membership and the potential for a split. The two are related in some interesting ways. A split, firstly, would create an additional centre party, to the right of Labour and to the Left of the Tories, and would give a massive boost to the centre vote at Labour’s expense. But what would happen then? Well, what happened last time it was tried?

Screen Shot 2016-07-02 at 18.22.53

As you can see, Labour were roundly beaten by the Tories in 1979, taking 36.9% of the vote to the Tories’ 43.9%; Labour’s vote share wouldn’t go above 40%, nor the Tories’ below 40%, until 1997. In all the next three elections, the Tory vote share was more or less unchanged, never falling as much as 2% below the 1979 level. What did happen over those three elections was that Labour lost ground massively to the ‘centre’ and then clawed it back. These were, of course, the years when the SDP was launched, swept all before it, formed an alliance with the Liberal Party, lost most of its MPs, merged into the new Liberal Democratic Party and was forgotten (the whole thing took less than a decade). The effect of the split was to create a centre-party surge; the effect of the centre-party surge was to split the Left and help keep the Tories in power; and the surge ended when Labour managed to recover the support they’d lost.

That’s one way that a centre-party surge can end – through Labour winning those voters back. Another surge, not driven by a party split, developed between 2001 and 2010, as a morbid symptom of the decline of Labour’s appeal under Tony Blair. The chart could also be extended back in time to the two elections of 1974, in both of which the Liberal vote share went above 15% – something not previously seen since 1929. Both of these third-party surges ended abruptly and ignominiously – the Liberal Democrats discredited by their period in office, the Liberals both by their period in office and by the trial of Jeremy Thorpe. Nor was there any discernible benefit to Labour; the votes of former ‘centre’ voters appear to have largely benefited the Conservatives in 1979, UKIP in 2015.

This suggests that, where a centre-party surge fades gradually, voters can be won back to the Left; where it collapses suddenly, the Right gains. Intuitively this rings true. An ascendant centre party – like the one led by Kennedy and Clegg – is one that is in the process of drawing voters away from Labour, and attracts people who see themselves primarily as ‘not Labour any more’; if such a party has a rapid loss of credibility, voters who have started moving away from Labour are likely to carry on. A slow fade, by contrast, takes place when a ‘centre’ identity (like that of the SDP) has been successfully established and then starts to lose its appeal; someone who ‘is’ SDP for a couple of years may drift back to Labour when the spell breaks. But the difference between a surge that turns into a slow fade and one that ends in a sudden collapse is secondary to the key similarity between the two, which is that they draw votes away from Labour without the centre party ever having any prospect of taking power in its own right; the result is therefore to entrench the Tories in power. This was the effect of the 1974 surge (collapsing in 1979), the 1983 surge (fading through 1987 and 1992) and the 2005-10 surge (collapsing in 2015). In the 37 years between May 1979 and the present day, the Liberal Democrats and their predecessors have been in power for five years, promptly followed by the collapse of their vote (from 22.9% to 7.8%); the Tories have been in power for 24. Under FPTP, a centre-party surge – or a fortiori a new centre party – will always help the Tories. Anyone advocating a new party needs to be aware that this will be the result.

As for Labour’s individual membership over the years, it looks like this (figures x1000).

Screen Shot 2016-07-02 at 22.58.10

Prior to 1980, constituency Labour Parties had been constitutionally required to have a membership of no less than 800; needless to say, the main result of this policy was to make official figures less than reliable. Figures from 1981 on – assembled here from several different sources – seem reasonably trustworthy. What we can see here is that nothing much happened, in terms of individual membership, from 1981 to 1993. There were a couple of small surges – in 1984 when Kinnock became leader; in 1989-90 after a rule change enabled members to join ‘centrally’ instead of through a party branch – but nothing with any major or lasting impact. The New Labour surge of 1994-6, which took the party membership from 260,000 to over 400,000 in three years, was extraordinary and unprecedented. So too was the New Labour slump which followed almost immediately, taking the party membership back down from 400,000 in 1998 to 300,000 in 2000, 250,000 (2001), 200,000 (2004) and on down to 2009’s trough of 150,000. Like Kinnock before him, Ed Miliband attracted some new members; membership jumped back up to 200,000, but then stuck there. In fact, membership hovered around this (historically low) level until the 2015 leadership election. At that point – and, more importantly, ever since then – the party has recruited like never before; if Tony Blair raised membership by 60% in three years, Jeremy Corbyn has more than doubled it in two.

What’s interesting is the politics of individual membership. In 1981 – where our chart begins – the Labour Party had reasonably well-functioning, if idiosyncratic, democratic structures for deciding policy, but elected its leaders by the votes of MPs alone. (The Conservative Party had a similar system, and still operates a Parliamentary ‘vote of no confidence’ system, administered by the 1922 Committee. It’s what you’d expect from a party founded as a supporters’ club for a group of MPs; it’s less appropriate for a party which began life as an extra-parliamentary movement.) Votes on policy matters were cast by constituency Labour parties and by affiliated unions, both of which often came down rather to the Left of the parliamentary party. The cause of “one member, one vote” was advanced in the early 1980s by right-wingers including Frank Field, who intended it as a brake on the Left: the assumption was that the left-wing domination of CLPs was only possible because organised minorities had hijacked branch structures, and that the views of individual party members would be a better reflection of the ‘common sense’ of the party.

The policy of ‘one member, one vote’ made very little headway in the party, partly because of the perceived importance of the union link and partly (not unrelatedly) because OMOV was embraced by the right-wing splitters who founded the SDP. OMOV for leadership elections had a very limited and qualified implementation in the form of an ‘electoral college’, whereby the votes of MPs, affiliated unions and individual members each counted for a third of the final vote. When it came to policy-making, many on the centre and Right of the party were concerned that party membership was too small to make OMOV work, particularly if it was implemented on a constituency-by-constituency basis. Neil Kinnock in 1992 expressed “fears that one-member-one-vote would leave the more moribund local parties, with only 120 or so members, open to Militant or other infiltration”. (The average CLP membership in 1992 was 425.) He concluded that “MPs will simply have to ensure membership is large enough to prevent cliques taking over”; the risks of OMOV could be mitigated by keeping membership high.

Throughout the 1980s, successive leaderships bemoaned the gap which they believed to exist between activists and ordinary party members, but did very little to resolve it; this was partly because introducing OMOV for policy-making would have alienated the third element of the party, the affiliated unions. The problem remained unsolved until New Labour’s ‘Party Into Power’ reforms cut the knot, not by empowering party members but by disempowering local parties – and affiliated unions – altogether, bringing the ruling National Executive largely under the control of the party leadership and turning the annual conference into a rally rather than a policy-making forum. Under these conditions, when membership offered no possibility of holding the party’s national representatives to account, it is not surprising that membership went into decline – or that it declined even more steeply than the party’s vote did in the same period. (At the 2010 General Election Labour took 64% of the votes it had won in 1997; Labour in 2009 had 37% of the individual members it had had in 1997, rising to 49% by the end of 2010.) The decline was reversed – and then some – when the 2015 election was run with a revised version of the electoral college, based on OMOV in three groups: party members, registered party sympathisers and individual members of affiliated organisations (trade unions and others). The party membership now stands at a historic high. While party members have no more power over policy decisions than they had under Blair, they do now have the power to vote for and against party leadership candidates, and this form of OMOV has proved to be quite a draw.

The Labour Party as a membership organisation has often been at odds with the Labour Party in Parliament. What’s striking about the current crisis is that Ed Miliband’s electoral reforms have both revitalised the membership and given it the power to articulate that antagonism – and all this using a reform which was originally intended to take decision-making powers out of the hands of the Left. I suppose it’s to the credit of some on the Right of the party that they realised what was at stake so quickly – although any credit for insight needs to be qualified to take account of their extraordinary lack of tactical knowhow. In July 2015, for example – while the election was still in progress – the Independent printed this:

Two internal polls … suggested a surge in support for Mr Corbyn, with one even suggesting he could win on 12 September. Although this result is still seen as a long shot, MPs said in the event of a Corbyn victory they would immediately start gathering the 47 names needed to trigger a coup. One said: “We cannot just allow our party, a credible party of government, to be hijacked in this summer of madness. There would be no problem in getting names. We could do this before Christmas.” Another Labour MP said a Corbyn victory would cause deep unhappiness among the current shadow cabinet, and suggested that few would want to serve under him.

Yet talk of a potential coup will cause uproar among grassroots Labour members because, in this scenario, Mr Corbyn would have won in the most democratic leadership contest the party has ever held. A second leadership contest could also lead to the same result.

Some Labour MPs would like the way of toppling a leader changed to ape the simpler, but more brutal, system used by the Conservatives. … “The 1922 is a good model for Labour to follow,” said one fast-rising Labour MP.

The courage and audacity of these people – choosing anonymity rather than come out as an enemy of somebody they didn’t expect to win – is only matched by their strategic insight: they knew they couldn’t win under the current system, and their solution was to (a) daydream about alternative systems that would let them win and (b) plan on going for it anyway. (I wonder who that ‘fast-rising Labour MP’ was, and if (s)he’s still rising fast.)

In August 2015, Prospect printed some bizarre musings from Peter Kellner, who was concerned that Corbyn – if elected – might do too well:

Labour could do deceptively well in polls, by-elections, European and local elections in the next three or four years. Corbyn’s Labour could harness the protest vote, as the Lib Dems did for decades, and the Social Democratic Party did in the 80s. … This is bad news for Labour MPs who hate the idea of Corbyn as their leader, and are hoping for early evidence that he is a vote-loser. … Corbyn’s internal opponents should not rely on him doing so badly as leader in the next year or two that he will have to quit. They may need a different and far more dramatic Plan B. The only way to escape his orbit may be for them to split the party.

The article ends there; presumably Kellner had just used up his wordcount and had no space to say any more. It’s a shame; I would have been interested to know how he reconciled denouncing Corbyn as a ‘vote-loser’ with a positive recommendation of splitting the party. To be fair to Kellner, he may not have intended to endorse ‘Plan B’ – or may have thought better of it – as in January this year he wrote this in the New Statesman:

Corbyn’s opponents should not split the party – at least not yet and not ­unless conditions make it absolutely inevitable. The only beneficiaries would be the Tories. But does this mean surrendering the doctrinal high ground to Corbyn by accepting that he has a mandate to impose his views? Emphatically not. Together, Labour MPs won 9.3 million votes last May. Just[sic] 250,000 people voted for Corbyn to be party leader. Their mandate is much greater than his. They should use it to insist that their ­policies and their doctrine prevail in the Parliamentary Labour Party and in votes in the House of Commons. If they work together they should also be able to wrest control of the shadow cabinet from him; if they can’t, then the anti-Corbyn MPs should leave the front bench and make clear their refusal to accept the shadow cabinet’s authority over how they vote.

If the PLP cannot ­depose him – and it now looks as if it can’t, for if it was to force a new leadership election, he would have the right to stand and would probably win – then its best option is to undermine his leadership at Westminster so completely that he has no alternative but to stand down. Then Labour could have a new leadership contest, in which MPs ensure that nobody with Corbyn’s views receives enough nominations to become a candidate. The far left would kick and scream. Fine. They might tear up their membership cards. Even better. The Labour Party, and the still-powerful Labour brand, would be back in safe hands.

So it’s a No to splitting the party, or at least a Not Yet. Destroying the party in order to save it, however, seems to be very much on the agenda. The insouciance with which Kellner contemplates bullying a democratically elected leader into resigning, then throwing away a 200,000-strong influx of members, is startling. But it’s also instructive. Anyone awake and reasonably sober during the New Labour experiment (which had a powerful tendency to intoxicate) will have noticed the conjunction of a leadership supremely confident in its own decision-making powers, the erosion or dismantling of party democracy and a stampede away from anything that looked like socialism. As shiny and bizarre as New Labour indubitably was, I’m coming to the conclusion that this combination of qualities wasn’t accidental, and that it was an extreme case of a malady that had long afflicted Old Labour.

I mentioned the extra-parliamentary origins of the Labour Party earlier on: Labour began life as the Labour Representation Committee, a group campaigning – necessarily outside Parliament – for the political representation of working people (I owe this point to a rather fine article by Geoffrey Alderman in the Spectator, of all places). Socialism as a direction of travel – the progressive emancipation and empowerment of working people – is of its nature democratic; it cuts with the grain of effective democracy (and I owe that point to my Dad). I think we’re seeing now, with They Live-like clarity, something that’s probably always been there: the fact that there are people in and around the Labour Party whose opposition to socialist policies isn’t temporary or tactical, but absolute and entrenched – and whose view of democracy is strictly instrumental. If wider recruitment and greater party democracy will impede the development of a socialist Labour Party, they’re all in favour. If, as at present, those same things will tend to hasten the development of a socialist Labour Party, they’ll throw those principles overboard without so much as a reasoned argument (who wouldn’t want the Labour Party to be in ‘safe hands’?). And if the only thing that’ll halt the creeping advance of socialism is to split the party and throw the next couple of elections to the old enemy – well, they’ll consider it. Purely as a last resort, you understand.

Caveat lector: I don’t claim to know what’s going on in the PLP. Far more MPs have signed up for the anti-Corbyn cause than could possibly be accounted for by wreckers like Progress, even supported by more reasonable right-wingers like Labour First. But then, I get the impression that the atmosphere at Westminster is both unpleasant and febrile, with hardly anyone thinking straight (this applies to the whole period since the referendum, come to think of it). If Keir Starmer’s resignation letter is anything to go by, a number of Labour MPs have gone along with the coup purely because it appears to be happening, and they don’t want to end up in a bunker with Corbyn and Seumas Milne (literally or figuratively). It looks as if Milne the Media was a bad choice in more ways than he was a good one, and in general I don’t think Corbyn’s been the world’s greatest party leader – although I think at this point we can surely agree that he’s far more sinned against than sinning. But at the end of the day – at the end of several days – Corbyn stands for socialism and democracy, against austerity and against imperialist war. In short, he is the most consistently socialist leader the Labour Party has ever had, as well as being elected by the most democratic procedure the party has ever used – a conjunction which, incidentally, is tremendously hopeful for the future of the party, if that future is allowed to happen.

We need to avoid a split and keep as much as possible of the new membership, and it may not be possible to do either of those things if Corbyn is forced to resign; we certainly won’t be able to do both. If Corbyn does resign between now and 2020, it must be on his own terms – terms which allow his programme for the party to continue, bridge the gap between the PLP and the base, and enable the newly-recruited 50% of the Labour Party to continue as members. Only thus can the party hope to resume its historic function as an instrument of working-class emancipation – which will also enable it to regain relevance to ordinary people’s lives. This would, of course, represent the 180-degree reversal of Peter Kellner’s hopes and the complete failure of the coup. That’s as it should be. The coup must fail.

100 Years Ago (3)

In the last post I discussed a narrative of Labour decline – particularly in predominantly white working class communities – which got a lot of exposure before the Oldham West and Royton by-election. The idea was that Labour was losing the white working class and plugging the gap by appealing either to well-meaning middle-class liberal types or to local ethnic minorities – both of which tactics could only work temporarily, as they would both repel the white working class even more. One exponent of this theory, Stephen Bush, went so far as to apply it directly to Oldham West and Royton, although when asked he explained that he was referring to the General Election result in the constituency:

In Oldham West and Royton, Labour sought salvation in the seat’s Asian vote – but white working-class constituents defected in large numbers, to Nigel Farage’s party, or simply by staying at home.

Is that the kind of thing that’s been happening? Let’s look at some figures. Here’s the vote share in Oldham West and Royton, going back to 1997.

Screen Shot 2015-12-07 at 10.45.24

Note the steady decline in vote share through the New Labour period, following the national trend. (If ever there was a time when the working class was being told Labour wasn’t all about them…) Also note the big third- and fourth-party votes; never, since Nick Griffin stood for the BNP in 2001, have Labour and the Tories together taken as much as 75% of the vote in this constituency. (There was even a couple of percent each for the Socialist Labour Party and the Referendum Party in 1997.) There’s a sizeable sod-the-lot-of-’em vote in Oldham West – and a lot of those people aren’t too fussed about not being called racists.

Now look at the last three results – the 2010 and 2015 General Elections and then the by-election. Do you see white working-class constituents defect[ing] in large numbers from Labour to UKIP, or from Labour full stop? No, me neither. Between the two General Elections, two big changes seem to account for almost all the other differences. Firstly, the BNP didn’t stand, for the first time since 1997: cue a windfall for UKIP. Secondly, a previously strong Lib Dem vote collapsed almost to nothing, as it did in so many other places; most former Lib Dems seem to have gone to Labour, but some to UKIP. Add a little Tory-to-UKIP switching and you’re basically there. I’m not saying there was no Labour-to-UKIP traffic – masked by larger flows into Labour from the Lib Dems – just that this doesn’t seem to have happened on a large scale. My analysis depends on a third or more of the Lib Dem vote going to UKIP, but it’s not as if that’s hard to imagine; as anyone who’s read local election literature knows, local Lib Dem campaigners are adept at picking up protest votes and attracting people who are disaffected with both the major parties. (That’s the polite version.)

As for the by-election result, this looks even simpler: the Lib Dems stayed irrelevant and both Labour and UKIP put on voters at the expense of the Tories, Labour more successfully than UKIP. This doesn’t necessarily mean that there were any vote shifts at all: what may have happened is that UKIP and Labour mobilisation kept turnout relatively high, while Tory apathy, incompetence or simple lack of feet on the ground permitted the turnout of their voters to plummet. (If we compare the numbers of votes cast in the two elections, Labour and UKIP were both down about 27%; the Tory vote was down 70%.) Either way there is – once again – no obvious evidence for the two shifts Stephen Bush wrote about – from Labour to UKIP and from Labour to abstention. It looks more like straightforward polarisation, with Labour and UKIP fighting over Tory votes in much the same way that, seven months earlier, they’d fought over the spoils of the local Lib Dems.

Can we make Stephen’s model work? Voters only have to vote – there’s no requirement to fill in a form detailing their previous voting history; three- and four-way shifts are increasingly common, making a mockery of simple ‘swingometer’ pictures of vote movements. We know what the headline figures look like, but is it possible that the process Stephen describes was going on in Oldham West and Royton, in May 2015, in December 2015 or both? If it’s going to work at all, in fact, it does need to work for both elections: nobody has suggested that the supposed disaggregation of the Labour base is something that wasn’t happening at all before Corbyn was elected leader, still less that Corbyn’s election stopped it happening. These are long-term trends which, it’s generally agreed, haven’t been rectified by Corbyn’s election, and may even have been exacerbated.

If they exist, that is.

The next bit involves numbers, so buckle up. The proposition we’re testing is “white working-class constituents defected in large numbers”, from Labour to UKIP and from a Labour vote to abstention. I’ll define ‘large numbers’ as 5% of the turnout: Labour losing 2-3% of its support would hardly qualify as a trickle turning into a flood (and I think a party attracting voters in ‘large numbers’ would be able to keep its deposit!). So that’s 2,000 people in the General Election, 1,400 at the by-election. I’m also assuming that, when Stephen wrote that Labour voters defected (in large numbers) “to Nigel Farage’s party, or simply by staying at home” the implication is that large numbers of voters did both of these things: 2%-worth of UKIP switchers would look more like a trickle than a flood, even accompanied by 3% abstention.

So: between 2010 and 2015 Labour in Oldham West and Royton lost 2,000 votes to UKIP and 2,000 to abstention (but “sought salvation in the seat’s Asian vote”). Can this possibly be true?

The first problem here is that, between 2010 and 2015, the Labour vote rose by 4,000. (UKIP’s vote was up 7,500; the Tories were down 2,000 and the Lib Dems down 6,500, while the BNP (not standing) were in effect down 3,000.) Assume a 2,000-vote flow from Labour to UKIP and you have to assume that the Labour vote actually went up 6,000, presumably taking almost all of the Lib Dem vote. I don’t have any difficulty believing that the 2010 Lib Dem vote broke disproportionately towards Labour – it happened all over the country – but I do find it hard to believe it broke towards Labour by a factor of 12:1.

As for turnout, here we need to look at the demographics. “Around a fifth of the electorate is of Bangladeshi or Pakistani heritage”, said Rafael Behr. He may have better data than me, but the 2011 Census said that the population of Oldham is 80% White British and 13.5% Asian, which is a bit different. The Asian population of Oldham is concentrated in five wards, two of which are in the Oldham West and Royton constituency, so I wouldn’t expect the Census figures to be far out; I’ll work on the basis of 80% White British and 14% Asian, which is to say that there are approximately 55,000 White British people on the electoral roll and 10,000 Asians.

The contention we’re dealing with here is that White working-class Labour voters abstained in “large numbers” – say 2,000 of them, above and beyond any transfers between parties. So 2,000 White voters abstained, and their place was taken by 2,000 additional Asian voters. Instead of an overall turnout of 60% reflecting 60% turnout across all groups, turnout was lower among Whites and higher among Asians. 60% x 55,000 = 33,000; actual White turnout, without those 2,000 votes, would be 31,000 or 56%. And 60% x 10,000 = 6,000; actual Asian turnout would be 8,000… or 80%. As turnout figures go, that’s staggeringly high. As with the 12:1 split of Lib Dem votes to Labour rather than UKIP, it’s not outright impossible, but it’s very hard to believe without compelling evidence in its favour. (And in this case there’s basically no evidence in its favour, other than word of mouth from disgruntled Labour voters – a topic I’ll come back to.)

What really kills this theory, though, is the by-election. OK, so you’ve staved off disaster by replacing one lot of UKIP defectors with most of the Lib Dem diaspora, and another lot with hyper-mobilisation of the local Asian community: what’s going to happen next time? If “white working-class constituents” had “defected in large numbers” in May, there would have been absolutely no reason not to expect another tranche of defections in December; on the contrary, electing Corbyn to replace Miliband – who did at least look good in a suit – should have stepped up the defection rate. Let’s suppose that we start from the basis that everything happens in December just like it did in May, but on a 2/3 scale, as there’s a 40% overall turnout instead of 60%. So we’re expecting roughly 16,000 Labour votes, 6,000 UKIP and 5,400 Tory, on the basis of 38% White turnout and 54% Asian turnout (2/3 of 56% and 80%, respectively). In fact 17,000 people voted Labour, so we’ve got to gain 1,000 votes from somewhere. But – whoops – there go 1,400 White Labour voters, abstaining and being replaced seamlessly by Asian voters; turnout is now 35%, while Asian turnout has shot up to 74%. Perhaps that’s not outright impossible, but both the 2:1 disparity between communities and the figure of 74% itself would be very, very unusual, particularly in a by-election. It’s far more likely that Asian turnout would stay around about where it was, the White Labour abstainers would not be replaced – and the Labour vote would fall, instead of going up by 1,000. And as if that wasn’t bad enough, we’ve apparently lost another 1,400 Labour defectors to UKIP, so we’re short by 2,400 votes. Where are they going to come from? Not from the Lib Dems – we’re only expecting 1,000 of those to start with (which is also how many we got). Tory voters transferring to Labour – Corbyn’s Labour? Hardly.

In short, and with less maths, the “white working-class constituents defect in large numbers” story, in Oldham West and Royton, will hold up in the face of one good result for Labour – but only one. Those Lib Dem transfers and those newly-mobilised Asian voters are non-renewable resources: if the drift away from Labour had happened in May 2015 and then again in December, the Labour share of the vote would inevitably, necessarily have gone down. Even if the drift away from Labour had started after the General Election – which of course wasn’t what Stephen Bush was suggesting – the disappearance of local support for the Lib Dems would by now have taken away the only place Labour could get reinforcements. If “white working-class constituents” were “defect[ing] in large numbers” to abstention and UKIP, there is no way in the world that Labour’s share of the vote would not have gone down substantially at the by-election. And (new readers start here) it didn’t – it went up, from 55% to 62%.

In part 4: why? I mean, seriously, why?


TCM 3 – When the government falls

Andrew Rawnsley in Sunday’s Obs, pushing the line that Labour is not one party but two:

Those with a vote in the contest who are still unsure which Labour party they should be backing have been provided with a clarifying test by Dave Ward, the general secretary of the Communication Workers Union. Announcing the CWU’s endorsement of the MP for Islington North, Mr Ward declared that the union’s executive had acted on medical advice: “There is a virus within the Labour party and Jeremy Corbyn is the antidote.” “The virus” being the Blairites.

Incidentally, this was not an off-the-cuff remark in an interview. It is the language used in the formal declaration made by the union so we must assume that the CWU weighed its words before deciding to compare the former Labour prime minister to a disease. If you think New Labour was the political equivalent of Ebola, then you probably belong in the Corbyn Labour party. If you think that three election victories and 13 years in power had something to commend them, you should probably be in the non-Corbyn Labour party.

This isn’t the Scottish post – I’m putting that off for a second time. This one is sparked by that reference to “three election victories and 13 years in power”. I thought it would be worth having a systematic look at the figures in the first post and drawing some conclusions about what actually happened at all those elections. For each election that led to a change of government, I’ll highlight what seem to have been the main causal factors; I’ll also flag up any elections where the government has fallen despite the government vote not having dropped, the opposition vote not having risen, or both.

At the 1950 election Labour hung on to power, but it was a close thing; relative to 1945 the Labour vote grew by 10% but the Tory vote grew by nearer 30%, helped by a substantial rise in turnout. The Tories were still some way short of an overall majority, though. The 1951 election saw another increase in the vote for the Conservative Party, this time by 12%, giving them 13.6 million votes compared to 12.1 million in 1950 (and 8.6 million in 1945); however, the vote for the governing Labour Party also rose, from 13.3 to 13.9 million votes. So what happened?

A conventional explanation is that the Labour Party didn’t win votes in the right places, “piling up majorities in safe seats” while those wily Tories targeted their efforts at winning winnable seats. However, there’s a much simpler explanation, which is that the Liberal Party was broke. The Liberals had had a dreadful election in 1950, losing 300 deposits; another general election a year later was the last thing they wanted. In 1951 the Tories made a net gain of 20 seats, based almost exclusively on 21 seats that went directly from Labour to the Tories. Almost all of these were two-way fights – and in almost all of those there had been a Liberal candidate at the previous election. While there was a small rise in abstentions, the Liberal vote broke disproportionately towards the Tories; the main effect of the absence of a Liberal candidate was to bolster the Tory vote. So there’s our first data-point:

1951 Change of government due to MINOR-PARTY COLLAPSE; government vote UP

The Tories, bless their black hearts, hung on to power until 1964. In 1955 and 1959 the Tory vote gradually increased; the Liberal vote collapsed and then rebounded under the forward-looking leadership of Jo Grimond; and the Labour vote steadily declined. If we compare the election at which the Tories finally lost power with the previous one, however, the Labour vote was all but unchanged; in fact it had continued to fall, if only by 10,000. The big difference is a slump in the Conservative vote, offset by a rise in the Liberal vote; the two parties’ votes together dropped by 300,000 between the two elections, but the Tories’ vote alone fell by 1.7 million. The long period of Conservative domination had created the conditions for the Liberals to undermine a complacent government and differentiate themselves from an ineffectual opposition; Eric Lubbock’s 1962 by-election victory in Orpington, in particular, put wind in the Liberals’ sails. The result, ironically, was victory for a party whose vote had declined at every election since its defeat in 1951, and was now 1.7 million lower.

1964 Change of government due to GOVERNMENT DECLINE and MINOR-PARTY RESURGENCE; opposition vote DOWN

In 1966 Labour consolidated its position in government with an early election, in which it took votes from both the Conservatives and the Liberals. In 1970, however, the Conservatives won an election called tactically a year early, to the great surprise of the government and the opinion polls. The electorate had recently expanded with the enfranchisement of 18- to 20-year-olds; however, this does not appear to have affected the result, other than in a sudden increase in the number of non-voters. The government lost, the Opposition won; for once there isn’t a lot more to say.

1970 Change of government due to GOVERNMENT DECLINE and OPPOSITION STRENGTH

In February 1974, the governing Conservatives failed to achieve an overall majority owing to a combination of effects: a slight decline in the Conservative vote; an increase in the number of Labour MPs (despite a drop in the Labour vote); a near-tripling of the Liberal vote, eroding the votes of the two main parties and draining the pool of non-voters; and, not least, the decision of the Ulster Unionists not to take the Conservative whip (this alone would have stopped the Conservatives being the largest single party). The process whereby this combination of circumstances led to a minority Labour government was complex; what can be said, though, was that it had very little to do with voters preferring the Labour Party to the Tories. (In fact the Tories took more votes than Labour – and, compared to the previous election, both parties’ votes had declined.)

1974 Change of government due to MINOR-PARTY RESURGENCE and MINOR-PARTY REALIGNMENT; opposition vote DOWN

As 1966 had followed 1964, the minority government of February 1974 was followed by a fresh election in October; this was marked by a slump in turnout which hit the Tories and Liberals harder than the governing Labour party. 1979 is an interesting one, partly because (in retrospect) it was the end of British politics as we knew it, but mainly because the Tories’ victory had nothing to do with any change in the Labour vote. Compared to (October) 1974 Labour’s vote actually went up – Winter of Discontent, “crisis? what crisis?” and all. The figures suggest that Thatcher won by poaching votes from the Liberals and the SNP (1.5 million votes) and by mobilising non-voters and new voters (1.9 million votes). This isn’t too surprising when you think about it: Thatcher was a classic populist opposition leader – the politician who said things the others wouldn’t dare, who was going to teach the others a lesson, shake up the system, etc. In other words, she was the politician who people disillusioned with politics would vote for. And they did.

1979 Change of government due to OPPOSITION STRENGTH and MOBILISATION OF NON-VOTERS; government vote STATIC

In 1983 both main parties were affected by the advent of the SDP – Labour, for obvious reasons, more than the Tories. The SDP campaigned on ‘centrist’ policies: a series of posters portrayed the party as splitting the difference between Labour and Tory, or simply promising to maintain what was then the status quo instead of moving to the Right or Left. (One poster associated Labour with nationalisation and the Conservatives with privatisation; the SDP, by implication, would pursue neither. Taken literally, this would have meant keeping British Gas, British Telecommunications, British Coal and British Steel in the public sector, among much else.) There were some shibboleth issues, mainly in foreign and defence policy – maintaining an independent nuclear deterrent (pending multilateral nuclear disarmament) and ending any thought of withdrawing from the EEC or NATO – but in retrospect the SDP seems much less of a break with Labour’s past than New Labour would be. The new party’s positioning demanded sustained denunciations of Labour as excessively left-wing; whether there was enough of an ideological gulf between the two parties to justify this approach now looks rather dubious.

In any case, the new party’s 3.5 million votes coincide with a drop of 700,000 in the Tory vote and 3 million in the Labour vote. Blame for the low Labour vote is generally assigned to Michael Foot’s leadership and to the party’s left-wing manifesto. I think there’s some blame to spare for four former Cabinet ministers – including the best Home Secretary Labour ever had – who used extensive social and media connections to advertise their own rectitude and denounce the party which had enabled them to achieve anything, but that may just be me. It’s certainly hard to imagine that three million voters would have been sufficiently revolted by the state of the Labour Party to vote Liberal or abstain if the Gang of Four hadn’t left the party. (As it was, abstentions rose by 1.5 million; the euphoria of the SDP moment wasn’t for everyone.)

Over the next two elections some normality returned; the centre vote and the pool of non-voters were squeezed as both Labour and Tory votes rose. The Tory vote didn’t start falling until 1997, when John Major’s first full term limped to an end and another era began. How did Labour win? Like Thatcher, they squeezed some votes out of the Liberal (Democrat) area, but there the similarity ends. Unlike 1979, the governing party’s vote collapsed; also unlike 1979, the number of non-voters rose sharply. (Labour vote: up 1.9 million. Tory vote: down 4.5 million. Non-voters: up 3.3 million.) Yes, New Labour made the news and set the agenda – as the SDP did before – but in large parts of the country it looks as if what they really succeeded in doing was (in the immortal words of Willie Whitelaw) “stirring up apathy”.


2001 and 2005 were 1987 and 1992 in reverse: the Labour vote declined steadily (to levels below those of 1992 and 1987 respectively), but Labour won both elections handsomely. In 2005 I myself was one of those arguing that left-wingers shouldn’t vote Labour. Labour had been alienating its historic working-class base since 1997; the Left was more tenacious, but after Iraq a lot of us followed suit. That said, for me at least the advice not to vote Labour was explicitly on the basis that Labour wasn’t going to lose, whatever we did; the after-effects of the landslide of 1997 meant that the Tories still had a mountain to climb, even in 2005. But 1997 was a wasting asset. Like the Conservatives, Labour had a decade under the charismatic leader who had spearheaded their original victory, followed by a partial term under that leader’s successor; unlike the Conservatives, these were years of steadily diminishing electoral returns, culminating when Gordon Brown departed from John Major’s example by failing to win an election as leader in his own right.

2010 was in some ways an unremarkable election, repeating trends from the last election or two. Labour’s vote had been dropping; it dropped further. Abstentions had fallen in 2005; they fell again (although the level of non-voting was still extremely high). Votes for the Tories, Lib Dems and UKIP had grown; all three grew again. The difference was that Labour no longer had enough of a cushion of votes to withstand these trends; the party duly lost its overall majority, ushering in a new age of coalition politics (we wish). In 2015, of course, things got complicated – and the government didn’t change – but that’s another story.


Seven election defeats isn’t much of a dataset, but let’s see if anything jumps out. The simple, seesaw pattern of vote change – more votes for the opposition party, fewer for the government – is involved in only three defeats, two of them of a Labour government (the third was the 1997 Labour victory). The Tory victory in 1979 rested on the strength of the opposition party, combined with voter mobilisation; Labour’s victory in 1964 rested on declining government support, even though the beneficiaries were the Liberals and not Labour. The other two changes of government – the Tories’ victory in 1951 and Labour’s in 1974 – rested mainly on minor-party effects.

Minor-party effects were involved in four results overall: a collapse benefiting the Tories; two resurgences, both benefiting Labour; and two realignments, one benefiting the Tories and one Labour. An increased opposition vote was a factor in three out of four Tory election victories, but only one out of the three Labour victories (1997). A declining government vote was a factor in four of the seven (two Tory, two Labour). There are some odd effects if we compare vote changes and election outcomes more broadly. Ten elections led to the Tories either taking or remaining in power; the Tory vote increased in all of these except 1983, when the Tory vote fell by 0.7 million. The other eight post-1945 elections led to Labour either taking or remaining in power; in as many of six of these, the Labour vote fell. Conversely, the Labour vote rose in five out of its ten defeats (1979 included); the Tories’ vote rose in only two of their eight defeats. Labour never seems to have won on the back of increased voter mobilisation, nor the Tories on the basis of demobilisation. The evidence generally suggests that the Tory vote is more solid than Labour’s and easier to mobilise; an uncomfortable number of those Labour wins look narrow or lucky. Moreover, despite the increase in the Labour vote between 1992 and 1997, this clearly isn’t a problem that New Labour fixed – or not without also driving down the Labour vote, with ultimately self-destructive consequences.

If Labour is going to win again, the party is going to have to repeat some of those tricks and learn some new ones. Specifically, I think they’ll need to learn to mobilise, if the next Labour victory isn’t going to be as chancy as 1964, as fragile as February 1974 or as unsustainable as 2005; that means having something to offer new voters and non-voters. Even if they don’t break with the low-mobilisation past – or rather, especially if they don’t – they’ll need three things: a strong centre, to chip away at the Tory base as in 1964 and 1974; a tired, discredited, and ineffectual government, as in 1964 and 1997, which means both making them look tired and discredited and making sure they are ineffectual; and strength in numbers, which means (among other things) Scotland.

Ah, Scotland…

TCM 2 – Here comes success

Before I get on to Scotland, here’s another way of looking at the figures in the last post.

Leader 1: +1,300,000, +670,000, -640,000
Leader 2: -1,090,000
Leader 3: -10,000 (W), +890,000 (W), -890,000, -560,000 (W), -190,000 (W)
Leader 4: +70,000
Leader 5: -3,070,000
Leader 6: +1,570,000, +1,530,000
Leader 7: n/a
Leader 8: +1,960,000 (W), -2,800,000 (W), -1,170,000 (W)
Leader 9: -940,000
Leader 10: +740,000

Or to look at it another way:

Leader 6: +3,100,000
Leader 10: +740,000
Leader 4: +70,000
Leader 1: -10,000
Leader 3: -760,000, 4 election wins
Leader 9: -940,000
Leader 2: -1,090,000
Leader 8: -2,010,000, 3 election wins
Leader 5: -3,070,000

Judged over their whole careers as leader, Labour’s three biggest vote-winners – in fact, their only vote-winners – are Neil Kinnock, Ed Miliband and James Callaghan, in descending order; the three biggest vote-losers are Hugh Gaitskell, Tony Blair and Michael Foot, whose stewardship of the party saw it lose one, two and three million votes respectively. (Honourable mention to Gordon Brown, in a close fourth place with a net loss of over 900,000 votes.) Tony Blair, like Harold Wilson before him, drove the party’s vote up and then drove it right down again. Wilson became leader after a 12.2 million-vote defeat and resigned after winning an election with 11.5 million votes (a post-war low), paving the way for the defeat of 1979; Blair took over after an 11.6 million-vote defeat and resigned after winning an election with 9.6 million votes (from a substantially larger electorate), making the defeat of 2010 all but inevitable. A lot of recent commentary has bracketed Ed Miliband with Michael Foot, as left-wing leaders who presided over humiliating defeats (never mind the fact that Foot’s defeat was brought about by leading members of his own party). But the answer to the quiz question “which Labour leader had the second largest loss of votes?” isn’t Ed Miliband.

You could say that this is beside the point; what matters is to win elections, and on that metric Harold Wilson (say) beats Ed Miliband 4-0, despite having lost Labour more votes than Miliband gained. In other words, we should praise Wilson – and praise Blair – for finding tactics that won the party elections, even if they also drove supporters away. The problem with this argument is threefold. Firstly and most obviously, it lets New Labour (and any remaining Wilson apologists) have their cake and eat it: attracting 1.9 million new votes in 1997 shows how popular Blairism was, but driving them all away (and then some) four years later doesn’t matter, since after all the election was won anyway. We can’t really have it both ways: if New Labour was popular, the figures say that its popularity very rapidly ebbed away. Secondly, however effective New Labour’s laser-like targeting of swing voters may have been, any strategy that alienates that many of the party’s own voters is by definition of very limited use. If the election in which Ed Miliband’s leadership gained Labour 700,000 votes did more damage to the party’s standing than Blair’s massive losses in 2001 and 2005, it’s because Blair had the luxury of being able to lose all those votes. Thirdly and perhaps most importantly, “do what wins elections (even if it loses votes)” isn’t a strategy that a party can actually use. As we saw in the last post, apart from increased votes, “what wins elections” is mostly outside the winning party’s control: Labour’s election victories were created, to a large extent, by the parties led by Jo Grimond, Jeremy Thorpe and John Major. Of course, there are questions about making gains in the right places – better a narrow win over a Tory than a thumping majority in a safe seat. But, if we discount a pure swing-voter focus-group strategy – with its proven risk of alienating existing voters – what’s left is mostly technical questions of party organisation: any party, Left or Right, needs to target resources on winnable and vulnerable seats. (Of course, calling this a technical question isn’t to say that it’s insignificant. According to election post mortems several of the English seats Labour lost in 2015 could have been held with better organisation; if all seven had been held the Tories would have been three seats short of a majority. What might have been…)

Winning elections by hook or by crook is handy, but it makes more sense to judge success for a party – or party leader – in terms of numbers of votes; there are fewer factors involved, and a lot fewer factors outside the party’s control. And on that metric Ed Miliband really doesn’t look that bad – particularly if you bring Scotland in. Let’s suppose – as a lot of commentary does – that the SNP landslide was, in effect, just that: an unstoppable natural phenomenon, which couldn’t be predicted precisely but was bound to happen sooner or later. (I don’t think this is correct, but we’ll stick with it for the sake of argument.) Overall, the difference between Labour’s 2010 and 2015 performances (discounting by-elections) was a gain of 740,000 votes and a loss of 26 seats, but if we separate out the constituent nations of Great Britain the figures look a bit different.

Wales: +20,000 votes, 1 gain, 1 loss
Scotland: -330,000 votes, 40 losses
England: +1,050,000 votes, 21 gains, 7 losses

Labour put on over a million votes in England – in an election where the Green vote also increased by a million. (Some of the latter will have been former Lib Dems, but not all of them.) As you can see from the list above, this is a kind of increase in votes which Labour has only managed a handful of times since 1945 – once under Blair, once under Attlee and twice under Kinnock. Another interesting perspective shift is imagining what would have happened if somebody had kicked the crucial pebble ten years earlier, so that the landslide election was the one in 2005, not 2015. Take another 330,000 votes and 40 seats from Labour; suddenly Labour are nine seats short, and questions are being asked about Tony Blair’s ability to lose the party 1,500,000 votes, 87 seats and its majority.

That last part is a counter-factual – and, as it goes, I don’t think the SNP landslide was either inevitable or unpredictable, let alone that it could have happened as early as 2005. But the 2015 gain of a million votes in England is real. Doubtless much of this was a one-off gain from a minor-party squeeze – just as much of the vote loss in 1983 derived from a one-off minor-party surge – but the numbers do suggest that Ed Miliband’s leadership was doing something right, at least in England: something which should be built on rather than being repudiated. And there’s certainly nothing here to suggest that Blairism is a proven vote-winning strategy. If anything it’s a proven vote-losing strategy, which also wins elections – but only if the party’s support is already strong enough to absorb the loss of votes.

Next: the Scottish play (and surprise everyone).

TCM 1 – The past is prologue

This is the first in a series of posts on Jeremy Corbyn’s leadership campaign and the issues it raises.

The charge most commonly thrown at Corbyn from the Right is that Labour couldn’t possibly win in 2020 under his leadership. So I’m going to start by looking at how Labour’s won before, and at trends in voting at General Elections more generally. I’m going to argue that vote shifts in General Elections since 1945 can be modelled using a reasonably small set of effects:

1. Straight Vote Switch

2. Mobilisation Effects
2.1. Selective Mobilisation (Benefiting Incumbent)
2.2. Selective Mobilisation (Benefiting Opposition)
2.3. Selective Demobilisation (Benefiting Incumbent)
2.4. Selective Demobilisation (Benefiting Opposition)
2.5. General Demobilisation
(I haven’t got any examples of general mobilisation.)

3. Minor-Party Effects
3.1. Minor Party Surge (Benefiting Incumbent)
3.2. Minor Party Surge (Benefiting Opposition)
3.3. Minor Party Squeeze (Benefiting Incumbent)
3.4. Minor Party Squeeze (Benefiting Opposition)

4. Incumbency Effects
4.1. Consolidation In Office
4.2. Fightback In Opposition


Disclaimers: I’m not a political scientist or a historian of the period (although I have lived through most of it); for the most part I’m going to be ignoring day-to-day or even month-to-month politics and discounting the two- or three-way shifts in loyalty which take place in reality. It’s a simplistic model, but I think we can make it talk.

Labour won the 1945 election with a majority of 146, gained from 12 million votes – just under 48% of the total votes cast, and more than the votes gained by the Tories and Liberals combined. (The thin green line you can just make out in the left-most column, incidentally, represents 300,000 votes cast for the Communist Party and the left-wing Common Wealth party – votes which elected two MPs and one, respectively.) The figure to focus on here is the 8.9 million non-voters (turnout was 72.8%). In 1950 the voting public made up a significantly larger proportion (83.9%) of a larger electorate. Labour picked up some of the increase, but the Tories picked up more; Labour won with an overall majority of five. This is the first pattern I want to highlight: (2.2) Selective Mobilisation. Mobilisation in this sense isn’t a matter of winning voters from one party to another, or even ‘getting the vote out’ in the door-knocking and lift-to-the-polls sense, but of motivating potential voters: making the political weather to the point where voting for a particular party seems sensible. The point is simply to persuade supporters to vote rather than not bothering; getting them to feel that turning out to vote is a good idea, even if it hadn’t seemed to be before. Given the increased size of the electorate it’s hard to be certain where the votes for any party came from, but the fact that abstentions fell by 3.4 million and the Tory vote rose by 2.5 million looks decidedly suggestive.

The second pattern I want to draw attention to makes its appearance in 1951, when the Attlee government ill-advisedly called an election in the hope of increasing its majority. Labour certainly increased its vote, winning the most votes the party had (and has) ever won, but the Tories increased theirs more and did so more effectively. The key mechanism here was the (3.4) Minor Party Squeeze. The 1.8 million votes shown here for Liberals includes 1.1 million votes for the ‘National Liberals’, a Tory-allied splinter dating back to Ramsay MacDonald’s coalition government. The National Liberal vote went up slightly between 1950 and 1951, but the true Liberal vote plummeted from 2.6 million to 700,000. A third-party squeeze allowed the second party to achieve a gain of 22 seats – and a change of government – even while the Labour vote increased; Labour vote share in 1951 was 48.8%, even higher than it had been in 1945.

If we look at the next two elections – 1955 and 1959 – two patterns are discernible. One is (4.1) Consolidation In Office: over three successive elections the Tory vote grows, little by little, and the Labour vote declines. The other, particularly apparent in 1955, is (2.3) Selective Demobilisation, benefiting the party in office. The reverse of mobilisation, this – again – isn’t a matter of persuading opposition voters to switch parties, but simply demoralising them to the point where they stay at home. (I’m not saying that this was Tory party strategy – or even that anyone set out to achieve it; the main agents of Labour voter demoralisation may well have been the Labour Party. I’m just saying that the figures seem to suggest that it happened: electorate up by half a million or less, non-voters up by 2.1 million, Labour vote down by 1.5 million.) Patterns 3 and 4 are both essentially Labour/Tory phenomena in these years; the Liberal vote is unaffected, holding up between 1951 and 1955, then growing markedly in 1959 under the incoming leadership of Jo Grimond. (What looks like a collapse in 1955 is down to the almost total dissolution of the National Liberals into the Tory Party.)

The Tories’ reign came to an end in 1964, when the Labour vote leapt from 12.2 million to, er, 12.2 million; the national vote was actually 10,000 lower in 1964 than 1959. What had changed, however, was the Tory/Liberal share of the vote – or, to be more precise, the Tory/Liberal/non-voter share of voters. Here we see another pattern: the (3.2) Minor Party Surge, cracking the political pack-ice to the benefit of the main opposition party (in this case, Labour). Which in turn called an early election in 1966, pulling off a classic example of (4.1) Consolidation In Office by taking votes both from the Liberals and directly from the Tories.

What happened next?


The size of the electorate jumped between 1966 and 1970, due to Wilson’s government giving 18-year-olds the vote. As you can see, Labour didn’t gain from this. In fact none of the patterns identified up to now really fits the way that Edward Heath’s Tory government came to power. Let’s just call it a (1) Straight Vote Switch – the simplest and (one might assume) most common way for an election to change things in a two-party system, appearing now for the first time in seven elections.

After the February 1974 election, Harold Wilson formed a minority government reliant on, among others, the Ulster Unionists, who had just broken with the Tories. What made it possible was, once again, a (3.2) Minor Party Surge to the benefit of the opposition. (Thanks again, Liberals!) The October 1974 election was called in an attempt at (4.1) Consolidation In Office; unfortunately the Labour vote actually dropped. However, the Tory vote dropped by a lot more; Labour achieved the desired result – a parliamentary majority – through (2.3) Selective Demobilisation. (Labour’s majority was 3 – smaller than the majority which had prompted Attlee to call the 1951 election.) This was also the period of the SNP’s first surge, from 100,000 votes in 1966 to 300,000 in 1970, 600,000 in February 1974 and now 800,000. The devolution referendum in 1979, closely followed by Margaret Thatcher’s election, would put this into reverse.

As for Thatcher, what these figures suggest very strongly is that her victory in 1979 was almost entirely a question of (2.2) Selective Mobilisation, with a bit of (3.4) Minor Party Squeeze on the side. The Tories may have taken votes from the Liberals and the SNP, but there is no sign that they took any directly from Labour; the Labour vote actually went up compared to October 1974. (An alternative reading is that a Labour->Tory outflow was balanced by SNP->Labour and Liberal->Labour inflows. Either way, Labour didn’t get any less popular.) The main contributor to the massive increase in the Tory vote – from 10.5 million to 13.7 million – seems to have been a drop of nearly 2 million in the number of non-voters. The thin dark blue line you can make out near the top of the column, incidentally, is Britain’s first substantial far-Right vote: 200,000 votes for the National Front.

Bring on phase three:

Screen shot 2015-07-29 at 11.52.51

Research published recently has argued that the ‘Falklands Factor’ had very little effect on the Tories’ poll ratings, and had dissipated by the time of the election the following year. If anything accounted for Thatcher’s second victory, the paper argued, it was our old friend the economy, and people’s subjective perceptions that their prospects were improving. I’m happy to bid the Falklands farewell, but I’m not sure that the voting figures support the second argument. What leaps out is the huge success of the SDP/Liberal Alliance, who put on 3.5 million votes relative to the Liberals’ vote in 1979; Labour’s vote, meanwhile, fell by 3 million compared to four years earlier. The Tories’ vote actually fell, as did overall turnout (from 76% back down to the 73% of October 1974). In short, what we’re looking at here is a rare example of (3.1) Minor Party Surge to the benefit of the incumbent.

The next couple of elections are interesting (to look back on; they were heartbreaking to live through). There’s (4.1) Consolidation In Office, with the Tory vote increasing at both elections; John Major’s 14.1 million is the highest vote ever obtained by any UK political party. But there’s also (4.2) Fightback In Opposition, with the Labour vote also increasing both times – and by substantially more. (The Tories’ share of the vote drops slightly over the period, from 42.4% to 41.9%. Labour’s share increases from 27.6% to 34.4%.) These increases are paid for by a combination of (2.1/2) Selective Mobilisation (benefiting both the leading parties – although Labour more than the Tories) and (3.4) Minor Party Squeeze (in Labour’s favour). Politics over the period was getting steadily more interesting – only 9.5 million non-voters in 1992, the lowest figure since February 1974 – and more polarised between the two main parties; and the balance between the two was steadily shifting towards Labour.

Then came 1997, which seemed to represent a triumphal culmination to the growth of Labour in opposition. Actually, as we can see, it reversed most of the trends which had been operating. The (3.4) Minor Party Squeeze continued to work for Labour, but it was accompanied by something that hadn’t been seen since 1970 – a large-scale (1) Straight Vote Switch – and another shift which hadn’t been seen at all since the War: (2.4) Selective Demobilisation of the incumbent party’s support. The increase in Labour’s vote (1.9 million) was huge; the drop in Tory support was more than twice as big. For those of us who remember this period, this both is and isn’t surprising. We can certainly remember the wheels coming off the Tory Party: John Major’s genius move, putting Thatcher’s transformation of the political landscape in the bank and fronting it with the appearance of reasoned moderation, stopped working more or less as soon as he had to start coming up with policies of his own. The sense that absolutely tons of people were voting Labour nowadays – and that hardly anyone was voting Tory any more – was certainly in the air that May. But look at that huge increase in non-voters – and look at the size of that dark blue stripe (100,000 UKIP votes and 800,000 for Jimmy Goldsmith’s Referendum Party). The idea that Tony Blair’s success, right from the outset, might have depended in part on encouraging a lot of Labour’s opponents not to bother voting – and, perhaps, encouraging another slice of people to go right to the extreme and cast a harmless ‘expressive’ vote – is unpleasantly thought-provoking. All the more so in the light of what happened in phase four:

Screen shot 2015-07-29 at 11.53.02

Just look at 2001. Ladies and gentlemen, I give you, for the first time in a British post-war General Election, (2.5) General Demobilisation. The Greens’ vote had more than doubled and UKIP’s nearly quadrupled – although this didn’t make up for the loss of the Referendum Party – but all the major parties showed big declines in their vote as compared to 1997: the Lib Dems were down 10%, the Tories 12% and Labour 20%. Not only was Labour’s vote was below the level at which Kinnock had lost in 1992; it was below the number of votes gained by Callaghan’s Labour Party in 1979, despite the electorate having grown by nearly 10% over those 22 years. The number of non-voters was unprecedentedly high – for the first time ever, non-voters outnumbered voters for the winning party. Helped, perhaps, by the former more than the latter, Labour secured a second three-figure majority.

At the 2005 election things got still worse for Labour: general demobilisation was replaced by (2.2) Selective Mobilisation of both the other two parties, together with a (1) Straight Vote Switch away from Labour, for reasons that don’t need repeating here. The Lib Dems, who opposed the Iraq war, were the main beneficiary; they gained 1.2 million votes where Labour lost 1.1 million. (The number of non-voters also fell by a million in this period, however; it’s impossible to identify flows with any certainty from data at this level of generality.) There were also the early signs of an impending (3.2) Minor Party Surge from the Kippers. In 2010, epoch-making election though it was, nothing much actually changed. That is, exactly the same things happened as had happened in 2005: the number of non-voters fell; the Tory, Lib Dem and UKIP votes rose; and the Labour vote (already in 2005 the second lowest since the War) fell again. The only significant change was that, in this period, the Tories were more effective than the Lib Dems in re-mobilising their dormant vote (and, perhaps, attracting Labour voters); the two parties’ votes grew by 20% and 13% respectively between 2005 and 2010, as opposed to 5% and 25% between 2001 and 2005. Labour’s vote fell by 10% in both periods. If 2010 was the end of New Labour, then – judging purely in terms of electoral success – New Labour left the party in an appallingly bad state.

The 2015 election was the first since 1997 when the Labour Party’s vote actually increased relative to the previous General Election. So that’s a (4.2) Fightback In Opposition. Unfortunately for Labour it was accompanied by (4.1) Consolidation In Office, along with the bizarre and unprecedented combination of a (3.3) Minor Party Squeeze (Benefiting Incumbent) with a (3.1) Minor Party Surge (Benefiting Incumbent). To be precise, there were three separate minor party surges; while UKIP will certainly have cost the Tories votes, the other two hurt Labour far more. The non-Labour centre-Left wind is blowing far harder against Labour now than it ever did against New Labour (for reasons that may not be mysterious); if we assume that half of the increase in Green, SNP and Plaid Cymru votes in 2015 came from erstwhile Labour voters (a fairly conservative estimate), then Labour effectively needed to gain a million votes just to stand still. Labour actually put on 700,000 votes. For comparison, in 1997 – the only New Labour election which is comparable, as Labour lost votes in all the others – Labour’s vote was up by 1.9 million on the previous election; the Greens and the two nationalist parties between them were down by 100,000 votes. The dog that didn’t bark this year, at least as far as the overall figures can tell us, was mobilisation; a lot of people have got out of the habit of voting since 1997, and the 2015 vote wasn’t enough to get them back into it. Here again there may be a more complex picture if we drill down – mobilisation in Scotland, demobilisation in Lib Dem country? – but the overall picture is static.

So, what have we learnt?

Why Do Governments Fall?

That’s a very interesting question, which I’ll answer if I may by pointing out that nobody has any idea. But my simplistic model does suggest that, on the seven occasions when the government has changed hands since 1945, the following factors have been at work in vote changes:

1. Straight Vote Switch (1970, 1997 and 2010): voters for one party switch to another
2.2. Selective Mobilisation (1979 and 2010): ‘dormant’ voters for the main opposition party become more likely to vote
2.4. Selective Demobilisation (1997): voters for the incumbent become less likely to vote
3.2. Minor Party Surge (1964, 1974, 2010): a rise in support for a minor party cuts away the incumbent’s base
3.4. Minor Party Squeeze (1951, 1979, 1997): the main opposition party poaches support from a minor party

However, most of these have also been a factor on one on the ten occasions when government hasn’t changed hands. The only ones which haven’t are

2.4. Selective Demobilisation
3.2. Minor Party Surge

Which seems to suggest that the best way to win an election is to join a different party. Politics is hard.

We can draw a few conclusions, though. Here are four.

1. Vote Switches Are Rare

In 17 elections (from 1950 to 2015), large-scale vote switching from party A to party B is only identifiable on four occasions – 1970, 1997, 2005 and 2010. One of those didn’t lead to a change of government; out of the other three, 1970 is the only case where large-scale vote-switching is the only identifiable factor.

2. Minor Parties Are Crucial

A minor party squeeze is identifiable in six of these elections – 1951, 1979, 1987, 1992, 1997 and 2015; a minor party surge in five (1964, 1974, 1983, 2010 and 2015 again). Six of these ten elections led to a change of government.

3. Mobilisation Is Important

Selective mobilisation – which rests, not on getting the vote out, but on ‘making the weather’ in your party’s favour; doing the Gramscian thing and getting your ideas into the common sense of the age – seems to have been a factor in 1950, 1979, 1987, 1992, 2005 and 2010. Admittedly only two of those elections led to a change of government, but all the other four either laid the groundwork for a change of government or seemed to at the time (1997, as I’ve argued above, was as much a break with the earlier trend as a continuation). The story of demobilisation is more interesting. It’s at work in 1955, 1959 and October 1974, in each case helping consolidate a previous election victory. It’s turned against the incumbent in 1997, and effectively goes viral in 2001; the total number of non-voters never reached 12 million before 1997, and has never fallen below 15 million since 2001.

4. New Labour Was Weird

For Labour to win a landslide victory, on a wave of public euphoria, with promises to transform the entire political landscape, on the back of a substantial fall in turnout (from 77.7% to 71.6%) was, in retrospect, odd. Following this with a second landslide victory on the back of an even bigger decline in turnout was very odd indeed, particularly when Labour’s vote fell considerably more than any of the other parties’. And gaining a third victory – not a landslide this time, admittedly, but a very substantial majority – on an even smaller number of votes, when both overall turnout and the other major parties’ votes were starting to pick up; that was downright flukey. Neither Blair nor anyone else was going to ride that kind of luck to a fourth election victory. Perhaps 2010 was New Labour’s Best Before date.

Still, New Labour did make some enduring changes to British politics, and I fear that lower turnout may turn out to be one of them. In an odd way there may be some truth in the absurd story put about by some Labour people after the 1983 election – that people hadn’t bothered to vote because they were so happy with how things were going. Part of the positive message of New Labour was that there was a whole new approach to doing politics – an approach which didn’t have anything to do with class conflict (or any other kind of conflict), which promoted a combination of practitioner expertise and scientific management techniques, and which generally looked a lot like managerialism. It would be easy to take from that the message that politics wasn’t something ordinary people needed to worry about – the machine would go on working, in much the same way, whether we tried to get involved or not. The negative message of New Labour, on the other hand, was that this new way of doing politics was going to be the only game in town whether you liked it or not; if you weren’t going to be part of the solution, well, sod you. I think a lot of people – mostly but by no means exclusively on the Right – picked up on this and thought well, sod you then. And gave up on voting – either for good, or just until a “sod the lot of ’em” candidate came along.

Maybe managerialism on one side and disengagement on the other is the modern (post-modern?) condition; maybe weird is the new normal. Or maybe New Labour is over; maybe the belief that, underneath it all, the elitist managerial approach to politics was about something has gone for good. Maybe the only way to win elections on that basis is to be cynical, divisive, dishonest and lucky. In which case Labour is definitely going to need some new tricks for 2020.

Why The Post-War History Of Britain Shows That We Must Support My Politics

Can Labour win again? It’s going to be hard to win any kind of majority on the basis of simple vote-switching; the Labour vote is just too low – and the Tory vote is still well below its Thatcher-era highs, suggesting that 2015 Tory voters are likely to be relatively hard to detach. If we imagine the Labour vote going up by two million, entirely at the expense of the Tories – which is more than Labour achieved in 1997 – the resultant vote would put Labour on the level of 1992, a whisker ahead of 1979.

Labour needs to think much more strategically. Tory-to-Labour switches are nice to have, but what the party really needs is the reversal of the other main trends at work in 2010 and 2015. In other words, what’s needed is a (3.4) Minor Party Squeeze, a (3.2) Minor Party Surge and (2.4) Selective Demobilisation. Firstly, Labour needs to win (back) votes from the SNP and (to a lesser extent) the Greens; if we can win votes back from UKIP as well, so much the better. Secondly, we need a strong Liberal Democrat Party – but one that’s strong against the Tories, as it was from 1997 to 2010. Thirdly, while I’m loath to discourage anyone from voting, it would help Labour a lot if people leaning towards the Tories were that much less likely to vote; if, when you said ‘Vote’, people tended to complete the sentence with ‘Labour’. At the last election the precise opposite seems to have been the case, with the effects that we know (on the polls as well as the result).

The question then is, what kind of party is going to be able to do those things and/or foster the conditions in which those things happen? Whose approach will be better at winning voters back from the Greens and the SNP – a Labourite hack who sees one lot as tree-huggers and the other as tartan Tories, or a principled socialist who sees them both as friendly rivals? Which approach will do less damage to Tim Farron’s crusade to retake the West Country – scrapping over the middle ground and denouncing the Lib Dems as soft on drugs and civil liberties, or seeing them as nice well-meaning halfway-house merchants and leaving them to it? What’s the best way to make voting Labour seem a sensible, normal part of everyday life – tell the workers you understand their resentments and hatreds, or talk to them about their working conditions?

I believe Labour’s going to have to move to the Left; anything else really is throwing the next election, if not the one after. (The fact that I have always believed that Labour should move to the Left is merely a happy coincidence.)

Next: we need to talk about Scotland.

Come write me down

There’s a particular form of serendipity that comes from learning something in one area which resolves a puzzle, or fills a gap in your thinking, in another area entirely. It’s all the more serendipitous – and pleasant – if you didn’t realise the gap was there.

This line of thought was prompted by this piece on the excellent FactCheck blog, which made me realise that I’d always been a bit dubious about the notion of “policy-based evidence”. OK, it’s a neat reversal – and all too often people who say they’re making evidence-based policy are doing nothing of the sort – but is the alternative really policy-based evidence? Doesn’t that amount to accusing them of just making it up?

Thanks to Cathy Newman at FactCheck, I realise now that I was looking at this question the wrong way. Actually “policy-based evidence” means something quite specific, and it hasn’t (necessarily) got anything to do with outright fraud. Watch closely:

Iain Duncan Smith has been celebrating the government’s benefits cap. Part of the welfare reform bill, state handouts will be capped at £26,000 a year so that “no family on benefits will earn more than the average salary of a working family,” i.e. £35,000 a year before tax.

Today, the work and pensions secretary was delighted to cite figures released by his department which he said were evidence that the policy is already driving people back into work. Of 58,000 claimants sent a letter saying their benefits were to be capped, 1,700 subsequently moved into work. Another 5,000 said they wanted support to get back into work, according to the figures.

OK, this is fairly simplistic thinking – We did a new thing! Something happened! Our thing worked! – but it’s something like a legitimate way to analyse what’s going on, surely. It may need more sophisticated handling, but the evidence is there, isn’t it?

Well, no, it isn’t.

In order to know how effective the policy had been, we would need to know the rate at which people on benefits worth more than £26,000 went into work before the letter announcing the changes was sent, and compare it to after the letter was received. But those figures aren’t available.

“[These figures do] not reveal the effect of the policy,” Robert Joyce, senior researcher at the Institute for Fiscal Studies told us. Mr Joyce went on: “Indeed, this number is consistent with the policy having had no effect at all. Over any period, some fraction of an unemployed group will probably move into work, regardless of whether a benefits cap is about to be implemented. The number of people who moved into work as a result of the policy is 1,700 minus the number of people who would have moved into work anyway. We do not know the latter number, so we do not know the effect of the policy.”

The number of people, in a given group of claimants, who signed off over a given period is data. Collecting data is the easy part: take five minutes and you can do it now if you like. (Number of objects on your desk: data. Number of stationary cars visible from your window: data. Number of heartbeats in five minutes: data.) It’s only when the data’s been analysed – it’s only when we’ve compared the data with other conditions, compared variations in the data with variations in those conditions and eliminated chance fluctuations – that data turns into evidence. The number of people who moved into work as a result of the policy is 1,700 minus the number of people who would have moved into work anyway: that number would be evidence, if we had it (or had reliable means of estimating it). The figure of 1,700 is data.

One final quote:

A spokesman for the Department for Work and Pensions said: “The Secretary of State believes that the benefits cap is having an effect.”

Et voilà: policy-based evidence.

A man he may grow

Michael Rosen’s written a long and thoughtful piece about his experience of the grammar school system in the 1950s. I don’t know if it’s going to appear in print or on a higher-profile blog, but at the moment it’s just a post on his own blog – and he’s such a prolific poster that it’s going to roll off the bottom of the front page at any moment.

So catch it while you can – it’s a must-read for anyone who’s interested in the debate around grammar schools, or interested in debates about selective education, or secondary education in general. And anyone who’s got kids at school, has kids at school or is ever likely to. And anyone who went to a grammar school, or a selective school, or a comprehensive, or a secondary modern… Basically, you should read this.

It rings so many bells, both positively and negatively (really? we didn’t do that) that I’m tempted to live-blog my reactions to it, but that would be rather self-indulgent. I’ll just mention one small detail of Rosen’s story. He mentions that he was born in 1946, his mother’s second son, and that she died in 1976, aged 55. My own mother had her 55th birthday in 1976; I had my 16th. The coincidence of one date, and the differences of the others, raise all sorts of questions. I can’t begin to imagine my life if my mother had died in her 50s; it was hard enough when it did happen, thirty years later. Then: is it easier for an adult to lose a parent who dies relatively young? Then: easier than what?

But back to school, and a detail of Rosen’s story that sparked off a problem-solving train of thought. He writes:

the pass rate for the 11-plus wasn’t the same for boys and girls and it wasn’t the same from area to area. That’s to say, it panned out at the time that girls were generally better than boys at passing this exam. However, the places for boys and girls was split evenly between us. Somehow or another they engineered what was in reality something like a 55-45% split into a 50-50% cent split. Clearly, some five per cent of girls were serious losers in this and some five per cent of boys some kind of gainers – at least as far as the system thought of us.

But that last sentence can’t be right.

Say for the sake of simplicity that the children taking the test were evenly divided between boys and girls, rather than being 49:51 or 48:52. Then we want to know how many kids passed, and then how many were pushed up or down to even up the figures. Another thing I learned from Rosen’s post is that the pass rate varied from region to region(!), depending on the availability of grammar school places(!!), but let’s forget that for the moment and assume that about one in five passed the 11-plus (in fact the proportion ranged from 30% down to 10%).

So we’ve got, oh, let’s say 10,000 kids, made up of 5,000 boys and 5,000 girls, and 2,000 of them are going to Grammar School, the lucky so-and-so’s. Now, 55% of those 2,000 – 1,100 – are girls, and only 900 are boys. So we need to balance things up, and we skim off the dimmest 100 girls who passed and promote the brightest 100 boys who didn’t (each and every one of whom is officially less bright, and hence less able to benefit from grammar school, than the 100 girls we’ve just sent to the secondary mod, but we avert our eyes at this point).

So that’s 5% of girls demoted, 5% of boys promoted? No – it’s 100/5000, or 2%. When you massage that 55% down to 50%, the 5% that’s lost is 5% of the cohort that passed the exam (male and female), not of the girls (passed and failed). You could also say that the really serious losers – the ones who have been unfairly discriminated against even by the system’s own standards – are 100 out of the 1,100 girls who passed: roughly 9.1%. The serious gainers, on the other hand, are 100 out of the 4,100 boys who failed, roughly (reaches for calculator) 2.4%.

So there you go: applied maths for real-world problem-solving.

Clearly, some two per cent of girls (or nine per cent of the girls who passed the exam) were serious losers in this and some two per cent of boys some kind of gainers – at least as far as the system thought of us.

At which point I feel a bit like Babbage correcting Tennyson, but it’s right, dammit. And besides, without the maths I wouldn’t have arrived at the figure of nine per cent – for the girls who passed the eleven-plus but were artificially failed to even up the numbers – which is pretty shocking.

Cold water in the face

A remarkable variety of people have poured scorn on Clegg Minor’s contribution to the Sun, and rightly so. The point I want to make, following on from that fourth link, is that we need to watch the Liberal Democrats – now more than ever. (‘Watch’ here includes ‘exacerbate the contradictions within’; there are some good people in Clegg’s party, even now.) The problem is not just that the party’s support is going down the drain, or that the party’s reputation as a byword for unscrupulous vote-whoring has escaped the politically active minority and gone viral: trust can always be regained, to a greater or lesser extent. (And at the end of the day they don’t have to outrun the bear: it doesn’t matter if they don’t look whiter-than-white any more, just as long as they look cleaner than the other two parties.) What’s more to the point is that the reputational capital the party built by coherently positioning itself to the Left of New Labour was thrown to the winds last May; a sizeable chunk of the party’s 2010 vote went with it, and it’s not coming back. On top of that, the experience of coalition – the extraordinarily passive and timorous experience of coalition – is surely chipping away at the party’s bedrock support: from David Steel back to Jo Grimond, the party always stood for something, whatever that might actually be in any given period. The ‘standing for’ part seems to elude the party at the moment – quite possibly because they’ve been stitched up like a kipper by their coalition partner – and their former supporters have noticed.

The problem for the Lib Dem leadership is that they need to stem the flow of disaffected supporters. (The party took 23% of the vote last May; UK Polling Report currently has them averaging 9%, and doesn’t record a single poll when they’ve exceeded 15% since the beginning of November.) Or if they can’t do that – and they haven’t had much luck so far – they need to get support from somewhere else. And cue “Alarm Clock Britain”:

There are millions of people in Alarm Clock Britain. People, like Sun readers, who have to get up every morning and work hard to get on in life. People who want their kids to get ahead. People who don’t want to rely on state handouts. People who don’t need politicians to tell them what to think or how to live their lives. People who are not poor but struggle to stay out of the red.

They are the backbone of Britain. These are the people who will get this country moving again. It is their hard graft, day in, day out, that will get us out of the hole Labour left us in.

This Government is formed by a coalition of two parties and we want to join the people of Alarm Clock Britain in another coalition. A coalition of people prepared to roll up their sleeves and get the nation back on its feet. Ed Miliband may be prepared to hide under his duvet from the problems Labour left us with. But we will get up every morning and face up to them. In Alarm Clock Britain, people don’t want a handout but they appreciate a helping hand. And that is exactly what the Coalition Government is offering them.

I know that times are difficult right now. We are having to make cuts to pay off Labour’s debts and some bills are going up. Now more than ever, politicians have to be clear who they are standing up for. Be in no doubt, I am clear about who that is.

That is why the Liberal Democrats made a promise to voters on the front of our manifesto. That no basic rate taxpayer will pay any tax on the first £10,000 they earn. We’ve already taken the first steps which will take nearly 900,000 out of paying tax altogether. From April, every single taxpayer earning less than £42,500 a year will see their income tax bill cut by £200. By the time of the next election, 23 million people will be paying £700 less.

The Government is lending a hand in other ways, too.

(That’s enough Lib Dem promises – Ed.)

“Now more than ever, politicians have to be clear who they are standing up for. Be in no doubt, I am clear about who that is.” And who is he standing up for? Why, it’s you, you lucky Sun-reader! “People, like Sun readers, who have to get up every morning and work hard to get on in life.” People in work, in other words. Follow it through: these are also people who “want their kids to get ahead”, “don’t want to rely on state handouts” and (bizarrely) “don’t need politicians to tell them what to think or how to live their lives”. And they’re “the backbone of Britain”: Nick Clegg thinks they’re great, he really does.

Obviously life isn’t always quite that neat, but that’s OK too. Maybe you are receiving benefits of some sort or other – lots of working people do – but that’s all right: you’re just one of those people who “don’t want a handout but … appreciate a helping hand”. Maybe you’ve found that you just can’t “get on in life”, no matter how early you start work, but not to worry – you’re not poor, it’s just that you “struggle to stay out of the red”.

Which is just as well, because if you were poor, or – God forbid – if you didn’t have a job to get up for in the morning, then this offer would no longer apply. You would no longer be putting in the “hard graft, day in, day out, that will get us out of the hole Labour left us in”; on the contrary, you would be digging that hole deeper with every day you lived on benefits, and making life harder for “the backbone of Britain” with every morning that you didn’t stir from your lazy idle bed.

Who Nick Clegg is standing up against turns out to be just as important as who he’s standing up for. The message seems to go something like this: Tired after a long day? Taking on extra shifts? Working unpaid overtime? Blame them – blame the workshy, blame the bone-idle, blame all those people living on benefits. They don’t know the meaning of a hard day’s work, not like you do… This would be nasty, vindictive stuff at the best of times. At a time when the unemployment rate stands at 7.9%, or 2.5 million people – and when (as Clegg well knows) the government is poised to throw many more people out of work – it’s outrageous.

Having abandoned any pretence of a position to the Left of Labour, Clegg seems to have decided that fishing for support to the left of the Tories isn’t working either, and he’s trying out the populist far Right. I’ve got a nasty feeling this isn’t going to be a one-off: Clegg may be staring into the abyss, but he’s not going down without a fight. In 2011, watch out for our Deputy Prime Minister celebrating Crimestoppers Britain (“people who don’t want to see lynch law, but can’t let petty criminals make their lives a misery”), Easter Egg Britain (“people who are not racist, but simply know how to value their own traditions”), Beside The Seaside Britain (“people who don’t hate other nations, but know the truth of that old adage – east, west, home’s best!”) and (of course) Poppy Day Britain (“people who don’t glory in war for its own sake, but know that sometimes it is the only honourable choice”).

On the plus side, by the end of the year they’ll probably still be stuck on 9%.

Update Oldham East and Saddleworth: Labour 42.1% (up 10.3%), Liberal Democrat 31.9% (up 0.3%), Conservative 12.8% (down 13.6%); turnout 48.1% (down 14.1%). An interesting result, not least because the shares of the vote aren’t that different from earlier results:

Votes for the main parties in Oldham East and Saddleworth, 1997-2011 (rounded to nearest %)

Year Labour Lib Dem Tory Tory + LD
1997 42 35 20 55
2001 39 33 16 49
2005 41 33 18 51
2010 32 32 26 58
2011 42 32 13 45

At every election from 1997 to 2005, Labour has been at least 6% ahead of the Liberal Democrats, with the Tories taking less than 20% in third place. You could see 2010’s result as a local example of last year’s swing against Labour, and last night’s result as the return of business as usual. But if 42% and 32% are around what you’d expect Labour and the Lib Dems to be getting in OE&S, 13% is very low indeed for the Tories; there will have been some defection to the extreme right, but not a lot (the combined BNP and UKIP vote share went up by a little over 1% against last May). The best explanation is surely that the consistency of the Lib Dem vote is deceptive, and that some – perhaps quite a lot – of last night’s 32% were tactical Tory votes. It’s also worth noting that the combined Tory and Lib Dem vote was lower last night than it’s been at any time since 1997; it’s only the second time it’s been below 50% (and 2001 was an unusual election; this was the year of the BNP’s big push in Oldham, when they took 11% of the vote).

However, unlike Tom Clark, I don’t believe that this result supports Clegg’s apparent new direction:

YouGov this week reported that by 51% to 16% , the small band of remaining Liberal Democrats would prefer a Tory government led by Cameron to an Ed Miliband Labour administration.

The shrinking Lib Dem electorate, then, is now much more inclined to the centre-right than it has been historically, and Oldham suggests that as it retreats from the left it can hope to make good some of the losses by advancing on the right.

Dear oh dear. The Lib Dems have lost 14% of the 23% support it had in May 2010 – more than half; 51% of 9% equates to 20% of 23%. Lib Dem voters are more right wing than they used to be because there are fewer of them, and the left-leaning voters are the ones that have given up on the party. (As UK Polling Report puts it, “the remaining rump support for the Liberal Democrats is made up of those more positively inclined towards the Tories”.) This doesn’t mean that there are votes to be gained by “advancing on the right”; in fact it specifically and precisely means that that’s a good way to lose votes.

Nor does OE&S suggest that there are votes to be won on the Right; actually what it suggests is that the party’s vote is only holding up thanks to the generosity of Tory voters. This kind of grace and favour arrangement may keep the lights on for a while, but it doesn’t bode well for the party’s future; it suggests that a party with Liberal in the name is, once again, locked into a decaying orbit around the Conservative Party. Into which, precedent suggests, they would disappear without a trace.

Update 19/1/11 Polling data bears out my speculation that the unchanged Lib Dem percentage vote masked a partial collapse in the vote, propped up by borrowed Tory votes. UK Polling Report:

of 2010 Lib Dem voters, only 55% of those who voted in the by-election stuck with the party, with 29% instead defecting to Labour … This drop in Lib Dem support was cancelled however out by Conservative tactical voting: of 2010 Conservative voters, 33% who voted in the by-election ended up backing the Liberal Democrats.

Only 49% of the 2010 Conservative voters in the sample voted Tory in 2010; 91% of the 2010 Labour voters stayed loyal, but then there were fewer of them. Shift all the Tory-LD defectors back to the Conservatives and you get a notional Tory vote share of 22%, vying for second place with the Lib Dems on 23%. Of course, this is working back from answers to a phone poll to the actual result, which isn’t really legitimate, but what’s interesting about these figures is how much of the shift in voting patterns they do in fact seem to account for. You can do it yourself if you’ve got a spreadsheet handy:

2011 Labour = 91% 2010 Labour + 29% 2010 LD + 5% 2010 Tory (!)
2011 LD = 5% 2010 Labour (!!) + 55% 2010 LD + 33% 2010 Tory
2011 Tory = 0% 2010 Labour + 3% 2010 LD + 49% 2010 Tory

Let 2010 Labour = 32%, 2010 LD = 32% and 2010 Tory = 26%, and the 2011 figures come out at 40%, 28% and 14%; you only need to massage the figures a bit to cover variable turnout and you’ve got the real results of 42%, 32% and 13%.

These figures bear out the big difference between the Tory base and its Lib Dem counterpart. Tory support is flexible, and will go under other colours if it’s for the good of the party. Lib Dem support is just soft – and, given what they’re currently being asked to support, it’s no wonder.

A treasure hunt, but the treasure’s gone

Recent discussion on CT has made me aware of some startling disparities:

UK(2001) Oxford admissions (2009)
White 71.1% 84.9%
Mixed 3.2% 4.6%
Asian 12.1% 4.6%
Black 10.9% 1.0%
Chinese 1.1% 1.8%
Other 1.6% 0.3%

A massive over-representation of the White majority, together with a really glaring under-representation of British Asian and especially Black students, who are being rejected literally nine times out of ten, whereas…

Hang on, wrong figures. That first column is the ethnic breakdown of the population of London (which is where David Lammy MP was born and has lived most of his life, not to mention the obvious point that it’s where he works). Here’s the UK:

UK(2001) Oxford admissions (2009)
White 92.1% 84.9%
Mixed 1.2% 4.6%
Asian 4.0% 4.6%
Black 2.0% 1.0%
Chinese 0.4% 1.8%
Other 0.4% 0.3%

White majority: slightly under-represented. Chinese and mixed-race groups: over-represented. British Asians: very slightly over-represented. Black British…

Well, OK, Lammy has got something here, but it’s not quite as big an issue as it might look if you’re coming at it from an ethnically-mixed background (also known as a ‘city’). The UK population in 2001 was still 92% White – there are whole areas of the country where you just won’t see a brown face, or if you do you’ll go home and tell somebody. I won’t be surprised if the figure that comes out of the 2011 Census is a bit lower, but I’ll be amazed if it’s below 90%. So the fact that the Oxford student intake is 85% White is not, in itself, a problem, except insofar as it suggests that recruitment from Scotland, Wales and the North-East might need a bit of work.

All the same, it’s true that Black students are seriously under-represented; a factor of 2 isn’t as bad as a factor of 10, but it’s not good. But this seems to be a point specifically about Black students and not about non-Whites more generally. If racism on the part of Oxford admissions tutors is at the root of what’s going on here, either it’s specifically anti-Black racism or there are other factors outweighing racist attitudes towards other groups.

Or is the problem at the application stage? Here’s how applications look in comparison to UK population figures (bearing in mind that these are 2001 figures and hence almost certainly out of date). In 2009, there were approximately 185 Oxford applications for every 1,000,000 UK citizens. If the same figure is calculated for each ethnic group, you get the following:

Applications per million Over/under
White 155 83.5%
Mixed 703 379.4%
Asian 353 190.7%
Black 192 103.8%
Chinese 918 495.2%
Other 364 196.6%

Relative to the size of their ethnic group within the population as a whole, White students are under-represented. Asians and the ‘Other’ group – which consists mainly of people who declined to state their ethnic group – are over-represented; Chinese and the ‘Mixed’ group are massively over-represented. Black students are right in the middle of the distribution, a fairly small population represented – relative to the total of applications – proportionately to its size.

Here are the admission figures again, this time side by side with the application figures:

Applications Admissions Success Over/under
White 76.9% 84.9% 27.6% 110.0%
Mixed 4.4% 4.6% 26.5% 105.6%
Asian 7.6% 4.6% 15.3% 61.0%
Black 2.0% 1.0% 12.2% 48.6%
Chinese 2.1% 1.8% 21.6% 86.1%
N/K 6.3% 2.8% 11.1% 44.2%

The “over/under” figure gives the relative success of each group as compared with the overall success rate of 25.1%. And it’s an interesting figure. Relative to applications, White students are quite substantially over-represented, while every other group is under-represented, with the exception of the ‘Mixed’ group (the cynical explanation that they’re seen as ‘white enough’ suggests itself).

Here, finally, is what it looks like if you put it all together. (These are the same numbers I’ve been crunching so far. The ‘Over/under’ figure for applications is the ratio between the number of applicants per million in each group and the number of applicants per million UK residents. The ‘Over/under’ figure for admissions is the ratio between the success rate of applicants in each group and the overall success rate of applicants.)

% of population % of applications Over/under % of admissions Over/under
White 92.1% 76.9% 0.835 84.9% 1.103
Mixed 1.2% 4.4% 3.794 4.6% 1.057
Asian 4.0% 7.6% 1.907 4.6% 0.610
Black 2.0% 2.0% 1.038 1.0% 0.488
Chinese 0.4% 2.1% 4.952 1.8% 0.862
Other 0.4% 0.8% 1.966 0.3% 0.428

Every line tells a slightly different story. The Mixed ethnic group comes off best, with a massive over-representation in applications which is entrenched at the admissions stage; Chinese students are also over-represented, with a larger over-representation among applicants only slightly scaled back at the admission stage. A smaller over-representation over Asian students is almost entirely reversed by the rejection of 85% of applicants. The White group is significantly under-represented among applicants, although the admissions process partially compensates for this with a slight over-representation, relative to applications. Alone among all the major ethnic groups, Black students apply to Oxford at roughly the same rate as the population as a whole, neither over-represented among applicants (like most others) nor under-represented (like White students). However, the Black group suffers enormously at the admission stage, with a rejection rate of nearly 88%; this compares with 74.9% for all applicants and 72.4% for White students.

So what is going on? A large part of what’s going on seems to be that White schoolchildren aren’t getting the top grades in the numbers we’d expect – although this is still being compensated during admissions. Where Black Oxford applicants are concerned, it seems undeniable that something is going wrong somewhere in the admission process. The numbers of Asian – and to a lesser extent Chinese – applicants are cut down fairly significantly in the admissions process, but this is compensated by a massive over-representation of those groups among applicants. Black students get hit both ways: they’re not over-represented (although I would find it hard to label this as a fault, particularly given the performance of my own ethnic group), and they’re turned away at an even higher rate than Asian applicants. Oxford’s own investigation concludes that subject choice must bear some (most? all?) of the blame:

BME students apply disproportionately for the most oversubscribed courses. Oxford’s three most oversubscribed large (over 70 places) courses (Economics & Management, Medicine and Mathematics) account for 43% of all BME applicants and 44% of all Black applicants – compared to just 17% of all white applicants.

Well, maybe, but I can’t help feeling that this explanation stops where it ought to start. It’s hard to believe that subject choice is the only reason why Black students’ faces so consistently fail to fit; more to the point, the ‘good’ and ‘bad’ subject choices themselves are not entirely weightless and without a history.  I passed this snippet on to my wife (we met at Cambridge). Apparently Black students aren’t being advised to choose the right subjects, I said, and that’s why not many of them get into Oxford. What, she said, they’re not applying to do Land Economy?

A parting on the right

The police forces of England and Wales implemented a new set of rules for recording crimes in 2002-3, following earlier piecemeal adjustments in 1998-9. The National Crime Recording Standard (NCRS) was designed to be more victim-friendly than the counting rules which had preceded it: rather than the police insisting on corroborating evidence before a crime was recorded to have happened, a crime was to be recorded whenever one was reported unless there was evidence to the contrary. There was a certain amount of resistance to these changes, which had the direct effect of apparently increasing the crime rate and the indirect effect of lowering the police’s clear-up rate. Nevertheless, the Home Office felt very strongly that police figures were far too low – the British Crime Survey, based on reports from a representative sample of individual victims of crime, suggested that only about 25% of predatory crimes were getting into the police figures – and the changes duly went through. Comparability was also an issue, although less of an issue with each passing year of data being produced under the new rules. The Home Office has in any case made it very clear that there is no comparability of police crime figures between 2002 and 2003, making available figures like the ones from which the graph below was compiled.

As you can see, there’s a strong correlation between the impact of the NCRS and the amount of evidence typically left by the offence. Recorded burglaries weren’t greatly affected, but recorded crimes of personal violence – where supporting evidence is particularly thin on the ground – went up by almost a quarter from one year to the next, on the basis of nothing other than a change in counting rules.

Now, there is no particular reason why the average member of the public should know about all this. It’s inconceivable that anyone with a professional or academic interest in crime or policing wouldn’t know about it, though; it would be like claiming expertise in English history and getting the date of the Battle of Hastings wrong. So this was an interesting story about the Shadow Home Secretary, Chris Grayling.

Sir Michael Scholar, chairman of the UK Statistics Authority, has warned [Grayling] that the way he used figures for violent crime were “likely to mislead the public”. … Mr Grayling’s office arranged for a press release to go out in every constituency in England and Wales, purporting to show that violent crime had risen sharply under Labour, as part of a campaign spearheaded by Mr Cameron about “broken Britain”. But Mr Grayling had failed to take into account a more rigorous system for recording crime figures introduced by the Home Office in 2002. … Mr Grayling has used comparison between the figures before and after the rule change to suggest that the Labour government has presided over a runaway rise in violent crime.

“I do not wish to become involved in political controversy but I must take issue with what you said about violent crime statistics, which seems to me likely to damage public trust in official statistics,” Sir Michael wrote in a letter to Mr Grayling yesterday.

Mr Grayling replied by promising to “take account of the request by the Statistics Authority, particularly with regard to the changes to recording practices made in 2002-03”. But he insisted that he would “continue to use recorded crime statistics, because they reflect an important reality; that the number of violent crimes reported to police stations, and particularly serious violent crimes, has increased substantially over the past decade, even taking into account any changes to data collection”.

But we don’t know the number of violent crimes reported to police stations, because we don’t know the number which are reported but not recorded; that number is not recorded, surprisingly enough. (There was a proposal a few years back to keep separate tabs on ‘incidents’ (i.e. everything that comes over the front desk or over the phone) and ‘calls for service’ (the subset of incidents that the police do anything about), but as far as I’m aware it didn’t come to anything.) In other words, Grayling has not only managed to ignore a really basic piece of statistical general knowledge; he’s gone on to ignore a correction by an expert in the field, responding in a way which demonstrates a complete lack of understanding of what he’d just been told.

The question this leaves is, is David Cameron’s first choice for Home Secretary very, very dishonest, or just very, very stupid?

Not one of us

Nick Cohen in Standpoint (via):

a significant part of British Islam has been caught up in a theocratic version of the faith that is anti-feminist, anti-homosexual, anti-democratic and has difficulties with Jews, to put the case for the prosecution mildly. Needless to add, the first and foremost victims of the lure of conspiracy theory and the dismissal of Enlightenment values are British Muslims seeking assimilation and a better life, particularly Muslim women.

It’s the word ‘significant’ that leaps out at me – that, and Cohen’s evident enthusiasm to extend the War on Terror into a full-blown Kulturkampf. I think what’s wrong with Cohen’s writing here is a question of perspective, or more specifically of scale. You’ve got 1.6 million British Muslims, as of 2001. Then you’ve got the fraction who take their faith seriously & probably have a fairly socially conservative starting-point with regard to politics (call it fraction A). We don’t really know what this fraction is, but anecdotal evidence suggests that it’s biggish (60%? 70%?) – certainly bigger than the corresponding fraction of Catholics, let alone Anglicans. Then there’s fraction B, the fraction of the A group who sign up for the full anti-semitic theocratic blah; it’s pretty clear that fraction B is tiny, probably below 1% (i.e. a few thousand people). Finally, you’ve got fraction C, the proportion of the B group who are actually prepared to blow people up or help other people to do so – almost certainly 10% or less, i.e. a few hundred people, and most of them almost certainly known to Special Branch.

I think we can and should be fairly relaxed about fraction A; we should argue with the blighters when they come out with stuff that needs arguing with, but we shouldn’t be afraid to stand with them when they’re raising just demands. (Same as any other group, really.) Fraction B is not a good thing, and if it grows to the point of getting on the mainstream political agenda then it will need to be exposed and challenged. But it hasn’t reached that level yet, and I see no sign that it’s anywhere near doing so. (Nigel Farage gets on Question Time, for goodness’ sake. Compare and contrast.) The real counter-terrorist action, it seems to me, is or should be around fraction C. Let’s say there are 5,000 believers in armed jihad out there – 500 serious would-be jihadis and 4,500 armchair jihadis, who buy the whole caliphate programme but whose own political activism doesn’t go beyond watching the Martyrdom Channel. What’s more important – eroding the 5,000 or altering the balance of the 500/4,500 split? In terms of actually stopping people getting killed, the answer seems pretty obvious to me.

Nick Cohen and his co-thinkers, such as the Policy Exchange crowd, focus on fraction B rather than fraction A. In itself this is fair enough – I think it’s mistaken, but it’s a mistake a reasonable person can make. What isn’t so understandable is the urgency – and frequency – with which they raise the alarm against this tiny, insignificant group of people, despite the lack of evidence that they’re any sort of threat. “A small minority of British Muslims believe in the Caliphate” is on a par with “A small minority of British Conservatives would bring back the birch tomorrow” or “A small minority of British Greens believe in Social Credit”. It’s an advance warning of possible weird nastiness just over the horizon; it’s scary, but it’s not that scary.

What explains the tone of these articles, I think, is an additional and unacknowledged slippage, from fraction B back out to fraction A. What’s really worrying Cohen, in other words, isn’t the lure of conspiracy theory and the dismissal of Enlightenment values so much as the lure of Islam (in any form) and the dismissal of secularism. (What are these Enlightenment values, anyway? Nobody ever seems to specify which values they’re referring to. Somebody should make a list). Hence this sense of a rising tide of theocratic bigotry, and of the need for a proper battle of values to combat it. This seems alarmingly wrongheaded. Let’s say that there’s a correlation between religious devotion and socially conservative views (which isn’t always the case) – then what? A British Muslim who advocates banning homosexuality needs to be dealt with in exactly the same way as a British Catholic who advocates banning abortion – by arguing with their ideas. (Their ideas are rooted in their identities – but then, so are mine and yours.) And hence, too, that odd reference to British Muslims seeking assimilation and a better life, as if stepping out of the dark ages must mean abandoning your faith – or, at least, holding it lightly, in a proper spirit of worldly Anglican irony. Here, in fact, Cohen is a hop and a skip from forgetting about all the fractions and identifying the problem as Muslims tout court. Have a care, Nick – that way madness lies.

The high and the low

(Updated Christmas Eve, after spotting a flaw in my statistical analysis. I am deeply sad.)

Now that it’s well and truly over, two things really stick in my mind about the Manchester Congestion Charge vote. (Strictly speaking, the Manchester Transport Innovation Fund vote – but I don’t think it’s a fund that we voted to reject.)

One is the sheer strangeness of the Yes campaign. As you’ll already know if you live anywhere in Greater Manchester, this was a huge campaign. The public transport companies were in favour anyway, so you couldn’t get on a bus or a tram without being invited to vote Yes. But you couldn’t wait for a bus – or look out of the window once it started moving – without your eyes being met by the dull-eyed, faintly reproachful gaze of the Vote Yes People. (Click around the site for more. Perhaps not late at night.) They were everywhere. According to that Web site, the campaign was sponsored by TCS (a property company) and Practicus (an ‘interim management’ company, which seems to be something like middle-management recruitment only not quite; perhaps you don’t get an actual job at the end of it). Those two companies must be doing remarkably well, to have all that money to spend on someone else’s publicity; clearly names to watch. From the Vote Yes campaign’s point of view, though, I do wonder that nobody seems to have considered the potential downside of this level of saturation publicity. People don’t generally like being told what to do, least of all by spud-faced pod-people who purport to represent them.

Perhaps it wouldn’t have been so bad if the content of the campaign had been different. There were three waves of pod-people posterage, each a variation on the basic theme of What An Ordinary Manchester Person Is Thinking. (And ‘thinking’ is the word: nobody was actually speaking in those pictures. Look into my eyes! Hear my thoughts!) The first wave was the deeply annoying “I won’t be paying” theme. This wasn’t encouraging civil disobedience (which would probably be fairly futile with the level of surveillance required by the scheme). Rather, it was based on the idea that most people wouldn’t be making car journeys which would be hit by the charge – supposedly ‘eight out of ten people wouldn’t pay’ – and therefore most people ought to vote Yes.

This was a bad approach on so many levels. On the face of it, it was a straightforward appeal to self-interest: you want better public transport? you don’t want to pay more? lucky you, you won’t have to! But anyone who was already concerned about the charge, or suspected that they might be affected, had already had ample opportunities to do the sums for their own situations. (Full disclosure: I worked out that I’d be charged once a week. I really resented that.) Even if only 20% of the population was likely to be charged – and I’m sure people like me, incurring weekly charges, weren’t included in those calculations – the appeal to self-interest, for those people, would immediately backfire: saying that four out of five people wouldn’t pay isn’t much of a selling-point if you’re number 5.

For anyone who hadn’t given the charge much thought, on the other hand, the campaign could almost have been calculated to raise suspicions – precisely because of that weird and phony “we are ordinary people like you” framing. I won’t pay, says an actor representing a typical Manchester resident, because I only go into town at the weekend / I get to college by bus / I never go out of the house (I may have made up the last one). I suppose our reaction to these was supposed to be “good for us – tough luck on those people who insist on commuting by car”. Actually my instinctive reaction was “good for fictional you, but what about me?” If you’re going to appeal to self-interest, you need to get the story straight – once you start thinking in terms of “can I get something for nothing?”, you’re also thinking “am I going to get ripped off?”

The second wave was all about fairness. This time the pod people had talking points that they were mulling over (although where they got them was a mystery to me – the publicity about the actual details of the scheme was woefully limited). The emphasis was on the commitment to get the improvements to public transport into place before the charge came in; a typical poster read “Bus fares are frozen, and then the charge comes in? Sounds fair to me.” This wasn’t as actively repellent as the first phase, but it was extraordinarily weak – what do you mean, it sounds fair to you? What is this imitation of reasoning – are you saying it is fair or not – and if not, why not? Come to think of it, what’s fairness got to do with the timing of the introduction of the charge? There’s no sense in which the benefits gained in the first couple of years offset the costs imposed from that point on. Once again, this “we are ordinary people” approach provokes the very suspicions it’s apparently meant to allay – maybe it sounds ‘fair’ to you, mate, but to me it just sounds like a sweetener… And, once again, the underlying appeal is not to collective benefits or to fairness (despite the language), but to self-interest. Two years benefits upfront, free of charge? I’ll have some of that. What would genuinely sound fair would be “We’ll pay more when we drive at peak times, but we’ll get the benefit when we use public transport” – but that message never appeared.

The idea of actually paying the charge did surface in the third and final stage of the campaign, but yet again the appeal was to individual self-interest. The message here was “I want to [get from A to B quickly]. That’s why I’m voting Yes.”, with examples ranging from getting to the building site on time to putting the kids to bed. I don’t mind paying, the logic runs, because I know that other people won’t want to pay, and so the roads I drive down will be much clearer. Essentially this was the “get the plebs off the road” phase of the campaign. It seems to tap into the same vein of narcissistic fantasy that brought us the remake of SurvivorsWhat if everyone stopped using their cars to get to work except me? Wouldn’t that be brilliant?

This isn’t a full picture of the Yes campaign; there was some publicity which focused on improvements to public transport. More to the point, a lot of the actual campaigning went on by word of mouth, and here the idea that the charge might be paid for in collective benefits did get an airing. Overall, though, the Yes campaign was woeful as well as creepy. What it was trying to get us to do was assent to an additional tax, for the sake of benefits which (by government decree) couldn’t be funded any other way. The question, in other words, was “do you agree to start making a payment you’ve never had to make before and carry on paying it indefinitely, with no guarantee that the scheme won’t be extended or the toll increased, for no reason except that that’s the only offer on the table?” (The TIF was to consist of a £1500 million grant plus a £1200 million loan, a quarter of which would need to be spent on setting up the machinery to administer the scheme. And no, we couldn’t just have the £1500 million.) It appeals to a certain combination of public-spiritedness and submissive ‘realism’: you can say “yes, because I believe the investment in public transport will be worth it, and besides it’s the only offer on the table” or “yes, because I believe we should be encouraged to use our cars less (and besides…)”, but those are arguments for agreeing to a collective tax, arbitrarily imposed, in return for collective benefits. There’s just no way to sell a Yes vote in terms of individual self-interest, and it was pretty shabby of the Yes campaign to make the attempt.

The other thing that struck me about the campaign was the consistency of the voting figures, with one interesting exception. There are ten boroughs within the old Greater Manchester region; the plan was to implement two charging zones, one following the M60 and an inner ring further in towards the centre (not far enough in for my liking, but that’s by the way). Out of the ten boroughs, Bolton and Wigan are entirely outside the M60, and Rochdale almost entirely; these three boroughs presumably have the largest proportion of people who would be completely unaffected by the charge. Bury, Oldham, Tameside, Stockport and Trafford are all crossed by the M60. Manchester and Salford, finally, are divided both by the M60 and by the inner ring.

Here are the voting figures. I’ve given the percentage turnout and the No vote (as a percentage of those who voted). The dotted lines represent percentages across all ten boroughs. (Region-wide turnout: 53.2%; region-wide No vote: 78.8%.) I’ve graphed the No vote because it turns out that there was very little variation in the Yes vote, calculated as a percentage of eligible voters: 4% in total (from a low of 8.9% to a high of 12.8%), with six boroughs within 0.5% of the overall figure of 11.3%.

Congestion charge 1

Here are the same figures, normalised around those region-wide percentages: 90% means ‘90% of the regional percentage turnout/No vote’.

Congestion charge 2

And here are the percentages again, sorted by No vote rather than by turnout.

Congestion charge 3

What do we see? The first thing is that turnout was respectable everywhere (the Wigan low of 45% would be very good for a local election) and better than that in a few places (over 60% in Tameside and Trafford). The second is that the No vote was overwhelming (and the Yes vote miserable) pretty much everywhere: the No vote ranged from 84.5% in Salford all the way down to 72.2% in Manchester. This wasn’t a multiple-choice question or a choice between several candidates: 27.8% of people who voted in Manchester voted Yes, and 72.2% voted No. For the proposal to pass, the vote had to be over 50% in seven out of ten boroughs; it didn’t even reach 30% in one.

Then there’s the correlation of turnout and No vote, which is particularly striking in the third graph: three boroughs had a below-average No vote and a below-average turnout; six had an above-average turnout and an above-average No vote. (Bolton was in between.) Look at the first graph and compare Trafford, Tameside and Stockport (crossed by the M60) with Rochdale, Bolton and Wigan (outside the M60). Outer boroughs: low turnout, relatively low No vote. Inner: high turnout, relatively high No vote. As I noted above, the Yes turnout varied between 8.9% and 12.8%, for an overall average of 11.3%. There was much more variation in the No turnout, which was 41.9% across the area, but ranged from over 50% in Trafford and Tameside to just over 33% in Wigan and Manchester. (Trafford also had an above-average Yes turnout, at 12.5%. I guess they just take voting seriously in Trafford.) There seems to be a definite correlation with geography; it looks as if, where geography made a difference, the difference was both that the congestion charge interested fewer people (lower turnout in outer boroughs) and that those who bothered to vote were more motivated by self-interest (lower No vote in outer boroughs). In short, the geographical patterning of the Yes vote is highly suggestive of an appeal to self-interest, while the overall level of the Yes vote suggests that this appeal has very little power to mobilise.

Lastly, there’s a glaring exception to this correlation: Manchester, the borough covering most of the city centre and hence the only borough, apart from Salford, which is crossed by both inner and outer charging rings. Salford has the record No vote, at 84.5%; turnout was a respectable 57%. Manchester, by contrast, is out there with Wigan: a turnout of only 46%, of whom 27.8% voted Yes. Clearly, the model which explains the differences between inner and outer boroughs in terms of individual self-interest can’t deal with these figures.

I haven’t got an explanation, either for the high Yes vote or for the equally puzzling low turnout. Anecdotal evidence suggests that Manchester (or at least South Manchester) may have an unusually high concentration of people sympathetic to the aims of the Congestion Charge, or of non-drivers, or both. As for the low turnout, Manchester City Council hasn’t changed hands since 1974; the council’s motto is Concilio Et Labore, and it is. Perhaps conditions like that – compounded by the fug of neo-Blairite ex-municipal-socialist hortatory corporate righteousness which has enveloped the Town Hall for the last decade – tend to promote cynicism and disengagement: they’ll do it anyway, so why encourage them? The day the vote came through the Manchester Evening News results page included a poll: “Is the Congestion Charge dead and buried?” When I looked at the page, votes were running 4:1 in favour of “It’ll be back in some form”. White Van Man won’t resist the Future forever. (And a Merry Christmas to you too, Mr Leese sir!)

All those numbers

I like a good fallacy; I managed to get the Base Rate Fallacy, the Hawthorne Effect and Goodhart’s Law into one lecture I gave recently. So I was intrigued to run across this passage in Jock Young’s 2004 essay “Voodoo Criminology and the numbers game” (you can find a draft in pdf form here):

Legions of theorists from Robert K Merton through to James Q Wilson have committed Giffen’s paradox: expressing their doubts about the accuracy of the data and then proceeding to use the crime figures with seeming abandon, particularly in recent years when the advent of sophisticated statistical analysis is, somehow, seen to grant permission to skate over the thin ice of insubstantiality.

I like a good fallacy, but paradoxes are even better. So, tell me more about Giffen’s paradox:

Just as with Giffen’s paradox, where the weakness of the statistics is plain to the researchers yet they continue to force-feed inadequate data into their personal computers

Try as I might, I wasn’t seeing the paradox there. A footnote referenced

Giffen, P. (1965), ‘Rates of Crime and Delinquency’ in W. McGrath (ed.), Crime Treatment in Canada

I didn’t have W. McGrath (ed.), Crime Treatment in Canada by me at the time, so I did the next best thing and Googled. I rapidly discovered that Giffen’s paradox is also known as the Giffen paradox, that it’s associated with Giffen goods, and that it’s got nothing to do with Giffen, P. (1965):

Proposed by Scottish economist Sir Robert Giffen (1837-1910) from his observations of the purchasing habits of the Victorian poor, the Giffen paradox states that demand for a commodity increases as its price rises.

Raise the price of bread when there are people on the poverty line – ignoring for the moment the fact that this makes you the rough moral equivalent of Mengele – and those people will buy more bread, to substitute for the meat they’re no longer able to afford. It’s slightly reassuring to note that, notwithstanding Sir Robert’s observations of the Victorian poor, economists have subsequently questioned whether the Giffen paradox has ever actually been observed.

But none of this cast much light on those researchers force-feeding their personal computers with inadequate data. Eventually I tracked down W. McGrath (ed.), Crime Treatment in Canada. It turns out that the less famous Giffen did in fact describe the willingness of researchers to rely on statistics, after having registered caveats about their quality, as a paradox (albeit “one of the less important paradoxes of modern times”). I still can’t see that this rises to the level of paradox: surely being upfront about the quality of the data you’re processing is what a statistical analyst should do. If initial reservations don’t carry through into the conclusion that’s another matter – but that’s not a paradox, that’s just misrepresentation.

Paradoxical or not, Giffen’s observation accords with Young’s argument in the paper, which is that criminologists, among other social scientists, place far too much trust in statistical analysis: statistics are only as good as the methods used to produce them, methods which in many cases predictably generate gaps and errors.

It’s a good argument but not a very new or surprising one (perhaps it was newer in 1965). Moreover, Young pushes it in some odd directions. The paper reminded me of Robert Martinson’s 1974 study of rehabilitation programmes, “What Works?” – or rather, of how that paper was received. Martinson demonstrated that no study had conclusively shown any form of rehabilitation to work consistently, and that very few studies of rehabilitation showed any clear result; his paper was seized on by advocates of imprisonment and invoked as proof that nothing worked. This was unjustified on two levels. Firstly, while Martinson’s negatives would justify scepticism about a one-size-fits-all rehabilitation panacea, the detail of his research did suggest that some things worked for some people in some settings. Subsequent research – some of it by Martinson himself – bore out this suggestion, showing reasonably clear evidence that tailored, flexible and multiple interventions can actually do some good. Secondly, if Martinson was sceptical about rehabilitation, he wasn’t any less sceptical about imprisonment: his conclusion was that ex-offenders could be left alone, not that they should be kept locked up (“if we can’t do more for (and to) offenders, at least we can safely do less”). For Martinson, rehabilitation couldn’t cut crime by reforming bad people, because crime wasn’t caused by bad people in the first place. Sadly, the first part of this message was heard much more clearly than the second.

Like Martinson, Young is able to present a whole series of statistical analyses which seem obviously, intuitively wrong. However, what his examples suggest is that statistics from different sources require different types and levels of wariness: some are dependably more trustworthy than others, and some of the less trustworthy are untrustworthy in knowably different ways. But rather than deal individually with the different types of scepticism, levels of scepticism and reasons for scepticism which different analyses provoke, Young effectively concludes that nothing works, or very little:

Am I suggesting an open season on numbers? Not quite: there are … numbers which are indispensable to sociological analysis. Figures of infant mortality, age, marriage and common economic indicators are cases in point, as are, for example, numbers of police, imprisonment rates and homicide incidences in criminology. Others such as income or ethnicity are of great utility but must be used with caution. There are things in the social landscape which are distinct, definite and measurable; there are many others that are blurred because we do not know them – some because we are unlikely ever to know them, others, more importantly, because it is their nature to be blurred. … There are very many cases where statistical testing is inappropriate because the data is technically weak – it will simply not bear the weight of such analysis. There are many other instances where the data is blurred and contested and where such testing is simply wrong.

(In passing, that’s a curious set of solid, trustworthy numbers to save from the wreckage – it’s hard to think of an indicator more bureaucratically produced, socially constructed and culture-bound than “infant mortality”, unless perhaps it’s “marriage”.)

I’ve spent some time designing a system for cataloguing drug, alcohol and tobacco statistics – an area where practically all the data we have is constructed using “blurred and contested” concepts – so I sympathise with Young’s stance here, up to a point. Police drug seizure records, British Crime Survey drug use figures and National Treatment Agency drug treatment statistics are produced in different ways and tell us about different things, even when they appear to be talking about the same thing. (In my experience, people who run archives know about this already and find it interesting, people who use the statistics take it for granted, and IT people don’t know about it and want to fix it.) But: such testing is simply wrong? (Beware the persuasive adverb – try re-reading those last two sentences with the word ‘simply’ taken out.) We know how many people answered ‘yes’ to a question with a certain form of words; we know how many of the same people answered ‘yes’ to a different question; and we know the age distribution of these people. I can’t see that it would be wrong to cross-tabulate question one against question two, or to calculate the mean age of one sub-sample or the other. Granted, it would be wrong to present findings about the group which answered Yes to a question concerning activity X as if they were findings about the group who take part in activity X – but that’s just to say that it’s wrong to misrepresent your findings. Young’s broader sceptical claim – that figures constructed using contested concepts should not or cannot be analysed mathematically – seems… well, wrong.

Young then repeats the second of the errors of Martinson’s audience: if none of that works, then we can stick with what we know. In this case that means criminology reconceived as cultural ethnography: “a theoretical position which can enter in to the real world of existential joy, fear, false certainty and doubt, which can seek to understand the subcultural projects of people in a world riven with inequalities of wealth and uncertainties of identity”. Fair enough – who’d want a theoretical position which couldn’t enter in to the real world? But the question to ask about creeds is not what’s in them but what they leave out. Here, the invocation of culture seems to presage the abandonment not only of statistical analysis but of materialism.

The usual procedure … is to take the demographics and other factors which correlate with crime in the past and attempt to explain the present or predict the future levels of crime in terms of changes in these variables. The problem here is that people (and young people in particular) might well change independently of these variables. For in the last analysis the factors do not add up and the social scientists begin to have to admit the ghost in the machine.

People … might well change independently of these variables – how? In ways which don’t find any expression in phenomena that might be measured (apart from a drop in crime)? It seems more plausible to say that, while people do freely choose ways to live their lives, they do not do so in circumstances of their own choosing – and that those choices in turn have material effects which create constraints as well as opportunities, for themselves and for others. To put it another way, if the people you’re studying change independently of your variables, perhaps you haven’t got the right variables. Young’s known as a realist, which is one way of being a materialist these days; but the version of criminology he’s proposing here seems, when push comes to shove, to be non- or even anti-materialist (“the ghost in the machine”). That’s an awfully big leap to make, and I don’t think it can be justified by pointing out that some statisticians lie.

What arguments based on statistics need – and crime statistics are certainly no exception – is scepticism, but patient and attentive scepticism: it’s not a question of declaring that statistics don’t tell us anything, but of working out precisely what particular uses of statistics don’t tell us. A case in point is this story in last Friday’s Guardian:

An 8% rise in robberies and an 11% increase in vandalism yesterday marred the latest quarterly crime figures, which showed an overall fall of 2% across all offences in England and Wales.

The rise in street crime was accompanied by British Crime Survey indicators showing that public anxiety about teenagers on the streets, noisy neighbours, drug dealing, drunkenness and rowdiness has continued to increase despite the government’s repeated campaigns against antisocial behaviour. … But police recorded crime figures for the final three months of 2006 compared with 12 months earlier showed that violent crime generally was down by 1%, including a 16% fall in gun crime and an 11% fall in sex offences.

The more authoritative British Crime Survey, which asks 40,000 people about their experience of crime each year, reported a broadly stable crime rate, including violent crime, during 2006. … The 11% increase in vandalism recorded by the BCS and a 2% rise in criminal damage cases on the police figures underlined the increase in public anxiety on five out of seven indicators of antisocial behaviour.

Confused? You should be. Here it is again:

  Police BCS
All crime down 2% stable (up 1%*)
Violent crime down 1% stable
Robbery up 8% stable (down 1%*)
Vandalism up 2% up 11%

* Figures in italics are from the BCS but weren’t in the Guardian story.

Earlier on in this post I made a passing reference to statistical data being bureaucratically produced, socially constructed and culture-bound. Here’s an example of what that means in practice. Police crime figures are a by-product of the activities of the police in dealing with crime, and as such are responsive to changes in the pattern of those activities: put a lot more police resources into dealing with offence X, or change police procedure so that offences of type X are less likely to go by unrecorded, and the crime rate for offence X will appear to go up (see also cottaging). Survey data, on the other hand, is produced by asking people questions; as such, it’s responsive to variations in the type of people who answer questions and to variations in those people’s memory and mood, not to mention variations in the wording of the questions, the structure of the questionnaire, the ways in which answers are coded up and so on. The two sets of indicators are associated with different sets of extraneous influences; if they both show an increase, the chances are that they’ve both been affected by the same influence. The influence in question may be a single big extraneous factor which affects both sets of figures – for example, a massively-publicised crackdown on particular criminal offences will give them higher priority both in police activities and in the public consciousness. But it may be a genuine increase in the thing being measured – and, more to the point, the chances of it being a genuine increase are much higher than if only one indicator shows an increase.

In this case, the police have robberies increasing by 8%; the BCS has theft from the person dropping by 1%. That’s an odd discrepancy, and suggests that something extraneous is involved in the police figure; it’s not clear what that might be, though. Vandalism, on the other hand, goes up by 2% if you use police figures but by all of 11% if you use the BCS. Again, this discrepancy suggests that something other than an 11% rise in the actual incidence of vandalism might be involved, and in this case the story suggests what this might be:

British Crime Survey indicators showing that public anxiety about teenagers on the streets, noisy neighbours, drug dealing, drunkenness and rowdiness has continued to increase despite the government’s repeated campaigns against antisocial behaviour

Presumably the government’s repeated campaigns against antisocial behaviour have raised the profile of anti-social behaviour as an issue. Perhaps this has made it more likely that people will feel that behaviour of this type is something to be anxious about, and that incidents of vandalism will be talked about and remembered for weeks or months afterwards (the BCS asks about incidents in the past twelve months).

That’s just one possible explanation: the meaning of figures like these is all in the interpretation, and the interpretation is up to the interpreter. The more important point is that there are things that these figures will and won’t allow you to do. You can say that police figures, unlike the BCS, are a conservative but reliable record of things that have actually happened, and that robbery has gone up by 8% and criminal damage by 2%. You can say that victim surveys, unlike police figures, are an inclusive and valid record of things that people have actually experienced, and that vandalism has gone up by 11% while robbery has gone down by 1%. What you can’t do is refer to An 8% rise in robberies and an 11% increase in vandalism – there is no way that the data can give you those two figures.

But that’s not a paradox or even a fallacy – it’s just misuse of statistics.

None of you stand so tall

In the previous post, I showed that the canonical ‘power law’ chart which underlies the Long Tail image does not, in fact, represent a power law. What it represents is a ranked list, which happens to have a similar shape to a power law series: as it stands, the ‘power law’ is an artifact of the way the list has been sorted. In particular, the contrast which is often drawn, in this context, between a power law distribution and a normal distribution is inappropriate and misleading. If you sort a list high to low, it can only ever have the shape of a descending curve.

There are counter-arguments, which I’ll go through in strength order (weakest first).

Counter-argument 1: the Argument from Inconsequentiality.

In the post which started it all, Clay wrote:
the shape of Figure #1, several hundred blogs ranked by number of inbound links, is roughly a power law distribution.

Note weasel wordage: it would be possible to argue that what Clay (and Jason Kottke) identified wasn’t really a power law distribution, it was just some data which could be plotted in a way which looked oddly like a power law curve. Thankfully, Clay cut off this line of retreat, referring explicitly to power law distributions:

power law distributions are ubiquitous. Yahoo Groups mailing lists ranked by subscribers is a power law distribution. LiveJournal users ranked by friends is a power law … we know that power law distributions tend to arise in social systems where many people express their preferences among many options.

And so on. When we say ‘power law’, we mean ‘power law distribution’: we’re all agreed on that.

Except, of course, that what we’re talking about isn’t a power law distribution. Which brings us to…

Counter-argument 2: the Argument from Intuition.

The pages I excerpted in the previous post specifically contrast the power law distribution with the ‘normal’ bell curve.

many web statistics don’t follow a normal distribution (the infamous bell curve), but a power law distribution. A few items have a significant percentage of the total resource (e.g., inbound links, unique visitors, etc.), and many items with a modest percentage of the resources form a long “tail” in a plot of the distribution.

we find a very few highly connected sites, and very many nearly unconnected sites, a power law distribution whose curve is very high to the left of the graph with the highly connected sites, with a long “tail” to the right of the unconnected sites. This is completely different than the bell curve that folks normally assume

The Web, like most networks, has a peculiar behavior: it doesn’t follow standard bell curve distributions … [it] follows a power law distribution where you get one or two sites with a ton of traffic (like MSN or Yahoo!), and then 10 or 20 sites each with one tenth the traffic of those two, and 100 or 200 sites each with 100th of the traffic, etc.

One of my Latin teachers at school had an infuriating habit, for which (in the best school-story tradition) I’m now very grateful. If you read him a translation which didn’t make sense (grammatically, syntactically or literally) he’d give you an anguished look and say, “But how can that be?” It was a rhetorical question, but it was also – infuriatingly – an open question: he genuinely wanted you to look again at what you’d written and realise that, no, actually that noun in the ablative couldn’t be the object of the verb… Good training, and not only for reading Latin.

If you’ve got this far, do me a favour and re-read the excerpts above. Then ask yourself: how can that be?

As long as we’re talking about interval/ratio variables – the only type for which a normal distribution can be plotted – it’s hard to make sense of this stuff. What, to put it bluntly, is being plotted on the X axis? The best I can do is to suppose that the X axis plots number of sites: A few items have a significant percentage of the total resource; a very few highly connected sites; one or two sites with a ton of traffic. There’s your spike on the left: a low X value (a few items) and a high Y (a significant percentage of the total resource).

But this doesn’t really work either. Or rather, it could work, but only if every group of sites with the same number of links had a uniquely different number of members – and if the number of members in each group were in inverse proportion to the number of links (1 site with n links, 2 sites with n/2 links, 3 sites with n/3 links, 4 sites with n/4 links…). This isn’t impossible, in very much the same way that the spontaneous development of a vacuum in this room isn’t impossible; a pattern like that wouldn’t be a power law so much as evidence of Intelligent Design.

This is an elaborate and implausible model; it’s also something of a red herring, as we’ll see in a minute. It’s worth going into in detail, though; as far as I can see, it’s the only way of getting these data into a power law distribution, with high numbers of links on the left, without using ranking. And cue…

Counter-argument 3: the Argument from Ranking.

Over to Clay:

The basic shape is simple – in any system sorted by rank, the value for the Nth position will be 1/N. For whatever is being ranked — income, links, traffic — the value of second place will be half that of first place, and tenth place will be one-tenth of first place. (There are other, more complex formulae that make the slope more or less extreme, but they all relate to this curve.)

“The value for the Nth position will be 1/N” (or proportionate to 1/N, to be more precise); alternatively, you could say that N items have a value of 1/N or greater. (Have a think about this one – we’ll be coming back to it later.) Either way, it’s a power law, right? Well, yes – and no. It’s certainly true to say that a ranked list with these properties confirms to a version of the power law – specifically, Zipf’s law. It’s also true to say that Zipfian rankings are associated with Pareto-like power law distributions: we may yet be able to find a power law in this data. But we’re not there yet – and Clay’s presentation of the data doesn’t help us to get there. (Jason’s has some of the same problems, but Clay’s piece is a worse offender; it’s also much more widely known.)

The first problem is with the recurrent comparison of ranked graphs with bell curves. Adam: “a ranked graph … by definition is *always* decreasing, and can *never* be a bell curve”. If anyone tells you that such and such a phenomenon follows a power law rather than a normal distribution, take a good look at their X axis. If they’ve got ranks there, the statement is meaningless.

Secondly, the graph Clay presented – a classic of the ‘big head, long tail’ genre – isn’t actually a Zipfian series, for the simple reason that it includes tied ranks: it’s not a list of ranks but a list of nominals sorted into rank order.

I’ll clarify. Suppose that we’ve got a series which only loosely conforms to Zipf’s Law, perhaps owing to errors in the real world:

Rank Value
1 1000
2 490
3 340
4 220
5 220
6 180
7 140

Now, what happens on the graph around values 4 and 5? If the X axis represents ranking, it makes no sense to say that the value of 220 corresponds to a rank of 4 and a rank of 5: it’s a rank of 4, followed by no ranking for 5 and a rank of 6 for the value of 180. We can see the point even more clearly if we take the alternative interpretation of a Zipfian list and say that the X axis tracks ‘number of items with value greater than or equal to Y’. Clearly there are 6 items greater than or equal to 180 and 5 greater than or equal to 220 – but it would be nonsensical to say that there are also 4 items greater than or equal to 220. Either way, if you have a ranked list with tied rankings this should be represented by gaps in the graph.

This may seem like a minor nitpick, but it’s actually very important. Back to Adam:

One nice thing about a ranked graph is that the “area” under the curve is equal to the total value associated with the items spanned on the ranked axis

Or, in the words of one of the pieces I quoted in the previous post:

In such a curve the distribution tapers off slowly into the sunset, and is called a tail. What is most intriguing about this long tail is that if you add up all the traffic at the end of it, you get a lot of traffic

What we’re talking about, clearly, is the Long Tail. Looking at some actual figures for inbound linkage (collected from NZ Bear earlier this year), there are few tied ranks in the higher rankings and more as we go further out: 95 unique values in the first 100 ranks and 79 in the next 100. Further down, the curve grows flatter, as we’d expect. The first ten rankings (ranging from 5,389 down to 2,142 links) correspond to ten sites; the last ten (ranging, predictably, from 9 down to zero) correspond to a total of 14,445. As Adam says, if you were to graph these data as a list of nominals ranked in descending order, the ‘area’ covered by the curve would give you a good visual impression of the total number of links accounted for by low-linked sites: the Long Tail, no other. But this graphic does not conform to a power law – not even Zipf’s Law. A list conforming to Zipf’s Law would drop tied ranks – it would exclude duplicates, if that’s any clearer. Instead of a long tail, it would trail off to the right with a series of widely-spaced fenceposts. (“In equal 9126th place, blogs with 9 links; in equal 9593rd place, 8-linkers…”)

Long Tail, power law: choose one.

You can have a Long Tail, but only by graphing a list of nominals ranked in descending order.

You can have a power law series with rankings, but only by replacing the long tail with scattered fenceposts.

Even more importantly, neither of these is a power law distribution. Given the appropriate data values, you can derive a power law distribution from a ranked list – but it doesn’t look like the ‘long tail’ graphic we know so well. I’ll talk about what it does look like in the next post.

Put your head back in the clouds

OK, let’s talk about the Long Tail.

I’ve been promising a series of posts on the Long Tail myth for, um, quite a while. (What’s a month in blog time? A few of those.) The Long Tail posts begin here.

Here’s what we’re talking about, courtesy of our man Shirky:

We are all so used to bell curve distributions that power law distributions can seem odd. The shape of Figure #1, several hundred blogs ranked by number of inbound links, is roughly a power law distribution. Of the 433 listed blogs, the top two sites accounted for fully 5% of the inbound links between them. (They were InstaPundit and Andrew Sullivan, unsurprisingly.) The top dozen (less than 3% of the total) accounted for 20% of the inbound links, and the top 50 blogs (not quite 12%) accounted for 50% of such links.

Figure #1: 433 weblogs arranged in rank order by number of inbound links.

It’s a popular meme, or it would be if there were any such thing as a meme (maybe I’ll tackle that one another time). Here’s one echo:

many web statistics don’t follow a normal distribution (the infamous bell curve), but a power law distribution. A few items have a significant percentage of the total resource (e.g., inbound links, unique visitors, etc.), and many items with a modest percentage of the resources form a long “tail” in a plot of the distribution. For example, a few websites have millions of links, more have hundreds of thousands, even more have hundreds or thousands, and a huge number of sites have just one, two, or a few.


if we measure the connectivity of a sample of 1000 web sites, (i.e. the number of other web sites that point to them), we might find a bell curve distribution, with an “average” of X and a standard deviation of Y. If, however, that sample happened to contain, then things would be off the chart for the “outlier” and normal for every other one.If we back off to see the whole web’s connectivity, we find a very few highly connected sites, and very many nearly unconnected sites, a power law distribution whose curve is very high to the left of the graph with the highly connected sites, with a long “tail” to the right of the unconnected sites. This is completely different than the bell curve that folks normally assume

And another:

The Web, like most networks, has a peculiar behavior: it doesn’t follow standard bell curve distributions where most people’s activities are very similar (for example if you plot out people’s heights you get a bell curve with lots of five- and six-foot people and no 20-foot giants). The Web, on the other hand, follows a power law distribution where you get one or two sites with a ton of traffic (like MSN or Yahoo!), and then 10 or 20 sites each with one tenth the traffic of those two, and 100 or 200 sites each with 100th of the traffic, etc. In such a curve the distribution tapers off slowly into the sunset, and is called a tail. What is most intriguing about this long tail is that if you add up all the traffic at the end of it, you get a lot of traffic

All familiar, intuitive stuff. It’s entered the language, after all – we all know what the ‘long tail’ is. And when, for example, Ross writes about somebody who started blogging about cooking at the end of the tail and is now part of the fat head and has become a pro, we all know what the ‘fat head’ is, too – and we know what (and who) is and isn’t part of it.

Unfortunately, the Long Tail doesn’t exist.

To back up that assertion, I’m going to have to go into basic statistics – and trust me, I do mean ‘basic’. In statistics there are three levels of measurement, which is to say that there are three types of variable. You can measure by dividing the field of measurement into discrete partitions, none of which is inherently ranked higher than any other. This car is blue (could have been red or green); this conference speaker is male (could have been female); this browser is running under OS X (could have been Win XP). These are nominal variables. You can code up nominals like this as numbers – 01=blue, 02=red; 1=male, 2=female – but it won’t help you with the analysis. The numbers can’t be used as numbers: there’s no sense in which red is greater than blue, female is greater than male or OS X is – OK, bad example. Since nominals don’t have numerical value, you can’t calculate a mean or a median with them; the most you can derive is a mode (the most frequent value).

Then there are ordinal variables. You derive ordinal variables by dividing the field of measurement into discrete and ordered partitions: 1st, 2nd, 3rd; very probable, quite probable, not very probable, improbable; large, extra-large, XXL, SuperSize. As this last example suggests, the range covered by values of an ordinal variable doesn’t have to exhaust all the possibilities; all that matters is that the different values are distinct and can be ranked in order. Numeric coding starts to come into its own with ordinals. Give ‘large’ (etc) codes 1, 2, 3 and 4, and a statement that (say) ‘50% of size observations are less than 3’ actually makes sense, in a way that it wouldn’t have made sense if we were talking about car colour observations. In slightly more technical language, you can calculate a mode with ordinal variables, but you can also calculate a median: the value which is at the numerical mid-point of the sample, when the entire sample is ordered low to high.

Finally, we have interval/ratio or I/R variables. You derive an I/R variable by measuring against a standard scale, with a zero point and equal units. As the name implies, an I/R variable can be an interval (ten hours, five metres) or a ratio (30 decibels, 30% probability). All that matters is that different values are arithmetically consistent: 3 units minus 2 units is the same as 5 minus 4; there’s a 6:5 ratio between 6 units and 5 units. Statistics starts to take off when you introduce I/R variables. We can still calculate a mode (the most common value) and a median (the midpoint of the distribution), but now we can also calculate a mean: the arithmetic average of all values. (You could calculate a mean for ordinals or even nominals, but the resulting number wouldn’t tell you anything: you can’t take an average of ‘first’, ‘second’ and ‘third’.)

You can visualise the difference between nominals, ordinals and I/R variables by imagining you’re laying out a simple bar chart. It’s very simple: you’ve got two columns, a long one and a short one. We’ll also assume that you’re doing this by hand, with two rectangular pieces of paper that you’ve cut out – perhaps you’re designing a poster, or decorating a float for the Statistical Parade. Now: where are you going to place those two columns? If they’re nominals (‘red cars’ vs ‘blue cars’), it’s entirely up to you: you can put the short one on the left or the right, you can space them out or push them together, you can do what you like. If they’re ordinals (‘second class degree awards’ vs ‘third class’) you don’t have such a free rein: spacing is still up to you, but you will be expected to put the ‘third’ column to the right of the ‘second’. If they’re I/R variables, finally – ‘180 cm’, ‘190 cm’ – you’ll have no discretion at all: the 180 column needs to go at the 180 point on the X axis, and similarly for the 190.

Almost finished. Now let’s talk curves. The ‘normal distribution’ – the ‘bell curve’ – is a very common distribution of I/R variables: not very many low values on the left, lots of values in the middle, not very many high values on the right. The breadth and steepness of the ‘hump’ varies, but all bell curves are characterised by relatively steep rising and falling curves, contrasting with the relative flatness of the two tails and the central plateau. The ‘power law distribution’ is a less common family of distributions, in which the number of values is inversely proportionate to the value itself or a power of the value. For example, deriving Y values from the inverse of the cube of X:

X value Y formula Y value
1 1000 / (1^3) 1000
2 1000 / (2^3) 125
3 1000 / (3^3) 37.037
4 1000 / (4^3) 15.625
5 1000 / (5^3) 8
6 1000 / (6^3) 4.63

As you can see, a power law curve begins high, declines steeply then ‘levels out’ and declines ever more shallowly (it tends towards zero without ever reaching it, in fact).

Got all that? Right. Quick question: how do you tell a normal distribution from a power-law distribution? It’s simple, really. In one case both low and high values have low numbers of occurrences, while most occurrences are in the central plateau of values around the mean. In the other, the lowest values have the highest numbers of occurrences; most values have low occurrence counts, and high values have the lowest counts of all. In both cases, though, what you’re looking at is the distribution of interval/ratio variables. The peaks and tails of those distribution curves can be located precisely, because they’re determined by the relative counts (Y axis) of different values (X axis) – just as in the case of our imaginary bar chart.

Back to a real bar chart.

Figure #1: 433 weblogs arranged in rank order by number of inbound links.

The shape of Figure #1, several hundred blogs ranked by number of inbound links, is roughly a power law distribution.

As you can see, this actually isn’t a power law distribution – roughly or otherwise. It’s just a list. These aren’t I/R variables; they aren’t even ordinals. What we’ve got here is a graphical representation of a list of nominal variables (look along the X axis), ranked in descending order of occurrences. We can do a lot better than that – but it will mean forgetting all about the idea that low-link-count sites are in a ‘long tail’, while the sites with heavy traffic are in the ‘head’.

[Next post: how we could save the Long Tail, and why we shouldn’t try.]

A trick of the eye

A long time ago on a Web site far, far away, Clay Shirky wrote:

“We are all so used to bell curve distributions that power law distributions can seem odd.”

He then traced Pareto-like ‘power law’ curves operating in a number of domains where large numbers of people make unconstrained choices – most memorably, inbound link counts for blogs. The inverse ‘power law’ curve dives steeply, then levels out, glides downwards almost to zero and peters out slowly. And thus was born the ‘Long Tail’.

As I wrote here, there’s a problem with this article, and hence with the ‘Long Tail’ image itself. Despite repeated references to ‘power law distributions’, none of the curves Clay presented were distributions. They were histograms representing ranked lists: in other words series of numbers ordered from high to low.

What’s the difference? A short answer is that the data Clay presents makes his own comparison with ‘bell curve’ (normal) distributions unsustainable: order from high to low and you will only ever get a downward curve.

For a longer answer, you’ll have to look at some numbers. Here are some x,y values which would give you a normal distribution. (For anyone in danger of glazing over, that’s ‘x’ as in horizontal axis, low to high values running left to right; ‘y’ values are on the vertical axis, low to high running bottom to top).

1 1
2 30
3 100
4 240
5 400
6 600
7 750
8 900
9 960
10 1000
11 1000
12 960
13 900
14 750
15 600
16 400
17 240
18 100
19 30
20 1

OK? And here are some co-ordinates which would give you an inverse power-law distribution:

1 1000
2 444
3 250
4 160
5 111
6 82
7 63
8 49
9 40
10 33
11 28
12 24
13 20
14 18
15 16
16 14
17 12
18 11
19 10
20 9

Just for the hell of it, here are some numbers that would give you a direct (ascending) power law distribution:

1 9
2 10
3 11
4 12
5 14
6 16
7 18
8 20
9 24
10 28
11 33
12 40
13 49
14 63
15 82
16 111
17 160
18 250
19 444
20 1000

Finally, by way of contrast, here’s a series of numbers.


I’ve sorted these numbers high to low, but – unlike the other three examples – there’s nothing in the data that told me to do that. You could arrange them that way; you could sort them low to high instead; you could even hack them about manually to produce a rather lumpy and uneven bell curve. It’s up to you.

I’m not saying that a ranked listing – arranging numbers like these high to low – is meaningless. The ranked histogram is quite a good graphic – it’s informative (within limits) and easy to grasp. What I am saying is that it’s an arbitrary ordering rather than a distribution. Which is to say, it’s not the best way of representing this data – let alone the only way. It’s a relatively information-poor representation, and one which tends to promote perverse and unproductive ways of thinking about the data.

More about this – and a couple of constructive suggestions – next time I post.

When is a spike not a spike?

When it’s a long tail. Maybe.

David Weinberger writes:

In a conversation with Erica George at the Berkman she pointed out that the demographics of Live Journal don’t always represent one’s experience of Live Journal — the demographics say that teenage girls are the largest users, but if you’re a 25 year old, your social group there may not look that way at all.

Which raises an issue about the way the “long tail” is pictured. Clay’s charts are accurate depictions of his data, but they have a mythic power that’s misleading: The long tail looks like, well, a long tail when in fact it’s a fractal curlicue of relationships.

This is an interesting point in itself – perhaps the blogosphere would be better viewed as a series (archipelago? galaxy?) of more or less closed, more or less interlinked ‘spheres’. I’m not sure how you’d visualise that, though – perhaps something like the Jefferson High School network diagram?.

But there’s a broader point about the accuracy of those ‘long tail’ graphics. Adam Marsh made an interesting point here about a recently-discovered ‘long tail’:

Clay refers to “the characteristic long tail of people who use many fewer tags than the power taggers.” While this chart does exhibit a “long tail,” this is simply a result of the fact that the users were ordered by decreasing tag usage (also true of the following three charts) — the X axis here doesn’t represent a value, it is just a sequence of users.

The phrase “long tail” usually refers to the observation that for many distributions, the number of elements with outlying values (the “tail”) may be cumulatively significant compared to the number of elements clustered near the average.

On inspection, it turns out that this is also true of the celebrated ‘Power law and Weblogs’ graphic: there are no values on the X axis, just a list of blogs arranged in descending order of number of links. This matters, because in a graphical representation of a statistical distribution both axes carry information. Typically, values of the variable being measured run low to high on the X axis, left to right, while the count of occurrences of each value runs high to low on the Y axis, top to bottom. Clay wrote, “We are all so used to bell curve distributions that power law distributions can seem odd.” But Clay’s own graphics aren’t so much odd as misleading, and not only because he’s put high values on the left of the graph rather than the right. In effect, he’s got two axes conveying one piece of information. Andrew Sullivan’s blog and Instapundit get a high Y value (lots of links) and a high X value (because all the sites with lots of links have been sorted to the left).

If you took the same numbers and plotted them on an X axis with values – if you produced a graph showing how many blogs had how many links, with zero at the origin on both scales… Well, I don’t know what would happen – but five minutes’ experimentation tellsreminds me that, if you wanted to produce a nice clear series of vertical bars rather than a line that wanders all over the place, you’d need to put ‘number of blogs’ on the Y axis and ‘number of inbound links’ on the X axis, rather than vice versa. (There’s a simple reason for this: some values are unique by definition, others aren’t.) Which in turn means that any vertical spike would represent large numbers of blogs (say, for example, blogs with small numbers of inbound links) while any long tail would represent small numbers (say, for example, the few blogs with lots of links).

Caveat: I haven’t crunched any actual numbers, or even mumbled them gently. But maybe we’ve been looking at this the wrong way round, statistically speaking. Perhaps the long tail is the spike; perhaps the spike is really the long tail.

For Tomorrow (I) – 126 as a limit

Who’s Backing Blair? Probably not Chris Applegate, who says tactical voting is rubbish. Not Ken MacLeod, who fears we’re sleepwalking towards a Tory government. Certainly not Tom Watson MP, who says that making a protest vote is “one hell of a risk”.

This is the first in a series of posts inspired by Backing Blair and its critics: it began as an attempt to identify exactly what was wrong with Tom Watson’s arguments against protest voting. It grew from there; I’m going to be writing about electoral blackmail, Howard Dean’s presidential campaign, the state of the Left and Paul Anderson’s recent revival of Neville’s Inch, among other things. But to begin with, here’s some arithmetic. (Thanks to Electoral Calculus, UK Polling Report and, and in particular this site at Keele University, for the figures.)

At present, the Labour Party has 409 MPs out of 658 – a theoretical majority of 160. The number of Scottish constituencies will be reduced by 13 at the next election. In effect, Labour will go into the election with 400 MPs out of 645 – a majority of 155. The figures for the Conservatives and the Liberal Democrats are 164 and 54. (Boring but relevant information: in what follows I’ll use the by-election figures for the two seats which have changed hands at by-elections since 2001 (Leicester South and Brent East), but use the 2001 figures for the four by-election holds (Hartlepool, Birmingham Hodge Hill, Ogmore, Ipswich). I’ll also use the 2001 figures for two seats which have changed hands without an election (Wantage, Shrewsbury & Atcham) and for the 59 redefined Scottish seats; this includes one seat, the Scottish Conservative marginal of Galloway & Upper Nithsdale

%d bloggers like this: