May 2, 2011. Jack Layton led the federal New Democrats beyond the limitations of their history, and within sight of the promised land of government. Just over one hundred days later, his death opened the New Democrats to a leadership contest at a point when the party is stronger and more vulnerable than at any time in history. The choice it makes in March 2012, at its leadership convention in Toronto, will be a good sign of whether the party is ready to challenge the Conservatives for government or will sink back into obscurity.

Only one candidate is positioned to complete Layton’s transformation of the NDP into a party that can take on Stephen Harper. To become leader Thomas Mulcair will have to overcome the suspicions of a very tribal party and develop a new base for a party uncomfortable with change. But change used to be, and could be again, the driving force for Canada’s progressive party. Change is needed. Canada needs a party able to speak for the 99 per cent, not with slogans but with real reforms. A party that translates the values of a caring and cooperative Canada, where social justice is another way of describing the way people want their country to be: their communities, their relationships, their lives. Canada became a rich country despite the problems of exploitation and exclusion because we have been a progressive country, overcoming our problems together, becoming better.

New Democrats who see balanced books and strong social programs as complementary, not contradictory, are scattered across Canada. While the Saskatchewan and Manitoba branches of the NDP have long inspired social democrats with their combination of solid financial management and efficient, activist government, the NDP elsewhere has tended toward woolly thinking. Sensible socialists have had recent successes. In 2009 Nova Scotia became the first new province to elect a NDP government since Ontario in 1990. Unlike now-Liberal Bob Rae, who alternately appeased and enraged the union movement, business and just about every other group, Darrell Dexter has managed a careful government, even paying down provincial debt after an unexpected budget surplus in 2011.

For most of its history the federal NDP was the less serious option for ambitious prairie New Democrats. It became a playground for the more extreme activists, held at bay by an often rigid party establishment experienced in deflecting new energy and ideas. In Regina and Winnipeg, and sometimes in other provinces, the NDP offered a path to power. Ottawa offered the sterility of a party comfortable with its also-ran status.

A decade ago British sociologist Anthony Giddens described Canada’s federal New Democrats as “the last unreconstructed social democratic party in the Western world.” By that he meant the party had not confronted the failures of centralized economic planning, or absorbed the energy of the New Left movements of the 1960s. Before returning to New Brunswick and politics in my home province, I worked for ten years with a major international NGO (the National Democratic Institute) offering advice on political party development in many developing countries. My experience is that most parties have a lot in common. But the NDP stands out as the only party I have worked with where winning elections is not a key goal for members. Many are honestly comfortable with the New Democrats serving as the self-appointed “conscience” of Parliament. The NDP may revere Tommy Douglas, but except in Saskatchewan, Manitoba, the Yukon and, decades later, Nova Scotia, his ambition had minimal impact on NDP leaders.

This reflects the federal NDP’s complicated past. Its precursor, the federal Cooperative Commonwealth Federation (CCF), was the underachieving brother to a Saskatchewan party used to winning consecutive majorities. Decimated by the Diefenbaker landslide in 1958, the federal CCF, in partnership with the Canadian Labour Congress, created the New Democratic Party. Godfather to the CCF-labour marriage was David Lewis, who persuaded Tommy Douglas to leave Regina, after four terms as premier, to become the new party’s first leader.

When Douglas made the move to Ottawa, he found a very different party from the one he left behind. Shaped by an uneasy alliance of trade unions and intellectuals, this party did not look to Saskatchewan for inspiration but to the British Labour Party. After a mid-1940s spike in support, the Ontario CCF was weakened by fierce internal battles with the Communists, leaving the party a marginal player in the province. Nonetheless, the Ontario party exerted a large influence on the new federal party. Staff and financial support, most critically from the auto and steel workers unions, flowed from Toronto to Ottawa and back again.

Because it so dominated thinking among Ontario New Democrats, the history of the British Labour Party bears mention. Born in 1900, it was a creation of the Trades Union Congress. Riven by infighting, it did replace Britain’s crumbling Liberals as one of two dominant parties but realized only two transformative election victories, in 1945 and 1997.

The most powerful advocate of the British Labour Party as model for the NDP was David Lewis. Born in Poland and raised in the Jewish Bund tradition of socialism, he came to Canada when his family fled eastern Europe after the Bolshevik Revolution. A brilliant student, he attended Oxford as a Rhodes Scholar and immersed himself in Labour politics. He rejected the offer of a safe Labour seat so he could return to Canada to build the CCF. Where Douglas’s rhetoric inspired the party, Lewis built the structures and party culture that endured for 50 years. Lewis believed a formal trade union alliance was essential to building a social democratic party, and that non-Communist unions were an organizational force to protect the left from the then-powerful Communist movement.

In a series of lengthy battles, first as organizer and then as leader, Lewis purged the party, and the unions over which he and his supporters had influence, of Communists. They did this in the name of a social democracy that could best be defined as “Liberals in a hurry.” This model of iron discipline in protection of a tepid ideology failed. The NDP continued to languish in third place.

By the time he was angling to replace Douglas as leader in the early 1970s, Lewis’s command-and-control style of politics had gone out of style, replaced by the cacophony of the sixties New Left. The Movement for an Independent Socialist Canada, known by its nickname “the Waffle,” called for a radical move to the left, and brought the voices of feminists, young people and new social movements into the NDP, where they clashed with the party old guard. At the 1971 convention, it took Lewis four ballots and the concerted efforts of organized labour to beat back Waffle candidate Jim Laxer and become leader.

Party elders had struggled against Communism in the forties and fifties; now a new generation of activists had prevailed in battle with a new generation of leftists. For aspiring NDP leaders the lesson to be drawn from the war against the Waffle was the danger of new ideas and new policies. The way to get ahead was to be a professional organizer, preferably with a background in or connections to a trade union. The party presented a false choice: embrace a stagnant party culture that offered promotion or choose the life of a powerless activist. Many chose to leave the party altogether – to join single-interest groups or even move to the Liberals, where they could speak freely and have some hope of being part of a party in power. New Democrats scorn social democrats who defect to the Liberals; we should have looked inward and asked why the NDP was so flawed that the Liberals looked appealing.

The battle with the Waffle made the NDP the most conservative party in Canada. While the Liberals and Progressive Conservatives experimented with new ideas, policies and structures, the New Democrats dug in. Personal and professional connections trumped talent. Centre-left ideas circa 1970 became party dogma: universal health care, knee-jerk anti-Americanism, distrust of provincial power and of markets.

Drawing on Marx’s 18th Brumaire, if the evolution of the post-1917 Russian Bolsheviks was tragedy on a Napoleonic scale, the post-1970 evolution of the NDP was farce in the tradition of Napoleon III. Both Bolsheviks and NDP were transformed from parties of ideas and ideals into parties of apparatchiks. Through the leadership of Ed Broadbent, Audrey McLaughlin and Alexa McDonough, organizers ran the party, centralizing power in the leader’s office. Sometimes the party achieved a random resonance with the public, as when Broadbent took the party to short-lived leads in 1980s opinion polls. These spikes in popularity confirmed to party insiders that all was well. When the public subsequently abandoned its flirtation with the party, insiders took this as a sign, not that a change in our approach was needed, but that Canada was still not mature enough to appreciate our worldview.

NDP leaders became adept at listening to the subtle signals from internal factions. The demands of public sector trade unions and Toronto intellectuals were balanced with grousing from provincial party sections not aligned with the federal party’s positions. The NDP became a brokerage party in the same mould as the Liberals, but brokering the interests of party factions instead of the Canadian electorate.

This is the party Jack Layton inherited in 2003. He was elected without the support of many unions, who stood behind longtime MP Bill Blaikie. Despite his long history in the party and his experience as a Toronto city councillor, Layton was seen as an outsider. He was linked to the 2002 New Politics Initiative (NPI), which had called for the creation of a new party from a New Democratic Party that, in the 2000 general election, had slipped to 13 seats and 8.5 per cent of the vote. The NPI’s call for new links with the emerging antiglobalization movement and its support from the small, Trotskyist NDP Socialist Caucus contributed to concerns that the Waffle had been reincarnated in a new century.

Layton became leader in 2003 thanks to a party reform movement called NDProgress, created just before the 2000 election and dedicated to modernization of the party. The leaders of the group – I was one of them – were frustrated party staff or activists. We looked with envy to the innovative campaign tactics and recast ideas proposed by the Bill Clinton Democrats and Tony Blair’s New Labour.

NDProgress recognized that structural change must precede policy renewal. The major demand was “one member one vote,” which meant the elimination of union bloc voting at party conventions. Who could object to the principle of one person having one vote, or limiting the power of groups to influence the self-styled people’s party? The process of objecting to the reforms would strengthen the call for change. Despite a last-ditch effort by some unions to quash the reform at the Winnipeg convention in November 2001, the new system was put in place. Layton won on the first ballot in 2003 with 53.5 per cent of the vote. Without the NDProgress reforms, he would likely have lost.

The team around Layton was a mix of the old and the new. Party reformers, including a much-needed group of young people with roots in student politics, mixed with elders like Broadbent who, despite his establishment status, recognized that the NDP’s fire had grown so dim it was in danger of blowing out. Layton ran four election campaigns. The New Democrats grew incrementally in the first three, winning 19 seats in 2004, 29 in 2006 and 37 in 2008. The popular vote rose modestly from 16 per cent in 2004 to 18 per cent in 2008. Layton brought energy to the leadership that had not been seen since Broadbent in his prime. The party looked and sounded different, and much more professional, than anything that had come before.

Layton’s inner circle, including the 2006 and 2008 campaign director and now leadership candidate Brian Topp, were nervous reformers aware of how conservative our party can be. They admired the British Labour Party for its communications and public image departments; they were not interested in replicating Tony Blair’s experiment in prodding his party to think differently. On gaining power within Labour, Blair aggressively confronted the flaws behind many of Labour’s central policies and embraced new policies never imagined by the party’s previous generation. Immediately on winning the leadership, he pushed changes to Labour’s constitution, replacing an explicitly socialist preamble written in 1918 with a more modern communitarian version. With Blair promising to quit as leader if his reform was rejected, the new preamble was approved in a party-wide referendum by 58 per cent. To his party and his country, Blair had confirmed his reputation as a tough-minded reformer.

In contrast, under Layton policy reform, when it took place, happened by stealth or on the fringes, with easy-to-sell campaigns that, among other initiatives, proposed limits to ATM fees. Layton waited until 2011, eight years and four elections after he became leader, to introduce changes to the NDP’s preamble comparable to those Blair had introduced. There was no referendum, no consultation; the old preamble simply disappeared from the party’s website. A new version was floated just two days before the party’s 2011 convention in Vancouver. In the face of likely defeat Topp, the newly elected party president, tabled the new clause.

From 2004 to 2011 there had been little time for serious policy debate: minority governments meant ever-looming campaigns that discouraged party dissidents. Weaknesses in the NDP platform were largely overlooked by the media and political class, except in the context of Layton’s efforts to use his deal-making skills, honed by years at Toronto City Hall, to exact concessions from the Liberals and the Conservatives. Even Layton’s opponents conceded he was more successful than any New Democrat had been since David Lewis with his tactical manoeuvres during Pierre Trudeau’s 1972–74 minority government.

Building the NDP across Canada coincided with Layton’s efforts in Quebec, where it had been shut out for decades. The Quebec left was more open to decentralization within Canada and to free trade with the United States. They found the anti–free trade and centralizing Anglo-dominated NDP an uncomfortable home. Quebec social democrats rose to leadership positions in the Parti Québécois and Quebec Liberal Party in the 1970s, but subsequently lost influence.

In Layton’s first two elections there were behind-the-scenes gains in Quebec. The skeleton of an organization appeared. The party appealed to those Quebecers who wanted social democracy, not separatism, and who were tired of the Bloc Québécois’s monotonous drumbeat. A symbolic gesture: the federal party passed the Sherbrooke Declaration in 2005, emphasizing the right of Quebec to separate if it wished on the basis of a 50-per-cent-plus-1 vote – effectively reversing the party caucus’s previous support for the Clarity Act enacted in the wake of the 1995 referendum.

In 2006 Thomas Mulcair, the prominent Quebec Liberal and environmentalist, joined the party after leaving Premier Jean Charest’s cabinet in a dispute over development in a provincial park. Against the odds, the fiery lawyer ran and won in a 2007 byelection in Outremont, becoming only the second New Democrat ever to win a seat in the province. Winning reelection in the 2008 general election, Mulcair showed that the NDP in Quebec had become an organizational, if not yet an electoral, force.

As late as early April 2011, the NDP was seen as an also-ran in the May 2 election. It was frequently stated that Bloc leader Gilles Duceppe and Layton were as extraneous to the real choice of who should be prime minister as Green Party Leader Elizabeth May.

There will be arguments over what caused the Quebec NDP campaign to spring back to life over the Easter weekend. The Bloc campaign was stale; Ignatieff was not catching on and the Liberal campaign was battered from sustained Conservative attack. Quebecers warmed to Layton’s astonishing cheerfulness and eagerness and his command of colloquial Quebec French. Francophone Quebecers made a shift comparable to that which swept Mulroney’s Conservatives to power in 1984 and the Bloc to Official Opposition status in 1993.

Layton deserves credit for the fact that the Quebec NDP was ready to ride the wave that broke on May 2. The team he assembled, led provincially by Mulcair and organized by Raymond Guardia, a union man now managing Topp’s leadership bid, was stretched tight but well resourced. The media did their best to push the party off balance, but did not succeed.

The party’s new discipline was apparent when the 103-strong NDP Official Opposition ran an efficient filibuster against back-to-work legislation directed at the postal workers in June. It was on show again when the party voted unanimously to support the United Nations mission in Libya, a hard pill to swallow for a party with a strong pacifist wing. Over a terrible summer marked by the announcement of Jack Layton’s new cancer and then his death, the party was professional, united and calm.

There are parallels between the start of the NDP leadership campaign now under way and the 1994 Labour Party campaign that brought Blair to the leadership. In both cases, the former leader died unexpectedly. In 1994 Tony Blair, then working with Gordon Brown as a tag-team of modernizers inside the Labour Party, took advantage of the days after the death of John Smith. Brown had been seen as Smith’s natural successor, but he refused to mobilize his troops until after Smith’s funeral. This gave Blair’s supporters a week to frame the coming race. While many were shocked and some were offended, Blair’s initial success persuaded Brown not to contest the leadership.

Brian Topp did something similar. A longtime union leader and party organizer, he had no public profile. With Ed Broadbent at this side he seized the initiative and announced his candidacy quickly. In the weeks after his announcement Topp bludgeoned the party with endorsements from one heavyweight after another: former Saskatchewan Premier Roy Romanow was supposed to sew up the prairie pragmatist vote while Vancouver MP Libby Davies, standard bearer for the party’s old left, was supposed to sew up the left.

Topp, like Blair, needed to keep his heavyweight challenger out of the race. Here the parallel stops. Topp failed to stop Mulcair’s entry, in part because – unlike Blair and Brown – Topp and Mulcair have little in common. Out of convenience or conviction, Topp has become the voice of the status quo: he has gone so far as to defend the right of unions to preserve their bloc vote in the leadership race, despite that right having disappeared six years before. In contrast Mulcair has stated positions that, while not exceptional to regular Canadians, violate the NDP’s unwritten conventions. He has praised free trade, noting environmental protections in NAFTA he had helped draft. He has made it clear he wanted the NDP to move beyond its base. He has recounted how he told the head of the Steelworkers union that he would stand against special voting rights for any group within the party.

Topp’s failure to keep Mulcair out of the race has led to a crowded field. Topp secured support from Lewis’s union, the Steelworkers. But the Lewis machine has fragmented. Peggy Nash, a bilingual MP and former Canadian Auto Workers leader, entered the race, pitting the two big private sector unions against each other. Another candidate, Robert Chisholm, who led the Nova Scotia NDP to Official Opposition status in 1998, worked for many years for the Canadian Union of Public Employees. This fragmentation of union support between different candidates is unprecedented, and a sign of the challenges facing the NDP. Of Topp, Nash and Mulcair, the three fluently bilingual candidates with potential national organization, only Mulcair is offering a shift from a party obsessed with preserving its internal orthodoxy to one that is intent on changing Canada.

To win, Mulcair will have to sign up thousands of new members. He is at a disadvantage in a race where many existing members are comfortable with the old nostrums. But if Canada is a fundamentally social democratic country – as I believe – then Mulcair is far more likely to turn social democracy into a national force than Topp or Nash. Mulcair can cross partisan lines. If the NDP is going to occupy 24 Sussex Drive, the path is through the concerns of regular Canadians, not the interest groups courted traditionally by the NDP or now by the Occupy movement. Mulcair has the potential to turn Layton’s calls for a “kindler gentler Canada” into programs, plans and results that will change lives.

My hope is that New Democrats new and old will break our tradition of back-room dealing, and openly confront some serious questions: What do Canadians want from our federal government? How will we pay for rising health costs as the baby boomers age? When we talk about the public sector, is our first responsibility to protect public servants or public services? Are Canadians ready to overhaul the tax code and support a meaningful carbon tax? Will we confront the deprivation on reserves and the reality of poverty among urban Aboriginals? What responsibility do we have toward each other, to the citizens of other provinces and other countries? How will this be reflected in our foreign aid and military policies?

Mulcair is best placed to answer these questions. He is emphatic in stating the importance of sound financial management. As Tommy Douglas and Allan Blakeney never tired of saying, if you’re running a social democratic government you don’t want to be in debt to bankers. On the environment, Mulcair has an international record in government. In the pursuit of social justice, we can leave the lamentations to the Occupy movement. Canada’s problems are clear; our job as social democrats is to build broad support around solutions and then get the job done. For too long all Canadian parties have been offering a “Father knows best” style of government that infantilizes voters, suggesting we can have it all: low taxes, more generous social programs, a constantly rising share of GDP devoted to health costs. We cannot. A Thomas Mulcair–led NDP offers a national party that will embrace the challenges of the 21st century the way Tommy Douglas embraced the challenges of the 20th.

Photo: M.J. Coldwell and David Lewis looking over some papers together, September 1947