The Unknowable Country: What does Reporters Without Borders’ press freedom index actually mean?

Canada enjoyed one of the highest levels of press freedom in the world last year. But how is that measured? Columnist Sean Holman questions why Reporters Without Borders won’t release the questionnaires that determine a country’s level of press freedom. By Sean Holman Last year, the country's freedom of information commissioners called for a modernization…

By Sean Holman

Last year, the country's freedom of information commissioners called for a modernization of the antiquated laws that are supposed to allow, but in many cases, now frustrate journalists' access to public records.

Indeed, according to the Centre for Law and Democracy, 55 other nations have stronger information rights laws than we do.

Anecdotally, Canadian reporters have also complained about their shoddy access to public officials, with senior members of the governing federal Conservative Party making statements that have the effect of discrediting the media.

Nevertheless, according to a report by Reporters Without Borders, Canada continued to enjoy one of the highest levels of press freedom in the world last year. But we don't know exactly why that is because the advocacy group won't release the data underpinning that ranking.

Related content on J-Source:

Reporters Without Borders has been compiling its press freedom index since 2002. A country's position on that list is partially determined by the amount of violence (including imprisonment and arrests) against journalists within its borders. But the results of a survey—which is sent to 18 freedom of expression groups, Reporters Without Borders's 150 correspondents, as well as journalists, researchers, jurists and human rights activists—also factor in.

That survey asks questions about six different topics:

• the degree to which different opinions are represented in the media;

• media independence;

• media self-censorship;

• press freedom, freedom of information and freedom of expression legislation;

• the transparency of the "institutions and procedures that affect the production of news and information;"

• the "quality of the infrastructure that supports the production of news and information."

But when I emailed asking for the results of the Canadian surveys, this is the response I received from Camille Soulier, head of the Reporters Without Borders Americas desk: "I'm sorry but in the interest of our sources protection and so as to not jeopardize the independence of our work, our questionnaires are confidential. I know sources in Canada are not seriously at risk, as opposed to those in Honduras or Iran, but we apply the same rules to all. This is clearly unfortunate considering you are writing a column on transparency… but I hope you understand and I do apologize."

It's understandable Reporters Without Borders would want to protect its sources—which, according to Soulier, usually amounts to a panel of about five experts per country. But why doesn't the group release the results of its questionnaires?

Well, according to Soulier, it's because those results could be used to create different indexes about the survey's topics. So Reporters Without Borders keeps that information "confidential in the interest of publishing one single index."

But that's just one of several of limitations associated with that single index.

Reporters Without Borders acknowledges their rankings "should in no way be taken as an indication of the quality of the media in the countries concerned."

The group has also changed its method of compiling the index over the years.

And, as the Washington Post's Max Fisher observed in an interview with NPR's “On The Media,” "Most of the criteria [in the questionnaire] are actually subjective. And they send these forms out to hundreds of people around the world and ask them to rate things like self-censorship, how hard is it to get a TV license on a one-to-10 scale. And the thing that a few political scientists pointed out to me when I started asking questions about this is that somebody in the U.S. and somebody in Russia or Namibia is going to have a very different 10-point scale."

None of this is to say Reporters Without Borders isn't doing laudable work. It is. And the very existence of the Press Freedom Index should be applauded.

But the group seems to be asking reporters to take the accuracy of that index at its word—something any journalist should be loathe to do.

And that puts into question whether Canada has the 18th freest press in the world or something else entirely.

Sean Holman is a journalism professor at Mount Royal University, award-winning investigative reporter and director of the documentary Whipped: the secret world of party discipline. You can find more of his writing at the Unknowable Country.

Related content on J-Source:

Tamara Baluja

Tamara Baluja is an award-winning journalist with CBC Vancouver and the 2018 Michener-Deacon fellow for journalism education. She was the associate editor for J-Source from 2013-2014.