The delicate art of negotiating for data
It’s easier than you think, says Glen McGregor. In this piece for Media magazine, McGregor explains how to make effective access to information and freedom of information requests, the steps to successfully negotiating with coordinators, and why IT people may be your saving grace when asking for data from governments.
It’s easier than you think, says Glen McGregor. In this piece for Media magazine, McGregor explains how to make effective access to information and freedom of information requests, the steps to successfully negotiating with coordinators, and why IT people may be your saving grace when asking for data from governments.
Intro by Media magazine editor David McKie
When Opposition MPs rose in the House of Commons last month to question the motives behind the Conservative's government decision to limit the number of lakes and rivers protected by federal legislation, they were reacting the Glen McGreagor's story in The Ottawa Citizen. Entitled, "After federal changes to waterways rules, 90 per cent of protected lakes lap on Conservative shores," the piece suggested that the ruling party's choices may have been guided by politics — not science.
Opposition politicians raised similar questions about political motives, after stories Glen reported with Stephen Maher in the fall of 2009 that demonstrated a disproportionate amount of federal stimulus money during the last recession went Conservative ridings.
In both instances, Glen used mapping techniques that he's explained in past editions of Media magazine. On many occasions, the data used to achieve the results can be obtained from public sources, as was the case in the two stories I've just mentioned.
Fortunately, governments at all levels are making an increasing amount of data publicly available, a new reality which will force more journalists to learn mapping techniques that will be a major focus of columns in Media magazine.
However, in many cases, the data we need is not public. And in those instances, we must develop strategies to obtain that data from bureaucrats, something Glen explains in his column in the edition of Media magazine that I've just posted.
The goal is to make these columns must-reads for journalists who want to stay ahead of the curve. What makes this easy, is when we can point to impactful stories that employ these techniques, and then get the journalists to explain how they put those stories together.
***
By Glen McGregor
If you’ve ever called the city to complain about barking dogs or early morning jackhammers outside your window, you’ve created electronic data.
If you’ve applied for a marriage licence, driven on a toll highway, received a speeding ticket or obtained a passport, you have contributed to the growing pool of electronic data retained by governments.
Virtually every interaction we have with municipal, provincial and federal government is electronically captured and entered in a database somewhere. And under open records laws, you’re entitled to have a copy of the databases. Like no time in history, journalists can get access to these enormous number of records to use in their reporting just by asking.
The challenge is getting these datasets off government mainframes, onto your own computer, and into your news story.
The open data movement has compelled some governments to proactively publish electronic data, but these typically are of little interest to journalists. The City of Ottawa’s open data site, for example, pushes maps of swimming pools and tennis courts. Rarely do governments voluntarily release records, such as the locations of bike thefts, that might lead to an embarrassing analysis of their own performance.
To get the good stuff, you’ll still need to file freedom-of information (at the municipal and provincial levels) or access toinformation (at the federal level) requests for electronic data.
All this takes is a letter to the correct agency and a request that specifies you want the records in electronic format. I typically ask for “all records in all fields of the database of [whatever you’re looking for] in machine-readable, electronic format, in either tab-delimited or comma delimited ASCII text file, stored on USB thumb drive or CD-ROM or DVD-ROM.”
Once you’ve filed your request and sent in the fee, $5 in most jurisdictions, you’ll have to wait for response, which, more often than not, will be a polite “No.” Thus begins your negotiation.
When your request is rejected, it will be accompanied by a letter with instructions on launching an appeal of the decision. Hold off, for now.
The next step, if you haven’t done so already, is get in touch with the freedom of-information or access-to-information coordinator handling your file. Tell her you will be appealing, but would first like a chance to see if there’s a way to avoid going this route by modifying your request.
Governments keep careful tabs on the statistics these kinds of requests and most coordinators want to avoid appeals. Tell the coordinator you’d like to set up a meeting, via conference call or, preferably, in person, with the information technology people and bureaucrats who hold the data you’re after.
Many access-to-information staff are data illiterate, incapable of answering questions about the database you want. Having an information technology geek in the room solves a lot of problems.
One misconception is that the number of records requested will make more work. You may even be given you a massive fee estimate. When I asked the City of Ottawa for five-years’ of parking ticket data, officials told me it would cost $140,000 to produce copies of the 2.4 million tickets.
[node:ad]Until officials got the city’s information technology people involved, they didn’t understand that it doesn’t take any more effort to save a file with a million records than one with only 10 records. We eventually got the entire dataset for $40.
The information technology people will also help you overcome another common hurdle in data requests — privacy concerns.
Most databases contain sensitive personal information, such as the names and addresses of people interacting with government. With printed records, someone must manually go through each page of the document and eliminate the offending information.
But with a request for data, you can easily modify your request to exclude any field that is likely to contain personal information — such as fields that contain a narrative of what happened. Ask the IT people to bring a copy of a few sample records from the database so you and the freedom-of-information coordinator can see which fields to exclude from the database you will request. When they create a “query” to export the data for you, computer people will eliminate these fields
Street addresses, cities and postal codes are usually in separate fields, so you may only need to exclude the street field or part of the postal code. Often, you can obtain the first three characters of the postal code — known in post-office speak as the forward sortation area or FSA — which won’t identify any one person, but will give you an approximate location.
At the Ottawa Citizen, we spent four years fighting with Health Canada to get a copy of the database used to track applications for the federal medical marijuana program. The department initially refused to release virtually every field in the database, including the applicant’s province.
We appealed and continued to negotiate until, finally, the department relented and released a complete list of applications, including the medical conditions of the patients and their FSAs.
If the bureaucrats refuse to meet or remain intransigent, remind them that you have a legal right to obtain a copy of the data, allowing for certain exemptions. Then launch an appeal.
It’s handy to check with the appeal body — the Information and Privacy Commissioner of Ontario, for example — to see if there are any rulings on release of similar data.
Most open-records laws in Canada require some sort of mandated mediation process that will force the government to the table.
Again, this is another chance to find a way to overcome any technical hurdles or privacy concerns by amending your request slightly.
At this stage, the public servants you deal with will usually want to avoid a ruling against them, so they tend to be more cooperative.
Some tips
* Always include your phone number and email address in your original request letter, and indicate that you are willing to talk anytime to clarify or discuss your request. If the freedom-of-information person contacts you with problems, propose a meeting.
* Make sure that the government department or agency understands you want machine-readable data, not scanned versions of printed records. A common problem with data requests are image files of records saved as PDFs. You can’t work with this data. It is next to useless.
* Most open-records laws allow you to receive data in the format in which it was created. Today, that is almost always an electronic format. Don’t settle for printouts.
* Don’t let the IT people snow you with geek talk about “Oracle” or “MySQL database formats”. Most modern database systems will easily allow export to text files you can work with in Microsoft Excel or Access.
* Never accept a huge fee estimate. Exporting data is a relatively simple job and shouldn’t take too long, unless your request is particularly complex.
* Be prepared to bend a bit. Every database structure is different and the information technology people might have legitimate reasons for raising concernsAsk them to suggest a better way to get all or most of the data you want without creating too much work for them.
* Don’t lose your cool. It can be frustrating dealing with bureaucrats who raise silly arguments to keep data out of your hands, but if you remain polite and respectful, they’ll eventually understand you have a legal right to the data.
Glen McGregor is a national affairs reporter with the Ottawa Citizen. He is available to give web-scraping and datajournalism seminars to your newsroom or classroom. Contact sushiboy21@gmail.com
This article first appeared in the latest issue of Media magazine, the CAJ's media news publication, edited by David McKie. It has been reprinted here with permission.
