Data Issues

In 2010 the World Bank, the Food and Agricultural Organization of the United Nations (FAO), and many of the UN family of agencies and organizations began providing free access to data bases which in the past were available only on a subscription basis. This "open data" policy has made formerly difficult to obtain data available to anyone with Internet access. Since 2010 many of the international organizations have added new databases to their listings as well as providing new more user friendly interfaces and data formats (links to many of these data bases are available in the Data Sources section of this web site.)

Linking data from different data bases is relatively simple if you are using data for only one country or region, even if you are using a spreadsheet. However, linking data from multiple countries and different data sources is a serious challenge. To link countries you need different organizations to use a consistent method method of identifying countries. While there is an internationally accepted set of country name spellings and country codes available from the International Standards Organization (ISO), and while most of the international organizations accept these ISO standards in principle,  in practice, different data base providers often use different spellings of country names and different numeric or alphabet codes to identify countries or regions in their data,  making automatic linking almost impossible.

Data base systems require a common field in the "rows" being connected from different tables. In the case of these data bases the common field would be a country name or an agreed upon country code. While all the databases have a field for country name, the spelling of these names must be exactly the same-- data bases will not recognize Cote d'Ivoire as being the equivalent of Côte d'Ivoire--the smallest difference means that the fields will not link automatically and have to be linked by hand. Country names are an unfortunate choice for a common field because many of the names are complicated and even when an attempt to enter the name correctly is made, spelling errors are common. In addition, the data base developers often use the name by which a country is commonly known, rather than the agreed upon ISO standard -- for example it is much more convenient to enter "Bolivia" than the ISO standard "Bolivia, Plurinational State of".  One method of linking tables by country is to build a series of "look-up tables" for database provide a common link to the particular spelling of the country name. Using, for example, the three country ISO code for the country--in the case of Bolivia it would be BOL. These look-up tables allow the different tables to be easily linked.

The following pages will make accessible look-up tables for some of the more frequently used international data-bases. (Note of caution when using the look-up tables-- it will still be necessary to check for missing links-- country name spellings can be different for databases within the same agency--name spellings may be changed from time to time, etc., but the look-up tables should make the problem of linking pages easier.)The menu on the left hand side will give access to the look-up tables by agency.  The ISO provides both alpha and numeric codes-- the three character alpha codes will be the standard--unfortunately different pieces of software handle the ISO numeric codes differently -- for example the ISO numeric code is defined as a text character with preceding 0s when necessary-- code 004 for example will often be read by software as simply "4" and either as a numeric or text field--meaning that automatic linking is not possible.

The left side menu gives access to a page which describes linking issues for the listed organization and provides access to a downloadable look-up table which links the name of the country using the organization's spelling to the ISO three character alpha code. One page is also devoted to the a download of the ISO codes. Codes for organizations will be added as they are developed. The page titled "Country Names" illustrates the problem of country name spellings in some of the major data bases.

Copyright ©1998/2015 GRI    Updated 6 May 2015