Downloading Data

Each of the major data providers has a slightly different interface for downloading data (they also often have different data formats--this will be dealt with in the section "using data"). There are too many interfaces to deal with them all, but most can be figured out once you are familiar with the main models. Some providers provide searchable data bases, others provide spreadsheets which can be down loaded directly and some provide  both. The main data providers for which examples will be given are 1) the World Bank; 2) the Food and Agriculture Organization of the United Nations (FAO); 3) the United Nations Population Division; 4) the US Census International Database, 5) International Aid Transparency, 6) Open Street Map (OSM)/Geofabrik data and 7) Demographic and Health Surveys.

There are two formats of table design which are used by most data providers. Most of the data available for downloading is available in the form of a time series--that is a table with columns identifying characteristics of the variable in a number of columns on the left,followed by a number of columns for individual data by time periods--often by year. The attached table is an excerpt of a World Bank, World Development Indicators, which uses the format-- the range of the time period has been shortened, it is usually 1950 to the most current year.

Country Name Country Code Series Name Series Code 2015 [YR2015] 2016 [YR2016] 2017 [YR2017]
Australia AUS Agricultural land (sq. km) AG.LND.AGRI.K2 3659130 3710780 ..
Azerbaijan AZE Agricultural land (sq. km) AG.LND.AGRI.K2 47698 47730 ..
Bahrain BHR Agricultural land (sq. km) AG.LND.AGRI.K2 86 86 ..
Barbados BRB Agricultural land (sq. km) AG.LND.AGRI.K2 110 100 ..

This format may be changed slightly by adding adding a second classification to the top categories-- for example "urban', "rural" and "total".

The second option is called normalization or normal form. There are different levels of normal form, but the main characteristic is that that for time series data there is a single column for the time data and there would be a separate row for each year. A more complete description of normalization is available at https://en.wikipedia.org/wiki/Database_normalization . A downloaded table in normal form will need to be modified before it can be linked to data from other sites which are typically not in normal form. FAO provides tables both in the time series format and "normalized." An excerpt from a "normalized" FAO table is shown below.

Area Code Area Item Code Item Element Code Element Year Code Year Unit Value Flag
2 Afghanistan 1062 Eggs, hen, in shell 5313 Laying 1961 1961 1000 Head 4000 F
2 Afghanistan 1062 Eggs, hen, in shell 5313 Laying 1962 1962 1000 Head 4400 F
2 Afghanistan 1062 Eggs, hen, in shell 5313 Laying 1963 1963 1000 Head 4600 F
2 Afghanistan 1062 Eggs, hen, in shell 5313 Laying 1964 1964 1000 Head 4800 F
2 Afghanistan 1062 Eggs, hen, in shell 5313 Laying 1965 1965 1000 Head 5200 F
2 Afghanistan 1062 Eggs, hen, in shell 5313 Laying 1966 1966 1000 Head 5500

Examples -- It is not possible to comment on all of the possible models of downloading in use. However, several organizations have been selected to demonstrate the various options.

World Bank: The World Bank has more than 40 databases available on its World DataBank page. Most of these data bases have a  similar interface and produce results in standard formats. Most have interfaces in English, French, Spanish, Arabic and Chinese.  The World Development Indicators (the most used data base) has data on 260 countries and geographic areas for about 1,600 indicators with time series running from 1960 to the present. In addition to simply downloading data,  the data bases often allow the creation of maps, graphics and the calculation of statistics for the data selected. For detailed instructions on downloading the World Bank data bases click here.

FAO:  The main FAOSTAT database was developed in Windows using the Chrome browser. While many of the functions work with other operating systems and browser combinations, for some functions (pivot tables), important data is not loaded. The Windows 10/Firefox combination produces errors, however Linux Mint/Firefox seems to work fine.  It has obviously not been possible to test all the possible combinations. For Windows users it is safest to use the Chrome browser.   For detailed instructions click here.

UN Population Division: The UN Population Division provides for downloading data both as prepared XLS formatted tables or as output from an on-line data base. For data on more than a few countries the prepared XLS tables are the most efficient source, while for five countries or less the data base works well. This data source is restricted to population data and projections (to 2100) at both the national and regional levels. The United Nations Population Information Network provides links to a large number of other population data sources from multiple organizations in database and other formats. For detailed instructions click here.

International Aid Transparency Initiative (IATI)-- IATI is an international collaborative effort to provide an authoritative and transparent source of data on the spending of foreign development aid. More background is available at https://en.wikipedia.org/wiki/International_Aid_Transparency_Initiative . Click here for more detailed information on downloading data.

Open Street Maps (OSM)--Although downloads of specific data from the OSM site are available on many sites, two sites provided free and easy access to the raw data itself. The sites Geofabrik and HOT Export Tool provide different interfaces and emphasis to the downloads. For more information click here.

Demographic and Health Surveys (DHS)-- DHS has carried out demographic and health surveys in almost 100 developing countries. Some are one off, while others are covered every few years. The website provides data in both spreadsheet and GIS data. Click here for more information.

FEWS.NET-- The Famine Early Warning System Network (FEWS.NET) provides reports, spreadsheet data and shapefile data on current food security emergencies. Click here to access the database .

Copyright 1998/2019 GRI Updated 17 March 2019