(Last updated 2020-06-12 10:57:31)
Measures and basic correlations
We have now gathered enough data to start answering the research question you selected at the beginning of the term. You designed a classification system in tutorials, and individually gathered data on lots of countries (103 collected by 79 students to be exact). You will need to use the information in this post (and the next few posts) in the “Data and methods” and “General Results” sections of the essay template - please use this template to write your essay!In this first post, I will tell you a bit about the quality of this data. One way of figuring out if your data is any good is to compare it to professionally collected measures of democracy.
To construct the POLS209 measure of democracy, I simply added your answers to each question for each year and country and rescaled them to the 0-1 range. So each country gets a number for each year from 0 (no democracy) to 1 (full democracy). Sometimes a country was researched by more than one person; I then averaged the scores for each person.
After we do this, I find that a measure of democracy extracted from your answers correlates at 0.84 with the polyarchy index of the V-Dem (“Varieties of Democracy”) dataset (the gold standard in political science), which is really good! (Remember correlation coefficients go from -1 to 1; higher numbers mean greater correlations; a correlation of .8 is huge, and a correlation of 0 means no relationship). Your data also correlates at 0.85 with Freedom House’s numbers, and at 0.85 with Polity IV, two other highly professional datasets used to measure democracy. Well done!
In figure 1 below, you can see the correlations of the POLS209 measure of democracy with several other measures: the Freedom House index of freedom, the Polity IV measure of political institutions, and several of the V-Dem indexes of democracy (
v2x_ variables):
This is not bad at all; as you can see, the correlations between professionally-collected measures of democracy are sometimes a bit higher (often a little above 0.9), but your correlations with these other measures are very high (indeed, higher than the correlation between some of these large-scale indexes and some of the more obscure measures of democracy out there). Professionals do not always agree about whether a country is democratic or not. These results are great!
More specifically, the POLS209 measure of democracy correlates best with “liberal” measures of democracy (the “polyarchy” measure of democracy, Freedom House), and least well with “egalitarian” measures of democracy (V-Dem’s measure of participatory [v2x_partip] and egalitarian democracy [v2x_egaldem]). So it’s clearly measuring something about the standard “liberal” conception of democracy.
Time trends
Most countries were classified by only one student, but some were rated at different time periods by two or more students. Some of you chose longer periods, others chose shorter periods of time, but most chose periods of at least 10 years (median 15 years, min 1 year, max 154), which gives us a reasonable amount of regime data (2509 country-years).The simplest thing to do with this data is simply to plot the results per country, showing the variation over time and space in the POLS209 “democracy score”. In the figure below, the coloured lines represent your measure of democracy, while the light grey broken lines represent the V-Dem “polyarchy” measure of liberal democracy. I’ve split your measure into three categories (democracy, hybrid, non-democracy) for ease of visualization:
As you can see, your measure is very close to V-Dem’s judgments in most cases; mostly you’re a bit more generous, but not much. Some notable disagreements I can see are Jordan (where you claim the country is more democratic than th V-Dem experts think), and Guinea (same). We can be a bit more rigorous, calculating the mean squared error of the difference between the V-Dem score and the POLS209 score. This calculation indicates that the following countries have the largest absolute divergence from V-Dem since the year 2000:
| Country | Mean squared error from V-Dem |
|---|---|
| South Sudan | 0.1538341 |
| Armenia | 0.1408646 |
| United Arab Emirates | 0.1163336 |
| Benin | 0.1110339 |
| Jordan | 0.1009932 |
| Afghanistan | 0.1008580 |
| Zambia | 0.0863890 |
| Montenegro | 0.0822454 |
| Thailand | 0.0807592 |
| Guinea | 0.0802737 |
This doesn’t mean the POLS209 measure is wrong in these cases! Your concept of democracy is different, and judgments about how democratic a country is are fraught with uncertainty. So it may be that your view is correct, and that of V-Dem is wrong.
The raw data (including justifications, by Username and country) for this post is available here. The processed dataset (including the measure of democracy I calculated) is available here. The first file may be useful if you want to peruse the justifications given for particular countries; the second may be useful if you want to do any further analysis (but you can probably ignore it unless you are technically adept).
I will be adding more analysis as the week goes on, but this should get you started.
This is really cool! I was wondering though, on the countries with shorter or no data in grey on the time trend graphs, is this because they have not been observed by people/organisations that observe democracy before? Or are they mainly new states?
ReplyDeleteHi Jean,
DeleteMost of those (e.g., South Sudan) are new states (so not coded before independence). The V-Dem data goes back to until the date of independence or 1900 for most countries - and for a few the countries are coded even if not independent.