IV in Exactly Identified Models

Today I'm getting into the weeds a bit about how "finicky" (as I describe it) the IV estimator is. Before describing some of the downside to IV, I think it is worth saying that I think IV gets a very bad wrap. It's a great solution to a very common problem ... under the right assumptions. If those assumptions hold, don't be ashamed to use it. Clearly, that's a big "if" and, as they say, therein lies the rub.

So, one thing that makes IV "finicky" is that it a consistent estimator, but it is not unbiased. I think most people know this (but I do see references to IV producing "unbiased causal effects" far too often). Perhaps less well known is that in exactly identified models the expectation of the estimator does not exist! For those who perhaps don't recall, consistency is an asymptotic property based on taking plims. Bias is a finite sample property based on expectations. So, in finite samples (and I have yet to see an infinite sample in a paper), IV based on an exactly identified model does not even have an expectation. This is a problem. It's hard enough to find one valid instrument, now we need an overidentified model? Yes, if you have a small sample. How small? What does it mean to not have an expectation?

 The expectation does not exist because the distribution of the estimator has such fat tails (going off to +/- infinity) that the expectation is not defined. To show what this looks like in practice and get a sense of how small of a sample is too small, check out my simulation results.

Based on a 250,000 sims with a very strong IV and all assumptions satisfied, can you spot the distribution of IV with N=50? It is essentially the x-axis. Undefined expectation in visual form. It does improve with sample size. OK, so at this point you may be saying who has N=50?

 I'm sure you can find such papers out there, but I'll grant you it may be rare. But, what if the instrument is not that strong? Well, then the problem becomes much worse. The correlation between the covariate and instrument was 0.5 above. What if it is 0.15, but all other assumptions continue to hold? Well, then we get the graph below. Now you can't find the IV distribution even when N=250. And, even at N=1000, that's not a picture that should inspire confidence.

Just how weak is an instrument with a correlation of 0.15? Below is the corresponding distribution of first-stage F-stats. For N=250, the modal F-stat is around 5, but there is a significant mass of F-stats above the usual threshold of 10.

When N=1000, the instrument is usually strong, yet still gives rise to the above distribution. So, if you didn't know, tread lightly when using exactly identified IV models. And that's why no one wants to know how the sausage is made.

Note: Code is available here: http://faculty.smu.edu/millimet/blog.html

Popular posts from this blog

There is Exogeneity, and Then There is Strict Exogeneity

Image
Panel data has become quite abundant. As a result, fixed effects models have become prevalent in applied research. But, this week I handled a paper at one of the journals for which I am on the editorial board and was reminded of a mistake that I have seen all too frequently over the years. Most researchers are probably aware of the issue I am highlighting, but indulge me. In a cross-sectional regression model, we have y_i = a + b*x_i + e_i OLS is unbiased if x is exogenous , which requires that Cov(x_i,e_i) = 0. In other words, the covariates need to be uncorrelated with the error term from the same time period. In contrast, in a panel regression model with individual effects, we have y_it = a_i + b*x_it + e_it If we wish to allow for the possibility that Cov(a_i,x_it) differs from zero, then a_i is a "fixed" effect instead of a "random" effect. To understand what we need to assume to obtain unbiased estimates of b, we need to understand how the model i
Read more

Different, but the Same

Image
To say that difference-in-differences (DID) as a strategy to estimate causal effects has seen a resurgence in the past decade would be an understatement. A search for the term on Google Scholar produces nearly 2,000 hits. In 2020. About 22,000 since 2010. And, to say that I am not much of a fan of DID would also be a bit of an understatement. In case you are curious, I feel this way for two reasons. First, the lack of originality , as suggested by Google Scholar. Admittedly, this is an unfair critique. If it works, then it should be popular. But, I have my suspicions. I also have inner demons that make me detest the popular. I'm sure it has nothing to with my childhood and being, well, less than popular.  Second, most DID papers refer to the policy change or intervention as a  natural experiment . I equally detest that phrase in the majority of its usages. However, if one is going to pursue the DID strategy to examine the causal effect of a policy change or other intervention, then
Read more

Introverts Not Allowed

Image
Introverts are special people. Trust me. I know. I am one. Now, not all introverts are the same. However, one thing that nearly all introverts have in common is the desire to avoid, or at least minimize, interactions. We can only take so much.  I was thinking about interactions this week. Not social ones, but rather econometric ones. Not only did I happen to cover this in class this week, but then I came across this tweet  by @StatModeling. It led to a blog post mentioning a recent publication on Covid-19 vaccine effectiveness. I am not interested in this article per se  for the purposes of this post. However, one of the study's authors posted this as an introduction to the blog post Logistic regression  can mean different things. It can mean a regression of the log odds ratio on covariates using Ordinary Least Squares (OLS). It can also, perhaps, refer to a logit model when the outcome is binary (since the odds ratio is undefined in this case). Since the study above refers to va
Read more