IV in Exactly Identified Models
Today I'm getting into the weeds a bit about how "finicky" (as I describe it) the IV estimator is. Before describing some of the downside to IV, I think it is worth saying that I think IV gets a very bad wrap. It's a great solution to a very common problem ... under the right assumptions. If those assumptions hold, don't be ashamed to use it. Clearly, that's a big "if" and, as they say, therein lies the rub.
So, one thing that makes IV "finicky" is that it a consistent estimator, but it is not unbiased. I think most people know this (but I do see references to IV producing "unbiased causal effects" far too often). Perhaps less well known is that in exactly identified models the expectation of the estimator does not exist! For those who perhaps don't recall, consistency is an asymptotic property based on taking plims. Bias is a finite sample property based on expectations. So, in finite samples (and I have yet to see an infinite sample in a paper), IV based on an exactly identified model does not even have an expectation. This is a problem. It's hard enough to find one valid instrument, now we need an overidentified model? Yes, if you have a small sample. How small? What does it mean to not have an expectation?
The expectation does not exist because the distribution of the estimator has such fat tails (going off to +/- infinity) that the expectation is not defined. To show what this looks like in practice and get a sense of how small of a sample is too small, check out my simulation results.
Based on a 250,000 sims with a very strong IV and all assumptions satisfied, can you spot the distribution of IV with N=50? It is essentially the x-axis. Undefined expectation in visual form. It does improve with sample size. OK, so at this point you may be saying who has N=50?
I'm sure you can find such papers out there, but I'll grant you it may be rare. But, what if the instrument is not that strong? Well, then the problem becomes much worse. The correlation between the covariate and instrument was 0.5 above. What if it is 0.15, but all other assumptions continue to hold? Well, then we get the graph below. Now you can't find the IV distribution even when N=250. And, even at N=1000, that's not a picture that should inspire confidence.
Just how weak is an instrument with a correlation of 0.15? Below is the corresponding distribution of first-stage F-stats. For N=250, the modal F-stat is around 5, but there is a significant mass of F-stats above the usual threshold of 10.
When N=1000, the instrument is usually strong, yet still gives rise to the above distribution. So, if you didn't know, tread lightly when using exactly identified IV models. And that's why no one wants to know how the sausage is made.
Note: Code is available here: http://faculty.smu.edu/millimet/blog.html
So, one thing that makes IV "finicky" is that it a consistent estimator, but it is not unbiased. I think most people know this (but I do see references to IV producing "unbiased causal effects" far too often). Perhaps less well known is that in exactly identified models the expectation of the estimator does not exist! For those who perhaps don't recall, consistency is an asymptotic property based on taking plims. Bias is a finite sample property based on expectations. So, in finite samples (and I have yet to see an infinite sample in a paper), IV based on an exactly identified model does not even have an expectation. This is a problem. It's hard enough to find one valid instrument, now we need an overidentified model? Yes, if you have a small sample. How small? What does it mean to not have an expectation?
The expectation does not exist because the distribution of the estimator has such fat tails (going off to +/- infinity) that the expectation is not defined. To show what this looks like in practice and get a sense of how small of a sample is too small, check out my simulation results.
Based on a 250,000 sims with a very strong IV and all assumptions satisfied, can you spot the distribution of IV with N=50? It is essentially the x-axis. Undefined expectation in visual form. It does improve with sample size. OK, so at this point you may be saying who has N=50?
I'm sure you can find such papers out there, but I'll grant you it may be rare. But, what if the instrument is not that strong? Well, then the problem becomes much worse. The correlation between the covariate and instrument was 0.5 above. What if it is 0.15, but all other assumptions continue to hold? Well, then we get the graph below. Now you can't find the IV distribution even when N=250. And, even at N=1000, that's not a picture that should inspire confidence.
Just how weak is an instrument with a correlation of 0.15? Below is the corresponding distribution of first-stage F-stats. For N=250, the modal F-stat is around 5, but there is a significant mass of F-stats above the usual threshold of 10.
When N=1000, the instrument is usually strong, yet still gives rise to the above distribution. So, if you didn't know, tread lightly when using exactly identified IV models. And that's why no one wants to know how the sausage is made.
Note: Code is available here: http://faculty.smu.edu/millimet/blog.html
