The Great Divide, Part II
My previous post entitled "The Great Divide" has generated a lot of engagement on Twitter and some on Blue Sky. I am very humbled by that. I am humbled that people read any of my blog posts. There has been some push back, some wholehearted agreement, and perhaps some muting of which I am not aware. Nonetheless, given the feedback, I thought I would continue the discussion and (maybe) better frame my thoughts.
Most of the (vocalized) pushback has centered on the steps taken by theorists, particularly since the Credibility Revolution started, to be relevant. Many theorists are attuned to various degrees to applied work. Some theory journals as well as pointed out by Jaap Abbring, the current managing editor of Econometrics Journal, which describes itself as follows on its website:
This is great.
Many theory papers are indeed motivated by concrete examples, contain running examples, and may even include an application. Publicly provided code is not uncommon either. I think this practice is on the rise. I don't dispute these facts at all.
Not only don't I dispute this, I truly applaud the efforts, particularly in making code publicly available. That is an enormous public good. Applied researchers often don't want to share their data even though the data are "done" in order to write the paper. For theorists, there is a difference between code used in a single paper and making code generic for public consumption. So, that is extra work that is not compensated other than -- hopefully -- future citations.
My point is, and where the divide comes in, is that this is not enough. Not even by a long shot.
Let me draw a parallel that may resonate with all of us, applied and theoretical econometricians. In my department, we have one seminar per week, not differentiated by topic. As a communal good, everyone attends when they are available. We support one another, it provides the speaker with a good-sized audience, and, frankly, it's important to see papers outside of one's narrow field. And, when we are hiring, everyone in the department has a vote regardless of field.
As a result, I attend a large number of micro theory talks. Both job talks (last year and in the distant past) and regular seminars. It seems this field faces somewhat similar issues. There is "applied" theory and more esoteric theory. The difference between the two is not so much methodology, skill, etc., as it is just about motivation. Nearly all of the micro theory seminars I have attended spend the first 10-15 minutes motivating the paper with some real world stylized facts or situations, be it targeting advertising or motivating effort by a worker or conveying information to customers about product quality or about the effects of Yelp reviews or about fake news or about political gridlock, etc. etc.
It's brutal. And, in the end, I get very little out of it except feeling like an idiot. But, at least we have happy hour after!
Anyway, this is the state of econometric theory. Yes, it is well motivated. The introduction is even probably interesting, relevant, and comprehensible to nearly all applied researchers. Then Section 2 or 3 arrives. Then there's notation, assumptions about Weiner processes and compact sets and blah blah blah.
Again, not picking on the authors, but I just went to the page of forthcoming articles in Econometrica. Immediately there is one that ought to be of interest to every applied micro researcher, "Bias-Aware Inference in Fuzzy Regression Discontinuity Designs." Sounds important, I better read this and include it in my lecture notes, right? Well, here is the start of Section 4.
This is when even I check out.
Again, I know the authors' work and they are incredibly intelligent people doing important work. My point is not that this type of paper should not be published. My point is that (i) proper motivation of this type of paper, (ii) relegation of proofs, etc., to an appendix, and (iii) even providing code is not sufficient for this to make its way into the applied econometricians toolbox.
My belief is that this is the core reason for the domination of difference-in-differences. And, if we don't fix the system, 100% of papers will be DID in short order. Or at least the toolkit will shrink to 3 or 4 methods. As discussed with someone, perhaps a fair characterization is that applied work has improved over time, but it has also become much more homogeneous.
Instead, the applied toolbox ought to be growing exponentially, yet the opposite is occurring. As I said in the prior post, this is bad, bad science. We are asking for a replication crisis. We are also ignoring questions that don't fit in the DID framework or do but -- gasp! -- don't follow parallel trends.
So, in my view, where the divide comes in is not in what is published, but in what is not. At least not consistently and in outlets commensurate with the importance of such work to ensuring the best possible applied research is being done. Reka Sundaram stated it succinctly and much better than I ever could on Twitter. She framed it as a
"balance between innovation of econometric technique versus innovation in application of existing technique"
The former is healthy and robust. Lots of outlets for work in this vein. Lots of recognition for tenure and promotion. The latter, however, is viewed as a public good rather than serious research and relegated to journals with much smaller reach and much less recognition. This is the gap that must be closed. This work must become much more common, more valued, and published in outlets that encourage such articles to be written without forcing them to have technical appendices with 30 pages of math. Moving the math to an appendix is not the issue; it is the fact that having such math, wherever you put it, is more and more of a requirement to publish even in historically applied journals such as Journal of Applied Econometrics.
As mentioned on Twitter, political science is fantastic at this. Or, at least much better than Economics. Sociology is also pretty good at this. Their flagship journals regularly publish methods papers. No, not always new methods, but "how-to" papers. See, e.g., here, here, here, here, here, here, and here.
Are we really going to be out-done by these other fields?
Or are we too "good" to do this since these "other" fields do this? Me ...
I think the only reason people read my blog posts is because I expose them to interesting econometric methods and insights that they would not get otherwise if it comes with loads of formality. And, by not getting it, applied researchers are not as knowledgeable as they need to be.
For example, I started this blog prior to Covid. One of the earliest posts was on a common error that I have repeatedly come across when refereeing papers using fixed effects: the distinction between exogeneity and strict exogeneity. OK, fine. Important and somewhat subtle point when using fixed effects models. All these years later and this post is the most read post on this blog every week when I don't have a new post. It has been read more than 15,500 times.
I can only imagine it comes up if people google these terms. Or, bots really want to learn about strict exogeneity so they are not lax with it once they become our overlords. Theorists and top journals would roll their collective eyes at a paper trying to make a simple point about this. But, clearly something that theorists would characterize as beyond trivial is a source of confusion for many applied researchers (and referees who don't catch such errors).
As admitted by some on Twitter, when either being a "pure" applied person or putting on one's applied hat for a project, yet choosing to discuss the project with theorists, the conversations are akin to interdisciplinary discussions that most of us frown upon but university administrators repeatedly try to force on us. Theorists want to take the project somewhere not intended, but more importantly, will leave the applied consumer behind. This only encourages a lack of communication.
Some comments on Twitter related to coauthoring. We know about comparative advantage and specialization. I don't dispute that there is a place for this for sure. We should write papers with those with complementary, not redundant, skills. [Although, mostly we should just coauthor with people we like! Hello, my coauthors!] However, this misses the point to some extent. As I tried to illustrate above, turning every paper into a technical theory paper loses part of the audience. As Reka stated, there can be technical papers that innovate on methods, but there needs to be standalone papers that innovate on the application of methods.
My son is a rising college sophomore and a D3 college pitcher. A senior pitcher on the team last year was an Economics and Data Science major. After graduation he moved to Detroit and works with the Detroit Tigers (major league baseball if you're not a fan) in their analytics department. His job is not to do the analytics. His job is not to offer baseball advice. His job is purely to translate the analytics from the computer geeks to the baseball people and translate baseball stuff from the baseball people to the computer geeks.
When will we listen?
If you stuck with me this far, thank you. I'm done.
