Appealing to Large-Language-Model-as-Judge: Annotating and Analyzing the Universe of U.S. Appellate Opinions (with Jonathan Kastellec)

Abstract: We automate annotation of social science texts by manually converting expert codebooks into discrete facts, then automatically extracting those facts using a reproducible, multi-step large language model pipeline. Applying this approach to the U.S. Courts of Appeals, we build a universe-level database of all 440,000 published opinions—twenty times larger than existing hand-coded samples—over 500 times cheaper and 5,000 times faster than hand coding. Our annotations agree with human coders 85–95% of the time on metadata, substantive issues, and litigant types; and 80% on ideological direction (liberal or conservative). Against independent administrative records, they outperform human coders on nearly every available variable, suggesting residual disagreement reflects judgment rather than machine error. Performance depends on our domain-informed task design, not model choice, examples, or fine-tuning. Using our data, we show that partisan polarization is driven largely by dissenting votes and predicts outcomes better than litigant identity.

Download the latest version here.