BIG DATA

A sophisticated quantitative analysis of U.S. Supreme Court opinions by University of Virginia School of Law and Dartmouth College researchers is yielding new insights into the influence of clerks and the evolution of justices’ writing styles.

While many believe that Supreme Court clerks are highly influential, others disagree fervently and argue that justices, not clerks, run the show.

“We set out to try to test whether we could find clerk influence,” said study co-author Michael Livermore, an associate professor at U.Va. Law. “We looked at trends in writing style to determine if we could see any differences – focusing particularly on consistency in writing style.”

The analysis of 25,000 Supreme Court opinions shows that as clerks, typically recent law school graduates, took on a more important role on the court in the 20th century, the justices’ individual writing styles became less consistent, presumably because the pool of clerks turns over each year. At the same time, the individual differences between justices diminished, leading to greater institutional writing style consistency on the court.

The research – the first computational analysis of general writing style ever conducted on all Supreme Court cases – also revealed that justices’ opinions are less “friendly” now than in the past, but that they are also easier to understand than their predecessors. The study, “A Quantitative Analysis of Writing Style on the U.S. Supreme Court,” will be published in the Washington University Law Review.

“The opinions of the U.S. Supreme Court can have profound legal consequences, but they are also a major contributor to the genre of English legal writing,” said co-author Dan Rockmore, a professor of mathematics and computer science at Dartmouth who, in an earlier paper, used similar techniques to examine the evolution of writing style in the body of English literature. “In this study, we focus on the court’s role as an author rather than as a legal decision-maker.”

The U.Va.-Dartmouth researchers analyzed the frequency of “function words” – articles, pronouns, conjunctions and other words that are grammatical connectors, like “a,” “the,” “an,” “he,” “she,” “then” and “well” – which has been found to be a useful “stylistic fingerprint” in a host of settings, including for forensic identification of criminal suspects and in determining the authors of the Federalist Papers.

“It would take about 12 years of full-time labor, reading one opinion per hour, to work through the entire body of Supreme Court decisions – a task that could be held to be ‘cruel and unusual’ under the Eighth Amendment if assigned as punishment,” Livermore said. “Advances in mathematics and computer science allow us to approach these massive textual data sets and perceive patters that no human could, or would want to, find on their own.”

The study found that:

• Law clerks have increased the consistency of writing style of the court as an institution, but have reduced the consistency of the individual justices;
• Highly cited justices don’t appear to exert greater stylistic influence on other justices;
• Modern justices write more similarly to their peers than to earlier justices;
• Modern justices tend to produce more words than their predecessors;
• Modern justices’ opinions are grumpier – or much less “friendly” (the percentage of positive versus negative words) – than the opinions of earlier justices; and
• Modern justices’ opinions are written at a lower grade level than the opinions of their predecessors.

Livermore said he first had the idea to examine Supreme Court opinions in such an in-depth manner when he read about Rockmore’s work and got in contact with him in 2009, but the project gained steam recently with help from Jon Ashley, a U.Va. Law business and empirical research librarian.

The project required tagging when each justice’s opinions, dissents and concurrences began in a digital format.  

“While the entire Supreme Court corpus is widely available, finding the markup needed for this project was not,” Ashley said. “Given the number of opinions we were considering, marking these up one-by-one was out of the question, so I started writing scripts in [the programming language] Python to insert tags into the text in the appropriate location and to determine authorship. After several months of testing and tweaking, we had 25,000-plus Supreme Court opinions beautifully marked up and ready for analysis.”

That’s when Rockmore and co-author Keith Carlson, a Dartmouth Ph.D. student in computer science, stepped in to work on the advanced computational analysis.

Livermore said that though many variables influence writing style – for example, the increase in the frequency of negative words could be related to the increase in the number of dissents, or broader social trends in writing style – the team also looked at break periods, when the role of clerks changed, as a point of comparison.

“When we conduct statistical tests on those ‘breaks’ in the timeline, we find that they appear meaningful,” Livermore said. “This gave us some confidence that we were correctly identifying the influence of clerks, rather than some other factor.”

The realization that justices are grumpier in writing was a surprise.

“We didn’t really expect to see a time trend. We did that analysis mostly for fun,” Livermore said. “It was a basic examination of the digitized texts to warm up the computer and get started working with the data.”

Another surprise is that influential justices don’t tend to have influential writing styles.

“We looked at whether the most-cited justices tend to be the most influential in the sense that they have similar writing styles to contemporary justices, and we find that that’s not the case,” he said. “What we primarily find is that justices and their clerks produce opinions that are similar to their contemporaries.”

There’s also not a divide in writing styles between conservative and liberal justices.

“Not finding that is kind of interesting,” he said.

Livermore said that this type of computational textual analysis presents many avenues for future research and study. One follow-up project that Livermore is interested in pursuing is an examination of the effect that law school writing programs have had on legal writing. He’s also working with the Dartmouth team on what subject matter the Supreme Court addresses.

“The possibilities are pretty expansive,” he said.