This policy note is based on the SSRN paper “Measuring the Cost of Regulation: A Text-Based Approach”. The views expressed are those of the authors and not necessarily those of the institutions the authors are affiliated with.
Abstract
Regulatory costs come from various government bodies, take different forms, are of varying importance, can rise or fall, and may affect firms in the same industry differently. Natural language processing of corporate Earnings Calls offers a new way to overcome these measurement challenges. We use discussions of regulation from Earnings Calls to measure its costs and show that our measure has important consequences for sales growth, asset growth, leverage, expected stock returns, and profits, although the largest effects are on growth. We validate our approach in several ways, including by an event study of the 2016 election. Effects are larger on small firms, which is consistent with economies of scale in managing regulation.
Prior academic literature has produced evidence of substantial costs of regulation for regulated firms. That evidence is based mainly on measures of the physical cost of regulation, such as the salaries of employees charged with ensuring compliance. But regulation does much more than impose physical costs of compliance on current activities; it can prohibit profitable activities from being undertaken, and those opportunity costs are not captured by measures of compliance costs. Also, a shift toward regulatory enforcement that is more permissive (restrictive) about the activities in which firm can engage might have positive (negative) consequences for firms by encouraging (discouraging) growth. Finally, moments of high regulatory risk may affect firms’ growth plans, even before any clear prospects for positive or negative regulatory reform or enforcement changes are apparent.
Further complicating the measurement of regulation is the fact that a firm is regulated by multiple entities: governments of different countries, and within countries, governments of different states. Still another complicating factor is the growing importance of informal regulation – in the US, so-called “guidance,” which is not contained in any formal rule, but rather creates implicit requirements that are not clearly stated as rules but nonetheless are understood by regulated firms as constraints on their operations, which have attendant costs.
How can one construct a measure of regulatory news – from the perspective of each firm – that captures positive or negative news about changes in rules or their enforcement that are relevant for that firm, while also considering regulatory risks? And how can one do so in a way that encompasses changes that originate in many different governments, including both formal and informal rules? And if one can find a way to make a comprehensive list of all those various elements of regulatory news, how does one weight each of them according to its importance, so that one can put all the information together to derive an index of the net amount of positive or negative regulatory news?
It is a daunting task, especially given that news about the same regulation will have different consequences for different firms, even within the same industry. The literature on the political economy of regulation has long argued that large firms have more political power and financial wherewithal to manage the regulatory process, so the same rule applied to a given industry should have less consequences for large firms than for small. And some regulations are inherently firm- or location-specific (e.g., drug approvals, limits on utility fees).
Empirical methods associated with natural language processing (NLP) have produced new approaches for collecting information, and it is natural to apply those methods to measure regulatory costs. For example, George Mason University developed an approach that counts the words of new federal government rules, and associates them with each two-digit industry. That method, however, captures only formal federal laws, not informal guidance, does not distinguish between increases and decreases in regulation, does not gauge the importance of the change, and does not measure any firm-level differences in the relevance of the new rules within the industry.
Figure 1. George Mason and George Washington University regulatory measures
As Figure 1 shows, the aggregation across industries of the George Mason measure trends smoothly over time. The smooth picture of regulatory change provided by this measure stands in sharp contrast to a different measure by George Washington University researchers that captures the number of important new regulations (those that surpass a hurdle of projected cost), which is also shown in Figure 1. For example, according to the George Mason measure, President Trump’s election in 2016 had no discernible effect on regulatory change, while according to the George Washington measure, it was associated with a decline in important new regulations.
In our paper, “Measuring the Cost of Regulation: A Text-Based Approach” (with Ruoke Yang), we develop a new, NLP-based approach to measuring regulatory news. We begin from the premise that one should focus on the individual firms that bear the costs of regulation when measuring regulatory news. Specifically, we devise a means of capturing what their communications tell us is important for them about positive or negative regulatory news. We adopt a simple approach that focuses on discussions of regulation in Earnings Calls, which we validate in a number of ways.
Earnings Calls are the obvious place to find communications about individual public companies’ news of all kinds. Quarterly Earnings Calls are the forum in which firms’ top executives present news about their current condition and prospects to investors (in the Presentation Section of the Earnings Call) and then field investors’ questions (in the Q&A Section of the Earnings Call). Other features of Earnings Calls that make them particularly useful for NLP are their consistent format, their consistent length, and their brevity. Those ensure that the presence of any discussion of a topic in either the Presentation or Q&A Sections is comparably important. The importance of any topic must be weighed before the manager or investor decides to devote scarce time to discussing it. That is not to say that discussions of regulation in Earnings Calls always, or even mainly, reference specific rule changes or particular enforcement process changes. Our reading of many examples of regulatory discussions, perhaps surprisingly, indicates that general discussions of the regulatory climate, which are not focused on any specific rule or enforcement change, are quite common. We take these as an indication that regulatory issues can be “front-of-mind” for executives or investors even when the specific prospects about particular regulations are not discussed in the Earnings Calls.
To identify discussion of regulation, we look for the word “regulation” (or other words with the root “regulat”) in the transcripts of Earnings Calls, and omit mentions where the context clearly refers to engineering uses of the word, as opposed to regulation in the economic sense. Because it is important to distinguish between increasing or decreasing regulation, we identify lists of positive or negative modifier words within the sentence that discusses regulation to indicate whether the mention of regulation is likely associated with an increase or a decrease in the cost of rules or their enforcement. Our lists of increasing or decreasing words were developed by reading a large number of regulatory discussions in Earnings Calls and applying our subjective judgment to determine useful lists of words that typically identify increases or decreases. Because those judgments are subjective, to validate them, we asked OpenAI’s ChatGPT model to read the Earnings Calls to locate mentions of regulation, and in each case, opine on whether the discussion pointed to prospective increases or decreases. ChatGPT validated our judgments; its measures are highly correlated with ours, and when we use its score instead of ours we get obtain similar empirical findings to those reported below.
We analyze the Presentation and Q&A Sections separately because the two discussions differ in structure and in how topics are chosen (in the Presentation Section, by executives as part of a statement of issues of importance, as opposed to in the Q&A Section, by investors, sometimes about issues not raised in the Presentation Section, or about differences of interpretation of what was discussed in the Presentation Section).
For each Presentation and Q&A Section of a firm’s quarterly Earnings Call, we compute a regulatory news score, which consists of the net number of mentions of increasing regulation (the number of times regulation is mentioned in the context of an increasing regulatory intensity word minus the number of times regulation is mentioned in the context of a decreasing regulatory word). Thus, we have a quarterly time series for each firm that measures net mentions of increased regulation. We experimented with more inclusive approaches, such as searching for words like “legal” or “rule,” but these were used too generally outside of the area of regulatory news to be useful. A time series for the average of our measure across firms, for both the Presentation and Q&A Sections of Earnings Calls, is presented in Figure 2.
Figure 2. Time series of regulatory intensity from Presentation and Q&A Sections of Earnings Calls
Note that the measure varies greatly, both positively and negatively, as one would expect of a measure of news. Interestingly, regulatory news in the Presentation and Q&A Sections are not highly correlated (with a correlation of only 0.03), which may indicate that if regulatory news is covered adequately by management it is not typically a subject of questioning by investors. Also note that, similar to the George Washington University measure of regulatory change, Trump’s 2016 election was associated with a large average decline in both measures.
Having identified a measure for each firm of regulatory news over the period 2009-2019, we use that measure to forecast changes in key dependent variables. Our list of dependent variables includes sales growth, asset growth, various measures of profit margin or its change, leverage, and the post-Earnings Call stock market risk premium (which is captured by measuring average abnormal returns after the Call, after removing the effects of the five Fama-French factors and momentum). A focus on a broad range of dependent variables is useful (1) to identify whether the news is associated with changes in near-term operating costs, or with firm growth, (2) to see whether regulation is associated with subsequent changes in firm risk (e.g., increases in risk would cause stock returns to rise and leverage to fall), and (3) as a form of cross-validation (e.g., we expect to see results for sales and asset growth that are similar, and results for leverage and returns that are opposite).
Our sample consists of about 75 thousand firm-quarter observations. In about 28 thousand of those observations, regulation is mentioned. Roughly 10 percent of firms never had a mention of regulation. Our forecasting regressions controls for zero mentions within the Earnings Call by including a special indicator variable for that outcome, and we also include a special indicator variable for firms whose Earnings Calls never mention regulation. It is noteworthy that our measure of regulatory news is not closely related to NLP indicators from prior studies that try to measure firm-level exposures to political or regulatory news.
Most of the variation in regulatory news within the quarter-firm panel is within-firm variation. We perform a variance decomposition of our regulatory news measures, which allows us to identify how much of the total panel variation in those measures reflects average sectoral effects, firm fixed effects, time effects, or the residual (within-firm variation, after controlling for all of those). We perform a Cholesky-type decomposition which assigns to each category of influence a percentage of variance. The results are striking. Close to 90 percent of the variation in our measure of regulatory news in Presentation Sections, and close to 95 percent of its variation in Q&A Sections, reflect within-firm variation in earnings call discussions of regulation, which is unrelated to the variation captured by quarter, industry, or firm fixed effects. Obvious examples of such within-firm variation include approval of new drugs, or anti-trust policies related to mergers. Our variance decomposition results suggest that such idiosyncratic events related to the changes that differ in their relevance across firms are the most common examples of firm-level discussions of regulation.
To build intuition for how our measure captures the regulatory environment faced by firms, we examine the example of Duke Energy, a large utility operating in many regulated markets. Duke devotes a relatively large fraction of its earnings calls, both in the Presentation and the Q&A sections, to discussing its regulatory landscape. Duke mentions regulations in most of its earnings calls, as can be seen from its nearly complete time series of our measure.
The August 7, 2013 Duke Energy earnings call contains the Presentation section with the lowest regulatory news score, indicating a decrease in regulatory intensity, among all Presentation sections with five or more decreasing regulatory modifier words. We expect this call, therefore, to indicate an improving regulatory environment. Indeed, we first hear that “we [i.e., management] expect the second half of the year to be relatively stronger than the first half, primarily as a result of 3 items: First, constructive rate case outcomes.” As a regulated utility, Duke periodically asks its state regulators to approve rate increases, and here expresses satisfaction with the allowed rate increases in this cycle. The company then discusses the regulatory approval of a decision to retire a nuclear power plant called Crystal River 3 on the west coast of Florida. It goes on to say that “2013 is an important year, with a number of regulatory proceedings to position the company for the future. We operate in constructive regulatory jurisdictions and have 5 approved or pending settlements with annual revenue increases of around $600 million. This will result in less regulatory risk to the company, as well as more rate certainty for our customers.” Note the emphasis on the risk aspect of regulation. The company does not mention the physical cost of its regulatory compliance, but rather the lower level of future regulatory risk. It goes on to say that given its “focus on resolution of near-term priorities and constructive regulatory outcomes, we have positioned Duke for low risk, primarily regulated growth through 2015.” Duke is pleased about the low risk of its regulated growth. Finally, the company points out that “[l]ow load growth, new technologies, new regulations and ongoing cost pressures are just some of the forces that require new thinking and action. This includes innovation and technology deployment, continuous improvement [of] regulatory mechanisms.” Again, the management of Duke is focused on, among other matters, evolving regulations and improvement of the regulatory mechanism.
There are two main takeaways from this example. First, our scoring methodology does indeed identify this quarter as a very positive one for Duke Energy from a regulatory perspective. Second, Duke is concerned primarily about the risk associated with its regulations, as well as with the smooth functioning of the regulatory process. Nowhere is there mention of the explicit cost of regulation to the firm or of the resources the firm expends to manage its regulatory environment. Risk and a rational regulatory process are the primary concerns.
Consider, in contrast, regulatory mentions in the Earnings Calls of American Axle, an automotive parts manufacturer, and of AutoNation, a national car dealer. American Axle and AutoNation face lower regulatory scrutiny than does Duke Energy. These two firms discuss regulations on their earnings calls very infrequently. When they do, it is often in response to unusual regulatory developments. In 2018, American Axle is concerned about an unanticipated regulator-mandated electricity rate increase. In 2015, AutoNation discusses the increased “regulatory burden” of additional consumer protection regulations. These examples illustrate the strength of our method: we can identify infrequent, but important, regulatory mentions covering a variety of topics; obtaining similar information from other data sources is nearly impossible.
We find that discussions about regulation by executives and their investors (visible in the transcripts of quarterly Earnings Calls) have important predictive consequences for a wide range of dependent variables. By tracking the implications of regulatory discussions for subsequent profit margins and firm growth we are able to measure the relative importance of growth effects, leverage effects and risk premia (which conceptually should be particularly relevant when news entails either prospective regulation increases or increased regulatory uncertainty) and profit margin effects (which are more related to current changes in actual physical costs of compliance). While discussions of increased regulation are associated both with diminished profitability and with diminished growth, the largest effects of regulation discussions in Earnings Calls are diminished asset and sales growth. Most of our economically and statistically significant results arise in response to discussions of regulation in the Presentation Section. We interpret this as indicating that firms generally raise important regulatory news themselves. However, regulatory news that predicts a significant increase in the risk premium tends to come from the Q&A Section, a result we find plausible: forward-looking discussions of regulation initiated by investors are the ones most relevant for stock pricing.
A standard deviation increase in our measure of regulation in the Presentation Section produces between a one and two percentage point decline in sales or asset growth, on average over the next four quarters, and the effect is much larger for small firms, which apparently incur higher costs of managing regulatory risks. The larger effects for small firms are consistent with the political economy literature, which argues that large firms are better able to manage regulatory costs. We find similar negative effects of increased regulation on firms’ leverage. The decline in leverage appears to reflect the heightened risk associated with discussions of increasing regulation. This increased risk is also visible in higher post-Earnings Call stock returns, which likely result from investors’ beliefs about increased regulatory uncertainty.
One can reasonably question whether our measure of regulatory news is exogenous, or alternatively, might be symptomatic of some other aspect of firm behavior or outcomes. For example, it is conceivable that firms might mention regulations more when they are anticipating bad financial performance, as a way to make excuses for management. To test for this and other endogeneity possibilities, we use a variety of lagged firm performance measures to try to forecast regulatory news. We find that none of those measures predicts regulatory news. Regulatory news appears to be statistically exogenous in the sense that it does not reflect observable changes in measures of firm performance.
One way of validating our measure and its effects on dependent variables is to perform an event study on the 2016 Presidential election. The 2016 election outcome was considered surprising by many observers, and at least one can say that its outcome was highly uncertain. That means that it constituted news that should have led to revised beliefs about government behavior. It is likely that the election produced news that was especially relevant for regulation because, both as a candidate and as President-elect, Trump referred explicitly to his desire to reduce regulatory costs to spur economic growth. If our measure of regulatory news is a useful indicator, it should fall for many firms immediately after the election, and that fall should have important consequences for our dependent variables (post-election growth in sales, assets, profits and leverage, and declines in risk).
In President Trump’s first term, he was not able to do much to repeal existing regulations outright. But his statements, executive actions and appointments could have mattered by reducing new regulations and regulatory enforcement risk. New rules and strict interpretations of existing rules should have become less likely after November 2016. If one were to focus only on the physical costs of compliance arising from regulation, such as the wages paid to the staff that oversees compliance with regulations, it isn’t clear that a Presidential election should have much of an immediate effect on regulatory cost. For example, firms have hired staff to handle compliance, which is overseen by various administrative agencies, whose rules are the result of the cumulative actions of prior legislatures and Presidents. A Presidential election seems unlikely to lead to much of an immediate impact on the size of the firm’s compliance staff.
But Presidents can – either through their Cabinet appointees, or through Executive Orders – make changes in how existing rules are enforced. They can also change the immediate risk profile of regulation by signaling support for loosening rules going forward. Additionally, firms bear the cost of uncertainty about how rules will be enforced or changed. Uncertainty about regulation is a major component of its cost, which can lead firms to delay growth plans. In fact, our research shows that Trump’s first term saw a major reduction in regulatory costs, which was associated with a large boost to growth.
Our approach to investigating how Trump’s election mattered for regulatory costs begins by measuring all firms’ regulatory news in the quarter immediately prior to the election (when Trump’s election was not forecastable, and was viewed by most as unlikely). We investigate whether Trump’s inauguration as President was associated with a diminution of the effect on sales growth that would have happened in the year after Trump took office if he had not been elected. We investigate this, both for the average of sales growth across all firms, and differentially across firms.
To capture firms whose prospects were particularly sensitive to regulatory concerns related to Trump’s election we construct the variable Trump, which measures the stock return of the firm in the three-day window around the November 8, 2016 election. According to this measure, the firms that benefited most from Trump’s election were firms involved in natural resources extraction, for-profit education, and various heavy industries.
We find that, immediately after the election, discussions between firms’ executives and their investors indicated perceptions of reduced regulatory costs going forward. The firms whose stocks benefited more from Trump’s election were also those with greater regulatory intensity in the pre-election period. Furthermore, we find that the negative effect of regulatory news on sales growth from the pre-election period was diminished by Trump’s election. Finally, firms that were the greatest stock market beneficiaries from Trump’s election experienced a bigger post-election reversal in the negative effects associated with their regulatory news intensity in the pre-election period. These findings point toward significant reductions in regulatory risk during Trump’s first term.
In summary, we use Earnings Calls to construct a new measure of regulation and document its effects on firm growth, profitability, leverage, and stock returns. We believe Earnings Calls are an ideal setting in which to measure the impact of regulations on firms, and that this setting is equally relevant for all industries regardless of their specific regulatory bodies. The majority of variation in our regulatory measure is due to idiosyncratic within-firm changes in regulatory exposure, suggesting that our measure captures information that industry- and economy-wide measures cannot reflect.
News of increasing regulation has substantial negative effects on future sales growth, asset growth, and leverage. There is also a negative (but lesser) effect on profit margins. Excess stock returns are substantially higher after earnings calls exhibiting increased regulatory news. This suggests that regulatory exposures identified in earnings calls reflect, at least in part, regulatory risks which drive down contemporaneous stock returns via an increased risk premium.
Effects of regulation are smaller for large firms, indicating substantial economies of scale in managing exposure to regulation. The decreased impact of our regulatory measure on future corporate outcomes after the Trump inauguration for firms with high pre-election regulatory news and strong positive stock reactions to Trump’s election is a form of validation that demonstrates that our measure captures important exogenous variation in firm-level regulatory exposure.
