On Balance: Retrospective Analyses Are Hard: A Cautionary Tale
Under the 1990 Clean Air Act Amendments, the U.S. Environmental Protection Agency (EPA) was required to establish standards limiting air toxics emissions from industrial plants. An article in the latest issue of the Journal of Benefit Cost Analysis (JBCA), “Retrospective Analyses Are Hard: A Cautionary Tale from EPA's Air Toxics Regulations,” takes a retrospective look at 5 of the largest rules issued by EPA in the initial round of air toxics rulemaking over the period 1995 to 2000. For the rules examined, our estimates suggest mixed results, in terms of the reductions in emissions that were achieved. However, our efforts during the project also reinforce the difficulty of obtaining adequate plant emissions data, even where there is an established database--in this case, the Toxic Release Inventory (TRI).
The project reported in this paper is part of the broader Regulatory Performance Project (RPP) launched by Resources for the Future. Since 2014, RPP has funded nine retrospective analyses covering a total of 34 cost-benefit or cost-effectiveness comparisons from a highly diverse set of environmentally oriented rules, in order to compare observed outcomes for these rules with a range of credible baselines and ex ante estimates. A recent article by Richard Morgenstern in the JBCA looks more broadly at these nine studies.
Initially, the ambition of the current project was to cover a large number of the Clean Air Act (CAA) air toxics rules issued during the first phase of the program in the 1990s, to add to our understanding of what technology-based standards actually accomplished. We also hoped that the project would provide lessons on ways to improve ex ante analysis and identify what might be done to facilitate future retrospective analyses. We identified 13 air toxics rules--Maximum Achievable Control Technology (MACT) standards--addressing manufacturing industry categories in the first tier of air toxics rules issued from 1994 to December 1998. However, small sample size and data limitations forced us to drop 8 of the 13 rules from further study. As a result, our analysis focused on EPA’s air toxic rules for the petroleum refining, pharmaceutical, pulp and paper, printing and publishing, and wood furniture industries. These five 5 rules accounted for a large share of the total cost imposed by the air toxics rules issued over the study period.
We used TRI emissions data for those air toxics typically covered by EPA’s Maximum Achievable Control Technology (MACT) rules for the period from 1995 to 2003. Unfortunately, within these 5 industry categories, many of the plants did not report TRI data—and missing data for those plants reporting to TRI further limited our analysis. In fact, because of these issues with the TRI emissions data, our study covered only about half of the plants regulated under the Cluster Rule (EPA’s integrated air and water regulation focusing on toxic releases from pulp and paper mills) and the petroleum refining MACT, and only a substantially smaller proportion of the plants in the other industries.
We used both difference-in-difference and simple ordinary-least-squares (OLS) first difference models to explore the ex post change in air toxics emissions. We adopted two alternative approaches to provide a counterfactual baseline for emissions. For two of the industries—pulp and paper and printing and publishing—we identified similar but unregulated industry categories to provide a baseline. In addition, we compared emissions over the pre- and post-regulation period for all 5 industries with the emissions behavior of the surface coating operations of plants in 6 unregulated industries (like the metal furniture industry). The plants in these “control” industries were eventually regulated by EPA under the MACT provisions in the mid-2000s.
We did not find statistically significant changes in emissions for the petroleum refining, pharmaceutical, and wood furniture industries, suggesting that these industries achieved little or no additional reduction in air toxics emissions over the compliance period in response to EPA’s air toxics rules. The control group industries also made modest reductions over the relevant post-regulation period in response to other CAA-related regulatory actions. In the absence of MACT-regulation, we believe these other CAA-related regulations would have operated to reduce emissions in the MACT industries over the study period.
Our estimates suggest that plants in the printing and publishing and pulp and paper industries realized important reductions in their air toxics emissions in the period between publication of the final rule and the effective date for compliance with the rule—although the reduction in air toxics emissions by pulp and paper mills falls short of EPA’s ex ante projections. For pulp and paper, Gray and Shadbegian (2015) report similar results—that is, some reduction in air toxics, but “...smaller than the ex ante prediction and not always significant.”
Our finding and the similar Gray and Shadbegian results represent an important complement to EPA’s conclusion in its recent retrospective cost study that it overestimated the capital cost of the Pulp and Paper Cluster Rule by 30 percent to 100 percent. We believe several factors may have contributed to this result, including an EPA overestimate of plant emissions at the time it issued its final rule. As noted above, plants in our control groups also made modest reductions over the relevant post-regulation period.
However, there is a “cautionary tale” to these results. The limited number of plants reporting consistently to TRI over the study period limits our confidence in the study results. To facilitate future retrospective studies, EPA should include as a part of its final rules a specific plan of study including a provision for the collection of the necessary data.
References
Gray, W and Shadbegian, RJ. 2015. Multimedia Pollution Regulation and Environmental Performance: EPA’s Cluster Rule, Availible through the member portal here.