In recent years, a number of articles in the Journal of Benefit Cost Analysis have addressed the question of how best to conduct—and evaluate—a regulatory benefit-cost analysis. Two of these subsequently resulted in posts to On Balance: Consumers Guide to Regulatory Impact Analysis, by Susan Dudley, and Two Decades of Benefits and Costs: Promise and Pitfalls, by Clark Nardinelli. Another article by Richard Morganstern, Retrospective Analysis of U.S. Federal Environmental Regulation, examines various estimates for environmental rules and analyzes issues relevant to developing credible baselines.

These articles remind us that baselines in a regulatory impact analysis can be slippery: in estimating benefits and costs the comparison is not between before-and-after regulation but between a counterfactual with-and-without regulation. In turn, the “counterfactual” should take into account the effect of other regulations, the evolution of the market in the absence of regulation, and other external factors. In short: developing this counterfactual requires understanding how the baseline has been changing and how it is likely to change.

We do our best to develop reasonable counterfactuals but it is not easy. Two examples from my experience illustrate how hard it can be. I present them here for comments and responses to open questions about our procedures.

My first example is what we (in Canada) call “Tier3” standards, which include lower limits on the sulphur content of gasoline and stricter limits on air pollutant emissions from new passenger cars, light-duty trucks and certain heavy-duty vehicles. These regulations, published in 2014, were accompanied by a Regulatory Impact Analysis. The analysis employed a sophisticated, multi-stage approach to estimating changes in emissions, their impacts on air quality, and the effects and value of changes in air quality on the health and environment of Canadians.

Given the analytical complexity, we did not fully model either the base or the policy case. Instead, we modelled the annual emission trends, then picked two representative years (2020 and 2030) and completed the full multi-stage air quality and benefits modelling for those years. Intervening years were interpolated linearly. Admittedly, this is a somewhat heroic assumption, as few natural processes move in a straight line and changes in air emissions may have counter-intuitive results for changes in ambient air quality and health outcomes.

Is it better to oversimplify or to avoid representing how the world is changing? Different agencies have answered this question differently; how would you?

My second example involves regulations designed to reduce greenhouse gas (GHG) emissions from on-road vehicles: both “light-duty” vehicles such as passenger cars, pick-up trucks, and vans; and “heavy-duty” vehicles such as heavier cars and trucks, commercial vans, buses, and tractor-trailers. The most recent version, Regulations Amending the Heavy-duty Vehicle and Engine Greenhouse Gas Emission Regulations, were published in May 2018, also accompanied by a Regulatory Impact Analysis. Baseline emissions in this analysis were taken from the most recent ECCC projected GHG emissions given existing policies and trends, and incorporated the latest available set of economic and energy projections. This baseline assumed, for the most part, that vehicle technologies affecting GHG emissions will remain essentially unchanged over the analytical time frame, in the absence of new regulations. Thus, emissions in the baseline may not fully reflect changes in emissions that might occur due to technological changes that could occur in the market in the absence of any regulations. We find that net benefits for Canadians under the proposed policy are positive, although we state that net benefits might be lower than stated under some forms of technological change.

Projecting how potential technology change and market forces might affect emissions over time, in the absence of new regulation, is difficult to do. Thus, it is equally difficult to determine whether the omission of technological change might cause us to over- or under-estimate the true cost-effectiveness of the policy. Moreover, policymakers and stakeholders care about the scale of net impacts, so our reassurances of net benefits under any baseline estimate of natural technology adoption may not be fully satisfactory to those who are looking for a compelling result to justify regulatory intervention.

Are these reasons enough to abandon our assumption that technology will not change (much) in the baseline? Forecasting technology changes and costs is difficult and we feel this assumption is transparent enough and easily understood by policymakers; what do you think?

I ask readers to consider my examples and feel free to provide some feedback or further insights because I believe that the only way to avoid pitfalls is constant vigilance; an ongoing effort is necessary if we are to serve the policymakers and stakeholders who rely on our regulatory analyses. You can contact me directly at [email protected]