Who is this research for? Economists, data scientists, policy analysts, and business leaders working with causal analysis and program evaluation.
Top Answer
Research suggests that standard methods for measuring policy or business impacts can give misleading results when important variables shift after a change is introduced. This study presents a new approach that separates the overall impact into what comes directly from the policy and what happens through related changes, helping analysts better understand both the size and the source of an effect.
Executive Summary
This research by Dr. Nicholas L. Brown (Florida State University), Dr. Kyle Butts (Dept. of Economics, Sam M. Walton College of Business, University of Arkansas), and Dr. Joakim Westerlund (Lund University and Deakin University) introduces a new way to measure the impact of policies and business changes using panel data.
Many common approaches assume that outcomes would have followed similar trends without the policy change and that key variables remain unaffected. In practice, those assumptions often do not hold. For example, a policy aimed at improving competition might also change costs or productivity, which then influence the final outcome. This creates a challenge: including those variables can distort results, but ignoring them can also lead to bias.
To address this, the authors developed a method called Treatment and Common Correlated Effects (TCCE). This approach accounts for hidden factors affecting trends over time and allows important variables to respond to the policy itself. It also separates the total impact into two parts: a direct effect (the impact of the policy itself) and an indirect effect (the portion that works through other changing variables).
Using simulations and an example based on China’s entry into the World Trade Organization, the study finds that a substantial share of the overall impact—about half in their application—operates through these indirect channels. This highlights how traditional methods may miss or misinterpret key drivers of change, while the new approach provides a clearer picture of both impact and mechanism.
Expert Insights: What should leaders understand about measuring policy and business impacts?
Why can standard methods mislead decision-makers?
Dr. Kyle Butts notes: “First, policies are often implemented in certain places for certain reasons, e.g. places that are struggling might be supported by interventions. This means that if the policy didn’t get implemented, those places might grow less quickly than other areas. This makes standard methods implausible. Our method allows us to try and figure out what is causing those areas to grow less quickly and adjust for that when evaluating the policy.”
→ Takeaway: Policies often target places already on different trajectories, making it important to account for underlying trends when evaluating results.
What makes treatment-affected variables so challenging to handle?
Dr. Kyle Butts explains: “Treatment-affected variables have to be treated very carefully. Say I was looking at the jobs training program. If I ask ‘what was the effect of the program, after I control for a person’s skills,’ then I would understate the effects of policies. I would, in effect, take away credit from the program for improving people’s skills! The method helps us give proper credit to the program.”
→ Takeaway: Controlling for variables changed by the policy itself can unintentionally hide part of the policy’s true impact.
When should organizations consider using more advanced methods like TCCE?
Dr. Kyle Butts adds: “One advantage of the method is that it is relatively easy to implement! We have open-source software implementing the method to try and make it as accessible as possible! In my research, I always prioritize making software to improve practices; helping others do better work is a multiplier!”
→ Takeaway: More advanced analytical methods are becoming increasingly accessible through open-source tools and easier implementation.
What practical value does this approach offer for decision-makers?
Dr. Kyle Butts notes: “This method allows you to dig deeper than ‘what was the overall effect’ of a policy and try to understand how these effects occur. By better understanding what is driving the change can inform how to improve on the policy design.”
→ Takeaway: Understanding how a policy creates change can help organizations refine and improve future decisions.
Link to the Original Research
Published in Journal of Applied Econometrics (2026)
Frequently Asked Questions
Why are traditional treatment effect methods sometimes unreliable?
Many standard approaches assume that treated and untreated groups would follow similar trends over time. When this assumption fails—especially due to hidden factors or changing conditions—estimates can become biased or misleading.
What are “treatment-affected covariates”?
These are variables that change as a result of the treatment itself. For example, a policy aimed at improving productivity may also affect wages or costs, which then influence the outcome being measured.
What is the difference between direct and indirect treatment effects?
Direct effects capture the impact of the treatment itself, while indirect effects measure how the treatment works through other variables (such as productivity or costs). Together, they make up the total effect.
Why does separating these effects matter?
It helps decision-makers understand not just whether something works, but how it works. This can improve strategy design, policy targeting, and resource allocation.
Where is this method most useful?
It is especially valuable in business and policy settings with complex systems, such as trade policy, pricing strategies, or operational changes, where multiple factors evolve simultaneously.
