After the first several months of the pandemic, decisions were left to the states about the most effective policies to balance health risks. What should be done to monitor and control the spread? Should people be mandated to stay home? Should there be a lockdown of the economy and if so when and how should it be opened? Should the wearing of masks be mandated? Should schools close and if so for how long? How can vaccination levels be increased? What can be done to develop trust with the public and manage misinformation? How to balance the protection of the public health with individual freedom? All of these decisions had tradeoffs with different consequences that in some cases seemed to be contradictory, and there were no obvious answers. COVID-19 put stress on healthcare in every state as they responded to the pandemic differently.
It was, to be sure, a learning experience with different states and their governors and legislators making their own policy choices and employing different strategies with varying degrees of success. And, they are still learning. In a way this “pandemic federalism” provided a laboratory to look back and try to understand what worked and what did not with the purpose of improving pandemic management in the future.
What are the key measures related to the management of the pandemic?
In the initial phases of the pandemic there were extensive efforts to measure the spread and the severity of the disease, which included such things as monitoring infection rates, testing levels, positivity rates, hospitalizations, intensive care usage, mortality rates, and eventually vaccination rates. It was also recognized that COVID-19 could have a devastating impact on the economy, resulting in loss of employment and income, with implications for businesses and individuals. Finally, it became apparent that there were broader societal implications in terms of such things as emotional well-being, food insecurity, educational development, drug usage, and affordable housing.
Accordingly, in order to evaluate state efforts in managing the pandemic, there needed to be a broad definition of “key indicators,” which fall into three interrelated categories:
- Health related indicators, including mortality directly related to COVID-19, measures to monitor the prevalence and severity of the illness (i.e., testing levels, hospital and intensive care usage), “excess mortality” or death rates that exceed historical norms, and vaccination rates.
- Economic indicators, including unemployment rates and changes in Gross Domestic Product (GDP), both of which have an impact on health and well-being.
- Social Determinant of Health (SDOH),  including such things as access to healthcare, economic stability (unemployment, poverty, food security), quality housing, and access to education, all of which were impacted by the COVID-19 pandemic.
Measuring the relative performance of Utah with other states
No comprehensive studies address all of the key indicators mentioned above. However, we evaluated three studies that analyzed some of these key indicators:
- COVID’s Deadly Trade-offs, by the Numbers: How each state has fared in the pandemic. This study was published by Politico in December 2021.
- Final Report Card on the State Response to COVID-19. This study was part of a working paper series that was published for review by the National Bureau of Economic Research (NBER) in April 2022.
- 2022 Scorecard on Health System Performance—How did States do During the COVID-19 Pandemic? This study was published by the Commonwealth Fund in June 2022.
The Politico Study—COVID’s Deadly Trade-offs, by the Numbers
Politico is a political journalism company that covers politics and policy. They “primarily distribute content online but also with printed newspapers, radio and podcasts.” Politico refers to their study as a scorecard with the stated purpose of demonstrating how state decisions impacted lives, jobs, education, and social well-being. The scorecard analyzes data in four areas that include at least one measure in each of the three categories mentioned above.
- Heath performance was measured by studying deaths per capita, hospitalizations per capita, COVID tests completed compared to hospital admissions, and vaccine doses administered per capita.
- Economic performance, was measured by analyzing changes in GDP growth, job creation, and unemployment rates.
- Social well-being indicators included food insecurity, ability to afford household expenses, and violent crime change from 2019 to 2020.
- Educational measures included changes in reading and math achievement,   and in person days of instruction. It should be noted that there were no current federal test scores that reflect performance of all students and all states. Accordingly, private sources were aggregated to create the largest data base possible. The completeness of the data varied by state, which made the education score less reliable.
Summary Results of the Politico Scorecard
The following table summarizes the results for the 10 top performing states according to the Politico analysis. Outcomes of the data analysis generate a score in each area between zero and 100. Scores in each of the categories are then averaged generating a total score which is the basis for the final rankings.
Top Ten Performing States Average Score
Politico State Scorecard
|Rank||State||Overall Average Score||Health Score/Rank||Economy Score/Rank||Social Score/Rank||Education Score/Rank|
Politico Key Indicators in Each Area of Focus
(Out of a potential 100)
|Score Out of 100||Utah Rank||Score Out of 100||Utah Rank|
|Deaths||96||2||State GDP Growth||93||2|
|Vaccination Levels||28||33||Job Creation||99||1|
|Score Out of 100||Utah Rank||Score Out of 100||Utah Rank|
|Violent Crime||24||39||Math Score||23||29|
|Food Insecurity||45||32||Reading Score||67||18|
|Household Expense||72||12||In-person days of Instruction||99||1|
In the Politico analysis, Utah was tied for second of the 50 states with an overall score of 66 out of 100. Utah scored high on the economy, ranking high on all three measures and 2nd overall. For social well-being Utah ranked much lower overall (33rd) with low scores for violent crime and food insecurity and a relatively high score for household expenses. Utah’s health score ranked 15th overall with a score of 61, with very high scores for fewer deaths (95) and fewer hospital admissions (76) but very low for immunizations (28) ranking 33rd. For education, Utah had a score of 61 ranking 18th with the highest score (99) for public school in person instruction, with lower scores for math achievement (23) and reading (16). No state scored high across all four categories and although the overall Utah score of 66 out of 100 ranks high compared to other states, it is not impressive. According to the Politico analysis, every state has the potential for significant improvement, including Utah.
The strength of the Politico analysis was the inclusion of measures for social well-being, which during the course of the pandemic became very important as broader societal implications of COVID became apparent. The challenge is determining which social indicators are most related to the pandemic and could be influenced by state decision makers. The Politico analysis had some weaknesses. The economic data was not adjusted for differences in the economies of different states, and mortality data was not adjusted for age and pre-pandemic diabetes levels, both of which proved to be important factors in the severity of the disease for certain population cohorts. Politico also expanded their analysis of educational impacts beyond days of in-person education to include math and reading performance. This is perhaps an innovative appropriate consideration. However, by Politico’s own admission, the data was not available to support their analysis. For example, there were states with virtually no in-person days of instruction that show no learning loss related to math and education while other states including Utah scored low with high levels of in-person days. The Politico analysis was contradicted by the recently completed National Assessment of Educational Progress (NAEP) produced by the National Center of Educational Statistics (NCES). It is not clear why Politico chose violent crime as an indicator for social well-being because it is not apparent how violent crime was impacted by state decision making. Politico used the experimental Household Pulse Survey conducted by the Census Bureau to measure food security and ability to cover household expenses. The low Utah score for food security raises an important question about how Politico measured relative state performance because in a much more robust study by Feeding America, Utah ranked very high.
NBER Study—The Final Report Card on the State Response to COVID-19
The NBER is a private nonprofit research organization that has the purpose of “undertaking and disseminating unbiased economic research among public policymakers, business professions and the academic community.” As a part of this effort, new research (working papers) by NBER affiliates are circulated for discussion and comment. These papers, which have been produced by professional researchers, have not been peer reviewed. The final Report Card on the State Response to COVID-19 is one of these working papers.
- Economic performance was evaluated using unemployment and GDP by state, adjusted for industry composition. This is because “state populations are heterogenous and their economies emphasize different industries.” The pandemic had a much more negative impact on economic output in some industries (such as entertainment, energy, production, mining, hotels, and food) that made this adjustment essential.
- Educational performance was evaluated using the Burbio cumulative in-person instruction percentage for 2020-2021 school year with hybrid instruction weighted half.
- Mortality included COVID-associated deaths reported to the CDC and all-cause excess mortality. Both measures were adjusted for age and pre-pandemic prevalence of diabetes and obesity, which are highly correlated with higher death rates from the virus.
Summary Results of the NBER Study
The following table summarizes the results for the ten best performing states as well as performance in each category. The overall ranking and combined rankings within each category were calculated by using a Z-score (which shows relative performance compared to the mean). The Z-scores of the three components were equally weighted and transformed to a zero to 100 scale to illustrate relative overall performance. The mean and lowest scores are also included for reference.
The Top Ten Performing States
NBER Working Paper Analysis
|Rank||State||Combined Adjusted Z-Score||Scaled (0-100) Combined Score||Economy Z Score/Rank||Percent Cumulative In-Person Days/Rank||Mortality Z-Score/Rank|
NEBR Key Indicators Detailed
|NBER Key Indicators||Utah||Utah Rank||Best State||Worst State|
|Economy Average Z-Score||1.42||4||1.91||-1.74|
|Percent GDP Growth||0.6%||2||0.8%||-6.4%|
|Percent Cumulative In-Person Days in School||87.3%||5||100%||19.2%|
|Mortality Average Z-Score||0.84||8||0.41||-1.99|
|Total Score Scaled||100||1||100||0|
Utah achieved the top rank overall in the NBER study and received a grade of A+ for overall performance along with two other states. Utah scored relatively high for each key indicator, ranking 4th for economic performance, 5th for health (mortality) and eighth for education. The strength of the NBER study is the sophistication of research techniques and the quality of the data included in the analysis. Their study recognized that state populations are heterogeneous and their economies emphasize different industries that were affected differently by the pandemic. Accordingly, unemployment and GDP data were adjusted for the industry composition of each state. Additionally, because age and pre-pandemic obesity and diabetes are highly correlated with higher COVID death rates the mortality data was adjusted to reflect this difference. These adjustments are important because Utah has a fairly diverse economy and the population is young and healthy. Accordingly, Utah was ranked lower on the economy than in the Politico study (4th vs. 2nd) and significantly lower for mortality (13th vs. 2nd).
The weakness of the NBER study is the limited number of indicators measured. For health, there were two measures, deaths and excess mortality, whereas the Politico analysis used two additional measures and the Commonwealth used six. For education, a single metric was used (in-persons days of instruction), whereas Politico added measures of educational performance. NBER took the position that other respected studies concluded that COVID and school closures had a negative impact on learning and that other measures were unnecessary. For the economy, in addition to unemployment and GDP growth, Politico used data on job creation that were not part of the NBER study.
Commonwealth Fund—2022 Scorecard on Health System Performance: How did States do During the COVID-19 Pandemic?
The Commonwealth Fund is a private foundation whose stated purpose is to “promote a high-performing health care system that achieves better access, improved quality, and greater efficiency, particularly for society’s most vulnerable, including low-income people, the uninsured, and people of color.” The Commonwealth Fund study is unique in that it focuses only on how COVID impacted health system performance and does not address other economic and social issues. This analysis was an expansion of their 2022 scorecard on health system performance that added seven COVID-specific measures in the following areas: Vaccination rates; COVID-related hospitalization rates and health system stress; and COVID-related mortality. These measures were used to construct a composite ranking of state health system performance. The summary analysis of these measures referenced below can be found in the appendices to the study.
- Vaccination performance included two measures: the percentage of adults vaccinated against COVID-19 with a booster and the number of days until 70 percent of the population age 12 and older was fully vaccinated.
- COVID-19 hospitalization rates were based on the hospital admission rate per 100,000 population and were intended to be a measure of the prevalence and severity of the disease.
- Health system stress was based on two measures: the number of days ICUs exceeded 80 percent capacity and the number of days hospitals experienced staffing shortages.
- Excess mortality was measured using two indicators: the number of deaths beyond the norm for the period of the pandemic and deaths from COVID-19 in nursing homes, per 100 beds.
Summary Results—The Commonwealth Fund Study
The following table includes the top 10 performing states in the Commonwealth study.
The following table summarizes how Utah ranked in each of the key indicators that were included in the Commonwealth study, with the addition of drug overdose deaths, which was not included in the COVID-19 dimension ranking but was included in the health lives ranking. This is an important indicator of the social impact of COVID.
Health System Performance
|Key Indicator||Utah Results||Utah Rank||United States Average||Best State Result|
|% Adults vaccinated with booster||38%||22||37%||55%|
|Number of days until 70% of state Population age 12 and over was fully vaccinated||360||24||354||182|
|Number of days with high ICU stress during the pandemic||24||13||112||0|
|Number of days with hospital staffing shortages||0||1||50||0 (16 States)|
|COVID-19 hospital admissions per 100,000 population||1075||12||1,443||564|
|Excess deaths per 100,000 population||210||5||345||110|
|Deaths from COVID-19 in nursing homes per 100 beds||69||8||94||19|
|Drug overdose deaths per 100,000 population||20.5||16||28.3||10.3|
The Commonwealth Fund analysis does include direct measures of health (excess mortality and nursing home deaths) but is primarily a study of preparedness in that it focused on the ability of state health care systems to manage the pandemic. It is not obvious how state level decisions were related to the health system preparedness of the individual states but preparedness[D1] was a critical factor in overall management of the pandemic. The Commonwealth Fund has considerable experience and expertise in the assessment of state level data, and the data used for their rankings is reliable. Utah was ranked 7th overall and was in the top quartile for the other measures with the exception of those related to vaccination levels and drug overdoses. Utah’s high relative performance is most likely more related to the preparation and quality of the health care system itself and not state level decision making.
Aside from the limited scope of the study, there really is no notable weakness in the data analysis; for health-related indicators it is comprehensive and reliable. For indicators that were the same, there was consistency with the other studies except for vaccination levels where the Commonwealth ranking for Utah was higher than the Politico. This is most likely because of different methodologies used by each study.
The following table is a comparative summary of the Utah ranking in each study.
Selective Comparative Rankings for Utah in Each Study
|COMBINED OVERALL RANKING||2 Tied with three states||1|
|Days to 70% Vaccinations||24|
|Days ICU Stress||13|
|Hospital Staffing Shortage||1|
|Nursing Home Deaths||8|
|In-Person Days of Instruction||1||5|
Did state interventions make a difference?
The introduction to this paper noted that decision making for managing the pandemic was left to the states and there were a range of different approaches to the management of the disease. The comparative studies that are summarized in this paper were intended to highlight the relationship between state level decision making and the impact of these decisions on selected indicators of performance. The studies varied considerably, however, in documenting the relationship between state interventions (or lack thereof) and state level performance for the indicators used in their analysis. The following are selected observations by the authors of each of the studies.
- States that imposed more restraints such as stay at home orders and mask mandates had lower rates of deaths and hospital admissions but had poorer economic outcomes.
- States that had fewer shutdowns and more rural geography tended to fare better economically than more urban states that imposed more public health restrictions.
- Isolated states (Hawaii and Alaska) and those with smaller population and geographic footprints fared better.
- States whose economies are heavily dependent on tourism suffered the most economically.
- No state performed well in every policy area.
- Overall, rural states tended to fare better than more urbanized states on economic and educational outcomes.
- Many rural states, despite being less densely populated, ranked poorly in health outcomes.
- Schools which had more days of in-person school had less learning loss in their students.
- Economic and educational performance were impacted by decisions made by policy makers and were positively correlated, (correlation coefficient = 0.43).
- The correlation between health and economic indicators was essentially zero, which suggests that states that withdrew from economic activity did not significantly improve health by doing so. (Hawaii is a notable exception. It ranked last on the economic index and sixth from last on schooling but first on health.)
- There is no clear pattern in which states had high and low mortality. Whether or not political leaders can be considered responsible for mortality outcomes is therefore unclear.
- The economy and education scores were likely influenced by decisions made by policymakers but it is unclear if that is the case for mortality measures.
- School closures may ultimately prove to be the costliest decision of the pandemic era in both economic and mortality terms.
- State decision making did negatively impact deaths in nursing homes in some states.
- There was little health benefit to closing schools.
- The COVID-19 pandemic created considerable stress on healthcare in every state. However, states responded differently with different results.
- COVID-19 vaccines were effective in decreasing transmission of the virus and reducing hospitalizations and deaths for those that were infected with the virus.
- States that were well organized and responded quickly to get shots in the arm had higher vaccination rates.
- All states have made progress in vaccinating their residents against COVID-19 but there are opportunities for improvement.
- COVID-19 pushed hospitals to the breaking point as many operated close to capacity while being understaffed.
- The death toll from COVID-19 was high and extended beyond deaths directly. attributable to the virus; and every state experienced higher than expected mortality from all causes.
- The number of excess deaths varied fivefold across states.
There is some degree of consistency between the three studies where there are common indicators. Where there are inconsistencies it is because of the indicators selected and the quality of data used to measure each indicator. For example, Politico concluded that states’ restrictions and lockdowns improved health outcomes while NBER’s findings were just the opposite. The NBER analysis was supported by statistical analysis. It is not apparent how Politico reached its conclusions. It is clear that state level rankings can be impacted by which indicators are selected and the quality of the data used to support the analysis. Comparative rankings with other states are one way to look at the analysis; the other and perhaps more important is what the data say about potential for improvement. It is important for every state, including Utah, to carefully look at ways to improve the management of future pandemics and even how to increase ongoing public health efforts to improve population health.
How does the Utah experience measure up compared to other states
Utah decision makers were relatively aggressive in eliminating mask mandates, limiting lockdowns of the economy and opening schools. Overall, Utah fared well in each of the three studies, ranking 1st in NBER, tied for 2nd in Politico and 7th in Commonwealth. Utah ranked well for economic performance in both the Politico and NBER studies and high for both mortality and excess mortality indicators in both the NBER and Commonwealth analysis. Utah also ranked high for educational performance in the NBER study and in-person education days in the Politico analysis. In the recent NAEP study Utah’s learning performance was comparable to other states for the 4th grade and was better than other states at the 8th grade level and was the only state that showed no learning loss for math education.
State decision making may have contributed to Utah’s relative performance for both economic and educational measures but was likely not a factor for low mortality and health system performance. Utah’s youngest median age (31.4) and healthy population, combined with the quality of health care in Utah, were most likely positive contributing factors to Utah’s low mortality rate and overall health performance. The quality and performance of the Utah Health Care System was well documented in the Commonwealth study.
Potential for Improvement
The most notable weakness in Utah’s performance related to measures of COVID-19 vaccinations, ranking 33rd in Politico and 22nd and 24th in the Commonwealth study. Vaccination rates were well below top performing states as noted in the following table.
Effort of Top Ten States and Utah
Getting Their Populations Vaccinated
|State||Days Until 70% of population over age 12 Vaccinated|
Utah required 360 days to get 70% of the population over age 12 vaccinated, 178 more days than the top performing state, Vermont, and 102 days more than New York, which ranked tenth. It should be noted that both Maine, and Washington also performed well economically. It should also be noted that Utah’s vaccination rates for childhood infectious diseases appear to be declining and for school year 2020-21 were below the national average and the median for other states. Why does it matter?
Historically the development of safe and effective vaccines against serious and deadly diseases has been “one of the foremost scientific advances of the 21st century”.” Importantly, vaccinations have also proven to be the most significant intervention in getting the pandemic under control. The Commonwealth Fund estimated that “from December 2020 through November 2022 the COVID-19 vaccination program prevented more than 18.5 million additional hospitalizations and 3.2 million additional deaths. Without vaccinations, there would have been nearly 120 million more COVID-19 infections.”
The impact of vaccines in preventing COVID related deaths, is demonstrated in the following table based on data prepared by the Centers for Disease Control.
The Utah Department of Health did an excellent job in producing Utah specific data that illustrates the power of the COVID-19 vaccines in preventing hospitalizations and deaths in Utah during the Omicron period since December 21, 2021.
|Unvaccinated Utahns Had|
|2.1 X greater risk of being hospitalized than those who were fully vaccinated||3.5 X times greater risk of dying than those who were fully vaccinated|
|3.6 X greater risk of being hospitalized than those who were boosted||14 X times great risk of dying than those who were boosted|
We are concerned that vaccine hesitancy in the U.S and in Utah during the pandemic will cause us to forget the significant impact that vaccinations have had on our well-being today and be a negative factor as we continue management of Covid-19 and plan to improve future responses to pandemics.
Conclusions and Recommendations
The overall relative performance of Utah in managing the COVID-19 pandemic was positive in comparison to other states. Economic performance was among the best in the country and was a major focus of decision makers. And, it is important to note that there is a strong relationship between the strength of the economy and several measures of population health. Decisions to open the schools have proven to be well founded and were encouraged by policy makers.
“Health Performance” was relatively high as measured by low mortality and lower than average rates of inpatient hospitalizations. The quality of the Utah Healthcare system was a significant factor in Utah’s performance and decisions by policy makers probably had no positive impact.
Higher vaccination rates would have most likely resulted in lower hospitalization rates and fewer deaths. The broader problem of declining vaccination rates for childhood infectious diseases and the specific challenge of getting the population vaccinated for COVID-19 may be related and should be a significant public policy concern. In spite of the remarkable historical achievements, the pandemic has had the unanticipated effect of discrediting public health in the U.S. In many states, including Utah, the traditional role of public health has been challenged. This may have impacted vaccine hesitancy in Utah.
It is recommended that
a study forum be created consisting of professionals with credible experience
and expertise in preventive health, COVID-19 management and the use of vaccines
to improve population health. The study
should focus on declining immunization levels in Utah, why vaccinations for
COVID-19 were lower than hoped for and how to better define the role of public
health in the future.