Is the Healthcare System Stable?

By Rocco Perla

Twenty years ago, the Dartmouth Atlas provided a first glimpse into the glaring variation in how medical resources are distributed and used in the United States. Why do some primary care physicians order more than twice as many CT scans as their colleagues in the same practice? Why is the cost of a hip replacement three times more expensive in one region in the U.S. than another? Not only did the Atlas signal alarming variation in cost, quality and utilization across the country, it demonstrated the health system was not stable – that is, the cost or quality for a procedure in one health system did not predict the cost or quality in another system. But what specifically is the relationship between stability and variation? And why does it matter?

To know if a system is stable, we first have to understand variation. There is variation in all aspects of our lives: our household expenses, behaviors, stress levels, and commute times. There is variation among institutions. Test scores for students in different schools vary from year-to-year. Crime rates in different communities change from month-to-month. Profit margins differ between companies from quarter-to-quarter. Success rates for the same surgical procedures vary between different hospitals. We constantly make decisions in our daily lives around this variation: Is it time to change schools? Is the crime rate getting worse?  Should I abandon the bus and start taking the train?

In 1931, Walter Shewhart helped us understand that variation can be stable or unstable over time – a clue as to whether we should do something about it. If the variation is similar to past experience, then the system is stable (relatively predictable from point-to-point); if the variation is all over the place and past experience provides no guidance about how the system will perform in the future, then the system is unstable. Unusual patterns of variation merit examination and/or action, while random variation that is similar to what has been experienced in the past likely do not require action. Mistakes – overreacting and under-reacting to data – play out millions of times each day across our knowledge economy, at a significant cost. There are two mistakes we can make around variation in data:

  • Assume the variation is not random and is due to some special cause, when in fact it is random(i.e., part of the normal fluctuation of data over time). Calling a special meeting to discuss why this month’s patient satisfaction score is lower than last month’s score, even though it’s not actually that different from past performance, is an example of this type of mistake.
  • Assume the variation is random, when in fact it is not. For example, one surgeon has an unusually high infection rate this month, but it’s missed because the data are analyzed by aggregating all surgeons together each month which masks the performance of this one surgeon.

Failure to understand variation causes confusion, inhibits learning, produces poorer outcomes, and demoralizes leaders and workforces. And if we do not know if the data are stable over time, we cannot know what to expect in the future. Shewhart didn’t stop with just a theory and a framework, he developed a simple tool— called the control chart—to distinguish between variation that was random (i.e., due to “common causes”) and variation that was non-random (i.e., due to “special causes”). The control chart presents data over time (e.g., week, month or quarter) and includes 3 lines: the center line which is the average of all the data points, and upper and lower limits based on the average and an estimate of the standard error. All three lines come together define the parameters of a process moving forward in time (see Figure 1). By analyzing data over time using this approach, we can assess if the variation is random or not. One sign of non-random variation would be a single data point below or above the limits.

Figure 1 represents a control chart looking at the number of unique patients who screen positive for an unmet social need in seven states across 21 clinical sites. This information is critical for health systems to understand—in the context of health reform—given that 60% of the modifiable factors of health are linked to social determinants. What we learn from this data is that the screen positive rate is stable over the last 28-week period. Based on this data, if conditions remain the same, we can predict the average screen positive rate will continue to be 300, going as low as 205 or as high as 395 – the upper and lower limits. This is the predictable output of the system, which can help determine resource needs and staffing. As tempting as it might be to react to the highest value at week 41, that would be equivalent to explaining why you flipped a coin and got heads – just random variation.

figure1

Today, for all the talk and focus on big-data, data-driven decision-making, and business intelligence, uptake of Shewhart’s framework and tool for understanding variation has not been proportional to its potential for impact. We have spent decades and billions of dollars implementing and evaluating interventions not knowing if our experiments are occurring in stable or unstable systems – and therefore misunderstanding or over/underestimating their impact.  

This one approach to analysis can be transformational. Take the stability of a clinical process in our healthcare system. Until recently, performing coronary artery bypass surgery was an unstable system in which outcomes (including death) varied depending on the surgeon or the hospital where the surgery was performed. But no one knew it. Only by studying variation over time for this procedure and recognizing its association with specific surgeons and hospitals were stable outcomes achieved in states like California and New York – resulting in dramatic decreases in mortality rates to less than 2%. As one researcher recently observed, “It is now almost impossible to identify a surgeon or hospital in either state that is better or worse than other surgeons or hospitals.”

So why hasn’t Shewhart’s approach become a dominant analytic framework in the U.S. and what are the consequences? My colleagues and I attempt to answer this question in “Understanding Variation – 26 Years Later,” a recent article in Quality Progress. In this piece, we demonstrate through a series of examples the distortion that occurs when data (especially those derived from large publicly available sources) are interpreted – and decisions made —without understanding if the outcomes of the system are stable.

For example, every year, the Bureau of Labor Statistics (BLS) analyzes fatal work injury data and reports a color-coded map showing whether a state’s number of fatal work injuries increased, decreased or stayed the same from the previous year. We created the control chart in Figure 2 using BLS data from 1992-2013, but we also added the year-to-year assessment you get from the color-coded map approach that indicates if things are getting better or worse. The control chart shows that from 1992 – 2010 the system was stable – that is, the average fatal work injury rate was 4 per 100,000 people and could fluctuate between a low of 1.7 and a high of 6.4. For 18 years, that system was stable without a single value exceeding the upper limit – until 2011. In 2012, North Dakota officials grew concerned about the increased frequency of fatal injuries, which some attributed to an increased number of workers with riskier jobs in sectors such as the oil industry, as the energy sector grew. Unlike the Shewhart chart in Figure 2, which provides a system view of all data and reveals the upper and lower limits of North Dakota’s fatal injuries over time — the color-coded map limits the analysis to whether conditions are getting better or worse from the prior year. And in 2013 the rate is lower than 2012, but are things really better? That’s what you would think from the year-to-year comparison, but from the control chart we see 2013 still exceeds the upper limit. The system is not stable and we need to know why. What should a worker in North Dakota think, or the family member of a worker who was killed in 2012?

The opportunities to improve how we learn from and act on data using Shewhart’s framework are limitless. For example, the USDA recently reported that food insecurity across U.S. households declined significantly from 14.0% in 2014 to 12.7% in 2015.  Their website states that “the cumulative decline from 2011 (14.9%) to 2014 (14.0%) was statistically significant, and that downward trend continued in 2015.” But the real story is that food insecurity was stable between 1995 – 2007 (first center line = 11.1%), but in 2008 we see an upward shift that stabilizes at rate of 14.5% for seven years (second center line). This means the “new normal” for food insecure households increased by 30.6% in a single year. That cumulative decline from 2011 – 2014 was not a signal of non-random variation in the control chart, and even though it was directionally desirable – nothing was changing fundamentally. It is not until 2015 that we get a data point outside what we would expect – below the lower limit. The greatest learning comes from studying the two non-random patterns we see in the data: the shift that occurs from 2007 to 2008 and the most recent data point in 2015. These are signs that something unusual is occurring and it makes economic sense to invest time and resources to study these periods – beyond any others. What can we learn from the signal in 2015? Are there certain policies or programs that led to this result that we need to keep investing in to get us back to pre-recession levels? These are the questions we must ask.

figure3

An understanding of variation using Shewhart’s framework can unlock some of our most fundamental public policy challenges by enabling data to reveal where we should focus our finite time, dollars, and research capacity. This approach has three levels of impact.

First, it’s a method that people intuitively understand – whether you’re an oil refinery worker in North Dakota, a coronary artery bypass graph patient in California or a hungry family in the U.S., it is easy to visualize a stable process and understand limits.

Second, the framework allows decision-makers and leaders to minimize risk of over and under-reacting to data.

Lastly, this approach can guide the design of systems, which are effectively the accumulated effects of decisions that are made over time.

Some might think that all this is simply an issue of methods but it is not; it is about the way we construct meaning from experience.

Recent Resources

Args

post_type
0post
post_statuspublish
posts_per_page4
meta_query
relationAND
0Array ( )
cat
01
post__not_in
01795
orderby
dateASC
relevanssitrue

Module Settings

post_type_chooseoff|on|off|off|off|off|off|off|off|off|off
loop_stylecustom_loop_layout
shortcode_name[de_loop_template_shortcode]
loop_templatesdivi-blog
custom_loop_templatenone
loop_layout8931
filter_update_animationload-6
no_posts_layoutnone
no_posts_layout_textSorry, No posts.
is_main_loopoff
include_current_termsoff
groupping_post_objectoff
groupping_taxonomynone
show_empty_onloadoff
post_statuspublish
show_current_postoff
posts_number4
post_offset0
post_display_typerelated
saved_typewishlist
acf_linked_acfnone
related_contentcategories
related_acf_fieldnone
related_content_categoriespost_cats
is_category_loopoff
disable_sticky_postsoff
specific_post_objectsoff
related_content_tagsdefault
tax_name_relatednone
acf_name_relatednone
custom_tax_choosepost
acf_namenone
include_sticky_postson
include_sticky_posts_onlyoff
onload_tax_choosepost
sort_orderdate
acf_sort_fieldnone
acf_sort_typestring
acf_date_picker_fieldnone
acf_date_picker_methoddefault
acf_date_picker_custom_day30
order_asc_descASC
sec_acf_sort_fieldnone
sec_acf_sort_typestring
sec_acf_date_picker_fieldnone
sec_order_asc_descASC
enable_loadmoreoff
pagination_positionbottom
scrolltoon
scrollto_fine_tune0px
loadmore_textLoad More
loadmore_text_loadingLoading...
enable_resultcountoff
resultcount_positionright
result_count_single_textShowing the single result
result_count_all_textShowing all %d results
result_count_pagination_textShowing %d-%d of %d results
has_mapoff
map_all_postsoff
map_all_posts_limit-1
map_infoview_layoutnone
map_infoview_layout_ajaxoff
hide_marker_labeloff
map_clusteron
map_center_first_poston
link_whole_girdoff
link_whole_gird_externaloff
external_acfnone
content_section_layoutnone
grid_layoutgrid
columns4
columns_tablet2
columns_mobile1
custom_gutter_widthoff
grid_list_defaultlist
grid_list_positionleft
grid_list_cookie_time30
grid_view_icon||divi||400
list_view_icon||fa||900
icon_padding7px|7px|7px|7px
icon_margin0px|10px|0px|0px
enable_overlayon
show_featured_imageon
show_read_moreoff
read_more_textRead More
show_authoron
show_dateon
date_formatF j, Y
show_categorieson
show_contentoff
excerpt_length270
excerpt_more...
show_commentsoff
meta_separator|
content_visibilityhover
image_hover_animationnone
loop_template_content_alignmentcenter_center
loop_template_color_themelight
loop_template_same_heighton
image_min_height150px
image_max_height500px
pagination_item_background#fff
pagination_item_background_active#ebe9eb
_builder_version4.21.0
_module_presetdefault
grouping_tax_title_font_size14px
grouping_tax_title_letter_spacing0px
grouping_tax_title_line_height1em
title_font_size14px
title_letter_spacing0px
title_line_height1em
excerpt_font_size14px
excerpt_letter_spacing0px
excerpt_line_height1em
loop_template_meta_font_size14px
loop_template_meta_letter_spacing0px
loop_template_meta_line_height1em
loop_template_meta_a_font_size14px
loop_template_meta_a_letter_spacing0px
loop_template_meta_a_line_height1em
loop_template_a_font_size14px
loop_template_a_letter_spacing0px
loop_template_a_line_height1em
pagination_font_letter_spacing0px
active_pagination_letter_spacing0px
background_enable_coloron
use_background_color_gradientoff
background_color_gradient_repeatoff
background_color_gradient_typelinear
background_color_gradient_direction180deg
background_color_gradient_direction_radialcenter
background_color_gradient_stops#2b87da 0%|#29c4a9 100%
background_color_gradient_unit%
background_color_gradient_overlays_imageoff
background_color_gradient_start#2b87da
background_color_gradient_start_position0%
background_color_gradient_end#29c4a9
background_color_gradient_end_position100%
background_enable_imageon
parallaxoff
parallax_methodon
background_sizecover
background_image_widthauto
background_image_heightauto
background_positioncenter
background_horizontal_offset0
background_vertical_offset0
background_repeatno-repeat
background_blendnormal
background_enable_video_mp4on
background_enable_video_webmon
allow_player_pauseoff
background_video_pause_outside_viewporton
background_enable_pattern_styleoff
background_pattern_stylepolka-dots
background_pattern_colorrgba(0,0,0,0.2)
background_pattern_sizeinitial
background_pattern_widthauto
background_pattern_heightauto
background_pattern_repeat_origintop_left
background_pattern_horizontal_offset0
background_pattern_vertical_offset0
background_pattern_repeatrepeat
background_pattern_blend_modenormal
background_enable_mask_styleoff
background_mask_stylelayer-blob
background_mask_color#ffffff
background_mask_aspect_ratiolandscape
background_mask_sizestretch
background_mask_widthauto
background_mask_heightauto
background_mask_positioncenter
background_mask_horizontal_offset0
background_mask_vertical_offset0
background_mask_blend_modenormal
custom_buttonoff
button_text_size18
button_bg_use_color_gradientoff
button_bg_color_gradient_repeatoff
button_bg_color_gradient_typelinear
button_bg_color_gradient_direction180deg
button_bg_color_gradient_direction_radialcenter
button_bg_color_gradient_stops#2b87da 0%|#29c4a9 100%
button_bg_color_gradient_unit%
button_bg_color_gradient_overlays_imageoff
button_bg_color_gradient_start#2b87da
button_bg_color_gradient_start_position0%
button_bg_color_gradient_end#29c4a9
button_bg_color_gradient_end_position100%
button_bg_enable_imageon
button_bg_parallaxoff
button_bg_parallax_methodon
button_bg_sizecover
button_bg_image_widthauto
button_bg_image_heightauto
button_bg_positioncenter
button_bg_horizontal_offset0
button_bg_vertical_offset0
button_bg_repeatno-repeat
button_bg_blendnormal
button_bg_enable_video_mp4on
button_bg_enable_video_webmon
button_bg_allow_player_pauseoff
button_bg_video_pause_outside_viewporton
button_use_iconon
button_icon_placementright
button_on_hoveron
custom_loop_template_buttonoff
loop_template_button_text_size18
loop_template_button_bg_use_color_gradientoff
loop_template_button_bg_color_gradient_repeatoff
loop_template_button_bg_color_gradient_typelinear
loop_template_button_bg_color_gradient_direction180deg
loop_template_button_bg_color_gradient_direction_radialcenter
loop_template_button_bg_color_gradient_stops#2b87da 0%|#29c4a9 100%
loop_template_button_bg_color_gradient_unit%
loop_template_button_bg_color_gradient_overlays_imageoff
loop_template_button_bg_color_gradient_start#2b87da
loop_template_button_bg_color_gradient_start_position0%
loop_template_button_bg_color_gradient_end#29c4a9
loop_template_button_bg_color_gradient_end_position100%
loop_template_button_bg_enable_imageon
loop_template_button_bg_parallaxoff
loop_template_button_bg_parallax_methodon
loop_template_button_bg_sizecover
loop_template_button_bg_image_widthauto
loop_template_button_bg_image_heightauto
loop_template_button_bg_positioncenter
loop_template_button_bg_horizontal_offset0
loop_template_button_bg_vertical_offset0
loop_template_button_bg_repeatno-repeat
loop_template_button_bg_blendnormal
loop_template_button_bg_enable_video_mp4on
loop_template_button_bg_enable_video_webmon
loop_template_button_bg_allow_player_pauseoff
loop_template_button_bg_video_pause_outside_viewporton
loop_template_button_use_iconon
loop_template_button_icon_placementright
loop_template_button_on_hoveron
positioningnone
position_origin_atop_left
position_origin_ftop_left
position_origin_rtop_left
widthauto
max_widthnone
min_heightauto
heightauto
max_heightnone
filter_hue_rotate0deg
filter_saturate100%
filter_brightness100%
filter_contrast100%
filter_invert0%
filter_sepia0%
filter_opacity100%
filter_blur0px
mix_blend_modenormal
animation_stylenone
animation_directioncenter
animation_duration1000ms
animation_delay0ms
animation_intensity_slide50%
animation_intensity_zoom50%
animation_intensity_flip50%
animation_intensity_fold50%
animation_intensity_roll50%
animation_starting_opacity0%
animation_speed_curveease-in-out
animation_repeatonce
hover_transition_duration300ms
hover_transition_delay0ms
hover_transition_speed_curveease
link_option_url_new_windowoff
sticky_positionnone
sticky_offset_top0px
sticky_offset_bottom0px
sticky_limit_topnone
sticky_limit_bottomnone
sticky_offset_surroundingon
sticky_transitionon
motion_trigger_startmiddle
hover_enabled0
grouping_tax_title_text_shadow_stylenone
grouping_tax_title_text_shadow_horizontal_length0em
grouping_tax_title_text_shadow_vertical_length0em
grouping_tax_title_text_shadow_blur_strength0em
grouping_tax_title_text_shadow_colorrgba(0,0,0,0.4)
title_text_shadow_stylenone
title_text_shadow_horizontal_length0em
title_text_shadow_vertical_length0em
title_text_shadow_blur_strength0em
title_text_shadow_colorrgba(0,0,0,0.4)
excerpt_text_shadow_stylenone
excerpt_text_shadow_horizontal_length0em
excerpt_text_shadow_vertical_length0em
excerpt_text_shadow_blur_strength0em
excerpt_text_shadow_colorrgba(0,0,0,0.4)
loop_template_meta_text_shadow_stylenone
loop_template_meta_text_shadow_horizontal_length0em
loop_template_meta_text_shadow_vertical_length0em
loop_template_meta_text_shadow_blur_strength0em
loop_template_meta_text_shadow_colorrgba(0,0,0,0.4)
loop_template_meta_a_text_shadow_stylenone
loop_template_meta_a_text_shadow_horizontal_length0em
loop_template_meta_a_text_shadow_vertical_length0em
loop_template_meta_a_text_shadow_blur_strength0em
loop_template_meta_a_text_shadow_colorrgba(0,0,0,0.4)
loop_template_a_text_shadow_stylenone
loop_template_a_text_shadow_horizontal_length0em
loop_template_a_text_shadow_vertical_length0em
loop_template_a_text_shadow_blur_strength0em
loop_template_a_text_shadow_colorrgba(0,0,0,0.4)
pagination_font_text_shadow_stylenone
pagination_font_text_shadow_horizontal_length0em
pagination_font_text_shadow_vertical_length0em
pagination_font_text_shadow_blur_strength0em
pagination_font_text_shadow_colorrgba(0,0,0,0.4)
active_pagination_text_shadow_stylenone
active_pagination_text_shadow_horizontal_length0em
active_pagination_text_shadow_vertical_length0em
active_pagination_text_shadow_blur_strength0em
active_pagination_text_shadow_colorrgba(0,0,0,0.4)
button_text_shadow_stylenone
button_text_shadow_horizontal_length0em
button_text_shadow_vertical_length0em
button_text_shadow_blur_strength0em
button_text_shadow_colorrgba(0,0,0,0.4)
loop_template_button_text_shadow_stylenone
loop_template_button_text_shadow_horizontal_length0em
loop_template_button_text_shadow_vertical_length0em
loop_template_button_text_shadow_blur_strength0em
loop_template_button_text_shadow_colorrgba(0,0,0,0.4)
box_shadow_stylenone
box_shadow_colorrgba(0,0,0,0.3)
box_shadow_positionouter
box_shadow_style_productnone
box_shadow_color_productrgba(0,0,0,0.3)
box_shadow_position_productouter
box_shadow_style_grid_list_view_buttonnone
box_shadow_color_grid_list_view_buttonrgba(0,0,0,0.3)
box_shadow_position_grid_list_view_buttonouter
box_shadow_style_buttonnone
box_shadow_color_buttonrgba(0,0,0,0.3)
box_shadow_position_buttonouter
box_shadow_style_loop_template_buttonnone
box_shadow_color_loop_template_buttonrgba(0,0,0,0.3)
box_shadow_position_loop_template_buttonouter
text_shadow_stylenone
text_shadow_horizontal_length0em
text_shadow_vertical_length0em
text_shadow_blur_strength0em
text_shadow_colorrgba(0,0,0,0.4)
disabledoff
global_colors_info{}
0.353949069977 seconds