DATA & MEASUREMENT

How Will You Know if Things are Improving?

Examine & Analyze Population Data

At the beginning of a project, it is useful to examine the population data that is available to you. Population data refers to information and data about a specific group of people. In this project, all states had access to the following population data:

  • the number of children tested for lead poisoning
  • the average blood lead level (BLL) per month
  • the number of children who tested positive for lead poisoning and received a confirmatory test

Data like this guided teams in identifying the focus areas that needed improvement and helped them prioritize how to invest their time and effort in this improvement work. 

Next, it is important to analyze the available population data to identify more specific areas in which to focus. Then, after there is a successful track record of positive changes that result in improvement for a specific focus area, the next step is to plan how to scale this area of focus into a statewide population success (go to the “Using Data for Improvement” step). 

Pro Tip: Troubleshooting data limitations 

A state may lack the ability to drill down into its health data and may be guided by using hot spot maps that show high density of aging homes, high risk industrial areas, and areas that are underserved by primary care. These hot spot maps are created using available data or even by using subject matter knowledge and experience of public health workers. For example, Missouri identified a rural faith-based area with very low screening rates and confirmatory testing which had large amounts of farm building and older homes containing lead paint. They focused their work on this population as an area to target due to need.

Use Data to Inform Your Team Aim

After examining and analyzing their population data, each state in this CoIIN developed an aim statement or a statement of purpose and answered four questions in developing these statements:

  • When is the project deadline?
  • What does the state want to do?  
  • For whom do you want to make this improvement?
  • And how much improvement, in concrete and measurable terms, do they want to achieve?  

All states were given guidance to follow this format:  

By August 2020, our team will decrease exposure to lead from major sources and/or increase access to systems of care for children ages 0- < 72 months so that they:

  • Decrease by 10% blood lead levels in children ages 0- <72 months.
  • Increase, by 25% or more, the # of children ages 0- <72 months that receive a screening test for blood lead levels.
  • Increase, by 25% or more, the # of children ages 0- <72 months with confirmed elevated blood lead levels who receive care in a medical home.
  • Increase, by 25% or more, the # of providers who are following the CDC recommendations for follow-up of children ages 0- <72 months with confirmed elevated blood lead levels.

Here is an example of how the Alabama CoIIN team customized the aim statement:

  • By August 2020, we want to reduce childhood lead poisoning prevalence by 10% through education and outreach and collaborate effectively with internal partners to provide Alabama’s lead affected children with the most up-to-date and innovative care, increasing care coordination referrals by 25%

A useful improvement aim includes:

  • By when
  • What you want to accomplish
  • For whom
  • How much improvement, in measurable terms that you want to achieve   

Case study: Guided by their aim statement, Alabama made some systemic changes that “hard wired” improvements in their state and led to state-wide changes and improvements. They standardized recommendations for testing and follow up of BLLs. In addition, they updated provider reporting forms to include full address which enabled census tract-level analysis. They ensured that more children were identified early and offered care coordination. This resulted in a new care coordination protocol that refers children 37-48 months of age to their Local Education Agency so that local case managers could automatically refer all children with confirmed elevated BLLs to Early Intervention/Child Find or their local education agency for services. Alabama also noted an increase in confirmatory testing related to improved care coordination.  

Figure 1 shows the increase in confirmatory tests for elevated blood lead levels reported by the Alabama CoIIN team throughout the CoIIN project period.

Use Data for Improvement

Several CoIIN state teams used their population data and other local contextual information to select a population of focus. Louisiana’s aim was to improve lead screening in children ages 0-6 years by 25% within 18 months.

After looking at their state level data and consulting the driver diagram (a guide to making improvements), the Louisiana team decided to leverage partnerships and formed an alliance with a large Women, Infants, and Children (WIC) Center in a New Orleans suburb. Their plan was to meet families there, take lead samples, and talk to families about lead testing and follow up. If this worked, the plan was to spread the changes across the state in all WIC sites.  

Below is their first Plan-Do-Study-Act (PDSA) cycle. Note the size and scope of it, their learning questions, and what they wanted to do next if their plan succeeded.  

During their tests of change, or PDSA cycles, the team learned how to work together with WIC to determine how and when testing will happen, by whom, and what resources were needed.  They developed a detailed plan of action to test children at a high attendance time when parents received WIC vouchers. They also coordinated resources in order to promote the Lead Testing Days. The Louisiana Lead Team distributed lead poisoning prevention educational materials and National Lead Poisoning Prevention Week awareness packets to parents at the Crescent City WIC Clinic and trained families on the importance of childhood lead testing and ways to prevent childhood lead poisoning. WIC Clinic staff were trained on the importance of childhood lead testing and ways to prevent childhood lead poisoning. Lead Team staff created a spreadsheet of all training events that occurred in September and October and events that occurred during National Lead Poisoning Prevention Week. This spreadsheet was a tool that was used to track the following: 

  • Number of people educated
  • Number of educational materials distributed 
  • Number of children tested for lead poisoning during the months of September and October.

Eventually, the Lead team added a screening question about lead testing to the WIC intake form. This is an example of making a lasting system change. 

The Louisiana team wrote, “As a result of the Louisiana Health Homes and Childhood Lead Poisoning Prevention Program (LHHCLPPP) participating in the MCEH CoIIN and through the targeted approaches we have used such as the PDSA Worksheet and the 30-60-90 Day Action Planning Worksheet, we have increased lead testing at one clinic. This has increased our overall lead testing rate and we have tested more children in a shorter period of time. We can see how the change in strategies did drive improvement. We hope that through the lessons learned at this one clinic, we can replicate our strategies at other clinics. Essentially, we hope that this will result in increased lead testing among children ages one and two.”

Important Considerations to Keep in Mind When Working with Data

  • If you only have state-level summary data and you want to drill down to a smaller population of focus, you will need a way to collect data from the narrower population of focus to understand if you are improving. Providers and/or local public health departments can partner with you for data collection. When working with a narrow or small population of focus, you may not see an effect on your overall state data.  
  • When doing an improvement project, data are for learning, not for judgment. While it is tempting to look at data like that displayed above in graph P2 to compare states to each other, remember that each state is its own system. The states have real differences and procedures; comparisons can lead to faulty conclusions. That said, looking at small multiples (data across populations that are measuring the same data point) can spark lively conversations among participating states about how they do their work and why they do it that way, all of which promotes learning. 
  • Some measures have promise as measures for improvement but end up being not very useful. An example from this project can be seen in this measure “The Percentage of Providers Following CDC Guidelines for Lead Poisoning Follow Up.” Note in the figure below the lack of variation in the data. Four of the five teams reporting this measure show 100% each month because follow up providers were state care coordinators trained in the CDC guidelines. When data is 100% month after month with no variation, there is not much to learn from it. A better measure might be looking at rare events, that is failures that happen infrequently. In this case, a more sensitive measure might be the number of follow up contacts associated with a failure to comply with CDC standards. Such a measure would be useful for learning and it is important not to use the data for judgment but rather to guide learning about the sorts of situations that result in a failure. Rare event data like this is harder to collect. A balance is needed between utility of the data, the ability to learn, and the burden of data collection.
Figure 2 shows the percentage of providers that follow CDC recommendations for lead poisoning follow-up among five states from August 2018 – August 2020. The four states that indicate 100% follow-up have providers that are trained in the CDC recommendations.

Tips and Take-aways for Data Collection and Using Data for Improvement

  • Define the population of focus based on your state/jurisdiction’s local and identified need. For example, in the MCEH CoIIN, Illinois and Michigan wanted to focus on women who were pregnant and Iowa focused on pediatric patients at three clinics.
  • Select an outcome for your population of focus. For example, if your population of focus was pediatric patients at a clinic, an outcome measure could be that “85% or more pediatric patients at xyz clinic receive lead testing based on Bright Futures recommendations.”
  • Select key process measures that are sensitive to change and monitor them. For example, % of children caught up on lead testing during sick visits and during well child care visits
  • Use “good enough” data and work with data sources for frequent reporting of data. In this CoIIN, testing laboratories agreed to report more frequent data and more timely data. Provisional data is also useful for monitoring change and can be used as a surrogate while waiting for final data.
  • Review data with your improvement team. Determine whether variation in the data is showing improvement. If the data does not indicate improvement, use this information to generate a series of tests of change to learn if your strategies are working.

Other Quality Improvement and Data Resources: 

CASE STUDIES

Learn from our partners

Our CoIIN members have developed statewide actions and initiatives for lead poisoning prevention and mitigation. Included in these case studies are resources & summaries from the CoIIN state teams and partners.

Browse all case studies

Have a case study to contribute? Click Here

Can’t find what you need?

We have a full index of resources to help you find the information you need.

Browse the resource index