Before You Trust a Stratified Number
Breaking a measure down by language, race and ethnicity, age, or neighborhood is now standard advice in health care quality work, and for good reason. A breakdown, though, is a new number with its own weaknesses. Before anyone acts on a gap between two groups, it is worth asking a plainer question: how good is the demographic field that created those groups?
This explainer covers the checks that come before interpretation. None of them require advanced statistics. They require knowing how the data was collected and setting a few rules in advance.
Why stratification depends on the field
CMS explains that it aims to measure and publicly report health care disparities “in a standardized and systematic way,” and through its stratified reporting it publishes Medicare Advantage measures by race and ethnicity, sex, rural or urban residence, and dual-eligible or low-income subsidy status. AHRQ’s National Healthcare Quality and Disparities Report tools similarly allow comparisons across populations, places, and settings.
The key word in the CMS description is standardized. National reporting can compare groups because the categories are defined consistently. Inside a single clinic or program, that consistency can’t be assumed. It has to be checked.
Check 1: How often is the field blank?
Start with completeness. For each demographic field you plan to use, look at the share of records marked blank, unknown, or declined, and look at it by site, by registration channel, and over time.
Missing values are rarely random. If one site records far more unknowns than another, a difference in outcomes between sites may partly reflect a difference in registration workflow. If online scheduling skips the language question that phone scheduling asks, the online group will look different for reasons unrelated to the people in it.
A useful rule of thumb: report the size of the unknown group alongside every breakdown. If it is larger than the groups being compared, the comparison needs a caveat or a pause.
Check 2: Who chose the category?
Self-reported demographic information is generally more reliable than categories assigned by staff. When staff fill in race, ethnicity, or preferred language based on a name, appearance, or accent, the field reflects the observer as much as the person.
Find out, for each field:
- Whether the person was asked directly, filled it in themselves, or had it entered for them.
- Whether the answer options match what people would choose, including an option to decline.
- Whether the value is ever updated, or carried forward from a record created years ago.
- Whether data from different sources, such as a health plan file and a registration form, are being mixed without a rule for which one wins.
If the collection method varies, consider analyzing only self-reported values, or at least flagging which method produced each record. People should also hear why the information is requested and how it will be protected, and staff benefit from practical guidance on asking respectfully. Both improve completeness over time.
Check 3: Are the categories fit for the question?
Broad categories can hide as much as an overall average. A single category covering many language groups or nationalities may combine people with very different experiences of the service. On the other hand, splitting categories too finely leads straight to small numbers.
Pick categories based on the decision at hand. If the question is whether translated reminders are working, preferred language matters more than any other field, and it should be specific enough to match the languages the reminders are actually sent in.
Check 4: Set small-number rules before you look
A rate based on a handful of people can swing widely from one month to the next. One missed appointment can move it by many percentage points. Small groups also raise privacy risk, because a reader may be able to identify the individuals behind a figure.
Decide in advance, and write down:
- A minimum group size below which a rate is not shown.
- A suppression rule, including suppressing a second cell when the first could be calculated from the total.
- What happens instead. Combine several reporting periods, combine sites, or report the count with a note rather than a percentage.
- Who approves exceptions, so decisions aren’t made one report at a time.
Setting these rules before results are seen matters. Rules invented after a striking number appears tend to bend toward the story the number seems to tell.
Suppression is not a reason to ignore a small group. When a group is too small to report reliably, other sources can fill in: conversations with the people affected, input from staff and navigators who work with them, or a process review with a trusted community partner.
Check 5: Report the data quality with the result
Every stratified table should travel with a short note covering the share unknown, how the field was collected, and which small-number rules were applied. This takes a few lines and prevents a great deal of misreading. It also builds the case for improving collection, because the gaps become visible to the people who can fund and fix them.
A short pre-release list
- Unknown and blank values counted and shown.
- Collection method known for each field.
- Categories chosen for the decision being made.
- Minimum group size and suppression rules applied.
- Data-quality note attached.
A stratified number that passes these checks deserves attention. One that doesn’t may still be worth investigating, but as a question about the data before it is treated as an answer about people.
Sources
- CMS Office of Minority Health: Stratified Reporting
- AHRQ: National Healthcare Quality and Disparities Report Data Tools
About the author
Sherry Sayari, MPH, is an Operational Standards Evaluator who has led population health, community outreach, and social services programs. More on the About page.
