Re-examining CMS’ 5-star rating system

In 1998, the Centers for Medicare and Medicaid Services (CMS) began posting information regarding the quality of care provided by nursing homes on it Nursing Home Compare (NHC) website. The purpose of the NHC website was to assist people in choosing a nursing home. A decade later, in 2008, CMS implemented its Five-Star Quality Rating System to provide additional information that enabled consumers to evaluate a specific nursing home and compare its ratings to other nursing facilities.

CMS rates each of the 15,600 nursing homes that participate in the Medicare program with an overall “star rating” as well as ratings in three discrete categories—health inspections (both annual and complaint surveys), quality measures, and staffing. The star ratings are based on a five-star scale with CMS defining a 1-star rating as “much below average,” a 3-star rating as “average” and a 5-star rating as “much above average.”

Although CMS has made a number of changes to the Five-Star System, the long-term care industry—as well as the Government Accountability Office (GAO)—have long expressed concerns regarding the usefulness of the Five-Star System. For example, in a March 2012 report, the GAO recommended that CMS “use strategic planning to establish how its planned efforts will help meet the goals of the Five-Star System, and develop milestones and timelines for each of its planned efforts.”

CMS added quality measures regarding antipsychotic medications and staffing levels to its Five-Star System in February 2015. However, those changes “caused an estimated 4,777 of 15,500 centers nationwide to lose one or more stars in their individual quality rankings even though nothing about the quality of care those centers offered changed,” Greg Crist, Senior Vice President for Public Affairs for the American Health Care Association (AHCA), told Bloomberg BNA in August 2015. If AHCA is correct and the quality remained the same, why indeed, would a facility lose a star-rating?

Calls for improvement

Even with the additional measures that CMS recently added to the Five-Star System, those changes do not “impact the accuracy and reliability of the measures reported,” according to Senator Bob Casey (D-Pa). In a jointly signed letter to the GAO, Senators Casey and Ron Wyden (D-Ore.), both members of the Senate Finance Committee, noted that there are “questions about the [Five-Star] system’s integrity.”[i] Consequently, they requested that the GAO again examine the Five-Star Rating System to address its reliability and accuracy.

GAO recommendations

The GAO report, published in November 2016, found fault with the fact that the NHC website does not include consumer satisfaction survey information, “leaving consumers to make nursing home decisions without this important information.”

The GAO also demonstrated the problems created by the lack of a resident satisfaction tool. It provided the hypothetical example of two nursing homes with the same star rating but vastly different resident recommendation rates. Presumably, the resident satisfaction data would be an important consideration for a family trying to choose a nursing home for a loved one.

What if one nursing home has a 5-star rating with a resident satisfaction of only 45 percent, while another nursing home has a 3-star rating with a resident satisfaction of 85 percent? Would people actively incorporate—or at least consider—the satisfaction rates into their decision-making?

Nikki Clowers is the Managing Director of the GAO Health Care Team that led the research underpinning the GAO report. In an interview, Clowers noted another limitation of the Five-Star System: “Consumers can't use the ratings to compare nursing homes nationally. This is particularly concerning if you live in an area of a state that borders another state, or you have multi-state options. If I lived in Maryland, I may be looking at nursing homes in Maryland, Virginia, West Virginia and Pennsylvania.”

The problem is, ratings in one state don’t necessarily equate with the ratings in another state. “What we [GAO] found is that a nursing home in Pennsylvania with a 4 rating does not necessarily mean that it's of higher quality than a nursing home in Maryland with a 3 rating. This is because the ratings are calculated on a state-by-state basis,” Clowers explained. “When we recalculated the ratings using a national distribution, we found that the ratings for the nursing homes changed significantly in some cases.”[ii]

CMS admits there is variation among states: “There are some differences in how different states carry out the inspection process, even though the standards are the same across the country,” CMS writes on its Strengths and Limitations of the Five-Star Rating System web page. “You should be careful if you're trying to compare a nursing home in one state with a nursing home in another state.”

The GAO also recommends that CMS establish a systematic process for reviewing the NHC website and to improve the Five-Star System’s utility as a tool to assist consumers to understand nursing home quality.

Voices from the field

“I believe the most important of the four recommendations by the GAO is the recommendation to include consumer satisfaction in the rating system,” said Cari Levy, MD, Vice President of AMDA–The Society for Post-Acute and Long-Term Care Medicine. “In a 2014 study of Ohio nursing home residents, many nursing homes that received five stars on the NHC overall rating had very low consumer satisfaction, and many nursing homes with one-star NHC ratings had high to very high consumer satisfaction. Other investigators have similarly found no relationship between family satisfaction and the NHC quality domain.”

Consistency is also a problem, Levy added. “These disparate ratings suggest that consumer ratings are not consistently reflected in NHC ratings and with such heavy reliance on NHC data and inconsistency with consumer satisfaction, attention needs to be turned to the area of patient satisfaction.”

Lynn Fieldhouse, Vice President and General Counsel of Litigation-Risk for Signature Healthcare, a long-term care provider operating in ten different states, echoes Levy's concerns, albeit from a slightly different angle. “The LTC industry creates homes and provides care for others' relatives and loved ones, making customer and family satisfaction every bit as critical in how one should assess or make a decision when choosing a resident center as are quality measures,” Fieldhouse said. “The current five-star system misses this very important consideration.”

Another potential limitation of the Five-Star System—and perhaps the overall survey system—is the surveys that directly impact the star ratings may be comparing apples to oranges. For example, should a 250-bed nursing facility that cares for an average of 35 to 50 ventilator-dependent residents, many of whom have HIV AIDS, require hemodialysis and are medically challenging be compared with and rated on the same scale as an 85-bed nursing facility that does not admit ventilator-dependent residents with such medically complex conditions? Yet, both nursing facilities are compared using the same measuring stick.

“The survey system which directly impacts on a facility’s Five-Star Ratings fails to take into account the varying levels of acuity and size in the nursing home industry,” said Mark Yost, CEO of NMS Healthcare, a multi-site Maryland provider that specializes in pulmonary services and complex wound care. “The system acts like every nursing home is stuck in 1985, and forgets the modern reality of sub-acute care present in today’s system.”

While the Five-Star System has undergone several improvements and iterations, it is far from perfect and is only one among a number of tools consumers can utilize when selecting a nursing home. As CMS continues to refine the Five-Star System and incorporate the recommendations from the GAO and consider the legitimate concerns of the long-term care industry, the Five-Star System will continue to evolve.

At its heart, the Five-Star System relies on statistics. However, one is reminded of the statistician who drowned while crossing a river with an average depth of two-feet. Statistics are only valuable if they are accurate, reliable and reflective of reality.

Alan C. Horowitz

Alan C. Horowitz, Esq., is a partner at Arnall Golden Gregory LLP, where he focuses his legal practice on regulatory compliance for skilled nursing homes, hospices and home health agencies and manages cases where the Centers for Medicare and Medicaid Services (CMS) has imposed an enforcement action. He is a former assistant regional counsel Office of the General Counsel, U.S. Department of Health and Human Services. As counsel to CMS, he was involved with hundreds of enforcement actions and successfully handled appeals before administrative law judges, the HHS Departmental Appeal Board and in federal court. He also has clinical healthcare experience as a registered respiratory therapist and registered nurse. He can be reached at alan.horowitz@agg.com.