US News & World Report is a popular bi-weekly news magazine with a trusted brand, best known for its rankings of universities, graduate schools and hospitals. In the financially embattled media industry, magazines and newspapers that rank institutions or jurisdictions (e.g., on wait times for hospital services) will sell like cheap lemonade in Central Park in summer. News editors and public relations experts know this. Some think tanks and advocacy organizations understand this, too. Even if your methodology is flawed – especially if it is flawed – you will enjoy attention among policy-makers, the media and the public. In the self-preoccupied media industry, mere news about the allegedly flawed methodology of the rankings used by competitor news organizations receives heaps of news attention.

This brings me to an attention-getting analysis of the US News hospital system, published April 19th in the well-reputed Annals of Internal Medicine, by Ashwini Sehgal. Dr. Sehgal analyzed US News’ measures and “found little relationship between rankings and objective quality measures for most specialties.” He concludes: “users should understand that the relative standings of US News & World Report’s top 50 hospitals largely indicate national reputation, not objective measures of hospital quality.”

Sehgal studied the association between “reputation score” and the total score and the connection of objective measures to “reputation score”. According to Sehgal’s compelling analysis, the statistical association is strong between the total US News score and the reputation score. However, the association between the total US News score and total objective scores is variable, with a minimal link established between reputation score and objective scores. The top five heart and heart-surgery hospitals based on just reputation score were the same as those of the top five heart hospitals (the Cleveland Clinic, Mayo Clinic-Rochester, Johns Hopkins University, Massachusetts General Hospital, and the Texas Heart Institute), and 80 percent of the 20 heart and heart-surgery hospitals with the best reputation scores were also on the US News top-20 heart and heart-surgery centers.

“How can anyone trust US News again?” a performance measurement expert might ask after reviewing the analysis. The answer lies in something called ‘the human condition’. We are all invested in taking reputation seriously, both for our own success and for the organizations with which we’re associated. Consider the US News’ ranking of Universities and Colleges. In 2007, the majority of college presidents in the Annapolis Group, which describes itself as an alliance of the “nation’s leading independent liberal arts colleges,” [emphasis added] said it did not want to participate in the US News reputational rankings. Search the Web sites of most US colleges and universities and you’ll find many touting their “leading” status.

In response, US News issued a communiqué in 2007: “…the reputation of a school can help get that all-important first job and plays a key part in which grad school someone will be able to get into. The peer survey is by nature subjective, but the technique of asking industry leaders to rate their competitors is a commonly accepted practice. The results from the peer survey also can act to level the playing field between private and public colleges.”

Of particular interest to me is my (non-statistical) observation that the reputational scores of business, law, medicine and graduate schools as reported in publications everywhere in the world – i.e., when a school’s reputation gets evaluated anonymously by deans of competitor schools, or by recruiters – are more statistically consistent, year over year, than the variability seen in many other comparatively objective measures (such as publications per capita).

Students who matriculate from Harvard know about the “H-bomb”: merely telling someone you went to Harvard makes you seem instantly more attractive (to potential romantic mates and to potential employers). I find highly educated people are seduced by the H-bomb far more than others, since educational credentials are tools to signal social status, ‘face,’ and reputation.

What is most illuminating about Sehgal’s analysis is how he shows that a hospital’s reputational score is self-reinforcing. Since a hospital’s reputation score in US News is determined by asking roughly 250 specialists to identify the five best hospitals in their specialty, only nationally recognized hospitals are likely to be named frequently. What’s more, a high ranking solidifies the hospital’s gold reputation, making it more likely that surveyed physicians will rate it highly in the future. It’s analogous to asking 250 acclaimed authors to identify their favorite books; books from small publishing houses by first-time authors will not be likely to make the list.

This observation is more salient today since reputation can melt away in 24 hours following a single negative reputational wound – such as a civil suit launched against Goldman Sachs. A ‘blogstorm’ can erupt after ethically questionable corporate emails emerge on the Web. For a company like Goldman, the consequence of abrupt reputational demise is its stock price. In the case of a hospital against which a baseless lawsuit is filed, the consequence can affect the decisions of clients (some may avoid seeking needed care). For this reason, institutions of every type have a responsibility – in the interests of protecting stakeholder interests, or, in the case of a hospital, patient safety – to protect their corporate reputation against slurs. And so, reputation will rise in importance as a measure of value for responsible boards of directors.

Meanwhile, the performance measurement and scientific academy ignores the growing importance of reputation and Web popularity at the cost of losing its own relevance and impact. As of the time of writing, Annals had less than 300 followers on Twitter™; US News had over 7,000. Based on the public’s sense of these sites’ comparative reputation, Annals has a Twitter ‘trust grade’ of 22/100; US News has a ‘trust grade’ of 99.8/100.

Network theory expert Albert-László Barabási demonstrates that if a Web site ekes out a small reputational advantage over its competitors, visitors will post more links to it, which then drives more visitors to the site and it becomes exponentially more popular. A high-reputation site such as usnews.com ranks in the top 1,000 in America. Over 18,000 sites link to the US News site, including all the major global news sites. Meanwhile, the Web site for the “high-impact” Annals of Internal Medicine, where Sehgal’s analysis appeared, ranks 28,810 in the US, with 2,743 sites linking to it. Unless scientists use network theory better to disseminate knowledge, the US News survey will always be more read and more trusted than any criticism of it.

Our instinctive willingness to trust in the value of reputation above all else is hard to fight. This is a product of availability bias, first observed by Amos Tversky and Daniel Kahneman: if something pops into our mind as relevant – a rushed calculation based on reputational judgment for instance – then it becomes ipso facto valuable. Physicians typically make rushed decisions based on reputation. The most data-driven physician, when asked for a referral to another physician or to another health provider, will often make an instinctive decision based on her perception of the reputation of one of the providers in her referral network. Further, surgical decisions over how to treat and when to treat may be based on the perception of legal consequence and on the demands of the patient.

So even if every hospital and clinic in the world were measured objectively based on outcomes measures – such as readmissions rates for selected surgical procedures – this would be a drop in the bucket compared to the multitudes of subjective and reputation-driven decisions that influence patient care every minute and every day.  Even if only objective data were made available to the public, they would still have to be filtered through a third party, such as US News, and their accuracy would still depend on reporters and editors with little or no training in the evaluation of performance measures.

Reputation matters. It will matter vastly more in years to come, as the speed of information and misinformation flows ever faster. Reputation is, as my colleague Carlos Rizo points out, just one of many pieces of information upon which people make health decisions.  Outside of healthcare, we base decisions on everything (from whom to marry to what to feed our children) on reputational factors. In the case of hospitals or colleges, it is better to shine sunlight on our instinctive reputational judgments rather to ignore these judgments – whether or not they match objective outcomes data.

The placebo effect of sending a sick child to a prestigious hospital may well make a demonstrably positive difference to a parent’s health and happiness – in particular, to his or her peace of mind over whether they made the right decision to attend that particular hospital. Meanwhile, the science of measuring reputation – figuring out the amalgam of known and unknown biases that feed into our minute-by-minute decisions – will improve. Humans are complicated. It’s in the interests of public health to try our best to measure what should guide our health decisions – and to measure what actually does.

Comments

Be the first to comment on this!