Surveys, Smurveys: Who do They Think They’re Kidding?

In my statistics class many moons ago, we were taught that as statistical measures, surveys were at the bottom of the heap of ways to establish meaningful evidence.  It was implied that if anyone in the class planned to use surveys in their studies, he or she would be laughed out of the classroom.

So low and behold, all these many moons later, surveys are being promoted as something reliable enough to be used to help determine health care provider reimbursment

Who do the promoters of surveys as reasonable ways to establish patient satisfaction—and then basing physician reimbursement on it—think they are kidding?

Dr. Robert Rowley in “Measuring the Patient Experience in Health Care” tells us that “this kind of measurement…is increasingly becoming important” as payers are trying to find some way to relate payments for providers to patient satisfaction with their care.

Let me be really blunt here.  There are enough “what if’s” here to sink a battleship.

Surveys are based on opinion. No one disputes surveys which measure patient satisfaction about the color of the hospital carpet really don’t tell us anything useful about quality of the care. However, Dr. Rowley says that a “sentinel article in the British Medical Journal in 1999 highlighted the increasing importance of patient surveys, given that sound survey methods now exist, and can guide improvement in healthcare delivery” [my emphasis].

Two issues here.  Dr. Rowley says surveys, now that the measurement techniques have improved, “can guide improvement” in healthcare delivery. Yes, opinions are appropriate for guiding improvements.

However, let’s take a look at the article referenced, “The Increasing Importance of Patient Surveys” by Paul D. Cleary, Department of Health Care Policy, Harvard Medical School. First of all, this article was published in the British Medical Journal in 1999, over ten years ago. In terms of research, this is an aged article. Not quite Rip Van Winkle, but approaching it.

In this article, Paul Cleary admits that surveys were flawed measures of healthcare quality before “rigorous methods” began to be used to gather these opinions.  Rigorous methods are defined as no longer using “clinical conversations” to elicit patient views regarding treatment decisions and quality of care, but instead asking patients questions about their specific experiences with the care offered (rather than, for example, the color of the carpet). With these “improved measures,” Cleary states that “such questions are less subjective and less influenced by patient characteristics, are more interpretable, and thus may be acted on for quality improvement purposes.”

Wait a minute.  A survey, then, is still a survey. That is, opinion is subjective. This change in topic from carpet color to patient care is somehow perceived as less subjective and less influenced by patient opinion. Huh?

As for “more interpretable,” more interpretable than what?  Surveys based on poorer questions?

I have no objections to surveys being used to find out what kinds of things might make a patient experience more pleasant.  Actually, most health care entities wouldn’t need to bother with surveys if they paid any attention to the complaints patients reported.

My problem is with the implication that surveys are somehow a reliable measure of anything statisticians consider meaningful. Surveys are totally dependent upon asking the right question at the right time, and in the right context, but no amount of rightness of the question is going to change the fact that we are dealing with people’s opinions here.  What one person loves another may hate. There’s an underlying assumption here that everyone will recognize quality health care when they experience it.  Anyone who spent any time visiting with physicians would know this simply is not true.

To the credit of Paul Cleary, he does clearly state in his article that from the outset, those in his study agreed “to use the data not to judge ‘winners and losers’ but to educate and inform hospitals and consumers and to focus and facilitate quality improvement efforts.” Paul Cleary made it very plain in his article that the “focus is not on identifying ‘bad apples’ but on stimulating and guiding quality improvement efforts.”

So how, then, do we come to the issue of how surveys, because studies have shown survey designers have learned to better hone their questions, have somehow been magically elevated to highly respectable statistical data—the kind of data one might have some justification for basing provider reimbursement.

Dr. Robert Rowley in his article on measuring patient experience has extrapolated the findings on surveys to conclude “As we move away from a fee-for-service dominated way of paying for health care, and toward one that values performance and outcomes more heavily, using patient surveys is an important domain of measuring such performance.”

Measuring? No.  Sampling what kinds of care people like or dislike.  Sure.  It’s opinion. But measuring, as in basing practitioner reimbursement on the perceptions of patients who may or may not recognize quality health care when they see it—absolute rubbish!





Some young HR person once looked at my CV and asked me, quite seriously, if I had really done everything I had listed there. Well, yes. Because I am someone who can't sit in a Morris Miller cubicle every day, much less for any great stretch of time. Once the problem is solved, I get bored and I'm ready to move on to the next challenge. This hasn't afforded me any great stability in my work life. I simply arrive in places about ten years ahead of time. So far, at least, that penchant for early arrival hasn't been accompanied with a pocketbook full of door knobs.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Show Buttons
Hide Buttons