There’s a lot of talk about quality metrics, pay for performance, value-based care and penalties for poor outcomes.
In this regard, it’s useful to ask a basic question. What is quality? Or an even simpler question, who is the better physician?
Let’s consider two fictional radiologists: Dr. Singh and Dr. Jha.
Dr. Singh is a fast reader. Her turn-around time for reports averages 15 minutes. Her reports are brief with a paucity of differential diagnoses. The language in her reports is decisive and her reports contain very few disclaimers. She has a high specificity meaning that when she flags pathology it is very likely to be present.
The problem is her sensitivity. She is known to miss subtle features of pathology.
There’s another problem. Sometimes when reading her reports one isn’t reassured that she has looked at every organ. For example, her report of a CAT scan of the abdomen once stated that “there is no appendicitis. Normal CT.” The referring physician called her wondering if she had looked at the pancreas, since he was really worried about pancreatitis not appendicitis. Dr. Singh had, but had not bothered to enlist all normal organs in the report.
Dr. Jha is not as fast a reader as Dr. Singh. His turn-around time for reports averages 45 minutes. His reports are long and verbose. He meticulously lists all organs. For example, when reporting a CAT of the abdomen of a male, he routinely mentions that “there is no gross abnormalities in the seminal vesicles and prostate,” regardless of whether pathology is suspected or absence of pathology in those organs is of clinical relevance.
He presents long list of possibilities, explaining why he thinks a diagnosis is or is not. He rarely comes down on a specific diagnosis.
Dr. Jha almost never misses pathology. He picks up tiny lung cancers, subtle thyroid cancers and tiny bleeds in the brain. He has a very high sensitivity. This means that when he calls a study normal, and he very rarely does, you can be certain that the study is normal.
The problem with Dr. Jha is specificity. He often raises false alarms such as “questionable pneumonia,” “possible early appendicitis” and “subtle high density in the brain, small punctate hemorrhage not entirely excluded.”
In fact, his colleagues have jokingly named a scan that he recommends as “The Jha Scan Redemption.” These almost always turn out to be normal.
Which radiologist is of higher quality, Dr. Singh or Dr. Jha?
If you were a patient who would you prefer read your scan, the under calling, decisive Dr. Singh or the over calling, painfully cautious Dr. Jha?
If you were a referring physician which report would you value more, the brief report with decisive language and a paucity of differential diagnoses or the lengthy verbose report with long lists on the differential?
If you were the payer which radiologist would you wish the hospital employed, the one who recommended fewer studies or the one who recommended more studies?
If you were a hospital administrator which radiologist would you award a higher bonus, the fast reading Singh or the slow reading Jha? This is not a slam dunk answer because the slow-reading over caller generates more billable studies.
If you were hospital’s Quality and Safety officer or from Risk Management, who would you lose more sleep over, Dr. Singh’s occasional false negatives or Dr. Jha’s frequent false positives? Note, it takes far fewer false negatives to trigger a lawsuit than false positives.
I suppose you would like hard numbers to make an “informed” decision. Let me throw this one to you.
For every 10, 000 chest x-rays Dr. Singh reads, she misses one lung cancer. Dr. Jha does not miss a single lung cancer, but he recommends 200 CAT scans of the chest for “questionable nodule” per 10, 000 chest x-rays. That is 200 more than Dr. Singh. And 199/ 200 of these scans are normal.
I can hear the siren song of an objection. Why can’t a physician have the sensitivity of Dr. Jha and the specificity of Dr. Singh? The caution of Jha and the speed of Singh? The decisiveness of Singh and the comprehensiveness of Jha?
You think I’m committing a bifurcation fallacy by enforcing a false dichotomy. Can’t we have our specificity and eat it?
Sadly, I’m not. It is a known fact of signal theory that no matter how good one is, there is a trade-off between sensitivity and specificity. Meaning if you want fewer false negatives, e.g. fewer missed cancers on chest X-ray, there will be more false positives, i.e. negative CAT scans for questioned findings on chest X-ay.
Trade-off is a fact of life. Yes, I know it’s very un-American to acknowledge trade-offs. And I respect the sentiment. The country did, after all, send many men to the moon.
Nevertheless, whether we like it or not trade-offs exist. And no more so than in the components that make up the amorphous terms “quality” and “value.”
Missing cancer on a chest x-ray is poor quality (missed diagnosis). Over calling a cancer on a chest x-ray which turns out to be nothing is poor quality (waste). But now you must decide which is poorer. Missed diagnosis or waste? And by how much is one poorer than the other.
That’s a trade-off. Because if you want to approach zero misses there will be more waste. And if we don’t put our cards on the table, “quality” and “value” will just be meaningless magic talk. There, I just gave Hollywood an idea for the next Shrek, in which he breaks the iron triangle of quality, access and costs and rescues US healthcare.
If I had a missed cancer on a chest x-ray I would have wanted Dr. Jha to have read my chest x-ray. If I had no cancer then I would have wanted Dr. Singh to have read my chest x-ray. Notice the conditional tense. Conditional on knowing the outcome.
In hindsight, we all know what we want. Hindsight is just useless mental posturing. The tough proposition is putting your money where your mouth is before the event. Before you know what will happen.
This is the ex-ante ex-post dilemma. In case you want a clever term for what is patently common sense.
Dr. Singh is admired until she misses a subtle cancer on a chest x-ray. Then Risk Management is all over her case wondering why? How? What systems must we change? What guidelines must we incorporate?
Really? Must you ask?
Dr. Jha, on the other hand, is insidiously despised and ridiculed by everyone. All who remain unaware that he is merely a product of the zero risk culture in the bosom of which all secretly wish to hide.
The trouble with quality is not just that it is nebulous in definition and protean in scope. It can mean whatever you want it to mean on a Friday. It is that it comprises elements that are inherently contradictory.
Society, whatever that means these days, must decide what it values, what it values more and how much of what it values less is it willing to forfeit to attain what it values more.
Before you start paying physicians for performance and docking them for quality can we be precise about what these terms mean, please?
Thank you.
Categories: Uncategorized
This is a fantastic article outlining the dilemma radiologist face in clinical practice. We cannot be perfect and have 100% sensitivity and 100% specificity. I have seen radiologists in both the Singh and Jha schools of diagnostic radiology. I personally try to strike a balance between the two schools for my practice but naturally I am still inclined to be in the Singh side because I feel that I am helping the patients and giving a more specific and definitive answer to referring clinicians. I have seen some radiologists in the Jha school of radiology overcalling everything you can possibly think of creating unnecessary extra investigations, biopsy, anxiety and taxpayers money. That makes me sometimes wonder what the role of radiologist is if our job is to be like Dr Jha. Unfortuantely the legal system expects us to be 100% sensitive disregarding the importance of specificity, thus encouraging radiologists to follow the Dr Jha’s school and generate reports which won’t get them in trouble but may be entirely useless in guiding referring clinicians in clinical management. That is very sad.
These are some awesome tips to find the best radiologist. Thanks for sharing this information. We provide back surgery treatment and care in West Palm Beach. You can contact us for assistance.
Excellent article. The diagnostic imaging for a multitude of problems has exploded. Insecure primary care providers, including nurse practioners, order studies with reckless abandon. The vast majority of MRI scans for low back pain are useless or harmful. Over-reads on disc changes promote medicalization of routine problems. In my area, Dr. Jha easily does more harm than good. Incidentalomas are an epidemic. Get out of the darkroom and look at the whole clinical picture. The priority should be on patient care. I know it is a complex issue, but let’s keep quality patient care on top.
So far I have seen many thoughtful comments and suggestions. However, the answer is in plain sight…..BOTH and ALL…get paid less. The fight is all about how much less, and for which one, and by whom?
“Sometimes in error, but never in doubt”
That’s absolutely correct Legacy. I’ve found that clinicians favour certainty and consistency over doubt and accuracy.
Which means that radiologists might have to internalize some risk in order to be considered useful.
Legacyflyer signing in. My signal is strong but my content is weak ….. or something like that.
Dr. Jha is absolutely correct and he has explained this concept well.
When I was in training I heard the following two statements which have had a great impact on me:
“Don’t be a ditzel caller!” From a professor I admired greatly (John Diaconis -and if you knew him, you knew that most everything he said had an exclamation point behind it.) The point of his statement is that there are many ill defined gray (not black and white) object on an image. If you call them all, you can send your clinical colleagues running in 20 different directions.
“Sometimes in error, but never in doubt”. On the face of it, this seems foolish, but in reality is not. It is possible to become so paralyzed by doubt that one endlessly equivocates and become useless. Depending on the sensitivity and specificity, it is sometimes better to miss a few things, but send your referring docs on a LOT fewer wild goose chases (Dr. Singh).
As Barry Carol knows, I used to do the recruiting for a large group. But when you hire people fresh out of training, you usually don’t know where they stand on the “Singh – Jha Spectrum”. And where they are on this spectrum changes over time. The “kids” coming out of training now are trained to be “overcallers” and people tend to drift toward calling less as they get older.
In practice, as you read the reports of your partners, you tend to get a sense of where they fall on that spectrum. Personally, I tend to fall on the “Singh” side of the spectrum – although I hope my false positive rate is equally low and my false negative rate is as good.
The trend is, unfortunately, toward more “ditzel calling”, more follow up of questionably relevant findings and more expensive testing.
Oh, and I would be remiss if I didn’t point out the message that our Government, through its Judicial Branch, is giving us: If you miss something – for any reason (including trying to save the taxpayers money) YAOYOMF – You are on your own my friend – or something like that.
Peter, my comment was the general blight of booking advanced imaging studies with a one word entry on the “Clinical Details” section.
In my institution in North America, it is considered acceptable to perform a CT pulmonary angiography study with the words “CTPA protocol” entered in the clinical details section, and patients get CT brains, carotids and circle of Willis with the words “hot stroke protocol” entered. ER patients often get near full body x ray and CT scans with only the word “trauma”.
The radiologist has the option of dialing for every study he/she is reading, to get more info, or just interpreting them with the information the referrer chose to provide.
We are expected to get through large worklists, and ringing for information on every study would treble turnaround time, unacceptable for patient care.
A fundamental ignorance here is that the positive predictive value of any scientific test, medical tests included, is the product of test sensitivity and pre test probability.
An unknown pretest probability (0.5) means that for an imaging study, no matter how good the machine, the tech or the radiologist, will have several findings, the accuracy of which are halved!
This is why so many reports by good radiologists read “may be this, could be that, clinical correlation advised” -they literally had no clue what was wrong with the patient.
Think about it, would any competent doctor routinely examine patients and formulate a diagnosis without taking a history? You would fail med school with that attitude, yet this is what many physicians expect radiologists to do!
As for the second opinion thing, radiology has a full grading system for disagreeing with reports of others, we are asked for second reads all the time, knowing what the initial report read is crucial in any comparison.
Yes, I agree. If imaging was an exact science, and there is a movement within imaging increasing more quantification, then no context would be necessary.
My wife describes Radiology as making black and white decisions based on shades of gray. My opinion is that making a radiographic diagnosis without complete information on the patient is like making a clinical diagnosis without taking a history. This is why the best radiologists make themselves readily available to consult by phone or in person when the clinician has a question on the result.
The other problem with not knowing the history is that the wrong test may be ordered for the potential condition, making the test worthless.
The most recent issue of JAMA actually has an interesting Opinion Editorial on the role of the radiologist in controlling costs and limiting unecessary studies.
I agree that there are better and worse ROC curves. But even within an ROC curve there is a trade off.
One of the problems is discerning between radiologist moving along an ROC curve and radiologist moving to a higher (greater AUC) curve. Often, the two are conflated.
Actually I, too, would prefer the blank slate approach. The problem is that there are so many shades of gray (possible adrenal cancer, possible this and possible that) that context helps reduce the noise, particularly when there will be an after the fact expert witness who will tell me how I should have known better all along.
There is computer aided detection. But that must decide as well where it wishes to draw the line between being wrong (false positive) or being really wrong (false negative).
It is easier to make the decisive radiologist over cautious than it is to make the over cautious radiologist decisive.
That is it is easier to improve sensitivity at the expense of specificity than it is to improve specificity at the expense of sensitivity.
That’s my conjecture. I have no empirical evidence to back that up.
Thanks for reading Perry.
Specificity, like good Scotch, comes with age and experience!
Now Vik, you are allowing your single malt bias to creep in!
The problem is that the system is killing the decisive Singh. And that’s ultimately bad care.
I’m all for checking the blind spot of the rear view mirror without exception, and punishing those that don’t. But the safety culture has gone out of control.
I would love to hear from Legacyflyer on this one.
Since absolute perfection is an impossible and unreasonable expectation, if you’re hiring a radiologist for a large group practice, how many missed positives per 1,000 or 10,000 readings are acceptable? As a patient, I would opt for Dr. Singh. A 1 in 10,000 chance of missing a lung cancer is an acceptable risk to me in exchange for cost-effectiveness and the avoidance of unnecessary additional testing. If I’m the primary care doctor, I would prefer the radiologist who can give me the information I need in a clear and concise manner without wasting my time and without suggesting additional unnecessary testing. This is especially relevant if the primary care doctor has a capitated or shared savings contract. For the hospital that wants to maximize its revenue by doing more testing, get over it and embrace capitated or shared risk / shared savings contracts. To help this along, reform the legal system so that doctors are allowed to miss something occasionally. Both patients and payers pay a high price for unnecessary defensive medicine. How prevalent are failure to diagnose lawsuits in other developed countries and how successful are they when brought? The answers to those two questions would speak volumes, I think.
For a single radio receiver, if I turn up its sensitivity, I am more likely to detect a weak station I want to hear and I am also more likely to detect others that I don’t (1-specificity.) The relationship between these two characteristics can generally be plotted as a continuous function, a receiver operator curve (ROC.) You describe the tradeoff beautifully. But, then you make a mistake.
Each radio has its own ROC. Similarly, so does each radiologist.
ROCs can intersect, resulting in the tradeoffs you describe between Dr. Jha and Dr. Singh. The other possibility is that their curves never cross, that one does perform better than the other.
One corollary to your conclusion is that education, training and experience resulting in improvements in sensitivity must incur a loss of specificity because there is always a tradeoff between the two. Or vice versa. Another is that the area under the curve for every radiologist is the same. Certainly, neither is true.
None of this is intended to refute that pay for performance is a tricky and nuanced concept. P4P can be a ham-fisted instrument resulting in tremendous collateral damage and no measurable improvement. Let’s do everything we can to avoid these accidents through small trials and experiments Because, I don’t think that we should profit from our mistakes, where avoidable error results in additional service, generating more fees. That is perverse.
Fergus, I think I would prefer to let the Rad interpret the picture without outside information that may lead them down a biased path. I have had experiences with dentists, who I was sent to for second opinion, just agreeing with the first doc as a matter of professional courtesy. Not saying that a Rad would do that but just putting a thought in their mind may be a subconscious influence.
I’d prefer a blank slate analysis.
Another point here relating to so called “consumer directed health plans” (CDHP) is that how is the consumer supposed to evaluate cost/quality, especially when they are not privy to inside information.
There is an inverse relationship between clarity of information shared with the radiologist prior to reading the study, and amount of gibberish and pointless waffle in the report.
One comment here suggested the radiologist read the study, then track down and ring the referrer, which could take 10-15 minutes, to ask what they were concerned about. How about letting the radiologist know the concern when the scan was booked by filling the clinical question box correctly, doesn’t the patient deserve that basic courtesy?
I have heard that a computer program can be more accurate than an expert cardiologist in interpreting EKGs. Is there a prospect of something similar occurring in radiology?
Absolutely not, it’s serious business for the rads.
Well I guess in an imperfect world that’s all we can hope for, just as long as the peer review does not favor billing volume over patient safety.
Actually, there is peer review, Peter, the rads in the group review a percentage of each other’s cases. You can also bet if there’s a miss, that one will be reviewed extensively.
I can also tell you that misses will happen whether reading fast or slow. Busy day, multiple interruptions, radiologist on night call, being human. I bet even the ficitonal Dr. Jha will eventually have a miss despite his overcaution.
Don’t both radiologists need to go back to school? Is there anything here justifying peer review and additional training?
You nailed it. The problem is that the overly-cautions doc is not clinically helpful a lot of times to the doctor who ordered the test. I often get radiology reports that give so much information that it makes it quite hard to find the answer to the question I was asking. The ideal situation is when the radiologist calls me and gives me a verbal report. Then I can ask if they see the thing I am looking for (or hoping to rule out). Of course, I need to be sure I send enough clinical information to the radiologist so they can know what I am looking for.
As an aside, as a pediatrician, I am constantly frustrated with the radiologists’ inability to call an infant/young child x-ray normal. They nearly always (in this town) say “cannot rule out bronchitis,” or something ridiulous like that. Sorry, that’s a pet peeve.
I agree, great piece. Being married to a radiologist, I can appreciate the nuances in how these two are perceived. I like to think my wife is a nice combination of the two, which does happen.
Dr. Jha will be appreciated by the specialists, but also perhaps by the hospital because his hedging reports mean more revenue from more studies.
Dr Singh will be a favorite of Primary Care for her succint reports, and fast TAT will endear her to the hospital admin as well.
The radiology group will really like Dr. Singh over Dr Jha because she will produce more RVUs.
It is most important above all to be able to discern a good Scotch. I prefer Talisker myself.
This is a GREAT essay. Thank you, Saurabh for painting the picture very nicely of the kinds of trade-offs we have to eventually deal with but pretend don’t exist. The healthcare industry is the Gordian knot that it is because these confounders have been ignored as we added even more layers of complexity figuring that someone would eventually sort it all out. There is no someone and eventually is now…or, maybe never, because I believe the entire construct of American healthcare is irretrievably broken.
By the way, I vote for Dr. Jha, not only because he is apparently careful to the point of being a nuisance (something people in my household accuse me of from time to time…especially the nuisance part), but because I know he knows his Scotch, and that must count for something. At least it does in my book.