Despite a deeply disturbing personal experience, data scientist and medical doctor, Dr. Candace Makeda Moore, explains why she has high hopes for combating bias with AI in the healthcare industry

 

Philosophers like Kant and John Rawls have written a great deal on fairness. But to translate these ideas into actual code, which often implies complex mathematics including everything from linear algebra to probability theory, is a really interesting challenge to which many med-tech companies have sadly not risen.

AI based on historical patient data often reveals that we have had an issue with unfairness in medicine. Some of my colleagues might be shocked to learn that not only racist people but also computer algorithms, from straightforward signal processing ones to complex opaque deep learning style algorithms, are in many cases currently exacerbating the historical unfairness of medicine in terms of vulnerable populations.

There is an unfortunate tendency for people to believe anything that comes out of a computerized system is correct (automation bias) and this could entrench inequalities in the healthcare system on a huge scale while simultaneously allowing the human culprits to sidestep responsibility.

I spend a lot of time programming computers and like every other programmer, I know I make mistakes all the time. Sometimes, those mistakes mean what was written will not compile or that my code gives me an error message when run. Less obvious are mistakes in code that compile and give results but not the ones intended mathematically.

I think many people would be surprised how often released code, code in medical products in use already, have this kind of problem. Particularly when algorithms for healthcare have been written by engineers or computer scientists who do not have a lot of insight into medicine. But the fact that multiple algorithms with mistakes biased against black patients – such as the Optum algorithm – ran live in so many hospitals that millions of patients were affected, with little resistance from physicians, should cause some reflection in any doctor who imagines him or herself to be dedicated to the care of all patients regardless of race.

The physician workforce in many countries is predominantly drawn from upper socioeconomic status – people whose lives will never be threatened by a biased AI program like the Optum algorithm. Not only do most members of the physician workforce not have to worry about algorithmic bias affecting them or their children, but they are also usually blissfully unaware of how they participate actively in creating a bias against patients different than themselves.

I have decided to write about my personal experiences in the hope they catch the eyes of people who live in a bubble bordered by news about stock prices, tech IPOs, global markets, or whatever else the various algorithmically-driven newsfeeds of the upper-middle class feature. I do not suppose it is lost on my colleagues in the USA that we are in the middle of dramatic protests about violence against black people. But my colleagues may be, in many cases, less aware of how the healthcare workforce, and now some of the algorithms it uses, actively create similar problems.

As a patient, who also happens to be a black woman, I have dealt with more than my fair share of discrimination. I hope that AI can improve medicine because my experiences as a black patient in the US left me with little hope that humans alone can change the problem of discrimination in medicine.

I wanted a child at some point after I finished medical school so I looked into adoption when I was single. Adoption was practically impossible to negotiate given my student loans, so I thought perhaps I would have a biological child instead and started pursuing fertility care. To be honest, I can’t say with certainty that all the behaviors from clinicians I experienced and suspected to be racist were definitely coming from racism because I can’t know with any certainty what other people are thinking.

However, I really doubt if I were a well dressed white person carrying health insurance typical of someone affluent, that I would have been exposed to the attitudes and comments that resulted in me feeling extremely uncomfortable.

As a physician, I was able to request specific tests and treatments from the doctors I saw, but many flat-out refused to help me. I remember one physician who bluntly told me, “People like you should not be allowed to have children”. Others lectured me, telling me that I should be on birth control when I told them I was in my thirties and desperate to have a child. You may find it surprising but often, when I told other physicians that I was also a medical doctor, they would assume, based on my appearance and the kind of insurance I carried, that I was lying.

I recall a particular incident when I was doing a voluntary externship. Halfway through, I knew something was wrong with my health. I didn’t want my colleagues to know, so I went to the emergency room of a different hospital. I told the physicians I could feel something was not right. Naturally, my vital signs were examined and they realized, that indeed, something was off. A relatively young person with no significant medical history should not have a racing pulse and very high blood pressure. The physicians asked if I was on drugs. This was logical; something like cocaine or some kind of stimulant would explain some of my symptoms like my racing pulse.

I tried explaining several times that I was also a medical doctor and I thought I understood what was going on. But the treating doctors refused to listen. They even looked at me and said, “I don’t know why your drug test came back negative but we can help you with the fact that you think you are a doctor. You have been ordered psychiatry consult”. To their great credit, a psychiatrist eventually spoke with me, realized that I was indeed a doctor and wished me good luck with my career.

During that visit, my systolic blood pressure ended up going well over 180 and I thought, ‘Racism is literally going to kill me…they would never discharge a white person with no history of hypertension and this blood pressure without doing something.’ Of course, I was aware that acutely lowering my blood pressure could have caused problems, but I have a feeling that a patient with a different appearance might have received a recommendation to at least follow up a primary care physician. Or perhaps even some interest in my idea of thyroid disease as a more valid differential diagnosis than illicit drugs that were not showing up on testing. I was later diagnosed with antibodies against my thyroid.

Although the lack of fertility treatment, and the lectures on how I should not have children that I received from American physicians were not life-threatening, it was what actually bothered me the most as a patient. This was not because I lack insight about the meaning of my untreated blood pressure or thyroid problems. To put this all in a historical perspective, I was born in 1976 in North Carolina, around the time when the Eugenics Board of North Carolina (EBNC) was being abolished. EBNC was originally set up to sterilize “mentally defective” individuals. But gradually, the program began supporting the sterilization of people from entire populations, and specifically, people from my population. In truth, all over the USA, if you bother to listen to or collect oral history accounts, black and other non-white women would often be sterilized without their permission even during procedures not related to childbearing, like routine dental work.

Laws have changed, but I can say based on my experiences, the attitudes of many in the physician workforce have not. Physicians do know there may be legal consequences of explicit racism so they will not always say out loud what has been deemed inappropriate. But I suspect strongly that race came into play in the many interactions I had with the other doctors so I chose to leave the US and move to Israel.

In some sense, the physicians who treated me in a racist way have won. Not only will I never be able to have biological children as I might have if I had received decent timely treatment, I have also left the country unwittingly ensuring that even more black women will be subjected to their racist attitudes and actions.

I am quite sure the smug physicians who treated me will never pay any price for or even be aware of how much harm they perpetuate against black people. They probably believe themselves to be heroes for dealing with poor black patients, unaware that interactions with doctors like them tend to drive many patients away from seeking proper medical care altogether.

They would blame black patients for not seeking care quickly or assume it is due to a lack of education, never wondering if they themselves might be part of the problem. These are not merely academic issues in the days of COVID, and in fact, perhaps the threat to their health caused by patients from underserved populations may eventually cause some soul-searching among even the most oblivious healthcare providers in the US.

I hope that by sharing my personal story I can make it easier for doctors to put themselves in the shoes of someone else because they should be able to imagine that I am just like them. But it shouldn’t take an Ivy leaguer with a medical degree to be an example of someone who might deserve bias-free care. The norms of human rights and medical ethics should promise this to us all.

I wish I could claim the US is an outlier in terms of systemic racism cursed by its particular cultural and historical background, but it is not. Many countries have different castes or classes of people who are treated on average very differently. I fled to Israel, a relatively pronatalist country with universal healthcare as I thought this would be a good place to have a child. I filed for adoption but in Israel, differences in terms of how people are treated are not drawn based only on ethnic groups but also religion.

My husband is non-Jewish while I am Jewish. As far as I am concerned, these are just classifications on our identity cards but unfortunately, they have tremendous bureaucratic significance. For example, unable to marry in Israel, my husband and I had to get married in Europe and return. The same legal distinctions come into play for adoption. Children are sorted based on their religions and children of a particular religion are not allowed to be adopted into a family of a different religion.

For complex reasons, children regarded as Muslims are more likely to wait longer before they are taken into an adoptive family. Certain Muslim populations are also exposed to attitudes like the ones I encountered in doctors in the USA. Attitudes that border on the eugenic, but I have not personally seen these attitudes realized in the healthcare system towards Muslims. But interestingly, I have seen them realized against other vulnerable populations with less representation in the ranks of doctors.

The idea that women from an African refugee or Ethiopian background should have fewer children has been openly voiced by certain officials here in Israel. But I suspect even they would balk at the idea put forth by colleagues in the US, that women of my color should simply be stopped from ever having children – the logical consequence of trying to stop a childless woman over a certain age like myself from having children.

I share my personal experience because I want physicians to think about why people from certain communities may not trust doctors more deeply. I have often read work implying that health disparities around infertility could come down to certain types of patients not seeking proper care. But why would they if they already know how they might be treated?

If I could, I simply would not have gone through the emotionally painful experience of being told by physicians that I should be wiped from the human gene pool (physicians who were sometimes themselves visibly pregnant and perhaps even younger than me) and spoken to as if I had the intelligence of a child and was deluded to think that I might have some insight into my own health or desires. Patient autonomy is taught as a pillar of medical ethics, however, it seems de facto to only apply to certain patients from favored populations.

Depending on how it’s used, I believe AI and technology can serve to make this issue both better and worse. The recent racial justice protests are illustrative of the potential of technology. Since before I was born, many communities experienced strained or outright terrible relationships with law enforcement. But technology, in this case, smartphone cameras and the internet, has put that under an unforgiving objective light for the entire public to see. One thing programmers like me can do, with statistical analyses, data, and some of the same algorithms common in industrial AI programs, is to show certain phenomena in a much more objective light.

In the proper hands, AI algorithms could help improve this de facto disaster of medical ethics and improve health disparities and even change patterns of discrimination. However, I doubt it will happen very quickly because most organizations are bound by profits. So, what you see is a lot of people silencing themselves. They know a better world is possible but they are afraid to speak up publicly. Until there are financial gains to address health disparities or significant penalties for not addressing them, biased care including algorithms will continue to exist.

I often imagine algorithms that would help reduce bias in care, but instead of working on them, I’m constrained by my financial needs to work on data more closely tied to some organization’s “bottom line.” And as I do, I often watch people who feel they have an intuition about fairness or that they are not racist or exceptionally discriminating, make mistakes that exacerbate racism. They don’t know that dealing with fairness is a complex problem.

Solutions won’t necessarily be easy and will probably involve some trade-offs. We may actually not see substantial changes until better regulations come in and force these issues upon med-tech companies. Nevertheless, in spite of, and because of my own personal experiences, I remain optimistic that where racism and fairness are concerned, our industry can still make significant progress by judiciously using algorithms.