“AI is undergoing a paradigm shift with the rise of models (e.g., BERT, DALL-E, GPT-3)… We call these models foundation models to underscore their critically central yet incomplete character. The term ‘foundation models’ is meant to evoke the central importance of these systems within whole ecosystems of software applications. There is the aspirational idea of having a foundation model that is…the bedrock of AI systems.”

Percy Laing, computer scientist at Stanford

A foundation model, based on deep learning and transfer learning concepts, was first introduced in May 2020 with OpenAI’s large language model (LLM) Generative Pre-trained Transformer 3 (GPT-3) model with its impressive 175 billion machine learning parameters (remember that parameters are internal to the model and are learned during training, whereas hyperparameters are external to the model and are set before the training begins so that the learning algorithm can learn the parameters.) This foundation model, which used to be large language models (LLM) but now functions beyond language processing (such as vision and audio), is trained on broad data using self-supervision and is designed to be adapted to a diverse set of downstream tasks (such as generating content, answering questions, recognizing objects, etc). Other foundation models include Bidirectional Encoder Representations from Transformers (BERT) and DALL-E (a clever portmanteau of the artist Salvador Dali and WALL-E from Pixar).

These huge foundation models have an impressive array of capabilities such as language, vision, and even reasoning) with applications in the form of search engines, interactive games, speech recognition, computer vision, and app creation software. An interesting aspect of foundation models is their ability to scale and provide homogenization, which is a harmonization of techniques. In addition, their adaptability can result in many downstream tasks being handled efficiently by the foundation model.

A particularly interesting application in healthcare is protein sequence search that can yield dividends in drug development. This potential for scaling across many possible related areas is particularly valuable for a domain like healthcare, which has basic and central anatomic and physiological tenets across many organ systems and clinical subspecialties. A well-established and reliable foundation model embedded with sound clinical concepts can be immensely useful to set forth variations in many clinical areas. This capability has potential for healthcare to finally achieve its Quintuple Aim as well as precision medicine goals.

Despite its full potential, there have been some challenges and disappointments in the use of these foundation models. It is precisely the advantage of homogenization that is also its inherent weakness: any flaw in the “progenitor” model is replicated and then passed on to all its model “congeners”. More accurate data and complete data sets are also key to the success of foundation models. This is particularly concerning given its potential vulnerability to perpetuating biases. In addition, there may be new security risks to foundation models given its prominent central role and huge size. Furthermore, given its high cost of creation, foundation models will be the projects that Will only be possible in large corporations and therefore open to public scrutiny and even criticism. Lastly, related to the aforementioned issue of “essential facilities”, is the public data privacy issue that may be exaggerated in the much larger models as compared with the smaller models.

What will be key for these foundation models to have a permanent place in AI in healthcare is a close collaboration between the AI experts and clinicians. The lead researcher of the foundation model study, Stanford professor, Percy Liang, so aptly stated that “AI has been primarily driven by computer scientists, technologists who for decades were trying to get anything working. What’s blindingly clear now is that this is not sufficient.”

This fascinating topic of foundation model in AI in healthcare along with others will be discussed at the annual AIMed Global Summit 2023 scheduled for June 5-7th of 2023 in San Diego. Hope to see you before then as well as there!

We at AIMed believe in changing healthcare one connection at a time. If you are interested in discussing the contents of this article or connecting, please drop me a line – [email protected]