Professionalism and Value Olive Branch Icon

Setting a Research Agenda for the Use of Artificial Intelligence & Machine Learning in Primary Care

Virtual Summit: March 18-19, 2020

FOR QUESTIONS CONTACT: Andrew Bazemore, & Mikel Severson at 1-202-600-9447

Event Details

Thursday, March 18, 2021

1:00PM – 5:00 PM EST | 10:00AM – 2:00 PM PST

1:00 – 1:15 PM EST | 10:00 – 10:15 AM PST

Welcome & Brief Meeting Overview


1:15 – 1:45 PM EST | 10:15 – 10:45 AM PST

Brief Introductions


1:45 – 2:50 PM EST | 10:45 – 11:50 AM PST

AI/ML & Primary Care: Opportunities to Transform Delivery
(Panel Presentation & Discussion)


2:50 – 3:00 PM EST | 11:50 – 12:00 PM PST

Break


3:00 – 4:00 PM EST | 12:00 – 1:00 PM PST

AI/ML & Primary Care: Opportunities to Transform Research (Panel Presentation & Discussion)


4:00 – 4:30 PM EST | 1:00 – 1:30 PM PST

Breakout Session A: Small Group Discussions on Advancing AI/ML application in Primary Care Research & Practice


4:30 – 4:50 PM EST | 1:30 – 1:50 PM PST

Small Group Report Outs


4:50 – 5:00 PM EST | 1:50 – 2:00 PM PST

Wrap up, Review Day 2 Agenda & Adjourn

Friday, March 19, 2021

10:00AM – 1:00 PM EST | 7:00AM – 10:00 AM PST

10:00 – 10:20 AM EST | 7:00 – 7:20 AM PST

Welcome, Review Lessons from Day 1, Agenda for Day 2


10:20 – 11:00 AM EST | 7:20 – 8:00 AM PST

Breakout Session B: Small Group Discussions on Setting Priorities for the AI/ML to advance primary care agenda:


11:00 – 12:00 PM EST | 8:00 – 9:00 AM PST

Small Group Report Outs, Reactions & Ranking Exercise


12:00 – 12:15 AM EST | 9:00 – 9:15 AM PST

Break


12:15 – 1:00 PM EST | 9:15 – 10:00 AM PST

Next steps/Working together to advance an agenda


1:00 PM EST | 10:00 AM PST

Adjourn


Download the pdf

Thursday, March 18, 2021

1:00PM – 5:00 PM EST
10:00AM – 2:00 PM PST

1:00 – 1:15 PM EST
10:00 – 10:15 AM PST

Welcome & Brief Meeting Overview


1:15 – 1:45 PM EST
10:15 – 10:45 AM PST

Brief Introductions


1:45 – 2:50 PM EST
10:45 – 11:50 AM PST

AI/ML & Primary Care: Opportunities to Transform Delivery
(Panel Presentation & Discussion)


2:50 – 3:00 PM EST
11:50 – 12:00 PM PST

Break


3:00 – 4:00 PM EST
12:00 – 1:00 PM PST

AI/ML & Primary Care: Opportunities to Transform Research (Panel Presentation & Discussion)


4:00 – 4:30 PM EST
1:00 – 1:30 PM PST

Breakout Session A: Small Group Discussions on Advancing AI/ML application in Primary Care Research & Practice


4:30 – 4:50 PM EST
1:30 – 1:50 PM PST

Small Group Report Outs


4:50 – 5:00 PM EST
1:50 – 2:00 PM PST

Wrap up, Review Day 2 Agenda & Adjourn

Friday, March 19, 2021

10:00AM – 1:00 PM EST
7:00AM – 10:00 AM PST

10:00 – 10:20 AM EST
7:00 – 7:20 AM PST

Welcome, Review Lessons from Day 1, Agenda for Day 2


10:20 – 11:00 AM EST
7:20 – 8:00 AM PST

Breakout Session B: Small Group Discussions on Setting Priorities for the AI/ML to advance primary care agenda:


11:00 – 12:00 PM EST
8:00 – 9:00 AM PST

Small Group Report Outs, Reactions & Ranking Exercise


12:00 – 12:15 AM EST
9:00 – 9:15 AM PST

Break


12:15 – 1:00 PM EST
9:15 – 10:00 AM PST

Next steps/Working together to advance an agenda


1:00 PM EST
10:00 AM PST

Adjourn


Download the pdf
With the digitization of everything from videos to voices and documents, artificial intelligence and machine learning (AI/ML) have revolutionized industries, including medicine, but have yet to transform primary care. A review of primary care AI/ML concluded that the field remains in “early stages of maturity,” despite a history spanning nearly 35 years.1 Only 1 out of every 7 of these papers includes a primary care author; therefore, one barrier to greater impact is engagement from the primary care community.

Transforming primary care is a necessity if we are to reduce waste and reverse declines in life expectancy.2,3 Primary care touches all Americans,4,5, and its presence in communities extends lives.6 Despite this benefit, primary care remains underfunded and overwhelmed.7 To care for patients holistically, primary care practices must coordinate with specialists, hospitals, mental health providers, and public health – a function that is critical to the effort to combat the current coronavirus disease 2019 pandemic.8 The resulting avalanche of data exceeds what individuals and teams can realistically manage, and practices struggle to turn these data into the insights needed to improve quality and health. Once obtained, these data must be stored in electronic health records (EHRs) and compiled into quality measures, a necessary but time-consuming process that erodes face time with patients and contributes to burnout.9–11

Turning data into knowledge is a challenge across all fields, and many have turned to computer science. In primary care, AI/ML can ensure that EHRs are updated in real-time, conversations are accurately and efficiently converted into notes, patients receive the preventive services they need, and high-risk individuals are connected to appropriate interventions. However, in the absence of input from end-users, including patients and clinicians, AI/ML will not automatically lead to better outcomes. Critics warn that AI/ML could increase costs, magnify biases, and disrupt relationships.

To avoid this fate, the primary care and AI/ML communities need to work in a transdisciplinary manner to create new frameworks and methods tailored to the complexity and longitudinality of primary care. To do so, they must collaboratively answer the questions critical to this field. For example, given scarce resources, how should key questions be prioritized? What are the most promising applications of primary care AI/ML? What investments in primary care AI/ML partnerships will yield the greatest returns? What infrastructure is needed to facilitate connections between primary care and AI/ML researchers?

TTo address these gaps, the American Board of Family Medicine plans to convene a virtual meeting titled “Setting a Research Agenda for the use of Artificial Intelligence & Machine Learning in Primary Care” in February 2020. Over two days, a small group of experts with combined knowledge of AI/ML, large datasets, policy, and primary care research will convene to discuss the state of AI/ML techniques and their use in the primary care setting, identify barriers and opportunities for further use, declare an agenda for future research and a priority list of questions.

The specific aims of the meeting will be to:
  1. Review a summary of ongoing efforts to incorporate AI/ML techniques into primary care research.
  2. Identify barriers to be addressed, assets to be leveraged in pursuit of greater integration between AI/ML and primary care.
  3. Develop consensus around a research agenda for the application of these techniques in primary care.
  4. Declare priority domains where the techniques may offer the most-needed insights and priority questions that need the most immediate attention.
  5. Discuss a plan for engaging the primary care research community, funding community, data, and policy stakeholders in advancing both the agenda and priority research questions.

A report will summarize these discussions and serve as fodder for peer-reviewed publications to follow.

Developing a shared language and standardizing definitions are important as we bring together disparate stakeholders. For example, primary care has been defined differently by organizations. The World Health Organization defines primary health care as “essential health care based on practical, scientifically sound and socially acceptable methods and technology made universally accessible to individuals and families in the community through their full participation and at a cost that the community and country can afford to maintain.” Its Declaration of Alma Ata goes on to state that “[primary health care] forms an integral part both of the country’s health system, of which it is the central function and main focus, and of the overall social and economic development of the community. It is the first level of contact of individuals, the family and community with the national health system bringing health care as close as possible to where people live and work, and constitutes the first element of a continuing health care process.” Finally, “[It] addresses the main health problems in the community, providing promotive, preventive, curative and rehabilitative services accordingly.” 12 Barbara Starfield echoed some of these ideas when she characterized primary care as first-contact, continuous, comprehensive, and coordinated care provided to populations undifferentiated by gender, disease, or organ system.13 Similarly, the Institute of Medicine defined primary care as “the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and community.”14

John McCarthy coined the term AI, calling it the idea of getting a computer to do things which, when done by people, are said to involve intelligence.15 Marvin Minsky added to this definition by calling it “the study of ideas to bring into being machines that respond to stimulation consistent with the traditional response from humans, given the human capacity for contemplation, judgment, and intention.”16 These tasks include problem-solving, reasoning, understanding language, and learning. A subset of AI – machine learning – focuses on the learning aspect of intelligence. In his book Machine Learning, Tom Mitchell defined the field as “concerned with the question of how to construct computer programs that automatically improve with experience.” He goes on to describe machine learning as a computer program that can “learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.”17

Primary Care research, like the enterprise itself, is complex, with questions that lend themselves to approaches with commensurate complexity. Yet the use of AI in family medicine and primary care research is limited. In their recent commentary “Family Medicine & AI: Better Together”, Liaw et al note:

Without our input, AI risks following the path of EHRs. When the Health Information Technology for Economic and Clinical Health (HITECH) Act was passed, policy makers believed that EHRs would lead to care that was more efficient, effective, and equitable,15 and EHRs have led to important advances in population health and quality.16 However, with increasing burnout and decreasing time with patients, many lament that EHRs cater to the needs of administrators and EHR vendors rather than physicians and patients.17 The usability and interoperability failures underlying these complaints are not the result of gaps in technological expertise. Instead, these failures emerged, in part, because end-users like ourselves have been insufficiently engaged in relevant design, policy, and implementation decisions.

They further note the natural ability of AI wed itself to integrate multiple data sources including geographic, EHR, claims, and pharmacy data to identify those individuals at high risk for multiple chronic diseases, to facilitate timely referral and appropriate treatment, to streamline and facilitate quality measurement and to enhance primary care of patients directly.

APubMed search conducted in 2019 revealed no AI/ML papers in Family Medicine, Annals of Family Medicine, or the Journal of the American Board of Family Medicine. For comparison, AI/ML papers number 18, 77, and 8 for Academic Medicine, JAMA, and the Journal of General Internal Medicine, respectively. For context, a bibliometric analysis conducted in the same year identified 1,473 AI-related health care papers using Web of Science.18

In the summer editions of Annals of Family Medicine & Family Medicine, however, four relevant papers emerged:

  • Kueper JK, Terry AL, Zwarenstein M, Lizotte DJ. Artificial Intelligence and Primary Care Research: A Scoping Review. Ann Fam Med. 2020;18(3):250-258. doi:10.1370/afm.2518
  • Wingrove P, Liaw W, Weiss J, Petterson S, Maier J, Bazemore A. Using Machine Learning to Predict Primary Care and Advance Workforce Research. Ann Fam Med. 2020;18(4):334-340. doi:10.1370/afm.2550
  • Liaw W, Kakadiaris IA. Artificial Intelligence and Family Medicine: Better Together. Fam Med. 2020;52(1):8-10. https://doi.org/10.22454/FamMed.2020.881454.
  • Liaw W, Kakadiaris IA. Primary Care Artificial Intelligence: A Branch Hiding in Plain Sight. Annals of Family Medicine. May 2020; 18(3): 194-195.

Kueper et al. found 405 primary care AI articles since 1986.1 Two thirds of these articles focused on developing or modifying AI methods while the remaining supported diagnostic or treatment recommendations. Papers were included if they referenced primary care data, settings, or personnel. As previously noted, only 1 in 7 included an author with a primary care appointment. Noting that few tools were ready for widespread implementation, the authors called for the inclusion of frontline clinicians in these studies and the evaluation of these tools in primary care settings.

Partnerships and Planning Committee

A Planning Committee will guide the development of this Conference. Prior to the meeting date, Committee members will meet monthly to provide direction for the Conference, give feedback on Conference materials, select and recruit participants, and interpret participant responses to pre-conference prompts. Committee members will consist of representatives from the American Board of Family Medicine, Stanford University, the University of Houston, and federal agencies.

Conference Format

Before the meeting, the Planning Committee will engage in activities that will enhance the effectiveness of the time spent communicating synchronously. First, a Conference intern will conduct 60-minute interviews with participants (Table 1). During this interview, the intern will ask questions related to primary care AI/ML exemplars, opportunities, challenges, assets, and priority domains/questions. This interview will also touch on why participants are attending and what they hope to accomplish. The intern will record field notes that will inform the virtual conversation. Prior to the meeting, participants will receive relevant readings and a synthesis of the interviews.

In this 2-day, virtual meeting, we will have panels and small group breakouts to maximize participant engagement. The discussion will cover four themes: Current Landscape, Research Agenda, Infrastructure, and Dissemination. Each small group will include a student reporter, who will present back to the large group.

We plan to recruit 20 national experts to participate. To select these individuals, we will consider diversity in expertise, gender, race/ethnicity, and region. Attendees will include stakeholders from primary care, artificial intelligence, research, frontline practice, industry, insurance, government, community-based organizations, and law.

Related Conferences

Multiple associations and conferences bring together health care and AI/ML stakeholders. For example, Datapalooza, the Healthcare Information and Management Systems Society Conference, and Ai4 Healthcare provide content at the intersection of health care and technology, though each targets different audiences. Despite these opportunities, these meetings neither focus on primary care specifically nor address the aims documented in this proposal.

Conference Work Products and Dissemination Plan

Several products will come from this event. First, we will produce a white paper describing the event and lessons learned from this process (Table 2). This document will include the participants, agenda, bibliography, participant interviews, and themes. Second, we will submit a manuscript for peer-review that outlines the priority research domains and questions for primary care AI/ML.

  1. Kueper J, Terry AL, Zwarenstein M, Lizotte DJ. Artificial Intelligence and Primary Care Research: A Scoping Review. Ann Fam Med. Published online 2020.
  2. Institute of Medicine. Better Care at Lower Cost: The Path to Continuously Learning Health Care in America. National Academies Press; 2013.
  3. Case A, Deaton A. Rising morbidity and mortality in midlife among white non-Hispanic Americans in the 21st century. Proc Natl Acad Sci. 2015;112(49):15078-15083. doi:10.1073/pnas.1518393112
  4. Petterson S, McNellis R, Klink K, Meyers D, Bazemore A. The State of Primary Care in the United States. Robert Graham Center; 2018.
  5. Green LA, Fryer GE, Yawn BP, Lanier D, Dovey SM. The Ecology of Medical Care Revisited. N Engl J Med. 2001;344(26):2021-2025.
  6. Basu S, Berkowitz SA, Phillips RL, Bitton A, Landon BE, Phillips RS. Association of Primary Care Physician Supply With Population Mortality in the United States, 2005-2015. JAMA Intern Med. Published online February 18, 2019. doi:10.1001/jamainternmed.2018.7624
  7. Martin S, Phillips RL, Petterson S, Levin Z, Bazemore AW. Primary Care Spending in the United States, 2002-2016. JAMA Intern Med. 2020;180(7):1019. doi:10.1001/jamainternmed.2020.1360
  8. Westfall JM, Petterson S, Rhee K, et al. A New “PPE” For A Thriving Community–Public Health, Primary Care, Health Equity. Health Affairs Blog. Published September 25, 2020. Accessed October 2, 2020.
  9. Young RA, Burge SK, Kumar KA, Wilson JM, Ortiz DF. A Time-Motion Study of Primary Care Physicians’ Work in the Electronic Health Record Era. Fam Med. 2018;50(2):91-99. doi:10.22454/FamMed.2018.184803
  10. Casalino LP, Gans D, Weber R, et al. US Physician Practices Spend More Than $15.4 Billion Annually To Report Quality Measures. Health Aff (Millwood). 2016;35(3):401-406. doi:10.1377/hlthaff.2015.1258
  11. Shanafelt TD, Hasan O, Dyrbye LN, et al. Changes in Burnout and Satisfaction With Work-Life Balance in Physicians and the General US Working Population Between 2011 and 2014. Mayo Clin Proc. 2015;90(12):1600-1613. doi:10.1016/j.mayocp.2015.08.023
  12. World Health Organization. Declaration of Alma-Ata International Conference on Primary Health Care.; 1978:159-161. Accessed October 1, 2020.
  13. Starfield B. Primary Care: Balancing Health Needs, Services, and Technology. Oxford University Press; 1998.
  14. Institute of Medicine. Defining Primary Care: An Interim Report. The National Academies Press; 1994.
  15. McCarthy J, Minsky M, Rochester N, Shannon C. A Proposal for the Dartmouth Summer Research Project on Artificial Intelligence: August 31, 1955. AI Mag. 2006;27(4):12-14.
  16. Dean R & D Desh Bhagat University, Mandi Gobindgarh, India, Grewal PDS. A Critical Conceptual Analysis of Definitions of Artificial Intelligence as Applicable to Computer Engineering. IOSR J Comput Eng. 2014;16(2):09-13. doi:10.9790/0661-16210913
  17. Mitchell TM. Machine Learning. McGraw-Hill; 1997.
  18. Guo Y, Hao Z, Zhao S, Gong J, Yang F. Artificial Intelligence in Health Care: Bibliometric Analysis. J Med Internet Res. 2020;22(7):e18228. doi:10.2196/18228