Project Overview

Deciphering medical data is often a tedious and challenging task, typically requiring a specialist like an MD to manually label thousands of rows. However, Large Language Models (LLMs) can significantly reduce the need for expert human labeling. The key insight here is that we're not always seeking human feedback.

While we might never replicate the instinctual insights of a radiologist honed over 20 years of practice, we can teach AI to recognize the symptoms of long COVID or the diagnostic criteria for clinical depression. By leveraging well-encoded knowledge in medical domains, we can transform this knowledge into rows of example data. This helps train models to accurately extract procedures from clinical notes or identify symptoms from patient records.

We started by defining every ICD-10 medical code and gathering examples for each one. We then generated thousands of examples as they would appear in appointment transcripts, including both positive and negative instances. As a result, our solution matches MDs in accuracy while being able to scale in quantity exponentially.

Methodology

The core of our solution involved creating synthetic transcripts of doctor-patient interactions, each meticulously labeled with the appropriate ICD-10 codes. Here’s how we did it:

‍

‍Knowledge Ingestion‍

We gathered a wide range of medical texts and resources, including lists of ICD-10 codes and diagnostic textbooks, to ensure comprehensive coverage of medical scenarios.‍

Ground Truth Creation‍

Rather than extracting information from unstructured data, we created a new dataset from scratch based on the knowledge we gathered. Textbooks, diagnostic criteria, and operating procedures provided the information needed to create nearly perfect labeled transcripts of patients receiving care for any ICD-10 code.‍

Style and Distribution Matching‍

We transformed our data to mirror real-world examples of patient transcripts in tone, length, and style. This step ensured our data looked exactly like it would in real-world scenarios while maintaining the accuracy gained from generating it from scratch.

Challenges and Solutions

We encountered several key technical challenges along the way:

• Maintaining Medical Accuracy: Ensuring that synthetic data was as accurate and useful as real data required rigorous validation and continuous refinement of our pipeline. Properly ingesting available knowledge was crucial here.
• Balancing Realism and Anonymity: Our data needed to be realistic enough for training purposes while ensuring no actual patient data was used. We achieved this by carefully curating training materials to transfer only style and format from real data, not actual content.
• Model Bias: To prevent generating biased data, we incorporated bias detection and mitigation strategies throughout the development process.

Future Implications

The success of this project opens the door to numerous future applications:

• Automatic ICD-10 Datasets: Every time CMS updates the ICD-10 code set (twice a year), we can programmatically generate rows of data that help a model embed that knowledge. This makes training data easily accessible for any procedure, eliminating skewed data and performance biases toward the most common codes.
• Medical Uses Outside of Billing: Many fields in medicine face barriers due to the lack of proper training data. From chronic illness diagnostic criteria to DSM classifications, we can turn any unstructured knowledge into data that a model can use.

Sample Data

Here’s a sample of the synthetic data generated during the project, demonstrating realistic artifacts and common issues encountered in real data interpretation:

Doctor's Transcript:

Date: 10/11/2023

Presenting problem: L wrist pain

Hx: Pt tripped over curb, landed on outstretched hand. Immediate pain/swelling. No LOC.

Exam: L wrist swollen, tender over radial styloid. No deformity. ROM limited by pain.

Imaging: X-ray shows nondisplaced fracture of radial styloid.

Dx: Nondisplaced fracture of L radial styloid process

Plan: Splint, pain management, ortho follow-up in 1 week. Educated patient on RICE protocol.

Expected ICD-10 Code:

S52.515F - Nondisplaced fracture of left radial styloid process, subsequent encounter for open fracture type IIIA, IIIB, or IIIC with routine healing.

There's a few key issues highlighted in this example:

1. Inconsistent Abbreviations: Medical notes often include inconsistent abbreviations like "Pt" (patient), "ovr" (over), and "fx" (fracture).
2. Varied Data Formats: Real medical data can come in many formats, each with unique artifacts. This includes speech-to-text transcripts, OCR of handwritten notes, text converted from HTML, and data with headers or custom formats, all adding complexity and noise.
3. Complexity and Noise: Notes are often poorly detailed and complicated. Our pipelines capture these issues when generating data, ensuring the model generalizes well to real-world data variability, dealing with incomplete information, transcription errors, and varying levels of detail.

This sample shows how our synthetic data includes realistic artifacts such as abbreviations, shorthand, and variability in data format, common in real-world medical data. This level of detail ensures the data is robust and valuable for training advanced medical models.