Agentic computer-vision pipeline May 2026
DermAssist
DermAssist takes a dermatoscopic photo of a skin lesion and writes a structured report backed by medical literature. A person has to review every report before it's final. It's a research and teaching tool, not a medical device, and nothing it says is a diagnosis.
The pipeline
It's a LangGraph graph: ingest, preprocess, classify, interpret, look up literature, build the report, wait for human review, then finalize. If the reviewer rejects or edits the report, the graph goes back to the interpretation step with their notes.
The part I cared about most is the orchestration and the review gate, more than the model.
Reading the image
Preprocessing loads and resizes the photo, corrects the color with Shades-of-Gray, and removes hair with a DullRazor-style filter. A pretrained ViT from Hugging Face then gives probabilities over the seven HAM10000 lesion classes. The classifier owns those probabilities; Claude is told never to produce numbers of its own.
Explaining it
Claude looks at the preprocessed image, goes through the ABCDE features (asymmetry, border, color, diameter, evolution), and writes an interpretation that has to line up with the classifier. Its answer is forced through a Pydantic schema with tool calls, and the prompt is cached.
Backing it with literature
The pipeline searches PubMed through an MCP server, using the top diagnosis as the query. It embeds the abstracts with PubMedBERT, stores them in a local ChromaDB collection, and pulls the most relevant ones into the report. I bundled a PubMed MCP server so this works out of the box.
The review gate
When the graph reaches human review, it really stops. It uses LangGraph's interrupt() with a SQLite checkpointer, so a run can pause in one process and resume in another.
A Streamlit app shows the original and preprocessed images, the classifier's probabilities, Claude's interpretation and the citations side by side, with Approve, Reject and Edit buttons. The disclaimer is pinned to every screen.
Testing
The tests cover pausing at the gate, resuming to finalize, rejecting and looping back, and resuming from a brand-new graph instance to show the state really lives on disk. Every stage falls back to a mock when its dependencies or API keys are missing, so the whole graph runs anywhere.