On the platform I work on, a lot of patients arrive by referral. Their GP fills in a form with the patient’s details and why they’re being referred, and sends it over. A member of staff then adds the patient to our platform.
That last step was the problem.
The GP had already written everything down, and we were asking someone to type it all in again by hand. It’s slow, and every re-typed field is a chance for a mistake.
The regex scraper
We already had a system that tried to read these forms automatically. It used regular expressions to pull fields out of the PDF’s text, with one set of rules for each of the 23 fields it knew about.
Regex is a reasonable tool when the input is predictable. These forms aren’t. They come from different GP practices and different clinical systems, as PDFs, Word documents and scans. The same field is laid out differently on each:
- Blood results sit beside their label on one form, on the row underneath on another, and in a four-column grid on a third.
- A blood pressure row prints the date, the value and the units in separate cells, so a simple match grabs the date instead of the reading.
- Some practices leave address boxes empty and start the address one box later.
- Scanned forms have no text layer at all, so there’s nothing for a pattern to match.
Every new layout meant another pattern to write, and a pattern fixed for one form could quietly break another.
When a rule failed, the field arrived blank or wrong, and staff corrected it by hand. A wrong HbA1c is worse than a blank one, because the care pathway branches on it.
Using Amazon Bedrock instead
We replaced it with an AI form reader built on Amazon Bedrock
The model is Amazon Nova, called through the Converse API. I wanted something that could read a page the way a person does, and Nova reads PDFs visually, as rendered. That matters more than it sounds. Some of our forms store their filled-in values in form-field annotations whose position on the page the PDF parser can’t recover, so no amount of text clean-up can pair that label with its value. Reading the rendered page sidesteps the problem.
Structured output
We don’t ask the model for JSON and hope. We give it a single tool, extract_referral_fields, and force it to call that tool:
1'toolConfig' => [ 2 'tools' => [['toolSpec' => [ 3 'name' => 'extract_referral_fields', 4 'inputSchema' => ['json' => [ 5 'type' => 'object', 6 'properties' => [ 7 'dob' => [ 8 'type' => 'string', 9 'description' => "Patient's date of birth, copied exactly as written. "10 .'Omit if not present in the document.',11 ],12 // ...around 35 more13 ],14 ]],15 ]]],16 'toolChoice' => ['tool' => ['name' => 'extract_referral_fields']],17],
The schema is deliberately flat, because flat schemas extract more reliably than nested ones. Every field description lists the label variants we’ve seen (DOB, D.o.B, Date of Birth) and ends with the same “omit if not present” line. That repetition is on purpose. We have blank template forms in our test set, and a blank form is exactly what tempts a model into inventing something plausible.
Showing it the page and the text together
Wide tables were the model’s weak spot. On a four-column results grid, a label and its own value are separated by an unrelated pair, and that’s where every wrong lipid result came from.
The fix used something we already had. We were extracting the plain text of each form anyway, to verify the answers. Sending that text alongside the page gave the model both views. Cholesterol went from 70% correct to 100%, triglycerides from 62% to 100%, and the HDL/LDL swaps disappeared.
Checking what comes back
The model never gets the last word. Every value has to be copied verbatim from the document, and we check that in code rather than trusting the prompt:
- Is the value actually printed in the document?
- Is it printed under its own label, and not a neighbour’s?
- Is a date being returned as a blood pressure reading?
When a check raises a question, the model gets one targeted second look at only the flagged fields, with the reason stated as a fact. The rule that made this work is that a check asks, it never deletes. A correct value that gets flagged survives even if the second look comes back empty.
Where a form prints a value in a cell labelled with its own name, such as Mobile Tel: or HbA1c Value:, we read it straight from the document and skip the model. Each of those readers had to match our answer key before we let it override the model. The one for email addresses landed on the form’s own header address 15 times in 19, so it never got switched on.
A human still reviews it
The AI fills in the wizard. It doesn’t create the patient. A member of staff reviews the fields and saves, and the screen tells them the values were read by AI.
If the Bedrock call fails outright, the request falls back to the old regex scraper, so nobody ever gets less than they had before.
Handling clinical data carefully
Sending clinical documents to an AI model raises obvious questions, and the answers matter more than the feature.
- Region. Requests go to Bedrock in
eu-west-2(London), and the model is invoked from our own AWS account. - Training. AWS states that Bedrock doesn’t use your prompts or completions to train its models, or share them with third parties.
- Storage. For each scrape we log the file type, the status, the email address (where one was found), the token counts and the timing of each model call. The log doesn’t hold the document or the extracted clinical values.
On the model’s behaviour, the system prompt is blunt: copy, don’t interpret. It never derives one field from another, so no BMI calculated from height and weight and no gender inferred from a name. We also don’t clean the data. An early safeguard silently dropped readings outside a “normal” range, until a real form printed a serum cholesterol of 51 mmol/L and the scraper blanked it. The clinician reading the referral has context we don’t, and a silently missing value hides the referrer’s mistake. Unusual readings are now questioned, never quietly removed.
What went wrong along the way
- Prompt rules backfired. Adding a rule to fix one field broke others, three times. Precise field descriptions and checks in code worked. More prompt didn’t.
- Nova rejects its own tool calls. It occasionally answers with a
ModelErrorException(“invalid sequence as part of ToolUse”), mostly on near-empty forms. We retry, and the last attempt drops the text transcript, which on those forms is what provokes the error. - The 30-second limit. One family of forms produced a text extraction that contradicted the page, and the model spent over 100 seconds trying to reconcile the two. We now compare the two extractions and withhold the text when they disagree. That form went from 106.9 seconds to about 20.7, and all 96 of its fields stayed correct.
- Scans. With no text layer, every value the model read failed verification, because there was nothing to check it against. Transcribing the page first gives the checks something to work with.
The results
- The client reports a 40% increase in staff productivity and a 30% increase in referral output since it went live. These are their figures, not ours.
- The step that used to need manual intervention is now automated, and staff review fields instead of typing them.
- The regex system’s 23 rule sets, and the maintenance that came with every new layout, are gone as the first choice. It now only runs as the fallback.
- On a test set of 26 real-layout forms, scraped 4 times each (104 scrapes), 99 were fully correct (95.2%), and 3,312 of 3,328 fields were right (99.5%). A scrape only counted if all 32 scored fields were right.
- None of the five failing runs put a wrong value in front of staff. The field was left blank.
- It costs about 2.3p per referral on Nova Pro, so roughly £230 per 10,000 referrals.
What I learnt
The best use of AI here wasn’t anything flashy. It was taking a task where people re-type information that already exists, and removing it.
If you’ve got a pile of regex trying to understand semi-structured documents, that’s a strong sign a model will do the job better. But the model was the easy part. What made it trustworthy was building the answer key first, measuring every change against it, and throwing away the ideas the numbers didn’t support.
Check what it gives you in code, and keep a human in the loop where it matters.