Falcon CDI shadow pilot

What was tested, and what it showed

Healthful AI ran a shadow pilot of Falcon CDI over 23 de-identified episodes supplied by a private hospital group in the Kingdom of Saudi Arabia: 10 de-identified discharge summaries (inpatient) and 13 de-identified outpatient notes. Falcon is a point-of-care clinical documentation improvement (CDI) tool. It reads the clinical note as it is written and surfaces suggestions to the treating clinician for the specific detail that coding and classification will later depend on.

To measure Falcon, an independent Australian credentialed coder first coded the same records. The coder's documentation queries, written in the formal audit query format Australian health fund audits accept (the IHACPA guideline format), define what a professional retrospective audit would raise on each record. The pilot then measured how much of that audit value Falcon surfaces on the document itself, before any auditor is involved.

Every Falcon run in this case study was captured from server logs with each chart cryptographically bound to its logged run, and all but two charts were run repeatedly, so findings are reported as frequencies rather than single outputs.

Beyond the coder's own list. Falcon also raised findings that fell outside the independent coder's baseline entirely. Twelve such findings were identified across the corpus, and all twelve were put to the coder for review. Four were validated as carrying clinical evidence worth querying, and two of those moved the episode to a higher complexity level: one on a neonate in neonatal intensive care, from minor to major complications, the other from intermediate to major. Section 05 sets them out. This is a separate measurement from the headline above and the two are never combined. Two of the four carry no DRG consequence at all, and they matter for a different reason: an incomplete record is relied on by the next clinician who treats the patient.

From here. Section 09 sets out the path a shadow pilot of this kind takes into a formal pilot: a technical session with hospital IT, extension of the same measurement to a larger corpus, test environment access, and conversion to a formal pilot on the hospital's own records.

What the pilot was

The pilot compared independent views of the same records.

1. The records

   23 de-identified episodes supplied by the hospital group: 10 de-identified discharge summaries (inpatient) and 13 de-identified outpatient notes. All records were de-identified before any processing.

2. The benchmark

   An independent Australian credentialed coder reviewed every record. On the inpatient records the coder worked blind from the record as documented, producing a baseline DRG, a set of documentation queries in the IHACPA audit format, and the post-query DRG achievable if each query is answered. On the outpatient records the coder audited the hospital's submitted coding and recorded amendments. The coder is a benchmark, not ground truth: where Falcon and the coder diverge, this case study presents the divergence as an open question for adjudication, not as an error on either side.

3. The system under test

   Falcon CDI was run over the same records the coder received, surfacing suggestions for the clinical specificity each record lacks.

The comparison logic

The coder's queries define what documentation was missing from each record. Falcon is scored on whether it surfaces a suggestion for that same detail. If the clinician acts on the suggestion, the coder codes the improved record, and the movement from baseline DRG to post-query DRG under AR-DRG Version 9.0 is the result. The evidence unit is query-level convergence: for each coder query, did Falcon raise a suggestion for it.

Two cohorts, two distinct comparisons

1. Inpatient

   The hospital supplied no inpatient coding for these records, so the baseline is the independent coder's own first pass. The inpatient result is therefore the CDI-answerable uplift on a competent baseline, which is the improvement available even when the coding is already professionally done. It is not a claim about the hospital's inpatient coding, and none is made.

2. Outpatient

   The hospital's own submitted coding was available for these visits, so the outpatient cohort additionally compares against that coding. The outpatient result is expressed as coding accuracy and completeness rather than DRG movement.

Classification context

All coding is to ICD-10-AM 10th Edition, grouped with AR-DRG Version 9.0, the hospital's current classification framework, shared with Australian practice. That shared classification is what makes an independent Australian credentialed coder directly portable as a benchmark for records from the Kingdom.

How the measurement was made

The measurement standard applied in this pilot is the one we would want applied to us.

1. Evidence is captured from server logs, with cryptographic binding

   Every Falcon run reported here is taken from server logs, and every chart is bound to its logged run by a cryptographic hash of the record text. No result in this case study rests on a screen capture alone. A run that cannot be tied to its exact source record is excluded rather than approximated.

2. Findings are frequencies over repeated runs, not single outputs

   Generative systems vary between runs. Charts were therefore run repeatedly, nine to eleven runs each with the two exceptions disclosed below, and every finding is reported as a frequency. Throughout this document, 8/10 means the suggestion appeared in 8 of that chart's 10 runs. No catch is quoted without its frequency.

3. Outright, partial and missed

   Each coder query is scored at one of three levels.

   Score	Applies when
   Outright	Falcon raised the specific detail the query targets
   Partial	Falcon raised the right area, usually as an indication question against a medication or an order, without asserting the coded detail itself
   Missed	No run raised it

   On the obstetric emergency, for example, the anaemia was reached through the transfusion order in 10/10 runs while the haemorrhage assertion appeared in 2/10, so that query scores partial. Partials are reported separately throughout and are never counted as catches.

The instrument was hardened during the pilot

Where re-measurement under the hardened instrument produced a different reading from an earlier pass, the harder reading is the one reported, including where it is lower.

Results by cohort

How to read the results

1. The spine of the inpatient result is DRG movement under AR-DRG Version 9.0.

   Stated as classification change. Where a percentage change in reimbursement appears, it is an illustrative valuation only. No figure in this case study is a revenue claim.

2. The DRG movement column is the coder's own post-query coding.

   The ceiling that answering those queries reaches. Falcon's contribution is the coverage column, and the uplift is addressable uplift, contingent on clinician response. The pilot does not measure clinician response rates.

3. Coverage is measured against one credentialed coder's single independent pass.

   Not against adjudicated truth. Divergences are reported as open questions. Every quoted catch carries its run frequency.

4.1 Inpatient cohort

† Reported from a single fully captured run; see the disclosed exception in section 03. Single-run absences are reported as “not surfaced”, not as capability verdicts.

4.2 Outpatient cohort

The outpatient comparison is against the hospital's own submitted coding for the same visits, expressed as accuracy and completeness. There is no ambulatory grouper in the Kingdom, so no DRG axis is reported for this cohort.

Findings only Falcon could make

A retrospective audit, however good, can only find what is already in the record. A point-of-care tool can cause information to enter the record, because it asks the clinician while the patient is still under their care and the details are still current. An auditor or coder raises the same question at least a week later, when the episode is closed and the clinician's recall of it has faded. Any finding in this class sits above the ceiling of retrospective review.

How these findings were selected. Falcon surfaces many suggestions on each record. Those suggestions were scored into findings and compared against the independent coder's baseline. Most aligned with the coder, and that alignment is what section 04 measures. This section covers the remainder: every finding Falcon raised that fell outside that baseline entirely. Twelve findings met that description across the corpus, and all twelve were put to the coder for review. This is the complete set, not a selection from it.

* Percentage change over the starting DRG's price in the CHI schedule, calculated at the “Medical city and tertiary type provider” tier. The hospital's own contracted tier was not available for this pilot, so the tier is a stated assumption; the schedule's tiers are fixed multiples of one another, so the percentage is the same at every tier. The schedule is a draft CY 2021 price list. Any provider can apply its own contracted values to the same classification movements, which do not change. Each movement is contingent on the query being answered, which is the nature of a documentation query. One DRG applies per episode, so the neonate's movements are stages on a single path, E70B to E41B in section 04 and E41B to E41A here; they are sequential, not additive, and are never added together. The obstetric emergency is different: the coder's queries in section 04 and the coagulopathy finding here are two independent routes to the same O01A. Its 46% is the same movement already counted once in the section 04 average, not an addition to it.

Accuracy is not only a reimbursement question. Two of the four validated findings carry no DRG consequence at all, and they matter for a different reason. A record that does not say why a medication was given, or why an image was ordered, is an incomplete account of the patient's care. That record is relied on by the next clinician who treats them. Capturing the record accurately is as important as the reimbursement attached to it, and an incorrect or incomplete medical record can lead to harm. Every one of the four findings above is, first, a gap between what was done for the patient and what the record says was done.

What the two DRG movements represent. Neither is a realised gain. Each is a query which, answered one way, moves the episode. On the neonate the movement sits on top of the coder's own audit, so it represents what the tool adds beyond a completed expert review. On the obstetric emergency the tool reaches the same classification the coder reached, by a different route and from different evidence, which means the movement is available on a record that receives no retrospective audit at all. Most records do not receive one.

What this means in a hospital workflow

The CDI officer's view, during the admission

For a CDI program, the operational question is whether the deficiencies worth querying the treating physician about are visible while the patient is still admitted, not months later. The pilot evidences exactly that query-surfacing workflow: Falcon's suggestions repeatedly matched the independent coder's audit-format queries (the IHACPA guideline format, the Australian audit-query convention), in several cases near-verbatim. What an auditor would raise retrospectively appeared as a suggestion on the document itself.

Worked examples, from the scored runs:

1. Outpatient 3.

   The coder's audit query: no indication documented for the iron infusion. Falcon, in 10/10 runs, asked “What diagnosis or clinical indication is documented for the administration of FERINJECT 50MG/ML VIAL?”, quoted exactly as Falcon produced it and with the wording confirmed on the clinician-facing screen and answer options that carry the same distinction the coder's amendment coded (iron deficiency with or without anaemia). A near-verbatim point-of-care pre-emption of the audit query.

2. The infant in paediatric intensive care.

   The coder's query: oedema due to surgery, the item behind the episode's E02B → E02A movement. Falcon raised the corresponding medication-indication question in 10/10 runs, as a question on the relevant drug order rather than a proposed diagnosis.

3. The neonate in neonatal intensive care.

   The coder's query: renal fullness suggestive of hydronephrosis, the finding behind the episode's E70B → E41B movement. Falcon keyed and queried the renal-fullness line in 10/10 runs.

One product characteristic should be read alongside the catch numbers: on several charts the load-bearing catch arrived as an indication question rather than an asserted diagnosis. Clinicians experience a question to answer, not a label to accept, and the coverage figures are reported with that form disclosed.

The question arrives with its answers already drafted

Where Falcon raises a question it also supplies the answer set, and both are captured in the server logs. Across the corpus, every logged question carried one: 41% a multiple-choice list, the remainder a yes or no with a free-text follow-up. The lists are specific to the order they hang from, not generic. On the infant in paediatric intensive care the adrenaline nebuliser order carried the choices bronchodilator therapy, upper airway obstruction, acute bronchiolitis, postoperative airway oedema and other, and postoperative airway oedema is the item behind that episode's E02B → E02A movement. On Outpatient 3 the iron infusion carried iron deficiency anaemia, anaemia of chronic disease, iron deficiency without anaemia and other, in all ten runs, which is the same distinction the coder's amendment turned on.

The clinician is therefore not composing a response to a query. They are selecting from a list of options that includes the detail classification depends on, at the moment the patient is in front of them. That is the practical difference between a retrospective query, which requires the treating clinician to be located weeks later and to reconstruct the episode from memory, and a prompt answered while the record is being written.

On some of the partially-surfaced items the answer set carried exactly the detail the coder queried, even where Falcon did not assert it. On Outpatient 7 the question of which skin condition was actually being treated was never asserted, while the option list offered rosacea against the retinoid order in 9/9 runs. On the older adult general medicine episode the treated comorbidities were never proposed as diagnoses, while the option lists named them against the medications treating them in 10/10 runs. A partial of that kind is not a near miss. It is a question raised with the relevant options already in front of the clinician.

Answer sets and the coverage figures. These answer sets are offers rather than assertions. They are assessed on whether they are clinically plausible for the case, not against what the record happens to document, and they are not counted in the coverage figures above.

As calibration: an independent clinically-informed reviewer, working without time pressure, predicted 16 of the coder's 27 inpatient query items outright, with 3 more partially, 19 in total. Falcon surfaced 15 outright, with 6 more partially, 21 of the same 27, at the point of care. The two do not catch the same items, the comparison is recall-only, and Falcon's misses include driver-grade items both the reviewer and the coder found. Read with those caveats, a point-of-care tool landing in the same range as an unhurried clinical read is the pilot's calibration result.

Falcon does not assign codes. In the deployed workflow Falcon performs documentation improvement in real time at the point of care, and coding and grouping are then carried out separately by the hospital's coders. Grouping is a coder-initiated step rather than an automatic one.

Earlier in the journey: admission and pre-authorisation documentation

The Kingdom's forthcoming DRG implementation requires pre-admission authorisation requests through Nafis carrying principal and secondary diagnoses, documented by the physician at admission. The capability demonstrated in this pilot is not specific to discharge summaries: Falcon applies the same specificity mechanism to any clinical document, including admission requests and progress notes. The suggestions shown here at discharge can equally be raised at admission, where the pre-authorisation record is created.

Specificity and first-pass denials

Research indicates that 15 to 25% of claims in the Kingdom are denied at least once, and documentation specificity gaps are among the drivers of first-pass denials under NPHIES. The outpatient cohort in this pilot shows the relevant mechanism directly: queries an auditor would later raise on submitted coding being surfaced up front, before the claim is formed. The pilot did not measure denial rates; what it evidences is the mechanism by which denial risk is reduced.

Known limitations and product findings

Reported in full, because the measurement is only worth what its limits are understood to be.

1. Run-to-run variability, quantified.

   Falcon's output varies between runs on the same record, and the size of the variation is chart dependent. On the same build and day, five charts returned their lead suggestion in every run at constant rank, while the least stable chart ranged from 2 to 23 suggestions per run on identical input with no suggestion common to all of its runs, and two charts produced unions of 44 and 47 distinct suggestions against intersections of 2 and 5. No single run is representative in either direction: on the older adult general medicine episode, one run carried a high-significance occlusion finding that none of the other nine reproduced; on the adult ENT episode, the full-specificity discharge diagnosis appeared in only 1/10 runs while other runs dropped whole suggestion classes. This is why every finding in this case study is a frequency over repeated runs rather than a single output.

2. Detail held only in narrative prose is a systematic gap.

   Falcon's recall is strongest where a finding has a structured hook in the record, such as a medication, a laboratory value, or an existing diagnosis line. Findings documented only in narrative prose can be missed in every run: peritoneal adhesions 0/11 (the elective repeat caesarean section), ventilation coding 0/10 (the infant in paediatric intensive care), the device-complication driver 0/10 (the spinal surgery episode), therapy start and stop times 0/10 (the neonate in neonatal intensive care), the malnutrition complexity item 0/10 (the cardiology episode). This was identified and quantified during the pilot and is on the product improvement path.

3. Administrative and data-quality items are out of scope.

   Falcon's suggestions target clinical documentation specificity. It does not flag administrative or data-quality issues in the record: episodes filed under a different specialty than the care described, and implausible recorded observations, drew no suggestion in any run (0 of 81 runs across the eight repeatedly-run inpatient charts). None are scored in this pilot.

4. Precision was not measured by this instrument.

   The pilot measures Falcon's coverage of the coder's queries. Whether every Falcon suggestion merits a clinician's attention is a precision question that requires a clinician review pass, and it is not measured here.

5. Clinician response is a mechanism, not a measured result.

   The reported uplift depends on clinicians acting on point-of-care suggestions. A suggestion raised during writing is more likely to be acted on than a query raised months later, but this pilot did not measure response rates, and no response rate is claimed.

6. Scope of the product lane.

   Falcon addresses documentation specificity at the point of care. Retrospective error detection, including over-coding review, is a separate capability; a companion retrospective audit product (Horizon) exists and is outside the scope of this pilot.

What this pilot changed in our development plan

A pilot that only confirmed what we already believed would not have been worth running. Most of the value of this exercise to us is in the misses, and this corpus has changed what we build next. The items below are set out so a reader can judge the direction of the product and not only its present state.

1. Reading records exactly as they are written.

   These records carry a high volume of spelling variation and irregular formatting, considerably more than the material Falcon had previously been developed against. Falcon identifies the exact clinical text a suggestion refers to so it can be highlighted in place, and that link is harder to make reliably when the underlying text is inconsistently written. The results in this case study were produced on a build where a failure to make that link does not prevent the suggestion being shown, so the figures here are unaffected by it. What this pilot changed is that tolerance of imperfect source text is now a design requirement rather than an assumption. The standard we are working to is that the quality of typing in a record has no bearing on what a clinician is shown.

2. Detail that exists only in prose.

   Section 07 records this as the systematic gap. Falcon is strongest where a finding has a concrete hook in the record, such as a medication, a result, or an existing diagnosis line, and weakest where the only evidence is a sentence of narrative. Several of the misses in this case study are of exactly that kind, and they include findings that carry classification weight. This is the largest single improvement available to the product. The independent coder's queries from this pilot give us a precise specification to build against, which is something we did not have before this exercise.

3. Consistency between readings.

   As section 07 sets out, Falcon does not return an identical set on every reading of the same record, which is why every figure here is a frequency rather than a single output. Raising that consistency means a clinician sees the strongest findings on the first reading rather than on a later one. It is being addressed directly, and it lifts the floor of the results reported here rather than the ceiling.

4. Records of this scale.

   Two of the inpatient records are larger than anything in our earlier testing, large enough to exceed limits in our own capture layer. Records of this size are now a defined requirement rather than an assumption, which is a change this pilot caused.

5. Tailoring answer sets to the patient.

   The options offered against an order should reflect who the patient is. On a neonatal record, choices drawn from general dermatology are less useful than choices pitched at a newborn, and the same applies to sex-specific conditions. The date of birth is available to Falcon, but neonatal dates of birth are recorded inconsistently between hospitals and between fields within the same record, so we are making the patient's age and sex explicit inputs to answer-set generation rather than leaving them to be inferred.

None of these ask anything of the hospital. They are ours to close, and they arrive as improvements to the product.

From shadow pilot to formal pilot

A shadow pilot of this kind is a measurement exercise, not a deployment. Where a provider chooses to take it further, the path is the same four steps.

1. Technical session with hospital IT.

   A working session between the provider's IT team and the Healthful AI team to cover integration, hosting, and security questions.

2. Further records.

   The measurement approach in section 03 applies unchanged to a larger corpus, and a larger corpus strengthens the frequencies reported here.

3. Test environment access.

   Access to a test environment for the provider's own teams, as a subsequent step following the technical session.

4. Conversion to a formal pilot.

   Subject to the outcome of the steps above, conversion of the shadow pilot into a formal pilot on the provider's own records. Scope, timeline and terms are set out separately.

Evidence behind the findings

This annex records what each headline claim rests on, and lists the captures from the clinician-facing screen reproduced in the body. It is provided so that the claims can be checked rather than taken on trust.

A.1 How the evidence was captured

1. Server logs, not screenshots

   Every figure here is taken from Falcon's server logs. Each chart is bound to its logged run by a cryptographic hash of the record text, so a result cannot be attributed to the wrong record. A run that could not be tied to its exact source record was excluded rather than approximated.

2. Repeated runs

   Each chart was run repeatedly, nine to eleven times, and every finding is reported as a frequency over those runs. Two inpatient records (the older adult intensive care admission and the older adult general medicine episode with critical care) exceed a storage limit in the capture layer and are reported from a single fully captured run each; that exception is disclosed in section 03 and marked in the results tables.

3. Screen corroboration

   Alongside the logged runs, one attended run per chart was captured as a series of screenshots of the clinician-facing application, 269 images in total, each bound to its chart by the patient banner visible in the frame. These were used to confirm that what the logs record is what a clinician sees. Section A.4 reports the result of that comparison.

A.2 What each claim rests on

A.3 The exhibits

Three captures from the clinician-facing application are reproduced in the body, each placed with the finding it evidences. All are from the attended runs described in A.1 and all records are de-identified. In each image the left pane is the record as the clinician sees it, with the text Falcon has keyed highlighted; the right pane lists Falcon's suggestions; and the panel at lower right is the question as it reaches the clinician, with its answer options and a single Submit action. The patient banner across the top of each capture has been redacted for publication. The note date in Exhibit 1 has also been redacted. No other part of any frame has been altered.

A.4 The screen agreed with the logs

Because the figures here are drawn from server logs, the screenshots were used to test whether the logs and the clinician's view agree. Across the attended charts, every suggestion recorded in the logs was also present on the clinician-facing screen. On 14 of the 21 charts compared the two matched exactly. On the remaining seven the screen additionally showed cards generated from terms previously approved elsewhere in the same organisation, which are not counted anywhere in this case study. There was no chart on which a logged suggestion failed to appear.

Answer options are assessed on whether they are clinically plausible for the case, not against what the record happens to document, and they are not counted in the coverage figures in section 04.