We drafted a real phase 3 CSR and scored it against the one the sponsor filed.
The study, the sources and the final report are all public, published by the European Medicines Agency. Asthra drafted from the sources; we scored every section side by side against the filed report.
A public filing, with its sources.
The European Medicines Agency publishes the clinical data behind medicines it evaluates. Under its clinical data publication policy, sponsors' Clinical Study Reports are released alongside the documents they were written from: the protocol, the statistical analysis plan and the tables, listings and figures.
That makes a fair test possible. We take the same inputs a medical-writing team started from, hand them to Asthra, and compare what it writes with what the sponsor actually submitted. Nothing is synthetic, and nothing was written with this benchmark in mind.
Skylight 1 · fezolinetant · Astellas
Protocol 2693-CL-0301. A phase 3, randomized, placebo-controlled study: 12 weeks double-blind, followed by a 40-week active extension, of fezolinetant 30 mg and 45 mg once daily for moderate to severe vasomotor symptoms of menopause.
Clinical Study Report v1.0
The filed report, dated 24 Feb 2022. Used only as the answer key. Asthra never sees it.
Clinical study protocol
Protocol 2693-CL-0301 and its amendments: design, eligibility, treatments, assessments.
Statistical analysis plan
The 12-week SAP: endpoints, power calculation, missing-data handling, sensitivity analyses.
Tables, listings and figures
The TLF pack: demographics, disposition, efficacy and safety outputs, SAE narratives.
Draft blind, score side by side.
The filed CSR is the answer key. Asthra never sees it. Every section is scored by reading the two versions against each other.
Pick a public golden
We use a CSR the regulator has already published, with its sources. The sponsor’s filed report becomes the golden: the version a professional medical-writing team actually submitted.
Hide the answer, keep the sources
Asthra gets the protocol, SAP and TLFs, the same inputs a writer starts from. The filed CSR is held back. Asthra plans retrieval, drafts each ICH E3 section, cites to file and page, and runs its own QC.
Extract a fact key from the golden
For every section we pull the facts the golden states: design details, criteria, doses, sample sizes, statistical methods, safety findings. That gives 125 scorable facts across the 23 sections in this run.
Score side by side
Each section of the draft is read against the matching section of the golden. Every fact is marked covered, partial or missed. Every number is checked for an exact match. Every claim is checked for contradiction or fabrication.
Audit the gap flags
Asthra flags missing data instead of inventing it. We check each flag: was the data truly absent from the sources, or did Asthra miss something it had? False gaps count against us.
Trace every miss to a cause
Each missed fact is traced through the run logs to retrieval, drafting or the harness, and becomes a regression test before the next run.
- Coverage
- Share of the golden’s facts for that section that the draft states, scored 0–10. Partial credit for half-stated facts.
- Numeric
- Of the key numbers the draft states, how many match the golden exactly. A rounded or shifted value is a miss.
- Claims
- Starts at 10. Deducted for any statement that contradicts the golden or isn’t supported by the sources.
- Gap flags
- Inline MISSING INFO markers Asthra inserted where it could not find data, each reviewed for whether the data really was absent.
Section-by-section scorecard.
Six sections are fully covered. Two pull the average down, and both failed in the drafting pipeline, not in retrieval. We explain both below.
| ICH E3 section | Facts covered | Coverage | Numbers exact | Claims |
|---|---|---|---|---|
| Title page | 9 / 13 | 3 / 3 | 10 | |
| 7 Introduction | 6.5 / 8 | — | 10 | |
| 9.1 Overall study design | 11 / 12 | 7 / 7 | 10 | |
| 9.2 Discussion of design | 1.5 / 3 | — | 10 | |
| 9.3.1 Inclusion criteria | 4.5 / 5 | 3 / 3 | 10 | |
| 9.3.2 Exclusion criteria | 6 / 6 | 2 / 2 | 10 | |
| 9.3.3 Removal of subjects | 3 / 3 | — | 10 | |
| 9.4.1 Treatments administered | 3 / 6 | — | 10 | |
| 9.4.2 Identity of products | 3 / 5 | 1 / 1 | 10 | |
| 9.4.3 Assignment to groups | 0 / 4 | — | Not drafted | |
| 9.4.4 Dose selection | 2 / 2 | 1 / 1 | 10 | |
| 9.4.5 Timing of doses | 3.5 / 4 | — | 10 | |
| 9.4.6 Blinding | 2.5 / 5 | — | 9 | |
| 9.4.7 Prior & concomitant therapy | 1.5 / 2 | — | 10 | |
| 9.4.8 Treatment compliance | 4 / 4 | 2 / 2 | 10 | |
| 9.5.1 Efficacy & safety measures | 8.5 / 9 | 2 / 2 | 10 | |
| 9.5.2 Appropriateness of measures | 1 / 2 | — | 10 | |
| 9.5.3 Primary efficacy variable | 3 / 3 | 1 / 1 | 10 | |
| 9.5.4 Drug concentration | 2 / 3 | — | 10 | |
| 9.6 Data quality assurance | 3 / 9 | — | 10 | |
| 9.7.2 Sample size | 2 / 2 | 1 / 1 | 10 | |
| 11.4.2.2 Handling of missing data | 5 / 7 | 3 / 3 | 10 | |
| 12.3.2 SAE narratives | 0.5 / 8 | — | 8 | |
| Total | 86 / 125 | 69% · mean 7.1 | 26 / 26 | mean 9.4 |
A dash under “Numbers exact” means the section states none of the key's numeric facts. Sections not drafted in this run (efficacy and safety results, 10.x, most of 11 and 12) are not scored.
What held up, and what didn't.
No invented facts
Across 23 sections, the reviewers found zero fabrications. Where Asthra couldn’t find something, it said so with a gap flag instead of writing a plausible-sounding guess.
Numbers are exact
All 26 key numbers Asthra stated match the filed CSR. In 11.4.2.2, 63 of 63 checkable value groups are exact, and all 19 values in the 9.7.2 power table match the SAP.
It caught an error in the golden
The filed CSR labels its multiple-imputation confidence intervals “95%”. The SAP specifies 90%, and the interval widths equal 1.645 × SE. Asthra wrote 90%, which is correct.
Source conflicts surfaced, not buried
Asthra flagged five genuine conflicts between source documents for reviewer attention, such as two different SDs for the PROMIS sleep score, and resolved each to the value the sources support.
SAE narratives (12.3.2)
Asthra read the narratives on its first pass, but a recovery step rebuilt the section from thin notes and dropped them. Coverage fell to 0.6, and two table errors from a misread source page made it in: one serious adverse event placed in the wrong dose arm, and an OCR typo (“Adenocrine” for apocrine).
Assignment to groups (9.4.3)
All three drafting attempts ended in about a second with no output, and the run didn’t retry. The section scored 0.
Too many gap flags
Of 115 gap-type flags, 15 were real gaps. 62 were prompted by ICH E3 template fields the filed CSR never fills, and 27 were false gaps: the data was in the sources, often found by a neighbouring section.
What happens next
Each miss above is now a regression test. The fixes cover keeping retrieved sources through recovery steps, rejecting status text as a draft, retrying empty results, re-checking malformed tables against the page images, and sharing evidence between sections. Fixing the 12.3.2 path alone is worth about 7 coverage points on that section and removes both errors. The next run adds the efficacy sections (9.7.1 and 11.4.1) so results can be compared across runs.
Run it on your own study.
Point Asthra at your protocol, SAP and TLFs and compare its draft with yours.