Evidence Infrastructure

How We Extract & Organize Evidence

From raw papers to structured, connected, transparent claims. Here's exactly how we do it.

The Pipeline

1

Extract

Parse structure

2

Classify

Study type

3

Understand

Extract claims

4

Profile

Quality signals

5

Link

Evidence network

● Validated   ● Experimental

The Problem

Science runs on attention signals, not evidence signals

Current system rewards:

  • Citations (popularity, not validity)
  • Journal prestige (brand, not rigor)
  • Novelty (interesting, not replicated)

What actually matters:

  • Methodology quality
  • Replication status
  • Statistical power
  • Conflicts of interest

We're building infrastructure to surface evidence signals—extraction, organization, and connection at scale.

Our approach: Build the extraction layer first. Validate it. Quality assessment comes after we've proven we can accurately extract and connect claims.

Why Claims, Not Papers

Papers are containers. A single paper might contain 15 distinct claims—some well-supported, some speculative. Treating a paper as a unit means we can't say "the main finding replicated but the secondary analysis didn't."

Claims are the atomic unit of scientific knowledge. By extracting individual claims, we can:

Track Replications

Know which specific findings have been independently confirmed

Identify Contradictions

Find conflicts between papers at the claim level, not just citations

Build Evidence Graphs

Show how claims support or undermine each other across the literature

Answer Real Questions

"What do we actually know about X?" with claim-level granularity

Step 1: Extract Structure

Validated●○○○○ Easy

We adapt extraction to the source format. No one-size-fits-all approach.

PMC XML

Tier 1 • Preferred

Already structured. Direct parsing:

  • • Authors, affiliations, ORCID
  • • Sections, tables, figures
  • • References with DOIs

No AI needed

PDF

Tier 2 • No XML

Multi-tool extraction pipeline:

  • • GROBID → Structure
  • • Azure Doc Intel → Tables
  • • Cross-reference for accuracy

ML-assisted extraction

Website HTML

Tier 3 • No PDF

Publisher pages, preprints:

  • • Meta tags (citation_*)
  • • JSON-LD / schema.org
  • • Site-specific parsers

bioRxiv, paywalled content

XML→PDF→HTML(fallback order)

Why this matters: Using structured XML when available is more accurate than throwing ML at everything. We use the right tool for the job.

Step 2: Classify Study Type

Validated●●○○○ Moderate

Before extracting claims, we classify the study type. Different designs need different extraction prompts and quality frameworks.

Study Types & Frameworks

Systematic Review / Meta-Analysis

PRISMA, AMSTAR 2, GRADE

Randomized Controlled Trial

CONSORT, Cochrane RoB 2

Prospective Cohort

STROBE, Newcastle-Ottawa

Case-Control

STROBE, Newcastle-Ottawa

Cross-Sectional

STROBE

Case Report / Series

CARE, JBI

Study type is detected from article metadata, methods section keywords, and structural patterns. This determines which extraction prompt and quality framework we use.

View Full Quality Guidelines & Checklists

Step 3: Extract Claims

Validated●●●○○ Challenging

With study type identified, we use a type-specific extraction prompt to extract structured data. A meta-analysis prompt looks for forest plots and heterogeneity; an RCT prompt looks for randomization and blinding.

14-Entity Evidence Schema

v3.0

We use a normalized, versioned data model with field-level provenance tracking. Study → Estimand → Arms → Results → Comparisons, with full audit trail.

View Full Schema Documentation

Anti-Hallucination Rules

  • Mandatory source quotes — every claim requires exact text from paper
  • null for missing data — never infer, guess, or fabricate
  • Tables override text — numerical precision from table cells
  • Provenance tracking — text_only, table_only, or text_and_table

Step 4: Surface Quality Signals

Experimental●●●○○ Challenging

Experimental — Not Yet Validated

Quality assessment requires validation against expert human review. We're building that. Until then, treat these as signals to investigate, not verdicts.

We surface quality indicators from extracted data. You assess what they mean for your use case.

What We Surface

Reporting Completeness

Which checklist items are present/missing

Extraction confidence: Medium

Statistical Details

Effect sizes, CIs, p-values, sample sizes

Extraction confidence: High

Funding & COI

Disclosed sources and conflicts

Extraction confidence: High

Registration Status

Pre-registration on ClinicalTrials.gov, PROSPERO

Extraction confidence: High

Frameworks We Reference

Systematic Reviews
  • • PRISMA 2020 (27 items)
  • • AMSTAR 2 (16 items)
  • • GRADE (certainty)
RCTs
  • • CONSORT (25 items)
  • • Cochrane RoB 2 (5 domains)
  • • Jadad Scale (5 points)
Observational
  • • STROBE (22 items)
  • • Newcastle-Ottawa (9 stars)
  • • ROBINS-I (7 domains)

Evidence Grade (A-F)

Experimental
Grade = (Quality × 0.40) + (Reporting × 0.20) + (Power × 0.20) + (Clinical × 0.20)
A

≥85%

B

70-84%

C

55-69%

D

40-54%

F

<40%

⚠️ These weights are not empirically validated. We're working on calibration against expert assessment.

Quality Signal Categories

Integrity
  • Retraction status
  • GRIM test (math check)
  • Trial registration
  • Paper mill patterns
  • Journal reputation
Statistical
  • Sample size (N)
  • Effect size (ES)
  • Confidence intervals
  • Power estimate
  • P-values
Transparency
  • Funding sources
  • Conflicts of interest
  • Data availability
  • Protocol registration
  • Author disclosures

Important Caveats

  • • GRIM test can flag legitimate papers due to rounding
  • • Paper mill detection uses heuristics with false positives
  • • These are signals to investigate, not automatic verdicts

Explore Quality Frameworks

Interactive checklists with good/bad examples

Step 5: Link to Evidence Network

Experimental●●●●○ Hard

We connect claims across papers—not just citations, but actual evidence relationships. This requires four interconnected capabilities.

Entity Normalization

"aspirin" = "ASA" = "acetylsalicylic acid"

Prerequisite for cross-paper matching

Relationship Detection

14 types: Agreement, Disagreement, Evolution, Building

How claims relate to each other

Discourse Linking

Original → Critique → Response → Replication

The scientific conversation structure

Temporal Tracking

When evidence emerged, velocity, time-in-state

Time is the critical dimension

Why This Matters

Citation Networks

"Paper A cites Paper B"

But does A support or contradict B? When? Unknown.

Evidence Networks

"Paper A's Claim 3 contradicts Paper B's Claim 1 (2023)"

Specific, directional, temporal, meaningful.

Together, these four capabilities transform a pile of papers into a living map of scientific knowledge—showing not just what we know, but how we came to know it and how certain we should be.

Entity Normalization

Experimental●●●○○ Challenging

Paper A says "aspirin." Paper B says "ASA." Paper C says "acetylsalicylic acid." Paper D says "Bayer." These are all the same drug. Without normalization, we can't match claims across papers.

Why This Is Challenging

Ambiguity

"CAD" = coronary artery disease OR computer-aided design?

Granularity

"heart disease" vs "ischemic heart disease" vs "acute MI" — which level?

Evolution

Disease definitions change. "Senility" in 1970 → "Alzheimer's" today

Multiple Ontologies

MeSH, SNOMED, ICD-10, OMIM all carve up disease space differently

Tiers of Solution

Tier 0
Plain Text (Current)
Effort: None

Store raw text as-is. No normalization.

Enables: Basic display. Cannot match "aspirin" to "ASA".

Tier 1
LLM-Suggested Canonical Forms
Effort: Low

During extraction, LLM suggests a canonical form. Store both raw + normalized.

Enables: Consistent naming within our system. "ASA" → "aspirin". No external lookup.

Tier 2
UMLS Concept Linking
Effort: Medium

Use ScispaCy/MetaMap to link to UMLS concepts. Store CUI alongside text.

Enables: Standard medical vocabulary. Cross-paper matching. Synonym expansion in search.

Tier 3
Multi-Ontology + Hierarchies
Effort: High

Link to MeSH, DrugBank, SNOMED, NCBI Gene, UniProt. Include hierarchical relationships.

Enables: "aspirin IS_A NSAID IS_A drug". Cross-domain queries. Full synonym resolution.

Target Ontologies (Tier 3)
DrugBank
Drugs
MeSH/SNOMED
Diseases
NCBI Gene
Genes
UniProt
Proteins

Discourse Linking

Experimental●●●○○ Challenging

Science is a conversation. Paper A makes a claim. Paper B critiques it. Paper A's authors respond. Paper C attempts replication and fails. Paper D does a meta-analysis.

A Scientific Dialogue

Paper A: "We found that X causes Y"Original
Paper B: "We question their methodology..."Critique
Paper A authors: "Our response to the critique..."Response
Paper C: "We failed to replicate..."Replication
Paper D: "Meta-analysis of 5 studies shows..."Synthesis

Without discourse linking, these are just 5 unrelated papers. With it, we can show the full scientific dialogue around a claim.

Claim Lifecycle Tracking

Experimental●●●○○ Challenging

Scientific claims have lives. They're born as hypotheses, tested, replicated (or not), contested, and eventually reach consensus or get superseded. Time is the critical dimension.

Lifecycle States (with Temporal Context)

Hypothesized2018
→
Tested2019
→
Replicated2020-21
→
Contested2023
→
Consensus?

Not just the state—but when each transition happened.

Why Time Is Critical

Age of Evidence

10 supporting studies from 1995 vs 2 contradictions from 2024. Recency matters.

Temporal Ordering

Original → Replication → Contradiction → Meta-analysis. The sequence tells the story.

Evidence Velocity

Stable (no new evidence in 10 years) vs active debate (5 papers this year). How "hot" is this?

Time-in-State

"Contested for 8 years" vs "contested for 2 months." Duration signals uncertainty.

Time to First Replication

2 years = healthy field. 15 years = nobody tried. Replication gap is a red flag.

Recency-Weighted Confidence

Should recent evidence count more? Flag when all support is old? Open design question.

What Temporal Lifecycle Enables

"This claim was consensus until..."

Show when and why understanding changed. Not just current state—the full history.

"Last evidence: 2019"

Flag stale claims. If nothing new in 5+ years, is this settled or abandoned?

"Active debate (12 papers in 2024)"

Show velocity. High activity = unsettled. Low activity = either consensus or dead field.

What We're Building Toward

Honest quality assessment requires validation. Here's our roadmap.

Step 1: Extract Structure

Done

PMC XML + PDF + HTML parsing with provenance tracking

Step 2: Classify Study Type

Done

Automatic detection from metadata and structure

Step 3: Extract Claims

Done

Type-specific prompts with mandatory source quotes

Step 4: Surface Quality Signals

In Progress

Integrity, statistical, and transparency indicators

Step 5: Evidence Network

In Progress

Entity normalization, contradiction detection, discourse linking

Expert Annotations

In Progress

Building ground truth dataset for validation

Accuracy Metrics

Planned

Published precision/recall on extraction

Quality Calibration

Planned

Validate quality signals against expert assessment

Our commitment: We'll publish our validation metrics as we develop them. No black boxes. No unearned authority.

Our Commitment

Open Methodology

This page documents exactly how we process papers. No black boxes.

Honest Uncertainty

We label what's validated vs experimental. No fake rigor.

Traceable Extraction

Every claim links back to source text with provenance.

Validation Transparency

We'll publish accuracy metrics as we develop them.

Format Transparency

We show whether data came from XML parsing or PDF extraction.

Questions about our methodology?

We're building in public. Reach out with questions or feedback.

hello@socratic.science