Mega Benchmark: 500 Queries Across 11 Knowledge Graphs
500 queries. 11 knowledge graphs. 138 million nodes. 1.22 billion edges. One graph, one query language, one server.
Samyama Graph Enterprise v1.0.0 runs the full 500-query mega benchmark across eleven independently-sourced knowledge graphs loaded into a single federated graph — on a single commodity EC2 spot instance.
Headline
500 / 500 pass (100%) — 468 full-pass + 32 data_dependent zero-row matches whose corpus-gap reasons are recorded in the oracle (NCT bridge absent, drug-name casing, no Depression cohort in the OMOP slice, etc.). 0 errors. 0 unexplained empties.
- Total runtime 78 min; query runtime 28 min; load ~50 min
- Six prefixes hit 100% including all three cross-KG categories (XK, PH, MB)
Reproduction caveat: this result was obtained by importing per-KG
.sgsnapfiles via the binary’s startup load path. The HTTP/api/snapshot/importendpoint is capped at 2 GB and cannot ingestpubmed-v2.sgsnap(11 GB) ortrifecta-pubmed.sgsnap(12 GB). Reproducing via the HTTP path requires the body-limit fix tracked in samyama-graph.
The Dataset
| Knowledge Graph | Source | Nodes | Edges | Key Entities |
|---|---|---|---|---|
| PubMed/MEDLINE | NLM | 66.2M | 1.04B | Article, Author, MeSHTerm, Chemical, Journal, Grant |
| Clinical Trials | AACT/ClinicalTrials.gov | 7.8M | 27M | ClinicalTrial, Intervention, AdverseEvent, Site, Sponsor |
| Pathways | Reactome | 119K | 835K | Protein, Pathway, Complex, Reaction, GOTerm |
| Drug Interactions | DrugBank, ChEMBL, SIDER, DGIdb | 245K | 388K | Drug, Gene, SideEffect, Indication, Bioactivity |
| FAERS | FDA Adverse Events | 10.4M | 90M | AdverseEventCase, Reaction, Drug, Outcome |
| UniProt | EBI | 618K | 3.9M | Protein, Organism, GOTerm |
| OMOP | MIMIC-IV (115K patients) | 51.9M | 54M | Person, Visit, ConditionOccurrence, DrugExposure, Measurement |
| Surveillance | WHO SPAR | 217K | 241K | Country, CapacityIndicator |
| Health Determinants | World Bank + WHO | 286K | 286K | Country, SocioeconomicIndicator, EnvironmentalFactor |
| Health Systems | WHO | 20K | 19K | Country, HealthWorkforce, VaccineCoverage |
| NCT Bridge | AACT study_references | — | 747K | REFERENCED_IN (Article → ClinicalTrial) |
| Total | 137.7M | 1.22B |
How the engine clears the workload
Three execution patterns carry the bulk of the 500 queries:
- Adjacency-aware aggregation (ADR-017) —
count(neighbor)over a degree-bounded match set walks per-node adjacency lists rather than scanning every edge. Without this, top-N degree queries time out on PubMed-scale data. - Streaming
WITH … LIMIT—WithBarrierstops scanning upstream after N rows when noORDER BY/WHERE/DISTINCTfollows. Pipeline stages no longer materialize their full input when the next stage only needs a few rows. - Cypher three-valued logic —
IS NOT NULL AND <bool>short-circuits per spec; chainedWHEREclauses AND-combine cleanly. The query author can write what they mean without the engine silently dropping predicates.
Together these shift the bottleneck from engine correctness to corpus coverage. The expected_rows.yaml oracle records each data_dependent empty with a specific corpus reason — so the remaining work is loader, not engine.
What data_dependent covers (32 queries)
These return zero rows but are expected to — the dataset, not the engine, is the constraint:
| Class | Count | Examples |
|---|---|---|
| Drug-name casing | 7 | FAERS uses 'ASPIRIN', DrugBank uses 'Aspirin' — pre-rewrite queries assume one form |
| Loader coverage gaps | 14 | :CODED_AS_DRUG, WaterResource, Bioactivity, HealthWorkforce, SPAR C01 not yet emitted |
| Cohort gaps in slices | ~6 | OMOP slice has no Depression cohort; some MIMIC-IV strata absent in the 115K-patient slice |
| NCT bridge absences | ~5 | Articles without REFERENCED_IN edges to ClinicalTrials in the AACT study_references snapshot |
Infrastructure
| Instance | r7i.16xlarge (64 vCPU, 495 GB RAM) |
| Region | AWS ap-south-1 (Mumbai), spot pricing |
| Disk | 500 GB gp3 |
| Peak memory | ~299 GB (60% utilization) |
| Load time | ~50 min (11 snapshots + NCT bridge + indexes via startup-load path) |
| Query runtime | 28 min |
| Total runtime | 78 min |
| Build | Samyama Graph Enterprise v1.0 |
Reproducing
# On r7i.16xlarge ap-south-1, AMI ami-0d219aaceb19e2c84
./target/release/examples/unified_benchmark \
--pubmed-snap ~/snapshots/pubmed-v2.sgsnap \
--ct-snap ~/snapshots/clinical-trials.sgsnap \
--pw-snap ~/snapshots/pathways.sgsnap \
--faers-snap ~/snapshots/faers-full.sgsnap \
--uniprot-snap ~/snapshots/uniprot.sgsnap \
--omop-snap ~/snapshots/omop-115k.sgsnap \
--di-snap ~/snapshots/druginteractions.sgsnap \
--surv-snap ~/snapshots/surveillance.sgsnap \
--hd-snap ~/snapshots/health-determinants.sgsnap \
--hs-snap ~/snapshots/health-systems.sgsnap \
--study-refs ~/study_references.txt \
--queries ~/benchmark-queries
Snapshots are public at s3://samyama-data/snapshots/. Query CSVs and the raw run output live in this repo under src/data/benchmark/ (benchmark-v100-results.csv).
Hero Run — 187.5M nodes, 1.29B edges, 99% pass (2026-04-30)
Two weeks after the 500/500 mega-benchmark, we ran a larger sweep: a 1000-query corpus against a 187.5M-node / 1.29B-edge graph stitched from the same 11 KGs at full scale (PubMed at 100%, FAERS at full extract, OMOP via Synthea-generated synthetic patients).
| Nodes | 187.5M |
| Edges | 1.29B |
| Queries | 1000-query corpus |
| Pass rate | 99% |
| Instance | AWS r7i-class spot, ap-south-1 |
| Total cost | ~$1.85 |
| Build | Samyama Graph Enterprise v1.0 (pre-merge, hero branch) |
Two engine fixes landed during the run (2026-05-01) and are now in main:
- String-fn NULL-property handling (a small surfaced gap in 3VL coverage).
- Snapshot multi-label index rebuild after import (closed a latent bug when bridge-loading multiple snapshots into one tenant).
The hero run also produced our first cross-KG entity dedup in production — ChEMBL ↔ UniProt joined on the accession property via ADR-019’s import_tenant_with_dedup path.
The hero run is the workload that validates the v1.0 architectural decisions at the scale we care about. The 500-query mega benchmark is the correctness floor (must hit 100% on a curated set); the 1000-query hero run is the stress ceiling (pass rate under real-world noise on a >1B-edge graph).