- https://arxiv.org/pdf/2311.07509.pdf
- Source: arxiv
TL;DR
An enterprise-domain benchmark shows that adding an ontology-based knowledge graph layer over SQL triples GPT-4 question-answering accuracy from 16% to 54% in zero-shot settings.
Summary
Enterprise SQL databases are hard for LLMs to query accurately because schemas alone carry no business semantics. This benchmark uses an insurance-domain enterprise schema paired with an ontology-based knowledge graph to measure how much semantic context affects accuracy. Testing GPT-4 with zero-shot prompting, direct SQL achieves only 16% accuracy while posing the same questions over a KG representation reaches 54%, demonstrating that semantic enrichment is a decisive factor independent of model capability.
Key contributions
- Introduces a benchmark using a real enterprise insurance schema to evaluate LLM question-answering accuracy.
- Demonstrates a threefold accuracy improvement (16% → 54%) by adding a KG layer over the same raw SQL data.
- Shows that ontology and R2RML mappings alone — without fine-tuning — supply sufficient semantic context for GPT-4.
- Establishes that popular text-to-SQL benchmarks underrepresent real enterprise complexity.
When to cite
- When arguing that knowledge graph enrichment is more impactful than model upgrades for enterprise Q&A accuracy.
- When supporting claims that schema structure alone is insufficient context for LLM-based SQL querying.
- When motivating the use of ontology-to-SQL mappings (OBDA/R2RML) as a production engineering asset.
- When benchmarking LLM readiness on domain-specific enterprise data with realistic query complexity.
LLM Knowledge Graph Prompt Engineering RAG Knowledge Graph Prompting for Multi-Document Question Answering Unifying Large Language Models and Knowledge Graphs - A Roadmap