How Sokrates Understands Data
A walkthrough of the Sokrates data pipeline for people who know what APIs and databases are but don’t spend their weekends drawing system architecture diagrams.
The Problem
Every business runs on dozens of software systems — an ERP, a CRM, a helpdesk, accounting software, HR tools. Each one stores data in its own format, its own database, behind its own API. If you want to ask a question that spans two systems (“which customers who filed support tickets last month also have overdue invoices?”), you’re writing custom integration code for each pair of systems.
That’s fine with 2 systems. With 10 it’s 45 integrations. With 20 it’s 190. It doesn’t scale.
What if an AI agent could understand the shape of any system’s data — what entities exist, how they relate, what actions are possible — and then reason across all of them through a single interface?
That’s what Sokrates builds.
The Library Card Catalog
Imagine you have 20 libraries, each with a different catalog system. Some use Dewey Decimal, some use alphabetical, some use a weird custom spreadsheet. You can’t search across them.
What you want is a master catalog that doesn’t hold the books themselves, but knows:
- What books exist in each library (the entities — Customer, Invoice, Ticket)
- How they relate to each other (Customer has many Invoices)
- What you can do with them (create an order, close a ticket)
- What the rules are (an invoice can’t exceed the credit limit)
That master catalog is what we’re building. The source systems keep their data. We extract and unify the map.
Phase 1: Reading the Schema — “What Shape Is This Data?”
Business systems describe their data structure in several standard formats:
| Format | Who Uses It | What It Says |
|---|---|---|
| OpenAPI (REST) | Most modern web APIs | ”I have a /customers endpoint that returns objects with fields name, email, phone” |
| JSON Schema | Configuration files, document stores | ”A Customer object must have a name (string) and may have an email (string)“ |
| DDL (SQL) | Relational databases | ”CREATE TABLE customers (id INT, name VARCHAR, email VARCHAR)“ |
| GraphQL Schema | Modern APIs with flexible querying | ”type Customer { name: String!, email: String, orders: [Order!]! }” |
These are all saying the same thing in different languages. Hyle (the ingestion pipeline) reads any of these formats and normalizes them into a single internal representation. A Customer from Salesforce’s REST API and a Customer from a PostgreSQL database both become the same kind of object.
The heavy lifting is done by a code generator called DMCG (datamodel-code-generator) that parses these schema formats and produces Python type definitions. The coverage is broad — OpenAPI, JSON Schema, GraphQL, and DDL between them cover essentially every way modern software describes its data.
Phase 2: Classification — “What Kind of Thing Is This?”
This is where it gets interesting. Not all data objects are the same kind of thing. Sokrates classifies every schema into one of four ontological primitives:
| Primitive | What It Means | Examples |
|---|---|---|
| Entity | Things that exist | Customer, Product, Employee, Department |
| Process | Things that happen | Order, Payment, Shipment, Approval |
| Law | Things that constrain | CreditLimit, ReturnPolicy, ApprovalThreshold |
| Observation | Things that were noticed | LineItem, LogEntry, Measurement, SupportTicket |
An AI classifier looks at each schema — its field names, its relationships — and tags it. A table with OrderDate, ShippedDate, Status is obviously a Process. A table with CompanyName, Phone, Address is obviously an Entity.
Why does this matter? Because when an AI agent later asks “what business processes involve this customer?”, the graph already knows which nodes are processes vs. entities. It’s not doing string matching — it has semantic understanding of the data’s role in the business.
Phase 3: Loading the Map — “Put It in the Graph”
Each classified schema becomes a node in a Neo4j knowledge graph with:
- Its fields as properties
- Its ontological type (Entity/Process/Law/Observation) as a label
- Its relationships to other nodes as graph edges
After processing an ERP’s API specification, the graph might look like:
(Customer:Entity) ──[HAS_MANY]──> (Order:Process) ──[CONTAINS]──> (OrderLine:Observation)
──[SHIPS_VIA]──> (Shipment:Process)
Do that for 10 different source systems and you have a unified knowledge graph that maps the entire data landscape of a business.
The Query Surface — Eidos
Eidos is the API that sits in front of the graph. It provides:
- A unified query language — one way to search across all sources (text search, semantic similarity, label filtering, relationship traversal)
- An MCP endpoint — so AI agents can directly query the graph as a tool
When a user asks the AI assistant “what data do we have about customers?”, the agent calls Eidos, which searches the graph across all ingested sources and returns: “Salesforce has Customer with 12 fields, the ERP has Client with 8 fields, and the helpdesk has Contact with 5 fields — here are the relationships between them.”
Source Systems Hyle Pipeline Neo4j Graph AI Agent
───────────── ───────────── ─────────── ────────
ERP (OpenAPI) ──┐
CRM (OpenAPI) ──┤ Parse → Classify → Unified "What do we
DB (DDL) ──┼──→ Transform → Load ──→ Knowledge ←── know about
Helpdesk (API) ──┤ Graph customers?"
GraphQL svc ──┘ ↑
Eidos API (query + MCP)
We’re not moving data. We’re mapping structure. The source systems keep their data. We extract the shape — what exists, how it relates, what it means — into a graph that an AI can reason over. When the AI needs the actual data, it knows which system to ask and what fields to expect.
Beyond Static Maps: Hyperedges
Everything above gives you a static map of data structure. Useful, but real business questions aren’t about structure — they’re about patterns that emerge from the data itself:
- “Which purchase orders over 500k ISK cross more than two departments before reaching a budget holder?”
- “Which approval chains are bottlenecks this quarter?”
- “Where do cross-department processes create friction?”
You can’t answer these by looking at the schema. You need to actually query the data and let the answer define the relationship. That’s where hyperedges come in.
Edges vs. Hyperedges
In a normal graph, an edge connects exactly two nodes: Alice → Bob (“reports to”). Simple.
A hyperedge connects an arbitrary set of nodes. Think of it as drawing a circle around a group of things and saying “these belong together.” The circle around “all purchase orders, departments, and budget holders involved in approval bottlenecks” is a hyperedge — it connects dozens of nodes simultaneously.
Queries as Identity
Here’s the critical insight: you don’t store the circle. You store the question that draws it.
Think of it like a saved search in your email — “unread messages from this week with attachments.” The result changes every day. You don’t store the list of emails; you store the search criteria. Re-run it tomorrow and you get tomorrow’s answer.
In Sokrates, a hyperedge definition is literally a SQL query stored as a node in the graph:
Name: "bottleneck_approval_chain"
Type: Law (a constraint/rule)
Query: SELECT po.*, d.*, r.*
FROM purchase_orders po
JOIN departments d ON ...
WHERE po.value > d.approval_threshold
AND chain_length > 2
Run this query against the actual data and the result set is the hyperedge — the group of nodes connected by this pattern. Run it next quarter and the membership changes because the data changed. There’s no sync process, no stale cache — the computation is the truth.
Where DuckDB Fits In
This creates a natural split between two kinds of work:
| Neo4j (graph database) | DuckDB (analytical database) | |
|---|---|---|
| Stores | Schemas, relationships, hyperedge definitions, ontological types | Actual row data from source systems (CSV imports, API extracts, database dumps) |
| Good at | ”Trace the approval chain from this PO to the budget holder" | "Find all orders over 500k ISK, group by department, compute averages” |
| Think of it as | The map | The territory |
Graph databases excel at traversing relationships but struggle with analytical queries over large tables. “Find all orders over 500k grouped by department” is a SQL question, not a graph question. DuckDB is purpose-built for exactly that kind of analytical work — it’s an embedded columnar database that runs in-process and handles millions of rows effortlessly.
A generating query bridges the two: the definition lives in Neo4j (as a hyperedge node), but the SQL it contains executes against DuckDB where the actual data lives. Results flow back into the graph as hyperedge membership.
The Three Layers
Layer 0 — Ground Facts (Hyle + DuckDB)
Raw data from source systems. Purchase orders, invoices,
employees, departments. Loaded into DuckDB tables.
Schema mapped into Neo4j via the Hyle pipeline.
↓ Generating queries execute SQL against DuckDB
Layer 1 — Hyperedges (stored in Neo4j as Law nodes)
Named computations. "bottleneck_approval_chain" is a SQL
query whose result set IS the hyperedge membership.
Can be MATERIALIZED (cached, refreshed on schedule)
or VIRTUAL (computed fresh on demand).
↓ Higher-order queries reference Layer 1 by name
Layer 2 — Composed Hyperedges
Patterns of patterns. "cross_department_friction" references
"bottleneck_chains" from Layer 1. An AI agent can WRITE
new Layer 2 definitions — the knowledge graph grows
by learning new queries.
Self-Healing
When underlying data changes (an ERP updates its schema, new transactions come in), you don’t need a reconciliation process. Re-evaluate the generating queries and the graph reflects current reality. This property — called fixed-point evaluation in the theory — means the system self-heals by design. There’s no drift to detect because the queries are the truth.
The AI Agent Writes New Queries
This is where Layers 1 and 2 get powerful. The AI agent doesn’t just run queries — it writes new ones based on patterns it discovers. “I notice that every purchase order over 500k ISK traverses four departments but only needs two.” That observation becomes a new generating query, which becomes a new hyperedge, which becomes a new fact that other queries can reference.
The knowledge graph grows not by importing more data, but by learning new questions to ask about the data it already has.
The Full Picture
┌─────────────────────────────────────────────────────────────────────┐
│ Source Systems │
│ ERP (OpenAPI) · CRM (REST) · DB (DDL) · Helpdesk (GraphQL) │
└──────────┬──────────────────────────────────────────────────────────┘
│ Schema specs (OpenAPI, JSON Schema, GraphQL, DDL)
▼
┌─────────────────────────────────────────────────────────────────────┐
│ HYLE — Schema Ingestion Pipeline │
│ │
│ Parse ──→ Generate types ──→ Classify ──→ Transform ──→ Load │
│ (DMCG) (Pydantic models) (AI agent) (AST rewrite) (Neo4j) │
└──────────┬──────────────────────────────────────────────────────────┘
│
┌─────┴─────┐
▼ ▼
┌─────────┐ ┌────────────────────────────────────────────────────────┐
│ DuckDB │ │ Neo4j Knowledge Graph │
│ │ │ │
│ Layer 0 │ │ Layer 0: Schema nodes (Entity, Process, Law, Obs.) │
│ Ground │ │ Layer 1: Hyperedge definitions (generating queries) │
│ facts │ │ Layer 2: Composed hyperedges (patterns of patterns) │
│ (rows) │ │ │
└────┬────┘ └───────────────────────────┬────────────────────────────┘
│ SQL queries execute │ Graph traversal
│ against DuckDB data │ and relationship
└──────────┬───────────────────────┘ reasoning
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ EIDOS — Unified Query API + MCP │
│ │
│ Semantic search · Label filtering · Relationship traversal │
│ Hyperedge evaluation · MCP endpoint for AI agents │
└──────────────────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ AI Agent (Hermes) │
│ │
│ Answers questions across all data sources │
│ Discovers patterns → writes new generating queries │
│ The knowledge graph grows by learning │
└─────────────────────────────────────────────────────────────────────┘
Why This Matters for the Business
For customers: They connect their existing systems (ERP, CRM, helpdesk) to a Sokrates appliance sitting on their network. The appliance maps their data landscape, classifies it, and provides an AI agent that can answer questions across all their systems — without any of their data leaving their premises.
For scale: Adding a new data source means feeding its schema to Hyle. No custom integration code. The ontological classification and hyperedge machinery work the same regardless of whether the source is a REST API, a SQL database, or a GraphQL service.
For intelligence: The system doesn’t just store what it’s told — it learns. Hyperedges that start as analyst-defined queries can be discovered by AI agents examining the data. The knowledge compounds over time.
The traditional approach to business intelligence is: extract data from 10 systems into a data warehouse, write reports, run them weekly. When the data changes, the reports are stale until next refresh.
Sokrates inverts this. The queries themselves are the knowledge. They live in the graph as first-class objects. They self-heal when data changes. And the AI can write new ones. The map and the territory stay in sync — by design, not by maintenance.