Multi-Agent Knowledge Graphs

Leonard Weise

I wanted to better understand how multi-agent AI systems automatically enrich knowledge graphs. Not to become a top-tier expert, but to gain an appreciation for the abstractions I use every day — especially as AI research papers flood in at ever-increasing rates.

This post highlights what I've learned about building and orchestrating multiple specialized agents to handle tasks like document retrieval, entity extraction, schema alignment, and more. It's written for engineers new to knowledge graph enrichment, but who are already comfortable with concepts like large language models and AI pipelines.

Knowledge Graph Enrichment

A knowledge graph captures the relationships between entities (e.g., technical concepts, authors, publications) and provides a foundation for reasoning or advanced analytics. The challenge is constant maintenance and scalability:

Key Challenges
  1. High Volume of New Research: With hundreds of new papers appearing daily, manually integrating or validating each data point is laborious.
  2. Complex Terminology: AI research often introduces novel concepts, relationships, and contradictions.
  3. Evolving Data: Findings in one subfield can drastically change how existing information should be interpreted.

In a traditional pipeline, you might try single-agent solutions that parse documents, perform summarization, and update the knowledge graph. However, large tasks like conflict resolution or schema alignment often need specialized logic¹. That's where multi-agent systems shine.

Introducing Multi-Agent Systems

Multi-agent means exactly what it sounds like: multiple AI agents, each with distinct roles, collaborating to complete an overarching objective. Recent work from Peking University's Biomedical AI and Computer Technology department² highlights a nine-agent architecture for automated knowledge graph enrichment:

Nine-Agent Architecture Overview

  1. Controller Agent

    • Coordinates the entire process
    • Receives inputs, delegates tasks, and aggregates outputs

  2. Retrieval Agent

    • Fetches large volumes of new content (PDFs, HTML)
    • Focuses on data ingestion efficiency

  3. Segmentation Agent

    • Splits documents into smaller sections
    • Filters irrelevant segments by referencing existing graph context

  4. Summarization Agent

    • Produces concise text from relevant sections
    • Preserves the relationships between entities

  5. Entity Extraction Agent

    • Detects and normalizes domain-specific entities (e.g., novel AI methods)

  6. Relationship Extraction Agent

    • Maps out connections between entities
    • Uses multi-label classification for overlapping relationships

  7. Schema Alignment Agent

    • Ensures new entities and relationships conform to the existing ontology
    • Flags items requiring ontology expansion

  8. Conflict Resolution Agent

    • Identifies and handles contradictions
    • Uses LLM-driven debate and evidence review to decide next steps

  9. Evaluator Agent

    • Validates final results before permanently updating the knowledge graph
    • Scores factors like coherence and relevance

Why Not Fewer Agents?

Understanding the Need for Multiple Agents

You might wonder: Why not collapse this system into fewer agents? Single or fewer-agent solutions often fall short when faced with the specialized demands of large-scale scientific updates — particularly conflict resolution and schema alignment. Researchers found that omitting any one specialized agent decreased overall performance. Even modest dips in metrics lead to more errors in final knowledge graph quality. The overhead of an extra agent is quickly offset by better accuracy and reduced confusion down the pipeline.

Hierarchical vs. Self-Learning Agents

Hierarchical Approach

A central "controller" orchestrates each specialized agent in a predefined sequence. This is simpler to implement, though less flexible.

Self-Learning Approach

These architectures aim to evolve new agent behaviors on-the-fly in response to emerging complexities or newly introduced domains. Higher adaptability but more complex debugging.

Aside: Modular Upgrades

One appealing aspect of multi-agent systems is that each agent's prompt or code can be upgraded without disrupting everything else. Suppose the "Relationship Extraction Agent" needs to incorporate advanced multi-hop reasoning for robotics. You can rewrite its prompt engineering or training approach while leaving the other agents intact.

Example "Agent" Definition (Simplified Pseudocode)

View Code Example
# Define prompts and config for a specialized agent.
class RelationshipExtractionAgent:
    def __init__(self, llm, rules):
        self.llm = llm
        self.rules = rules

    def extract_relationships(self, text_segment):
        # Possibly a structured LLM prompt call
        # e.g., "Identify all cause-effect and part-of relationships."
        llm_input = create_prompt(text_segment, self.rules)
        return self.llm.process(llm_input)

# Additional steps like scoring, conflict detection, etc.

Each agent orchestrates a piece of your pipeline. The controller agent coordinates them based on your knowledge graph's state and real-time logic, reminiscent of a microservices architecture in traditional software engineering.

Conflict Resolution

When a new paper contradicts a known relationship in the graph, the "Conflict Resolution Agent" triggers a sub-workflow:

Conflict Resolution Workflow
  1. Identify: Pinpoint the conflicting assertions
  2. Debate: Engage a short LLM-driven discussion using domain prompts and external references to gauge credibility
  3. Decide: Either reject the new claim, refine it, or revise existing graph nodes
  4. Log & Update: Note the changes for transparency and future auditing

This approach ensures the knowledge graph remains consistent and up-to-date. Attempting such logic with a single agent typically leads to partial or overlooked conflicts.

Evaluation & Integration

Once all subtasks — from retrieval to conflict resolution — are complete, the "Evaluator Agent" calculates scores for final acceptance.

Evaluation Metrics
  • Confidence: Probability that the extracted facts are correct
  • Clarity: Whether the new data is interpretable within the current graph schema
  • Coherence: Overall consistency of the newly added subgraph with existing knowledge

Only segments meeting specific thresholds get integrated into the main knowledge graph. The rest can be stored in a "quarantine" area, pending manual or further automated review.

Key Takeaways

  • Scalability: Handling daily volumes of AI research becomes more manageable when tasks are broken down

  • Specialization: Agents designed for distinct subtasks (e.g., summarization, schema alignment) collectively outperform monolithic approaches

  • Conflict Handling: Multi-agent systems systematically detect and resolve contradictory findings before they pollute the graph

  • Modular Updates: Each agent can be individually improved, speeding iteration and reducing integration risk

It's fascinating to see how dividing a large, intricate task into specialized subtasks, each powered by a domain-adaptive LLM, transforms an unmanageable update pipeline into a structured, consistent knowledge graph enrichment engine.

Getting Started Tip: If you're intrigued by building these systems, consider experimenting with a small pipeline. Even starting with two or three specialized agents can quickly validate whether multi-agent orchestration fits your knowledge graph needs.

Footnotes

¹: In practice, conflict resolution demands specialized domain knowledge. A single summarizing or classification agent rarely suffices.

²: Biomedical AI College of Computer Technology, Peking University. This multi-agent approach was demonstrated on 1,200 domain papers spanning genomics, proteomics, and metabolomics.