Generative AI in the Real World: Tom Smoker on Getting Started with GraphRAG
Do you really need a formal graph? Or is structure all you need?
Join Ben Lorica and Tom Smoker for a discussion of GraphRAG, one of the hottest topics of the last few months. GraphRAG goes a step beyond RAG to make the output of language models more consistent, accurate, and explainable. But what is a graph? A graph is a way of structuring data. In the end, it’s the structure that’s important, along with the work you do to create that structure.
Check out other episodes of this podcast or the full-length version of this episode on the O’Reilly learning platform.
Learn faster. Dig deeper. See farther.
About the Generative AI in the Real World podcast: In 2023, ChatGPT put AI on everyone’s agenda. In 2024, the challenge will be turning those agendas into reality. In Generative AI in the Real World, Ben Lorica interviews leaders who are building with AI. Learn from their experience to help put AI to work in your enterprise.
Timestamps
- 0:00: Introduction
- 0:15: GraphRAG is RAG with a knowledge graph. Do you have a more strict definition?
- 1:00: A lot of what I do is the R in RAG: retrieve. Retrieval is better if you have structured data. I’ve yet to find a definition for GraphRAG. You want to bring in structured data.
- 2:03: At the end of the day, the lesson is structure. Sometimes structure is an SQL database. Don’t lose hope if you don’t have a knowledge graph.
- 2:49: A knowledge graph is a knowledge base and a list of axioms (rules). The knowledge base is just a word connected to another word through a third word. Fundamentally, the benefit comes from the list of triples. The value is in having extracted and defined those triples.
- 4:01: What are your favorite examples of GraphRAG in production?
- 4:06: A new application is a veterinary radiology example. Without GraphRAG, the LLM kept recommending conditions specific to Labradors not bulldogs. GraphRAG controlled the problem.
- 5:02: The underlying data was almost exclusively text. It’s difficult to build up a consistent dataset for veterinary radiology because animals move.
- 5:44: What are the key bottlenecks? How do I get a knowledge graph?
- 5:51: The biggest question is: Do you need a graph in the first place? There’s a whole spectrum. It’s in most people’s interest to stop before they get to the end.
- 6:28: For people who come to us brand-new, we say, “You should try vector RAG first. If that doesn’t work, there’s a lot of good that structuring data can provide.”
- 7:14: If the chunks are structured, and a lot of the work is done up front, then it’s possible to navigate through structured information. At that point, you get value out of vector RAG.
- 8:28: Academic papers have to follow a certain structure. If you spend time making sure you know what the chunks are, where they’re split and why, and they’re labeled, you can get a lot of value.
- 8:49: What are some of your pointers about how to get started?
- 9:28: The knowledge base is often a compressed representation. That means less tokens. That means better rate limits and less cost. So some people want a graph to help scale. That’s one start. Another is the desire for a system to be explainable. Getting that information into a structured representation and tracing back that structured representation can be very useful.
- 10:24: The first pass is extracting in a consistent way. Pydantic and JSON are often a good first step. You can prompt an LLM and ask it to extract for you.