A few months ago, during an enterprise GenAI discussion, someone told me:

I paused and thought about the words I used to answer, as the response should not be considered against ChatGPT, because I’ve seen this exact assumption derail multiple GenAI initiatives.

Early on, even I underestimated how tricky this part really is.

That’s when I learned a critical lesson about RAG.

The Real Problem (From Experience)

Most teams think RAG is a technology problem.

In reality, it’s a thinking problem.

What I saw repeatedly:

• Teams indexed everything
• No one asked what should not be retrieved
• No governance
• No ownership
• And then… surprise: inconsistent answers

The issue wasn’t the model.
The issue was context discipline.

My Shift in Understanding

I used to explain RAG like this:

That explanation is technically correct and practically misleading.

Now I explain RAG like this:

This view has changed how teams design their systems.

What Actually Worked (In Practice)

From real implementations, three things mattered most:

1 - Curated Knowledge Beats Complete Knowledge

Teams that selected high-quality, trusted documents performed better than teams that indexed everything.

Less data --- better answers.

2 - RAG Is a Governance Decision, Not Just Architecture

Who owns the documents?
Who approves updates?
Who is accountable for wrong answers?

If those questions aren’t answered, RAG will fail quietly.

3 - RAG Is a appraoch, not the End Goal

RAG works best when:

• answering questions
• explaining policies
• supporting decisions

But when teams tried to automate actions without additional controls, problems surfaced fast.

RAG is memory, not autonomy.

Where RAG Fits in the Real World (My View)

From what I’ve seen:

• Early teams use RAG to build trust
• Mature teams use RAG to scale knowledge
• Advanced teams combine RAG, Rules & Agents to drive outcomes

This is why RAG sits in the middle of the GenAI maturity journey, not at the beginning or the end.

Personal Takeaway

Today, when someone asks me to “just add RAG,” I don’t start with embeddings or vector databases.

I start by asking three questions:

1. Which documents are allowed to influence decisions?
2. Who owns the truth when answers conflict?
3. What happens when the Gen AI is confidently wrong?

If those questions don’t yet have answers, the technology discussion is premature.

That shift from how to build RAG to when to trust it has saved teams months of rework.

That’s the lesson I learned the hard way.

That's it for this week, thanks for reading!

Reply with your thoughts or your favorite section. Found it useful? Share it with a friend or colleague to expand the AI community.

Until next Saturday,

Kashif

The opinions expressed here are solely my conjecture based on experience, practice, and observation. They do not represent the thoughts, intentions, plans, or strategies of my current or previous employers or their clients/customers. The objective of this newsletter is to share and learn with the community.