RAG in .NET: answer questions from your own documents with an LLM

Short answer: retrieval-augmented generation (RAG) means finding the few passages of your documents that are relevant to a question and putting them into the prompt, with an instruction to answer only from them and cite them. The model doesn't need to have been trained on your data; it reads it at request time. A RAG system has three steps: index (split documents into chunks and make them searchable), retrieve (find the best chunks for a question) and generate (answer from those chunks, or say you don't know).

A real example: "ask the terms page"

A shop has purchase terms, a refund policy and FAQ pages. Customers ask the same questions by email: "If I renew early, do I lose the remaining days?" The answer is in the terms, but nobody reads terms.

A general model doesn't know this shop's rules, and if you ask it anyway it may produce a confident, generic answer that is wrong for this shop. With RAG, the system finds the paragraph about renewals, gives it to the model, and the model answers "No, renewing adds a full year to the current expiry date [1]", with a citation you can show.

The three steps

Index. Split each document into chunks of a few sentences to a few paragraphs, with a little overlap so an idea cut at a boundary still appears whole in one chunk. Store each chunk with where it came from (document, heading, URL).

Retrieve. Turn the question into the same representation as the chunks and find the nearest ones. In production this is usually embeddings: a model turns each chunk into a vector of numbers so that texts with similar meaning are close together, and a vector store finds the nearest vectors. Many systems combine this with keyword search (hybrid search), because embeddings can miss exact codes and names.

Generate. Put the retrieved chunks in the prompt, clearly delimited, and instruct the model to answer only from them, cite them, and say "I don't know" when they don't contain the answer.

Runnable code

This example keeps retrieval in plain C# (a small TF-IDF index) so it runs with nothing but the SDK; the comments show where embeddings slot in. Create a console app, dotnet add package Anthropic, set ANTHROPIC_API_KEY, and replace Program.cs:

using Anthropic;
using Anthropic.Models.Messages;

// 1. Index: split documents into overlapping chunks (done once, offline).
string[] documents =
[
    "Refunds: if the app does not run and support cannot fix it, the full amount is refunded within 7 days of approval.",
    "Licenses are valid for 365 days. Renewing adds a full year to the current expiry date, so remaining days are not lost.",
    "Orders are approved by hand after the card-to-card transfer is matched, usually the same day and within 72 hours at most.",
    "DiskWatch never deletes a file without confirmation. Keep backups of important data before cleaning up.",
];
var index = new KeywordIndex(documents.SelectMany(d => Chunker.Split(d, maxWords: 40, overlapWords: 8)));

// 2. Retrieve: find the chunks closest to the question.
var question = "If I renew two weeks early, do I lose those two weeks?";
var hits = index.Search(question, top: 2);

// 3. Generate: answer from the retrieved chunks only, and say so when they don't cover it.
var context = string.Join("\n", hits.Select((h, i) => $"<source id=\"{i + 1}\">{h}</source>"));
AnthropicClient client = new();
var response = await client.Messages.Create(new MessageCreateParams
{
    Model = "claude-opus-5",
    MaxTokens = 1024,
    System = "Answer only from the <source> blocks. Cite sources like [1]. If they don't contain the answer, say you don't know.",
    Messages = [new UserMessageParam { Content = $"{context}\n\nQuestion: {question}" }],
});

Console.WriteLine(response.Content.Select(b => b.Value).OfType<TextBlock>().First().Text);

static class Chunker
{
    public static IEnumerable<string> Split(string text, int maxWords, int overlapWords)
    {
        var words = text.Split(' ', StringSplitOptions.RemoveEmptyEntries);
        for (var start = 0; start < words.Length; start += maxWords - overlapWords)
        {
            yield return string.Join(' ', words.Skip(start).Take(maxWords));
            if (start + maxWords >= words.Length)
            {
                yield break;
            }
        }
    }
}

/// <summary>A tiny TF-IDF retriever. In production, swap it for embeddings in a vector store; the RAG flow stays the same.</summary>
sealed class KeywordIndex
{
    private readonly List<(string Text, Dictionary<string, double> Vector)> _chunks;
    private readonly Dictionary<string, double> _idf;

    public KeywordIndex(IEnumerable<string> chunks)
    {
        var tokenized = chunks.Select(c => (Text: c, Terms: Tokens(c))).ToList();
        _idf = tokenized.SelectMany(c => c.Terms.Distinct()).GroupBy(t => t)
            .ToDictionary(g => g.Key, g => Math.Log(1.0 + tokenized.Count / (double)g.Count()));
        _chunks = tokenized.Select(c => (c.Text, Vectorize(c.Terms))).ToList();
    }

    public IReadOnlyList<string> Search(string query, int top)
    {
        var q = Vectorize(Tokens(query));
        return _chunks.OrderByDescending(c => Cosine(q, c.Vector)).Take(top).Select(c => c.Text).ToList();
    }

    private Dictionary<string, double> Vectorize(IEnumerable<string> terms) =>
        terms.GroupBy(t => t).ToDictionary(g => g.Key, g => g.Count() * _idf.GetValueOrDefault(g.Key));

    private static double Cosine(Dictionary<string, double> a, Dictionary<string, double> b)
    {
        var dot = a.Sum(kv => kv.Value * b.GetValueOrDefault(kv.Key));
        var norm = Math.Sqrt(a.Values.Sum(v => v * v)) * Math.Sqrt(b.Values.Sum(v => v * v));
        return norm == 0 ? 0 : dot / norm;
    }

    private static string[] Tokens(string text) =>
        text.ToLowerInvariant().Split([' ', ',', '.', ':', '?', '!'], StringSplitOptions.RemoveEmptyEntries);
}

What matters in it:

• The system prompt is the grounding rule. "Answer only from the sources, cite them, say you don't know otherwise." Without the last part, the model fills gaps with general knowledge.
• <source id="…"> delimiters keep document text clearly separate from your instructions, and give the model something to cite.
• The retriever is replaceable. KeywordIndex.Search is the only thing that changes when you move to embeddings: embed the chunks once, store the vectors, embed the question, and return the nearest chunks.

Moving to embeddings

When keyword search stops being good enough (users ask in different words than the documents use, or in another language), add embeddings:

1. Pick an embedding model. Anthropic doesn't offer one itself; its documentation points to providers such as Voyage AI. Use the same model for chunks and questions.
2. Embed every chunk once and store the vector next to the chunk text and its source.
3. Store vectors where you already have data if you can: SQL Server 2025 adds a vector type, PostgreSQL has pgvector. A dedicated vector database is worth it at larger scale.
4. Keep keyword search as well and merge the results (hybrid). Codes like AB12CD34EF are found by keywords, not by meaning.

Measuring before tuning

Build a list of 20 to 50 real questions with the chunk that should answer each. Measure two things separately: did retrieval return the right chunk (in the top 3 or 5), and was the answer correct and grounded in it. Most bad answers trace back to retrieval, so fix chunking and search before you rewrite the prompt.

When to use RAG, and when not

Use it when

• Answers must come from your own documents, policies or records.
• The data changes; re-indexing is cheap compared to retraining.
• You need citations users or auditors can check.

Skip it when

• The documents fit comfortably in the context window and don't change often: put them all in the prompt (with prompt caching) and skip retrieval.
• The question needs a calculation or a live lookup: that's a tool call (see agents and tool calling) or an MCP server (see MCP in C#), not retrieval.

More context on where this fits: the AI engineering roadmap.

Want "ask our documents" in your product, built on your existing .NET stack? See how I can help.