How to Use n8n with ._Template 151 Rag Documentos Locais Ollama Privado

You have hundreds of internal documents — contracts, SOPs, product specs, support tickets — and every time someone needs an answer buried in them, they either dig manually or paste sensitive text into a public chatbot. The first wastes hours. The second leaks your data to a third party. For a founder handling client contracts or an ops team sitting on regulated records, neither is acceptable. What you need is a system that answers questions from your own documents, runs entirely on infrastructure you control, and never sends a byte to an external API.

The Problem: RAG Without the Privacy Tax

Retrieval-Augmented Generation (RAG) is the standard pattern for "ask questions about my documents." You embed your files into vectors, store them, retrieve the relevant chunks when a question comes in, and feed those chunks to a language model to compose an answer. The problem is that almost every tutorial assumes you'll use OpenAI or Anthropic embeddings and completions — which means your document contents leave your network on every single query.

For most real businesses that's a dealbreaker. Legal documents, financial records, HR files, and customer PII often can't legally or contractually be sent to a US-based inference provider. Even when it's technically allowed, "we send all our contracts to a third-party LLM" is a hard sentence to say to a client during a security review. The result is that teams who most need document search are the ones who can't use the easy version of it.

Template 151 solves this by keeping the entire pipeline local. Ollama runs open-weight models (embeddings and chat) on your own machine or VPS. n8n orchestrates ingestion, retrieval, and generation. Nothing touches an external inference endpoint. You get the RAG experience with zero data egress.

The Solution: n8n + Ollama + a Local Vector Store

The architecture has three moving parts, and n8n glues them together:

• Ollama — serves two models locally over HTTP at http://localhost:11434: an embedding model (nomic-embed-text or mxbai-embed-large) and a chat model (llama3.1 or qwen2.5). Pull them once with ollama pull nomic-embed-text and ollama pull llama3.1.
• A vector store — the template uses the in-memory or Qdrant vector store node. For anything beyond a demo, run Qdrant locally in Docker so your index survives restarts.
• n8n — two workflows: one that ingests documents into the vector store, and one that answers questions against it.

The key insight is that n8n's LangChain nodes let you swap the model provider without rewriting logic. Instead of pointing the embedding and chat nodes at OpenAI, you point them at the Ollama credential. Everything downstream — chunking, retrieval, prompt assembly — stays identical.

Step-by-Step Setup in n8n

1. Configure the Ollama credential. In n8n, create a new "Ollama" credential and set the base URL to http://localhost:11434 (or http://host.docker.internal:11434 if n8n runs in Docker and Ollama runs on the host). Test it to confirm the connection.

2. Build the ingestion workflow. Start with a Manual Trigger (or a Local File Trigger watching a folder). Chain these nodes:

• Read/Write Files from Disk — load your PDFs, .txt, or .md files from a local directory.
• Default Data Loader (Document Loader) — parse the binary into text. For PDFs, set the data type to binary and let the loader extract text.
• Recursive Character Text Splitter — set chunk size to around 1000 characters with a 100–200 character overlap. Overlap matters: it prevents answers from being cut off at chunk boundaries.
• Embeddings Ollama — select your Ollama credential and the nomic-embed-text model.
• Vector Store (Qdrant) in "Insert Documents" mode — point it at your local Qdrant instance and give the collection a name like docs_local.

Run this workflow once per batch of documents. Every file gets chunked, embedded locally, and stored.

3. Build the question-answering workflow. This is where the RAG happens:

• Chat Trigger (or a Webhook, if you're wiring it into an internal tool) — receives the user's question.
• Question and Answer Chain — the LangChain node that coordinates retrieval and generation.
• Attach a Vector Store (Qdrant) node in "Retrieve" mode as the retriever, pointing at the same docs_local collection and using the same Embeddings Ollama model. Set "top K" to 4–6 chunks.
• Attach an Ollama Chat Model node as the language model, using llama3.1.

The chain embeds the incoming question with Ollama, retrieves the most similar chunks from Qdrant, stuffs them into the prompt as context, and asks the local chat model to answer using only that context. The response comes back through the Chat Trigger. No external call anywhere in the loop.

4. Tune the system prompt. In the Q&A chain, override the default prompt to instruct the model: "Answer only from the provided context. If the answer isn't in the context, say you don't know." This single line cuts hallucinations dramatically and keeps answers grounded in your actual documents.

The Benefits: Why Local Wins for Real Businesses

Zero data egress. Your documents never leave the machine. This is the entire point, and it turns "we can't use AI on this data" into "our AI runs where the data already lives." It clears legal, compliance, and client-security-review hurdles in one move.

Predictable cost. There is no per-token bill. Once the hardware is running, you can embed a million chunks and answer ten thousand questions for the cost of electricity. For high-volume internal use, this is often cheaper than an API by an order of magnitude.

No rate limits, no outages. Your RAG system doesn't go down because a provider had an incident or throttled your account. It's yours end to end.

Full control over the stack. Because n8n exposes every step as a node, you can insert filtering, logging, access control, or metadata tagging anywhere in the pipeline — something a black-box hosted RAG service won't let you do.

Common Pitfalls (and How to Avoid Them)

Mismatched embedding models. The single most common failure: you ingest with nomic-embed-text but retrieve with a different model. Vectors from different models are incompatible, so retrieval returns garbage. Use the exact same embedding model and credential in both workflows.

Using the in-memory vector store in production. It's fine for a first test, but it evaporates when n8n restarts. Move to Qdrant (or PGVector) early so you don't re-embed everything after every deployment.

Chunks too large or too small. Huge chunks blow past the model's context window and dilute relevance; tiny chunks lose meaning. Start at 1000 characters with 150 overlap and adjust based on your document type — dense contracts often benefit from smaller chunks, narrative docs from larger ones.

Underpowered hardware for the chat model. Embeddings are cheap, but a 7B–8B chat model wants real RAM and ideally a GPU. On a CPU-only box, expect slow responses. If latency matters, use a quantized model (llama3.1:8b-instruct-q4_K_M) or a smaller model like qwen2.5:3b.

Docker networking. If n8n and Ollama run in separate containers, localhost won't resolve between them. Use host.docker.internal or put them on the same Docker network and reference the container name. This trips up nearly everyone on first setup.

No "I don't know" guardrail. Without an explicit instruction to answer only from context, local models will confidently invent answers. Always include the grounding instruction in your system prompt and set top-K high enough to actually contain the answer.

Set this up once and you have a private, self-hosted knowledge base that any teammate can query in plain language — with the confidence that no document ever left your infrastructure.

Get the n8n Template: RAG Documentos Locais com Ollama Privado →