Showing posts with label Loki. Show all posts

11/09/2025

Building Your Own Local Grafana AI Assistant with Ollama, Prometheus, and Loki

In this guide, we’ll walk through setting up Grafana’s new LLM-powered Assistant — entirely locally — using open-source components:
Grafana OSS, Ollama for local AI models, Prometheus for metrics, and Loki + Promtail for logs.

Overview

We’ll deploy a local observability stack that includes:

Component	Purpose
Grafana OSS + LLM App	Dashboards + built-in AI Assistant
Ollama	Runs Llama 3 or other open-source LLMs
Prometheus	Metrics collection
Loki + Promtail	Log aggregation
Node Exporter	Host metrics
Grafana MCP (optional)	Connect Grafana data to external AI clients like Claude Desktop

All of this runs locally with docker-compose — no cloud dependencies, no API billing, and complete data privacy.

Step 1 — Download and Extract

Clone the ready-made project:

https://github.com/dhanuka84/grafana-llm-observability

cd grafana-llm-observability

Step 2 — Fix Docker Root for Promtail

Check your Docker data root:

docker info --format '{{ .DockerRootDir }}'

Then create an .env file so Promtail can find container logs:

echo "DOCKER_ROOT=$(docker info --format '{{ .DockerRootDir }}')" > .env

Step 3 — Start the Stack

docker-compose up -d

This launches Grafana, Prometheus, Loki, Promtail, Node Exporter, Ollama, and optionally the MCP server.

Step 4 — Pull a Local Model

docker-compose exec ollama ollama pull llama3

Confirm it’s available:

docker-compose exec ollama ollama list

Step 5 — Configure Grafana LLM App

Open Grafana → http://localhost:3000
Login: admin / admin
Go to Administration → Plugins → LLM → Configuration

Use the following settings:

Field	Value
Provider	Custom API
API URL	http://ollama:11434
API Path	✅ /v1
API Key	any value (e.g. local)
Model (Base & Large)	llama3:latest

Save → you should see
✅ LLM provider health check succeeded!

Step 6 — Check Metrics and Logs

Prometheus: http://localhost:9090
Loki: via Grafana → Explore → Loki → query {job="docker"}

Grafana auto-loads an Observability Starter dashboard that shows Prometheus scrapes and Docker logs.

Step 7 — Enable and Use the AI Assistant

In Grafana, go to Administration → Plugins → LLM
Toggle Enable all LLM features in Grafana
Refresh your browser (Ctrl + Shift + R)

Now you can:

Hover over any panel → ⋯ → Ask AI → Explain this panel
Use Explain query in Explore
Add a persistent LLM Chat panel:
+ Create → Dashboard → Add Visualization → “LLM Chat”

Step 8 — (Advanced) Enable MCP Server

If you’d like to integrate Grafana with external AI clients like Claude Desktop:

Create a Grafana API key (Editor role)

Add it to docker-compose.yml:

Restart MCP:

docker-compose up -d --force-recreate mcp

Visit http://localhost:8000/healthz → ok

Then configure your AI client to use:

SSE URL: http://localhost:8000/sse

Step 9 — Verify Everything

# LLM test

curl http://localhost:11434/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{"model":"llama3:latest","messages":[{"role":"user","content":"Say hi!"}]}'

→ returns Hi!

# Prometheus test

curl http://localhost:9090/metrics | head

Step 10 — Enjoy Your Offline AI Observability Assistant

You now have:

A full local monitoring stack
A private LLM Assistant powered by Ollama
AI-driven insights and explanations directly inside Grafana dashboards

9/21/2025

Hands-On Study Guide: Runtime Security Monitoring with Falco & Grafana Stack

📖 Runtime Security Analogy: The Investigator and the Doctor

This guide explains the distinct but complementary roles of Falco, Falcosidekick, Prometheus, Grafana, and Loki.
To make it intuitive, we’ll use an analogy: a Crime Scene Investigator (CSI), their Dispatcher, their Doctor, and a shared Command Center.

👉 So the distinction is:

Falco => Sidekick (Dashboard + Fanout) => Loki = Incident Reports (logs, JSON events)
Falco => Prometheus => Grafana = Health Reports (numeric metrics)

🔎 Introduction: Two Jobs, One Mission

In any security operation, two functions are equally important:

Investigating threats – detecting malicious or suspicious activity.
Monitoring the health of the investigator – ensuring the security tools themselves are working reliably.

A tired or overworked investigator might miss a crucial clue. Similarly, a misconfigured security tool might miss a real threat.

Our runtime security stack mirrors this exact model. Different tools play different roles, working together to provide both detection and health monitoring.

👥 The Characters

The Crime Scene Investigator (CSI): An expert patrolling the city, spotting crimes based on a rulebook, and writing detailed reports.
The Dispatcher: The operator who instantly forwards the CSI’s reports to multiple destinations.
The Doctor: A physician whose job is to check on the CSI’s health and performance.
The Command Center: The central hub where the Police Chief can view both the crime reports and the investigator’s health charts.

🔄 Mapping the Analogy to the Technology

🕵️ The Crime Scene Investigator = Falco

The City: Your Kubernetes cluster.
Patrolling the Streets: Falco uses an eBPF probe to observe every system call: processes, files, network connections.
The Rulebook: custom-rules.yaml. If an action matches a rule (e.g., “Terminal shell in container”), Falco flags it.
Filing Reports: For each event, Falco generates a JSON log with rich details: who, what, where, when. These are sent via http_output.

📡 The Dispatcher’s Live Feed = Falcosidekick

The Dispatcher: Falcosidekick listens for Falco’s reports.
The Live Feed: The Falcosidekick Web UI (http://localhost:2802) shows a real-time stream of Falco alerts — the best way to see what’s happening right now.
Forwarding Reports: Falcosidekick can send these alerts to multiple systems simultaneously (Loki, Slack, Elasticsearch, etc.).

📁 The Case File Archive = Loki

Forwarding to the Archive: Falcosidekick pushes every Falco report to Loki.
The Archive: Loki stores the full JSON logs, creating a searchable history of every detected security event.
Deep Analysis: Using LogQL, you can slice and dice events, filter by rule, namespace, severity, or process.

🩺 The Investigator’s Doctor = Prometheus

Routine Check-ups: Prometheus polls Falco’s /metrics endpoint.
Vital Signs Collected:

falco_events_total → how many reports Falco has filed.
falco_syscalls_total → how much workload Falco has handled.
up → is Falco alive and responding?

Health Reports: Prometheus stores these as time-series metrics. It doesn’t know the contents of crime reports, only statistics about Falco’s activity.

🖥️ The Unified Command Center = Grafana + FalcoSidekick

Grafana brings both views together for the Police Chief:

Investigation Board (FalcoSideKick):
View Falco’s detailed JSON alerts:

Perfect for forensic analysis.
Health Dashboard (Prometheus):
View charts of Falco’s performance and workload, e.g.:

“Spike in high-priority alerts over the last hour?”
“Is Falco consuming too much CPU?”

✅ Conclusion: Why Separation is Powerful

This Investigator + Doctor model reflects industry best practice:

Loki → for detailed, text-rich, searchable event logs.
Prometheus → for fast, efficient, numerical health metrics.

By keeping logs and metrics separate but unifying them in Grafana, you get the best of both worlds:
🔹 Deep forensic detail for every incident.
🔹 Operational visibility into the reliability of your detection engine.

🛡️ Hands-On Study Guide: Runtime Security Monitoring with Falco & Grafana Stack

1. Prerequisites

A Kubernetes cluster (Minikube recommended: minikube start --cpus=4 --memory=8192)
Helm 3 installed
kubectl installed
Docker installed (for building sample apps)
Checkout the below github repository:

https://github.com/dhanuka84/grafana-prometheus-app-monitoring/tree/falco