documentation

How it works

Complete reference for Telegram Search Engine: architecture, configuration, pipeline commands, API endpoints, and how search, graphs, and scoring work.

# Overview

A self-hosted engine that discovers, analyzes, ranks, and maps public Telegram channels. It reads public channels with a residential IP, classifies them with a local LLM, scores by quality + activity + influence + freshness, and builds an interactive graph of how they reference each other.

Open source, fully self-hosted. One docker compose up runs the whole stack. Use as a data layer for structured Telegram data, or run the web UI as a discovery product.

This demo is read-only over a frozen snapshot of tech-community channels.

# Architecture

Two deliberately separated layers:

Pipeline (crawl → analyze → graph): runs on your machine with residential IP + GPU. Writes to Postgres.
Serving layer (web + API + search): runs anywhere via Docker, reads only.

YOUR MACHINE (residential IP + GPU)
  ingestion (Telethon)  ──┐
  analysis  (Ollama)    ──┼──► Postgres ──► API (FastAPI) ──► web (Next.js)
  graph     (networkx)  ──┘       └─────► Meilisearch

Pipeline reads public channels with Telethon, stores in Postgres, analyzes with Ollama, computes graph metrics with networkx. Serving layer reads Postgres / Meilisearch, exposes HTTP API, serves web UI.

# Configuration

Set environment variables in .env (dev) or .env.prod (production).

Required

TG_API_ID (int) — Telegram app ID from my.telegram.org

TG_API_HASH (string) — Telegram app hash from my.telegram.org

TG_SESSION_STRING (string) — Saved Telegram session (from --print-session)

DATABASE_URL (string) — PostgreSQL connection (e.g., postgresql://user:pass@localhost/tg_db)

Telegram Crawl

TG_MIN_DELAY_SECONDS (float, default 4.0) — Min delay between API calls

TG_JITTER_SECONDS (float, default 3.0) — Random jitter added to delay

TG_MESSAGES_PER_CHANNEL (int, default 40) — Posts sampled per channel

TG_MAX_CHANNELS_PER_RUN (int, default 50) — Safety cap per run

TG_ALLOW_JOIN (bool, default false) — **Must be false.** Crawler refuses to join.

LLM & Search

OLLAMA_BASE_URL (string, default http://localhost:11434) — Ollama endpoint

OLLAMA_MODEL (string, default llama3.1:8b) — Model name

MEILI_URL (string, default http://localhost:7700) — Meilisearch endpoint (blank = use Postgres FTS fallback)

MEILI_MASTER_KEY (string) — Meilisearch admin key

API Server

API_HOST (string, default 0.0.0.0) — Bind address

API_PORT (int, default 8000) — Port

CORS_ORIGINS (string) — Comma-separated allowed frontend origins. Empty in dev (allows all); set in production.

# Pipeline Commands

All commands run from repo root with venv active. From your machine with residential IP + GPU.

0. Capture Telegram Session (one-time)

python -m app.ingestion.crawl --print-session

Interactive login. Copy printed session string into TG_SESSION_STRING in .env.

1a. Crawl by Keywords

python -m app.ingestion.crawl --keywords phones addis crypto jobs

Search each keyword, deduplicate by tg_id, sample ~40 messages, store in Postgres, enqueue discovered refs into frontier (depth 1).

• Caps at TG_MAX_CHANNELS_PER_RUN

• Records edges (t.me links, @mentions, forwards)

1b. DB-driven Keyword Expansion

python -m app.ingestion.crawl --from-db --max-queries 20 --min-age-hours 24

Generate bases × modifiers, pick due queries (crawled >24h ago), run them, record in keyword_runs. Enables iterative discovery without re-crawling.

• --max-queries: limit this run (default 20)

• --min-age-hours: skip recent crawls (default 24.0)

1c. Seed a Known Channel

python -m app.ingestion.add_channel solodevchronicles
python -m app.ingestion.add_channel https://t.me/SoloDevChronicles
python -m app.ingestion.add_channel @channel1 https://t.me/channel2

Add channels to frontier at depth 0. Picked up by next --link-graph crawl. Accepts usernames, @handles, t.me URLs.

2. Link-graph Discovery

python -m app.ingestion.crawl --link-graph --max-depth 2 --limit 30
python -m app.ingestion.crawl --link-graph --limit 5 --messages 1000

Drain frontier queue: resolve candidates, sample messages, extract edges, enqueue refs up to max-depth.

• --link-graph: drain frontier instead of keyword search

• --max-depth: max hops from seed (default 2). Children at max-depth don't harvest further.

• --limit: max candidates to process (default 30)

• --messages: override TG_MESSAGES_PER_CHANNEL (high value for deep history on seeds)

3. Analyze with LLM

python -m app.analysis.run --limit 200

Pull un-analyzed channels, classify with Ollama, compute quality/activity/freshness scores, upsert analysis, mirror into Meilisearch.

• --limit: max channels per run (default 50)

• Non-fatal if Meili down — Postgres is source of truth

4. Compute Graph Metrics

python -m app.graph.metrics

Build networkx directed graph from edges, compute PageRank/betweenness/Louvain clusters, write to channel_graph, rescore all channels with influence, re-sync Meili.

• No arguments

• Recomputes: final_score = quality·40% + activity·30% + influence·20% + freshness·10%

5. Backfill Edges

python -m app.graph.backfill_edges

Rebuild edges from stored messages (no Telegram API calls). Extracts t.me links and @mentions from message text. Useful after bulk imports.

6. Reindex Search

python -m app.search.reindex

Bulk-load all analyzed channels from Postgres into Meilisearch. Use once after setup or to rebuild index.

7. Run API Server

uvicorn app.api.main:app --reload --port 8000

Start read-only FastAPI server. All endpoints are GET. CORS restricted to CORS_ORIGINS.

# API Reference

Base: http://localhost:8000. All endpoints are read-only GET.

GET /health

Health check.

curl http://localhost:8000/health
# {"status": "ok"}

GET /search

Search channels by query. Typo-tolerant ranking via Meilisearch (or Postgres FTS fallback).

q (required, string): search query

limit (optional, int, default 20, max 100): result count

curl "http://localhost:8000/search?q=crypto&limit=10"
curl "http://localhost:8000/search?q=phones+addis&limit=50"

GET /channel/{channel_id}

Full channel detail: metadata, sample messages (20), analytics, all scores.

channel_id (required, path, int): database ID

curl http://localhost:8000/channel/123

GET /categories

List all categories with channel counts.

curl http://localhost:8000/categories

GET /stats

Pipeline statistics: total channels, analyzed, frontier status.

curl http://localhost:8000/stats

GET /graph

Channel reference graph: nodes + edges. Paginated by score, optionally filtered by cluster.

limit (optional, int, default 250, max 1000): max nodes

cluster_id (optional, int): filter to Louvain cluster

curl "http://localhost:8000/graph?limit=500"
curl "http://localhost:8000/graph?limit=250&cluster_id=5"

GET /graph/hubs

Most influential channels (highest PageRank).

limit (optional, int, default 20, max 100): count

curl "http://localhost:8000/graph/hubs?limit=50"

GET /graph/bridges

Channels bridging communities (highest betweenness centrality).

limit (optional, int, default 20, max 100): count

curl "http://localhost:8000/graph/bridges?limit=20"

GET /graph/clusters

Louvain communities with aggregate stats and top channels.

curl http://localhost:8000/graph/clusters

# How Search Works

Search ranks via Meilisearch (typo tolerance, word/proximity), with automatic fallback to Postgres FTS if Meili is down or unconfigured.

Meilisearch Mode

• Indexes: title, username, summary, category, why_recommended

• Ranking: text relevance (fuzzy match + word/proximity) first, then final_score tie-breaker

• Channels pushed automatically by analyzer; bulk reindex available

Postgres FTS Fallback

• Used if MEILI_URL blank or Meili unreachable

• Full-text search via GIN indexes (title, username, summary)

• Ranked by ts_rank, then final_score

• Slightly slower but ensures search always works

# How Graph Works

Channel-to-channel references become weighted directed edges. Graph metrics compute influence, bridges, and communities.

Edge Types

• t.me links: explicit channel references

• @mentions: channel usernames in posts

• forwards: message forwarded from another channel (live crawls only)

Computation

• PageRank: centrality/hub score

• Betweenness: bridges communities (on undirected projection)

• Louvain: community detection (undirected projection); falls back to connected components

• In/out-degree: reference counts

# How Scoring Works

final_score = quality·40% + activity·30% + influence·20% + freshness·10%

quality (Ollama): channel usefulness. Spam penalized; informative content scores higher. Range 0–100.

activity (messages): volume, image ratio, low repetition. Range 0–100.

influence (PageRank): normalized network centrality. Not member count. Range 0–100.

freshness (timestamps): recency of newest sampled message. Range 0–100.

Computed by analyzer on first analysis. Graph metrics recompute influence and refresh final_score for all channels. Scores synced into Meili for search ranking.

# Safety / ToS

Crawler is intentionally read-only and throttled. Never joins, honors rate limits, runs on dedicated aged account on residential IP.

Bans come from behavior (mass-join, ignoring limits), not from reading public channels carefully. Keep crawl volume modest. Store session string for fast swaps.

✓ Read-only (never joins)

✓ Throttled by construction (TG_MIN_DELAY_SECONDS + TG_JITTER_SECONDS)

✓ Residential IP + dedicated account

✓ TG_ALLOW_JOIN must be false (enforced)

# Deployment

See DEPLOY.md in repo for single-VPS Docker deploy (Caddy + HTTPS), MIGRATE_DB.md for moving data into dockerized Postgres, and server-infra/README.md for multi-project reverse proxy setup.

git clone github.com/your/telegram-search-engine
cp .env.prod.example .env.prod
# fill in: TG_API_ID, TG_API_HASH, TG_SESSION_STRING, DATABASE_URL, etc.
docker compose --env-file .env.prod up -d --build

Full Reference on GitHub ↗