Architecture

Single binary Go API serving three domains — Admin, Portal, Data API

System Data Flow

API Server — Three Domains

Data Pipeline

Zero-loss path from exchange feed to client-facing API

Latency Budget

Services & Ports

All services run as Docker containers with idxmdp- prefix

Credentials (Dev Environment)

All credentials below are for local development only

Admin Panel Access

PORT=18090
DATABASE_URL=postgres://admin:secret@localhost:25432/idx_admin?sslmode=disable
REDIS_ADDR=localhost:26379
REDIS_PASSWORD=idxmdp_redis_dev_2026
QUESTDB_HOST=localhost
QUESTDB_HTTP_PORT=19000
CLICKHOUSE_DSN=http://localhost:28123
JWT_PRIVATE_KEY_PATH=secrets/jwt_private.pem
JWT_PUBLIC_KEY_PATH=secrets/jwt_public.pem
JWT_ISSUER=idx-market-data
CHART_LIB_PATH=../charting_library-master
SUPERADMIN_PASSWORD=Admin123456

Docker Compose

All containers use idxmdp- prefix on idxmdp_net network

# Start all services
docker compose up -d

# View parser logs
docker compose logs -f idxmdp-parser

# Check container status
docker ps --filter "name=idxmdp"

# Restart parser
docker compose restart idxmdp-parser

Volume Mounts

Storage & Retention

Hot/cold storage strategy with automatic data lifecycle

Public Endpoints

No authentication required

// Request
POST /v1/auth/token
Content-Type: application/json

{ "api_key": "idx_live_a1b2c3d4..." }

// Response
{
  "ok": true,
  "data": {
    "token": "eyJhbGciOiJSUzI1NiIs...",
    "expires_in": 900,
    "tier": "pro"
  }
}

Response Envelope

All API responses use a standard envelope:

{ "ok": true,
  "data": {...},
  "meta": { "latency_us": 142 } }

{ "ok": false,
  "error": "description",
  "meta": { ... } }

Data API

JWT required — Authorization: Bearer {token}

GET /v1/ohlcv/BBCA?tf=1m&from=2026-03-25&to=2026-03-26
Authorization: Bearer eyJ...

{
  "ok": true,
  "data": [{
    "ticker": "BBCA",
    "date": "2026-03-25T02:00:00Z",
    "bar_time": "09:00",
    "freq": "1m",
    "open": 8900, "high": 8925,
    "low": 8875, "close": 8925,
    "volume": 123456,
    "freq_cnt": 42,
    "hd": 0
  }]
}

Plugin Endpoints

AmiBroker DLL wire protocol — API key in body, not JWT

Streaming (WS/SSE)

Real-time OHLCV bars via WebSocket or Server-Sent Events

// Subscribe
{"action":"subscribe", "tickers":["BBCA","BBRI"], "freq":"1m"}

// Unsubscribe
{"action":"unsubscribe", "tickers":["BBRI"]}

// Incoming bar
{"ticker":"BBCA", "bar_time":"09:15", "open":8900,
 "high":8925, "low":8875, "close":8925,
 "volume":45000, "hd":0}

Tier Limits

TradingView UDF Datafeed

UDF-compatible endpoints for the TradingView charting library

// Success (columnar format)
{ "s": "ok",
  "t": [1711324800, 1711411200],
  "o": [8900, 8925],
  "h": [8950, 8975],
  "l": [8875, 8900],
  "c": [8925, 8950],
  "v": [123456, 98765] }

// No data (with hint)
{ "s": "no_data", "nextTime": 1711238400 }

Authentication

Dual auth model — JWT for API, sessions for Admin/Portal

API Key Lifecycle

{
  "sub": "user-uuid",
  "iss": "idx-market-data",
  "tier": "pro",
  "key_id": "key-uuid",
  "machine_id": "sha256(...)",  // plugin only
  "exp": 1711929600,
  "iat": 1711928700
}

Tier System

Single source of truth in internal/config/tier.go

Tier Override Priority

RBAC & HD Access

Role-based access control + tier-based data filtering

Roles

HD (Hidden Delta) Access by Tier

Monitoring

Real-time observability across the full platform stack

Monitoring Stack

Dashboards

Access Points

# Request metrics
http_requests_total{method, path, status}
http_request_duration_seconds{method, path}

# Rate limiting
rate_limit_hits_total
rate_limit_fail_open_total

# Audit
audit_writes_total
audit_fallback_writes_total

# Plugin
plugin_activations_active
plugin_heartbeat_failures_total

# WebSocket / SSE
ws_connections_active
sse_connections_active

# Start monitoring stack
docker compose -f docker-compose.dev.yml up -d grafana prometheus alertmanager

# Check status
docker ps --filter "name=idxmdp-grafana"
docker ps --filter "name=idxmdp-prometheus"

# View Grafana logs
docker logs idxmdp-grafana --tail 20

# Restart monitoring
docker compose -f docker-compose.dev.yml restart grafana prometheus

Alert Thresholds

HD & RSI Metrics

Proprietary market indicators computed in real-time by the Rust metric worker

HD (Hidden Delta) — 7-Step Pipeline

Gate Times (Configurable via Admin)

RSI (Relative Strength Index)

// Standard RSI-14 on 1-minute close prices
gain = avg(positive_deltas, 14)
loss = avg(negative_deltas, 14)
rs   = gain / loss
rsi  = 100 - (100 / (1 + rs))

Key Decisions

Architecture and design choices with rationale

Project Stages

Scope-driven development stages with phases inside each — not time-boxed sprints

Project Repository

idx-market-data-platform/
  go-api/                      Go/Fiber API server (:18090)
  rust-workers/                Parser + metric worker + aggregator + ch-drain
  docker/                      Docker compose configs + Grafana/Prometheus
  schema/                      PostgreSQL (16 tables) + ClickHouse schemas
  fixture/                     Test fixtures, HD accuracy CSV, replay data
  charting_library-master/     TradingView charting library (licensed)
  docs/                        Tech specs, decisions log, operations guide

Service Architecture Summary

Telegram Alerts Setup

Get notified on your phone the moment a stock matches your criteria

The Screener can deliver alerts directly to your Telegram. This guide walks you through the one-time setup and creating your first alert. Once linked, every alert you create gets delivered to your chat — not a shared channel.

What you'll need

• A Telegram account on your phone
• An account on this platform (login required)
• About 5 minutes

Step 1 — Find your Telegram chat ID

The platform needs to know which Telegram chat to send your alerts to. Find your numeric chat ID:

1. Open Telegram on your phone
2. Search for the bot @userinfobot
3. Tap Start
4. The bot replies with your Id: — a positive integer like 560442208
5. Copy that number — you'll need it in Step 3

Step 2 — Start a chat with the IDX Alerts bot

1. In Telegram, search for @idx_testhink_bot
2. Open the chat
3. Tap Start (or send any message — /start is the convention)

That's it. You don't need to interact with the bot beyond this. It's purely for delivery.

Step 3 — Link Telegram in the dashboard

1. Log into the dashboard
2. Open Screener → Alerts tab
3. In the Telegram delivery card, click Link Telegram
4. A 6-digit code appears (e.g. 451096)
5. In the form below the code, fill in the inputs left-to-right:
   • Left field: your chat ID from Step 1 (e.g. 560442208)
   • Right field: the 6-digit code from above — auto-filled, but verify it matches
6. Click Verify

The card flips to show Linked to chat <your ID> with a green Linked badge. You're set up.

Step 4 — Create your first alert

In the same Alerts tab, scroll down to the Create alert form:

Worked example — notify me when BBRI's RSI drops below 30:

• Ticker: BBRI
• Alert name: BBRI oversold
• Condition: field RSI, operator <, value 30

Click Create alert. The new alert appears in the Active alerts table above with status enabled.

Step 5 — Wait for delivery

The alert worker checks every 10 seconds. When the condition transitions from false to true, a message lands on your phone:

🔔 BBRI
BBCA oversold

The message includes the ticker and your alert name.

How delivery actually works

• Each transition fires once. If RSI drops below 30, you get one message. If it rises back above 30 then drops again, you get another. Mid-condition (already true), no re-fire.
• Quotas: 50 alerts per user.
• Disable temporarily: toggle the alert in the Active alerts list to pause without deleting.
• One ticker per alert: a single alert tracks one ticker. To watch multiple tickers, create one alert per ticker.
• Cross-ticker conditions are not allowed for alerts (e.g. you cannot reference index membership). Use the Screener tab for cross-ticker scans.

Troubleshooting

Stop receiving alerts

• Delete individual alerts from the Active alerts list, OR
• Toggle an alert to disabled to pause it without losing the configuration, OR
• Click Re-link in the Telegram delivery card and verify a different chat ID to redirect alerts elsewhere

IDX Symbol Names Refresh

Regenerate the ticker → company-name map shown in the chart's symbol search

The chart's symbol search displays the full company name next to each IDX ticker (e.g. BBCA → "PT Bank Central Asia Tbk"). That mapping lives in go-api/internal/refdata/idx_symbols.json and is embedded into the API binary at build time. This guide explains how to regenerate the file when new tickers list on IDX or when names change.

When to refresh

• New IPO — ticker shows the bare code instead of company name in symbol search
• Ticker symbol change after merger / corporate action (e.g. EXCL → XLSmart Telecom Sejahtera)
• Quarterly hygiene — KSEI republishes ownership data monthly; pulling a fresh snapshot picks up renames you might have missed

Typical cadence: every 2–3 months, or whenever the screener shows an unmapped ticker that traders ask about.

Why we don't fetch live from IDX

Source of truth: KSEI ownership data

KSEI (the Indonesian central securities depository) publishes monthly stock-ownership snapshots which include the full issuer_name for every listed company. The community repo aryakdaniswara/idx-stock-ownership mirrors these as structured CSV. We use the latest CSV's share_code + issuer_name columns.

Refresh procedure

The whole pipeline is a single bash session — no committed script, since the source URL changes per snapshot. Run from the repo root.

Step 1 — Find the latest KSEI CSV

# List recent CSV files in the source repo
curl -sS "https://api.github.com/repos/aryakdaniswara/idx-stock-ownership/contents/data" \
  | grep '"download_url".*\.csv'

Pick the most recent file (filename pattern kepemilikan_saham_YYYYMMDD.csv) and copy its download_url.

Step 2 — Download and extract unique pairs

# Replace URL with the latest from Step 1
curl -sS "https://raw.githubusercontent.com/aryakdaniswara/idx-stock-ownership/main/data/kepemilikan_saham_YYYYMMDD.csv" \
  -o /tmp/ksei.csv

# Extract unique (ticker, name) pairs from columns 2 + 3
awk -F',' 'NR>1 {print $2"\t"$3}' /tmp/ksei.csv | sort -u > /tmp/ksei_pairs.tsv
wc -l /tmp/ksei_pairs.tsv   # expect ~950

Step 3 — Pull current QDB ticker universe

We only emit entries for tickers that actually trade in our QDB — warrants and delisted codes get the bare-ticker fallback automatically.

curl -sS -G "http://localhost:19000/exec" \
  --data-urlencode "query=SELECT DISTINCT ticker FROM idx_ohlcv ORDER BY ticker" \
  | python3 -c "import json,sys; d=json.load(sys.stdin); [print(r[0]) for r in d['dataset']]" \
  > /tmp/qdb_tickers.txt

# Build the intersection (preserve all KSEI variants per ticker)
awk -F'\t' '
  NR==FNR { ksei[$1] = ksei[$1] "\n" $2; next }
  ($1 in ksei) {
    n = split(ksei[$1], variants, "\n")
    for (i=2; i<=n; i++) print $1"\t"variants[i]
  }
' /tmp/ksei_pairs.tsv /tmp/qdb_tickers.txt > /tmp/intersection.tsv

Step 4 — Title-case + write JSON

python3 << 'PYEOF' > go-api/internal/refdata/idx_symbols.json
import re, json

ticker_to_names = {}
with open('/tmp/intersection.tsv') as f:
    for line in f:
        line = line.rstrip('\n')
        if '\t' not in line: continue
        ticker, name = line.split('\t', 1)
        ticker_to_names.setdefault(ticker, []).append(name)

def pick_canonical(names):
    # Drop sub-class variants like "MVS GOTO ..." in favor of plain "GOTO ..."
    candidates = [n for n in names if not re.match(r'^(MVS|DR)\s', n)]
    if not candidates: candidates = names
    return min(candidates, key=len)

pairs = {t: pick_canonical(ns) for t, ns in ticker_to_names.items()}

def fix_parens(s):
    s = re.sub(r'\(\s+', '(', s); s = re.sub(r'\s+\)', ')', s); return s

ACRONYMS = {
    'PT', 'XL', 'CIMB', 'BNI', 'BCA', 'BRI', 'BTPN', 'BFI', 'AKR',
    'MNC', 'GMF', 'KAI', 'IDX', 'OCBC', 'NISP', 'UOB', 'HSBC', 'IFG',
    'BSI', 'WSBP', 'KB', 'IBK', 'QNB', 'SMBC', 'BTN', 'CBP',
}

def smart_title(s):
    s = fix_parens(s); out = []
    for word in s.split():
        m = re.match(r'^(\W*)(.*?)(\W*)$', word)
        prefix, body, suffix = (m.group(1), m.group(2), m.group(3)) if m else ('', word, '')
        if not body: out.append(word); continue
        upper = body.upper()
        if body.lower() == 'tbk':       out.append(prefix + 'Tbk' + suffix)
        elif upper in ACRONYMS:          out.append(prefix + upper + suffix)
        elif body[0].isalpha():          out.append(prefix + body.capitalize() + suffix)
        else:                            out.append(word)
    return ' '.join(out)

out = {
    "_comment": "IDX ticker -> full company name. Sourced from KSEI ownership data, filtered to tickers actually present in our QDB universe, title-cased. Tickers absent fall back to the bare ticker."
}
for ticker in sorted(pairs):
    cleaned = smart_title(pairs[ticker])
    if not cleaned.startswith('PT '): cleaned = 'PT ' + cleaned
    out[ticker] = cleaned

print(json.dumps(out, indent=2, ensure_ascii=False))
PYEOF

Step 5 — Hand-clean the diff

Smart-title gets ~95% right, but a handful of brand acronyms come out wrong (e.g. XLSmart → Xlsmart). Diff against the previous version and patch the obvious ones in-place:

git diff go-api/internal/refdata/idx_symbols.json | head -100

Common touch-ups:

• Add new acronym to the ACRONYMS set if it appears in >1 company name
• Single-occurrence quirks: just hand-edit the JSON value
• Verify BBCA, TLKM, BMRI, BBRI, GOTO as smoke-check anchors

Step 6 — Rebuild + verify live

docker compose -f docker-compose.dev.yml build api
docker compose -f docker-compose.dev.yml up -d api

# Smoke test: search "BBCA" should return the full company name
curl -sS -b /tmp/cookies.txt "http://localhost:18090/udf/search?query=BBCA&limit=3" \
  | python3 -m json.tool

Expected first result:

{
  "description": "PT Bank Central Asia Tbk",
  "exchange": "IDX",
  "full_name": "IDX:BBCA",
  "symbol": "BBCA",
  "ticker": "BBCA",
  "type": "stock"
}

How the lookup is wired

• File: go-api/internal/refdata/idx_symbols.json — flat { "TICKER": "Full Name" } map plus a leading _comment key.
• Loader: go-api/internal/refdata/refdata.go — //go:embed's the JSON, parses on package init, exposes refdata.CompanyName(ticker).
• Consumer: go-api/internal/handler/udf.go — UDFSearch and UDFSymbolResolve call the helper stockDescription(ticker) which falls back to the bare ticker for unmapped equities.
• Search filter: UDFSearch also matches against the company name — typing “Bank Central” finds BBCA.

Adding a single ticker without a full refresh

If only one or two new tickers need adding (e.g. an IPO this week), skip the pipeline and patch the JSON directly:

{
  …
  "NEWX": "PT New Company Tbk",
  …
}

Then rebuild api. Keep the file alphabetically sorted to keep diffs reviewable.

Troubleshooting

Operations Guide

Start · Stop · Status · Monitoring · Logging — everything you need to run the platform day-to-day

Quick Links

1. Start / Stop

# Start everything (all containers, dependency order)
docker compose -f docker-compose.dev.yml up -d

# Start individual services
docker compose -f docker-compose.dev.yml up -d questdb
docker compose -f docker-compose.dev.yml up -d parser
docker compose -f docker-compose.dev.yml up -d metric-worker

# Stop everything (preserves volumes)
docker compose -f docker-compose.dev.yml down

# Full wipe (removes data volumes — re-apply schemas after)
docker compose -f docker-compose.dev.yml down -v

2. Status Check

# Visual (recommended)
# → http://localhost:18082 — auto-refresh every 10s

# CLI
docker compose -f docker-compose.dev.yml ps
docker logs idxmdp-parser --tail 20
docker logs idxmdp-metric-worker --tail 20
docker logs idxmdp-ch-drain --tail 20

3. Monitoring Commands

Kafka / Redpanda

docker exec idxmdp-redpanda rpk topic list
docker exec idxmdp-redpanda rpk topic consume idx.ohlcv --num 1
docker exec idxmdp-redpanda rpk group describe metric-worker-hd
docker exec idxmdp-redpanda rpk group describe ch-drain

QuestDB (hot storage)

curl -s "http://localhost:19000/exec?query=SELECT%20count()%20FROM%20idx_ticks"
curl -s "http://localhost:19000/exec?query=SELECT%20count()%20FROM%20metrics_hd"

ClickHouse (cold storage)

docker exec idxmdp-clickhouse clickhouse-client --database market_data \
  --query "SELECT count() FROM idx_ticks"

docker exec idxmdp-clickhouse clickhouse-client \
  --query "SHOW TABLES FROM market_data"

Redis (cache)

docker exec idxmdp-redis redis-cli -a idxmdp_redis_dev_2026 GET last:hd:BBCA
docker exec idxmdp-redis redis-cli -a idxmdp_redis_dev_2026 KEYS "last:hd:*"
docker exec idxmdp-redis redis-cli -a idxmdp_redis_dev_2026 DBSIZE

4. Logs & Levels

# Follow all / specific service
docker compose -f docker-compose.dev.yml logs -f
docker compose -f docker-compose.dev.yml logs -f parser

# Last N lines
docker logs idxmdp-parser --tail 50

# RUST_LOG levels (set in docker-compose.dev.yml)
RUST_LOG=idx_parser=info        # default
RUST_LOG=idx_parser=debug       # every skipped message
RUST_LOG=idx_parser=warn        # errors only

All services use Docker json-file driver with rotation: max 50 MB per file × 3 files = ~150 MB per service.

5. Maintenance

# QuestDB retention (keep 14 days)
./scripts/questdb-retention.sh 14 --dry-run
./scripts/questdb-retention.sh 14

# Recommended cron — daily after market close (16:30 WIB)
# 30 16 * * 1-5 /home/testhink/idx-market-data-platform/scripts/questdb-retention.sh 14

# Rebuild after code change
docker compose -f docker-compose.dev.yml build parser
docker compose -f docker-compose.dev.yml up -d

6. Daily Workflow (Trading Day)

07:50   docker compose -f docker-compose.dev.yml up -d
        Open http://localhost:18082 — verify all green

08:30   Pre-market: check logview for SNAP/BID/ASK messages

09:00   Market opens: check TRADE messages in logview
        Dashboard candles building (BBCA, TLKM etc.)
        ops dashboard: metrics_hd row count growing

During  Monitor via ops dashboard (:18082)
        docker logs idxmdp-parser --tail 5
        docker logs idxmdp-metric-worker --tail 5

16:15   Market closes — aggregator flushes open bars
16:30   Run retention: ./scripts/questdb-retention.sh 14

7. Graceful Shutdown & Morning Startup

7.1 Shutdown — Phase 1: Pre-shutdown drain check

Do not start shutdown until you confirm the pipeline is idle (after 16:15 WIB post-trade window).

# 1a. Verify no new ticks are flowing (should be stable between two checks ~10s apart)
curl -s "http://localhost:19000/exec?query=SELECT%20count()%20FROM%20idx_ticks%20WHERE%20ts%20%3E%20dateadd('m',-1,now())"

# 1b. Verify Kafka consumer lag is zero for every group / every partition
docker exec idxmdp-redpanda rpk group describe ch-drain
docker exec idxmdp-redpanda rpk group describe metric-worker-hd
docker exec idxmdp-redpanda rpk group describe metric-worker-rsi
# Every partition must show LAG = 0. If not, WAIT — do not proceed.

7.2 Shutdown — Phase 2: Run daily retention (optional)

./scripts/questdb-retention.sh 14

7.3 Shutdown — Phase 3: Stop writers upstream-first (drain the pipeline)

# 3a. Stop parser FIRST — cuts off new data at the source
docker compose -f docker-compose.dev.yml stop parser

# 3b. Wait ~30s for the aggregator to flush in-memory 1m bars to Kafka
sleep 30

# 3c. Re-verify consumer lag is zero
docker exec idxmdp-redpanda rpk group describe metric-worker-hd
docker exec idxmdp-redpanda rpk group describe ch-drain

# 3d. Stop metric-worker (drains idx.ohlcv → metrics_hd / metrics_rsi)
docker compose -f docker-compose.dev.yml stop metric-worker

# 3e. Stop ch-drain (drains idx.* → ClickHouse)
docker compose -f docker-compose.dev.yml stop ch-drain

7.4 Shutdown — Phase 4: Stop read-side tier

docker compose -f docker-compose.dev.yml stop api dashboard logview ops
docker compose -f docker-compose.dev.yml stop grafana alertmanager prometheus node-exporter redis-exporter

7.5 Shutdown — Phase 5: Snapshot data stores

# 5a. Redis — force a background save so in-memory state hits disk
docker exec idxmdp-redis redis-cli -a idxmdp_redis_dev_2026 BGSAVE
sleep 3
docker exec idxmdp-redis redis-cli -a idxmdp_redis_dev_2026 LASTSAVE

# 5b. ClickHouse — optional lazy merge (skip if in a hurry; merges happen next start)
docker exec idxmdp-clickhouse clickhouse-client --database market_data \
  -q "OPTIMIZE TABLE idx_ticks FINAL" 2>/dev/null || true
docker exec idxmdp-clickhouse clickhouse-client --database market_data \
  -q "OPTIMIZE TABLE idx_ohlcv FINAL" 2>/dev/null || true

# 5c. QuestDB auto-commits its WAL on clean stop — nothing to do

7.6 Shutdown — Phase 6: Stop infrastructure bottom-up

# Redpanda must stop AFTER all consumers are gone (done in §7.3)
docker compose -f docker-compose.dev.yml stop redpanda

# Then storage
docker compose -f docker-compose.dev.yml stop clickhouse questdb redis postgres

# Verify all stopped
docker compose -f docker-compose.dev.yml ps

7.7 Startup — Phase 1: Infrastructure first (T-60min, 08:00 WIB)

cd /home/testhink/idx-market-data-platform

# Bring up storage + Kafka first; wait for healthchecks
docker compose -f docker-compose.dev.yml up -d questdb redpanda clickhouse redis postgres

# Poll until all five are (healthy) — usually <30s
for i in 1 2 3 4 5 6; do
  docker compose -f docker-compose.dev.yml ps questdb redpanda clickhouse redis postgres
  sleep 5
done

7.8 Startup — Phase 2: Smoke-test infrastructure

# QuestDB — row counts persisted from last session
curl -s "http://localhost:19000/exec?query=SELECT%20count()%20FROM%20idx_ticks"
curl -s "http://localhost:19000/exec?query=SELECT%20count()%20FROM%20idx_ohlcv"

# Redpanda — topics present (disk-persisted)
docker exec idxmdp-redpanda rpk topic list

# ClickHouse — tables present
docker exec idxmdp-clickhouse clickhouse-client -q "SHOW TABLES FROM market_data"

# Redis — ping
docker exec idxmdp-redis redis-cli -a idxmdp_redis_dev_2026 PING

# Postgres — users table
docker exec idxmdp-postgres psql -U idxmdp -d idxmdp -c "SELECT count(*) FROM users"

If any check fails, STOP here. Do not start writers on top of broken storage.

7.9 Startup — Phase 3: Start consumers before the producer

# ch-drain + metric-worker first — so they're already consuming
# by the time parser starts firing. Prevents startup lag spikes.
docker compose -f docker-compose.dev.yml up -d ch-drain metric-worker

# Confirm they joined their consumer groups cleanly
docker logs --since 30s idxmdp-ch-drain 2>&1 | tail -10
docker logs --since 30s idxmdp-metric-worker 2>&1 | tail -10

7.10 Startup — Phase 4: API, monitoring, read-side tier

docker compose -f docker-compose.dev.yml up -d api
curl -sf http://localhost:18090/health && echo ' API OK'

docker compose -f docker-compose.dev.yml up -d dashboard logview ops
docker compose -f docker-compose.dev.yml up -d prometheus grafana alertmanager node-exporter redis-exporter

7.11 Startup — Phase 5: Parser LAST (T-30min, 08:30 WIB)

# Parser is the data source — start it last so everything downstream is listening
docker compose -f docker-compose.dev.yml up -d parser

# Watch it connect to RabbitMQ and start consuming
docker logs -f --since 10s idxmdp-parser
# Expect: "RabbitMQ connected", message decode counts climbing

7.12 Startup — Phase 6: Pre-market verification (T-30 to T-0)

# 6a. All containers running and healthy
docker compose -f docker-compose.dev.yml ps

# 6b. Parser metrics flowing
curl -s http://localhost:9464/metrics | grep -E 'idx_parser_itch_bytes_total|idx_parser_messages_total'

# 6c. Consumer groups rejoined with zero lag
docker exec idxmdp-redpanda rpk group describe ch-drain
docker exec idxmdp-redpanda rpk group describe metric-worker-hd

# 6d. Pre-market ticks flowing after 08:45 WIB
curl -s "http://localhost:19000/exec?query=SELECT%20count()%20FROM%20idx_ticks%20WHERE%20ts%20%3E%20dateadd('m',-5,now())"

# 6e. Ops dashboard visual check
# → http://localhost:18082  — all 15+ services green
# → http://localhost:13000/d/pipeline-flow  — Grafana pipeline-flow dashboard

7.13 Startup — Phase 7: Market open (09:00 WIB)

# First trades should arrive within the first minute
docker logs --since 2m idxmdp-parser 2>&1 | grep -i trade | head -5

# 1m bars start building at 09:01
curl -s "http://localhost:19000/exec?query=SELECT%20ticker%2C%20ts%2C%20close%20FROM%20idx_ohlcv%20WHERE%20ts%20%3E%20dateadd('m'%2C-2%2Cnow())%20LIMIT%205"

# Chart at /chart?symbol=BBCA should show live candles updating

7.14 Recovery — if something goes wrong

8. Troubleshooting

Monitoring Guide

Grafana · Prometheus · Alertmanager · Telegram — the authoritative monitoring runbook

1. Scope

In scope: container health, QuestDB ILP throughput, ClickHouse inserts, Redpanda consumer lag, Redis memory & clients, host CPU/disk/mem, Go API RPS/latency/errors, Telegram alert routing.

Out of scope: business metrics, billing state, user analytics, market-data correctness (HD accuracy — see Backfill Guide).

2. Scrape Targets

3. Access Points

4. Start / Verify Stack

# Bring up monitoring
docker compose -f docker-compose.dev.yml up -d grafana prometheus alertmanager \
  node-exporter redis-exporter

# Verify each scrape target is UP
curl -s http://localhost:19090/api/v1/targets | jq '.data.activeTargets[] | {job:.labels.job, health:.health}'

# Check a specific metric is being scraped
curl -s 'http://localhost:19090/api/v1/query?query=up' | jq '.data.result'

5. Dashboards (pre-provisioned)

6. Alert Thresholds

7. Telegram Routing

# Required env vars (set in .env before starting alertmanager)
TELEGRAM_BOT_TOKEN=...
TELEGRAM_CHAT_ID=...

# Test send
docker exec idxmdp-alertmanager amtool alert add test severity=warning \
  --alertmanager.url=http://localhost:9093

8. Common Troubleshooting

Backfill Guide

Recover · Recompute · Restore — the data-recovery swiss army knife

Quick Reference

1. Tiers Rebuilt

All subcommands re-use the live pipeline's own Aggregator and MetricEngine code so output is byte-identical. Every run is idempotent.

2. Binary Location

The backfill binary lives at rust-workers/target/release/backfill after cargo build --release --bin backfill. For production runs, always invoke it inside the parser container (docker exec idxmdp-parser backfill…) so it uses the same network, env vars and config as the live workers. The host-side binary is only for --dry-run validation.

3. Subcommand Reference

3.1 ohlcv — re-aggregate 1-minute bars

# One day, all tickers
docker exec idxmdp-parser backfill ohlcv \
  --from 2026-04-07 --to 2026-04-08

# A single minute, single ticker
docker exec idxmdp-parser backfill ohlcv \
  --from 2026-04-07T09:14:00 --to 2026-04-07T09:15:00 --ticker BBCA

# Dry-run (count ticks, no writes)
docker exec idxmdp-parser backfill ohlcv \
  --from 2026-04-07 --to 2026-04-08 --dry-run

3.2 metric — recompute HD or RSI

docker exec idxmdp-parser backfill metric \
  --engine hd --from 2026-04-01 --to 2026-04-08

docker exec idxmdp-parser backfill metric \
  --engine rsi --from 2026-04-07 --to 2026-04-08 --ticker BBCA

3.3 restore — import CSV into ClickHouse

docker cp /home/testhink/dumps/2026-Q1.csv idxmdp-parser:/tmp/2026-Q1.csv
docker exec idxmdp-parser backfill restore /tmp/2026-Q1.csv

4. Time Format & UTC Gotcha

Accepted forms (all interpreted as UTC, --from inclusive, --to exclusive):

• YYYY-MM-DD — e.g. 2026-04-07 (midnight UTC)
• YYYY-MM-DDTHH:MM:SS ISO — 2026-04-07T09:14:00
• YYYY-MM-DD HH:MM:SS — 2026-04-07 09:14:00

IDX trades on WIB (UTC+7). Subtract 7 hours before passing to backfill.

5. Runbook A — Recover a missing minute bar

# 1. Detect (QuestDB)
SELECT ticker, count() FROM idx_ohlcv
WHERE ts >= '2026-04-07T09:14:00.000Z'
  AND ts <  '2026-04-07T09:15:00.000Z'
GROUP BY ticker;

# 2. Dry-run
docker exec idxmdp-parser backfill ohlcv \
  --from 2026-04-07T09:14:00 --to 2026-04-07T09:15:00 --dry-run

# 3. Run
docker exec idxmdp-parser backfill ohlcv \
  --from 2026-04-07T09:14:00 --to 2026-04-07T09:15:00

# 4. Verify — both tiers should now match

6. Runbook B — Recompute HD/RSI for a full day

# 1. Detect gaps
SELECT count() FROM metrics_hd
WHERE ts >= '2026-04-07T02:00:00.000Z'
  AND ts <  '2026-04-07T09:30:00.000Z';

# 2. Dry-run (watch for "Warm-up complete: N bars replayed")
docker exec idxmdp-parser backfill metric \
  --engine hd --from 2026-04-07 --to 2026-04-08 --dry-run

# 3. Run
docker exec idxmdp-parser backfill metric \
  --engine hd --from 2026-04-07 --to 2026-04-08

# 4. Verify
SELECT symbol, count(), min(ts), max(ts)
FROM metrics_hd WHERE ts IN '2026-04-07'
GROUP BY symbol LIMIT 10;

Note: IN '2026-04-07' is QuestDB shorthand for the entire UTC day.

7. Runbook C — Restore ClickHouse from CSV

docker cp /home/testhink/dumps/idx_ohlcv-2026Q1.csv idxmdp-parser:/tmp/restore.csv
docker exec idxmdp-parser backfill restore /tmp/restore.csv --dry-run
docker exec idxmdp-parser backfill restore /tmp/restore.csv

# Then re-run Runbook B to rebuild dependent metrics

8. Runbook D — Repair HD/RSI after a metric-worker cold-start

Use this when an unexpected restart leaves metric-worker running with empty in-memory state — the symptom is HD value=0.0 (or RSI stuck at 50.0) for every bar of the affected day across all symbols. Root cause: metric-worker started before ClickHouse was ready to serve queries, so warm-up logged warm-up failed: ... — starting cold and the engine never seeded.

1. Detect — confirm the cold-start case (not a real flat market):

-- QuestDB. zero_cnt ≈ total_cnt means the whole day is bogus.
SELECT count(*) AS total_cnt,
       sum(case when value=0 then 1 else 0 end) AS zero_cnt,
       count(distinct symbol) AS syms
FROM metrics_hd WHERE ts >= '2026-05-07';

# Cross-check metric-worker startup log
docker logs idxmdp-metric-worker --since 24h | grep -E "warm-up failed|warm-up: [0-9]+ bars replayed"

A warm-up failed: ... line on the most recent restart confirms the diagnosis.

2. One-command fix:

# today (UTC)
./scripts/repair-metrics.sh

# a specific past day
./scripts/repair-metrics.sh 2026-05-07

The script does four steps (not three — the first version of this runbook missed step 4, leaving the chart showing huge |delta| bars at the day boundary because ClickHouse still held the cold-start zeros while QuestDB had been repaired):

1. Restarts idxmdp-metric-worker and waits for warm-up: N bars replayed (rebuilds in-memory state from ClickHouse).
2. Drops the bad day's partition in QuestDB metrics_hd and metrics_rsi.
3. Re-runs backfill metric --engine hd and --engine rsi for that day — both subcommands do their own warm-up before processing, so output values are continuous with the prior day. Writes to QuestDB only.
4. Runs sync-metrics-hd-to-ch.py and sync-metrics-rsi-to-ch.py to mirror the corrected rows into ClickHouse metrics_hd / metrics_rsi. Charts at 1D+ resolution query CH (not QDB), so without this step the histogram shows tall green+red bars at the boundary where CH has stale zeros while QDB is clean.

3. Verify — pick a known symbol and confirm continuity:

-- QuestDB. Today's first value should equal yesterday's last value
-- (first bar of a new day inherits prev-day frozen DMAPO).
SELECT ts, value FROM metrics_hd
WHERE symbol = 'PIPA' AND ts >= '2026-05-06' AND ts < '2026-05-08'
ORDER BY ts;

9. Troubleshooting

Where to look first

docker logs --since 1h idxmdp-parser
docker logs --since 1h idxmdp-metric-worker
docker logs --since 1h idxmdp-ch-drain

# QuestDB console
# → http://localhost:19000

# ClickHouse client
docker exec -it idxmdp-clickhouse clickhouse-client --database market_data

# Prometheus (backfill + aggregator counters)
# → http://localhost:19090/graph?g0.expr=aggregator_bars_emitted_total
# → http://localhost:19090/graph?g0.expr=backfill_rows_written_total

Data Pipeline Flow

End-to-end flow from IDX ITCH feed to client dashboards and AmiBroker plugin

Stage Map

Topic & Table Matrix

Retention

Partition TTL runs as a cron job after market close. See Operations Guide §5.

Parser Tech Spec

Rust ITCH parser — 9-thread pipeline, zero-alloc decoder, 3.1M msg/s

Architecture — 9 Thread Pipeline

RabbitMQ ──▶ Consumer (x2, prefetch=500)
                  │
                  ▼
           Parser threads (x1, zero-alloc decoder)
                  │
        ┌─────────┼──────────┐
        ▼         ▼          ▼
   Aggregator  ILP Writers  Kafka Producer
   (1m bars)   (x6, NODELAY) (async)
        │         │          │
        ▼         ▼          ▼
     idx_ohlcv  6 QDB tables Redpanda (6 topics)

Supported Message Types

Performance

Board Filter

IDX has three boards: RG (Regular — real market price discovery), NG (Negotiated — off-market), TN (Tunai / Cash). Only RG ticks flow into idx_ohlcv. Filtering out NG/TN matches the Go reference implementation and prevents negotiated trades from distorting OHLCV.

Build Flags

# Cargo.toml release profile
[profile.release]
lto = "fat"
codegen-units = 1
panic = "abort"

# Build command
RUSTFLAGS="-C target-cpu=native" cargo build --release

Bench & Testing Guide

Criterion benchmarks, TSV history tracking, HD accuracy fixtures

Run Benchmarks

# Full bench suite (from host, NOT container)
cd rust-workers && cargo bench

# Record a bench point with a note in docs/bench-history.tsv
./scripts/bench-record.sh "hd engine v2 with EMA fast-path"

# View historical trend
cat docs/bench-history.tsv

HD Accuracy Fixture

The HD engine has a 100% accuracy gate in CI: the Rust implementation is compared byte-for-byte against the Go reference across 606 tickers. The fixture lives at fixture/hd-accuracy.csv.

# Run the accuracy test locally
cd rust-workers && cargo test --release hd_accuracy

# If it fails, diff is written to /tmp/hd-diff.csv

Unit Tests

# Rust workers
cd rust-workers && cargo test --release

# Go API
cd go-api && go test ./...

# Go API with race detector
cd go-api && go test -race ./...

Replay Fixtures

# Replay a recorded ITCH dump into the live pipeline
docker exec idxmdp-parser replay /etc/idx-parser/fixture/itch-low26.txt

# Large fixture (10.3M messages)
docker exec idxmdp-parser replay /etc/idx-parser/fixture/full-day.txt

Project Status

Stage-by-stage progress snapshot — what's done, what's next, what's on hold

Current Status

Stage Roster

Recent Milestones

• 2026-04-08 — Operations / Monitoring guides expanded; backfill-guide quality fixes (TL;DR, RSI warm-up note, troubleshooting "where to look").
• 2026-04-07 — S2-B5 HD accuracy fixture at 100% match vs Go reference across 606 tickers.
• 2026-04-03 — S3 API DONE: 1 binary, 3 domains, 16 PG tables, HTMX admin, Xendit prep.
• 2026-04-02 — S2 backfill tool + CI/CD gate DONE.

Enterprise Code Review

Comprehensive E2E audit — 2026-04-12 — Branch: feat/ch-qdb-hybrid-query

Audit Scope

Five parallel review agents examined distinct domains simultaneously. Each agent performed an independent deep read of all files in its domain.

Findings Overview

Data Flow with Failure Points

The diagram below traces a single market tick from vendor Redpanda through every processing stage to the client API response. Red markers indicate where data can be silently lost.

Severity Definitions

Data Pipeline Audit

End-to-end trace from Kafka ingest to API response — every failure point identified

C4: ch_drain Auto-Commits Before ClickHouse Insert

CRITICAL rust-workers/src/bin/ch_drain.rs:313-314

What happens: The ch_drain Kafka consumer uses enable.auto.commit = true with a 5-second interval. The consumer accumulates rows in memory and flushes to ClickHouse when the batch reaches 1,000 rows or 5 seconds pass. Since auto-commit fires on wall time (not on successful insert), there is a race window on every cycle.

Failure scenario:

1. Consumer polls and buffers 800 rows over 4.5 seconds
2. At 5.0 seconds, Kafka auto-commit fires — offsets for those 800 messages are committed to the broker
3. At 5.0 seconds, the flush timer also fires — HTTP INSERT to ClickHouse begins
4. ClickHouse returns 503 (overloaded). All 3 retry attempts fail
5. The 800 rows are discarded (see C5 below). The offsets are already committed. Those messages will never be replayed

Impact: Up to 1,000 rows per table per event are permanently lost from ClickHouse cold storage. The user sees gaps in historical charts that don’t appear in QuestDB (which got the data via direct ILP write).

Fix: Switch to enable.auto.commit = false and commit offsets manually after ch.insert_json_rows() succeeds. The metric_worker already does this correctly at worker/mod.rs:177.

C5: ch_drain Clears Row Buffer on Insert Failure

CRITICAL rust-workers/src/bin/ch_drain.rs:424-431

What happens: After the insert attempt (success or failure), buf.rows.clear() and buf.last_flush = Instant::now() execute unconditionally — they are outside the match arms.

match ch.insert_json_rows(&database, buf.table, &buf.rows).await {
    Ok(()) => { total_inserted += count; ... }
    Err(e) => { total_errors += count; ... }
}
// OUTSIDE the match — runs regardless:
buf.rows.clear();          // ← rows gone forever
buf.last_flush = Instant::now();

Combined with C4: This guarantees that any ClickHouse hiccup results in permanent data loss. The rows cannot be retried (cleared from memory) and cannot be replayed from Kafka (offset already committed).

Fix: Move buf.rows.clear() into the Ok(()) arm only. On failure, leave rows in the buffer for retry on the next flush cycle.

C6: db_writer Drops Entire Batch on QuestDB Double-Failure

CRITICAL rust-workers/src/db_writer.rs:148-156

What happens: The ILP writer attempts a TCP write. On failure, it reconnects and retries once. If the retry also fails, the function returns without error — and the caller’s macro always calls buf.clear() afterward.

fn flush(stream: &mut TcpStream, buf: &str, addr: &str, label: &str) {
    if let Err(e) = stream.write_all(buf.as_bytes()) {
        *stream = connect(addr, label);             // reconnect
        if let Err(e2) = stream.write_all(buf.as_bytes()) {
            tracing::error!("{}: retry write failed, {} bytes lost", ...);
            // ← returns normally, caller will .clear() the buffer
        }
    }
}

Impact: A QuestDB outage lasting more than one flush cycle (typically seconds) causes silent loss of ticks and OHLCV bars from the hot store. Since the parser doesn’t retry at the channel level, these rows are gone.

Fix: Return Result from flush(). On failure, skip buf.clear() so the batch is preserved for the next flush attempt. Add a idx_parser_db_writer_rows_lost_total counter.

I1: Aggregator OHLCV Bars Silently Dropped

IMPORTANT rust-workers/src/aggregator.rs:277-279

What happens: Completed OHLCV bars are sent via let _ = self.ohlcv_tx.try_send(cb). The let _ = pattern discards the Result — if the channel is full, the bar vanishes with no log, no metric, no alert.

Why it matters: A slow QuestDB writer causes the OHLCV channel to fill. Bars dropped here affect both QuestDB and ClickHouse (since the Kafka producer is downstream). Unlike tick drops (logged with a warning), OHLCV drops are completely invisible.

Fix: Log on drop and increment idx_parser_aggregator_ohlcv_drops_total.

I3: Snapshot Published to Kafka Before QDB Drop Check

IMPORTANT rust-workers/src/pipeline.rs:251,272

What happens: The Kafka publish (kp.send_snapshot) fires before the try_send to the QuestDB channel. If the QDB channel is full, the snapshot is sent to Kafka (and eventually to ClickHouse via ch_drain) but never reaches QuestDB.

Impact: ClickHouse has snapshot/index rows that QuestDB doesn’t. Any query that reads from QuestDB exclusively will see gaps. The same pattern affects idx_index messages.

I4: Orderbook Depth Indexing Mismatch

IMPORTANT rust-workers/src/db_writer.rs:82 vs ch_drain.rs:212

Any cross-database join or comparison on orderbook depth is off-by-one. Screener features that merge both sources will silently mis-label price levels.

I5: MetricHistory Returns Wrong Data Type from ClickHouse

IMPORTANT go-api/internal/handler/data.go:337-355

What happens: The MetricHistory handler (for HD/RSI values) calls ch.QueryOHLCV() for the ClickHouse time window. QueryOHLCV reads from idx_ohlcv (price bars: open, high, low, close, volume) — not from metrics_hd or metrics_rsi (metric values: value, mapo, direction).

Impact: For any request where the date range extends more than 1 hour into the past, the CH portion returns OHLCV price bars instead of HD/RSI metric values. The QDB portion (last hour) is correct. The merged response is structurally valid JSON but semantically wrong — the client chart displays price data where it expects metric indicator values.

Fix: Create ch.QueryMetrics(ctx, symbol, metric, from, to) that reads from metrics_hd/metrics_rsi and call it from MetricHistory.

I10: HDChart hotCut Computed Per-Goroutine

IMPORTANT go-api/internal/handler/hdchart.go:127

What happens: The HD chart handler spawns 6 goroutines (one per timeframe: 1m, 5m, 15m, 30m, 1h, 1d). Each goroutine computes hotCut = time.Now().Add(-1h) independently. If goroutine scheduling crosses a second boundary, different timeframes use different split points.

Impact: The 6-timeframe chart response has inconsistent overlap boundaries. One timeframe may show a gap or duplicate bar at the split point while others are clean. Visible as chart glitches near the 1-hour mark.

Fix: Compute hotCut once before launching goroutines and pass it as a parameter.

Rust Workers Review

Consumer, parser, aggregator, db_writer, Kafka producer, engines, ch_drain, metric_worker

C2: Credentials in Committed config.toml

CRITICAL rust-workers/config.toml:6,12,34

The Kafka SASL password (bridge2025!) and Redis password (idxmdp_redis_dev_2026) are in config.toml, which is committed to git and COPY-ed into the Docker image (Dockerfile line 72). Anyone with repo access or image registry access can extract these credentials.

Why this matters: The Kafka credentials grant read access to the live IDX market data feed. The Redis password grants access to cached metrics, session data, and tier configuration.

Fix: Replace config.toml values with placeholders. Supply real credentials only via IDX__* environment variables at runtime. The settings loader already supports this — config.toml should be a template, not a credential store.

I6: Unsafe UTF-8 in kafka_producer.rs

IMPORTANT rust-workers/src/kafka_producer.rs:58-98

What happens: The producer uses unsafe { self.buf.as_mut_vec() } to write JSON directly into a String’s internal buffer via serde_json::to_writer. If serde_json encounters an IO error mid-write, the String may contain partial UTF-8, violating its invariant and causing undefined behaviour on any subsequent string operation.

Fix: Replace with serde_json::to_string(tick) which is safe and has negligible cost since the buffer is cloned anyway.

I8: metric_worker auto.offset.reset=latest

IMPORTANT rust-workers/src/worker/mod.rs:81

What happens: When the metric worker starts with a new consumer group (no committed offset), it only processes bars arriving after startup. The warm-up from ClickHouse compensates, but if ch_drain is also lagging, the warm-up produces incomplete state.

Impact: HD/RSI values appear correct after ~14 bars but are stale-seeded for the first few minutes post-restart. This is visible as a small “jump” in the metric chart immediately after parser restart.

I9: warmup.rs Buffers Entire CH History in RAM

IMPORTANT rust-workers/src/worker/warmup.rs:52-53

What happens: resp.text().await? buffers the complete ClickHouse response (all rows from idx_ohlcv) into a single String before line-by-line parsing begins. At 700 symbols × 330 bars/day × N days, this easily reaches hundreds of MB.

Fix: Add WHERE ts >= now() - INTERVAL 30 DAY to bound the warm-up query, or use reqwest::Response::bytes_stream() for streaming line-by-line processing.

I21: String Keys in HD/RSI Engine Hot Path

IMPORTANT rust-workers/src/engine/hd.rs:117

What happens: The engine uses FxHashMap<String, HdTickerState>. The .entry(symbol.to_owned()) call allocates a new heap String on every bar (~1,400/s), even for tickers already in the map. The main parser correctly uses fixed [u8; 16] keys.

Fix: Use SmallVec<[u8; 16]> or a fixed-size array key, matching the aggregator’s pattern.

M1: parse_u32 Wrapping Overflow

MINOR rust-workers/src/parser.rs:403-408

The parser uses wrapping_mul/wrapping_add to avoid panics, but a 10-digit volume string like "5000000000" passes the length guard (len ≤ 10) yet overflows a u32 (max 4,294,967,295). The wrapping produces a silently wrong value (705,032,704 instead of 5,000,000,000).

Fix: Replace with checked_mul(10)?.checked_add(...)? to return None on overflow.

M3: Pipeline Comment Says “RabbitMQ”

MINOR rust-workers/src/pipeline.rs:19

Comment says “Raw bytes from RabbitMQ consumer” — should say Kafka/Redpanda after the migration.

M4: Redis KEYS Command in Ops Dashboard

MINOR rust-workers/src/bin/ops.rs:277

KEYS last:hd:* is O(N) and blocks Redis. Safe at current scale (~2,800 keys) but runs every 10 seconds on the ops dashboard. Replace with SCAN for non-blocking enumeration.

Go API Review

Handlers, middleware, DB clients, cache, audit, tier system, tests

C9: Redis Key Injection via Unvalidated metric Param

CRITICAL go-api/internal/handler/data.go:281

key := "last:" + metric + ":" + symbol

symbol is validated by ValidTicker (regex ^[A-Z0-9]{1,10}$), but metric is taken directly from c.Query("metric") with zero validation. An attacker can supply ?metric=../../session to probe arbitrary Redis key namespaces, potentially reading session tokens or rate-limit buckets.

Fix:

var validMetrics = map[string]struct{}{"hd": {}, "rsi": {}}
if _, ok := validMetrics[metric]; !ok {
    return c.Status(400).JSON(models.Err("invalid metric"))
}

C10: Silent Date Parse Failure Bypasses Tier Limits

CRITICAL go-api/internal/handler/data.go:322-323

from, _ := time.Parse("2006-01-02", fromStr)
to, _   := time.Parse("2006-01-02", toStr)

Parse errors are silently discarded. When time.Parse fails, it returns zero-time (year 0001-01-01). The history-depth clamp on line 327 clamps this to earliest = now - HistoryDays, which works for non-enterprise tiers. But enterprise tiers with IsUnlimitedHistory() == true skip the clamp entirely, passing year-0001 to the database and returning the entire ClickHouse history.

Compare with: OHLCVCached (lines 169-175) correctly returns HTTP 400 on parse failure.

Fix: Return 400 on bad date, matching the existing pattern.

I2: indexCache Thundering Herd

IMPORTANT go-api/internal/handler/udf.go:56-75

What happens: The index name cache uses a read-unlock → check-freshness → re-lock-and-write pattern with no singleflight guard. When the 5-minute TTL expires, all concurrent TradingView chart loads simultaneously call qdb.Indices(ctx) instead of just one. TradingView fires multiple /udf/symbols and /udf/search requests per chart load.

Fix: Use sync.Once or golang.org/x/sync/singleflight to deduplicate concurrent refreshes.

I7: Tier Subscriber Has No Reconnect

IMPORTANT go-api/internal/config/tier_subscriber.go:41-44

When the Redis pubsub channel closes (Redis restart, network blip), the subscriber goroutine logs a warning and exits permanently. After this, no tier hot-reloads will be applied to this API instance until the process restarts. This is the only background job without reconnect logic — StartAuditReplay and StartKeyExpiry both have retry loops.

Fix: Wrap the subscribe/listen loop in an outer reconnect loop with exponential backoff.

I20: Admin Self-Promotion of Tier

IMPORTANT go-api/internal/handler/admin/users_edit.go:109-111

The admin UserEdit handler correctly gates role changes to superadmin-only. But tier changes have no such restriction. An admin can pass their own user ID and promote their account tier to “enterprise”, bypassing billing entirely.

Fix: Prevent admins from editing their own tier, or require superadmin role for tier changes.

Other Important Findings

Minor Findings

Security Audit

OWASP Top 10 — authentication, authorization, injection, credential management

C1: Telegram Bot Token in .env.example

CRITICAL .env.example:151

TELEGRAM_BOT_TOKEN=8715922974:AAHWx7cmM6WL1QD2CfMoqzZDwTPRVMA5a0s

.env.example is explicitly allowed through .gitignore (line 6: !.env.example), meaning it is committed and visible in git history. This token follows the exact structure of a real Telegram bot API token. Possession grants full bot control: receiving alert messages, sending to channels, enumerating chat IDs.

Immediate action: Revoke via @BotFather (/revoke), generate new token, replace in .env.example with CHANGE_ME_YOUR_BOT_TOKEN.

C3: /ops/* Endpoints Have Zero Authentication

CRITICAL go-api/cmd/server/main.go:177-180

Four routes registered before the JWT auth group:

app.Get("/ops/latency", handler.OpsLatencyPage())
app.Get("/ops/ranking", handler.OpsRankingPage())
app.Get("/ops/api/latency", handler.OpsLatencyAPI(qdb, ch, rdb, pool))
app.Get("/ops/api/ranking", handler.OpsRankingAPI(qdb))

/ops/api/latency returns: QuestDB table names and row counts, ClickHouse row counts, Redis DB size, PostgreSQL pool stats, Redpanda consumer group lag with topic names. This is a full infrastructure inventory that directly aids targeted attacks.

Fix: Wrap in SessionAuth + RequireSessionRole("admin").

C7: ClickHouse Has Empty Password

CRITICAL .env:31, docker-compose.dev.yml:98

The default ClickHouse user operates with no password. The clickhouse-users.xml network restriction allows connections from the entire Docker subnet 172.0.0.0/8. Any container on any bridge network on the host can query, insert, or drop all market data tables with no authentication.

C8: Prometheus Metrics Exposed Externally

CRITICAL go-api/cmd/server/main.go:358

http.ListenAndServe(":2112", mux)

Bound to 0.0.0.0:2112 and published via Docker. The /metrics endpoint leaks rate limit counters, session counts, active WebSocket connections, audit buffer stats. For a financial SaaS, this is operational intelligence for an attacker.

Fix: Bind to 127.0.0.1:2112. Let Prometheus scrape via internal Docker network only.

I15: /udf/history Bypasses All Auth and Tier Limits

IMPORTANT go-api/cmd/server/main.go:199-204

The TradingView UDF endpoints are entirely unauthenticated. /udf/history proxies to the same QuestDB and ClickHouse backends as the paid /v1/ohlcv endpoint, returning full OHLCV data with no tier enforcement, no rate limiting, and no history depth limits. Anyone who reverse-engineers the UDF URL gets free access to data that paying customers pay for.

Fix: Gate the UDF group behind SessionAuth, or apply the same tier-based limits from OHLCVCached to UDFHistory.

I16: Rate Limiter Fails Open

IMPORTANT go-api/internal/middleware/ratelimit.go:39-42

This is a documented design decision, but in a financial SaaS context where tier enforcement is revenue-critical, any Redis disruption (OOM, network partition, or key eviction under allkeys-lru) makes all rate limits disappear. A free-tier user becomes unlimited.

I18: Plugin Report Accepts Unauthenticated Data

IMPORTANT go-api/internal/handler/plugin.go:159-208

POST /v1/plugin/report is outside the JWT group. An unauthenticated attacker can write arbitrary strings into the reports table and inject content into structured log output. If logs are forwarded to a SIEM, this is log injection.

Other Security Findings

Infrastructure Review

Docker Compose, Dockerfiles, schemas, monitoring, networking

C11: go-api/Dockerfile References Go 1.25 (Does Not Exist)

CRITICAL go-api/Dockerfile:1

FROM golang:1.25-alpine AS builder

Go 1.25 does not exist (latest stable is 1.24.x as of April 2026). This causes docker build to fail with an image-not-found error. The CI job and dev compose api service both use this Dockerfile. The API image cannot be built.

Fix: Change to golang:1.24-alpine.

C12: Postgres Audit Log Partitions Only Cover Through 2025-06

CRITICAL schema/postgres.sql:91-95

Only two partitions exist: audit_log_2025_01 (Jan 2025) and audit_log_2025_06 (Jun 2025). Today is 2026-04-12. Any audit write with created_at ≥ 2025-07-01 will fail with a PostgreSQL partition constraint violation. All current audit logging is broken.

Fix: Create partitions for the current date range:

CREATE TABLE audit_log_2025_07 PARTITION OF audit_log
  FOR VALUES FROM ('2025-07-01') TO ('2026-01-01');
CREATE TABLE audit_log_2026_01 PARTITION OF audit_log
  FOR VALUES FROM ('2026-01-01') TO ('2027-01-01');

I11: Alertmanager Has No Active Receiver

IMPORTANT monitoring/alertmanager/alertmanager.yml:57

The telegram-ops receiver has no telegram_configs (commented out). Alerts fire in Prometheus, reach Alertmanager, and are recorded only in the in-memory UI — no Telegram, no email, no PagerDuty. A QuestDBDown or APIDown critical alert will silently expire after 5 minutes without operator notification.

I12: No Memory/CPU Limits on Any Container

IMPORTANT docker-compose.dev.yml (all 18 services)

No service has deploy.resources.limits. On a single-host deployment, a runaway ClickHouse query or Redpanda log storm can OOM the entire host and take down all 18 containers simultaneously.

I13: Parser Missing depends_on for Outbound Redpanda

IMPORTANT docker-compose.dev.yml:247

The parser publishes to idxmdp-redpanda:9092 but only depends on questdb. If outbound-redpanda is not yet healthy at parser startup, initial publish attempts fail silently.

Other Infrastructure Findings

Action Plan

Prioritised fix schedule with effort estimates

Before Monday Trading (Tonight)

These items are either actively losing data, actively exploitable, or blocking builds. Fix before 08:45 WIB Monday.

This Week

Reliability and defense-in-depth improvements. Schedule across sprint.

Backlog