1. Executive Summary & Core Architectural Philosophy
Managing ISP infrastructure requires handling thousands of edge routers (MikroTik/RouterOS), millions of concurrent subscriber data-accounting packets per second, real-time M-Pesa payments, automated access provisioning, and live policy enforcement (speed throttling and disconnects).
To scale without suffering from database bottlenecks or unhandled concurrency spikes, the platform follows an Event-Driven Microservices + Modular Monolith hybrid model using Rust (Axum + Tokio).
flowchart TD
subgraph Clients["Clients & Edge Network"]
Users["WiFi / PPPoE Users"]
Admins["ISP Vendors / Admins"]
Routers["MikroTik Routers (WireGuard / RADIUS)"]
end
subgraph Ingress["Ingress & Edge Layer"]
Gateway["api-gateway (Axum Edge)"]
VueUI["Vue 3 Single-Page App"]
end
subgraph CoreProduct["Product Core (Modular Monolith)"]
ProductAPI["product-api (Axum - 26 Modules)"]
end
subgraph DataStore["Systems of Record"]
Postgres[(PostgreSQL + PgBouncer)]
OutboxTable[["outbox table"]]
end
subgraph OutboxWorker["Outbox Publisher"]
OutboxRelay["outbox-relay"]
end
subgraph EventSpine["Event Bus Spine"]
Redpanda{{"Redpanda (Kafka API Event Bus)"}}
end
subgraph Workers["Hot-Path Microservice Workers"]
RadiusSvc["radius-svc (AAA / CoA)"]
GrantSvc["grant-svc (Access Grant)"]
SessionSvc["session-svc (Redis State + Reconciler)"]
EnforceSvc["enforcement-svc (Quota & Disconnects)"]
AcctConsumer["accounting-consumer (Analytics Pipeline)"]
RouterControl["router-control (WireGuard / RouterOS API)"]
RealtimeSvc["realtime-svc (WebSocket Server)"]
NotifSvc["notifications-svc (SMS Engine)"]
end
subgraph StorageTier["State & Analytics Data Stores"]
Redis[(Redis Live Session Cache)]
ClickHouse[(ClickHouse Usage Analytics)]
end
Users -->|HTTP/Portal| Gateway
Admins -->|HTTPS| VueUI
VueUI -->|REST API| ProductAPI
Gateway --> ProductAPI
ProductAPI -->|Write Business Data| Postgres
ProductAPI -->|Write Outbox Event| OutboxTable
OutboxTable -->|Poll / Stream| OutboxRelay
OutboxRelay -->|Publish Events| Redpanda
Routers -->|RADIUS Auth / Accounting| RadiusSvc
Routers <-->|WireGuard Tunnels| RouterControl
RadiusSvc -->|Publish accounting.events| Redpanda
Redpanda -->|payment.events / access.events| GrantSvc
Redpanda -->|accounting.events| SessionSvc
Redpanda -->|accounting.events| AcctConsumer
Redpanda -->|accounting.events / usage| EnforceSvc
Redpanda -->|all events| RealtimeSvc
Redpanda -->|notification.events| NotifSvc
GrantSvc -->|Update radcheck| Postgres
SessionSvc <-->|Read / Write Live State| Redis
AcctConsumer -->|Batch Insert Usage| ClickHouse
EnforceSvc -->|CoA Disconnect Request| RadiusSvc
RealtimeSvc -->|Live WebSockets| VueUI
Core Architecture Principles:
- Stateless Services: All worker microservices and API nodes carry no local in-memory state. State resides strictly in Redis, PostgreSQL, or ClickHouse.
- Zero Synchronous Calls on the Hot-Path: Cross-service interaction happens via Redpanda (Kafka API). A payment or accounting event does not block API requests; it publishes an event that downstream workers consume asynchronously.
- Transactional Outbox Pattern: Database mutations (e.g. subscriber payment) write the business row and an outbox event in the same ACID PostgreSQL transaction.
outbox-relayreads the outbox and publishes to Redpanda, guaranteeing zero lost events. - Idempotent Consumers: Every event includes an
event_id. Consumers track processed IDs to ensure at-least-once delivery results in exactly-once side-effects.