Epic: TVM-1698 Date: 2026-02-02 Author: Raymond Mintz
Problem: ClickHouse consumer stuck since 1/27, massive lag preventing event ingestion
Root Cause: 4 bad test messages blocking consumer across all 3 Kafka partitions:
- Partition 0, Offset 58:
{"test":"connection_test",...} - Partition 1, Offset 56:
{"test":"direct_publish",...} - Partition 2, Offset 113:
{"test":"message",...}
Hereβs a concise summary + work items focused on implementationβrelevant points from the Kafka meeting notes.
Summary (implementationβrelevant highlights)
- The events endpoint currently requires auth, but thereβs consensus that unauthenticated (guest) events are needed. This is a potential DDoS vector, so any unauthenticated mode needs rateβlimiting or an API key strategy.
- Clients were hitting 500s on the events endpoint. Root cause likely in validation/headers. Once the correct request + headers were used, responses became sane (401 or validation errors). Accept header absence can trigger HTML response instead of JSON.
- Logging quality is important for debugging bad payloads; logs should be useful in dev.
- Feature flag is required on mobile/web so event sending can be toggled off by default, especially in production builds.
- Batching events is expected; payloads should accept arrays and clients can send every 5β10 seconds.
- Brand/tenant identification s
-
Send Yan the metrics endpoint and payload (explicitly requested)
- Endpoint already exists at
POST /v1/events(no/apiprefix) - Controller:
app/Http/Controllers/EventController.php - Returns
202 {"success": true}
- Endpoint already exists at
-
Increase payload size limit - Support batch events (~500KB suggested)
| Build Name | Static CR | Dyn CR @68Β° | HP Range | TQ Range | Notes |
|---|---|---|---|---|---|
| 347 TFS 11R 170/53 Flat-Top | 11.39 | 8.86 | 420β450 hp | 400β430 lb-ft | High comp, CNC heads, pump gas 93 required |
| 347 TFS 11R 170/53 Dish | 9.95 | 7.77 | 395β425 hp | 380β410 lb-ft | Safe pump gas, still strong street combo |
| 347 Liberty 180/64 Flat-Top | 9.95 | 7.77 | 410β440 hp | 395β420 lb-ft | Non-CNC, lower comp, safer but slightly softer |
| 347 Liberty 180/64 Dish | 8.86 | 6.95 | 375β410 hp | 365β400 lb-ft | Very safe CR, better for boost than NA |
Moderation Process Flow
βββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββ
β User ββββββΆβ API Server ββββββΆβ PostgreSQL β
β Upload β β β (FastAPI) β β‘ β Database β
βββββββββββββββ ββββββββββββββββββββ βββββββββββββββββββ
β β’
βΌ
NewerOlder