newsReddit r/googlecloudTrust 52 · CommunityPublished 1mo agoLive · 1mo ago
Cheapest way to ingest ~270M JSON events/hour from an on-prem Kafka into GCP
We need to stream network telemetry events (JSON, ~2KB each) from an on-prem Kafka cluster into GCP for storage/analytics. Volume is large: ~270M events/hour (~390 TB/month at 2KB avg size). What I tested: Custom Python producer → Pub/Sub topic → BigQuery subscription → BigQuery table, with periodic GCS backup exports. I validated this empirically with a 10M-message load test (real event schema, 8 parallel producer threads, ~1000 m
Why these links exist
Every edge carries a method, confidence, and the source snippet that justified it — so bad links are debuggable.
- PossiblePossibly related (embedding) · 54%quixio/quix-streams →
- PossiblePossibly related (embedding) · 46%RumbleDB/rumble →
