Software Engineer Data Platform

We're looking for an experienced Software Engineer to join our Data Platform Group. In this key role, you will build the company’s data platform: cloud-based microservices and data pipelines that process approximately 1M records per second at low latency. Because the platform is the foundation other groups build on, your work directly impacts our customers and enables engineering, product, and research teams across the organization.
The Streaming team owns the real-time heart of the platform: every record enters the company through our pipelines, and we process it in motion. We build high-throughput Go services and Kafka-based pipelines that ingest, enrich, aggregate, and route our data in real time, at low latency. We also build external integrations that deliver our data to customers, partnering closely with Product to define and implement connectors that meet customer needs.
At this scale, efficiency is the product. We care about how many records per second each core handles and how many bytes each record costs in memory. If you enjoy profiling hot paths, pushing CPU utilization to the limit, and cutting memory footprint without giving up correctness, this is the role.

Responsibilities

• Take end-to-end ownership of our real-time stream processing layer: ingestion, enrichment, aggregation, and routing of high-volume data at low latency..
• Design and develop high-performance stream processing services in Go, built on Kafka/MSK..
• Build and evolve stateful stream processing pipelines, including Kafka Streams-based applications..
• Partner with Product to define, design, and implement external connectors that export and deliver our data to customers’ systems..
• Own performance and resource efficiency: profile and optimize hot paths, get the most out of every CPU core, reduce memory footprint and allocations, and lower compute costs at scale..
• Design for reliability under load, including backpressure, partitioning and consumer-group strategies, ordering, delivery guarantees, and idempotency..
• Research new technologies in stream processing and adapt them for use in our product..
• Work closely with Product, DevOps, Security, and the other teams in the Data Platform Group..

Requirements

• 5+ years of hands-on experience designing and developing high-throughput, low-latency distributed systems in production..
• Deep, hands-on experience with Kafka at significant scale: partitioning, consumer groups, offset management, producer/consumer tuning, and throughput vs. latency trade-offs..
• Familiarity with fundamantals of Go and Java is a major advantage..
• At least basic hands-on experience using AI coding assistants or development tools to accelerate implementation, testing, and debugging, with the ability to review and validate generated code..
• Experience building and running microservices on Kubernetes, including tuning resource requests and limits, and configuring autoscaling..
• Hands-on experience with AWS, particularly MSK, EKS, and EC2..
• A motivated, independent learner and strong problem solver..
• A team player with excellent collaboration and communication skills..
• B.Sc. in Computer Science, Software Engineering, or a related field, or equivalent practical experience..