
Posted 1 month ago
Senior Software Engineer - Data Infrastructure
AI Summary
Builds and operates a petabyte-scale data lakehouse for multimodal spatial and sensor data, covering storage layout, table formats, versioning, ingest, and distributed processing into training-ready datasets.
About this role
About us:
Calder is solving the data bottleneck for embodied AI and world models. Today's AI models lack the real-world, 4D data needed to reason on spatial problems and operate in physical environments. We capture, process, and deliver high-fidelity human-generated spatial data at scale, providing the substrate that trains the next generation of generalizable models for manipulation, navigation, and interaction.
About the role
We're looking for a Senior Data Infrastructure Engineer to help build and run the petabyte-scale data platform behind Calder's spatial data products. Every capture — robot rigs, egocentric headsets, multi-camera video, depth, IMU — and everything derived from it — reconstructions, Gaussian Splats, labels, training datasets — lands in this platform, gets processed and versioned, and is served to customers and training clusters. You'll work on storage layout, table formats, versioning, ingest, and the distributed processing that turns raw sensor data into training-ready datasets, and help the platform hold up as volume grows by orders of magnitude. This is a hands-on role: you'll write, deploy, and operate core parts of the platform alongside the team. We're looking for someone who has personally built and run a PB-level platform in production, not only someone who has used one.
Build and scale Calder's PB-scale data lake / lakehouse for multimodal spatial data — object storage layout, partitioning, open table formats (Iceberg or equivalent), catalogs, and metadata
Work on data versioning and lineage (LakeFS or equivalent): branches, commits, reproducible dataset snapshots, and provenance from raw capture to customer delivery
Build high-throughput, fault-tolerant ingest for field and lab captures — upload, validation, checksumming, content-addressed deduplication, and safe promotion through bronze → silver → gold
Build and operate large-scale distributed processing across CPU and GPU fleets — video decode and frame extraction, sync and calibration application, SLAM / reconstruction / 3DGS batch jobs, and model-based auto-labeling (pose, depth, detection)
Build and maintain data pipelines at scale: idempotent jobs, retries, backfills, partitioned reprocessing, and clean recovery when a multi-day run fails halfway
Build the dataset curation and query layer — metadata indexing, SQL over the lake (Trino / DuckDB / Spark SQL), and search so researchers and customers can slice data without copying it
Keep GPUs fed: sharded training formats (WebDataset, Mosaic Streaming, or similar), caching, prefetching, and efficient cross-region movement between capture sites and training regions
Work on platform reliability, observability, and cost on AWS — S3 tiering and lifecycle, egress, spot and right-sizing, capacity planning, and alerting
Contribute to infrastructure-as-code (Terraform) and CI/CD for the data platform; write design docs and runbooks so others can operate it without tribal knowledge
Work with capture, calibration, SLAM, 3DGS, labeling, and QA teams to turn their workload requirements into platform capabilities
About you
B.S. or M.S. in Computer Science, Software Engineering, or a related field, or equivalent practical experience
6+ years in data infrastructure, data platform, or distributed systems engineering
Hands-on experience setting up and operating a data platform at petabyte scale in production — you can speak concretely to the volumes, throughput, object counts, and failure modes you personally handled
Deep object storage experience (S3 or equivalent): key/prefix design, request-rate and throughput limits, the small-file problem, lifecycle and storage-class tiering, and cost control
Production experience with open table formats and lakehouse components — Iceberg, Delta Lake, or Hudi; catalogs (Glue, Polaris, Nessie, Unity, or similar); compaction and snapshot management
Strong experience with distributed processing — Spark, Ray, Flink, or Dask — tuned for large production jobs, not only used through notebooks
Experience with workflow orchestration (Airflow, Argo, Prefect, or similar) and the discipline that goes with it: idempotency, retries, backfills, and data contracts
Strong programming in Python plus at least one of Go, Rust, C++, Java, or Scala
Solid AWS infrastructure skills — EC2, EKS/ECS, S3, IAM, VPC, networking — and Terraform or equivalent IaC
Production operations experience: you've been on call for a data platform and care about its uptime, correctness, and cost
Clear written communication: design docs and runbooks other engineers can follow
Nice to Have
Multimodal or unstructured data at scale — video, images, LiDAR / point clouds, sensor logs, or 3D assets — ideally from autonomous driving, robotics, mapping, or simulation
Data versioning systems (LakeFS, DVC, Nessie) and content-addressed storage / deduplication design
Feeding large-scale ML training: WebDataset, Mosaic Streaming, Petastorm, Ray Data, or custom loaders for multi-node GPU training
GPU-accelerated data processing (NVIDIA DALI, RAPIDS, NVDEC / FFmpeg pipelines)
Query engines over lake data (Trino, DuckDB, ClickHouse) and embedding / metadata search for dataset curation
Multi-region or hybrid cloud / on-prem architectures, including large-scale data transfer and egress optimization
Observability stacks (Datadog, Prometheus / Grafana, OpenTelemetry)
Robotics log formats (ROS bags, MCAP) and time-synchronized multi-sensor data
Contributions to open-source data infrastructure projects
Calder is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status. Learn more here.
By applying for this position, you consent to the collection, storage, and processing of your personal data by Calder for recruitment purposes. We will handle your information securely and in accordance with local privacy policies. Your data may be shared with authorized third-party service providers aiding in the hiring process.
Skills
Explore related jobs
More jobs at Calder
Similar Airflow jobs
- J
Airflow Platform EngineerJade Global Software Pvt LTD · Hyderabad, Telangana - S
Airflow Python DeveloperSutherland · Hyderabad, TS - C
Category Manager - AirflowCFO Trane U.S. Inc.-CFO · Davidson, North Carolina - B
Airflow Operator 3rd ShiftBurke Aerospace · Farmington, Connecticut - MSystems Engineer, Thermal and AirflowMill · San Bruno, California
- C
Sr. Mechanical Engineer - AirflowCFO Trane U.S. Inc.-CFO · Clarksville, Tennessee
Browse these categories
Market data for software engineer roles
All reports →- Role reportSoftware engineer jobs, Sep 2026: new listings down 9.7%New software engineer listings fell 9.7% to 1,506 this week, the first weekly drop in a month. Prioritize applications while 1,506 fresh roles remain open.
- Salary reportSoftware engineer salary 2026: median midpoint is $190,587The median published range midpoint for US software engineer roles is $190,587, with the middle half from $166,000 to $222,000.
- Market overviewState of tech hiring, September 2026: up 4.8%Tech hiring rose 4.8% month over month in September 2026, with 411,122 new listings. Customer support and account executive roles led the growth.