
Posted 12 days ago
Senior Databricks Data Engineer
AI Summary
Leads design and optimization of data pipelines and data platforms in a Databricks-on-AWS environment, building scalable, secure, governed data solutions for analytics, reporting and AI initiatives.
About this role
KIBO is a registered United States company dedicated to providing governments worldwide its services of inspection, analysis, certification, and compliance. The Senior Databricks Data Engineer will lead the design, development, and optimization of data pipelines and data platforms within a Databricks-on-AWS environment.
Working closely with the Principal AI Architect, this role will build scalable, secure, and governed data solutions that support analytics, reporting, and AI initiatives. The ideal candidate brings deep expertise in Databricks, cloud-based data engineering, and modern data architecture.
Key Responsibilities
- Data platform development. Configure, manage, and optimize Databricks workspaces and supporting AWS infrastructure.
- Data pipeline engineering. Design and build scalable batch and streaming data pipelines using PySpark, Spark SQL, Delta Lake, and Auto Loader.
- Data architecture. Implement and maintain Medallion Architecture (Bronze, Silver, Gold) and analytics-ready data models.
- Data governance & security. Configure Unity Catalog, permissions, row-level security, column masking, and data-access controls.
- Data integration. Ingest data from third-party APIs, file transfers, and operational systems using modern Databricks patterns.
- CI/CD & quality. Implement Git-based development workflows, automated testing, and deployment processes across environments.
- Agile delivery. Collaborate with cross-functional teams in an Agile environment to deliver reliable, high-quality data solutions
Required Qualifications
- 5+ years of experience in Data Engineering, including 2+ years of hands-on Databricks experience in production.
- Strong experience with Databricks Lakehouse Platform, Delta Lake, Unity Catalog, and PySpark.
- Advanced SQL and Python development skills.
- Experience implementing Medallion Architecture and data models for analytics and reporting.
- Experience with Delta Live Tables, change data capture (CDC), incremental processing, and data-quality controls.
- Working knowledge of AWS services including S3, IAM, storage credentials, and external locations.
- Experience with Git, CI/CD pipelines, and automated testing practices.
- Bachelor's degree in Computer Science, Information Systems, Engineering, or equivalent experience.
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience with Spark performance tuning and cost optimization.
- Software engineering background with API development experience.
- Experience developing and testing locally using PySpark, Delta Lake, and pytest.
- Experience supporting AI/ML, financial, trade, customs, or other regulated-data environments.
What Success Looks Like in Your First Few Months
- Establish and secure the Databricks environment and governance framework.
- Deliver initial ingestion pipelines and curated Bronze, Silver, and Gold data layers.
- Implement CI/CD processes and data-quality monitoring standards.
- Support the transition from batch processing to near real-time incremental data ingestion.
Reporting and Working Relationships Reports to the Principal AI Architect and works closely with data, AI, analytics, and engineering stakeholders across the organization.
What KIBO Offers
- Competitive compensation and comprehensive benefits.
- Opportunity to build a modern Databricks data platform from the ground up.
- Exposure to cutting-edge data engineering and AI initiatives.
- A collaborative, mission-driven environment with significant ownership and impact.
Join Us If you are passionate about building scalable cloud data platforms and want to help shape the future of data and AI solutions, we encourage you to apply.
KIBO Laboratories LLC is an equal-opportunity employer and welcomes qualified applicants of all backgrounds. Offers are contingent upon background and right-to-work verification where applicable.
Skills
Explore related jobs
More jobs at Securiport
Similar Databricks jobs
- P
Loader Operator | Auto Salvage & RecyclingPull-A-Part · Memphis, Tennessee - FCDL Driver - Automated Side Loader (Asheville, NC)FCC ENVIRONMENTAL SERVICES FLORIDA LLC · Asheville, North Carolina
- FCDL Driver- Automated Side Loader (Omaha, NE)FCC ENVIRONMENTAL SERVICES FLORIDA LLC · Omaha, Nebraska
- FCDL Driver- Automated Side Loader (Venice, FL)FCC ENVIRONMENTAL SERVICES FLORIDA LLC · Venice, Florida
- FCDL Driver - Automated Side Loader (St. Augustine, FL)FCC ENVIRONMENTAL SERVICES FLORIDA LLC · St. Augustine, Florida
- FCDL Driver - Automated Side Loader (Green Cove Springs, FL)FCC ENVIRONMENTAL SERVICES FLORIDA LLC · Green Cove Springs, Florida
Jobs in Reston
- SLeasing Consultant - Signature at Reston Town CenterSimpson Housing LLLP · Reston, VA
- A
Mechanical Engineer, Intelligence SystemsAnduril Industries · Reston, Virginia - CStaff Accountant – General Ledger & Cash ManagementChugach Government Solutions · RESTON, Virginia
- O
Phlebotomist / Front DeskOne Medical · Reston, Virginia - O
Float Phlebotomist / Front DeskOne Medical · Reston, Virginia - OSenior Principal Site Reliability Engineer - US Top Secret ClearanceOracle · Reston, VA
Browse these categories
Market data for data engineer roles
All reports →- SeriesRole reportsOne role family at a time: how many openings, what changed this week, who is hiring, what it pays.
- SeriesSalary reportsWhat employers publish in job postings, by level and workplace. Not self-reported pay.
- Market overviewState of tech hiring, September 2026: up 4.8%Tech hiring rose 4.8% month over month in September 2026, with 411,122 new listings. Customer support and account executive roles led the growth.