Senior Databricks Engineer

About the role

The Senior Databricks Engineer will design, build, and optimize scalable data ingestion and transformation pipelines to support enterprise‑grade data modernization programs. This role focuses on implementing ELT workflows within modern Data Lakehouse architectures, ensuring high‑quality, reliable, secure, and efficient movement of data from diverse source systems into curated analytical and application‑ready layers.

You will work closely with Data Architects, Data Modelers, Cloud Engineers, and Application teams to develop repeatable processing frameworks, enforce data quality rules, and operationalize Medallion‑layered datasets (Bronze/Silver/Gold or equivalents). This role is domain‑agnostic but requires the ability to translate complex business and technical requirements into performant data pipelines. Experience with high‑volume operational data domains (e.g., logistics, telematics, IoT, ERP, mobility platforms) is a plus.

Key Qualifications:

• Bachelor’s/Master’s in Computer Science, Information Systems, Engineering, or related field (or equivalent experience).

• 5–8 years of experience building data ingestion and transformation pipelines.

• Strong proficiency in SQL and PySpark for large‑scale data processing.

• Hands‑on experience with dbt (data build tool) for building, testing, and documenting modular data transformation pipelines — a core requirement for this role.

• Hands‑on experience with Databricks, Spark jobs, and Delta Lake‑based transformations.

• Experience with ELT patterns and Medallion Architectures (Bronze/Silver/Gold).

• Practical knowledge of Airbyte, ADF, or equivalent orchestration tools.

• Experience with data quality frameworks such as Great Expectations or Databricks expectations.

• Strong understanding of schema evolution, partitioning, optimization, and Delta Lake performance tuning.

• Familiarity with Azure components including ADLS Gen2, Event Hub, Synapse Serverless/Dedicated SQL.

• Experience integrating pipelines with applications and APIs (read/write patterns).

• Strong debugging skills for data integrity, ingestion failures, schema drift, and performance issues.

• Excellent communication, documentation, and cross‑team collaboration skills.

Responsibilities:

• Build scalable ELT pipelines using Databricks (PySpark/SQL), dbt, and Delta Lake.

• Develop Bronze, Silver, and Gold transformations aligned with enterprise modeling standards.

• Implement ingestion using tools like Airbyte, ADF, or custom connectors.

• Collaborate with Data Modelers to operationalize business rules, mappings, and canonical definitions.

• Integrate data pipelines with downstream applications (mobile apps, web apps, APIs).

• Implement data quality and validation rules, error handling, and automated remediation steps.

• Manage schema evolution, CDC ingestion, late‑arriving data, and incremental refresh patterns.

• Optimize Spark/Delta workloads through partitioning, indexing, caching, Z‑ordering, and cluster tuning.

• Develop reusable pipeline templates and frameworks for cross‑domain scalability.

• Collaborate with Cloud Engineers to ensure pipelines meet security, monitoring, and DevOps standards.

• Produce technical documentation, lineage, and metadata for governance teams.

• Support ML workloads by preparing feature datasets and inference‑ready tables.

• Participate in cross‑functional design sessions, reviews, and architectural decisions.

About Aubrant & Our Culture

About Aubrant

Aubrant Digital is a leader in multi-shore custom application development. We are passionate about solving our clients’ business problems through consultative teamwork, innovative software, and proven processes. We’ve served more than 50 clients and delivered hundreds of high quality, custom enterprise applications. Our clients value us as integral team members who get the job done on time and on spec, and we are proud of our high client retention rate and under 2% staff turnover. With offices in New Jersey, Boston, Costa Rica, and Eastern Europe, we execute the full software lifecycle, from architecture and design through development, QA and application maintenance & support. Our company culture emphasizes client service, trust-based relationships, and innovation.