
Posted 1 day ago
Senior Databricks Engineer
AI Summary
Leads design and implementation of cloud-based enterprise data platforms on Databricks, Snowflake and AWS, building ETL/ELT pipelines, lakehouse architectures and data foundations for analytics and AI, with technical leadership and mentoring of engineering teams.
About this role
Overview
Responsibilities
- Design and architect scalable, secure, and highly available enterprise data platforms using Databricks, Snowflake, and AWS Cloud.
- Lead the development and optimization of large-scale ETL/ELT pipelines, supporting both batch and real-time data processing workloads.Design and implement modern data lakehouse architectures and cloud-native data ecosystems.
- Architect and deploy AWS-based data platform components while ensuring compliance with enterprise security, IAM, governance, and networking standards.Build and optimize data ingestion frameworks integrating Databricks APIs, REST APIs, GraphQL APIs, and third-party data sources.
- Develop real-time data processing solutions for business-critical, low-latency use cases using modern streaming technologies.
- Design and implement scalable data models, incorporating both traditional data warehousing methodologies (Kimball, Inmon, Star Schema, Snowflake Schema) and modern semi-structured/unstructured data patterns.
- Collaborate with business stakeholders, architects, data scientists, and engineering teams to define and execute data strategies.Drive performance optimization, cost management, scalability, and reliability initiatives across the data platform landscape.
- Provide technical leadership, architectural governance, code reviews, and mentorship to engineering teams.
- Support AI and Machine Learning initiatives by designing data foundations that enable LLMs, Generative AI, Machine Learning pipelines, and advanced analytics solutions.
- Establish best practices for data engineering, architecture, governance, observability, and platform operations.Prepare and maintain architecture documentation, design standards, and technical roadmaps.
.
Requirements
- 12 to 16 years of experience in Data Engineering, Data Architecture, and Cloud Data Platforms.
- Strong expertise in Databricks, Snowflake, and AWS Cloud Services.
- Proven experience designing and delivering large-scale enterprise data architectures and modern data lakehouse solutions.
- Hands-on experience building and optimizing ETL/ELT pipelines, data ingestion frameworks, and distributed data processing systems.
- Strong knowledge of real-time and batch data processing architectures.
- Expertise in Databricks Workflows, Delta Lake, Spark, Structured Streaming, and Databricks APIs.
- Experience integrating REST and GraphQL APIs for enterprise-scale data ingestion and orchestration.
- Strong understanding of Data Warehousing concepts including Kimball, Inmon, Star Schema, and Snowflake Schema methodologies.
- Experience designing scalable and secure solutions within the AWS ecosystem, including IAM, networking, and governance controls.
- Strong proficiency in SQL, Python, and Spark-based development.
- Experience with performance tuning, optimization, data quality, metadata management, and platform observability.
- Ability to provide technical leadership, solution architecture guidance, and mentorship to engineering teams.
- Excellent stakeholder management, communication, problem-solving, and decision-making skills.
Preferred Qualifications
Experience with modern data ecosystem tools such as Apache Kafka, Airflow, dbt, Iceberg, and Kubernetes. Exposure to Generative AI, LLMs, Machine Learning platforms, MLOps, and advanced analytics architectures will be highly preferred. Experience with data governance, security, compliance frameworks, and enterprise-scale cloud modernization programs is advantageous. Relevant certifications in Databricks, Snowflake, AWS Solutions Architecture, or Data Engineering will be considered a strong advantage. Candidates with experience supporting AI-ready data platforms and leading enterprise data transformation initiatives will be highly valued.
Skills
Explore related jobs
More jobs at Prodapt
Similar Databricks jobs
- J
Airflow Platform EngineerJade Global Software Pvt LTD · Hyderabad, Telangana - S
Airflow Python DeveloperSutherland · Hyderabad, TS - C
Category Manager - AirflowCFO Trane U.S. Inc.-CFO · Davidson, North Carolina - B
Airflow Operator 3rd ShiftBurke Aerospace · Farmington, Connecticut - C
Sr. Mechanical Engineer - AirflowCFO Trane U.S. Inc.-CFO · Clarksville, Tennessee - E
Solutions Engineer(Python/Airflow)Edb · Remote - United States
Jobs in Bengaluru
Browse these categories
Market data for this role
All reports →- SeriesRole reportsOne role family at a time: how many openings, what changed this week, who is hiring, what it pays.
- SeriesSalary reportsWhat employers publish in job postings, by level and workplace. Not self-reported pay.
- Market overviewState of tech hiring, September 2026: up 4.8%Tech hiring rose 4.8% month over month in September 2026, with 411,122 new listings. Customer support and account executive roles led the growth.