Posted 1 month ago
Senior Data Engineer (Databricks Lead)Prodapt · Bengaluru
Senior Data Engineer (Databricks Lead)
BengaluruOn-site
AI Summary
Leads the design, development, and optimization of enterprise-scale data platforms using Databricks, Snowflake, and AWS, with deep expertise in Spark, Kafka, Python, and SQL.
About this role
Overview
We are looking for a Senior Data Engineer (Databricks Lead) with 12-16 years of experience to lead the design, development, and optimization of enterprise-scale data platforms. The ideal candidate will have deep expertise in Databricks, Apache Spark, Snowflake, Kafka, Python, SQL, and AWS, along with experience building scalable data architectures that support analytics, real-time processing, AI/ML, and Generative AI initiatives. .
Responsibilities
- Lead the architecture, design, and implementation of scalable, secure, and high-performance data platforms using Databricks, Snowflake, and AWS.
- Design and optimize large-scale batch and real-time data pipelines using Spark and Kafka.
- Develop and maintain robust ETL/ELT frameworks to ensure data reliability, scalability, and performance.
- Implement real-time data ingestion and processing solutions using Kafka and Spark Streaming.
- Build automated data integration frameworks leveraging REST APIs, GraphQL APIs, and Databricks APIs.
- Design cloud-native data solutions aligned with enterprise governance, security, and compliance standards.
- Develop data models using Star Schema, Snowflake Schema, Kimball, and Inmon methodologies.
- Provide technical leadership, architecture guidance, and mentorship to engineering teams.
- Collaborate with Analytics, Data Science, and AI teams to enable Machine Learning, Generative AI, and LLM-based solutions.
- Troubleshoot and resolve platform performance bottlenecks and scalability challenges.
- Drive engineering best practices, code quality standards, and data governance initiatives.
Requirements
Technical Skills
- Strong expertise in Databricks and Apache Spark.
- Hands-on experience with Snowflake and modern data warehouse architectures.
- Proficiency in Kafka and real-time stream processing.
- Strong programming skills in Python and advanced SQL.
- Experience with AWS Cloud Services.
- Expertise in designing and implementing ETL/ELT architectures.
- Strong understanding of data modeling concepts and methodologies.
- Experience integrating systems using REST and GraphQL APIs.
Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.
- 12-16 years of experience in Data Engineering and Data Platform Architecture.
- Proven experience leading enterprise-scale data modernization and cloud migration initiatives.
Preferred Skills
- Experience with AI/ML data platforms, Generative AI, and LLM enablement.
- Exposure to Data Governance and Data Quality frameworks.
- Strong stakeholder management, communication, and technical leadership skills.
Skills
Apache SparkAWSDatabricksETL/ELTGraphQL APIsInmonKafkaKimballPythonREST APISnowflakeSnowflake SchemaSpark StreamingSQLStar Schema
Explore related jobs
More jobs at Prodapt
Similar Databricks jobs
- B
Apache Spark DeveloperBright Vision Technologies · Edison / Middlesex County, New Jersey - E
Apache Spark Java DeveloperElite Workforce Inc. · Brooklyn Park, MN - E
JAVA and Scala/ Apache Spark developer.EROS Technologies Inc. · New York, NY - N
Senior Machine Learning Engineer, Accelerated Apache SparkNVIDIA USA · US, CA - DSr. Designated Support Engineer, Apache Spark™Databricks · Sao Paulo, Brazil
- LPrincipal Data Engineer (Apache Spark, dbt, Airflow)LE001-ASXOPS ASX Operations Pty Ltd · Sydney Office
Jobs in Bengaluru
- BEngineer I, SEE 3D ModellerBrpcareers Se · Bangalore, Karnataka
- BSenior Technical Trainer IBrpcareers Se · Bangalore, Karnataka
- BTechnical Trainer IIBrpcareers Se · Bangalore, Karnataka
- CEngineer II, Quality Continuous ImprovementConfidential Se · Bangalore, Karnataka
- CSenior Engineer II, CostingConfidential Se · Bangalore, Karnataka
- CSolutions ArchitectCelonis · Bangalore, India
Browse these categories
Market data for data engineer roles
All reports →- SeriesRole reportsOne role family at a time: how many openings, what changed this week, who is hiring, what it pays.
- SeriesSalary reportsWhat employers publish in job postings, by level and workplace. Not self-reported pay.
- Market overviewState of tech hiring, September 2026: up 4.8%Tech hiring rose 4.8% month over month in September 2026, with 411,122 new listings. Customer support and account executive roles led the growth.