G
Open since Apr 2026
Lead Data EngineerGo Digital Technology Consulting LLP · Mumbai
Lead Data Engineer
MumbaiOn-siteFull-time
AI Summary
Designs and builds scalable, fault-tolerant data pipelines and end-to-end data engineering solutions on AWS, while leading requirements gathering and stakeholder communication.
About this role
Technologies / Skills:
Advanced SQL, Python and associated libraries like Pandas, Numpy etc., Pyspark , Shell scripting, DataModelling, Big data, Hadoop , Hive, ETL pipelines and IaC tools like Terraform etc.
Responsibilities:
- Proven success in communicating with users, other technical teams, and senior management to
- Collect requirements, describe data modeling decisions and develop data engineering strategy.
- Ability to work with business owners to define key business requirements and convert to user stories with required technical specifications.
- Communicate results and business impacts of insight initiatives to key stakeholders to collaboratively solve business problems.
- Working closely with the overall Enterprise Data & Analytics Architect and Engineering practice leads to ensure adherence with the best practices and design principles.
- Assures quality, security and compliance requirements are met for supported area.
- Design and create fault-tolerance data pipelines running on cluster
- Excellent communication skills with the ability to influence client business and IT teams
- Should have design data engineering solutions end to end. Ability to come up with scalable and modular solutions
Required Qualification:
- 7 years of relevant IT experience
- 4+ years of hands-on experience Designing and developing Data Pipelines for Data Ingestion or Transformation using Python (PySpark)/Spark SQL in AWS cloud
- Experience in design and development of data pipelines and processing of data at scale using technologies like EMR, Kinesis, Kafka, Lambda, Glue, Athena, Redshift, Step Functions.
- Advanced experience in writing and optimizing efficient SQL queries with Python and Hive handling Large Data Sets in Big-Data Environments
- Experience in debugging, tunning and optimizing PySpark data pipelines
- Should have implemented concepts and have good knowledge of Pyspark dataframes, joins, caching, memory management, partitioning, parallelism etc.
- Understanding of Spark UI, Event Timelines, DAG, Spark config parameters, in order to tune the long running data pipelines.
- Experience working in Agile implementations
- Experience with building data pipelines in streaming and batch mode.
- Experience with Git and CI/CD pipelines to deploy cloud applications
- Good knowledge of designing Hive tables with partitioning for performance
Desired Qualification:
- Experience in data modelling
- Hands on creating workflows on any Scheduling Tool like Autosys, CA Workload Automation
- Proficiency in using SDKs for interacting with native AWS services
- Strong understanding of concepts of ETL, ELT and data modeling
Skills
Advanced SQLAutosysAWS AthenaAWS EMRAWS GlueAWS KinesisAWS LambdaAWS RedshiftAWS Step FunctionsCI/CDData ModellingETLGitHadoopHiveKafkaNumPyPandasPySparkPythonShell ScriptingSpark SQLTerraform
Explore related jobs
More jobs at Go Digital Technology Consulting LLP
Similar Advanced SQL jobs
- S
Java Scripting (strong), Advanced sql QueriesSQUIRCLE IT CONSULTING SERVICES PVT. LTD · Hyderabad, Telangana - U
QA/ testing resource with strong/ advanced SQLUSM · Arlington, VA - I
SAS DI Studio and Advanced SAS/SQLINENT INC · Charlotte, NC - W
Senior, Data Analyst - Advanced Analytics (SQL, Tableau, Business Fluency)Wal-Mart Canada Corp · Bentonville, Arkansas - S
Vendavo Uii DeveloperSA Technologies Inc. · Fort Worth, TX - F
WMS ConsultantFulfillment IQ · Atlanta, Georgia
Jobs in Mumbai
- PExecutive -Direct TaxationP G BHAGWAT LLP · Mumbai, MH
- PArticleship- Statutory Audit- APS (Mumbai)P G BHAGWAT LLP · Mumbai, MH
- PAssociate- (Bank and NBFC Audits)P G BHAGWAT LLP · Mumbai, MH
- PSenior Manager- Banks & NBFC AuditsP G BHAGWAT LLP · Mumbai, MH
- PChartered Accountant- Statutory AuditP G BHAGWAT LLP · Mumbai, MH
- PTax Assistant- DSM Team- MumbaiP G BHAGWAT LLP · Mumbai, MH
Browse these categories
Market data for data engineer roles
All reports →- SeriesRole reportsOne role family at a time: how many openings, what changed this week, who is hiring, what it pays.
- SeriesSalary reportsWhat employers publish in job postings, by level and workplace. Not self-reported pay.
- Market overviewState of tech hiring, September 2026: up 4.8%Tech hiring rose 4.8% month over month in September 2026, with 411,122 new listings. Customer support and account executive roles led the growth.