
Posted 3 days ago
Data Engineer, Sr
AI Summary
Designs, develops and maintains scalable data pipelines on Databricks and Azure, owning platform reliability, production incident resolution and DataOps practices to support analytics and AI use cases.
About this role
Overview
The Data & Insights organization at Insight Global is responsible for delivering Trusted Data & Verified Insights to drive business decisions across the company. As a Senior Data Engineer, you will be part of the Data Engineering & Operations team responsible for enhancing and maintaining scalable data pipelines, supporting and improving platform operations processes, and enabling analytics, data science, and AI use cases.
This is a hands-on role reporting to the Data Operations Manager with direct impact on Insight Global’s ability to deliver high-quality, actionable data to the business.
Responsibilities
- Design, develop, and enhance scalable data pipelines supporting analytics, reporting, AI/ML, and operational business processes.
- Own the reliability and stability of enterprise data platforms by proactively monitoring pipelines, troubleshooting production issues, and ensuring adherence to SLAs and operational standards.
- Lead root cause analysis and incident resolution efforts, driving permanent fixes, automation opportunities, and preventative controls to reduce recurring issues.
- Develop and maintain data solutions using Databricks, Azure Data Factory, SQL, and PySpark, following enterprise architecture and engineering standards.
- Drive DataOps best practices, including CI/CD implementation, automated testing, code reviews, deployment automation, release management, and environment governance.
- Improve data quality, performance, observability, and operational efficiency through monitoring, alerting, lineage tracking, logging, and automation initiatives.
- Work with structured, semi-structured, and streaming data, building scalable ingestion and transformation frameworks.
- Collaborate with business stakeholders, analytics teams, architects, product owners, and application teams to deliver reliable and business-aligned data solutions.
- Identify opportunities for continuous improvement in processes, architecture, governance, security, cost optimization, and platform scalability.
Qualifications
Required Skills
- 5-7 years of experience in Data Engineering, DataOps, Analytics Engineering, or Enterprise Data Platforms.
- Strong expertise in advanced SQL development, query optimization, data analysis, and troubleshooting.
- Proficiency in Python and PySpark for large-scale data transformation, pipeline development, and automation.
- Hands-on experience with Databricks, Delta Lake, Azure Data Factory, and Azure data services.
- Experience designing and supporting batch, near real-time, and streaming data solutions.
- Strong understanding of CI/CD pipelines, Git-based source control, pipeline testing, and DevOps/DataOps practices.
- Experience with monitoring, observability, logging, alerting, and production support processes.
- Knowledge of data modeling, data warehousing concepts, and dimensional design principles
Preferred Skills
- Experience with Databricks Unity Catalog.
- Knowledge of data governance, data quality frameworks, security, and compliance requirements.
- Proven experience managing production incidents, performing root cause analysis, and driving long-term remediation plans.
- Ability to prioritize operational stability while balancing enhancement and project delivery commitments.
- Experience working in Agile, Scrum, or Scrumban environments.
This Role Is Ideal If You:
- Are excited to be part of a data team supporting data pipelines in Databricks.
- Thrive in Agile environments and enjoy working across disciplines.
- Take ownership of operational responsibilities and ability to fix data issues.
- Are passionate about delivering reliable, high-quality data solutions.
- Embody Insight Global’s shared values and contribute to a collaborative, high-performance culture.
Skills
Explore related jobs
More jobs at Insight Global
Similar Azure jobs
Jobs in Atlanta
- A
Review Nurse-Medical ClaimsAlliant Health Solutions · Atlanta, GA - M
Retail Sales Ambassador - Dresses Multi-Vendor, Cumberland Mall - Full TimeMacy's · Atlanta, GA - M
Seasonal Retail Sales and Recovery Associate, Lenox Square MallMacy's · Atlanta, GA - M
Retail Sales Ambassador - Dresses Multi-Vendor, Perimeter Mall - Full TimeMacy's · Atlanta, GA - N
Licensed Practical Nurse (LPN) | Atlanta CampusNorthside Hospital Inc. · Atlanta, GA - N
Licensed Practical Nurse (LPN) | Atlanta CampusNorthside Hospital Inc. · Atlanta, GA
Browse these categories
Market data for data engineer roles
All reports →- SeriesRole reportsOne role family at a time: how many openings, what changed this week, who is hiring, what it pays.
- SeriesSalary reportsWhat employers publish in job postings, by level and workplace. Not self-reported pay.
- Market overviewState of tech hiring, September 2026: up 4.8%Tech hiring rose 4.8% month over month in September 2026, with 411,122 new listings. Customer support and account executive roles led the growth.