Data Engineer

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Data Engineer based in United States.

This role offers an opportunity to help build and evolve a scalable, self-service data and AI ecosystem within a modern enterprise environment.
You will contribute to data integration, ETL, cloud-based pipelines, and analytics solutions that support critical business needs.
Working primarily with AWS and Databricks, you will develop reliable batch and near-real-time data pipelines.
The position combines hands-on engineering with production support, troubleshooting, automation, and continuous improvement.
You will collaborate closely with senior engineers, architects, analytics teams, infrastructure specialists, business stakeholders, and external partners.
This is a strong opportunity to deepen your expertise in modern data engineering technologies while contributing to high-quality, dependable data solutions.
The role is remote, with candidates in the Chicago metropolitan area preferred for occasional office collaboration.

• Participate throughout the Software Development Life Cycle for data integration, ETL, and cloud-based data pipeline initiatives, including requirements analysis, design, development, testing, deployment, and production support.
• Design, develop, test, deploy, and support batch and near-real-time data pipelines using AWS and Databricks.
• Build data transformations and integration solutions using SQL, Python, Spark, and Delta Lake.
• Develop and maintain Databricks notebooks, workflows, jobs, Delta tables, and related data components.
• Contribute to requirements analysis, estimation, solution design, code reviews, testing, production deployment, and technical documentation.
• Create unit tests and support integration, reconciliation, performance, regression, and data quality testing.
• Monitor data pipelines, ETL processes, and cloud platform components, investigating production issues and supporting timely resolution.
• Diagnose data integration and pipeline failures, participate in root cause analysis, and contribute to corrective and preventive actions.
• Maintain source-to-target mappings, transformation rules, data flow diagrams, troubleshooting guides, and operational documentation.
• Collaborate with senior engineers, architects, analytics teams, infrastructure teams, business stakeholders, and external partners to deliver scalable data solutions.
• Support platform upgrades, deployment automation, monitoring improvements, and operational process enhancements.
• Participate in a scheduled on-call rotation and provide occasional evening or weekend support for production environments and project requirements.
• Stay current with AWS, Databricks, Spark, Python, SQL, Delta Lake, Unity Catalog, and emerging data engineering practices.
• Identify automation, process improvement, and operational efficiency opportunities that strengthen platform reliability and delivery quality.

Requirements:

• Bachelor’s degree in Information Technology, Computer Science, Statistics, Economics, Mathematics, Engineering, or another quantitative or technical discipline.
• 1–3 years of experience in software development, data engineering, ETL development, data integration, analytics, or a related technical field within a complex enterprise environment.
• Experience or exposure to data integration, ETL processes, data warehousing, or cloud-based data platforms.
• Working knowledge of SQL and relational databases for data extraction, transformation, and analysis; Oracle and PL/SQL experience is a plus.
• Basic programming experience with Python, SQL, or similar languages used in data engineering and analytics.
• Working knowledge of Apache Spark, preferably PySpark, and distributed data processing concepts.
• Familiarity with Databricks, Delta Lake, Databricks Workflows and Jobs, and cloud-based data engineering technologies.
• Basic understanding of AWS, including services such as Amazon S3, IAM, and cloud-native data storage concepts.
• Understanding of data warehousing, data modeling, data mapping, data transformation, data quality, governance, and integration principles.
• Familiarity with modern data platform concepts, including data lakes, data warehouses, lakehouse architecture, Medallion Architecture, and batch and streaming processing.
• Exposure to incremental data loading, Change Data Capture (CDC), streaming data fundamentals, and scalable data integration patterns.
• Familiarity with Git-based version control, CI/CD concepts, automated deployments, environment management, code reviews, and software development best practices.
• Basic understanding of data security, access controls, governance, data lineage, compliance, and enterprise data lifecycle management.
• Familiarity with Databricks Unity Catalog or similar governance tools is preferred.
• Understanding of cloud cost management, performance monitoring, and optimization concepts is a plus.
• Familiarity with Agile methodologies and tools such as Jira is beneficial.
• Strong analytical, problem-solving, troubleshooting, communication, and collaboration skills, with the ability to learn new technologies quickly.
• Demonstrated attention to detail and commitment to delivering reliable, maintainable, and high-quality solutions.
• Familiarity with data visualization concepts, Microsoft Office Suite, enterprise data platforms, and transactional system integrations is preferred.
• Asset leasing industry experience, particularly in the rail sector, is a plus.
• Relevant certifications such as Databricks Certified Data Engineer, AWS Certified Data Analytics, or Azure Data Engineer are preferred.
• Working knowledge of Unix shell scripting is a plus.
• Ability to accommodate occasional travel and scheduled production support outside standard working hours when required.

Benefits:

• Annual salary range of $80,800–$96,000 USD.
• Eligibility for a short-term incentive plan, subject to applicable plan terms.
• Remote work arrangement, with periodic access to an office environment for candidates in the Chicago metropolitan area.
• Opportunities to work with modern cloud and data technologies, including AWS, Databricks, Spark, Python, SQL, and Delta Lake.
• Career development opportunities through hands-on experience, training, self-development, and exposure to enterprise data and AI initiatives.
• Benefits and employee programs designed to support professional and personal well-being.
• Equal opportunity employment and a commitment to an inclusive workplace.