Data Engineer

Data Engineer:

The Everything Managed Group is a dynamic, innovative and technology driven business committed to leveraging data-driven insights to optimise operations and drive strategic decision-making. We are now seeking a talented Data Engineer to play a significant part in the design, delivery and ongoing evolution of the EMG data platform.

Summary:

As a Data Engineer, you will play a critical role in transforming data into actionable insights and supporting our data infrastructure. You will collaborate with cross-functional teams to gather requirements, design data pipelines and transformations, and develop analytical solutions to address business challenges.

Responsibilities:

1. Collaborate with cross-functional teams, including system administrators, developers / engineers, external partners, and business stakeholders, to understand requirements and deliver solutions.
2. Design, build and maintain scalable data pipelines to ingest, process and transform structured and unstructured data from diverse sources.
3. Optimise solutions for performance, reliability, and scalability, ensuring efficient data processing and storage.
4. Perform exploratory profiling and validation on new data sources to assess quality, structure and fitness for purpose prior to pipeline development.
5. Implement and manage orchestration workflows to automate data pipeline scheduling monitoring and error handling.
6. Design and implement data quality and consistency schemes to ensure the integrity and consistency of data, management information and reporting.
7. Design and implement well-structured optimised data models that enable accurate, performant reporting and analysis by downstream stakeholders both within the team and across the wider organisation.
8. Ensure the data platform is fully documented, maintaining up-to-date information requirements, data models, data dictionaries, and ETL mappings.
9. Provide ad-hoc deep dive analysis, reporting requests and management information to business users.
10. Communicate technical concepts and findings to non-technical audiences in a clear and understandable manner.
11. Provide training and support to others, sharing best practices and fostering culture of continuous learning.
12. Plan and prioritise work to ensure on time delivery of tasks.

Skills and Experience:

• Degree in Mathematics, Statistics, Computer Science, Engineering or a related field or equivalent practical experience in a data engineering role. .
• Ideally experienced as a Data Engineer with hands-on experience designing and delivering data pipelines using industry-standard tools and cloud technologies.
• Knowledge of data engineering concepts, data pipeline development, ETL processes, data lake and data warehousing technologies.
• Proficiency in working with at least one major cloud data platform or lakehouse technology.
• Data modelling skills (including 3NF and dimensional modelling).
• Good knowledge of relational database technologies and concepts.
• Excellent analytical and problem-solving skills with a keen attention to detail.
• Good verbal and written communication skills with the ability to collaborate across teams and present findings to stakeholders.
• Experience with data transformation and/or distributed processing concepts, technologies and best practices is a benefit.
• Proficiency in Python for data engineering tasks (e.g. pipeline scripting, data transformation, automation)
• Experience using version control (Git) and familiarity with CI/CD practices for data engineering workflows.
• Ability to work both independently and collaboratively in a dynamic environment.
• Strong customer focus and stakeholder skills.
• Ability to prioritise tasks and work effectively in a fast-paced environment.