Posted 4 months ago

Lead Software Engineer, Data Engineering

ChennaiOn-siteFull-time

AI Summary

Leads the design, development, and operation of data ingestion, transformation, and orchestration pipelines and Lakehouse architectures on Azure for life-sciences clients.

About this role

About ValGenesis

ValGenesis is a leading digital validation platform provider for life sciences companies. ValGenesis suite of products are used by 30 of the top 50 global pharmaceutical and biotech companies to achieve digital transformation, total compliance and manufacturing excellence/intelligence across their product lifecycle.

Learn more about working for ValGenesis, the de facto standard for paperless validation in Life Sciences: https://www.valgenesis.com/about

About the Role:

Responsibilities

Architect, Design, develop, and maintain data ingestion, transformation, and orchestration pipelines (batch and real-time).

Build and optimize data Lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.

Integrate and manage structured and unstructured data sources (SQL/NoSQL, files, documents, IoT streams).

Develop and operationalize ETL/ELT pipelines using Azure Data Factory, Databricks, or Apache Spark.

Collaborate with Data Scientists to prepare and serve ML-ready datasets for model training and inference.

Implement data quality, lineage, and governance frameworks across pipelines and storage layers.

Work with BI tools (Power BI, Superset, Tableau) to enable self-service analytics for business teams.

Deploy and maintain data APIs and ML models in production using Azure ML, Kubernetes, and CI/CD pipelines.

Ensure scalability, performance, and observability of data workflows through effective monitoring and automation.

Collaborate cross-functionally with engineering, product, and business teams to translate insights into action.

Mentor junior team members and review their work

Requirements

Experience: 6 to 12 years in Data Engineering or equivalent roles.

Programming: Strong in Python, SQL, and at least one compiled language (C#, Java, or Scala).

Databases: Experience with relational (SQL Server, PostgreSQL, MySQL) and NoSQL (MongoDB, Cosmos DB) systems.

Data Platforms: Hands-on experience with Azure Data Lake, Databricks.

ETL/ELT Tools: Azure Data Factory, Apache Airflow, or dbt.

Messaging & Streaming: Kafka, Event Hubs, or Service Bus for real-time data processing.

AI/ML Exposure: Familiarity with ML frameworks (TensorFlow, PyTorch) and MLOps concepts.

Visualization & Analytics: Power BI, Apache Superset, or Tableau.

Cloud & DevOps: Azure, Docker, Kubernetes, GitHub Actions/Azure DevOps.

Best Practices: Solid understanding of data modeling, version control, and CI/CD for data systems.

Preferred skills

Preferred Skills (Nice to Have):

Experience in knowledge graph or semantic search solutions.

Understanding of LLM-based data retrieval (RAG) patterns.

Exposure to data mesh, data fabric, or domain-oriented data architecture.

Familiarity with MLflow, Delta Live Tables, or Data Bricks Unity Catalog.

Soft skills

Strong analytical and problem-solving ability.

Excellent communication and collaboration skills.

Attention to detail and ability to work with large, complex datasets.

Creativity and ability to automate repetitive workflows.

Passion for continuous learning and innovation in data and AI technologies.

Skills

Apache AirflowApache SupersetAzureAzure Data FactoryAzure Data LakeAzure DevOpsC++CI/CD For DataCosmos DBDatabricksData ModelingDbtDockerEvent HubsGitHub ActionsJavaKafkaKubernetesMongoDBMySQLPostgreSQLPower BIPythonPyTorchScalaService BusSQLSQL ServerTableauTensorFlowVersion Control

Lead Software Engineer, Data Engineering

About this role

Responsibilities

Requirements

Preferred skills

Soft skills

Skills

Explore related jobs

More jobs at ValGenesis

Similar Apache Airflow jobs

Jobs in Chennai

Browse these categories

Market data for data engineer roles