AWS Lead Data Engineer - R01564133

Job Description – AWS Senior Data Engineer

Company: Brillio Technologies
Role: Senior Data Engineer
Experience: 5–10 Years
Location: India (Hybrid/Remote)
Employment Type: Full-time

Role Overview

We are looking for a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with Spark, PySpark, AWS ecosystem, and modern data architectures, along with exposure to APIs and real-time data processing.

Key Responsibilities

• Design and build scalable data pipelines using Spark / PySpark on Databricks
• Develop and optimize ETL/ELT workflows for large-scale data processing
• Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing
• Build and maintain data lake and data warehouse solutions
• Develop and integrate APIs for data ingestion and consumption
• Implement data processing workflows using Airflow / Autosys
• Optimize data performance using partitioning, caching, and query tuning
• Handle structured and semi-structured data from multiple sources
• Ensure data quality, governance, and reliability of pipelines
• Collaborate with cross-functional teams (Analytics, Product, Engineering)

---

Technical Skills (Must-Have)

• Strong expertise in:
• Apache Spark / PySpark
• Databricks (mandatory)
• SQL (PostgreSQL or similar)
• Hands-on experience with:
• AWS EMR, S3
• Hadoop, Hive ecosystem
• Programming skills:
• Python (must-have)
• Scala (good exposure)
• Strong knowledge of:
• UNIX / Shell scripting
• ETL pipelines and data engineering fundamentals

---

Additional Skills

• Experience with:
• Elasticsearch (data storage & retrieval)
• Workflow orchestration tools (Airflow, Autosys)
• Exposure to:
• API development & integration
• Version control tools (Git / SVN)
• Basic knowledge of HTML (for web/data tasks)

---

Experience Required

• Minimum 5+ years of Data Engineering experience
• Strong hands-on experience in Databricks + Big Data stack
• Proven experience working on large-scale distributed systems

---

Education

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field

---

Why Join Brillio

• Work on cloud-native, large-scale data platforms
• Exposure to cutting-edge Databricks & AWS ecosystems
• Collaborative and high-performance engineering culture