Lead Data Engineer

Primary Skills

- Athena, Step Functions, Spark - Pyspark, ETL Fundamentals, SQL (Basic Advanced), Glue, Python, Lambda, Data Warehousing, EBS /EFS, AWS EC2, Lake Formation, Aurora, S3, Modern Data Platform Fundamentals, PLSQL, Data Modelling Fundamentals, Cloud front

Specialization

- AWS Data EngineerIng Basic: Senior Data Engineer

Job requirements

- AWS Senior Data Engineer

Company: Brillio Technologies

Role: Senior Data Engineer

Experience: 5-10 Years

Location: India (Hybrid/Remote)

Employment Type: Full-time

Role Overview

We are looking for a Senior Data Engineer with strong Databricks and Big Data expertise to design, build, and optimize scalable data pipelines and platforms. The ideal candidate will have hands-on experience with Spark, PySpark, AWS ecosystem, and modern data architectures, along with exposure to APIs and real-time data processing.

Key Responsibilities

- Design and build scalable data pipelines using Spark / PySpark on Databricks

- Develop and optimize ETL/ELT workflows for large-scale data processing

- Work with AWS EMR, S3, and Hadoop ecosystem for distributed data processing

- Build and maintain data lake and data warehouse solutions

- Develop and integrate APIs for data ingestion and consumption

- Implement data processing workflows using Airflow / Autosys

- Optimize data performance using partitioning, caching, and query tuning





- Handle structured and semi-structured data from multiple sources

- Ensure data quality, governance, and reliability of pipelines

- Collaborate with cross-functional teams (Analytics, Product, Engineering)

Technical Skills (Must-Have)

- Strong expertise in:

- Apache Spark / PySpark

- Databricks (mandatory)

- SQL (PostgreSQL or similar)

- Hands-on experience with:

- AWS EMR, S3

- Hadoop, Hive ecosystem

- Programming skills:

- Python (must-have)

- Scala (positive exposure)

- Strong knowledge of:

- UNIX / Shell scripting

- ETL pipelines and data engineering fundamentals

Additional Skills

- Experience with:

- Elasticsearch (data storage & retrieval)

- Workflow orchestration tools (Airflow, Autosys)

- Exposure to:

- API development & integration

- Version control tools (Git / SVN)

- Basic knowledge of HTML (for web/data tasks)

Experience Required

- Minimum 5 years of Data Engineering experience

- Strong hands-on experience in Databricks Big Data stack

- Proven experience working on large-scale distributed systems

Education

- Bachelor's or Master's degree in Computer Science, Engineering, or related field

Why Join Brillio

- Work on cloud-native, large-scale data platforms

- Exposure to cutting-edge Databricks & AWS ecosystems

- Collaborative and high-performance engineering culture

📌 Sr Data Specialist133 (Bengaluru)
🏢 Brillio
📍 Bengaluru