Summary :

Exp. range : 7 yrs (With 4 yrs of relevant experience as AWS Data Engineer)

Education : B.E / MCA or equivalent

Working model : Work from office (5-day week)

This is a hands-on technical leadership role. The successful candidate will contribute directly to architecture, coding, code reviews, troubleshooting, performance optimization, and implementation while leading and mentoring the data engineering team.

Key Responsibilities :

Technical Architecture & Design :

- Lead technical architecture and design efforts for data engineering solution components to deliver business solutions
- Design and implement scalable, secure and resilient data engineering solutions using AWS and Databricks, leveraging Databricks Lakehouse architecture, Apache Spark, Delta Lake and associated data engineering capabilities.
- Evaluate and select appropriate technology stack for data engineering solution components - cloud-native services, open-source tools, data storage, data processing frameworks etc.

Data Engineering Development :

- Lead, mentor and manage a team of agile data engineers for development, testing, and implementation of data pipelines, ensuring quality, performance, and on-time delivery
- Responsible for detailed technical design and data mapping specifications
- Ensure adherence to industry best practices, coding standards and guidelines driving continuous improvement in code quality and delivery excellence
- Optimize data solutions for performance, cost-efficiency and scalability
- Ensure data quality and integrity for downstream systems and applications

Technical Project Management :

- Plan, prioritize and manage data engineering projects, ensuring timely delivery and quality
- Identify and mitigate risks, troubleshoot issues, handle customer escalations
- Ensure alignment with business requirements and priorities, working closely with stakeholders - both internal and customer





Collaboration and Documentation :

- Maintain comprehensive technical documentation, including architecture diagrams, high-level/low-level data integration design, data mapping specifications, database design etc.
- Work closely with peers, BI engineers and data scientists to enable advanced analytics, machine learning models and reporting solutions

Requirements :

- 7 years of hands-on experience in delivering data engineering solutions
- Experience leading a team of 5 data engineers, including technical mentoring, solution/design reviews, code reviews, estimation, task prioritization, technical guidance and stakeholder communication.
- AWS & Databricks: Extensive hands-on experience designing and implementing data engineering solutions using AWS and Databricks. Strong experience with AWS services such as S3, Glue, Redshift and Athena, along with Databricks Lakehouse, Apache Spark/PySpark and Delta Lake.
- Data Engineering: Proven experience designing and implementing scalable ETL/ELT pipelines using Python, PySpark and Databricks. Experience with Databricks Jobs/Workflows, cluster configuration, performance optimization and data quality.
- Data Architecture: Strong understanding of data lake, data warehouse and Lakehouse architecture patterns, including data ingestion, transformation, storage, partitioning, schema evolution and consumption.
- Databases & SQL: Expert-level SQL skills with experience in relational and NoSQL databases, including query and performance optimization.
- DevOps & CI/CD: Experience with Git/GitHub and implementing CI/CD pipelines for data engineering solutions, including deployment of Databricks notebooks/jobs and associated infrastructure.
- AWS Security: Good understanding of AWS IAM, encryption, access controls, secrets management, and secure data engineering practices.
- Strong analytical and problem-solving skills.
- Excellent communication and collaboration skills

📌 AWS Lead Data Engineer - Databricks (India)
🏢 Spectrum Consulting India
📍 India