Locations: Multiple, Bangalore, Chennai, Hyderabad, India
Work Arrangement: Remote Work from Home
About This Role:
We are seeking a talented and experienced Senior Data Engineer to join our data engineering team. You will be responsible for designing, building, and maintaining scalable data pipelines and infrastructure on cloud platforms. This role requires expertise in modern data engineering tools and a strong foundation in data transformation and optimization.
Key Responsibilities
- Design and develop robust, scalable data pipelines using Azure Data Factory (ADF)
- Build and optimize data processing workflows using Databricks and Apache Spark
- Implement declarative pipeline solutions using Lakehouse architecture and best practices
- Write and optimize complex SQL queries for data transformation and analysis
- Develop PySpark applications for large-scale data processing
- Collaborate with data scientists, analysts, and stakeholders to understand data requirements
- Ensure data quality, security, and compliance across all pipelines
- Monitor,
troubleshoot, and optimize data pipeline performance
- Document technical solutions and maintain best practices documentation
- Mentor junior team members and contribute to team knowledge sharing
What we're Looking For
We need a Senior Data Engineer who is:
- Proficient in building scalable data solutions using Azure and Databricks
- Skilled in SQL and PySpark for data transformation and processing
- Capable of implementing declarative, maintainable pipeline architectures
- Detail-oriented with a focus on data quality and performance optimization
- Team-oriented and able to work effectively with cross-functional teams
- Proactive in staying current with emerging data engineering technologies and best practices
- Communicative and able to explain complex technical concepts clearly