19 Nov
|
Jobaaj.com
|
Noida
Purpose of the Job
We are looking for a Senior Data Engineer who is equally strong in data modelling, data quality, SQL, and business understanding, as well as engineering skills such as Python, PySpark, CI/CD, performance optimization, and reliability.
The candidate will be responsible for designing and delivering production-ready data platforms, from gathering requirements to deployment and production support. The role also involves working directly with clients and sharing technical knowledge with the wider engineering team.
Technology Stack: Python, SQL, PySpark, Azure Databricks, Microsoft Fabric, Azure Data Factory, Delta Lake, Lakehouse, and Data Warehouse architectures.
Key Responsibilities
1. Own the end-to-end design and delivery of data pipelines and platforms, including requirements gathering, architecture, development, testing, deployment, and production support.
2. Design data models, including dimensional modelling, SCD, and Medallion Architecture (Bronze/Silver/Gold), based on business requirements.
3. Build and maintain scalable ETL/ELT pipelines using Python, SQL, PySpark, and Azure Data Factory on Azure Databricks or Microsoft Fabric.
4. Implement batch, incremental/CDC, and streaming data loads.
5. Optimize Spark and SQL performance, ensuring scalability and cost efficiency.
6. Implement data quality checks, monitoring, and alerting across pipelines.
7. Apply data governance, lineage, and access control practices.
8. Automate testing and deployment using Git and CI/CD while following engineering best practices.
9. Work directly with client stakeholders to gather requirements, explain technical design decisions, and establish acceptance criteria.
10. Integrate data from APIs, transactional systems, and external platforms.
11. Investigate and resolve complex production issues and document their root causes.
12.
Maintain technical documentation and contribute to team knowledge-sharing through peer reviews and technical discussions.
Essential Qualifications
- Bachelor's degree in any discipline, preferably in Computer Science, Information Systems, or a related field.
- 58 years of relevant experience in Data Engineering.
Required Skills and Qualifications
Data Engineering Skills:
- Strong SQL knowledge.
- Experience in dimensional modelling and Medallion/Lakehouse architecture.
- Understanding of data quality practices.
- Ability to translate business requirements into effective data solutions.
Technical and Engineering Skills:
- Strong hands-on experience with Python and PySpark.
- Knowledge of Git, CI/CD, and automated testing.
- Experience in performance tuning of high-volume workloads.
- Hands-on experience with Azure Databricks or Microsoft Fabric.
- Experience with Azure Data Factory or an equivalent orchestration tool.
- Proven experience in developing batch and incremental/CDC pipelines using Delta Lake or similar technologies.
Client Management Skills:
- Experience in gathering client requirements.
- Strong stakeholder communication and collaboration skills.
- Ability to prepare explicit technical documentation.
- Ability to demonstrate measurable improvements in runtime, cost, or data quality.
Good to Have
- Experience with streaming technologies such as Spark Structured Streaming and Kafka.
- Knowledge of governance and lineage tools such as Unity Catalog and Azure Purview.
- Understanding of Power BI and semantic-layer concepts.
- Microsoft certifications such as DP-203 or DP-600.
- Experience with Infrastructure as Code (Terraform).
Benefits and Additional Information
- Medical insurance coverage for employees.
- Provident Fund (PF) facility.
- Quarterly team parties and annual outings/trips.
- Regular leadership check-ins for professional growth and feedback.
- Recognition awards for high performance.
- Team engagement activities and events throughout the year.
📌 Senior Data Engineer (Noida)
🏢 Jobaaj.com
📍 Noida