Job Title: Senior Data Engineer

Location : Hinjewadi Phase 1, Pune

Qualification: B.E./B. Tech, M.E./M. Tech, MCA, or equivalent in Computer Science/IT or a related field.

Years of experience: 5–8 Years (Minimum 5 Years Relevant Experience)

Employment type: Full Time | Permanent

Overview/ job purpose/ about the company:

We are looking for a Senior Data Engineer to design, develop, optimize, and operate scalable data pipelines on the Data bricks platform. The role involves end-to-end data engineering, legacy ETL modernization, PySpark development, Delta Lake, data quality, performance optimization, and collaboration with cross-functional teams.

Key Responsibilities:

· Design, build, and maintain scalable ETL/ELT data pipelines using Data bricks, PySpark, Python, and SQL.

· Develop end-to-end data workflows using Data bricks Jobs and Workflows with robust monitoring, error handling, and alerting.

· Refactor legacy ETL code and migrate traditional ETL processes to PySpark and up-to-date ELT patterns on Data bricks.

· Design and maintain Delta Lake tables and implement Lake house/Medallion Architecture.

· Optimize Spark jobs and pipeline performance using partitioning, caching, Z-ORDER, OPTIMIZE, and other techniques.

· Implement data quality, validation, testing, and reconciliation frameworks.

· Follow software engineering practices including Git, version control, CI/CD, code reviews, and reusable frameworks.

· Troubleshoot production data pipeline issues and ensure reliability and maintainability.

· Collaborate with data architects, analysts,



business stakeholders, Infrastructure, Applications, and Cyber teams.

· Mentor junior data engineers and document technical solutions and best practices.

Competencies:

· Strong Data Engineering, ETL/ELT, and data pipeline design expertise.

· Hands-on expertise in Data bricks, PySpark, Python, Spark SQL, and SQL.

· Strong experience with Data bricks Workspace, Cluster Management, Notebooks, Jobs, and Workflows.

· Experience with Delta Lake, Lake house Architecture, and Medallion Architecture.

· Experience in legacy code/ETL migration to PySpark and Data bricks.

· Strong understanding of Spark performance optimization.

· Experience with Data Modeling, Dimensional Modeling, Data Vault, or Lake house Architecture.

· Knowledge of Cloud Platforms – Azure, AWS, or GCP.

· Familiarity with Data Governance, Security, Structured Streaming, Git, CI/CD, and Data Quality frameworks.

Additional Information:

· Mandatory Certification: Data bricks Certified Data Engineer Associate OR Data bricks Certified Data Engineer Professional.

· Minimum 2–3 years hands-on Data bricks experience.

· Experience building and maintaining production data pipelines at scale.

· Experience working across multiple data sources and formats.

· Experience in Agile development environments.

· Working Model: Hybrid | 5 Days Working.

· Shift Timing: 8:00 AM – 5:00 PM IST.

· Interview Process: Virtual & Face-to-Face Rounds.

· Strong problem-solving, analytical, communication, and collaboration skills.

📌 Senior Data Engineer (India)
🏢 Insight Advisors Private
📍 India