Key Responsibilities

- Design, build, and maintain scalable batch and streaming data pipelines using Apache Spark and cloudnative data technologies.

- Develop and optimize ETL/ELT workflows to ingest, transform, and curate data from diverse source systems into analyticsready datasets.

- Implement data modeling and transformation logic to support reporting, dashboards, and downstream analytical and machine learning workloads.

- Build and manage data processing workloads within modern Lakehouse platforms, including Microsoft Fabric / One Lake (preferred).

- Ensure data quality, reliability, and consistency by implementing validation checks, monitoring, and reconciliation processes.

- Optimize Spark jobs for performance, cost efficiency, and scalability across large and complex datasets.

- Manage and evolve data schemas while handling schema drift and upstream source changes.

- Develop reusable frameworks, libraries, and standardized patterns to improve data engineering productivity and consistency.

- Implement CI/CD pipelines for data workloads to enable automated testing, deployment, and rollback.

- Monitor data pipelines and jobs, troubleshoot failures, and resolve performance or data quality issues.

- Partner with analytics engineers, BI developers, and data scientists to understand data requirements and deliver curated datasets.

- Collaborate with platform, security, and governance teams to ensure data security, compliance, and proper access controls.





- Contribute to Agile delivery processes, including sprint planning, design reviews, and continuous improvement initiatives.

Required Qualifications

- Solid experience as a Data Engineer building and operating production data pipelines.

- Handson experience with Apache Spark for largescale data processing.

- Proficiency in Python, SQL, and data transformation best practices.

- Experience with cloudbased data platforms and storage (e.g., Data Lakes, Lakehouse architectures).

- Familiarity with Microsoft Fabric, One Lake, or similar analytics platforms (strong plus).

- Experience designing and optimizing data models for analytical workloads.

- Understanding of distributed data processing concepts, performance tuning, and fault tolerance.

- Experience with CI/CD, version control, and infrastructureascode concepts.

- Strong problemsolving skills and ability to troubleshoot complex data issues.

- Excellent communication skills and ability to collaborate across technical and nontechnical teams.

- 4+ years of experience in data engineering or equivalent role (preferred).

Preferred / NicetoHave Skills

- Experience with realtime or eventdriven data processing.

- Familiarity with data governance, metadata management, and data quality frameworks.

- Exposure to orchestration tools and workflow management systems.

- Experience supporting analytical, reporting, or machine learning use cases.Role & responsibilities

📌 Data Engineer (Bengaluru)
🏢 Allstate Solutions (ASPL)
📍 Bengaluru