For Data Engineer – Databricks Data PlatformRole Overview

Experienced Data Engineer to design, build, and maintain scalable data pipelines and analytics-ready data products on Databricks and AWS. End-to-end ownership of data ingestion, modelling, transformation, quality, and performance optimisation.

Key Responsibilities

- Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.
- Ingest data from databases, APIs, event streams, and flat files.
- Design and implement Medallion Architecture (Bronze / Silver / Gold layers).
- Write data transformation logic in PySpark, Spark SQL, and Python.
- Implement data quality checks, validation rules, and reconciliation processes.
- Optimise Spark workloads for performance, cost, and scalability.
- Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.
- Support CI/CD for pipelines using Git, DevOps tooling, and release processes.
- Monitor pipelines, triage failures, and restore services within agreed SLAs.
- Maintain technical documentation: data models, pipeline flows, runbooks.
- Adhere to data security, privacy, and regulatory compliance standards.

Must-Have Skills
- Databricks: Workflows, Delta Live Tables, Unity Catalog, Databricks SQL.
- Apache Spark: PySpark and Spark SQL for large-scale data processing.
- Delta Lake: ACID transactions, schema evolution, time travel, merge.
- Medallion Architecture or equivalent layered data platform design.
- Python for pipeline development and transformation logic.
- AWS: S3,



Glue, IAM, EC2, Lambda.
- Batch, micro-batch, and streaming ingestion (Structured Streaming / Kafka).
- Data modelling: star schema, snowflake schema, Data Vault.
- CI/CD, Git, version control, and Agile delivery practices.

Good-to-Have Skills
- Unity Catalog for data governance and lineage.
- AWS Kinesis for real-time streaming.
- Data Warehouse and Informatica PWC Knowledge
- dbt for SQL-based transformation workflows.
- AWS Redshift or Athena for cloud data warehousing.
- Terraform or AWS CloudFormation for infrastructure-as-code.
- AWS Step Functions for workflow orchestration.
- Data observability tools (Monte Carlo, AWS Glue Catalog, Atlas).
- Industry experience in insurance or financial services.
- Databricks or AWS Data Analytics certification.

Technical Environment
- Platform: Databricks Lakehouse, Delta Lake, Unity Catalog.
- Languages: Python, PySpark, SQL, Spark SQL.
- Ingestion: Auto Loader, APIs, JDBC, CDC tools.
- Processing: Apache Spark, Structured Streaming, Delta Live Tables.
- Cloud: AWS.
- Orchestration: Databricks Workflows, AWS Step Functions.
- DevOps: Git, CI/CD, Terraform, GitHub Actions.
- Analytics: Power BI, Tableau, Looker.

Experience And Qualifications

- Bachelor's in Computer Science, Engineering, IT, or equivalent.
- 4–8 years in data engineering, ETL/ELT, and cloud data platforms.
- Production-grade pipeline delivery on AWS.
- Agile or DevOps-driven delivery workplace.

📌 Senior Data Engineering (Pune)
🏢 Mphasis
📍 Pune