Role Description
Job Description for Data Engineer – Databricks Data PlatformRole Overview
Experienced Data Engineer to design, build, and maintain scalable data pipelines and analytics-ready data products on Databricks and AWS. End-to-end ownership of data ingestion, modelling, transformation, quality, and performance optimisation.
Key Responsibilities
• Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.
• Ingest data from databases, APIs, event streams, and flat files.
• Design and implement Medallion Architecture (Bronze / Silver / Gold layers).
• Write data transformation logic in PySpark, Spark SQL, and Python.
• Implement data quality checks, validation rules, and reconciliation processes.
• Optimise Spark workloads for performance, cost, and scalability.
• Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.
• Support CI/CD for pipelines using Git, DevOps tooling, and release processes.
• Monitor pipelines, triage failures, and restore services within agreed SLAs.
• Maintain technical documentation: data models,
pipeline flows, runbooks.
• Adhere to data security, privacy, and regulatory compliance standards.
Must-Have Skills
• Databricks: Workflows, Delta Live Tables, Unity Catalog, Databricks SQL.
• Apache Spark: PySpark and Spark SQL for large-scale data processing.
• Delta Lake: ACID transactions, schema evolution, time travel, merge.
• Medallion Architecture or equivalent layered data platform design.
• Python for pipeline development and transformation logic.
• AWS: S3, Glue, IAM, EC2, Lambda.
• Batch, micro-batch, and streaming ingestion (Structured Streaming / Kafka).
• Data modelling: star schema, snowflake schema, Data Vault.
• CI/CD, Git, version control, and Agile delivery practices.
Valuable-to-Have Skills
• Unity Catalog for data governance and lineage.
• AWS Kinesis for real-time streaming.
• Data Warehouse and Informatica PWC Knowledge
• dbt for SQL-based transformation workflows.
• A
📌 Senior Data Engineering (Pune)
🏢 Mphasis
📍 Pune