Job Summary

Key Responsibilities

- Design, develop, and maintain scalable data pipelines using Databricks.
- Build and optimize ETL/ELT processes for structured and unstructured data.
- Develop solutions using PySpark, Spark SQL, Delta Lake, and Databricks Workflows.
- Implement data ingestion frameworks from various source systems including APIs, databases, cloud storage, and streaming platforms.
- Design and maintain Bronze, Silver, and Gold data layers following Medallion Architecture.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Integrate Databricks solutions with cloud platforms such as Azure, AWS, or GCP.
- Collaborate with business analysts, data scientists, architects, and stakeholders to gather requirements.
- Ensure data quality, governance, security, and compliance standards.
- Monitor and troubleshoot production data pipelines and workflows.
- Participate in code reviews and implement data engineering best practices.

Key Responsibilities

Required Qualifications

- 7+ years of experience in Data Engineering and Data Warehousing.
- 4+ years of hands-on experience with Databricks.
- Robust expertise in:
- - PySpark
- Spark SQL
- Python
- SQL
- Delta Lake

- Experience building large-scale batch and streaming data pipelines.
- Strong understanding of data modeling and data warehousing concepts.




- Experience with cloud platforms (Azure, AWS, or GCP).
- Knowledge of CI/CD and DevOps practices.

Skill Requirements

Technical Skills

Databricks & Big Data

- Databricks Lakehouse Platform
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake
- Databricks Workflows
- Unity Catalog

Programming

- Python
- SQL
- Scala (Preferred)

Cloud Platforms

- Azure Databricks
- Azure Data Factory (ADF)
- AWS EMR / Glue
- Google Cloud Platform (GCP)

Data Storage

- Azure Data Lake Storage (ADLS)
- Amazon S3
- Snowflake
- Synapse Analytics
- SQL Server
- Oracle

Data Integration

- Kafka
- Event Hub
- REST APIs
- CDC Implementation

DevOps & Monitoring

- Git
- Azure DevOps
- Jenkins
- Terraform
- Monitoring and Logging Tools

Must Have Skill

Azure Data Factory (ADF),Azure Databricks,Snowflake

Good to Have Skill

Other Requirements

Preferred Qualifications

- Experience with Azure Data Factory (ADF), Synapse Analytics, or Snowflake.
- Knowledge of Data Governance and Unity Catalog.
- Experience building data lakes and lakehouse architectures.
- Exposure to machine learning workflows and MLOps.
- Databricks Certification:
- - Databricks Certified Data Engineer Associate
- Databricks Certified Data Engineer Professional

Experience working in Agile/Scrum environments.

📌 Senior Data Lead - Azure Data Factory (ADF), Databricks (India)
🏢 HCLTech
📍 India