Experience : 7 - 11 Years

Location : Hyderabad / Chennai / Bangalore

Role &

Responsibilities :

- Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and Spark SQL.
- Build and orchestrate complex, parameterized data pipelines using Azure Data Factory (ADF).
- Develop and implement end-to-end data engineering solutions using Azure Data Lakehouse and Medallion Architecture.
- Process structured and unstructured data across batch and streaming environments.
- Optimize Spark jobs, SQL queries, and data pipelines for performance, scalability, and reliability.
- Implement metadata-driven frameworks for data ingestion, transformation, and data quality.
- Develop exception handling, logging, monitoring, and troubleshooting mechanisms for data pipelines.
- Work with Databricks Notebooks, Workflows, Unity Catalog, and Delta Lake.
- Ensure data security, governance, and lifecycle management across the Azure environment.
- Collaborate with data architects, data scientists, BI teams, and other stakeholders to deliver data solutions.
- Take ownership of critical modules, conduct code reviews,



and guide team members on best practices.

Required Skills :

- Strong experience in Azure Databricks, PySpark, and Spark SQL.
- Hands-on experience in Azure Data Factory (ADF) and pipeline orchestration.
- Strong understanding of Apache Spark architecture, including RDDs and DataFrames.
- Experience in designing and developing scalable data pipelines and ETL/ELT workflows.
- Strong SQL skills, including complex query development, optimization, and troubleshooting.
- Experience with Azure Data Lakehouse and Medallion Architecture.
- Knowledge of Delta Lake, Parquet, Avro, JSON, and CSV file formats.
- Experience in batch and streaming data processing.
- Strong understanding of data modeling, data warehousing, and dimensional modeling.
- Experience with Databricks Workflows, Unity Catalog, and cloud data platforms.
- Proficiency in Python, PySpark, SQL, debugging, and unit testing.
- Familiarity with Git, CI/CD, and automated deployment practices.
- Robust problem-solving, collaboration, and ownership skills.

📌 Senior Data Engineer - Azure Databricks/PySpark (India)
🏢 Careernet
📍 India