Must-Have Skills:
- Robust hands-on experience with Azure Databricks
- Expertise in Apache Spark (PySpark) and Spark SQL
- Strong programming skills in Python and SQL
- Experience with Delta Lake including ACID transactions and schema evolution
- Hands-on experience in ETL/ELT pipeline development
- Strong understanding of Lakehouse Architecture
- Experience building and maintaining Medallion Architecture (Bronze, Silver, Gold) data layers
- Experience with batch and streaming data processing
- Experience with Azure Data Factory (ADF) and Azure Data Engineering ecosystem
- Strong understanding of Data Modeling and Analytics data structures
- Experience with Unity Catalog or Data Governance frameworks
- Experience in performance tuning, query optimization, partitioning, and caching
- Data Engineering experience
- Azure Databricks experience

Good-to-Have Skills:
- Git, Azure DevOps, GitHub Actions
- CI/CD implementation and deployment automation
- Terraform (Infrastructure as Code)
- Docker and Kubernetes
- Airflow
- Kafka or Azure Event Hub
- Power BI or Tableau




- Databricks Workflows and Job Orchestration
- Cluster Management and Optimization
- Experience with Delta Table features such as MERGE, UPSERT, and Time Travel
- BFSI domain experience

Key Responsibilities:
- Design, develop, and optimize scalable batch and streaming data pipelines using Azure Databricks.
- Develop data processing frameworks using PySpark and Spark SQL.
- Implement Delta Lake-based storage solutions to ensure high reliability and data consistency.
- Build and maintain Bronze, Silver, and Gold data layers within a Lakehouse architecture.
- Ingest data from APIs, databases, Azure storage, and streaming platforms.
- Develop scalable data models supporting analytics and reporting requirements.
- Implement data quality, validation, and reconciliation processes.
- Monitor and optimize Spark workloads for performance and cost efficiency.
- Establish and support data governance practices through U

📌 Azure Databricks (India)
🏢 Tata Consultancy Services
📍 India