Role- AWS Databricks, Pyspark

Desired Experience Range- 7-11 Years

Location- Chennai, Pan India

Roles & Responsibility -

- Robust hands-on experience with AWS Databricks, Pyspark, Delta Lake, Unity Catalog, Workflows/Jobs and Lakehouse architecture
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Delta Lake, Auto Loader and Delta Live Tables (DLT).
- Design and implement metadata-driven data ingestion and transformation frameworks to enable scalable, reusable and configuration-based pipeline development across multiple data sources.
- Develop framework components for managing source configurations, ingestion rules, schema evolution, data quality validations, audit logging and pipeline execution monitoring.
- Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality checks, validation frameworks, testing, and monitoring.
- Develop ingestion frameworks using Databricks Auto Loader, including checkpointing, schema evolution and handling structured/semi-structured data.
- Configure and manage Unity Catalog including catalogs, schemas, access controls, audit logging, data lineage and security policies.
- Optimize Spark workloads by tuning Spark jobs, cluster configurations, partitioning strategies,



and Delta Lake storage layouts for performance and cost efficiency.
- Build and manage workflows using Databricks Jobs/Lakeflow Jobs for batch and streaming data processing.
- Implement CI/CD practices and integrate with Git-based DevOps processes.
- Integrate Databricks solutions with AWS services such as S3, IAM, Glue, Step Functions and other AWS data services.
- Enable analytics consumption through BI tools such as Power BI, Tableau or Looker using optimized connectivity patterns.
- Collaborate with data architects, analysts, data scientists and business stakeholders to deliver enterprise data platform solutions.
- Troubleshoot pipeline failures, performance issues and operational challenges while maintaining technical documentation.

Nice to Have-

- Experience with Databricks advanced capabilities such as MLflow, Feature Store, Vector Search and GenAI workloads.
- Knowledge of Delta Sharing and Databricks Marketplace.
- Experience migrating workloads from platforms such as Snowflake, Azure Databricks or traditional data warehouses.
- Exposure to enterprise-scale data platform implementations.

📌 AWS Databricks, Pyspark (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru