Job Description: AWS Databricks Data Engineer

Role Overview:

We are looking for an experienced AWS Databricks Data Engineer to design, developand optimize scalable Lakehouse data solutions. The ideal candidate should have robust hands-on experience with DatabricksAWS cloud services, PySpark, Spark SQLand Delta Lakewith the ability to build reliableproduction-grade data pipelines and translate business requirements into scalable data engineering solutions.

Experience Level
- Data Engineer: 5-7 years of overall data engineering experience, including at least 3 years of hands-on experience with Databricks, Spark and Lakehouse implementations.
- Senior Data Engineer / Lead Engineer: 7-12 years of experience with strong expertise in Databricks architecture, data pipeline design, performance optimization, production deployments and leading enterprise-scale data engineering solutions.

Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, Delta Lake, Auto Loader and Delta Live Tables (DLT).




- Design and implement metadata-driven data ingestion and transformation frameworks to enable scalable, reusable and configuration-based pipeline development across multiple data sources.
- Develop framework components for managing source configurations, ingestion rules, schema evolution, data quality validations, audit logging and pipeline execution monitoring.
- Implement Medallion Architecture (Bronze, Silver, Gold) with appropriate data quality checks, validation frameworks, testing, and monitoring.
- Develop ingestion frameworks using Databricks Auto Loader, including checkpointing, schema evolution and handling structured/semi-structured data.
- Configure and manage Unity Catalog including catalogs, schemas, access controls, audit logging, data lineage and security policies.
- Optimize Spark workloads by tuning Spark jobs, cluster configurations, partitioning strategies, and Delta Lake storage layouts for performance and cost efficien

📌 Aws Databricks (Chennai)
🏢 Conoscenza Technology Solutions
📍 Chennai