- Data Pipeline Design: Architect and implement scalable, reliable data pipelines using Databricks, Spark, and Delta Lake
- Medallion Architecture Implementation: Ingest raw data into the Bronze layer, process and clean it in the Silver layer, and aggregate/transform it for analytics in the Gold layer
- Data Modeling: Design and optimize data models for effective storage, retrieval, and analytics

• Data Quality & Security: Implement data validation, quality checks, and security controls throughout the pipeline

• Automation & Monitoring: Automate pipeline execution, monitor performance, and troubleshoot issues

- Collaboration: Work closely with data scientists, analysts, and business stakeholders to deliver high-quality, analytics-ready data

Essential Skills





- Databricks Certified Associate Developer for Apache Spark (or equivalent)

• Strong Python/Scala for Spark development

• Experience with Delta Lake and Spark DataFrame API

- Proven experience migrating from Cloudera to Databricks
- Data pipeline orchestration (Databricks Jobs, Airflow, etc.)
- Data quality and security best practices

Desirable Skills

- Databricks Certified Data Engineer Associate
- SQL and advanced query optimization
- Cloud platforms (Azure, AWS, GCP)
- Real-time/streaming data processing (Spark Structured Streaming)
- DevOps/MLOps (CI/CD, Docker, Kubernetes)
- Experience with data governance and lineage tools

📌 Databricks Developer (Chennai)
🏢 Tekskills
📍 Chennai