Exp- 5+ yrs

Location- Remote (Preferred candidates to be in bangalore)

Notice- Looking candidates with to be joining within 30 Days

Key Responsibilities:

- Design, implement, and optimize scalable data pipelines using Databricks and Apache Spark.

- Architect data lakes using Delta Lake, ensuring reliable and productive data storage.

- Manage metadata, security, and lineage through Unity Catalog for governance and compliance.

- Ingest and process streaming data using Apache Kafka and real-time frameworks.

- Collaborate with ML engineers and data scientists on LLM-based AI/GenAI project pipelines.

- Apply CI/CD and DevOps practices to automate data workflows and deployments (e.g., with GitHub Actions, Jenkins, Terraform).

- Optimize query performance and data transformations using advanced SQL.

- Implement and uphold data governance, quality, and access control policies.

- Support production data pipelines and respond to issues and performance bottlenecks.





- Contribute to architectural decisions around data strategy and platform scalability.

Requirements

Required Skills & Experience:

- 5+ years of experience in data engineering roles.

- Proven expertise in Databricks, Delta Lake, and Apache Spark (PySpark preferred).

- Deep understanding of Unity Catalog for fine-grained data governance and lineage tracking.

- Proficiency in SQL for large-scale data manipulation and analysis.

- Hands-on experience with Kafka for real-time data streaming.

- Solid understanding of CI/CD, infrastructure automation, and DevOps principles.

- Experience contributing to or supporting Generative AI / LLM projects with structured or unstructured data.

- Familiarity with cloud platforms (AWS, Azure, or GCP) and data services.

- Strong problem-solving, debugging, and system design skills.

- Excellent communication and collaboration abilities in cross-functional teams.

📌 Sr Data Engineer - Databricks (Bengaluru)
🏢 Important Business
📍 Bengaluru