What You''ll Do

- Own and evolve our Databricks Lakehouse platform Unity Catalog governance, Delta Lake table management, and Lakeflow Declarative Pipelines.
- Design and build reliable, scalable data pipelines using Apache Spark and dbt for batch and near-real-time processing.
- Stream and ingest data at scale via Kafka; orchestrate workflows with Airflow.
- Partner with the integrations team to onboard current data sources and support AI/ML and analytics use cases.
- Own data quality and observability end-to-end using Datadog; proactively identify and resolve discrepancies across data layers.
- Manage infrastructure as code using Terraform; ensure platform reliability and cost efficiency on AWS.
- Build and maintain MongoDB and PostgreSQL data stores; define schema standards and access patterns.
- Deliver data products surfaced in Tableau; collaborate with analytics and product teams on reporting requirements.
- Mentor engineers, lead code reviews, and drive engineering best practices across the team.
- Participate in hiring and onboarding of new team members. Required




- Deep hands-on expertise with Databricks Unity Catalog, Delta Lake, and Lakeflow Declarative Pipelines.
- Strong Apache Spark experience tuning, optimization, and production operations.
- Proficiency in Scala (required) and Python for data pipeline development.
- Experience with Kafka for real-time data ingestion and streaming architectures.
- Solid dbt skills for SQL-based transformation and data modeling.
- Familiarity with Airflow for workflow orchestration.
- Infrastructure experience with Terraform and AWS (S3, IAM, networking fundamentals).
- Experience with MongoDB and PostgreSQL for operational and analytical workloads.
- Strong communication skills able to translate complex technical concepts for non-technical audiences.

Keywords: Data Platform: Databricks (Unity Catalog, Delta Lake, Lakeflow Declarative Pipelines), Apache Spark

Databases: MongoDB, PostgreSQL

Cloud: AWS (S3, EMR, Lambda, Glue, IAM, VPC)

Streaming & Messaging: Apache Kafka

Orchestration: Apache Airflow

📌 Sr Data Engineer (HITEC City)
🏢 Zeta Global
📍 HITEC City