Job Title: Data Engineer + Databricks Developer
- Location: Hyderabad, India
- Experience Required: 5+ Years
- Technical Knowledge:
Databricks (PySpark, SQL, Notebooks), Delta Lake, Apache Spark, Python, Azure Data Factory (ADF), Azure Synapse Analytics, AWS S3, Glue, Lambda, CI/CD Pipelines, GitLab, Data Quality Tools, Power BI/Tableau, Data Cataloging, Data Governance, Unity Catalog, MLflow, Job Clusters
- Role Summary:
- Key Responsibilities:
Design, build, and optimize robust ETL/ELT pipelines on Databricks using PySpark, SQL, and Delta Lake.

Lead the development of reusable and modular Notebooks, Jobs, and Workflows for both batch and streaming workloads.

Collaborate with data architects and engineers to build scalable data lakes and data warehouse layers on cloud platforms.

Implement best practices for coding standards, version control, testing, and performance tuning in Databricks.

Use Unity Catalog to manage metadata, lineage, access policies, and data governance across workspaces.

Integrate Databricks with orchestration tools like ADF, AWS Glue, or Airflow,



depending on the workplace.

Build and manage CI/CD pipelines using GitLab, GitHub Actions, or other DevOps tools for automated deployments.

Work with business users and data scientists to develop high-performance pipelines for analytics and ML models.

Monitor and troubleshoot Databricks Jobs, Job Clusters, and compute usage, optimizing for cost and speed.

Ensure security and compliance across data assets, leveraging row-level security, tokenization, and audit logging.

Maintain comprehensive documentation of pipelines, job configs, and data processing logic.
- Requirements:
5+ years of hands-on experience working with Databricks, building scalable, distributed data pipelines using PySpark and SQL.

Strong understanding of Apache Spark internals, cluster tuning, job performance optimization, and partitioning strategies.

Expertise in working with Delta Lake, schema evolution, and ACID-compliant

📌 Data Engineer + Data Bricks Developer (Hyderabad)
🏢 Databeat
📍 Hyderabad