Senior Data Engineer - Azure Databricks & Microsoft FabricRole Overview :We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Python, PySpark, SQL, Azure Databricks, and Microsoft Fabric. The candidate will be responsible for designing and maintaining scalable batch and real-time data pipelines, developing lakehouse architectures, and enabling reliable data solutions for analytics and AI/ML applications.Key Responsibilities :Data Pipeline & Lakehouse Engineering :- Design, implement, and maintain high-performance batch and real-time streaming data pipelines using Python, PySpark, and Spark SQL. Build multi-tier Medallion architectures (Bronze, Silver, and Gold) using Delta Lake principles.Data Platform Delivery :- Leverage OneLake, Direct Lake, Lakehouses, and Data Factory Gen2 to streamline data ingestion, optimise semantic models, and reduce redundant ETL processes.API & Systems Integration :- Build resilient API connectors, Change Data Capture (CDC) workflows, and data integration pipelines to ingest data from multiple heterogeneous sources.AI & Analytics Enablement :- Structure, clean, and optimise lakehouse data layers for Power BI reporting and AI/ML applications,



ensuring data is validated and structured for AI grounding.Data Quality & Governance :- Implement data validation, access controls, data lineage, and metadata management using Unity Catalog, platform governance capabilities, and automated testing frameworks.Technical Mentorship & CI/CD :- Conduct code reviews, enforce engineering best practices, optimise query performance, and manage automated deployment pipelines.Required Technical Skills :- Advanced hands-on proficiency in Python, PySpark, and SQL, including Spark SQL and T-SQL.- Strong experience with Azure Databricks, including Delta Lake, Unity Catalog, Delta Live Tables, Auto Loader, and Genie.- Demonstrated experience developing data pipelines and workspace components within Microsoft Fabric.- Expertise in Azure cloud services, including ADLS Gen2, Azure Data Factory (ADF), Azure Key Vault, and Azure SQL / Azure Synapse.- Robust understanding of data engineering requirements for AI/ML workloads, including knowledge of vector embeddings, RAG data ingestion, and feature stores.Education :- Bachelor's or Master's degree in Computer Science, Information Technology, or a related quantitative field. (ref:hirist.tech)

📌 Senior Data Engineer - Azure Databricks (Hyderabad)
🏢 Primesoft
📍 Hyderabad