Job Summary

Updated - Senior Azure Data Engineer

Location: Hyderabad

Summary

Seeking a Senior Data Engineer to architect and deliver enterprise-scale Azure data platforms, lead modernization and migration initiatives, and mentor engineering teams. The ideal candidate should possess deep expertise in Azure Databricks, Azure Synapse, Data Lakehouse architecture, PySpark, and up-to-date data engineering practices. In addition, the candidate should effectively leverage AI-powered engineering tools and agents to accelerate development, improve quality, automate repetitive tasks, and increase delivery efficiency.

Key Responsibilities

Data Platform & Lakehouse Engineering

- Design end-to-end Azure data architecture using Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage Gen2, Delta Lake, and Medallion Architecture.
- Lead migrations from Informatica PowerCenter and legacy ETL platforms to Azure-native solutions.
- Perform migration assessments, complexity analysis, effort estimation, and modernization planning.
- Design scalable ingestion, transformation, and consumption layers following Data Lakehouse best practices.
- Implement data modeling, partitioning strategies, schema evolution, and performance optimization techniques.

Data Engineering & Pipeline Development

- Build batch and near real-time data pipelines using Databricks, PySpark, Spark SQL, Azure Data Factory, and Synapse.
- Design and implement CDC, SCD, UPSERT/MERGE, reconciliation, and data quality frameworks.
- Develop reusable ingestion and transformation frameworks to improve delivery consistency and speed.
- Optimize Databricks workloads for performance, scalability, reliability, and cost efficiency.

AI-Assisted Engineering & Agentic Development

Leverage GitHub Copilot, Microsoft Copilot,



Databricks Assistant, and AI-powered tools to accelerate SQL/PySpark development, ETL pipelines, test data generation, documentation, troubleshooting, and overall engineering productivity. Develop and utilize AI agents and intelligent workflows to automate repetitive data engineering tasks, improve code quality, and accelerate delivery while ensuring security, governance, and maintainability.

DevOps, Governance & Security

- Enforce coding standards, automated testing, code reviews, and release governance.
- Promote observability, monitoring, alerting, operational readiness, and platform governance.

Leadership & Delivery

- Lead technical design discussions and architecture reviews.
- Mentor junior and mid-level engineers.
- Drive engineering excellence, reusable frameworks, and best practices.
- Collaborate with architects, business stakeholders, and global teams to deliver successful outcomes.

Required Skills

- Expert-level experience in Azure Databricks, Azure Synapse Analytics, Azure Data Factory, PySpark, Spark SQL, and Azure Data Lake Storage Gen2.
- Strong hands-on experience with Informatica PowerCenter (preferred for migration programs).
- Advanced SQL expertise across Oracle, SQL Server, PostgreSQL, and modern cloud data platforms.
- Deep understanding of Delta Lake, Lakehouse Architecture, Unity Catalog, Data Governance, and Medallion Architecture.




- Experience implementing CDC, SCD, Data Quality Frameworks, Data Reconciliation, and Data Lineage solutions.
- Hands-on experience with Git, GitHub, CI/CD, Azure DevOps, and automated deployment practices.
- Strong knowledge of Azure security services including Key Vault, Managed Identity, and Entra ID.

AI & Productivity Engineering Skills

- Proven experience using GitHub Copilot, Microsoft Copilot, Databricks Assistant, or similar AI-powered development tools.
- Ability to leverage AI agents for:
- Code generation
- Pipeline development
- Test data generation
- Automated documentation
- Unit test creation
- Data validation and reconciliation
- Code review and optimization

- Demonstrated ability to improve engineering productivity and delivery efficiency using AI-assisted development practices.

Preferred Skills

- Terraform and Infrastructure as Code.
- Azure AI Foundry, Azure OpenAI, Agentic AI solutions.
- AI-assisted testing and synthetic test data generation.
- Experience integrating data platforms with Machine Learning and AI workloads.
- Insurance and Financial Services domain experience.

Soft Skills

- Strong technical leadership and mentoring capabilities.
- Excellent stakeholder management and communication skills.
- Ability to work effectively with global and cross-functional teams.
- Strong problem-solving mindset with focus on automation and continuous improvement.
- Comfortable working in Agile and DevOps delivery models.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Software Engineer I (Hyderabad)
🏢 MetLife
📍 Hyderabad