ADF (Azure Data Factory)Databricks Pyspark Technology->Big Data - Data Processing->PySpark Technology->Cloud Integration->Azure Data Factory (ADF) Technology->Data Engineering->Databricks Preferred Qualifications:

- Experience designing medallion/layered data architectures and implementing reusable transformation patterns in Databricks • Strong understanding of data modeling concepts and building curated datasets for analytics consumption • Experience implementing CI/CD practices for data pipelines and notebooks, including automated testing and deployment • Proven ability to lead technical discussions, mentor team members, and drive engineering best practices • Experience improving observability (metrics, alerts, dashboards) and reducing pipeline failures through proactive monitoring Key Responsibilities: Data Engineering & Delivery • Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing • Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability • Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing • Build reusable frameworks, templates,



and standards for pipeline development and deployment Architecture & Performance • Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices • Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing) • Establish monitoring, alerting, and operational runbooks for production pipelines Leadership & Collaboration • Provide technical leadership, code reviews, and mentoring to ensure high engineering standards • Collaborate with stakeholders to translate business requirements into scalable data solutions • Drive delivery planning, estimation, and risk management for data engineering initiatives Minimum Qualifications:
- BTECH, MTECH, MCA, MSC (or equivalent) in Computer Science, Engineering, or related field • 7–9 years of experience in data engineering with strong hands-on delivery ownership • Strong expertise in Azure Data Factory (ADF) for pipeline orchestration, scheduling, and integration patterns • Solid expertise in Databricks for building scalable data processing solutions • Hands-on proficiency with PySpark for building and optimizing distributed data transformations • Experience building production-grade pipelines with logging, error handling, and operational support readiness

📌 Senior Data Engineer - Azure Data Factory, Databricks & PySpark (Bengaluru)
🏢 Infosys
📍 Bengaluru