Technology->Big Data - Data Processing->PySpark Technology->Cloud Integration->Azure Data Factory (ADF) Technology->Data Engineering->Databricks

Key Responsibilities: Data Engineering & Delivery

- Lead end-to-end development of data pipelines using ADF for orchestration and Databricks for scalable processing
- Design and implement robust ETL/ELT workflows, ensuring data quality, reliability, and maintainability
- Develop optimized transformations and jobs using PySpark in Databricks for batch and incremental processing
- Build reusable frameworks, templates, and standards for pipeline development and deployment Architecture & Performance
- Define solution architecture for ingestion, transformation, and serving layers aligned to platform best practices
- Tune Spark jobs for performance and cost efficiency (partitioning, caching, shuffle optimization, file sizing)
- Establish monitoring, alerting, and operational runbooks for production pipelines Leadership & Collaboration
- Provide technical leadership, code reviews, and mentoring to ensure high engineering standards
- Collaborate with stakeholders to translate business requirements into scalable data solutions
- Drive delivery planning, estimation, and risk management for data engineering initiatives Minimum Qualifications:
- BTECH, MTECH, MCA,



MSC (or equivalent) in Computer Science, Engineering, or related field
- 7–9 years of experience in data engineering with solid hands-on delivery ownership
- Strong expertise in Azure Data Factory (ADF) for pipeline orchestration, scheduling, and integration patterns
- Strong expertise in Databricks for building scalable data processing solutions
- Hands-on proficiency with PySpark for building and optimizing distributed data transformations
- Experience building production-grade pipelines with logging, error handling, and operational support readiness Preferred Qualifications:
- Experience designing medallion/layered data architectures and implementing reusable transformation patterns in Databricks
- Strong understanding of data modeling concepts and building curated datasets for analytics consumption
- Experience implementing CI/CD practices for data pipelines and notebooks, including automated testing and deployment
- Proven ability to lead technical discussions, mentor team members, and drive engineering best practices
- Experience improving observability (metrics, alerts, dashboards) and reducing pipeline failures through proactive monitoring

📌 ADF (Azure Data Factory)Databricks+Pyspark (Bengaluru)
🏢 Infosys
📍 Bengaluru