ETL, PYSPARK, DATABRICKS, Delta Lake, Spark SQL, Data Modeling, Workflow Orchestration, Performance Tuning

Key Responsibilities

- Develop and maintain data pipelines and transformations using Databricks and PySpark.
- Implement scalable ETL/ELT workflows to ingest, cleanse, and curate data for downstream analytics and reporting.
- Optimize Spark jobs for performance and cost by tuning partitions, caching, joins, and cluster configurations.
- Build reusable notebooks and modular code to support consistent development and easier maintenance.
- Perform data validation, reconciliation, and quality checks to ensure accuracy and reliability of datasets.
- Collaborate with cross-functional teams to gather requirements, clarify data definitions, and deliver aligned solutions.
- Support deployments and production operations by troubleshooting failures, analyzing logs, and resolving incidents.
- Contribute to documentation, coding standards, and best practices for Databricks-based development.

Minimum

Qualifications:

- Bachelor’s or Master’s degree in BTECH, MTECH, MCA, or MSC (or equivalent).




- 2–3 years of hands-on experience working with Databricks in data engineering or analytics engineering projects.
- Strong experience in PySpark for building transformations and distributed data processing.
- Solid understanding of data pipeline concepts, data modeling basics, and structured/semi-structured data handling.
- Ability to debug and troubleshoot Spark jobs and collaborate effectively within delivery teams.
- Experience with Spark optimization techniques and practical performance tuning in Databricks environments.
- Familiarity with Delta Lake concepts such as ACID tables, schema evolution, and incremental processing patterns.
- Exposure to orchestrating workflows and managing dependencies for end-to-end pipeline execution.
- Experience working in agile delivery models with robust ownership of tasks, timelines, and quality outcomes.
- Strong communication skills to translate requirements into implementable data solutions and clearly document outcomes.

📌 Databricks (Bengaluru)
🏢 Infosys
📍 Bengaluru