Key Requirements:

- Contract duration - Initially 3 months and it will be extendable upto 6months to 1year

- Core Skills: SQL, Python, PySpark, and Databricks

- AI Tools: Experience with AI coding assistants (e.g., Claude Code, GitHub Copilot, Codex, or Cursor)

- Additional Skills: Airflow for workflow orchestration, Linux administration, shell scripting, data modeling, and ETL/data warehousing processes

Responsibilities:

Data Pipeline Development:

· Design, develop, and maintain scalable data pipelines using Python, PySpark, and SQL to ingest, process, and transform data from various sources.

· Implement data quality checks and monitoring to ensure the reliability and integrity of data pipelines.

Data Warehouse Management:

· Build and optimize data warehouse solutions for efficient storage and retrieval of large volumes of structured and unstructured data.

· Collaborate with cross-functional teams to design and implement data models for analytical purposes.

Databricks Expertise:

· Utilize Databricks platform for data engineering tasks, including data ingestion, ETL processing, and data analysis.

· Optimize Databricks jobs and clusters for performance and cost efficiency.

Airflow Workflow Orchestration:

· Configure and manage Airflow workflows to automate data pipeline orchestration and scheduling.





· Implement error handling and monitoring within Airflow DAGs to ensure reliable execution of data workflows.

Linux Administration:

· Manage Linux-based servers and environments for data engineering tasks, ensuring security, reliability, and performance.

· Troubleshoot and resolve issues related to Linux infrastructure and server configurations.

Collaboration and Documentation:

· Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

· Document data engineering processes, workflows, and best practices for knowledge sharing and future reference.

Skills:

· Bachelor's degree in Computer Science, Engineering, or related field.

· Proven experience as a Data Engineer or similar role, with a minimum of 5 to 8 years of relevant experience.

· Proficiency in Python, PySpark, SQL, and Databricks.

· Familiarity with AI Coding tools like Claude code OR Github Copilot OR Cursor

· Familiarity with Airflow for workflow orchestration.

· Experience with Linux administration and shell scripting.

· Strong understanding of data modelling, ETL processes, and data warehousing concepts.

· Excellent problem-solving skills and attention to detail.

· Strong communication and collaboration skills, with the ability to work effectively in a team setting.

Work Location: In person

📌 Databricks Data Engineers (Chennai)
🏢 Infinity Crest Group Private
📍 Chennai