We are seeking a skilled Data Engineer with expertise in PySpark, Databricks, and SQL to design, develop, and maintain scalable data pipelines and data platforms. The ideal candidate will have experience working with large-scale datasets, cloud-based data solutions, and up-to-date data engineering practices.

Key Responsibilities
- Design, develop, and maintain ETL/ELT pipelines using PySpark and Databricks.
- Build scalable data processing solutions for structured and unstructured data.
- Develop and optimize complex SQL queries, stored procedures, and data models.
- Implement data quality checks and ensure data accuracy and consistency.
- Work with Delta Lake and Databricks workflows for data ingestion and transformation.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Collaborate with business analysts, data scientists, and stakeholders to understand data requirements.
- Monitor and troubleshoot production data pipelines.
- Implement best practices for data governance, security, and documentation.
- Support cloud-based data platforms such as Azure, AWS, or GCP.

📌 Data Engineer with Pyspark and data bricks (Chennai)
🏢 Tata Consultancy Services
📍 Chennai