Key Responsibilities : - Design, develop, and maintain scalable data processing pipelines using PySpark and Apache Spark.- Build and optimize ETL/ELT workflows to process large volumes of structured and unstructured data.- Develop complex SQL queries and perform data transformation, cleansing, validation, and aggregation.- Optimize Spark jobs for performance, scalability, and productive resource utilization.- Integrate data from multiple sources, including databases, APIs, data lakes, and enterprise systems.- Implement data quality checks, error handling, logging, and monitoring for reliable data pipelines.- Collaborate with data engineers, data architects, analysts, and business teams to deliver data solutions.- Support production deployments, troubleshoot pipeline failures, and resolve data processing issues.- Follow coding standards, version control, testing, documentation, and CI/CD best practices.Required Skillsets : - Strong hands-on experience in PySpark,



Python, SQL, and Apache Spark.- Experience developing ETL/ELT pipelines and processing large datasets.- Strong understanding of Spark architecture, DataFrames, RDDs, Spark SQL, joins, partitioning, and caching.- Experience in performance tuning, debugging, and optimizing distributed data processing jobs.- Knowledge of data warehousing, data modeling, and data lake concepts.- Experience with databases such as Oracle, PostgreSQL, or other relational/NoSQL databases.- Familiarity with cloud platforms such as AWS, Azure, or GCP and their data engineering services.- Knowledge of workflow orchestration tools such as Airflow and version control tools such as Git.- Understanding of data quality, security, and governance practices. (ref:hirist.tech)

📌 ValueLabs - PySpark Developer (Chennai)
🏢 ValueLabs
📍 Chennai