ETL/PySpark Data Engineer

The ETL/PySpark Data Engineer will play a crucial role in designing, developing, and maintaining data pipelines that ensure high-quality, reliable data for our analytics and reporting needs. This position is vital for transforming raw data into meaningful insights, driving data-driven decisions across the organization. The ideal candidate will have a strong background in data engineering, proficiency in PySpark, and a passion for data-driven solutions.

Experience Required:

3+ years

Key Responsibilities:

- Design and implement ETL processes to extract, transform, and load data from various sources.
- Develop and optimize PySpark scripts for large-scale data processing.
- Collaborate with data scientists and analysts to understand data requirements and deliver data solutions.
- Ensure data quality and integrity throughout the data pipeline lifecycle.
- Troubleshoot and resolve data processing issues to maintain pipeline performance.
- Automate repetitive data tasks to improve efficiency and reduce manual effort.
- Participate in the design and implementation of data warehousing solutions.
- Create and maintain documentation for data processes and data models.
- Conduct performance tuning and optimization of data pipelines.




- Stay updated with the latest trends and technologies in data engineering.

Qualification Criteria:

- Bachelor's degree in Computer Science, Data Science, or a related field.
- Proven experience in ETL processes and data pipeline development.
- Strong proficiency in PySpark and Python.
- Experience with data warehousing solutions such as Amazon Redshift, Snowflake, or Google BigQuery.
- Familiarity with database management systems like MySQL, PostgreSQL, or MongoDB.
- Excellent problem-solving skills and attention to detail.
- Strong understanding of data modeling and data architecture.
- Ability to work collaboratively in a fast-paced, team-oriented environment.
- Solid communication skills to articulate technical concepts to non-technical stakeholders.
- Experience with CI/CD tools and practices.

Primary Skills:

- ETL processes
- PySpark
- data warehousing

Key Skills:

- Python
- data modeling
- SQL
- CI/CD
- data quality
- cloud platforms
- big data
- data governance
- data visualization
- APIs
- version control
- cloud computing
- automation
- machine learning
- cloud services

Skills

ETL processesPySparkdata warehousing

📌 ETL / PySpark Data Engineer (Chennai)
🏢 SPECTRUM TALENT MANAGEMENT
📍 Chennai