The ETL/PySpark Data Engineer will play a crucial role in designing, developing, and maintaining data pipelines that ensure high-quality, reliable data for our analytics and reporting needs. This position is vital for transforming raw data into meaningful insights, driving data-driven decisions across the organization. The ideal candidate will have a strong background in data engineering, proficiency in PySpark, and a passion for data-driven solutions.
Experience Required:
3+ years
Key Responsibilities:
Design and implement ETL processes to extract, transform, and load data from various sources.
Develop and optimize PySpark scripts for large-scale data processing.
Collaborate with data scientists and analysts to understand data requirements and deliver data solutions.
Ensure data quality and integrity throughout the data pipeline lifecycle.
Troubleshoot and resolve data processing issues to maintain pipeline performance.
Automate repetitive data tasks to improve efficiency and reduce manual effort.
Participate in the design and implementation of data warehousing solutions.
Create and maintain documentation for data processes and data models.
Conduct performance tuning and optimization of data pipelines.
Stay updated with the latest trends and technologies in data engineering.
Qualification Criteria:
Bachelor's degree in Computer Science, Data Science, or a related field.
Proven experience in ETL processes and data pipeline development.
Robust proficiency in PySpark and Python.
Experience with data warehousing solutions such as Amazon Redshift, Snowflake, or Google BigQuery.
Familiarity with database management systems like MySQL, PostgreSQL, or MongoDB.
Excellent problem-solving skills and attention to detail.
Solid understanding of data modeling and data architecture.
Ability to work collaboratively in a fast-paced, team-oriented environment.
Solid communication skills to articulate technical concepts to non-technical stakeholders.
Experience with CI/CD tools and practices.
Primary Skills:
ETL processes
PySpark
data warehousing
Key Skills:
Python
data modeling
SQL
CI/CD
data quality
cloud platforms
big data
data governance
data visualization
APIs
version control
cloud computing
automation
machine learning
cloud services
Skills
ETL processesPySparkdata warehousing
📌 Etl / Pyspark Data Engineer Chennai
🏢 Spectrum Talent Management
📍 Chennai
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.