We are seeking a skilled and proactive Data Engineer with strong expertise in Apache Spark, Python, and DevOps practices to join our data engineering team. The ideal candidate will be responsible for building scalable data pipelines, optimizing Spark jobs, automating data workflows, and contributing to CI/CD processes for data engineering artifacts.
Key Responsibilities:
- Design, develop, and optimize large-scale data pipelines using Apache Spark.
- Knowledge on Modern Lakehouse Architecture and management of Delta tables.
- Optimize Spark jobs for performance, scalability, and cost-efficiency across large datasets.
- Implement performance tuning strategies for Spark jobs to ensure productive data processing.
- Develop automation scripts in Python to streamline integration, deployment, and maintenance of data engineering workflows.
- Collaborate with cross-functional teams to automate manual tasks and improve operational efficiency.
- Contribute to CI/CD pipelines for data engineering artifacts using tools like Azure DevOps, GitHub Actions.
- Ensure data quality, reliability, and security across all stages of the data lifecycle.
- Monitor and troubleshoot production data pipelines and resolve issues proactively.
- Document processes, workflows, and best practices for data engineering operations.
Required Skills & Qualifications:
- 6+ years of hands-on experience in data engineering roles.
- Strong hands-on experience with Apache Spark (PySpark or SparkSQL) and Spark job optimization techniques.
- Proficiency in Python for scripting, automation, and data manipulation.
- Should have experience in working with APIs and other Source systems.
- Basic understanding of DevOps principles and experience with CI/CD tools.
- Familiarity with cloud platforms such as Azure.
- Experience with version control systems like Git.
- Ability to work independently and collaboratively in a fast-paced environment.