Responsibilities

- Establish scalable, productive, automated processes for data analysis, data model development, validation, and implementation.
- Work closely with analysts/data scientists to understand impact to the downstream data models.
- Write efficient and well-organized software to ship products in an iterative, continual release environment.
- Contribute to and promote good software engineering practices across the team
- Communicate clearly and effectively to technical and non-technical audiences.

Minimum Qualifications:

- University or advanced degree in engineering, computer science, mathematics, or a related field
- Strong hands-on experience in Databricks using PySpark and Spark SQL (Unity Catalog, workflows, Optimization techniques)
- Experience with at least one cloud provider solution (GCP preferred)
- Strong experience working with relational SQL databases.
- Strong experience with object-oriented/object function scripting language: Python.
- Working knowledge in any transformation tools, DBT preferred.
- Ability to work with Linux platform.
- Strong knowledge of data pipeline and workflow management tools (Airflow)
- Working knowledge of Git hub /Git Toolkit
- Expertise in standard software engineering methodology, e.g. unit testing, code reviews, design documentation
- Experience creating Data pipelines that prepare data for ingestion & consumption appropriately.
- Experience in maintaining and optimizing databases/filesystems for production usage in reporting, analytics.
- Working in a collaborative environment and interacting effectively with technical and non-technical team members equally well.



Good verbal and written communication skills.

Qualifications & Skills

- University or advanced degree in engineering, computer science, mathematics, or a related field
- 10+ years' experience developing and deploying data pipelines both batch and streaming into production.
- Strong experience working with a variety of relational SQL and NoSQL databases.
- Extensive experience with the cloud-native data services of Google Cloud Platform (BigQuery, Vertex AI, Pub/Sub, Cloud Functions, etc.).
- Deep expertise in one of the popular data warehousing tools such as Snowflake, Big Query, RedShift, etc
- Hands-on experience with dbt (Data Build Tool) for data transformation
- Experience working with big data tools and frameworks such as Hadoop, Spark, Kafka, etc. Familiarity with Databricks is a plus.
- Experience with object-oriented/object function scripting languages: Python, Java, C++, Scala, etc.
- Hands-on experience in the Data engineering Spectrum, e.g. developing metadata-based framework-based solutions for Ingestion, Processing, etc., building Data Lake/Lake House solutions.
- Strong knowledge of Apache Airflow for orchestration and workflow management.
- Working knowledge of Github /Git Toolkit.
- Experience with providing operational support to stakeholders.
- Expertise in standard software engineering methodology, e.g. unit testing, test automation, continuous integration, code reviews, design documentation.
- Experience working with CI/CD pipelines using Jenkins and Github Actions.
- Experience with data visualization using Tableau, PowerBI, Looker or similar tools is a plus

📌 Gcp Data Engineer (Pune)
🏢 Purview Services
📍 Pune