This job involves creating data pipelines using a configurable framework (developed using Databricks / pyspark), which are used to load data from multiple sources to Google Cloud. It also involves extensive collaboration with other team members (India USA) to follow a plan, execute workflows, ensure data load accuracy and debug in case of issues.
Essential Responsibilities
- Design, develop, and implement robust, high-performance data pipelines using Databricks/PySpark to meet diverse data integration requirements. Understand data and its relation / significance from business perspective
- Leverage extensive SQL expertise for complex data manipulation, analysis, and query optimization.
- Experience in Google Cloud services like Dataproc, BigQuery, Airflow, etc.
- Create custom scripts in python / pyspark
- Minor development and enhancement activities as needed to handle recent / custom requirements
- Validate data between source and target highlight issues and rerun jobs as needed
- Lead other developers,
review their code and ensure quality optimal code is being delivered
- Provide regular updates on the tasks, status and risks to project manager
The experience we are looking to add to our team
Required
- Bachelors degree from a reputed university
- 7 to 12 years overall experience
- Excellent SQL / data knowledge (6+ years) in creating complex SQL queries, Stored Procedures etc.
- Experience in developing enterprise grade ETL applications (any tool)
- Experience in Google BigQuery or any cloud database
- Experience in Google Cloud services like Managed service for Apache Spark(Dataproc), Airflow etc.
- Working knowledge in Pyspark / Python
- Good communication and positive attitude
- Attitude to take ownership and drive tasks without follow-ups
Good to have
- Experience in Big data technologies like Hadoop, Hive, etc.
- Knowledge in Iceberg, Starburst, DBT
- Knowledge in US Healthcare domain
- Knowledge in Agile
📌 Consultant - Cloud Data Engineer (Tamil Nadu)
🏢 enGen Global
📍 Tamil Nadu
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.