Job Overview:

We are looking for a skilled and motivated Lead Data Engineer with solid experience in Python programming, PySpark, Databricks and Google Cloud Platform (GCP) to join our data engineering team. The ideal candidate will be responsible for requirements gathering, designing, architecting the solution, developing, and maintaining robust and scalable ETL (Extract, Transform, Load) & ELT data pipelines. The role involves working with customers directly, gathering requirements, discovery phase, designing, architecting the solution, using various GCP services, implementing data transformations, data ingestion, data quality, and consistency across systems, and post post-delivery support.

Experience Level:

10 to 16 years of relevant IT experience
n

Key Responsibilities:
Design, develop, test, and maintain scalable ETL data pipelines using Python, PySpark, Databricks & GCP / Azure.
Architect the enterprise solutions with various technologies like GCP, Azure, Databricks, PySpark and Spark SQL.
Work extensively on Google Cloud Platform (GCP) services such as:




Dataflow for real-time and batch data processing
Cloud Functions for lightweight serverless compute
BigQuery for data warehousing and analytics
Cloud Composer for orchestration of data workflows (on Apache Airflow)
Google Cloud Storage (GCS) for managing data at scale
IAM for access control and security
Cloud Run for containerized applications

Should have experience in the following areas :
Develop production-grade Databricks notebooks and workflows.
Build data transformation pipelines using PySpark and Spark SQL.
Implement Delta Lake architecture.
Design Bronze, Silver, and Gold data layers using the Medallion Architecture.
Implement Databricks Workflows/Jobs and dependency management.
Tune Spark jobs for large-scale data processing.
Optimize cluster configuration and compute utilization.
Implement appropriate partitioning, caching, and file-size optimization strategies.
Perform data ingestion from var

📌 Lead Data Engineer (Hyderabad)
🏢 Egen
📍 Hyderabad