Job Requirements

Key Responsibilities -Pipeline Development: Design, build, and maintain highly scalable, fault-tolerant batch and streaming data pipelines (ETL/ELT) using Python and PySpark.

PySpark Optimization: Tune and optimize complex PySpark applications for performance, efficiently managing data skew, memory (OOM issues), and cluster resources.

Multi-Cloud Architecture: Build and orchestrate data integrations between AWS (S3, EMR, Glue) and GCP (Cloud Storage, Dataproc, BigQuery).

Data Modeling & Storage: Design robust data models and manage data lakes/warehouses using modern columnar formats (Parquet, ORC) and table formats (Apache Iceberg, Delta Lake, or Apache Hudi).

Orchestration & Automation: Develop and maintain complex directed acyclic graphs (DAGs) using Apache Airflow (MWAA or Cloud Composer) to automate data workflows.

Code Quality: Write clean, modular, and thoroughly tested Python code following software engineering best practices (CI/CD, version control, unit/integration testing).

Mentorship: Act as a technical subject matter expert, mentoring junior engineers and performing rigorous code reviews.

Required Qualifications -Experience: 8+ years of dedicated Data Engineering or Software Engineering experience.

Python:



Advanced proficiency in Python programming, including object-oriented programming, data structures, and packaging.

PySpark (Mandatory): Deep, hands-on expertise with PySpark and the Apache Spark ecosystem. You must understand Spark's internal workings (Catalyst optimizer, DAG execution, RDDs vs. DataFrames, memory management, and partitioning).

AWS Stack: Solid experience with AWS data services, particularly S3, EMR, Athena, AWS Glue, and IAM.

GCP Stack: Strong experience with Google Cloud Platform data services, primarily Google BigQuery, Dataproc, and Google Cloud Storage (GCS).

SQL: Expert-level SQL skills for data querying, profiling, and warehouse modeling.

Orchestration: Hands-on experience with Apache Airflow or similar workflow orchestration tools.

Preferred Qualifications (Nice-to-Have) -Experience with real-time data streaming technologies (Apache Kafka, AWS Kinesis, or GCP Pub/Sub).

Knowledge of Infrastructure as Code (IaC) tools such as Terraform or AWS CloudFormation.

Experience handling cross-cloud networking, egress cost optimization, and Workload Identity Federation.

Work Experience

7-10Years

📌 Lead Software Engineer (Hyderabad)
🏢 Virtusa
📍 Hyderabad