Job Requirements

Key Responsibilities
-Pipeline Development: Design, build, and maintain highly scalable, fault-tolerant batch and streaming data pipelines (ETL/ELT) using Python and PySpark. PySpark Optimization: Tune and optimize complex PySpark applications for performance, efficiently managing data skew, memory (OOM issues), and cluster resources. Multi-Cloud Architecture: Build and orchestrate data integrations between AWS (S3, EMR, Glue) and GCP (Cloud Storage, Dataproc, BigQuery). Data Modeling & Storage: Design robust data models and manage data lakes/warehouses using modern columnar formats (Parquet, ORC) and table formats (Apache Iceberg, Delta Lake, or Apache Hudi). Orchestration & Automation: Develop and maintain complex directed acyclic graphs (DAGs) using Apache Airflow (MWAA or Cloud Composer) to automate data workflows. Code Quality: Write clean, modular, and thoroughly tested Python code following software engineering best practices (CI/CD, version control, unit/integration testing). Mentorship: Act as a technical subject matter expert, mentoring junior engineers and performing rigorous code reviews. Required Qualifications -

Experience:
8+ years of dedicated Data Engineering or Software Engineering experience. Python:



Advanced proficiency in Python programming, including object-oriented programming, data structures, and packaging. PySpark (Mandatory): Deep, hands-on expertise with PySpark and the Apache Spark ecosystem. You must understand Spark's internal workings (Catalyst optimizer, DAG execution, RDDs vs. DataFrames, memory management, and partitioning). AWS Stack: Strong experience with AWS data services, particularly S3, EMR, Athena, AWS Glue, and IAM. GCP Stack: Solid experience with Google Cloud Platform data services, primarily Google BigQuery, Dataproc, and Google Cloud Storage (GCS). SQL: Expert-level SQL skills for data querying, profiling, and warehouse modeling. Orchestration: Hands-on experience with Apache Airflow or similar workflow orchestration tools. Preferred Qualifications (Nice-to-Have) -Experience with real-time data streaming technologies (Apache Kafka, AWS Kinesis, or GCP Pub/Sub). Knowledge of Infrastructure as Code (IaC) tools such as Terraform or AWS CloudFormation. Experience handling cross-cloud networking, egress cost optimization, and Workload Identity Federation. Work Experience 7-10Years

📌 Lead Software Engineer (Hyderabad)
🏢 Virtusa
📍 Hyderabad