We are seeking an experienced and leadership-minded Senior Data Engineer - Pyspark & AWS to drive the reliability, scaling, and operation of our enterprise data infrastructure. In this role, you will balance core development with production support and operational excellence, ensuring high availability through proactive monitoring and debugging of cloud data streams.
As a senior member of the data team, you will take full ownership of the data ecosystem, troubleshoot complex live incidents, and actively mentor and upskill junior engineers.
Key Responsibilities
Data Engineering & Pipeline Design:
Orchestration: Program complex workflow dependencies and multi-stage DAGs within Apache Airflow.
ETL Architecture: Construct serverless data preparation workflows using AWS Glue and PySpark.
Compute Processing: Deploy targeted, cost-optimized scripts via AWS Lambda for event-driven backend tasks.
Storage Management: Structure secure Amazon S3 data lakes with transparent landing, raw,
and gold curation tiers.
Leadership & Mentorship
Technical Ownership: Accept end-to-end accountability for platform health, security compliance, and architecture updates.
Team Mentorship: Guide junior developers through systematic code reviews, architectural advice, and pair programming sessions.
Process Standards: Establish baseline engineering documentation, testing frameworks, and on-call rotation protocols.
Technical Skills
Primary Skills (Required)
AWS Production Environment: Background supporting enterprise-scale cloud infrastructures under tight data freshness targets.
Observability & Triage: Hands-on triage using AWS CloudWatch, CloudTrail, X-Ray, or third-party log collectors.
Serverless Ingestion: Advanced setup of AWS Glue (Crawlers, Catalogs, Jobs) and event-driven AWS Lambda components.
Data Lakes & Orchestration: Deep mastery of Amazon S3 object storage policies and workflow
📌 Senior Data Engineer (Pune)
🏢 Spore N Sprouts
📍 Pune
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.