Job Description
We are hiring a Data Engineer (5+ Years Experience) for our heavy data analytics project.
Location:
Pune
Experience:
5-10 Years Key
Skills:
Spark, PySpark, Scala, Python, SQL, Databricks, Data Lake, DWH, Snowflake, Azure ADF/Synapse/ADLS
Job Description
We are looking for an experienced Data Engineer (5+ years) with strong hands‑on expertise in building high‑performance data pipelines for a heavy data analytics environment. The ideal candidate must excel in Spark (PySpark/Scala), complex SQL, data lake architectures, and data warehouse modeling. Experience with cloud data platforms such as Databricks, Azure, or Snowflake will be a solid advantage.
Key Responsibilities
• Data Pipeline & ETL/ELT Development
• Develop and optimize Spark (PySpark/Scala) pipelines
• Ingest, transform, cleanse, and aggregate large datasets
• Build scalable batch and near real‑time pipelines
• Apply Delta Lake optimization, partitioning, caching, and performance tuning
• Heavy Data Analytics
• Write complex aggregation logic (window functions, grouping sets, analytical functions)
• Understand KPIs, metrics, and analytical use cases
• Translate business logic into technical transformations
• Validate outputs against business requirements
• Data Lake Engineering
• Build multi‑layer data lake (Bronze/Silver/Gold)
• Work with Parquet, Delta Lake, ORC and columnar formats
• Implement schema evolution, metadata management, and auditing
• Data Warehouse Engineering
• Design dimensional models: Star Schema, Snowflake Schema
• Build fact and dimension tables
• Optimize table structures and partition strategies
• Databricks (Preferred)
• Develop notebooks and workflows in PySpark/Scala
• Manage clusters, jobs,
and Delta Live Tables
• Apply best practices for cost and performance
• SQL Engineering
• Write and optimize complex SQL queries
• Perform data profiling, validation, and analytical computations
• Support dashboards and reporting layers
• Cloud Data Platforms
• Azure: ADF, Synapse, ADLS Gen2 (preferred)
• Snowflake: Warehouses, Snowpipe, Streams/Tasks
• Optional: Azure Functions
• Data Quality & Documentation
• Validate data transformations and business rules
• Document data flows, transformation logic, and metadata
• Collaborate with QA and analysts for data accuracy Required Qualifications
• 5+ years of hands-on experience in Data Engineering
• Strong programming: Spark, Scala, Python
• Strong SQL (complex joins, analytical functions, aggregations)
• Experience in Data Lake & Data Warehouse concepts
• Spark performance tuning (Delta, shuffle tuning, partitioning)
• Experience with at least one cloud ecosystem (Azure/AWS/GCP) Preferred Skills
• Experience with Databricks
• Snowflake or modern cloud DWH
• ADF/Synapse/Airflow/dbt
• CI/CD for data pipelines
• Large-scale analytics environment exposure Soft Skills
• Strong understanding of business logic behind analytics
• Ability to convert metrics into technical transformations
• Excellent debugging & problem‑solving capabilities
• Good communication and cross‑functional collaboration Required Skills Spark, PySpark, Scala, Python, SQL (advanced), Data Lake, Data Warehouse, ETL/ELT pipelines, Delta Lake optimization, Azure/AWS/GCP cloud experience, Data modeling, Complex aggregations, Performance tuning Supported Skills Databricks, Snowflake, ADF, Synapse, Airflow, dbt, CI/CD for data pipelines, Large‑scale analytics e
📌 Data Engineers Expert (Pune)
🏢 KPIT
📍 Pune