20 Nov
|
Hoonartek
|
Pune
Pune
About Us
We empower enterprises globally through intelligent, creative, and insightful services for data integration, data analytics and data visualization.
Hoonartek is a leader in enterprise transformation, data engineering and an acknowledged world-class Ab Initio delivery partner.
Using centuries of cumulative experience, research and leadership, we help our clients eliminate the complexities & risk of legacy modernization and safely deliver big data hubs, operational data integration, business intelligence, risk & compliance solutions and traditional data warehouses & marts.
At Hoonartek, we work to ensure that our customers, partners and employees all benefit from our unstinting commitment to delivery, quality and value. Hoonartek is increasingly the choice for customers seeking a trusted partner of vision, value and integrity
How We Work?
Define, Design and Deliver (D3) is our in-house delivery philosophy. It’s culled from agile and rapid methodologies and focused on ‘just enough design’. We embrace this philosophy in everything we do, leading to numerous client success stories and indeed to our own success.
We embrace change, empowering and trusting our people and building long and valuable relationships with our employees, our customers and our partners. We work flexibly, even adopting traditional/waterfall methods where circumstances demand it. At Hoonartek, the focus is always on delivery and value.
Job Description
Job Description | Data Engineer – PySpark & SQL
Data Engineer to build and maintain data pipelines using PySpark and SQL.
The role involves transforming large datasets, troubleshooting production issues, and improving pipeline performance and data quality.
Key Responsibilities
- Develop and maintain batch ETL pipelines using PySpark and SQL.
- Translate business requirements into data transformation rules and reusable datasets.
- Optimize Spark jobs and SQL queries by reviewing execution plans and resolving issues such as data skew, excessive shuffles, and inefficient joins.
- Implement data quality checks for duplicates, missing values, and source-to-target reconciliation.
- Monitor production pipelines, investigate failures, and support secure reruns and data backfills.
- Work with business analysts, QA, and platform teams to validate and release changes.
Must-Have Skills
- Hands-on experience with PySpark DataFrame APIs, including joins, aggregations, window functions, and deduplication.
- SQL skills, including CTEs, joins, functions, and query optimization.
- Experience building incremental data loads and handling late-arriving or duplicate records.
- Understanding of Spark partitioning, shuffle behavior, and performance tuning.
- Ability to write maintainable Python code and troubleshoot pipelines using logs and the Spark UI.
Good to Have
- Experience with Databricks, Delta Lake, or another cloud-based Spark platform.
- Familiarity with Airflow or Databricks Workflows, Git, and CI/CD.
- Experience with data warehousing, dimensional modeling, or the Bronze, Silver, and Gold architecture.
📌 Pyspark + SQL (Pune)
🏢 Hoonartek
📍 Pune