Job Purpose

We are seeking a highly skilled Data Engineer with a strong software engineering mindset and deep expertise in Python-based data engineering. This is not a traditional SQL-centric data engineering role. We are specifically looking for engineers with strong programming, debugging, and distributed data processing experience, who can build scalable and maintainable data platforms using modern cloud and lakehouse technologies.

The ideal candidate will have experience developing data pipelines using Python and Spark, working with modern data lake platforms such as Databricks and Snowflake, and deploying solutions on AWS.

Experience with Dagster for data orchestration is highly desirable.

Key Responsibilities

Data Platform Development

- Design, develop, and maintain scalable data pipelines using Python and distributed processing frameworks.
- Build reusable, production-grade data engineering components and frameworks.
- Develop and optimize high-volume batch and streaming data processing solutions.
- Implement robust testing, monitoring, and observability for data pipelines.
- Troubleshoot complex data processing and performance issues.

Lakehouse & Data Engineering

- Build and maintain modern lakehouse architectures using Databricks, Snowflake, and AWS data services.
- Design scalable data models and storage strategies for structured and semi-structured data.
- Develop ingestion, transformation, validation, and enrichment pipelines.
- Support data quality, lineage, and governance initiatives.

Orchestration & Automation





- Develop and manage workflow orchestration solutions using Dagster or similar orchestration frameworks.
- Automate data operations and deployment processes.
- Monitor pipeline performance and optimize reliability and efficiency.

Cloud Engineering

- Build cloud-native data solutions on AWS.
- Integrate data services across storage, processing, orchestration, and analytics platforms.
- Implement security, scalability, and resiliency best practices.

Engineering Excellence

- Write clean, maintainable, and well-tested code.
- Participate in peer code reviews and technical design discussions.
- Collaborate with product managers, architects, and engineering teams.
- Follow software engineering best practices, including CI/CD and version control.

Key competencies

Experience

- 5+ years of professional experience in Data Engineering.
- Hands-on experience building production-grade data platforms and pipelines.
- Solid software engineering and coding background.

Technical Skills

Primary Skills (Must Have)

- Python (Expert Level)
- PySpark / Apache Spark
- Data Pipeline Development
- Object-Oriented Programming
- Debugging and Performance Optimization
- REST APIs
- Git

Modern Data Platforms

- Experience with one or more:
- Databricks
- Snowflake
- Delta Lake
- Apache Iceberg
- Modern Lakehouse Architectures

AWS Experience Experience with multiple services such as:

- S3
- Lambda
- Glue
- EMR
- ECS/EKS
- Step Functions
- RDS
- CloudWatch

Data Engineering Practices

- ETL/ELT Development
- Data Modeling
- Data Quality Frameworks
- Data Validation and Monitoring
- CI/CD for Data Pipelines

📌 Senior Associate-Date Engineer (Bengaluru)
🏢 Acuity Analytics
📍 Bengaluru