Role & Responsibilities

- Design, develop, and maintain scalable ETL/ELT data pipelines using Python, PySpark, SQL, AWS Glue, Azure Data Factory, and Apache Airflow.

- Build and manage cloud-based data platforms across AWS and Microsoft Azure, supporting enterprise data engineering and analytics requirements.

- Develop data ingestion pipelines to collect data from databases, APIs, files, streaming platforms, and other structured and semi-structured sources.

- Implement data processing and transformation solutions using PySpark, Apache Spark, Python, and SQL.

- Develop and optimize data lakes and data warehouses using Amazon S3, Amazon Redshift, Snowflake, Azure Data Lake Storage (ADLS), and Azure Synapse Analytics.

- Work with AWS services including AWS Glue, Lambda, EMR, Athena, S3, Redshift, Step Functions, SQS, SNS, and CloudWatch.

- Work with Azure services including Azure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse Analytics, Azure Functions, and Azure Monitor.

- Design and implement scalable data models, including fact tables, dimension tables, star schemas, and analytical data models.

- Develop complex SQL queries, stored procedures, transformations, and data integration processes for reporting and analytics.

- Implement data quality, validation, reconciliation, and monitoring frameworks to ensure data accuracy, completeness, and reliability.

- Optimize Spark jobs, SQL queries, ETL workflows, and cloud resources for improved performance and cost efficiency.

- Develop Python-based automation scripts, data processing frameworks, utilities, and integrations.

- Configure and maintain workflow orchestration using Apache Airflow for reliable scheduling, dependency management,



and pipeline monitoring.

- Collaborate with business analysts, data architects, developers, and stakeholders to understand requirements and deliver scalable data solutions.

- Support analytics, reporting, and BI initiatives by providing trusted, curated, and high-quality datasets.

- Implement CI/CD and version-control practices using Git, GitHub/Azure DevOps, and cloud-native deployment processes.

- Troubleshoot pipeline failures, data issues, performance bottlenecks, and production incidents.

- Follow best practices for coding, testing, documentation, security, governance, and Agile/Scrum development.

Preferred Candidate Profile

- Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, Engineering, or a related field.

- 38 years of experience in Data Engineering, Cloud Data Engineering, Data Analytics, or related domains.

- Strong programming experience in Python, SQL, and PySpark.

- Hands-on experience with AWS cloud data services, particularly AWS Glue, S3, Athena, Redshift, Lambda, EMR, Step Functions, and CloudWatch.

- Hands-on experience with Microsoft Azure data services, particularly Azure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse Analytics, and Azure Monitor.

- Strong understanding of ETL/ELT, data warehousing, data lakes, data lakehouse architecture,



and big-data processing.

- Experience with Apache Spark and PySpark for large-scale distributed data processing.

- Experience with Apache Airflow for data pipeline orchestration and workflow automation.

- Knowledge of cloud data warehouses such as Snowflake, Amazon Redshift, and Azure Synapse Analytics.

- Experience with Databricks, Delta Lake, and Medallion Architecture is preferred.

- Strong understanding of data modeling, dimensional modeling, partitioning, file formats, and performance optimization.

- Familiarity with Git, CI/CD, DevOps practices, Agile/Scrum, and cloud security principles.

- Experience working with Parquet, JSON, CSV, Avro, and other structured/semi-structured data formats.

- Robust analytical, troubleshooting, problem-solving, and communication skills.

- Ability to work independently and collaboratively in a fast-paced, enterprise environment.

Perks & Benefits

- Hybrid and remote work opportunities.

- Health insurance and employee wellness benefits.

- Learning and certification support for AWS, Azure, Snowflake, Databricks, and other cloud technologies.

- Opportunities to work on enterprise-scale AWS and Azure data platforms.

- Exposure to modern technologies in cloud computing, big data, analytics, and data engineering.

- Career growth and continuous learning opportunities.

- Flexible work environment and employee-friendly policies.

- Employee recognition and rewards programs.

- Paid leaves, holidays, and work-life balance initiatives.

- Opportunity to work on challenging Data Engineering, Cloud Migration, Data Lake, Lakehouse, and Analytics projects.

📌 Data Engineer / Python Developer/ Data Analyst (Bengaluru)
🏢 Aetherrise Solutions
📍 Bengaluru