Job Overview :

We are seeking an experienced Cloud Data Engineer to the architecture, development, Data migration, ETL pipeline in cloud and optimization of a next-generation data platform. This is a development role for an individual with over 4 years of deep data engineering and data migration, data pipeline in AWS expertise, capable of driving technical direction, and delivering high-impact solutions in a fast-paced environment.

Key Responsibilities :

1.

Platform Development :

- Knowledge of best practices for our end-to-end data pipelines in Cloud, ensuring scalability, reliability, and security across the platform.
- Aggressively following the project timeline to ensure rapid delivery of data migration in AWS RDS.
- Working with Data Analysts, and business stakeholders to translate complex requirements into robust, production-ready ETL data solutions.

2.

Data Pipeline Development :

- Ingestion &
- Transformation : Design, build, and optimize high-volume data ingestion and transformation jobs using tools like dbt Core, iceberg, AWS Glue, ensuring data quality and integrity.
- Building ELT/ETL pipelines and using workflow orchestration tools like Apache Airflow or dbt.
- Optimize large-scale data processing algorithms and cloud data warehouse performance.
- Workflow : Develop and maintain sophisticated data pipelines using AWS glue/lambda, focusing on modularity and reusability.
- Streaming &
- Real-time Integration :



Implement and manage real-time data flows utilizing native AWS streaming services (e.g., Kinesis) for immediate data availability.
- Data Security and Privacy : Data Anonymization, Compliance with Regulations.
- Good knowledge of CI/CD for ETL pipeline automation and Data migration.

3. Drive the Data Ecosystem Management &

- Monitoring :
- Open Table Formats &
- Management : Implement and maintain the Iceberg open table format, utilizing tools for efficient schema evolution and data management.
- Compute Engine Optimization : Optimize query performance and cost efficiency across our primary compute engines : Amazon Redshift, and AWS Athena.
- Rapidly identify bottlenecks, and troubleshoot production issues.

Candidate Profile :

- 4 Years of hands-on, progressive experience in Cloud Data Engineering, cloud ecosystems AWS, Data Architecture.
- Deep conceptual understanding of core data engineering principles, ETL/ELT patterns, and metadata management.
- Ability to translate complex business needs into actionable technical requirements.
- Solid troubleshooting and problem-solving skills in distributed systems.
- Agile/Scrum teamwork capabilities and version control practices.
- Insurance industry experience preferred but not mandatory.

Tools

1.

Cloud Environment : AWS (S3, IAM, VPC, Glue, Lambda etc.)

2.

Experience with dbt Core, Iceberg, AWS Glue Catalog, AWS lambda, Redshift, Athena, Splunk, AmazonQ, AWS streaming services, Git.

- Strong SQL, Pyspark and Python.

📌 CG-VAK Software & Exports - AWS Data Engineer - Python/Spark (India)
🏢 CG-VAK Software u0026 Exports
📍 India