Key Responsibilities

- Lead the development of scalable ETL/ELT pipelines and data models to process high-volume data.

- Design and optimize data processing applications across platforms like Databricks, Snowflake, AWS, and Azure.

- Build and manage data pipelines using Apache Airflow for orchestration and Airbyte for data integration.

- Architect solutions for Change Data Capture, batch processing, and real-time/streaming data processing.

- Develop and manage REST APIs for data ingestion and distribution to applications and users.

- Leverage AI tools to enhance the development and deployment of data pipelines.

- Oversee CI/CD pipelines and DevOps deployments using Docker, Kubernetes, and monitoring tools.

- Mentor team members in best practices for data governance, quality, and security.

Overview

- As a Lead Data Engineer at our consulting firm, you will play a pivotal role in shaping the data architecture and engineering landscape.

- This position requires an expert with multi-platform proficiency, particularly in Databricks, Snowflake, AWS, and Azure, to lead scalable data solutions.

- You will work closely with cross-functional teams to integrate MLOps/DevOps practices and guide a team in delivering production-ready solutions.





- The role demands innovation in solving complex data challenges using a diverse toolkit and requires robust leadership skills to mentor and develop team members.

- Your expertise will help build robust, high-volume data processing pipelines and drive the adoption of AI-augmented deliverables.

- The position is based in Bangalore and Gurgaon, catering to high-profile clients in the consulting sector.

- You will be responsible for ensuring data governance, quality, and security while staying ahead of industry trends in MLOps and GenAI.

- This role offers an opportunity to collaborate with data scientists and analysts to ensure data readiness for machine learning projects and contribute to the evolution of CI/CD pipelines.

Requirements

- 5+ years of experience in Data Engineering with expertise in data lakes and warehouses.

- Proficiency in Change Data Capture (CDC), batch, and real-time data processing.

- Experience with multiple cloud data platforms such as Databricks, Snowflake, AWS, or Azure.

- Advanced SQL skills for large-scale data processing and analytics.

- Strong programming skills in Python, with experience in asynchronous programming and threading.

- Experience in creating and managing REST APIs using frameworks like FastAPI or Flask.

- Hands-on experience with Apache Airflow for workflow orchestration and Airbyte for data integration.

📌 Lead Data Engineer (Bengaluru)
🏢 Net Connect
📍 Bengaluru