Sr. Data Engineer

Location: Pune

Responsibilities

· Lead the end-to-end design and development of scalable, high-performance data pipelines and ETL/ELT frameworks aligned with contemporary data engineering best practices.

· Architect complex data integration workflows that bring together structured, semi-structured, and unstructured data from both cloud and on-premise sources.

· Build robust real-time, batch, and on-demand pipelines with built-in observability—monitoring, alerting, and automated error handling.

· Partner with analysts, data scientists, and business leaders to define and deliver reliable data models, quality frameworks, and SLAs that power key business insights.

· Ensure optimal pipeline performance and throughput, with clearly defined SLAs and proactive alerting for data delivery or quality issues.

· Collaborate with platform, DevOps, and architecture teams to build secure, reusable, and CI/CD-enabled data workflows that align with enterprise architecture standards.

· Establish and enforce the best practices in source control, code reviews, testing automation, and continuous delivery for all data engineering components.

· Lead root cause analysis (RCA) and preventive maintenance for critical data failures, ensuring minimal business impact and continuous service improvement.

· Guide the team in establishing standards for data modeling, transformation logic, and governance, ensuring long-term maintainability and scalability.

· Design and execute comprehensive testing strategies—unit, integration, and system testing—ensuring high data reliability and pipeline resilience.

· Monitor and fine-tune data pipeline and query performance, optimizing for reliability, scalability, and cost-efficiency.

· Create and maintain detailed technical documentation,



including data architecture diagrams, process flows, and integration specifications for internal and external stakeholders.

· Facilitate and lead discussions with business and operational teams to understand data requirements, prioritize initiatives, and drive data strategy forward.

Qualifications

· 7 to 10 years of hands-on experience in data engineering roles with a proven record of building scalable and secure data platforms.

· Over 5 years of experience in scripting languages such as Python for data processing, automation, and ETL development.

· 4+ years of experience with Snowflake, including performance tuning, security model design, and advanced SQL development.

· 5+ years of experience with data integration tools such as Azure Data Factory, Fivetran, or Matillion.

· 5+ years of experience in writing complex, highly optimized SQL queries on large datasets.

· Proven experience integrating and managing APIs, JSON, XML, and webhooks for data acquisition.

· Hands-on experience with cloud platforms (Azure/AWS) and orchestration tools like Apache Airflow or equivalent.

· Experience with CI/CD pipelines, automated testing, and code versioning tools (e.g., Git).

· Familiarity with dbt or similar transformation tools and best practices for modular transformation development.

· Exposure to data visualization tools like Power BI for supporting downstream analytics is a plus.

· Strong interpersonal and communication skills with the ability to lead discussions with technical and business stakeholders.

Education

Bachelor’s or Master’s degree in Computer Science/Information technology or related field.

Equivalent academic and work experience can be considered.

Skills:- Data engineering, Snow flake schema, Python, Amazon Web Services (AWS) and Azure OpenAI

📌 sr. Data Engineer (Pune)
🏢 TalentOne HR Consulting
📍 Pune