Responsibilities

- Design, develop, test, and maintain optimal data pipeline and ETL architectures.
- Map out data systems and define/design required integrations, ETL, BI, and AI systems/processes.
- Prepare and optimise data for predictive and prescriptive modelling.
- Collaborate with teams to integrate ERP data into the enterprise data lake, ensuring seamless flow and quality.
- Enhance cloud data infrastructure on AWS or Azure for scalability and performance.
- Utilise big data tools and frameworks to optimise data acquisition and preparation.
- Build architectures to move data to/from data lakes and data warehouses for advanced analytics.
- Develop and curate data models for analytics, dashboards, and reports.
- Conduct code reviews, maintain production-level code, and implement testing approaches.
- Monitor, troubleshoot, and resolve data ingestion workflows to maintain reliability and uptime.
- Drive innovation and implement efficient new approaches to data engineering tasks.

Requirements

- Bachelor's degree in Computer Science, Mathematics, Engineering, or a related field.




- 5+ years of experience working with enterprise data platforms, including building and managing data lakes.
- QX | Public: 3-5 years of experience designing and implementing data warehouse solutions.
- Expertise in SQL, including developing stored procedures (SPs) and applying advanced data design concepts
- Proficiency in Spark (Python/Scala) and Spark Streaming for real-time data pipelines.
- Experience with AWS or Azure services (e. g., AWS Glue, Azure Data Factory, Redshift, Snowflake).
- Familiarity with big data tools such as Apache Kafka, Apache Spark, or Flink.
- Hands-on experience with orchestration tools (e. g., Apache Airflow, Prefect).
- Knowledge of CI/CD processes, version control (e. g., Git, Jenkins), and deployment automation.
- Solid problem-solving, communication, and collaboration skills.

Good-to-Have Skills

- Experience in integrating ERP data into data lakes.
- Experience with traditional ETL tools (e. g., Talend, Pentaho).

This job was posted by Gnana Ramya from QuaXigma.

📌 Senior Data Engineer (Tirupati)
🏢 QX Impact
📍 Tirupati