Requisition Number: US Req# XXXX; S1S# XXXX; GSS# – 2375760

Position: Data Engineer, Shift – 11:00 AM – 8:30 PM IST
Grade: 26 –Sr Data Engineering Analyst

Job Description:

The Data Engineer (Grade 26) will support the design, development, and maintenance of scalable Big Data solutions on cloud platforms. This role is ideal for data engineers with hands-on experience in Spark-based data processing, Azure cloud services, real-time data processing and data pipeline orchestration.

The individual will work closely with senior engineers, architects, and cross-functional teams to build reliable data pipelines, improve existing solutions, and ensure secure and efficient data operations. The role emphasizes strong fundamentals in data engineering, cloud technologies, and production support, while providing opportunities to grow into advanced Big Data and cloud-native architectures.

Primary Job Responsibilities:

- Design, code, test, document, and maintain high-quality, scalable data applications using Python, PySpark and Scala Spark on Azure Cloud platforms.
- Develop and manage data pipelines and schedule workflows using Apache Airflow, ensuring proper job dependencies and execution order.
- Securely manage secrets and credentials using Azure Key Vault, following enterprise security best practices.
- Analyze existing data pipelines and applications to identify gaps, risks, and opportunities for improvement.
- Assist in analyzing data architecture and design frameworks, working with multiple databases and data warehouses.
- Create prototypes and proof-of-concepts (POCs) and participate in design and code reviews.
- Write and maintain technical documentation for data pipelines, workflows,



and operational procedures.
- Participate in production support activities, including monitoring, troubleshooting, and issue resolution.
- Perform performance optimization of data pipelines and assist with application migration efforts across environments.

Skill Set Required:

- Bachelor’s or Master’s degree in Computer Science, Information Technology, or equivalent, with more than 4 years of relevant work experience.
- Hands-on experience with Python for data engineering and application development.
- Working experience with Apache Spark and good understanding of Hadoop ecosystem concepts.
- Preferable experience in developing applications using FastAPI.
- Experience with job scheduling and orchestration tools, particularly Apache Airflow.
- Good experience working in cloud environments, preferably Microsoft Azure.
- Hands-on exposure to Azure Data Lake, Azure Databricks, Azure Data Factory, and Azure Key Vault.
- Strong experience in writing complex SQL and PL/SQL queries.
- Experience with Snowflake and writing Shell scripts for automation and operational tasks.
- Exposure to CI/CD pipelines using tools such as Jenkins and GitHub Actions.
- Hands-on exposure to AI-assisted development tools such as Microsoft Copilot, with a basic understanding of prompt engineering to improve coding efficiency, data analysis productivity, and documentation quality.
- Basic understanding of software development best practices, version control, and collaborative development.
- Positive analytical skills, attention to detail, and willingness to learn new technologies and frameworks.
- Ability to work effectively in a team-oriented and fast-paced environment.

📌 Sr Data Engineering Analyst-Python, Scala, Azure (Noida)
🏢 Optum
📍 Noida