We are seeking an experienced Data Engineering &
Automation Lead to design, automate, and optimize large-scale data processing and web scraping pipelines. The role involves leading a team to build and maintain high-performance ETL workflows using Apache Airflow, Apache Spark, and AWS services, while integrating AI/NLP solutions powered by OpenAI GPT and other GenAI models for intelligent data extraction and analytics.

Key Responsibilities:
Design, automate, and maintain ETL and data processing pipelines using Apache Airflow and Apache Spark.
Build, monitor, and optimize web scraping and data extraction workflows for global compliance and risk data sources.
Lead and manage web scraping and data engineering teams, ensuring delivery excellence, code quality, and scalability.
Create, design, and document automation workflows and secure data-sharing systems using AWS (Lambda, S3, API Gateway, SQS).
Implement AI and NLP integrations using OpenAI GPT and GenAI models for intelligent data extraction, tagging, and analytical automation.
Analyze large-scale datasets to identify quality gaps, improve accuracy, and optimize indexing and retrieval performance.
Collaborate with Backend, DevOps, and Frontend teams for data modeling, monitoring, and visualization.




Work closely with clients to gather and translate business requirements into scalable automation and analytics solutions.
Author HLD/LLD documentation, mentor junior engineers, and continuously improve automation processes and data workflows.

Key Requirements:Positive to Have:
Robust understanding of Large Language Models (LLMs) and Generative AI for building intelligent data extraction and analytics agents.
Familiarity with risk and compliance domains, including Sanctions, PEP (Politically Exposed Persons), and AMS (Adverse Media Screening) data and processes.

Core Technical Skills:
Programming: Python, SQL, JavaScript
Data Engineering &
Automation: Apache Airflow, Apache Spark, Web Scraping (Scrapy, Selenium), Pandas, NumPy
Databases &
Storage: Elasticsearch, MongoDB, MySQL
Cloud &
Backend: AWS (Lambda, S3, EC2, CloudWatch, SQS, SNS, EKS), Docker, Django, Flask
AI/ML &
NLP: OpenAI GPT APIs, NER, Sentiment Analysis, Embeddings, Information Extraction
Monitoring &
Tools: Grafana, Git, Postman, Jupyter, VS Code

Soft Skills:
Leadership &
Team Mentoring
Problem-Solving &
Analytical Thinking
Transparent Technical Communication
Cross-functional Collaboration
Remote Team Coordination

📌 Lead Data Engineer Pune
🏢 Solytics Partners
📍 Pune