We are seeking an experienced Data Engineering & Automation Lead to design, automate, and optimize large-scale data processing and web scraping pipelines. The role involves leading a team to build and maintain high-performance ETL workflows using Apache Airflow, Apache Spark, and AWS services, while integrating AI/NLP solutions powered by OpenAI GPT and other GenAI models for intelligent data extraction and analytics.

Key Responsibilities:

- Design, automate, and maintain ETL and data processing pipelines using Apache Airflow and Apache Spark.
- Build, monitor, and optimize web scraping and data extraction workflows for global compliance and risk data sources.
- Lead and manage web scraping and data engineering teams, ensuring delivery excellence, code quality, and scalability.
- Create, design, and document automation workflows and secure data-sharing systems using AWS (Lambda, S3, API Gateway, SQS).
- Implement AI and NLP integrations using OpenAI GPT and GenAI models for intelligent data extraction, tagging, and analytical automation.
- Analyze large-scale datasets to identify quality gaps, improve accuracy, and optimize indexing and retrieval performance.
- Collaborate with Backend, DevOps, and Frontend teams for data modeling, monitoring, and visualization.




- Work closely with clients to gather and translate business requirements into scalable automation and analytics solutions.
- Author HLD/LLD documentation, mentor junior engineers, and continuously improve automation processes and data workflows.

Key Requirements:Positive to Have:

- Strong understanding of Large Language Models (LLMs) and Generative AI for building intelligent data extraction and analytics agents.
- Familiarity with risk and compliance domains, including Sanctions, PEP (Politically Exposed Persons), and AMS (Adverse Media Screening) data and processes.

Core Technical Skills:

- Programming: Python, SQL, JavaScript
- Data Engineering & Automation: Apache Airflow, Apache Spark, Web Scraping (Scrapy, Selenium), Pandas, NumPy
- Databases & Storage: Elasticsearch, MongoDB, MySQL
- Cloud & Backend: AWS (Lambda, S3, EC2, CloudWatch, SQS, SNS, EKS), Docker, Django, Flask
- AI/ML & NLP: OpenAI GPT APIs, NER, Sentiment Analysis, Embeddings, Information Extraction
- Monitoring & Tools: Grafana, Git, Postman, Jupyter, VS Code

Soft Skills:

- Leadership & Team Mentoring
- Problem-Solving & Analytical Thinking
- Clear Technical Communication
- Cross-functional Collaboration
- Remote Team Coordination

📌 Lead Data Engineer (Pune)
🏢 Solytics Partners
📍 Pune