? We're Hiring: Senior Data Engineer | Python | PySpark | Databricks | SQL - WFH/Remote

AgileEngine is looking for a strong, hands-on Senior Data Engineer to join an exciting engagement focused on scalable data engineering, modern data platforms, and AI-powered automation.

We are particularly interested in Data Engineers with a strong Software Engineering background who can combine data engineering expertise with clean coding practices, system design, software architecture, and API development.

We are looking for engineers who can independently design, develop, optimize, and troubleshoot scalable production systems — not just build and maintain ETL pipelines.

? Location: Remote – India

? Core Skills Required

- Strong hands-on experience with Python, Advanced SQL, PySpark/Spark, Airflow, and dbt.
- Strong software engineering fundamentals, including clean coding, modular design, debugging, and code optimization.
- Experience with system design, software architecture, REST APIs, and backend development.
- Production experience with Databricks and BigQuery/Snowflake or equivalent data platforms.
- Strong practical understanding of Spark, including joins, partitioning, data skew, performance optimization, and troubleshooting.
- Experience designing and developing scalable ETL/ELT pipelines.
- Solid understanding of data modelling, incremental data loads, and SCD Type 1 / Type 2.
- Experience implementing data quality checks, validation, testing, monitoring, and source-to-target reconciliation.
- Understanding of production deployment, monitoring, troubleshooting, and performance optimization.
- Practical experience using AI coding assistants, including reviewing and validating AI-generated code.





⭐ Added Advantages
- GIS/Geospatial data experience.
- Mapping and location-based applications.
- Go programming language.
- Agentic AI, AI Agents, and MCP (Model Context Protocol) experience.
- Experience building AI-powered data engineering and workflow automation solutions.

? What You'll Work On
- Design, develop, and optimize scalable data solutions using Python, PySpark, SQL, and modern data platforms.
- Build and maintain production-grade ETL/ELT workflows.
- Develop clean, modular, reusable, and maintainable software components.
- Contribute to system design, architecture, and API development.
- Work with Databricks, BigQuery/Snowflake, Airflow, and dbt.
- Handle large-scale data processing and optimize Spark workloads.
- Implement data quality, validation, monitoring, and reconciliation frameworks.
- Explore AI-powered automation and agentic workflows to improve engineering productivity.
- Collaborate with engineering and business teams to deliver reliable and scalable data solutions.

? Important: Technical Interview Expectations

Candidates should be able to demonstrate:

- Strong understanding of their projects, architecture, databases, APIs, and orchestration tools.
- Strong Python programming skills, including clean coding, modularity, and problem-solving.




- Practical SQL and PySpark/Spark knowledge, including code walkthroughs.
- Ability to explain system design and architectural decisions.
- Understanding of Spark joins, partitioning, data skew, and optimization techniques.
- Experience handling growing data volumes and production performance challenges.
- Understanding of data modelling, incremental loads, and SCD Type 1/2.
- Independent problem-solving and technical decision-making.
- Clear communication and concrete examples from real production environments.

⚠️ Important Hiring Consideration

This is a hands-on engineering role requiring a combination of Software Engineering and Data Engineering expertise.

Our preference is for engineers who have strong Python development experience, understand software architecture and system design, and can build scalable data-intensive applications.

Candidates with experience limited to traditional ETL development or pipeline orchestration, without strong software engineering fundamentals, may not be the right fit.

Theoretical knowledge alone will not be sufficient. Strong practical experience in Python, SQL, Spark/PySpark, and production engineering is essential.

? Interested? Send me a direct message with:

- Email ID
- Updated resume
- Total years of experience
- Python experience
- SQL experience
- PySpark/Spark experience
- Databricks experience
- Airflow and dbt experience
- BigQuery/Snowflake experience
- Software Engineering / Backend Development experience
- System Design and API development experience
- Notice period / Last Working Day (LWD)
- Current CTC / Expected CTC
- Location

📌 Data Engineer – Python | PySpark | Databricks | Software Engineering - WFH/Remote - India
🏢 Remote_WFH_INDIA
📍 India