Our partner is looking for a Senior Data Engineer based in India. We are seeking an experienced Senior Data Engineer to design, build, and optimize scalable data platforms that power advanced analytics and data-driven business decisions. In this role, you will develop high-performance batch and real-time data pipelines using Python, SQL, PySpark, and Databricks, working with terabyte-scale datasets and distributed processing architectures. You will contribute to modern lakehouse solutions, cloud migration initiatives, and secure data management practices across major cloud platforms. You will also automate deployment workflows and infrastructure provisioning using modern DevOps tools and CI/CD practices. This remote opportunity is suited to a technically strong qualified who enjoys solving complex data challenges, improving platform performance, and mentoring colleagues in a fast-paced environment. Accountabilities: - Design, develop, and optimize scalable ETL/ELT data pipelines for batch and real-time processing using Python, SQL, PySpark, Spark Structured Streaming, and Delta Live Tables. - Implement and maintain Medallion Architecture, organizing data into Bronze, Silver, and Gold layers to support data quality, performance, and analytical requirements. - Implement Unity Catalog for metadata management, granular access controls, OAuth-based authentication, and enterprise-level data security and governance. - Develop and execute cloud migration strategies for legacy databases, including Oracle, Teradata, and SQL Server, to cloud-based data platforms and storage solutions such as Google Cloud Storage, Amazon S3,



Azure Data Lake Storage, Snowflake, and Amazon Redshift. - Build and maintain CI/CD pipelines using Terraform and GitHub Actions to automate infrastructure provisioning, data pipeline deployments, testing, and monitoring. - Develop custom data processing scripts and data-driven APIs using Python and web frameworks such as Flask. - Use Docker to containerize data processing workloads and support portable, consistent deployment environments. - Evaluate emerging technologies and approaches for data-intensive machine learning workloads and other advanced engineering use cases. - Collaborate with business stakeholders, data teams, and cross-functional engineering partners to translate business needs into technical architectures and actionable data solutions. - Apply best practices for SQL query optimization, data processing efficiency, security, monitoring, and operational reliability. - Contribute to technical problem-solving, project ownership, knowledge sharing, and mentoring to strengthen team capabilities and delivery quality. Requirements: - Four to seven years of professional experience in data engineering, big data development, or a closely related technical field. - Advanced hands-on proficiency in Python, SQL, and PySpark, including expert-level SQL query optimization for large-scale datasets. - Strong practical experience with Apache Spark, Spark Structured Streaming, Delta Live Tables, and the design of scalable batch and real-time data pipelines. - Demonstrated experience working with terabyte-scale datasets and distributed data processing architectures. -

📌 (Senior) Data Engineer (ML, Big Data)
🏢 Jobgether
📍 India