We are seeking a highly skilled Senior Data Engineer with 5+ years of experience to design, develop, and optimize scalable data solutions. The ideal candidate will have strong expertise in Python, SQL, ETL/ELT processes, and big data technologies. You will be responsible for building reliable data pipelines, ensuring data quality, and enabling seamless access to data for analytics and business intelligence initiatives.

Key Responsibilities:

- Design, develop, and maintain scalable data pipelines and data integration workflows.
- Build and optimize ETL/ELT processes for ingesting, transforming, and loading data from multiple sources.
- Develop high-performance data processing solutions using Python and Spark/PySpark.
- Design and maintain data warehouse structures, data marts, and analytical datasets.
- Implement workflow orchestration and scheduling using Airflow or similar tools.
- Monitor data pipelines, troubleshoot failures, and optimize performance.
- Ensure data quality, consistency, and reliability across all data platforms.
- Collaborate with Data Scientists, Analysts, and Engineering teams to support reporting and analytics requirements.
- Create and maintain technical documentation for data architecture, pipelines, and processes.
- Follow best practices for data governance, security, and operational excellence.

Required Skills





- 5+ years of experience in Data Engineering or a similar role.
- Strong proficiency in Python for data processing and automation.
- Advanced SQL skills with experience in query optimization and performance tuning.
- Hands-on experience designing and implementing ETL/ELT pipelines.
- Experience with Apache Spark/PySpark for large-scale data processing.
- Strong understanding of data warehousing concepts and dimensional data modeling.
- Experience with workflow orchestration tools such as Airflow.
- Strong knowledge of relational databases such as PostgreSQL or MySQL.
- Experience working with large datasets in a production environment.
- Excellent analytical, problem-solving, and debugging skills.
- Strong communication and collaboration abilities.

Preferred Skills

- Experience with Kafka or other real-time streaming platforms.
- Exposure to Databricks.
- Knowledge of Snowflake, BigQuery, or similar analytical databases.
- Experience with AWS, Azure, or GCP.
- Familiarity with Docker, Kubernetes, and CI/CD practices.
- Experience supporting Machine Learning or AI data pipelines.

Pay: ₹509,764.81 - ₹1,868,691.87 per year

Perks:

- Flexible schedule
- Paid sick time
- Paid time off
- Provident Fund

Work Location: In person

📌 Senior Data Engineer (India)
🏢 Crest Infosystems
📍 India