Senior Data Engineer Experience: 5+ Years
Department: Data Engineering & Analytics
Job Type: Full-Time
We are looking for an experienced Senior Data Engineer with 5+ years of experience in designing, developing, and maintaining scalable data pipelines and modern data platforms. The ideal candidate should have strong hands-on expertise in Python, SQL, ETL/ELT, Databricks, Snowflake, Apache Airflow, and cloud-based data engineering solutions .
The candidate will be responsible for building robust data pipelines, implementing lakehouse and data warehouse architectures, orchestrating complex workflows, and optimizing data processing performance. Experience with Microsoft Purview and enterprise data governance frameworks is highly desirable.
The role requires collaboration with data architects, analysts, data scientists, and business stakeholders to deliver reliable, secure, governed, and analytics-ready data solutions.
Design, develop, and maintain scalable ETL/ELT pipelines using Python, SQL, PySpark, and modern data engineering technologies.
Build and optimize data processing solutions using Databricks, Apache Spark, Delta Lake, and lakehouse architecture .
Design and manage data warehouse solutions using Snowflake , including ingestion, transformation, performance optimization, and cost management.
Develop, schedule, and orchestrate complex data pipelines and dependencies using Apache Airflow DAGs .
Integrate Apache Airflow with Databricks, Snowflake, cloud storage, APIs, and other data services.
Implement batch and near-real-time data pipelines integrating databases, REST APIs, third-party applications, and cloud platforms.
Implement incremental data loading, Change Data Capture (CDC), schema evolution, and data partitioning.
Design dimensional data models, star schemas, slowly changing dimensions (SCD), and analytical data structures.
Establish data quality checks, validation rules, reconciliation processes, and automated pipeline monitoring.
Implement data governance, metadata management, data lineage, data cataloging, and data classification using Microsoft Purview and related tools.
Support data discovery, sensitive data classification, business glossary management, and data ownership practices.
Implement data access controls, least-privilege principles, and sensitive data protection.
Optimize SQL queries, Spark jobs, Snowflake workloads, and cloud infrastructure costs.
Implement CI/CD, automated testing, version control, and deployment workflows for data engineering solutions.
Troubleshoot production data issues, perform root-cause analysis, and maintain pipeline reliability.
Conduct code reviews, mentor junior engineers,



and contribute to data engineering architecture and best practices.
Programming & Data Processing
Strong proficiency in Python and SQL.
Hands-on experience with PySpark and Apache Spark.
Experience developing production-grade ETL/ELT pipelines.
Knowledge of incremental processing, CDC, data validation, and performance tuning.
Experience with Databricks, Apache Spark, PySpark, Spark SQL, and Delta Lake.
Understanding of Bronze, Silver, and Gold layers in medallion architecture.
Experience with Delta tables, optimization, partitioning, and incremental processing.
Snowflake
Hands-on experience with Snowflake data warehousing and Snowflake SQL.
Understanding of virtual warehouses, databases, schemas, and query optimization.
Experience with data loading, transformation, and workload cost optimization.
Familiarity with Snowpark, role-based access control, and data security.
Hands-on experience developing and maintaining Apache Airflow DAGs for production data pipelines.
Experience integrating Airflow with Databricks, Snowflake, cloud storage, databases, and APIs.
Experience monitoring DAG execution, troubleshooting failed tasks, and configuring alerts.
Understanding of Airflow connections, variables, secrets management, and environment-specific configurations.
Familiarity with Airflow deployment, CI/CD, and production operations is an advantage.
Cloud Technologies
Hands-on experience with at least one major cloud platform:
AWS: S3, AWS Glue, Lambda, Redshift, EMR.
Microsoft Azure: Azure Data Factory, Azure Data Lake Storage, Azure Databricks, Synapse Analytics.
Google Cloud: BigQuery, Cloud Storage, Dataflow.
Experience integrating cloud services with Databricks, Snowflake, and Airflow is preferred.
Data Governance & Microsoft Purview
Experience with Microsoft Purview for data cataloging, discovery, and governance.
Understanding of metadata management, data lineage, data classification, and business glossaries.
Knowledge of data ownership, stewardship, and governance policies.
Familiarity with sensitive data identification, PII protection, access governance, and data lifecycle management.
Understanding of governance integration with Databricks, Snowflake, and cloud data services.




Familiarity with Unity Catalog and Snowflake governance capabilities is an advantage.
Databases & Data Modeling
Experience with PostgreSQL, SQL Server, MySQL, or other relational databases.
Familiarity with Parquet, Avro, JSON, and other common data formats.
Knowledge of data warehouse, data lake, and lakehouse design principles.
DevOps & CI/CD
Experience with Git and version control.
Familiarity with CI/CD pipelines, automated testing, and deployment automation.
Knowledge of Docker, logging, monitoring, alerting, and production support.
Experience managing code promotion across development, testing, and production environments.
Experience working across Databricks and Snowflake within enterprise data platforms.
Experience orchestrating multi-stage pipelines using Apache Airflow.
Microsoft Purview implementation or administration experience.
Knowledge of dbt, Fivetran, Airbyte, or similar data integration tools.
Exposure to Power BI, Tableau, or other BI platforms.
Understanding of GDPR, CCPA, and other applicable data privacy regulations.
Experience preparing governed datasets for AI, machine learning, and LLM-based applications.
Relevant certifications in Databricks, Snowflake, cloud data engineering, or data governance are an advantage.
Experience & Qualifications
Minimum 5+ years of professional experience in data engineering.
Strong hands-on experience building and supporting production data pipelines.
Practical experience with Databricks, Snowflake, and Apache Airflow.
Experience with cloud data platforms, orchestration, and enterprise data architectures.
Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or equivalent practical experience.
Strong ownership of data engineering deliverables.
Ability to design scalable, secure, and maintainable data solutions.
Solid understanding of data governance and data quality principles.
Excellent troubleshooting and root-cause analysis skills.
Commitment to engineering quality, documentation, security, and continuous improvement.
Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional.
Snowflake: SnowPro Core Certification or a relevant SnowPro Advanced certification, particularly SnowPro Advanced: Data Engineer.

Opportunities to work with modern data platforms, including Databricks and Snowflake.
Exposure to Apache Airflow orchestration and enterprise data governance using Microsoft Purview.
Opportunities to contribute to data platform architecture and technical decision-making.
Collaborative engineering environment and professional development opportunities.

📌 Senior Data Engineer - AWS, DevOps (Ahmedabad)
🏢 Aix Solutions
📍 Ahmedabad