Job Summary
We are looking for an experienced Senior Data Engineer with 4+ years of experience in designing, developing, and maintaining scalable data pipelines and modern data platforms. The ideal candidate will have strong hands-on expertise in Python, SQL, ETL/ELT, Databricks, Snowflake, Apache Airflow, cloud data engineering, and enterprise data governance. The candidate will be responsible for building robust data pipelines, implementing lakehouse and data warehouse architectures, orchestrating complex workflows, optimizing data processing performance, and ensuring data quality, security, and compliance.
Experience with Microsoft Purview, metadata management, data lineage, and data governance frameworks is highly desirable.
The role requires collaboration with data architects, analysts, data scientists, and business stakeholders to deliver reliable, secure, governed, and analytics-ready data solutions.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Python, SQL, PySpark, and modern data engineering technologies.
- Build and optimize data processing solutions using Databricks, Apache Spark, Delta Lake, and lakehouse architecture.
- Design, develop, and manage Snowflake data warehouse solutions, including data ingestion, transformation, performance optimization, and cost management.
- Develop, schedule, and orchestrate complex data pipelines using Apache Airflow, including DAG development, task dependencies, scheduling, retries, and failure handling.
- Integrate Airflow with Databricks, Snowflake, cloud storage, APIs, databases, and other data services.
- Implement batch and near-real-time data pipelines integrating relational databases, REST APIs, third-party applications, and cloud platforms.
- Implement incremental data loading, Change Data Capture (CDC), schema evolution, data partitioning, and data synchronization.
- Design dimensional data models, star schemas, slowly changing dimensions (SCD), and analytical data structures.
- Establish data quality checks, validation rules, reconciliation processes, and automated pipeline monitoring.
- Implement data governance, metadata management, data lineage, data cataloging, and data classification using Microsoft Purview and related tools.
- Configure data discovery, sensitive data classification, business glossaries, and data asset documentation.
- Support data ownership, data stewardship, access governance, and least-privilege access principles.
- Optimize SQL queries, Spark jobs, Snowflake workloads, and cloud infrastructure costs.
- Implement CI/CD, automated testing, version control, and deployment workflows for data engineering solutions.
- Troubleshoot production data issues, perform root-cause analysis, and ensure pipeline reliability.
- Conduct code reviews, mentor junior engineers, and contribute to data engineering architecture and best practices.
Required Technical Skills
A. Programming & Data Processing
- Strong proficiency in Python and SQL.
- Hands-on experience with PySpark and Apache Spark.
- Experience developing and maintaining production-grade ETL/ELT pipelines.
- Knowledge of incremental processing, CDC, data validation, and performance tuning.
- Ability to write reusable, maintainable, and well-tested code.
B. Databricks
- Hands-on experience with Databricks, Apache Spark, PySpark, Spark SQL, and Delta Lake.
- Understanding of Bronze, Silver, and Gold layers in medallion architecture.
- Experience with Delta tables, partitioning, optimization, and incremental data processing.
- Experience developing and maintaining data engineering workloads in Databricks.
- Familiarity with Databricks Workflows and Unity Catalog is desirable.
C. Snowflake
- Hands-on experience with Snowflake data warehousing and Snowflake SQL.
- Understanding of Snowflake architecture, databases, schemas, tables, and virtual warehouses.
- Experience with data loading, transformation, query optimization, and workload cost management.
- Knowledge of Snowpipe, Streams, Tasks, and Change Data Capture concepts.
- Familiarity with Snowpark, role-based access control (RBAC), and Snowflake security features.
D. Apache Airflow & Workflow Orchestration
- Hands-on experience developing and maintaining Apache Airflow DAGs for production data pipelines.
- Strong understanding of DAG design, task dependencies, scheduling, operators, sensors, retries, and error handling.
- Experience integrating Airflow with Databricks, Snowflake, cloud storage, databases, and APIs.
- Knowledge of backfills, catchup behavior, idempotency, and dependency management.
- Experience monitoring DAG execution, troubleshooting failed tasks, and configuring alerts.
- Understanding of Airflow connections, variables, secrets management, and environment-specific configurations.
- Familiarity with Airflow deployment, CI/CD, and production operations.
E. Data Governance & Microsoft Purview
- Experience with Microsoft Purview for data discovery, cataloging, metadata management, and governance is preferred.
- Understanding of enterprise data governance frameworks, policies, and standards.
- Experience with technical and business metadata, data lineage, data classification, and business glossary management.
- Knowledge of data ownership, data stewardship, and governance responsibilities.
- Familiarity with identifying and classifying sensitive data, including Personally Identifiable Information (PII).
- Understanding of data access policies, RBAC, least-privilege principles, and sensitive data protection.
- Knowledge of data lifecycle management, retention policies, and applicable regulatory requirements.
- Familiarity with governance integrations across Databricks, Snowflake, and cloud data services.
- Experience with Databricks Unity Catalog and Snowflake governance capabilities is an advantage.
F. Cloud Technologies Hands-on experience with at least one major cloud platform:
- AWS: Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift, and Amazon EMR.
- Microsoft Azure: Azure Data Factory, Azure Data Lake Storage, Azure Databricks, and Azure Synapse Analytics.
- Google Cloud Platform: BigQuery, Cloud Storage, and Dataflow.
Experience integrating cloud services with Databricks, Snowflake, and Apache Airflow is preferred. G. Databases & Data Modeling
- Experience with PostgreSQL, SQL Server, MySQL, or other relational databases.
- Strong understanding of dimensional modeling, star schemas, snowflake schemas, and slowly changing dimensions.
- Familiarity with Parquet, Avro, JSON, and other common data formats.
- Understanding of data warehouses, data lakes, and lakehouse architectures.
H. DevOps, CI/CD & Engineering Practices
- Experience with Git and version control.
- Familiarity with CI/CD pipelines, automated testing, and deployment automation.
- Knowledge of Docker, logging, monitoring,
alerting, and production support.
- Experience managing deployments across development, testing, and production environments.
4. Preferred Qualifications
- Experience working across Databricks and Snowflake within enterprise data platforms.
- Experience orchestrating multi-stage data pipelines using Apache Airflow.
- Microsoft Purview implementation, configuration, or administration experience.
- Knowledge of dbt, Fivetran, Airbyte, or similar data integration tools.
- Exposure to Power BI, Tableau, or other business intelligence platforms.
- Understanding of GDPR, CCPA, and other applicable data privacy and compliance requirements.
- Experience preparing governed, high-quality datasets for AI, machine learning, and LLM-based applications.
- Experience with data platform modernization, migration, and performance optimization.
5. Certifications — Good to Have
- Databricks: Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional.
- Snowflake: SnowPro Core Certification or a relevant SnowPro Advanced certification, particularly SnowPro Advanced: Data Engineer.
6. Experience & Qualifications
- Minimum 5+ years of professional experience in data engineering or a closely related role.
- Strong hands-on experience building and supporting production data pipelines.
- Practical experience with Databricks, Snowflake, and Apache Airflow.
- Experience with at least one major cloud platform and up-to-date data engineering architectures.
- Understanding of enterprise data governance, data quality, and security principles.
- Strong analytical, problem-solving, communication, documentation, and stakeholder management skills.
- Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related discipline, or equivalent practical experience.
7. Key Competencies
- Strong ownership of data engineering deliverables.
- Ability to design scalable, secure, and maintainable data solutions.
- Strong understanding of data governance and data quality principles.
- Excellent troubleshooting and root-cause analysis skills.
- Ability to work independently and collaborate with cross-functional teams.
- Commitment to engineering quality, documentation, security, and continuous improvement.
What We Offer
- Opportunities to work with modern data platforms, including Databricks and Snowflake.
- Exposure to Apache Airflow orchestration and enterprise data governance using Microsoft Purview.
- Opportunities to contribute to data platform architecture and technical decision-making.
- Collaborative engineering environment and professional development opportunities.
Ready to take your career to the next level? Submit your resume to
[email protected] or call our HR team +919265320742 to get started!
Build Your Future @DevsTree
Ready to make an impact? At DevsTree, we're more than just a tech company; we're a community of innovators, problem-solvers, and dream-weavers dedicated to crafting cutting-edge digital solutions. If you're passionate about technology and eager to grow, you've found your next home.
Learning Excellence
Innovative thinking A Learning attitude Opportunities Learn Next-Gen Technologies
Global Exposure
Work on forward-thinking projects with international clients.
Your Growth, Our Priority.
Career Path Planning with a Culture of Rewards & Recognition
Work - Life Balance
Celebrate life with balanced work
Open Doors Culture
Flat Hierarchy. Transparent Culture.
Work Smarter. Celebrate Harder
Fun, Culture & Team Bonding Activities
📌 Senior Data Engineer ( 4+ years Exp.) (Ahmedabad)
🏢 Devstree IT Services
📍 Ahmedabad