– Data Engineer (3–4 Years Experience)

Job Title

Data Engineer – Databricks, Python, SQL & Cloud Migration

Experience

3 – 4 Years Relevant Experience

Location

Hyderabad / Hybrid

Job Summary

We are seeking a motivated and skilled Data Engineer with 3–4 years of experience in building scalable data pipelines, ETL processes, and cloud-based data solutions. The ideal candidate should have strong hands-on expertise in Databricks, Python, SQL, and Apache Airflow, along with experience supporting data migration initiatives from on-premises platforms to modern cloud environments.

The role involves designing, developing, optimizing, and maintaining data pipelines that support analytics, reporting, and business intelligence requirements.

Key Responsibilities

Data Engineering & ETL Development

 Design, develop, and maintain scalable ETL/ELT pipelines using Python, PySpark,

Databricks, and SQL.

 Build reusable data ingestion, transformation, and data quality frameworks.

 Develop batch and near real-time data processing solutions.

 Perform data cleansing, transformation, validation, and enrichment activities.

Databricks Development

 Build and optimize Spark/PySpark applications in Azure Databricks.

 Work with Delta Lake, data partitioning, caching, and performance tuning techniques.

 Create and maintain notebooks, jobs, workflows, and reusable libraries.

 Implement medallion architecture (Bronze, Silver, Gold layers).

Workflow Orchestration

 Develop and maintain Apache Airflow DAGs for scheduling and orchestration.





 Monitor pipeline execution and troubleshoot failures.

 Implement alerting, logging, and retry mechanisms.

Classification

Internal

Cloud Migration Activities

 Participate in migration of on-premise data assets and ETL jobs to cloud platforms.

 Analyze legacy ETL workflows and redesign them using cloud-native services.

 Support data validation, reconciliation, and migration testing activities.

 Assist in cutover and production deployment activities.

SQL & Data Modeling

 Develop complex SQL queries, stored procedures, and performance optimization techniques.

 Design dimensional and normalized data models.

 Support data warehousing and reporting requirements.

Collaboration & Governance

 Collaborate with business analysts, architects, and stakeholders to understand data requirements.

 Participate in Agile ceremonies, sprint planning, and code reviews.

 Ensure adherence to security, governance, and data quality standards.

Required Technical Skills

Skill Experience

Azure Databricks 3+ Years

Python / PySpark 3+ Years

SQL 3+ Years

Apache Airflow 2+ Years

ETL / ELT Development 3+ Years

Cloud Migration Projects 1+ Years

Git / Version Control Working





Knowledge

Azure Data Lake Storage (ADLS) Preferred

CI/CD (Azure DevOps, GitHub

Actions) Preferred

Classification

Internal

Required Qualifications

 Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.

 3–4 years of experience in Data Engineering.

 Strong understanding of Data Warehousing concepts.

 Experience working with large-scale structured and semi-structured datasets.

 Good knowledge of cloud-based analytics platforms.

Preferred Skills

 Azure Data Factory (ADF)

 Delta Lake

 Snowflake

 Kafka/Event Streaming

 Azure Synapse Analytics

 Data Quality and Data Governance frameworks

 Terraform or Infrastructure as Code

 Exposure to CI/CD pipelines

Key Competencies

 Strong analytical and problem-solving skills.

 Excellent communication and stakeholder management.

 Ability to work independently and within Agile teams.

 Solid debugging and performance tuning capabilities.

 Focus on quality, automation, and continuous improvement.

Nice to Have

 Azure Data Engineer Associate Certification.

 Experience with enterprise-scale cloud migration programs.

 Exposure to healthcare, finance, telecom, or retail data domains.

This JD is suitable for sourcing candidates for Azure Databricks Data Engineer / Cloud

Migration Data Engineer roles in large-scale migration programs such as

Teradata/Hadoop/DataStage to Azure Databricks migrations.

📌 Data Engineer – Databricks, Python, SQL & Cloud Migration (Hyderabad)
🏢 HCL INDIA
📍 Hyderabad