17 Nov
|
TuTeck Technologies
|
Kolkata
17 Nov
TuTeck Technologies
Kolkata
– Databricks Data Engineer
Location: Kolkata, India (On-site)
Experience: 5–8 Years
Notice Period: Immediate Joiners Preferred
Employment Type: Full-Time
Job Summary
We are looking for an experienced Databricks Data Engineer with strong hands-on expertise in PySpark, Python, and SQL to join our data engineering team in Kolkata.
The ideal candidate will be responsible for designing, developing, optimizing, and maintaining scalable data pipelines and data processing solutions using Databricks across cloud environments such as Azure or AWS. The candidate should have practical experience in implementing Medallion Architecture, working with large datasets, and building robust ETL/ELT pipelines.
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, and SQL.
- Integrate Databricks with multiple external data sources, including cloud-based storage and databases.
- Design and implement Medallion Architecture (Bronze, Silver, and Gold layers) based on business and technical requirements.
- Process, transform, and manage large datasets to support enterprise data engineering and analytics requirements.
- Optimize Spark jobs, queries, and data processing workflows for performance, scalability, and cost efficiency.
- Work extensively with Delta Lake for reliable data storage, processing, and management.
- Develop and automate data workflows using Databricks Workflows, Apache Airflow, or similar orchestration tools.
- Implement data quality checks, validation frameworks, and pipeline monitoring mechanisms.
- Collaborate with data scientists, business analysts, BI developers, and other stakeholders.
- Troubleshoot data pipeline failures, perform root cause analysis, and implement corrective actions.
- Follow best practices for data security, governance, version control, and CI/CD implementation.
- Participate in code reviews, technical discussions, and Agile development activities.
Required Skills & Experience
- 5–8 years of relevant experience in Data Engineering, with substantial hands-on experience in Databricks.
- Strong programming expertise in PySpark and Python.
- Advanced SQL skills, including complex queries, joins, aggregations, and query optimization.
- Hands-on experience with Databricks, Delta Lake, and Databricks Workflows.
- Strong understanding of Apache Spark architecture, distributed processing, and performance tuning.
- Experience implementing Medallion Architecture and building scalable data pipelines.
- Practical experience with at least one cloud platform: Microsoft Azure, AWS, or GCP.
- Understanding of Data Lake, Data Warehouse, and up-to-date data engineering concepts.
- Familiarity with Git, CI/CD pipelines, and code versioning practices.
- Good analytical, troubleshooting, and problem-solving skills.
- Experience working in Agile development environments.
Preferred Skills
- Experience with Azure Data Factory (ADF), Azure Synapse Analytics, or Snowflake.
- Knowledge of data modeling, data governance, and data management frameworks.
- Experience with cloud storage services such as Azure Data Lake Storage (ADLS) or Amazon S3.
- Exposure to MLflow and machine learning pipeline integration within Databricks.
- Familiarity with workflow orchestration tools such as Apache Airflow.
- Knowledge of data security, access management, and governance best practices.
Educational Qualification Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Engineering, or a related discipline.
📌 Databricks Data Engineer (Kolkata)
🏢 TuTeck Technologies
📍 Kolkata