Key Responsibilities: Design, build, and maintain robust ETL/ELT data pipelines using Apache Spark on Databricks. Implement data Lakehouse architecture using Delta Lake for cost-effective data storage and analytics.
Use Databricks
Workflows for orchestrating batch and streaming pipelines. Develop and maintain CI/CD pipelines for data applications using tools such as AWS, GitHub Actions or Databricks Repos. Monitor pipeline performance and troubleshoot data issues in real-time and batch environments. Document solutions, workflows, and technical standards.
Required Skills: Experience in data engineering with a strong focus on AWS Databricks and Apache Spark. Proficiency in PySpark, SQL, and Python.
Experience with Delta Lake, Databricks SQL, and Unity Catalog. Hands-on experience with cloud platforms Familiarity with data lakehouse architecture, data warehousing and streaming data Strong understanding of ETL best practices, data partitioning, and performance tuning.
Experience with CI/CD for data pipelines.
Experience in Machine learning and Data Analytics is added advantage. Strong exposure and understanding on the AI and Agentic AI technologies. Excellent problem-solving and communication skills #QualificationsKey Responsibilities: Design, build,
and maintain robust ETL/ELT data pipelines using Apache Spark on Databricks. Implement data Lakehouse architecture using Delta Lake for cost-effective data storage and analytics.
Use Databricks
Workflows for orchestrating batch and streaming pipelines. Develop and maintain CI/CD pipelines for data applications using tools such as AWS, GitHub Actions or Databricks Repos. Monitor pipeline performance and troubleshoot data issues in real-time and batch environments. Document solutions, workflows, and technical standards.
Required Skills: Experience in data engineering with a strong focus on AWS Databricks and Apache Spark. Proficiency in PySpark, SQL, and Python.
Experience with Delta Lake, Databricks SQL, and Unity Catalog. Hands-on experience with cloud platforms Familiarity with data lakehouse architecture, data warehousing and streaming data Strong understanding of ETL best practices, data partitioning, and performance tuning.
Experience with CI/CD for data pipelines.
Experience in Machine learning and Data Analytics is added advantage. Robust exposure and understanding on the AI and Agentic AI technologies. Excellent problem-solving and communication skills
📌 EFS FCRM 170694 (Telangana)
🏢 ADP
📍 Telangana