15 Nov
|
WebSenor®
|
Bengaluru
15 Nov
WebSenor®
Bengaluru
## Job highlights
- You will focus on creating scalable data architectures,pipelines,and real-time streaming frameworks using Databricks,Apache Spark,and Kafka,preferably on Azure
- Knowledge of Azure (preferred) or other clouds like AWS / GCP.
- Experience with data modeling,data lakes,and scalable solutions. ### Job Summary We are looking for a Senior Data Engineer skilled in Databricks and Big Data technologies. You will focus on creating scalable data architectures, pipelines, and real-time streaming frameworks using Databricks, Apache Spark, and Kafka, preferably on Azure. Your role will ensure data governance, security, and high performance in a cloud environment.
Location:
Bangalore. Employment type: Full-time. ### Responsibilities
- Develop end-to-end data solutions using Databricks for large-scale processing.
- Build and optimize data pipelines for high quality, reliability, and scalability.
- Perform complex data transformations with PySpark and Databricks.
- Integrate Kafka for real-time data streaming and processing.
- Collaborate with DevOps to optimize Databricks clusters and Kafka setups.
- Implement best practices for data governance, security, and privacy.
- Troubleshoot pipelines, resolve performance issues, and ensure data quality.
- Plan and scale Databricks and Kafka infrastructures for large workloads.
- Document workflows, processes, and configurations.
- Work with relational databases (e.g., MySQL, MS SQL) and understand complex SQL procedures. ### Requirements
- 3+ years experience in Big Data tools: Databricks, Apache Spark, Kafka, and Cloud platforms.
- Proficiency in programming: Python, PySpark, Scala, or Java.
- Solid SQL skills with relational/NoSQL databases.
- Experience with data modeling, data lakes, and scalable solutions.
- Knowledge of Azure (preferred) or other clouds like AWS/GCP.
- Expertise in data governance, security, and quality in Databricks.
- Familiar with Docker, Linux scripting, and Agile workflows.
- Certifications in Databricks or Azure Data Engineering.
- Experience with additional tools like Snowflake.
- Strong documentation, communication, and stakeholder management skills. ### Skills
- Databricks
- Apache Spark
- Kafka
- PySpark
- Python
- SQL
- Azure
- Scala
- Java Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.
📌 Senior Data Engineer (Bengaluru)
🏢 WebSenor®
📍 Bengaluru