21 Nov
|
HCLTech
|
Bengaluru
Job Summary
The Databricks Developer will design, develop, test, and optimize scalable data-processing solutions using Apache Spark, PySpark, Delta Lake, and the Databricks platform. The role requires practical experience in reliable batch and change-data-capture pipelines, cloud integration, workload troubleshooting, and storage and processing optimization.
Key Responsibilities
- Design, build, and maintain scalable Spark and PySpark data pipelines.
- Develop Delta Lake solutions with effective schema handling, data quality controls, and read/write patterns.
- Implement batch and CDC pipelines for incremental processing.
- Optimize Spark jobs, storage layouts, and pipeline execution for performance, reliability, and cost.
- Develop Databricks notebooks, workflows, jobs, and reusable components.
- Integrate Databricks workloads with AWS services such as S3, EC2, ECS, and DynamoDB.
- Troubleshoot pipeline failures, data-quality issues, performance bottlenecks, and production incidents.
- Apply modular development, version control, testing, code review, deployment automation, and documentation practices.
- Use REST APIs and platform automation where required.
Skill Requirements
- Apache Spark knowledge from core concepts through distributed processing patterns.
- Production-grade Python and PySpark development.
- Practical Delta Lake and Delta table experience.
- Storage, file-layout, Spark execution, and pipeline optimization.
- Databricks notebooks, jobs, workflows, compute, and development practices.
- Strong in Python data structures and algorithms.
- AWS data-engineering services, particularly S3, with familiarity with EC2, ECS, and DynamoDB.
- CDC and incremental data-processing solutions.
- REST APIs for integration and automation.
- Ability to develop solutions using Databricks Apps when required.
Must Have Skill
Databricks,PySpark,Python
Good to Have Skill
RESTAPI
Other Requirements
Positive-to-Have Qualifications
- Spark Structured Streaming fundamentals.
- Exposure to Mosaic AI capabilities.
- Relevant Databricks certification or preparation toward certification.
Expected Developer Profile
The preferred candidate is a hands-on developer who can translate requirements into maintainable Databricks solutions. The strongest emphasis is on Spark, PySpark, Delta Lake, pipeline and storage optimization, AWS integration, and CDC.
📌 Senior Technical Specialist (Bengaluru)
🏢 HCLTech
📍 Bengaluru