We are looking for a motivated Databricks Developer with 23 years of experience in data engineering and big data technologies. The ideal candidate should have hands-on expertise in Databricks, PySpark, SQL, ETL development, and cloud platforms. The role involves building scalable data pipelines, optimizing data workflows, and supporting analytics and business intelligence initiatives.
Key Responsibilities
- Design, develop, and maintain data pipelines using Azure Databricks/AWS Databricks.
- Develop ETL/ELT processes using PySpark and Spark SQL.
- Ingest, transform, and process large-scale structured and unstructured datasets.
- Optimize Databricks jobs and clusters for performance and cost efficiency.
- Implement and manage Delta Lake architecture for reliable data processing.
- Collaborate with business stakeholders, data analysts, and engineering teams to understand data requirements.
- Ensure data quality,
governance, and security best practices.
- Troubleshoot production issues and provide timely resolutions.
- Participate in code reviews and adhere to development standards.
- Create and maintain technical documentation for solutions and processes.
Required Skills
- 2-5 years of experience in Databricks development.
- Strong proficiency in PySpark and Python.
- Hands-on experience with Databricks Workspace, Jobs, Notebooks, and Workflows.
- Solid knowledge of Spark SQL, DataFrames, and distributed computing concepts.
- Experience with Delta Lake and data lake architectures.
- Strong SQL skills and experience with relational databases.
- Understanding of ETL/ELT frameworks and data warehousing concepts.
- Experience with version control tools such as Git.
- Knowledge of Linux/Unix environments.
- Strong analytical and problem-solving skills.