We are looking for an experienced Databricks Lead Developer / Architect with 7+ years of experience in data engineering, cloud data platforms, and distributed data processing. The ideal candidate will have strong hands-on expertise with Databricks, Apache Spark, Python, SQL, and modern data lake/lakehouse architectures, with the ability to lead technical design and development initiatives.

Experience working with AWS cloud services and a Databricks certification will be an added advantage.

Preferred Location: Mohali/Noida

Key Responsibilities:

● Design and architect scalable, secure, and high-performance Databricks/Lakehouse solutions.

● Lead the development of enterprise-grade data engineering pipelines using Databricks, Apache Spark, Python, and SQL.

● Define technical architecture, development standards, best practices, and reusable frameworks.

● Design and implement data ingestion, transformation, integration, and processing pipelines.

● Work extensively with Delta Lake and develop optimized data models and lakehouse architectures.

● Optimize Spark and Databricks workloads for performance, scalability, reliability, and cost.

● Provide technical leadership and mentoring to data engineers and development teams.

● Collaborate with business stakeholders, data architects, analysts, and engineering teams to translate requirements into technical solutions.

● Implement data quality, validation, monitoring, logging, and error-handling frameworks.

● Design solutions for batch and, where applicable, streaming data processing.

● Establish CI/CD and deployment practices for Databricks workloads.

● Ensure solutions follow enterprise standards for security, governance, access control, and data management.

● Troubleshoot complex production issues and provide technical guidance to development teams.





● Evaluate new Databricks and cloud capabilities and recommend improvements to the data platform.

Skills & Experience:

● 7+ years of overall experience in data engineering, software development, or data platform engineering.

● Strong hands-on experience with Databricks and Apache Spark.

● Strong programming experience in Python and SQL.

● Solid understanding of Delta Lake, Lakehouse architecture, data warehousing, and data lake concepts.

● Experience designing and developing scalable ETL/ELT data pipelines.

● Strong understanding of Spark architecture, optimization, partitioning, caching, joins, and performance tuning.

● Experience with data integration and processing of structured and semi-structured data.

● Experience with Databricks Workflows/Jobs, notebooks, clusters, and deployment processes.

● Good understanding of data modeling and distributed data processing.

● Experience with version control and CI/CD practices, preferably using Git and automated deployment pipelines.

● Strong problem-solving, analytical, communication, and technical leadership skills.

● Ability to independently own architecture and technical decisions for complex data engineering solutions.

AWS Experience — Preferred

● Experience working with AWS-based data platforms is highly desirable, including one or more of:

○ Amazon S3

○ AWS Glue

○ AWS Lambda

○ Amazon Redshift

○ Amazon EMR

○ AWS IAM





○ Amazon CloudWatch

○ AWS networking and security concepts

Experience integrating Databricks with AWS services and designing cloud-native data architectures will be an added advantage.

Preferred / Nice-to-Have

● Databricks certification such as Databricks Certified Data Engineer or Databricks Certified Data Engineer Qualified.

● Experience with Delta Live Tables / Lakeflow Declarative Pipelines.

● Experience with Unity Catalog and Databricks governance.

● Experience with Databricks SQL and BI/data analytics integrations.

● Experience with real-time/streaming technologies such as Kafka or Spark Structured Streaming.

● Experience with orchestration tools such as Airflow or similar platforms.

● Knowledge of infrastructure-as-code tools such as Terraform.

● Experience implementing enterprise data governance, security, and observability.

● Experience working in Agile/Scrum development environments.

Leadership Expectations

● Act as the technical lead and subject-matter expert for Databricks and data engineering initiatives.

● Review technical designs and code and establish engineering best practices.

● Mentor developers and help build technical capability within the team.

● Drive architecture decisions and provide guidance on scalability, performance, security, and maintainability.

● Communicate complex technical concepts effectively to both technical and non-technical stakeholders.

Education : Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field is preferred.

Experience Level: 7+ years of overall experience, with significant hands-on experience in Databricks and data engineering and preferably experience in AWS cloud environments.

📌 Databricks Lead Developer / Architect (Noida)
🏢 SourceFuse
📍 Noida