We are looking for a Lead Data Engineer with strong Databricks expertise who can also design and build accelerators for automation The role requires hands-on development in Databricks PySpark Spark SQL Delta Lake Autoloader DLT etc along with experience in creating frameworks for test data generation report validation and backend validation You will be leading technical work mentoring team members and building reusable solutions that speed up project delivery Key Responsibilities Design develop and optimize ETL ELT pipelines on Databricks using PySpark Spark SQL Delta Lake DLT live Build automation accelerators such as Synthetic test data generators for large-scale data sets Automated report validation tools to compare dashboards with backend data Backend validation frameworks for reconciliation and data quality checks Implement data validation frameworks Great Expectations Deequ or custom solutions Ensure data quality lineage and governance using Unity Catalog Collaborate with QA BI and business teams to integrate accelerators into workflows Drive best practices in coding performance tuning and cost optimization on Databricks Lead and mentor data engineers review code and set technical standards Must-Have Skills 5-7 years of experience in data engineering with at least 3-5 years hands-on experience in databricks Databricks PySpark Spark SQL Delta Lake Delta Live Tables DLT Automation Accelerators Experience creating test data generators validation frameworks report reconciliation tools Programming language Robust in Python PySpark and SQL Validation Tools open source tools like Great Expectations Deequ or equivalent custom frameworks Data Pipelines Batch and streaming pipeline design Data Governance Unity Catalog for access lineage and compliance DevOps CI-CD Git Azure DevOps Jenkins or GitHub Actions for deployment automation Cloud Storage Azure preferred AWS GCP with hands-on in ADLS S3 GCS Good-to-Have Skills Streaming Kafka EventHub or Kinesis Infra-as-Code Terraform for Databricks and cloud provisioning Reporting Tools Power BI Tableau or Looker for validation accelerators Testing Exposure to pytest or unittest for automation Qualifications Bachelor s or Master s degree in Computer Science Engineering or a related field Labcorp is proud to be an Equal Opportunity Employer Labcorp strives for inclusion and belonging in the workforce and does not tolerate harassment or discrimination of any kind We make employment decisions based on the needs of our business and the qualifications and merit of the individual Qualified applicants will receive consideration for employment without regard to race religion color national origin sex including pregnancy childbirth or related medical conditions family or parental status marital civil union or domestic partnership status sexual orientation gender identity gender expression personal appearance age veteran status disability genetic information or any other legally protected characteristic Additionally all qualified applicants with arrest or conviction records will be considered for employment in accordance with applicable law We encourage all to apply If you are an individual with a disability who needs assistance using our online tools to search and apply for jobs or needs an accommodation please visit our accessibility site or contact us at Labcorp Accessibility For more information about how we collect and store your personal data please see our Privacy Statement

📌 Lead Data Engineer (Karnataka)
🏢 Labcorp
📍 Karnataka