Role Overview

We are looking for an experienced Big Data Data Engineer with strong hands-on expertise in Databricks, PySpark, AWS Glue and Data Governance. The ideal candidate should have experience designing, developing and optimizing scalable data pipelines, managing large datasets and implementing data governance practices across cloud-based data platforms.

Key Responsibilities

- Design, develop and maintain scalable data pipelines using Databricks and PySpark.
- Build and optimize ETL/ELT workflows for large-scale data processing.
- Develop data ingestion, transformation and integration solutions using AWS Glue.
- Work with Databricks Lakehouse architecture, Delta Lake and data processing frameworks.
- Implement data governance practices, including data quality, data lineage, metadata management and access controls.
- Ensure data security, consistency, reliability and compliance across data platforms.
- Optimize Spark jobs, cluster performance and data processing workflows.
- Collaborate with Data Architects, Data Scientists and cross-functional engineering teams.
- Troubleshoot data pipeline issues and ensure performance, scalability and reliability.
- Follow best practices for data engineering, CI/CD,



version control and deployment.

Required Technical Skills
- 5–8 years of experience in Big Data Engineering / Data Engineering.
- Strong hands-on experience with Databricks and PySpark.
- Experience with AWS Glue for ETL and data integration.
- Robust understanding of Data Governance, Data Quality and Data Management.
- Experience with Delta Lake, Spark SQL and Lakehouse architecture.
- Proficiency in Python and SQL.
- Experience working with cloud-based data platforms, preferably AWS.
- Understanding of data lineage, metadata management, data cataloguing and access control.

Certifications
- Databricks Certification is required / strongly preferred.
- AWS certification or relevant AWS Glue experience will be an added advantage.

Preferred Skills
- Experience with Unity Catalog and Databricks governance.
- Exposure to AWS S3, Athena, Redshift and Lake Formation.
- Experience with Apache Airflow or other workflow orchestration tools.
- Knowledge of CI/CD, Git and automated deployment practices.
- Experience handling high-volume enterprise datasets.

📌 Big Data Developer (Karnataka)
🏢 NexionPro Services
📍 Karnataka