Key Responsibilities

- Design and implement scalable data platforms leveraging Data Lake, Lakehouse, Data Mesh, and modern data architecture patterns.

- Build and optimize data pipelines for batch and real-time processing using Databricks, Apache Spark, dbt, and cloud-native AWS services.

- Develop robust data ingestion frameworks for structured, semi-structured, and unstructured data from APIs, files, databases, and streaming sources.

- Design and develop scalable Python-based microservices to enable secure and productive data sharing across systems and applications.

- Build RESTful APIs and event-driven services for exposing curated datasets from Data Lake and Lakehouse platforms.

- Work extensively with Python, PySpark, and SQL for data transformation, processing, and data engineering workflows.

- Implement streaming data pipelines using Kafka/Kinesis and integrate them with downstream analytics and data platforms.

- Design and manage large-scale datasets using formats such as Parquet, JSON, CSV, and IoT/sensor data.

- Optimize data storage, partitioning, and query performance for high-volume analytical workloads.

- Design and implement data orchestration workflows using Apache Airflow and/or Databricks Workflows.

- Implement infrastructure and data platform components using Terraform and Infrastructure as Code (IaC) practices.

- Containerize data applications and services using Docker where applicable.

- Collaborate with cross-functional teams, including Data Architects, Data Analysts, and BI teams, to operationalize Data Lake and Lakehouse solutions.

- Contribute to data modeling in Lakehouse environments, including Medallion Architecture and dimensional modeling.

- Implement data quality, validation, monitoring, and observability practices across data pipelines.

- Use data quality and observability tools to identify data issues, monitor pipeline health, and improve data reliability.

- Ensure data quality, reliability, security,



and observability across data pipelines.

- Implement serverless data processing and pipeline solutions where applicable.

Required Skills & Experience

- 6-8 years of experience in Data Engineering or related roles.

- Strong understanding of modern data architectures such as Data Lake, Lakehouse, Data Mesh, and Data Products.

- Strong proficiency in Python, PySpark, and SQL.

- Hands-on experience with Databricks, Delta Lake, and/or Snowflake.

- Experience with ETL/ELT frameworks and data orchestration tools.

- Hands-on experience with Apache Airflow and/or Databricks Workflows.

- Practical experience with streaming technologies such as Kafka and/or AWS Kinesis.

- Strong understanding of batch processing frameworks and technologies such as Apache Spark, AWS Glue, and dbt.

- Proficiency in handling structured, semi-structured, and unstructured data.

- Experience with modern data modeling techniques, including Star Schema, Snowflake Schema, and 3NF.

- Familiarity with vector databases and data architectures supporting AI/ML use cases.

- Hands-on experience with AWS services including S3, Glue, Glue Data Catalog, Athena, and Redshift.

- Experience with Infrastructure as Code (IaC) tools such as Terraform.

- Experience with containerization technologies such as Docker.

- Experience with Git, CI/CD, and DevOps practices.

- Experience implementing data quality, monitoring, and observability solutions using relevant tools and frameworks.

- Good understanding of cloud security concepts, including IAM, encryption, access control, and data governance.

- Strong understanding of Data Lakehouse concepts, architecture, and implementation patterns.

- Familiarity with serverless architectures and Python-based serverless data processing solutions.

Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Sr. Software Engineer Database Level 1 (Chandigarh)
🏢 Bebo Technologies
📍 Chandigarh