Key Responsibilities
Develop and maintain scalable data pipelines for batch and real-time data processing
Build robust data ingestion frameworks using APIs, file-based ingestion, and streaming sources
Design and implement Python and PySpark-based data transformation logic for large-scale datasets
Develop Python-based microservices and REST APIs for secure and productive data access and sharing
Integrate data pipelines with platforms such as Databricks, Delta Lake, and Snowflake
Experience with serverless data processing (e.g., AWS Lambda, Azure Functions with Python
Implement and manage streaming pipelines using Kafka / Kinesis
Work with structured, semi-structured, and unstructured data (Parquet, JSON, CSV, sensor/IoT data
Optimize data processing jobs for performance, scalability, and cost efficiency
Apply best practices for data partitioning, storage optimization, and query performance tuning
Ensure data quality, validation, and monitoring across pipelines
Collaborate with data architects, analysts,



and downstream consumers to deliver reliable datasets
Support data modeling and transformation aligned with Lakehouse principles (e.g., medallion architecture)
Contribute to automation and deployment of pipelines using orchestration and CI/CD practices

Required Skills & Experience
2–4 years of hands-on experience in Data Engineering
Strong programming expertise in Python and PySpark
Solid experience in SQL for data transformation and analysis
Hands-on experience building data pipelines (ETL/ELT) in production environments
Experience working with APIs and microservices for data ingestion and exposure
Strong understanding of data processing frameworks such as Spark, AWS Glue, or Dbt
Experience handling large-scale datasets across multiple formats (Parquet, JSON, CSV, etc)
Good understanding of Data Lake / Lakehouse concepts and implementation patterns

📌 Database Engineer (Chandigarh)
🏢 Bebo Technologies
📍 Chandigarh