– Data Engineer (AWS + Databricks)

Experience: 5+ Years

Location: Versatile / Hybrid

Key Responsibilities

- Build scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).

- Design and implement Medallion Architecture (Bronze, Silver, Gold) for Lakehouse.

- Develop batch and real-time pipelines using Spark Streaming, AWS Kinesis, or Glue Streaming.

- Manage data ingestion from diverse sources – RDBMS, APIs, S3, Kafka.

- Perform data profiling, validation, and quality checks.

- Optimize Spark jobs, partitions, and cluster configurations for performance.

- Implement and manage data infrastructure using AWS S3, Glue, Athena, Lambda, Step Functions, Redshift, EMR.

- Set up CI/CD pipelines (GitHub Actions, CodePipeline,



Jenkins).

- Collaborate with analysts and data scientists to deliver analytical datasets.

Technical Skills

- Databricks: PySpark, Spark SQL, Delta Lake, Unity Catalog, Job Workflows.

- AWS: S3, Glue, Athena, Kinesis, Lambda, Step Functions, EMR, Redshift, CloudWatch.

- Programming: Python (pandas), SQL (window functions, optimization).

- IaC & DevOps: Terraform / CloudFormation, Git, Jenkins, GitHub Actions.

- Performance Tuning: Spark optimizations, Z-ordering, partitioning, caching.

📌 Data Engineer - Databricks (Chennai)
🏢 Sedin Technologies
📍 Chennai