Company:IT Services Organization

Key Skills:

Pyspark, AWS, Databricks, SQL, Data Engineer, AWS Lambda

Roles and Responsibilities:

- Design, develop, and maintain end-to-end data pipelines using Databricks, PySpark, SQL, and AWS services.

- Build scalable data processing solutions using Delta Lake, Unity Catalog, and Apache Spark capabilities.

- Develop SQL queries, stored procedures, data marts, views, and reporting-layer components for relational data platforms.

- Analyze Spark and Databricks workloads, reduce job runtimes, and implement cost and performance optimization measures.

- Participate in solution design, code reviews, production support, proof-of-concept initiatives, and technical documentation.

- Support CI/CD pipelines and infrastructure automation using Terraform and GitHub or equivalent version-control platforms.

- Provide technical mentoring to junior engineers and coordinate delivery for midsized engineering teams.

Skills Required:

- 5 to 8 years of IT experience with solid hands-on experience in Databricks on AWS.





- Proficiency in PySpark and Python for data engineering and distributed data processing.

- Strong SQL development experience, including stored procedures, data marts, and relational database views.

- Experience designing ETL workflows, end-to-end data pipeline architectures, data warehouse layers, and reporting layers.

- Working knowledge of AWS data and orchestration services, including Amazon S3, AWS Glue, Amazon SNS, and AWS Step Functions.

- Experience with Delta Lake, Unity Catalog, Spark troubleshooting, workload tuning, and Databricks cost optimization.

- Exposure to CI/CD pipelines, Terraform-based infrastructure automation, GitHub, Agile delivery, and SDLC practices.

Good to Have

- Experience using AWS Lambda in data engineering or cloud integration workflows.

Education:

Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.

📌 Data Engineer - AWS + Databricks (Bengaluru)
🏢 MyCareernet
📍 Bengaluru