Job Summary:
Key Responsibilities:
- Design, build, and optimize data pipelines using Databricks, Python, and AWS services to ingest, transform, and load large datasets from various sources.
- Develop and maintain ETL/ELT processes, ensuring data quality, reliability, and performance.
- Utilize Databricks platform features, including notebooks, Delta Lake, and Spark SQL, for data processing, analysis, and machine learning workflows.
- Implement and manage data solutions on AWS, leveraging services such as S3, EC2, Lambda, Glue, Redshift, Kinesis, and IAM.
- Write clean, productive, and well-documented code in Python and PySpark for data manipulation, scripting, and automation.
- Collaborate with data scientists, analysts,



and other engineering teams to understand data requirements and deliver appropriate data solutions.
- Monitor data pipelines and systems for performance, errors, and data integrity, implementing necessary alerts and troubleshooting procedures.
- Optimize existing data infrastructure and processes for cost efficiency, scalability, and improved performance.
- Ensure data security and compliance with industry best practices and company policies.
- Participate in code reviews, contribute to architectural discussions, and mentor junior team members.
- Stay up-to-date with emerging technologies and best practices in data engineering, cloud computing, and big data.

📌 Databrics,python,aws (Chennai)
🏢 Archelons Solutions Private
📍 Chennai