Candidates who can join immediately will be preferred.
Job Description:
As a lead data engineer you will oversee data architecture, ETL processes, and analytics
pipelines, ensuring efficiency, scalability, and quality.
Key Responsibilities:
Job Requirements:
- Participate in the customers system design meetings and collect the functional/technical requirements.
- Build up data pipelines for consumption by the data science team.
- Skillful in ETL process and tools.
- Clear understanding and experience with Azure Data Factory, Python and PySpark or Spark, with HIVE, Airflow, Impala, and Hadoop and RDBMS architecture.
- Experience in writing Python programs and SQL queries.
- Experience in SQL Query tuning.
- Experienced in Shell Scripting(Unix/Linux).
- Build and maintain data pipelines in Spark/Pyspark with SQL and Python or SCALA.
- Knowledge of Cloud (Azure/AWS/GCP, etc..) technologies is additional.
- Good to have knowledge of Kubernetes, CI/CD concepts,
Apache Kafka
- Suggest and implement best practices in data integration.
- Guide the QA team in defining system integration tests as needed.
- Split the planned deliverables into tasks and assign them to the team.
- Needs to Maintain/Deploy the ETL code and follow the Agile methodology
- Needs to work on optimization wherever applicable.
- Positive oral, written and presentation skills.
Preferred Qualifications:
- Degree in Computer Science, IT, or similar field; a Masters is a plus.
- Hands-on experience with Python and Pyspark, Azure Data Factory
- Hands-on experience with Spark
- Great numerical and analytical skills.
- Working knowledge of cloud platforms such as MS Azure, AWS, etc...
- Technical expertise with data models, data mining, and segmentation techniques.
Tech Stack:
Minimum of 5+ years working experience with Spark, Hadoop eco systems.
Minimum of 4+ years working experience on designing data
📌 Senior/ Lead Data Engineer- Blackstraw- Immediate Joiner- Chennai
🏢 Blackstraw Technologies
📍 Chennai
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.