- Databricks & Spark: Hands-on experience with Databricks, Apache Spark, and PySpark for big data processing.
- Cloud Platforms: Proficiency in Azure (preferred), AWS, or GCP for data engineering workflows
- SQL & Data Modeling: Strong knowledge of SQL, schema design, and dimensional data modeling.
- ETL/ELT Tools: Experience with tools like Azure Data Factory, Informatica, or Talend.
- Data Warehousing: Familiarity with technologies such as Azure Synapse, Snowflake, or BigQuery.
- Strong problem-solving and debugging skills.
- Excellent communication and teamwork abilities.
- Ability to work in a fast-paced, agile environment.
Good-to-Have
- Knowledge of Event Hub, IoT Hub,
and streaming analytics.
- Exposure to Power BI, Azure DevOps, and CI/CD pipelines.
- Understanding of machine learning libraries (MLlib) and predictive analytics.
SN Responsibility of / Expectations from the Role
1
Design and maintain scalable data pipelines using Databricks and PySpark.
2
Implement and optimize ETL processes for large datasets.
3
Collaborate with cross-functional teams to define data requirements.
4
Ensure data integrity and quality across the lifecycle.
5
Integrate solutions with cloud platforms and manage large-scale datasets.