1. Expertise in PySpark, Python
2. Knowledge and Skill in Databricks
3. Strong proficiency in Azure services: Data Factory, Synapse, Data Lake, Databricks.
4. Hands-on experience with ETL tools and data pipeline orchestration.
5. Proficiency in Python or Scala for data processing.
6. Knowledge of SQL and NoSQL databases.
7. Familiarity with data modeling and data warehousing concepts.
8. Understanding of security best practices for data in AWS.
9. Good hands on experience on Python, Numpy , pandas.
10. Experience in building ETL/ Data Warehouse transformation process.
11. Experience working with structured and unstructured data.
12.
Developing Big Data and non-Big Data cloud-based enterprise solutions in PySpark and SparkSQL and related frameworks/libraries,
13. Developing scalable and re-usable, self-service frameworks for data ingestion and processing,
14. Integrating end to end data pipelines to take data from data source to target data repositories ensuring the quality and consistency of data,
15. Knowledge of big data frameworks (Spark, Hadoop).
Positive-to-Have
- Experience with streaming data (Azure Event Hub, Kafka).
- Knowledge of big data frameworks (Spark, Hadoop).
- Exposure to machine learning data preparation.
- Familiarity with Agile methodologies.
- Knowledge of data management principles
- Knowledge of Microsoft Fabric architecture and services