Implement Data Architecture: Implement scalable, secure, and efficient data architecture on on-prem and cloud platforms (Azure/GCP/AWS) to support business growth and data-driven decision-making.
Collaborate with data engineers, and product teams to identify data requirements and develop data models that meet business needs.
Data Ingestion and Integration:
Develop and maintain data ingestion pipelines using various tools and technologies, such as Apache Spark, PySpark, Kafka, and Flume.
Experience with GenAI tooling: LangChain, OpenAI API, Amazon Bedrock, or Vertex AI integrations, Claude, Copilot
Integrate data from multiple sources, including relational databases, NoSQL databases, APIs, and files.
Batch and Stream Processing:
Develop and maintain batch and stream processing pipelines using tools like Apache Spark & databricks.
Integrate with messaging systems, such as Apache Kafka, Amazon Kinesis, and Google Cloud Pub/Sub.
SQL Knowledge:
Very strong SQL knowledge, including query optimization, indexing, and database design.
Delta Lake and Data Warehouse:
Design and implement Delta Lake and data warehouse/mart solutions to support business intelligence, reporting, and analytics.
Develop and maintain data pipelines to ingest, process, and store data in Delta Lake and data warehouses.
Distributed Databases and Data Warehousing:
Implement and maintain data warehouses, such as Amazon Redshift, Google BigQuery, and Azure Synapse Analytics.
Database Design and Development:
Design, develop, and maintain effective and scalable database systems across different platforms of relational databases (such as Oracle, MySQL, PostgreSQL, SQL Server).
Collaborate with cross-functional teams to understand data requirements and translate them into effective database solutions.
Implement and design data models and database schemas that align with business needs, ensuring data integrity and efficient data retrieval.
Develop and optimize database queries, stored procedures, and functions
📌 Senior Data Engineer (Pune)
🏢 Crisil
📍 Pune