Roles and Responsibilities:

- Plan, design, and implement scalable data solutions using GCP and Databricks technologies.

- Design, develop, and maintain up-to-date data pipelines and real-time data streaming solutions.

- Manage and optimise ETL processes to ensure efficient data processing, transformation, and integration.

- Develop and maintain data marts to support defined business and data-driven use cases.

- Pre-process, clean, and transform structured and unstructured data, while creating and optimising queries as required.

- Establish and implement best practices, frameworks, and standards for data testing, validation, and quality assurance.

- Contribute to the development of data-driven products, including user journey analysis and editorial intelligence tools.

- Collaborate closely with cross-functional technology teams to align data initiatives with business and technical objectives.

- Perform root cause analysis on internal and external data issues to solve business problems and identify improvement opportunities.

- Build and maintain processes supporting data transformation, metadata management, dependency handling, and workload optimisation.

- Analyse, manipulate, and extract actionable insights from large and complex datasets across multiple disconnected sources.

Key Requirements:

- 5+ years of hands-on experience as a Data Engineer, designing and building scalable data pipelines and cloud-based data processing solutions.

- Strong hands-on expertise in Python and SQL, including data transformation, performance optimisation, data modelling, workflow automation and large-scale data processing.

- Experience working with Google Cloud Platform (GCP) services such as BigQuery, Cloud Storage and Cloud Functions, or equivalent cloud-native data platforms, is mandatory.

- Hands-on experience with Databricks, including notebooks, clusters, Spark/PySpark development, data transformation, and pipeline orchestration.





- Strong understanding of distributed data processing frameworks, particularly PySpark/Spark DataFrames, for large-scale data transformation and analytics workloads.

- Experience working with relational databases such as PostgreSQL, SQL Server, Oracle or similar platforms, including complex SQL development, query optimisation and data modelling.

- Proven experience developing and maintaining ETL/ELT pipelines, including ingestion from APIs, databases, cloud storage platforms and third-party data sources.

- Understanding of modern data architecture patterns such as Medallion Architecture (Bronze/Silver/Gold), Data Lake, Data Warehouse, and Lakehouse architectures.

- Experience working with version control systems and CI/CD pipelines using tools such as GitHub Actions, GitLab CI/CD, Azure DevOps or equivalent platforms.

- Experience working with modern data processing and analytics platforms such as Databricks, Snowflake, Redshift, Synapse Analytics or similar technologies is desirable.

- Experience designing and implementing reusable Python libraries, utility packages, frameworks, and automation solutions for data engineering workflows.

- Familiarity with cloud-based orchestration and workflow management tools such as Apache Airflow, Dataform, Azure components - Azure Data Lake, Azure SQL DW, Azure Synapse, etc. or equivalent platforms is good to have.

- Experience integrating and processing data from diverse sources, including APIs, cloud storage, structured databases and file formats such as JSON, CSV and Parquet.

- Exposure to containerization and cloud-native deployment technologies such as Docker and Kubernetes would be an advantage.

- Any understanding of Web Analytics data, such as Adobe Analytics and Google Analytics, alongside any Ad tech like Google Ad Manager (working with programmatic and affiliate partners), is a plus, but not essential.

- Strong problem-solving skills with the ability to independently analyse business requirements, propose scalable data solutions, and collaborate effectively with analysts and stakeholders.

📌 Senior Data Engineer (Bengaluru)
🏢 Expian Technologies
📍 Bengaluru