Role & responsibilities

- Lead the design, development, and implementation of scalable enterprise Data Platform solutions supporting analytics, AI/ML, and business intelligence initiatives.
- Architect and develop robust workflow orchestration pipelines using Apache Airflow for scheduling, monitoring, dependency management, and workflow automation.
- Design, build, and optimize distributed data processing applications using Apache Spark for large-scale batch and real-time data processing.
- Develop scalable ETL/ELT frameworks for ingesting structured, semi-structured, and unstructured data from multiple enterprise data sources.
- Design and implement modern Data Lake, Data Warehouse, and Lakehouse architectures using technologies such as Apache Iceberg, Delta Lake, or Apache Hudi.
- Build high-performance analytical data models and OLAP solutions using Apache Kylin to enable low-latency multidimensional analytics.
- Develop and optimize real-time analytics platforms using Apache Druid for interactive dashboards, time-series analytics, and high-speed query performance.
- Integrate streaming data pipelines using Apache Kafka or similar messaging platforms for real-time data ingestion and processing.
- Design scalable data models that support reporting, business intelligence, advanced analytics, and machine learning workloads.
- Optimize Spark applications through partitioning, caching, memory tuning, resource allocation, and query optimization to maximize performance.
- Configure, monitor, and optimize Apache Airflow environments, including DAG development, scheduling strategies, retries, alerting, logging, and failure recovery mechanisms.
- Develop reusable data engineering frameworks, metadata-driven pipelines, and common libraries to improve development efficiency.
- Implement data governance, metadata management, lineage, quality validation, and monitoring across the enterprise data platform.
- Develop APIs and data services to enable secure and efficient data consumption across applications and business platforms.
- Integrate data from databases, cloud storage, APIs, IoT platforms, enterprise applications,



and third-party systems into centralized data platforms.
- Design and optimize SQL queries, distributed processing workflows, and data storage strategies for maximum scalability and performance.
- Deploy and manage data platform components using Docker and Kubernetes, ensuring scalability, reliability, and high availability.
- Work with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform to design, deploy, and manage cloud-native data engineering solutions.
- Implement CI/CD pipelines using Git, Jenkins, GitHub Actions, or GitLab CI to automate application deployment and infrastructure changes.
- Collaborate closely with Data Scientists, BI Developers, Product Managers, DevOps Engineers, and Business stakeholders to deliver enterprise data solutions.
- Lead architecture discussions, establish coding standards, conduct code reviews, and mentor Data Engineers on best engineering practices.
- Monitor platform performance, troubleshoot production issues, perform root cause analysis, and drive continuous improvements in platform reliability and efficiency.
- Ensure platform security, access controls, compliance, backup strategies, and disaster recovery planning.
- Prepare technical documentation, architecture diagrams, deployment guides, and operational runbooks.
- Evaluate emerging technologies and recommend enhancements to modernize the organization's data platform ecosystem.
- Drive technical strategy, innovation, and platform roadmap initiatives while ensuring alignment with organizational business objectives.

Preferred candidate profile

- 58 years of hands-on experience in Data Engineering, Big Data, or Data Platform development.
- Strong expertise in Apache Airflow for workflow orchestration, scheduling, monitoring, and pipeline automation.




- Hands-on experience with Apache Spark (Spark Core, Spark SQL, DataFrames, and performance tuning) for large-scale distributed data processing.
- Proficiency in Python, SQL, and Shell scripting for developing scalable data pipelines and automation.
- Strong understanding of ETL/ELT, Data Warehousing, Data Lakes, and modern Lakehouse architectures.
- Experience designing and implementing scalable, high-performance data platforms capable of handling large volumes of structured and unstructured data.
- Hands-on experience with Apache Kylin for OLAP analytics and multidimensional data modeling is highly preferred.
- Experience with Apache Druid for real-time analytics, time-series data processing, and low-latency query performance is an added advantage.
- Knowledge of Apache Kafka or similar streaming technologies for real-time data ingestion and processing.
- Familiarity with distributed storage technologies such as Apache Iceberg, Delta Lake, or Apache Hudi.
- Experience working with relational and NoSQL databases such as PostgreSQL, MySQL, MongoDB, Cassandra, or similar databases.
- Exposure to cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform.
- Experience with Docker, Kubernetes, Git, and CI/CD tools such as Jenkins or GitHub Actions is preferred.
- Robust understanding of data governance, data quality, metadata management, and security best practices.
- Proven ability to optimize Spark workloads, SQL queries, and large-scale data processing pipelines for performance and scalability.
- Experience in mentoring junior engineers, participating in architecture discussions, conducting code reviews, and contributing to technical decision-making.
- Strong analytical, troubleshooting, and problem-solving skills with the ability to resolve complex production issues.
- Excellent communication, stakeholder management, and collaboration skills, with the ability to work effectively in cross-functional Agile teams.
- Self-motivated, proactive, and passionate about building scalable enterprise data platforms and adopting modern data engineering technologies.

📌 Data Platform Lead (Karnataka)
🏢 Trinity Mobility
📍 Karnataka