Job Requirements*

- Experience in data engineering/pipeline development
- Solid hands-on experience with Apache NiFi (flow design, custom processors, registry)
- Solid understanding of Apache Kafka (topics, partitions, consumer groups, throughput optimization)
- Experience with Cloudera Data Platform (CDP) ecosystem
- Working knowledge of Kubernetes and container orchestration
- Experience with OpenStack or similar cloud infrastructure
- Familiarity with monitoring tools (Grafana, Prometheus)
- Understanding of networking concepts (DNS, load balancing, firewalls)

Key Responsibilities*

- Design, develop, and maintain Apache NiFi data pipelines for real-time and batch data ingestion across multiple sources and sinks
- Integrate NiFi pipelines with Kafka (Logstash Kafka, Kafka Kafka) and Cloudera Data Platform (CDP) components
- Architect and deploy NiFi clusters on OpenStack-based Container-as-a-Service (CaaS) environments




- Configure NiFi clustering and High Availability to ensure fault tolerance and zero downtime
- Perform performance tuning and load optimization of NiFi processors and Kafka topics (partitions, throughput, consumer groups)
- Design and implement monitoring and alerting solutions using NiFi reporting tasks, Grafana, and Prometheus
- Create and maintain deployment, networking, scaling, and configuration documentation
- Collaborate with platform teams on Kubernetes/container orchestration for pipeline deployments
- Implement security best practices including TLS encryption, RBAC, and Kerberos authentication
- Develop and maintain CI/CD pipelines for automated NiFi flow deployment and version control
- Troubleshoot production issues, perform root cause analysis, and implement preventive measures
- Participate in architecture reviews and provide recommendations for pipeline scalability and resilience

📌 Apache NIFI (Hyderabad)
🏢 Tata Consultancy Services
📍 Hyderabad