Job Summary:
Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership. You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.
Key Responsibilities
- Lead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more.
- Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO).
- Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka.
- Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet/Avro file formats for efficient querying and cost management.
- Build and maintain high-quality ETL pipelines,



ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance.
- Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices.
- Mentor junior engineers (SDE-I) and foster a collaborative, growth-oriented environment.
- Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency.
- Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, Current Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health.
- Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase.
- Stay curr

📌 Data Engineer II (Chennai)
🏢 Clear Demand
📍 Chennai