Job Summary:
Building on the foundation of the SDE-I role, the DE- II position takes on a greater level of responsibility and leadership. You'll play a crucial role in driving the evolution and efficiency of our data collection and analytics platform, capable of handling terabyte-scale data and billions of data points.
Key Responsibilities
Lead the design, development, and optimization of large-scale data pipelines and infrastructures using technologies like Apache Airflow, Spark, Kafka, and more.
Architect and implement distributed data processing solutions to handle terabyte-scale datasets and billions of records efficiently across multi-region cloud infrastructure (AWS, GCP, DO).
Develop and maintain real-time data processing solutions for high-volume data collection operations using technologies like Spark Streaming and Kafka.
Optimize data storage strategies using technologies such as Amazon S3, HDFS, and Parquet/Avro file formats for efficient querying and cost management.
Build and maintain high-quality ETL pipelines,



ensuring robust data collection and transformation processes with a focus on scalability and fault tolerance.
Collaborate with data analysts, researchers, and cross-functional teams to define and maintain data quality metrics, implement robust data validation, and enforce security best practices.
Mentor junior engineers (SDE-I) and foster a cooperative, growth-oriented workplace.
Participate in technical discussions, contributing to architectural decisions, and proactively identifying improvements for scalability, performance, and cost-efficiency.
Ensure application performance monitoring (APM) is in place, utilizing tools like Datadog, Current Relic, or similar to proactively monitor and optimize system performance, detect bottlenecks, and ensure system health.
Implement effective data partitioning strategies and indexing for performance optimization in distributed databases such as DynamoDB, Cassandra, or HBase.
Stay curr

📌 Data Engineer Ii Chennai (India)
🏢 Clear Demand
📍 India