At Lilly we unite caring with discovery to make life better for people around the world We are a global healthcare leader headquartered in Indianapolis Indiana Our employees around the world work to discover and bring life-changing medicines to those who need them improve the understanding and management of disease and give back to our communities through philanthropy and volunteerism We give our best effort to our work and we put people first We re looking for people who are determined to make life better for people around the world As a Senior Data Engineer you will demonstrate expert skills in ETL ELT data integration ML Ops and SQL as well as intermediate to advanced skills in Python Pyspark AI ML and data visualization demonstrate the ability to review optimize document and mentor data visualization engineers on data pipelines mapping cleansing and visual design using various tools and platforms possess ability to break down moderately complex problems to implement for increased business impact support other team members and helps them to be successful Actively shares learnings with team members drive and enforce the team process improvements ensuring others are brought along in understanding the benefits and tradeoffs actively promote new and creative ideas across multiple teams and capabilities Key Responsibilities Hands-On Development 75 Build and maintain scalable data platforms and infrastructure on AWS Implement end-to-end data pipelines for batch and real-time data processing Build robust ETL ELT workflows to ingest transform and load data from diverse sources Implement data lake Lakehouse architectures using AWS S3 Glue Athena and Lake Formation Design and optimize data warehouse solutions Redshift Snowflake for analytics and reporting Establish data quality frameworks and automated monitoring systems Write production-quality Python code for data processing transformation and automation Build scalable data pipelines using Apache Airflow AWS Step Functions or similar orchestration tool Develop streaming data solutions using Kinesis Kafka or AWS MSK Optimize SQL queries and database performance for large-scale datasets Implement data validation cleansing and quality checks Build APIs and microservices for data access and integration Create monitoring alerting and observability solutions for data pipelines Debug and resolve data pipeline failures and performance bottlenecks Technical Leadership Collaboration 25 Mentor junior and mid-level data engineers through code reviews and technical guidance Establish best practices for data engineering testing and deployment Collaborate with data scientists analysts and business stakeholders to understand data requirements Work with ML engineers to build data pipelines supporting machine learning workflows Partner with platform infrastructure teams on cloud architecture and cost optimization Lead technical design discussions and architectural reviews Document data architectures pipelines and processes Evangelize data engineering best practices across the organization Required Qualifications Technical Expertise 10 years of professional experience in data engineering or related roles Expert-level proficiency in Python for data engineering Data processing libraries Pandas PySpark Dask Polars API development FastAPI Flask Testing Pytest unittest Strong AWS expertise with hands-on experience in Data Storage S3 RDS Aurora DynamoDB Redshift Data Processing Glue ETL jobs crawlers Data Catalog EMR Athena Streaming Kinesis Data Streams Firehose Analytics MSK Managed Kafka Orchestration Step Functions EventBridge Lambda Analytics QuickSight Athena Redshift Spectrum Data Lake Lake Formation Glue Data Catalog Infrastructure CloudFormation CDK IAM VPC CloudWatch Workflow Orchestration Apache Airflow strong preference Big Data Technologies Apache Spark PySpark for distributed data processing Experience with EMR Databricks or similar platforms Understanding of distributed computing concepts Parquet Avro ORC file formats Architecture Design Solid understanding and implementation knowledge of data modelling dimensional modelling star snowflake schemas Experience with both batch and streaming data processing patterns Knowledge of data lake data warehouse and lake-house architectures Understanding of data partitioning bucketing and optimization strategies Expertise in designing for data quality lineage and governance DevOps Best Practices Strong experience with CI CD pipelines for data engineering GitHub Actions GitLab CI Jenkins Infrastructure as Code using Terraform CloudFormation or AWS CDK Containerization with Docker experience with ECS Fargate Kubernetes is a plus Git version control and branching strategies Monitoring and observability tools CloudWatch Grafana Data pipeline testing strategies and frameworks Preferred Qualifications Bachelor s or Master s degree in Computer Science Engineering Data Science or related field or equivalent experience Experience in regulated industries healthcare pharma finance government with compliance requirements Hands-on experience with Additional AWS services Glue DataBrew AppFlow Data Pipeline Lambda SageMaker Streaming platforms Apache Kafka Confluent AWS MSK Data quality tools Great Expectations dbt Monte Carlo Bigeye Data cataloging AWS Glue Data Catalog Alation Collibra Alternative clouds GCP BigQuery Dataflow Azure Synapse Data Factory Data orchestration dbt for transformation workflows Experience with clinical data life sciences or statistical computing domains CDISC standards clinical trials data Knowledge of data mesh or data fabric architectures Experience building data platforms for ML AI workloads Familiarity with data governance and metadata management frameworks Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce ensuring equal opportunities when vying for positions If you require accommodation to submit a resume for a position at Lilly please complete the accommodation request form https careers lilly com us en workplace-accommodation for further assistance Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response Lilly does not discriminate on the basis of age race color religion gender sexual orientation gender identity gender expression national origin protected veteran status disability or any other legally protected status WeAreLilly

📌 Sr. Data Engineer/tech Lead (Karnataka)
🏢 Eli Lilly
📍 Karnataka