Role Summary

We are seeking a highly experienced Senior Data Engineer with strong expertise in Teradata Administration, Databricks, and AI/ML to support a Teradata Utilization Analysis engagement. The ideal candidate will analyze Teradata platform utilization using DBQL logs, metadata, and workload statistics to identify cost optimization opportunities and deliver scalable, log-driven analytical solutions using the Databricks platform. Candidates with Databricks Machine Learning or Databricks Generative AI Associate certification are highly preferred.

Key Responsibilities Teradata Utilization Analysis

- Ingest, validate, and analyze 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and execution steps.
- Analyze Teradata system metadata and workload statistics to identify unused datasets, inactive partitions, and read-only data.
- Capture CPU, IO, and workload utilization metrics using Teradata ResUsage.
- Develop recommendations for dataset archival, storage optimization, and platform cost reduction.
- Classify datasets into hot, warm, and cold storage tiers based on usage patterns.

Databricks Engineering

- Design and develop scalable data pipelines using Databricks and Apache Spark.
- Build reusable notebooks and workflows for log-driven analytics.
- Develop Delta Lake-based data pipelines for reliable and performant processing.
- Utilize Databricks SQL and Power BI to build dashboards, heatmaps, and analytical reports.
- Implement data governance using Unity Catalog.

AI / Machine Learning

- Build ML models to detect workload anomalies and predict cold-data candidates.
- Apply clustering and classification algorithms for dataset categorization.
- Develop feature engineering pipelines using time-series log data.




- Integrate LLMs for SQL log interpretation and automated recommendation generation.
- Track experiments using MLflow for reproducibility.

Data Integration

- Integrate metadata from Autosys, DataStage, MagicWand, and other enterprise systems.
- Assess ETL pipelines and recommend decommissioning of unused workflows.
- Support enterprise data modernization and cloud migration initiatives.

Reporting Stakeholder Management

- Prepare Observation Reports, recommendation documents, workshop notes, and executive presentations.
- Present technical findings and cost optimization recommendations to customer stakeholders.
- Collaborate with architects, DBAs, data engineers, and business teams throughout the engagement.

Required Qualifications

- Bachelors or Masters degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
- 10+ years of relevant experience in Data Engineering, Teradata Administration, and Analytics.
- Robust experience in enterprise-scale data platforms and cloud-native data engineering.
- Experience delivering advisory or consulting engagements is preferred.

Required Technical Skills Teradata

- Teradata Administration
- Teradata DBQL
- Teradata System Views
- Space Metadata Analysis
- Teradata SQL
- Performance Tuning
- ResUsage
- Workload Statistics
- BTEQ
- FastExport
- Teradata Parallel Transporter (TPT)
- Data Classification
- ETL Assessment
- Autosys
- DataStage

Databricks

- Databricks Workspaces




- Databricks Jobs
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake
- Databricks Notebooks
- Databricks Workflows
- Unity Catalog
- Databricks SQL

Programming Analytics

- Python
- Pandas
- NumPy
- SQL
- Plotly
- Matplotlib
- Power BI

AI / Machine Learning

- Machine Learning
- Scikit-learn
- MLflow
- Generative AI
- Large Language Models (LLMs)
- Feature Engineering
- Time-Series Analytics
- Clustering Algorithms
- Classification Algorithms
- Natural Language Processing (NLP)

Preferred Skills

- Experience with DataStage orchestration log parsing.
- Experience in cloud migration readiness assessments.
- Data platform cost optimization expertise.
- Healthcare payer domain experience with HIPAA awareness.
- Strong technical documentation and advisory reporting skills.
- Experience creating executive-level presentations and customer-facing deliverables.

Preferred Certifications

- Databricks Machine Learning Associate (Highly Preferred)
- Databricks Generative AI Associate (Highly Preferred)
- Databricks Data Engineer Professional
- Azure Databricks Certification
- AWS Certified Data Engineer
- Microsoft Azure Data Engineer Associate
- Teradata Certification

Key Competencies

- Data Engineering
- Teradata Administration
- Databricks Development
- Machine Learning
- Generative AI
- Data Platform Optimization
- Performance Tuning
- Data Analytics
- Solution Architecture
- Problem Solving
- Stakeholder Management
- Technical Consulting
- Executive Communication

Disclaimer : This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Senior Data Engineer Teradata & Databricks AI/ML (Karnataka)
🏢 Stack Digital
📍 Karnataka