Bengaluru, India

IIT PatnaM.Tech · AI & Data Science Engineering · Ongoing

Data & Applied AI Engineer

Data Engineering · BI & Analytics · Applied AI

At HP, I modernise Databricks pipelines and build ontology-backed knowledge graphs and GraphRAG retrieval for pipeline troubleshooting. At AMAZON, I built BI datasets and automated supply-chain reporting with SQL and Python.

Explore my work View project

HP · Data and AI Platform Engineering

Reliable data. Useful analysis.

Selected HP contributions across pipeline modernisation, knowledge graphs and GraphRAG, protected data workflows, migration and platform reliability.

01Pipeline modernisation

Legacy Databricks workflows, modernised

AI-assisted refactoring and a contract-driven ODPS framework, with YAML data contracts, shared-runtime Python utilities, source-backed review and output-parity checks.

50+ legacy pipelinesNine reusable Copilot skillsDeveloper approval gates

02PII protection and enrichment

Protected data and useful case-note context

Databricks workflows for PII detection, masking and OpenPGP encryption using Presidio, spaCy and domain rules, alongside LLM-generated summaries for operational review and analytics.

PII maskingOpenPGP encryptionLLM-generated summaries

03Data-aware engineering

Knowledge graphs and GraphRAG for pipeline troubleshooting

Designed and deployed an OntoBricks knowledge graph on Databricks using Unity Catalog data and RDF/OWL and R2RML mappings. Built hybrid GraphRAG combining vector search and graph traversal, with read-only retrieval in GitHub Copilot via MCP.

Hybrid GraphRAGRead-only Copilot retrievalApproximately $250K annualized team savings contribution

04Pipeline migration

Reusable migration and refactoring tools

Migrated and deployed enterprise production pipelines with a reusable logging and refactoring framework.

40+ production pipelinesApproximately 90% less manual migration effortApproximately 25-30% less runtime per migrated job

05Processing optimisation

Efficient PySpark transformations

Optimised partitioning, joins and caching, and implemented Spark deduplication to reduce redundant processing.

Approximately 75% less runtime21 billion → 400 million records

06Pipeline monitoring

Routine checks with execution evidence

Automated monitoring and recurring operational checks through reusable execution logging, validation and incident-diagnosis utilities.

54 pipelines85-90% fewer manual checks

07Lakehouse observability

Pipeline health, SLAs and incident diagnosis

Developed observability for lakehouse operations, contributing to annual infrastructure savings.

Pipeline health and SLA trackingApproximately $1.55M annual savings contribution

AI development tools: GitHub Copilot · Claude · Codex ↗

Professional experience

Data platforms and operational context.

My work spans enterprise data pipelines, business analysis and centre operations, with an emphasis on reliable inputs and useful operational outputs.

HP Inc.

Jun 2024 - Present

Data Scientist II | Data and AI Platform Engineering

Databricks pipeline modernisation, ontology-backed knowledge graphs and hybrid GraphRAG for pipeline troubleshooting, alongside PySpark optimisation and platform reliability.

AMAZON

Oct 2022 - Jun 2024

Business Analyst | SSD/AMXL - India Supply Chain Analytics

Built SQL pipelines and BI datasets for supply-chain reporting. Automated reporting with SQL and Python, reducing manual effort by approximately 70%, and improved CPT query runtime by approximately 30%. Received Minesweeper and Automation Expert awards.

BYJU’S

Oct 2021 - Oct 2022

Centre Head - Operations

Led a 16-member cross-functional team and built SQL-driven Student Performance Index dashboards combining attendance, assessment and tutor metrics.

Independent project

Data Platform Reliability Agent

An AI-assisted independent build using synthetic data. The case study describes read-only decision support, source code and validation limits.

Synthetic dataAI-assisted build

Data Platform Reliability Agent

A tool-using reliability agent with an optional OpenAI planner that investigates synthetic pipeline incidents and proposes, but never executes, remediation.

Decision supportEvidence-grounded incident diagnosis with citations, tool trace and an approval gate.

Read case study →Source on GitHub ↗

Professional contact

Let's connect around data.

Data engineering, analytics and practical Applied AI.

Email Shivanand