Focused Impact Case Study (Confidential)

Designing Data Integrity for an Internal LLM Platform

Preventing false AI insights by identifying and correcting upstream analytics ingestion failures

TL;DR

Role: Contributing UX Designer — AI & Data Systems
Scope: Analytics ingestion, data validation risk, AI input integrity
Problem: Corrupted analytics data produced confident but incorrect LLM outputs
Impact: Prevented silent failure by identifying and correcting upstream data ingestion issues
Outcome: Increased trust, reliability, and decision safety in an internal AI platform

Details are intentionally abstracted due to confidentiality.

Context

Ask AT&T is an internal, LLM-powered tool designed to help business teams analyze data, surface insights, and answer questions across large and complex datasets.

A common workflow involved business units exporting analytics data (primarily CSV files from Adobe Analytics) and uploading them into Ask AT&T for summarization and insight generation.

At a glance, the system appeared to be functioning correctly:

• Files uploaded successfully

• AI responses were fluent and plausible

• No explicit errors were surfaced to users

However, the reliability of these insights depended entirely on the integrity of the data entering the system.

My Role & Scope

I supported Ask AT&T as a contributing UX designer with a focused scope on how business-unit data was funneled into the AI system.

What I owned

• Evaluating the data ingestion experience from a UX and systems perspective

• Identifying risks in how analytics files were parsed and interpreted

• Partnering with engineering and product to surface upstream failures affecting AI output quality

What I did not own

• Overall product strategy

• Backend architecture or model selection

• End-to-end feature ownership

This was a narrow but high-leverage intervention, focused on protecting system trust rather than expanding surface-level functionality.

The Problem: Plausible but Incorrect AI Insights

Ask AT&T relied on user-uploaded analytics data to generate AI-assisted insights. In several workflows, business teams uploaded CSV exports from Adobe Analytics that appeared to process successfully.

While AI outputs were fluent and plausible, I identified inconsistencies between source data characteristics and generated insights. The system treated malformed data as valid input, producing confident responses without signaling underlying data integrity issues.

This created a high-risk failure mode: incorrect insights without visible error states.

Discovery: CSV Conversion and Ingestion Failure

Tracing the issue upstream revealed that certain Adobe Analytics CSV exports were being misinterpreted during ingestion. Column structures, data types, and contextual metadata were altered or flattened during upload.

Because ingestion completed without error, the AI had no signal that the data was incomplete or corrupted. From the system’s perspective, the input was valid — even though the meaning was not.

This distinction became critical: the model was functioning correctly given incorrect inputs.

Why This Mattered (Principal-Level Risk)

This issue was not detectable through interface review or conventional usability testing. The failure occurred before prompts, UI, or interaction design — at the data boundary.

Because outputs appeared reasonable, teams had no reason to question them. At scale, this type of silent failure introduces compounding decision risk.

Addressing this required systems-level UX thinking rather than surface-level design changes.

My Intervention

I escalated the issue cross-functionally and worked with product and engineering to:

• Clearly identify the ingestion step as the root cause

• Demonstrate how corrupted analytics inputs led to misleading AI outputs

• Advocate for data validation and normalization before AI analysis

• Push for UX-level signals that communicate ingestion success or failure meaningfully

Rather than designing new UI surfaces, the focus was on system correctness and trust.

Solution Direction (High-Level)

While implementation details evolved beyond my direct ownership, the design direction centered on:

• Validation checks during file upload and parsing

• Guardrails preventing AI analysis on malformed datasets

• Clear user feedback when data could not be reliably interpreted

• Internal guidance on proper analytics exports for AI consumption

The goal was not perfection — it was preventing silent failure.

Impact & Outcome

• Prevented corrupted analytics data from generating false AI insights

• Increased internal confidence in Ask AT&T’s analytical outputs

• Surfaced data ingestion as a critical UX concern in AI systems

• Helped shift focus upstream, from interface polish to system trust

The impact of this work was largely preventative, measured in avoided risk rather than visible features.

Reflection

This work reinforced that AI UX does not begin at the interface. Trustworthy systems require designers to engage with how data is sourced, transformed, and validated — not just how results are presented.

At platform scale, the most impactful design decisions are often preventative: identifying where systems can fail silently and designing guardrails before those failures reach users.