Designing Data Integrity for an Internal LLM Platform

Context

Ask AT&T is an internal, LLM-powered tool designed to help business teams analyze data, surface insights, and answer questions across large and complex datasets.

A common workflow involved business units exporting analytics data (primarily CSV files from Adobe Analytics) and uploading them into Ask AT&T for summarization and insight generation.

AI Insights Pipeline — Data Integrity Failure Point (Public view)

Click to enlarge

Tip: click/tap to zoom. ESC closes.

My Role & Scope

I supported Ask AT&T as a contributing UX designer with a focused scope on how business-unit data was funneled into the AI system.

What I owned

• Evaluating the data ingestion experience from a UX and systems perspective
• Identifying risks in how analytics files were parsed and interpreted
• Partnering with engineering and product to surface upstream failures affecting AI output quality

What I did not own

• Overall product strategy
• Backend architecture or model selection
• End-to-end feature ownership

This was a narrow but high-leverage intervention, focused on protecting system trust rather than expanding surface-level functionality.

The Problem: Plausible but Incorrect AI Insights

Ask AT&T relied on user-uploaded analytics data to generate AI-assisted insights. In several workflows, business teams uploaded CSV exports from Adobe Analytics that appeared to process successfully.

While AI outputs were fluent and plausible, I identified inconsistencies between source data characteristics and generated insights. The system treated malformed data as valid input, producing confident responses without signaling underlying data integrity issues.

This created a high-risk failure mode: incorrect insights without visible error states.

Discovery: CSV Conversion and Ingestion Failure

Tracing the issue upstream revealed that certain Adobe Analytics CSV exports were being misinterpreted during ingestion. Column structures, data types, and contextual metadata were altered or flattened during upload.

Because ingestion completed without error, the AI had no signal that the data was incomplete or corrupted. From the system’s perspective, the input was valid — even though the meaning was not.

This distinction became critical: the model was functioning correctly given incorrect inputs.

AI Insights Pipeline — Failure Mode Detail (Gated)

Click to enlarge

Tip: click/tap to zoom. ESC closes.

Why This Mattered (Systems-Level UX)

This issue was not detectable through interface review or conventional usability testing. The failure occurred before prompts, UI, or interaction design — at the data boundary.

Because outputs appeared reasonable, teams had no reason to question them. At scale, this type of silent failure introduces compounding decision risk.

Addressing this required systems-level UX thinking rather than surface-level design changes.

My Intervention

I escalated the issue cross-functionally and worked with product and engineering to:

• Clearly identify the ingestion step as the root cause
• Demonstrate how malformed analytics inputs could produce misleading AI outputs
• Advocate for data validation and normalization before AI analysis
• Push for UX-level signals that communicate ingestion success or failure meaningfully

Rather than designing new UI surfaces, the focus was on system correctness and trust.

Solution Direction (High-Level)

While implementation details evolved beyond my direct ownership, the design direction centered on:

• Validation checks during file upload and parsing
• Guardrails preventing AI analysis on malformed datasets
• Clear user feedback when data could not be reliably interpreted
• Internal guidance on proper analytics exports for AI consumption

The goal was not perfection — it was preventing silent failure.

Outcome

• Surfaced ingestion as the root cause of unreliable AI insights (not prompt quality or “model intelligence”)
• Improved organizational awareness that data conversion is a UX problem, not just a technical one
• Helped shift focus upstream toward validation, guardrails, and meaningful failure signals
• Increased confidence in AI-assisted analysis by reducing risk of silent corruption

The impact was largely preventative, measured in avoided risk rather than visible features.

Reflection

This work reinforced that AI UX does not begin at the interface. Trustworthy systems require designers to engage with how data is sourced, transformed, and validated — not just how results are presented.

At platform scale, the most impactful design decisions are often preventative: identifying where systems can fail silently and designing guardrails before those failures reach users.