Prompting Is Not Enough: The Case for Unstructured AI Insights

If you are only using AI to generate prompt-based analyses you are missing out on valuable insights.  

I say this for two reasons.

Two Issues with Prompt-Only Analysis

First, what if you don’t ask the right questions?  Good AI analysis tools are trained not to hallucinate.  This means that, by design, the objective of a tool is to respond to prompts directly to ensure accuracy.  

This is a critical first step, but it tends to lead to analyses that are literal, meaning they are accurate, but in a “just the facts” way.  This doesn’t capture the full scope of effective analysis because it is missing something crucial- the unexpected insights that come from unstructured analysis (more on what I mean by this below).  

The second reason why prompt-only analysis is not sufficient is that a key promise of AI is to make efficient the ability to handle large data sets that may have previously been cost or time-prohibitive to analyze.  It’s hard to argue that, historically, important insights have been left on the table because it was difficult or impractical to make the connections between disparate data points to uncover them.  AI can solve for this from the “data crunching” perspective, but at that point we are back to the first issue- what if the user doesn’t ask the right questions to surface these insights?

Something is missing and I suggest that it is unstructured analysis.

Unstructured Analysis

Let’s define the term “unstructured analysis.”  By this I mean analytical output produced autonomously, not in response to a specific prompt.  As researchers, we use both structured and unstructured questions and data collection methods because we know that they result in different types of data.  

This distinction is also present when analyzing data.  Structured analysis is the metaphorical equivalent to traditional quantitative methodologies- questions with specific response options provided, analyses based on predefined comparison groups, etc.  

In contrast, unstructured analysis is the spontaneous production of unexpected and potentially valuable insights outside of a formal analysis framework.  This is akin to a “bolt from the blue” or a “Eureka!” moment you might have while taking a shower or making dinner.  These insights can reveal hidden opportunities by providing a deeper layer of analysis to enhance your overall understanding of the available data.  

Example- Media Monitoring

So what does this look like?

I am running an AI-enabled media monitor to gather and analyze news related to the 2024-25 NHL season that started recently.  The monitor works like a Google Alert, with two crucial differences: 

• It not only finds articles and mentions of the target topic, it also analyzes them

• It does not hallucinate, meaning its knowledge base is limited to what I have put into it- in this case media reports and social media commentary about the new season

  
  

After letting it run for a few days, I asked it the following question: What are the key insights from the 2024-25 season so far?

In response, I got what I consider to be accurate and reasonable responses, which focused on the prospects of particular teams and players, such as:

• Continued Excellence and Strong Offseasons- The likely continued success of the Florida Panthers and Edmonton Oilers

• Emerging Teams- Expectations related to teams in transition like the Vancouver Canucks, Ottawa Senators, Dallas Stars, New Jersey Devils and the newly constituted (and yet-to-be-named) Utah Hockey Club

• Emerging Players- The development of potential future stars like Emil Hemming and rising stars like Cole Caufield

  
  

In my view, the findings accurately reflect the data that has been provided to the app for analysis and the themes noted above broadly reflect key themes present in media coverage of the new season to date.  However, I’m also able to generate autonomous, unstructured insights from the data.  Here are a few examples:

• The NHL’s Global Series, which led the league to kick off this season in Czechia points to a desire by the league to embrace and capitalize on the global profile of the sport

• The introduction of new rules around warm up periods for backup goaltenders and the banning of players sitting on the boards with their skates exposed, as well as new approaches to roster and injury management demonstrate an increase in focus on player safety and health

• The focus on individual players’ stories and development journeys reflects a focus on narratives related to breakout stars, underdogs and redemption stories

  
  

What is interesting here is that, based on my review of the data, both sets of insights are “correct” and reflect the underlying data.  However, the outputs are different.  

The “structured” analyses are more literal in that they reflect specific teams and players mentioned in the data set that was analyzed.  The “unstructured” analyses on the other hand reflect broader themes developed from the spontaneous connection of data points. 

In my view, you need both to fully understand the available information, whether you are a researcher, marketer, executive at the brand or a competitor. 

How to Get Started

Reach out to me today to discuss how I can help you deepen your understanding of what matters to you- your brand, your competitors, your industry, or just a topic of interest. 

Let's Talk