Will AI take over the world and create havoc for the human race? It might, but before HAL shows up, the mess caused will be because of the flawed systems humans built. While cloud providers and consulting companies are rushing to sell AI, it’s worth asking the question: does it actually work and can you trust it? It’s easy to get seduced by seeing AI generate text or computer code to your prompt, but try something more complex, like using AI for content and data analysis, and a very different picture emerges. This article is about our experience and we hope it’s helpful as you’re thinking about how to use AI, perhaps in your content marketing or elsewhere in your company. 

We started out by simply wanting to analyze the 100 best books of the century recently published by the NYT. We were interested in analyzing some simple statistics, like gender distribution and the number of books in translation. We also were curious to see if any clear themes emerged from the books that were selected by “503 novelists, nonfiction writers, poets, critics” and others invited by the NYT. We wondered whether these 100 books could help us understand something about our times, our public discourse, and our future. We didn’t get very far because the AI tool we used (ChatGPT v4o) never got it right. 

We started off by asking some straightforward questions to ensure the AI was reading our uploaded file correctly. The first response was not encouraging as it provided two different sets of numbers for gender distribution. In the first instance, AI told us there were 46 books written by women and 54 by men. Then, later in the same response, the answer changed to 40 books by women and 32 by men (screenshot below). 

Next, we wanted to know how many of the 100 were fiction and nonfiction. The response we go was 81 fiction books and 19 nonfiction books (screenshot below):

We presumed the information was correct and proceeded to ask more complex questions, like categorizing the books by theme (screenshot below):

The list we got back included titles that were not among the 100 books we specified in our original file. After several more attempts with similarly erroneously answers, we tried to reset the prompt (screenshot below):

The AI tool provided a very long response recapitulating its process and all of the books. We’ll pick-up on the response we got after several tries. The total count we got back was 60 fiction books and 22 books. What happened to the rest? 

We inquired. Here’s the (partial) response we got. While the AI tool says there are 81 fiction books, it only provided 58 titles (screenshot below).

So we asked again why only 58 titles were provided when AI said there were 81. Here’s the partial response, we’ve truncated the bottom where non-fiction titles were listed (screenshot below).

We tried many more times. Finally, success . . . sort of. The number of fiction books changed to 76 but at least we got 76 fiction titles . . . (screenshot below):

. . . and 24 nonfiction titles. So we provided a little praise (screenshot below):

We got a little courage and, though we had already spent a lot of time (or rather because of it), we decided to proceed with asking more interesting questions. We asked the AI tool to categorize just the fiction titles (screenshot below).

It didn’t quite feel like 76 titles, so we asked for enumeration. It took many tries, but finally we thought we had it (partial list shown in screenshot below):

Next, we wanted the tool to create a chart for us, but even after several tries, we kept getting the same response: issue with generating the chart (screenshot below).

So then we asked it to export the data so we could create a chart ourselves. But it kept getting the count wrong, so we reentered the list of 76 (partial shown in screenshot below):

In the output we noticed something strange, the inclusion of an author who had been dead for a long time and certainly isn’t publishing posthumously: Jean-Paul Sartre (screenshot below):

Our Take Away on the State of AI

Our experience with AI (specifically ChatGPT) being unable to correctly analyze data, “hallucinating” by erroneously inserting data not present in the original dataset, and failing to consistently generate charts makes AI an unreliable tool. For tasks beyond rewriting simple text, such as emails, or basic code for a website, it’s hard to see how the current iteration of AI could assist with more complex tasks in the near future. In other disciplines where AI is touted as a solution for data analysis, such as healthcare, accounting, or travel, it seems highly improbable that AI could be trusted anytime soon.

While we will continue to review the state of AI for its potential in content creation, the results we are seeing indicate that anything requiring research, understanding of data, and factual accuracy still demands significant human involvement at every step in the process. This is why our clients rely on us instead of defaulting to AI-generated content.

About the Author
Ren Agarwal is the CEO of StoryAZ Studio. With a passion for helping clients develop and implement successful marketing strategies, Ren brings a wealth of experience and insight into the ever-changing world of marketing and communications.