In the first two parts of this series, we explored the foundational aspects of using generative AI for data analysis. We discussed how to avoid the wow-effect trap by preparing data and crafting detailed prompts. We also looked at analysing survey responses and open-ended answers, emphasising the importance of a step-by-step, methodical approach.

Now, in the third part, we will dive deeper into sentiment analysis, a crucial tool for understanding public opinion, feedback, and the emotional tone of texts.

Sentiment analysis: a definition

Sentiment analysis is a technique used to identify and categorise opinions expressed in a piece of text, determining the writer’s attitude towards a particular topic. This process involves analysing the emotional tone conveyed in the text, whether it is positive, negative, or neutral.

For journalists, sentiment analysis is invaluable. It allows us to gauge public reaction to news stories, assess the sentiment behind reader comments, and analyse feedback on social media or other platforms. By understanding these sentiments, we can better tailor our content and responses to align with our audience’s views and emotions.

Remember that, while sentiment analysis is a powerful tool, we need to recognise its limitations:

context dependency: sentiments are often context-dependent, and both a human or an AI might miss the nuances without sufficient context
non-verbal cues: sentiment analysis focuses on textual data and misses non-verbal cues such as tone, facial expressions, and body language, which are crucial for fully understanding the sentiment
sarcasm and irony: sarcasm and irony are meta-textual and cultural-based. Again, both for a human and for an AI, even with examples provided, could be difficult to detect sarcasm, because these rely heavily on meta-textual and cultural nuances
bias and misinterpretation: both humans and AI models can inherit biases, from their points of view, backgrounds, or from their training data and may misinterpret sentiments, especially in diverse linguistic and cultural contexts, but also in very well known contests.

Keeping in mind these limitations will help us to limit mistakes and misinterpretations.

Different kinds of sentiment analysis

Before starting with a tool, it’s important to know also that we can work on different kinds of sentiment analysis

basic sentiment analysis focuses on identifying whether a piece of text expresses a positive, negative, or neutral sentiment. This type of analysis is straightforward but has its limitations. It often struggles with context sensitivity, sarcasm, and nuanced emotions. For instance, the phrase “I am thrilled to wait in line for hours” may be tagged as positive due to the word “thrilled,” missing the sarcastic tone entirely. It also struggles with ambiguous content like “I loved the movie but…”
aspect-based sentiment analysis goes a step further by identifying sentiments related to specific aspects or features within a text. For example, in a restaurant review, the analysis would separately evaluate sentiments about the food, service, and ambiance. This granular approach provides more detailed insights, helping to pinpoint exactly what aspects are praised or criticised
fine-grained sentiment analysis moves beyond the basic positive, negative, and neutral categories, identifying a broader range of emotions such as happiness, anger, sadness, or fear, or giving points to different parts of a comment or a sentence. This level of detail is particularly useful for understanding complex emotional responses and can provide deeper insights into the emotional landscape of the audience.
emotion detection is a subset of sentiment analysis that focuses on identifying specific emotions expressed in the text. This can be particularly useful for gauging the deeper emotional responses of an audience to certain events or news stories. For example, analysing comments on a sensitive news topic might reveal prevalent emotions like grief or outrage, which are crucial for understanding public sentiment.

Case of study with ChatGPT

Even if there are specific tools providing sentiment analysis, I still prefer to use one of the major LLMs: it’s tricky even to choose these tools because you can never know if they will be still there in the next months, if you can trust them and so on. This is valid also for major LLMs, since they are “black boxes”, but at least they are the most used.

As I mentioned, you can choose the tool that you prefer. I propose ChatGPT just because I’ve fully integrated it in my workflow. Again, it’s not the ready-to-copy-and-past prompts that matters, but the method.

Let’s take my survey about the event I organised about AIs in journalism in Italy and let’s work on the open answers.

First of all, since I had questions about the feelings about the event, I created a complex prompt with an example of clustering. I want to go deeper than the classical division into positive-negative-neutral and I propose a prompt with some examples.

Prompt: Now, let’s work on the sentiment analysis of these answers.
Examples:

“No. But thanks for including many women in the panel! It doesn’t happen often… Keep it up!🤩.” –> Very positive

“I know it’s not your fault, but a bit more agility in switching between speakers, computers, slides, etc. would be helpful, especially since we’re discussing technology. Sometimes it’s amusing how we get tangled up sending slides to the video to talk about artificial intelligence 🙂 The rest: topics, guests, schedule, etc. all perfect. Maybe a bit long for those who followed the entire day, but I personally endured it with pleasure. I take this opportunity to thank you: after years of poor and useless courses followed as a journalist to update myself, this event has restored my hope :-)” –> Very positive, but critics about technical parts

“No remote speakers. Active involvement of participants.” –> Negative

“Perfect as is.” –> Very positive

“Plan some exercises” –> Neutral with suggestions.

Cluster the comments and provide me with sentiment analysis about the event, thanks.

Since I’m very clear about the context and there are suggestions for classification, the AI is good enough to “understand” that I need a scale from Very positive to Highly negative and provides me a proper classification, with some correction needed.

Then, asking for a technical report about the analysis, you can obtain the frequency distribution.

Keeping in mind that the question was: “Do you have any suggestions to improve the event?”, we understand that the neutral with suggestions are a normal kind of comments. Also the AI “understand that”, providing these considerations:

Having the help and evaluation of a non-emotional machine is crucial to avoid the human-kind of misunderstandings.

It’s important to keep in mind, once again, that we are in charge and we, as human beings, must decide if the analysis is right or not.

While I was reading the answers and clustering about requests I showed in part two of this guide, I realised that I could analyse an unintended sentiment: the feelings pro or against AIs. This is a classic part of a healthy approach to these tools: working with them as assistants, you have time and energy free to think!

And so, I asked ChatGPT something like this:

Prompt: May you provide me with a sentiment analysis trying to understand and extract feelings from the audience about the whole AIs ecosystem? Are they scared? Happy? Enthusiasts? Are they with doubts? Write a technical report about this topic.

In this case, I’m not sure what I’m exploring and I don’t want to give suggestions: I’m always working on the same chat and my theory is that ChatGPT has got enough information to work by itself at least for a draft version of what I need.

And, in fact, this is the case, as you can see by the output (and as I’ve verified from the original file, of course).

Last question, at least on my side and to show another possible approach: I want to select only the negative answers, asking for concerns and fears. If we have done everything right, preparing and following the step-by-step method, this is pretty easy to ask for the human (e.g. “Which are the fears or concerns?”) and to answer for the AI.

Best practices

Let’s finally see the best practices that you can add to the ones in part one and part two of the guides:

Craft detailed prompts: explain the context of your data, describe each type of sentiment you are interested in, and provide sample outputs.
- Example prompt: “Cluster the comments and provide sentiment analysis about the event. Examples: ‘No. But thanks for including many women in the panel! It doesn’t happen often… Keep it up!🤩.’ –> Very positive. ‘No remote speakers. Active involvement of participants.’ –> Negative.”
Iterative refinement: start with a basic prompt and refine it based on the AI’s responses. Adjust the prompt to address any errors or misinterpretations.
Human oversight: always validate the AI’s outputs with human oversight. Review the AI’s analysis to catch any errors, biases, or misinterpretations. Human intuition and context understanding are invaluable for ensuring accuracy. Validate AI-generated insights about sentiment analysis with traditional methods where possible. This ensures the reliability and credibility of the results.
Addressing limitations: be aware that AIs may struggle with context sensitivity, non-verbal cues, sarcasm and irony, bias and misinterpretation. Just like humans!

Using generative AI for data analysis: sentiment analysis, part 3

Alberto Puliafito

Analyse public opinion, feedback, and emotional tones in text, while understanding the limitations and best practices for accurate and ethical data analysis

Author

Alberto Puliafito

‌

‌‌‌

‌‌‌