Ever since ChatGPT 3 was released for public use, several people assumed that AI is already the new way to get almost everything done! User research analysis almost fell to that assumption. However, Hold your horses! That transition is not happening anytime soon. Most of the AI analysis software, whether hurriedly built on user research platforms, video calling software like MS-teams or several meeting softwares like Grain AI recorder, Otter.ai, Read.ai, etc. fall well short of delivering sensible insights from qualitative data.
Analysis of qualitative data, whether recordings from unmoderated usability testing, user interviews, Diary studies or Focus group discussions is an art. Extracting meaningful insights via Analysis and Synthesis is the job of a UX researcher and the researchers need to master these skills. UXArmy DeepDive user interview software is built to speed up the analysis and synthesis of such qualitative data through easy collaboration, filtering and sorting.
Ask a new researcher about qualitative data analysis, these terms will ring in – “Thematic analysis”, “Tagging” and “Pattern finding”. However, huge quantities of data and researchers’ bias are challenging problems to correctly analyze qualitative data. The data from research can be coming from machine / manual Transcriptions and Notes taken by teammates and researchers themselves. Each noteworthy text has a meaning depending upon the context (action which the user was doing on screen / real life, etc.) which researchers know. It is important for Researchers to practice and learn how to consider each information that is relevant irrespective of how many times that information appears in the data.
While Thematic analysis is the most commonly used technique to analyze qualitative data, other methods exist namely, Grounded theory, Discourse analysis and Content analysis. These techniques give clusters of similar information however, these still remain “facts”. They are not Insights yet. Qualitative data needs to be synthesized to arrive at insights. The end objective is to identify insights, instead of just reporting facts.
Broadly, two main approaches are used most often.
1) Inductive
This is a bottom-up approach of identifying themes from the existing data. The name of the themes are not known from the outset, they emerge as the analysis proceeds forward and new themes are found. This approach is most often used in exploratory research topics, when not much is known about user behavior. Using an Inductive approach helps the researchers to develop new theories and gather new insights from the qualitative data.
2) Deductive
Opposite of Inductive, this approach is top-down. Themes are known before the thematic analysis is started. Themes might be known due to a preceding research or with clear objectives of research e.g. in case of usability studies with clear problem definition. Researchers look for evidence of the known themes in the data and then Code the data.
The entire qualitative data is read through and potential patterns are identified. Codes are created and assigned to each of the likely patterns. Pre-existing codes (aka Tags) in Analysis Space of DeepDive can be used as an inventory of codes applicable throughout the account.
In the past few months, AI based software have made several claims about automatic analysis of qualitative data. However, most of those software have fallen well short of the expectations of researchers. The analysis from those AI softwares that we evaluated at UXArmy have been awfully inaccurate and sometimes even ridiculous. With due regards to creators of those interview summary softwares, we reckon that the technology is not mature enough yet to be useful. UXArmy research team tried ChatGPT to process transcripts from a project with 8 interview transcripts. We requested ChatGPT to generate codes, quotes and interview summaries. For each of the interview scripts, several different codes were generated. Some of the quotes were totally meaningless, requiring our research team to do the analysis on DeepDive Analysis Space.
As Nielsen Norman quoted in their article written in July, 2023 “Be skeptical of the marketing claims being made by AI tools designed for UX researchers. Many of these systems are not able to do everything they claim.”
The skill to analyse Qualitative data is quintessential for user reseachers. While doing the analysis is not difficult, it is time consuming and honestly, not all that fun 🙂 A thematic analysis approach produces useful insights, provided it is meticulously done and researchers’ biases eliminated as much as possible. Software used for qualitative analysis like DeepDive, are efficient to work with, offering flexibilty and collaboration.