Guide to Running an Impactful Diary Study
By
Digital Of Things
October 10, 2023
Technology has evolved so much over the last few years that the once time-consuming and laborious diary study has evolved into something super streamlined. Gone are the days of quill and ink…now participants just fill out a form and send the link. It’s as simple as that.
Running a diary study is one of the best ways to develop your skill as a researcher. A study such as this allows you to get to the heart of a question by probing and prompting participants when necessary. These days it’s easier than ever to do. So long as the participants have an internet connection or a smartphone, it’s possible to get their innermost thoughts without having to leave the office (or sit with them in an interview situation).
The data that gets collected from a diary study can be much richer and more in-depth, but what is a diary study – and what’s the best way to run one?
You might not want to go back and look at your own teenage diaries (hello, embarrassing crushes and bad fashion choices). But if you did, you’d find they were a stream of information about your day-to-day life and the minutiae of your experiences in education, with friends and family. A diary study for UX research is pretty much the same thing. Participants are given a task to undertake or a process to go through (this could be anything from installing a new app to perhaps booking a flight), and then they go away and do it. Years ago, it would’ve been done with notebooks, and participants mailing in their responses. Nowadays, it can be done virtually – even using video and audio.
UX diary study research is a hugely important tool, as it can be a fantastic way of gaining long-term insights into the behaviour of users, and open a window into their thoughts and feelings. A UX diary study can give a researcher a bird’s eye view into how a participant makes a decision or how they feel about making changes over a period of time.
Of course, there are pros and cons to using a UX diary study. Let’s start with the pros though:
Now for the cons:
The most important thing to consider when designing a diary study is to ensure that the product actually does require a diary study as a research method and that it's not just a random decision!
To run a diary study, you need to have a product or service that people use frequently for a certain period of time. A good example of this would be booking a ticket for travel, using Spotify for entertainment or Slack for work.
A second consideration is to think about exactly who you’re asking to participate in a diary study. Making sure you have the right people and understanding the goals to analyse is crucial:
This really depends on what you're testing and the development of the product or service.
Say you're trying to test something like booking a flight, it won't take weeks, it'll probably take maybe ten days. However, if you're testing a feature or something that people use, such as a messaging tool and you're introducing features to it, you'll probably need a longer period of time.
If you’re building new features and want to test them with people, you might want your recruits to see what you’re building feature by feature or have them test out certain ways of uncovering features. This would be useful in an online game, where users have to pass a certain level. In cases like this, you might want to do a diary study about the game itself. You could also use a diary study to look into the gamification of different kinds of apps.
One of the most important factors in a diary study is time. However, this can create issues. Sometimes, the more time there is, the less engaged people feel. So make sure that your plan to communicate with them is easy.
Also the longer a diary study continues, the more likely it is participants might drop out and not continue. Therefore, it's important to either give them a good incentive so that they can stay longer, or just make sure that it's not too long and it doesn't take them a lot of time to participate or give feedback. It should be quite easy and to-the-point.
Again, it depends on the study and what’s being tested. If it's a really long study, done over a long period (say for up to six months), entries could be done every week, once a week – or it’s possible they could be done every couple of days.
We chatted with Rana Mansour, one of our brilliant UX wizards, and she gave us some great insights from a diary study she conducted with people who were booking flights. She said: “There was a lot of input needed from them because we wanted to capture feedback every step of the way”.
This included information on when the participants were researching, to when they were looking for places to go. The next stage was for the participants to look at tickets and discover when they were actually booking them. It was important to look at what kind of decisions they were making, and who the main decision maker was, too.
Rana said that for a study like this, input was needed every two to three days. Regularity and consistency were required because time was of the essence. Responses would need to match the steps that the participants were actually using to book the flight tickets.
In terms of how to question respondents in a study like this Rana says: “You don't want to ask too many questions, but also you don't want to ask or engage them too little so that they forget about what they're doing. There needs to be a balance”.
As with any other form of research, you need to take a core sample group, or the participants need to represent the demographics. Look at who your target audience is and consider these factors:
Here Rana tells us that: “An important thing is not to make the task tedious! Maybe having multiple questions or asking participants to record a voice note, take a screenshot, something that is fast for them to do”.
She adds that it’s very important to incentivise participants and to really appreciate the time that they're taking to fill out the answers. Incentivising people properly, being very empathetic, and treating them as human beings are super important in studies like this. After all, people are giving their time to the diary study to help you.
It’s a great idea to use tools that the participants are already familiar with, for example, WhatsApp. What’s really key is to ensure you’re being considerate to different types of people or different personalities and confidence levels. Some people might be happy to record a video, others not. In these cases, asking them to record a voice note or sending text messages instead can be helpful.
Rana comments that it’s very important to always take an interest in what your participants are doing and engage with them as human beings: “Explaining to them what we're trying to understand or what we're trying to uncover, what exactly are we looking for” is essential. As an example, she says: “If we're doing a diary study about people travelling, then we should ask them what kind of platforms they use, what tools they usually search with, or what their processes are”. When they answer, we can then dive deeper.
It really depends on the type of study. Let’s say there’s going to be a new feature on a product, and it needs to be tested. It’s important that people are given enough time to explore it first, before any studies are carried out.
Another way around it is to give users a task to actually do while using this feature – without explicitly telling them about it. This is a good way to help us understand how they find it, and give them enough time to explore. Here, you can ask participants to record answers and then, after a couple of days, ask them to update and record their findings after they’ve spent more time using the feature.
Rana adds her own experiences from working on her flight booking diary study: “If it's for booking a ticket, we need to understand their timelines and match our questions to their timeline.” In this instance, questions have to be answered pretty instantly – not a week after they do their task.
In essence, Rana says that timing can be considered in two ways: “The first is making sure it’s based on our timelines and how we release things. If it's based on a specific event, then it needs to be tied to their process and their own timeline”.
The ideal way to analyse data is to look at all the entries that have come through and try to segregate them by behaviour.
First, try to understand who the different personas are. From that, you can attribute certain behaviours. Once you can see those, you can look at personas and see if there are patterns between them.
Rana comments: “It's really like diving deep into all their insights, and trying to make sense of them by finding patterns or certain actions that trigger a type of behaviour. We’re trying to make sense of the timeline, with the actions or the behaviours that are attributed to it”.
Rana says that she mainly uses Miro to track data because it's very visual. She has also used WhatsApp to collect feedback and follow up with people. However, she’s recently seen a tool called Indeemo that looks as though it could be an incredibly useful platform to trial.
She says that many older diary study tools are now very outdated and often users need to go out of their way to report things, which often isn’t very easy or convenient, so her two main choices work the best.
There are online survey platforms, which can provide instant, secure ways of recording participants' thoughts and feelings online. This can be done in the person’s own time. A similar idea is the use of mobile diary apps – allowing for written, sometimes video and audio communication, on a research topic.
Qualitative data analysis software is another very useful diary study tool to have at your disposal. Why? For interpreting and analysing the results when they come in and then pulling your information together coherently.
Main limitation is budget. With this, it’s always best to try and explain to users how long it's going to take and determine the duration of a diary study, so that they know how much effort they’ll need to put in. Finish by mentioning the incentive they can enjoy for taking part.
The second limitation is the risk of people dropping out of the study. It’s important to make sure that there’s just enough engagement with participants and that a good rapport is built with them in order to explain exactly what’s required and when.
You’ve weighed up the pros and cons of all the above and decided that a diary study is the best way forward to get maximum impact in terms of research, so the next question is…how do you go about creating one?
There are five main steps to consider, depending on the type of study you want to undertake and how long you think it might take to complete:
What do you want to know? Who do you want to hear from?
Finding the right people in the right demographic and location.
What’s the best way to collect the diary study examples – an online platform, or via recorded audio and video?
Making sense of the information you receive.
Sharing the UX diary study research results with your colleagues and stakeholders to get their thoughts.
A diary study for UX research might not be exactly the right fit for you, but there are similar alternatives that can work just as well.
Methods like field studies, where an interview or study is conducted in the environment the participants would use the platform/app. These are very viable and can produce similar results over a shorter time frame. They do rely more on the participant and researcher living in close proximity and as with other forms of study, like one-on-one interviews, there can sometimes be a lean towards unconscious bias (where a participant is answering with what they think the researcher wants to know, rather than their genuine feelings).
As we’ve seen, the diary study for UX research is one of the best ways of getting to the heart of the thoughts, feelings and behaviour of participants – as well as discovering what guides and influences them to make their decisions. It’s an invaluable tool and can really assist with problem-solving.
