Evaluating the recommender – undergraduate focus groups

We held more focus groups over the summer holidays, which is kind of tricky when your audience for testing are undergraduates!  Not many are left on campus during the summer months, but we did manage to find some undergraduates still around the Manchester Metropolitan University (MMU) campus.  This might be a reflection of the heavy rainfall we’ve been having in Manchester this summer.

We chose MMU as we wanted to test the recommender on undergraduates who didn’t already have a recommender on their own library catalogue, as they might complete the tests as a comparison.   We spoke to 11 undergraduates in total and they tested the recommender 42 times between them.

The Results

The students were positive about the concept of the book recommender and were keen to use this as another tool in their armoury to find out about new resources.  A key bonus to them was the lack of input the recommender needed in order to gain maximum output. To a time poor, pressured undergraduate this is a huge plus point.

‘Yeah, I would use it, I don’t have to do anything’

‘I would always look at it if it was on the MMU library catalogue’.

The recommender also offered an alternative source of materials to the ubiquitous reading list.  This is absolutely crucial because it quickly became apparent that our participants struggled to find resources;

‘I go off the reading list, all those books are checked out anyway’

‘I’ve used Google scholar when I’ve had nowhere else to go but it returned stuff I couldn’t get hold of and that just frustrated me’.

So in theory it offered more substance to their reading lists.  The additional books it found were more likely to be in the library and came with the advantage that they were suggested because of student borrowing patterns.  Our respondents liked having this insider knowledge about what their peers have read in previous years.

‘It would be useful as I had to read a book for a topic area and when we got the topic area there were already 25 reservations on the book, so if I could click on something and see what the person who did this last year read, that would be very useful’.

Testing the prototype

In testing it proved difficult to conclude if the recommender was useful or not, as some testers seemed to have more luck then others in finding resources that were useful to them. Obviously within the data collection method some margin of error needs to be accounted for.

Of course, you could argue that whether a book is useful or not is a highly subjective decision.  One person’s wildcard might be another’s valuable and rare find, and whereas one tester might be searching for similar books others might be looking for tangentially linked books.   As an example of this, in our group, History students wanted old texts and Law students wanted new ones.

Positively, 91.4 % of the recommendations looked useful and only 3 searches returned nothing at all of any use to the user. 88.6% of searches generated at least one item that the user wanted to borrow and only 4 searches resulted in not a single thing that the user would borrow. Even with a deviation due to subjectivity these are compelling results. As the recommender requires the user not to submit anything substantial in order to get results, a low percentage returning nothing is acceptable to all users we interviewed.

Privacy concerns?

As in previous research, none of the undergraduates attending the focus groups expressed any concern in respect of privacy issues and they understood that the library collected circulation data and the results by the book recommender are generated by that circulation data.

‘I would benefit from everyone else’s borrowing as they are benefitting from mine, so I haven’t got a problem’.

‘It would be nice to be informed about it and given the option to opt out, but I don’t have a problem with it. No.’

Although a ‘nice to be asked’ was expressed by more than one attendee, they wouldn’t want this to delay the development of the book recommender.

In conclusion, the time poor, pressured student struggling to find books off the reading list still in the library would welcome another way of finding precious resources. The majority of students in our groups would use the recommender and, although some recommendations are better than others, they would be willing to forgive this if it just gave them one more resource, when the coursework deadline looms!