Readers of this blog are not new to my ramblings on soft peer review, social metrics and post-publication impact measures:

• can we measure the impact of scientific research based on usage data from collaborative annotation systems, social bookmarking services and social media?
• should we expect major discrepancies between citation-based and readership-based impact measures?
• are online reference management systems more robust a data source to measure scholarly readership than traditional usage factors (e.g. downloads, clickthrough rates etc.)?

These are some of the questions addressed in my COOP ’08 paper. Jason Priem also discusses the prospects of what he calls “scientometrics 2.0″ in a recent First Monday article and it is really exciting to see a growing interest in these ideas from both the scientific and the STM publishing community.

We now need to think of ways of putting these ideas into practice. Science Online London 2010 earlier this month offered a great chance to test a real-world application of these ideas in front of a tech-friendly audience and this post is meant as its official announcement.

ReaderMeter is a proof-of-concept application showcasing the potential of readership data obtained from reference management tools. Following the announcement of the Mendeley API, I decided to see what could be built on top of the data exposed by Mendeley and the first idea was to write a mashup aggregating author-level readership statistics based on the number of bookmarks scored by each of one’s publications. ReaderMeter queries the data provider’s API for articles matching a given author string. It parses the response and generates a report with several metrics that attempt to quantify the relative impact of an author’s scientific production based on its consumption by a population of readers (in this case the 500K-strong Mendeley user base):

The figure above shows a screenshot of ReaderMeter’s results for social scientist Duncan J Watts, displaying global bookmark statistics, the breakdown of readers by publication as well as two indices (the H_R index and the G_R index) which I compute using bookmarks as a variable by analogy to the two popular citation-based metrics. Clicking on a reference allows you to drill down to display readership statistics for a given publication, including the scientific discipline, academic status and geographic location of readers of an individual document:

A handy permanent URL is generated to link to ReaderMeter’s author reports (using the scheme: [SURNAME].[FORENAME+INITIALS]), e.g.:

http://readermeter.org/Watts.Duncan_J

I also included a JSON interface to render statistics in a machine-readable format, e.g.:

http://readermeter.org/Watts.Duncan_J/json

Below is a sample of the JSON output:

{
	"author": "Duncan J Watts",
	"author_metrics":
	{
		"hr_index": "15",
		"gr_index": "26",
		"single_most_read": "140",
		"publication_count": "57",
		"bookmark_count": "760",
		"data_source": "mendeley"
	},
	"source": "http://readermeter.org/Watts.Duncan_J",
	"timestamp": "2010-09-02T15:41:08+01:00"
}

Despite being just a proof of concept (it was hacked in a couple of nights!), ReaderMeter attracted a number of early testers who gave a try to its first release. Its goal is not to redefine the concept of research impact as we know it, but to complement this notion with usage data from new sources and help identify aspects of impact that may go unnoticed when we only focus on traditional, citation-based metrics. Before a mature version of ReaderMeter is available for public consumption and for integration with other services, though, several issues will need to be addressed.

1. Author name normalisation

The first issue to be tackled is the fact the same individual author may be mentioned in a bibliographic record under a variety of spelling alternates: Rod Page was among the first to spot and extensively discuss this issue, which will hopefully be addressed in the next major upgrade (unless a provision to fix this problem is directly offered by Mendeley in a future upgrade of their API).

2. Article deduplication

A similar issue affects individual bibliographic entries, as noted by Egon Willighagen among others. Given that publication metadata in reference management services can be extracted by a variety of sources, the uniqueness of a bibliographic record is far from given. As a matter of fact, several instances of the same publication can show up as distinct items, with the result of generating flawed statistics when individual publications and their relative impact need to be considered (as is the case when calculating the H- and G-index). To what extent crowdsourced bibliographic databases (such as those of Mendeley, CiteULike, Zotero, Connotea, and similar distributed reference management tools) can tackle the problem of article duplication as effectively as manually curated bibliographic databases, is an interesting issue that sparked a heated debate (see this post by Duncan Hull and the ensuing discussion).

3. Author disambiguation

A way more challenging problem consists in disambiguating real homonyms. At the moment, ReaderMeter is unable to tell the difference between two authors with an identical name. Considering that surnames like Wang appear to be shared by about 100M people on the planet, the problem of how to disambiguate authors with a common surname is not something that can be easily sorted out by a consumer service such as ReaderMeter. Global initiatives with a broad institutional support such as the ORCID project are trying to fix this problem for good by introducing a unique author identifier system, but precisely because of their scale and ambitious goal they are unlikely to provide a viable solution in the short run.

4. Reader segmentation and selection biases

You may wonder: how genuine is data extracted from Mendeley as an indicator of an author’s actual readership? Calculating author impact metrics based on the user population of a specific service will always by definition result in skewed results due to different adoption rates by different scientific communities or demographic segments (e.g. by academic status, language, gender) within the same community. And how about readers who just don’t use any reference management tools? Björn Brembs posted some thoughtful considerations on why any such attempt at measuring impact based on the specific user population of a given platform/service is doomed to fail. His proposed solution, however – a universal outlet where all scientific content consumption should happen–sounds not only like an unlikely scenario, but also in many ways an undesirable one. Diversity is one of the key features of the open source ecosystem, for one, and as long as interoperability is achieved (witness the example of the OAI protocol and its multiple software implementation), there is certainly no need for a single service to monopolise the research community’s attention for projects such as ReaderMeter to be realistically implemented. The next step on ReaderMeter’s roadmap will be to integrate data from a variety of content providers (such as CiteULike or Bibsonomy) that provide free access to article readership information: although not the ultimate solution to the enormous problem of user segmentation, data integration from multiple sources should hopefully help reduce biases introduced by the population of a specific service.

What’s next

I will be working in the coming days on an upgrade to address some of the most urgent issues, in the meantime feel free to test ReaderMeter, send me your feedback and feature requests, follow the latest news on the project via Twitter or just help spread the word!

Just take a look at the way newspapers and magazines display articles online.

Many newspapers and magazines insist on splitting up long articles into separate pages, and there are a couple of reasons for this. The first benefits the reader: it’s a lot easier to bookmark a page in a long article and come back to the same place in that article if it’s split over several pages. The second, which benefits the publisher, is that a single article gets more page impressions, and – in some cases – more ad revenue. We’ve always been proponents of putting the reader first, so for short and medium-length articles, we always advocate single-page display, and we think that even for longer articles, readers should have the option of viewing the whole thing on a single page. (For example, see how this article at nytimes.com is paginated, with an option to display on a full page).

We do understand the pressures publishers are under to monetise their content though, so with Granta.com, we paginated articles, and subscribers are given the option to view the whole thing on a single page. Likewise with the Golden Notebook Project, pagination allowed far more focussed attention to be given to the text by the readers who were commenting closely on it.

eBook Pagination

With ebooks, however, there’s no need to do this – and attempts to do so often smack of a wrong-headed desire to recreate the physical book with little thought to the affordances of the new format. Many of the complaints about the Kindle and Sony Reader centre on the delay of page flipping, which is an unfortunate necessity of e-ink devices, but which has also been implemented in many iPhone and other readers (like classics2go). It’s much like those “page-flip” books and magazines you find on the web, which needlessly attempt to recreate the experience of a physical artefact with a load of Flash and fiddly interactions (e.g.). Man, we really don’t like those.

With Enhanced Editions, chapters are displayed as one continuous text, meaning there’s no waiting to flip the page every 20 or 30 lines. In this context, pagination is a hangover from the physical book, and wholly unnecessary – just think of Jack Kerouac typing ‘On The Road’ on one continuous roll of paper.

This also means that resizing and altering the text has no effect on the layout of the book, and our paragraph-level bookmarking and full-text search means it’s easy to return to the same point in the story, regardless of pagination. It’s also much more accessible.

Tilt Scrolling

Finally, there’s the ultimate iPhone benefit: tilt-scrolling. Scrolling is of huge benefit to readers, but it’s very hard to get the speed right automatically. The iPhone’s accelerometer (the bit of the device that knows which way it is facing) means that with a flick of the wrist, you can move the page up and down at a variable speed you feel comfortable reading at.

We first saw tilt scrolling implemented perfectly by Marco Arment’s glorious Instapaper apps, one of the first inspirations for Enhanced Editions. We even asked Marco to build Enhanced Editions for us, but he was too busy.

A year on, we’ve found that every page we read on an iPhone needs to have both tilt-scrolling, and rotation lock. Hopefully you’ll agree – but if you don’t, you can just turn them off.

Enhanced Editions allows you to bookmark the text anywhere you like, and save these bookmarks to come back to later. You can also quote portions of the text, and send these via email to yourself or to a friend.

Love that particular passage in Bunny Munro? Just select, click, and send it on its way. Your friends will thank you.

Every chapter is presented as one long continuous stream of text (there’s more about this in another blog post), with arrows at the top and bottom to skip quickly back and forth between chapters. At any time, you’re just a couple of taps away from the full table of contents, which breaks the book down into its constituent sections and chapters, and you can easily return to your place in the text either directly or via the bookmarks.

The real key here is Authorial intention vs. Ebook affordances, or to put it simply, what the author wanted you to read, combined with the advantages of electronic reading over traditional books.

Authors structure their books a certain way – with chapters and sections – because these have relevance to the story, and must be preserved. But beyond that, anything is possible, and with Enhanced Editions we’ve tried to focus on what makes the best experience for the reader.

After a long (very long) time spent using lots of reading apps and devices, we decided that a reader’s time is best spent reading rather than turning pages. And so that’s what we focused on.

And then we added tilt-scrolling.