ContentMine Scrapers and other harvesters of scholarly content

[jenny]

We had a conversation with Mike Taylor about the harvesters of scholarly content being built for The One Repo and how they differ from the scrapers ContentMine are using with quickscrape. Mike is looking for partner organisations who might be willing to sponsor the creation of harvesters and in the spirit of collaboration and reducing unnecessary duplication of work we wondered if we could use each other's tools.

Mike responded:

The big differences, from what I saw when I looked at your scrapers, are:
* Scrapers seem to work on one page at a time, while harvesters crawl
around sites and extract multiple pages.
* Harvesters have lots of a heavy machinery to deal with awkward pages
that obscure content with JavaScript, cookies or crazy authentication
schemes.
* Scrapers are more neatly positioned as tools that can be integrated
into any system, whereas harvesters are wired into a bigger ecosystem.
* Crucially, scrapers are driven by an open-source engine, whereas the
harvester engine is encumbered by co-ownership.

Peter Murray-Rust:

I use "crawl" instead of harvest. It's about discovery and it's 90%+ political and non-technical.

Our scrapers have Javascript machinery but don't do wget - they get precisely what they are asked for

I think your harvesters overlap at one end with crawlers and at the other end with scrapers.

We hope the projects can find useful collaborative ground and we're interested to hear from anyone else building scraper/harvester/interfaces for the scholarly literature.

[MikeTaylor]

To clarify: in writing this, I wasn't trying to make a distinction between the meanings of the words "scraper" and "harvester" -- just describing the difference between ContentMine's software (which they call "scrapers") and Index Data's (which we call "harvesters"). I'm not sure that the two terms significantly differ in their actual meaning.

[antonin]

The Internet Archive currently leads a project called OpenJournal bringing together people who work on indexing/archiving/mining scholarly publications. You are very welcome to join the discussion! We have already had very interesting discussions between BASE, CiteSeerX and dissemin, among others.

Mike, reading your description of the differences between QuickScrape and the OneRepo I understand that OneRepo is not an open source project. Can you elaborate on that?

[jenny]

Thanks Mike - yes the title for the thread was misleading - I've updated!

[MikeTaylor]

In spirit, it's an open-source project. More crucially, it's an open-data project. Everything that we harvest, we republish openly for anyone to reuse for any purpose. But for historical reasons, some of the infrastructure that we use to harvest sites is co-owned by a commercial partner who are not willing for us to release it as open source. (If necessary, we could re-write that part as open-source software, but it would take us time that we presently have more pressing uses for.)

[antonin]

I see. I think it would be strategic to opensource at least some parts of it, for instance the scraping part. For instance, Zotero's translators are actively maintained by the community: it would be very expensive to build an equivalent tool in a closed way. Another interesting model is DuckDuckGo's Instant Answers, allowing anyone to improve DuckDuckGo's search results by adding custom sources.

[MikeTaylor]

The parts that we can open-source, we have -- e.g. the UI widget set that we use to query the database for the demo is at https://mkws.indexdata.com/

We (Index Data) have been releasing open-source software for more than two decades -- since 1995, long before the term "open source" was coined. Don't worry, we're well aware of the advantages!