Daily Crawling of the Scholarly literature

[petermr]

This is a new topic to record and plan our imminent start of systematic crawling of the literature. This is the high-level strategy:

• use getpapers to download daily metadata for each paper including its URL. Current sources are CrossRef, EuropePMC (EPMC) and arXiv.
• filter the query on very broad filters (e.g. journal-article for CrossRef)
• extract the top metadata items (perhaps 10-20) for each paper.
• filter (blacklist) sites which cause technical problems (e.g. hanging)
• write metadata to daily CSV and bib.json

Constraints and comments.

• About 10K (+-50%) papers/day
  • each metadata.json ca 50Kb , so maybe 20-50 Mb per day
  • each paper (in HTML) ca 300Kb so 5 GB/day

Attempt to download paper, using quickscrape on Cambridge Univ subscription. Possible results:
* success. HTTP 200
* graceful failure (e.g. HTTP 520) with no download
* problem failure (crash or timeout). This will require iterative alterations to quickscrape.

Summary data
* types of artefact (PDF, HTML, supplemental), with sizes
* HTTP status

At this stage we will probably use the HTML for metadata extraction (html/head) but not use the PDF.

Metadata
* Metadata to be created as bibjson, with additional fields from other systems
* CrossRef gives crossref_metadata.json which has ca 20 first-level keys
* HTML landing pages have html/head
* HTML fulltext has html/head. It may be different from the landing page
* Quickscrape has scrapers directory which dictates the html/head fields to extract
* EPMC has a JATS-derived eupmc_metadata.json - quite extensive with some fields related to housekeeping
* JATS also has (often copious) metadata - e.g. names/addresses of editors.

[petermr]

I have used CrossRef to extract metadata for a complete week of scholarship, starting on 20160601 until 20160607. I did this by manually constructing the CrossRef RESTful URLs (API) although since then it can be done better by getpapers. This gets metadata for "everything" - books, "components", journal-articles, etc.
The coverage depends on what "publishers" submit. It is very probably weak on the Global South, on theses, and will be skewed to commercial publishers and large institutions.
Here's the scale for 7 days:

18556
25599
18571
8446
43150
16483
19576

Treat this with caution as we haven't listed the types yet. I think some contributors don't deposit every day, but it gives an idea of the scale - 150K scholarly objects / week. I imagine that most/all have DOIs.

Here's an analysis of the types in CrossRef. I imagine these are assigned by the publisher:

journal-article,106863
book-chapter,19487
component,11071
proceedings-article,4953
dataset,1806
journal-issue,1383
other,1022
report,806
monograph,766
book,670
journal,670
reference-entry,363
dissertation,215
standard,154
report-series,67
proceedings,33
journal-volume,26
book-section,12
reference-book,12

All these data will be published as CC0.

There were ca 1000 publishers during that one week; here are the leading 50. Again caution - we haven't normalized the names and some publishers may deposit irregularly. But there were some surprises for me. Divide by 7 to get the daily totals, and multiply by ~50 to get the yearly ones.

Elsevier BV,21614
Springer Nature,16576
Informa UK Limited,9945
Wiley-Blackwell,7703
Public Library of Science (PLoS),6909
Cambridge University Press (CUP),6673
Walter de Gruyter GmbH,5383
Organisation for Economic Co-Operation and Development (OECD),3309
Institute of Electrical & Electronics Engineers (IEEE),2511
Virginia Tech Libraries,2429
SAGE Publications,2163
Oxford University Press (OUP),2097
MDPI AG,2097
Brill Academic Publishers,1718
University of Arizona,1701
Copernicus GmbH,1486
Royal Society of Chemistry (RSC),1454
American Chemical Society (ACS),1352
OpenEdition,1351
PERSEE Program,1306
Ovid Technologies (Wolters Kluwer Health),1111
China Science Publishing & Media Ltd.,1108
Thieme Publishing Group,1054
Shanghai Institute of Optics and Fine Mechanics,929
AIP Publishing,892
Edward Elgar Publishing,889
Association for Computing Machinery (ACM),862
IOP Publishing,796
Harvard University Press,722
EDP Sciences,659
QIRT Council,619
RCN Publishing Ltd.,587
SWGE Sistemas,566
eLife Sciences Organisation, Ltd.,564
American Physical Society (APS),523
Medknow,512
American Scientific Publishers,494
American Society of Civil Engineers (ASCE),492
National Institute of Standards and Technology (NIST),484
OMICS Publishing Group,477

[petermr]

Mining journal-articles only

installed getpapers locally and used on crossref using the command:

./bin/getpapers.js -o test20160602 --api crossref \
    --filter-from-created-date 2016-06-02 \
    --filter-until-created-date 2016-06-02 \
    --filter "type:journal-article"

The output was:

info: Searching using crossref API
[ 'type:journal-article' ]
type:journal-article
info: Found 18547 results
info: Duplicate records found: 18538 unique results identified
info: Saving result metadata
info: Full CrossRef result metadata written to crossref_results.json
info: Individual CrossRef result metadata records written

The files are located in test20160602

localhost:getpapers pm286$ ls -lt test20160602 | wc
   18540  166853 1392303

and

ls -lt test20160602/crossref_results.json 
-rw-r--r--  1 pm286  staff  50057539 20 Jun 18:15 test20160602/crossref_results.json

i.e. ca 50 Mbytes for 18K results ==> 3Kbytes for each item

First entry is:

[
  {
    "indexed": {
      "date-parts": [
        [
          2016,
          6,
          2
        ]
      ],
      "date-time": "2016-06-02T11:40:22Z",
      "timestamp": 1464867622590
    },
    "reference-count": 23,
    "publisher": "Springer Nature",
    "license": [
      {
        "content-version": "vor",
        "delay-in-days": 0,
        "start": {
          "date-parts": [
            [
              2016,
              6,
              2
            ]
          ],
          "date-time": "2016-06-02T00:00:00Z",
          "timestamp": 1464825600000
        },
        "URL": "http://www.springer.com/tdm"
      }
    ],
    "DOI": "10.1007/s11695-016-2250-9",
    "type": "journal-article",
    "created": {
      "date-parts": [
        [
          2016,
          6,
          2
        ]
      ],
      "date-time": "2016-06-02T10:46:07Z",
      "timestamp": 1464864367000
    },
    "update-policy": "http://dx.doi.org/10.1007/springer_crossmark_policy",
    "source": "CrossRef",
    "title": [
      "Factors Leading to Self-Removal from the Bariatric Surgery Program After Attending the Orientation Session"
    ],
    "prefix": "http://id.crossref.org/prefix/10.1007",
"author": [
      {
        "affiliation": [],
        "family": "Yang",
        "given": "Kai"
      },
[... snipped ...]
      {
        "affiliation": [],
        "family": "Sockalingam",
        "given": "Sanjeev"
      }
    ],
    "member": "http://id.crossref.org/member/297",
    "published-online": {
      "date-parts": [
        [
          2016,
          6,
          2
        ]
      ]
    },
    "container-title": [
      "Obesity Surgery",
      "OBES SURG"
    ],
    "link": [
      {
        "intended-application": "text-mining",
        "content-version": "vor",
        "content-type": "application/pdf",
        "URL": "http://link.springer.com/content/pdf/10.1007/s11695-016-2250-9.pdf"
      },
      {
        "intended-application": "text-mining",
        "content-version": "vor",
        "content-type": "text/html",
        "URL": "http://link.springer.com/article/10.1007/s11695-016-2250-9/fulltext.html"
      }
    ],
    "deposited": {
      "date-parts": [
        [
          2016,
          6,
          2
        ]
      ],
      "date-time": "2016-06-02T10:46:08Z",
      "timestamp": 1464864368000
    },
    "score": 1,
    "subtitle": [],
    "issued": {
      "date-parts": [
        [
          2016,
          6,
          2
        ]
      ]
    },
    "alternative-id": [
      "2250"
    ],
    "URL": "http://dx.doi.org/10.1007/s11695-016-2250-9",
    "ISSN": [
      "0960-8923",
      "1708-0428"
    ],
    "subject": [
      "Surgery",
      "Nutrition and Dietetics",
      "Endocrinology, Diabetes and Metabolism"
    ]
  },

These files will be extracted using norma software:

[petermr]

I am now about to mine the complete metadata from Crossref exhaust. Here's the repeat of what I did earlier:

 ../../getpapers/bin/getpapers.js -o crossref --api crossref --filter-from-index-date "2016-06-01" --filter-until-index-date "2016-06-01" --filter "type:journal-article"

which gives

info: Searching using crossref API
[ 'type:journal-article' ]
type:journal-article
info: Found 26517 results
info: Saving result metadata
info: Full CrossRef result metadata written to crossref_results.json
info: Individual CrossRef result metadata records written
localhost:getpapers pm286$ ls -lt crossref/crossref_results.json 
-rw-r--r--  1 pm286  staff  67659071 26 Jun 11:51 crossref/crossref_results.json

that's ca 3 Kb of metadata from each article. In discussions with Geoff Bilder it seems that some journals point directly to the fulltext, so possibly no need to use Quickscrape. Let's see.

First thing will be to create spreadsheets (*.csv) of the metadata from Crossref. They'll be large (25K rows) and probably ca 30 columns. Let's see