Indexing Open Access Theses

[petermr]

Indexing Open Access Theses

The AMI software has been upgraded to use PDFBox2 and this makes it attractive for indexing scientific theses. Most public thesis indexes do not use semantic science tools so we expect this to be an important step forward.

Methodology

This is a proof of concept, not a work of scholarly bibliography. The conclusions are based on a small arbitrary sample and designed to find out what citizens want to see.

I have selected ca. 30 theses from OATD (https://oatd.org/faq.html ) , a global collection of pointers to "Open Access Theses". I'll search for angiosperm (The technical term for "flowering plants" - an area I know something about but not at a professional level - I can read papers, but not write them.) I will download them manually (as repositories are not automated), run them through AMI and show a dashboard of plant science.

"Open Access" is not a precisely defined term

• at its lowest means "You can read this now but may not do anything with it and it may disappear at any time"
• at the most liberal (CC BY or CC0) it means
  • free to download
  • free to read
  • free to use (e.g. mine or compute)
  • free to re-use (e.g. extract chunks, transform by humans or software, mix with other documents)
  • free to redistribute (either intact or modified)

The only restriction is that you must normally credit the author. There is no formal way of doing this and I shall copy the bibliographic record on the OATD site.

thanks and caveats

I thank OATD for the repository and the authors of the theses (any criticism is not aimed at them). OATD provides links to (University) repositories where the theses are held, and cannot provide a frictionless machine processable system.

Every University provides a different approach to managing and exposing theses. These are arbitrary, based on Handle and DSpace and similar tools and few (?none?) provide APIs for bulk download. Generally it is impossible to find the thesis without human navigation (e.g. "Download thesis" button.

[petermr]

"Open Notebook" log of my exploration

I would like to provide a scientific , reproducible record of what I do. The material is "open" but often with express instruction from the University that it can only be used for personal purposes. This means that although I can record what I did I can't post the material to Github or this Discourse without permission.

I am allowed to mine the theses without permission as the UK "Hargreaves" exception allows me to in the UK for non-commercial purposes. It does not allow me to repost the original thesis, nor to post a derivative though I can post "facts" as these are uncopyrightable under the Berne Convention.

This makes it almost impossible to do responsible science without a serious investment of time writing for permissions

By default the Notebook will cover about 30 (arbitrary) entries retrieved from OATD.

[petermr]

The OATD repository

NOTE: do not confuse with the excellent OATP project for tracking #openaccess https://cyber.harvard.edu/hoap/OATPFAQ_

I reproduce some of the FAQ here https://oatd.org/faq.html (hopefully OK...)

What can I search for here? (from OATD)

This is an index of over 3.5 million electronic theses and dissertations (ETDs). To the extent possible, the index is limited to records of graduate-level theses that are freely available online.

What can I do with the ETDs I find on this site? (from OATD)

Most documents made available for open access allow you to download and read them for personal use. Without specific permission from the copyright owner, do not assume further permissions, such as redistributing ETDs from another web site or using them for commercial purposes.

With few exceptions, the documents indexed on OATD are the property of their authors and are legally protected by copyright. Individual ETDs and host repositories may have specific statements of the author’s rights and your permissions to use the document. In the event that the local repository or the ETD itself states different permissions than the record indexed by OATD, the local statement takes priority.

What definition of "Open Access" do you use? (from OATD)

We define open access broadly to cover ETDs that are free to access and read online. We encourage authors to consider formally specifying use permissions, for example by publishing their theses with a Creative Commons license.

Where does the full text live? (from OATD)

The full text of all papers lives on the original hosting site, usually the repository of the university that granted the degree. OATD indexes about the first 30 pages of some theses in order to show search hits, but in no case does OATD index or store the full text of the paper.

PMR's comments

Note that the restrictions above (not OATD's fault) are serious drawbacks to automatic mining. There is massive legal uncertainty and difficulty in automatic navigation. This is particularly true in US (where Universities often give the rights to commercial companies), UK where there is no centralisation of theses and countries where theses are published as books.

There are communal repositories with APIs such as HAL for France and I have started to index them, but here I want to challenge the system with variety.

querying OATD

I used the home page and Advanced Search to specify

• The text contains angiosperm and probably stemmings from angiosperms , angiospermic, etc. I suspect that this could occur anywhere (title, abstract, references). In general this whole-document indexing gives a huge percentage of false positives, especially where words are polysemic or common.
• Nothing earlier than 2010 because earlier documents are lower technical typographic quality (although the science is doubtless as good as today).
• That the theses should carry "Creative Commons Licences". Note that a CC licence is not de facto liberal - "Some Rights Reserved" normally provides no rights at all. In practice many of the theses di not carry licences. The most we can hope for is that the thesis is visible and (probably) downloadable.

[petermr]

Corpus for mining.

The results from OATD search are not easy for machines to follow automatically. Like so many sites they are arranged as a series of single references. Here are the first five.

NOTE: There is no indication of licence or permissions on this page. I downloaded this by hand (like all the others). As I said before I have no complaints with the authors

Thesis 1

Author: Anastasia Gousseva
Title: _Investigating the Expansion of Angiosperms during the Late
Cretaceous using a Modeling Approach_
Licence: None
Copyright: © Copyright by Anastasia Gousseva 2010
Pages: 211

Thesis 2

Author: CORINNE ALEXANDRA FAY
Title: _MID-CRETACEOUS pCO2, CARBON-CYCLING
AND THE RISE OF THE FLOWERING PLANTS_
Licence: None
Copyright: None
Pages: 387

Thesis 3

Author: Karina I. Neimanis
Title: AN INVESTIGATION OF ALTERNATIVE OXIDASE PRESENCE, EXPRESSION, AND REGULATION IN THE MOSS PHYSCOMITRELLA PATENS
Licence: None
Copyright: None explicit
Pages: 211

Includes: Follow this and additional works at: http://scholars.wlu.ca/etd
Part of the Bioinformatics Commons, Biology Commons, and theMolecular Biology Commons
Unfortunately these "Commons" now probably belong to #elsevier

Wilfrid Laurier University

Thesis 4

Author: Charles Stuart Piper Foster
Title: Using Phylogenomic Data to Untangle the Patterns and Timescale of Flowering Plant Evolution
Licence: None
Copyright: None explicit
Pages: 249

Thesis 5

Author: Lisa Maria Ebner
Title: Untersuchung an Angiospermen- und Gymnospermenpollenkörnern aus der Potomac-Formation (Unterkreide) in den USA
Licence: None
Copyright: All rights reserved (site)
Pages: 129

NOTE: the text is in German but AMI can still index much of it. We can use Wikipedia later to index the German words.

Files were manually copied to a directory:

└── oatd
    ├── 2014-06-09_0801231.pdf
    ├── CORINNE_ALEXANDRA_FAY.pdf
    ├── Foster_CSP_Thesis.pdf
    ├── Gousseva_Anastasia_201011_MSc_thesis.pdf
    └── OXIDASE_IN_PHYSCOMITRELLA.pdf

(The filenames were edited to remove non-printing and URL-escaped characters).

terms

• oatp is called a CProject in the AMI system.
• The PDFs will all become CTrees after this.

full corpus

I selected 30 records from OATD. Of these:

• 2 sites failed to respond
• 1 link was broken
• 1 was embargoed til 2021
• 1 required me to sign in even though it was CC BY-NC
• 1 was "not available"

Leaving 24 that I'll work with.

So 20% of links in OATD won't give theses.

Conclusions

It took me about 30 mins to download 30 theses. There were at least 15 different "styles" to the repository, most were clunky and give a clear impression that libraries regard theses with a C20th mind, not a C21st. I agree that they are part of the educational and assessment process, but many are funded by Research funders and industry and IMO the theses themselves are excellently produced. It's a shame that they are not better deployed on the sites.

from now on almost everything is automatic ...

[petermr]

AMI search and indexing

The 24 theses range from 50-600. It takes 30-300 secs on my laptop to process a thesis, with an average of ca 45 secs. Unfortunately some theses show quadratic performance, and can get a memory leak. This can be due to large numbers of images and also many graphic components. Will fix.

Junk

This horror consists of 60 separate images. Why not use SVG which would do this in 20 Kb, would be pastable and re-usable??

Running the search

cTree: 2013-04-05_0105406
pages include: [0, 1, 2, 3, 4,  ...196, 197, 198, 199, 200]

[1][.1][2][3][4][5][.1][6][7][.1][8][9][10][11][12][13][14][15][16][17][18] ...
...[114][115][.1][116][117][.1][118][119][.1][120]
writing to: target/totalIntegration/oatd/2013-04-05_0105406/svg

This shows a 120 page thesis with images ([7][.1] means 1 image on page 7. The resultant SVG pages are then written to disk. This is now converted to HTML, and then indexed.

We'll look at that later. the actual formatting is not perfect but almost all the words are correct so we can index on terms.

Here's a small taster of the dashboard:

Plant scientists will recognize species and genera and also the parts of plants. We have done all 24 theses and got a dense table of high quality extractions.