AMI-EPPI CM-UCL table extraction project

petermr 2017-01-30 15:17:07 UTC #1

UCL (London) have asked CM to collaborate on a project to extract data from tables on a high-throughput basis and measure the results. The project will last ca 2 months from now and we are grateful to UCL (James Thomas, Alison O'Mara-Eves) for funding. Another partner is Fala Cramond (Imperial, London).

@chjh and @petermr are providing the ContentMine input. @chjh is designing the evaluation strategy and supervising its implementation. @petermr is enhancing AMI to extract tables.

The project is OpenNoteBook (https://en.wikipedia.org/wiki/Open_notebook_science ) and everything is public. There are no closed emails other than admin. We believe that the whole project should be replicable including Open software, Open corpus, immediately published results and Open discussion. Anyone can read and comment.

Currently we have a corpus of 54 articles, with ca. 170 tables corresponding to ca. 20 "PubStyles". All data is at https://github.com/ContentMine/cm-ucl/ and many issues have been posted at https://github.com/ContentMine/cm-ucl/issues . The balance between discussion on Github (pages and issues) and this Discourse thread will evolve. At present we suggest formal issues on Github; narrative on Discourse.

petermr 2017-01-30 23:31:45 UTC #2

Raw-final dual presentation

This shows the raw table (left) manually captured by @chjh compared to the annotated version from AMI-table. The agreements for "almost all" of the table is excellent. Note that I have manually adjusted the scales to make the comparison easier. Not yet sure whether we can automate this. Probably.

Note the slight discrepancy between the subtables which include the subscripts. This is a bug which would be reported as an issue.

petermr 2017-01-31 08:49:43 UTC #3

Raw-final landscape presentation

Landscape tables may be displayed vertically. This has been scaled by hand but we are optimistic of automating this. Can you spot the character corruption?

The "minus" and the "plus-minus" are not Unicode so will have to be converted heuristically.

petermr 2017-01-31 10:24:47 UTC #4

Legacy characters and corruption

The previous reply shows the problem of legacy characters (in http://dx.doi.org/10.1371/journal.pbio.1000481 - open up the PDF Table 1 if you want to verify ). Look at the text under "K_D" It reads (μM)^c. This means "micromolar" i.e. a concentration of 10^-6 Molar. Try to cut and paste this from the PDF and you'll probably get

(mM)c

Any scientist would interpret this as "millimolar" - 1000X more concentrated. This could easily kill the organism, which might be you.

AMI-pdf2svg is able to convert characters heuristically.

The raw SVG shows:

   <text stroke="none" fill="#000000" clip-path="url(#clipPath9)" svgx:fontName="NPJKJN+AdvP697C" 
svgx:charName="m" svgx:charCode="956" svgx:hexCode="3bc" x="331.086" y="109.232"
 font-size="6.485" font-family="TimesNewRoman" font-weight="normal" id="text8722" 
svgx:width="750.0" xmlns:svgx="http://www.xml-cml.org/schema/svgx">μ</text>

Because these fonts are undocumented I have created an equivalence table by hand - very boring. It maps character "m" in Font "AdvP697C" to character "0x03bc" which is "mu"

However this has to be done for every font. For the "minus" in s^-1 we have

<text stroke="none" fill="#000000" clip-path="url(#clipPath1)" svgx:fontName="NPJKLP+AdvP7DA6" 
svgx:width="833.0" x="280.403" y="106.114" font-size="4.931" font-family="TimesNewRoman" 
font-weight="normal" id="text8708" xmlns:svgx="http://www.xml-cml.org/schema/svgx">2</text>

We do not have a converter for this font and so its (legacy) "codepoint" "2" is translated into the wrong character. (I thought I trapped unknown fonts - will have to check). Any other tool will select "2".

chjh 2017-01-31 14:22:31 UTC #5

Just committed a draft of the results of the grobid automated retrieval. Feel free to make revisions in this file https://github.com/ContentMine/cm-ucl/blob/master/interim-reports/grobid-table-retrieval.Rmd

It seems like precision/recall is promising for future work, but we'd need to have some additional applications to map the grobid output for automated cutting out of tables. I do note that it will be a large increase in efficiency, after having cut out the tables myself (somewhat tedious; @petermr can probably corroborate this).

petermr 2017-01-31 23:53:56 UTC #6

NIcely presented. Summary 61%. Were there examples pf partial tables or tables with non-tables included?

Agreed it's tedious - and you also scraped the PNGs as well! It took me about 5 hours I think. for 172 tables. That's about 10 papers / hour == 6 mins/paper.

I am reasonably optimistic that we can adapt AMI-table to do this. It's the same algorithm and code as for tables, so it's mainly tweaking. And it will be needed for Figures as well.

petermr 2017-02-01 00:07:21 UTC #7

The first extracted table corpus-oa-pmr/10.1007_s00213-015-4198-1/pdftable/doubleTable1.html is an excellent example. It's an APA-like table with Title/Header/Body/Footer with split columns and subtables. Everything has been identified and compartmentalised. Note:

There are 4 top-level columns, split into 5. These can be semantically captured
Some rows have 5 cells, others have only a leading cell.
The full rows are indented.
3/4 are consistent with three subtables (we have "striped" the background to show them)
The only failing is the subscripts - which we intend to tackle in the next week.

petermr 2017-02-01 00:30:58 UTC #8

More tables - in serial order - with observations

This is a table with uncleasr semantics in the headers. It's not immediately clear what to do with them

Perfect simple table

Perfect capture of multisplit column tables (bar subscripts). Note the capture of (R) symbol

Perfect capture of dense table including bold characters

Changing PubStyle. This has a box, but AMI-table parses it well

triple nesting of columns

petermr 2017-02-01 00:35:07 UTC #9

Problem tables

Running text in cells.

This is almost impossible to parse as it has nested paragraphs of discourse. We capture the character stream OK, but the semantics are probably not worth pursuing

Double nesting of subtables

The semantic of this are probably too obscure. (The character stream is captured)

whitespace problems

The algorithm has proposed a gap for a slightly wide inter-word space and added a column. This can probably be solved by heauristics to match columns and headers

petermr 2017-02-01 00:51:56 UTC #10

failing PubStyles

outer box not programmed

impenetrable semantics

(Even if we programm the outer box the contents are too complex to semantify)

Gridded table not yet supported

petermr 2017-02-01 17:52:35 UTC #11

Browsable tables from `AMI-table`

see https://github.com/ContentMine/cm-ucl/blob/master/tables.csv for an automatically generated CSV of all the tables with their original images and the AMI-table output for comparison and comment.

The directory https://github.com/ContentMine/cm-ucl/tree/master/corpus-oa-pmr-show holds all the browsable tables with the pair{raw + extracted} in "*/doubleTable%d.html".

Will soon post a *.xslx or similar stylesheet.

chjh 2017-02-03 13:46:44 UTC #12

There were several continuation tables included. GROBID identified several non-tables as well, due to table format being used to present objects in a structured manner (for example).

petermr 2017-02-06 23:45:59 UTC #14

Pubstyles will probably be based on (a) an example document with (b) regexes for variation in the components. Will do this as soonas I have done the HTML

PDFBox. on my todo. Suspect it's a single hair-tearing day

Good about the validation corpus.

petermr 2017-02-06 23:48:55 UTC #15

Have written code that creates the HTML for tables. Works for simple tables. Fails for subscripts and nested tables. see table%d.annot.svg.html files.

Will gradually tackle the errors in the following rough order:
* sub/superscripts
* indents for subtables
* boxes and grids

see example of perfectly extracted table!

chjh 2017-02-07 09:15:14 UTC #16

Oh damn one of my posts got deleted because I flagged it first (accidentally posted too soon), then changed it to show final post... Missing a post somewhere around #12..

petermr 2017-02-07 11:00:30 UTC #17

There is a hidden one (in red) from you. Is that the one? If so I will repost

chjh 2017-02-07 11:27:14 UTC #18

Just called with @petermr and agreed upon a few points:

Chris will run the entire pipeline to retrieve tables on the validation corpus independently from @petermr, documenting all the steps (in due time)
@petermr and I will independently walk through all tables and categorize them as
a. in scope
b. out of scope
c. questionable/unclear
This will help build a clear table definition wrt scope as well. The files are all available in table-imgs-scope/ folder and coding is done in data/table-imgs-scope.csv. @petermr it is important to both do this locally first so we have independence (opening the table-imgs-scope/ files might be a bit of a hassle with the extension, but should work)
@petermr will list some development problems and @chjh will coordinate with UCL to rank-order them based on priority. This will structure possible future development or if there is time left on this project to start tackling these.
Agreed that we are limited in generalizing the findings from this project (focused on psychology/neuroscience journals mainly) to other, vastly different fields.

chjh 2017-02-07 11:27:28 UTC #19

Yes please do repost it

petermr 2017-02-07 11:45:22 UTC #20

deleted message from @chjh

chjh:

This is looking really promising. @petermr, will there be an argument in ami-table to specify the pubstyle doc if it is not default (e.g., --pubstyle FILE)? I get the concept of the pubstyles but am still wrapping my head around how to implement. I really like the template of the pubstyle. Here are some of my progressions on other areas.

PDFBox

Major API call differences between 2.0.X and 1.8.X here . This is primarily useful if at somepoint ami will be migrated.
Seems like cropping is doable via cropBox; here is an example of code. I think this is Java code, but how grobid coordinates map on to this I do not know at the moment (I guess they directly map on there? Worth trying out at some point).

Test corpus

Now wrote up the description of the test corpus, see here

Validation corpus

Starting to set this up, but will be blind to @petermr so I will only discuss high level progress on this, no specifics.

Building the stack

I've posted a Github issue documenting my findings. See here.

Fala 2017-02-07 11:47:57 UTC #21

Hi Chris,

Please can I have some clarification regarding point 4 and the generalisability: does this mean that the PubStyle differs between journals from the same publisher? Because for the preclinical work as most journals come from a small handful of publishers I would have expected it to be generalisable?

If it would be helpful I can look for non-neuroscience preclinical PDFs/ journals.

Many thanks,
Fala