Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
S
Semanticextraction
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
7
Issues
7
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Ulrich Rabenstein
Semanticextraction
Commits
d461f401
Commit
d461f401
authored
Apr 27, 2017
by
Ulrich
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
evaluation as csv
parent
b3d5e764
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
33 additions
and
118 deletions
+33
-118
evaluation/evaluation.csv
evaluation/evaluation.csv
+17
-0
evaluation/results.txt
evaluation/results.txt
+0
-110
thesis/tex/implementation.tex
thesis/tex/implementation.tex
+16
-8
No files found.
evaluation/evaluation.csv
0 → 100644
View file @
d461f401
Document,astro-ph0001117,astro-ph0008271,astro-ph0011211,astro-ph0012434,astro-ph0102308,cond-mat0001199,cond-mat0002133,cond-mat0003070,Cond-mat0007243,cond-mat0009225,cond-mat0011175,cond-mat0102209,hep-ex0006009,hep-lat0007038,hep-lat0008012,hep-ph0010254,hep-th0011187,math0101174,physics0007034,physics0011001
correct Qes (TP),51,9,47,5,33,19,3,0,2,5,1,8,33,19,11,11,0,0,5,5
missed Qes (FN),7,5,6,0,3,0,0,0,0,0,0,0,0,2,1,4,0,0,0,0
false Qes (FP),3,3,2,1,5,1,0,5,1,1,2,0,0,31,2,0,1,3,0,2
,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,
Total Documents ,=COUNTA(B1:Z1),,,,,,,,,,,,,,,,,,,
Total TP,=SUM(B2:Z2),,,,,,,,,,,,,,,,,,,
Total FN,=SUM(B3:Z3),,,,,,,,,,,,,,,,,,,
Total FP,=SUM(B4:Z4),,,,,,,,,,,,,,,,,,,
Total Qes,=B10+B11,,,,,,,,,,,,,,,,,,,
,,,,,,,,,,,,,,,,,,,,
Precision,=B10/(B10+B12),,,,,,,,,,,,,,,,,,,
Recall,=B10/(B10 + B11),,,,,,,,,,,,,,,,,,,
F-Score,=2*B15*B16/(B15 + B16),,,,,,,,,,,,,,,,,,,
evaluation/results.txt
deleted
100644 → 0
View file @
b3d5e764
Number of documents: 20
Total:
QEs detected correctly: 267
missed QEs: 28
false QEs: 63
astro-ph0001117:
QEs detected correctly: 51
missed QEs: 7
false QEs: 3
astro-ph0008271:
QEs detected correctly: 9
missed QEs: 5
false QEs: 3
astro-ph0011211:
QEs detected correctly: 47
missed QEs: 6
false QEs: 2
astro-ph0012434:
QEs detected correctly: 5
missed QEs: 0
false QEs: 1
astro-ph0102308:
QEs detected correctly: 33
missed QEs: 3
false QEs: 5
cond-mat0001199:
QEs detected correctly: 19
missed QEs: 0
false QEs: 1
cond-mat0002133:
QEs detected correctly: 3
missed QEs: 0
false QEs: 0
cond-mat0003070:
QEs detected correctly: 0
missed QEs: 0
false QEs: 5
cond-mat0007243:
QEs detected correctly: 2
missed QEs: 0
false QEs: 1
cond-mat0009225:
QEs detected correctly: 5
missed QEs: 0
false QEs: 1
cond-mat0011175:
QEs detected correctly: 1
missed QEs: 0
false QEs: 2
cond-mat0102209:
QEs detected correctly: 8
missed QEs: 0
false QEs: 0
hep-ex0006009:
QEs detected correctly: 33
missed QEs: 0
false QEs: 0
hep-lat0007038:
QEs detected correctly: 19
missed QEs: 2
false QEs: 31
hep-lat0008012:
QEs detected correctly: 11
missed QEs: 1
false QEs: 2
hep-ph0010254:
QEs detected correctly: 11
missed QEs: 4
false QEs: 0
hep-th0011187:
QEs detected correctly: 0
missed QEs: 0
false QEs: 1
math0101174:
QEs detected correctly: 0
missed QEs: 0
false QEs: 3
physics0007034:
QEs detected correctly: 5
missed QEs: 0
false QEs: 0
physics0011001:
QEs detected correctly: 5
missed QEs: 0
false QEs: 2
thesis/tex/implementation.tex
View file @
d461f401
...
...
@@ -547,18 +547,26 @@ at the end:
\ednote
{
Discuss with Michael
}
\subsubsection
{
Quantitative Evaluation
}
\ednote
{
do this
}
\begin{itemize}
\item
how many documents were handled successful?
\item
annotate samples in KAT and calculate F-Score
\end{itemize}
The implementation was tested on a set of about 35000 documents and could successfully process nearly all documents.
Fatal errors occurred only in about 150 documents. The overall runtime was about 80 hours on nine 2.00 GHz cores of a Intel Xeon E5-2650
processor. This is equivalent to a runtime of about 70 seconds per document on a single core. It includes preprocessing as well as spotting
and scoring of quantity expressions. Note that, the scoring task involves the execution of Schäfer's declaration spotter which adds
a significant amount of runtime.
The author evaluated the quality of the implementation by manually validating 20 randomly selected documents.
They include 295 quantity expressions of which 267 were successfully recognized (true positives).
We regard the detection of a quantity expression as correct when its highest scored possible meaning reflects the correct
meaning of the expression. 28 quantity expressions were not detected (false negatives) and 63 times expressions were
marked as quantity expressions although they are not (false positives).
In this setup, true negatives are equivalent to non-quantity expressions which were successfully not detected.
However, there is no meaningful quantification of these expressions in this case.
The evaluation results in a precision of
$
267
\,
/
\,
(
267
+
63
)
\approx
81
\%
$
and in a recall of
$
267
\,
/
\,
(
267
+
28
)
\approx
91
\%
$
. This gives us an F-score of about 85
\%
.
\subsubsection
{
Qualitative Evaluation
}
\ednote
{
Describe some errors and their causes here
}
\begin{itemize}
\item
problems in bigger formuals (astro-ph9211002), due to me and due to MathML
\item
\item
abkürzungen (MC)
\end{itemize}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment