Evaluate NERSchemas

Operation

You can apply an evaluation operation comparing two NERSchemas to calculate the annotation recall, precision and f-score between a NERSchema taken as a gold standard (typically taken from a corpus with manual annotations) and a NERSchema to compare (typically from running a specific algorithm or pipeline). To start the evaluation process, right click on the NERSchema Datatype to be selected as gold standard and select "Evaluate -> NER Schema".

Select NERSchema to compare

A GUI will be launched to choose the NERSchema Datatype (in blue) to be compared with the gold standard selected on the clipboard.

NOTES: Only NERSchemas with the same Normalization process as the one NERSchema gold standard will appear.

Result

An NER schema evaluation report will appear after the evaluation process with two tabs related to overall scores and scores per class type.

In the overall scores tab, the annotation recall, precision and f-score measures are presented for all annotated entities.

In the scores per class type, the annotation recall, precision and f-score measures are presented for each annotated entity class.