Review Mode¶
Once you have labeled data from your annotators and models, you’ll want to review it to know if it is any good. LightTag’s Review Mode makes this easy.
Span Annotation Review¶
The goal of review is to pick one right answer. For example in the case below one or more annotators made the following annotation:
A span annotation displayed in review mode. One or more Annotators may have made this annotation, but LightTag will show it once¶
Sometimes your annotators will disagree, either by either choosing different tags for the same span, or annotating overlapping spans. In that case, you’ll need to resolve the conflict to choose a correct answer:
In this case, both annotators agreed on the tag, but one included the $ and the other did not. We need to decide which is the correct annotation¶
What Gets Reviewed¶
An annotated span will only need to be reviewed once unless a new annotation conflicts with it. So if you reviewed the work of your first annotator, and then asked another group to annotate the data you would only have to review new annotations, that is annotations that the original annotator hadn’t made.
Tip
Many LightTag users will start a project by labeling and reviewing a small dataset themselves, then ask their team to label the same dataset. The teams performance will be available instantly in the analytics section, and those metrics can be used to evaluate the quality of your guidelines and/or the quality of each team members work.
Tip
Another common use case is to upload suggestions from multiple models on the same dataset and then review their outputs at once. You can use this reviewed data to train a new model on a large amount of high precision data very fast.
How To Review¶
While in review mode, you’ll be shown one annotation (or conflicting group of annotations) at a time from the dataset and schema you chose. LightTag will fetch a batch of examples for you to review, and will ask you to save your work at the end of each batch.
You can configure the batch size with the batch size slider on the left hand side, and save your work at any time by pressing the save button.
Accept an annotation with the checkmark button or by pressing A
Reject an annotation by clicking the trash can or pressing X.
Go Back by pressing the Undo button or ←
Skip an annotation by pressing space
Resolve A conflict by clicking the checkmark next to the correct answer or pressing the number associated with it.
Reject All conflicting annotations by pressing the REJECT ALL button or Shift-X