How to rank & review your annotators

Some useful tips and suggestions that may be useful for anyone who is managing a team of annotators

By Uxío García Andrade(👐Open Link to this article)

In our latest update, we have included a new 🆕 feature to our search engine, which allows users to filter documents based on whether a given user has confirmed them (or any user has confirmed them). This feature, despite it may not look like such a big deal, it can be used for many different things. In this post I will discuss some of the applications 🚀.

Image for post
Your annotators are working hard. Help them a little.

First, let’s see how it is possible to use the query in the tagtog web application. You just need to navigate to a project (e.g. Then input this query in the search bar located at the top left:


where username is the name of the user you want to know which documents has completed. If you use * (“star”) instead of an username, you will get the documents any user has completed ✅.

Image for post
Searching in the GUI

You can also use logical operators, such as OR, AND or NOT to make more complex queries. Here you can see an example of a query which retrieves the documents that have been completed by the users demo or uxio:

Image for post
Complex query example

And now, we will get to the first of the applications I mentioned before. Let’s review step by step a script that allows you to get a report 📘 of how many documents any member of a given project have completed.

Before doing anything else, I define some variables that will be useful later on:

Image for post
Constant definitions of your project

First, I get a list of project members, who have annotation permissions, so I exclude those users who just have the role reader assigned:

Image for post
Get list of annotating members

Then, I have written this function, which uses the search engine API to retrieve the documents completed by a given user:

Image for post
Getting the number of confirmed documents

Finally, we just need to output the results in a practical way. I have chosen to print the results to a txt file, but it could be any other format 📜.

Image for post
Printing the results in a file

Below is the output. As you can imagine, this allows you to periodically track and rank which member is doing more annotations:

Image for post
Output example

Moreover, another possible application of this feature would be to get a list of documents which have been annotated by any user 📝. This is very useful for those members who review the annotations of others.

For this, we will use a similar function to the previous one, but we will use * instead of an username:

Image for post
Get all the documents completed by anyone

Afterwards, I iterate over the list of documents I have obtained in the response, and I print the document ids in a different txt file:

Image for post
Printing the results to a file

And again, this would be the output:

Image for post
Output example

Curious to apply similar searches to your project? Click here to obtain this post’s full script’s code. Moreover, we have other code samples in our github repository:

I hope you found this new search feature useful! Let us know in the comments your experience 👇🏻.

Needing training data for #NLP? Find it, create it, and share it on: 🍃tagtog

Like birds? Follow us on Twitter 🐦

👏👏👏 Clap if you like the post, and want to share it with others! 💚🧡

The text annotation tool to train #AI. Easy. 🔗

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store