How to rank & review your annotators

Some useful tips and suggestions that may be useful for anyone who is managing a team of annotators

By Uxío García Andrade(👐Open Link to this article)

In our latest update, we have included a new 🆕 feature to our search engine, which allows users to filter documents based on whether a given user has confirmed them (or any user has confirmed them). This feature, despite it may not look like such a big deal, it can be used for many different things. In this post I will discuss some of the applications 🚀.

Your annotators are working hard. Help them a little.

First, let’s see how it is possible to use the query in the tagtog web application. You just need to navigate to a project (e.g. Then input this query in the search bar located at the top left:


where username is the name of the user you want to know which documents has completed. If you use * (“star”) instead of an username, you will get the documents any user has completed ✅.

Searching in the GUI

You can also use logical operators, such as OR, AND or NOT to make more complex queries. Here you can see an example of a query which retrieves the documents that have been completed by the users demo or uxio:

Complex query example

And now, we will get to the first of the applications I mentioned before. Let’s review step by step a script that allows you to get a report 📘 of how many documents any member of a given project have completed.

Before doing anything else, I define some variables that will be useful later on:

Constant definitions of your project

First, I get a list of project members, who have annotation permissions, so I exclude those users who just have the role reader assigned:

Get list of annotating members

Then, I have written this function, which uses the search engine API to retrieve the documents completed by a given user:

Getting the number of confirmed documents

Finally, we just need to output the results in a practical way. I have chosen to print the results to a txt file, but it could be any other format 📜.

Printing the results in a file

Below is the output. As you can imagine, this allows you to periodically track and rank which member is doing more annotations:

Output example

Moreover, another possible application of this feature would be to get a list of documents which have been annotated by any user 📝. This is very useful for those members who review the annotations of others.

For this, we will use a similar function to the previous one, but we will use * instead of an username:

Get all the documents completed by anyone

Afterwards, I iterate over the list of documents I have obtained in the response, and I print the document ids in a different txt file:

Printing the results to a file

And again, this would be the output:

Output example

Curious to apply similar searches to your project? Click here to obtain this post’s full script’s code. Moreover, we have other code samples in our github repository:

I hope you found this new search feature useful! Let us know in the comments your experience 👇🏻.

Needing training data for #NLP? Find it, create it, and share it on: 🍃tagtog

Like birds? Follow us on Twitter 🐦

👏👏👏 Clap if you like the post, and want to share it with others! 💚🧡

The text annotation tool to train #AI. Easy. 🔗

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store