How to rank & review your annotators
Some useful tips and suggestions that may be useful for anyone who is managing a team of annotators
By Uxío García Andrade — (👐Open Link to this article)
In our latest update at tagtog, we have included a new 🆕 feature to our search engine, which allows users to filter documents based on whether a given user has confirmed them (or any user has confirmed them). This feature, despite it may not look like such a big deal, it can be used for many different things. In this post I will discuss some of the applications 🚀.
First, let’s see how it is possible to use the query in the web application. You just need to navigate to a project. Then input this query in the search bar located at the top left:
members_anncomplete:usernamewhere username is the name of the user you want to know which documents has completed. If you use * (“star”) instead of an username, you will get the documents any user has completed ✅.
You can also use logical operators, such as OR, AND or NOT to make more complex queries. Here you can see an example of a query which retrieves the documents that have been completed by the users demo or uxio:
And now, we will get to the first of the applications I mentioned before. Let’s review step by step a script that allows you to get a report 📘 of how many documents any member of a given project have completed.
Before doing anything else, I define some variables that will be useful later on:
First, I get a list of project members, who have annotation permissions, so I exclude those users who just have the role reader assigned:
Then, I have written this function, which uses the search engine API to retrieve the documents completed by a given user:
Finally, we just need to output the results in a practical way. I have chosen to print the results to a txt file, but it could be any other format 📜.
Below is the output. As you can imagine, this allows you to periodically track and rank which member is doing more annotations:
Moreover, another possible application of this feature would be to get a list of documents which have been annotated by any user 📝. This is very useful for those members who review the annotations of others.
For this, we will use a similar function to the previous one, but we will use * instead of an username:
Afterwards, I iterate over the list of documents I have obtained in the response, and I print the document ids in a different txt file:
And again, this would be the output:
I hope you found this new search feature useful! Let us know in the comments your experience 👇🏻.
👏👏👏 Clap if you like the post, and want to share it with others! 💚🧡
