How to rank & review your annotators
Some useful tips and suggestions that may be useful for anyone who is managing a team of annotators
In our latest update, we have included a new 🆕 feature to our search engine, which allows users to filter documents based on whether a given user has confirmed them (or any user has confirmed them). This feature, despite it may not look like such a big deal, it can be used for many different things. In this post I will discuss some of the applications 🚀.
First, let’s see how it is possible to use the query in the tagtog web application. You just need to navigate to a project (e.g. https://www.tagtog.net/tagtog/Neeq_Annual_Reports). Then input this query in the search bar located at the top left:
username is the name of the user you want to know which documents has completed. If you use
* (“star”) instead of an username, you will get the documents any user has completed ✅.
You can also use logical operators, such as
NOT to make more complex queries. Here you can see an example of a query which retrieves the documents that have been completed by the users demo or uxio:
And now, we will get to the first of the applications I mentioned before. Let’s review step by step a script that allows you to get a report 📘 of how many documents any member of a given project have completed.
Before doing anything else, I define some variables that will be useful later on:
First, I get a list of project members, who have annotation permissions, so I exclude those users who just have the role reader assigned:
Then, I have written this function, which uses the search engine API to retrieve the documents completed by a given user:
Finally, we just need to output the results in a practical way. I have chosen to print the results to a txt file, but it could be any other format 📜.
Below is the output. As you can imagine, this allows you to periodically track and rank which member is doing more annotations:
Moreover, another possible application of this feature would be to get a list of documents which have been annotated by any user 📝. This is very useful for those members who review the annotations of others.
For this, we will use a similar function to the previous one, but we will use
* instead of an username:
Afterwards, I iterate over the list of documents I have obtained in the response, and I print the document ids in a different txt file:
And again, this would be the output:
Curious to apply similar searches to your project? Click here to obtain this post’s full script’s code. Moreover, we have other code samples in our github repository: https://github.com/tagtog/tagtog-doc/tree/develop/examples
I hope you found this new search feature useful! Let us know in the comments your experience 👇🏻.