Connect your NLP models to tagtog using webhooks

By Jorge Campos

When you finish this article, you will understand:

• What is a webhook
• How to connect your model to tagtog (or other services) using webhooks
• How to test webhooks locally
• How using webhooks will make the training of your model more accessible

First of all, what is a webhook?

A webhook is a notification mechanism that allows your system to receive events from a different service in real-time. Event notifications are sent via HTTP POST requests to an endpoint defined by you.

Think of it as an SMS notification. You make a change to your bank account details (event), and you receive an SMS asking you for confirmation (event notification). When you opened your bank account, you gave your phone number (endpoint), so they know how to reach you.

Webhooks

In tagtog, there are two events that trigger a notification:

• Import new document: a notification is sent once a user uploads a document. Use this notification, for example, to annotate the document using your models and push the annotated document back.
• Save a document: a notification is sent once a user saves a document. Use this notification, for example, to get the annotations done by your team and add them to your training data.

In the docs, you find the detailed specification for webhooks.

An example

We will build a simple app in Python that attends the event notifications.

Let's go step by step. Our use case:

1. A non-technical user uploads a document to their project
2. Our app receives a notification, annotates this document and, push it back
3. The document is shown annotated to the non-technical user

Let's start by creating the app and serve it using Flask.

Create a virtual environment

Before starting a new Python project, we create a virtual environment to isolate the new project from other project's dependencies.

# Create the virtual environment myenv
# Python 3.3+
$ python3 -m venv myenv

# Activate the virtualenv (OS X & Linux)
$ source myenv/bin/activate

Install Flask

Flask is a lightweight WSGI web application framework. Flask will help us serve the application and make it reachable. We’ll use pip for package management.

$ pip install Flask

Install spaCy

To annotate the document, we will use a spaCy model (en_core_web_sm). Please check this post for more details about how to integrate tagtog and spaCy.

$ pip install -U pip setuptools wheel
$ pip install -U spacy
$ python3 -m spacy download en_core_web_sm

Create a minimal Flask app

Create an app.py file with the following code:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
  return "Hello, World!"

if __name__ == "__main__":
  app.run()

Here we tell Flask to respond with a "Hello, World!" message once a request reaches our web app at the "/" path.

$ python3 app.py

Now you can open http://127.0.0.1:5000 in your browser to receive the “Hello World!” response. For more details about this sample go to the Flask docs.

Create our custom Flask app

Following the same logic, once an event notification reaches our web app, we want to tell Flask to get the related document, annotate it and send it back to tagtog.

This is what our app will do upon receiving an event notification:

1. Get the document’s content using the document ID coming within the POST event notification.
2. Annotate the text using the model and transform the spaCy annotations into the ann.json format.
3. Push the annotated document back

Setting and testing a webhook

First, let's create a project. You can do this by using the Cloud version; it is as simple as signing up.

Our model will extract from text three entity types: people, organizations, and money terms. We create these entity types in our project's settings.

Now, let’s add a webhook to our project. But before that, where tagtog should reach our app? Your app is reachable within your network in your development environment, but it might be tricky to get tagtog (or other external services) accessing it.

To easily expose your app to the world, you can use ngrok. This tool forwards the incoming requests to your local app. Run your Flask app and then start ngrok:

$ ./ngrok http 5000

This command, among other data, will print the forwarding URLs to reach your app from the outside.

Now we are ready to define the webhook in the Webhooks section under the project's settings:

1. Endpoint URL: write here the forwarding URL provided by ngrok
2. Payload format: select tagtogID to send the document's ID as a result of an event. You can choose ann.json if you prefer to receive the annotations of the document instead of the ID.
3. Trigger only if change originates in the GUI: yes, we want to send an event notification only when the user uses the web app (not the API).
4. Authentication: in this example, we don't use any authentication mechanism, meaning our app doesn't require an authenticated request.

From the moment you save the webhook, an event notification is sent to the endpoint URL as a result of an event

OK, time to test. Make sure you are running your Flask app and ngrok, then go to the documents of your project and import a piece of text.

🪄Magic! Your text is automatically annotated by your spaCy model.

Just import some text, and it gets automatically annotated

Other use cases

• Document review: the end-user uploads a document (e.g., PDF), your model annotates the document, and push it back annotated, helping the user to complete the review.
• Let your domain experts train your model: the end-user uploads text to tagtog, your model annotates the text, and push it annotated. The domain expert corrects the required annotations, and your model receives this feedback to train itself further.

Please reach out for any questions!

I hope you enjoyed the example! The whole project is open so take a look🍃 at https://www.tagtog.com/jorge/demo-webhook/

👏👏👏 if you like the post, and want to share it with others! 💚🧡