Moderate comments on your website using Azure Content Moderator

In this post we are going to see in detail about how Azure Content Moderator can be used to moderate comments on your website/blog. For those who prefer to view code instead of reading through blog post, here is the GitHub Link to my article.

First, what is Azure Content Moderator? According to Microsoft,

The Azure Content Moderator API is a cognitive service that checks text, image, and video content for material that is potentially offensive, risky, or otherwise undesirable.

In simple terms, it is a cognitive service that can be used to validate and moderate various user generated contents. In this post we focus on moderating comments aka text.

Where is it useful?

Blocking harmful comments with just the help of a blocklisted words might allow some content to slip through it. This service allows you detect and prevent them and also do additional things like ensuring that Personal info is not posted in public forums or comments.

Setting up things on Azure

We will start by setting up things on Azure first and then proceed to the coding.

1. It’s pretty obvious by now that you will need an Azure account.
2. Login into your Azure Portal and create a Content Moderation service.
3. Once the resource is created, go to the resource.
4. Navigate to the Keys and Endpoint settings.
5. You will see two key values here. Any one value can be used in place of another. These redundant keys are here to make key regeneration easier. Note down anyone key and the endpoint. We will need it later.

Setting up code

For the purposes of this demo, I will be using a C# console application. The application will read my input, analyse it and take corresponding action. You can easily modify this code for rest API or any other format that fits you best.

1. My first step will be to create a Console application.
2. Install the Nuget package Microsoft.Azure.CognitiveServices.ContentModerator.

Authenticating with Azure service

1. I am creating a new ModerationLib class to contain all the corresponding functions.
2. Create two variable to hold the key and endpoint we noted down earlier.

private static readonly string SubscriptionKey = "EXAMPLE_KEY";
private static readonly string Endpoint = "https://example.cognitiveservices.azure.com/";

3. Add the following references to the ModerationLib class.

using Microsoft.Azure.CognitiveServices.ContentModerator;
using Microsoft.Azure.CognitiveServices.ContentModerator.Models;

4. Create a function that creates a client that can authenticate into the Azure service.

private ContentModeratorClient Authenticate()
{
   var client = new ContentModeratorClient(new ApiKeyServiceClientCredentials(SubscriptionKey))
   {
       Endpoint = Endpoint
   };
   return client;
}

Calling the service to Analyse the text

The function we need to call to analyse the text via Azure is ContentModeratorClient.TextModeration.ScreenText. It requires the text to be passed as stream. We also need to use a ContentModeratorClient that has key and endpoint details i.e) the one we created in the previous step. Here is a simple wrapper function I use for Calling the Azure service,

private Screen CallAzureModerator(ContentModeratorClient client, string text)
{
   var textBytes = Encoding.UTF8.GetBytes(text);
   using var stream = new MemoryStream(textBytes);
   return client.TextModeration.ScreenText("text/plain", stream, language: "eng", autocorrect: false, pII: true, listId:null, true);
}

Notice the parameters I pass to the ScreenText function.

• language - I pass the language of the text, if you don’t know the language you can set the Azure Service to auto detect the language for you.
• autocorrect - This parameter indicates whether or not you want Azure to perform auto correction on text.
• pII - Personal Identifiable Information. If set to true it finds the PII within the text you send.
• listId - You can use this to have a custom blocklist and block words from them.

Processing the response

Here is a response in JSON format I got for a sample text,

{
    "OriginalText": "Hello My email is [email protected]",
    "NormalizedText": "Hello  email  [email protected]",
    "AutoCorrectedText": null,
    "Misrepresentation": null,
    "Classification": {
        "Category1": {
            "Score": 0.0016730600036680698
        },
        "Category2": {
            "Score": 0.20461790263652802
        },
        "Category3": {
            "Score": 0.0669788047671318
        },
        "ReviewRecommended": false
    },
    "Status": {
        "Code": 3000,
        "Description": "OK",
        "Exception": null
    },
    "PII": {
        "Email": [
            {
                "Detected": "[email protected]",
                "SubType": "Regular",
                "Text": "[email protected]",
                "Index": 18
            }
        ],
        "SSN": [],
        "IPA": [],
        "Phone": [],
        "Address": []
    },
    "Language": "eng",
    "Terms": null,
    "TrackingId": "578522d7-9e53-418b-abff-985c186e7377"
}

This provides us with a lot of information. For example, the index at which every PII starts is given to you. You can use it react the personal info and then process the comment. Here is how I intend process the info,

public void ModerateText(string text)
{
    var azClient = Authenticate();
    var azResult = CallAzureModerator(azClient, text);

    // TODO:
    // Check if the text contains a blocklisted word using List

    if (azResult.PII != null && (azResult.PII.Email.Count > 0 || azResult.PII.Phone.Count > 0))
    {
        // Don't post the comment and inform the user
        Console.WriteLine("Contains Personal Data");
    }
    else if (azResult.Classification.ReviewRecommended == true)
    {
        
        if (azResult.Classification.Category1.Score > 0.9)
        {
            // Automatically reject comments with greater confidence in inappropriate content.
            // Category 1 refers to potential presence of language that may be considered sexually explicit or adult in certain situations
            // There are a total of 3 categories.
            // Ref: https://docs.microsoft.com/en-us/azure/cognitive-services/content-moderator/text-moderation-api#classification
            Console.WriteLine("Rejected the comment");
        }
        else
        {
            // Send the comment to your manual review system
            Console.WriteLine("Comment held for review");
        }
    }
    else
    {
        Console.WriteLine("Comment Published");
    }
}

You can even combine them with your existing custom workflows such as finding and removing links, etc…

What’s next These blog posts are WIP. Links to these blog posts will be updated once they are published.

1. Creating & Managing custom blocklists in the Azure Content Moderator service
2. Using the review system within Azure for Manual reviews

Useful Links

• Full Source Code - GitHub
• Content Moderation Documentation
• Language Support
• Pricing for Azure Content Moderation service