New AI Framework Powers LinkedIn’s Content Moderation via @sejournal, @martinibuster

LinkedIn rolled out a new content moderation framework that’s a breakthrough in optimizing moderation queues, reducing the time to catch policy violations by 60%. This technology may be the future of content moderation once the technology becomes more available.

How LinkedIn Moderates Content Violations

LinkedIn has content moderation teams that work on manually reviewing possible policy-violating content.

They use a combination of AI models, LinkedIn member reports, and human reviews to catch harmful content and remove it.

But the scale of the problem is immense because there are hundreds of thousands of items needing review every single week.

What tended to happen in the past, using the first in, first out (FIFO) process, is that every item needing a review would wait in a queue, resulting in actual offensive content taking a long time to be reviewed and removed.

Thus, the consequence of using FIFO is that users were exposed to harmful content.

LinkedIn described the drawbacks of the previously used FIFO system:

“…this approach has two notable drawbacks.

First, not all content that is reviewed by humans violates our policies – a sizable portion is evaluated as non-violative (i.e., cleared).

This takes valuable reviewer bandwidth away from reviewing content that is actually violative.

Second, when items are reviewed on a FIFO basis, violative content can take longer to detect if it is ingested after non-violative content.”

LinkedIn devised an automated framework using a machine learning model to prioritize content that is likely to be violating content policies, moving those items to the front of the queue.

This new process helped to speed up the review process.

New Framework Uses XGBoost

The new framework uses an XGBoost machine learning model to predict which content item is likely to be a violation of policy.

XGBoost is shorthand for Extreme Gradient Boosting, an open source machine learning library that helps to classify and rank items in a dataset.

This kind of machine learning model, XGBoost, uses algorithms to train the model to find specific patterns on a labeled dataset (a dataset that is labeled as to which content item is in violation).

LinkedIn used that exact process to train their new framework:

“These models are trained on a representative sample of past human labeled data from the content review queue and tested on another out-of-time sample.”

Once trained the model can identify content that, in this application of the technology, is likely in violation and needs a human review.

XGBoost is a cutting edge technology that has been found in benchmarking tests to be highly successful for this kind of use, both in accuracy and the amount of processing time it takes, outperforming other kinds of algorithms..

LinkedIn described this new approach:

“With this framework, content entering review queues is scored by a set of AI models to calculate the probability that it likely violates our policies.

Content with a high probability of being non-violative is deprioritized, saving human reviewer bandwidth and content with a higher probability of being policy-violating is prioritized over others so it can be detected and removed quicker.”

Impact On Moderation

LinkedIn reported that the new framework is able to make an automatic decisions on about 10% of the content queued for review, with what LinkedIn calls an “extremely high” level of precision. It’s so accurate that the AI model exceeds the performance of a human reviewer.

Remarkably, the new framework reduces the average time for catching policy-violating content by about 60%.

Where New AI Is Being Used

The new content review prioritization system is currently used for feed posts and comments. LinkedIn announced that they are working to add this new process elsewhere in LinkedIn.

Moderating for harmful content is super important because it can help improve the user experience by reducing the amount of users who are exposed to harmful content.

It is also useful for the moderation team because it helps them scale up and handle the large volume.

This technology is proven to be successful and in time it may become more ubiquitous as it becomes more widely available.

Read the LinkedIn announcement:

Augmenting our content moderation efforts through machine learning and dynamic content prioritization

Featured Image by Shutterstock/wichayada suwanachun