AI Management

Overview

As part of its security suite, Link11 includes robust bot management, including tracking of individual types of bots. There is some additional granularity for managing AI crawlers.

The AI Management Dashboard

This Dashboard provides analytics that reveal their activity and composition of AI crawler activity. It is divided into three sections, in this order (top to bottom):

• The date/time range for the analytics display, similar to the Query Specification section of the main analytics Dashboard.

• A graphical summary of all crawler activity for the specified time range. Dragging the cursor over a portion of the chart will adjust the display to focus only on this time period.

• Activity by individual crawlers, and the disposition of their requests.

The Passed and Blocked metrics are the number of requests Link11 passed to the origin or blocked, respectively. For these purposes, requests which made it to the origin but were blocked there are counted as "passed."

At this time, it is not possible to break out statistics for Copilot and Gemini crawlers, as they are indistinguishable from the standard Bing and Google spiders. However, visitors arriving from these services (along with other AI services) can be identified, enabling their requests to be handled differently if desired. See AI Referrals, below.

How to Manage AI Crawlers

The AI Management Dashboard is for displaying crawler activity.

The crawlers' activity can also be managed; this is configured in a different part of L11WAAP, based on a dedicated set of Global Filters that identify them.

To control crawler activity within the protected systems, two approaches can be taken, depending on the desired scope.

Global blocking

To globally block all requests for a specific crawler, edit the appropriate Global Filter and set its Action parameter to Global filter block. For example, the AI ChatGPT Filter can be used to block ChatGPT's crawler.

Path-dependent management

To block or otherwise manage AI bot requests for specific parts of the system, follow this procedure:

• Open the appropriate Global Filter for editing (the one which identifies the crawler in question).

• Ensure that the Filter’s Action parameter is set to monitor (tag only).

• In the Filter’s Tags field, note the specific tag that the system will attach to all requests generated by the crawler. Or, you can define your own.

• Next, for each part of the protected system where crawler activity should be controlled, determine the ACL Profile that is in effect. (ACL Profiles are assigned to paths by Security Policies.)

• Add the crawler’s Tag to the appropriate column within each ACL Profile. (To block the requests, the Enforce Deny column can be used.)

AI Referrals

Sometimes it is desirable to set up unique management for visitors that originate from AI services.

Link11 includes specific "Referral" Global Filters for this purpose. For example, the AI ChatGPT Referral Filter will match every request that has its referer header set to https://chatgpt.com/.

Using the procedures explained above, these Filters can be used to block, or trigger other specific handling, of these requests.