AI Management

Overview

As part of its security suite, Link11 includes robust bot management, including tracking of individual types of bots.

The AI Management Dashboard provides additional granularity for managing AI crawlers specifically.

The AI Management Dashboard

This Dashboard provides analytics that display AI crawler activity. It is divided into three sections, in this order (top to bottom):

• The query filters for the analytics, similar to the Query Specification section of the main analytics Dashboard.

• A graphical summary of crawler activity for the specified time range. Dragging the cursor over a portion of the chart will adjust the display to focus only on this time period.

• Activity by individual crawlers, and the disposition of their requests. The entries here also allow filtering of analytics results, and provide some ability to manage crawler requests.

The Passed and Blocked metrics are the number of requests Link11 passed to the origin or blocked, respectively. For these purposes, requests which made it to the origin but were blocked there are counted as "passed."

At this time, it is not possible to break out statistics for Copilot and Gemini crawlers, as they are indistinguishable from the standard Bing and Google spiders. Note that in the crawler-specific analytics, some entries have a Type of Search crawler: these entries contain combined statistics for the search spiders and the AI bots.

How to Manage AI Crawlers

Crawler activity can be managed by configuring a dedicated set of Global Filters that identify them.

The availability of configuration options are dependent on the user's Access Level.

This configuration can be done in different ways, depending on the desired scope.

Global management

To globally configure handling for all requests from a specific crawler, edit the appropriate Global Filter and set its Action parameter to the desired setting. This will then be reflected in the Action column in the bottom section of the Dashboard.

Example: to globally block the ChatGPT crawler, edit the AI ChatGPT Global Filter and set its Action to Global filter block.

Note that this Action only applies to AI/search crawlers; it does not apply to visitors referred from the AI/search platforms. To manage the latter, see How to Manage AI Referrals, below.

To edit a Global Filter, the most convenient way is to find the appropriate crawler entry in the bottom section of the window. Each entry includes a three-dot menu where:

• The relevant Filter's Action can be directly set to Block or Monitor (tag only).

• Or, if other Filter parameters need to be updated too, the Filter itself can be edited.

The Dashboard shows which Global Filters have blocking or monitoring Actions. If a Global Filter has a different type of Action (for example, https redirect), its Action is displayed as Custom.

After changing a Global Filter's Action, or editing the Filter directly and saving the edits, the updates must be published.

Path-dependent management

To block or otherwise manage AI bot requests for specific parts of the system, follow this procedure:

• Open the appropriate Global Filter for editing (the one which identifies the crawler in question).

• Ensure that the Filter’s Action parameter is set to monitor (tag only). If the Action or any other parameter is changed, save the changes.

• In the Filter’s Tags field, note the specific tag that the system will attach to all requests generated by the crawler. Or, you can define your own.

• Next, for each part of the protected system where crawler activity should be controlled, determine the ACL Profile that is in effect. (ACL Profiles are assigned to paths by Security Policies.)

• Add the crawler’s Tag to the appropriate column within each ACL Profile. (To block the requests, the Enforce Deny column can be used.)

• Publish the changes.

How to Manage AI Referrals

In the Dashboard's bottom section, visitors that originate from AI/search services are reported in the Referral Visits column. Sometimes it is desirable to set up unique management for them.

Referred-visitor management is not related to the Action column, because this column shows the current settings of the crawler-specific Global Filters. Thus, those Actions only apply to crawler requests.

Instead, Link11 includes specific "Referral" Global Filters that can be used for this purpose. For example, the AI ChatGPT Referral Filter will match every request that has its referer header set to https://chatgpt.com/.

These Filters can be used to block, or trigger other specific handling, of these requests, by using the procedures explained above. The only difference is that the editing is performed on the referral-specific Filters, instead of the crawler-specific Filters.

Filtering Query Results

As noted above, the analytics provided by the Dashboard reflect the following:

• The query filters for the analytics display (top section of the window), as in the Query Specification section of the main analytics Dashboard.

• The graphical summary (middle section), which can be narrowed by dragging the cursor over the desired time period.

In the bottom section, each crawler entry includes an eye icon. This toggles the inclusion of that crawler's statistics in the main analytics display.

Each entry also has a three-dot control, which will open this contextual menu:

This provides the ability to:

• Edit the Global Filter for that crawler, or change the Filter's Action, as discussed previously.

• Hide (i.e., exclude the statistics from) all other crawlers.

• Open the Events Log, filtered to display the crawler's requests, for more in-depth analysis.