# Adding public URLs as knowledge sources for automations

> In addition to your public knowledge base, you can make other public content available to the AI Agent.

Source: https://help.kustomer.com/en_us/add-public-urls-r1IqUxWKJg

Last updated: 2026-06-01T14:30:32.119Z

Knowledge Sources let you expand what an automation can reference beyond your public Help Center. You can add public webpages, sitemaps, or a Zendesk Help Center.

This article explains how to add, manage, and troubleshoot knowledge sources.

**Who can access this feature?**

**User types**

Admins can access the Knowledge Sources page.

  

### In this article

*   [Add knowledge sources](#add)
*   [Manage your knowledge sources](#manage)
*   [Nightly re-indexing and URL filtering](#reindexing)
*   [Editing knowledge sources](#editing)
*   [Filtering out URLs from your knowledge sources](#filtering)
*   [Deleting knowledge sources](#deleting)
*   [FAQ](#faq)

### Add knowledge sources

You can add public content so your automation can reference it when responding to customers.

**Supported source types**

You can add:

*   Individual public URLs (single pages).
*   Sitemap URLs (to ingest multiple pages at once).
*   Zendesk Help Center URLs (to import an entire help center).

### Before you begin

Some public webpages block web crawlers and search engines. These pages cannot be used as a data source.

In addition:

*   Some websites load content dynamically (for example, after scrolling or authentication).
*   Our ingestion system supports many dynamic pages, but it may not capture all content exactly as displayed in a browser.
*   Automations use only text content. They cannot use text from images, videos, alt text, or embedded files.

If the content is critical, add it directly to your Help Center or another structured source that your automation can reliably access.

### Adding new knowledge sources

To add a knowledge source:

1.  From the left side navigation, click **Kustomer AI**. The **Concierge** screen opens.
2.  Click **Knowledge Sources**.
3.  Click **Add source**.
4.  Select the source type.
5.  Enter a name and description for the source.
6.  Click **Add URL**, then enter the URL in one of the following formats:
    *   Public URL: `https://www.example.com`
    *   Sitemap: `https://example.com/sitemap.xml`
    *   Zendesk Help Center: `yourcompany.zendesk.com/hc`
7.  Click **Create**.

### **Manage your knowledge sources**

After you add a source, you can view its status on the **Knowledge Sources** screen. Sources with a status of Indexed are ready for AI Agents to use.

![](https://cdn.kustomerhostedcontent.com/media/570fad9d9001bc1000163b28/0c2c5cfe06db206badf7e40e6e9d000a.png)

If the status does not update, refresh your browser.

In addition:

*   Sitemap and Zendesk collections are capped at 5,000 entries.
*   Manual Public URLs do not have an explicit count limit, but the request payload is limited to ~5MB.
*   AI Agents view you page in plain text, ignoring images like infographics and screenshots.
*   AI Agents will not view embedded content.
*   Sitemap and Zendesk collections are refreshed daily. Public URLs need a manual refresh when content is added.

**Note:** Click the refresh button next to each knowledge to update the sync status. Syncing usually occurs within 1 minute.

### Nightly re-indexing and URL filtering

Public URL knowledge sources are **re-indexed nightly**, consistent with **Zendesk** and **sitemap** knowledge sources. When syncing fails, you can retry syncing, view reasons for the failure, and use sitemap search filtering to view only URLs that failed to sync and retry syncing one by one.

### Editing knowledge sources

To add additional URLs to an existing source:

1.  On the **Knowledge Sources** page, locate the source.
2.  Click **Edit**.
3.  Add or update URLs.
4.  Click **Save**.
5.  Click **Options**, then select **Refresh** to sync changes.

### Filtering out URLs from your knowledge source

When uploading a knowledge source sitemap, you may wish to exclude URLs (either certain URLs or URLs by groups). You can add inclusion or exclusion rules to determine when your sitemap syncing will include or exclude URLs.

To filter URLs from sitemap syncing:

1.  From the **Knowledge Sources** page, add your sitemap.
2.  Click **\+ Add Filter**. A new URL prefix rule appears.
    1.  Select **Include** to include URLs that match the defined prefix.
    2.  Select **Exclude** to exclude URLs that match the defined prefix.

**URL filtering** will include or exclude specific URLs in order during indexing.

### Deleting knowledge sources

To delete a source:

1.  On the **Knowledge Sources** page, locate the source.
2.  Click **Options**.
3.  Select **Delete**.

### FAQ

**Q. How long does it take to sync a new knowledge source?**

In most cases, a new knowledge source is synced within a minute. To confirm it was synced, refresh your browser.

**Q. My page has an infographic. Will automations use that to respond to a customer?**

AI Agents only use text found on a page to respond. Any text found in videos, images, or descriptive alt text will not be used.

**Q. How does Kustomer Concierge handle knowledge sources based on brands?**

Kustomer Concierge can identify knowledge articles and external knowledge sources by brands based on conversation context. The AI first looks for a matching brand and uses all knowledge source content associated to that brand. If a knowledge source has no brand, the AI will include it regardless of the brand the conversation is associated to and perform a Retrieval-Augmented Generation (RAG) search across all included sources. 

**Q. I embedded a Google sheet on one of my pages. Can automations read the embedded sheet?**

No, content embedded in a page cannot be used to respond.

**Q. How often do sources refresh?**

Collections based on website sitemaps and Zendesk Help Centers are refreshed automatically every day. Individual public webpages need to be manually refreshed.