Your website’s content was not fetched correctly, so your AI can’t use it.
This is usually caused by URL issues, access restrictions, or crawl settings.
1. Check the URL #
What to verify
- URL is correct and complete
- Includes https://
- Page loads in your browser
Fix:
- Re-enter the correct URL
- Try opening it manually to confirm
2. Page is not publicly accessible #
Problem
- Page requires login or authentication
- Blocked by permissions
Fix:
- Use only publicly accessible pages
- For private content → upload files or use Q&A instead
3. Wrong crawl scope selected #
Check your option
- This page only
- Entire site
- Sitemap
Fix:
- Start with This page only to test
- Expand later once it works
4. Website blocks crawlers #
Problem
- Site has restrictions (robots.txt, firewall, bot protection)
Fix:
- Allow crawler access on your site
- Disable strict bot blocking (if safe)
5. JavaScript-heavy pages #
Problem
- Content loads dynamically (not visible in page source)
Fix:
- Use static pages where possible
- Or upload content as files
6. Crawl completed but not trained #
Common issue
Check:
- Are URLs in New?
Fix:
- Click Refresh Agent
7. Sitemap issues #
If using sitemap
- Sitemap may be missing or broken
Fix:
- Check sitemap URL manually
- Ensure it lists valid pages
8. Large site or timeout #
Problem
- Too many pages or slow response
Fix:
- Crawl smaller sections first
- Use subpages instead of full site
Quick checklist #
- URL is valid and accessible
- Page is public
- Correct crawl option selected
- Crawl completed successfully
- Data trained (Refresh Agent)
Tip #
Start small:
Crawl one page → Refresh Agent → Test → Then expand.
Next, try crawling a single page, click Refresh Agent, and test Preview.
If you have any questions, concerns, or need further clarification, please contact support@highengage.com for assistance.
