From bfee917049ceef8ef8dbf73de635ef50e664bb29 Mon Sep 17 00:00:00 2001 From: Russell Ballestrini Date: Thu, 27 Nov 2025 10:11:10 -0500 Subject: [PATCH] docs: Add ethical web crawler documentation page Created /crawler.html page explaining: - Our user agent (uncloseai.com/1.42) - What we use the crawler for (research, Q&A, content aggregation) - Ethical practices (robots.txt, crawl delays, caching, smart depth) - How to block the crawler - Link to Discord bot Added crawler link to sidebar navigation in inference.html This provides transparency for webmasters and explains our ethical crawling practices in human-friendly terms. --- public/crawler.html | 246 ++++++++++++++++++++++++++++++++++++++++++ public/inference.html | 1 + 2 files changed, 247 insertions(+) create mode 100644 public/crawler.html diff --git a/public/crawler.html b/public/crawler.html new file mode 100644 index 0000000..b27d27e --- /dev/null +++ b/public/crawler.html @@ -0,0 +1,246 @@ + + + + + + + + + Our Ethical Web Crawler | uncloseai.com + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+

uncloseai.

 +

Our Ethical Web Crawler

+
+ +
+ +

About Our Web Crawler

+

Our team uses an ethical web crawler for research and development on the open web. We believe in responsible data collection that respects website owners and follows industry best practices.

+ +

User Agent

+

Our crawler identifies itself with the following user agent:

+ 
uncloseai.com/1.42 (ethical web crawler; +https://uncloseai.com)
+

This clearly identifies who we are and provides a link back to this page for more information.

+ +

What We Do

+

Our crawler is used by our AI systems to:

+ + +

Our Ethical Practices

+

We follow strict ethical guidelines to ensure we're good web citizens:

+ +

1. Robots.txt Compliance

+

We always respect robots.txt files. If your site's robots.txt disallows our crawler, we won't access those pages.

+ 
# Example: Block our crawler from specific paths
+User-agent: uncloseai.com
+Disallow: /private/
+Disallow: /admin/
+ +

2. Crawl Delays

+

We respect crawl delays specified in robots.txt. Our default is 2 seconds between requests to the same domain, but we'll honor any delay you specify:

+ 
# Example: Set custom crawl delay
+User-agent: uncloseai.com
+Crawl-delay: 5
+ +

3. Intelligent Caching

+

We cache fetched content for 7 days by default. This means:

+ + +

4. Smart Depth Crawling

+

We use intelligent depth-based crawling:

+ +

Our crawler automatically decides how deep to go based on whether it found relevant information, so we don't waste resources fetching unnecessary pages.

+ +

5. Query-Aware Relevance

+

We score pages based on relevance to the user's question, prioritizing:

+ + +

Transparency

+

We believe in being open about our crawling practices:

+ + +

How to Block Our Crawler

+

If you don't want our crawler accessing your site, you can block it using robots.txt:

+ 
# Block uncloseai.com crawler entirely
+User-agent: uncloseai.com
+Disallow: /
+

Or use your server's firewall/WAF to block our user agent string.

+ +

Questions or Concerns?

+

If you have questions about our crawler or want to discuss our access to your site, please reach out:

+ + +

Try Our Discord Bot

+

Our ethical web crawler powers the research capabilities in our Discord bot. The bot can:

+ +

Want to try it? Visit our Discord bot page at discord.agents.ai.unturf.com to add the bot to your server and see our ethical crawler in action.

+ + + + + + +
+ + + + + + diff --git a/public/inference.html b/public/inference.html index 42fa278..9a31ca7 100644 --- a/public/inference.html +++ b/public/inference.html @@ -63,6 +63,7 @@
  • Swift Examples
  • uncloseai.js Docs
  • Inference Setup
    • +
  • Our Crawler
  • 🔗 All Languages
  • 📚 Book