engineering/uncloseai.com
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: Add ethical web crawler documentation page

Browse source 
Created /crawler.html page explaining:
- Our user agent (uncloseai.com/1.42)
- What we use the crawler for (research, Q&A, content aggregation)
- Ethical practices (robots.txt, crawl delays, caching, smart depth)
- How to block the crawler
- Link to Discord bot

Added crawler link to sidebar navigation in inference.html

This provides transparency for webmasters and explains our ethical
crawling practices in human-friendly terms.
This commit is contained in:
Russell Ballestrini 2025-11-27 10:11:10 -05:00
parent baa0a1fd7c
commit bfee917049
2 changed files with 247 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

1
public/inference.html
View file

@ -63,6 +63,7 @@
<li><a href="/swift-examples.html">Swift Examples</a></li>
<li><a href="/uncloseai-js.html">uncloseai.js Docs</a></li>
<li><a href="/inference.html" class="active">Inference Setup</a></li>
<li><a href="/crawler.html">Our Crawler</a></li>
<li><a href="/languages" target="_blank">🔗 All Languages</a></li>
<li><a href="https://shop.unturf.com/p/8486f492-a93e-11f0-b477-02dfe05770ee/uncloseai-machine-learning-reference-guide-to-inference-clients" target="_blank">📚 Book</a></li>
</ul>