Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Server Farms 2026 Continuing discussion of hosting and data center IP ranges |
6 | blend27 | 3:27 pm Feb 21, 2026 | ||||
| Server Farms 2025 Continuing discussion of hosting and data center IP ranges |
5 | not2easy | 5:09 pm Mar 1, 2025 | ||||
| AI UAs - Identifying the crawlers AI uses |
1 | not2easy | 5:38 pm Oct 18, 2023 | ||||
| At Home Without the Robots: 2023 edition |
5 | lucy24 | 11:39 am Jul 29, 2023 | ||||
| At Home with the Law-Abiding Robots (2021) |
2 | lucy24 | 8:32 pm Sep 6, 2021 | ||||
| At Home with the Robots: 2020 edition |
11 | lucy24 | 2:48 pm Aug 13, 2020 | ||||
| At Home with the Robots: 2019 edition |
3 | lucy24 | 3:52 am Mar 13, 2019 | ||||
| At Home with the Robots: 2017 edition Critique of the Year's Active User Agents |
15 | lucy24 | 3:52 pm May 18, 2017 | ||||
| User Agents of Programming Libraries and Command Line Tools Common user agents |
14 | incrediBILL | 8:54 pm Apr 29, 2008 | ||||
| what do they want? [3] ( 1 2 3 ) |
88 | lucy24 | 9:40 pm Apr 29, 2026 | ||||
| 0 | 12:00 am Jan 1, 1970 | ||||||
| My website was vacuumed up by OpenAI |
3 | SumGuy | 6:27 pm Apr 13, 2026 | ||||
| Microsoft, AS8075. Google, AS396982. |
7 | shawnb61 | 2:59 pm Apr 10, 2026 | ||||
| Fixed Line ISP- gtt.net Are they Hu-men, AS3257? |
13 | blend27 | 3:25 am Apr 9, 2026 | ||||
| SleepBot |
6 | Bewenched | 3:59 pm Apr 7, 2026 | ||||
| "traceparent" header ...part of OpenTelemetry SDK |
1 | blend27 | 8:07 pm Apr 1, 2026 | ||||
| PerplexityBot What's the verdict? |
3 | SumGuy | 12:16 am Mar 30, 2026 | ||||
| Youbot (new crawler?) and google-inspection tool |
1 | SumGuy | 11:50 pm Mar 13, 2026 | ||||
| FeedFetcher-Google running amok |
7 | Pfui | 5:12 pm Mar 9, 2026 | ||||
| Procter & Gamble bot-running (not just soap 'n' stuff) |
9 | Pfui | 11:57 pm Feb 19, 2026 | ||||
| A useful resource for bot / proxy detectives |
2 | SumGuy | 7:26 pm Jan 6, 2026 | ||||
| Softblocking 35.0.0.0/8 |
22 | IJustWanttoPost | 11:26 pm Dec 21, 2025 | ||||
| FBI Seeks To Unmask Anonymous Web Archiving Service Owner Archive.today |
4 | SumGuy | 11:46 pm Dec 17, 2025 | ||||
| Is anyone getting hits from 121.99.4.0/24 ? |
15 | SumGuy | 3:28 am Dec 16, 2025 | ||||
| Where is grok bot / xAI? Does grok use anonymous VPNs to crawl sites? |
18 | SumGuy | 9:03 pm Dec 9, 2025 |
