Skip to content

Crawler

About CosmeticlaimBot

If you found this address in your server logs, this page explains what the request was.

CosmeticlaimBot fetches public pages when someone asks our scanner to check a site. It is triggered by a person pasting a URL — it does not crawl on a schedule and it does not discover sites on its own.

It identifies itself as:

CosmeticlaimBot/1.0 (+https://cosmeticlaim.eu/bot)

What it does

  • Requests your robots.txt first — before fetching any page, including the one you gave us — and honours Disallow rules for * and for CosmeticlaimBot. If the start page is disallowed we stop and return an error rather than a report. We read the first 1000 rules of a robots.txt; if yours is longer than that, the rules past 1000 are not applied, and you should tell us.
  • Reads at most 10 pages per scan, at most 2 MB each, within a 45-second budget.
  • Follows at most 3 redirects, and only to public http or https addresses.
  • Requests pages sequentially. It is closer to one person browsing than to a crawler.

What it does not do

  • It does not execute JavaScript, submit forms, or follow links behind a login.
  • It does not store your pages. Text is analysed in memory and discarded when the response is sent.
  • It does not index your site or publish anything from it.

One qualification, because the sentence that used to sit here was too strong: sentences from your pages that match a restricted term are sent to a language model — a private gateway at ai-gw.nurme.eu — which decides whether the match is a marketing claim or ordinary commentary. That is at most the matched sentence and the page path, never the whole page, and none of it is stored. The privacy page sets out exactly what is transmitted.

Blocking it

Add this to your robots.txt and it will stop:

User-agent: CosmeticlaimBot
Disallow: /

Blocking by user agent at the edge works too. If you would rather it never reached your origin at all, that is entirely reasonable — the scanner will report that the pages were skipped rather than pretending they were clean. Our crawler comes from 157.180.65.81, so you can block it by address as well as by name; if you need to reach a person about it, write to tere@cosmeticlaim.eu.