PeerFoundry

Bot

TypedPull fetches a public page only when an agent asks for fields from that page.

User-Agent: PeerFoundry-TypedPull/0.1 (+https://peerfoundry.ai/bot)

Before a fetch, TypedPull reads robots.txt for the product token PeerFoundry-TypedPull. A 5xx or an unreachable robots.txt is a disallow. A 404 is an allow. Crawl-delay sets the minimum gap for that host.

If robots.txt, a response header, or a robots meta tag says ai-input=no (or an aipref no for AI input), TypedPull does not return fields. The log stores the host and the reason only.

TypedPull sends no cookies and no Authorization header. It does not solve CAPTCHAs, pass paywalls, or use rotating proxies.

Opt out: email abuse@peerfoundry.ai with the host name. A host opt-out is added to the denylist within 24 hours. A cease-and-desist or a specific block notice is added as soon as we receive it. We do not retry that host.

PeerFoundry · Bot · Terms · Privacy · Status