Transparency
How we read public websites
To show a shop in the directory we read only what its website already shows every visitor. Here is what we do and what we do not do.
How we identify ourselves
Every request we send carries the User-Agent "Mozilla/5.0 (compatible; AgentStorefrontBot/1.0; +https://storefront.fleetq.net/crawler)", both for shop catalogs and when we read the pages of a site to build a profile. You can recognize us by it in your site logs.
We read only public addresses that visitors see too: product pages, the site search and the publicly reachable catalog interfaces the site itself uses. Some shops load their catalog from a third-party search service. Then we send the same public request the site makes for its visitors, and it goes to the domain of that service.
robots.txt
Before we open a page to write the profile text from, we read the site's robots.txt. We follow the rules for "AgentStorefrontBot", or the rules for all robots ("*") when there are none for us. We do not open a page that robots.txt disallows.
We read a shop's catalog when an agent asks, through the public search or catalog interface the site itself uses. For some shops that means opening the search results page, the page of the product the agent asks about, or the home page to read the category menu. When a shop is added, we check its home page and catalog interface once to find out whether its catalog can be read live. robots.txt is not checked yet for any of these requests; the rule above applies to the pages we read to write the profile text.
If you do not want us to read your site, tell us on the opt-out page. We will hide the shop from the directory and from the search across all shops and stop reading your site.
How often
We send at most one request per second to the catalog of one shop. The exception is supermag.bg: the gap there is 250 milliseconds, because one search reads 32 product lists. If the site answers with 429 or a server error, we wait 1, 2 and 4 seconds and try again.
For a profile we read at most 8 pages of one site.
Search results are kept for up to 1 hour, so repeated questions do not reach the shop every time.
What we never open
We do not open pages for the cart, checkout, user account and login. We do not sign in, submit forms or order on anyone's behalf.
Addresses on internal networks are blocked.
What we do with what we read
We show name, price, stock and a link to the shop page, always with source_url and the time of checking. The customer is sent to the shop to buy there.
For a shop that is not claimed by its owner we accept no orders and do not pose as the shop.