UseCookie -- defines whether to use per-session cookies during crawling



UseCookie {yes | no}


UseCookie defines whether to use HTTP cookies during crawling.

Some sites may require cookies to crawl dynamic pages, e.g. database driven documents, CGIs, etc. Once indexer hits a dynamically created page, the site can request setting up a cookie containing session id or other kind of session information, and require that cookie for each subsequent request to other dynamically created documents.

When UseCookie is set to yes, indexer remembers all cookies set by the crawled sites in RAM, and uses them for subsequent requests to the same servers (taking into account the domain and path cookie attributes).

The default value is no.


UseCookie yes

See also

HTTPHeader, UseSitemap.