Spider crawling pattern

Spiders can be divided into three types: primary spiders, mid-polar spiders and advanced spiders. The three types of spiders have different functions and permissions.

The junior spider generally only goes to the new site, and the crawling speed is very slow. When it crawls the web page, it will crawl in stages. When the first level is submitted to the search engine database and is recognized as valuable, the spider will patronize again and submit the crawling to the second level.

Intermediate spiders will crawl to your website from external links, and then grab the content of the website, and then compare the existing content in the database. If the content is of high value, it is easy to be collected, which can effectively increase the weight of the website.

Advanced spiders will only go to websites with high weight, and the speed of advanced spiders is very fast. He has the ability to collect in seconds. So posting on forums with high weights will soon be accepted by search engines. The crawling depth and interview frequency of advanced spiders are also very high, basically all linked pages will crawl, and basically visit every day.

Spider crawling is divided into two ways, one is depth first, and the other is breadth first.

Depth-first is to reach a page and find a link, then crawl to another page, and then find the link again, and continue to crawl until all the pages are crawled.

Breadth first is to reach a page and find that the link does not go in, but after crawling all pages, then enter another page of all links in a unified manner until crawling the entire website.