Activate LiteSpeed Crawler

You are here:
Estimated reading time: 2 min

About this Article

If you have one of our Super Fast Web Hosting Packages, these packages use LiteSpeed Web Server. LiteSpeed comes with something called LiteSpeed Crawler. This article describes the steps to activate the crawler and its settings.

What is a Crawler?

A crawler is a programme that scans your website pages using a map of your site and caches all of the pages it can. As an example, the major search engines use crawlers to visit your site to read its content and let your website in their search listings.

What is Caching

Caching means the storing of static assets that make up your website. These files are then called quickly when a page is loaded. This speeds up the loading of your web pages. For example, your web browser will cache certain parts of websites that you visit to speed up loading. Web servers also have caching functionality built-in too.

Static assets are those things that don’t change very often – such as images, CSS, JavaScript. These get stored for a short period of time then get deleted then get recached.

What Does the Crawler Do?

Typically, caching happens when someone visits your site. Then, when the next person visits, the site loads a lot quicker. The crawler scans all pages in your site and makes sure that it’s fast for everyone.

How to Activate the Crawler

The crawler is activated on the server but you still need to activate it locally. It’s available for WordPress sites and can be turned on the LiteSpeed WordPress plugin.

To activate:

  • Login to WordPress
  • Locate LiteSpeed Cache from your left menu
  • Select Crawler
  • If this is the first time you are using the crawler, click Generate Crawler file
  • Switch Enable Crawler Cron to Enabled

You can watch the crawler do its thing by clicking Watch Crawler Status.

What Are the Downsides

While this all sounds great, there is a downside. The crawler is resource heavy and uses a lot of CPU. Although it will run within the limits of your hosting package, you may see a slightly degraded performance while the crawler is running. The crawler runs in brief periods to make sure everything doesn’t grind to a halt.

Crawler Settings

To reduce the impact the crawler has you can adjust some of its settings. This is done in the main Litespeed Plugin settings page and clicking the Crawler Tab. You can set:

  • Run Duration – How long the crawler will run for before taking a break. The longer the run the more the impact
  • Interval – the break in-between runs – the shorter the interval the bigger the impact
  • Threads – How many threads to use. Your account will have between 20 and 30 threads available. The more you use the larger the impact
There is really no benefit to speeding up crawling. It’s best to leave the settings at the recommended levels to maximise performance during crawling.

Help With the Crawler?

Then please just ask.

Was this article helpful?
Dislike 0
Views: 151