Technical SEO is considered among the most important metrics by Google. In order for large-scale sites to be better navigated by Google, attention should be paid to this detail. Among the different methods that can be used for this, the most challenging is log analysis. Log analysis basically works as follows; Suppose there is a computer and a server. The two communicate with each other. Inside the server, all requests from computers, that is, users, are recorded even if they are not for SEO purposes.
How to Utilize the Recorded Data for SEO?
- It is clearly shown which IP is used by the people who send you requests.
- It becomes clear when Google bots come to your page.
- It is named what kind of requests from Google bots are. Is Google requesting data or sending data? The answer to this question also becomes clear.
- The request URL, i.e. the server IP address, requests data from the videos and log analysis folder via the http protocol.
- The response to the server IPs is revealed.
- The user has to indicate his identity directly. Identity does not mean detailed information. What Google requires here is the browser and the type of device used.
All these processes actually make it clear from which IP number the users are coming from, from which browser they are visiting you and which pages they are requesting you to GET.
Log analysis can even be millions of pages long for large pages. Because each movement on the site is processed into a one-line log analysis. For large websites and e-commerce sites, log analyzes of these transactions have extremely large files. These files are generally archived monthly by businesses. Different information such as the total number of bytes uploaded, host names and the details of the IP sending the request are included in the reports.
All of the data may also vary depending on where the data is provided. There are two types of data providers. The first one is the data coming directly from the server, and the second one can be the devices, hardware, or software administrators that redirect according to the availability of several servers in the back when a direct request comes to the server.
Metrics to Be Used
Error codes and server codes: When Google bots come to the server, which request code is responded with. Examples include 404 page not found error code and 500 server error or 200 positive.
Total crawl area of the bots: Suppose you have more than 2 million pages on your website. Do Google bots really crawl all your pages or is your crawl area more limited? Google bots may not be crawling pages that are valuable and important to you. You can clearly see this situation after log analysis.
Bots’ crawling priorities: Do Google bots spend enough time on pages that are important for your website? If you are a news site, how long does it take Google bots to get to that page after you enter a news item?
Crawl budget and its misinterpretation: Google bots spending too much time on unwanted pages.
Last Crawl dates: You may want to keep the pages on your website up to date. However, this does not make much sense when Google bots do not see the update and do not visit your page.
What Should You Question?
What is your crawling frequency: After all your pages have been crawled, which groups of pages have a strong relationship with Google bots or other search engine spiders? Once you have a clear picture of this, you can see problems in your linking structure and realize that Google is indexing the wrong pages due to these problems.
The percentage of pages Google crawls per month: When you look at data such as whether Google crawls 20% of your pages or 70% of your pages on a monthly basis, you can also realize what problems there are in your link structure and off-site basic SEO work.
User agent: Is there a difference between the crawling frequency of Google bots and the bots of other search engines? Servers may be responding differently to the requests of Google bots that arrive periodically. When looking at the log analysis, the progress of Google bots can be seen clearly.
Which Tools Can Be Used
Screaming Frog Log Analyzer: Provides you with information about which pages the bots visit and how often.
Oncrawl Log Analyzer: It is an open source log analysis tool. With this tool, you can access data on how you can perform log analysis on your own site.
Botify Log Analyzer: When the extracted logs are uploaded to Botify, the movements of the bots are analyzed. Thus, problem and performance measurement can be done.
Logz.io: This tool can be used when a large-scale e-commerce site wants to perform log analysis simultaneously.
Splunk: If you want to analyze a 1-year log, it is among the most ideal tools for long-term reviews.
Latest Posts
