Log file analysis reveals exactly how search engines crawl your site — data you can't get from any other source. Screaming Frog's Log File Analyser turns raw Apache, IIS, and Nginx logs into actionable insights about crawl budget waste, orphaned pages, and indexing problems that Google Search Console doesn't show.
This guide walks you through importing server logs, analyzing crawl patterns, and identifying technical issues that hurt your search visibility. You'll need access to your server logs and a crawl of your site to compare bot behavior against your actual site structure.
What You'll Need
Server log files (Apache, IIS, or Nginx format), the Screaming Frog SEO Spider with Log File Analyser add-on, and administrator access to download logs from your hosting panel or server. Most shared hosting providers offer log downloads through cPanel or similar interfaces.
Step 1: Download and Prepare Server Logs
Time: 10 minutes | Tool: Server Admin Panel Access your hosting control panel and locate the raw access logs section — usually under "Logs" or "Statistics." Download at least one month of logs to capture meaningful crawl patterns. Googlebot doesn't visit every page daily, so shorter periods miss important trends. Extract the compressed log files to a dedicated folder on your computer. Most servers compress logs with gzip, so you'll need extraction software. Check the file sizes — large sites generate massive logs that can overwhelm analysis tools if not filtered properly.
Step 2: Configure Log File Import Settings
Time: 5 minutes | Tool: Screaming Frog Open Screaming Frog and navigate to Configuration > Log File Analyser > Import. Set the log format to match your server type — Apache Combined is most common, but IIS and Nginx have different formats. Getting this wrong produces garbage data that looks legitimate but contains no useful insights. Enable the "Filter by User Agent" option and add Googlebot variations: Googlebot, Googlebot-Mobile, Googlebot-Image, and Googlebot-Video. This eliminates crawls from other bots that don't impact your Google rankings. Don't filter too aggressively — legitimate crawlers like Bingbot provide valuable comparative data.
Step 3: Import and Process Log Data
Time: 15 minutes | Tool: Screaming Frog Click "Import Log Files" and select your prepared log files. Screaming Frog processes the data chronologically, showing real-time progress. Large sites with millions of log entries take substantial processing time — don't interrupt the import or you'll lose partial data. Watch the import statistics window for obvious problems like zero valid entries or suspicious user agent counts. Valid imports show thousands of Googlebot requests for active sites. Single-digit request counts usually indicate import configuration errors or extremely low-traffic sites.
Step 4: Analyze Crawl Frequency and Response Codes
Time: 10 minutes | Tool: Screaming Frog Review the Response Codes tab to identify crawl budget waste. High volumes of 404 errors indicate broken internal links or outdated sitemaps that force Googlebot to crawl non-existent pages. 500-series errors suggest server instability during peak crawl times. Check the Crawl Frequency report to spot pages consuming disproportionate crawl budget. Pages crawled daily but rarely updated waste resources that could index new content. Sort by crawl frequency to find over-crawled pagination, search results, or administrative pages that shouldn't rank anyway.
Step 5: Compare Logs Against Site Structure
Time: 10 minutes | Tool: Screaming Frog Run a fresh crawl of your site and compare it against log file data using the "Compare with Crawl" feature. This reveals orphaned pages — URLs that receive bot traffic but aren't linked from your site structure. These pages often perform well in search results but risk losing rankings without proper internal linking. Identify uncrawled important pages by filtering crawl data against your sitemap or priority page lists. Pages that never appear in server logs aren't getting indexed, regardless of their technical optimization. This data helps prioritize internal linking improvements and sitemap submissions.
Step 6: Generate Actionable Reports
Time: 10 minutes | Tool: Screaming Frog Export findings to comprehensive reports focusing on specific issues. The "Crawl Budget Analysis" report shows which page types consume the most bot attention relative to their business value. Use this data to implement crawl budget optimization through robots.txt directives or noindex tags. Create separate reports for 404 errors, server errors, and crawl frequency anomalies. These targeted reports make it easier to communicate technical issues to development teams and track improvement progress over time.
Pro Tips
Set up automated log downloads if your hosting provider supports it — manual log collection becomes tedious for regular monitoring. Many enterprise hosting solutions offer API access or scheduled FTP uploads to streamline the process. Cross-reference crawl timing with your publishing schedule to understand bot response patterns. Sites that publish content at specific times often see corresponding crawl frequency spikes that indicate healthy bot engagement.
Common Pitfalls
Don't analyze single days of log data — crawl patterns vary significantly based on content updates, server performance, and Google's algorithmic changes. Week-over-week comparisons provide much more reliable insights than daily snapshots. Avoid filtering out all non-Googlebot traffic without reviewing it first. Excessive crawling from unknown user agents might indicate security issues, while legitimate tools like Screaming Frog itself appear in logs and help validate your analysis setup.
Expected Results
You'll identify specific URLs wasting crawl budget, discover orphaned pages that need internal links, and pinpoint server errors that prevent indexing. This data directly informs technical SEO improvements that increase organic visibility within weeks of implementation.
Quick Facts
About Screaming Frog
Industry-standard website crawler for technical SEO auditing
View tool review →Screaming Frog Alternatives
6 alternatives compared
View alternatives →