1. Confirm the exact URL
Request the URL and check its HTTP status. Follow any redirects and record the final destination.
2. Review crawling controls
Read the site’s robots.txt file and check for a matching disallow rule. Review security plugins and firewall rules as well.
3. Check indexing directives
A noindex directive is different from a crawling restriction. Inspect the HTML and response headers before changing settings.
4. Verify the change
Test again without an administrator session and use your search engine inspection tools. Keep a record of what changed.