check Foollowing pages :
robots.txt
sitemap_index.xml
Tools to use
whois - cli tool
netcraft - https://sitereport.netcraft.com/arrow-up-right
dnsrecon -d <url>
dnsdumpster - https://dnsdumpster.com/arrow-up-right
wafw00f - tells what firefall a website is using
google dorks
site:website.com : it will show results only related to website.com
site:
site:*.website.com shows subdomains.
inurl:title to search within the domain (can use with site)
inurl:
Last updated 2 years ago