fix(fetch): handle 5xx responses and add timeout in robots.txt check

fix(fetch): handle 5xx responses and add timeout in robots.txt check #8424