Could not request seed URL error

[fernstedt]

katana version:

Expected Behavior:

It worked fine for a list of urls 400+

trying again I just get errors

Steps To Reproduce:

Example: steps to reproduce the behavior:

1. Run 'katana ..' with -f url -v to see verbose and -o for file.
   Error

[WRN] Could not request seed URL: GET https://xxxxx.com giving up after 2 attempts: Get "https://xxxx.com": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

This is for all urls, no crawling

[ehsandeep]

@math-ematics katana is an active crawler, as it's very likely that some website ban the IPs that will result into a timeout error, let me know if you have an example to share that we can look into it. Otherwise this is more of expected behavior if your IP got banned from target website.

[coffeegist]

I'm also experiencing this issue, the crawler then shuts down. Confirmed the IP is not blocked, the warning pops after the timeout is exceeded (though I only see the error up to context deadline exceeded, nothing specifically calling out the timeout.

[WRN] "https://domain.com/?parameters=words" on wait load: context deadline exceeded
[WRN] "https://domain.com/?parameters=words" on wait load: context deadline exceeded
[WRN] Could not request seed URL: could not get dom: context deadline exceeded

This is all with headless mode. I've tried changing timeouts, using system chrome, and changing the max response size to no avail.

Using -sb I've seen that a lot of the requests just sit in a pending state, but never actually load. In a normal browser these requests have no issue fulfilling. Since discovering this, I also tried upping the concurrent fetchers used, but no changes observed.

Additionally, I've confirmed that expected results are obtained from a different crawler using the playwright python engine. Happy to provide any details I can (though unfortunately the website I'm crawling can't be shared as it's internal)

[yangyang5214]

my case

katana -u  http://58.56.78.6:81 -sb -hl -d 10 -j

runtime log

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3                                                    

                projectdiscovery.io

{"timestamp":"2023-01-17T13:46:23.150052+08:00","endpoint":"http://58.56.78.6:81/pages/login.jsp","source":"http://58.56.78.6:81/","tag":"a","attribute":"href"}
....
[ERR] "http://58.56.78.6:81" on wait load: context deadline exceeded
[ERR] "http://58.56.78.6:81" on wait idle: context deadline exceeded
[ERR] "http://58.56.78.6:81/pages/login.jsp" on wait load: context deadline exceeded
....

[ehsandeep]

my case

katana -u  http://58.56.78.6:81 -sb -hl -d 10 -j

runtime log

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3                                                    

                projectdiscovery.io

{"timestamp":"2023-01-17T13:46:23.150052+08:00","endpoint":"http://58.56.78.6:81/pages/login.jsp","source":"http://58.56.78.6:81/","tag":"a","attribute":"href"}
....
[ERR] "http://58.56.78.6:81" on wait load: context deadline exceeded
[ERR] "http://58.56.78.6:81" on wait idle: context deadline exceeded
[ERR] "http://58.56.78.6:81/pages/login.jsp" on wait load: context deadline exceeded
....

This works fine if you increase the timeout.

./katana -u  http://58.56.78.6:81 -hl -d 10 -timeout 20 -v

   __        __                
  / /_____ _/ /____ ____  ___ _
 /  '_/ _  / __/ _  / _ \/ _  /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.3							 

		projectdiscovery.io

[a] http://58.56.78.6:81/pages/login.jsp
[link] http://58.56.78.6:81/dtui/theme/dtui.css
[link] http://58.56.78.6:81/css/new.style.css
[script] http://58.56.78.6:81/dtui/js/jquery/jquery.js
[script] http://58.56.78.6:81/dtui/js/jquery/jquery.ba-resize.js
[script] http://58.56.78.6:81/dtui/js/jqueryui/jquery-ui.js
[script] http://58.56.78.6:81/dtui/js/ztree/jquery.ztree.all.js
[script] http://58.56.78.6:81/dtui/js/jqgrid/grid.locale-cn.js
[script] http://58.56.78.6:81/dtui/js/jqgrid/jquery.jqGrid.js
[script] http://58.56.78.6:81/dtui/js/layout/jquery.layout-latest.js
[script] http://58.56.78.6:81/dtui/js/ckeditor/ckeditor.js
[script] http://58.56.78.6:81/dtui/js/gooflow/GooFlow.js
[script] http://58.56.78.6:81/dtui/js/gooflow/GooFunc.js
[script] http://58.56.78.6:81/dtui/js/dtui.all.js
[script] http://58.56.78.6:81/dtui/js/jqueryui/jquery.ui.datepicker-zh-CN.js
[script] http://58.56.78.6:81/js/func.base64.js
[script] http://58.56.78.6:81/js/core/dt.moduleloader.js
[a] http://58.56.78.6:81/a
[a] http://58.56.78.6:81/dtui/js/jqgrid/%5C%22%22
[link] http://58.56.78.6:81/dtui/js/ckeditor/'+a+'
[a] http://58.56.78.6:81/dtui/js/%27%20+%20url%20+%20%27
[a] http://58.56.78.6:81/dtui/js/%22%20+item.url+%20%22
[img] http://58.56.78.6:81/dtui/js/%27+%20icon+%27
[iframe] http://58.56.78.6:81/dtui/js/%27%20+%20this.options.url%20+%20%27
[img] http://58.56.78.6:81/verifycode.do?width=70&height=20&codecount=4&codestyle=digit
[script] http://58.56.78.6:81/js/func.login.js
[script] http://58.56.78.6:81/js/md5.js

[ehsandeep]

I'm also experiencing this issue, the crawler then shuts down. Confirmed the IP is not blocked, the warning pops after the timeout is exceeded (though I only see the error up to context deadline exceeded, nothing specifically calling out the timeout.

[WRN] "https://domain.com/?parameters=words" on wait load: context deadline exceeded
[WRN] "https://domain.com/?parameters=words" on wait load: context deadline exceeded
[WRN] Could not request seed URL: could not get dom: context deadline exceeded

This is all with headless mode. I've tried changing timeouts, using system chrome, and changing the max response size to no avail.

Using -sb I've seen that a lot of the requests just sit in a pending state, but never actually load. In a normal browser these requests have no issue fulfilling. Since discovering this, I also tried upping the concurrent fetchers used, but no changes observed.

Additionally, I've confirmed that expected results are obtained from a different crawler using the playwright python engine. Happy to provide any details I can (though unfortunately the website I'm crawling can't be shared as it's internal)

Thank you for sharing this! probably something to investigate when we have examples to use/replicate, as of now moving this to the discussion section.

[MetzinAround]

A couple answers in here for now, leaving it open for @ehsandeep since he moved it here recently.