You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When I open the scrapy program, the program executes normally. But after dozens of minutes, my program recorded the keet print
I found that scrapy went through the request middleware and got the proxy, but no parse was parsed. In other words, it might be scrapy to get an empty url or just not get the url. Just after going to a proxy, I don't know what to do. I added a log to the top of each parsed parse method to print the url of the current request. I found that this log was not returned, and all the requests were broken after DOWNLOADER_MIDDLEWARES and have not been reported. My log level is info.
When I open the scrapy program, the program executes normally. But after tens of minutes my program log keet printing:
2019-06-25 17:42:12 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 0 pages/min), scraped 64 items (at 0 items/min)
2019-06-25 17:43:12 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 0 pages/min), scraped 64 items (at 0 items/min)
2019-06-25 17:44:12 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 0 pages/min), scraped 64 items (at 0 items/min)
2019-06-25 17:45:12 [scrapy.extensions.logstats] INFO: Crawled 64 pages (at 0 pages/min), scraped 64 items (at 0 items/min)
But at this point my redis queue still has data, so I can only close the program and restart it. What should I do to solve this problem? Thank you!
The text was updated successfully, but these errors were encountered: