Last task of radiozamaneh.com_persian takes too long

[benoit74]

Recipe URL

https://farm.openzim.org/recipes/radiozamaneh.com_persian

Task URL

https://farm.openzim.org/pipeline/38678175-a393-4a75-a172-0022cd2f863f

Details

Every page seems to follow the same pattern:

{"timestamp":"2024-11-14T11:49:21.669Z","logLevel":"info","context":"worker","message":"Starting page","details":{"workerid":0,"page":"https://www.radiozamaneh.com/701411/"}}
{"timestamp":"2024-11-14T11:49:21.670Z","logLevel":"info","context":"crawlStatus","message":"Crawl statistics","details":{"crawled":2368,"total":6360,"pending":1,"failed":0,"limit":{"max":0,"hit":false},"pendingPages":["{\"seedId\":0,\"started\":\"2024-11-14T11:49:21.669Z\",\"extraHops\":0,\"url\":\"https:\\/\\/www.radiozamaneh.com\\/701411\\/\",\"added\":\"2024-11-13T12:55:31.241Z\",\"depth\":3}"]}}
{"timestamp":"2024-11-14T11:49:22.123Z","logLevel":"info","context":"general","message":"Awaiting page load","details":{"page":"https://www.radiozamaneh.com/701411/","workerid":0}}
{"timestamp":"2024-11-14T11:49:24.084Z","logLevel":"info","context":"behavior","message":"Running behaviors","details":{"frames":1,"frameUrls":["https://www.radiozamaneh.com/701411/"],"page":"https://www.radiozamaneh.com/701411/","workerid":0}}
{"timestamp":"2024-11-14T11:49:24.084Z","logLevel":"info","context":"behavior","message":"Run Script Started","details":{"frameUrl":"https://www.radiozamaneh.com/701411/","page":"https://www.radiozamaneh.com/701411/","workerid":0}}
{"timestamp":"2024-11-14T11:50:40.531Z","logLevel":"warn","context":"recorder","message":"Request failed","details":{"url":"https://www.radiozamaneh.info/wp-json/wp/v2/comments","errorText":"net::ERR_CONNECTION_TIMED_OUT","type":"Fetch","status":0,"page":"https://www.radiozamaneh.com/701411/","workerid":0}}
{"timestamp":"2024-11-14T11:50:40.531Z","logLevel":"info","context":"behavior","message":"Run Script Finished","details":{"frameUrl":"https://www.radiozamaneh.com/701411/","page":"https://www.radiozamaneh.com/701411/","workerid":0}}
{"timestamp":"2024-11-14T11:50:40.531Z","logLevel":"info","context":"behavior","message":"Behaviors finished","details":{"finished":1,"page":"https://www.radiozamaneh.com/701411/","workerid":0}}
{"timestamp":"2024-11-14T11:50:41.533Z","logLevel":"info","context":"pageStatus","message":"Page Finished","details":{"loadState":4,"page":"https://www.radiozamaneh.com/701411/","workerid":0}}

I.e. it always has ERR_CONNECTION_TIMED_OUT on https://www.radiozamaneh.info/wp-json/wp/v2/comments. And it takes more than one minute to timeout, so the crawl is very very slow.

It is important to note that this URL is not a page but a sub-resource of the current page. It can hence only blocked by --blockRules ... which are not yet exposed by the scraper (see openzim/zimit#433).

[benoit74]

Nota: I cancelled the task and disabled the recipe for now