You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a problem that I cannot resolve with the the javascript code execution contained in a page.
In dev under Vagrant/Homestead the javascript code in the html page is executed correctly, but not on the prod server ...
The OS and all the libraries are the same on the dev and prod machine :
Ubuntu Server 18.04.5
Laravel/Framework 6.19.1
Spatie/Crawler 4.5.7
puppeteer 5.3.1 with all the necessary dependencies
My code (very simple ...)
Crawler::create()
->setCrawlObserver(new GGCrawlObserver())
->executeJavaScript()
->setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:82.0) Gecko/20100101 Firefox/82.0')
->setMaximumCrawlCount(1)
->setConcurrency(1)
->ignoreRobots()
->setDelayBetweenRequests(500)
->startCrawling('https://....');
In Observer
public function crawled(UriInterface $url, ResponseInterface $response, ?UriInterface $foundOnUrl = null)
{
$domCrawler = new Crawler(
(string)$response->getBody()
);
$name = $domCrawler
->filter('h1')
->first()
->text();
In dev I easily get the content of H1 which is generated by the page javascript code.
But in prod the error The current node list is empty. is fired. The body content is correctly loaded but its javascript code does not execute !
Maybe a delay issue (faster prod server), but I don't know how to fix it.
This discussion was converted from issue #327 on December 16, 2020 08:02.
Heading
Bold
Italic
Quote
Code
Link
Numbered list
Unordered list
Task list
Attach files
Mention
Reference
Menu
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello the team,
I have a problem that I cannot resolve with the the javascript code execution contained in a page.
In dev under Vagrant/Homestead the javascript code in the html page is executed correctly, but not on the prod server ...
The OS and all the libraries are the same on the dev and prod machine :
My code (very simple ...)
In Observer
In dev I easily get the content of H1 which is generated by the page javascript code.
But in prod the error The current node list is empty. is fired. The body content is correctly loaded but its javascript code does not execute !
Maybe a delay issue (faster prod server), but I don't know how to fix it.
Thank you for your help
Beta Was this translation helpful? Give feedback.
All reactions