Set curl option for each request

[asefsoft]

Hello
Is there any ways to set separate option for each request?
for example I need to dispatch request across all of my server IP addresses by setting CURLOPT_INTERFACE, so I will need to set this option before sending each request.

Thanks

[freekmurze]

Currently this isn't possible. If the code to enable this is easy, I'd accept a PR with tests and an updated readme.

[asefsoft]

I did it for myself but I dont think that you would do that because it needs to extend Pool class of guzzle

to accomplish that i extend Pool class and changed its construct to add my code to alter request options.
I also extend Crawler class of your package and changed startCrawlingQueue function to use new extended Pool class ...

[asefsoft]

changes to startCrawlingQueue function

<?php
    protected function startCrawlingQueue()
    {

        while ($this->crawlQueue->hasPendingUrls()) {
            $pool = new MyPool($this->client, $this->getCrawlRequests(), [
                'concurrency' => $this->concurrency,
                'options' => $this->client->getConfig(),
                'fulfilled' => new $this->crawlRequestFulfilledClass($this),
                'rejected' => new $this->crawlRequestFailedClass($this),
                'enable_balancer' => $this->isEnableBalancer(),
                'request_id' => $this->request_id
            ]);

            $promise = $pool->promise();

            $promise->wait();
        }
    }

changes to construct of Pool class

<?php
public function __construct(ClientInterface $client, $requests, array $config = [])
    {
        if (!isset($config['concurrency'])) {
            $config['concurrency'] = 25;
        }

        if (isset($config['options'])) {
            $opts = $config['options'];
            unset($config['options']);
        } else {
            $opts = [];
        }

        $enable_balancer = isset($config['enable_balancer']) && $config['enable_balancer'] == true;
        $request_id = empty($config['request_id']) ? Str::random(5) : $config['request_id'];

        $iterable = P\Create::iterFor($requests);
        $requests = static function () use ($request_id, $iterable, $client, $opts, $config, $enable_balancer) {

            orig_opts = $opts;

            foreach ($iterable as $key => $rfn) {

                // load balance
                if($enable_balancer) {
                    $opts     = $orig_opts;
                    $balancer = balancer()->getNextBalancer('all', $request_id);
                    $balancer->configGuzzleClient($rfn, $opts);
                }

                if ($rfn instanceof RequestInterface) {
                    yield $key => $client->sendAsync($rfn, $opts);
                } elseif (\is_callable($rfn)) {
                    yield $key => $rfn($opts);
                } else {
                    throw new \InvalidArgumentException('Each value yielded by the iterator must be a Psr7\Http\Message\RequestInterface or a callable that returns a promise that fulfills with a Psr7\Message\Http\ResponseInterface object.');
                }
            }
        };

        $this->each = new EachPromise($requests(), $config);
    }

[Redominus]

More SOLID approach two comments below. 😛

[asefsoft]

I use this opportunity to thank you for your great packages 🌹

[Redominus]

For your current needs there is already a facility in place in the Guzzle code: Middlewares.

<?php

namespace App\Infrastructure\Network;

use ArrayIterator;
use GuzzleHttp\Promise\PromiseInterface;
use InfiniteIterator;
use Iterator;
use Psr\Http\Message\RequestInterface;

class InterfacesMiddleware
{
    /** @var callable */
    private $nextHandler;
    private Iterator $interfaces;

    public function __construct(
        callable $nextHandler,
        string ...$interfaces
    ) {
        $this->nextHandler = $nextHandler;
        $this->interfaces = new InfiniteIterator(new ArrayIterator($interfaces));
        $this->interfaces->rewind();
    }

    public function __invoke(RequestInterface $request, array $options): PromiseInterface
    {
        $fn = $this->nextHandler;
        $options['curl'][CURLOPT_INTERFACE] = $this->interfaces->current();
        $this->interfaces->next();
        return $fn($request, $options);
    }
}

Now in Crawler construction time:

$handler = HandlerStack::create(new CurlMultiHandler());
$handler->push(
            function (callable $handler) use ($interfaces){
                return new InterfacesMiddleware($handler, ...$interfaces);
            },
            'Interfaces manager or any other name'
        );
$options = [
   'handler' => $handler,
];
$crawler = Crawler::create($options);