-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Some fixes and improvements #19
Changes from all commits
90a04a9
1d6ea84
12806e4
0558831
570df12
97390c1
b9254fb
8f5ae76
8c46e56
ab058aa
c928027
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1491,6 +1491,9 @@ def to_json(self) -> T_JSON_DICT: | |
|
||
@classmethod | ||
def from_json(cls, json: T_JSON_DICT) -> CookiePartitionKey: | ||
if isinstance(json, str): | ||
# Some chrome versions return partitionKey as string | ||
return cls(json, False) | ||
Comment on lines
+1494
to
+1496
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. just out of curiosity are there any docs on this? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I didn't find something in docs, simple logged CDP message and got from practice usage of chrome >=129 |
||
return cls( | ||
top_level_site=str(json["topLevelSite"]), | ||
has_cross_site_ancestor=bool(json["hasCrossSiteAncestor"]), | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -276,6 +276,11 @@ async def get( | |
await connection.sleep(0.25) | ||
return connection | ||
|
||
async def communicate(self) -> tuple[bytes, bytes]: | ||
if self._process is None: | ||
raise ValueError("Browser process not running") | ||
return await self._process.communicate() | ||
|
||
async def start(self) -> Browser: | ||
"""launches the actual browser""" | ||
if not self: | ||
|
@@ -358,6 +363,12 @@ async def start(self) -> Browser: | |
break | ||
|
||
if not self.info: | ||
stderr = None | ||
try : | ||
_, stderr_bytes = await self._process.communicate() | ||
stderr = stderr_bytes.decode()[:1000] | ||
except Exception : | ||
pass | ||
Comment on lines
+367
to
+371
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is the try/except needed? when would this cause an exception? if this is to handle There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. also, why are we truncating to first 1000 characters? maybe if anything getting the last 1000 characters would be more useful? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. about exception handling : communicate is hard method (it use system calls and can raise exception by much reasons), and raise exception in block that already process exception isn't good for me ... in this case we can live with original exception without output (and debug output separatly) |
||
raise Exception( | ||
( | ||
""" | ||
|
@@ -366,7 +377,8 @@ async def start(self) -> Browser: | |
--------------------- | ||
One of the causes could be when you are running as root. | ||
In that case you need to pass no_sandbox=True | ||
""" | ||
""" + | ||
("Browser error output:" + stderr if stderr else '') | ||
Comment on lines
+380
to
+381
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. thanks, I've been meaning to add something like this. very useful |
||
) | ||
) | ||
|
||
|
@@ -702,8 +714,6 @@ async def save(self, file: PathLike = ".session.dat", pattern: str = ".*"): | |
# return | ||
# if not connection.websocket: | ||
# return | ||
# if connection.websocket.closed: | ||
# return | ||
cookies = await self.get_all(requests_cookie_format=False) | ||
included_cookies = [] | ||
for cookie in cookies: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -201,7 +201,7 @@ def __init__( | |
): | ||
super().__init__() | ||
self._target = target | ||
self.__count__ = itertools.count(0) | ||
self._cdp_id_generator = itertools.count(0) | ||
self._owner = _owner | ||
self.websocket_url: str = websocket_url | ||
self.websocket = None | ||
|
@@ -230,7 +230,7 @@ def target(self, target: cdp.target.TargetInfo): | |
def closed(self): | ||
if not self.websocket: | ||
return True | ||
return self.websocket.closed | ||
return (not self.websocket) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see you added websocket = None below, so this will work when we close the connection, but aren't there also other ways for the websocket to get closed? what if the browser closes the connection? to handle all cases, we should probably do like
(this applies to the other places you changed this as well) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. closed property removed in 14 and logic of websockets authors : |
||
|
||
def add_handler( | ||
self, | ||
|
@@ -282,7 +282,7 @@ async def aopen(self, **kw): | |
:return: | ||
""" | ||
|
||
if not self.websocket or self.websocket.closed: | ||
if not self.websocket: | ||
try: | ||
self.websocket = await websockets.connect( | ||
self.websocket_url, | ||
|
@@ -308,11 +308,12 @@ async def aclose(self): | |
""" | ||
closes the websocket connection. should not be called manually by users. | ||
""" | ||
if self.websocket and not self.websocket.closed: | ||
if self.websocket: | ||
if self.listener and self.listener.running: | ||
self.listener.cancel() | ||
self.enabled_domains.clear() | ||
await self.websocket.close() | ||
self.websocket = None | ||
logger.debug("\n❌ closed websocket connection to %s", self.websocket_url) | ||
|
||
async def sleep(self, t: Union[int, float] = 0.25): | ||
|
@@ -411,7 +412,7 @@ async def send( | |
:return: | ||
""" | ||
await self.aopen() | ||
if not self.websocket or self.closed: | ||
if not self.websocket: | ||
return | ||
if self._owner: | ||
browser = self._owner | ||
|
@@ -425,9 +426,7 @@ async def send( | |
try: | ||
tx = Transaction(cdp_obj) | ||
tx.connection = self | ||
if not self.mapper: | ||
self.__count__ = itertools.count(0) | ||
tx.id = next(self.__count__) | ||
tx.id = next(self._cdp_id_generator) | ||
self.mapper.update({tx.id: tx}) | ||
if not _is_update: | ||
await self._register_handlers() | ||
|
@@ -603,15 +602,17 @@ async def listener_loop(self): | |
# breathe | ||
# await asyncio.sleep(self.time_before_considered_idle / 10) | ||
continue | ||
except (Exception,) as e: | ||
# break on any other exception | ||
# which is mostly socket is closed or does not exist | ||
# or is not allowed | ||
|
||
except (websockets.exceptions.ConnectionClosedError,) as e: | ||
logger.debug( | ||
"connection listener exception while reading websocket:\n%s", e | ||
) | ||
break | ||
except (Exception,) as e: | ||
# we don't expect here other exceptions, need to debug if it will appear. | ||
logger.exception( | ||
"connection listener exception while reading websocket" | ||
) | ||
break | ||
|
||
if not self.running: | ||
# if we have been cancelled or otherwise stopped running | ||
|
@@ -646,9 +647,7 @@ async def listener_loop(self): | |
try: | ||
event = cdp.util.parse_json_event(message) | ||
event_tx = EventTransaction(event) | ||
if not self.connection.mapper: | ||
self.connection.__count__ = itertools.count(0) | ||
event_tx.id = next(self.connection.__count__) | ||
event_tx.id = next(self.connection._cdp_id_generator) | ||
self.connection.mapper[event_tx.id] = event_tx | ||
except Exception as e: | ||
logger.info( | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hey, this change needs to be reverted if you want this PR to get merged - I am sticking with
>=3.10
for theX | Y
union syntax for type annotations.BTW, sorry for the delay in responding, I need to do some further research to understand the websockets==14.0 changes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I replace
|
with typing.Union ? I need 3.9 compatibility …There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I recreated MR, (can't find way to change branch in already created MR), I can't rollback python 3.9 adaptation in main and created MR with other branch : #22