Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft: feat: introduce HFPath #422

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Draft: feat: introduce HFPath #422

wants to merge 1 commit into from

Conversation

bbtfr
Copy link
Member

@bbtfr bbtfr commented Nov 5, 2024

  1. 实现一个 fsspec.FileSystem 到 megfile.SmartPath 的 wrapper
  2. 通过上述实现适配 huggingface

TODO:

  1. 补测试

@bbtfr bbtfr marked this pull request as draft November 5, 2024 11:20
github-advanced-security[bot]
github-advanced-security bot found potential problems Nov 5, 2024
View reviewed changes
megfile/fsspec.py
from datetime import datetime
from typing import IO, BinaryIO, Callable, Iterator, List, Optional, Tuple

import fsspec

Check failure

Code scanning / Pyre

Type Error Error

Undefined import [21]: Could not find a module corresponding to import fsspec.
megfile/fsspec.py

class BaseFSSpecPath(URIPath):
protocol: str
filesystem: fsspec.AbstractFileSystem

Check failure

Code scanning / Pyre

Type Error Error

Undefined or invalid type [11]: Annotation fsspec.AbstractFileSystem is not defined as a type.
megfile/fsspec.py

:returns: All contents have in the path in ascending alphabetical order
"""
entries = list(self.scandir(followlinks=followlinks))

Check failure

Code scanning / Pyre

Type Error Error

Unexpected keyword [28]: Unexpected keyword argument followlinks to call BaseFSSpecPath.scandir.
megfile/fsspec.py
self.path_without_protocol, dst_path, recursive=True
)

def rename(self, dst_path: PathLike, overwrite: bool = True) -> "BaseFSSpecPath":

Check failure

Code scanning / Pyre

Type Error Error

Incompatible return type [7]: Expected BaseFSSpecPath but got implicit return value of None.
megfile/fsspec.py
"""
self.filesystem.mv(self.path_without_protocol, dst_path, recursive=False)

def move(self, dst_path: PathLike, overwrite: bool = True) -> "BaseFSSpecPath":

Check failure

Code scanning / Pyre

Type Error Error

Incompatible return type [7]: Expected BaseFSSpecPath but got implicit return value of None.
megfile/utils/__init__.py


class cached_classproperty(cached_property):
def __get__(self, instance, cls) -> object:

Check failure

Code scanning / Pyre

Type Error Error

Inconsistent override [14]: megfile.utils.cached\_classproperty.\_\_get\_\_ overrides method defined in cached\_property inconsistently. Could not find parameter cls in overridden signature.
megfile/utils/__init__.py


class cached_classproperty(cached_property):
def __get__(self, instance, cls) -> object:

Check failure

Code scanning / Pyre

Type Error Error

Inconsistent override [14]: megfile.utils.cached\_classproperty.\_\_get\_\_ overrides method defined in cached\_property inconsistently. Could not find parameter instance in overridden signature.
megfile/utils/__init__.py


class cached_classproperty(cached_property):
def __get__(self, instance, cls) -> object:

Check failure

Code scanning / Pyre

Type Error Error

Inconsistent override [14]: megfile.utils.cached\_classproperty.\_\_get\_\_ overrides method defined in cached\_property inconsistently. Could not find parameter self in overridden signature.
megfile/utils/__init__.py
)
val = getattr(cls, self.attrname, self)
if val is self:
with self.lock:

Check failure

Code scanning / Pyre

Type Error Error

Undefined attribute [16]: cached\_classproperty has no attribute lock.
megfile/utils/__init__.py
val = getattr(cls, self.attrname, self)
if val is self:
val = self.func(cls)
setattr(cls, self.attrname, val)

Check failure

Code scanning / Pyre

Type Error Error

Incompatible parameter type [6]: In call setattr, for 2nd positional argument, expected str but got Optional[str].
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@bbtfr