-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[xdoctest] reformat example code with google style in paddle/jit
#55645
Conversation
22bed9f
to
a5d5b96
Compare
有一些交互环境下无法获取源码的示例代码,比如一个非常简单的例子: import inspect
def foo(): ...
inspect.getsource(foo) 在交互环境下是无法获取源码的,这样会报错,这种情况是要如何跳过呢?整段代码 |
@megemini 另外我发现一个问题 非连续的 这是 xdoctest 的已知问题嘛?所以我需要在空行上加上 |
是的!如果无法执行的跳过就行! 如果同段代码后面有需要继续执行的,可以用 |
不连续的 可以手动加上 我把 convert_doctest 改一下,把空行加上 |
#55629 增加了本地验证的步骤,可以一并试一下~ |
这个可以!不然与 CI 交互总归是不够及时 |
不过奇怪的是,我本地跑一直报这个错误
看样子像是 复现方式: xdoctest --global-exec "import paddle\npaddle.device.set_device('cpu')" python/paddle/jit/api.py |
如果直接测试项目中的文件,就会出这种错误! 不能用 xdoctest 直接测试 paddle 整个项目也是这个原因~ 他对于 module 的查找跟咱们不一样~ 所以我在描述里面也建议了,把 docstring 封装到另外一个空文件空函数里面去测试就行~ |
好的,我稍后试一下,话说 xdoctest 是否有参数 prevent 这个行为呢? |
应该不行~ 命令行执行 xdoctest 就会去找 module~ 不过,用之前做的 In [31]: from sampcd_processor_xdoctest import Xdoctester
In [32]: dt = Xdoctester()
In [33]: with open('/paddle/python/paddle/distribution/bernoulli.py') as f:
...: codes = ''.join(f.readlines())
...:
In [34]: codes = dt.convert_directive(codes)
In [35]: dt.run('bernoulli', codes)
====== <exec> ======
* DOCTEST : <modpath?>::bernoulli:0, line 75 <- wrt source file
Tensor(shape=[], dtype=float32, place=Place(cpu), stop_gradient=True,
0.30000001)
Tensor(shape=[], dtype=float32, place=Place(cpu), stop_gradient=True,
0.21000001)
Tensor(shape=[], dtype=float32, place=Place(cpu), stop_gradient=True,
0.61086434)
[100, 1]
[100]
[100, 2]
[100, 2, 2]
[100, 1]
[100]
[100, 2]
[100, 2, 2]
DOCTEST RESULT
* SUCCESS: <modpath?>::bernoulli:0
====== </exec> ======
Out[35]: [TestResult(name='<DocTest(<modname?> bernoulli:0 ln 75)>', nocode=False, passed=True, skipped=False, failed=False, time=0.004740715026855469, test_msg=None, extra_info=None)] 如果直接全文检查的话,会有一个问题,因为他把整个 python 文件当作 docstring,所以,如果有一个地方有 所以,如果没有 skip 之类的,上面的方法应该可以,或者把上面的 codes 改为每一段 docstring,也行~ |
这个 PR 把 jit 下的都改了,可惜的是,因为 AST 动转静需要通过 |
@megemini 我发现有一些空行会让 xdoctest 报错,比如
xdoctest --global-exec "import paddle\npaddle.device.set_device('cpu')" test.py 会报
此时删掉 |
应该是~ 我看了一下 xdoctest,上面的流程:
其中前两步如果出问题,是咱们的问题, 他用 具体问题等有时间再定位吧,不过应该可以确认是 xdoctest 的 bug ... ... 😮💨 |
paddle/jit
paddle/jit
paddle/jit
@megemini @sunzhongkai588 这个 PR 可以来 review 一下了~ CI 只需要看 PR-CI-Static-Check、PR-CI-APPROVAL 其他的不出意外不会不过的 |
除了上面 review 的问题之外,从
不过, 2023-07-26 13:52:29 API check -- Example Code
2023-07-26 13:52:29 sample_test running under python 3.7.0
2023-07-26 13:52:31 [2023-07-26 05:52:31,307] [ INFO] dygraph_sharding_optimizer.py:27 - g_shard_use_reduce 0
2023-07-26 13:52:31 [2023-07-26 05:52:31,308] [ INFO] dygraph_sharding_optimizer.py:29 - g_shard_norm_align_dp 1
2023-07-26 13:52:31 [2023-07-26 05:52:31,308] [ INFO] hybrid_parallel_optimizer.py:43 - g_shard_norm_align_dp 1
2023-07-26 13:52:31 [2023-07-26 05:52:31,315] [ INFO] pipeline_parallel.py:48 - g_shard_use_reduce 0
2023-07-26 13:52:31 paddle.jit.TranslatedLayer in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.TranslatedLayer.program in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.enable_to_static in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.ignore_module in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.load in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.not_to_static in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.save in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.set_code_level in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.set_verbosity in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.to_static in dev is *******, different from pr's *******
2023-07-26 13:52:31 paddle.jit.TranslatedLayer' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.TranslatedLayer.program' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.enable_to_static' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.ignore_module' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.load' code block (name:code-example1, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.load' code block (name:code-example2, id:2) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.not_to_static' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.save' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.set_code_level' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.set_verbosity' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 paddle.jit.to_static' code block (name:None, id:1) is wrapped by PS1(>>> ), which will be tested by xdoctest.
2023-07-26 13:52:31 -----API_PR.spec is the same as API_DEV.spec----- 另外,私有方法也没有提取,这个是 |
上面 review 的问题是?是不是没有 submit review 还处于 pending 状态?
我之前看过 https://github.com/PaddlePaddle/docs/blob/develop/docs/api/gen_doc.py 部分代码,其逻辑是将 Paddle/python/paddle/jit/__init__.py Lines 26 to 36 in 669bcf5
那么就会暴露
这些没抽取到接口的,是因为他们不是公开 API,这样看的话,示例代码的抽取和公开 API 的抽取逻辑是一样的,都是基于 |
python/paddle/jit/api.py
Outdated
>>> # create data loader | ||
>>> dataset = RandomDataset(BATCH_NUM * BATCH_SIZE) | ||
>>> loader = paddle.io.DataLoader(dataset, | ||
>>> feed_list=[image, label], | ||
>>> places=place, | ||
>>> batch_size=BATCH_SIZE, | ||
>>> shuffle=True, | ||
>>> drop_last=True, | ||
>>> return_list=False, | ||
>>> num_workers=2) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
python/paddle/jit/api.py
Outdated
... self._linear = nn.Linear(IMAGE_SIZE, CLASS_NUM) | ||
|
||
... @paddle.jit.to_static |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
缺少 ...
python/paddle/jit/api.py
Outdated
>>> adam = opt.Adam(learning_rate=0.001, parameters=fc.parameters()) | ||
>>> loader = paddle.io.DataLoader(dataset, | ||
>>> places=place, | ||
>>> batch_size=BATCH_SIZE, | ||
>>> shuffle=True, | ||
>>> drop_last=True, | ||
>>> num_workers=2) | ||
>>> for epoch_id in range(EPOCH_NUM): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
同上
>>> print([i.name for i in inputs]) | ||
>>> # [u'generated_tensor_0'] the feed input Tensor name representing x | ||
>>> print([o.name for o in outputs]) | ||
>>> # [u'_generated_var_4'] the fetch output Tensor name representing x_v |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ProgramTranslator 是一个废弃 API,不会公开
lgtm :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
emmm,话说 pytorch 这样的行也是只删除 |
https://pytorch.org/docs/stable/generated/torch.einsum.html?highlight=einsum#torch.einsum 看样子 torch 会把整行删掉,是怎么做到的捏? |
啊…… 再提需求感觉官网想打人了…… |
判断只有 |
前端得到的是渲染后的 HTML 吧?让前端去删 DOM 我觉得不是很好的处理方式 |
我看了下 torch 的处理,看起来应该是这里: 我们是否可以利用 UPDATE: 我在本地搭的 sphinx 环境里测试是可以的: source: before: after: |
嗯嗯 |
PR types
Others
PR changes
Others
Description
修改如下文件的示例代码为新的格式,并通过
xdoctest
检查:python/paddle/jit/api.py
python/paddle/jit/dy2static/convert_call_func.py
python/paddle/jit/dy2static/logging_utils.py
python/paddle/jit/dy2static/program_translator.py
python/paddle/jit/translated_layer.py
预览:
@sunzhongkai588 @SigureMo @megemini
PCard-66962