Report raw, cumulative rewards in all five lives during training regardless of settings in reward clipping and episodic life in Atari games

[CloudyDory]

Hi, it seems that Tianshou currently logs the reward coming after all environment wrappers of Atari games. Is it possible to add an option to log the raw, cumulative rewards of all five lives regardless of settings in reward clipping and episodic life?

[Trinkle23897]

https://tianshou.readthedocs.io/en/master/tutorials/cheatsheet.html#handle-batched-data-stream-in-collector
something like this, add an extra wrapper layer for logging right above atari env:

tianshou/docs/tutorials/cheatsheet.rst

Lines 223 to 253 in 4ac407c

	Another solution is to create a vector environment wrapper through :class:`~tianshou.env.VectorEnvWrapper`, e.g.
	::
	import numpy as np
	from collections import deque
	from tianshou.env import VectorEnvWrapper
	class MyWrapper(VectorEnvWrapper):
	def __init__(self, venv, size=100):
	self.episode_log = None
	self.main_log = deque(maxlen=size)
	self.main_log.append(0)
	self.baseline = 0
	def step(self, action, env_id):
	obs, rew, done, info = self.venv.step(action, env_id)
	n = len(rew)
	if self.episode_log is None:
	self.episode_log = [[] for i in range(n)]
	for i in range(n):
	self.episode_log[i].append(rew[i])
	rew[i] -= self.baseline
	for i in range(n):
	if done[i]:
	self.main_log.append(np.mean(self.episode_log[i]))
	self.episode_log[i] = []
	self.baseline = np.mean(self.main_log)
	return obs, rew, done, info
	env = MyWrapper(env, size=100)
	collector = Collector(policy, env, buffer)