v0.0.4-beta

What's Changed

• [CI, Doc] Update functorch source installation command by @zou3519 in #446
• [BugFix] TransformedEnv attributes inheritance by @vmoens in #467
• [Feature] Cleanup mocking envs init and new by @vmoens in #469
• [Tests] Adding tensordict __repr__ tests by @sladebot in #435
• [Logging]: implement MLFlow logging integration by @rayanht in #432
• [BugFix] MLFlow import fix by @vmoens in #473
• [BugFix] Fixed pip install by @brandonsj in #475
• [Features]: Changed _inplace_update cls parameter passing in __new__ by @nicolas-dufour in #464
• [Feature]: ModelBased Envs by @nicolas-dufour in #333
• [Feature] make ReplayBufferTrainer compatible with storing trajectories by @vmoens in #476
• [Tutorial] DQN tutorial by @vmoens in #474
• [Feature] reader hooks for GymLike by @vmoens in #478
• [BugFix] TensorSpec.zero(None) failure fix by @vmoens in #483
• [Feature]: Support for planners and CEM by @nicolas-dufour in #384
• [Feature] Replaced device_safe() with device by @ordinskiy in #485
• [Feature]: TensorDictPrimer transform by @nicolas-dufour in #456
• [Feature]: erase() method for torchrl.timeit by @nicolas-dufour in #480
• [Feature] Added support for single collector in sync_async_collector by @nicolas-dufour in #482
• [BugFix] removing unwanted device_safe() by @vmoens in #486
• [Refactoring] Refactored get_stats_random_rollout by @nicolas-dufour in #481
• [Feature] VIP Integration by @JasonMa2016 in #487
• [Refactoring] Minor tweaks to recorder and logger by @nicolas-dufour in #489
• [Feature]: Deactivate typechecks in envs by @nicolas-dufour in #490
• [BugFix] Vectorized td_lambda with gamma tensor does not match the serial version by @vmoens in #400
• [BugFix] Fix TensorDictPrimer init by @vmoens in #491
• [Feature] Optional auto-reset when done for collectors and batched envs by @vmoens in #492
• [BugFix] Defaulting passing_devices to None by @himjohntang in #477
• Revert "[BugFix] Defaulting passing_devices to None" by @vmoens in #494
• [BugFix] Multi-agent fixes by @vmoens in #488
• [BugFix] Defaulting passing_devices to None by @vmoens in #495
• [Feature] Lazy initialization of CatTensors by @vmoens in #497
• [Cleanup] Removing cuda 10.2 references by @vmoens in #498
• [BugFix] Migration to pytorch org by @vmoens in #499
• [Refactoring] Import at root to enable vmap monkey-patching by @vmoens in #500
• [BugFix] python version for linting checks by @vmoens in #502
• [Feature] Replay Buffers refactor by @bamaxw in #330
• [Feature] Rename step_tensordict in step_mdp by @romainjln in #512
• [Lint] re-instantiate F821 by @vmoens in #516
• [BugFix] run_type_checks for TransformedEnvs by @vmoens in #513
• [BugFix] making first_dim and last_dim negative in FlattenObservation when a parent is set by @vmoens in #511
• [Feature] Add info dict key-spec pairs to observation_spec by @tcbegley in #504
• [BugFix] Changing the dm_control import to fail if not installed by @zeenolife in #515
• [CI] Add coverage with codecov by @silvestrebahi in #523
• Revert "[CI] Add coverage with codecov" by @vmoens in #525
• [Quality] Use relative imports for local c++ deps by @apbard in #526
• [Feature] Nightly release by @vmoens in #519
• [Feature] Add make_tensordict() function by @sicong-huang in #522
• [Doc] Misc readme fixes by @GavinPHR in #532
• [BugFix] Replacing inference_mode decorator with no_grad to fix state_dict loading error by @GavinPHR in #530
• [BugFix] Transformed ParallelEnv meta data are broken when passing to device by @vmoens in #531
• [Doc] Add coverage banner by @vmoens in #533
• [BugFix] Fix colab link of coding_dqn.ipynb by @Benjamin-eecs in #543
• [BugFix] Fix optional imports by @vmoens in #535
• [BugFix] Restore missing keys in data collector output by @tcbegley in #521
• [Lint] reorganize imports by @apbard in #545
• [BugFix] Single-cpu compatibility by @vmoens in #548
• [BugFix] vision install and other deps in optdeps by @vmoens in #552
• [Feature] Implemented device argument for modules.models by @yushiyangk in #524
• [BugFix] Fix ellipsis indexing of 2d TensorDicts by @vmoens in #559
• [BugFix] Additive gaussian exploration spec fix by @vmoens in #560
• [BugFix] Disabling video step for wandb by @vmoens in #561
• [BugFix] Various device fix by @vmoens in #558
• [Feature] Allow collectors to accept regular modules as policies by @tcbegley in #546
• [BugFix] Fix push binary nightly action by @psolikov in #566
• [BugFix] TensorDict comparison by @vmoens in #567
• [BugFix] Fix SyncDataCollector reset by @jrobine in #571
• [Doc] Banners on README.md by @vmoens in #572
• [Feature] Log printing in alphabetical order when creating a replay buffer by @nikhlrao in #573
• [BugFix] Add eps to reward normalization by @vmoens in #574
• [BugFix] Fix argument for PPOLoss.get_entropy_bonus() by @vmoens in #578
• [Feature] Restructure torchrl/objectives by @sgrigory in #580
• [Docs] Documentation revamp by @vmoens in #581
• [Doc] Publishing on pytorch.org by @vmoens in #582
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in #584
• [Doc] Publishing on pytorch.org by @vmoens in #585
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in #586
• [Doc] Publishing on pytorch.org by @vmoens in #587
• [Feature] More restrictive tests on docstrings by @vmoens in #457
• [BugFix] Wrong stack import in tests by @vmoens in #590
• [Feature] Exclude "_" out_keys in tensordictmodel by @jlesuffleur in #589
• [Feature]: Dreamer support by @nicolas-dufour in #341
• [Doc] Missing doc for prototype RB by @vmoens in #595
• [Feature] Update list of supported libraries by @vmoens in #594
• [BugFix] Fix timeit count registration by @vmoens in #598
• [Naming] Renaming ProbabilisticTensorDictModule keys by @vmoens in #603
• [Feature] Categorical encoding for action space by @artkorenev in #593
• [BugFix] ReplayBuffer's storage now signal back when changes happen by @paulomarciano in #614
• [Doc] Typos in tensordict tutorial by @PaLeroy in #621
• [Doc] Integrate knowledge base in docs by @hatala91 in #622
• [Doc] Updating docs requirements by @vmoens in #624
• [Feature] Make torchrl runnable without functorch and with gym==0.13 by @vmoens in #386
• [Feature] Habitat integration by @vmoens in #514
• [Feature] Checkpointing by @vmoens in #549
• Add support for null dim argument in TensorDict.squeeze by @jgonik in #608
• [Version] Updating to torch 1.13 by @vmoens in #627
• [Feature] Sub-memmap tensors by @vmoens in #626
• [BugFix] copy_ changes the index if the dest and source memmap tensors share the same file location by @vmoens in #631
• [F...

v0.0.4

What's Changed

• [CI, Doc] Update functorch source installation command by @zou3519 in #446
• [BugFix] TransformedEnv attributes inheritance by @vmoens in #467
• [Feature] Cleanup mocking envs init and new by @vmoens in #469
• [Tests] Adding tensordict __repr__ tests by @sladebot in #435
• [Logging]: implement MLFlow logging integration by @rayanht in #432
• [BugFix] MLFlow import fix by @vmoens in #473
• [BugFix] Fixed pip install by @brandonsj in #475
• [Features]: Changed _inplace_update cls parameter passing in __new__ by @nicolas-dufour in #464
• [Feature]: ModelBased Envs by @nicolas-dufour in #333
• [Feature] make ReplayBufferTrainer compatible with storing trajectories by @vmoens in #476
• [Tutorial] DQN tutorial by @vmoens in #474
• [Feature] reader hooks for GymLike by @vmoens in #478
• [BugFix] TensorSpec.zero(None) failure fix by @vmoens in #483
• [Feature]: Support for planners and CEM by @nicolas-dufour in #384
• [Feature] Replaced device_safe() with device by @ordinskiy in #485
• [Feature]: TensorDictPrimer transform by @nicolas-dufour in #456
• [Feature]: erase() method for torchrl.timeit by @nicolas-dufour in #480
• [Feature] Added support for single collector in sync_async_collector by @nicolas-dufour in #482
• [BugFix] removing unwanted device_safe() by @vmoens in #486
• [Refactoring] Refactored get_stats_random_rollout by @nicolas-dufour in #481
• [Feature] VIP Integration by @JasonMa2016 in #487
• [Refactoring] Minor tweaks to recorder and logger by @nicolas-dufour in #489
• [Feature]: Deactivate typechecks in envs by @nicolas-dufour in #490
• [BugFix] Vectorized td_lambda with gamma tensor does not match the serial version by @vmoens in #400
• [BugFix] Fix TensorDictPrimer init by @vmoens in #491
• [Feature] Optional auto-reset when done for collectors and batched envs by @vmoens in #492
• [BugFix] Defaulting passing_devices to None by @himjohntang in #477
• Revert "[BugFix] Defaulting passing_devices to None" by @vmoens in #494
• [BugFix] Multi-agent fixes by @vmoens in #488
• [BugFix] Defaulting passing_devices to None by @vmoens in #495
• [Feature] Lazy initialization of CatTensors by @vmoens in #497
• [Cleanup] Removing cuda 10.2 references by @vmoens in #498
• [BugFix] Migration to pytorch org by @vmoens in #499
• [Refactoring] Import at root to enable vmap monkey-patching by @vmoens in #500
• [BugFix] python version for linting checks by @vmoens in #502
• [Feature] Replay Buffers refactor by @bamaxw in #330
• [Feature] Rename step_tensordict in step_mdp by @romainjln in #512
• [Lint] re-instantiate F821 by @vmoens in #516
• [BugFix] run_type_checks for TransformedEnvs by @vmoens in #513
• [BugFix] making first_dim and last_dim negative in FlattenObservation when a parent is set by @vmoens in #511
• [Feature] Add info dict key-spec pairs to observation_spec by @tcbegley in #504
• [BugFix] Changing the dm_control import to fail if not installed by @zeenolife in #515
• [CI] Add coverage with codecov by @silvestrebahi in #523
• Revert "[CI] Add coverage with codecov" by @vmoens in #525
• [Quality] Use relative imports for local c++ deps by @apbard in #526
• [Feature] Nightly release by @vmoens in #519
• [Feature] Add make_tensordict() function by @sicong-huang in #522
• [Doc] Misc readme fixes by @GavinPHR in #532
• [BugFix] Replacing inference_mode decorator with no_grad to fix state_dict loading error by @GavinPHR in #530
• [BugFix] Transformed ParallelEnv meta data are broken when passing to device by @vmoens in #531
• [Doc] Add coverage banner by @vmoens in #533
• [BugFix] Fix colab link of coding_dqn.ipynb by @Benjamin-eecs in #543
• [BugFix] Fix optional imports by @vmoens in #535
• [BugFix] Restore missing keys in data collector output by @tcbegley in #521
• [Lint] reorganize imports by @apbard in #545
• [BugFix] Single-cpu compatibility by @vmoens in #548
• [BugFix] vision install and other deps in optdeps by @vmoens in #552
• [Feature] Implemented device argument for modules.models by @yushiyangk in #524
• [BugFix] Fix ellipsis indexing of 2d TensorDicts by @vmoens in #559
• [BugFix] Additive gaussian exploration spec fix by @vmoens in #560
• [BugFix] Disabling video step for wandb by @vmoens in #561
• [BugFix] Various device fix by @vmoens in #558
• [Feature] Allow collectors to accept regular modules as policies by @tcbegley in #546
• [BugFix] Fix push binary nightly action by @psolikov in #566
• [BugFix] TensorDict comparison by @vmoens in #567
• [BugFix] Fix SyncDataCollector reset by @jrobine in #571
• [Doc] Banners on README.md by @vmoens in #572
• [Feature] Log printing in alphabetical order when creating a replay buffer by @nikhlrao in #573
• [BugFix] Add eps to reward normalization by @vmoens in #574
• [BugFix] Fix argument for PPOLoss.get_entropy_bonus() by @vmoens in #578
• [Feature] Restructure torchrl/objectives by @sgrigory in #580
• [Docs] Documentation revamp by @vmoens in #581
• [Doc] Publishing on pytorch.org by @vmoens in #582
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in #584
• [Doc] Publishing on pytorch.org by @vmoens in #585
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in #586
• [Doc] Publishing on pytorch.org by @vmoens in #587
• [Feature] More restrictive tests on docstrings by @vmoens in #457
• [BugFix] Wrong stack import in tests by @vmoens in #590
• [Feature] Exclude "_" out_keys in tensordictmodel by @jlesuffleur in #589
• [Feature]: Dreamer support by @nicolas-dufour in #341
• [Doc] Missing doc for prototype RB by @vmoens in #595
• [Feature] Update list of supported libraries by @vmoens in #594
• [BugFix] Fix timeit count registration by @vmoens in #598
• [Naming] Renaming ProbabilisticTensorDictModule keys by @vmoens in #603
• [Feature] Categorical encoding for action space by @artkorenev in #593
• [BugFix] ReplayBuffer's storage now signal back when changes happen by @paulomarciano in #614
• [Doc] Typos in tensordict tutorial by @PaLeroy in #621
• [Doc] Integrate knowledge base in docs by @hatala91 in #622
• [Doc] Updating docs requirements by @vmoens in #624
• [Feature] Make torchrl runnable without functorch and with gym==0.13 by @vmoens in #386
• [Feature] Habitat integration by @vmoens in #514
• [Feature] Checkpointing by @vmoens in #549
• Add support for null dim argument in TensorDict.squeeze by @jgonik in #608
• [Version] Updating to torch 1.13 by @vmoens in #627
• [Feature] Sub-memmap tensors by @vmoens in #626
• [BugFix] copy_ changes the index if the dest and source memmap tensors share the same file location by @vmoens in #631
• [F...

v0.0.4-alpha

What's Changed

• [CI, Doc] Update functorch source installation command by @zou3519 in #446
• [BugFix] TransformedEnv attributes inheritance by @vmoens in #467
• [Feature] Cleanup mocking envs init and new by @vmoens in #469
• [Tests] Adding tensordict __repr__ tests by @sladebot in #435
• [Logging]: implement MLFlow logging integration by @rayanht in #432
• [BugFix] MLFlow import fix by @vmoens in #473
• [BugFix] Fixed pip install by @brandonsj in #475
• [Features]: Changed _inplace_update cls parameter passing in __new__ by @nicolas-dufour in #464
• [Feature]: ModelBased Envs by @nicolas-dufour in #333
• [Feature] make ReplayBufferTrainer compatible with storing trajectories by @vmoens in #476
• [Tutorial] DQN tutorial by @vmoens in #474
• [Feature] reader hooks for GymLike by @vmoens in #478
• [BugFix] TensorSpec.zero(None) failure fix by @vmoens in #483
• [Feature]: Support for planners and CEM by @nicolas-dufour in #384
• [Feature] Replaced device_safe() with device by @ordinskiy in #485
• [Feature]: TensorDictPrimer transform by @nicolas-dufour in #456
• [Feature]: erase() method for torchrl.timeit by @nicolas-dufour in #480
• [Feature] Added support for single collector in sync_async_collector by @nicolas-dufour in #482
• [BugFix] removing unwanted device_safe() by @vmoens in #486
• [Refactoring] Refactored get_stats_random_rollout by @nicolas-dufour in #481
• [Feature] VIP Integration by @JasonMa2016 in #487
• [Refactoring] Minor tweaks to recorder and logger by @nicolas-dufour in #489
• [Feature]: Deactivate typechecks in envs by @nicolas-dufour in #490
• [BugFix] Vectorized td_lambda with gamma tensor does not match the serial version by @vmoens in #400
• [BugFix] Fix TensorDictPrimer init by @vmoens in #491
• [Feature] Optional auto-reset when done for collectors and batched envs by @vmoens in #492
• [BugFix] Defaulting passing_devices to None by @himjohntang in #477
• Revert "[BugFix] Defaulting passing_devices to None" by @vmoens in #494
• [BugFix] Multi-agent fixes by @vmoens in #488
• [BugFix] Defaulting passing_devices to None by @vmoens in #495
• [Feature] Lazy initialization of CatTensors by @vmoens in #497
• [Cleanup] Removing cuda 10.2 references by @vmoens in #498
• [BugFix] Migration to pytorch org by @vmoens in #499
• [Refactoring] Import at root to enable vmap monkey-patching by @vmoens in #500
• [BugFix] python version for linting checks by @vmoens in #502
• [Feature] Replay Buffers refactor by @bamaxw in #330
• [Feature] Rename step_tensordict in step_mdp by @romainjln in #512
• [Lint] re-instantiate F821 by @vmoens in #516
• [BugFix] run_type_checks for TransformedEnvs by @vmoens in #513
• [BugFix] making first_dim and last_dim negative in FlattenObservation when a parent is set by @vmoens in #511
• [Feature] Add info dict key-spec pairs to observation_spec by @tcbegley in #504
• [BugFix] Changing the dm_control import to fail if not installed by @zeenolife in #515
• [CI] Add coverage with codecov by @silvestrebahi in #523
• Revert "[CI] Add coverage with codecov" by @vmoens in #525
• [Quality] Use relative imports for local c++ deps by @apbard in #526
• [Feature] Nightly release by @vmoens in #519
• [Feature] Add make_tensordict() function by @sicong-huang in #522
• [Doc] Misc readme fixes by @GavinPHR in #532
• [BugFix] Replacing inference_mode decorator with no_grad to fix state_dict loading error by @GavinPHR in #530
• [BugFix] Transformed ParallelEnv meta data are broken when passing to device by @vmoens in #531
• [Doc] Add coverage banner by @vmoens in #533
• [BugFix] Fix colab link of coding_dqn.ipynb by @Benjamin-eecs in #543
• [BugFix] Fix optional imports by @vmoens in #535
• [BugFix] Restore missing keys in data collector output by @tcbegley in #521
• [Lint] reorganize imports by @apbard in #545
• [BugFix] Single-cpu compatibility by @vmoens in #548
• [BugFix] vision install and other deps in optdeps by @vmoens in #552
• [Feature] Implemented device argument for modules.models by @yushiyangk in #524
• [BugFix] Fix ellipsis indexing of 2d TensorDicts by @vmoens in #559
• [BugFix] Additive gaussian exploration spec fix by @vmoens in #560
• [BugFix] Disabling video step for wandb by @vmoens in #561
• [BugFix] Various device fix by @vmoens in #558
• [Feature] Allow collectors to accept regular modules as policies by @tcbegley in #546
• [BugFix] Fix push binary nightly action by @psolikov in #566
• [BugFix] TensorDict comparison by @vmoens in #567
• [BugFix] Fix SyncDataCollector reset by @jrobine in #571
• [Doc] Banners on README.md by @vmoens in #572
• [Feature] Log printing in alphabetical order when creating a replay buffer by @nikhlrao in #573
• [BugFix] Add eps to reward normalization by @vmoens in #574
• [BugFix] Fix argument for PPOLoss.get_entropy_bonus() by @vmoens in #578
• [Feature] Restructure torchrl/objectives by @sgrigory in #580
• [Docs] Documentation revamp by @vmoens in #581
• [Doc] Publishing on pytorch.org by @vmoens in #582
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in #584
• [Doc] Publishing on pytorch.org by @vmoens in #585
• Revert "[Doc] Publishing on pytorch.org" by @vmoens in #586
• [Doc] Publishing on pytorch.org by @vmoens in #587
• [Feature] More restrictive tests on docstrings by @vmoens in #457
• [BugFix] Wrong stack import in tests by @vmoens in #590
• [Feature] Exclude "_" out_keys in tensordictmodel by @jlesuffleur in #589
• [Feature]: Dreamer support by @nicolas-dufour in #341
• [Doc] Missing doc for prototype RB by @vmoens in #595
• [Feature] Update list of supported libraries by @vmoens in #594
• [BugFix] Fix timeit count registration by @vmoens in #598
• [Naming] Renaming ProbabilisticTensorDictModule keys by @vmoens in #603
• [Feature] Categorical encoding for action space by @artkorenev in #593
• [BugFix] ReplayBuffer's storage now signal back when changes happen by @paulomarciano in #614
• [Doc] Typos in tensordict tutorial by @PaLeroy in #621
• [Doc] Integrate knowledge base in docs by @hatala91 in #622
• [Doc] Updating docs requirements by @vmoens in #624
• [Feature] Make torchrl runnable without functorch and with gym==0.13 by @vmoens in #386
• [Feature] Habitat integration by @vmoens in #514
• [Feature] Checkpointing by @vmoens in #549
• Add support for null dim argument in TensorDict.squeeze by @jgonik in #608
• [Version] Updating to torch 1.13 by @vmoens in #627
• [Feature] Sub-memmap tensors by @vmoens in #626
• [BugFix] copy_ changes the index if the dest and source memmap tensors share the same file location by @vmoens in #631
• [F...

v0.0.3

The main changes introduced by this release are:

• dependency on the standalone tensordict repo;
• refactoring of the "next" API

What's Changed

• [Versioning] MacOs versioning and release bugfix by @vmoens in #247
• [Versioning] Setup metadata by @vmoens in #248
• [BugFix] Fix setup instructions by @vmoens in #250
• [BugFix] Fix a bug when segment_tree size is exactly 2^N by @xiaomengy in #251
• [Feature] Added test for RewardRescale transform by @nicolas-dufour in #252
• [Feature] Empty TensorDict population in loops by @vmoens in #253
• [BugFix] Memmap del bugfix by @vmoens in #254
• [Feature] Implement padding for tensordicts by @ajhinsvark in #257
• [BugFix]: recursion error when calling permute(...).to_tensordict() by @vmoens in #260
• [Feature] Differentiable PPOLoss for IRL by @vmoens in #240
• [BugFix]: avoid deleting true in_keys in TensorDictSequence by @vmoens in #261
• [Feature] Add issue and pull request template by @Benjamin-eecs in #263
• [Feature] Nested tensordicts by @vmoens in #256
• [Feature]: Index nested tensordicts using tuples by @vmoens in #262
• [Feature]: flatten nested tensordicts by @vmoens in #264
• [Test]: test nested CompositeSpec by @vmoens in #265
• [Test]: test squeezed TensorDict by @vmoens in #269
• [Doc] Added TensorDict tutorial by @nicolas-dufour in #255
• [Test]: TensorDict: test tensordict created on cuda and sub-tensordict indexed along 2nd dimension by @vmoens in #268
• Refactor the torch.stack with destination by @khmigor in #245
• [Feature]: faster meta-tensor API for TensorDict by @vmoens in #272
• [Feature]: Refactored logging to be able to support other loggers easily by @nicolas-dufour in #270
• Small tweaks to make the replay buffer code more consistent by @shagunsodhani in #275
• [BugFix]: Minor bugs in docstrings by @vmoens in #276
• [Doc]: TorchRL demo by @vmoens in #284
• [BugFix]: update wrong links in issue and pull request template by @Benjamin-eecs in #286
• [BugFix]: quickfix: force gym 0.24 installation until issue with rendering is resolved by @vmoens in #283
• [Doc]: remove pip install from CONTRIBUTING.md by @vmoens in #288
• [Feature]: faster safetanh transform via C++ bindings by @vmoens in #289
• [BugFix]: fix GLFW3 error when installing dm_control by @vmoens in #291
• [BugFix]: Fix examples by @vmoens in #290
• [Doc] Simplify PR template by @vmoens in #292
• [BugFix]: Replay buffer bugfixes by @vmoens in #294
• [Doc] MacOs M1 troubleshooting by @ramonmedel in #296
• [Feature]: Improving training efficiency by @vmoens in #293
• [Feature] Wandb logger by @nicolas-dufour in #274
• [QuickFix]: update issue and pr template by @Benjamin-eecs in #303
• [Test] tests for BinarizeReward by @srikanthmg85 in #302
• [BugFix]: L2-priority for PRB by @vmoens in #305
• [Feature] Transforms: Compose.insert and TransformedEnv.insert_transform by @rmartimov in #304
• [BugFix] Fix flaky test by waiting for procs instead of sleep by @nairbv in #306
• [BugFix] Fix a build warning, setuptools/distutils import order by @nairbv in #307
• ufmt issue if imports in order requested by distutils by @nairbv in #308
• [BugFix]: Conda to pip for circleci by @vmoens in #310
• [BugFix] Support list-based boolean masks for TensorDict by @benoitdescamps in #299
• [Feature] Truly invertible tensordict permutation of dimensions by @ramonmedel in #295
• [Doc] Tensordictmodule tutorial by @nicolas-dufour in #267
• [Feature] Rename _TensorDict into TensorDictBase by @yoavnavon in #316
• [Release]: v0.0.1b versioning by @vmoens in #317
• [Feature] Adding additional checks to TensorDict.view to remove unnecessary ViewedTensorDict object creation by @bamaxw in #319
• [BugFix]: Safe state normalization when std=0 by @vmoens in #323
• [BugFix]: gradient propagation in advantage estimates by @vmoens in #322
• [BugFix]: make training example gracefully exit by @vmoens in #326
• [Setup]: Exclude tutorials from wheels by @vmoens in #325
• [BugFix]: Tensor map for subtensordict.set_ by @vmoens in #324
• [Versioning]: Wheels v0.0.1c by @vmoens in #327
• [BugFix] Fixed compose which ignored inv_transforms of child by @nicolas-dufour in #328
• [BugFix] functorch installation in CircleCI by @vmoens in #336
• [Refactor] VecNorm inference API by @vmoens in #337
• [BugFix] TransformedEnv sets added Transforms into eval mode by @alexanderlobov in #331
• [Refactor] make to_tensordict() create a copy of the content by @nicolas-dufour in #334
• [CircleCI] Fix dm_control rendering by @vmoens in #339
• [BugFix]: joining processes when they're done by @vmoens in #311
• [Test] pass the OS error in case the file isn't closed by @tongbaojia in #344
• [Feature] Make default rollout tensordict contiguous by @vmoens in #343
• [BugFix] Clone memmap tensors on regular tensors and other replay buffer improvements by @vmoens in #340
• [CI] Using latest gym by @vmoens in #346
• [Doc] Coding your first DDPG tutorial by @vmoens in #345
• [Doc] Minor: typos in DDPG by @vmoens in #354
• [Feature] Register lambda and gamma in buffers by @vmoens in #353
• [Feature] Implement eq for TensorSpec by @omikad in #358
• [Doc] Multi-tasking tutorial by @vmoens in #352
• [Feature] Env refactoring for model based RL by @nicolas-dufour in #315
• [Feature]: Added support for TensorDictSequence module subsampling by @nicolas-dufour in #332
• [BugFix] Add lock to vec norm transform by @jaschmid-fb in #356
• [Perf]: Improve PPO training performance by @vmoens in #297
• [BugFix] Functorch-Tensordict bug fixes by @vmoens in #361
• Revert "[BugFix] Functorch-Tensordict bug fixes" by @vmoens in #362
• [BugFix] Functorch-Tensordict bug fixes by @vmoens in #363
• [Feature] CSVLogger (ABBANDONED) by @vmoens in #371
• [Feature] Support tensor-based decay in TD-lambda by @tcbegley in #360
• [Feature] CSVLogger by @vmoens in #372
• [BugFix] Fewer env instantiations for better mujoco rendering by @vmoens in #378
• [Feature] change imports of environment libraries (gym and dm_control) at lower levels by @guabao in #379
• [BugFix] Representation of indexed nested tensordict by @vmoens in #370
• [BugFix] In-place __setitem__ for SubTensorDict by @vmoens in #369
• [Feature] Add ProbabilisticTensorDictModule dist key mapping support by @nicolas-dufour in #376
• [Feature]: R3M integration by @vmoens in #321
• [Feature] static_seed flag for envs, vectorized envs and collectors by @vmoens in #385
• [Feature] AdditiveGaussian exploration strategy by @vmoens in #388
• [Feature] Multi-images R3M by @vmoens in #389
• [Feature] Flatten multi-images in R3M by @vmoens in #391
• [Quality] Code cleanup for fbsync by @vmoens in #392
• [Feature] In-house functional modules for TorchRL using TensorDict by @vmoens in https://github.com/pytorch...

0.0.2a

What's Changed

• [BugFix] Fixed compose which ignored inv_transforms of child by @nicolas-dufour in #328
• [BugFix] functorch installation in CircleCI by @vmoens in #336
• [Refactor] VecNorm inference API by @vmoens in #337
• TransformedEnv sets added Transforms into eval mode by @alexanderlobov in #331
• [Refactor] make to_tensordict() create a copy of the content by @nicolas-dufour in #334
• [CircleCI] Fix dm_control rendering by @vmoens in #339
• [BugFix]: joining processes when they're done by @vmoens in #311
• [Test] pass the OS error in case the file isn't closed by @tongbaojia in #344
• [Feature] Make default rollout tensordict contiguous by @vmoens in #343
• [BugFix] Clone memmap tensors on regular tensors and other replay buffer improvements by @vmoens in #340
• [CI] Using latest gym by @vmoens in #346
• [Doc] Coding your first DDPG tutorial by @vmoens in #345
• [Doc] Minor: typos in DDPG by @vmoens in #354
• [Feature] Register lambda and gamma in buffers by @vmoens in #353
• [Feature] Implement eq for TensorSpec by @omikad in #358
• [Doc] Multi-tasking tutorial by @vmoens in #352
• [Feature] Env refactoring for model based RL by @nicolas-dufour in #315
• [Feature]: Added support for TensorDictSequence module subsampling by @nicolas-dufour in #332
• [BugFix] Add lock to vec norm transform by @jaschmid-fb in #356
• [Perf]: Improve PPO training performance by @vmoens in #297
• [BugFix] Functorch-Tensordict bug fixes by @vmoens in #361
• Revert "[BugFix] Functorch-Tensordict bug fixes" by @vmoens in #362
• [BugFix] Functorch-Tensordict bug fixes by @vmoens in #363
• [Feature] CSVLogger (ABBANDONED) by @vmoens in #371
• [Feature] Support tensor-based decay in TD-lambda by @tcbegley in #360
• [Feature] CSVLogger by @vmoens in #372
• [BugFix] Fewer env instantiations for better mujoco rendering by @vmoens in #378
• [Feature] change imports of environment libraries (gym and dm_control) at lower levels by @guabao in #379
• [BugFix] Representation of indexed nested tensordict by @vmoens in #370
• [BugFix] In-place __setitem__ for SubTensorDict by @vmoens in #369
• [Feature] Add ProbabilisticTensorDictModule dist key mapping support by @nicolas-dufour in #376
• [Feature]: R3M integration by @vmoens in #321
• [Feature] static_seed flag for envs, vectorized envs and collectors by @vmoens in #385
• [Feature] AdditiveGaussian exploration strategy by @vmoens in #388
• [Feature] Multi-images R3M by @vmoens in #389
• [Feature] Flatten multi-images in R3M by @vmoens in #391
• [Quality] Code cleanup for fbsync by @vmoens in #392
• [Feature] In-house functional modules for TorchRL using TensorDict by @vmoens in #387
• [Quality] Code cleanup for fbsync by @vmoens in #397
• [Doc] Add charts to examples by @nicolas-dufour in #374
• [Feature] Vectorized GAE by @vmoens in #365
• [BugFix] Temporarily fix gym to 0.25.1 to fix CI by @vmoens in #411
• [Feature] Create a Squeeze transform and update Unsqueeze transform by @reachsumit in #408
• [Naming] Recurse kwarg to match pytorch by @matt-fff in #410
• [Feature] Add all implemented loggers to the init of loggers by @flinder in #402
• [BugFix] Fix gym 0.26 compatibility by @vmoens in #403
• [BugFix] Remove submodules by @vmoens in #414
• [Feature] lock tensordict when calling share_memory_() by @fdabek1 in #412
• [BugFix] Updated TensorDict.expand to work as Tensor.expand by @AnshulSehgal in #409
• [BugFix] Looser check for test_recorder assertion by @vmoens in #415
• [Feature] Allow spec to be passed directly to exploration wrappers by @vmoens in #418
• [BugFix] Collector revert to default exploration mode if empty string is passed by @vmoens in #421
• [Naming] Rename _TargetNetUpdate to TargetNetUpdater, making it public by @yushiyangk in #422
• [Doc] Re-run tutorials by @vmoens in #381
• Revert "[Doc] Re-run tutorials" (colab links broken) by @vmoens in #423
• [Feature] Switch back to latest gym by @vmoens in #425
• [Feature] TensorDict without device by @tcbegley in #413
• Updated the README.md file by @bashnick in #427
• [Feature] Adding support for initialising TensorDicts from nested dicts by @zeenolife in #404
• [Features] Make image_size a cfg param by @nicolas-dufour in #430
• Make TensorDict.expand accept Sequence arguments by @nicolasgriffiths in #424
• [Doc] Readme revamp for efficiency/modularity display by @vmoens in #382
• [Feature] New biased_softplus semantic to allow for minimum scale setting by @nicolas-dufour in #428
• [Tutorial] Re-run tutos by @vmoens in #434
• [BugFix] mixed device_safe vs device by @vmoens in #429
• [BugFix] Explicit params and buffers by @agrotov in #436
• [BugFix] Fixed Additive noise by @nicolas-dufour in #441
• [Tests] Test loggers video saving by @bashnick in #439
• Revert "[BugFix] Fixed Additive noise" by @vmoens in #442
• [Refactor] Rename TensorDictSequence to TensorDictSequential by @ronert in #440
• [Refactor] Refactoring set*() methods for TensorDictBase class by @zeenolife in #438
• [Cleanup] Removing gym-retro interface by @vmoens in #444
• [BugFix]: Fix additive noise by @nicolas-dufour in #447
• [BugFix] CatTensors: Prepended next_ to the out_key by @ggimler3 in #449
• [BugFix] Fix AdditiveGaussian exploration tests by @vmoens in #450
• [BugFix] Wrong call to device_safe in replay buffer code by @vmoens in #454
• [BugFix] Add transform_observation_spec _R3MNet by @ymwdalex in #443
• [Doc] Add a knowledge base by @shagunsodhani in #375
• [Feature] Allow for actions and rewards to be in the reset tensordict by @vmoens in #458
• [Doc] Readme for knowledge base by @vmoens in #459
• [Feature] Added batch_lock attribute in EnvBase by @nicolas-dufour in #399
• [BugFix] deepcopy specs before transforming by @vmoens in #461
• [BugFix]: Fixed dm_control action type casting by @nicolas-dufour in #463
• [Versioning] Version 0.0.2a0 by @vmoens in #465

New Contributors

• @alexanderlobov made their first contribution in #331
• @tongbaojia made their first contribution in #344
• @omikad made their first contribution in #358
• @jaschmid-fb made their first contribution in #356
• @tcbegley made their first ...

v0.0.1-gamma

What's Changed

• Adding additional checks to TensorDict.view to remove unnecessary ViewedTensorDict object creation by @bamaxw in #319
• [BugFix]: Safe state normalization when std=0 by @vmoens in #323
• [BugFix]: gradient propagation in advantage estimates by @vmoens in #322
• [BugFix]: make training example gracefully exit by @vmoens in #326
• [Setup]: Exclude tutorials from wheels by @vmoens in #325
• [BugFix]: Tensor map for subtensordict.set_ by @vmoens in #324
• [Release]: Wheels v0.0.1c by @vmoens in #327

New Contributors

• @bamaxw made their first contribution in #319

Full Changelog: v0.0.1b...v0.0.1c

v0.0.1-beta

Highlights

Supports nested tensordicts:

• [Feature] Nested tensordicts by @vmoens in #256
• [Feature]: Index nested tensordicts using tuples by @vmoens in #262
• [Feature]: flatten nested tensordicts by @vmoens in #264

Padding for tensordicts:

• [Feature] Implement padding for tensordicts by @ajhinsvark in #257

Speed improvements:

• [Feature]: faster meta-tensor API for TensorDict by @vmoens in #272
• [Feature]: faster safetanh transform via C++ bindings by @vmoens in #289
• [Feature]: Improving training efficiency by @vmoens in #293

Logging capabilities:

• [Feature]: Refactored logging to be able to support other loggers easily by @nicolas-dufour in #270
• [Feature] Wandb logger by @nicolas-dufour in #274

Doc

• [Doc]: TorchRL demo by @vmoens in #284
• [Doc] Added TensorDict tutorial by @nicolas-dufour in #255
• [Doc] Tensordictmodule tutorial by @nicolas-dufour in #267

What's Changed

• MacOs versioning and release bugfix by @vmoens in #247
• Setup metadata by @vmoens in #248
• Fix setup instructions by @vmoens in #250
• Fix a bug when segment_tree size is exactly 2^N by @xiaomengy in #251
• Added test for RewardRescale transform by @nicolas-dufour in #252
• Empty TensorDict population in loops by @vmoens in #253
• Memmap del bugfix by @vmoens in #254
• [BugFix]: recursion error when calling permute(...).to_tensordict() by @vmoens in #260
• Differentiable PPOLoss for IRL by @vmoens in #240
• [BugFix]: avoid deleting true in_keys in TensorDictSequence by @vmoens in #261
• [Feature] Add issue and pull request template by @Benjamin-eecs in #263
• [Test]: test nested CompositeSpec by @vmoens in #265
• [Test]: test squeezed TensorDict by @vmoens in #269
• [Test]: TensorDict: test tensordict created on cuda and sub-tensordict indexed along 2nd dimension by @vmoens in #268
• Refactor the torch.stack with destination by @khmigor in #245
• Small tweaks to make the replay buffer code more consistent by @shagunsodhani in #275
• [BugFix]: Minor bugs in docstrings by @vmoens in #276
• [BugFix]: update wrong links in issue and pull request template by @Benjamin-eecs in #286
• [BugFix]: quickfix: force gym 0.24 installation until issue with rendering is resolved by @vmoens in #283
• [Doc]: remove pip install from CONTRIBUTING.md by @vmoens in #288
• [BugFix]: fix GLFW3 error when installing dm_control by @vmoens in #291
• [BugFix]: Fix examples by @vmoens in #290
• [Doc] Simplify PR template by @vmoens in #292
• [BugFix]: Replay buffer bugfixes by @vmoens in #294
• [Doc] MacOs M1 troubleshooting by @ramonmedel in #296
• [QuickFix]: update issue and pr template by @Benjamin-eecs in #303
• [Test] tests for BinarizeReward by @srikanthmg85 in #302
• [BugFix]: L2-priority for PRB by @vmoens in #305
• [Feature] Transforms: Compose.insert and TransformedEnv.insert_transform by @rmartimov in #304
• [BugFix] Fix flaky test by waiting for procs instead of sleep by @nairbv in #306
• [BugFix] Fix a build warning, setuptools/distutils import order by @nairbv in #307
• ufmt issue if imports in order requested by distutils by @nairbv in #308
• [BugFix]: Conda to pip for circleci by @vmoens in #310
• [BugFix] Support list-based boolean masks for TensorDict by @benoitdescamps in #299
• [Feature] Truly invertible tensordict permutation of dimensions by @ramonmedel in #295
• [Feature] Rename _TensorDict into TensorDictBase by @yoavnavon in #316

New Contributors

• @nicolas-dufour made their first contribution in #252
• @ajhinsvark made their first contribution in #257
• @ramonmedel made their first contribution in #296
• @srikanthmg85 made their first contribution in #302
• @rmartimov made their first contribution in #304
• @nairbv made their first contribution in #306
• @benoitdescamps made their first contribution in #299
• @yoavnavon made their first contribution in #316

Full Changelog: v0.0.1...v0.0.1b

v0.0.1-alpha

TorchRL Initial Alpha Release

TorchRL is the soon-to-be official RL domain library for PyTorch.
It contains primitives that are aimed at covering most of the modern RL research space.

Getting started with the library

Installation

The library can be installed through

$ pip install torchrl

Currently, torchrl wheels are provided for linux and macos (not M1) machines. For other architectures or for the latest features, refer to the README.md and CONTRIBUTING.md files for advanced installation instructions.

Environments

TorchRL currently supports gym and dm_control out-of-the-box. To create a gym wrapped environment, simply use

from torchrl.envs import GymEnv, GymWrapper
env = GymEnv("Pendulum-v1")
# similarly
env = GymWrapper(gym.make("Pendulum-v1"))

Environment can be transformed using the torchrl.envs.transforms module. See the environment tutorial for more information.
The ParallelEnv allows to run multiple environments in parallel.

Policy and modules

TorchRL modules interacts using TensorDict, a new data carrier class. Although it is not necessary to use it and one can find workarounds for it, we advise to use the TensorDictModule class to read tensordicts:

from torchrl.modules import TensorDictModule
>>> policy_module = nn.Linear(n_obs, n_act)
>>> policy = TensorDictModule(policy_module, 
...   in_keys=["observation"],  # keys to be read for the module input
...   out_keys=["action"],  # keys to be written with the module output
)
>>> tensordict = env.reset()
>>> tensordict = policy(tensordict)
>>> action = tensordict["action"]

By using TensorDict and TensorDictModule, you can make sure that your algorithm is robust to changes in configuration (e.g. usage of an RNN for the policy, exploration strategies etc.) TensorDict instances can be reshaped in several ways, cast to device, updated, shared among processes, stacked, concatenated etc.

Some specialized TensorDictModule are implemented for convenience: Actor, ProbabilisticActor, ValueOperator, ActorCriticOperator, ActorCriticWrapper and QValueActor can be found in actors.py.

Collecting data

DataColllectors is the TorchRL data loading class family. We provide single process, sync and async multiprocess loaders. We also provide ReplayBuffers that can be stored in memory or on disk using the various storage options.

Loss modules and advantage computation

Loss modules are provided for each algorithm class independently. They are accompanied by efficient implementations of value and advantage computation functions.
TorchRL is devoted to be fully compatible with functorch, the functional programming PyTorch library.

Examples

A bunch of examples are provided as well. Check the examples directory to learn more about exploration strategies, loss modules etc.