Code

RLHF with PPO#1005

Code

Merged

SalmanMohammadi merged 44 commits intopytorch:mainpytorch/torchtune:mainfrom SalmanMohammadi:ppoSalmanMohammadi/torchtune:ppoCopy head branch name to clipboardAug 5, 2024

+3,639-245

Commits

Commits on May 9, 2024

• Refactoring TransformerDecoder and adding value-head transformers

  SalmanMohammadi

  committed
  11d88a2

Commits on May 10, 2024

• adding ppo config and recipe to registry

  SalmanMohammadi

  committed
  2849ec5

Commits on May 12, 2024

• Merge branch 'pytorch:main' into ppo

  SalmanMohammadi

  authored
  f0c1410

Commits on May 15, 2024

• implemented ppo recipe structure, advantage and return estimation, trajectory generation, tests for advantage and return estimation

  SalmanMohammadi

  committed
  57c67bf

• finished first pass implementation of ppo. added tests for ppo loss

  SalmanMohammadi

  committed
  03cba4b

• reverting changes

  SalmanMohammadi

  committed
  f50f047

Commits on May 16, 2024

• adding lora to ppo recipe, adding lora value head component and model builders

  SalmanMohammadi

  committed
  b034af7

Commits on May 19, 2024

• added lora training, added value head checkpointing and recipe resuming from checkpoints

  SalmanMohammadi

  committed
  466b683

Commits on May 21, 2024

• removing test model builders, adding batched generation to ppo recipe, adding support for saving value head checkpoints

  SalmanMohammadi

  committed
  928037d

• fixing bug in _checkpointer.py

  SalmanMohammadi

  committed
  68b6162

Commits on May 30, 2024

• Adding support for user-provided masks in attention

  SalmanMohammadi

  committed
  65ca12a

Commits on May 31, 2024

• Merge branch 'pytorch:main' into ppo

  SalmanMohammadi

  authored
  9d8c5a8

Commits on Jun 4, 2024

• merging transformer custom masking, adding support for generation with padded inputs, added tests for new generation

  SalmanMohammadi

  committed
  b99102c

• adding functionality for truncation in generation, and further tests for generation

  SalmanMohammadi

  committed
  a1cde1c

Commits on Jun 6, 2024

• updated lora recipe to use custom generation

  SalmanMohammadi

  committed
  b032778

• Merge branch 'pytorch:main' into ppo

  SalmanMohammadi

  authored
  f126e9a

Commits on Jun 7, 2024

• added support for correct truncation and padding of responses, added rejection sampling and reward model masking, moved utils

  SalmanMohammadi

  committed
  04d514a

Commits on Jun 8, 2024

• added correct mask and position id trajectory generation, score rejection sampling masking, tests for get_causal_mask

  SalmanMohammadi

  committed
  4854908

• bugfixing in ppo recipe. refactoring ppo_utils and tests to individual files, added tests for ppo collation

  SalmanMohammadi

  committed
  c885833

• updating ppo_utils namespace

  SalmanMohammadi

  committed
  57d57fa

Commits on Jun 10, 2024

• fixing bug in collation, updating loss tests

  SalmanMohammadi

  committed
  cce5548

Commits on Jun 12, 2024

• bugfixes in masking and indexing logprobs and values, added fixed kl loss coefficient and refactored kl controllers

  SalmanMohammadi

  committed
  c289566

Commits on Jun 14, 2024

• added loss and value masking

  SalmanMohammadi

  committed
  a3fa1ea

Commits on Jun 16, 2024

• some refactoring, lots of testing and docs

  SalmanMohammadi

  committed
  c3db142

• improved early training stability by adding value head init. from reward model

  SalmanMohammadi

  committed
  589bf7d

Commits on Jun 18, 2024

• updating metrics

  SalmanMohammadi

  committed
  346c30b

• reworking causal masking

  SalmanMohammadi

  committed
  2e9d779

• freeing up memory after steps to avoid mem leaks

  SalmanMohammadi

  committed
  46b75be

Commits on Jul 16, 2024

• Merge branch 'main' into ppo

  SalmanMohammadi

  committed
  0fd885e

• cleaning up; verifying results; switching to full finetune

  SalmanMohammadi

  committed
  1942b0f

• tidying up

  SalmanMohammadi

  committed
  58d92ab

Commits on Jul 18, 2024

• detaching losses for metric logging

  SalmanMohammadi

  committed
  1fbb6dc

Commits on Jul 25, 2024

• removing 1b, merging main

  SalmanMohammadi

  committed
  65ef9dc

• merging

  SalmanMohammadi

  committed
  c7bbff1

Commits on Jul 29, 2024

• deleting logits in loss

  SalmanMohammadi

  committed
  1129f9e

Commits on Aug 2, 2024

• Merge branch 'main' into ppo

  SalmanMohammadi

  authored
  fe87dfb

• cleaning conf

  SalmanMohammadi

  committed
  662ab2c

• pYdOcLiNt

  SalmanMohammadi

  committed
  76b124f

Commits on Aug 3, 2024

• downloading weights

  SalmanMohammadi

  committed
  dc4887c

Commits on Aug 5, 2024

• addressing comments

  SalmanMohammadi

  committed
  ef85dba

• updating test

  SalmanMohammadi

  committed
  fd87fe6

• let's finish this the way we started... together

  SalmanMohammadi

  committed
  ba365a8

• Merge branch 'main' into ppo

  SalmanMohammadi

  authored
  e76304c

• lInTiNG

  SalmanMohammadi

  committed
  4e6be43