Code

Bnb/training with obs#252

Code

Draft

bnb32 wants to merge 28 commits intomainNREL/sup3r:mainfrom bnb/training_with_obsNREL/sup3r:bnb/training_with_obsCopy head branch name to clipboard

+2,101-1,130

Commits

Commits on Jan 19, 2025

• dual sampler, queue, and batch handler with obs. modifying Sup3rDataset to work with three data members.

  bnb32

  committed
  cee4058

  Copy full SHA for cee4058

• training with obs test

  bnb32

  committed
  61f6206

  Copy full SHA for 61f6206

• split up interface and abstact model

  bnb32

  committed
  e6b8818

  Copy full SHA for e6b8818

• made dual batch queue flexible enough to account for additional obs member in data

  bnb32

  committed
  ee79c13

  Copy full SHA for ee79c13

• tensorboard mixin moved to model utilities. dual queue completely absorbed WithObs variant.

  bnb32

  committed
  1b61a66

  Copy full SHA for 1b61a66

• integrated dual sampler with obs into base dual sampler.

  bnb32

  committed
  df698e1

  Copy full SHA for df698e1

• examples added to DataHandler doc string. Some instructions on sup3rwind model training added to examples README.

  bnb32

  committed
  0cb253c

  Copy full SHA for 0cb253c

• removed namedtuple from Sup3rDataset to make Sup3rDataset picklable.

  bnb32

  committed
  13664d1

  Copy full SHA for 13664d1

• parallel batch queue test added.

  bnb32

  committed
  c31ed72

  Copy full SHA for c31ed72

• namedtuple -> DsetTuple missing attr fix

  bnb32

  committed
  89c6cde

  Copy full SHA for 89c6cde

• gust added to era download variables. len dunder added to Container class to enable passing xr.Dataset and like objects directly to BatchHandlers without needing to invoke .data

  bnb32

  committed
  5976aa2

  Copy full SHA for 5976aa2

• computing before reshaping is 2x faster.

  bnb32

  committed
  ec8b739

  Copy full SHA for ec8b739

• obs_index fix - sampler needs to use hr_out_features for the obs member.

  bnb32

  committed
  e265cd5

  Copy full SHA for e265cd5

• split up calc_loss and calc_loss_obs

  bnb32

  committed
  d4c009d

  Copy full SHA for d4c009d

• Optional run_qa flag in DualRasterizer. Queue shape fix for queues with obs data

  bnb32

  committed
  ae564ac

  Copy full SHA for ae564ac

• run_qa=True default for DualRasterizer

  bnb32

  committed
  4085a07

  Copy full SHA for 4085a07

• better tracking of batch counting. (this can be tricky for parallel queueing, since batches can be sampled directly if there are none in the queue).

  bnb32

  committed
  dc933f9

  Copy full SHA for dc933f9

• missed compute call for slow batching. this was hidden by queueing and dequeueing since this would cast to tensors.

  bnb32

  committed
  f26f304

  Copy full SHA for f26f304

• Included convert to tensor in sample_batch. Test for training with max_workers > 1.

  bnb32

  committed
  b43e807

  Copy full SHA for b43e807

• cc batch handler test fix

  bnb32

  committed
  9b542be

  Copy full SHA for 9b542be

• added test for new disc with "valid" padding

  bnb32

  committed
  10dbc9c

  Copy full SHA for 10dbc9c

• parallel sampling batch sampling test.

  bnb32

  committed
  2e6ed14

  Copy full SHA for 2e6ed14

• removed workers tests. max_workers > 1 still not consistently faster. just sampling is, except for macos, but training is not.

  bnb32

  committed
  8f2218d

  Copy full SHA for 8f2218d

• Sup3rGanWithObs model subclass. Other misc model refactoring.

  bnb32

  committed
  02a9ecc

  Copy full SHA for 02a9ecc

• moved _run method to bias correction interface AbstractBiasCorrection

  bnb32

  committed
  dd70a57

  Copy full SHA for dd70a57

• moved _run method to bias correction interface AbstractBiasCorrection

  bnb32

  committed
  d93a6b1

  Copy full SHA for d93a6b1

• fix: tensorboard issue with loss obs details

  bnb32

  committed
  5afa9ed

  Copy full SHA for 5afa9ed

• Adding obs loss to logging of loss gen

  bnb32

  committed
  32edc38

  Copy full SHA for 32edc38