Integrate with pynbody v2 #250

apontzen · 2024-01-11T20:43:05Z

Some small discrepancies known and not to be fixed:

black holes in changa+AHF may be assigned to a different halo. BH_central will be the same as it previously was, but BH now uses the AHF-provided parent relationship to assign to a top-level halo, rather than checking whether the BH itself is a member of a higher-level halo.
gadget+rockstar pynbody calculations look different but actually are far more sensible in v2. This appears to reflect a bug in the calculation of finder_offset, which was used for pynbody v1; now that tangos is passing through finder_ids instead, this has been fixed

Outstanding discrepancies still to be fixed:

subfind-reported masses are coming out in Msol/h units rather than 1e10 Msol/h units as previously. This arises from allowing units to be attached to the quantity then stripping them out through a float(...) call. Since the philosophy is to import the raw values, this should be fixed, and it could be addressed at the same time as implementing a unified way to get property arrays from halos in pynbody.
Gadget 4 tutorial is not importing at all, seemingly because pynbody v2 won't load the gadget4 subfind hdf5 files

…ion-blocked

…r from processing other quick messages

… in shared-mem runs

…rapper for tests

…'t exist. Add test.

…halos. Transfer the KDTree over shared memory in server-shared-mem mode.

While pynbody won't build a tree that already exists, the tangos logging didn't reflect this

These may be related to async message processing, the justification for which is anyway marginal

…ting multiple properties with same region spec

…f manually polling pipes

…ssed

…o longer needed with pynbody-2?]

Make our tiny test amiga.grp files use the -1 'no-halo' and 0 'halo', rather than 0 and 1 as they used to

Make sure timesteps are available during db property writing, to fix black hole property writing

…instances running

…istent with pynbody v2)

…ng catalogues in ramses build)

Clear up un-needed imports at package level

apontzen added 30 commits January 11, 2024 19:37

Keep track of what processes are waiting for when they are communicat…

14e5bc7

…ion-blocked

Fix performance issue with getting halos

58fb151

Make pynbody server work asynchronously so it doesn't block the serve…

e33a838

…r from processing other quick messages

Fix bug with out-of-order initialisation of message classes

ae3791b

Fix bug with out-of-order initialisation of message classes

0e7cd10

Fix py3.13 deprecation warning

7c23de2

Formalise remote index list fetching in preparation for factoring out…

45f0f31

… in shared-mem runs

WIP to access halo catalogues using shared memory

2a1c895

Remove boilerplate from test_pynbody_server by introducing a launch w…

2dfdfc2

…rapper for tests

Progress towards working shared memory halo cats

77466fc

Add pyproject.toml and update import sorting

85e82bf

First working implementation of a shared memory halo catalogue

88b425b

Fix pre-commit tests

da15c6c

Fix missing import

250732c

Fix inconsistent use of object number vs object index

2a9fa3d

Performance fix

a3bde8b

Stop multiple requests being issued to the server for arrays that don…

924bb6f

…'t exist. Add test.

Use pynbody KDTree when performing region queries on large number of …

692bdad

…halos. Transfer the KDTree over shared memory in server-shared-mem mode.

Don't build the tree repeatedly

c31060f

Add test to ensure tree only gets built once

a064f67

Only build tree / log attempting to build tree once per snapshot

df056ce

While pynbody won't build a tree that already exists, the tangos logging didn't reflect this

Help track down CI hang by making pytest verbose

4e5eb5e

Attempt to address hangs in CI

214c228

These may be related to async message processing, the justification for which is anyway marginal

Cache regions so that time isn't wasted in big snapshots when calcula…

b25a31f

…ting multiple properties with same region spec

Try running CI tests in random order to see if hangs still in same place

490d24e

Fix issues with out-of-order tests

208b0df

Shot in the dark: try tidying up better in test_region_loading

154ade5

Squash warning about return values from pytest

fe3ae12

Make multiprocessing backend more efficient by using select instead o…

46d0ad6

…f manually polling pipes

Don't waste time getting a shared lock every time a new halo is proce…

a8402a3

…ssed

apontzen and others added 18 commits February 29, 2024 22:26

Add failing unit test to pick up problem in integration tests

ff050a4

Progress towards preventing any lazy-loading in property writer code

598fd4b

Fix issue with preventing lazy loading during property writer sessions

9e1bcc6

Remove sqlalchemy 1.4 from support

e1fbd7f

Fix missing quotes around python version number

3653cf3

Install pynbody 2 beta for integration testing

a1010e2

Fix changa BH importer for new pynbody

88f2710

Fix catalogue class names

1b26ac1

Fix use of deleted method HaloCatalogue.get_halo_properties

e53f4e9

tangos serve: enable specifying or finding a free port

da187fb

Correct use of finder_offset to finder_id [perhaps finder_offset is n…

542fcef

…o longer needed with pynbody-2?]

Fix issues arising from changed pynbody halo numbering

4a37d7c

Make our tiny test amiga.grp files use the -1 'no-halo' and 0 'halo', rather than 0 and 1 as they used to

Clarify the sqlalchemy load options used by during live calculation

03a6811

Make sure timesteps are available during db property writing, to fix black hole property writing

Enable webserver to be given a distinguishing name, in case multiple …

b3a908e

…instances running

Switch off very verbose sql logging

e55567a

Style fixes

be0f1d0

Clear information about errors between test runs

432c1e1

Update crossmatching to use finder_id rather than finder_offset (cons…

bc63ba4

…istent with pynbody v2)

apontzen changed the title ~~Fix some performance issues~~ Integrate with pynbody v2 Mar 9, 2024

apontzen added 9 commits March 13, 2024 08:23

Update pynbody input handler for latest pynbody v2 revisions

6bb32e3

Add safety iteration limiter to BH halo assignment

541e5de

Fix name of number_mapper in pynbody v2 beta

35b8a8c

Make integration testing use git version of pynbody (temporarily)

661b043

Fix missing 'child' property name

d47b50c

Point to updated zenodo integration test data (fixes issue with missi…

cad3973

…ng catalogues in ramses build)

Point to updated reference database/input files for integration test

978cad9

Switch back to pre-release pypi channel rather than git to get pynbody

99f8e2c

Bump version number and mark as beta until pynbody 2 is out of beta

b3a3fd2

Clear up un-needed imports at package level

apontzen merged commit 9178f5d into master Mar 16, 2024
23 checks passed

apontzen deleted the pt-loop-resume branch March 16, 2024 20:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate with pynbody v2 #250

Integrate with pynbody v2 #250

apontzen commented Jan 11, 2024 •

edited

Loading

Integrate with pynbody v2 #250

Integrate with pynbody v2 #250

Conversation

apontzen commented Jan 11, 2024 • edited Loading

apontzen commented Jan 11, 2024 •

edited

Loading