Updating profile & consensus docs #75

michaelbornholdt · 2021-07-21T18:13:52Z

Related to issue #73

michaelbornholdt · 2021-07-21T18:14:23Z

@shntnu @gwaygenomics @FloHu let's continue the discussion here

michaelbornholdt · 2021-07-21T18:15:49Z

michaelbornholdt · 2021-07-21T18:17:58Z

I think I still need to add a part where I explain where the profiles and consensus can be found in the repo. That should go in the top level README

michaelbornholdt · 2021-07-21T20:20:34Z

@gwaygenomics

DMSO vs. Compound Profiles

Note we generated per-well DMSO consensus signatures and per compound-dose pair consensus signatures for compounds.
The per-well DMSO profiles can help to assess plate-associated batch effects.

I have read this part of the consensus/README.md several times and I don't understand what it is trying to tell me. Can we rewrite that?

gwaybio · 2021-07-26T14:46:49Z

I have read this part of the consensus/README.md several times and I don't understand what it is trying to tell me. Can we rewrite that?

Yes! We should clarify anything that isn't clear.

It's just saying that the consensus signatures for DMSO create profiles per well instead of per treatment. Does that make sense? One might assume that we treat DMSO like any other perturbation when creating consensus profiles - that is, combine ALL replicates into a single signature. We don't do that. Instead, we collapse per well, thus creating ~24 or so (I don't remember the exact number) DMSO consensus profiles.

michaelbornholdt · 2021-07-27T17:39:07Z

@niranjchandrasekaran Can you review and merge this?
I don't have any rights here :)

@FloHu any last comments?

niranjchandrasekaran

@michaelbornholdt I made a few inline comments. Also there is typo in the feature_select box of the figure in the word "operations".

I am not approving or merging this PR yet as I want @gwaygenomics to take a look at your changes since he should be the one who decides what goes where in this repo :)

niranjchandrasekaran · 2021-07-27T19:18:00Z

README.md

@@ -19,10 +19,11 @@ The [Morphology Connectivity Hub](https://clue.io/morphology) is the primary sou
 We apply a unified, image-based profiling pipeline to all 136 384-well plates from `LINCS Pilot 1`, and all 135 384-well plates from `LKCP`.
 We use [pycytominer](https://github.com/cytomining/pycytominer) as the primary tool for image-based profiling.

-We process and store profiles in the [profiles/](profiles/) directory.
-See [`profiles/README.md`](profiles/README.md) for more details and for instructions on how to reproduce the pipeline.
+We process and store profiles in the [profiles/](profiles/) directory. By 'profiles' we refer to the level 3 data, which contains the aggregate of each well.


We call everything from level 3-5 profiles. Also doesn't this folder contain level 4a and 4b profiles?

profiles/README.md

niranjchandrasekaran · 2021-07-27T19:30:40Z

profiles/README.md


 Note here that we do not include the intermediate step of generating `.sqlite` files per plate using a tool called [cytominer-database](https://github.com/cytomining/cytominer-database).
 This repository and workflow begins after we applied cytominer-database.

+
+### Aggregation
+The [aggregation method](https://github.com/cytomining/pycytominer/blob/master/pycytominer/aggregate.py) is used twice in the workflow. Firstly, the median of all cells within a well is aggregated to one profiler per well. The aggregation method doesn't persist the metadata which is why this step is followed by an annotation step to add the MOA data and others. 


I like the idea of thinking about creating the consensus profiles as another aggregation step. But given that you have a separate section for consensus below perhaps you could talk about aggregating 4b to consensus profiles in that section and not here.

@michaelbornholdt - can you address this?

I have, there are two different sections now that explain aggregation and consensus.

niranjchandrasekaran · 2021-07-27T19:32:16Z

profiles/README.md

+[Pycytominer](https://github.com/cytomining/pycytominer) is a code base built by @gwaygenomics and @niranjchandrasekaran. 
+It allows easy processing CellProfiler data and contains all functions that were used to create the data in this repository. Below, we describe the different steps of the pipeline. Please check the pycytominer repo for more details. 
+
+Part of the pipeline, from Level 3 to Level 4b, can be found in the [profile_cells](https://github.com/broadinstitute/lincs-cell-painting/blob/master/profiles/profile_cells.py) script and the final aggregation to the consensus data is found in this [notebook](https://github.com/broadinstitute/lincs-cell-painting/blob/master/consensus/build-consensus-signatures.ipynb).  


Do you think including the notebook that contains the code for sphering should also be mentioned here?

@michaelbornholdt - can you address this?

profiles/README.md

FloHu · 2021-07-28T07:42:47Z

@niranjchandrasekaran Can you review and merge this?
I don't have any rights here :)

@FloHu any last comments?

I would still clarify what is the difference in the plate normalization and the normalization by spherization, i.e. which artifacts are corrected in the former and which ones in the latter and where they occur. For people from outside the imaging field, e.g. in some screen types people just normalize on a plate by plate basis and that takes care of the batch effects.

Overall, great job, things are a lot clearer now. Thanks for the effort!

gwaybio · 2021-07-28T13:19:59Z

Thanks @michaelbornholdt - I will provide my review once you address @niranjchandrasekaran's comments

Co-authored-by: Niranj Chandrasekaran <[email protected]>

michaelbornholdt · 2021-07-28T16:10:17Z

@gwaygenomics
Ready for you. I addressed Flos and Niranj's comments.

gwaybio

Looks great @michaelbornholdt - I made several suggested changes (note that you can commit these directly from the github GUI, so there is no need to change again on your end)

The three primary comments are:

Make sure you make formatting consistent. This means proper spacing after headings and converting to one-sentence-per-line. I tried to adjust every instance, but it's likely i missed some.
Try your best to edit this document to active (instead of passive) voice. For example, write like this: "Unlike the other normalizations, we spherize the data using the full batch (all plates)." and not like this: "Unlike the other normalizations, spherizing is done on the full batch (all plates)" It makes reading such a more pleasant experience :)
Make your png overwrite the old png - no need for a new file name

Thanks again!

README.md

consensus/README.md

profiles/README.md

Co-authored-by: Greg Way <[email protected]>

Co-authored-by: Gregory Way <[email protected]>

michaelbornholdt · 2021-08-04T16:36:13Z

@niranjchandrasekaran @gwaygenomics
I tried to rewrite in active language and accepted all the suggestions above.

michaelbornholdt · 2021-08-04T17:34:22Z

Why do the suggestions of Grag still appear in the other tab?

batch commit to clear my suggestions

gwaybio · 2021-08-04T18:07:55Z

@michaelbornholdt - it also looks like you need to add back your new-and-improved workflow figure: profiles/media/cytominer_workflow.png

michaelbornholdt · 2021-08-04T18:40:58Z

@michaelbornholdt - it also looks like you need to add back your new-and-improved workflow figure: profiles/media/cytominer_workflow.png

The newest version is there!? Or what am I missing?

gwaybio

Gotcha! Yeah, I misread the commit

LGTM! I'll go ahead and merge. Thanks for this contribution!

profiles/README.md

README.md

profiles/README.md

shntnu · 2021-08-22T13:00:25Z

Just dropping in to say that this figure was really helpful to me – I had forgotten the order of operations leading up to spherizing and this cleared it up. Thanks again @michaelbornholdt ! 🥇

michaelbornholdt added 3 commits July 16, 2021 17:26

initial push

871add3

updated information and image

b76efd9

new workflow

5b8fd3d

michaelbornholdt mentioned this pull request Jul 21, 2021

Make decisions up to consensus clearer #73

Closed

Update other readmes

5f63a0a

michaelbornholdt marked this pull request as ready for review July 27, 2021 17:37

niranjchandrasekaran requested changes Jul 27, 2021

View reviewed changes

michaelbornholdt and others added 6 commits July 28, 2021 11:40

Update profiles/README.md

f5ccc83

Co-authored-by: Niranj Chandrasekaran <[email protected]>

Update profiles/README.md

d194158

Co-authored-by: Niranj Chandrasekaran <[email protected]>

Update profiles/README.md

cb6ca41

Co-authored-by: Niranj Chandrasekaran <[email protected]>

Niranjes comments

c8f0662

Merge remote-tracking branch 'origin/updating_docs' into updating_docs

23febf0

Add a section about norm vs sphere

824021a

gwaybio requested changes Jul 28, 2021

View reviewed changes

michaelbornholdt and others added 7 commits July 29, 2021 15:25

Update README.md

ec2762e

Co-authored-by: Greg Way <[email protected]>

Update consensus/README.md

dcdc46b

Co-authored-by: Greg Way <[email protected]>

Update consensus/README.md

0b96e17

Co-authored-by: Greg Way <[email protected]>

Update consensus/README.md

640142c

Co-authored-by: Greg Way <[email protected]>

Update profiles/README.md

d5e6e99

Co-authored-by: Greg Way <[email protected]>

Update profiles/README.md

6e4717c

Co-authored-by: Greg Way <[email protected]>

Update profiles/README.md

9cc2891

Co-authored-by: Greg Way <[email protected]>

michaelbornholdt and others added 5 commits August 4, 2021 11:39

Update profiles/README.md

cc5bf02

Co-authored-by: Gregory Way <[email protected]>

Update profiles/README.md

61de818

Co-authored-by: Gregory Way <[email protected]>

Update profiles/README.md

408d3d0

Co-authored-by: Gregory Way <[email protected]>

rename image

197ccf3

move to active language

df3d539

Apply suggestions from code review

e94b9e5

batch commit to clear my suggestions

gwaybio approved these changes Aug 4, 2021

View reviewed changes

profiles/README.md Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

profiles/README.md Show resolved Hide resolved

profiles/README.md Show resolved Hide resolved

profiles/README.md Show resolved Hide resolved

profiles/README.md Show resolved Hide resolved

gwaybio added 6 commits August 4, 2021 15:11

Update profiles/README.md

c629b02

Update profiles/README.md

75449fb

Update profiles/README.md

6092f0b

Update README.md

c6d3be5

Update profiles/README.md

5fff2c6

Update profiles/README.md

5749325

gwaybio merged commit 1769b32 into broadinstitute:master Aug 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updating profile & consensus docs #75

Updating profile & consensus docs #75

michaelbornholdt commented Jul 21, 2021

michaelbornholdt commented Jul 21, 2021

michaelbornholdt commented Jul 21, 2021

michaelbornholdt commented Jul 21, 2021

michaelbornholdt commented Jul 21, 2021

DMSO vs. Compound Profiles

gwaybio commented Jul 26, 2021

michaelbornholdt commented Jul 27, 2021

niranjchandrasekaran left a comment

niranjchandrasekaran Jul 27, 2021

niranjchandrasekaran Jul 27, 2021

gwaybio Aug 4, 2021

michaelbornholdt Aug 4, 2021

niranjchandrasekaran Jul 27, 2021

gwaybio Aug 4, 2021

michaelbornholdt Aug 4, 2021

FloHu commented Jul 28, 2021

gwaybio commented Jul 28, 2021

michaelbornholdt commented Jul 28, 2021

gwaybio left a comment

michaelbornholdt commented Aug 4, 2021

michaelbornholdt commented Aug 4, 2021

gwaybio commented Aug 4, 2021

michaelbornholdt commented Aug 4, 2021

gwaybio left a comment

shntnu commented Aug 22, 2021

Updating profile & consensus docs #75

Updating profile & consensus docs #75

Conversation

michaelbornholdt commented Jul 21, 2021

michaelbornholdt commented Jul 21, 2021

michaelbornholdt commented Jul 21, 2021

michaelbornholdt commented Jul 21, 2021

michaelbornholdt commented Jul 21, 2021

DMSO vs. Compound Profiles

gwaybio commented Jul 26, 2021

michaelbornholdt commented Jul 27, 2021

niranjchandrasekaran left a comment

Choose a reason for hiding this comment

niranjchandrasekaran Jul 27, 2021

Choose a reason for hiding this comment

niranjchandrasekaran Jul 27, 2021

Choose a reason for hiding this comment

gwaybio Aug 4, 2021

Choose a reason for hiding this comment

michaelbornholdt Aug 4, 2021

Choose a reason for hiding this comment

niranjchandrasekaran Jul 27, 2021

Choose a reason for hiding this comment

gwaybio Aug 4, 2021

Choose a reason for hiding this comment

michaelbornholdt Aug 4, 2021

Choose a reason for hiding this comment

FloHu commented Jul 28, 2021

gwaybio commented Jul 28, 2021

michaelbornholdt commented Jul 28, 2021

gwaybio left a comment

Choose a reason for hiding this comment

michaelbornholdt commented Aug 4, 2021

michaelbornholdt commented Aug 4, 2021

gwaybio commented Aug 4, 2021

michaelbornholdt commented Aug 4, 2021

gwaybio left a comment

Choose a reason for hiding this comment

shntnu commented Aug 22, 2021