P2 01. Testing the method on subsets of the LINCS dataset

[EchteRobert]

LINCS contains 6 dose points: 0.04 µM, 0.12 µM, 0.37 µM, 1.11 µM, 3.33 µM, and 10 µM. For my experiments, I will use the highest dose (10 µM) as the training set and the validation set. The model is trained to create profiles that attract replicate compound profiles and repel non-replicate compound profiles. It is then validated by evaluating the ability of these profiles to predict MoAs (or find sister compounds). Finally, the model will be tested on the 3.3 µM dose point data as a hold-out set. This data should look significantly different from the training and validation data.

I will follow the same data exclusion protocol as Michael did in his research:

“The evaluation metrics all require the number of compounds in an MOA to be at least two (MOA class size of at least two). A compound that does not have other compounds annotated with the same MOA cannot have a precision value and does not affect Enrichment and Hit@k. Additionally, there is no use for compounds with unknown MOA labels, because I can not apply any metrics. Finally, I used only a single dose per compound to make experimental iterations practical. I picked the 10uM dose point because we found that this dose produces the strongest phenotypes across compounds without reducing the ability to group MOAs [42]. These three requirements lead to a subset of the LINCS data in which I 1) only keep the highest dose of each perturbation, 2) delete all compounds that have unknown MOA labels, and 3) drop all compounds with MOA sizes smaller than two. All further processing and experiment are based on this subset of data. This subset process is documented in my repository.”

Which should result in similar data numbers:

“The resulting (sub-selected) dataset comprises 8,818 wells spanning over 136 plates, five different batches, and 18.2 million cells. Each of the 1,144 perturbations (compounds) have five technical replicates - each on a separate plate but in the same position on the plate (same well position). Four of these technical replicate wells were produced in the same batch. This is highly relevant for training and technical artifacts, because I expect cells in the same well and same batch to have similar technical artifacts, but the corresponding wells from a different batch may look very different from one another. Each of the 136 plates has 24 negative controls (DMSO) distributed across the plate (3264 DMSOs in total). The 1,144 compounds can be categorized into 235 MOAs, and every MOA has at least two compounds (MOA class size > 1). Some MOAs, such as phosphodiesterase inhibitor, have an MOA class size of 35, while most other MOAs are only found for two compounds.”

I have put some relevant quotes from Michael's work and the LINCS manuscript here: https://docs.google.com/document/d/1z2U5o91vzBwB-4xtryYn5d3kSWJi8ZSerE_MAzdLT_0/edit#heading=h.sbi6l2r6p5ec

[EchteRobert]

Experiment 0: test the method on 3 plates from LINCS (mixed doses)

I will start off with plates SQ00015231, SQ00015232, and SQ00015233. First I will have to remove all low dose (<3.33 µM) wells from each plate. Then I will train the model as usual, using the highest dose (10 µM) replicates for the contrastive learning objective. Finally, the model will be evaluated on MoA prediction using the highest dose (10 µM) wells. I will not use the 3.33 µM wells during training/validation and only use those during the final testing phase.

I have now updated the scripts to work for the LINCS data, but I still need to filter the dose points before training. This is next up on the list.

I have already trained a model using the original setup but without filtering doses. This means that replicate compound profiles are forced to attract during training, even if they were created with different doses.

Main takeaway

The replicate mAP was very high but no improvement was found in MoA prediction (both were also evaluated using different dose points).

Click to expand!

Training and validation curves

MoA prediction

plate	mAP model	mAP BM
SQ00015232	0.04	0.04
SQ00015233	0.02	0.03
SQ00015231	0.03	0.04

Replicate prediction

plate	Training mAP model	Training mAP BM
SQ00015232	0.89	0.38
SQ00015233	0.86	0.35
SQ00015231	0.85	0.37

[EchteRobert]

Experiment 1: test the method on 3 plates from LINCS (only 10 µM dose)

I trained a model on plates SQ00015231, SQ00015232, and SQ00015233 using only the 10 µM dose point and training on forming replicates between compounds across plates. The latter is new compared to previous experiments. Earlier on when experimenting on the Stain datasets, I found that training on across plate replicates reduced the model's ability to generalize to hold-out compounds. This may also negatively influence the model's ability to predict MoAs. It's possible that more plates are needed for generalization anyways (3 plates did not work well for Stain data either).

Main takeaways

The model improves replicate prediction significantly: it's nearly perfect (1.0 mAP). However, as expected it does not improve MoA prediction. I will probably have to use more plates during training to improve generalization.

Results

Training curves

Loss curves of experiment 0 and experiment 1

mAP of replicate and MoA prediction

mAPs replicate prediction

plate	Training mAP model	Training mAP BM
SQ00015231_SQ00015232_SQ00015233	1	0.59

mAP model MoA prediction

compound	AP	precision at R
cyclooxygenase inhibitor	0.0385604	0
leukotriene synthesis inhibitor	0.0255024	0
integrin antagonist	0.0220234	0

mAPs MoA prediction

plate	mAP model	mAP BM
SQ00015231_SQ00015232_SQ00015233	0.03	0.08

Baseline LINCS

It looks like the LINCS results were generated by

1. creating the profiles
2. normalizing/spherizing per plate to correct for batch effects
3. creating consensus profiles per compound
4. applying feature selection on the consensus profiles per plate
5. putting all of these processed consensus profiles together in a single heap and calculating the metrics between them per dose point (and thus across plate).

From the manuscript:

"Because the negative control DMSO profiles were noisy due to technical artifacts, we applied a spherize transform (also known as whitening) to mitigate the impact of well positioning. More specifically, we used the zero-phase whitening filters (ZCA) solution calculated on the profile correlation matrix (ZCA-cor) to minimize the absolute distance between the transformed profiles and the untransformed profiles. We also formed consensus signatures (level 5) by moderated z-score (MODZ) aggregating all replicate wells across plate maps into a single signature. We applied feature selection to the consensus signatures and batch effect corrected profiles separately using the same operations as described above. We applied the same pipeline to batch 1 (A549) and batch 2 (A549, MCF7, and U2OS) Cell Painting datasets."

[EchteRobert]

Experiment 2: test the method on 6 plates from LINCS (only 10 µM dose)

Using plates SQ00015224, SQ00015223, SQ00015230, SQ00015231, SQ00015229, SQ00015233, and SQ00015232 as the training data and considering across plate replicates, the model was trained in the same way as in Exp. 1. Using more plates during training is expected to improve MoA prediction performance compared to Exp. 1.

Main takeaways

The model is still not generalizing to MoA prediction. This is probably due to the lack of data. In the next experiment I will try to use ~30 plates.

Results

The model is converging slower than the previous models - I think because there are more compounds that can be used for replicate training, increasing the training task complexity. We can also see that there are more MoAs that are taken into account for the evaluation of the MoA prediction task. The model is beating BM performance in the replicating task, but not in the MoA prediction task - the performance for this task remains near random (as in Exp. 1). The total number of samples, due to filtering of lower dose points, is only the size of one plate (~360 samples). This could explain the lack of generalization of the model. It's possible I will need to at least use 5 times as many plates (~30).

Loss curves here!

mAP of replicate and MoA prediction

mAPs replicate prediction

plate	Training mAP model	Training mAP BM
SQ00015223_SQ00015224_SQ00015230_SQ00015231_SQ00015232_SQ00015233	0.87	0.67

mAP model MoA prediction

compound	AP	precision at R
leukotriene synthesis inhibitor	0.0475004	0
cyclooxygenase inhibitor	0.0436489	0
histamine receptor antagonist	0.0307984	0
integrin antagonist	0.0279527	0
HCV inhibitor	0.0162529	0
MEK inhibitor	0.0158495	0
dopamine receptor antagonist	0.0113624	0

mAPs MoA prediction

plate	mAP model	mAP BM
SQ00015223_SQ00015224_SQ00015230_SQ00015231_SQ00015232_SQ00015233	0.03	0.09

[EchteRobert]

Experiment 3: test the method on 26 plates with 1439 wells on the server (bugged)

I have now moved the python environment and all of the data to John's server to accommodate for the larger memory requirements of the LINCS dataset. Effectively these 26 plates only correspond to ~4 plates in training data, which is nowhere close to the 15 I used for training the final model on the Stain datasets. I have monitored all the steps I had to take and updated those in the README of this repository for future reference. The model was trained using the same default hyperparameters as before. The updated pipeline uses 1781 features instead of 1783, possibly due to some Image or Metadata features being filtered out correctly now.

Main takeaways

Something went wrong during training. I expect the problem to lie with the data. I will investigate the issue and improve the training pipeline.

After investigating a bit, the issue might be related to same compounds getting different labels.

Results

The loss curves show that the model was not able to learn the task correctly. The validation mAP does keep increasing (although on a much smaller scale than it would normally), which may indicate that the training procedure is correct. I will investigate the training data to see if something is wrong there.

Loss curves

[EchteRobert]

Experiment 3: test the method on 26 plates with 458 wells on the server

I resolved the issue and reran the experiment as described in #12 (comment). I now aggregate all the wells from all the plates first. I now also add the perturbation and moa information per well and then remove any wells that do not contain any perturbation information. Finally, I remove all compounds which occur only once in the entire dataset. This results in:

Removed 318 wells due to missing annotation of pert_iname and moa.
Removed 663 unique compound wells.
Using 458 wells

Note that this is only equal to ~1.5 plates worth of data!

Main takeaways

The model is still not better at MoA prediction than the benchmark. In previous experiments, the model only started beating this performance after I started using >~9 plates. Another method would be to add more data augmentation, i.e., random sampling from the wells.

Results

The model now trains correctly and we can see that the loss curves converge properly. Percent replicating is nearly perfect, while the model still lacks improved MoA prediction compared to the benchmark.

Loss curves

mean average precisions scores

mAP replicate prediction

plate	Training mAP model	Training mAP BM	Validation mAP model	Validation mAP BM
all plates*	0.98	0.51	nan	nan

plate	mAP model	mAP BM
all plates*	0.04	0.07

*SQ00015199_SQ00015200_SQ00015201_SQ00015202_SQ00015203_SQ00015204_SQ00015205_SQ00015206_SQ00015207_SQ00015208_SQ00015209_SQ00015210_SQ00015211_SQ00015212_SQ00015214_SQ00015215_SQ00015216_SQ00015217_SQ00015218_SQ00015219_SQ00015220_SQ00015221_SQ00015222_SQ00015223_SQ00015224_SQ00015230

[EchteRobert]

Experiment 4: test the method on 38 plates with 1008 wells

I added more plates and repeated the experiment. The number of wells used is still only equal to ~3 plates total. I will have to add even more to make this work.

Removed 477 wells due to missing annotation of pert_iname and moa.
Removed 624 unique compound wells.
Using 1008 wells

Results here!

replicate prediction

plate	Training mAP model	Training mAP BM	Validation mAP model	Validation mAP BM
all plates*	0.89	0.38	nan	nan

MoA prediction

plate	mAP model	mAP BM
all plates*	0.07	0.08

*SQ00015153_SQ00015167_SQ00015168_SQ00015169_SQ00015170_SQ00015171_SQ00015172_SQ00015173_SQ00015194_SQ00015195_SQ00015196_SQ00015198_SQ00015199_SQ00015200_SQ00015201_SQ00015202_SQ00015203_SQ00015204_SQ00015205_SQ00015206_SQ00015207_SQ00015208_SQ00015209_SQ00015210_SQ00015211_SQ00015212_SQ00015214_SQ00015215_SQ00015216_SQ00015217_SQ00015218_SQ00015219_SQ00015220_SQ00015221_SQ00015222_SQ00015223_SQ00015224_SQ00015230

[EchteRobert]

Experiment 5: test the method on 81 plates with 3306 - BS 36 and BS 72

Removed 958 wells due to missing annotation of pert_iname and moa.
Removed 226 unique compound wells.
Using wells 3306
Loading 81 plates with 783 unique compounds. 

Main takeaways

• Training on this number of plates improves the MoA prediction accuracy slightly
• Using a batch size of 72 instead of 32 achieves slightly better performance

Results

batch size 36
MoA prediction

plate	mAP model	mAP BM
all plates	0.0573	0.04

Replicate prediction

plate	Training mAP model	Training mAP BM
all plates	0.8	0.3

batch size 72
MoA prediction

plate	Training mAP model	Training mAP BM
all plates	0.0594	0.04

Replicate prediction

plate	Training mAP model	Training mAP BM
all plates	0.82	0.3

MLP

compound	AP	precision at R
phosphodiesterase inhibitor	0.03	0.06
estrogen receptor antagonist / selective estrogen receptor modulator (SERM)	0.03	0
cyclooxygenase inhibitor	0.03	0.08
fungal squalene epoxidase inhibitor	0.04	0
isocitrate dehydrogenase inhibitor	0.05	0
topoisomerase inhibitor	0.05	0.12
ribonucleotide reductase inhibitor	0.05	0
diacylglycerol O acyltransferase inhibitor	0.08	0.08
AKT inhibitor	0.09	0.29
endothelin receptor antagonist	0.1	0.02
mTOR inhibitor / PI3K inhibitor	0.12	0.09
vitamin D receptor agonist	0.12	0.3
p38 MAPK inhibitor	0.13	0.18
PLK inhibitor	0.18	0.21
JAK inhibitor	0.19	0.36
PARP inhibitor	0.2	0.03
dihydroorotate dehydrogenase inhibitor	0.21	0.34
HSP inhibitor	0.22	0.1
dihydrofolate reductase inhibitor	0.25	0.43
mTOR inhibitor	0.29	0.52
HDAC inhibitor	0.3	0.72
CDK inhibitor	0.3	0.43
sulfonylurea	0.33	0.28
Aurora kinase inhibitor	0.38	0.73
voltage-gated sodium channel blocker	0.39	0.19
RAF inhibitor	0.41	0.18
proteasome inhibitor	0.41	0.19
tubulin polymerization inhibitor	0.45	0.43
MEK inhibitor	0.53	0.69
dehydrogenase inhibitor / inositol monophosphatase inhibitor	1	1

BM

compound	AP	precision at R
histamine receptor antagonist	0.02	0.03
adrenergic receptor antagonist	0.02	0.02
cyclooxygenase inhibitor	0.02	0.02
Bruton's tyrosine kinase (BTK) inhibitor	0.02	0
PI3K inhibitor	0.02	0
topoisomerase inhibitor	0.03	0.09
phosphodiesterase inhibitor	0.03	0.03
HMGCR inhibitor	0.03	0.04
fungal squalene epoxidase inhibitor	0.03	0.03
cholesterol inhibitor	0.03	0
AKT inhibitor	0.03	0.07
retinoid receptor agonist	0.04	0.08
HIV protease inhibitor	0.04	0.07
dihydroorotate dehydrogenase inhibitor	0.07	0.07
JAK inhibitor	0.08	0.11
mTOR inhibitor / PI3K inhibitor	0.09	0
dihydrofolate reductase inhibitor	0.09	0.18
mTOR inhibitor	0.09	0.04
HDAC inhibitor	0.14	0.43
RAF inhibitor	0.19	0.14
HSP inhibitor	0.23	0.22
MEK inhibitor	0.26	0.5
voltage-gated sodium channel blocker	0.28	0.12
CDK inhibitor	0.3	0.44
PARP inhibitor	0.31	0.31
Aurora kinase inhibitor	0.35	0.56
PLK inhibitor	0.37	0.54
tubulin polymerization inhibitor	0.43	0.52
proteasome inhibitor	0.47	0.35
dehydrogenase inhibitor / inositol monophosphatase inhibitor	0.8	0.9

Plate barcodes

*SQ00014813_SQ00014814_SQ00014815_SQ00014816_SQ00014817_SQ00014818_SQ00014819_SQ00014820_SQ00015041_SQ00015042_SQ00015043_SQ00015044_SQ00015045_SQ00015046_SQ00015047_SQ00015048_SQ00015049_SQ00015050_SQ00015051_SQ00015052_SQ00015053_SQ00015054_SQ00015055_SQ00015056_SQ00015057_SQ00015058_SQ00015059_SQ00015096_SQ00015097_SQ00015098_SQ00015099_SQ00015100_SQ00015101_SQ00015102_SQ00015103_SQ00015105_SQ00015106_SQ00015107_SQ00015108_SQ00015109_SQ00015110_SQ00015111_SQ00015112_SQ00015153_SQ00015167_SQ00015168_SQ00015169_SQ00015170_SQ00015171_SQ00015172_SQ00015173_SQ00015194_SQ00015195_SQ00015196_SQ00015198_SQ00015199_SQ00015200_SQ00015201_SQ00015202_SQ00015203_SQ00015204_SQ00015205_SQ00015206_SQ00015207_SQ00015208_SQ00015209_SQ00015210_SQ00015211_SQ00015212_SQ00015214_SQ00015215_SQ00015216_SQ00015217_SQ00015218_SQ00015219_SQ00015220_SQ00015221_SQ00015222_SQ00015223_SQ00015224_SQ00015230

[EchteRobert]

Experiment 6: test the method on 108 plates with 4691 wells - BS 72

Main takeaways

Similar to the results from Experiment 5.

Next up

I will try some different training approaches to see if I can improve model performance further. However, it's possible that this small increase is all that we can get due to the large number of different compounds in the dataset.
Things I want to try:

• Only use compounds with X replicates or more (probably with X=4). [Note: I have already tried calculating the mAP using min_replicates=4 and the performance only slightly decreased.]
• Train with a subset of compounds. Perhaps this number of compounds is too high making the training task too complex and training on only 90 different compounds, like in the JUMP data, would increase performance.
• Add more data augmentation
• Add more data
• Hyperparameter optimization

I will do the last step last as optimizing these parameters again might mean that the current hyperparameter settings are not robust to new data.

Loss curves

Results

Replicate prediction

plate	Training mAP model	Training mAP BM
all plates	0.8	0.26

MoA prediction

plate	mAP model	mAP BM
all plates	0.07	0.05

compound	AP	precision at R
ribonucleotide reductase inhibitor	0.06	0.03
topoisomerase inhibitor	0.06	0.17
HIV protease inhibitor	0.06	0.09
EGFR inhibitor	0.07	0.23
JAK inhibitor	0.08	0.21
BCL inhibitor	0.1	0
retinoid receptor agonist	0.1	0.39
vitamin D receptor agonist	0.1	0.28
dihydroorotate dehydrogenase inhibitor	0.13	0.25
glucocorticoid receptor agonist	0.13	0.26
mTOR inhibitor	PI3K inhibitor	0.13
voltage-gated sodium channel blocker	0.15	0.04
inosine monophosphate dehydrogenase inhibitor	0.15	0
AKT inhibitor	0.16	0.41
mTOR inhibitor	0.2	0.24
HDAC inhibitor	0.21	0.6
PLK inhibitor	0.23	0.46
PARP inhibitor	0.25	0.42
Aurora kinase inhibitor	0.27	0.41
dihydrofolate reductase inhibitor	0.33	0.67
protein synthesis inhibitor	0.33	0.61
CDK inhibitor	0.33	0.54
tubulin polymerization inhibitor	0.55	0.65
MEK inhibitor	0.68	0.87
proteasome inhibitor	0.69	0.68
HSP inhibitor	0.81	0.83
bacterial 50S ribosomal subunit inhibitor	0.9	0.82
MDM inhibitor	0.94	0.98
purine antagonist	0.97	1
dehydrogenase inhibitor	inositol monophosphatase inhibitor	1

compound	AP	precision at R
dopamine receptor antagonist	0.03	0.08
phosphodiesterase inhibitor	0.03	0.04
dihydroorotate dehydrogenase inhibitor	0.04	0.04
RAF inhibitor	0.04	0.12
EGFR inhibitor	0.04	0.08
glucocorticoid receptor agonist	0.05	0.12
topoisomerase inhibitor	0.05	0.22
estrogen receptor antagonist	selective estrogen receptor modulator (SERM)	0.06
ribonucleotide reductase inhibitor	0.07	0.1
JAK inhibitor	0.07	0.15
retinoid receptor agonist	0.07	0.22
dihydrofolate reductase inhibitor	0.09	0.18
mTOR inhibitor	PI3K inhibitor	0.1
PARP inhibitor	0.1	0.15
inosine monophosphate dehydrogenase inhibitor	0.12	0.07
mTOR inhibitor	0.13	0.18
MDM inhibitor	0.15	0.05
bacterial 50S ribosomal subunit inhibitor	0.15	0.2
voltage-gated sodium channel blocker	0.16	0.17
HDAC inhibitor	0.18	0.6
protein synthesis inhibitor	0.19	0.34
HSP inhibitor	0.28	0.47
Aurora kinase inhibitor	0.29	0.47
CDK inhibitor	0.32	0.51
MEK inhibitor	0.38	0.74
tubulin polymerization inhibitor	0.43	0.54
PLK inhibitor	0.46	0.69
proteasome inhibitor	0.6	0.59
dehydrogenase inhibitor	inositol monophosphatase inhibitor	0.68
purine antagonist	0.71	0.61

Plate barcodes

SQ00014813_SQ00014814_SQ00014815_SQ00014816_SQ00014817_SQ00014818_SQ00014819_SQ00014820_SQ00015041_SQ00015042_SQ00015043_SQ00015044_SQ00015045_SQ00015046_SQ00015047_SQ00015048_SQ00015049_SQ00015050_SQ00015051_SQ00015052_SQ00015053_SQ00015054_SQ00015055_SQ00015056_SQ00015057_SQ00015058_SQ00015059_SQ00015096_SQ00015097_SQ00015098_SQ00015099_SQ00015100_SQ00015101_SQ00015102_SQ00015103_SQ00015105_SQ00015106_SQ00015107_SQ00015108_SQ00015109_SQ00015110_SQ00015111_SQ00015112_SQ00015127_SQ00015128_SQ00015129_SQ00015130_SQ00015131_SQ00015132_SQ00015133_SQ00015134_SQ00015135_SQ00015136_SQ00015137_SQ00015138_SQ00015139_SQ00015140_SQ00015141_SQ00015142_SQ00015143_SQ00015144_SQ00015145_SQ00015146_SQ00015147_SQ00015148_SQ00015149_SQ00015150_SQ00015151_SQ00015152_SQ00015153_SQ00015154_SQ00015167_SQ00015168_SQ00015169_SQ00015170_SQ00015171_SQ00015172_SQ00015173_SQ00015194_SQ00015195_SQ00015196_SQ00015198_SQ00015199_SQ00015200_SQ00015201_SQ00015202_SQ00015203_SQ00015204_SQ00015205_SQ00015206_SQ00015207_SQ00015208_SQ00015209_SQ00015210_SQ00015211_SQ00015212_SQ00015214_SQ00015215_SQ00015216_SQ00015217_SQ00015218_SQ00015219_SQ00015220_SQ00015221_SQ00015222_SQ00015223_SQ00015224_SQ00015230

[EchteRobert]

Experiment 6: results updated with randomly shuffled baseline and normalized mAP scores

I recalculated the results with updates to the eval script. A randomly shuffled baseline is also calculated and subtracted from the mAP scores. Here I divide the mAP by the sum of the labels (i.e., the number of positive samples in the rank order). This was incorrect and the mAP is recalculated again in a new comment below (#12 (comment)).

Main takeaways

• The mAP values are much lower (~3x)
• The shuffled value indeed approximates the formula's outcome
• Due to the shuffled mAP being so low, I now feel more confident that the model's profiles significantly contribute to the Replicate and MoA prediction.

normalized mAP scores (with shuffled baseline)

Replicate prediction

plate	Training mAP model	Training mAP BM	Training mAP shuffled
all plates	0.2526	0.0731	0.0002

Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=56.98555674994308, pvalue=0.0)

MoA prediction

plate	mAP model	mAP BM	mAP shuffled
all plates	0.0054	0.0032	0.0002

Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=4.663893266927154, pvalue=3.176144361395387e-06)

Plate barcodes

SQ00014813_SQ00014814_SQ00014815_SQ00014816_SQ00014817_SQ00014818_SQ00014819_SQ00014820_SQ00015041_SQ00015042_SQ00015043_SQ00015044_SQ00015045_SQ00015046_SQ00015047_SQ00015048_SQ00015049_SQ00015050_SQ00015051_SQ00015052_SQ00015053_SQ00015054_SQ00015055_SQ00015056_SQ00015057_SQ00015058_SQ00015059_SQ00015096_SQ00015097_SQ00015098_SQ00015099_SQ00015100_SQ00015101_SQ00015102_SQ00015103_SQ00015105_SQ00015106_SQ00015107_SQ00015108_SQ00015109_SQ00015110_SQ00015111_SQ00015112_SQ00015127_SQ00015128_SQ00015129_SQ00015130_SQ00015131_SQ00015132_SQ00015133_SQ00015134_SQ00015135_SQ00015136_SQ00015137_SQ00015138_SQ00015139_SQ00015140_SQ00015141_SQ00015142_SQ00015143_SQ00015144_SQ00015145_SQ00015146_SQ00015147_SQ00015148_SQ00015149_SQ00015150_SQ00015151_SQ00015152_SQ00015153_SQ00015154_SQ00015167_SQ00015168_SQ00015169_SQ00015170_SQ00015171_SQ00015172_SQ00015173_SQ00015194_SQ00015195_SQ00015196_SQ00015198_SQ00015199_SQ00015200_SQ00015201_SQ00015202_SQ00015203_SQ00015204_SQ00015205_SQ00015206_SQ00015207_SQ00015208_SQ00015209_SQ00015210_SQ00015211_SQ00015212_SQ00015214_SQ00015215_SQ00015216_SQ00015217_SQ00015218_SQ00015219_SQ00015220_SQ00015221_SQ00015222_SQ00015223_SQ00015224_SQ00015230

[EchteRobert]

Experiment 7: training with data with at least 4 replicates

I now filtered the training data for compounds that had at least 4 replicates (though 4 is the highest number of replicates I found in this dataset).

Main takeaways

The results are worse than using all data. I will try training with minimum replicates = 3 and if that does not work I will always include all data in the future.

normalized mAP scores (with shuffled baseline)

Replicate prediction

plate	Training mAP model	Training mAP BM	Training mAP shuffled
all plates	0.1953	0.0731	0.0002

MoA prediction

plate	mAP model	mAP BM	mAP shuffled
all plates	0.0046	0.0032	0.0002

Plate barcodes

SQ00014813_SQ00014814_SQ00014815_SQ00014816_SQ00014817_SQ00014818_SQ00014819_SQ00014820_SQ00015041_SQ00015042_SQ00015043_SQ00015044_SQ00015045_SQ00015046_SQ00015047_SQ00015048_SQ00015049_SQ00015050_SQ00015051_SQ00015052_SQ00015053_SQ00015054_SQ00015055_SQ00015056_SQ00015057_SQ00015058_SQ00015059_SQ00015096_SQ00015097_SQ00015098_SQ00015099_SQ00015100_SQ00015101_SQ00015102_SQ00015103_SQ00015105_SQ00015106_SQ00015107_SQ00015108_SQ00015109_SQ00015110_SQ00015111_SQ00015112_SQ00015127_SQ00015128_SQ00015129_SQ00015130_SQ00015131_SQ00015132_SQ00015133_SQ00015134_SQ00015135_SQ00015136_SQ00015137_SQ00015138_SQ00015139_SQ00015140_SQ00015141_SQ00015142_SQ00015143_SQ00015144_SQ00015145_SQ00015146_SQ00015147_SQ00015148_SQ00015149_SQ00015150_SQ00015151_SQ00015152_SQ00015153_SQ00015154_SQ00015167_SQ00015168_SQ00015169_SQ00015170_SQ00015171_SQ00015172_SQ00015173_SQ00015194_SQ00015195_SQ00015196_SQ00015198_SQ00015199_SQ00015200_SQ00015201_SQ00015202_SQ00015203_SQ00015204_SQ00015205_SQ00015206_SQ00015207_SQ00015208_SQ00015209_SQ00015210_SQ00015211_SQ00015212_SQ00015214_SQ00015215_SQ00015216_SQ00015217_SQ00015218_SQ00015219_SQ00015220_SQ00015221_SQ00015222_SQ00015223_SQ00015224_SQ00015230

[EchteRobert]

Experiment 8: training with only 88 different compounds with each 4 replicates

To reduce complexity I only train on 10% of the currently preprocessed data. The idea is to mimic the complexity of the JUMP data. However, the amount of data is drastically reduced with respect to JUMP as there are no replicate plates of the same compounds, reducing the total number of replicates of a single compound to 4 instead of 4 times the number of replicate plates.

Main takeaways

Due to the lack of training samples, this task is more complex than when using more different compounds but also more samples. In the future, I will use all samples I have at my disposal.

Results

mAP results

Replicate prediction
Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=-1.0605392922634074, pvalue=0.28892709582574516)

plate	Training mAP model	Training mAP BM	Training mAP shuffled	Validation mAP model	Validation mAP BM	Validation mAP shuffled
all plates	0.1957	0.1048	0.0002	0.057	0.0695	0.0002

MoA prediction

compound	AP	precision at R
purine antagonist	0.122	0.25
purine antagonist	0.1304	0.5
MDM inhibitor	0.1343	0.5
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1353	0.5
purine antagonist	0.1411	0.75
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1433	0.75
MDM inhibitor	0.1459	0.5
purine antagonist	0.1604	0.5
MDM inhibitor	0.162	0.75
purine antagonist	0.1652	0.75
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1702	0.75
MDM inhibitor	0.1706	0.75
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1752	0.75
purine antagonist	0.1752	0.75
MDM inhibitor	0.1753	0.75
purine antagonist	0.176	0.75
inosine monophosphate dehydrogenase inhibitor	0.1806	0.75
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1822	1
voltage-gated sodium channel blocker	0.1859	0.5
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1886	1
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1933	1
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1933	1
MDM inhibitor	0.1933	1
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.2	1
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.2	1
MDM inhibitor	0.2	1
MDM inhibitor	0.2	1
purine antagonist	0.2	1
MDM inhibitor	0.2	1
purine antagonist	0.25	1

compound	AP	precision at R
voltage-gated sodium channel blocker	0.0562	0.25
proteasome inhibitor	0.0577	0.5
proteasome inhibitor	0.0596	1
protein synthesis inhibitor	0.0602	1
inosine monophosphate dehydrogenase inhibitor	0.0607	0
inosine monophosphate dehydrogenase inhibitor	0.0625	0
bacterial 50S ribosomal subunit inhibitor	0.0753	0.5
purine antagonist	0.0797	0.25
purine antagonist	0.0798	0.25
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.085	0.25
MDM inhibitor	0.0894	0.25
inosine monophosphate dehydrogenase inhibitor	0.0907	0.25
purine antagonist	0.1008	0.5
bacterial 50S ribosomal subunit inhibitor	0.1115	0.5
voltage-gated sodium channel blocker	0.1127	0.25
purine antagonist	0.1128	0.5
voltage-gated sodium channel blocker	0.1137	0.25
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1198	0.5
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.145	0.5
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1525	0.75
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1531	0.75
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1534	0.75
purine antagonist	0.1545	0.25
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1633	1
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1661	1
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.1689	0.75
purine antagonist	0.2219	0.75
purine antagonist	0.25	1
purine antagonist	0.25	1
purine antagonist	0.25	1

Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=1.9471959035173303, pvalue=0.05155618670367708)

plate	mAP model	mAP BM	mAP shuffled
all plates	0.004	0.0032	0.0002

Plate barcodes

SQ00014813_SQ00014814_SQ00014815_SQ00014816_SQ00014817_SQ00014818_SQ00014819_SQ00014820_SQ00015041_SQ00015042_SQ00015043_SQ00015044_SQ00015045_SQ00015046_SQ00015047_SQ00015048_SQ00015049_SQ00015050_SQ00015051_SQ00015052_SQ00015053_SQ00015054_SQ00015055_SQ00015056_SQ00015057_SQ00015058_SQ00015059_SQ00015096_SQ00015097_SQ00015098_SQ00015099_SQ00015100_SQ00015101_SQ00015102_SQ00015103_SQ00015105_SQ00015106_SQ00015107_SQ00015108_SQ00015109_SQ00015110_SQ00015111_SQ00015112_SQ00015127_SQ00015128_SQ00015129_SQ00015130_SQ00015131_SQ00015132_SQ00015133_SQ00015134_SQ00015135_SQ00015136_SQ00015137_SQ00015138_SQ00015139_SQ00015140_SQ00015141_SQ00015142_SQ00015143_SQ00015144_SQ00015145_SQ00015146_SQ00015147_SQ00015148_SQ00015149_SQ00015150_SQ00015151_SQ00015152_SQ00015153_SQ00015154_SQ00015167_SQ00015168_SQ00015169_SQ00015170_SQ00015171_SQ00015172_SQ00015173_SQ00015194_SQ00015195_SQ00015196_SQ00015198_SQ00015199_SQ00015200_SQ00015201_SQ00015202_SQ00015203_SQ00015204_SQ00015205_SQ00015206_SQ00015207_SQ00015208_SQ00015209_SQ00015210_SQ00015211_SQ00015212_SQ00015214_SQ00015215_SQ00015216_SQ00015217_SQ00015218_SQ00015219_SQ00015220_SQ00015221_SQ00015222_SQ00015223_SQ00015224_SQ00015230

[EchteRobert]

Experiment 6: recalculating the mAP again

I now calculate the mAP by correcting it with the random baseline:
mAP = mAP - sum(labels)/len(labels)
This increases the mAP again as the random baseline is very low: ~0.00086 on average.

Main takeaways

The mAP is now closer to what would be expected according to the results found in the LINCS manuscript, where the majority of the results are found between 0 and 0.1.

Results replicate prediction

Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=89.0721529525361, pvalue=0.0)

plate	Training mAP model	Training mAP BM	Training mAP shuffled
all plates	0.8006	0.2555	0

Results MoA prediction

compound	AP	precision at R
glucose dependent insulinotropic receptor agonist	0.0464	0.0317
ribonucleotide reductase inhibitor	0.0526	0.0471
HIV protease inhibitor	0.0535	0.1053
EGFR inhibitor	0.0549	0.0671
JAK inhibitor	0.073	0.0957
retinoid receptor agonist	0.0942	0.129
BCL inhibitor	0.0944	0
vitamin D receptor agonist	0.1001	0.1015
dihydroorotate dehydrogenase inhibitor	0.1253	0.1786
glucocorticoid receptor agonist	0.1302	0.1813
mTOR inhibitor/PI3K inhibitor	0.1317	0.0772
voltage-gated sodium channel blocker	0.1508	0
inosine monophosphate dehydrogenase inhibitor	0.1535	0
AKT inhibitor	0.1546	0.128
mTOR inhibitor	0.1953	0.29
HDAC inhibitor	0.2015	0.238
PLK inhibitor	0.2221	0.212
PARP inhibitor	0.252	0.3376
Aurora kinase inhibitor	0.2659	0.342
CDK inhibitor	0.3264	0.3326
dihydrofolate reductase inhibitor	0.3312	0.3333
protein synthesis inhibitor	0.3317	0.3492
tubulin polymerization inhibitor	0.5497	0.512
MEK inhibitor	0.6669	0.6212
proteasome inhibitor	0.6842	0.6451
HSP inhibitor	0.8049	0.7313
bacterial 50S ribosomal subunit inhibitor	0.8962	0.86
MDM inhibitor	0.9344	0.86
purine antagonist	0.9699	0.9333
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.9989	1

compound	AP	precision at R
HMGCR inhibitor	0.0234	0.0381
AKT inhibitor	0.0234	0.026
EGFR inhibitor	0.03	0.0588
dihydroorotate dehydrogenase inhibitor	0.0357	0.0369
topoisomerase inhibitor	0.0378	0.0551
RAF inhibitor	0.0389	0.0536
glucocorticoid receptor agonist	0.0445	0.0551
estrogen receptor antagonist/selective estrogen receptor modulator (SERM)	0.0624	0.0933
retinoid receptor agonist	0.0627	0.1038
ribonucleotide reductase inhibitor	0.0628	0.0647
JAK inhibitor	0.0653	0.0792
dihydrofolate reductase inhibitor	0.0845	0.0933
mTOR inhibitor/PI3K inhibitor	0.0987	0.0963
PARP inhibitor	0.1022	0.1293
inosine monophosphate dehydrogenase inhibitor	0.1213	0.0571
mTOR inhibitor	0.1301	0.1767
MDM inhibitor	0.1475	0.06
bacterial 50S ribosomal subunit inhibitor	0.1486	0.18
voltage-gated sodium channel blocker	0.1605	0.1667
HDAC inhibitor	0.1727	0.2192
protein synthesis inhibitor	0.1886	0.1825
HSP inhibitor	0.2751	0.2518
Aurora kinase inhibitor	0.2848	0.297
CDK inhibitor	0.3093	0.3482
MEK inhibitor	0.3672	0.3741
tubulin polymerization inhibitor	0.4277	0.428
PLK inhibitor	0.4559	0.466
proteasome inhibitor	0.6011	0.5735
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.675	0.58
purine antagonist	0.7069	0.6

Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=5.879117847435402, pvalue=4.351771882547785e-09)

plate	mAP model	mAP BM	mAP shuffled
all plates	0.0617	0.0401	0

[EchteRobert]

Experiment 9: Increasing the level of data augmentation

I reran Experiment 6, but now using 8 sets of cells sampled per compound type (instead of 4). To keep the true batch size the same, I reduced the batch size from 72 to 36. The model crashed near the end so the results are inconclusive, but may still give an idea of which direction this type of training is headed.

Main takeaways

Although training loss and replicate prediction performance are worse than Experiment 6 and even though training was not completed, the mAP for MoA prediction is higher than before (0.0658 instead of 0.0617). This indicates that adding more data augmentation will benefit generalization.

Results mAP MoA prediction

compound	AP	precision at R
tachykinin antagonist	0.0495	0.0943
RAF inhibitor	0.0548	0.11
topoisomerase inhibitor	0.0659	0.1033
retinoid receptor agonist	0.0712	0.1325
BCL inhibitor	0.08	0
HIV protease inhibitor	0.0844	0.1165
JAK inhibitor	0.0859	0.1197
vitamin D receptor agonist	0.0992	0.1015
glucocorticoid receptor agonist	0.1144	0.1665
estrogen receptor antagonist/selective estrogen receptor modulator (SERM)	0.1304	0.2267
AKT inhibitor	0.1321	0.102
mTOR inhibitor/PI3K inhibitor	0.1475	0.1123
voltage-gated sodium channel blocker	0.1588	0.0556
inosine monophosphate dehydrogenase inhibitor	0.1797	0
mTOR inhibitor	0.2365	0.3367
HDAC inhibitor	0.2613	0.2955
Aurora kinase inhibitor	0.2933	0.3032
PLK inhibitor	0.2997	0.31
dihydrofolate reductase inhibitor	0.3324	0.3333
PARP inhibitor	0.342	0.3536
protein synthesis inhibitor	0.3619	0.381
CDK inhibitor	0.3654	0.4187
tubulin polymerization inhibitor	0.5223	0.442
MDM inhibitor	0.6252	0.5
MEK inhibitor	0.7936	0.7151
HSP inhibitor	0.8016	0.7122
proteasome inhibitor	0.8687	0.799
bacterial 50S ribosomal subunit inhibitor	0.948	0.92
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.9766	0.96
purine antagonist	0.999	1

compound	AP	precision at R
HMGCR inhibitor	0.0234	0.0381
AKT inhibitor	0.0234	0.026
EGFR inhibitor	0.03	0.0588
dihydroorotate dehydrogenase inhibitor	0.0357	0.0369
topoisomerase inhibitor	0.0378	0.0551
RAF inhibitor	0.0389	0.0536
glucocorticoid receptor agonist	0.0445	0.0551
estrogen receptor antagonist/selective estrogen receptor modulator (SERM)	0.0624	0.0933
retinoid receptor agonist	0.0627	0.1038
ribonucleotide reductase inhibitor	0.0628	0.0647
JAK inhibitor	0.0653	0.0792
dihydrofolate reductase inhibitor	0.0845	0.0933
mTOR inhibitor/PI3K inhibitor	0.0987	0.0963
PARP inhibitor	0.1022	0.1293
inosine monophosphate dehydrogenase inhibitor	0.1213	0.0571
mTOR inhibitor	0.1301	0.1767
MDM inhibitor	0.1475	0.06
bacterial 50S ribosomal subunit inhibitor	0.1486	0.18
voltage-gated sodium channel blocker	0.1605	0.1667
HDAC inhibitor	0.1727	0.2192
protein synthesis inhibitor	0.1886	0.1825
HSP inhibitor	0.2751	0.2518
Aurora kinase inhibitor	0.2848	0.297
CDK inhibitor	0.3093	0.3482
MEK inhibitor	0.3672	0.3741
tubulin polymerization inhibitor	0.4277	0.428
PLK inhibitor	0.4559	0.466
proteasome inhibitor	0.6011	0.5735
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.675	0.58
purine antagonist	0.7069	0.6

Total mean mAP shuffled: 0.0
Total mean precision at R shuffled: 0.014760876116724603

Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=6.754822157536942, pvalue=1.5725563418210224e-11)

plate	mAP model	mAP BM	mAP shuffled
all plates	0.0658	0.0401	0

Results mAP replicate prediction

Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=78.30556567539497, pvalue=0.0)

plate	Training mAP model	Training mAP BM	Training mAP shuffled
all plates	0.7532	0.2555	0

[EchteRobert]

Experiment 9: continued

I reran the evaluation after completing the training loop (100 epochs) for the model described in Experiment 9. Because I also preprocessed all remaining plates overnight, the evaluation is done on some plates which were not included in training resulting in the evaluation of some compounds which were not seen before. This may deflate the mAP numbers somewhat.

Main takeaways

Although training loss and replicate prediction performance are worse than in Experiment 6, the mAP for MoA prediction is higher. This indicates that adding more data augmentation will benefit generalization.

Next up

Train a model on ALL LINCS data!

MoA prediction results

compound	AP	precision at R
tachykinin antagonist	0.0603	0.1031
PARP inhibitor	0.067	0.1058
glucose dependent insulinotropic receptor agonist	0.0734	0.08
RAF inhibitor	0.0813	0.123
glucocorticoid receptor agonist	0.0833	0.11
HMGCR inhibitor	0.0858	0.1425
estrogen receptor antagonist/selective estrogen receptor modulator (SERM)	0.0889	0.1533
BCL inhibitor	0.0943	0.0222
vitamin D receptor agonist	0.0984	0.1203
inosine monophosphate dehydrogenase inhibitor	0.1068	0
AKT inhibitor	0.1168	0.1
topoisomerase inhibitor	0.1234	0.1709
voltage-gated sodium channel blocker	0.1564	0.06
Aurora kinase inhibitor	0.1633	0.2256
mTOR inhibitor/PI3K inhibitor	0.1637	0.1033
mTOR inhibitor	0.2132	0.2067
PLK inhibitor	0.2181	0.224
HDAC inhibitor	0.2579	0.292
CDK inhibitor	0.2926	0.3062
dihydrofolate reductase inhibitor	0.329	0.3333
protein synthesis inhibitor	0.3963	0.3968
focal adhesion kinase inhibitor	0.557	0.5
tubulin polymerization inhibitor	0.5715	0.528
bacterial 50S ribosomal subunit inhibitor	0.7708	0.68
MEK inhibitor	0.7752	0.6696
proteasome inhibitor	0.8682	0.8133
HSP inhibitor	0.8813	0.8231
MDM inhibitor	0.941	0.88
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.9924	0.96
purine antagonist	0.9992	1

compound	AP	precision at R
EGFR inhibitor	0.0289	0.0623
dihydroorotate dehydrogenase inhibitor	0.0338	0.0467
prostanoid receptor agonist	0.0361	0.02
glucocorticoid receptor agonist	0.0379	0.056
PARP inhibitor	0.0435	0.0642
retinoid receptor agonist	0.0577	0.0949
estrogen receptor antagonist/selective estrogen receptor modulator (SERM)	0.0584	0.0867
JAK inhibitor	0.0693	0.094
voltage-gated sodium channel blocker	0.0701	0.08
ribonucleotide reductase inhibitor	0.0786	0.0967
topoisomerase inhibitor	0.0806	0.1142
dihydrofolate reductase inhibitor	0.0823	0.0933
mTOR inhibitor	0.1165	0.17
MDM inhibitor	0.1215	0.06
mTOR inhibitor/PI3K inhibitor	0.1216	0.12
focal adhesion kinase inhibitor	0.1242	0.0286
HMGCR inhibitor	0.1366	0.1579
bacterial 50S ribosomal subunit inhibitor	0.1371	0.18
HDAC inhibitor	0.1632	0.2094
inosine monophosphate dehydrogenase inhibitor	0.1757	0.06
protein synthesis inhibitor	0.1882	0.1825
Aurora kinase inhibitor	0.2571	0.2756
CDK inhibitor	0.2818	0.3196
HSP inhibitor	0.3248	0.301
MEK inhibitor	0.3387	0.3488
tubulin polymerization inhibitor	0.4142	0.4347
PLK inhibitor	0.419	0.442
proteasome inhibitor	0.633	0.6233
dehydrogenase inhibitor/inositol monophosphatase inhibitor	0.6606	0.58
purine antagonist	0.7005	0.6

Exp. 9:
Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=6.830722767928342, pvalue=9.18749483493789e-12)

Exp. 6:
Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=6.310555991504098, pvalue=2.9500067552577934e-10)

plate	mAP model	mAP BM	mAP shuffled
all plates Exp. 9	0.0592	0.0367	0
all plates Exp. 6	0.0566	0.0367	0

Replicate prediction results

Exp. 9:
Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=78.43263356928172, pvalue=0.0)

Exp. 6:
Welch's t-test between mlp mAP and bm mAP: Ttest_indResult(statistic=82.86209673792646, pvalue=0.0)

plate	Training mAP model	Training mAP BM	Training mAP shuffled
all plates Exp. 9	0.7381	0.2698	0
all plates Exp. 6	0.7592	0.2698	0