HPC: Sporadic model crashes on Helix

[linusseelinger]

Multiply by 2 test jobs (modified to take 10 seconds per evaluation) occasionally log Quit after Listening on port x.... I'm running 100 instances via HQ, queried from the test script below. The issue happens around once every 300 runs. As a result, the test script waits infinitely for the failed job to return something.

#!/bin/bash 
 
echo "Sending requests..." 
 
for i in {1..100} 
do 
   # Expected output: {"output":[[200.0]]} 
   # Check if curl output equals expected output 
   # If not, print error message 
 
   if [ "$(curl -s http://localhost:4242/Evaluate -X POST -d '{"name": "forward", "input": [[100.0]]}')" == '{"output":[[200.0]]}' ]; then 
       echo -n "y" 
   else 
       echo -n "n" 
       #echo "Error: curl output does not equal expected output" 
   fi & 
 
done 
 
echo "Requests sent. Waiting for responses..." 
 
wait

A possible workaround is to set a minimum port of 60000. I therefore suspect the issue is just an occupied port.

Either the port finder does not work as intended on Helix, or maybe there is a race condition due to the short time between port finder and reserving the port by launching the model (seems unlikely).