You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now steps (at least 03.split) that write their output to $TMPDIR. As stated by @jelmervdl, this is probably because LUMI has not $SCRATCH set and files are written to a temporary dir in the node that is owned by the user and not the group. We should have something like this for LUMI that creates the temporary directory and changes ownerships.
Example of giashard doing it well but 03.split not:
$ ll output_wide15_filtered_sample3-shards/es/*/*
output_wide15_filtered_sample3-shards/es/0/1:
total 376M
-rw-r----- 1 zaragoza2 project_462000252 188M May 9 18:04 plain_text.gz
-rw-rw---- 1 zaragoza2 zaragoza2 187M May 10 16:40 sentences.gz
-rw-r----- 1 zaragoza2 project_462000252 12K May 9 18:04 source.gz
-rw-r----- 1 zaragoza2 project_462000252 655K May 9 18:04 url.gz
output_wide15_filtered_sample3-shards/es/1/1:
total 322M
-rw-r----- 1 zaragoza2 project_462000252 162M May 9 18:04 plain_text.gz
-rw-rw---- 1 zaragoza2 zaragoza2 160M May 10 16:39 sentences.gz
-rw-r----- 1 zaragoza2 project_462000252 11K May 9 18:04 source.gz
-rw-r----- 1 zaragoza2 project_462000252 661K May 9 18:04 url.gz
The text was updated successfully, but these errors were encountered:
Right now steps (at least 03.split) that write their output to
$TMPDIR
. As stated by @jelmervdl, this is probably because LUMI has not$SCRATCH
set and files are written to a temporary dir in the node that is owned by the user and not the group. We should have something like this for LUMI that creates the temporary directory and changes ownerships.Example of giashard doing it well but 03.split not:
The text was updated successfully, but these errors were encountered: