Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix permissions of step files writen to $TMPDIR #27

Open
ZJaume opened this issue May 10, 2023 · 0 comments
Open

Fix permissions of step files writen to $TMPDIR #27

ZJaume opened this issue May 10, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@ZJaume
Copy link
Member

ZJaume commented May 10, 2023

Right now steps (at least 03.split) that write their output to $TMPDIR. As stated by @jelmervdl, this is probably because LUMI has not $SCRATCH set and files are written to a temporary dir in the node that is owned by the user and not the group. We should have something like this for LUMI that creates the temporary directory and changes ownerships.

Example of giashard doing it well but 03.split not:

$ ll output_wide15_filtered_sample3-shards/es/*/*
output_wide15_filtered_sample3-shards/es/0/1:
total 376M
-rw-r----- 1 zaragoza2 project_462000252 188M May  9 18:04 plain_text.gz
-rw-rw---- 1 zaragoza2 zaragoza2         187M May 10 16:40 sentences.gz
-rw-r----- 1 zaragoza2 project_462000252  12K May  9 18:04 source.gz
-rw-r----- 1 zaragoza2 project_462000252 655K May  9 18:04 url.gz

output_wide15_filtered_sample3-shards/es/1/1:
total 322M
-rw-r----- 1 zaragoza2 project_462000252 162M May  9 18:04 plain_text.gz
-rw-rw---- 1 zaragoza2 zaragoza2         160M May 10 16:39 sentences.gz
-rw-r----- 1 zaragoza2 project_462000252  11K May  9 18:04 source.gz
-rw-r----- 1 zaragoza2 project_462000252 661K May  9 18:04 url.gz
@ZJaume ZJaume added the bug Something isn't working label May 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant