Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

motdgen, issuegen: write staged changes to a tempfile #44

Merged
merged 2 commits into from
Jul 30, 2020

Conversation

rfairley
Copy link
Contributor

For the same reason as in #39
(copying the phrasing below):

With the staged file shared, there would be potential for two or
more processes executing motdgen to write to it resulting in
corrupted output, or the error in the cat command due to missing
file reported in #35 (comment). Currently, this is not a problem with
motdgen, but could be if motdgen were invoked by something like the
udev rules that invoke issuegen.

Instead, write the intermediate output to a variable before writing
to the final output file. This ensures only valid output is written
to the issue file shown to the terminal.

Additionally, perform code tidyups similar to those done for
issuegen in #40.

@rfairley rfairley force-pushed the rfairley-motdgen-staged branch 2 times, most recently from 88dab77 to c15a140 Compare April 20, 2020 22:40
@rfairley rfairley force-pushed the rfairley-motdgen-staged branch 2 times, most recently from a88708f to 0a5346c Compare May 22, 2020 20:56
@cgwalters
Copy link
Member

It's too bad bash doesn't expose O_TMPFILE, it's exactly what we want here.

But still though aren't there going to be races since we're still truncating and replacing files that the main issuegen process will re-read?

In other words don't we want to do something like "write to a file ending in .tmp, rename() into place", and ensure that issuegen ignores files ending in ".tmp" or so? (Which sadly we still need to do even with O_TMPFILE, see also https://lore.kernel.org/linux-fsdevel/[email protected]/ )

@rfairley
Copy link
Contributor Author

rfairley commented Jun 29, 2020

But still though aren't there going to be races since we're still truncating and replacing files that the main issuegen process will re-read?

I think you're right - it's still possible for the writes to the intermediate snippets going into issuegen/motdgen to overstep each other. I'll take a look at mktemp for this - having different .abcdef.tmp files for each process before renaming would fix this. Then mv to the main issuegen file can be done by processes simultaneously.

Also having doubts with the current change in this PR - redirection echo "${generated_string}" > "${generated_file}" between two or more processes may interleave in execution. Will check if this is a problem. If generalizing the tmpfiles logic above though, it'd be easy to replace this redirection with a similar mktemp and mv.

@rfairley rfairley force-pushed the rfairley-motdgen-staged branch from 0a5346c to 6afc0a4 Compare July 2, 2020 22:44
rfairley pushed a commit to rfairley/console-login-helper-messages that referenced this pull request Jul 3, 2020
Similar to `motdgen` (coreos#44), use `mktemp` to create a uniquely-named
tmpfile file to stage changes and then `mv` the file to the target
location. This ensures no interleaving of writes when two processes
are running `issuegen`. See: coreos#35
Robert Fairley added 2 commits July 2, 2020 20:57
With the `staged` file shared, there would be potential for two or
more processes executing `motdgen` to write to it resulting in
corrupted output, or the error in the `cat` command due to missing
file reported in coreos#35 (comment). Currently, this is not a problem with
motdgen, but could be if `motdgen` were invoked by something like the
udev rules that invoke `issuegen`. A bug reported in `issuegen` for
this reason is: coreos#35

Instead, write the intermediate output to a unique tempfile and mv the
tempfile to the final output location. If the final output is on the
same filesystem as the tempfile, this operation should be atomic. This
ensures only valid output is written to the issue file shown to the
terminal.

Additionally, perform code tidyups similar to those done for
`issuegen` in coreos#40.
Similar to `motdgen` (coreos#44), use `mktemp` to create a uniquely-named
tmpfile file to stage changes and then `mv` the file to the target
location. This ensures no interleaving of writes when two processes
are running `issuegen`. See: coreos#35
@rfairley rfairley force-pushed the rfairley-motdgen-staged branch from e446583 to 8fedbd3 Compare July 3, 2020 00:57
@rfairley
Copy link
Contributor Author

rfairley commented Jul 3, 2020

Updated to use mktemp and mv for file writes, and did the same for issuegen while at it. Interdiff from previous changes: https://github.com/coreos/console-login-helper-messages/compare/6afc0a4b0ee634e9507a90fa3be01c86d2159160..8fedbd386d8e456e0ec2e28d4b55f3600ca0a2d5.

Working on a make target to allow more easily testing this - should be up soon.

@rfairley rfairley changed the title motdgen: do not share a staged file motdgen, issuegen: write staged changes to a tempfile Jul 3, 2020
@rfairley rfairley marked this pull request as draft July 3, 2020 02:06
@rfairley rfairley changed the title motdgen, issuegen: write staged changes to a tempfile [WIP] motdgen, issuegen: write staged changes to a tempfile Jul 3, 2020
rfairley pushed a commit to rfairley/console-login-helper-messages that referenced this pull request Jul 5, 2020
Similar to `motdgen` (coreos#44), use `mktemp` to create a uniquely-named
tmpfile file to stage changes and then `mv` the file to the target
location. This ensures no interleaving of writes when two processes
are running `issuegen`. See: coreos#35
rfairley pushed a commit to rfairley/console-login-helper-messages that referenced this pull request Jul 5, 2020
Similar to `motdgen` (coreos#44), use `mktemp` to create a uniquely-named
tmpfile file to stage changes and then `mv` the file to the target
location. This ensures no interleaving of writes when two processes
are running `issuegen`. See: coreos#35
rfairley pushed a commit to rfairley/console-login-helper-messages that referenced this pull request Jul 6, 2020
Similar to `motdgen` (coreos#44), use `mktemp` to create a uniquely-named
tmpfile file to stage changes and then `mv` the file to the target
location. This ensures no interleaving of writes when two processes
are running `issuegen`. See: coreos#35
@rfairley rfairley marked this pull request as ready for review July 6, 2020 13:24
@rfairley rfairley changed the title [WIP] motdgen, issuegen: write staged changes to a tempfile motdgen, issuegen: write staged changes to a tempfile Jul 6, 2020
@rfairley
Copy link
Contributor Author

rfairley commented Jul 6, 2020

Now ready for review!

The Makefile and sync_to_vm.sh script in #57 can be used to test, if anyone would like to test the changes locally.

rfairley pushed a commit to rfairley/console-login-helper-messages that referenced this pull request Jul 6, 2020
Similar to `motdgen` (coreos#44), use `mktemp` to create a uniquely-named
tmpfile file to stage changes and then `mv` the file to the target
location. This ensures no interleaving of writes when two processes
are running `issuegen`. See: coreos#35
rfairley pushed a commit to rfairley/console-login-helper-messages that referenced this pull request Jul 8, 2020
Similar to `motdgen` (coreos#44), use `mktemp` to create a uniquely-named
tmpfile file to stage changes and then `mv` the file to the target
location. This ensures no interleaving of writes when two processes
are running `issuegen`. See: coreos#35
Copy link
Member

@sohankunkerkar sohankunkerkar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cgwalters
Copy link
Member

(somewhat superficial since I don't know this code) LGTM

Copy link
Member

@dustymabe dustymabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dustymabe
Copy link
Member

I found #64 while looking into this change, but this particular change doesn't help or hurt that issue. I say let's go with it.

@sohankunkerkar sohankunkerkar merged commit 64254b3 into coreos:master Jul 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants