Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filestore: add dirs(0775 def) and files(0664 def) chmod(2) perms cmdline options #1155

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions cmd/tusd/cli/composer.go
Original file line number Diff line number Diff line change
Expand Up @@ -144,11 +144,11 @@ func CreateComposer() {
}

stdout.Printf("Using '%s' as directory storage.\n", dir)
if err := os.MkdirAll(dir, os.FileMode(0774)); err != nil {
if err := os.MkdirAll(dir, os.FileMode(Flags.DirPerms)); err != nil {
stderr.Fatalf("Unable to ensure directory exists: %s", err)
}

store := filestore.New(dir)
store := filestore.New(dir, Flags.DirPerms, Flags.FilePerms)
store.UseIn(Composer)

locker := filelocker.New(dir)
Expand Down
29 changes: 29 additions & 0 deletions cmd/tusd/cli/flags.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@ package cli

import (
"flag"
"fmt"
"path/filepath"
"strconv"
"strings"
"time"

Expand Down Expand Up @@ -74,11 +76,36 @@ var Flags struct {
AcquireLockTimeout time.Duration
FilelockHolderPollInterval time.Duration
FilelockAcquirerPollInterval time.Duration
FilePerms uint32
DirPerms uint32
GracefulRequestCompletionTimeout time.Duration
ExperimentalProtocol bool
}

type ChmodPermsValue struct {
perms *uint32
}

func (v ChmodPermsValue) String() string {
if v.perms != nil {
return fmt.Sprintf("%o", *v.perms)
}
return ""
}

func (v ChmodPermsValue) Set(s string) error {
if u, err := strconv.ParseUint(s, 8, 32); err != nil {
return err
} else {
*v.perms = uint32(u)
}
return nil
}

func ParseFlags() {
Flags.DirPerms = 0775
Flags.FilePerms = 0664

fs := grouped_flags.NewFlagGroupSet(flag.ExitOnError)

fs.AddGroup("Listening options", func(f *flag.FlagSet) {
Expand Down Expand Up @@ -116,6 +143,8 @@ func ParseFlags() {
f.StringVar(&Flags.UploadDir, "upload-dir", "./data", "Directory to store uploads in")
f.DurationVar(&Flags.FilelockHolderPollInterval, "filelock-holder-poll-interval", 5*time.Second, "The holder of a lock polls regularly to see if another request handler needs the lock. This flag specifies the poll interval.")
f.DurationVar(&Flags.FilelockAcquirerPollInterval, "filelock-acquirer-poll-interval", 2*time.Second, "The acquirer of a lock polls regularly to see if the lock has been released. This flag specifies the poll interval.")
f.Var(&ChmodPermsValue{&Flags.DirPerms}, "dir-perms", "The created directory chmod(2) OCTAL value permissions.")
f.Var(&ChmodPermsValue{&Flags.FilePerms}, "file-perms", "The created file chmod(2) OCTAL value permissions.")
})

fs.AddGroup("AWS S3 storage options", func(f *flag.FlagSet) {
Expand Down
4 changes: 4 additions & 0 deletions docker/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,8 @@ set -o pipefail

. /usr/local/share/load-env.sh

if [ -n "$UMASK" ]; then
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change seems unrelated to the PR itself. Can you please remove it if that's the case?

Copy link
Author

@avreg avreg Jul 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, it can be removed, but ...

On many UNIX systems, the default umask is 022.
This significantly limits the usefulness of the new options in the context of "group" and "other" permissions
If we don't use the UMASK environment variable, we have to call umask(2) directly in the go application (hard-coded zero or add new option like -fs-umask=umask).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how umask works. Can you provide some hints or links so I can understand what this is about?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is an example using Go code
https://stackoverflow.com/a/61645606/23561452

I needed this patch to get group write access to the intermediate directories of the downloaded file.

// pre-create hook:
return {
            ChangeFileInfo: {
                ID: "subdir1/subdir2/.../basename",
            },
        };

Copy link
Member

@Acconut Acconut Jul 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I meant that I don't know what umask is in general. Maybe you can provide a bit more detailed explanation showing the motivation/necessity behind your PR.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, now you exactly understand the mechanics of how umask(2) works.

diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
index 88a70ef..f1934bb 100644
--- a/docker/entrypoint.sh
+++ b/docker/entrypoint.sh
@@ -6,4 +6,18 @@ set -o pipefail
 
 . /usr/local/share/load-env.sh
 
+if printenv UMASK >/dev/null; then
+   umask "$UMASK"
+fi

With this and the other above patches and without explicitly setting env UMASK for the docker container, the tusd daemon will create new directories/files with 0755/0644 permissions (since usually linux distributions set umask 022 for users by default).

If a user needs something special, like "wx" access for a group, they will have to pass env UMASK=002 to the container. And then the permissions of newly created directories/files will become 0775/0664.

This changes is lightweight and should satisfy most users. Unless the user wants different permissions for files and directories, e.g. 0755 (g=rx) for directories but 0660 (g=rwx) for files ;)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patience. I'm slowly getting it. Do you know how other Docker images handle this? Do they expose a UMASK environment variable as well? Or are there other patterns to control file permissions from containers?

Copy link
Author

@avreg avreg Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docker ecosystem has "Isolate containers with a user namespace", which would be interesting to use with a TUSD container, but it won't solve the UMASK issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't really ask my question, so let me try to rephrase it: Do other Docker images also use the UMASK environment variable for configuring the umask of the process running in the Docker container? Does the Docker community have other defacto ways of configuring the umask?

Copy link
Author

@avreg avreg Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not used any other projects where I need to change the UMASK. So my answer to your question is that I don't know and I don't know any de facto solutions.

However, if you search on google, you can find many posts where people are looking for similar solutions in other projects.

Generally, regarding umask there are not so many solutions:

  1. pam_umask(8) (systemwide only),
  2. RC, like bashrc (systemwide and user),
  3. umask (CLI).

umask "$UMASK"
fi

exec tusd "$@"
2 changes: 1 addition & 1 deletion docs/_advanced-topics/usage-package.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ func main() {
// If you want to save them on a different medium, for example
// a remote FTP server, you can implement your own storage backend
// by implementing the tusd.DataStore interface.
store := filestore.New("./uploads")
store := filestore.New("./uploads", 0774, 0664)

// A locking mechanism helps preventing data loss or corruption from
// parallel requests to a upload resource. A good match for the disk-based
Expand Down
2 changes: 1 addition & 1 deletion examples/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ func main() {
// If you want to save them on a different medium, for example
// a remote FTP server, you can implement your own storage backend
// by implementing the tusd.DataStore interface.
store := filestore.New("./uploads")
store := filestore.New("./uploads", 0774, 0664)

// A locking mechanism helps preventing data loss or corruption from
// parallel requests to a upload resource. A good match for the disk-based
Expand Down
34 changes: 21 additions & 13 deletions pkg/filestore/filestore.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,29 +18,30 @@ import (
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"

"github.com/tus/tusd/v2/internal/uid"
"github.com/tus/tusd/v2/pkg/handler"
)

var defaultFilePerm = os.FileMode(0664)
var defaultDirectoryPerm = os.FileMode(0754)

// See the handler.DataStore interface for documentation about the different
// methods.
type FileStore struct {
// Relative or absolute path to store files in. FileStore does not check
// whether the path exists, use os.MkdirAll in this case on your own.
Path string

DirPerm fs.FileMode
FilePerm fs.FileMode
}

// New creates a new file based storage backend. The directory specified will
// be used as the only storage entry. This method does not check
// whether the path exists, use os.MkdirAll to ensure.
func New(path string) FileStore {
return FileStore{path}
func New(path string, dirPerms uint32, filePerms uint32) FileStore {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't change the signature of exported functions as this would be a breaking change. Instead, we can keep New unchanged and users can set the DirPerm and FilePerm fields after they called New.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I corrected the code, please take a look.

return FileStore{path, os.FileMode(dirPerms), os.FileMode(filePerms)}
}

// UseIn sets this store as the core data store in the passed composer and adds
Expand Down Expand Up @@ -73,14 +74,16 @@ func (store FileStore) NewUpload(ctx context.Context, info handler.FileInfo) (ha
}

// Create binary file with no content
if err := createFile(binPath, nil); err != nil {
if err := createFile(binPath, store.DirPerm, store.FilePerm, nil); err != nil {
return nil, err
}

upload := &fileUpload{
info: info,
infoPath: infoPath,
binPath: binPath,
dirPerm: store.DirPerm,
filePerm: store.FilePerm,
}

// writeInfo creates the file by itself if necessary
Expand Down Expand Up @@ -133,6 +136,8 @@ func (store FileStore) GetUpload(ctx context.Context, id string) (handler.Upload
info: info,
binPath: binPath,
infoPath: infoPath,
dirPerm: store.DirPerm,
filePerm: store.FilePerm,
}, nil
}

Expand Down Expand Up @@ -166,14 +171,17 @@ type fileUpload struct {
infoPath string
// binPath is the path to the binary file (which has no extension)
binPath string

dirPerm fs.FileMode
filePerm fs.FileMode
}

func (upload *fileUpload) GetInfo(ctx context.Context) (handler.FileInfo, error) {
return upload.info, nil
}

func (upload *fileUpload) WriteChunk(ctx context.Context, offset int64, src io.Reader) (int64, error) {
file, err := os.OpenFile(upload.binPath, os.O_WRONLY|os.O_APPEND, defaultFilePerm)
file, err := os.OpenFile(upload.binPath, os.O_WRONLY|os.O_APPEND, upload.filePerm)
if err != nil {
return 0, err
}
Expand Down Expand Up @@ -212,7 +220,7 @@ func (upload *fileUpload) Terminate(ctx context.Context) error {
}

func (upload *fileUpload) ConcatUploads(ctx context.Context, uploads []handler.Upload) (err error) {
file, err := os.OpenFile(upload.binPath, os.O_WRONLY|os.O_APPEND, defaultFilePerm)
file, err := os.OpenFile(upload.binPath, os.O_WRONLY|os.O_APPEND, upload.filePerm)
if err != nil {
return err
}
Expand Down Expand Up @@ -253,7 +261,7 @@ func (upload *fileUpload) writeInfo() error {
if err != nil {
return err
}
return createFile(upload.infoPath, data)
return createFile(upload.infoPath, upload.dirPerm, upload.filePerm, data)
}

func (upload *fileUpload) FinishUpload(ctx context.Context) error {
Expand All @@ -262,19 +270,19 @@ func (upload *fileUpload) FinishUpload(ctx context.Context) error {

// createFile creates the file with the content. If the corresponding directory does not exist,
// it is created. If the file already exists, its content is removed.
func createFile(path string, content []byte) error {
file, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, defaultFilePerm)
func createFile(path string, dirPerm fs.FileMode, filePerm fs.FileMode, content []byte) error {
file, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, filePerm)
if err != nil {
if os.IsNotExist(err) {
// An upload ID containing slashes is mapped onto different directories on disk,
// for example, `myproject/uploadA` should be put into a folder called `myproject`.
// If we get an error indicating that a directory is missing, we try to create it.
if err := os.MkdirAll(filepath.Dir(path), defaultDirectoryPerm); err != nil {
if err := os.MkdirAll(filepath.Dir(path), dirPerm); err != nil {
return fmt.Errorf("failed to create directory for %s: %s", path, err)
}

// Try creating the file again.
file, err = os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, defaultFilePerm)
file, err = os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, filePerm)
if err != nil {
// If that still doesn't work, error out.
return err
Expand Down
2 changes: 1 addition & 1 deletion pkg/handler/composer_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
func ExampleNewStoreComposer() {
composer := handler.NewStoreComposer()

fs := filestore.New("./data")
fs := filestore.New("./data", 0774, 0664)
fs.UseIn(composer)

ml := memorylocker.New()
Expand Down
2 changes: 1 addition & 1 deletion pkg/hooks/hooks_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ func TestNewHandlerWithHooks(t *testing.T) {
ctrl := gomock.NewController(t)
defer ctrl.Finish()

store := filestore.New("some-path")
store := filestore.New("some-path", 0774, 0664)
config := handler.Config{
StoreComposer: handler.NewStoreComposer(),
}
Expand Down