filter: FilesWithExtensions fails to be useable with filter.Keep.

[dmitshur]

An issue that @VojtechVitek made me aware of is that FilesWithExtensions does not work well with filter.Keep.

When we created it (/cc @slimsag @neelance), we had filter.Skip usage in mind, e.g., to skip .html and .go files (but not folders), one could do:

filter.Skip(filter.FilesWithExtensions(".html", ".go"))

And it'd do what you'd expect.

However, that fails when you try to use filter.FilesWithExtensions with filter.Keep, e.g., to keep only ".xyz" files:

filter.Keep(filter.FilesWithExtensions(".xyz"))

That fails because none of the directories are kept, so / is not included.

Need to think about how to best resolve this. Some options are to do nothing at all, to document this, to remove FilesWithExtensions (or turn it into a documentation example), to change FilesWithExtensions or to change Keep/Skip.

[slimsag]

(I haven't used this code in a very long time, so I'm fine with whatever and my response may be slightly off-topic due to a lack of context now)

Given it's named FilesWithExtensions I think it makes since that directories are not kept. I would expect to have to do something like:

filter.Keep(filter.And(filter.AllDirs(), filter.FilesWithExtensions(".xyz"))) // keep all .xyz files in all directories

Or something like that, anyway.

[VojtechVitek]

@dmitshur I hit this issue again today.

I wrote this code (yes, I'm embedding the source code into Go binary!):

var FS = filter.Keep(http.Dir("github.com/some/project"), func(path string, fi os.FileInfo) bool {
	if strings.HasSuffix(path, ".go") && !strings.HasSuffix(path, "_test.go") {
		return true
	}
	if fi.IsDir() {
		// This condition is missing in filter.FilesWithExtensions() and causes
		// vfsgen.Generate() to fail, since it can't walk a FS with no directories
		return true
	}
	return false
})

which adds directories to theFS, so vfsgen can walk all its files.

But, this actually add all the directories from the project, including those that didn't have any files matching the desired extensions (ie. thousands of /.git/objects/* directories!).

---

@slimsag The above proposal filter.And(filter.AllDirs(), filter.FilesWithExtensions(".xyz")) would suffer from the same issue.

---

I think we have two options:

1. Introduce filter.KeepWithDirs() that would return directories for all the "kept" files.
2. Introduce filter.FilesWithExtensionsAndDirs(".xyz") (I know, this func name is silly)