Discussion: Testing

[mintopia]

Speaking to @astrolox about how to go about testing the cache images.

Initial thoughts:

1. Attempt to download file directly, hash and save.
2. Clear cache data directory
3. Attempt to download file through cache. Ensure we have a cache miss, compare hash to original download
4. Attempt to download file again through cache, Ensure we have a cache hit, compare hash to original and previous version.

This only covers a simple case though. It would be good for advanced options for testing range requests, etag headers, etc.

Other people's thoughts and comments?

[GotenXiao]

We can invoke steamcmd to drive an offical Steam depot downloader behind a cache service - pretty trivial to script and orchestrate. Also means we can do some interesting test cases like deliberately corrupting files on the cache and testing client responses.

For the other services, we might be dependent on either trying to perform a replay of captured network traffic, or having to spin up a Windows VM somewhere with a game client installed, and doing some integration testing that way.

[VibroAxe]

Thought: @ilumos steam cache primer script could work nicely for this as it would test a full steam download through the cache.

Time game download
delete game
Retime game download

game download should be x faster?

[VibroAxe]

Worth keeping this up to date, we do now do some basic testing through goss... however we could definitely improve them / make them less reliable on external services

[ilumos]

@VibroAxe Using zeropingheroes/lancache-autofill and timing it would work, but I think it would be a better test to write a dedicated script to test Steam e.g:

1. Start SteamCMD downloading a free game anonymously, e.g. TF2, to a temporary directory
2. Watch the nginx access/error logs for requests from the IP of the test host
   • Output counts of cache hits/misses in the access log
   • Output any lines from the error log
   • If no log lines are found, abort SteamCMD and output "requests are not being routed via the cache, please check your DNS" or similar
3. Clear the temporary directory used to download the game
4. Re-run the SteamCMD game download
5. Repeat the watching of the logs as above
6. If the second time it's 100% cache hits then output a "test passed successfully" message, otherwise output a failure message, along with an excerpt of the access/error logs, perhaps Markdown formatted so it can be pasted into a Github issue directly

For Origin it could be a script that:

1. Takes a known URL to a free Origin game installer (if that's possible without special auth query strings) and perform a number of range requests via cURL
2. Count access log lines for the IP of the test host (but don't tally hit/miss as it is misreported by $cache_status when using slice and requesting something for the first time)
3. Output any error log lines for the IP of the test host
4. If no log lines are found, output "requests are not being routed via the cache, please check your DNS" or similar
5. cURL the URL with range requests again
6. If all log lines show as a cache hit, output a "test passed successfully" message, otherwise output a failure message as above

Is there already a script to test DNS is configured correctly?

@mintopia What do you think would be useful to check re: ETags?

[VibroAxe]

The testing we are talking about here is unit testing of the images, so GitHub "issue formatting" isnt a problem persay as they will be stopped at PR automated testing.

I really like the idea of steamcmd but having thought about it some more the infrastructure to set that up in a unit test is definitely non trivial as it requires DNS overrides and other features to work which obviously are not available to the base image

The benefit of the simple Goss script is that it has no external dependencies whilst still giving a go/no-go on basic cache functionality. We also have to pay for ci time if the test scripts take too long to run

[ilumos]

Will Goss give us enough to go on?

Here's what's available for checking a HTTP URL:

https://github.com/aelsabbahy/goss/blob/master/docs/manual.md#http