Skip to content

Latest commit

 

History

History
909 lines (710 loc) · 21 KB

lesson6.md

File metadata and controls

909 lines (710 loc) · 21 KB
Raw

Unit testing

Performing maintenance on an existing library can be quite a task, the initial intention might not always be obvious. Adding a few unit tests can make a big difference years down the line.

Objectives

  • Write unit tests in Jsonnet
  • Do test-driven development
  • Know how to avoid pitfalls

Lesson

Let's unit test the webserver library from the first exercise.

local k = import 'k.libsonnet';

{
  new(name, replicas=1): {
    container::
      k.core.v1.container.new('httpd', 'httpd:2.4'),

    deployment:
      k.apps.v1.deployment.new(
        name,
        replicas,
        [self.container]
      ),
  },

  withImage(image): {
    container+:
      k.core.v1.container.withImage(image),
  },
}

// example1/lib/webserver/main.libsonnet

This library provides a number of functions to create a webserver. Each function eventually renders a bit of JSON. The withImages() function is supposed to be mixed in with the new(). While doing maintenance on this library or adding new features, a number of things could go wrong. A few unit tests can catch unintended changes early.

local webserver = import 'webserver/main.libsonnet';
webserver.new('webserver1')

// example1/example0.jsonnet

Let's generate a base from our library to build our tests on:

jsonnet -J lib -J vendor -o base.json example0.jsonnet

{
   "deployment": {
      "apiVersion": "apps/v1",
      "kind": "Deployment",
      "metadata": {
         "name": "webserver1"
      },
      "spec": {
         "replicas": 1,
         "selector": {
            "matchLabels": {
               "name": "webserver1"
            }
         },
         "template": {
            "metadata": {
               "labels": {
                  "name": "webserver1"
               }
            },
            "spec": {
               "containers": [
                  {
                     "image": "httpd:2.4",
                     "name": "httpd"
                  }
               ]
            }
         }
      }
   }
}

// example1/base.json

The output of the webserver deployment will look like this. Note that it doesn't include the hidden container field. This rendered representation will be used as the base for the unit tests.

Initializing Testonnet

For the unit tests, the Testonnet library provides a few primitives to get us started.

$ jb install github.com/jsonnet-libs/testonnet

Check out the docs for Testonnet.

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/main.libsonnet';

test.new(std.thisFile)

// example1/example1.jsonnet

A test suite is initialized by calling new(name). The name will be printed during execution to help us find failing test cases.

When a test case fails, Testonnet will use error to ensure a non-zero exit code. This has the side effect that the corresponding stack trace will be from the Testonnet library, rather than the failing test. When using std.thisFile in the name, it will be easier to find the failing test case.

# jsonnet -J lib -J vendor example1.jsonnet
TRACE: vendor/testonnet/main.libsonnet:74 Testing suite example1.jsonnet
{
   "verify": "Passed 0 test cases"
}

// example1/example1.jsonnet.output

Running the test suite can be done with this:

$ jsonnet -J vendor -J lib example1.jsonnet

The output will either show the failing test cases or count the successful test.

Testing the webserver library

new()

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/main.libsonnet';

local base = import 'base.json';

test.new(std.thisFile)
+ test.case.new(
  'Basic',
  test.expect.eq(
    webserver.new('webserver1'),
    base
  )
)

// example1/example2.jsonnet
# jsonnet -J lib -J vendor example2.jsonnet
TRACE: vendor/testonnet/main.libsonnet:74 Testing suite example2.jsonnet
{
   "verify": "Passed 1 test cases"
}

// example1/example2.jsonnet.output

test.case.new(name, test) adds a new test case to the suite. The name can be an arbitrary string, test is an object that can created with test.expect. In this example test.expect.eq compares 2 objects with the expectation that they are equal.

The output from webserver.new() is compared to the rendered representation. Running the test suite returns a successful test.

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/main.libsonnet';

local webserverName = 'webserver1';
local base = import 'base.json';

test.new(std.thisFile)
+ test.case.new(
  'Basic',
  test.expect.eq(
    webserver.new(webserverName),
    base
  )
)
+ test.case.new(
  'Change default replicas',
  test.expect.eq(
    webserver.new(webserverName, 2),
    base { deployment+: { spec+: { replicas: 2 } } }
  )
)

// example1/example3.jsonnet
# jsonnet -J lib -J vendor example3.jsonnet
TRACE: vendor/testonnet/main.libsonnet:74 Testing suite example3.jsonnet
{
   "verify": "Passed 2 test cases"
}

// example1/example3.jsonnet.output

The new() function allows us to modify the replicas on the deployment, this will go into the 'actual' part of the test case.

On the 'expected' part base is added with only the replicas attribute modified.

This test ensures only the replicas are changed, it also reinforces the values tested in the 'Basic' test.

withImages()

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/main.libsonnet';

local webserverName = 'webserver1';
local base = import 'base.json';

local mapContainerWithName(name, obj) =
  {
    local containers = super.spec.template.spec.containers,
    spec+: { template+: { spec+: { containers: [
      if c.name == name
      then c + obj
      else c
      for c in containers
    ] } } },
  };

test.new(std.thisFile)
+ test.case.new(
  'Basic',
  test.expect.eq(
    webserver.new(webserverName),
    base
  )
)
+ test.case.new(
  'Change default replicas',
  test.expect.eq(
    webserver.new(webserverName, 2),
    base { deployment+: { spec+: { replicas: 2 } } }
  )
)
+ test.case.new(
  'Set alternative image',
  test.expect.eq(
    webserver.new(webserverName)
    + webserver.withImage('httpd:2.5'),
    base { deployment+: mapContainerWithName('httpd', { image: 'httpd:2.5' }) }
  )
)

// example1/example4.jsonnet
# jsonnet -J lib -J vendor example4.jsonnet
TRACE: vendor/testonnet/main.libsonnet:74 Testing suite example4.jsonnet
{
   "verify": "Passed 3 test cases"
}

// example1/example4.jsonnet.output

Testing withImages() is a bit more complex. In the library this function modifies the hidden container:: field, which eventually gets added to the deployment in new() through late-initialization.

Again new() is called to set the 'actual' part, this time withImages() is concatenated to get a deployment with an alternative image.

On the 'expected' side the container with name httpd in the deployment needs to be modified with the new image name, using the mapContainerWithName helper function to keep the test cases readable.

Note that mapContainerWithName also preserves any other containers that may exist in the deployment, future-proofing the unit tests.

Test-driven development

Let's write a test for a new function webserver.withImagePullPolicy(policy), which can then be added as a feature to the library.

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/wrong1.libsonnet';

local webserverName = 'webserver1';
local base = import 'base.json';

local mapContainerWithName(name, obj) =
  {
    local containers = super.spec.template.spec.containers,
    spec+: { template+: { spec+: { containers: [
      if c.name == name
      then c + obj
      else c
      for c in containers
    ] } } },
  };


test.new(std.thisFile)
+ test.case.new(
  'Basic',
  test.expect.eq(
    webserver.new(webserverName),
    base
  )
)
+ test.case.new(
  'Change default replicas',
  test.expect.eq(
    webserver.new(webserverName, 2),
    base { deployment+: { spec+: { replicas: 2 } } }
  )
)
+ test.case.new(
  'Set alternative image',
  test.expect.eq(
    webserver.new(webserverName)
    + webserver.withImage('httpd:2.5'),
    base { deployment+: mapContainerWithName('httpd', { image: 'httpd:2.5' }) }
  )
)
+ test.case.new(
  'Set imagePullPolicy',
  test.expect.eq(
    webserver.new(webserverName)
    + webserver.withImagePullPolicy('Always'),
    base { deployment+: mapContainerWithName('httpd', { imagePullPolicy: 'Always' }) }
  )
)

// example1/example5.jsonnet

The new test 'Set imagePullPolicy' is very similar to 'Set alternative image'.

To use the same base, new() is concatenated with withImagePullPolicy('Always') on 'actual'.

On 'expected' it uses the mapWithContainerName helper to set imagePullPolicy on the httpd container.

local k = import 'k.libsonnet';
local main = import 'main.libsonnet';

main {
  withImagePullPolicy(policy): {
    container:
      k.core.v1.container.withImagePullPolicy(policy),
  },
}

// example1/lib/webserver/wrong1.libsonnet

Extending the library (referenced as main) with the withImagePullPolicy() function is quite straightforward.

# jsonnet -J lib -J vendor example5.jsonnet
RUNTIME ERROR: Failed 1/4 test cases:
Set imagePullPolicy: Expected {"deployment": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"name": "webserver1"}, "spec": {"replicas": 1, "selector": {"matchLabels": {"name": "webserver1"}}, "template": {"metadata": {"labels": {"name": "webserver1"}}, "spec": {"containers": [{"imagePullPolicy": "Always"}]}}}}} to be {"deployment": {"apiVersion": "apps/v1", "kind": "Deployment", "metadata": {"name": "webserver1"}, "spec": {"replicas": 1, "selector": {"matchLabels": {"name": "webserver1"}}, "template": {"metadata": {"labels": {"name": "webserver1"}}, "spec": {"containers": [{"image": "httpd:2.4", "imagePullPolicy": "Always", "name": "httpd"}]}}}}}
	vendor/testonnet/main.libsonnet:(78:11)-(84:13)	thunk from <object <anonymous>>
	vendor/testonnet/main.libsonnet:(74:7)-(87:8)	object <anonymous>
	Field "verify"	
	During manifestation	


// example1/example5.jsonnet.output

Oh no, running the test shows a failure, how did that happen? The difference between expected and actual result can be found in the output...

Turns out that the test.expect.eq function output is quite inconvenient, let's improve that.

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/wrong1.libsonnet';

local webserverName = 'webserver1';
local base = import 'base.json';

local mapContainerWithName(name, obj) =
  {
    local containers = super.spec.template.spec.containers,
    spec+: { template+: { spec+: { containers: [
      if c.name == name
      then c + obj
      else c
      for c in containers
    ] } } },
  };

local eqJson = test.expect.new(
  function(actual, expected) actual == expected,
  function(actual, expected)
    'Actual:\n'
    + std.manifestJson(actual)
    + '\nExpected:\n'
    + std.manifestJson(expected),
);

test.new(std.thisFile)
+ test.case.new(
  'Basic',
  eqJson(
    webserver.new(webserverName),
    base
  )
)
+ test.case.new(
  'Change default replicas',
  eqJson(
    webserver.new(webserverName, 2),
    base { deployment+: { spec+: { replicas: 2 } } }
  )
)
+ test.case.new(
  'Set alternative image',
  eqJson(
    webserver.new(webserverName)
    + webserver.withImage('httpd:2.5'),
    base { deployment+: mapContainerWithName('httpd', { image: 'httpd:2.5' }) }
  )
)
+ test.case.new(
  'Set imagePullPolicy',
  eqJson(
    webserver.new(webserverName)
    + webserver.withImagePullPolicy('Always'),
    base { deployment+: mapContainerWithName('httpd', { imagePullPolicy: 'Always' }) }
  )
)

// example1/example6.jsonnet

To replace test.expect.eq, a new 'test' function needs to be created. This can be done with test.expect.new(satisfy, message).

The satisfy function should return a boolean with actual and expected as arguments.

The message function returns a string and also accepts the actual and expected results as arguments, these can be used to display the results in the error message.

# jsonnet -J lib -J vendor example6.jsonnet
RUNTIME ERROR: Failed 1/4 test cases:
Set imagePullPolicy: Actual:
{
    "deployment": {
        "apiVersion": "apps/v1",
        "kind": "Deployment",
        "metadata": {
            "name": "webserver1"
        },
        "spec": {
            "replicas": 1,
            "selector": {
                "matchLabels": {
                    "name": "webserver1"
                }
            },
            "template": {
                "metadata": {
                    "labels": {
                        "name": "webserver1"
                    }
                },
                "spec": {
                    "containers": [
                        {
                            "imagePullPolicy": "Always"
                        }
                    ]
                }
            }
        }
    }
}
Expected:
{
    "deployment": {
        "apiVersion": "apps/v1",
        "kind": "Deployment",
        "metadata": {
            "name": "webserver1"
        },
        "spec": {
            "replicas": 1,
            "selector": {
                "matchLabels": {
                    "name": "webserver1"
                }
            },
            "template": {
                "metadata": {
                    "labels": {
                        "name": "webserver1"
                    }
                },
                "spec": {
                    "containers": [
                        {
                            "image": "httpd:2.4",
                            "imagePullPolicy": "Always",
                            "name": "httpd"
                        }
                    ]
                }
            }
        }
    }
}
	vendor/testonnet/main.libsonnet:(78:11)-(84:13)	thunk from <object <anonymous>>
	vendor/testonnet/main.libsonnet:(74:7)-(87:8)	object <anonymous>
	Field "verify"	
	During manifestation	


// example1/example6.jsonnet.output

The output is now a bit more convenient. It turns out that the container is being replaced completely instead of having imagePullPolicy set.

local k = import 'k.libsonnet';
local main = import 'main.libsonnet';

main {
  withImagePullPolicy(policy): {
    container:
      k.core.v1.container.withImagePullPolicy(policy),
  },
}

// example1/lib/webserver/wrong1.libsonnet

Can you spot the mistake?

local k = import 'k.libsonnet';
local main = import 'main.libsonnet';

main {
  withImagePullPolicy(policy): {
    container+:
      k.core.v1.container.withImagePullPolicy(policy),
  },
}

// example1/lib/webserver/correct.libsonnet

Turns out a + was forgotten on container+:.

# jsonnet -J lib -J vendor example7.jsonnet
TRACE: vendor/testonnet/main.libsonnet:74 Testing suite example7.jsonnet
{
   "verify": "Passed 4 test cases"
}

// example1/example7.jsonnet.output

With that fixed, the test suite succeeds.

Pulling it together

example2/lib/webserver/
├── main.libsonnet
├── Makefile
└── test
    ├── base.json
    ├── jsonnetfile.json
    ├── lib/k.libsonnet
    └── main.libsonnet

With the test cases written, let's pull it all together in a test/ subdirectory so that the test dependencies from jsonnetfile.json are not required to install the library.

.PHONY: test
test:
	@cd test/ && \
		jb install && \
		jsonnet -J vendor -J lib main.libsonnet

// example2/lib/webserver/Makefile
# make test
TRACE: vendor/testonnet/main.libsonnet:74 Testing suite main.libsonnet
{
   "verify": "Passed 4 test cases"
}

// example2/lib/webserver/make_test.output

With a test target in a Makefile, running the test cases becomes trivial.

Pitfalls

Just like with any test framework, a unit test can be written in such a way that they succeed while not actually validating the unit.

Testing individual attributes

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/wrong2.libsonnet';

local simple = webserver.new('webserver1');
local imagePull =
  webserver.new('webserver1')
  + webserver.withImagePullPolicy('Always');

test.new(std.thisFile)
+ test.case.new(
  'Validate name',
  test.expect.eq(
    simple.deployment.metadata.name,
    'webserver1',
  )
)
+ test.case.new(
  'Validate image name',
  test.expect.eq(
    simple.deployment.spec.template.spec.containers[0].name,
    'httpd',
  )
)
+ test.case.new(
  'Validate imagePullPolicy',
  test.expect.eq(
    imagePull.deployment.spec.template.spec.containers[0].imagePullPolicy,
    'Always',
  )
)

// example1/pitfall1.jsonnet
# jsonnet -J lib -J vendor pitfall1.jsonnet
TRACE: vendor/testonnet/main.libsonnet:74 Testing suite pitfall1.jsonnet
{
   "verify": "Passed 3 test cases"
}

// example1/pitfall1.jsonnet.output

While the unit tests here are valid on their own, they only validate individual attributes. They won't catch any changes withImagePullPolicy() might make to other attributes.

local k = import 'k.libsonnet';
local main = import 'main.libsonnet';

main {
  withImagePullPolicy(policy): {
    container+:
      k.core.v1.container.withName(super.container.name + policy)
      + k.core.v1.container.withImagePullPolicy(policy),
  },
}

// example1/lib/webserver/wrong2.libsonnet

For example, here withImagePullPolicy() function also changes name on the container while this was explicitly tested on the 'simple' use case.

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/wrong2.libsonnet';

local simple = webserver.new('webserver1');
local imagePull =
  webserver.new('webserver1')
  + webserver.withImagePullPolicy('Always');

test.new(std.thisFile)
+ test.case.new(
  'Validate name',
  test.expect.eq(
    simple.deployment.metadata.name,
    'webserver1',
  )
)
+ test.case.new(
  'Validate image name',
  test.expect.eq(
    simple.deployment.spec.template.spec.containers[0].name,
    'httpd',
  )
)
+ test.case.new(
  'Validate imagePullPolicy',
  test.expect.eq(
    imagePull.deployment.spec.template.spec.containers[0].imagePullPolicy,
    'Always',
  )
)
+ test.case.new(
  'Validate name',
  test.expect.eq(
    imagePull.deployment.metadata.name,
    'webserver1',
  )
)
+ test.case.new(
  'Validate image name',
  test.expect.eq(
    imagePull.deployment.spec.template.spec.containers[0].name,
    'httpd',
  )
)

// example1/pitfall2.jsonnet

To cover for the name (and other tests), the unit tests for 'simple' need to be repeated for the 'imagePull' use case, resulting in an exponential growth of test case as the library gets extended.

# jsonnet -J lib -J vendor pitfall2.jsonnet
RUNTIME ERROR: Failed 1/5 test cases:
Validate image name: Expected httpdAlways to be httpd
	vendor/testonnet/main.libsonnet:(78:11)-(84:13)	thunk from <object <anonymous>>
	vendor/testonnet/main.libsonnet:(74:7)-(87:8)	object <anonymous>
	Field "verify"	
	During manifestation	


// example1/pitfall2.jsonnet.output

Adding the test shows the expected failure.

Testing hidden attributes

local test = import 'testonnet/main.libsonnet';
local webserver = import 'webserver/wrong3.libsonnet';

local webserverName = 'webserver1';
local base = import 'base.json';

test.new(std.thisFile)
+ test.case.new(
  'Basic',
  test.expect.eq(
    webserver.new(webserverName),
    base
  )
)
+ test.case.new(
  'Set alternative image',
  test.expect.eq(
    (webserver.new(webserverName)
     + webserver.withImagePullPolicy('Always')).container,
    {
      name: 'httpd',
      image: 'httpd:2.4',
      imagePullPolicy: 'Always',
    }
  )
)

// example1/pitfall3.jsonnet
# jsonnet -J lib -J vendor pitfall3.jsonnet
TRACE: vendor/testonnet/main.libsonnet:74 Testing suite pitfall3.jsonnet
{
   "verify": "Passed 2 test cases"
}

// example1/pitfall3.jsonnet.output

While a unit test can access and validate the content of a hidden attribute, it is likely not useful. From a testing perspective, the hidden attributes should be considered 'internals' to the function.

As Jsonnet does late-initialization before returning a JSON, validating the output should also be done on all visible attributes it might affect.

local k = import 'k.libsonnet';
local main = import 'main.libsonnet';

main {
  withImagePullPolicy(policy): {
    container+:::
      k.core.v1.container.withImagePullPolicy(policy),
  },
}

// example1/lib/webserver/wrong3.libsonnet

For example, here the withImagePullPolicy() function makes the container visible in the output, changing the intended behavior of new().

Conclusion

Writing unit tests can feel like a burden, but when done right they can be elegant and quite cheap to write.

And remember: "A society grows great when old men plant trees whose shade they know they shall never sit in."