Test cases for examples

[wang-boyu]

Hello there.

Two questions regarding the test cases for the examples:

1. pytest passed on the whole tests directory but failed at the single tests/test_examples.py file, i.e., the following works:

    pytest --cov=mesa tests/

   whereas the following results in a huge chunk of error:

   pytest tests/test_examples.py

   Error Messages

   $ pytest tests/test_examples.py
   ===================================== test session starts =====================================
   platform darwin -- Python 3.9.13, pytest-7.1.2, pluggy-1.0.0
   rootdir: /Users/boyu/GitHubProjects/mesa
   plugins: cov-3.0.0
   collected 1 item
   
   tests/test_examples.py F                                                                [100%]
   
   ========================================== FAILURES ===========================================
   _________________________________ TestExamples.test_examples __________________________________
   
   self = <tests.test_examples.TestExamples testMethod=test_examples>
   
       def test_examples(self):
           for example in os.listdir(self.EXAMPLES):
               if not os.path.isdir(os.path.join(self.EXAMPLES, example)):
                   continue
               if hasattr(self, f"test_{example.replace('-', '_')}"):
                   # non-standard example; tested below
                   continue
   
               print(f"testing example {example!r}")
               with self.active_example_dir(example):
                   try:
                       # model.py at the top level
   >                   mod = importlib.import_module("model")
   
   tests/test_examples.py:52:
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   
   name = 'model', package = None
   
       def import_module(name, package=None):
           """Import a module.
   
           The 'package' argument is required when performing a relative import. It
           specifies the package to use as the anchor point from which to resolve the
           relative import to an absolute import.
   
           """
           level = 0
           if name.startswith('.'):
               if not package:
                   msg = ("the 'package' argument is required to perform a relative "
                          "import for {!r}")
                   raise TypeError(msg.format(name))
               for character in name:
                   if character != '.':
                       break
                   level += 1
   >       return _bootstrap._gcd_import(name[level:], package, level)
   
   /usr/local/Cellar/[email protected]/3.9.13_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py:127:
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   
   name = 'model', package = None, level = 0
   
   >   ???
   
   <frozen importlib._bootstrap>:1030:
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   
   name = 'model', import_ = <function _gcd_import at 0x100456310>
   
   >   ???
   
   <frozen importlib._bootstrap>:1007:
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   
   name = 'model', import_ = <function _gcd_import at 0x100456310>
   
   >   ???
   E   ModuleNotFoundError: No module named 'model'
   
   <frozen importlib._bootstrap>:984: ModuleNotFoundError
   
   During handling of the above exception, another exception occurred:
   
   self = <tests.test_examples.TestExamples testMethod=test_examples>
   
       def test_examples(self):
           for example in os.listdir(self.EXAMPLES):
               if not os.path.isdir(os.path.join(self.EXAMPLES, example)):
                   continue
               if hasattr(self, f"test_{example.replace('-', '_')}"):
                   # non-standard example; tested below
                   continue
   
               print(f"testing example {example!r}")
               with self.active_example_dir(example):
                   try:
                       # model.py at the top level
                       mod = importlib.import_module("model")
                       server = importlib.import_module("server")
                       server.server.render_model()
                   except ImportError:
                       # <example>/model.py
                       mod = importlib.import_module(f"{example.replace('-', '_')}.model")
                       server = importlib.import_module(
                           f"{example.replace('-', '_')}.server"
                       )
   >                   server.server.render_model()
   
   tests/test_examples.py:61:
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   mesa/visualization/ModularVisualization.py:352: in render_model
       element_state = element.render(self.model)
   mesa/visualization/modules/BarChartVisualization.py:77: in render
       df = data_collector.get_agent_vars_dataframe().astype("float")
   mesa/datacollection.py:246: in get_agent_vars_dataframe
       df = df.set_index(["Step", "AgentID"])
   venv/lib/python3.9/site-packages/pandas/util/_decorators.py:311: in wrapper
       return func(*args, **kwargs)
   venv/lib/python3.9/site-packages/pandas/core/frame.py:5563: in set_index
       del frame[c]
   venv/lib/python3.9/site-packages/pandas/core/generic.py:4049: in __delitem__
       self._mgr = self._mgr.idelete(loc)
   venv/lib/python3.9/site-packages/pandas/core/internals/managers.py:1307: in idelete
       nbs = self._slice_take_blocks_ax0(taker, only_slice=True)
   venv/lib/python3.9/site-packages/pandas/core/internals/managers.py:827: in _slice_take_blocks_ax0
       max_len = max(len(mgr_locs), taker.max() + 1)
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
   
   a = array([0]), axis = None, out = None, keepdims = False, initial = <no value>, where = True
   
       def _amax(a, axis=None, out=None, keepdims=False,
                 initial=_NoValue, where=True):
   >       return umr_maximum(a, axis, None, out, keepdims, initial, where)
   E       TypeError: int() argument must be a string, a bytes-like object or a number, not '_NoValueType'
   
   venv/lib/python3.9/site-packages/numpy/core/_methods.py:40: TypeError
   ------------------------------------ Captured stdout call -------------------------------------
   testing example 'boid_flockers'
   testing example 'sugarscape_cg'
   [1, 100]
   [2, 100]
   [3, 94]
   [4, 92]
   [5, 90]
   [6, 82]
   [7, 79]
   [8, 76]
   [9, 75]
   [10, 71]
   testing example 'shape_example'
   Creating agent 0 at (8, 8)
   Creating agent 1 at (10, 3)
   Uid: 0, Heading: (-1, 0)
   Uid: 1, Heading: (-1, 0)
   Creating agent 0 at (17, 7)
   Creating agent 1 at (14, 5)
   testing example 'bank_reserves'
   testing example 'color_patches'
   testing example 'virus_on_network'
   testing example 'charts'
   ====================================== warnings summary =======================================
   tests/test_examples.py::TestExamples::test_examples
   tests/test_examples.py::TestExamples::test_examples
   tests/test_examples.py::TestExamples::test_examples
   tests/test_examples.py::TestExamples::test_examples
   tests/test_examples.py::TestExamples::test_examples
   tests/test_examples.py::TestExamples::test_examples
     /Users/boyu/GitHubProjects/mesa/venv/lib/python3.9/site-packages/pandas/__init__.py:11: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
       __import__(dependency)
   
   tests/test_examples.py::TestExamples::test_examples
     /Users/boyu/GitHubProjects/mesa/venv/lib/python3.9/site-packages/_pytest/_io/saferepr.py:71: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
       s = repr(x)
   
   -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
   =================================== short test summary info ===================================
   FAILED tests/test_examples.py::TestExamples::test_examples - TypeError: int() argument must ...
   ================================ 1 failed, 7 warnings in 2.70s ================================

   Btw, mesa was installed via python3 -m pip install -e ".[dev]". Not sure whether this is an issue with my own environment.

2. How to include an example in the test cases, if it has its own extra dependency?

[rht]

Adding all the examples dependencies into .[dev] is problematic. There will eventually be lots of packages having to be installed and updated. E.g. #1057 requires pyibl.

[wang-boyu]

Yep I agree. But in tests/test_examples.py aren't we looping through the examples/ folder and test run every model? How does a model run if its dependencies aren't installed properly?

[rht]

I think all of them so far have been covered by the current .[dev].

[wang-boyu]

As of now yes.

I'm asking this question because I'm trying to put this example into Mesa-Geo, but unfortunately it has some extra dependencies.

I think I'll just leave it as separated in its own repo for now. Those extra dependencies could be part of Mesa-Geo in the future when I implement more functions, and I'll include this example by then.

Thanks for the reply : )

[rht]

I can't reproduce your problem.

===================================================== test session starts =====================================================
platform linux -- Python 3.10.5, pytest-7.1.1, pluggy-1.0.0
plugins: anyio-3.6.1, cov-3.0.0
collected 1 item                                                                                                              

tests/test_examples.py .                                                                                                [100%]

====================================================== warnings summary =======================================================
tests/test_examples.py: 14 warnings
  /run/current-system/sw/lib/python3.10/site-packages/pandas/__init__.py:11: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
    __import__(dependency)

tests/test_examples.py::TestExamples::test_examples
  /run/current-system/sw/lib/python3.10/importlib/util.py:247: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
    self.__spec__.loader.exec_module(self)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================== 1 passed, 15 warnings in 69.68s (0:01:09) ==========================================

[wang-boyu]

Let me try it with a Linux environment.