Integration tests are good

I won't condemn ZopeTestCase entirely. It actually serves a good purpose: to test how a particular Zope 2 object (a Zope 2 meta type, a CMF portal tool, a content object, etc.) behaves in a Zope 2-like environment, which essentially means inside a folder hierarchy with acquisition and all the bells and whistles. Because of the way many Zope 2 solutions were written, this is pretty much the only way you would test them anyways, because they rely on having an acquisition context, the REQUEST available all the time, etc.

Still, calling a ZopeTestCase test a unit test is pure blasphemy. It's an integration test.

Unit tests are important, too

The problem with ZopeTestCase is that it's so convenient and it lures test authors into a type of test that they probably didn't want to write and definitely shouldn't. Sure, integration testing is important and, as said above, ZopeTestCase actually serves that purpose.

Unit tests are also important, though. And as Zope 2 software is being componentized, they're actually much easier to write. The CMF, for example, doesn't use ZopeTestCase (except for one test which probably shouldn't use it, either).

ZopeTestCase's problem

The problem with ZopeTestCase is really a technical one. It is the way it provides integration testing facilities:

• All test modules containing ZopeTestCase classes load a special file called framework.py. It starts up ZopeLite which is a small Zope setup with a ZODB DemoStorage and no HTTP servers.
• Test modules can make calls to installProduct at module level to have certain products initialized before tests are even found.
• ZopeTestCase instances have access to a folder hierarchy provided by the setup. Given that ZopeLite and certain products are initialized, they can then pretty much test anything as if users were doing things to a live Zope instance.

The problem: there's no tear down and therefore no isolation. If I have unit tests and ZopeTestCase tests running together, the unit tests will be executed in the context of ZopeLite and all the initialized products that other tests wanted to have installed.

For this reason, for example, you're definitely asking for trouble when doing installProducts('Five') because that loads all of Five's ZCML, therefore registers things that your unit tests might not expect to be there.

The new testrunner's solution

Since Zope 3.2/2.9, we have a new testrunner in zope.testing. The test.py script is a small facade for it. The new testrunner supports a concept called layers. Layers are levels of test setup and all tests of the same layer are executed at once. From zope.testing's testrunner.txt:

Most tests are unit tests. They don't depend on other facilities, or set up whatever dependencies they have. For larger applications, it's useful to specify common facilities that a large number of tests share. Making each test set up and and tear down these facilities is both ineffecient and inconvenient. For this reason, we've introduced the concept of layers, based on the idea of layered application architectures. Software build for a layer should be able to depend on the facilities of lower layers already being set up. (...)

How layers work

Layers are objects with setUp and tearDown methods. When using classes as layers, we must use class methods:

class MyLayer(object):

    @classmethod
    def setUp(self):
        # do something here

    @classmethod
    def tearDown(self):
        # undo it here

You can nest layers using class inheritance:

class MyExtendedLayer(MyLayer):

    @classmethod
    def setUp(self):
        # do additional stuff here
        # don't call super

    @classmethod
    def tearDown(self):
        # undo it only the additional stuff here
        # don't call super

The test runner will first set up MyLayer and run all tests of that layer. Then it will set up MyExtendedLayer and run all tests of that layer. When done, it will tear down MyExtendedLayer, then MyLayer.

As you can imagine, this allows very fine grained test setup. For example, there can be a very general layer that bootstraps common things; specific tests can then supply their own test setup based on that layer (by subclassing from the layer).

Using layers for integration tests

In Zope 3, unit tests simply have no layer. Integration tests (we call them "functional tests") that rely on a full-blown ZCML setup (starting at ftesting.zcml) can use zope.app.testing.functional.ZCMLLayer, for example. Other tests use their own layer.

I think ZopeTestCase, specifically ZopeLite, should be converted into a test layer. The trick is that it will also support tear down. The installProduct functionality should also be moved into layers, ones that obviously extend the ZopeLite layer. Again, tear down will be possible so that tests will be able to say which specific products they need (right now, all tests run with all the products installed that are somewhere specified as dependencies).

I think this might be a nice sprint task. Paris, Switzerland, EuroPython... anyone?