Philipp von Weitershausen

Professional Plone Development -- a review

Recently I volunteered to set up a website for a small student association at my university. Nothing fancy: corporate design, community-managed content, private member area for managing the association's internal affairs, etc. "Sure thing," I thought, "I'll just use Plone. It's something I know." But it turned out I didn't really. Anymore.

You see, I rarely do content-management projects, and the last time I customized and deployed Plone was when Archetypes was considered the cutting edge. Luckily, I had been given Martin Aspeli's book Professional Plone Development for review, so to kill two birds with one stone, I did what any Plone noob is supposed to: RTFB.

Professional Plone Development starts out with a gentle introduction to the Plone community and software. It goes on explaining how to set up a development environment. This chapter is very well sorted out, especially insofar as it's currently the only printed material that explains the techniques revolving around eggs and zc.buildout.

Over the next couple of chapters, the book then takes you on a typical walk through the Plone customization jungle: though-the-web customization, policy products and GenericSetup profiles, security and workflow, custom themes. These things changed drastically in the last Plone releases, and even though they're now often based on Zope 3 techniques, I had no idea about the specifics of the system. With the help of just this part of the book, I had my 90% of my website up and running. But then again, it's a small and simple site.

The rest of the book dives into the core of Plone and Zope programming. Here's where I have my only few negative points of criticism. Chapter 9 in which Martin explains a couple of core Zope concepts (Object Publishing, Acquisition, Persistence, Component Architecture, etc.) feels a bit disconnected from the rest and frankly overwhelming (not to me, mind you, but I'm imagining for others it might be). Some of these concepts could have easily explained earlier on and some of them only start to make sense when you're actually using them.

Admittedly, it is hard to talk about browser views or automated forms without some Component Architecture theory. Also, the slightly dry Chapter 9 does make a good reference for later. Lastly, the criticism is completely outweighed by another highlight that awaits the reader at the end of the book: a couple of very thorough chapters on deployment.

Looking at this book, I can't help but wonder: how can he cover it all in just onder 400 pages? This is not only a very good book, it's also good value for money.

P.S.: I wrote up this review a while ago already but forgot to post it, so apologies if it comes a bit late now...

From the PEP8 department...

Since a lot of people trip over this, let me repeat an important and useful passage from PEP8, the Python styleguide:

- Comparisons to singletons like None should always be done with
  'is' or 'is not', never the equality operators.

  Also, beware of writing "if x" when you really mean "if x is not None"
  -- e.g. when testing whether a variable or argument that defaults to
  None was set to some other value.  The other value might have a type
  (such as a container) that could be false in a boolean context!

In particular, you should avoid code like

  ob = retrieveObjectSomehow(..., default=None)
  if ob:
      # you'll never get here if ob is an IContainer and empty

If you know that None is the fallback, then check for it explicitly:

  ob = retrieveObjectSomehow(..., default=None)
  if ob is not None:

I don't know why people frequently trip over this, I blame tutorials that draw a misleading picture of Python's polymorphism.

It's not a bug...

... but at least the engine is where God intended it to be: aft.

Yup, that's her, a 1991 VW Transporter double-cab pickup. Not a beauty and not really fast (the 1.7L diesel develops a staggering 57 bhp), but she's in good shape and has loads of space as you can imagine. I'm already looking forward to some skiing trips.

She's also quite economical since she's taxed like a truck and will happily eat vegetable oil. Saw her for sale online when I was visiting the folks at Jarn in Norway and bought her the day I got back home. I mounted the Mexican number plate in reminiscence of my wonderful time in Mexico where I drove a Toyoto Hilux, a vehicle not unlike this one that changed my views on pickup trucks in general and Toyota in particular.

Meanwhile, I'm moving the bug to the workshop soon where it'll receive a complete make-over of the chassis and body.

Can't hardly wait: PloneConf 2007

With all the Zope 3 work I'm getting these days, I can't really contribute much to Plone or do customer work with it anymore. But I still love attending their get-togethers. Doesn't matter whether it's the Snow Sprint high up in the Alps or a conference in Vienna, Seattle or Naples - you always know you're in for endless loads of fun.

This year's conference starts this coming Wednesday. Looking at the agenda, I find an interesting recurring theme (among many others) that we haven't had at a Plone conference before: Looking over the fence and learning from others.

For instance, there's Andy McKay's talk What Plone can learn from Rails on Friday. On Wednesday, we have Lennart Regebro's talk about What Zope did wrong (and what to do instead). And then there's my talk on Thursday called Plone Age, in which I will look at lessons learned from other (Python) web frameworks, as well as recent developments in the Zope 3 scene.

With these and all the other fantastically sounding talks in the agenda, I can't hardly wait to get to Naples!

Little bit of history repeating...

The talk schedule for this year's Plone Conference is out. The amount and variety of talks are stunning. Sadly and funnily at the same time, I'm up against the exact same as last year: Introduction to KSS.

Announcing zopeproject 0.4

zopeproject makes it easy to get started with a web application based on Zope eggs, zc.buildout and WSGI/Paste. If you're not familiar with this world yet, zopeproject is your easy entry to it. If you are, zopeproject will save you lots of typing up the boilerplate.

Why eggs?

Because they are a great way to distribute Python software and declare its dependencies. Zope 3.4 is completely distributed as individual eggs and will from now on be developed as such.

What's zc.buildout?

It's a tool for repeatable deployments that's configuration-file driven. You tell it "install these eggs for me, create this script and let me develop packages in this directory" and it will go ahead and do all that. Over and over again, if necessary, and for everybody who wants the same setup as you (which is great for teams).

What are WSGI and Paste?

WSGI is a Python standard for web applications and web servers. It defines how they communicate so that potentially any WSGI application can run on any WSGI gateway (server). Zope has had support for WSGI for a while now, but it has never really been exposed, even though it offers great possibilities such as middlewares.

Paste is a collection of related to WSGI. One of its tools, PasteDeploy, is of particular interest. It allows you to plug WSGI applications, middlwares and gateways together using a simple configuration file.

zopeproject vs. instances

Apart from all the flashy technology that zopeproject brings us, it also implies a big conceptual change in the way we develop web applications with Zope.

In the world of regular instances, Zope is the server and the application. Your code is just like a plug-in:

+------------+         +---------+
|    Zope    |  runs   |your code|
|server & app|  ---->  | (plugin)|
+------------+         +---------+

Even though we've been doing it for years ("Products" ring any bells?), I think this is backwards and accounts for much of the irritation people coming to Zope have.

With zopeproject's approach, your code is the application. The web application, to be precise. And it happens to use Zope. As a library:

+------------+           +-------------+          +---------+
|WSGI gateway|  serves   |  your code  |   uses   |  Zope   |
|  (server)  |  ------>  |(application)|   ---->  |libraries|
+------------+           +-------------+          +---------+

Zope isn't the center of attention anymore, your application is. Even better, you can choose how much of Zope you'd really like to use. You may compare this approach to a GUI application that happens to use some sort of GUI framework to get some widgets on the screen, but otherwise it's just like any other, real application.

So how does it work?

It's pretty easy. All you need to do is install zopeproject first. You don't need to install Zope for this, zopeproject will do that for you later. Note that you will have to have setuptools installed for this to work (it provides the easy_install script):

$ easy_install zopeproject

Depending on where and how you installed Python, you may have to prefix this command with "sudo" to gain privileges for installing packages globally.

Now you can create a new web application:

$ zopeproject WebApp

Much like with mkzopeinstance, you'll be asked a couple of questions. The last question concerns the central download location for eggs (which will include the Zope eggs). After the questions, zopeproject will create a 'WebApp' directory with some initial directory structure, as well as download and install the Zope eggs.

To start the application, go to the newly created directory and invoke the server:

$ cd WebApp
$ bin/paster serve deploy.ini

You may also use the webapp-ctl script which works much like zopectl:

$ bin/webapp-ctl fg

You will then be able to go to http://localhost:8080 and see the default Zope screen. (The standard boilerplate simply uses the Zope 3 default, after all. It is now up to you to change that.)

More information and feedback

For more information, please refer to zopeproject's homepage on PyPI: http://pypi.python.org/pypi/zopeproject

I'm very much interested in getting your feedback. Please direct questions that are of public interest to the zope3-users@zope.org list. Bugs may be reported in Launchpad: https://launchpad.net/zopeproject.

The Online Search Warrant

And now for something completely different... (in other words, unreleated to Python or Zope)

When it comes to national security in Germany, there's currently one topic dominating the press: online search warrants. Or rather the lack thereof. You see, German politicians and authorities involved in national security have demanded for many months now that online searches of computers on behalf of the police should be allowed without a search warrant issued by a court.

At first sight, it seems to make total sense. Surely there are terrorist who want to erect theocracies all over the world and bomb the rest of us heathens back to the middle ages. To reach that goal they don't seem to shy away from using modern means of communication, combat and sabotage, though. So this is a war of technology and we need to fight back with everything we got now, right? Considering the success the police had investigating the bombing of the London Underground in 2005 and the attempted bombing of trains in Dortmund and Kassel in 2006 was thanks to video surveillance, it seems that betting on technology pays off.

So why not go a step further and put the bad guys under surveilance before they bomb us to pieces? Let's catch them making plans, not executing them. What better tool could online searches be for this? Well, to be honest, I see a whole range of problems with them.

It's unconstitutional

From my point of view, this is by far the strongest argument. "Naturally we have to change the constitution to allow them," the supporters of online searches say. The problem with this is that this touches essential civil rights (Americans will know this as the Bill of Rights) that every Western democracy holds up so high. Once we start giving these up, we might just as well get out of the U.N. and stop telling the world that we're so better at treating people. I can already hear China saying "Told you so, Germany".

Lack of proportionality

Authorities claim that they can only conduct (and afford) 5 to 10 online searches a year. Is it really too much trouble getting a judge to sign a search warrant for such few incidents? Is it worth changing our constitution for this? It's not like the police isn't going to collect some evidence on the guy they're going to put under surveillance in the first place. After all, hacking into another guy's computer takes time and money, you better make sure it's worth it. So you might just as well get a judge involved.

Have they actually thought this through?

The means by which the authorities want to conduct these online searches aren't well known. The press obviously likes to speculate wildly (and usually gets it all wrong due to their lack of understanding the details).

The most obvious choice seems to be to buy a zero-day exploit for whichever operating system the person in question is using. I've heard these sell for a couple hundred thousand dollars. So the whole thing sure is pricey. Then you install a trojan (dubbed "Federal Trojan"). According to the press, this has been done already in an investigation (unconstitutionally, I might add).

The problem with the trojan is that if the police can get in that way, so can anybody else, especially if the system isn't well maintained (security updates, virus scanner, etc.). So in the end, an online search is like catching two cars on a speed camera. You know at least one of them has been speeding but you can't tell which one. That's why I doubt they'll hold up in court.

Politicians realized that too, of course, and backpedaled. Now they wouldn't use a trojan but rather fall back to more conventional methods of gaining access to the computer: breaking into the suspect's house and install the software or hardware (e.g. a keylogger) locally. Well, I sure hope that those terrorist have no laptops, otherwise them cops are goin' be real mad when they find out the terrorists always take that computer with them when they leave the house...

To me, all of this doesn't look like this has been thought through properly. Those who want it seem to think it's a silver bullit. Those who actually know a little bit about this stuff can't seem to see that it would be. I sure hope that parliament will be wise enough not to let this one slide by.

zc.buildout: new feature and speed up

Jim Fulton released zc.buildout 1.0.0b30 last night. This release is worth mentioning for two reasons:

buildout can now be told to prefer final versions over alpha/beta/dev releases. In normal mode, buildout will simply try to get the newest version of a package that matches the package requirements (just like easy_install). This means, however, that development releases may be installed instead of stable final releases, which usually is undesired in production environments. To get buidout to prefer final releases, put the following statement in your buildout.cfg:
```
[buildout]
prefer-final = true
```
buildout now uses PyPI's simple package index by default. To setuptools (and zc.buildout), this index looks like the regular PyPI, except that it's much speedier (because it doesn't have to render fancy HTML) and it lists all releases of a certain package, even the hidden ones. A while back, Jim had already played with a simple index like that, dubbed ppix, which uses PyPI's XML-RPC API to mirror the information.

I've done a little test and timed updating the grok buildout with buildout -n using the various indexes (standard PyPI, simple PyPI, ppix). Buildout had to go to whichever index I configured and check for newer versions of roughly 100 eggs (times are in minutes and seconds):

standard PyPI	3:09
simple PyPI	2:05
ppix	1:54

So, ppix is only marginally faster than simple PyPI (depending on how "close" you are to either one), but they're both quite a lot faster than the standard PyPI.

pdb'ing to eggs installed with zc.buildout

If you use zc.buildout to install eggs, it sometimes happens that you can't pdb into their code.

The reason for this is that the egg is built in a temporary directory and then moved to its destination. During the build, setuptools creates .pyc files which contain an absolute path reference to the original .py file. Since it's done a temp directory, this reference will be outdated.

A quick fix is to simply remove all the .pyc files from the egg and have them rebuilt automatically next time you use the package.

Cheap binary Windows eggs

Thanks to Hanno Schlichting's howto, I've figured out how to create Windows eggs of those packages that have C extensions. This approach doesn't need Microsoft Visual Studio, nor does it require you to wade through a bunch of free Microsoft downloads that don't really work in the end anyway.

Here's what I did:

Installed the standard Python 2.4 distribution from the MSI package.
Installed the MingW compiler (into the standard location C:\MingW)
Created C:\Documents and Settings\Philipp\pydistutils.cfg and put the following text in it:
```
[build]
compiler=mingw32
```
This tells distutils to always use the MingW compiler whenever it has to compile something from C.
Went to the Control Panel -> System -> Advanced tab and clicked on the Environment Variables button. There I appended the following text to the the Path environment variable, adding the Python interpreter as well as MingW's programs to the search path:
```
;C:\Python24;C:\MingW\bin
```
Then I added another environment variabled called HOME with the following value:
```
C:\Documents and Settings\Philipp
```
This points distutils at the pydistutils.cfg file that I created earlier (you can put the pydistutils.cfg file anywhere you want, you just need to make sure that the HOME environment variable points to the directory).
With that in place, I am able to take any tarball (e.g. zope.interface-3.4.0.tgz), unzip it and create a Windows egg from it like so:
```
python setup.py bdist_egg
```

What's more, with a setup like this, it is easily possible to install Zope 3 completely from eggs (e.g. using zc.buildout) even if there are no pre-built Windows eggs on the Cheeseshop. More specifically, with this setup, zopeproject (which is really just a convenience tool over zc.buildout) works like a charm on Windows now.

Please stop using request.RESPONSE

The uppercasing of the REQUEST and RESPONSE variables is a Zope2ism that should die.

Sometimes it may still be necessary to write REQUEST (e.g. when acquiring it from somewhere or taking it as a parameter, but even then you should really ask yourself whether you shouldn't rather write a view component).

Either way, writing request.RESPONSE is completely unnecessary because the Zope 2 request also allows you to write request.response which will instantly work on Zope 3 as well, therefore increase the portability of your code.

Grok movie trailer

Zope is back (by Martijn Faassen).

mkzopeapp now called zopeproject

After expressing my wish to kill Zope instances in a recent blog post, I had been discussing these ideas with Jim Fulton and Martijn Faassen. Their valuable feedback had led to some refactorings of mkzopeapp, the tool that would set you up with a Zope development sandbox (which would basically replace instances as a concept).

Last week at EuroPython, I presented my ideas in a lightning talk and a 45-min brainstorming session with quite a few Zope and non-Zope folks. Especially the input from people like James Gardner of Pylons fame and Ignas Mikalajunas from the SchoolTool project was quite useful. This feedback led to even further refactorings of mkzopeapp. In particular, I dumped zope.paste for defining the application factory. Instead, each newly created application now contains its own application factory. It is only a few lines long: it processes zope.conf, loads site.zcml and opens the ZODB databases. The great thing is that developers now have much more control over what happens at startup and how they can configure their application from the PasteDeploy INI file. In particular, it will be much easier to write ZODB-less applications this way.

At the sprint after the conference, my main task was to bring Grok over to Zope 3.4 eggs. I subsequently improved grokproject, the sandbox creation tool we had for Grok, and I realized that it shared a lot of infrastructure with mkzopeapp. I therefore factored this common infrastructure out into mkzopeapp, now called zopeproject. grokproject (or at least a branch of it, for now) now merely uses zopeproject for the heavy lifting.

From the `zopeproject` README:

With zopeproject you can start a new Zope-based web application from scratch with just a two commands:

$ easy_install zopeproject
$ zopeproject MyZopeProj

This will ask you the name and password for an initial administrator user. It will also ask you where to put the Python packages ("eggs") that it downloads. This way multiple projects created with zopeproject can share the same packages and won't have to download them each time.

After asking the questions, zopeproject will download the zc.buildout package that will be used to build the sandbox, unless zc.buildout is already installed locally. Then it will invoke zc.buildout to download Zope and its dependecies. If you're doing this for the first time or not sharing packages between different projects, this may take a while.

When zopeproject is done, you will find a typical Python package development environment in the MyZopeProj directory: the package itself (myzopeproj) and a setup.py script. There's also a bin directory that contains scripts, such as paster which can be used to start the application:

$ cd MyZopeProj
$ bin/paster serve deploy.ini

Death to instances!

There. I've said it. You can beat me up now.

Finished? Ok, then I can get to my point: I think Zope instances are an overcome concept, now that we've finally sorted out that Zope isn't an application anymore.

Confused? Fair enough, I'll explain.

Overcoming an identity crisis

Zope is in an identity crisis. Both Zope 2 and 3 don't really know what they want to be. A content management application? An application server? A platform? A set of libraries? They're sort of everything and nothing. In its long life time of almost a decade, Zope has therefore attracted and disappointed different kinds of people:

People who really weren't developers were attracted by the ZMI and all the through-the-web stuff, but sooner or later they had to discover the "Z-shaped" learning curve: in the beginning Zope is really easy, but when you get past a certain point it becomes very hard, until you've mastered it. Usually that involved learning a substantial amount of Python (remember, we're talking about non-developers here).
For a long time, Zope was pretty much the only thing out there that did cool web stuff with Python. But by trying to appeal to non-developers in a way that's so un-intuitive to developers (through-the-web development, anyone?), it has scared off so many Pythonistas that Zope, even after trying very very hard with Zope 3, is forever branded as the most un-Pythonic piece of code out there (even if that wasn't even true at the time this prejudicial judgement was conceived...).

Tres Seaver's keynote at this year's DZUG Tagung in Potsdam revolved around this very problem. At the end, he suggested that Zope should make up its mind as to what it wants to be. His suggestion was that Zope should be a platform. Not an application. Not a server. Just a platform

with lots of libraries that help you create a web application (this is what's currently known as Zope 3),
with a particular set of those libraries (and a few legacy ones) that make up a "known good" assembly for a web application platform (this is what's currently known as Zope 2),
but without necessarily providing its own HTTP server implementation, its own process management tools, its own ways of installing software.

What's important to note here is that Zope would not be an application that "eats" your 3rd party code to be pluggable. Instead, it'd be you who would write the web application and this application happens to use the Zope platform (the Zope libraries), just like other applications happen to use TurboGears or Pylons. Think of writing a GUI application that happens to use wxPython. Would you expect that you would have to launch the "wxPython-the-application" in order to launch your GUI app? I think not.

So what's the problem with instances?

Primarily an instance is a place that contains your code which is plugged into Zope-the-application. So strictly speaking your code isn't really the application, it's just stuff that plugs into Zope-the-application. And the configuration that's in the instance, too, doesn't configure your application. It configures Zope-the-application. And the scripts that are also in the instance are aptly named "runzope" and "zopectl".

Instances therefore embody the concept of Zope-the-application. They're not very helpful in bringing forward the platform idea of Zope.

So how would I write an app with using Zope instead of a plug-in for Zope?

Simple. You start by writing Python code. And where does Python code go? Right, into a package. So you start by creating a package.

That's right, you don't start by making an instance. You simply start by doing something you'd do with any other piece of Python software you'd write: create a package with code. Naturally you're using Zope, so you'd put all the kinds of Zope-based components in that package that you know: ZODB-persistent classes, views, Page Templates, etc. The more of Zope's components you use, the more dependencies your application will grow, which can all be tracked using the a standard setup.py script that we know from other setuptools-based distributions.

What's the next step? Configuring your application's components. Surely you want to use the Zope publisher so that your views will be found. And you probably want Zope's security machinery as well. So before configuring all the components that you yourself have written, you want to load those components from Zope that you're reusing. So your package's configure.zcml file will start off by including lots of Zope bits and pieces, perhaps like so:

...
<include package="zope.publisher" />
<include package="zope.traversing" />
<include package="zope.app.securitypolicy" />
<include package="zope.app.authentication" />
<include package="zope.app.session" />
...

Great. Now how do you start your application? Essentially what you'd want during developing time is a web server that processes all requests by sending them to Zope publisher (which will then invoke your application's components). PasteDeploy is a small library that helps connecting applications to web servers. It is configured using a simple INI-style configuration file, such as this:

[app:main]
use = egg:zope.paste
site_definition = configure.zcml
file_storage = /path/to/datadir/Data.fs

[server:main]
use = egg:PasteScript#wsgiutils
host = 127.0.0.1
port = 8080

Here we configure the two bits we want to connect:

the application, represented by zope.paste, a PasteDeploy-friendly frontend for any Zope publisher-based web application, and configure.zcml of which we've seen an excerpt above.
and the web server, which happens to be a simple one from a package called WSGIUtils.

After that little bit of configuration (which might remind you a bit of zope.conf) you can start your application by invoking Paste:

$ paster serve your-app.ini

Now the good news: this already works. Today.

Automating the common case

So all in all, writing an application with Zope-the-platform (instead of writing a plug-in for Zope-the-application) is pretty simple:

Start with a package and put your application's code in it. Add to its dependencies as you're using more of Zope's own components.
Configure your application's components, not forgetting to configure those components of Zope that you're reusing (the publisher, the security machinery, etc.). This configuration can go right into the application's package as well.
Hook the application up to an HTTP server. Done.

Admittedly, that's a bit more work than with instances. With instances, you simply invoke mkzopeinstance and you ended up with a pre-configured runnable Zope-the-application. To get the same kind of convenience for beginners, we'd simply need a mkzopeapp script that creates a Python package with some pre-made component and server configuration.

Good news, everybody! It exists! Here's how you'd get Zope up and running with it. First of all, you'd need to install mkzopeapp (it is suggested to perform all of this in a workingenv to avoid pollution of your global Python installation):

$ easy_install mkzopeapp

Note that this doesn't mean you need to install Zope first or that Zope will be installed in the process. For now, that just installs the script that sets you up with an empty package layout.

Now that the mkzopeapp script is available, we can simply invoke it, just like mkzopeinstance:

$ mkzopeapp MyZopeProj

Inside the MyZopeProj directory, however, it looks nothing like an instance. It's simply a standard Python package layout with a setup.py. The Python package, though, contains a configure.zcml file that loads the most common Zope components. It also contains a develop.ini file that contains a simple HTTP server configuration not unlike the one we've seen above.

To get the application up and running, we need to install its dependencies (which happens to be a lot of Zope libraries). We do that by activating the package as a development egg:

$ cd MyZopeProj
$ python setup.py develop -f http://download.zope.org/distribution

The -f switch tells setuptools where to find the Zope packages. Note that this may take a few minutes until all the packages have been downloaded. When that is done and MyZopeProj has been activated as an egg, you will also get a new script that you can use to directly start the application:

$ startMyZopeProj
Starting server in PID 1217.
Starting HTTP server on http://127.0.0.1:8080

Conclusion

As we've seen, writing applications using Zope isn't any more difficult than plugging into Zope-the-application, thanks to mkzopeapp.

You might also have realized that this way of arranging a web application makes it much easier to install and distribute: you simply install that one package and it contains a runnable, configured application. And because it lists Zope's libraries as dependencies, they will be installed automatically as well. So you no longer install Zope and then add the application code. You install the application, and Zope happens to be pulled in as a consequence. And if your application package specifies a specific version of Zope, people won't have problems by accidentally having installed the wrong version of Zope.

I could go on and come up with lots more advantages of the platform-centric view. And certainly there are a few things where instances still deliver more features (debugging comes to mind). Certainly those can be sorted out in the near future, hopefully with your help!

Custom traversal with Grok

Marius Gedminas has written down how to implement custom traversal for a content object in Zope 3 (which also applies to Zope 2.10+ now, btw). As he admits, it's a bit complicated. In an email conversation, I showed him how easy it is in Grok:

When we created Grok, we realized this obvious complication in Zope 3 and made custom traversal very easy in Grok. You can either implement a traverse() method directly on a model:
class Appointment(grok.Model):
    pass

class Day(grok.Model):
    def traverse(self, number):
        appointment = self.getAppointment(int(number))
        if appointment.published():
            return appointment
        # if traverse() returns None, the default fallback
        # behaviour is assumed (looking up views usually).
Or you can implement an (exchangeable!) separate traverser component:
class Day(grok.Model):
    pass

class DayTraverser(grok.Traverser):
    grok.context(Day)

    def traverse(self, number):
        appointment = self.context.getAppointment(int(number))
        if appointment.published():
            return appointment

Update: Regarding that last example, Marius asked: "How's the binding done? Does grok look at the class name?" I answered:

grok.Traverser is actually a fully qualified IBrowserPublisher multi-adapter, like the one you have written in your blog. So it's registered as a multi-adapter (no class name magic going on!). The grok.Traverser base class simply lets you implement a hook -- the traverse() method -- and does all the boilerplate stuff for you, that way you don't have to know about IBrowserPublisher and all those fallbacks...

A bug I'll gladly fix

Here's for once a bug that I'll gladly spend my free time fixing: A 1971 Volkswagen 1302 S convertible I bought this past weekend.

Given the sunny days with temperatures well over 25C (77F), I'm already looking forward to taking it out to nearby Saxon Switzerland.

Why ZClasses are dead and why Grok lives

Recently there has been a bit of discussion on the zope-dev mailinglist about ZClasses. To those unfamiliar with this particular Zope 2 institution, ZClasses are a way to compose content types ("classes") using through-the-web (TTW) development techniques. You would define one or more base classes, give it a name and then add methods (= views implemented as DTML, ZPT or PythonScripts).

Among Zope developers, ZClasses have been discouraged for a while. As of Zope 2.9, ZClasses don't work anymore. Something broke their fragile code. They're officially dead. How has it come to that?

ZClasses: 1000 years of cruft

Why has nobody cared to fix ZClasses if they're so great? Well, because most developers first of all don't care about ZClasses (more below), and second of all they don't understand their magical code. In fact, one of my favourite quotes from Zope source code ever is from ZClasses:

def _setBasesHoldOnToYourButts(self, bases):
    # Eeeek
    copy=self.__class__(self.id, self.title, bases,
                        hasattr(self._zclass_, '_p_deactivate')
                        )

I'll let this code speak for itself.

Sure, they're agile, but...

Back in 2000, ZClasses were cool because they allowed for rapid prototyping and agile development like no other tool. It's 7 years later and the world has changed. Ruby on Rails, for example, has taught us that actual code, written in a dynamic language can also be agile, if you enforce or at least encourage certain defaults. Django and TurboGears have done the same in the Python world.

File system code has lots of advantages over TTW code, as I've pointed out in an earlier post. It is because of reasons presented in that article that people just don't care about ZClasses anymore. It is also why many people in the Plone community, for instance, care less and less about the TTW development model. I can understand them. Standard file-system based Python development, especially with Zope 3 components, is much more appealing to experienced developers, who after all make up the Plone and Zope core developer teams.

Zope 3 - the answer to life, the universe and everything?

Now Zope 3 is an incredibly powerful framework. Jeff Shell, who didn't always have nice things to say about it (apparently it drove him back to smoking), just now wrote:

We've been using Zope 3 in earnest for just over a year and a half now. I would like to report that in that year and a half our little company has achieved more re-use than at any time in our history.

But Zope 3 isn't exactly known for its agility. Those who've been defending the TTW use case, be it some of the Plone folks or Christopher Lozinski make a good point: how on earth is somebody going to learn interfaces, ZCML, security declarations, etc. just to customize a teeny bit in Plone? Or just to write a simple application for his boss?

"They don't have to" is Grok's answer, and the friendly caveman goes on to say: "But they also don't have to give up that flexiblity given to us by Zope 3."

Low bar for contributions

This ZClasses discussion has also made me aware of another good point. Christopher Lozinski, who's repeatedly come to the zope-dev mailinglist, willing to contribute to ZClasses to keep them alive, hasn't been able to write a single of code yet to so. Why? As a user of ZClasses, it's almost impossible for him to tackle the magic that lies in the code of ZClasses. Heck, if some core contributors can't do it without a considerable amount of time (hence money), how should he?

With Grok, on the other hands, we now have newbies contributing to the Grok source code itself only after a few months. Tim Terlegard, whom I trained in Zope 3 only a few months back at the Lovely Training in Copenhagen, has implemented JSON functionality for Grok. Darryl Cousins has been enhancing Grok's admin UI as well as fixing bugs and writing tests. Kevin Smith has been looking into implementing layers and skins. They've all been with Grok for only a few weeks or months.

This is what makes me confident enough to think that we're on the right track with Grok.

Meet Grok, the successor of through-the-web development

For the most part I think that through-the-web (TTW) development as we know it from Zope 2 is a failed approach. I understand that some people see a lot of advantages in it, such as

non-developers apparently being very productive without having to learn how to "program,"
the ability to develop collaboratively on one Zope instance,
restarts not being necessary,
the ability to fix bugs easily in the production system without needing shell access.

I see most of those points as disadvantages (except for the reload thing, as that clearly helps with agility). TTW development misleads Zope beginners and tricks them into going down a very steep path that's hard to go back.

TTW development vs. standard Python development

In order to write a TTW application, you'll end up having to learn Python, even if you're not a developer per se. The problem is that you don't actually use Python like it's taught in the books. Instead of writing functions, classes or methods in modules, you write Python Scripts, which looks like a function/method without a def statement, are compiled using a special compiler that adds lots of security checks, and have some pre-defined top-level variables.

To add obstacles to the weirdness, it's not possible to access standard Python functionality like opening files on the filesystem or importing a module from the standard library in TTW code, due to security concerns.

The Z-shaped learning curve

What happens if you hit said limitations of TTW development? You need to convert all that code in Python Scripts to actual functions, classes and methods on the filesystem. This is not a trivial task, filesystem-based code typically looks a lot different than TTW code. People have to basically unlearn what they've learned about Zope development so far and learn this whole other way of writing so-called "Products". This is why people jokingly referred to the "Z-shaped learning curve" when it comes to learning Zope: it's not a gradual process.

Lastly, there's no automation for such a conversion, at least not for the whole process. There is FSDump which gets you half way, and there's Frostbite which gets you there but produces incomprehensible code that's not really meant to be edited further. Also both only work with the CMF.

TTW development vs. established tools

TTW development encourages working on one development system, even if you're multiple developers. One developer can easily break the whole application for the others and make a whole team unproductive until the problem is fixed. I've been there, it sucks.

TTW development also makes it hard to impossible to use standard development tools such as an IDE, source control system and even a dead-simple thing like grep. Non-developers don't care about this, you might say. That makes it even worse. TTW development discourages using standard practices and tools such as version control. I've consulted on Zope projects that where 100% TTW and had no source control. It made me uneasy. I wouldn't accept client projects like that anymore.

Grok to the rescue!

Zope 3 gets many things right in this respect. It has pretty much done away with TTW development which means we don't have a Z-shaped learning curve anymore. That's good.

The problem is that Zope 3 is lacking the equivalent of TTW development as a way to ease people into the platform. Right now you have to drink the kool-aid in one big gulp: Component Architecture, interfaces, security, ZCML and much more.

The Grok project addresses that problem: it lets you get started with a Zope 3-style web application without having to learn about interfaces, components, or any of that stuff first. Its goal is to bring agility back into Zope and to flatten the learning curve for newbies. The good thing about Grok is that it doesn't make you unlearn things (unlike TTW development). You can gradually improve your Grok application and make use of interfaces, security, etc. once you learn about them. You don't have to throw away your old code.

Furthermore, Grok code is regular filesystem-based Python code. There's nothing funny about it. You write functions, classes and methods, just like that Python book taught you. You can use your favourite text editor. You can check your stuff into a source control system.

Is Grok really as easy as TTW development? I believe so. Kevin Teague recently shared his experience with Grok on the mailinglist after writing a pastebin-like application:

Consider the plain-text Paste view:

class PastePlainView(grok.View):
    "Plain-text view of a single Paste"
    grok.context(Paste)
    grok.name('plain')

    def render(self):
        return self.context.body

Where in Zope 2 I might write:

## Script (Python) "pastePlainText"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=
##title=Plain-text view of a single Paste

return context.body

The Grok version doesn't seem to be harder to write as the TTW version from Zope 2. Au contraire, the Grok version is something that somebody familiar with Python will immediately understand. Meet Grok, the successor of TTW development.

Gone to USA, Mexico for 1 month

I'm leaving for a 1 month trip thru Eastern USA and Mexico next week. Some cities I'll be stopping by:

Boston, MA
Zope Town a.k.a. F12g a.k.a. F'burg a.k.a. Fredericksburg, VA
Washington D.C. (to speak about grok at the March meeting of the Zope/Python Users Group of D.C.)
Chapel Hill, NC (to give the Camp5 Zope 3 training and attend the BBQ sprint)
Guadalajara, Jalisco, Mexico
Mexico D.F.

If you happen to live or be in any of these places and wanna meet up for a beer, drop me a line :)

Good time to get into Zope

The Zope and Plone community is looking for more talented developers. It's not a secret that virtually every Plone consultant is booked out in advance for several months. But it's not only a good time to get into Plone, it's also a good time to get into Zope 3. That's not just because Plone is adopting Zope 3 technology on a large basis (I'm really excited about Plone 3.0 in that respect, by the way). It's also because Zope companies like Zope Corporation, gocept, Lovely Systems, and others are looking for dedicated Zope 3 experts to hire.

To make sure that there are enough experts in the Plone community, people like Joel Burton are giving trainings on Plone. This is good, but apparently not enough. We need more of such trainings, especially dedicated to Zope 3 technologies, to ensure we can actually deliver Zope and Plone solutions to our customers. Because if we don't have enough manpower to do so, they're going some place else. This is why I'm very excited that people like Sasha Vincic of Lovely Systems and Chris Calloway of the Triangle Zope User Group are organizing Zope 3 trainings (in which I have the honour of playing the trainer role). The next training in America is coming up soon in March, one in Europe is planned for May.

And you thought Zope 3 wasn't fun

If you think that Zope 3, or even Zope in general, isn't fun, agile, developer-friendly, easily understandable or "Pythonic", enter Grok. Here's an excerpt from Grok's README file:

Grok uses the Component Architecture and builds on Zope 3 concepts like content objects (models), views, and adapters. Its simplicity lies in using convention over configuration and sensible defaults when wiring components together. That means neither a configuration language like ZCML nor a lot of repitition are needed to create a web application with grok.

For the last couple of months, Grok has been developed by a handful of Zope 3 core developers. Like Five and lxml, Grok was originally thought up by Martijn Faassen at the EuroPython 2006 sprint. After we gave some input to Martijn's early design notes, five of us met for a first Grok sprint in October. This past weekend I had the pleasure of hosting the second Grok sprint. A first release is only a few weeks away, with a 1.0 scheduled some time in the second quarter this year (after a third sprint).

Since there have been several blog articles mentioning Grok and its philosophy already, I'd like to answer a FAQ that I get from developers:

So what does Grok code look like?

Imagine a simple web application for managing a herd of mammoths. For the sake of simplicity, we only support adding a mammoth to the herd and searching for it afterwards (more functionality like listing all mammoths in the database or deleting mammoths can be added with few lines of code). Data persistence will be provided automatically and transparently by the ZODB. The searching will facilitate the Zope Catalog utility. All of this fits into one Python module:

"""
herd.py -- grok-based web application for managing mammoths
"""
import grok
from zope import schema, interface, component
from zope.app.intid import IntIds
from zope.app.intid.interfaces import IIntIds
from zope.app.catalog.catalog import Catalog
from zope.app.catalog.interfaces import ICatalog
from zope.app.catalog.field import FieldIndex

def setup_catalog(catalog):
    catalog['name'] = FieldIndex('name', IMammoth) # only index mammoths

class Herd(grok.Application, grok.Container):
    """Root application object and container for Mammoth objects.
    Also sets up a catalog with a field index for indexing mammoths."""

    grok.local_utility(IntIds, provides=IIntIds) # necessary for catalog
    grok.local_utility(Catalog, provides=ICatalog, setup=setup_catalog)


class HerdIndex(grok.View):
    """Default view for herd objects.  It lets you search for mammoths
    inside the herd and add new mammoths.

    Actual HTML rendering by herd_templates/herdindex.pt (which is
    automatically found according to the name of this Python module
    and the name of this view class)"""

    grok.context(Herd) # indicates which model the view is for
    grok.name('index') # URL name of the view

    def update(self):
        """This method is executed before the template is rendered.
        It allows us to perform operations such as processing request
        parameters or precomputing stuff that the template may need."""
        query = self.request.form.get('query')
        if not query:
            return
        catalog = component.getUtility(ICatalog)
        self.search_results = catalog.searchResults(name=(query, query))


class IMammoth(interface.Interface):
    """Data schema for mammoths"""

    name = schema.TextLine(
        title=u'Name',
        description=u'Name of the Mammoth',
        )

class Mammoth(grok.Model):
    grok.implements(IMammoth)

    def __init__(self, name):
        self.name = name # this attribute will be indexed by the FieldIndex


class AddMammoth(grok.AddForm):
    """Form that creates a new mammoth."""
    grok.context(Herd)

    form_fields = grok.Fields(id=schema.TextLine(title=u"URL name"))
    form_fields += grok.AutoFields(IMammoth) # deduce form fields from schema

    @grok.action('Add mammoth')
    def add(self, id, name):
        self.context[id] = Mammoth(name)
        self.redirect(self.url(self.context))

The HTML of the default view for herds is rendered by the herdindex.pt template. By putting it in the herd_templates directory, it will be picked up automatically and associated with the view class in herd.py according to the name of the template file and the name of the view class:

<html> <!-- herd_templates/herdindex.pt -->
<body tal:define="results view/search_results|nothing">

<div>
  <h3>Search for a mammoth:</h3>
  <form tal:attributes="action view/url">
    <input type="text" name="query" />
    <input type="submit" value="Search for mammoth" />
  </form>
</div>

<div tal:condition="results">
  <h3>Search results:</h3>
  <ul>
    <li tal:repeat="mammoth results">
      <a href=""
         tal:attributes="href python:view.url(mammoth)"
         tal:content="mammoth/name">
        mammoth name goes here
      </a>
    </li>
  </ul>
</div>

<p>
  <a href="" tal:attributes="href python:view.url('addmammoth')">
    Add a mammoth
  </a>
</p>

</body>
</html>

That's it! That's the whole application! The only teeny bit of ZCML that we need is for registering the application with Zope 3:

<grok:grok package="herd" />

This will tell Grok to "grok" the components we defined and register them accordingly. No ZCML or any other configuration needed. Just take it out of the box and have fun. (I should note though that currently, this example will require some ZCML so that the initial Herd container can be created. We hope to solve that problem by the first release or the 1.0)

And you thought Zope 3 wasn't fun? We're just warming up!

Update: After receiving lots of feedback, particularly about the before() method on grok views, we decided to rename before() to update() in an attempt to make the name less confusing and more similar to existing Zope 3 idioms. update() can now also issue a redirect which will cause the template not to be rendered subsequently. Thanks to Martin Aspeli for suggesting that. The code snippet above has been changed accordingly.

Update 2: Grok now has an Application base class for an application's root object, so I updated the Herd class to use that.

Update 3: Grok now has a web site at http://grok.zope.org!

Zope 3 benchmark revisited: PowerBook vs. MacBook Pro

After a little more than two and a half years, I retired my PowerBook G4 and got myself a brandnew MacBook Pro. While I was at the Plone Conference in Seattle, Apple had finally upgraded the MacBook Pro to the Core 2 Duo. I had been waiting for that upgrade to happen, not only for performance reasons, but also because I was betting that they would upgrade other things along with it -- and they did (FireWire 800, better DVD burner). With the MacBook Pro line being over half a year old, I was now also confident enough to buy it after the hardware trouble they had with those machines initially.

Back in April when I bought my folks a Mac mini, I compared my PowerBook G4 to the slowest Intel Mac avaiable and found that the Mac mini was basically twice as fast in processor speed. Now I tried to reproduce the same "benchmark" that I did back then with the MBP:

==============  ===================  =======================  =========================
                     PowerBook              Mac mini                 MacBook Pro
==============  ===================  =======================  =========================
processor       PowerPC G4 1.33 GHz  Intel Core Solo 1.5 GHz  Intel Core 2 Duo 2.33 GHz
RAM             1 GiB                512 MiB                  2 GiB
harddisk        80 GB ATA 5400 rpm   60 GB SATA 5400 rpm      80 GB SATA 5400 rpm
--------------  --------------------------------------------  -------------------------
OS              Mac OS X 10.4.5 with latest security patches  Mac OS X 10.4.8
python                    2.4.2 from darwinports              2.4.3 from darwinports
zope 3                         trunk, r66263                  (same)
--------------  --------------------------------------------  -------------------------
pystone 1.1              19800 py/s               34700 py/s                 58800 py/s
--------------  -------------------  -----------------------  -------------------------
compile[1]                     77 s                     52 s                       27 s
1st startup[2]                 63 s                     25 s                       22 s
2nd startup[3]                 21 s                      8 s                        5 s
--------------  -------------------  -----------------------  -------------------------
unit tests[4]                 344 s                    173 s                      105 s
ftests setup                   25 s                     11 s                        5 s
       run                    610 s                    245 s                      160 s
==============  ===================  =======================  =========================

[1]	`make` command, measured via Unix `time`, real time shown.

[2]	`z3.py` command after having done "something else" (Zope 3 files not in disk cache); time measurement by Zope 3 startup machinery, real time shown.

[3]	`z3.py` command right after having terminated the first one; time measurements as above.

[4]	`test.py -p --all` command, time measurement by the test runner.

Ignoring tasks that obviously max out due to the harddrive (1st startup), it seems that the MacBook Pro is somewhere between 1.5 to 2 times faster than the Mac mini, making it 3 to 4 times faster than the PowerBook.

Of course, the MacBook has other advantages just besides the amazing speed:

2 GiB of RAM speed up the system considerably when many applications are running in parallel. Considering that I almost never quit apps, this improves overall productivity a lot.
It's an Intel machine, which means I can use Parallels to boot Windows XP (e.g. to test Python software on this platform, check websites in IE, make screenshots for documentation) and Linux (use valgrind to debug nasty C code). I can also use WINE to run IE6 on OS X directly.
It's noticeably lighter than the PowerBook.

Overall I'm impressed by how smooth Apple made the transition to the Intel platform. A year and a half after they announced the switch to Intel, and half as much time after the first Intel-based product came out, all applications that I need are available as native binaries. There were no problems migrating the system whatsoever, even though I resisted the temptation of having all my files copied over, I didn't need more than an afternoon of setting up the new machine and copying over my home directory via FireWire.

How to install IE6 on Mac OS X (Intel)

With the advent of my gorgeous new MacBook Pro, I obviously want to take advantage of having an Intel-based machine. One of those advantages is executing Windows binaries using Wine.

Now, I've got Parallels set up with Windows XP and Ubuntu Linux, allowing me to test things in foreign browsrs like IE and Konqueror quite easily. However, those VMs are on an external hard drive (due to their size) and may not always be required if you're just checking whether something renders correctly in IE or not. The approach discussed here only needs approximatly 150 MiB and integrates IE6 as an X11 window.

Here's how I got IE6 running "natively" on OS X:

Install prerequisites:
- XCode including the X11 SDK (which didn't seem to be installed with a default XCode installation). Both are on the DVD that came with the machine. Oh, and you need X11 itself as well, of course.
- MacPorts (formerly known as DarwinPorts)
Install necessary ports (using sudo port install <name>):
- freetype (needed by WINE)
- fontforge (needed by WINE)
- wget (needed by ies4linux)
- cabextract (needed by ies4linux)
Get WINE sources and compiled them:
- I know there's Darwine which aims at integrating WINE nicely into OSX (you can double click on an .exe), but it's not sufficient for ies4linux.
- Add /usr/X11R6/bin to your PATH, without it the configure script doesn't seem to find the OpenGL headers (yeah, it sounds weird, but trust me):
```
export PATH=$PATH:/usr/X11R6/bin
```
- Execute the configure script. If it complains about any libraries missing even though you installed the corresponding ports, try pointing it to the right library and include paths manually:
```
export CPPFLAGS=-I/opt/local/include
export LDFLAGS=-L/opt/local/lib
```
- When there are no complaints about missing stuff from configure, execute make depend, make and finally sudo make install.
- Try out WINE by executing one of the simple Windows programs like NOTEPAD.EXE:
```
wine NOTEPAD.EXE
```
  If it complaints about freetype not being available, try symlinking the libraries to a place where WINE can find them, e.g.:
```
sudo ln -s /opt/local/lib/libfreetype.* /usr/local/lib
```
Get ies4linux and install it by simply executing the ies4linux script. It will guide you through the setup and automatically download IE6 and set it up. Note that I did not try installing IE7 yet.

Update: Screenshot