I want my software to be to-spec. This means spec tests, which is fine, because I love testing.
However, I don’t want to rewrite the entire spectest suite for every language or version of a piece of software that I create. My answer to this problem is as follows:
- Since these are tests of conformance to a spec or API, write the tests once only, expressing them as messages
- These messages are sent from a master program to a shim program
- The master, which is used for all spec tests, knows only about sending messages, accepting results, and comparing those results to a predefined set of expected results
- The shim, which must be written for each implementation, knows how to interpret the messages from the master program, and how to use its implementation to compute answers
The messages themselves will be JSON, using a grammar to be defined a little later.
What I’ve worked on so far is the master program. It’ll be written in Perl 5, and will use IPC::Open2 to talk bidirectionally to its shim programs, using pipes. Pipes are very Unix-y, and very well understood, but they also have problems (which are themselves well understood). The classical problem of a producer-consumer bidi pipe setup, like what I’ll have, is deadlock — where both ends are waiting for the other one to write.
My solution to this problem is to add another pipe to the master; a unidirectional one which talks to a tiny watchdog program.
Just before the master sends a test message to a shim program, it will tell the watchdog what it’s doing, its pid, and the pid of the shim program in use.
The watchdog will be in a loop, mostly sleeping, and checking for output from the master every time it wakes up. If it finds output, everything is swell. If it doesn’t, an elapsed timer is incremented. If too much time goes by with no output from the master, the watchdog declares that the master is in deadlock and kills both the shim and the master before shutting down.
This evening I got the master-watchdog bits working, in about 2 hours. If I’d ever done this kind of IPC work before, it would have gone faster, because I would already have known about the proper use of sysread and so on to avoid blocking (and locking up the watchdog) when checking for input.
Here’s a sample run of the system as it stands now (still with development scaffolding still in place, obviously):
mdxi@hydra:~/projects/Catechesis$ ./catechist This is the master program. My pid is 4181. wd: Watchdog pid is 4182 wd: Time since seen: 5 seconds master> foo master> bar wd: foo wd: bar master> baz wd: baz wd: Time since seen: 5 seconds wd: Time since seen: 10 seconds wd: Time since seen: 15 seconds master> quux wd: quux master> QUIT mdxi@hydra:~/projects/Catechesis$
As you can see, it’s called Catechesis — as in the process of catechism — because its job is to enforce adherence to dogma by call-and-response.
This should be coming along quickly. It’ll grow stepwise, as needed, because this is the bottom of the stack.
EDIT: Forgot to mention the crazy bit. The watchdog isn’t a separate script. It’s inside the master, stored as a string, which is passed as the argument to ‘perl -e’ when that command is called via the IPC form of ‘open’. And that’s how the watchdog launches.
