From tapiwa.gutu at gmail.com Sat Nov 10 17:41:17 2012 From: tapiwa.gutu at gmail.com (Tapiwa Gutu) Date: Sat, 10 Nov 2012 19:41:17 +0200 Subject: [fiveam-devel] Unit Test Framework Message-ID: Faithful hackers, I decided to take up the challenge laid down here http://fare.livejournal.com/169346.html and try to consolidate the Common Lisp unit testing frameworks. I have written a framework that aims to consolidate all the major features of all your frameworks mentioned in this blog http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html . You can find it on Github https://github.com/tgutu/clunit. I also wrote a blog on the development of the framework and reasons for it here http://ml.sun.ac.za/2012/11/09/developing-a-unit-test-framework-part-1/ if you are interested. I would very much appreciate it, if you could join me in this effort and we all work together torwards making this project a success. Regards, Tapiwa -------------- next part -------------- An HTML attachment was scrubbed... URL: From c-riesbeck at northwestern.edu Sat Nov 10 19:28:12 2012 From: c-riesbeck at northwestern.edu (Christopher K Riesbeck) Date: Sat, 10 Nov 2012 19:28:12 +0000 Subject: [fiveam-devel] Unit Test Framework In-Reply-To: References: Message-ID: On a quick skim, looks like nice work. If you want to see if it handles a lot of the cases (better or as well) that lisp-unit was designed for, see http://www.cs.northwestern.edu/academics/courses/325/programs/exercise-tests.lisp This is what tests my students' solutions to exercises from Graham and elsewhere. I'd prefer to use a standard test framework in my class, but I would want that particular file to continue to be simple for novices to read and understand, since they're the target audience. I'll try making the translation myself sometime but it won't be for the next few months at least, due to time constraints. I notice that you didn't include an "assert-prints" though obviously it's not hard to use some wrapper macro like (assert-equal "...." (collect-output (print-dots 4))) I would recommend adding something comparable to lisp-unit's assert-equality that lets you handle special equality testers, like set-equal, epsilon-equal, etc. (I used to call it assert-predicate but that implied a unary rather than binary predicate.) Nice job. On Nov 10, 2012, at 11:41 AM, Tapiwa Gutu wrote: Faithful hackers, I decided to take up the challenge laid down here http://fare.livejournal.com/169346.html and try to consolidate the Common Lisp unit testing frameworks. I have written a framework that aims to consolidate all the major features of all your frameworks mentioned in this blog http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html. You can find it on Github https://github.com/tgutu/clunit. I also wrote a blog on the development of the framework and reasons for it here http://ml.sun.ac.za/2012/11/09/developing-a-unit-test-framework-part-1/ if you are interested. I would very much appreciate it, if you could join me in this effort and we all work together torwards making this project a success. Regards, Tapiwa Christopher Riesbeck Home page: http://www.cs.northwestern.edu/~riesbeck Calendar: http://calendar.yahoo.com/criesbeck -------------- next part -------------- An HTML attachment was scrubbed... URL: From tapiwa.gutu at gmail.com Sun Nov 11 08:02:54 2012 From: tapiwa.gutu at gmail.com (Tapiwa Gutu) Date: Sun, 11 Nov 2012 10:02:54 +0200 Subject: [fiveam-devel] Unit Test Framework In-Reply-To: References: Message-ID: Hey Chris, Thanks for the quick response. I took a look at the link you posted. The syntax I used was most heavily influenced by that in lisp-unit because what any given function or macro did was very clear from the name without having to read the documentation first. Names like ASSERT-TRUE and ASSERT-EQUAL are pretty self explanatory! Also the syntax is very simple and easy to read and use. Because the syntax is so close to that of lisp-unit I will make a quick translation with a few changes and test it. The changes would be as shown below, the rest is pretty much the same. (in-package :cs325-user) ;;; Define a suite for all our tests since CLUnit only groups according to test suites instead of the implicit package grouping in lisp-unit. (defsuite cs325-user ()) (define-test testname . body) -> (deftest testname (cs325-user) . body) I had left out ASSERT-PRINTS simply because I had not considered its uses well enough. But after you pointed it out, I realised having a built in form to test the messages you print in your library would be very handy. The definition of ASSERT-TRUE that I used is actually a predicate tester according to its common use in Lisp conditional forms such as IF and WHEN. I figured if you want to test for the specific value T you would use (assert-eq t form). (assert-equality 'set-equal '(car) (functions-referenced '(car l))) -> (assert-true (set-equal '(car) (functions-referenced '(car l))) There is an idea I have been toying with in my head. It seems the library generally works now so I think its now safe to write unit tests for the library itself. Now, I obviously need test the lower level functions and make sure that they are doing the correct thing. So I have a situation were I want to test A and B, but since B depends of A, if A fails its test there is no point testing B. I would need to first fix A before I can trust what B will do. Basically, I want to be able to short circuit tests. I want to add this without changing the current behaviour. Since execution of tests in a test suite is unordered, you would place the dependent tests in a child test suite. So something like this: 1. (defsuite A ()) 2. (defsuite B (A)) 3. Define low level function tests in A. 4. Define tests in B that depend on functions tested in A. 5. (run-suite 'A :stop-on-fail t) The RUN-SUITE keyword argument :use-debugger, is for interactive tests, this option would work for short circuiting batch tests. What do you think? On 10 November 2012 21:28, Christopher K Riesbeck < c-riesbeck at northwestern.edu> wrote: > On a quick skim, looks like nice work. If you want to see if it handles > a lot of the cases (better or as well) that lisp-unit was designed for, see > > > http://www.cs.northwestern.edu/academics/courses/325/programs/exercise-tests.lisp > > This is what tests my students' solutions to exercises from Graham and > elsewhere. I'd prefer to use a standard test framework in my class, but I > would want that particular file to continue to be simple for novices to > read and understand, since they're the target audience. > > I'll try making the translation myself sometime but it won't be for the > next few months at least, due to time constraints. > > I notice that you didn't include an "assert-prints" though obviously > it's not hard to use some wrapper macro like > > (assert-equal "...." (collect-output (print-dots 4))) > > I would recommend adding something comparable to lisp-unit's > assert-equality that lets you handle special equality testers, like > set-equal, epsilon-equal, etc. (I used to call it assert-predicate but that > implied a unary rather than binary predicate.) > > Nice job. > > > > On Nov 10, 2012, at 11:41 AM, Tapiwa Gutu wrote: > > Faithful hackers, > > I decided to take up the challenge laid down here > http://fare.livejournal.com/169346.html and try to consolidate the Common > Lisp unit testing frameworks. I have written a framework that aims to > consolidate all the major features of all your frameworks mentioned in this > blog > http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html > . > > You can find it on Github https://github.com/tgutu/clunit. I also wrote a > blog on the development of the framework and reasons for it here > http://ml.sun.ac.za/2012/11/09/developing-a-unit-test-framework-part-1/if you are interested. > > I would very much appreciate it, if you could join me in this effort and > we all work together torwards making this project a success. > > Regards, > Tapiwa > > > Christopher Riesbeck > Home page: http://www.cs.northwestern.edu/~riesbeck > Calendar: http://calendar.yahoo.com/criesbeck > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rpgoldman at sift.info Sun Nov 11 19:40:34 2012 From: rpgoldman at sift.info (Robert Goldman) Date: Sun, 11 Nov 2012 13:40:34 -0600 Subject: [fiveam-devel] Unit Test Framework In-Reply-To: References: Message-ID: <509FFF32.4080505@sift.info> On 11/10/12 Nov 10 -11:41 AM, Tapiwa Gutu wrote: > Faithful hackers, > > I decided to take up the challenge laid down here > http://fare.livejournal.com/169346.html and try to consolidate the > Common Lisp unit testing frameworks.I have written a framework that aims > to consolidate all the major features of all your frameworks mentioned > in this blog > http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html. > > You can find it on Github https://github.com/tgutu/clunit. I also wrote > a blog on the development of the framework and reasons for it here > http://ml.sun.ac.za/2012/11/09/developing-a-unit-test-framework-part-1/ > if you are interested. > > I would very much appreciate it, if you could join me in this effort and > we all work together torwards making this project a success. > Executive summary: Hey! Not so fast! Longer: I'm in favor of Far?'s approach, but I feel like you are jumping in faster than I'd like to see us go. Far?'s article suggests these steps: 1. Pick your favorite problem domain 2. Identify all libraries that address it 3. Work with various library authors 4. Pick the most promising library 5. Declare it THE library It feels to me like you are skipping steps 3 and 4. I guess I'd like to see a wider discussion of this issue before jumping to #5. At SIFT, we ended up choosing FiveAM as THE library, for our purposes. Are you saying that cl-unit is the most promising? If so, why? Or are you saying that "None of the above" is the most promising, and we need to build YA framework from scratch? That is not at all what Far? was arguing for, as I read it. Certainly the write-up by Olin Shivers should be read as a push to fix an existing framework, rather than develop a new one. If you are proposing that we take cl-unit, what are you planning to do to provide a compatibility layer for people whose decisions are different for yours? Are you going to make it possible for people who use FiveAM to migrate their existing test suites without a lot of work? What's your plan to do this task that Far? identifies: "To fully solve the problem domain, we must identify all the libraries, and all their current and desired features, so that we may provide a 100% replacement to each and every one of these existing libraries, based on an architecture that makes all the future features possible." That sounds to me like the right thing to do now is try to come to consensus around a requirements document. I don't think that jumping into setting up a repo and slinging code is the right place to be now. Please do a little more consensus building and more "work with various library authors" and working with library users, so that you are not building YA 80% solution. Thanks! r From mjcurry at gmail.com Sun Nov 18 04:35:53 2012 From: mjcurry at gmail.com (Matthew Curry) Date: Sat, 17 Nov 2012 23:35:53 -0500 Subject: [fiveam-devel] small default formatting output patch Message-ID: Hi, Attached is a diff of check.lisp. I got annoyed with the (otherwise really really awesome) default 'is' macro output, which when testing large-ish s-exps, it all gets crammed to the right margin. I've also attached a before and after text file so you can see what I mean. If you can incorporate my change (or if you know a better way to do this) that would be great! -Matt -------------- next part -------------- CL-USER> (5am:run! 'kql::kql-compiling) ..f. Did 4 checks. Pass: 3 (75%) Skip: 0 ( 0%) Fail: 1 (25%) Failure Details: -------------------------------- ELIMINATES-A-REDUNDANT-NESTING []: (ELIMINATE-NESTING (PARSE "(NOT (apple AND orange)) OR (foo AND baz) AND NOT (quux AND bar)")) evaluated to (:OR (:NOT (:AND (:KEYWORD "apple") (:KEYWORD "orange"))) (:AND (:AND (:KEYWORD "foo") (:KEYWORD "baz")) (:NOT (:AND (:KEYWORD "quux") (:KEYWORD "bar"))))), which is not EQUALP to (:OR (:NOT (:AND (:KEYWORD "apple") (:KEYWORD "orange"))) (:AND (:KEYWORD "foo") (:KEYWORD "baz") (:NOT (:AND (:KEYWORD "quux") (:KEYWORD "bar"))))).. -------------------------------- -------------- next part -------------- A non-text attachment was scrubbed... Name: check-output.diff Type: application/octet-stream Size: 2092 bytes Desc: not available URL: -------------- next part -------------- CL-USER> (5am:run! 'kql::kql-compiling) ..f. Did 4 checks. Pass: 3 (75%) Skip: 0 ( 0%) Fail: 1 (25%) Failure Details: -------------------------------- ELIMINATES-A-REDUNDANT-NESTING []: (ELIMINATE-NESTING (PARSE "(NOT (apple AND orange)) OR (foo AND baz) AND NOT (quux AND bar)")) evaluated to (:OR (:NOT (:AND (:KEYWORD "apple") (:KEYWORD "orange"))) (:AND (:AND (:KEYWORD "foo") (:KEYWORD "baz")) (:NOT (:AND (:KEYWORD "quux") (:KEYWORD "bar"))))) which is not EQUALP to (:OR (:NOT (:AND (:KEYWORD "apple") (:KEYWORD "orange"))) (:AND (:KEYWORD "foo") (:KEYWORD "baz") (:NOT (:AND (:KEYWORD "quux") (:KEYWORD "bar"))))) .. -------------------------------- NIL From sionescu at cddr.org Sun Nov 18 04:45:10 2012 From: sionescu at cddr.org (Stelian Ionescu) Date: Sun, 18 Nov 2012 05:45:10 +0100 Subject: [fiveam-devel] patch to RESULTS-STATUS In-Reply-To: <504E3738.2020208@sift.info> References: <504E3738.2020208@sift.info> Message-ID: <1353213910.10772.4.camel@cathai.lan> On Mon, 2012-09-10 at 13:53 -0500, Robert Goldman wrote: > RESULTS-STATUS tells you if there are any non-"TEST-PASSED" results. > But it doesn't provide any way for you to get your hands on these > results. Originally, I thought that one could simply test for them, but > the results categories aren't actually exported from FiveAM, so this is > difficult. > > To make it easier to find "interesting" results, I modified > RESULTS-STATUS to return a second value. The first value is a boolean, > as before, telling you if the tests all passed. If they didn't, the > second value is now a list of the non-passing results. > > I believe this to be a benign change from the standpoint of backward > compatibility, since anyone just looking for a single return value will > get the same boolean value as before. > > It's possible that the actual value --- not interpreted as a boolean --- > will be different, since the new code uses REMOVE-IF instead of EVERY. > But if a caller takes information out of the results list in a function > whose documentation proclaims it to be boolean, I submit that the caller > is doing so at his/her own risk! > > Proposed patch is attached. Applied with some modifications, thanks :) -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From sionescu at cddr.org Sun Nov 18 04:45:29 2012 From: sionescu at cddr.org (Stelian Ionescu) Date: Sun, 18 Nov 2012 05:45:29 +0100 Subject: [fiveam-devel] small default formatting output patch In-Reply-To: References: Message-ID: <1353213929.10772.5.camel@cathai.lan> On Sat, 2012-11-17 at 23:35 -0500, Matthew Curry wrote: > Hi, > > Attached is a diff of check.lisp. I got annoyed with the (otherwise > really really awesome) default 'is' macro output, which when testing > large-ish s-exps, it all gets crammed to the right margin. > > I've also attached a before and after text file so you can see what I > mean. If you can incorporate my change (or if you know a better way > to do this) that would be great! Applied, thanks :) -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From rpgoldman at sift.info Sun Nov 25 00:52:06 2012 From: rpgoldman at sift.info (Robert Goldman) Date: Sat, 24 Nov 2012 18:52:06 -0600 Subject: [fiveam-devel] patch to RESULTS-STATUS In-Reply-To: <1353213910.10772.4.camel@cathai.lan> References: <504E3738.2020208@sift.info> <1353213910.10772.4.camel@cathai.lan> Message-ID: <50B16BB6.5020303@sift.info> On 11/17/12 Nov 17 -10:45 PM, Stelian Ionescu wrote: > On Mon, 2012-09-10 at 13:53 -0500, Robert Goldman wrote: >> RESULTS-STATUS tells you if there are any non-"TEST-PASSED" results. >> But it doesn't provide any way for you to get your hands on these >> results. Originally, I thought that one could simply test for them, but >> the results categories aren't actually exported from FiveAM, so this is >> difficult. >> >> To make it easier to find "interesting" results, I modified >> RESULTS-STATUS to return a second value. The first value is a boolean, >> as before, telling you if the tests all passed. If they didn't, the >> second value is now a list of the non-passing results. >> >> I believe this to be a benign change from the standpoint of backward >> compatibility, since anyone just looking for a single return value will >> get the same boolean value as before. >> >> It's possible that the actual value --- not interpreted as a boolean --- >> will be different, since the new code uses REMOVE-IF instead of EVERY. >> But if a caller takes information out of the results list in a function >> whose documentation proclaims it to be boolean, I submit that the caller >> is doing so at his/her own risk! >> >> Proposed patch is attached. > > Applied with some modifications, thanks :) > Question: should we bump the ASDF version lisp-expr to indicate that this return variable is available? I.e., move to version 1.1? Not a big deal, but it's nice to stick to the convention that a change in the API merits a .1 version bump.... Cheers, r From sionescu at cddr.org Sun Nov 25 00:54:27 2012 From: sionescu at cddr.org (Stelian Ionescu) Date: Sun, 25 Nov 2012 01:54:27 +0100 Subject: [fiveam-devel] patch to RESULTS-STATUS In-Reply-To: <50B16BB6.5020303@sift.info> References: <504E3738.2020208@sift.info> <1353213910.10772.4.camel@cathai.lan> <50B16BB6.5020303@sift.info> Message-ID: <1353804867.3735.1.camel@cathai.lan> On Sat, 2012-11-24 at 18:52 -0600, Robert Goldman wrote: > On 11/17/12 Nov 17 -10:45 PM, Stelian Ionescu wrote: > > On Mon, 2012-09-10 at 13:53 -0500, Robert Goldman wrote: > >> RESULTS-STATUS tells you if there are any non-"TEST-PASSED" results. > >> But it doesn't provide any way for you to get your hands on these > >> results. Originally, I thought that one could simply test for them, but > >> the results categories aren't actually exported from FiveAM, so this is > >> difficult. > >> > >> To make it easier to find "interesting" results, I modified > >> RESULTS-STATUS to return a second value. The first value is a boolean, > >> as before, telling you if the tests all passed. If they didn't, the > >> second value is now a list of the non-passing results. > >> > >> I believe this to be a benign change from the standpoint of backward > >> compatibility, since anyone just looking for a single return value will > >> get the same boolean value as before. > >> > >> It's possible that the actual value --- not interpreted as a boolean --- > >> will be different, since the new code uses REMOVE-IF instead of EVERY. > >> But if a caller takes information out of the results list in a function > >> whose documentation proclaims it to be boolean, I submit that the caller > >> is doing so at his/her own risk! > >> > >> Proposed patch is attached. > > > > Applied with some modifications, thanks :) > > > Question: should we bump the ASDF version lisp-expr to indicate that > this return variable is available? I.e., move to version 1.1? Yes, I'm going to make a new release soon. -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From mb at bese.it Fri Nov 30 18:42:43 2012 From: mb at bese.it (Marco Baringer) Date: Fri, 30 Nov 2012 19:42:43 +0100 Subject: [fiveam-devel] Unit Test Framework References: Message-ID: Tapiwa Gutu writes: > Faithful hackers, > I decided to take up the challenge laid down here > http://fare.livejournal.com/169346.html and try to consolidate the > Common Lisp unit testing frameworks. I have written a framework that > aims to consolidate all the major features of all your frameworks > mentioned in this blog > http://aperiodic.net/phil/archives/Geekery/notes-on-lisp-testing-frameworks.html. > You can find it on Github https://github.com/tgutu/clunit. I also > wrote a blog on the development of the framework and reasons for it > here http://ml.sun.ac.za/2012/11/09/ > developing-a-unit-test-framework-part-1/ if you are interested. i mean no disrespect to your effort, but i think it'd be better to improve one of the existing test frameworks (and i'm partial to fiveam) than to try and start from scratch. now, if fiveam really doesn't (or can't) do what you want/need it to do, then i'd love to talk about what that is. if you end up writing your own test framework, well, i can totally understand that too. i also think that this problem isn't nearly as bad as it seems, fiveam already does many of the things the other test suites do, but the documentation is pretty horrible (so nobody knows this). did you know about 5am:debug! ? (i didn't. until a few days ago). so, here's an attempt at improving fiveam's documentation (this is 60 or 70% complete): http://bese.it/fiveam/manual.html i'd love someone to have a quick look at it and tell me if it makes sense (i know fiveam too well to be a good judge) or if it needs to be fundamentally changed somehow. it all lives in this repo: https://github.com/segv/fiveam which, other than the docs contains a few minor cleanups and one new feature (no pull request yet). ps - you know what would be really cool? source location of the failing test (if i have 20 is checks in a test and one of them fails it'd be nice to know the filename and line number of those checks which failed. i'd settle form something that was #+(or sbcl ccl)...) pps - to get stefil's debugger when a test fails in fiveam, just do (setf 5am:*debug-on-failure* t), if you want low over head just ignore suites (the default suite, nil, always exists) and use (5am:run!) to run all the currently defined tests, if you want ever lower overhead then set 5am:*run-test-when-defined* and you can skip the call to run!. -- -marco From rpgoldman at sift.info Fri Nov 30 19:35:42 2012 From: rpgoldman at sift.info (Robert Goldman) Date: Fri, 30 Nov 2012 13:35:42 -0600 Subject: [fiveam-devel] Unit Test Framework In-Reply-To: References: Message-ID: <50B90A8E.1090607@sift.info> On 11/30/12 Nov 30 -12:42 PM, Marco Baringer wrote: > > ps - you know what would be really cool? source location of the failing test > (if i have 20 is checks in a test and one of them fails it'd be nice to > know the filename and line number of those checks which failed. i'd > settle form something that was #+(or sbcl ccl)...) I am familiar with how to set this up for ACL. If someone will create a framework for doing this (e.g., figure out how we decide which test to look for if the same name is used in two different suites, etc.), then I would be happy to contribute code for ACL. I suspect that a lot of this will be implementation-specific. We can all support our favorites to get a generally useful implementation.