[cffi-devel] how to treat expected failures in tests

[Robert Goldman]

On 1/10/12 Jan 10 -5:51 PM, Jeff Cunningham wrote:
> On 01/10/2012 04:00 AM, Anton Vodonosov wrote:
>> Hello.
>>
>> For my common lisp testing project I aggregate results of a library
>> test suite into single value - ok/fail.
>>
>> I just tested ECL and have the following output from CFFI test suite:
>>
>> 4 out of 228 total tests failed: DEFCFUN.NOOP, CALLBACKS.BFF.1,
>> STRING.ENCODING.UTF-16.BASIC,
>>     STRING.ENCODINGS.ALL.BASIC.
>> No unexpected failures.
>>
>> What meaning do you put into the term "expected failure"? Does it mean
>> the library is buggy, but
>> these bugs are known? Or it means that some non-required features are
>> absent, but the
>> library in general OK?
>>
>> I am interested in both short answer - as a library author, how do you
>> think CFFI test suite
>> should be marked if only expected failures present - OK or FAIL?
>>
>> And also I am curious in this concrete example, what these 4 failures
>> mean for CFFI on ECL?
>>
> 
> In a testing scenario, "expected failure" to me means the test was
> designed to fail and it did. Usually, these are set up to test error
> handling. In a large testing environment we would periodically toss in a
> couple tests we knew would fail - one thing it tests it that the people
> running the tests are actually running them. If they didn't come back
> with these failures we knew there was a breakdown in the process.

That is not how the term is used in the CFFI tests, or in most of the
unit testing libraries.  Those libraries offer facilities for a test to
PASS if and only if the code raises an error (or an error of a
particular type).

We should not conflate the notion of "raise an error" (or, for that
matter, "return NIL") with the notion of "fail a test."  Tests are about
verifying expectations, and we can have valid, test-passing expectations
that code will raise an error.

best,
r