[cffi-devel] how to treat expected failures in tests

[Luís Oliveira]

On Tue, Mar 13, 2012 at 9:31 PM, Anton Vodonosov <avodonosov at yandex.ru> wrote:
> 13.03.2012, 03:55, "Luís Oliveira" <luismbo at gmail.com>:
>> However, the results are a bit depressing. So many fails. :-)
>
> Not so many - 20 failed tests in total.The same failures repeat on different lisps.

Exactly. There are not that many failures, but the picture drawn by
the grid makes the current status look grimmer than it is. :-)



>> Perhaps you could show the ration of failed to total tests
> Can not do it now - we do not collect total number of tests in test suite.
> In my opinion it is not very useful information (comparing to the list
> of failed tests, which I hope will allow us to prevent new failures in future)

Agreed. It's not very important.


> Lisp compilation failures are present in logs, but in this case we have C library compilation
> error. This may be seen by component description in the ASDF error message:
> #<C-TEST-LIB "cffi-tests" "tests" "libtest">

Ah, my bad. While I'm making feature requests, perhaps a backtrace
would be useful, though? :-)


> In cffi-tests.asd the
>
> (defmethod perform ((o compile-op) (c c-test-lib))
>
> does not try to run "make" on windows,
>
>  #-windows
>  (unless (zerop (run-shell-command
>                  "cd ~A; make"
>
> and on windows the only error signaled in absense of the native library is
> cffi:load-foreign-library-error. I rely on it when detecting :no-resource.

Should we assume that 'make' is available on windows? Is that a common
setup for cygwin/mingw? What about other toolsets such as Microsoft's?
Do they ship with make?

Cheers,

-- 
Luís Oliveira
http://r42.eu/~luis/