[cffi-devel] how to treat expected failures in tests

[Anton Vodonosov]

Hello, thanks for the feedback

13.03.2012, 03:55, "Luís Oliveira" <luismbo at gmail.com>:
>
> However, the results are a bit depressing. So many fails. :-) 

Not so many - 20 failed tests in total.The same failures repeat on different lisps. 

Here is the breakdown by failures, for the quicklisp 2012-02-08 and the lisp 
implementations we tested:

"callbacks.bff.1" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-macosx-x64"
                      "ccl-1.7-f95-win-x86" "ccl-1.8-f95-macosx-x64"
                      "cmu-20c_release-20c__20c_unicode_-linux-x86"
                      "ecl-11.1.1-606449eb-linux-x86")
"callbacks.bff.2" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-macosx-x64"
                      "ccl-1.7-f95-win-x86" "ccl-1.8-f95-macosx-x64"
                      "cmu-20c_release-20c__20c_unicode_-linux-x86")
"callbacks.uninterned" => ("ecl-11.1.1-606449eb-linux-x86")
"defcfun.bff.2" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-win-x86"
                    "clisp-2.49-unix" "clisp-2.49-win"
                    "cmu-20c_release-20c__20c_unicode_-linux-x86"
                    "sbcl-1.0.54-linux-x86")
"defcfun.noop" => ("ccl-1.7-f95-linux-x86" "ccl-1.7-f95-macosx-x64"
                   "ccl-1.7-f95-win-x86" "ccl-1.8-f95-macosx-x64"
                   "ecl-11.1.1-606449eb-linux-x86")
"defcfun.stdcall.1" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win")
"defcfun.undefined" => ("cmu-20c_release-20c__20c_unicode_-linux-x86")
"defcfun.varargs.double" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win")
"defcfun.varargs.float" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win")
"foreign-symbol-pointer.1" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win")
"funcall.stdcall.1" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win")
"funcall.varargs.double" => ("ccl-1.7-f95-win-x86" "clisp-2.49-win")
"string.encoding.utf-16.basic" => ("ccl-1.7-f95-linux-x86"
                                   "ccl-1.7-f95-macosx-x64"
                                   "ccl-1.7-f95-win-x86"
                                   "ccl-1.8-f95-macosx-x64"
                                   "clisp-2.49-unix" "clisp-2.49-win"
                                   "cmu-20c_release-20c__20c_unicode_-linux-x86"
                                   "ecl-11.1.1-606449eb-linux-x86"
                                   "sbcl-1.0.49-linux-amd64"
                                   "sbcl-1.0.54-linux-x86"
                                   "sbcl-1.0.54.45-a2bef14-macosx-x64")
"string.encodings.all.basic" => ("ccl-1.7-f95-linux-x86"
                                 "ccl-1.7-f95-macosx-x64"
                                 "ccl-1.7-f95-win-x86"
                                 "ccl-1.8-f95-macosx-x64"
                                 "clisp-2.49-unix" "clisp-2.49-win"
                                 "cmu-20c_release-20c__20c_unicode_-linux-x86"
                                 "ecl-11.1.1-606449eb-linux-x86"
                                 "sbcl-1.0.49-linux-amd64"
                                 "sbcl-1.0.54-linux-x86"
                                 "sbcl-1.0.54.45-a2bef14-macosx-x64")
"struct.alignment.3" => ("ccl-1.7-f95-linux-x86")
"struct.alignment.4" => ("ccl-1.7-f95-linux-x86")
"struct.alignment.5" => ("ccl-1.7-f95-linux-x86")
"struct.alignment.6" => ("ccl-1.7-f95-linux-x86")
"struct.alignment.7" => ("ccl-1.7-f95-linux-x86")
"struct.alignment.8" => ("ccl-1.7-f95-linux-x86")

> Perhaps you could show the ration of failed to total tests 
Can not do it now - we do not collect total number of tests in test suite.
In my opinion it is not very useful information (comparing to the list
of failed tests, which I hope will allow us to prevent new failures in future)

> and maybe show know-fail/unexpected-ok in yellow/orange rather than red.
this is doable - added a TODO item

> I wonder if compilation errors could be printed. Error messages like
> this are not very helpful:
> <http://cl-test-grid.appspot.com/blob?key=AMIfv97suboJpeei-uBWzlkqcR7CTlyh0Izhvi7u_29HNBgu80ScYf0Mj6zWPjgbsosA-F0Q12HP8o9S5zhsEelTfss8_3C7sjgcuG_q_grR-jMfXPLLRzu6CNytLoNk23rwqlQ6AsajxTRYFubFbz3iBWl5uo8iZQ>.
>

Lisp compilation failures are present in logs, but in this case we have C library compilation
error. This may be seen by component description in the ASDF error message:
#<C-TEST-LIB "cffi-tests" "tests" "libtest">

Actually, I try to represent this case - absence of the native library and therefore inability to run tests
- by status :no-resource. But this failure is signaled differently on windows and linux,
I didn't noticed this and in result in this linux test run was unable to recognize
the failure as :no-resource.

Some details.

In cffi-tests.asd the 

(defmethod perform ((o compile-op) (c c-test-lib))

does not try to run "make" on windows, 

  #-windows
  (unless (zerop (run-shell-command
                  "cd ~A; make"

and on windows the only error signaled in absense of the native library is
cffi:load-foreign-library-error. I rely on it when detecting :no-resource.

Added TODO item to detect is as :no-resouce on linux too when compile-op
for the library fails.

Best regards,
- Anton