From Dr.Alain.Picard at gmail.com Tue Jan 4 05:20:20 2011 From: Dr.Alain.Picard at gmail.com (Alain Picard) Date: Tue, 04 Jan 2011 16:20:20 +1100 Subject: [elephant-devel] Missing method in elephant. Message-ID: <87ei8ttfcb.fsf@gmail.com> I think I discovered a (rather serious!) bug in elephant. Calling MAP-BTREE with both the :FROM-END and :END arguments on a BDB store controller leads to this condition: There is no applicable method for the generic function # when called with arguments (#). [Condition of type SIMPLE-ERROR] This can be replicated quite simply with the included example test code. FWIW, changing the definition of MAP-BTREE-FROM-END from (defun map-btree-from-end (fn btree start end collect) (with-map-wrapper (fn btree collect cur) (iterate-map-btree :start (if end (with-cursor-values (cursor-set-range cur end) (cond ((and exists? (lisp-compare-equal skey end)) (cursor-next-nodup cur) (cursor-prev cur)) (t (cursor-prev cur)))) (cursor-last cur)) :continue (or (null start) (lisp-compare>= key start)) :step (cursor-prev cur)))) to (defun map-btree-from-end (fn btree start end collect) (with-map-wrapper (fn btree collect cur) (iterate-map-btree :start (if end (with-cursor-values (cursor-set-range cur end) (cond ((and exists? (lisp-compare-equal skey end)) (cursor-next cur) (cursor-prev cur)) (t (cursor-prev cur)))) (cursor-last cur)) :continue (or (null start) (lisp-compare>= key start)) :step (cursor-prev cur)))) "fixes" the problem, in my simple test case, but I don't understand that function deeply enough to be convinced that this is a correct fix in all circumstances. If it's only ever called on btrees, which can contain no duplicates, it's presumably OK, but if it can be called on indexed trees, then it would not be OK, and we would need a definition of CURSOR-NEXT-NODUP on regular cursors. --alain ================================================================ ;;;; -*- Mode: lisp; Package: elephant; Syntax: Common-lisp -*- ;; ;; Copyright (C) 2011 TechNo Wait. ;; All rights reserved. ;; ;; Author: Alain Picard ;; ;;;; Code: (in-package :elephant) (defun *test-tree* () (or (get-from-root :test-tree) (add-to-root :test-tree (make-btree)))) (defun ap/prepare-1 () (dotimes (i 10) (setf (get-value i (*test-tree*)) (format nil "~:R" i)))) (defun ap-print2 (a b) (print (list a b))) (defun ap/test-1 () (map-btree 'ap-print2 (*test-tree*)) (print "ap/test-1 OK")) (defun ap/test-2 () (map-btree 'ap-print2 (*test-tree*) :from-end t) (print "ap/test-2 OK")) (defun ap/test-3 () (map-btree 'ap-print2 (*test-tree*) :from-end t :end 7) (print "ap/test-3 OK")) (defun ap/run-tests () (ap/test-1) (ap/test-2) (ap/test-3)) #+foo (eval-when (:load-toplevel :execute) (ap/prepare-1) (ap/run-tests)) ;;; BUG2.LISP ends here ================================================================ -- Please read about why Top Posting is evil at: http://en.wikipedia.org/wiki/Top-posting and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html From eslick at media.mit.edu Tue Jan 4 05:44:47 2011 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 3 Jan 2011 21:44:47 -0800 Subject: [elephant-devel] Elephant/BDB problems on recent 64-bit Linux systems In-Reply-To: <20101216.073007.918591125461212580.petter@pangea.home.gustad.com> References: <20101216.073007.918591125461212580.petter@pangea.home.gustad.com> Message-ID: <1F464311-2A6C-4CF5-885F-6F77AEA32171@media.mit.edu> Hi Petter, Any progress since you wrote this? Tracking down these kinds of errors is nasty - I had to do a bunch of it a few years ago when I was changing the serialization code and messing around with the serializer's bits and bytes. I'm afraid I don't have a good idea here. Was your problem in clojure also involving Berkeley DB? Cheers, Ian On Dec 15, 2010, at 10:30 PM, Petter Gustad wrote: > > Hi, > > I'm experiencing problems when I try to run Elephant with the Berkeley > DB backend on certain recent 64-bit Gentoo systems. I've enclosed a > sample script which will download and build SBCL, Elephant, and then > run a small test. > > > System A: Core-2 Duo somewhat older Gentoo. > /usr/bin/db4.7_deadlock -V > Berkeley DB 4.7.25: (2010-08-25) > gcc 4.1.2 > > System B: Core-2 Duo recent Gentoo installation > /usr/bin/db4.7_deadlock -V > Berkeley DB 4.7.25: (2010-11-18) > gcc 4.4.4 > > System C: Atom recent Gentoo installation > /usr/bin/db4.7_deadlock -V > Berkeley DB 4.7.25: (2010-11-18) > gcc 4.4.4 > > System D: Core-2 Quad Gentoo (somewhere between A and B) > /usr/bin/db4.7_deadlock -V > Berkeley DB 4.7.25: (2010-12-14) > gcc 4.3.2 > > > A: OK (*** GOT KEY ("Zappa rules!") IN SECOND INVOCATION) > B: Unhandled memory fault at #x7D0. > C: Unhandled memory fault at #x7D0. > D: *** GOT KEY ("Zappa rules!") IN SECOND INVOCATION > > Then B gcc was downgraded to 4.1.2. The Berkeley db was then > recompiled, all .fasl files were deleted, and the result was still: > > Unhandled memory fault at #x7D0. > > This occurs at: > (open-store cl-user::+ele-storage+) > > I'm also getting a similar failure with Clojoure so I don't think it's > releated to SBCL. > > Have anybody else experienced these kinds of problems, or have any > clues as what the problem might be? > > //Petter > P.S. The script will install QuickLisp if it's not installed > > #!/bin/sh > > # http://prdownloads.sourceforge.net/sbcl/sbcl-1.0.44-x86-64-linux-binary.tar.bz2 > # http://prdownloads.sourceforge.net/sbcl/sbcl-1.0.44-x86-linux-binary.tar.bz2 > # http://downloads.sourceforge.net/project/sbcl/sbcl/1.0.44/sbcl-1.0.44-source.tar.bz2 > > unset SBCL_HOME > unset SBCL_SOURCE_ROOT > unset CDPATH > LC_CTYPE=en_US.UTF-8 > export LC_CTYPE > > TARGET_DIR=/tmp/ele-test-dir > > mkdir -p ${TARGET_DIR} > > cd ${TARGET_DIR} > > wget http://prdownloads.sourceforge.net/sbcl/sbcl-1.0.44-x86-64-linux-binary.tar.bz2 > tar xvfj sbcl-1.0.44-x86-64-linux-binary.tar.bz2 > wget http://downloads.sourceforge.net/project/sbcl/sbcl/1.0.44/sbcl-1.0.44-source.tar.bz2 > tar xvfj sbcl-1.0.44-source.tar.bz2 > > mkdir root > cd sbcl-1.0.44 > > > OLD_PATH="$PATH" > PATH=${TARGET_DIR}/sbcl-1.0.44-x86-64-linux/src/runtime:$PATH > export PATH > SBCL_HOME=${TARGET_DIR}/sbcl-1.0.44-x86-64-linux/output > export SBCL_HOME > > > sh make.sh --prefix=${TARGET_DIR}/root > > unset SBCL_HOME > INSTALL_ROOT=${TARGET_DIR}/root sh install.sh > > PATH=${TARGET_DIR}/root/bin:${OLD_PATH} > export PATH > > cd ${TARGET_DIR} > > darcs get http://www.common-lisp.net/project/elephant/darcs/elephant-1.0 > > cat > elephant-1.0/my-config.sexp < #+(and (or sbcl allegro openmcl lispworks) (not (or mswindows windows)) (not (or macosx darwin))) > ((:compiler . :gcc) > (:berkeley-db-version . "4.7") > (:berkeley-db-include-dir . "/usr/include/") > (:berkeley-db-lib-dir . "/usr/lib/") > (:berkeley-db-lib . "/usr/lib/libdb-4.7.so") > (:berkeley-db-deadlock . "/usr/bin/db4.7_deadlock") > (:berkeley-db-cachesize . 20971520) > (:berkeley-db-map-degree2 . t) > (:clsql-lib-paths . nil) > (:prebuilt-libraries . nil)) > EOF > > echo "(defconstant +ele-storage+ '(:BDB \"${TARGET_DIR}/ele-storage\"))" > store-name.lisp > > mkdir ele-storage > > if test ! -e $HOME/quicklisp/setup.lisp > then > wget http://quickstart.quicklisp.org/quicklisp.lisp > sbcl --no-userinit --no-sysinit --load quicklisp.lisp \ > --eval '(quicklisp-quickstart:install")' \ > --eval '(sb-ext:quit)' > fi > > sbcl --no-userinit --no-sysinit --load $HOME/quicklisp/setup.lisp --load store-name.lisp < (in-package #:cl-user) > (asdf:initialize-source-registry '(:source-registry :ignore-inherited-configuration)) > (ql:quickload "bordeaux-threads") > (ql:quickload "cl-base64") > (ql:quickload "uffi") > (push (make-pathname :directory '(:relative "elephant-1.0")) asdf:*central-registry*) > (asdf:oos 'asdf:load-op :elephant) > (in-package :elephant) > (open-store cl-user::+ele-storage+) > (add-to-root "somekey" "Zappa rules!") > (when (equal (get-from-root "somekey") "Zappa rules!") > (format t "*** GOT KEY (~s) AFTER IT WAS WRITTEN~%" (get-from-root "somekey"))) > (sb-ext:quit) > EOF > > sbcl --no-userinit --no-sysinit --load $HOME/quicklisp/setup.lisp --load store-name.lisp < (in-package #:cl-user) > (asdf:initialize-source-registry '(:source-registry :ignore-inherited-configuration)) > (ql:quickload "bordeaux-threads") > (ql:quickload "cl-base64") > (ql:quickload "uffi") > (push (make-pathname :directory '(:relative "elephant-1.0")) asdf:*central-registry*) > (asdf:oos 'asdf:load-op :elephant) > (in-package :elephant) > (open-store cl-user::+ele-storage+) > (when (equal (get-from-root "somekey") "Zappa rules!") > (format t "*** GOT KEY (~s) IN SECOND INVOCATION~%" (get-from-root "somekey"))) > (sb-ext:quit) > EOF > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From elephant at gustad.com Tue Jan 4 08:40:46 2011 From: elephant at gustad.com (Petter Gustad) Date: Tue, 04 Jan 2011 09:40:46 +0100 (CET) Subject: [elephant-devel] Elephant/BDB problems on recent 64-bit Linux systems In-Reply-To: <1F464311-2A6C-4CF5-885F-6F77AEA32171@media.mit.edu> References: <20101216.073007.918591125461212580.petter@pangea.home.gustad.com> <1F464311-2A6C-4CF5-885F-6F77AEA32171@media.mit.edu> Message-ID: <20110104.094046.644644224815183338.petter@pangea.home.gustad.com> From: Ian Eslick Subject: Re: [elephant-devel] Elephant/BDB problems on recent 64-bit Linux systems Date: Mon, 3 Jan 2011 21:44:47 -0800 > Any progress since you wrote this? Unfortunately not. I haven't had much time during the last few weeks. > Tracking down these kinds of errors is nasty - I had to do a bunch > of it a few years ago when I was changing the serialization code and > messing around with the serializer's bits and bytes. I'm afraid I > don't have a good idea here. I see. The problem might as well be in the area of UFFI/BDB. I should also run some C based BDB tests to check that by BDB libraries are sane as well. > Was your problem in clojure also involving Berkeley DB? Yes. I ran the same code using Clojure rather than SBCL and got some corruption warning as well. Hence I assume that this problem is not related to SBCL. //Petter From elephant at gustad.com Tue Jan 4 12:21:28 2011 From: elephant at gustad.com (Petter Gustad) Date: Tue, 04 Jan 2011 13:21:28 +0100 (CET) Subject: [elephant-devel] Elephant/BDB problems on recent 64-bit Linux systems In-Reply-To: <20110104.094046.644644224815183338.petter@pangea.home.gustad.com> References: <20101216.073007.918591125461212580.petter@pangea.home.gustad.com> <1F464311-2A6C-4CF5-885F-6F77AEA32171@media.mit.edu> <20110104.094046.644644224815183338.petter@pangea.home.gustad.com> Message-ID: <20110104.132128.37709278787060538.petter@pangea.home.gustad.com> From: Petter Gustad Subject: Re: [elephant-devel] Elephant/BDB problems on recent 64-bit Linux systems Date: Tue, 04 Jan 2011 09:40:46 +0100 (CET) >> Was your problem in clojure also involving Berkeley DB? > > Yes. I ran the same code using Clojure rather than SBCL and got some > corruption warning as well. Hence I assume that this problem is not > related to SBCL. I'm sorry about the confusion. It was Clozure/CCL and not Clojure: (format nil "~A -- ~A" (lisp-implementation-type) (lisp-implementation-version)) "Clozure Common Lisp -- Version 1.5-r13651 (LinuxX8664)" I'm typing Clojure (which I was playing a bit with some time ago), but I'm thinking about CCL which I was using for my Elephant testing. //Petter From elephant at gustad.com Tue Jan 4 23:00:25 2011 From: elephant at gustad.com (Petter Gustad) Date: Wed, 05 Jan 2011 00:00:25 +0100 (CET) Subject: [elephant-devel] Elephant/BDB problems on recent 64-bit Linux systems In-Reply-To: <20110104.132128.37709278787060538.petter@pangea.home.gustad.com> References: <1F464311-2A6C-4CF5-885F-6F77AEA32171@media.mit.edu> <20110104.094046.644644224815183338.petter@pangea.home.gustad.com> <20110104.132128.37709278787060538.petter@pangea.home.gustad.com> Message-ID: <20110105.000025.472177616931667360.petter@pangea.home.gustad.com> From: Petter Gustad Subject: Re: [elephant-devel] Elephant/BDB problems on recent 64-bit Linux systems Date: Tue, 04 Jan 2011 13:21:28 +0100 (CET) This was a classical my-config.sexp error... I had my include directory specified as: (:berkeley-db-include-dir . "/usr/include/") But on older Gentoo system /usr/include/db.h is a symbolic link to db4.7/db.h whereas on more recent systems it is a symbolic link to db4.8/db.h. After specifying (:berkeley-db-include-dir . "/usr/include/db4.7/") it worked. The reason for the crash is that offset of env->set_lk_max_locks does not point to the right offset since I was using the a header file which did not correspond to the binary library. Ian, thank you for pointing me in the right direction. //Petter P.S. I will repeat the symtoms here in case somebody else runs into a similar problem. Hopefully you get a hit on this message if you search for any of the errors using Google. I could reproduce the problem by: (asdf:initialize-source-registry '(:source-registry :ignore-inherited-configuration)) (ql:quickload "bordeaux-threads") (ql:quickload "cl-base64") (ql:quickload "uffi") (ql:quickload "fiveam") (push (make-pathname :directory '(:relative "elephant-1.0")) asdf:*central-registry*) (asdf:oos 'asdf:load-op :elephant) (asdf:oos 'asdf:load-op :elephant-tests) ;; (in-package :elephant-tests) ;; (setf *default-spec* *testbdb-spec*) ;; (do-backend-tests) (asdf:oos 'asdf:load-op :ele-bdb) (in-package :db-bdb) (db-env-set-max-locks (db-env-create) 2000) Running SBCL 1.0.44 gives me a memory corruption error: Unhandled memory fault at #x7D0. Looking at the stack I see: 0: (SB-SYS:MEMORY-FAULT-ERROR) 1: ("foreign function: call_into_lisp") 2: ("foreign function: post_signal_tramp") 3: ("foreign function: strlen") 4: (DB-BDB::DB-ENV-SET-MAX-LOCKS # #) Using CCL I get a similar fault: exception in foreign context Exception occurred while executing foreign code at strlen + 18 ? for help [24895] Clozure CL kernel debugger: current thread: tcr = 0x7fbbe11ce570, native thread ID = 0x6140, interrupts enabled (#x00007FBBE0F697E0) #x00003020016FA5DC : # + 173 (#x00007FBBE0F69800) #x00003020016FA27C : # + 45 (#x00007FBBE0F69820) #x00003020017A96FC : # + 797 (#x00007FBBE0F69918) #x0000300000216E3C : # + 893 (#x00007FBBE0F69990) #x00003000001FAE1C : # + 269 (#x00007FBBE0F699F8) #x00003020011A9EF4 : # + 325