From aditya.siram at gmail.com Mon Nov 3 17:42:07 2008 From: aditya.siram at gmail.com (aditya siram) Date: Mon, 3 Nov 2008 11:42:07 -0600 Subject: [elephant-devel] Issues installing Elephant 0.9 Message-ID: <594f78210811030942k3adc97a1o26431e80134147b2@mail.gmail.com> I am trying to install elephant using asdf but I get a fatal error. Here is my-config.sexp: ;; Linux defaults #+(and (or sbcl allegro openmcl lispworks) (not (or mswindows windows)) (not (or macosx darwin))) ((:compiler . :gcc) (:berkeley-db-include-dir . "/usr/include/") (:berkeley-db-lib-dir . "/usr/lib/") (:berkeley-db-lib . "/usr/lib/libdb-4.5.so") (:berkeley-db-deadlock . "/usr/bin/db_deadlock") (:berkeley-db-cachesize . 20971520) (:berkeley-db-map-degree2 . t) (:clsql-lib-paths . nil) (:prebuilt-libraries . nil)) and here is the error: ; file: /home/deech/.sbcl/site/elephant/src/db-clsql/sql-controller.lisp ; in: DEFUN SQLITE3-HARMLESS-READ ; (HANDLER-CASE (CLSQL-SYS:QUERY "select count(*) from keyvalue") ; ((CLSQL-SYS:SQL-DATABASE-ERROR NIL NIL))) ; ; caught ERROR: ; (in macroexpansion of (HANDLER-CASE (CLSQL-SYS:QUERY "select count(*) from keyvalue") ((CLSQL-SYS:SQL-DATABASE-ERROR NIL NIL)))) ; (hint: For more precise location, try *BREAK-ON-SIGNALS*.) ; error while parsing arguments to DESTRUCTURING-BIND: ; invalid number of elements in ; ((CLSQL-SYS:SQL-DATABASE-ERROR NIL NIL)) ; to satisfy lambda list ; (TYPE SB-IMPL::LL &BODY SB-IMPL::BODY): ; at least 2 expected, but 1 found ; (LET ((DB-CLSQL::CON (DB-CLSQL::CONTROLLER-DB DB-CLSQL::SC))) ; (IF (EQUAL (CAR (CADR #)) :SQLITE3) ; (HANDLER-CASE (CLSQL-SYS:QUERY "select count(*) from keyvalue") ; ((CLSQL-SYS:SQL-DATABASE-ERROR NIL NIL))))) Appreciate the help ... deech From aditya.siram at gmail.com Mon Nov 3 19:40:13 2008 From: aditya.siram at gmail.com (aditya siram) Date: Mon, 3 Nov 2008 13:40:13 -0600 Subject: [elephant-devel] Issues installing Elephant 0.9 In-Reply-To: <594f78210811030942k3adc97a1o26431e80134147b2@mail.gmail.com> References: <594f78210811030942k3adc97a1o26431e80134147b2@mail.gmail.com> Message-ID: <594f78210811031140p347dd422v9db5d66af3d9a4b9@mail.gmail.com> I have solved the problem by making changes to the following function in ../src /db-clsql/sql-controller.lisp. (defun sqlite3-harmless-read (sc) (let ((con (controller-db sc))) (if (equal (car (cadr (controller-spec sc))) :sqlite3) (handler-case (clsql:query "select count(*) from keyvalue") ((clsql:sql-database-error () nil)) ;;<< wrote: > I am trying to install elephant using asdf but I get a fatal error. > Here is my-config.sexp: > ;; Linux defaults > #+(and (or sbcl allegro openmcl lispworks) (not (or mswindows > windows)) (not (or macosx darwin))) > ((:compiler . :gcc) > (:berkeley-db-include-dir . "/usr/include/") > (:berkeley-db-lib-dir . "/usr/lib/") > (:berkeley-db-lib . "/usr/lib/libdb-4.5.so") > (:berkeley-db-deadlock . "/usr/bin/db_deadlock") > (:berkeley-db-cachesize . 20971520) > (:berkeley-db-map-degree2 . t) > (:clsql-lib-paths . nil) > (:prebuilt-libraries . nil)) > > and here is the error: > ; file: /home/deech/.sbcl/site/elephant/src/db-clsql/sql-controller.lisp > ; in: DEFUN SQLITE3-HARMLESS-READ > ; (HANDLER-CASE (CLSQL-SYS:QUERY "select count(*) from keyvalue") > ; ((CLSQL-SYS:SQL-DATABASE-ERROR NIL NIL))) > ; > ; caught ERROR: > ; (in macroexpansion of (HANDLER-CASE (CLSQL-SYS:QUERY "select > count(*) from keyvalue") ((CLSQL-SYS:SQL-DATABASE-ERROR NIL NIL)))) > ; (hint: For more precise location, try *BREAK-ON-SIGNALS*.) > ; error while parsing arguments to DESTRUCTURING-BIND: > ; invalid number of elements in > ; ((CLSQL-SYS:SQL-DATABASE-ERROR NIL NIL)) > ; to satisfy lambda list > ; (TYPE SB-IMPL::LL &BODY SB-IMPL::BODY): > ; at least 2 expected, but 1 found > > ; (LET ((DB-CLSQL::CON (DB-CLSQL::CONTROLLER-DB DB-CLSQL::SC))) > ; (IF (EQUAL (CAR (CADR #)) :SQLITE3) > ; (HANDLER-CASE (CLSQL-SYS:QUERY "select count(*) from keyvalue") > ; ((CLSQL-SYS:SQL-DATABASE-ERROR NIL NIL))))) > > Appreciate the help ... > deech > From aditya.siram at gmail.com Mon Nov 3 20:29:24 2008 From: aditya.siram at gmail.com (aditya siram) Date: Mon, 3 Nov 2008 14:29:24 -0600 Subject: [elephant-devel] Issues adding to root Message-ID: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> I am having issues with the serializer and sbcl's alien types. A session transcript with a trace on ELEPHANT-SERIALIZER2::SERIALIZE-TO-UTF8 is below: CL-USER> (elephant:open-store '(:CLSQL (:SQLITE3 "elephantTest.db"))) # CL-USER> (elephant:add-to-root "a" "1") 0: (ELEPHANT-SERIALIZER2::SERIALIZE-TO-UTF8 "1" #S(ELEPHANT-MEMUTIL:BUFFER-STREAM :BUFFER # :SIZE 0 :POSITION 0 :LENGTH 10)) The value # is not of type SYSTEM-AREA-POINTER. [Condition of type TYPE-ERROR] Thanks for your help ... deech From aditya.siram at gmail.com Mon Nov 3 20:30:20 2008 From: aditya.siram at gmail.com (aditya siram) Date: Mon, 3 Nov 2008 14:30:20 -0600 Subject: [elephant-devel] Issues adding to root In-Reply-To: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> References: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> Message-ID: <594f78210811031230i9c4fab6lb4e920515e16f4c@mail.gmail.com> I am using SBCL 1.0.21. Thanks ... deech On Mon, Nov 3, 2008 at 2:29 PM, aditya siram wrote: > I am having issues with the serializer and sbcl's alien types. A > session transcript with a trace on > ELEPHANT-SERIALIZER2::SERIALIZE-TO-UTF8 is below: > > CL-USER> (elephant:open-store '(:CLSQL (:SQLITE3 "elephantTest.db"))) > # > CL-USER> (elephant:add-to-root "a" "1") > 0: (ELEPHANT-SERIALIZER2::SERIALIZE-TO-UTF8 "1" > #S(ELEPHANT-MEMUTIL:BUFFER-STREAM > :BUFFER > # > (UNSIGNED > > 8))> > :SIZE 0 > :POSITION 0 > :LENGTH 10)) > The value > # (UNSIGNED > 8))> > is not of type > SYSTEM-AREA-POINTER. > [Condition of type TYPE-ERROR] > > Thanks for your help ... > deech > From sky at viridian-project.de Mon Nov 3 21:11:38 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Mon, 3 Nov 2008 22:11:38 +0100 (CET) Subject: [elephant-devel] Issues adding to root In-Reply-To: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> References: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> Message-ID: <62312.88.73.210.243.1225746698.squirrel@mail.stardawn.org> Can you try some other SBCL versions, namely .20 and .22? What version of Elephant are you using? If you're using 0.9.1, try to update to the latest stable head. Leslie From sky at viridian-project.de Mon Nov 3 21:15:51 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Mon, 3 Nov 2008 22:15:51 +0100 (CET) Subject: [elephant-devel] Issues installing Elephant 0.9 In-Reply-To: <594f78210811031140p347dd422v9db5d66af3d9a4b9@mail.gmail.com> References: <594f78210811030942k3adc97a1o26431e80134147b2@mail.gmail.com> <594f78210811031140p347dd422v9db5d66af3d9a4b9@mail.gmail.com> Message-ID: <62461.88.73.210.243.1225746951.squirrel@mail.stardawn.org> > I have solved the problem by making changes to the following function in ../src > /db-clsql/sql-controller.lisp. Can you check whether this change is still needed with the latest head? If yes, can you make a patch? Thanks! :) Leslie From killerstorm at newmail.ru Mon Nov 3 21:26:44 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Mon, 3 Nov 2008 23:26:44 +0200 Subject: [elephant-devel] Issues adding to root References: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> <594f78210811031230i9c4fab6lb4e920515e16f4c@mail.gmail.com> Message-ID: as> I am using SBCL 1.0.21. but which version of elephant do you use? please try darcs version if you're not using it already From aditya.siram at gmail.com Tue Nov 4 01:15:02 2008 From: aditya.siram at gmail.com (aditya siram) Date: Mon, 3 Nov 2008 19:15:02 -0600 Subject: [elephant-devel] Issues adding to root In-Reply-To: <62312.88.73.210.243.1225746698.squirrel@mail.stardawn.org> References: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> <62312.88.73.210.243.1225746698.squirrel@mail.stardawn.org> Message-ID: <594f78210811031715r227c9915q55240e99ca13d42@mail.gmail.com> The following is something of a workaround and I'm not really sure why it works, but here goes. Starting with a fresh SBCL image I loaded "cffi-uffi-compat" (*not* "cffi" or "uffi") , then "cl-base64" and finally "elephant". After issuing some warnings about package name clashes between UFFI and CFFI-UFFI-COMPAT, adding and retrieving from a store seems to work. I have yet to run the tests. Thanks ... -deech On Mon, Nov 3, 2008 at 3:11 PM, Leslie P. Polzer wrote: > > Can you try some other SBCL versions, namely .20 and .22? > > What version of Elephant are you using? If you're using 0.9.1, > try to update to the latest stable head. > > Leslie > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From aditya.siram at gmail.com Tue Nov 4 01:15:55 2008 From: aditya.siram at gmail.com (aditya siram) Date: Mon, 3 Nov 2008 19:15:55 -0600 Subject: [elephant-devel] Issues installing Elephant 0.9 In-Reply-To: <62461.88.73.210.243.1225746951.squirrel@mail.stardawn.org> References: <594f78210811030942k3adc97a1o26431e80134147b2@mail.gmail.com> <594f78210811031140p347dd422v9db5d66af3d9a4b9@mail.gmail.com> <62461.88.73.210.243.1225746951.squirrel@mail.stardawn.org> Message-ID: <594f78210811031715u243b7185g4299716f28f30e5c@mail.gmail.com> Yes, this is also a problem with code from the darcs repo. I will send a patch. Thanks... -deech On Mon, Nov 3, 2008 at 3:15 PM, Leslie P. Polzer wrote: > >> I have solved the problem by making changes to the following function in ../src >> /db-clsql/sql-controller.lisp. > > Can you check whether this change is still needed with the latest > head? If yes, can you make a patch? > > Thanks! :) > > Leslie > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From sky at viridian-project.de Tue Nov 4 08:39:02 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 4 Nov 2008 09:39:02 +0100 (CET) Subject: [elephant-devel] Issues adding to root In-Reply-To: <594f78210811031715r227c9915q55240e99ca13d42@mail.gmail.com> References: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> <62312.88.73.210.243.1225746698.squirrel@mail.stardawn.org> <594f78210811031715r227c9915q55240e99ca13d42@mail.gmail.com> Message-ID: <62022.88.73.251.160.1225787942.squirrel@mail.stardawn.org> > The following is something of a workaround and I'm not really sure why > it works, but here goes. > > Starting with a fresh SBCL image I loaded "cffi-uffi-compat" (*not* > "cffi" or "uffi") , then "cl-base64" and finally "elephant". After > issuing some warnings about package name clashes between UFFI and > CFFI-UFFI-COMPAT, adding and retrieving from a store seems to work. I > have yet to run the tests. Glenn has nearly migrated Elephant and ELE-BDB to CFFI, so this probably won't be an issue in the mid-term. I'm surprised that CFFI-UFFI works for you with Elephant. Perhaps the CFFI guys fixed some bugs since I last tried it. From gtarcea at umich.edu Tue Nov 4 13:13:25 2008 From: gtarcea at umich.edu (Glenn Tarcea) Date: Tue, 4 Nov 2008 08:13:25 -0500 Subject: [elephant-devel] Issues adding to root In-Reply-To: <62022.88.73.251.160.1225787942.squirrel@mail.stardawn.org> References: <594f78210811031229l2bb81a80q759759688c807b68@mail.gmail.com> <62312.88.73.210.243.1225746698.squirrel@mail.stardawn.org> <594f78210811031715r227c9915q55240e99ca13d42@mail.gmail.com> <62022.88.73.251.160.1225787942.squirrel@mail.stardawn.org> Message-ID: <7A955DAE-E3EB-47E7-93D0-DECDFDEDAF1E@umich.edu> Just to follow up on this. I've been working with Leslie in the back ground (he's graciously taken the time to do some mentoring and reviews of code as well as to put up with some of my not so well thought out ideas :-) ). I have large pieces of Elephant switched over to CFFI, but largely due to my limited time have temporarily placed this on the back burner while working through a skiplist implementation with Leslie. I hope to get back to this project in the next couple of weeks. While it has not been quite as straight forward as I had anticipated (for example deftype assumptions) there isn't any reason (beyond spending some time on it) that I shouldn't be able to complete it. Thanks, Glenn V. Glenn Tarcea gtarcea at umich.edu On Nov 4, 2008, at 3:39 AM, Leslie P. Polzer wrote: > >> The following is something of a workaround and I'm not really sure >> why >> it works, but here goes. >> >> Starting with a fresh SBCL image I loaded "cffi-uffi-compat" (*not* >> "cffi" or "uffi") , then "cl-base64" and finally "elephant". After >> issuing some warnings about package name clashes between UFFI and >> CFFI-UFFI-COMPAT, adding and retrieving from a store seems to work. I >> have yet to run the tests. > > Glenn has nearly migrated Elephant and ELE-BDB to CFFI, so > this probably won't be an issue in the mid-term. > > I'm surprised that CFFI-UFFI works for you with Elephant. > Perhaps the CFFI guys fixed some bugs since I last tried it. > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > From ocorrain at gmail.com Tue Nov 4 18:59:08 2008 From: ocorrain at gmail.com (Tiarnan O'Corrain) Date: Tue, 4 Nov 2008 18:59:08 +0000 Subject: [elephant-devel] keeping btree indices up to date Message-ID: Hi-- I'm storing objects in a btree, and indexing one slot common to all of the objects for fast retrieval. There is more than one type of object in the btree, so I'm using btree indices rather than a slot index. I note that when the indexed slot changes, the index is not updated. As I'm using this slot to retrieve objects, this is something of a problem. The slot contains a string value. I could get around this by deleting and then recreating the index, but that seems ugly. Is there anything else I can do? I'm using the latest darcs elephant from the common-lisp.net repo with Berkeley DB 4.5. regards Tiarnan -- I have a vegetable garden...in about three months I'll have six cabbages, 12 lettuces and at least eight onions, so the recession won't really affect me. (ROXY) From sky at viridian-project.de Wed Nov 5 08:51:38 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Wed, 5 Nov 2008 09:51:38 +0100 (CET) Subject: [elephant-devel] keeping btree indices up to date In-Reply-To: References: Message-ID: <62455.88.73.216.197.1225875098.squirrel@mail.stardawn.org> > I'm storing objects in a btree, and indexing one slot common to all of > the objects for fast retrieval. There is more than one type of object > in the btree, so I'm using btree indices rather than a slot index. Can you provide a small example? > I note that when the indexed slot changes, the index is not updated. > As I'm using this slot to retrieve objects, this is something of a > problem. The slot contains a string value. > > I could get around this by deleting and then recreating the index, but > that seems ugly. Is there anything else I can do? Yes, try the not-so-unstable-on-BDB elephant-unstable[1]. If that doesn't help then we're going to find out what's wrong. Leslie [1] http://common-lisp.net/project/elephant/darcs/elephant-unstable/ From ocorrain at gmail.com Wed Nov 5 15:19:32 2008 From: ocorrain at gmail.com (Tiarnan O'Corrain) Date: Wed, 5 Nov 2008 15:19:32 +0000 Subject: [elephant-devel] keeping btree indices up to date In-Reply-To: <62455.88.73.216.197.1225875098.squirrel@mail.stardawn.org> References: <62455.88.73.216.197.1225875098.squirrel@mail.stardawn.org> Message-ID: Hi-- here's a small program that reproduces the problem. > Yes, try the not-so-unstable-on-BDB elephant-unstable[1]. Thanks, I will do so tonight. regards Tiarn?n -------------- next part -------------- A non-text attachment was scrubbed... Name: ele-test.lisp Type: application/octet-stream Size: 1078 bytes Desc: not available URL: From raison at chatsubo.net Fri Nov 7 04:14:20 2008 From: raison at chatsubo.net (Kevin Raison) Date: Thu, 06 Nov 2008 20:14:20 -0800 Subject: [elephant-devel] get-instances-by-range oddity Message-ID: <4913C09C.90305@chatsubo.net> Given this class defined with elephant-unstable in sbcl 10.22 using berkeleydb 4.6: (defpclass test-class () ((host :accessor host :initarg :host :type string :initform "" :index t) (timestamp :accessor timestamp :initarg :timestamp :initform (get-universal-time) :type integer :index t)) (:index t)) Can someone help me understand why this works: CL-USER> (make-instance 'test-class :host "test1" :timestamp 10000000000) # CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) (#) But this does not: CL-USER> (make-instance 'test-class :host "test1" :timestamp (get-universal-time)) # CL-USER> (get-instances-by-range 'test-class 'timestamp 0 (get-universal-time)) NIL CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) NIL When (type-of (get-universal-time)) is (INTEGER 0 1152921504606846975) If I do not drop all instances between the above two command sets, I get this: CL-USER> (make-instance 'test-class :host "test1" :timestamp 10000000000) # CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) (#) CL-USER> (make-instance 'test-class :host "test1" :timestamp (get-universal-time)) # CL-USER> (get-instances-by-range 'test-class 'timestamp 0 (get-universal-time)) (#) CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) (#) I have tried a number of variations, including not specifying types in the class definition, specifying "number" as the type of timestamp, coercing (get-universal-time) to an integer. Nothing has worked. Can anyone shed light on this for me? Am I missing something obvious? The hyperspec specifies that universal time "is an absolute time represented as a single non-negative integer." So why would it not be indexed like any other integer? Thanks in advance. Kevin Raison From eslick at media.mit.edu Sat Nov 8 05:55:42 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sat, 8 Nov 2008 00:55:42 -0500 Subject: [elephant-devel] get-instances-by-range oddity In-Reply-To: <4913C09C.90305@chatsubo.net> References: <4913C09C.90305@chatsubo.net> Message-ID: <6605A4F8-0DC9-4DD3-A1E2-5744B72DEF97@media.mit.edu> I can reproduce this, the serializer is treating this value as a bignum but the range value as a fixnum so the berkeley-db comparison function doesn't compare them properly. I'm not sure yet if what is happening has to do with the lisp type system or the innards of Elephant... Turns out your timing was excellent as I'm depending on this capability for a feature I'm rolling out this weekend, actually. Ian On Nov 6, 2008, at 11:14 PM, Kevin Raison wrote: > Given this class defined with elephant-unstable in sbcl 10.22 using > berkeleydb 4.6: > > (defpclass test-class () > ((host :accessor host :initarg :host :type string :initform > "" :index t) > (timestamp :accessor timestamp :initarg :timestamp :initform > (get-universal-time) > :type integer :index t)) > (:index t)) > > Can someone help me understand why this works: > > CL-USER> (make-instance 'test-class :host "test1" :timestamp > 10000000000) > # > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) > (#) > > But this does not: > CL-USER> (make-instance 'test-class :host "test1" :timestamp > (get-universal-time)) > # > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 > (get-universal-time)) > NIL > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) > NIL > > When (type-of (get-universal-time)) is (INTEGER 0 1152921504606846975) > > > If I do not drop all instances between the above two command sets, I > get > this: > CL-USER> (make-instance 'test-class :host "test1" :timestamp > 10000000000) > # > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) > (#) > CL-USER> (make-instance 'test-class :host "test1" :timestamp > (get-universal-time)) > # > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 > (get-universal-time)) > (#) > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) > (#) > > > I have tried a number of variations, including not specifying types in > the class definition, specifying "number" as the type of timestamp, > coercing (get-universal-time) to an integer. Nothing has worked. Can > anyone shed light on this for me? Am I missing something obvious? > The > hyperspec specifies that universal time "is an absolute time > represented > as a single non-negative integer." So why would it not be indexed > like > any other integer? > > Thanks in advance. > > Kevin Raison > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Sat Nov 8 06:42:41 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sat, 8 Nov 2008 01:42:41 -0500 Subject: [elephant-devel] get-instances-by-range oddity In-Reply-To: <4913C09C.90305@chatsubo.net> References: <4913C09C.90305@chatsubo.net> Message-ID: <4A8EE27D-336F-4C97-9FD3-1958ABAC67D6@media.mit.edu> I don't have time to dive into the details of 'why' this is failing, but here is a workaround in the meantime. (defmethod cursor-set-range :around ((cursor bdb-cursor) key) (multiple-value-bind (valid? k v) (cursor-set cursor key) (if valid? (values valid? k v) (cursor-next cursor)))) This is in the db-bdb package. For some reason cursor-next does the comparison properly while cursor- set-range does not. Something about the types returned by (get- universal-time) cause the serializer to do something unexpected and it will take some time to track it down due to issues of visibility in the serializer buffer streams. Let me know if this gets you unstuck. Ian On Nov 6, 2008, at 11:14 PM, Kevin Raison wrote: > Given this class defined with elephant-unstable in sbcl 10.22 using > berkeleydb 4.6: > > (defpclass test-class () > ((host :accessor host :initarg :host :type string :initform > "" :index t) > (timestamp :accessor timestamp :initarg :timestamp :initform > (get-universal-time) > :type integer :index t)) > (:index t)) > > Can someone help me understand why this works: > > CL-USER> (make-instance 'test-class :host "test1" :timestamp > 10000000000) > # > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) > (#) > > But this does not: > CL-USER> (make-instance 'test-class :host "test1" :timestamp > (get-universal-time)) > # > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 > (get-universal-time)) > NIL > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) > NIL > > When (type-of (get-universal-time)) is (INTEGER 0 1152921504606846975) > > > If I do not drop all instances between the above two command sets, I > get > this: > CL-USER> (make-instance 'test-class :host "test1" :timestamp > 10000000000) > # > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) > (#) > CL-USER> (make-instance 'test-class :host "test1" :timestamp > (get-universal-time)) > # > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 > (get-universal-time)) > (#) > CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) > (#) > > > I have tried a number of variations, including not specifying types in > the class definition, specifying "number" as the type of timestamp, > coercing (get-universal-time) to an integer. Nothing has worked. Can > anyone shed light on this for me? Am I missing something obvious? > The > hyperspec specifies that universal time "is an absolute time > represented > as a single non-negative integer." So why would it not be indexed > like > any other integer? > > Thanks in advance. > > Kevin Raison > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From ocorrain at gmail.com Sat Nov 8 17:00:26 2008 From: ocorrain at gmail.com (Tiarnan O'Corrain) Date: Sat, 8 Nov 2008 17:00:26 +0000 Subject: [elephant-devel] keeping btree indices up to date In-Reply-To: References: <62455.88.73.216.197.1225875098.squirrel@mail.stardawn.org> Message-ID: Hi-- I've tried the same test-case on elephant unstable and the indices are still not being updated, so the problem remains. One question: I'm using slot-value to update the indexed slots. Is this the problem, or should the indices be updated in any case? regards Tiarnan From spambait at CloudDancer.com Sun Nov 9 07:23:43 2008 From: spambait at CloudDancer.com (GP lisper) Date: Sat, 8 Nov 2008 23:23:43 -0800 Subject: [elephant-devel] cmucl problems in memutil.lisp Message-ID: in elephant 0.9 with cmucl snapshot from September 08, I find: CL-USER> (load "library:subsystems/berkeley-db") T CL-USER> (require :elephant) ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/elephant.asd". ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/cl-base64.asd". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/uffi.asd". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/package.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/primitives.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/objects.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/aggregates.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/functions.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/strings.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/libraries.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/os.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/package.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/encode.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/decode.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/package.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/convenience.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/locks.x86f". ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/os.x86f". Attempting to load libmemutil.so... ; Compiling: CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/memutil/memutil.lisp 28 APR 07 07:10:55 pm ... ; Compiling DEFUN RETURN-BUFFER-STREAM: ; Converted WITH-BUFFER-STREAMS. ; Compiling DEFMACRO WITH-BUFFER-STREAMS: Error in function ALIEN::%PARSE-ALIEN-TYPE: Unknown alien type: UNSIGNED-CHAR I'm not familiar enough with Alien to track this down. Perhaps CMUCL is not as supported as in the past? -- One of the strokes of genius from McCarthy was making lists the center of the language - kt From aditya.siram at gmail.com Sun Nov 9 19:14:10 2008 From: aditya.siram at gmail.com (aditya siram) Date: Sun, 9 Nov 2008 13:14:10 -0600 Subject: [elephant-devel] cmucl problems in memutil.lisp In-Reply-To: <594f78210811091102s31c3de35m8797d4181cfbe79c@mail.gmail.com> References: <594f78210811091102s31c3de35m8797d4181cfbe79c@mail.gmail.com> Message-ID: <594f78210811091114h271650ady47049ab2febee73d@mail.gmail.com> One other suggestion ... it is my understanding that elephant foreign calls are *only* made through clsql, so the problem maybe with clsql. I recompiled mine from scratch and haven't seen this issue since. -deech On Sun, Nov 9, 2008 at 1:02 PM, aditya siram wrote: > Hi there, > I was having similar problems with SBCL 1.0.21. My workaround is > outlined in the thread "Issues adding to root". Basically it seems > like you need to add the "cffi-uffi-compat" packages and *not* cffi or > uffi. > > This maybe an SBCL-only solution, so I hope this works you. > -deech > > On Sun, Nov 9, 2008 at 1:23 AM, GP lisper wrote: >> in elephant 0.9 with cmucl snapshot from September 08, I find: >> >> CL-USER> (load "library:subsystems/berkeley-db") >> T >> CL-USER> (require :elephant) >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/elephant.asd". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/cl-base64.asd". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/uffi.asd". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/package.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/primitives.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/objects.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/aggregates.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/functions.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/strings.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/libraries.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/os.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/package.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/encode.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/decode.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/package.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/convenience.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/locks.x86f". >> ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/os.x86f". >> Attempting to load libmemutil.so... >> ; Compiling: CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/memutil/memutil.lisp 28 APR 07 07:10:55 pm >> ... >> ; Compiling DEFUN RETURN-BUFFER-STREAM: >> ; Converted WITH-BUFFER-STREAMS. >> ; Compiling DEFMACRO WITH-BUFFER-STREAMS: >> >> >> Error in function ALIEN::%PARSE-ALIEN-TYPE: Unknown alien type: UNSIGNED-CHAR >> >> >> I'm not familiar enough with Alien to track this down. Perhaps CMUCL >> is not as supported as in the past? >> >> -- >> One of the strokes of genius from McCarthy was making lists the center of the language - kt >> >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > From aditya.siram at gmail.com Sun Nov 9 19:02:25 2008 From: aditya.siram at gmail.com (aditya siram) Date: Sun, 9 Nov 2008 13:02:25 -0600 Subject: [elephant-devel] cmucl problems in memutil.lisp In-Reply-To: References: Message-ID: <594f78210811091102s31c3de35m8797d4181cfbe79c@mail.gmail.com> Hi there, I was having similar problems with SBCL 1.0.21. My workaround is outlined in the thread "Issues adding to root". Basically it seems like you need to add the "cffi-uffi-compat" packages and *not* cffi or uffi. This maybe an SBCL-only solution, so I hope this works you. -deech On Sun, Nov 9, 2008 at 1:23 AM, GP lisper wrote: > in elephant 0.9 with cmucl snapshot from September 08, I find: > > CL-USER> (load "library:subsystems/berkeley-db") > T > CL-USER> (require :elephant) > ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/elephant.asd". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/cl-base64.asd". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/uffi.asd". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/package.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/primitives.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/objects.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/aggregates.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/functions.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/strings.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/libraries.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/uffi-1.5.15/src/os.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/package.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/encode.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/cl-base64-3.3.2/decode.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/package.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/convenience.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/locks.x86f". > ; Loading #P"CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/utils/os.x86f". > Attempting to load libmemutil.so... > ; Compiling: CMUCL-19e-SEP-08/subsystems/elephant-0.9/src/memutil/memutil.lisp 28 APR 07 07:10:55 pm > ... > ; Compiling DEFUN RETURN-BUFFER-STREAM: > ; Converted WITH-BUFFER-STREAMS. > ; Compiling DEFMACRO WITH-BUFFER-STREAMS: > > > Error in function ALIEN::%PARSE-ALIEN-TYPE: Unknown alien type: UNSIGNED-CHAR > > > I'm not familiar enough with Alien to track this down. Perhaps CMUCL > is not as supported as in the past? > > -- > One of the strokes of genius from McCarthy was making lists the center of the language - kt > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From sky at viridian-project.de Mon Nov 10 08:44:16 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Mon, 10 Nov 2008 09:44:16 +0100 (CET) Subject: [elephant-devel] cmucl problems in memutil.lisp In-Reply-To: <594f78210811091114h271650ady47049ab2febee73d@mail.gmail.com> References: <594f78210811091102s31c3de35m8797d4181cfbe79c@mail.gmail.com> <594f78210811091114h271650ady47049ab2febee73d@mail.gmail.com> Message-ID: <64072.88.73.213.174.1226306656.squirrel@mail.stardawn.org> > One other suggestion ... it is my understanding that elephant foreign > calls are *only* made through clsql, so the problem maybe with clsql. No, foreign calls are currently an essential part of the serializer arch, too. And the BDB backend. Leslie From lists at infoway.net Mon Nov 10 18:43:28 2008 From: lists at infoway.net (lists at infoway.net) Date: Mon, 10 Nov 2008 13:43:28 -0500 (EST) Subject: [elephant-devel] BDB 4.7 backend tests fail on OS X SBCL 1.0.22 Message-ID: <1226342608.354925005@192.168.1.71> Hi all, I'm just finding time again to get back into elephant. So I downloaded latest unstable darcs and when I run the tests, it aborts as shown below. Any ideas? Thanks - Waldo There is no class named PERSON. [Condition of type SIMPLE-ERROR] Restarts: 0: [ABORT] Return to SLIME's top level. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-PCL::FIND-CLASS-FROM-CELL PERSON # T) Locals: SB-PCL::CELL = # SB-PCL::ERRORP = T SYMBOL = PERSON 1: ((SB-PCL::FAST-METHOD ELEPHANT::CLOSE-CONTROLLER :BEFORE (STORE-CONTROLLER)) #(8 NIL 3 NIL) # # SB-DEBUG::ARG-2 = # 2: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0.)) # # # SB-DEBUG::ARG-1 = : SB-DEBUG::ARG-2 = # 3: (CLOSE-STORE #) Locals: ELEPHANT::SC = # 4: ((FLET #:CLEANUP-FUN-[DO-BACKEND-TESTS]204))[:CLEANUP] [No Locals] 5: (DO-BACKEND-TESTS (:BDB "/Users/devel/lisp/src/elephant-unstable/tests/testdb/")) Locals: SPEC = (:BDB "/Users/devel/lisp/src/elephant-unstable/tests/testdb/") 6: (SB-INT:SIMPLE-EVAL-IN-LEXENV (DO-BACKEND-TESTS) #) 7: (SWANK::EVAL-REGION "(do-backend-tests) ") 8: ((LAMBDA NIL)) 9: (SWANK::TRACK-PACKAGE #) From denis.papathanasiou at gmail.com Mon Nov 10 18:52:57 2008 From: denis.papathanasiou at gmail.com (denis.papathanasiou at gmail.com) Date: Mon, 10 Nov 2008 13:52:57 -0500 Subject: [elephant-devel] Installing 0.9.1 on CMU: "libmemutil" problem Message-ID: <49188309.90407@gmail.com> When I tried installing version 0.9.1 on CMU (version 19e), I got everything in src/utils to build, but then I got an error related to src/memutil as follows: End-of-File on # [Condition of type END-OF-FILE] Restarts: 0: [RETRY ] Retry performing # on #. 1: [ACCEPT ] Continue, treating # on # as having been successful. 2: [CONTINUE] Return NIL from load of "install-steps.lisp". 3: [ABORT ] Return to Top-Level. Debug (type H for help) (LISP::INPUT-CHARACTER # T (LISP::*EOF*)) Source: Error finding source: Error in function DEBUG::GET-FILE-TOP-LEVEL-FORM: Source file no longer exists: target:code/fd-stream.lisp. 0] The my-config.sexp file is the same the config.sexp file provided, except I changed all the :berkeley-db references to BerkeleyDB.4.7, which is the version of Berkeley I have installed. I did see that CMU is marked in red as "untested recently" in the platform list, so I was wondering if anyone had tried it with CMU, or could suggest a work-around? From raison at chatsubo.net Tue Nov 11 01:11:17 2008 From: raison at chatsubo.net (Kevin Raison) Date: Mon, 10 Nov 2008 17:11:17 -0800 Subject: [elephant-devel] get-instances-by-range oddity In-Reply-To: <4A8EE27D-336F-4C97-9FD3-1958ABAC67D6@media.mit.edu> References: <4913C09C.90305@chatsubo.net> <4A8EE27D-336F-4C97-9FD3-1958ABAC67D6@media.mit.edu> Message-ID: <4918DBB5.1000501@chatsubo.net> Thanks, Ian. This solved my problem for new objects. Now, is there an easy way to re-index existing objects? Cheers. Kevin Ian Eslick wrote: > I don't have time to dive into the details of 'why' this is failing, > but here is a workaround in the meantime. > > (defmethod cursor-set-range :around ((cursor bdb-cursor) key) > (multiple-value-bind (valid? k v) > (cursor-set cursor key) > (if valid? > (values valid? k v) > (cursor-next cursor)))) > > This is in the db-bdb package. > > For some reason cursor-next does the comparison properly while cursor- > set-range does not. Something about the types returned by (get- > universal-time) cause the serializer to do something unexpected and it > will take some time to track it down due to issues of visibility in > the serializer buffer streams. > > Let me know if this gets you unstuck. > > Ian > > On Nov 6, 2008, at 11:14 PM, Kevin Raison wrote: > >> Given this class defined with elephant-unstable in sbcl 10.22 using >> berkeleydb 4.6: >> >> (defpclass test-class () >> ((host :accessor host :initarg :host :type string :initform >> "" :index t) >> (timestamp :accessor timestamp :initarg :timestamp :initform >> (get-universal-time) >> :type integer :index t)) >> (:index t)) >> >> Can someone help me understand why this works: >> >> CL-USER> (make-instance 'test-class :host "test1" :timestamp >> 10000000000) >> # >> CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) >> (#) >> >> But this does not: >> CL-USER> (make-instance 'test-class :host "test1" :timestamp >> (get-universal-time)) >> # >> CL-USER> (get-instances-by-range 'test-class 'timestamp 0 >> (get-universal-time)) >> NIL >> CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) >> NIL >> >> When (type-of (get-universal-time)) is (INTEGER 0 1152921504606846975) >> >> >> If I do not drop all instances between the above two command sets, I >> get >> this: >> CL-USER> (make-instance 'test-class :host "test1" :timestamp >> 10000000000) >> # >> CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) >> (#) >> CL-USER> (make-instance 'test-class :host "test1" :timestamp >> (get-universal-time)) >> # >> CL-USER> (get-instances-by-range 'test-class 'timestamp 0 >> (get-universal-time)) >> (#) >> CL-USER> (get-instances-by-range 'test-class 'timestamp 0 10000000001) >> (#) >> >> >> I have tried a number of variations, including not specifying types in >> the class definition, specifying "number" as the type of timestamp, >> coercing (get-universal-time) to an integer. Nothing has worked. Can >> anyone shed light on this for me? Am I missing something obvious? >> The >> hyperspec specifies that universal time "is an absolute time >> represented >> as a single non-negative integer." So why would it not be indexed >> like >> any other integer? >> >> Thanks in advance. >> >> Kevin Raison >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From sky at viridian-project.de Tue Nov 11 08:56:58 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 11 Nov 2008 09:56:58 +0100 (CET) Subject: [elephant-devel] BDB 4.7 backend tests fail on OS X SBCL 1.0.22 In-Reply-To: <1226342608.354925005@192.168.1.71> References: <1226342608.354925005@192.168.1.71> Message-ID: <62106.88.73.193.146.1226393818.squirrel@mail.stardawn.org> > Hi all, > > I'm just finding time again to get back into elephant. So I downloaded latest unstable > darcs and when I run the tests, it aborts as shown below. Any ideas? I'll try to check this. From sky at viridian-project.de Tue Nov 11 08:57:19 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 11 Nov 2008 09:57:19 +0100 (CET) Subject: [elephant-devel] Installing 0.9.1 on CMU: "libmemutil" problem In-Reply-To: <49188309.90407@gmail.com> References: <49188309.90407@gmail.com> Message-ID: <62106.88.73.193.146.1226393839.squirrel@mail.stardawn.org> > When I tried installing version 0.9.1 on CMU (version 19e), I got > everything in src/utils to build, but then I got an error related to > src/memutil as follows: > > End-of-File on # "/opt/downloads/berkeley-db/common-lisp/Elephant/elephant/my-config.sexp"> > [Condition of type END-OF-FILE] Looks like a syntax error in your my-config.sexp. Missing any parens? From killerstorm at newmail.ru Tue Nov 11 10:56:16 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 11 Nov 2008 12:56:16 +0200 Subject: [elephant-devel] BDB 4.7 backend tests fail on OS X SBCL 1.0.22 References: <1226342608.354925005@192.168.1.71> Message-ID: lR> I'm just finding time again to get back into elephant. So I downloaded lR> latest unstable darcs and when I run the tests, it aborts as shown lR> below. Any ideas? lR> There is no class named PERSON. tests are broken -- something about class definitions/redefinitions, not related to elephant itself. i've already reported these errors some time ago. (well, i could try to fix them, but honestly i have no clue what are they trying to test and in which way) what's interesting, if you run tests multiple times, eventually you'll get it passing 100% of tests :) i guess that's how these broken tests went into repository From denis.papathanasiou at gmail.com Tue Nov 11 12:35:10 2008 From: denis.papathanasiou at gmail.com (denis.papathanasiou at gmail.com) Date: Tue, 11 Nov 2008 07:35:10 -0500 Subject: [elephant-devel] Installing 0.9.1 on CMU: "libmemutil" problem In-Reply-To: <62106.88.73.193.146.1226393839.squirrel@mail.stardawn.org> References: <49188309.90407@gmail.com> <62106.88.73.193.146.1226393839.squirrel@mail.stardawn.org> Message-ID: <49197BFE.3040801@gmail.com> > Looks like a syntax error in your my-config.sexp. Missing any parens? That's what I thought at first, too, since "Condition of type END-OF-FILE" usually means just that. But the only difference between my-config.sexp and config.sexp was that I changed all the :berkeley-db references to BerkeleyDB.4.7, which is the version of Berkeley I have installed. From sky at viridian-project.de Tue Nov 11 13:54:43 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 11 Nov 2008 14:54:43 +0100 (CET) Subject: [elephant-devel] Installing 0.9.1 on CMU: "libmemutil" problem In-Reply-To: <49197BFE.3040801@gmail.com> References: <49188309.90407@gmail.com> <62106.88.73.193.146.1226393839.squirrel@mail.stardawn.org> <49197BFE.3040801@gmail.com> Message-ID: <61959.88.73.193.146.1226411683.squirrel@mail.stardawn.org> > But the only difference between my-config.sexp and config.sexp was that > I changed all the :berkeley-db references to BerkeleyDB.4.7, which is > the version of Berkeley I have installed. Weird. Can you send me a copy of your tree? Leslie From denis.papathanasiou at gmail.com Tue Nov 11 16:27:38 2008 From: denis.papathanasiou at gmail.com (denis.papathanasiou at gmail.com) Date: Tue, 11 Nov 2008 11:27:38 -0500 Subject: [elephant-devel] Installing 0.9.1 on CMU: "libmemutil" problem In-Reply-To: <61959.88.73.193.146.1226411683.squirrel@mail.stardawn.org> References: <49188309.90407@gmail.com> <62106.88.73.193.146.1226393839.squirrel@mail.stardawn.org> <49197BFE.3040801@gmail.com> <61959.88.73.193.146.1226411683.squirrel@mail.stardawn.org> Message-ID: <4919B27A.3040702@gmail.com> > Weird. Can you send me a copy of your tree? Do you mean the "elephant" folder (and sub-folders) I downloaded from darcs? If so, let me know if I should copy the list, or email it just to you. From peter.eddy at gmail.com Fri Nov 14 20:10:20 2008 From: peter.eddy at gmail.com (Peter Eddy) Date: Fri, 14 Nov 2008 15:10:20 -0500 Subject: [elephant-devel] unique object IDs and OIDs Message-ID: <5aea26870811141210ice396fwd3bfb6467bf70561@mail.gmail.com> Hello, I'm a first time user of elephant and I'm using it to build a prototype application that will send and receive updates to persistent objects via Json. I'd like to use unique IDs in order for the Json data to refer to the elephant based persistent objects. I'm wondering if it is appropriate to use the OIDs that elephant assigns objects for this purpose. Will these values change during the lifetime of the associated object? If using OID values makes sense for this purpose, is there a user-API way to get an object by its OID? Should I instead add another indexed field on my object that contains the OID value? It seems like there should be a way to get an object by it's OID, maybe I'm just missing it? thanks in advance, Peter From killerstorm at newmail.ru Fri Nov 14 22:29:06 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 15 Nov 2008 00:29:06 +0200 Subject: [elephant-devel] unique object IDs and OIDs References: <5aea26870811141210ice396fwd3bfb6467bf70561@mail.gmail.com> Message-ID: PE> I'm wondering if it is appropriate to use the OIDs that elephant PE> assigns objects for this purpose. yep, i think it's fine PE> Will these values change during the lifetime of the associated object? they shouldn't. oids are not preserved across export/import operations (by design of these export/import operations), but typically it's not a problem. PE> If using OID values makes sense for this purpose, is there a user-API PE> way to get an object by its OID? yep, via class index: (get-value oid (find-class-index 'classname)) an internal function get-cached-instance does it in more straightforward way, though. PE> Should I instead add another indexed field on my object that contains PE> the OID value? definitely not, unless you want to use your own ids. From rsynnott at gmail.com Tue Nov 18 00:49:25 2008 From: rsynnott at gmail.com (Robert Synnott) Date: Tue, 18 Nov 2008 00:49:25 +0000 Subject: [elephant-devel] Postmodern performance question Message-ID: <24f203480811171649n6640111bpdaa44619872e854d@mail.gmail.com> Hi, I'm wondering does anyone have any tips on setting up postgres as a postmodern backend for Elephant. I suspect that my issues are just a case of not really having enough memory for my data set, but if there's anything obvious I should be doing, I might as well do that first. Thanks Rob From wkerr at cs.arizona.edu Tue Nov 18 03:57:27 2008 From: wkerr at cs.arizona.edu (Wesley Kerr) Date: Mon, 17 Nov 2008 20:57:27 -0700 Subject: [elephant-devel] ClozureCL Intel 64 bit problem Message-ID: <333e2fc20811171957q1822054dj4b03bdbd10ce03f7@mail.gmail.com> I've been reading through the archives and back in June Patrick had a problem installing onto a 64 bit PPC machine with ClozureCL. At one point he was getting some errors that looked similar to this: Undefined function #:G6340 called with arguments (# # # 2 10 # 0 10 0 # ..) [Condition of type CCL::UNDEFINED-FUNCTION-CALL] Ian mentioned that he would contact the developers of ClozureCL and see if they could help debug the problem since it was unclear what was causing it. I'm now getting these errors on a 64bit intel and I'm not sure where to even begin. I'm using BDB 4.5 and my configuration file is: ;; OSX Defaults #+(and (or sbcl allegro openmcl lispworks) (not (or mswindows windows win32)) (or macosx darwin)) ((:compiler . :gcc) (:berkeley-db-version . "4.5") (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.dylib") (:berkeley-db-cachesize . 20971520) (:berkeley-db-max-locks . 2000) (:berkeley-db-max-objects . 2000) (:berkeley-db-map-degree2 . t) (:clsql-lib-paths . nil) (:prebuilt-libraries . nil)) I have built BDB 4.5 for 64 bit mac and used darcs to get the unstable version of elephant. Any thoughts for how to get me going? Patrick moved away from the problem by switching to SBCL, but that isn't an option for our lab. Thanks Wes Kerr University of Arizona From sky at viridian-project.de Tue Nov 18 08:46:54 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 18 Nov 2008 09:46:54 +0100 (CET) Subject: [elephant-devel] Postmodern performance question In-Reply-To: <24f203480811171649n6640111bpdaa44619872e854d@mail.gmail.com> References: <24f203480811171649n6640111bpdaa44619872e854d@mail.gmail.com> Message-ID: <63056.88.73.205.17.1226998014.squirrel@mail.stardawn.org> > I'm wondering does anyone have any tips on setting up postgres as a > postmodern backend for Elephant. I suspect that my issues are just a > case of not really having enough memory for my data set, but if > there's anything obvious I should be doing, I might as well do that > first. Make sure you allow enough simultaneous connections for your users and enough shared mem. The PG manual has a section on tweaking these. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From killerstorm at newmail.ru Tue Nov 18 10:35:54 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 18 Nov 2008 12:35:54 +0200 Subject: [elephant-devel] Postmodern performance question References: <24f203480811171649n6640111bpdaa44619872e854d@mail.gmail.com> Message-ID: RS> I'm wondering does anyone have any tips on setting up postgres as a RS> postmodern backend for Elephant. I suspect that my issues are just a RS> case of not really having enough memory for my data set, but if RS> there's anything obvious I should be doing, I might as well do that RS> first. it is a good idea to enable logging in postgres to see query execution durations. normally you would see something on order of 0.01-0.1 ms. if it takes longer, perhaps it requires tuning. one of issues i've noticed is that postgresql's planner sometimes prefers hash joins to normal ones, and this causes problems when looking up values on large indices in elephant -- even if you lookup small range it will try to load whole index in memory. if it does so, it makes sense disabling hash joins, or tuning planner's stats (e.g. making random IO less costly). From tjg at pentaside.org Thu Nov 20 16:16:17 2008 From: tjg at pentaside.org (Tayssir John Gabbour) Date: Thu, 20 Nov 2008 17:16:17 +0100 Subject: [elephant-devel] Dropping DB connection during writes Message-ID: Hi! With Elephant under both Postmodern and CLSQL, I'm consistently losing my DB connection. This apparently happens when writing from fairly simultaneous threads. (Two threads may write to the same btree key at around the same time.) All I do is add objects to a BTree, keyed by their UUID. (And there's two simple indexes.) No persistent metaclasses or anything; I just use it as a persistent table. Am I doing something terribly wrong? Code and error messages at the end of this email. All the best, Tayssir ;;; How I add stuff to the DB (defmethod save-to-db (thing) (with-city-store (setf (get-value (moderator-ontology:uuid thing) (my-btree)) thing))) (defmacro with-city-store (&body body) `(if *store-controller* (progn , at body) (with-open-store (citycouncil-config:*citycouncil-store-spec*) ;; removing with-transaction doesn't have much effect (with-transaction () , at body)))) (defun create-indexes (&key (btree (my-btree))) (with-city-store (add-index btree :index-name *semantic-type-index-name* :key-form '(lambda (secondary-db primary-key value) (declare (ignore secondary-db primary-key)) (values t (type->storage-location value)))) (add-index btree :populate t :index-name *owner-index-name* :key-form '(lambda (secondary-db primary-key value) (declare (ignore secondary-db primary-key)) (if (typep value 'moderator-ontology:icon) (values t (moderator-ontology:owner value)) (values nil nil)))))) ;;; CLSQL messages Condition CONTROLLER-LOST-ERROR was signalled. [Condition of type CONTROLLER-LOST-ERROR] Restarts: 0: [CONTINUE] Open a new instance and continue? 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: ((SB-PCL::FAST-METHOD ELEPHANT::GET-CON (PERSISTENT)) # # # #) Locals: ELEPHANT::INSTANCE = # 1: ((SB-PCL::FAST-METHOD GET-VALUE (T DB-CLSQL::SQL-BTREE)) # # :STORE-2008-06-24 #) Locals: SB-DEBUG::ARG-0 = : SB-DEBUG::ARG-1 = : SB-DEBUG::ARG-2 = :STORE-2008-06-24 SB-DEBUG::ARG-3 = # 2: ((LAMBDA ())) [No Locals] 3: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-CLSQL::SQL-STORE-CONTROLLER T)) ..)[:EXTERNAL] Locals: SB-DEBUG::ARG-0 = 8 SB-DEBUG::ARG-1 = : SB-DEBUG::ARG-2 = : SB-DEBUG::ARG-3 = # There is no applicable method for the generic function # when called with arguments (NIL). [Condition of type SIMPLE-ERROR] Restarts: 0: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #) Locals: SB-KERNEL:*HANDLER-CLUSTERS* = : SWANK-BACKEND::*SLDB-STACK-TOP* = : SWANK-BACKEND::DEBUGGER-LOOP-FN = # 1: (SWANK::DEBUG-IN-EMACS #) 2: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) # #) 3: (SWANK::CALL-WITH-REDIRECTED-IO # #) 4: (SWANK::CALL-WITH-CONNECTION # #) 5: (SWANK:INVOKE-SLIME-DEBUGGER #) 6: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) # #) 7: (INVOKE-DEBUGGER #) 8: (INVOKE-DEBUGGER #)[:EXTERNAL] 9: ((SB-PCL::FAST-METHOD HUNCHENTOOT:MAYBE-INVOKE-DEBUGGER (T)) # # #) 10: (SIGNAL #)[:EXTERNAL] 11: (ERROR "~@")[:EXTERNAL] Locals: SB-DEBUG::ARG-0 = 3 SB-DEBUG::ARG-1 = "~@ # # SB-DEBUG::ARG-2 = : SB-DEBUG::ARG-3 = # 16: ((SB-PCL::FAST-METHOD LOCATE-ICON-BY-OWNER (STRING)) # # "bc3c7276-3a80-0780-23e1-40c33cd1b238") ;;; Postmodern messages Database error: # is closed [Condition of type CL-POSTGRES:DATABASE-SOCKET-ERROR] Restarts: 0: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: ((LAMBDA ())) 1: (DB-POSTMODERN::POSTGRES-FORMAT # :OBJECT) 2: ((LAMBDA ())) 3: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] 4: ((SB-PCL::FAST-METHOD (SETF DB-POSTMODERN::INTERNAL-GET-VALUE) (T T DB-POSTMODERN::PM-BTREE)) ..) 5: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T DB-POSTMODERN::PM-INDEXED-BTREE)) ..) 6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] 7: ((SB-PCL::FAST-METHOD ADD (MODERATOR-ONTOLOGY:UUID-MIXIN)) # # #) (#>) has a bad file descriptor. [Condition of type SIMPLE-ERROR] Restarts: 0: [REMOVE-THEM] Remove bogus handlers. 1: [RETRY-THEM] Retry bogus handlers. 2: [CONTINUE] Go on, leaving handlers marked as bogus. 3: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-IMPL::HANDLER-DESCRIPTORS-ERROR) 1: (SB-IMPL::SUB-SUB-SERVE-EVENT NIL NIL) 2: (SB-IMPL::SUB-SERVE-EVENT NIL NIL NIL) 3: (SB-SYS:WAIT-UNTIL-FD-USABLE 10 :INPUT NIL) 4: (SB-IMPL::REFILL-INPUT-BUFFER #) 5: (SB-IMPL::INPUT-UNSIGNED-8BIT-BYTE # T NIL) 6: (CL-POSTGRES::READ-UINT1 #) 7: (CL-POSTGRES::SEND-EXECUTE # # # #) 8: ((LABELS #:G303)) 9: ((LAMBDA ())) 10: (DB-POSTMODERN::POSTGRES-FORMAT # :OBJECT) 11: ((LAMBDA ())) 12: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] 13: ((SB-PCL::FAST-METHOD (SETF DB-POSTMODERN::INTERNAL-GET-VALUE) (T T DB-POSTMODERN::PM-BTREE)) ..) 14: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T DB-POSTMODERN::PM-INDEXED-BTREE)) ..) 15: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] 16: ((SB-PCL::FAST-METHOD ADD (MODERATOR-ONTOLOGY:UUID-MIXIN)) # # #) Database error: Connection to database server lost. [Condition of type CL-POSTGRES:DATABASE-CONNECTION-LOST] Restarts: 0: [RECONNECT] Try to reconnect. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (CL-POSTGRES::ENSURE-CONNECTION #) 1: (CL-POSTGRES:EXEC-QUERY # # CL-POSTGRES:IGNORE-ROW-READER) 2: (POSTMODERN:ABORT-TRANSACTION #) 3: ((SB-PCL::FAST-METHOD CONTROLLER-ABORT-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] 4: ((FLET #:CLEANUP-FUN-[EXECUTE-TRANSACTION]23))[:CLEANUP] 5: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] 6: ((SB-PCL::FAST-METHOD ADD (MODERATOR-ONTOLOGY:UUID-MIXIN)) # # #) 7: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) Condition CONTROLLER-LOST-ERROR was signalled. [Condition of type CONTROLLER-LOST-ERROR] Restarts: 0: [CONTINUE] Open a new instance and continue? 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: ((SB-PCL::FAST-METHOD ELEPHANT::GET-CON (PERSISTENT)) #) 1: (ELEPHANT::INITIALIZE-PERSISTENT-SLOTS # # # # # # # # # # # #) From killerstorm at newmail.ru Thu Nov 20 21:16:29 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 20 Nov 2008 23:16:29 +0200 Subject: [elephant-devel] Dropping DB connection during writes References: Message-ID: TJG> With Elephant under both Postmodern and CLSQL, I'm consistently losing TJG> my DB connection. This apparently happens when writing from fairly TJG> simultaneous threads. (Two threads may write to the same btree key at TJG> around the same time.) weird. could be broken Lisp implementation, i guess. (or maybe it has something to do with OS). can you please try our concurrency test suite, like this: (do-test-spec 'testthreads) it is supposed to work in postmodern. From eslick at media.mit.edu Thu Nov 20 23:38:15 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 20 Nov 2008 18:38:15 -0500 Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: References: Message-ID: <358361AA-7541-4296-8D30-FD03D47A5F75@media.mit.edu> with-open-store closes the store when you're done, null'ing out *store- controller*. With sufficient concurrency you are likely to get a race condition where the if stmt in with-city-store succeeds but the body fails since you aren't locking access to *store-controller*. Ian On Nov 20, 2008, at 11:16 AM, Tayssir John Gabbour wrote: > Hi! > > With Elephant under both Postmodern and CLSQL, I'm consistently losing > my DB connection. This apparently happens when writing from fairly > simultaneous threads. (Two threads may write to the same btree key at > around the same time.) > > All I do is add objects to a BTree, keyed by their UUID. (And there's > two simple indexes.) No persistent metaclasses or anything; I just use > it as a persistent table. > > Am I doing something terribly wrong? Code and error messages at the > end of this email. > > > All the best, > Tayssir > > > > ;;; How I add stuff to the DB > > (defmethod save-to-db (thing) > (with-city-store > (setf (get-value (moderator-ontology:uuid thing) (my-btree)) > thing))) > > (defmacro with-city-store (&body body) > `(if *store-controller* > (progn , at body) > (with-open-store (citycouncil-config:*citycouncil-store-spec*) > ;; removing with-transaction doesn't have much effect > (with-transaction () > , at body)))) > > (defun create-indexes (&key (btree (my-btree))) > (with-city-store > (add-index btree > :index-name *semantic-type-index-name* > :key-form '(lambda (secondary-db primary-key value) > (declare (ignore secondary-db primary-key)) > (values t (type->storage-location value)))) > (add-index btree > :populate t > :index-name *owner-index-name* > :key-form '(lambda (secondary-db primary-key value) > (declare (ignore secondary-db primary-key)) > (if (typep value 'moderator-ontology:icon) > (values t (moderator-ontology:owner > value)) > (values nil nil)))))) > > > > ;;; CLSQL messages > > > Condition CONTROLLER-LOST-ERROR was signalled. > [Condition of type CONTROLLER-LOST-ERROR] > > Restarts: > 0: [CONTINUE] Open a new instance and continue? > 1: [TERMINATE-THREAD] Terminate this thread (# "hunchentoot-worker-204" RUNNING {B90AFB9}>) > > Backtrace: > 0: ((SB-PCL::FAST-METHOD ELEPHANT::GET-CON (PERSISTENT)) # argument> # # #) > Locals: > ELEPHANT::INSTANCE = # > 1: ((SB-PCL::FAST-METHOD GET-VALUE (T DB-CLSQL::SQL-BTREE)) > # # :STORE-2008-06-24 > #) > Locals: > SB-DEBUG::ARG-0 = : > SB-DEBUG::ARG-1 = : > SB-DEBUG::ARG-2 = :STORE-2008-06-24 > SB-DEBUG::ARG-3 = # > 2: ((LAMBDA ())) > [No Locals] > 3: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION > (DB-CLSQL::SQL-STORE-CONTROLLER T)) ..)[:EXTERNAL] > Locals: > SB-DEBUG::ARG-0 = 8 > SB-DEBUG::ARG-1 = : > SB-DEBUG::ARG-2 = : > SB-DEBUG::ARG-3 = # elephant_test #1=griffie #1# ... > SB-DEBUG::ARG-4 = # > > > > There is no applicable method for the generic function > # > when called with arguments > (NIL). > [Condition of type SIMPLE-ERROR] > > Restarts: > 0: [TERMINATE-THREAD] Terminate this thread (# "hunchentoot-worker-204" RUNNING {B90AFB9}>) > > Backtrace: > 0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) # {ADF2D6D}>) > Locals: > SB-KERNEL:*HANDLER-CLUSTERS* = : > SWANK-BACKEND::*SLDB-STACK-TOP* = : > SWANK-BACKEND::DEBUGGER-LOOP-FN = # {ADF2D6D}> > 1: (SWANK::DEBUG-IN-EMACS #) > 2: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) # SWANK:SWANK-DEBUGGER-HOOK> #) > 3: (SWANK::CALL-WITH-REDIRECTED-IO # > #) > 4: (SWANK::CALL-WITH-CONNECTION # > #) > 5: (SWANK:INVOKE-SLIME-DEBUGGER #) > 6: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) # SWANK:SWANK-DEBUGGER-HOOK> #) > 7: (INVOKE-DEBUGGER #) > 8: (INVOKE-DEBUGGER #)[:EXTERNAL] > 9: ((SB-PCL::FAST-METHOD HUNCHENTOOT:MAYBE-INVOKE-DEBUGGER (T)) > # # # {B368FA9}>) > 10: (SIGNAL #)[:EXTERNAL] > 11: (ERROR "~@ ~2I~_~S~\n ~I~_when called with arguments > ~2I~_~S.~:>")[:EXTERNAL] > Locals: > SB-DEBUG::ARG-0 = 3 > SB-DEBUG::ARG-1 = "~@ generic function ~2I~_~S~\n .. > 12: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) # argument> # # DB-CLSQL::CONTROLLER-$ 13: (DB-CLSQL::SQL-GET-FROM-CLCN-NTH 0 > :STORE-2008-06-24 NIL 0) > 14: ((LAMBDA ())) > 15: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION > (DB-CLSQL::SQL-STORE-CONTROLLER T)) ..)[:EXTERNAL] > Locals: > SB-DEBUG::ARG-0 = 8 > SB-DEBUG::ARG-1 = : > SB-DEBUG::ARG-2 = : > SB-DEBUG::ARG-3 = # elephant_test #1=griffie #1# ... > SB-DEBUG::ARG-4 = # > 16: ((SB-PCL::FAST-METHOD LOCATE-ICON-BY-OWNER (STRING)) > # # > "bc3c7276-3a80-0780-23e1-40c33cd1b238") > > > > > > > > > ;;; Postmodern messages > > > > Database error: # is closed > [Condition of type CL-POSTGRES:DATABASE-SOCKET-ERROR] > > Restarts: > 0: [TERMINATE-THREAD] Terminate this thread (# "artificial-thread" RUNNING {BCA96B9}>) > > Backtrace: > 0: ((LAMBDA ())) > 1: (DB-POSTMODERN::POSTGRES-FORMAT # {BC9EED1}> :OBJECT) > 2: ((LAMBDA ())) > 3: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION > (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] > 4: ((SB-PCL::FAST-METHOD (SETF DB-POSTMODERN::INTERNAL-GET-VALUE) (T > T DB-POSTMODERN::PM-BTREE)) ..) > 5: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T > DB-POSTMODERN::PM-INDEXED-BTREE)) ..) > 6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION > (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] > 7: ((SB-PCL::FAST-METHOD ADD (MODERATOR-ONTOLOGY:UUID-MIXIN)) > # # > #) > > > > (# {BB053AD}>>) has a bad file descriptor. > [Condition of type SIMPLE-ERROR] > > Restarts: > 0: [REMOVE-THEM] Remove bogus handlers. > 1: [RETRY-THEM] Retry bogus handlers. > 2: [CONTINUE] Go on, leaving handlers marked as bogus. > 3: [TERMINATE-THREAD] Terminate this thread (# "artificial-thread" RUNNING {B9E68D9}>) > > Backtrace: > 0: (SB-IMPL::HANDLER-DESCRIPTORS-ERROR) > 1: (SB-IMPL::SUB-SUB-SERVE-EVENT NIL NIL) > 2: (SB-IMPL::SUB-SERVE-EVENT NIL NIL NIL) > 3: (SB-SYS:WAIT-UNTIL-FD-USABLE 10 :INPUT NIL) > 4: (SB-IMPL::REFILL-INPUT-BUFFER # socket" {B9F8AA1}>) > 5: (SB-IMPL::INPUT-UNSIGNED-8BIT-BYTE # socket" {B9F8AA1}> T NIL) > 6: (CL-POSTGRES::READ-UINT1 #) > 7: (CL-POSTGRES::SEND-EXECUTE # {B9F8AA1}> # # > #) > 8: ((LABELS #:G303)) > 9: ((LAMBDA ())) > 10: (DB-POSTMODERN::POSTGRES-FORMAT # {B9D1459}> :OBJECT) > 11: ((LAMBDA ())) > 12: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION > (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] > 13: ((SB-PCL::FAST-METHOD (SETF DB-POSTMODERN::INTERNAL-GET-VALUE) (T > T DB-POSTMODERN::PM-BTREE)) ..) > 14: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T > DB-POSTMODERN::PM-INDEXED-BTREE)) ..) > 15: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION > (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] > 16: ((SB-PCL::FAST-METHOD ADD (MODERATOR-ONTOLOGY:UUID-MIXIN)) > # # > #) > > > > Database error: Connection to database server lost. > [Condition of type CL-POSTGRES:DATABASE-CONNECTION-LOST] > > Restarts: > 0: [RECONNECT] Try to reconnect. > 1: [TERMINATE-THREAD] Terminate this thread (# "artificial-thread" RUNNING {ACBD271}>) > > Backtrace: > 0: (CL-POSTGRES::ENSURE-CONNECTION # {AD297E9}>) > 1: (CL-POSTGRES:EXEC-QUERY # {AD297E9}> # CL-POSTGRES:IGNORE-ROW-READER) > 2: (POSTMODERN:ABORT-TRANSACTION # {B221ED9}>) > 3: ((SB-PCL::FAST-METHOD CONTROLLER-ABORT-TRANSACTION > (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] > 4: ((FLET #:CLEANUP-FUN-[EXECUTE-TRANSACTION]23))[:CLEANUP] > 5: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION > (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T)) ..)[:EXTERNAL] > 6: ((SB-PCL::FAST-METHOD ADD (MODERATOR-ONTOLOGY:UUID-MIXIN)) > # # > #) > 7: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) > > > Condition CONTROLLER-LOST-ERROR was signalled. > [Condition of type CONTROLLER-LOST-ERROR] > > Restarts: > 0: [CONTINUE] Open a new instance and continue? > 1: [TERMINATE-THREAD] Terminate this thread (# "artificial-thread" RUNNING {B7F38D9}>) > > Backtrace: > 0: ((SB-PCL::FAST-METHOD ELEPHANT::GET-CON (PERSISTENT)) # printing object>) > 1: (ELEPHANT::INITIALIZE-PERSISTENT-SLOTS # > # # # argument> # :AROUND (PERSISTENT-OBJECT T)) # # argument> # # SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. SB-INT:&MORE > SB-PCL::.DFUN-MORE-CONTEXT. SB-PCL::.DFUN-MORE-COUNT.)) # 4: ((SB-PCL::FAST-METHOD MAKE-INSTANCE (CLASS)) # argument> # # DB-POSTMODERN::PM-INDEXED-BTREE$ 5: ((FLET DB-POSTMODERN::INIT-ROOT)) > 6: ((SB-PCL::FAST-METHOD ELEPHANT::OPEN-CONTROLLER > (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER)) ..)[:EXTERNAL] > 7: ((SB-PCL::FAST-METHOD ELEPHANT::OPEN-CONTROLLER :AROUND > (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER)) ..)[:EXTERNAL] > 8: (OPEN-STORE (:POSTMODERN (:POSTGRESQL "localhost" #1="griffie" > #1# #1# :PORT ...)))[:EXTERNAL] > 9: ((SB-PCL::FAST-METHOD ADD (MODERATOR-ONTOLOGY:UUID-MIXIN)) > # # > #) > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From tjg at pentaside.org Fri Nov 21 09:58:52 2008 From: tjg at pentaside.org (Tayssir John Gabbour) Date: Fri, 21 Nov 2008 10:58:52 +0100 Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: References: Message-ID: Hi Ian, The problem is, suppose I just open a store controller at the very beginning. If I execute either of the following: (sb-thread:make-thread (lambda () (get-value "foobar" (get-from-root *my-btree-name* :sc *my-store-controller*)))) (sb-thread:make-thread (lambda () (get-value "foobar" *my-btree*))) there's a new connection in the database, never dying and eventually overflowing Postgresql's limit. (That is, "select count(*) from pg_stat_activity;" has a new row each time). A couple DB people at work looked at the ele-postmodern code, and they say they don't see the connection being closed down. I must be doing something wrong, but I can't see where... All the best, Tayssir Ian Eslick wrote: > with-open-store closes the store when you're done, null'ing out *store- > controller*. With sufficient concurrency you are likely to get a race > condition where the if stmt in with-city-store succeeds but the body > fails since you aren't locking access to *store-controller*. > > Ian > > > On Nov 20, 2008, at 11:16 AM, Tayssir John Gabbour wrote: >> Hi! >> >> With Elephant under both Postmodern and CLSQL, I'm consistently losing >> my DB connection. This apparently happens when writing from fairly >> simultaneous threads. (Two threads may write to the same btree key at >> around the same time.) >> >> All I do is add objects to a BTree, keyed by their UUID. (And there's >> two simple indexes.) No persistent metaclasses or anything; I just use >> it as a persistent table. >> >> Am I doing something terribly wrong? Code and error messages at the >> end of this email. From tjg at pentaside.org Fri Nov 21 10:02:21 2008 From: tjg at pentaside.org (Tayssir John Gabbour) Date: Fri, 21 Nov 2008 11:02:21 +0100 Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: References: Message-ID: Alex Mizrahi wrote: > TJG> With Elephant under both Postmodern and CLSQL, I'm consistently losing > TJG> my DB connection. This apparently happens when writing from fairly > TJG> simultaneous threads. (Two threads may write to the same btree key at > TJG> around the same time.) > > weird. could be broken Lisp implementation, i guess. > (or maybe it has something to do with OS). It happened under Debian (Etch) and FreeBSD, on 3 different computers. Running SBCLs in the range of (IIRC) 1.0.12 - 1.0.20. > can you please try our concurrency test suite, like this: > (do-test-spec 'testthreads) > it is supposed to work in postmodern. Cool, I tried it out: foo-user> (asdf:oos 'asdf:load-op :elephant-tests) ... foo-user> (ele-tests::do-test-spec 'testthreads) NIL I guess that's good. ;) All the best, Tayssir From killerstorm at newmail.ru Fri Nov 21 10:49:14 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Fri, 21 Nov 2008 12:49:14 +0200 Subject: [elephant-devel] Dropping DB connection during writes References: <358361AA-7541-4296-8D30-FD03D47A5F75@media.mit.edu> Message-ID: IE> with-open-store closes the store when you're done, null'ing out *store- IE> controller*. With sufficient concurrency you are likely to get a race IE> condition where the if stmt in with-city-store succeeds but the body IE> fails since you aren't locking access to *store-controller*. well, it seems the root cause of the problem is that open-store uses cached controller instance, so multiple threads will share same instance, and when one closes it others get botched. maybe it makes sense to do some changes to prevent this. like, do no caching? From sky at viridian-project.de Fri Nov 21 10:53:20 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Fri, 21 Nov 2008 11:53:20 +0100 (CET) Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: References: Message-ID: <63680.88.73.214.232.1227264800.squirrel@mail.stardawn.org> > there's a new connection in the database, never dying and eventually > overflowing Postgresql's limit. (That is, "select count(*) from > pg_stat_activity;" has a new row each time). A couple DB people at > work looked at the ele-postmodern code, and they say they don't see > the connection being closed down. What version of Elephant do you use? Postmodern doesn't clean up its connections in 0.9.1. Upgrade to the latest stable Darcs and it should be fine. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From sky at viridian-project.de Fri Nov 21 11:59:58 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Fri, 21 Nov 2008 12:59:58 +0100 Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: References: Message-ID: <20081121115958.GA376@viridian-project.de> On Fri, Nov 21, 2008 at 11:02:21AM +0100, Tayssir John Gabbour wrote: > > foo-user> (asdf:oos 'asdf:load-op :elephant-tests) > ... > > foo-user> (ele-tests::do-test-spec 'testthreads) > NIL > > > I guess that's good. ;) In fact this should output a lot of stuff. How long does it take until returning? Leslie From tjg at pentaside.org Fri Nov 21 13:41:17 2008 From: tjg at pentaside.org (Tayssir John Gabbour) Date: Fri, 21 Nov 2008 14:41:17 +0100 Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: <20081121115958.GA376@viridian-project.de> References: <20081121115958.GA376@viridian-project.de> Message-ID: Tayssir John Gabbour wrote: > Leslie P. Polzer wrote: > > On Fri, Nov 21, 2008 at 11:02:21AM +0100, Tayssir John Gabbour wrote: > >> foo-user> (asdf:oos 'asdf:load-op :elephant-tests) > >> ... > >> foo-user> (ele-tests::do-test-spec 'testthreads) > >> NIL > > > > In fact this should output a lot of stuff. > > How long does it take until returning? > > Unfortunately, no time at all -- TIME consistently tells me 0.000 > seconds, in all fields. And it just returns NIL. Incidentally, that zero-time result is under: SBCL 1.0.22 Debian Etch Linux 2.6.18-6-686 just did "darcs pull" on Elephant Postmodern 1.13 PostgreSQL 8.1.13 All the best, Tayssir From tjg at pentaside.org Fri Nov 21 13:24:56 2008 From: tjg at pentaside.org (Tayssir John Gabbour) Date: Fri, 21 Nov 2008 14:24:56 +0100 Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: <20081121115958.GA376@viridian-project.de> References: <20081121115958.GA376@viridian-project.de> Message-ID: <4926B6A8.5080308@pentaside.org> Leslie P. Polzer wrote: > On Fri, Nov 21, 2008 at 11:02:21AM +0100, Tayssir John Gabbour wrote: >> foo-user> (asdf:oos 'asdf:load-op :elephant-tests) >> ... >> foo-user> (ele-tests::do-test-spec 'testthreads) >> NIL > > In fact this should output a lot of stuff. > How long does it take until returning? Unfortunately, no time at all -- TIME consistently tells me 0.000 seconds, in all fields. And it just returns NIL. Incidentally, I pulled the latest stable as you mentioned (using "darcs pull") and so far it seems that Elephant/Postmodern works fine! Haven't tested CLSQL. (So, I have just one store-controller open over the app lifetime. No weird races or dropped connections seem to happen. And the PostgreSQL connections stay at a low level.) Thanks Alex, Ian and Leslie, Tayssir From eslick at media.mit.edu Fri Nov 21 19:11:19 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 21 Nov 2008 14:11:19 -0500 Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: References: <358361AA-7541-4296-8D30-FD03D47A5F75@media.mit.edu> Message-ID: <7455B1A1-BCD6-4513-85E0-53DDBA4C2700@media.mit.edu> What interface do you use to deal with opening/closing separate connections in your system? You want to use the same handles in the BDB backend and I don't think we want to have different interfaces for different backends. There is some code for doing this automatically by looking at unique thread ids in the CL-SQL backend, I believe, that keeps the same interface. On Nov 21, 2008, at 5:49 AM, Alex Mizrahi wrote: > IE> with-open-store closes the store when you're done, null'ing out > *store- > IE> controller*. With sufficient concurrency you are likely to get > a race > IE> condition where the if stmt in with-city-store succeeds but the > body > IE> fails since you aren't locking access to *store-controller*. > > well, it seems the root cause of the problem is that open-store uses > cached controller instance, so multiple threads will share same > instance, > and when one closes it others get botched. > > maybe it makes sense to do some changes to prevent this. like, do no > caching? > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Fri Nov 21 19:14:22 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 21 Nov 2008 14:14:22 -0500 Subject: [elephant-devel] Bug fixes for elephant-unstable Message-ID: I think my prior e-mail on this topic failed to make it to the list... I checked in some patches this week to elephant-unstable: - Fixed (get-universal-time) issue reported earlier caused by error in bignum comparisons under BDB - Fixed test failures under BDB/Linux/SBCL - all green for me now. - Fixed some indexing and mapping bugs Associations still have some problems. I may have screwed up Leslie's new deadlock handling code - Leslie, could you check this out? Cheers, Ian From rme at clozure.com Fri Nov 21 20:31:03 2008 From: rme at clozure.com (R. Matthew Emerson) Date: Fri, 21 Nov 2008 15:31:03 -0500 Subject: [elephant-devel] ClozureCL Intel 64 bit problem In-Reply-To: <333e2fc20811171957q1822054dj4b03bdbd10ce03f7@mail.gmail.com> (Wesley Kerr's message of "Mon\, 17 Nov 2008 20\:57\:27 -0700") References: <333e2fc20811171957q1822054dj4b03bdbd10ce03f7@mail.gmail.com> Message-ID: wkerr at cs.arizona.edu (Wesley Kerr) writes: > I've been reading through the archives and back in June Patrick had a > problem installing onto a 64 bit PPC machine with ClozureCL. At one > point he was getting some errors that looked similar to this: > > Undefined function #:G6340 called with arguments (# #x804E00> # # 2 10 > # 0 10 0 # [stack-allocated] #xB0502E60> ..) [Condition of type > CCL::UNDEFINED-FUNCTION-CALL] > > Ian mentioned that he would contact the developers of ClozureCL and > see if they could help debug the problem since it was unclear what was > causing it. I'm now getting these errors on a 64bit intel and I'm not > sure where to even begin. Ian forwarded your note to me. At the top of src/db-bdb/berkeley-db.lisp is a declaim form that declaims a bunch of functions inline. (declaim #-elephant-without-optimize (optimize (speed 3) (safety 1) (debug 1) (space 1)) #-lispworks (inline %db-get-key-buffered db-get-key-buffered %db-get-buffered db-get-buffered db-get %db-put-buffered db-put-buffered ...)) Make that #-lispworks be #-(or lispworks ccl) and that should get you going. Long explanation: Consider the following DEF-FUNCTION form: (def-function ("db_get_raw" %db-get-key-buffered) ((db :pointer-void) (txn :pointer-void) ... (result-size :unsigned-int :out)) :returning :int) This macroexpands into something like (progn (defun #:G0 (arg0 arg1 ...) ...) (defun %db-get-key-buffered () (values (#:G0 arg0 arg1 ...) ...))) If %DB-GET-KEY-BUFFERED is declaimed INLINE, we save its lambda expression, which includes the gensym. When the lambda expression in substituted into a caller defined in some other file, it won't be referencing the same #:G0, and you get undefined function error. (Inlining across files used to be disabled, at least until several months ago, so if this cropped up recently, that might be why.) [from Gary Byers:] SBCL seems to handle a toplevel (DECLAIM (INLINE FOO)) (or the combination of the declamation and a subsequent DEFUN) as something like: (eval-when (:compile-toplevel) (save-lambda-definition (lambda-definition-for 'foo) *env*)) (eval-when (:load-toplevel) (save-global-lambda-definition (preprocessed-definition-for foo))) where the "preprocessed definition" appears to be fully macroexpanded (doesn't contain references to macros that may have been defined at compile time but might not be defined at load time.) That could be nice functionality to have, but there's no way that the load-time inline-expansion and the compile-time inline-expansion can be equivalent when uninterned symbols are involved. Once we've crossed a point where we can't guarantee that, I think that we're pretty much into an area where code is depending on non-portable implementation artifacts. Hope this helps. From ludwig at fh-worms.de Fri Nov 21 22:22:23 2008 From: ludwig at fh-worms.de (Christoph Ludwig) Date: Fri, 21 Nov 2008 23:22:23 +0100 Subject: [elephant-devel] Bug fixes for elephant-unstable In-Reply-To: References: Message-ID: <20081121222222.GA3243@castellio.local> On Fri, Nov 21, 2008 at 02:14:22PM -0500, Ian Eslick wrote: > I think my prior e-mail on this topic failed to make it to the list... > > I checked in some patches this week to elephant-unstable: > - Fixed (get-universal-time) issue reported earlier caused by error in > bignum comparisons under BDB > - Fixed test failures under BDB/Linux/SBCL - all green for me now. > - Fixed some indexing and mapping bugs > > Associations still have some problems. I may have screwed up Leslie's > new deadlock handling code - Leslie, could you check this out? Do you mean association problems as in missing features or as in buggy behaviour? We use associations in our project and, in general, things work fine and, after we fixed our algorithms, reasonably fast. However, we observed sporadic corruptions when we put a lot of objects into the store. we could not trace the corruptions back to any particular elephant feature yet; so if you already know a part of the code that is a likely source of bugs, we'd appreciate if you let us know. Regards Christoph -- FH Worms - University of Applied Sciences Fachbereich Informatik / Telekommunikation Erenburgerstr. 19, 67549 Worms, Germany From killerstorm at newmail.ru Fri Nov 21 22:30:10 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 22 Nov 2008 00:30:10 +0200 Subject: [elephant-devel] Dropping DB connection during writes References: <358361AA-7541-4296-8D30-FD03D47A5F75@media.mit.edu> <7455B1A1-BCD6-4513-85E0-53DDBA4C2700@media.mit.edu> Message-ID: IE> What interface do you use to deal with opening/closing separate IE> connections in your system? connection management is not a problem. problem is that close-store unconditionally makes controller totally unusable, and that might cause problem if there are some with-open-store commands going concurrently. if we're not going to create separate controller for each open-store, maybe it makes sense to add sort of refcounting, either on with-open-store on on open-store/close-store level. so each open-store should be matched with corresponding close-store, and controller is only finally closed when there are no references to it. i.e. if you call open-store three times (either in one thread or in many), first two close-stores will do nothing, and third one will close it. i've seen such "technique" being implemented in many APIs that are dealing with initialization/deinitialization, btw.. but as nobody complains (Tayssir had other problem), this is probably low-priority thing.. might save someone from WTF moments in future, maybe.. From eslick at media.mit.edu Sat Nov 22 00:03:02 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 21 Nov 2008 19:03:02 -0500 Subject: [elephant-devel] Dropping DB connection during writes In-Reply-To: References: <358361AA-7541-4296-8D30-FD03D47A5F75@media.mit.edu> <7455B1A1-BCD6-4513-85E0-53DDBA4C2700@media.mit.edu> Message-ID: <9DE05FAB-0CBB-487B-AE55-9E8721474FA3@media.mit.edu> For now the easiest approach is to define-away the problem. with-open-store is a convenience function for interactive development and testing. It should not be used in production, a multi-store or multi-threaded program. Opening and closing a store for small operations comes with a significant performance penalty and is not recommended! :) On Nov 21, 2008, at 5:30 PM, Alex Mizrahi wrote: > IE> What interface do you use to deal with opening/closing separate > IE> connections in your system? > > connection management is not a problem. > > problem is that close-store unconditionally makes controller totally > unusable, and that might cause problem if there are some with-open- > store > commands going concurrently. > > if we're not going to create separate controller for each open-store, > maybe it makes sense to add sort of refcounting, either on with-open- > store > on on open-store/close-store level. so each open-store should be > matched > with corresponding close-store, and controller is only finally > closed when > there are no references to it. i.e. if you call open-store three times > (either > in one thread or in many), first two close-stores will do nothing, > and third > one will close it. > > i've seen such "technique" being implemented in many APIs that are > dealing with initialization/deinitialization, btw.. > > but as nobody complains (Tayssir had other problem), this is probably > low-priority thing.. might save someone from WTF moments in future, > maybe.. > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel