From franks-muc at web.de Sat Aug 5 23:53:50 2006 From: franks-muc at web.de (franks-muc at web.de) Date: Sun, 06 Aug 2006 01:53:50 +0200 Subject: [elephant-devel] strings as Class Indices ? Message-ID: <156501808@web.de> I wonder if it is possible to use strings in class indices. The example from the manual using integers works well on my computer (windows,acl-trial,elephant,postgres). I modified this example to use strings: (defclass idx-four () ((slot1 :initarg :slot1 :accessor slot1 :index t)) (:metaclass persistent-metaclass)) (defun make-idx-four (val) (make-instance 'idx-four :slot1 val)) (with-transaction () (mapc #'make-idx-four '("1" "1" "1" "2" "2" "4" "5" "5" "5" "6" "10"))) (get-instances-by-value 'idx-four 'slot1 "1") ==> NIL I suspect that nil is not the desired result since other examples provided for elephant suggest that strings should work (see (get-instance-by-value 'user 'name name) in http://common-lisp.net/pipermail/elephant-devel/2006-July/000477.html). Could someone please explain where I am wrong ? Thank you in advance! Frank Schorr _____________________________________________________________________ Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! http://smartsurfer.web.de/?mc=100071&distributionid=000000000071 From eslick at csail.mit.edu Sun Aug 6 19:13:14 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 06 Aug 2006 15:13:14 -0400 Subject: [elephant-devel] strings as Class Indices ? In-Reply-To: <156501808@web.de> References: <156501808@web.de> Message-ID: <44D63F4A.8050501@csail.mit.edu> That's interesting, that code works fine on my system. Have you run the full Elephant regression tests on clisp? Did they all pass? Regards, Ian franks-muc at web.de wrote: > I wonder if it is possible to use strings in class indices. The example from the manual > using integers works well on my computer (windows,acl-trial,elephant,postgres). > I modified this example to use strings: > > (defclass idx-four () > ((slot1 :initarg :slot1 :accessor slot1 :index t)) > (:metaclass persistent-metaclass)) > > > (defun make-idx-four (val) > (make-instance 'idx-four :slot1 val)) > > (with-transaction () > (mapc #'make-idx-four '("1" "1" "1" "2" "2" "4" "5" "5" "5" "6" "10"))) > > (get-instances-by-value 'idx-four 'slot1 "1") > ==> NIL > > I suspect that nil is not the desired result since other examples provided > for elephant suggest that strings should work (see (get-instance-by-value 'user 'name name) > in http://common-lisp.net/pipermail/elephant-devel/2006-July/000477.html). > > Could someone please explain where I am wrong ? > Thank you in advance! > Frank Schorr > > > _____________________________________________________________________ > Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! > http://smartsurfer.web.de/?mc=100071&distributionid=000000000071 > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From franks-muc at web.de Sun Aug 6 21:13:12 2006 From: franks-muc at web.de (franks-muc at web.de) Date: Sun, 06 Aug 2006 23:13:12 +0200 Subject: [elephant-devel] strings as Class Indices ? Message-ID: <157186952@web.de> Thank you for the reply. This is still on ACL7trial, where elephant works fine and already stores useful data, except for class indexing, which I tried to introduce now. Up to now I'm iterating through the main btree stored in root to search by slot value. I'm also using the latest CVS version. I made no progress in clisp. The result of the tests (in ACL) is pasted below. CG-USER(135): (setf ele-tests::*default-spec* '(:clsql (:postgresql-socket "localhost" "clsql-tests" "postgres" "$postgres%"))) (:CLSQL (:POSTGRESQL-SOCKET "localhost" "clsql-tests" "postgres" "$postgres%")) CG-USER(136): (ele-tests::do-backend-tests) Doing 111 pending tests of 111 tests total. ELEPHANT-TESTS::FIXNUMS ELEPHANT-TESTS::FIXNUM-TYPE-1 ELEPHANT-TESTS::BIGNUMS ELEPHANT-TESTS::FLOATS ELEPHANT-TESTS::RATIONALS ELEPHANT-TESTS::BASE-STRINGS ELEPHANT-TESTS::STRINGS ELEPHANT-TESTS::SYMBOLS ELEPHANT-TESTS::CHARS ELEPHANT-TESTS::PATHNAMES ELEPHANT-TESTS::CONSES ELEPHANT-TESTS::HASH-TABLES-1 ELEPHANT-TESTS::HASH-TABLES-2 ELEPHANT-TESTS::ARRAYS-1 ELEPHANT-TESTS::ARRAYS-2 ELEPHANT-TESTS::TEST-DEEP-EQUALP ELEPHANT-TESTS::OBJECTS ELEPHANT-TESTS::CIRCULAR ELEPHANT:PERSISTENT ELEPHANT-TESTS::NON-TRANSIENT-CLASS-SLOT-1 ELEPHANT-TESTS::NON-TRANSIENT-CLASS-SLOT-2 ELEPHANT-TESTS::TRANSIENT-CLASS-SLOT ELEPHANT-TESTS::CLASS-DEFINERS ELEPHANT-TESTS::BAD-INHERITENCE ELEPHANT-TESTS::MIXES ELEPHANT-TESTS::MIXES-RIGHT-SLOTS ELEPHANT-TESTS::INHERIT ELEPHANT-TESTS::INHERIT-RIGHT-SLOTS ELEPHANT-TESTS::INITFORM-CLASSES ELEPHANT-TESTS::INITFORM-TEST ELEPHANT-TESTS::INITARG-TEST ELEPHANT-TESTS::NO-EVAL-INITFORM ELEPHANT-TESTS::REDEFCLASS MAKUNBOUND ELEPHANT-TESTS::UPDATE-CLASS CHANGE-CLASS ELEPHANT-TESTS::CHANGE-CLASS3 ELEPHANT-TESTS::BASICPERSISTENCE ELEPHANT-TESTS::TESTOID ELEPHANT-TESTS::BTREE-MAKE ELEPHANT-TESTS::BTREE-PUT ELEPHANT-TESTS::BTREE-GET ELEPHANT:REMOVE-KV ELEPHANT-TESTS::REMOVED ELEPHANT:MAP-BTREE ELEPHANT-TESTS::INDEXED-BTREE-MAKE ELEPHANT-TESTS::ADD-INDICES ELEPHANT-TESTS::TEST-INDICES ELEPHANT-TESTS::INDEXED-PUT ELEPHANT-TESTS::INDEXED-GET ELEPHANT-TESTS::SIMPLE-SLOT-GET ELEPHANT-TESTS::INDEXED-GET-FROM-SLOT1 ELEPHANT-TESTS::INDEXED-GET-FROM-SLOT2 ELEPHANT-TESTS::REMOVE-KV-INDEXED ELEPHANT-TESTS::NO-KEY-NOR-INDICES ELEPHANT-TESTS::REMOVE-KV-FROM-SLOT1 ELEPHANT-TESTS::NO-KEY-NOR-INDICES-SLOT1 ELEPHANT-TESTS::REMOVE-KV-FROM-SLOT2 ELEPHANT-TESTS::NO-KEY-NOR-INDICES-SLOT2 ELEPHANT-TESTS::MAP-INDEXED ELEPHANT-TESTS::GET-FIRST ELEPHANT-TESTS::GET-FIRST2 ELEPHANT-TESTS::GET-LAST ELEPHANT-TESTS::GET-LAST2 SET ELEPHANT-TESTS::SET2 ELEPHANT-TESTS::SET-RANGE ELEPHANT-TESTS::SET-RANGE2 ELEPHANT-TESTS::REM-KV ELEPHANT-TESTS::REM-IDEXKV ELEPHANT-TESTS::MAKE-INDEXED2 ELEPHANT-TESTS::ADD-INDICES2 ELEPHANT-TESTS::PUT-INDEXED2 ELEPHANT-TESTS::GET-INDEXED2 ELEPHANT-TESTS::GET-FROM-INDEX3 ELEPHANT-TESTS::DUP-TEST ELEPHANT-TESTS::NODUP-TEST ELEPHANT-TESTS::PREV-NODUP-TEST ELEPHANT-TESTS::PNODUP-TEST ELEPHANT-TESTS::PPREV-NODUP-TEST ELEPHANT-TESTS::CUR-DEL1 ELEPHANT-TESTS::INDEXED-DELETE ELEPHANT-TESTS::TEST-DELETED Test ELEPHANT-TESTS::INDEXED-DELETE2 failed Form: (ELEPHANT-TESTS::FINISHES (ELEPHANT:WITH-TRANSACTION (:STORE-CONTROLLER ELEPHANT:*STORE-CONTROLLER*) (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::CURS ELEPHANT-TESTS::INDEX3) (ELEPHANT:CURSOR-FIRST ELEPHANT-TESTS::CURS) (ELEPHANT:CURSOR-NEXT-DUP ELEPHANT-TESTS::CURS) (ELEPHANT:CURSOR-DELETE ELEPHANT-TESTS::CURS)))) Expected value: T Actual value: NIL. Test ELEPHANT-TESTS::TEST-DELETED2 failed Form: (VALUES (ELEPHANT:GET-VALUE 0 ELEPHANT-TESTS::INDEXED2) (ELEPHANT:GET-VALUE 0 ELEPHANT-TESTS::INDEX3) (ELEPHANT:GET-VALUE 1 ELEPHANT-TESTS::INDEXED2) (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::C ELEPHANT-TESTS::INDEX3) (ELEPHANT:CURSOR-FIRST ELEPHANT-TESTS::C) (MULTIPLE-VALUE-BIND (ELEPHANT-TESTS::M ELEPHANT-TESTS::K ELEPHANT-TESTS::V) (ELEPHANT:CURSOR-NEXT ELEPHANT-TESTS::C) ELEPHANT-TESTS::V))) Expected values: 0 0 NIL -2 Actual values: 0 0 -1 -1. Test ELEPHANT-TESTS::CUR-DEL2 failed Form: (ELEPHANT:WITH-TRANSACTION (:STORE-CONTROLLER ELEPHANT:*STORE-CONTROLLER*) (LET* ((ELEPHANT-TESTS::IBT (ELEPHANT:MAKE-INDEXED-BTREE ELEPHANT:*STORE-CONTROLLER*)) (ELEPHANT-TESTS::ID1 (ELEPHANT:ADD-INDEX ELEPHANT-TESTS::IBT :INDEX-NAME 'ELEPHANT-TESTS::IDX1 :KEY-FORM 'ELEPHANT-TESTS::ODD))) (LOOP ELEPHANT-TESTS::FOR ELEPHANT-TESTS::I ELEPHANT-TESTS::FROM 0 ELEPHANT-TESTS::TO 10 DO (SETF (ELEPHANT:GET-VALUE ELEPHANT-TESTS::I ELEPHANT-TESTS::IBT) (* ELEPHANT-TESTS::I ELEPHANT-TESTS::I))) (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::C ELEPHANT-TESTS::ID1) (ELEPHANT:CURSOR-FIRST ELEPHANT-TESTS::C) (ELEPHANT:CURSOR-NEXT-DUP ELEPHANT-TESTS::C) (ELEPHANT:CURSOR-DELETE ELEPHANT-TESTS::C)) (EQUAL (LIST (ELEPHANT:GET-VALUE 1 ELEPHANT-TESTS::ID1) (ELEPHANT:GET-VALUE 0 ELEPHANT-TESTS::ID1)) '(1 0)))) Expected value: T Actual value: #. ELEPHANT-TESTS::GET-BOTH ELEPHANT-TESTS::PGET-BOTH ELEPHANT-TESTS::PGET-BOTH-RANGE Test ELEPHANT-TESTS::PCURSOR failed Form: (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::C ELEPHANT-TESTS::INDEX3) (VALUES (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PFIRST ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT-NODUP ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT-DUP ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PPREV ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PPREV-NODUP ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PLAST ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PSET ELEPHANT-TESTS::C 300)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PSET-RANGE ELEPHANT-TESTS::C 199.5)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PGET-BOTH ELEPHANT-TESTS::C 10 101)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PGET-BOTH-RANGE ELEPHANT-TESTS::C 11 111.4)))) Expected values: 0 2 10 11 10 9 9999 3000 2000 101 112 Actual values: 0 1 10 11 10 9 9999 3000 2000 101 112. ELEPHANT-TESTS::NEWINDEX Test ELEPHANT-TESTS::PCURSOR2 failed Form: (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::C ELEPHANT-TESTS::INDEX4) (VALUES (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PFIRST ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT-NODUP ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT-DUP ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PPREV ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PPREV-NODUP ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PLAST ELEPHANT-TESTS::C)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PSET ELEPHANT-TESTS::C 300)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PSET-RANGE ELEPHANT-TESTS::C 199.5)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PGET-BOTH ELEPHANT-TESTS::C 10 101)) (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PGET-BOTH-RANGE ELEPHANT-TESTS::C 11 111.4)))) Expected values: 0 2 10 11 10 9 9999 3000 2000 101 112 Actual values: 0 1 10 11 10 9 9999 3000 2000 101 112. ELEPHANT-TESTS::ADD-GET-REMOVE ELEPHANT-TESTS::ADD-GET-REMOVE-SYMBOL ELEPHANT-TESTS::EXISTSP Warning: Manually finalizing class IDX-ONE ELEPHANT-TESTS::DISABLE-CLASS-INDEXING-TEST ELEPHANT-TESTS::INDEXING-BASIC-TRIVIAL ELEPHANT-TESTS::INDEXING-BASIC ELEPHANT-TESTS::INDEXING-INHERIT ELEPHANT-TESTS::INDEXING-RANGE ELEPHANT-TESTS::INDEXING-WIPE-INDEX ELEPHANT-TESTS::INDEXING-RECONNECT-DB ELEPHANT-TESTS::INDEXING-CHANGE-CLASS ELEPHANT-TESTS::INDEXING-REDEF-CLASS Ranged get of 40/500 objects = Linear: 25.937 sec Indexed: 14.0 sec ELEPHANT-TESTS::INDEXING-TIMING Single store mode: ignoring ELEPHANT-TESTS::REMOVE-ELEMENT Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-BASIC Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-BTREE Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-IDX-BTREE Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-PCLASS Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-IPCLASS 5 out of 111 total tests failed: ELEPHANT-TESTS::INDEXED-DELETE2, ELEPHANT-TESTS::TEST-DELETED2, ELEPHANT-TESTS::CUR-DEL2, ELEPHANT-TESTS::PCURSOR, ELEPHANT-TESTS::PCURSOR2. NIL CG-USER(137): > -----Urspr?ngliche Nachricht----- > Von: eslick at media.mit.edu, Elephant bugs and development > Gesendet: 06.08.06 21:24:03 > An: Elephant bugs and development > Betreff: Re: [elephant-devel] strings as Class Indices ? > That's interesting, that code works fine on my system. > > Have you run the full Elephant regression tests on clisp? Did they all > pass? > > Regards, > Ian > > franks-muc at web.de wrote: > > I wonder if it is possible to use strings in class indices. The example from the manual > > using integers works well on my computer (windows,acl-trial,elephant,postgres). > > I modified this example to use strings: > > > > (defclass idx-four () > > ((slot1 :initarg :slot1 :accessor slot1 :index t)) > > (:metaclass persistent-metaclass)) > > > > > > (defun make-idx-four (val) > > (make-instance 'idx-four :slot1 val)) > > > > (with-transaction () > > (mapc #'make-idx-four '("1" "1" "1" "2" "2" "4" "5" "5" "5" "6" "10"))) > > > > (get-instances-by-value 'idx-four 'slot1 "1") > > ==> NIL > > > > I suspect that nil is not the desired result since other examples provided > > for elephant suggest that strings should work (see (get-instance-by-value 'user 'name name) > > in http://common-lisp.net/pipermail/elephant-devel/2006-July/000477.html). > > > > Could someone please explain where I am wrong ? > > Thank you in advance! > > Frank Schorr > > > > > > _____________________________________________________________________ > > Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! > > http://smartsurfer.web.de/?mc=100071&distributionid=000000000071 > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel ______________________________________________________________________ XXL-Speicher, PC-Virenschutz, Spartarife & mehr: Nur im WEB.DE Club! Jetzt gratis testen! http://freemail.web.de/home/landingpad/?mc=021130 From eslick at csail.mit.edu Mon Aug 7 23:36:04 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 07 Aug 2006 19:36:04 -0400 Subject: [elephant-devel] strings as Class Indices ? In-Reply-To: <157186952@web.de> References: <157186952@web.de> Message-ID: <44D7CE64.7070006@csail.mit.edu> The website should probably be updated, but the CVS HEAD is not guaranteed to be stable. Some of the tests have failed on my local copy that I'll have to track down. I've been mucking with the serializer. If you use the 0.6.0 release you should be fine but I believe that the local CVS HEAD is incompatible with 0.6.0 under Allegro so you'd have to reconstitute your database. If this is a problem for you, let me know and I'll think about whether there is another way to go. Ian franks-muc at web.de wrote: > Thank you for the reply. > This is still on ACL7trial, where elephant works fine and already stores useful data, > except for class indexing, which I tried to introduce now. Up to now I'm iterating through > the main btree stored in root to search by slot value. I'm also using the latest CVS version. > > I made no progress in clisp. > > The result of the tests (in ACL) is pasted below. > > CG-USER(135): (setf ele-tests::*default-spec* '(:clsql (:postgresql-socket "localhost" "clsql-tests" "postgres" "$postgres%"))) > (:CLSQL (:POSTGRESQL-SOCKET "localhost" "clsql-tests" "postgres" "$postgres%")) > CG-USER(136): (ele-tests::do-backend-tests) > Doing 111 pending tests of 111 tests total. > ELEPHANT-TESTS::FIXNUMS ELEPHANT-TESTS::FIXNUM-TYPE-1 ELEPHANT-TESTS::BIGNUMS ELEPHANT-TESTS::FLOATS ELEPHANT-TESTS::RATIONALS > ELEPHANT-TESTS::BASE-STRINGS ELEPHANT-TESTS::STRINGS ELEPHANT-TESTS::SYMBOLS ELEPHANT-TESTS::CHARS ELEPHANT-TESTS::PATHNAMES > ELEPHANT-TESTS::CONSES ELEPHANT-TESTS::HASH-TABLES-1 ELEPHANT-TESTS::HASH-TABLES-2 ELEPHANT-TESTS::ARRAYS-1 ELEPHANT-TESTS::ARRAYS-2 > ELEPHANT-TESTS::TEST-DEEP-EQUALP ELEPHANT-TESTS::OBJECTS ELEPHANT-TESTS::CIRCULAR ELEPHANT:PERSISTENT > ELEPHANT-TESTS::NON-TRANSIENT-CLASS-SLOT-1 ELEPHANT-TESTS::NON-TRANSIENT-CLASS-SLOT-2 ELEPHANT-TESTS::TRANSIENT-CLASS-SLOT > ELEPHANT-TESTS::CLASS-DEFINERS ELEPHANT-TESTS::BAD-INHERITENCE ELEPHANT-TESTS::MIXES ELEPHANT-TESTS::MIXES-RIGHT-SLOTS ELEPHANT-TESTS::INHERIT > ELEPHANT-TESTS::INHERIT-RIGHT-SLOTS ELEPHANT-TESTS::INITFORM-CLASSES ELEPHANT-TESTS::INITFORM-TEST ELEPHANT-TESTS::INITARG-TEST > ELEPHANT-TESTS::NO-EVAL-INITFORM ELEPHANT-TESTS::REDEFCLASS MAKUNBOUND ELEPHANT-TESTS::UPDATE-CLASS CHANGE-CLASS ELEPHANT-TESTS::CHANGE-CLASS3 > ELEPHANT-TESTS::BASICPERSISTENCE ELEPHANT-TESTS::TESTOID ELEPHANT-TESTS::BTREE-MAKE ELEPHANT-TESTS::BTREE-PUT ELEPHANT-TESTS::BTREE-GET > ELEPHANT:REMOVE-KV ELEPHANT-TESTS::REMOVED ELEPHANT:MAP-BTREE ELEPHANT-TESTS::INDEXED-BTREE-MAKE ELEPHANT-TESTS::ADD-INDICES > ELEPHANT-TESTS::TEST-INDICES ELEPHANT-TESTS::INDEXED-PUT ELEPHANT-TESTS::INDEXED-GET ELEPHANT-TESTS::SIMPLE-SLOT-GET > ELEPHANT-TESTS::INDEXED-GET-FROM-SLOT1 ELEPHANT-TESTS::INDEXED-GET-FROM-SLOT2 ELEPHANT-TESTS::REMOVE-KV-INDEXED > ELEPHANT-TESTS::NO-KEY-NOR-INDICES ELEPHANT-TESTS::REMOVE-KV-FROM-SLOT1 ELEPHANT-TESTS::NO-KEY-NOR-INDICES-SLOT1 > ELEPHANT-TESTS::REMOVE-KV-FROM-SLOT2 ELEPHANT-TESTS::NO-KEY-NOR-INDICES-SLOT2 ELEPHANT-TESTS::MAP-INDEXED ELEPHANT-TESTS::GET-FIRST > ELEPHANT-TESTS::GET-FIRST2 ELEPHANT-TESTS::GET-LAST ELEPHANT-TESTS::GET-LAST2 SET ELEPHANT-TESTS::SET2 ELEPHANT-TESTS::SET-RANGE > ELEPHANT-TESTS::SET-RANGE2 ELEPHANT-TESTS::REM-KV ELEPHANT-TESTS::REM-IDEXKV ELEPHANT-TESTS::MAKE-INDEXED2 ELEPHANT-TESTS::ADD-INDICES2 > ELEPHANT-TESTS::PUT-INDEXED2 ELEPHANT-TESTS::GET-INDEXED2 ELEPHANT-TESTS::GET-FROM-INDEX3 ELEPHANT-TESTS::DUP-TEST ELEPHANT-TESTS::NODUP-TEST > ELEPHANT-TESTS::PREV-NODUP-TEST ELEPHANT-TESTS::PNODUP-TEST ELEPHANT-TESTS::PPREV-NODUP-TEST ELEPHANT-TESTS::CUR-DEL1 > ELEPHANT-TESTS::INDEXED-DELETE ELEPHANT-TESTS::TEST-DELETED > Test ELEPHANT-TESTS::INDEXED-DELETE2 failed > Form: (ELEPHANT-TESTS::FINISHES > (ELEPHANT:WITH-TRANSACTION (:STORE-CONTROLLER ELEPHANT:*STORE-CONTROLLER*) > (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::CURS ELEPHANT-TESTS::INDEX3) > (ELEPHANT:CURSOR-FIRST ELEPHANT-TESTS::CURS) > (ELEPHANT:CURSOR-NEXT-DUP ELEPHANT-TESTS::CURS) > (ELEPHANT:CURSOR-DELETE ELEPHANT-TESTS::CURS)))) > Expected value: T > Actual value: NIL. > Test ELEPHANT-TESTS::TEST-DELETED2 failed > Form: (VALUES (ELEPHANT:GET-VALUE 0 ELEPHANT-TESTS::INDEXED2) (ELEPHANT:GET-VALUE 0 ELEPHANT-TESTS::INDEX3) > (ELEPHANT:GET-VALUE 1 ELEPHANT-TESTS::INDEXED2) > (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::C ELEPHANT-TESTS::INDEX3) > (ELEPHANT:CURSOR-FIRST ELEPHANT-TESTS::C) > (MULTIPLE-VALUE-BIND (ELEPHANT-TESTS::M ELEPHANT-TESTS::K ELEPHANT-TESTS::V) > (ELEPHANT:CURSOR-NEXT ELEPHANT-TESTS::C) > ELEPHANT-TESTS::V))) > Expected values: 0 > 0 > NIL > -2 > Actual values: 0 > 0 > -1 > -1. > Test ELEPHANT-TESTS::CUR-DEL2 failed > Form: (ELEPHANT:WITH-TRANSACTION (:STORE-CONTROLLER ELEPHANT:*STORE-CONTROLLER*) > (LET* ((ELEPHANT-TESTS::IBT (ELEPHANT:MAKE-INDEXED-BTREE ELEPHANT:*STORE-CONTROLLER*)) > (ELEPHANT-TESTS::ID1 > (ELEPHANT:ADD-INDEX ELEPHANT-TESTS::IBT :INDEX-NAME 'ELEPHANT-TESTS::IDX1 :KEY-FORM 'ELEPHANT-TESTS::ODD))) > (LOOP ELEPHANT-TESTS::FOR ELEPHANT-TESTS::I ELEPHANT-TESTS::FROM 0 ELEPHANT-TESTS::TO 10 DO > (SETF (ELEPHANT:GET-VALUE ELEPHANT-TESTS::I ELEPHANT-TESTS::IBT) (* ELEPHANT-TESTS::I ELEPHANT-TESTS::I))) > (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::C ELEPHANT-TESTS::ID1) > (ELEPHANT:CURSOR-FIRST ELEPHANT-TESTS::C) > (ELEPHANT:CURSOR-NEXT-DUP ELEPHANT-TESTS::C) > (ELEPHANT:CURSOR-DELETE ELEPHANT-TESTS::C)) > (EQUAL (LIST (ELEPHANT:GET-VALUE 1 ELEPHANT-TESTS::ID1) (ELEPHANT:GET-VALUE 0 ELEPHANT-TESTS::ID1)) '(1 0)))) > Expected value: T > Actual value: #. > ELEPHANT-TESTS::GET-BOTH ELEPHANT-TESTS::PGET-BOTH ELEPHANT-TESTS::PGET-BOTH-RANGE > Test ELEPHANT-TESTS::PCURSOR failed > Form: (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::C ELEPHANT-TESTS::INDEX3) > (VALUES (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PFIRST ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT-NODUP ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT-DUP ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PPREV ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PPREV-NODUP ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PLAST ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PSET ELEPHANT-TESTS::C 300)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PSET-RANGE ELEPHANT-TESTS::C 199.5)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PGET-BOTH ELEPHANT-TESTS::C 10 101)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PGET-BOTH-RANGE ELEPHANT-TESTS::C 11 111.4)))) > Expected values: 0 > 2 > 10 > 11 > 10 > 9 > 9999 > 3000 > 2000 > 101 > 112 > Actual values: 0 > 1 > 10 > 11 > 10 > 9 > 9999 > 3000 > 2000 > 101 > 112. > ELEPHANT-TESTS::NEWINDEX > Test ELEPHANT-TESTS::PCURSOR2 failed > Form: (ELEPHANT:WITH-BTREE-CURSOR (ELEPHANT-TESTS::C ELEPHANT-TESTS::INDEX4) > (VALUES (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PFIRST ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT-NODUP ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PNEXT-DUP ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PPREV ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PPREV-NODUP ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PLAST ELEPHANT-TESTS::C)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PSET ELEPHANT-TESTS::C 300)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PSET-RANGE ELEPHANT-TESTS::C 199.5)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PGET-BOTH ELEPHANT-TESTS::C 10 101)) > (ELEPHANT-TESTS::PCURSOR-PKEY (ELEPHANT:CURSOR-PGET-BOTH-RANGE ELEPHANT-TESTS::C 11 111.4)))) > Expected values: 0 > 2 > 10 > 11 > 10 > 9 > 9999 > 3000 > 2000 > 101 > 112 > Actual values: 0 > 1 > 10 > 11 > 10 > 9 > 9999 > 3000 > 2000 > 101 > 112. > ELEPHANT-TESTS::ADD-GET-REMOVE ELEPHANT-TESTS::ADD-GET-REMOVE-SYMBOL ELEPHANT-TESTS::EXISTSP > Warning: Manually finalizing class IDX-ONE > ELEPHANT-TESTS::DISABLE-CLASS-INDEXING-TEST ELEPHANT-TESTS::INDEXING-BASIC-TRIVIAL ELEPHANT-TESTS::INDEXING-BASIC > ELEPHANT-TESTS::INDEXING-INHERIT ELEPHANT-TESTS::INDEXING-RANGE ELEPHANT-TESTS::INDEXING-WIPE-INDEX ELEPHANT-TESTS::INDEXING-RECONNECT-DB > ELEPHANT-TESTS::INDEXING-CHANGE-CLASS ELEPHANT-TESTS::INDEXING-REDEF-CLASS > Ranged get of 40/500 objects = Linear: 25.937 sec Indexed: 14.0 sec > ELEPHANT-TESTS::INDEXING-TIMING > Single store mode: ignoring ELEPHANT-TESTS::REMOVE-ELEMENT > Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-BASIC > Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-BTREE > Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-IDX-BTREE > Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-PCLASS > Single store mode: ignoring ELEPHANT-TESTS::MIGRATE-IPCLASS > 5 out of 111 total tests failed: ELEPHANT-TESTS::INDEXED-DELETE2, ELEPHANT-TESTS::TEST-DELETED2, ELEPHANT-TESTS::CUR-DEL2, > ELEPHANT-TESTS::PCURSOR, ELEPHANT-TESTS::PCURSOR2. > NIL > CG-USER(137): > > > > >> -----Urspr?ngliche Nachricht----- >> Von: eslick at media.mit.edu, Elephant bugs and development >> Gesendet: 06.08.06 21:24:03 >> An: Elephant bugs and development >> Betreff: Re: [elephant-devel] strings as Class Indices ? >> > > > >> That's interesting, that code works fine on my system. >> >> Have you run the full Elephant regression tests on clisp? Did they all >> pass? >> >> Regards, >> Ian >> >> franks-muc at web.de wrote: >> >>> I wonder if it is possible to use strings in class indices. The example from the manual >>> using integers works well on my computer (windows,acl-trial,elephant,postgres). >>> I modified this example to use strings: >>> >>> (defclass idx-four () >>> ((slot1 :initarg :slot1 :accessor slot1 :index t)) >>> (:metaclass persistent-metaclass)) >>> >>> >>> (defun make-idx-four (val) >>> (make-instance 'idx-four :slot1 val)) >>> >>> (with-transaction () >>> (mapc #'make-idx-four '("1" "1" "1" "2" "2" "4" "5" "5" "5" "6" "10"))) >>> >>> (get-instances-by-value 'idx-four 'slot1 "1") >>> ==> NIL >>> >>> I suspect that nil is not the desired result since other examples provided >>> for elephant suggest that strings should work (see (get-instance-by-value 'user 'name name) >>> in http://common-lisp.net/pipermail/elephant-devel/2006-July/000477.html). >>> >>> Could someone please explain where I am wrong ? >>> Thank you in advance! >>> Frank Schorr >>> >>> >>> _____________________________________________________________________ >>> Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! >>> http://smartsurfer.web.de/?mc=100071&distributionid=000000000071 >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > > > ______________________________________________________________________ > XXL-Speicher, PC-Virenschutz, Spartarife & mehr: Nur im WEB.DE Club! > Jetzt gratis testen! http://freemail.web.de/home/landingpad/?mc=021130 > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From lists at infoway.net Fri Aug 11 21:17:47 2006 From: lists at infoway.net (Daniel Salama) Date: Fri, 11 Aug 2006 17:17:47 -0400 Subject: [elephant-devel] Persistance and Model Validations Message-ID: <1C6C3914-0E35-4F08-886F-E60EC0CBD3B2@infoway.net> Hi, This may be more of a suggestion since I think I already know the answer. I come from the Ruby On Rails world and, as some of you may know, their Active Record module provides hooks that allow the automatic and customized evaluation of validation rules at different stages of the committing process. At first, I was going to as how can I hook validation rules into the persistent machinery. I guess I could (should) always wrap my commits in transactions and then I could perform the validation process within the transaction and, therefore, I could commit or abort depending on the validation results. However, as my Lisp knowledge is still limited, I'd like to suggest for some future version of Elephant, the inclusion of some type of macro that works somehow like a with-transaction type operation. This macro could support generic methods that could be "customized" per class just like CLOS generic methods to support model validation. Just for reference, Rails validation hooks support, among others: - before validation - after validation - before create - after create - before save - after save - before destroy - after destroy Maybe some of this functionality could be implemented in the near future. Maybe it's not that big of a deal, but, since my Lisp knowledge is limited, I am just throwing this suggestion. Then again, maybe there is a better way to do it and I just have too much of a Rails mentality for the time being :) Thanks, Daniel From read at robertlread.net Fri Aug 11 21:40:22 2006 From: read at robertlread.net (Robert L. Read) Date: Fri, 11 Aug 2006 16:40:22 -0500 Subject: [elephant-devel] Persistance and Model Validations In-Reply-To: <1C6C3914-0E35-4F08-886F-E60EC0CBD3B2@infoway.net> References: <1C6C3914-0E35-4F08-886F-E60EC0CBD3B2@infoway.net> Message-ID: <1155332422.4658.1490.camel@localhost.localdomain> Just a quick comment on this --- Ian may have a better idea. Yes, if we offered a nice clean place to put all of those, it would make it obvious where to put validation. (I will certainly think about that in the future.) Here is a technique that I use, that is very effective for me, but not generally applicable: I use DCM, which is in the contrib directory and basically provides wrappers for the create/read/update/destory operations, similar to those you name below. I use :after methods on these operations (specialized on Director classes, since it is generally class dependedent) to send objects to a text indexer in a background thread whenever they change. (I was using Lucene-ws but now use montezuma; it makes sense to do these operations in a back-ground thread since a web-service call might block the thread that needs to get a response on to the glass as fast as possible.) This is an extraordinarily powerful and elegant feature of LISP. If you have a method that gives you a "change point" (I believe that is what you are asking for) then you can easily put a :before method in a completely different file that performs validation and signals an exception or takes some other action that interrupts the intended operation. One of the beauties of this is that the validation rules can be easily kept quite separate from the rest of the code. I suppose this same technique could be applied even if you do not have clearly defined methods; but personally I consider this clarity to be one of the advantages of the DCM code. Of course, one can more or less easily create your own validation points, without becoming dependeing on the DCM code. Here's my example code: (defmethod register-obj :after ((dir indexing-director) (mo managed- object)) (pub-and-proc (list "indexdocument" dir (k (mid mo)))) ) (defmethod delete-obj :before ((dir indexing-director) (id key)) (pub-and-proc (list "removedocument" dir (k id))) ) "register-obj" and "delete-obj" are defined in DCM. On Fri, 2006-08-11 at 17:17 -0400, Daniel Salama wrote: > Hi, > > This may be more of a suggestion since I think I already know the > answer. > > I come from the Ruby On Rails world and, as some of you may know, > their Active Record module provides hooks that allow the automatic > and customized evaluation of validation rules at different stages of > the committing process. > > At first, I was going to as how can I hook validation rules into the > persistent machinery. I guess I could (should) always wrap my commits > in transactions and then I could perform the validation process > within the transaction and, therefore, I could commit or abort > depending on the validation results. > > However, as my Lisp knowledge is still limited, I'd like to suggest > for some future version of Elephant, the inclusion of some type of > macro that works somehow like a with-transaction type operation. This > macro could support generic methods that could be "customized" per > class just like CLOS generic methods to support model validation. > Just for reference, Rails validation hooks support, among others: > > - before validation > - after validation > - before create > - after create > - before save > - after save > - before destroy > - after destroy > > Maybe some of this functionality could be implemented in the near > future. Maybe it's not that big of a deal, but, since my Lisp > knowledge is limited, I am just throwing this suggestion. > > Then again, maybe there is a better way to do it and I just have too > much of a Rails mentality for the time being :) > > Thanks, > Daniel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Fri Aug 11 21:50:32 2006 From: lists at infoway.net (Daniel Salama) Date: Fri, 11 Aug 2006 17:50:32 -0400 Subject: [elephant-devel] Persistance and Model Validations In-Reply-To: <1155332422.4658.1490.camel@localhost.localdomain> References: <1C6C3914-0E35-4F08-886F-E60EC0CBD3B2@infoway.net> <1155332422.4658.1490.camel@localhost.localdomain> Message-ID: Robert, That's exactly what I was referring to, except that instead of using DCM, just use the standard persistence machinery in Elephant (for the main reason that I haven't learned your DCM yet :) ). Anyway, I guess as you guys get closer to integrating the DCM concept(s) into the standard machinery as we discussed a few weeks back, this would be a non-issue since you seem to have that (or similar) functionality already in place. Anyway, just food for thought. Thanks, Daniel On Aug 11, 2006, at 5:40 PM, Robert L. Read wrote: > Just a quick comment on this --- Ian may have a better idea. > > Yes, if we offered a nice clean place to put all of those, it would > make it obvious > where to put validation. (I will certainly think about that in the > future.) > > Here is a technique that I use, that is very effective for me, but > not generally > applicable: I use DCM, which is in the contrib directory and > basically provides > wrappers for the create/read/update/destory operations, similar to > those you > name below. I use :after methods on these operations (specialized > on Director > classes, since it is generally class dependedent) to send objects > to a text indexer > in a background thread whenever they change. (I was using Lucene- > ws but now > use montezuma; it makes sense to do these operations in a back- > ground thread since > a web-service call might block the thread that needs to get a > response on to the > glass as fast as possible.) > > This is an extraordinarily powerful and elegant feature of LISP. > If you have a method > that gives you a "change point" (I believe that is what you are > asking for) then you can > easily put a :before method in a completely different file that > performs validation and > signals an exception or takes some other action that interrupts the > intended operation. > One of the beauties of this is that the validation rules can be > easily kept quite separate from the rest of the code. > > I suppose this same technique could be applied even if you do not > have clearly defined > methods; but personally I consider this clarity to be one of the > advantages of the DCM code. > Of course, one can more or less easily create your own validation > points, without becoming > dependeing on the DCM code. > > Here's my example code: > (defmethod register-obj :after ((dir indexing-director) (mo managed- > object)) > (pub-and-proc (list "indexdocument" dir (k (mid mo)))) > ) > > (defmethod delete-obj :before ((dir indexing-director) (id key)) > (pub-and-proc (list "removedocument" dir (k id))) > ) > "register-obj" and "delete-obj" are defined in DCM. > > > On Fri, 2006-08-11 at 17:17 -0400, Daniel Salama wrote: >> Hi, >> >> This may be more of a suggestion since I think I already know the >> answer. >> >> I come from the Ruby On Rails world and, as some of you may know, >> their Active Record module provides hooks that allow the automatic >> and customized evaluation of validation rules at different stages of >> the committing process. >> >> At first, I was going to as how can I hook validation rules into the >> persistent machinery. I guess I could (should) always wrap my commits >> in transactions and then I could perform the validation process >> within the transaction and, therefore, I could commit or abort >> depending on the validation results. >> >> However, as my Lisp knowledge is still limited, I'd like to suggest >> for some future version of Elephant, the inclusion of some type of >> macro that works somehow like a with-transaction type operation. This >> macro could support generic methods that could be "customized" per >> class just like CLOS generic methods to support model validation. >> Just for reference, Rails validation hooks support, among others: >> >> - before validation >> - after validation >> - before create >> - after create >> - before save >> - after save >> - before destroy >> - after destroy >> >> Maybe some of this functionality could be implemented in the near >> future. Maybe it's not that big of a deal, but, since my Lisp >> knowledge is limited, I am just throwing this suggestion. >> >> Then again, maybe there is a better way to do it and I just have too >> much of a Rails mentality for the time being :) >> >> Thanks, >> Daniel >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Sun Aug 13 06:27:32 2006 From: lists at infoway.net (Daniel Salama) Date: Sun, 13 Aug 2006 02:27:32 -0400 Subject: [elephant-devel] Berkeley DB 4.4.20 Message-ID: Being that BDB is on version 4.4.20 and there seem to be some important fixes since the 4.3 version, does Elephant support this version? Is there any plans to support it? Thanks, Daniel From read at robertlread.net Sun Aug 13 21:36:57 2006 From: read at robertlread.net (Robert L. Read) Date: Sun, 13 Aug 2006 16:36:57 -0500 Subject: [elephant-devel] Berkeley DB 4.4.20 In-Reply-To: References: Message-ID: <1155505018.4658.1527.camel@localhost.localdomain> On Sun, 2006-08-13 at 02:27 -0400, Daniel Salama wrote: > Being that BDB is on version 4.4.20 and there seem to be some > important fixes since the 4.3 version, does Elephant support this > version? Is there any plans to support it? > > Thanks, > Daniel I don't have any plans to work on this in the near future. The Oracle purchase of Sleepycat clouds the issue slightly; personally I use the relational backend for all my work. I would estimate the work involved here at several hours to several days --- I really have not investigated it deeply except to note that it doesn't seem to work with 4.4. Of course, if someone wants to get it working, I would be both grateful and as helpful as I could be. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lists at infoway.net Mon Aug 14 03:27:06 2006 From: lists at infoway.net (Daniel Salama) Date: Sun, 13 Aug 2006 23:27:06 -0400 Subject: [elephant-devel] Berkeley DB 4.4.20 In-Reply-To: <1155505018.4658.1527.camel@localhost.localdomain> References: <1155505018.4658.1527.camel@localhost.localdomain> Message-ID: Robert/team, I know I've discussed this in the past, but somehow, your comment has awaken my concern as to the future of Elephant and Sleepycat. I know you've mentioned that you are using Elephant mainly (or only) for the relational backend. However, being that the BDB backend is about 5 times faster than the relational one, I was concentrating on using the BDB backend. Am I going down the wrong or uncertain path? Will future enhancements to Elephant be focused on the relational backend first? I wish I could say that I can volunteer to help enhance Elephant in anyway, but I just don't have the qualifications; at least not yet. I know there are others involved in the project, but not really familiar with everyone's contributions. The one thing I know is that, this, being an open-source project, is sort of like a side task for the current team members. After all, we "all" have to work to put food on the table. So, it's not that I'm asking the dedication of a team of developers such as one assigned by a company offering commercial licenses. Through the documents I've read, I think I came across one that said that there are about 15 people (entities) using Elephant. I don't know if those are just people known to use it and then many others that you just don't know about, or that it really is a low profile project. I can only comment/compare to CL-SQL, which is the other project I've come across more often. I don't use it but do see more traffic related to that project than to Elephant. From what I know, the two projects have different schools of thought and philosophies as to how persistence is implemented and managed. I can say that I like the way things are done in Elephant. They just seem so natural and elegant. It is for that reason that I'm concerned with my using of Elephant and BDB backend. I plan to migrate a large project over from Ruby on Rails to the UCW/Elephant framework. I just don't want to shoot myself on the foot down the road when my clients are using the system or, hopefully won't be the case, obsoleted framework. Thanks again for such great work. - Daniel On Aug 13, 2006, at 5:36 PM, Robert L. Read wrote: > On Sun, 2006-08-13 at 02:27 -0400, Daniel Salama wrote: >> Being that BDB is on version 4.4.20 and there seem to be some >> important fixes since the 4.3 version, does Elephant support this >> version? Is there any plans to support it? >> >> Thanks, >> Daniel > > I don't have any plans to work on this in the near future. The > Oracle purchase of > Sleepycat clouds the issue slightly; personally I use the > relational backend for all my > work. > > I would estimate the work involved here at several hours to several > days --- I really > have not investigated it deeply except to note that it doesn't seem > to work with 4.4. > > Of course, if someone wants to get it working, I would be both > grateful and as > helpful as I could be. > > > > >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Mon Aug 14 03:57:34 2006 From: read at robertlread.net (Robert L. Read) Date: Sun, 13 Aug 2006 22:57:34 -0500 Subject: [elephant-devel] Berkeley DB 4.4.20 In-Reply-To: References: <1155505018.4658.1527.camel@localhost.localdomain> Message-ID: <1155527855.4658.1577.camel@localhost.localdomain> On Sun, 2006-08-13 at 23:27 -0400, Daniel Salama wrote: > Robert/team, > I know I've discussed this in the past, but somehow, your comment has > awaken my concern as to the future of Elephant and Sleepycat. I know > you've mentioned that you are using Elephant mainly (or only) for the > relational backend. However, being that the BDB backend is about 5 > times faster than the relational one, I was concentrating on using the > BDB backend. > Am I going down the wrong or uncertain path? Will future enhancements > to Elephant be focused on the relational backend first? I wish I could > say that I can volunteer to help enhance Elephant in anyway, but I > just don't have the qualifications; at least not yet. I know there are > others involved in the project, but not really familiar with > everyone's contributions. You are not going down a wrong path; you are going down an uncertain path, which has a good contingency plan. The future of BDB itself and its licensing situation is slightly uncertain. If you had to switch to the relational backend, it would be painless; one of the things that Elephant does really well is to allow repository migration (I think.) Elephant insulates you from this consideration, to some extent. BDB is 5 times faster---are you sure that matters to you? One cannot emphasize too much that premature optimization is a bad thing. If someone would just improve the serializer, the relational backend would be much faster --- I know how to do this but have not done so, since, for me, it is irrelevant. > > The one thing I know is that, this, being an open-source project, is > sort of like a side task for the current team members. After all, we > "all" have to work to put food on the table. So, it's not that I'm > asking the dedication of a team of developers such as one assigned by > a company offering commercial licenses. Right. I committed (about a year ago, I think) to maintain it for some time, not necessarily to improve it. By good fortune, Ian Eslick has improved it a lot, and I have helped. > Through the documents I've read, I think I came across one that said > that there are about 15 people (entities) using That was my guess based on the downloads, questions, and the membership of the lists like this one. It could be much, much higher if the average user never asks a question. I honestly wish I know how many people are using it. > Elephant. I don't know if those are just people known to use it and > then many others that you just don't know about, or that it really is > a low profile project. I can only comment/compare to CL-SQL, which is > the other project I've come across more often. I don't use it but do > see more traffic related to that project than to Elephant. From what I > know, the two projects have different schools of thought and > philosophies as to how persistence is implemented and managed. I can > say that I like the way things are done in Elephant. They just seem so > natural and elegant. Elephant sits on top of CL-SQL; CL-SQL is the premier database connectivity system for LISP (it rocks.) CL-SQL also implements some object mapping stuff (ORM), whereas Elephant implements a straight Object database (more or less.) CL-SQL insulates Elephant from changing database issues, and let's us work with Postgres and SQLite3, and, with a little work, probably others; its evolution is partially driven by changes in those databases. > It is for that reason that I'm concerned with my using of Elephant and > BDB backend. I plan to migrate a large project over from Ruby on Rails > to the UCW/Elephant framework. I just don't want to shoot myself on > the foot down the road when my clients are using the system or, > hopefully won't be the case, obsoleted framework. You will certainly be able to migrate your data to a different backend; although I wish we had more real-world tests of this capability, it's definitely there. The real question is: do you really have a performance critical situation? I don't plan to work on BDB 4.4 in the next six months, and Ian and I are the only really active developers right now. I'm not planning to drop support for BDB either, unless they change the licensing in some way. Perhaps you should actually measure your object retrieval rate and see if you think it is acceptable. I personally use DCM (in the contrib dir) which is basically an prevalence-style caching system; in general, caching and indexing go a long way. > > > Thanks again for such great work. Thanks for using it! I wish I had more time to work on it, but I have children and college tuitions looming... ---- Robert L. Read, PhD read &T robertlread.net Consider visiting Progressive Engineering: http://robertlread.net/pe In Austin: 912-8593 "Think globally, Act locally." -- RBF -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Wed Aug 16 15:21:38 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 16 Aug 2006 11:21:38 -0400 Subject: [elephant-devel] Berkeley DB 4.4.20 In-Reply-To: References: <1155505018.4658.1527.camel@localhost.localdomain> Message-ID: <44E33802.10104@csail.mit.edu> Daniel (and others), I'm subjecting the BDB backend to extremely high throughput loads on a daily basis and it's the first backend I test when I check in changes. Between Robert and I we cover a large % of the uses of Elephant on Win32 and Mac OS. I'm in the middle of a demanding project so I haven't been able to implement the set of improvements I intended this summer but still intend to move towards the 0.6.1 and 0.6.2 TODO list located in the source directory. I have some plans to improve speed for both backends (relational & BDB) as well as eventually implement a native BTree backend similar to Rucksack and AllegroStore. I'm hoping to dig back into some of these issues in September and BDB 4.x is high on my priority list. Cheers, Ian Daniel Salama wrote: > Robert/team, > > I know I've discussed this in the past, but somehow, your comment has > awaken my concern as to the future of Elephant and Sleepycat. I know > you've mentioned that you are using Elephant mainly (or only) for the > relational backend. However, being that the BDB backend is about 5 > times faster than the relational one, I was concentrating on using the > BDB backend. Am I going down the wrong or uncertain path? Will future > enhancements to Elephant be focused on the relational backend first? I > wish I could say that I can volunteer to help enhance Elephant in > anyway, but I just don't have the qualifications; at least not yet. I > know there are others involved in the project, but not really familiar > with everyone's contributions. > > The one thing I know is that, this, being an open-source project, is > sort of like a side task for the current team members. After all, we > "all" have to work to put food on the table. So, it's not that I'm > asking the dedication of a team of developers such as one assigned by > a company offering commercial licenses. > > Through the documents I've read, I think I came across one that said > that there are about 15 people (entities) using Elephant. I don't know > if those are just people known to use it and then many others that you > just don't know about, or that it really is a low profile project. I > can only comment/compare to CL-SQL, which is the other project I've > come across more often. I don't use it but do see more traffic related > to that project than to Elephant. From what I know, the two projects > have different schools of thought and philosophies as to how > persistence is implemented and managed. I can say that I like the way > things are done in Elephant. They just seem so natural and elegant. > > It is for that reason that I'm concerned with my using of Elephant and > BDB backend. I plan to migrate a large project over from Ruby on Rails > to the UCW/Elephant framework. I just don't want to shoot myself on > the foot down the road when my clients are using the system or, > hopefully won't be the case, obsoleted framework. > > Thanks again for such great work. > > - Daniel > > On Aug 13, 2006, at 5:36 PM, Robert L. Read wrote: > >> On Sun, 2006-08-13 at 02:27 -0400, Daniel Salama wrote: >>> Being that BDB is on version 4.4.20 and there seem to be some >>> important fixes since the 4.3 version, does Elephant support this >>> version? Is there any plans to support it? >>> >>> Thanks, >>> Daniel >>> >> >> I don't have any plans to work on this in the near future. The >> Oracle purchase of >> Sleepycat clouds the issue slightly; personally I use the relational >> backend for all my >> work. >> >> I would estimate the work involved here at several hours to several >> days --- I really >> have not investigated it deeply except to note that it doesn't seem >> to work with 4.4. >> >> Of course, if someone wants to get it working, I would be both >> grateful and as >> helpful as I could be. >> >> >> >> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From bhyde at pobox.com Wed Aug 16 15:29:14 2006 From: bhyde at pobox.com (Ben Hyde) Date: Wed, 16 Aug 2006 11:29:14 -0400 Subject: [elephant-devel] Public API for slot btrees/cursors? Message-ID: <2E5FEDEF-68D3-4279-893B-12DB9AE8C62B@pobox.com> So... why is the public interface to the slot indexes limited to the get-instance-by-* methods? I find I'm writing routines like this: (defun map-over-unique-values-of-slot (function class-name slot) (ele::with-inverted-cursor (cur (find-class class-name) slot) (loop for count from 0 do (multiple-value-bind (ok? value) (cursor-next-nodup cur) (unless ok? (return)) (funcall function value))))) But I note that with-inverted-cursor is private. From Petter.Egesund at kunnskapsforlaget.no Fri Aug 18 18:41:02 2006 From: Petter.Egesund at kunnskapsforlaget.no (Petter Egesund) Date: Fri, 18 Aug 2006 20:41:02 +0200 Subject: [elephant-devel] Memory error with Elephant Message-ID: <4FEEF5F5BB5D344BAAB5004BB495C4F104077D@EPOST3.bokklubbene.no> Hi; I am trying to use Elephant, version 0.6 with Sbcl 0.9.15, Suse 9 on a AMD64 - I have only installed the DBD-backend - Berkley DB 4.3.29 Everything seems to compile fine (also the .so-files). When I try to do some simple things, I get a heap error from my Lisp. Any clues (dump is below)? Looks like a cool system, though... Cheers, Petter Egesund ----------- ; SLIME 2006-04-20 CL-USER> (asdf:operate 'asdf:load-op :ele-bdb) ; loading system definition from /home/pe/.sbcl/systems/ele-bdb.asd into ; # ; loading system definition from /home/pe/.sbcl/systems/uffi.asd into ; # ; registering # as UFFI ; registering # as ELE-BDB ; loading system definition from /home/pe/.sbcl/systems/elephant.asd into ; # ; registering # as ELEPHANT STYLE-WARNING: implicitly creating new generic function PERSISTENT-SLOTS STYLE-WARNING: implicitly creating new generic function OLD-PERSISTENT-SLOTS STYLE-WARNING: implicitly creating new generic function UPDATE-PERSISTENT-SLOTS STYLE-WARNING: implicitly creating new generic function INDEXED-RECORD STYLE-WARNING: implicitly creating new generic function OLD-INDEXED-RECORD STYLE-WARNING: implicitly creating new generic function UPDATE-INDEXED-RECORD STYLE-WARNING: implicitly creating new generic function MAKE-NEW-INDEXED-RECORD STYLE-WARNING: implicitly creating new generic function REMOVED-INDEXING? STYLE-WARNING: implicitly creating new generic function REGISTER-INDEXED-SLOT STYLE-WARNING: implicitly creating new generic function UNREGISTER-INDEXED-SLOT STYLE-WARNING: implicitly creating new generic function REGISTER-DERIVED-INDEX STYLE-WARNING: implicitly creating new generic function UNREGISTER-DERIVED-INDEX STYLE-WARNING: implicitly creating new generic function PREVIOUSLY-INDEXED ; loading system definition from /home/pe/.sbcl/systems/cl-base64.asd into ; # ; registering # as CL-BASE64 ; registering # as CL-BASE64-TESTS ; loading system definition from /home/pe/.sbcl/systems/kmrcl.asd into ; # ; registering # as KMRCL STYLE-WARNING: implicitly creating new generic function GET-CON STYLE-WARNING: implicitly creating new generic function CONTROLLER-PROPERTIES STYLE-WARNING: implicitly creating new generic function SET-ELE-PROPERTY STYLE-WARNING: implicitly creating new generic function GET-ELE-PROPERTY STYLE-WARNING: implicitly creating new generic function ENSURE-MARKED-VERSION STYLE-WARNING: implicitly creating new generic function CONTROLLER-VERSION STYLE-WARNING: implicitly creating new generic function UP-TO-DATE-P STYLE-WARNING: implicitly creating new generic function UPGRADABLE-P STYLE-WARNING: implicitly creating new generic function UPGRADE STYLE-WARNING: implicitly creating new generic function FLUSH-INSTANCE-CACHE STYLE-WARNING: implicitly creating new generic function DROP-POBJECT STYLE-WARNING: implicitly creating new generic function MAP-BTREE STYLE-WARNING: implicitly creating new generic function EMPTY-BTREE-P STYLE-WARNING: implicitly creating new generic function CLASS-INDEX-CACHED? STYLE-WARNING: implicitly creating new generic function DETERMINE-SYNCH-METHOD STYLE-WARNING: implicitly creating new generic function SET-DB-SYNCH STYLE-WARNING: implicitly creating new generic function INDEXED-SLOT-WRITER STYLE-WARNING: implicitly creating new generic function CLASS-INDEXEDP-BY-NAME STYLE-WARNING: implicitly creating new generic function FIND-INVERTED-INDEX-NAMES STYLE-WARNING: implicitly creating new generic function WIPE-CLASS-INDEXING STYLE-WARNING: implicitly creating new generic function COPY-BTREE-CONTENTS STYLE-WARNING: implicitly creating new generic function BUILD-BTREE-INDEX NIL CL-USER> (use-package "ELE") T CL-USER> (open-store '(:BDB "/home/pe/testdb3")) # CL-USER> (add-to-root "my key" "my value") "my value" CL-USER> (get-from-root "my key") Heap exhausted: 8520368128 bytes available, 468151435328 requested. PROCEED WITH CAUTION! [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8520368128 468151435328) 1: ("foreign function: call_into_lisp") 2: ("foreign function: funcall2") 3: ("foreign function: gc_heap_exhausted_error_or_lose") 4: ("foreign function: gc_find_freeish_pages") 5: ("foreign function: gc_alloc_large") 6: ("foreign function: alloc_tramp") 7: ((LABELS ELEPHANT::%DESERIALIZE) #) 8: ((SB-PCL::FAST-METHOD GET-VALUE (T SLEEPYCAT::BDB-BTREE)) # # # #) 9: (SB-INT:EVAL-IN-LEXENV (GET-FROM-ROOT "my key") #) ------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Fri Aug 18 20:04:03 2006 From: read at robertlread.net (Robert L. Read) Date: Fri, 18 Aug 2006 15:04:03 -0500 Subject: [elephant-devel] Memory error with Elephant In-Reply-To: <4FEEF5F5BB5D344BAAB5004BB495C4F104077D@EPOST3.bokklubbene.no> References: <4FEEF5F5BB5D344BAAB5004BB495C4F104077D@EPOST3.bokklubbene.no> Message-ID: <1155931444.21401.3.camel@localhost.localdomain> That mystifies me. I can only conjecture that it is somehow related to your environment. Even thought it may seem strange since that simplest of functionality doesn't work, you might wish to execute the test. If, for example, there were an infinite loop in the serializer when compiled in your environment, the larges suite of automated tests might reveal that, as opposed to a problem with BDB, for example. On Fri, 2006-08-18 at 20:41 +0200, Petter Egesund wrote: > Hi; I am trying to use Elephant, version 0.6 with Sbcl 0.9.15, Suse 9 > on a AMD64 - I have only installed the DBD-backend - Berkley DB 4.3.29 > > Everything seems to compile fine (also the .so-files). When I try to > do some simple things, I get a heap error from my Lisp. Any clues > (dump is below)? > > Looks like a cool system, though... > > Cheers, > > Petter Egesund > > ----------- > > ; SLIME 2006-04-20 > CL-USER> (asdf:operate 'asdf:load-op :ele-bdb) > ; loading system definition from /home/pe/.sbcl/systems/ele-bdb.asd > into > ; # > ; loading system definition from /home/pe/.sbcl/systems/uffi.asd into > ; # > ; registering # as UFFI > ; registering # as ELE-BDB > ; loading system definition from /home/pe/.sbcl/systems/elephant.asd > into > ; # > ; registering # as ELEPHANT > STYLE-WARNING: implicitly creating new generic function PERSISTENT- > SLOTS > STYLE-WARNING: > implicitly creating new generic function OLD-PERSISTENT-SLOTS > STYLE-WARNING: > implicitly creating new generic function UPDATE-PERSISTENT-SLOTS > STYLE-WARNING: implicitly creating new generic function INDEXED-RECORD > STYLE-WARNING: implicitly creating new generic function OLD-INDEXED- > RECORD > STYLE-WARNING: > implicitly creating new generic function UPDATE-INDEXED-RECORD > STYLE-WARNING: > implicitly creating new generic function MAKE-NEW-INDEXED-RECORD > STYLE-WARNING: implicitly creating new generic function REMOVED- > INDEXING? > STYLE-WARNING: > implicitly creating new generic function REGISTER-INDEXED-SLOT > STYLE-WARNING: > implicitly creating new generic function UNREGISTER-INDEXED-SLOT > STYLE-WARNING: > implicitly creating new generic function REGISTER-DERIVED-INDEX > STYLE-WARNING: > implicitly creating new generic function UNREGISTER-DERIVED-INDEX > STYLE-WARNING: implicitly creating new generic function PREVIOUSLY- > INDEXED > ; loading system definition from /home/pe/.sbcl/systems/cl-base64.asd > into > ; # > ; registering # as CL-BASE64 > ; registering # as CL-BASE64- > TESTS > ; loading system definition from /home/pe/.sbcl/systems/kmrcl.asd into > ; # > ; registering # as KMRCL > STYLE-WARNING: implicitly creating new generic function GET-CON > STYLE-WARNING: > implicitly creating new generic function CONTROLLER-PROPERTIES > STYLE-WARNING: implicitly creating new generic function SET-ELE- > PROPERTY > STYLE-WARNING: implicitly creating new generic function GET-ELE- > PROPERTY > STYLE-WARNING: > implicitly creating new generic function ENSURE-MARKED-VERSION > STYLE-WARNING: implicitly creating new generic function CONTROLLER- > VERSION > STYLE-WARNING: implicitly creating new generic function UP-TO-DATE-P > STYLE-WARNING: implicitly creating new generic function UPGRADABLE-P > STYLE-WARNING: implicitly creating new generic function UPGRADE > STYLE-WARNING: > implicitly creating new generic function FLUSH-INSTANCE-CACHE > STYLE-WARNING: implicitly creating new generic function DROP-POBJECT > STYLE-WARNING: implicitly creating new generic function MAP-BTREE > STYLE-WARNING: implicitly creating new generic function EMPTY-BTREE-P > STYLE-WARNING: implicitly creating new generic function CLASS-INDEX- > CACHED? > STYLE-WARNING: > implicitly creating new generic function DETERMINE-SYNCH-METHOD > STYLE-WARNING: implicitly creating new generic function SET-DB-SYNCH > STYLE-WARNING: implicitly creating new generic function INDEXED-SLOT- > WRITER > STYLE-WARNING: > implicitly creating new generic function CLASS-INDEXEDP-BY-NAME > STYLE-WARNING: > implicitly creating new generic function FIND-INVERTED-INDEX-NAMES > STYLE-WARNING: implicitly creating new generic function WIPE-CLASS- > INDEXING > STYLE-WARNING: implicitly creating new generic function COPY-BTREE- > CONTENTS > STYLE-WARNING: implicitly creating new generic function BUILD-BTREE- > INDEX > NIL > CL-USER> (use-package "ELE") > T > CL-USER> (open-store '(:BDB "/home/pe/testdb3")) > # > CL-USER> (add-to-root "my key" "my value") > "my value" > CL-USER> (get-from-root "my key") > Heap exhausted: 8520368128 bytes available, 468151435328 requested. > PROCEED WITH CAUTION! > [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] > > Restarts: > 0: [ABORT-REQUEST] Abort handling SLIME request. > 1: [TERMINATE-THREAD] Terminate this thread (# thread" {10035112F1}>) > > Backtrace: > 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8520368128 468151435328) > 1: ("foreign function: call_into_lisp") > 2: ("foreign function: funcall2") > 3: ("foreign function: gc_heap_exhausted_error_or_lose") > 4: ("foreign function: gc_find_freeish_pages") > 5: ("foreign function: gc_alloc_large") > 6: ("foreign function: alloc_tramp") > 7: ((LABELS ELEPHANT::%DESERIALIZE) #) > 8: ((SB-PCL::FAST-METHOD GET-VALUE (T SLEEPYCAT::BDB-BTREE)) > # # # argument> #) > 9: (SB-INT:EVAL-IN-LEXENV (GET-FROM-ROOT "my key") #) > > > ------- > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Fri Aug 18 20:51:24 2006 From: read at robertlread.net (Robert L. Read) Date: Fri, 18 Aug 2006 15:51:24 -0500 Subject: SV: [elephant-devel] Memory error with Elephant In-Reply-To: <4FEEF5F5BB5D344BAAB5004BB495C4F104077E@EPOST3.bokklubbene.no> References: <4FEEF5F5BB5D344BAAB5004BB495C4F104077E@EPOST3.bokklubbene.no> Message-ID: <1155934284.21401.11.camel@localhost.localdomain> Well, its good we know this ---- there is zero hope of getting Elephant to work until we solve this problem. My hunch would be this has to do with the AMD64. It is entirely possible that the serializer makes some assumption that isn't true on AMD64; the serializer gets into some byte-representation issues that could fail there. If you would like to debug it, I would be happy to answer your questions. I hope to use a 64-bit processor in my own business soon, but I don't have one available now. Does anyone else have this working on a 64 bit processor? On Fri, 2006-08-18 at 22:37 +0200, Petter Egesund wrote: > Thanks for answering! > > I did run one test, the not-so-good result was like below. Any clues? > > Cheers, > > Petter > > > > > ------------ > > ; SLIME 2006-04-20 > CL-USER> (asdf:operate 'asdf:load-op :ele-bdb) > ; loading system definition from /home/pe/.sbcl/systems/ele-bdb.asd into > ; # > ; loading system definition from /home/pe/.sbcl/systems/uffi.asd into > ; # > ; registering # as UFFI > ; registering # as ELE-BDB > ; loading system definition from /home/pe/.sbcl/systems/elephant.asd into > ; # > ; registering # as ELEPHANT > STYLE-WARNING: implicitly creating new generic function PERSISTENT-SLOTS > STYLE-WARNING: > implicitly creating new generic function OLD-PERSISTENT-SLOTS > STYLE-WARNING: > implicitly creating new generic function UPDATE-PERSISTENT-SLOTS > STYLE-WARNING: implicitly creating new generic function INDEXED-RECORD > STYLE-WARNING: implicitly creating new generic function OLD-INDEXED-RECORD > STYLE-WARNING: > implicitly creating new generic function UPDATE-INDEXED-RECORD > STYLE-WARNING: > implicitly creating new generic function MAKE-NEW-INDEXED-RECORD > STYLE-WARNING: implicitly creating new generic function REMOVED-INDEXING? > STYLE-WARNING: > implicitly creating new generic function REGISTER-INDEXED-SLOT > STYLE-WARNING: > implicitly creating new generic function UNREGISTER-INDEXED-SLOT > STYLE-WARNING: > implicitly creating new generic function REGISTER-DERIVED-INDEX > STYLE-WARNING: > implicitly creating new generic function UNREGISTER-DERIVED-INDEX > STYLE-WARNING: implicitly creating new generic function PREVIOUSLY-INDEXED > ; loading system definition from /home/pe/.sbcl/systems/cl-base64.asd into > ; # > ; registering # as CL-BASE64 > ; registering # as CL-BASE64-TESTS > ; loading system definition from /home/pe/.sbcl/systems/kmrcl.asd into > ; # > ; registering # as KMRCL > STYLE-WARNING: implicitly creating new generic function GET-CON > STYLE-WARNING: > implicitly creating new generic function CONTROLLER-PROPERTIES > STYLE-WARNING: implicitly creating new generic function SET-ELE-PROPERTY > STYLE-WARNING: implicitly creating new generic function GET-ELE-PROPERTY > STYLE-WARNING: > implicitly creating new generic function ENSURE-MARKED-VERSION > STYLE-WARNING: implicitly creating new generic function CONTROLLER-VERSION > STYLE-WARNING: implicitly creating new generic function UP-TO-DATE-P > STYLE-WARNING: implicitly creating new generic function UPGRADABLE-P > STYLE-WARNING: implicitly creating new generic function UPGRADE > STYLE-WARNING: > implicitly creating new generic function FLUSH-INSTANCE-CACHE > STYLE-WARNING: implicitly creating new generic function DROP-POBJECT > STYLE-WARNING: implicitly creating new generic function MAP-BTREE > STYLE-WARNING: implicitly creating new generic function EMPTY-BTREE-P > STYLE-WARNING: implicitly creating new generic function CLASS-INDEX-CACHED? > STYLE-WARNING: > implicitly creating new generic function DETERMINE-SYNCH-METHOD > STYLE-WARNING: implicitly creating new generic function SET-DB-SYNCH > STYLE-WARNING: implicitly creating new generic function INDEXED-SLOT-WRITER > STYLE-WARNING: > implicitly creating new generic function CLASS-INDEXEDP-BY-NAME > STYLE-WARNING: > implicitly creating new generic function FIND-INVERTED-INDEX-NAMES > STYLE-WARNING: implicitly creating new generic function WIPE-CLASS-INDEXING > STYLE-WARNING: implicitly creating new generic function COPY-BTREE-CONTENTS > STYLE-WARNING: implicitly creating new generic function BUILD-BTREE-INDEX > NIL > CL-USER> (asdf:operate 'asdf:load-op :elephant-tests) > ; loading system definition from /home/pe/.sbcl/systems/elephant-tests.asd > ; into # > ; registering # as ELEPHANT-TESTS > ; registering # as > ; ELEPHANT-TESTS-BDB > ; loading system definition from /home/pe/.sbcl/systems/rt.asd into > ; # > ; registering # as RT > NIL > CL-USER> (in-package "ELEPHANT-TESTS") > # > ELE-TESTS> (setf *default-spec* *testbdb-spec*) > (:BDB "/home/pe/.sbcl/site/elephant/tests/testdb/") > ELE-TESTS> (do-backend-tests) > ; compiling file "/home/pe/.sbcl/site/elephant/tests/testsleepycat.lisp" (written 19 FEB 2006 05:53:02 AM): > ; compiling (IN-PACKAGE "ELE-TESTS") > ; compiling (DEFVAR ENV) > ; compiling (DEFVAR DB) > ; compiling (DEFUN PREPARE-SLEEPYCAT ...) > ; compiling (DEFTEST PREPARES-SLEEPYCAT ...) > ; compiling (DEFUN TEST-SEQUENCE1 ...) > ; compiling (DEFTEST TEST-SEQ1 ...) > ; compiling (DEFUN TEST-SEQUENCE2 ...) > ; compiling (DEFTEST TEST-SEQ2 ...) > ; compiling (DEFUN CLEANUP-SLEEPYCAT ...) > ; compiling (DEFTEST CLEANSUP-SLEEPYCAT ...) > > ; /home/pe/.sbcl/site/elephant/tests/testsleepycat.fasl written > ; compilation finished in 0:00:00 > Doing 115 pending tests of 115 tests total. > FIXNUMS FIXNUM-TYPE-1 > Test BIGNUMS failed > Form: (ARE-NOT-NULL (IN-OUT-EQUAL 10000000000) (IN-OUT-EQUAL -10000000000) > (LOOP FOR I FROM 0 TO 2000 ALWAYS > (IN-OUT-EQUAL (EXPT 2 I))) > (LOOP FOR I FROM 0 TO 2000 ALWAYS > (IN-OUT-EQUAL (- (EXPT 2 I)))) > (LOOP FOR I FROM 0 TO 2000 ALWAYS > (IN-OUT-EQUAL (- (EXPT 2 I) 1))) > (LOOP FOR I FROM 0 TO 2000 ALWAYS > (IN-OUT-EQUAL (- 1 (EXPT 2 I)))) > (LOOP FOR I FROM 0 TO 2000 ALWAYS > (IN-OUT-EQUAL (EXPT 3 I))) > (LOOP FOR I FROM 0 TO 2000 ALWAYS > (IN-OUT-EQUAL (- (EXPT 3 I))))) > Expected values: T > T > T > T > T > T > T > T > Actual value: #. > FLOATS > Test RATIONALS failed > Form: (ARE-NOT-NULL (IN-OUT-EQUAL 1/2) (IN-OUT-EQUAL -1/2) > (IN-OUT-EQUAL (/ 1 MOST-POSITIVE-FIXNUM)) > (IN-OUT-EQUAL (/ 1 MOST-NEGATIVE-FIXNUM)) > (IN-OUT-EQUAL > (/ MOST-POSITIVE-FIXNUM MOST-NEGATIVE-FIXNUM)) > (IN-OUT-EQUAL (/ (EXPT 2 200) (EXPT 3 300))) > (IN-OUT-EQUAL (/ (EXPT 2 200) (- (EXPT 3 300))))) > Expected values: T > T > T > T > T > T > T > Actual value: #. > Test BASE-STRINGS failed > Form: (ARE-NOT-NULL > (IN-OUT-EQUAL (MAKE-STRING 0 :ELEMENT-TYPE 'BASE-CHAR)) > (IN-OUT-EQUAL (COERCE "this is a test" 'BASE-STRING)) > (IN-OUT-EQUAL > (MAKE-STRING 400 :INITIAL-ELEMENT (CODE-CHAR 127) :ELEMENT-TYPE > 'BASE-CHAR))) > Expected values: T > T > T > Actual value: #. > Heap exhausted: 8517255168 bytes available, 498216206416 requested. PROCEED WITH CAUTION! > [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] > > Restarts: > 0: [ABORT-REQUEST] Abort handling SLIME request. > 1: [TERMINATE-THREAD] Terminate this thread (#) > > Backtrace: > 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8517255168 498216206416) > 1: ("foreign function: call_into_lisp") > 2: ("foreign function: funcall2") > 3: ("foreign function: gc_heap_exhausted_error_or_lose") > 4: ("foreign function: gc_find_freeish_pages") > 5: ("foreign function: gc_alloc_large") > 6: ("foreign function: alloc_tramp") > 7: ((LABELS ELEPHANT::%DESERIALIZE) #) > 8: (IN-OUT-EQUAL "this is a test") > 9: (SB-INT:EVAL-IN-LEXENV (IN-OUT-EQUAL "this is a test") #) > 10: (SB-INT:EVAL-IN-LEXENV (PROGN (IN-OUT-EQUAL "this is a test")) #) > 11: (SB-INT:EVAL-IN-LEXENV (NULL (PROGN (IN-OUT-EQUAL "this is a test"))) #) > 12: (SB-INT:EVAL-IN-LEXENV (IS-NOT-NULL (IN-OUT-EQUAL "this is a test")) #) > 13: (SB-INT:EVAL-IN-LEXENV (ARE-NOT-NULL (IN-OUT-EQUAL "") (IN-OUT-EQUAL "this is a test") (IN-OUT-EQUAL (MAKE-STRING 400 :INITIAL-ELEMENT #))) #) > 14: ((FLET REGRESSION-TEST::%DO)) > 15: (REGRESSION-TEST::DO-ENTRY #S(REGRESSION-TEST::ENTRY :PEND T :NAME STRINGS :PROPS NIL :FORM (ARE-NOT-NULL (IN-OUT-EQUAL "") (IN-OUT-EQUAL "this is a test") (IN-OUT-EQUAL #)) :VALS (T T T)) #) > 16: (REGRESSION-TEST::DO-ENTRIES* #) > 17: (REGRESSION-TEST::DO-ENTRIES #) > 18: (DO-BACKEND-TESTS (:BDB "/home/pe/.sbcl/site/elephant/tests/testdb/")) > 19: (SB-INT:EVAL-IN-LEXENV (DO-BACKEND-TESTS) #) > > > > > > ________________________________ > > Fra: elephant-devel-bounces at common-lisp.net [mailto:elephant-devel-bounces at common-lisp.net] P? vegne av Robert L. Read > Sendt: 18. august 2006 22:04 > Til: Elephant bugs and development > Emne: Re: [elephant-devel] Memory error with Elephant > > > That mystifies me. I can only conjecture that it is somehow related to your environment. > > Even thought it may seem strange since that simplest of functionality doesn't work, > you might wish to execute the test. If, for example, there were an infinite loop in the > serializer when compiled in your environment, the larges suite of automated tests might > reveal that, as opposed to a problem with BDB, for example. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bhyde at pobox.com Sun Aug 20 16:22:39 2006 From: bhyde at pobox.com (Ben Hyde) Date: Sun, 20 Aug 2006 12:22:39 -0400 Subject: [elephant-devel] Installing elephant on Mac OS X using Darwin Ports to obtain bdb Message-ID: <070A3D46-0F8A-437C-82C3-CD37346F69B0@pobox.com> On Mac OSX or Darwin there are two entire systems for installing 3rd party open source code: Fink[1], and Darwin Ports[2]. Each of these systems has it's advantages. For example, Darwin Ports system is bundled with Mac OS. Which means it's always at hand. To install Berkeley DB using Darwin Ports you do: sudo port install db4 Currently elephant's instructions describe how to get Berkeley DB wired up based on having installed used Fink. If you installed it using Darwin Ports then you will need to arrange your elephant Makefile so that: DB43DIR=/opt/local/ DBINCDIR=$(DB43DIR)/include/db4/ DBLIBDIR=$(DB43DIR)/lib/ And then in config.lisp you'll need to arrange to have *sleepycat- foreign-library-path* set to "/opt/local/lib/libdb-4.3.dylib". At that point it all "just works" under sbcl 0.9.14 on my powerbook G4. Well actually not, these four tests fail: PCURSOR2, INDEXING-WIPE- INDEX, INDEXING-REDEF-CLASS, INDEXING-TIMING. But generally it does a good job of appearing to working well for me. My little 3 Gigbyte data base with a half dozen classes and a dozen indexes works quite nicely. - ben [1] http://fink.sourceforge.net/ [2] http://darwinports.opendarwin.org/ From Petter.Egesund at kunnskapsforlaget.no Tue Aug 22 12:18:30 2006 From: Petter.Egesund at kunnskapsforlaget.no (Petter Egesund) Date: Tue, 22 Aug 2006 14:18:30 +0200 Subject: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Message-ID: <4FEEF5F5BB5D344BAAB5004BB495C4F104079C@EPOST3.bokklubbene.no> Hi; I have done some debugging. It seems the problem of serializing integers is caused by mixing of the types int and integer in the foreign interface? I change the function read-int in the package memutil.lisp to: (defun pread-int (buf offset) "Read a 32-bit signed integer from a foreign char buffer." (declare (optimize speed (safety 1) (debug 1)) (type (alien (* char)) buf) (type fixnum offset)) (print "Offset: ") (print offset) (the (signed-byte 32) (deref (cast (sap-alien (sap+ (alien-sap buf) offset) (* char)) (* int))))) ;; !!!!!!!!!!!!! Changed from (* integer) After this the integer-test runs fine. I will do some more testing. Cheers, Petter Egesund -----Opprinnelig melding----- Fra: Ian Eslick [mailto:eslick at csail.mit.edu] Sendt: 21. august 2006 16:11 Til: Petter Egesund Kopi: Robert L. Read; eslick at media.mit.edu; Ian Eslick Emne: Re: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Interesting. That explains the source of our problem (serializing integers) but I still don't understand why. Even if the 4 byte assumption in lisp keeps us from expanding the buffer streams appropriately the write should still write the first 8 bytes correctly and thus be able to read it correctly. It almost looks like it is writing 4 bytes but reading 8. 1) Try replacing the #x45 with #x00 in my test code The other test is to see what is actually being put into the buffer stream. 2) replace #x45 with #x40302010 and add the following code instead of the buffer-read-int at the end: (loop for i from 0 upto 16 do (format t "byte ~A: ~A~%" i (buffer-read-byte bs))) Ian Petter Egesund wrote: > > > ---------------------------------------------------------------------- > -- > *Fra:* Robert L. Read [mailto:read at robertlread.net] > *Sendt:* 19. august 2006 22:26 > *Til:* Petter Egesund > *Kopi:* eslick at media.mit.edu; Ian Eslick > *Emne:* Re: SV: SV: SV: SV: [elephant-devel] Memory error with > Elephant > > Personally, the fact that buffer-write-int in memutil.lisp assumes a > 4-byte integer: > (defun buffer-write-int (i bs) > "Write a 32-bit signed integer." > (declare (optimize (speed 3) (safety 0)) > (type buffer-stream bs) > (type (signed-byte 32) i)) > (with-struct-slots ((buf buffer-stream-buffer) > (size buffer-stream-size) > (len buffer-stream-length)) > bs > (let ((needed (+ size 4))) > (when (> needed len) > (resize-buffer-stream bs needed)) > (write-int buf i size) > (setf size needed) > nil))) > While the c-code version of write-int in libmemutil.c uses the > "sizeof(int)" paradigm seems fragile on a 64-bit architecture: > > void write_int(char *buf, int num, int offset) { > memcpy(buf+offset, &num, sizeof(int)); } > > I would like to know if "sizeof(int)" is 8 or 4 as compiled. > A simple C program (or reading the f-ing manual) would answer that > question. > An awful hack that would let us test things would be to hard-wire "4" > in place of > "sizeof(int)" in the libmemutil.c file; I don't know enough about > UFFI and sap-alien to know if that would work. > > A better solution is to expose a C-function that simply gives the size > of an integer in bytes to LISP (it could be added to libmemutil.c > file) and then have memutil.lisp use that in place of the assumption > "4".) > > How this changes the typing in memutil.lisp, I'm not sure ---- can > we/should we force the > AMD64 to use 32-bit integers, or should we rewrite memutil.lisp to > determine based on its compilation environment the size of an integer? > Either solution is better than what we have now. > > > On Sat, 2006-08-19 at 20:53 +0200, Petter Egesund wrote: >> It looks like below. Certainly not 35. >> >> Petter >> >> ------- >> >> ELE> (defun fixnum-test () >> (with-buffer-streams (bs) >> (buffer-write-int #x23 bs) >> (buffer-write-int #x45 bs) >> (reset-buffer-stream bs) >> (buffer-read-int bs))) >> ; in: LAMBDA NIL >> ; (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 69 ELEPHANT::BS) >> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET >> WHEN ; --> COND IF PROGN ; ==> >> ; (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS >> ; ELEPHANT-MEMUTIL::NEEDED) >> ; >> ; note: doing signed word to integer coercion (cost 20) >> >> ; (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 35 ELEPHANT::BS) >> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET >> WHEN ; --> COND IF PROGN ; ==> >> ; (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS >> ; ELEPHANT-MEMUTIL::NEEDED) >> ; >> ; note: doing signed word to integer coercion (cost 20) ; ; >> compilation unit finished >> ; printed 2 notes >> FIXNUM-TEST >> ELE> (fixnum-test) >> 296352743459 >> ELE> >> >> >> >> >> >> >> -----Opprinnelig melding----- >> Fra: Ian Eslick [mailto:eslick at csail.mit.edu >> ] >> Sendt: 19. august 2006 20:21 >> Til: Petter Egesund >> Kopi: eslick at media.mit.edu ; Robert L. >> Read; Ian Eslick >> Emne: Re: SV: SV: SV: [elephant-devel] Memory error with Elephant >> >> What happens if you write two fixnums and read a fixnum by hand? >> >> (defun fixnum-test () >> (with-buffer-streams (bs) >> (buffer-write-int #x23 bs) >> (buffer-write-int #x45 bs) >> (reset-buffer-stream bs) >> (buffer-read-int bs))) >> >> Running this function should return 35 (#x23). >> >> Ian >> >> Petter Egesund wrote: >> > Hi, thanks for advices. >> > >> > I have had a quick look. It seems the problem lies in the >> > serializing, which might be illustrated by the fact that the error >> > occurs when entering these small lines; >> > >> > (in-package "ELEPHANT") >> > (defvar str (coerce "this is a test" 'base-string)) >> > (ser-deser-equal >> > str) >> > >> > I have had a look at the code, a rather wild guess might be that the problem is related to the function (buffer-read-fixnum bs) combined with reading from the buffer? Seems like byte-length, endian and stuff like this is relevant? >> > >> > I have not had other problems with the 64-bit Sbcl so far. I have been running a few other programs, like a web-server and some parser stuff. The 32-bits version did run, but it had problems with linking to libraries (which might probably be solved). On my private PC which is a 32-bits Slackware, everything seems to work well, Elephant also. >> > >> > I will try to do some tracing on Tonday/Tuesday. >> > >> > Cheers, >> > >> > Petter >> > >> > >> > >> > >> > >> > >> > >> > >> > -----Opprinnelig melding----- >> > Fra: Ian Eslick [mailto:eslick at csail.mit.edu >> > ] >> > Sendt: 19. august 2006 01:11 >> > Til: Robert L. Read >> > Kopi: Petter Egesund; Ian Eslick >> > Emne: Re: SV: SV: [elephant-devel] Memory error with Elephant >> > >> > I got my head wrapped around Elephant again recently so I'd be happy to assist. I'm on vacation this week and the latter part of next week so won't be highly responsive, but I think this is such a major problem it actually should be quite easy to track down. >> > >> > I did observe that it appears to be unbounded primitives like bignum and strings that fail. >> > >> > Some possible candidates off the top of my head: >> > - When reading a fixnum back for the length of a string, bignum, etc we are actually reading adjacent >> > values causing the allocator to request a string length in the gigabytes... >> > - Also, length calculation in memutil.lisp may mismatch with the >> > actual string length >> > - Some memutil.lisp system-dependent parameter such as byte-length, etc might not be setup properly for AMD 64 systems. >> > >> > Don't forget that the SBCL 64-bit implementation is new so if compiled >> > for native AMD-64 it might have a bug. Have you tried a 32-bit SBCL on >> > your AMD-64 CPU? >> > >> > Anyway, see if you can trace serialize and (labels serialize %serialize) as well as deserialize and %deserialize put a break at the entry point to each so you can allow computation to continue at each step and see why we're getting the strange memory requests. Then just create a buffer-stream using 'with-buffer-streams' and try serializing a string to it, resetting the stream and reading the string back using deserialize (all buffer stream methods are in src/memutil/memutil.lisp). >> > >> > That should get you close to the problem. >> > >> > Thanks for the help, I'll pitch in as I can. >> > >> > Cheers, >> > Ian >> > >> > >> > Robert L. Read wrote: >> > >> >> Ughh --- you make a kind and generous offer, which I would >> >> definitely accept if I weren't so busy with other things. >> >> >> >> How about this? >> >> You spend some time, a few hours maybe, trying to get the simplest >> >> serialization tests working. If you can clearly reduce it to the >> >> first first test that fails and perhaps track it down to the >> >> module or the assumption in question, then I will log into your >> >> computer and either debug it further, or see if there is (for >> >> example), a simple improvement in the serializer that will let it get further. >> >> >> >> On Fri, 2006-08-18 at 22:58 +0200, Petter Egesund wrote: >> >> >> >>> A quick guess is that this has nothing to do with the >> >>> BDB-backend, it seems it fails before it reaches the database point? >> >>> >> >>> I am not sure where to start debugging, but I can offer you a >> >>> user on the AMD64 if that could be of any help? >> >>> >> >>> Anyhow I will try to do some small debugging - this package works >> >>> so well on my private 32-bits Linux... >> >>> >> >>> Petter >> >>> >> >>> ----------------------------------------------------------------- >> >>> --- >> >>> - >> >>> --- >> >>> >> >>> *Fra:* Robert L. Read [mailto:read at robertlread.net >> >>> ] >> >>> *Sendt:* 18. august 2006 22:51 >> >>> *Til:* Petter Egesund >> >>> *Kopi:* Elephant-devel mailing list >> >>> *Emne:* Re: SV: [elephant-devel] Memory error with Elephant >> >>> >> >>> >> >>> Well, its good we know this ---- there is zero hope of getting >> >>> Elephant to work until we solve this problem. >> >>> >> >>> My hunch would be this has to do with the AMD64. >> >>> >> >>> It is entirely possible that the serializer makes some assumption >> >>> that isn't true on AMD64; the serializer gets into some >> >>> byte-representation issues that could fail there. >> >>> >> >>> If you would like to debug it, I would be happy to answer your >> >>> questions. I hope to use a 64-bit processor in my own business >> >>> soon, but I don't have one available now. >> >>> >> >>> Does anyone else have this working on a 64 bit processor? >> >>> >> >>> >> >>> On Fri, 2006-08-18 at 22:37 +0200, Petter Egesund wrote: >> >>> >> >>>> Thanks for answering! >> >>>> >> >>>> I did run one test, the not-so-good result was like below. Any clues? >> >>>> >> >>>> Cheers, >> >>>> >> >>>> Petter >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> ------------ >> >>>> >> >>>> ; SLIME 2006-04-20 >> >>>> CL-USER> (asdf:operate 'asdf:load-op :ele-bdb) >> >>>> ; loading system definition from >> >>>> /home/pe/.sbcl/systems/ele-bdb.asd >> >>>> into ; # ; loading system definition from >> >>>> /home/pe/.sbcl/systems/uffi.asd into ; # ; >> >>>> registering # as UFFI ; registering >> >>>> # as ELE-BDB ; loading system >> >>>> definition from /home/pe/.sbcl/systems/elephant.asd into ; >> >>>> # ; registering # >> >>>> as ELEPHANT >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> PERSISTENT-SLOTS >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function OLD-PERSISTENT-SLOTS >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> UPDATE-PERSISTENT-SLOTS >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> INDEXED-RECORD >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> OLD-INDEXED-RECORD >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> UPDATE-INDEXED-RECORD >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> MAKE-NEW-INDEXED-RECORD >> >>>> STYLE-WARNING: implicitly creating new generic function REMOVED-INDEXING? >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> REGISTER-INDEXED-SLOT >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> UNREGISTER-INDEXED-SLOT >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> REGISTER-DERIVED-INDEX >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> UNREGISTER-DERIVED-INDEX >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> PREVIOUSLY-INDEXED ; loading system definition from >> >>>> /home/pe/.sbcl/systems/cl-base64.asd into ; # ; >> >>>> registering # as CL-BASE64 ; >> >>>> registering # as >> >>>> CL-BASE64-TESTS ; loading system definition from >> >>>> /home/pe/.sbcl/systems/kmrcl.asd into ; # ; >> >>>> registering # as KMRCL >> >>>> STYLE-WARNING: implicitly creating new generic function GET-CON >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> CONTROLLER-PROPERTIES >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> SET-ELE-PROPERTY >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> GET-ELE-PROPERTY >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> ENSURE-MARKED-VERSION >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> CONTROLLER-VERSION >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> UP-TO-DATE-P >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> UPGRADABLE-P >> >>>> STYLE-WARNING: implicitly creating new generic function UPGRADE >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function FLUSH-INSTANCE-CACHE >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> DROP-POBJECT >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> MAP-BTREE >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> EMPTY-BTREE-P >> >>>> STYLE-WARNING: implicitly creating new generic function CLASS-INDEX-CACHED? >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> DETERMINE-SYNCH-METHOD >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> SET-DB-SYNCH >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> INDEXED-SLOT-WRITER >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> CLASS-INDEXEDP-BY-NAME >> >>>> STYLE-WARNING: >> >>>> implicitly creating new generic function >> >>>> FIND-INVERTED-INDEX-NAMES >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> WIPE-CLASS-INDEXING >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> COPY-BTREE-CONTENTS >> >>>> STYLE-WARNING: implicitly creating new generic function >> >>>> BUILD-BTREE-INDEX NIL >> >>>> CL-USER> (asdf:operate 'asdf:load-op :elephant-tests) >> >>>> ; loading system definition from >> >>>> /home/pe/.sbcl/systems/elephant-tests.asd >> >>>> ; into # >> >>>> ; registering # as >> >>>> ELEPHANT-TESTS ; registering #> >>>> {1003B28C91}> as ; ELEPHANT-TESTS-BDB ; loading system >> >>>> definition from /home/pe/.sbcl/systems/rt.asd into ; #> >>>> "ASDF0"> ; registering # as RT NIL >> >>>> CL-USER> (in-package "ELEPHANT-TESTS") >> >>>> # >> >>>> ELE-TESTS> (setf *default-spec* *testbdb-spec*) >> >>>> (:BDB "/home/pe/.sbcl/site/elephant/tests/testdb/") >> >>>> ELE-TESTS> (do-backend-tests) >> >>>> ; compiling file "/home/pe/.sbcl/site/elephant/tests/testsleepycat.lisp" (written 19 FEB 2006 05:53:02 AM): >> >>>> ; compiling (IN-PACKAGE "ELE-TESTS") ; compiling (DEFVAR ENV) ; >> >>>> compiling (DEFVAR DB) ; compiling (DEFUN PREPARE-SLEEPYCAT ...) >> >>>> ; compiling (DEFTEST PREPARES-SLEEPYCAT ...) ; compiling (DEFUN >> >>>> TEST-SEQUENCE1 ...) ; compiling (DEFTEST TEST-SEQ1 ...) ; >> >>>> compiling (DEFUN TEST-SEQUENCE2 ...) ; compiling (DEFTEST >> >>>> TEST-SEQ2 ...) ; compiling (DEFUN CLEANUP-SLEEPYCAT ...) ; >> >>>> compiling (DEFTEST CLEANSUP-SLEEPYCAT ...) >> >>>> >> >>>> ; /home/pe/.sbcl/site/elephant/tests/testsleepycat.fasl written >> >>>> ; compilation finished in 0:00:00 Doing 115 pending tests of 115 >> >>>> tests total. >> >>>> FIXNUMS FIXNUM-TYPE-1 >> >>>> Test BIGNUMS failed >> >>>> Form: (ARE-NOT-NULL (IN-OUT-EQUAL 10000000000) (IN-OUT-EQUAL -10000000000) >> >>>> (LOOP FOR I FROM 0 TO 2000 ALWAYS >> >>>> (IN-OUT-EQUAL (EXPT 2 I))) >> >>>> (LOOP FOR I FROM 0 TO 2000 ALWAYS >> >>>> (IN-OUT-EQUAL (- (EXPT 2 I)))) >> >>>> (LOOP FOR I FROM 0 TO 2000 ALWAYS >> >>>> (IN-OUT-EQUAL (- (EXPT 2 I) 1))) >> >>>> (LOOP FOR I FROM 0 TO 2000 ALWAYS >> >>>> (IN-OUT-EQUAL (- 1 (EXPT 2 I)))) >> >>>> (LOOP FOR I FROM 0 TO 2000 ALWAYS >> >>>> (IN-OUT-EQUAL (EXPT 3 I))) >> >>>> (LOOP FOR I FROM 0 TO 2000 ALWAYS >> >>>> (IN-OUT-EQUAL (- (EXPT 3 I))))) >> >>>> Expected >> >>>> values: T >> >>>> T >> >>>> T >> >>>> T >> >>>> T >> >>>> T >> >>>> T >> >>>> T >> >>>> Actual value: #. >> >>>> FLOATS >> >>>> Test RATIONALS failed >> >>>> Form: (ARE-NOT-NULL (IN-OUT-EQUAL 1/2) (IN-OUT-EQUAL -1/2) >> >>>> (IN-OUT-EQUAL (/ 1 MOST-POSITIVE-FIXNUM)) >> >>>> (IN-OUT-EQUAL (/ 1 MOST-NEGATIVE-FIXNUM)) >> >>>> (IN-OUT-EQUAL >> >>>> (/ MOST-POSITIVE-FIXNUM MOST-NEGATIVE-FIXNUM)) >> >>>> (IN-OUT-EQUAL (/ (EXPT 2 200) (EXPT 3 300))) >> >>>> (IN-OUT-EQUAL (/ (EXPT 2 200) (- (EXPT 3 >> >>>> 300))))) Expected values: T >> >>>> T >> >>>> T >> >>>> T >> >>>> T >> >>>> T >> >>>> T >> >>>> Actual value: #. >> >>>> Test BASE-STRINGS failed >> >>>> Form: (ARE-NOT-NULL >> >>>> (IN-OUT-EQUAL (MAKE-STRING 0 :ELEMENT-TYPE 'BASE-CHAR)) >> >>>> (IN-OUT-EQUAL (COERCE "this is a test" 'BASE-STRING)) >> >>>> (IN-OUT-EQUAL >> >>>> (MAKE-STRING 400 :INITIAL-ELEMENT (CODE-CHAR 127) :ELEMENT-TYPE >> >>>> 'BASE-CHAR))) Expected values: T >> >>>> T >> >>>> T >> >>>> Actual value: #. >> >>>> Heap exhausted: 8517255168 bytes available, 498216206416 requested. PROCEED WITH CAUTION! >> >>>> [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] >> >>>> >> >>>> Restarts: >> >>>> 0: [ABORT-REQUEST] Abort handling SLIME request. >> >>>> 1: [TERMINATE-THREAD] Terminate this thread (#> >>>> "repl-thread" {1003511531}>) >> >>>> >> >>>> Backtrace: >> >>>> 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8517255168 498216206416) >> >>>> 1: ("foreign function: call_into_lisp") >> >>>> 2: ("foreign function: funcall2") >> >>>> 3: ("foreign function: gc_heap_exhausted_error_or_lose") >> >>>> 4: ("foreign function: gc_find_freeish_pages") >> >>>> 5: ("foreign function: gc_alloc_large") >> >>>> 6: ("foreign function: alloc_tramp") >> >>>> 7: ((LABELS ELEPHANT::%DESERIALIZE) #) >> >>>> 8: (IN-OUT-EQUAL "this is a test") >> >>>> 9: (SB-INT:EVAL-IN-LEXENV (IN-OUT-EQUAL "this is a test") >> >>>> #) >> >>>> 10: (SB-INT:EVAL-IN-LEXENV (PROGN (IN-OUT-EQUAL "this is a >> >>>> test")) >> >>>> #) >> >>>> 11: (SB-INT:EVAL-IN-LEXENV (NULL (PROGN (IN-OUT-EQUAL "this is >> >>>> a >> >>>> test"))) #) >> >>>> 12: (SB-INT:EVAL-IN-LEXENV (IS-NOT-NULL (IN-OUT-EQUAL "this is >> >>>> a >> >>>> test")) #) >> >>>> 13: (SB-INT:EVAL-IN-LEXENV (ARE-NOT-NULL (IN-OUT-EQUAL "") >> >>>> (IN-OUT-EQUAL "this is a test") (IN-OUT-EQUAL (MAKE-STRING 400 >> >>>> :INITIAL-ELEMENT #))) #) >> >>>> 14: ((FLET REGRESSION-TEST::%DO)) >> >>>> 15: (REGRESSION-TEST::DO-ENTRY #S(REGRESSION-TEST::ENTRY :PEND >> >>>> T :NAME STRINGS :PROPS NIL :FORM (ARE-NOT-NULL (IN-OUT-EQUAL "") >> >>>> (IN-OUT-EQUAL "this is a test") (IN-OUT-EQUAL #)) :VALS (T T T)) >> >>>> #) >> >>>> 16: (REGRESSION-TEST::DO-ENTRIES* >> >>>> #) >> >>>> 17: (REGRESSION-TEST::DO-ENTRIES >> >>>> #) >> >>>> 18: (DO-BACKEND-TESTS (:BDB >> >>>> "/home/pe/.sbcl/site/elephant/tests/testdb/")) >> >>>> 19: (SB-INT:EVAL-IN-LEXENV (DO-BACKEND-TESTS) #) >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> ________________________________ >> >>>> >> >>>> Fra: elephant-devel-bounces at common-lisp.net >> >>>> >> >>>> > >>>> > >> >>>> [mailto:elephant-devel-bounces at common-lisp.net >> >>>> >> >>>> > >>>> >] P? vegne av >> >>>> Robert L. Read >> >>>> Sendt: 18. august 2006 22:04 >> >>>> Til: Elephant bugs and development >> >>>> Emne: Re: [elephant-devel] Memory error with Elephant >> >>>> >> >>>> >> >>>> That mystifies me. I can only conjecture that it is somehow related to your environment. >> >>>> >> >>>> Even thought it may seem strange since that simplest of >> >>>> functionality doesn't work, you might wish to execute the test. >> >>>> If, for example, there were an infinite loop in the serializer >> >>>> when compiled in your environment, the larges suite of automated tests might reveal that, as opposed to a problem with BDB, for example. >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> From eslick at csail.mit.edu Tue Aug 22 13:42:57 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 22 Aug 2006 09:42:57 -0400 Subject: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Message-ID: <200608221337.k7MDb8Ax011851@bayer.media.mit.edu> I don't have internet at the moment, but what is the definition of int vs integer in the sb-alien foreign interface? Just want to make sure we aren't asking for trouble in the future. (i.e. is int signed or unsigned vs integer? Is it 32bit for compatibility or whatever the clib it's linked to says it is?) Ian -----Original Message----- From: "Petter Egesund" To: eslick at media.mit.edu Cc: "Robert L. Read" ; "Ian Eslick" ; elephant-devel at common-lisp.net Sent: 8/22/06 8:18 AM Subject: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Hi; I have done some debugging. It seems the problem of serializing integers is caused by mixing of the types int and integer in the foreign interface? I change the function read-int in the package memutil.lisp to: (defun pread-int (buf offset) "Read a 32-bit signed integer from a foreign char buffer." (declare (optimize speed (safety 1) (debug 1)) (type (alien (* char)) buf) (type fixnum offset)) (print "Offset: ") (print offset) (the (signed-byte 32) (deref (cast (sap-alien (sap+ (alien-sap buf) offset) (* char)) (* int))))) ;; !!!!!!!!!!!!! Changed from (* integer) After this the integer-test runs fine. I will do some more testing. Cheers, Petter Egesund -----Opprinnelig melding----- Fra: Ian Eslick [mailto:eslick at csail.mit.edu] Sendt: 21. august 2006 16:11 Til: Petter Egesund Kopi: Robert L. Read; eslick at media.mit.edu; Ian Eslick Emne: Re: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Interesting. That explains the source of our problem (serializing integers) but I still don't understand why. Even if the 4 byte assumption in lisp keeps us from expanding the buffer streams appropriately the write should still write the first 8 bytes correctly and thus be able to read it correctly. It almost looks like it is writing 4 bytes but reading 8. 1) Try replacing the #x45 with #x00 in my test code The other test is to see what is actually being put into the buffer stream. 2) replace #x45 with #x40302010 and add the following code instead of the buffer-read-int at the end: (loop for i from 0 upto 16 do (format t "byte ~A: ~A~%" i (buffer-read-byte bs))) Ian Petter Egesund wrote: > > > ---------------------------------------------------------------------- > -- > *Fra:* Robert L. Read [mailto:read at robertlread.net] > *Sendt:* 19. august 2006 22:26 > *Til:* Petter Egesund > *Kopi:* eslick at media.mit.edu; Ian Eslick > *Emne:* Re: SV: SV: SV: SV: [elephant-devel] Memory error with > Elephant > > Personally, the fact that buffer-write-int in memutil.lisp assumes a > 4-byte integer: > (defun buffer-write-int (i bs) > "Write a 32-bit signed integer." > (declare (optimize (speed 3) (safety 0)) > (type buffer-stream bs) > (type (signed-byte 32) i)) > (with-struct-slots ((buf buffer-stream-buffer) > (size buffer-stream-size) > (len buffer-stream-length)) > bs > (let ((needed (+ size 4))) > (when (> needed len) > (resize-buffer-stream bs needed)) > (write-int buf i size) > (setf size needed) > nil))) > While the c-code version of write-int in libmemutil.c uses the > "sizeof(int)" paradigm seems fragile on a 64-bit architecture: > > void write_int(char *buf, int num, int offset) { > memcpy(buf+offset, &num, sizeof(int)); } > > I would like to know if "sizeof(int)" is 8 or 4 as compiled. > A simple C program (or reading the f-ing manual) would answer that > question. > An awful hack that would let us test things would be to hard-wire "4" > in place of > "sizeof(int)" in the libmemutil.c file; I don't know enough about > UFFI and sap-alien to know if that would work. > > A better solution is to expose a C-function that simply gives the size > of an integer in bytes to LISP (it could be added to libmemutil.c > file) and then have memutil.lisp use that in place of the assumption > "4".) > > How this changes the typing in memutil.lisp, I'm not sure ---- can > we/should we force the > AMD64 to use 32-bit integers, or should we rewrite memutil.lisp to > determine based on its compilation environment the size of an integer? > Either solution is better than what we have now. > > > On Sat, 2006-08-19 at 20:53 +0200, Petter Egesund wrote: >> It looks like below. Certainly not 35. >> >> Petter >> >> ------- >> >> ELE> (defun fixnum-test () >> (with-buffer-streams (bs) >> (buffer-write-int #x23 bs) >> (buffer-write-int #x45 bs) >> (reset-buffer-stream bs) >> (buffer-read-int bs))) >> ; in: LAMBDA NIL >> ; (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 69 ELEPHANT::BS) >> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET >> WHEN ; --> COND IF PROGN ; ==> >> ; (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS >> ; ELEPHANT-MEMUTIL::NEEDED) >> ; >> ; note: doing signed word to integer coercion (cost 20) >> >> ; (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 35 ELEPHANT::BS) >> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET >> WHEN ; --> COND IF PROGN ; ==> >> ; (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS >> ; ELEPHANT-MEMUTIL::NEEDED) >> ; >> ; note: doing signed word to integer coercion (cost 20) ; ; >> compilation unit finished >> ; printed 2 notes >> FIXNUM-TEST >> ELE> (fixnum-test) >> 296352743459 >> ELE> >> >> >> >> >> From eslick at csail.mit.edu Tue Aug 22 18:36:19 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 22 Aug 2006 14:36:19 -0400 Subject: SV: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant In-Reply-To: <4FEEF5F5BB5D344BAAB5004BB495C4F104079F@EPOST3.bokklubbene.no> References: <4FEEF5F5BB5D344BAAB5004BB495C4F104079F@EPOST3.bokklubbene.no> Message-ID: <44EB4EA3.4060706@csail.mit.edu> There are definitely some issues with string calculations. I've had some issues dealing with allegro's unicode support. Look in memutil.lisp: The byte-length macro Look in serializer.lisp: (method serialize %serialize) the string case statements Serialization of strings involves: 1) Serialize appropriate string tag based on Lisp character width for string type In SBCL this is 1 byte or 4 bytes depending on Unicode support 2) Serialize the total string length (bytes * num chars). The byte-length macro is used for this 3) Serialize the array of characters using buffer-write-string Hopefully that will help you figure out where the mismatch occurs. Cheers, Ian Petter Egesund wrote: > I am not sure, I have been looking for good documentation. I might exist > somewhere, though? I am guessing int in FFI is closer connected to int > as is defined by C and thereby the OS? Somebody can probably tell us... > > It seems also to be a problem computing the size of a buffer, as well, > which causes problems for strings. I will take a closer look tomorrow, > tonight I have to struggle with some deadlines... > > Petter > > > -----Opprinnelig melding----- > Fra: Ian Eslick [mailto:eslick at csail.mit.edu] > Sendt: 22. august 2006 15:43 > Til: Petter Egesund; eslick at media.mit.edu > Kopi: Robert L. Read; Ian Eslick; elephant-devel at common-lisp.net > Emne: RE: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with > Elephant > > I don't have internet at the moment, but what is the definition of int > vs integer in the sb-alien foreign interface? Just want to make sure we > aren't asking for trouble in the future. (i.e. is int signed or > unsigned vs integer? Is it 32bit for compatibility or whatever the clib > it's linked to says it is?) > > Ian > > -----Original Message----- > From: "Petter Egesund" > To: eslick at media.mit.edu > Cc: "Robert L. Read" ; "Ian Eslick" > ; elephant-devel at common-lisp.net > Sent: 8/22/06 8:18 AM > Subject: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with > Elephant > > Hi; > > I have done some debugging. It seems the problem of serializing integers > is caused by mixing of the types int and integer in the foreign > interface? > > I change the function read-int in the package memutil.lisp to: > > (defun pread-int (buf offset) > "Read a 32-bit signed integer from a foreign char buffer." > (declare (optimize speed (safety 1) (debug 1)) > (type (alien (* char)) buf) > (type fixnum offset)) > (print "Offset: ") (print offset) > (the (signed-byte 32) > (deref (cast (sap-alien (sap+ (alien-sap buf) offset) (* char)) > (* int))))) ;; !!!!!!!!!!!!! Changed from (* integer) > > > After this the integer-test runs fine. I will do some more testing. > > Cheers, > > Petter Egesund > > > > > > -----Opprinnelig melding----- > Fra: Ian Eslick [mailto:eslick at csail.mit.edu] > Sendt: 21. august 2006 16:11 > Til: Petter Egesund > Kopi: Robert L. Read; eslick at media.mit.edu; Ian Eslick > Emne: Re: SV: SV: SV: SV: SV: [elephant-devel] Memory error with > Elephant > > Interesting. That explains the source of our problem (serializing > integers) but I still don't understand why. Even if the 4 byte > assumption in lisp keeps us from expanding the buffer streams > appropriately the write should still write the first 8 bytes correctly > and thus be able to read it correctly. It almost looks like it is > writing 4 bytes but reading 8. > > 1) Try replacing the #x45 with #x00 in my test code > > The other test is to see what is actually being put into the buffer > stream. > > 2) replace #x45 with #x40302010 and add the following code instead of > the buffer-read-int at the end: > > (loop for i from 0 upto 16 do > (format t "byte ~A: ~A~%" i (buffer-read-byte bs))) > > Ian > > Petter Egesund wrote: > >> >> >> ---------------------------------------------------------------------- >> -- >> *Fra:* Robert L. Read [mailto:read at robertlread.net] >> *Sendt:* 19. august 2006 22:26 >> *Til:* Petter Egesund >> *Kopi:* eslick at media.mit.edu; Ian Eslick >> *Emne:* Re: SV: SV: SV: SV: [elephant-devel] Memory error with >> Elephant >> >> Personally, the fact that buffer-write-int in memutil.lisp assumes a >> 4-byte integer: >> (defun buffer-write-int (i bs) >> "Write a 32-bit signed integer." >> (declare (optimize (speed 3) (safety 0)) >> (type buffer-stream bs) >> (type (signed-byte 32) i)) >> (with-struct-slots ((buf buffer-stream-buffer) >> (size buffer-stream-size) >> (len buffer-stream-length)) >> bs >> (let ((needed (+ size 4))) >> (when (> needed len) >> (resize-buffer-stream bs needed)) >> (write-int buf i size) >> (setf size needed) >> nil))) >> While the c-code version of write-int in libmemutil.c uses the >> "sizeof(int)" paradigm seems fragile on a 64-bit architecture: >> >> void write_int(char *buf, int num, int offset) { >> memcpy(buf+offset, &num, sizeof(int)); } >> >> I would like to know if "sizeof(int)" is 8 or 4 as compiled. >> A simple C program (or reading the f-ing manual) would answer that >> question. >> An awful hack that would let us test things would be to hard-wire "4" >> in place of >> "sizeof(int)" in the libmemutil.c file; I don't know enough about >> UFFI and sap-alien to know if that would work. >> >> A better solution is to expose a C-function that simply gives the size >> > > >> of an integer in bytes to LISP (it could be added to libmemutil.c >> file) and then have memutil.lisp use that in place of the assumption >> "4".) >> >> How this changes the typing in memutil.lisp, I'm not sure ---- can >> we/should we force the >> AMD64 to use 32-bit integers, or should we rewrite memutil.lisp to >> determine based on its compilation environment the size of an integer? >> > > >> Either solution is better than what we have now. >> >> >> On Sat, 2006-08-19 at 20:53 +0200, Petter Egesund wrote: >> >>> It looks like below. Certainly not 35. >>> >>> Petter >>> >>> ------- >>> >>> ELE> (defun fixnum-test () >>> (with-buffer-streams (bs) >>> (buffer-write-int #x23 bs) >>> (buffer-write-int #x45 bs) >>> (reset-buffer-stream bs) >>> (buffer-read-int bs))) >>> ; in: LAMBDA NIL >>> ; (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 69 ELEPHANT::BS) >>> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET >>> WHEN ; --> COND IF PROGN ; ==> >>> ; (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS >>> ; ELEPHANT-MEMUTIL::NEEDED) >>> ; >>> ; note: doing signed word to integer coercion (cost 20) >>> >>> ; (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 35 ELEPHANT::BS) >>> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET >>> WHEN ; --> COND IF PROGN ; ==> >>> ; (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS >>> ; ELEPHANT-MEMUTIL::NEEDED) >>> ; >>> ; note: doing signed word to integer coercion (cost 20) ; ; >>> compilation unit finished >>> ; printed 2 notes >>> FIXNUM-TEST >>> ELE> (fixnum-test) >>> 296352743459 >>> ELE> >>> >>> >>> >>> >>> >>> > > From Petter.Egesund at kunnskapsforlaget.no Tue Aug 22 18:18:47 2006 From: Petter.Egesund at kunnskapsforlaget.no (Petter Egesund) Date: Tue, 22 Aug 2006 20:18:47 +0200 Subject: SV: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Message-ID: <4FEEF5F5BB5D344BAAB5004BB495C4F104079F@EPOST3.bokklubbene.no> I am not sure, I have been looking for good documentation. I might exist somewhere, though? I am guessing int in FFI is closer connected to int as is defined by C and thereby the OS? Somebody can probably tell us... It seems also to be a problem computing the size of a buffer, as well, which causes problems for strings. I will take a closer look tomorrow, tonight I have to struggle with some deadlines... Petter -----Opprinnelig melding----- Fra: Ian Eslick [mailto:eslick at csail.mit.edu] Sendt: 22. august 2006 15:43 Til: Petter Egesund; eslick at media.mit.edu Kopi: Robert L. Read; Ian Eslick; elephant-devel at common-lisp.net Emne: RE: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant I don't have internet at the moment, but what is the definition of int vs integer in the sb-alien foreign interface? Just want to make sure we aren't asking for trouble in the future. (i.e. is int signed or unsigned vs integer? Is it 32bit for compatibility or whatever the clib it's linked to says it is?) Ian -----Original Message----- From: "Petter Egesund" To: eslick at media.mit.edu Cc: "Robert L. Read" ; "Ian Eslick" ; elephant-devel at common-lisp.net Sent: 8/22/06 8:18 AM Subject: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Hi; I have done some debugging. It seems the problem of serializing integers is caused by mixing of the types int and integer in the foreign interface? I change the function read-int in the package memutil.lisp to: (defun pread-int (buf offset) "Read a 32-bit signed integer from a foreign char buffer." (declare (optimize speed (safety 1) (debug 1)) (type (alien (* char)) buf) (type fixnum offset)) (print "Offset: ") (print offset) (the (signed-byte 32) (deref (cast (sap-alien (sap+ (alien-sap buf) offset) (* char)) (* int))))) ;; !!!!!!!!!!!!! Changed from (* integer) After this the integer-test runs fine. I will do some more testing. Cheers, Petter Egesund -----Opprinnelig melding----- Fra: Ian Eslick [mailto:eslick at csail.mit.edu] Sendt: 21. august 2006 16:11 Til: Petter Egesund Kopi: Robert L. Read; eslick at media.mit.edu; Ian Eslick Emne: Re: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Interesting. That explains the source of our problem (serializing integers) but I still don't understand why. Even if the 4 byte assumption in lisp keeps us from expanding the buffer streams appropriately the write should still write the first 8 bytes correctly and thus be able to read it correctly. It almost looks like it is writing 4 bytes but reading 8. 1) Try replacing the #x45 with #x00 in my test code The other test is to see what is actually being put into the buffer stream. 2) replace #x45 with #x40302010 and add the following code instead of the buffer-read-int at the end: (loop for i from 0 upto 16 do (format t "byte ~A: ~A~%" i (buffer-read-byte bs))) Ian Petter Egesund wrote: > > > ---------------------------------------------------------------------- > -- > *Fra:* Robert L. Read [mailto:read at robertlread.net] > *Sendt:* 19. august 2006 22:26 > *Til:* Petter Egesund > *Kopi:* eslick at media.mit.edu; Ian Eslick > *Emne:* Re: SV: SV: SV: SV: [elephant-devel] Memory error with > Elephant > > Personally, the fact that buffer-write-int in memutil.lisp assumes a > 4-byte integer: > (defun buffer-write-int (i bs) > "Write a 32-bit signed integer." > (declare (optimize (speed 3) (safety 0)) > (type buffer-stream bs) > (type (signed-byte 32) i)) > (with-struct-slots ((buf buffer-stream-buffer) > (size buffer-stream-size) > (len buffer-stream-length)) > bs > (let ((needed (+ size 4))) > (when (> needed len) > (resize-buffer-stream bs needed)) > (write-int buf i size) > (setf size needed) > nil))) > While the c-code version of write-int in libmemutil.c uses the > "sizeof(int)" paradigm seems fragile on a 64-bit architecture: > > void write_int(char *buf, int num, int offset) { > memcpy(buf+offset, &num, sizeof(int)); } > > I would like to know if "sizeof(int)" is 8 or 4 as compiled. > A simple C program (or reading the f-ing manual) would answer that > question. > An awful hack that would let us test things would be to hard-wire "4" > in place of > "sizeof(int)" in the libmemutil.c file; I don't know enough about > UFFI and sap-alien to know if that would work. > > A better solution is to expose a C-function that simply gives the size > of an integer in bytes to LISP (it could be added to libmemutil.c > file) and then have memutil.lisp use that in place of the assumption > "4".) > > How this changes the typing in memutil.lisp, I'm not sure ---- can > we/should we force the > AMD64 to use 32-bit integers, or should we rewrite memutil.lisp to > determine based on its compilation environment the size of an integer? > Either solution is better than what we have now. > > > On Sat, 2006-08-19 at 20:53 +0200, Petter Egesund wrote: >> It looks like below. Certainly not 35. >> >> Petter >> >> ------- >> >> ELE> (defun fixnum-test () >> (with-buffer-streams (bs) >> (buffer-write-int #x23 bs) >> (buffer-write-int #x45 bs) >> (reset-buffer-stream bs) >> (buffer-read-int bs))) >> ; in: LAMBDA NIL >> ; (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 69 ELEPHANT::BS) >> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET >> WHEN ; --> COND IF PROGN ; ==> >> ; (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS >> ; ELEPHANT-MEMUTIL::NEEDED) >> ; >> ; note: doing signed word to integer coercion (cost 20) >> >> ; (ELEPHANT-MEMUTIL:BUFFER-WRITE-INT 35 ELEPHANT::BS) >> ; --> BLOCK ELEPHANT-MEMUTIL::WITH-STRUCT-SLOTS SYMBOL-MACROLET LET >> WHEN ; --> COND IF PROGN ; ==> >> ; (ELEPHANT-MEMUTIL:RESIZE-BUFFER-STREAM ELEPHANT-MEMUTIL::BS >> ; ELEPHANT-MEMUTIL::NEEDED) >> ; >> ; note: doing signed word to integer coercion (cost 20) ; ; >> compilation unit finished >> ; printed 2 notes >> FIXNUM-TEST >> ELE> (fixnum-test) >> 296352743459 >> ELE> >> >> >> >> >> From nowhere.man at levallois.eu.org Tue Aug 22 19:52:35 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 22 Aug 2006 21:52:35 +0200 Subject: [elephant-devel] Small glitches in the release tarball Message-ID: <20060822195235.GE13707@bateleur.arcanes.fr.eu.org> Hi, I'm just about to try to package some Lisp systems I use for Debian, and I noted some minor problems with the Elephant's tarballs: - it contains an 'elephant' dir, instead of 'elephant-0.6.0' (if you extract in a dir also containing the CVS dir, it will be a problem) - it contains the CVS dirs Quickly, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From read at robertlread.net Tue Aug 22 21:01:47 2006 From: read at robertlread.net (Robert L. Read) Date: Tue, 22 Aug 2006 16:01:47 -0500 Subject: [elephant-devel] Small glitches in the release tarball In-Reply-To: <20060822195235.GE13707@bateleur.arcanes.fr.eu.org> References: <20060822195235.GE13707@bateleur.arcanes.fr.eu.org> Message-ID: <1156280507.4669.8.camel@localhost.localdomain> OK --- I will try to fix this by Thursday at the latest. Thanks for pointing this out, and thanks for packaging it. On Tue, 2006-08-22 at 21:52 +0200, Pierre THIERRY wrote: > Hi, > > I'm just about to try to package some Lisp systems I use for Debian, and > I noted some minor problems with the Elephant's tarballs: > > - it contains an 'elephant' dir, instead of 'elephant-0.6.0' (if you > extract in a dir also containing the CVS dir, it will be a problem) > - it contains the CVS dirs > > Quickly, > Nowhere man > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Tue Aug 22 21:24:43 2006 From: read at robertlread.net (Robert L. Read) Date: Tue, 22 Aug 2006 16:24:43 -0500 Subject: [elephant-devel] Small glitches in the release tarball In-Reply-To: <20060822195235.GE13707@bateleur.arcanes.fr.eu.org> References: <20060822195235.GE13707@bateleur.arcanes.fr.eu.org> Message-ID: <1156281884.4669.11.camel@localhost.localdomain> Dear Pierre/Nowhere man, I have place a new elephant-0.6.0.tgz that fixes the two problems that noticed. I have not tested it; I just checked out the 0.6.0 release, changed the directory name, and removed all the CVS directories, and retarred. We tested that release extensively in May when it was first released. If I haven't fouled up the re-tarring, it should be good for you. Thanks! Let me know if there is any problem. On Tue, 2006-08-22 at 21:52 +0200, Pierre THIERRY wrote: > Hi, > > I'm just about to try to package some Lisp systems I use for Debian, and > I noted some minor problems with the Elephant's tarballs: > > - it contains an 'elephant' dir, instead of 'elephant-0.6.0' (if you > extract in a dir also containing the CVS dir, it will be a problem) > - it contains the CVS dirs > > Quickly, > Nowhere man > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From Petter.Egesund at kunnskapsforlaget.no Thu Aug 24 10:32:19 2006 From: Petter.Egesund at kunnskapsforlaget.no (Petter Egesund) Date: Thu, 24 Aug 2006 12:32:19 +0200 Subject: SV: SV: SV: SV: SV: SV: [elephant-devel] Memory error with Elephant Message-ID: <4FEEF5F5BB5D344BAAB5004BB495C4F10407BA@EPOST3.bokklubbene.no> Some thoughts; I am getting a little closer, passing all 110 test except 6. Storing of stings seems to work now. What does not work is storing of integers, and thereby rationals, bignums, and so on. This will not work without some larger rewriting. As far as I can see there are two options: 1. Force serializing of integers to 32-bit, not dependent on the underlying OS. This seems to be the easy way out. What must be rewritten is: - storing of fixnums (which can be larger then 32-bit on 64-bit os). Fixnums must be stored the same way as integers are stored today. - storing of bignums, as reading directly from the bignum using %bignum-ref can return 32-bit or 64-bit on different platsforms. A rewrite of storing bignums can be done on two ways, either by testing on platform, or by doing the work ourself, by doing some bitwise operations (something like shifting bytes doing some bitwise and, and then repeat until 0) I would wote for the last option, as this makes the codebase less platform-dependent (might be a small speed-penalty though (I don't know)). 2. We can start storing integers as 64-bits on 64-bits platform. Probably means a lot of rewring? This will probably give the fastest benchmark-results? I would personally go for suggestion 1, I do not think the differnce in speed will visible at all? Any opinions? Cheers, Petter Egesund From read at robertlread.net Thu Aug 24 18:28:38 2006 From: read at robertlread.net (Robert L. Read) Date: Thu, 24 Aug 2006 13:28:38 -0500 Subject: [elephant-devel] 64 bit issues In-Reply-To: <4FEEF5F5BB5D344BAAB5004BB495C4F10407BA@EPOST3.bokklubbene.no> References: <4FEEF5F5BB5D344BAAB5004BB495C4F10407BA@EPOST3.bokklubbene.no> Message-ID: <1156444118.4669.46.camel@localhost.localdomain> On Thu, 2006-08-24 at 12:32 +0200, Petter Egesund wrote: > Some thoughts; > > I am getting a little closer, passing all 110 test except 6. Storing of > stings seems to work now. What does not work is storing of integers, and > thereby rationals, bignums, and so on. Thank you! > > This will not work without some larger rewriting. As far as I can see > there are two options: > > 1. Force serializing of integers to 32-bit, not dependent on the > underlying OS. This seems to be the easy way out. What must be rewritten > is: > > - storing of fixnums (which can be larger then 32-bit on 64-bit os). > Fixnums must be stored the same way as integers are stored today. > - storing of bignums, as reading directly from the bignum using > %bignum-ref can return 32-bit or 64-bit on different platsforms. A > rewrite of storing bignums can be done on two ways, either by testing on > platform, or by doing the work ourself, by doing some bitwise operations > (something like shifting bytes doing some bitwise and, and then repeat > until 0) I would wote for the last option, as this makes the codebase > less platform-dependent (might be a small speed-penalty though (I don't > know)). > > 2. We can start storing integers as 64-bits on 64-bits platform. > Probably means a lot of rewring? This will probably give the fastest > benchmark-results? > > I would personally go for suggestion 1, I do not think the differnce in > speed will visible at all? Any opinions? IMHO, your solution #1 is an OK patch as a stopgap. However, we definitely eventually want to take full advantage of a 64 bit architecture --- this will be the wave of the future, and there is no serious reason why we can't rewrite it. Having said that, I'm in no position to do it for a while, so unless you or someone else wants to rewrite it, I propose the following: Let's produce a patch that works as much as possible on the 64 bit architecture, and document the defficiency that on that architecture, for example, fixnums don't work. As long as this continues to pass all tests on 32-bit architectures, we can release it as 0.6.1, for example. When Ian and I work on 0.7.0, if you or someone else will help us test on a 64-bit architecture, then we will try to make sure that we fully support things in that release. I'll write up the documentation based on what we decide. > > Cheers, > > Petter Egesund -------------- next part -------------- An HTML attachment was scrubbed... URL: From kykim_lists at mac.com Fri Aug 25 03:19:58 2006 From: kykim_lists at mac.com (kykim_lists at mac.com) Date: Thu, 24 Aug 2006 23:19:58 -0400 Subject: [elephant-devel] No more controller-root? Message-ID: <5077545.1156475999007.JavaMail.kykim_lists@mac.com> I want to get all the top-level keys in my data store. I use to do this: (let ((results '())) (ele:with-open-store (+data-store+) (ele:map-btree #'(lambda (k v) (push k results)) (ele:controller-root ele:*store-controller*))) (nreverse results))) But now, controller-root is no longer external. It looks like map-root would do what I want, but it's not external either (?) Any suggestions on what I can do? Thanks, -kevin From read at robertlread.net Fri Aug 25 03:37:35 2006 From: read at robertlread.net (Robert L. Read) Date: Thu, 24 Aug 2006 22:37:35 -0500 Subject: [elephant-devel] No more controller-root? In-Reply-To: <5077545.1156475999007.JavaMail.kykim_lists@mac.com> References: <5077545.1156475999007.JavaMail.kykim_lists@mac.com> Message-ID: <1156477056.4669.63.camel@localhost.localdomain> Ian can answer this better than I can, but I would say just use map-root anyway. It might be an oversight that it is not external. Perhaps Ian has a better idea. On Thu, 2006-08-24 at 23:19 -0400, kykim_lists at mac.com wrote: > I want to get all the top-level keys in my data store. > > I use to do this: > (let ((results '())) > (ele:with-open-store (+data-store+) > (ele:map-btree #'(lambda (k v) (push k results)) (ele:controller-root ele:*store-controller*))) > (nreverse results))) > > But now, controller-root is no longer external. > It looks like map-root would do what I want, but it's not external either (?) > > Any suggestions on what I can do? > > Thanks, > -kevin > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Fri Aug 25 19:35:38 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 25 Aug 2006 15:35:38 -0400 Subject: [elephant-devel] No more controller-root? In-Reply-To: <1156477056.4669.63.camel@localhost.localdomain> References: <5077545.1156475999007.JavaMail.kykim_lists@mac.com> <1156477056.4669.63.camel@localhost.localdomain> Message-ID: <44EF510A.4060709@csail.mit.edu> Go ahead and export map-root. I think controller-root was factored out and is now private to the backend and not the elephant package proper. Implementing the root index requires implementing the add/get/remove operations but not controller-root. map-root should be an interface standard, however. We can enforce a root as index policy as well, I forget the reasons I chose this partitioning. Ian Robert L. Read wrote: > Ian can answer this better than I can, but I would say just use > map-root anyway. > It might be an oversight that it is not external. > > Perhaps Ian has a better idea. > > > On Thu, 2006-08-24 at 23:19 -0400, kykim_lists at mac.com wrote: >> I want to get all the top-level keys in my data store. >> >> I use to do this: >> (let ((results '())) >> (ele:with-open-store (+data-store+) >> (ele:map-btree #'(lambda (k v) (push k results)) (ele:controller-root ele:*store-controller*))) >> (nreverse results))) >> >> But now, controller-root is no longer external. >> It looks like map-root would do what I want, but it's not external either (?) >> >> Any suggestions on what I can do? >> >> Thanks, >> -kevin >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Fri Aug 25 19:58:28 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 25 Aug 2006 15:58:28 -0400 Subject: [elephant-devel] 64 bit issues In-Reply-To: <1156444118.4669.46.camel@localhost.localdomain> References: <4FEEF5F5BB5D344BAAB5004BB495C4F10407BA@EPOST3.bokklubbene.no> <1156444118.4669.46.camel@localhost.localdomain> Message-ID: <44EF5664.6030700@csail.mit.edu> I've attached at the end of the e-mail the approach taken by Rucksack. The theory is to, entirely within lisp, reduce all objects to a byte sequence and only call out to C to do byte-writes into memory vectors. It might be a little slower, but it solves all these problems. However, including it is not backward compatible but philosophically strikes me as the right way to address all these issues. Implementing option #1 means testing in lisp whether the cl:most-positive-fixnum constant is > 2^32 and if so use the bignum tag to implement a bignum style store. This maintains backwards compatibility but upgrades 64-bit fixnums to bignums during serialization. We can address a better serialization philosophy in 0.7.0. In fact, I want a totally new serializer in 0.7.0 to clean up these annoying issues and improve performance on non-persistent objects. On the 64-bit lisp is buffer-write-uint writing 64-bits to the byte stream or 32? To address the %bignum-ref problem, you could go ahead and use the consing ldb operation used for allegro and other lisps. That's easy. The other way to go is to detect when you get 64-bit values back from %bignum-ref and then write two uints in a row so 32-bit lisps can read out the same stream as 64-bit lisps. When deserializing just know that all uints pulled from the stream are 32-bits, if the buffer-write-uint function is in fact serializing 4 bytes and not 8. Cheers, Ian >From serialize.lisp in Rucksack ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Integers ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; (defun serialize-byte-16 (integer stream) (serialize-byte (ldb (byte 8 0) integer) stream) (serialize-byte (ldb (byte 8 8) integer) stream)) (defun serialize-byte-24 (integer stream) (serialize-byte (ldb (byte 8 0) integer) stream) (serialize-byte (ldb (byte 8 8) integer) stream) (serialize-byte (ldb (byte 8 16) integer) stream)) (defun serialize-byte-32 (integer stream) (serialize-byte (ldb (byte 8 0) integer) stream) (serialize-byte (ldb (byte 8 8) integer) stream) (serialize-byte (ldb (byte 8 16) integer) stream) (serialize-byte (ldb (byte 8 24) integer) stream)) (defun serialize-byte-48 (integer stream) (multiple-value-bind (most-significant least-significant) (floor integer #x1000000) (serialize-byte-24 least-significant stream) (serialize-byte-24 most-significant stream))) (defun serialize-byte-64 (integer stream) (multiple-value-bind (most-significant least-significant) (floor integer #x100000000) (serialize-byte-32 least-significant stream) (serialize-byte-32 most-significant stream))) (defun deserialize-byte-16 (stream) (+ (deserialize-byte stream) (* (deserialize-byte stream) 256))) (defun deserialize-byte-24 (stream) (+ (deserialize-byte stream) (* (deserialize-byte stream) #x100) (* (deserialize-byte stream) #x10000))) (defun deserialize-byte-32 (stream) (+ (deserialize-byte stream) (* (deserialize-byte stream) #x100) (* (deserialize-byte stream) #x10000) (* (deserialize-byte stream) #x1000000))) (defun deserialize-byte-48 (stream) (+ (deserialize-byte-24 stream) (* (deserialize-byte-24 stream) #x1000000))) (defun deserialize-byte-64 (stream) (+ (deserialize-byte-32 stream) (* (deserialize-byte-32 stream) #x100000000))) (defmethod serialize ((obj integer) stream) ;; Serialize integers with least-significant bytes first. (cond ((zerop obj) (serialize-marker +zero+ stream)) ((= obj 1) (serialize-marker +one+ stream)) ((= obj -1) (serialize-marker +minus-one+ stream)) ((= obj 2) (serialize-marker +two+ stream)) (t (let* ((positive-p (>= obj 0) ) (unsigned (abs obj)) (nr-octets (nr-octets unsigned))) (serialize-integer positive-p unsigned nr-octets stream))))) (defun serialize-integer (positive-p unsigned nr-octets stream) (case nr-octets (1 (serialize-marker (if positive-p +positive-byte-8+ +negative-byte-8+) stream) (serialize-byte unsigned stream)) (2 (serialize-marker (if positive-p +positive-byte-16+ +negative-byte-16+) stream) (serialize-byte-16 unsigned stream)) (3 (serialize-marker (if positive-p +positive-byte-24+ +negative-byte-24+) stream) (serialize-byte-24 unsigned stream)) (4 (serialize-marker (if positive-p +positive-byte-32+ +negative-byte-32+) stream) (serialize-byte-32 unsigned stream)) ((5 6) (serialize-marker (if positive-p +positive-byte-48+ +negative-byte-48+) stream) (serialize-byte-48 unsigned stream)) ((7 8) (serialize-marker (if positive-p +positive-byte-64+ +negative-byte-64+) stream) (serialize-byte-64 unsigned stream)) (otherwise (let ((nr-bits (* 8 nr-octets))) (serialize-marker (if positive-p +positive-integer+ +negative-integer+) stream) (serialize nr-octets stream) (loop for position from (- nr-bits 8) downto 0 by 8 do (serialize-byte (ldb (byte 8 position) unsigned) stream)))))) (defun nr-octets (n) (ceiling (integer-length n) 8)) (defmethod deserialize-contents ((marker (eql +minus-one+)) stream) (declare (ignore stream)) -1) (defmethod deserialize-contents ((marker (eql +zero+)) stream) (declare (ignore stream)) 0) (defmethod deserialize-contents ((marker (eql +one+)) stream) (declare (ignore stream)) 1) (defmethod deserialize-contents ((marker (eql +two+)) stream) (declare (ignore stream)) 2) (defmethod deserialize-contents ((marker (eql +positive-byte-8+)) stream) (deserialize-byte stream)) (defmethod deserialize-contents ((marker (eql +negative-byte-8+)) stream) (- (deserialize-byte stream))) (defmethod deserialize-contents ((marker (eql +positive-byte-16+)) stream) (deserialize-byte-16 stream)) (defmethod deserialize-contents ((marker (eql +negative-byte-16+)) stream) (- (deserialize-byte-16 stream))) (defmethod deserialize-contents ((marker (eql +positive-byte-24+)) stream) (deserialize-byte-24 stream)) (defmethod deserialize-contents ((marker (eql +negative-byte-24+)) stream) (- (deserialize-byte-24 stream))) (defmethod deserialize-contents ((marker (eql +positive-byte-32+)) stream) (deserialize-byte-32 stream)) (defmethod deserialize-contents ((marker (eql +negative-byte-32+)) stream) (- (deserialize-byte-32 stream))) (defmethod deserialize-contents ((marker (eql +positive-byte-48+)) stream) (deserialize-byte-48 stream)) (defmethod deserialize-contents ((marker (eql +negative-byte-48+)) stream) (- (deserialize-byte-48 stream))) (defmethod deserialize-contents ((marker (eql +positive-byte-64+)) stream) (deserialize-byte-64 stream)) (defmethod deserialize-contents ((marker (eql +negative-byte-64+)) stream) (- (deserialize-byte-64 stream))) (defmethod deserialize-contents ((marker (eql +positive-integer+)) stream) (let ((nr-bytes (deserialize stream))) (assert (integerp nr-bytes)) (let ((result 0)) (loop for i below nr-bytes do (setf result (+ (ash result 8) (deserialize-byte stream)))) result))) (defmethod deserialize-contents ((marker (eql +negative-integer+)) stream) (- (deserialize-contents +positive-integer+ stream))) Robert L. Read wrote: > On Thu, 2006-08-24 at 12:32 +0200, Petter Egesund wrote: >> Some thoughts; >> >> I am getting a little closer, passing all 110 test except 6. Storing of >> stings seems to work now. What does not work is storing of integers, and >> thereby rationals, bignums, and so on. >> > Thank you! >> This will not work without some larger rewriting. As far as I can see >> there are two options: >> >> 1. Force serializing of integers to 32-bit, not dependent on the >> underlying OS. This seems to be the easy way out. What must be rewritten >> is: >> >> - storing of fixnums (which can be larger then 32-bit on 64-bit os). >> Fixnums must be stored the same way as integers are stored today. >> - storing of bignums, as reading directly from the bignum using >> %bignum-ref can return 32-bit or 64-bit on different platsforms. A >> rewrite of storing bignums can be done on two ways, either by testing on >> platform, or by doing the work ourself, by doing some bitwise operations >> (something like shifting bytes doing some bitwise and, and then repeat >> until 0) I would wote for the last option, as this makes the codebase >> less platform-dependent (might be a small speed-penalty though (I don't >> know)). >> >> 2. We can start storing integers as 64-bits on 64-bits platform. >> Probably means a lot of rewring? This will probably give the fastest >> benchmark-results? >> >> I would personally go for suggestion 1, I do not think the differnce in >> speed will visible at all? Any opinions? >> > IMHO, your solution #1 is an OK patch as a stopgap. However, we > definitely eventually want > to take full advantage of a 64 bit architecture --- this will be the > wave of the future, and there is > no serious reason why we can't rewrite it. Having said that, I'm in > no position to do it for a while, > so unless you or someone else wants to rewrite it, I propose the > following: > > Let's produce a patch that works as much as possible on the 64 bit > architecture, and document > the defficiency that on that architecture, for example, fixnums don't > work. As long as this > continues to pass all tests on 32-bit architectures, we can release it > as 0.6.1, for example. > > When Ian and I work on 0.7.0, if you or someone else will help us test > on a 64-bit architecture, then > we will try to make sure that we fully support things in that release. > > I'll write up the documentation based on what we decide. > >> Cheers, >> >> Petter Egesund >>