From jcorneli at planetmath.org Mon Sep 3 04:51:46 2007 From: jcorneli at planetmath.org (Joe Corneli) Date: Mon, 3 Sep 2007 00:51:46 -0400 (EDT) Subject: [elephant-devel] a job for the query system? In-Reply-To: <8D1E5530-E451-42EB-A216-6E835C16D347@csail.mit.edu> (message from Ian Eslick on Thu, 30 Aug 2007 21:45:34 -0400) References: <20070814221430.AB4F0826D@planetmath.cc.vt.edu> <8D1E5530-E451-42EB-A216-6E835C16D347@csail.mit.edu> Message-ID: <20070903045146.22A498223@planetmath.cc.vt.edu> OK, in response to your message: I certainly must abandon some assumptions about OIDs! I do need persistent unique handles for Things so I can consistently refer to them in my Triples and Theories. Keeping things unique by the data itself is an interesting minimalistic proposal... apparently even if OIDs are changing, compound objects like Triples are stored will still be stored "safely"? E.g. if I store the triple [ThingA, ThingB, ThingC] this will always be (de)serialized so that it points at the right Things (even if the OID of ThingC, say, happens to change)? If this works, that would be neat. (If not, I could easily enough maintain some persistent index and use it to create handles for Things.) Your statement of what I hope to accomplish in filtering-by-theory is correct. Having Theories just be subsets of Things would be fine for a prototype. Eventually it could be nice to be able to label the elements of a subset and store such labelings; Pseudo-SQL-wise, presumably all that amounts to is another column that is ignored before the "member" function is applied a la your example. From nowhere.man at levallois.eu.org Mon Sep 3 21:28:34 2007 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Mon, 3 Sep 2007 23:28:34 +0200 Subject: [elephant-devel] a job for the query system? In-Reply-To: <8D1E5530-E451-42EB-A216-6E835C16D347@csail.mit.edu> References: <20070814221430.AB4F0826D@planetmath.cc.vt.edu> <8D1E5530-E451-42EB-A216-6E835C16D347@csail.mit.edu> Message-ID: <20070903212834.GD5163@(none)> Scribit Ian Eslick dies 30/08/2007 hora 21:45: > There is some early prototype functionality that is partially in my > local development tree and partially in the current development tree. > Unfortunately I did not finish the query system this summer and so its > release has been indefinitely delayed. Is this local tree available online? Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Tue Sep 4 03:38:11 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 3 Sep 2007 23:38:11 -0400 Subject: [elephant-devel] a job for the query system? In-Reply-To: <20070903212834.GD5163@(none)> References: <20070814221430.AB4F0826D@planetmath.cc.vt.edu> <8D1E5530-E451-42EB-A216-6E835C16D347@csail.mit.edu> <20070903212834.GD5163@(none)> Message-ID: <3EA95101-75C8-4601-A850-EC4E48980FCC@csail.mit.edu> I do plan to integrate my sketch of the system in the coming weeks. However I haven't looked at the local tree recently so I'm not sure what shape it was in so would hesitate to make it available without that review. Ian On Sep 3, 2007, at 5:28 PM, Pierre THIERRY wrote: > Scribit Ian Eslick dies 30/08/2007 hora 21:45: >> There is some early prototype functionality that is partially in my >> local development tree and partially in the current development tree. >> Unfortunately I did not finish the query system this summer and so >> its >> release has been indefinitely delayed. > > Is this local tree available online? > > Curiously, > Pierre > -- > nowhere.man at levallois.eu.org > OpenPGP 0xD9D50D8A > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 186 bytes Desc: This is a digitally signed message part URL: From chris at terraminds.com Tue Sep 4 08:15:22 2007 From: chris at terraminds.com (Chris Laux) Date: Tue, 04 Sep 2007 10:15:22 +0200 Subject: [elephant-devel] String class index broken? Message-ID: <46DD141A.4090700@terraminds.com> Hi all, as I couldn't find information about the strange behaviour shown below elsewhere, I just wanted to report it here. I'm indexing class "user" over slot "nick" that contains strings. When I query the index using all lowercase letters, I get an ambiguous (and wrong!?) response, but when I stick to the original casing, it works as expected (and documented?). This isn't actually going to stop me from being able to use Elephant, but I thought it worth reporting. Regards, Chris Laux > (get-instances-by-value 'user 'nick "openhaus") (# #) > (mapcar #'nick *) ("OpenHaus" "OpenLate") > (get-instances-by-value 'user 'nick "OpenHaus") (#) > (mapcar #'nick *) ("OpenHaus") From henrik at evahjelte.com Tue Sep 4 08:33:35 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Tue, 4 Sep 2007 10:33:35 +0200 Subject: [elephant-devel] String class index broken? In-Reply-To: <46DD141A.4090700@terraminds.com> References: <46DD141A.4090700@terraminds.com> Message-ID: <50e8e4f60709040133u799ec0del7b6d922387566b4f@mail.gmail.com> To make debugging easier, which backend are you using? /Henrik Hjelte From chris at terraminds.com Tue Sep 4 08:39:19 2007 From: chris at terraminds.com (Chris Laux) Date: Tue, 04 Sep 2007 10:39:19 +0200 Subject: [elephant-devel] String class index broken? In-Reply-To: <50e8e4f60709040133u799ec0del7b6d922387566b4f@mail.gmail.com> References: <46DD141A.4090700@terraminds.com> <50e8e4f60709040133u799ec0del7b6d922387566b4f@mail.gmail.com> Message-ID: <46DD19B7.3050001@terraminds.com> Henrik Hjelte wrote: > To make debugging easier, which backend are you using? > /Henrik Hjelte Sorry, forgot about that kind of info: BerkeleyDB 4.5.20 with Elephant 0.9.0 with SBCL 1.0.5 on Linux/x86 32bit. Chris From read at robertlread.net Tue Sep 4 18:12:09 2007 From: read at robertlread.net (Robert L. Read) Date: Tue, 04 Sep 2007 13:12:09 -0500 Subject: [elephant-devel] String class index broken? In-Reply-To: <46DD141A.4090700@terraminds.com> References: <46DD141A.4090700@terraminds.com> Message-ID: <1188929529.9502.221.camel@penguin.yourdomain.com> Can you provide the code that generates the data for this class? If this is is still a problem for you, I think I will need a complete reproduction to look into it. On Tue, 2007-09-04 at 10:15 +0200, Chris Laux wrote: > Hi all, > > as I couldn't find information about the strange behaviour shown below > elsewhere, I just wanted to report it here. I'm indexing class "user" > over slot "nick" that contains strings. When I query the index using all > lowercase letters, I get an ambiguous (and wrong!?) response, but when I > stick to the original casing, it works as expected (and documented?). > > This isn't actually going to stop me from being able to use Elephant, > but I thought it worth reporting. > > Regards, > > Chris Laux > > > > (get-instances-by-value 'user 'nick "openhaus") > (# #) > > > (mapcar #'nick *) > ("OpenHaus" "OpenLate") > > > (get-instances-by-value 'user 'nick "OpenHaus") > (#) > > > (mapcar #'nick *) > ("OpenHaus") > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Tue Sep 4 18:50:28 2007 From: lists at infoway.net (lists at infoway.net) Date: Tue, 4 Sep 2007 14:50:28 -0400 Subject: [elephant-devel] Backend tests problem / OS X 10.4.10 / SBCL 1.0.6 Message-ID: <048D293C-3E0A-4118-A899-E4615BB85FFB@infoway.net> I just downloaded a fresh copy of elephant and ran the backend tests. 1 of 132 tests failed (1 out of 132 total tests failed: INDEXING- RANGE.) Any ideas? Below is the output: ELE-TESTS> (do-backend-tests) Attempting to load libmemutil.dylib... Loaded /Users/dev/lisp/elephant/src/memutil/libmemutil.dylib STYLE-WARNING: redefining COPY-BUFS in DEFUN STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining READ-INT32 in DEFUN STYLE-WARNING: redefining READ-INT64 in DEFUN STYLE-WARNING: redefining READ-UINT32 in DEFUN STYLE-WARNING: redefining READ-UINT64 in DEFUN STYLE-WARNING: redefining READ-FLOAT in DEFUN STYLE-WARNING: redefining READ-DOUBLE in DEFUN STYLE-WARNING: redefining WRITE-INT32 in DEFUN STYLE-WARNING: redefining WRITE-INT64 in DEFUN STYLE-WARNING: redefining WRITE-UINT32 in DEFUN STYLE-WARNING: redefining WRITE-UINT64 in DEFUN STYLE-WARNING: redefining WRITE-FLOAT in DEFUN STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN ; compiling file "/Users/dev/lisp/elephant/tests/ testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): ; compiling (IN-PACKAGE :ELE-TESTS) ; compiling (DEFVAR ENV) ; compiling (DEFVAR DB) ; compiling (DEFUN PREPARE-BDB ...) ; compiling (DEFTEST PREPARES-BDB ...) ; compiling (DEFUN TEST-SEQUENCE1 ...) ; compiling (DEFTEST TEST-SEQ1 ...) ; compiling (DEFUN TEST-SEQUENCE2 ...) ; compiling (DEFTEST TEST-SEQ2 ...) ; compiling (DEFUN CLEANUP-BDB ...) ; compiling (DEFTEST CLEANSUP-BDB ...) ; /Users/dev/lisp/elephant/tests/testbdb.fasl written ; compilation finished in 0:00:01 Doing 132 pending tests of 132 tests total. FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS COMPLEXES BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- CONSTRUCT CIRCULAR PERSISTENT; in: LAMBDA NIL ; NULL ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM REDEFCLASS MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES-SLOT2 MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET-RANGE SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM-INDEX3 DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST CUR-DEL1 INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC INDEXING-CLASS-OPT INDEXING-INHERIT Test INDEXING-RANGE failed Form: (PROGN (DEFCLASS IDX-FOUR NIL ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 :INDEX T)) (:METACLASS PERSISTENT-METACLASS)) (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) (DEFCLASS IDX-FOUR NIL ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 :INDEX T)) (:METACLASS PERSISTENT-METACLASS)) (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 VAL)) (WITH-TRANSACTION NIL (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 5 6 10))) (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) Expected values: T T T Actual value: #. INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS Ranged get of 10/700 objects = Linear: 0.364 sec Indexed: 0.007 sec INDEXING-TIMING ; in: LAMBDA NIL ; NULL ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring REMOVE-ELEMENT ; in: LAMBDA NIL ; NULL ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-BASIC ; in: LAMBDA NIL ; NULL ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-BTREE ; in: LAMBDA NIL ; NULL ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-IDX-BTREE ; in: LAMBDA NIL ; NULL ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-PCLASS ; in: LAMBDA NIL ; NULL ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-MULT-PCLASS ; in: LAMBDA NIL ; NULL ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 TEST-SEQ2 CLEANSUP-BDB 1 out of 132 total tests failed: INDEXING-RANGE. NIL From eslick at csail.mit.edu Tue Sep 4 19:16:55 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 4 Sep 2007 15:16:55 -0400 Subject: [elephant-devel] Backend tests problem / OS X 10.4.10 / SBCL 1.0.6 In-Reply-To: <048D293C-3E0A-4118-A899-E4615BB85FFB@infoway.net> References: <048D293C-3E0A-4118-A899-E4615BB85FFB@infoway.net> Message-ID: <585CDC03-B0AC-4C94-92AE-06B589DF29ED@csail.mit.edu> Is this 0.9.0 or the development branch? -Ian On Sep 4, 2007, at 2:50 PM, lists at infoway.net wrote: > I just downloaded a fresh copy of elephant and ran the backend > tests. 1 of 132 tests failed (1 out of 132 total tests failed: > INDEXING-RANGE.) Any ideas? Below is the output: > > ELE-TESTS> (do-backend-tests) > Attempting to load libmemutil.dylib... > Loaded /Users/dev/lisp/elephant/src/memutil/libmemutil.dylib > STYLE-WARNING: redefining COPY-BUFS in DEFUN > STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining READ-INT32 in DEFUN > STYLE-WARNING: redefining READ-INT64 in DEFUN > STYLE-WARNING: redefining READ-UINT32 in DEFUN > STYLE-WARNING: redefining READ-UINT64 in DEFUN > STYLE-WARNING: redefining READ-FLOAT in DEFUN > STYLE-WARNING: redefining READ-DOUBLE in DEFUN > STYLE-WARNING: redefining WRITE-INT32 in DEFUN > STYLE-WARNING: redefining WRITE-INT64 in DEFUN > STYLE-WARNING: redefining WRITE-UINT32 in DEFUN > STYLE-WARNING: redefining WRITE-UINT64 in DEFUN > STYLE-WARNING: redefining WRITE-FLOAT in DEFUN > STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN > STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN > STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN > STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN > STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN > STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN > STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN > STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN > STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN > STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN > STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN > STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN > STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN > STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN > STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN > STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN > ; compiling file "/Users/dev/lisp/elephant/tests/ > testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): > ; compiling (IN-PACKAGE :ELE-TESTS) > ; compiling (DEFVAR ENV) > ; compiling (DEFVAR DB) > ; compiling (DEFUN PREPARE-BDB ...) > ; compiling (DEFTEST PREPARES-BDB ...) > ; compiling (DEFUN TEST-SEQUENCE1 ...) > ; compiling (DEFTEST TEST-SEQ1 ...) > ; compiling (DEFUN TEST-SEQUENCE2 ...) > ; compiling (DEFTEST TEST-SEQ2 ...) > ; compiling (DEFUN CLEANUP-BDB ...) > ; compiling (DEFTEST CLEANSUP-BDB ...) > > ; /Users/dev/lisp/elephant/tests/testbdb.fasl written > ; compilation finished in 0:00:01 > Doing 132 pending tests of 132 tests total. > FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM > WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS > COMPLEXES > BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES > HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP > TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- > CONSTRUCT > CIRCULAR PERSISTENT; in: LAMBDA NIL > ; NULL > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION > UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 > NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS > BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS > INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM > REDEFCLASS > MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE > TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE > INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET > SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 > REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 > NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES- > SLOT2 > MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET- > RANGE > SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV > MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM-INDEX3 > DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST > CUR-DEL1 > INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 > GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 > ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET > DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC > INDEXING-CLASS-OPT INDEXING-INHERIT > Test INDEXING-RANGE failed > Form: (PROGN > (DEFCLASS IDX-FOUR NIL > ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 > :INDEX T)) > (:METACLASS PERSISTENT-METACLASS)) > (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) > (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) > (DEFCLASS IDX-FOUR NIL > ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 > :INDEX T)) > (:METACLASS PERSISTENT-METACLASS)) > (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 > VAL)) > (WITH-TRANSACTION NIL > (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 5 > 6 10))) > (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) > (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) > (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) > (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) > (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) > (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) > Expected values: T > T > T > Actual value: #. > INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB > INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS > Ranged get of 10/700 objects = Linear: 0.364 sec Indexed: 0.007 sec > INDEXING-TIMING > ; in: LAMBDA NIL > ; NULL > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring REMOVE-ELEMENT > ; in: LAMBDA NIL > ; NULL > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-BASIC > ; in: LAMBDA NIL > ; NULL > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-BTREE > ; in: LAMBDA NIL > ; NULL > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-IDX-BTREE > ; in: LAMBDA NIL > ; NULL > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-PCLASS > ; in: LAMBDA NIL > ; NULL > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-MULT-PCLASS > ; in: LAMBDA NIL > ; NULL > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 > TEST-SEQ2 CLEANSUP-BDB > 1 out of 132 total tests failed: INDEXING-RANGE. > NIL > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Tue Sep 4 19:21:27 2007 From: lists at infoway.net (lists at infoway.net) Date: Tue, 4 Sep 2007 15:21:27 -0400 Subject: [elephant-devel] Backend tests problem / OS X 10.4.10 / SBCL 1.0.6 In-Reply-To: <585CDC03-B0AC-4C94-92AE-06B589DF29ED@csail.mit.edu> References: <048D293C-3E0A-4118-A899-E4615BB85FFB@infoway.net> <585CDC03-B0AC-4C94-92AE-06B589DF29ED@csail.mit.edu> Message-ID: <6E7B62B5-D7B6-4AD3-96A4-35C0307586F9@infoway.net> Not quite sure. I'm using the info I was recommended from the list a while back: cvs -z3 -d :pserver:anonymous:anonymous at common-lisp.net:/project/ elephant/cvsroot co elephant Should I be using a different version? Thanks - Daniel On Sep 4, 2007, at 3:16 PM, Ian Eslick wrote: > Is this 0.9.0 or the development branch? -Ian > > On Sep 4, 2007, at 2:50 PM, lists at infoway.net wrote: > >> I just downloaded a fresh copy of elephant and ran the backend >> tests. 1 of 132 tests failed (1 out of 132 total tests failed: >> INDEXING-RANGE.) Any ideas? Below is the output: >> >> ELE-TESTS> (do-backend-tests) >> Attempting to load libmemutil.dylib... >> Loaded /Users/dev/lisp/elephant/src/memutil/libmemutil.dylib >> STYLE-WARNING: redefining COPY-BUFS in DEFUN >> STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining READ-INT32 in DEFUN >> STYLE-WARNING: redefining READ-INT64 in DEFUN >> STYLE-WARNING: redefining READ-UINT32 in DEFUN >> STYLE-WARNING: redefining READ-UINT64 in DEFUN >> STYLE-WARNING: redefining READ-FLOAT in DEFUN >> STYLE-WARNING: redefining READ-DOUBLE in DEFUN >> STYLE-WARNING: redefining WRITE-INT32 in DEFUN >> STYLE-WARNING: redefining WRITE-INT64 in DEFUN >> STYLE-WARNING: redefining WRITE-UINT32 in DEFUN >> STYLE-WARNING: redefining WRITE-UINT64 in DEFUN >> STYLE-WARNING: redefining WRITE-FLOAT in DEFUN >> STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN >> STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN >> STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN >> STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN >> STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN >> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN >> STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN >> STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN >> ; compiling file "/Users/dev/lisp/elephant/tests/ >> testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): >> ; compiling (IN-PACKAGE :ELE-TESTS) >> ; compiling (DEFVAR ENV) >> ; compiling (DEFVAR DB) >> ; compiling (DEFUN PREPARE-BDB ...) >> ; compiling (DEFTEST PREPARES-BDB ...) >> ; compiling (DEFUN TEST-SEQUENCE1 ...) >> ; compiling (DEFTEST TEST-SEQ1 ...) >> ; compiling (DEFUN TEST-SEQUENCE2 ...) >> ; compiling (DEFTEST TEST-SEQ2 ...) >> ; compiling (DEFUN CLEANUP-BDB ...) >> ; compiling (DEFTEST CLEANSUP-BDB ...) >> >> ; /Users/dev/lisp/elephant/tests/testbdb.fasl written >> ; compilation finished in 0:00:01 >> Doing 132 pending tests of 132 tests total. >> FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM >> WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS >> COMPLEXES >> BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES >> HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP >> TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- >> CONSTRUCT >> CIRCULAR PERSISTENT; in: LAMBDA NIL >> ; NULL >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION >> UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 >> NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS >> BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS >> INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM >> REDEFCLASS >> MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE >> TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE >> INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET >> SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 >> REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 >> NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES- >> SLOT2 >> MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET- >> RANGE >> SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV >> MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM-INDEX3 >> DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST >> CUR-DEL1 >> INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 >> GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 >> ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET >> DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC >> INDEXING-CLASS-OPT INDEXING-INHERIT >> Test INDEXING-RANGE failed >> Form: (PROGN >> (DEFCLASS IDX-FOUR NIL >> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 >> :INDEX T)) >> (:METACLASS PERSISTENT-METACLASS)) >> (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) >> (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) >> (DEFCLASS IDX-FOUR NIL >> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 >> :INDEX T)) >> (:METACLASS PERSISTENT-METACLASS)) >> (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 >> VAL)) >> (WITH-TRANSACTION NIL >> (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 5 >> 6 10))) >> (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) >> (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) >> (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) >> (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) >> (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) >> (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) >> Expected values: T >> T >> T >> Actual value: #. >> INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB >> INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS >> Ranged get of 10/700 objects = Linear: 0.364 sec Indexed: 0.007 sec >> INDEXING-TIMING >> ; in: LAMBDA NIL >> ; NULL >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring REMOVE-ELEMENT >> ; in: LAMBDA NIL >> ; NULL >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-BASIC >> ; in: LAMBDA NIL >> ; NULL >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-BTREE >> ; in: LAMBDA NIL >> ; NULL >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-IDX-BTREE >> ; in: LAMBDA NIL >> ; NULL >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-PCLASS >> ; in: LAMBDA NIL >> ; NULL >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-MULT-PCLASS >> ; in: LAMBDA NIL >> ; NULL >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 >> TEST-SEQ2 CLEANSUP-BDB >> 1 out of 132 total tests failed: INDEXING-RANGE. >> NIL >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Tue Sep 4 19:34:54 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 4 Sep 2007 15:34:54 -0400 Subject: [elephant-devel] Backend tests problem / OS X 10.4.10 / SBCL 1.0.6 In-Reply-To: <6E7B62B5-D7B6-4AD3-96A4-35C0307586F9@infoway.net> References: <048D293C-3E0A-4118-A899-E4615BB85FFB@infoway.net> <585CDC03-B0AC-4C94-92AE-06B589DF29ED@csail.mit.edu> <6E7B62B5-D7B6-4AD3-96A4-35C0307586F9@infoway.net> Message-ID: That's the latest patch of 0.9.0 and should be solid. I'll investigate when I get some time if Robert or someone else doesn't get to it first. Ian On Sep 4, 2007, at 3:21 PM, lists at infoway.net wrote: > Not quite sure. I'm using the info I was recommended from the list > a while back: > > cvs -z3 -d :pserver:anonymous:anonymous at common-lisp.net:/project/ > elephant/cvsroot co elephant > > Should I be using a different version? > > Thanks > - Daniel > > On Sep 4, 2007, at 3:16 PM, Ian Eslick wrote: > >> Is this 0.9.0 or the development branch? -Ian >> >> On Sep 4, 2007, at 2:50 PM, lists at infoway.net wrote: >> >>> I just downloaded a fresh copy of elephant and ran the backend >>> tests. 1 of 132 tests failed (1 out of 132 total tests failed: >>> INDEXING-RANGE.) Any ideas? Below is the output: >>> >>> ELE-TESTS> (do-backend-tests) >>> Attempting to load libmemutil.dylib... >>> Loaded /Users/dev/lisp/elephant/src/memutil/libmemutil.dylib >>> STYLE-WARNING: redefining COPY-BUFS in DEFUN >>> STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining READ-INT32 in DEFUN >>> STYLE-WARNING: redefining READ-INT64 in DEFUN >>> STYLE-WARNING: redefining READ-UINT32 in DEFUN >>> STYLE-WARNING: redefining READ-UINT64 in DEFUN >>> STYLE-WARNING: redefining READ-FLOAT in DEFUN >>> STYLE-WARNING: redefining READ-DOUBLE in DEFUN >>> STYLE-WARNING: redefining WRITE-INT32 in DEFUN >>> STYLE-WARNING: redefining WRITE-INT64 in DEFUN >>> STYLE-WARNING: redefining WRITE-UINT32 in DEFUN >>> STYLE-WARNING: redefining WRITE-UINT64 in DEFUN >>> STYLE-WARNING: redefining WRITE-FLOAT in DEFUN >>> STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN >>> STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN >>> STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN >>> STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN >>> STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN >>> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN >>> STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN >>> STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN >>> ; compiling file "/Users/dev/lisp/elephant/tests/ >>> testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): >>> ; compiling (IN-PACKAGE :ELE-TESTS) >>> ; compiling (DEFVAR ENV) >>> ; compiling (DEFVAR DB) >>> ; compiling (DEFUN PREPARE-BDB ...) >>> ; compiling (DEFTEST PREPARES-BDB ...) >>> ; compiling (DEFUN TEST-SEQUENCE1 ...) >>> ; compiling (DEFTEST TEST-SEQ1 ...) >>> ; compiling (DEFUN TEST-SEQUENCE2 ...) >>> ; compiling (DEFTEST TEST-SEQ2 ...) >>> ; compiling (DEFUN CLEANUP-BDB ...) >>> ; compiling (DEFTEST CLEANSUP-BDB ...) >>> >>> ; /Users/dev/lisp/elephant/tests/testbdb.fasl written >>> ; compilation finished in 0:00:01 >>> Doing 132 pending tests of 132 tests total. >>> FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM >>> WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS >>> COMPLEXES >>> BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES >>> HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP >>> TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- >>> CONSTRUCT >>> CIRCULAR PERSISTENT; in: LAMBDA NIL >>> ; NULL >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION >>> UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 >>> NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS >>> BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS >>> INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM >>> REDEFCLASS >>> MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE >>> TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE >>> INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET >>> SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 >>> REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 >>> NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES- >>> SLOT2 >>> MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET- >>> RANGE >>> SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV >>> MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM- >>> INDEX3 >>> DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST >>> CUR-DEL1 >>> INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 >>> GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 >>> ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET >>> DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC >>> INDEXING-CLASS-OPT INDEXING-INHERIT >>> Test INDEXING-RANGE failed >>> Form: (PROGN >>> (DEFCLASS IDX-FOUR NIL >>> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 >>> :INDEX T)) >>> (:METACLASS PERSISTENT-METACLASS)) >>> (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) >>> (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) >>> (DEFCLASS IDX-FOUR NIL >>> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 >>> :INDEX T)) >>> (:METACLASS PERSISTENT-METACLASS)) >>> (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 >>> VAL)) >>> (WITH-TRANSACTION NIL >>> (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 >>> 5 6 10))) >>> (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) >>> (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) >>> (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) >>> (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) >>> (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) >>> (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) >>> Expected values: T >>> T >>> T >>> Actual value: #. >>> INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB >>> INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS >>> Ranged get of 10/700 objects = Linear: 0.364 sec Indexed: 0.007 sec >>> INDEXING-TIMING >>> ; in: LAMBDA NIL >>> ; NULL >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring REMOVE-ELEMENT >>> ; in: LAMBDA NIL >>> ; NULL >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-BASIC >>> ; in: LAMBDA NIL >>> ; NULL >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-BTREE >>> ; in: LAMBDA NIL >>> ; NULL >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-IDX-BTREE >>> ; in: LAMBDA NIL >>> ; NULL >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-PCLASS >>> ; in: LAMBDA NIL >>> ; NULL >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-MULT-PCLASS >>> ; in: LAMBDA NIL >>> ; NULL >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 >>> TEST-SEQ2 CLEANSUP-BDB >>> 1 out of 132 total tests failed: INDEXING-RANGE. >>> NIL >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From chris at terraminds.com Tue Sep 4 21:34:33 2007 From: chris at terraminds.com (Chris Laux) Date: Tue, 04 Sep 2007 23:34:33 +0200 Subject: [elephant-devel] String class index broken? In-Reply-To: <1188929529.9502.221.camel@penguin.yourdomain.com> References: <46DD141A.4090700@terraminds.com> <1188929529.9502.221.camel@penguin.yourdomain.com> Message-ID: <46DDCF69.3020700@terraminds.com> Robert L. Read wrote: > Can you provide the code that generates the data for this class? These few lines will actually reproduce the bug: > (defclass test-class () ((test :initarg :test :accessor test :index t)) (:metaclass persistent-metaclass)) # > (make-instance 'test-class :test "OpenHaus") # > (make-instance 'test-class :test "OpenLast") # > (get-instances-by-value 'test-class 'test "openhaus") (# #) > If this is is still a problem for you, I think I will need a complete > reproduction to look into it. It isn't a problem for me right now, but it seems to be fairly fundamental? Thanks, Chris Laux From nowhere.man at levallois.eu.org Tue Sep 4 23:52:07 2007 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Wed, 5 Sep 2007 01:52:07 +0200 Subject: [elephant-devel] a job for the query system? In-Reply-To: <3EA95101-75C8-4601-A850-EC4E48980FCC@csail.mit.edu> References: <20070814221430.AB4F0826D@planetmath.cc.vt.edu> <8D1E5530-E451-42EB-A216-6E835C16D347@csail.mit.edu> <20070903212834.GD5163@(none)> <3EA95101-75C8-4601-A850-EC4E48980FCC@csail.mit.edu> Message-ID: <20070904235207.GF5163@(none)> Scribit Ian Eslick dies 03/09/2007 hora 23:38: > However I haven't looked at the local tree recently so I'm not sure > what shape it was in so would hesitate to make it available without > that review. You can make it available as a branch separate from the trunk, and making such local development visible is one of the benefits of a DVCS like Darcs. Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Wed Sep 5 02:22:57 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 4 Sep 2007 22:22:57 -0400 Subject: [elephant-devel] String class index broken? In-Reply-To: <46DDCF69.3020700@terraminds.com> References: <46DD141A.4090700@terraminds.com> <1188929529.9502.221.camel@penguin.yourdomain.com> <46DDCF69.3020700@terraminds.com> Message-ID: I found the bug, it's an optimization left in the BDB cursor code that would work with exact key matches but fails on non-exact matches due to the way I implemented map-index. Here's the patch to src/db-bdb/bdb-collections.lisp 593c593,596 < (values t key (deserialize val sc) --- > (values t > (progn (buffer-read-oid k) > (deserialize k sc)) > (deserialize val sc) I'll promote it to the 0.9p1 tree (CVS HEAD) after I investigate other functions for these screw cases and add some test cases as well. Ian On Sep 4, 2007, at 5:34 PM, Chris Laux wrote: >> (defclass test-class () > ((test :initarg :test :accessor test :index t)) > (:metaclass persistent-metaclass)) From eslick at csail.mit.edu Wed Sep 5 16:11:42 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 5 Sep 2007 12:11:42 -0400 Subject: [elephant-devel] Patch to 0.9.0 Message-ID: There was a disconnect between the case-insensitive string comparison inside BDB and that assumed in the lisp code (i.e. two lisp strings were not equal when they were equal in BDB causing the map function to misbehave). I believe I've fixed this. Are there any assumptions about case sensitivity in the Postmodern backend? I know that the SQL backend relies entirely on the lisp comparison function. I'll add some tests for this case and commit the patch to the 0.9.0 CVS tree as well as my local 0.9.1 darcs branch for near-term integration into the main darcs repository. Cheers, Ian PS - I'm not making my local 0.9.1 branch available not because I can't but because it is not in a coherent state. From chris at terraminds.com Sat Sep 8 15:55:47 2007 From: chris at terraminds.com (Chris Laux) Date: Sat, 08 Sep 2007 17:55:47 +0200 Subject: [elephant-devel] map-btree from-end arg not working? Message-ID: <46E2C603.2030003@terraminds.com> Hi again, I seem to have stumbled across another bug: the from-end argument doesn't seem to be working, map-btree always calls with the same (presumably last) entry when it is set. (defun test-btree-from-end (btree from-end) (let (acc (n 0)) (map-btree #'(lambda (k v) (push (list k v) acc) (when (= (incf n) 4) (return-from test-btree-from-end acc))) btree :from-end from-end))) > (test-btree-from-end ** nil) ((3398012969 #) (3398012898 #) (3398012815 #) (3398012772 #)) > (test-btree-from-end *** t) ((3398231130 #) (3398231130 #) (3398231130 #) (3398231130 #)) My setup is BerkeleyDB 4.5.20 with Elephant 0.9.0 with SBCL 1.0.4 on Linux/x86-64. Thanks, Chris Laux From chris at terraminds.com Sat Sep 8 19:28:46 2007 From: chris at terraminds.com (Chris Laux) Date: Sat, 08 Sep 2007 21:28:46 +0200 Subject: [elephant-devel] map-btree from-end arg not working? In-Reply-To: <46E2C603.2030003@terraminds.com> References: <46E2C603.2030003@terraminds.com> Message-ID: <46E2F7EE.3030306@terraminds.com> So I had a look at it myself and it turned out to be much easier than I had expected, which is what kept me from trying in the first place. The following method in src/db-bdb/bdb-collections.lisp is the culprit: (defmethod cursor-prev ((cursor bdb-cursor)) (if (cursor-initialized-p cursor) (let ((sc (get-con (cursor-btree cursor)))) (with-buffer-streams (key-buf value-buf) (multiple-value-bind (key val) (db-cursor-move-buffered (cursor-handle cursor) key-buf value-buf :prev t) (if (and key (= (buffer-read-oid key) (cursor-oid cursor))) (values t (deserialize key sc) (deserialize val sc)) (setf (cursor-initialized-p cursor) nil)))) (cursor-last cursor)))) That "cursor-last" call at the end looks very suspicious, and indeed removing it solves the problem. But I don't know where it belongs or if it can be safely removed. Any ideas? Cheers, Chris Laux Chris Laux wrote: > Hi again, > > I seem to have stumbled across another bug: the from-end argument > doesn't seem to be working, map-btree always calls with the same > (presumably last) entry when it is set. > > (defun test-btree-from-end (btree from-end) > (let (acc > (n 0)) > (map-btree #'(lambda (k v) > (push (list k v) acc) > (when (= (incf n) 4) > (return-from test-btree-from-end acc))) > btree :from-end from-end))) > >> (test-btree-from-end ** nil) > ((3398012969 #) (3398012898 #) > (3398012815 #) (3398012772 #)) > >> (test-btree-from-end *** t) > ((3398231130 #) (3398231130 #) > (3398231130 #) (3398231130 #)) > > My setup is BerkeleyDB 4.5.20 with Elephant 0.9.0 with SBCL 1.0.4 on > Linux/x86-64. > > Thanks, > > Chris Laux > From eslick at csail.mit.edu Sat Sep 8 19:49:11 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 8 Sep 2007 15:49:11 -0400 Subject: [elephant-devel] map-btree from-end arg not working? In-Reply-To: <46E2F7EE.3030306@terraminds.com> References: <46E2C603.2030003@terraminds.com> <46E2F7EE.3030306@terraminds.com> Message-ID: Let me look at the history for that file. I can't remember if I fixed some of this on the working branch (darcs) or back during the 0.9.0 release. There are a bunch of subtle interactions that can disturb each other, but that does look like a problem doesn't it? Ian On Sep 8, 2007, at 3:28 PM, Chris Laux wrote: > So I had a look at it myself and it turned out to be much easier > than I > had expected, which is what kept me from trying in the first place. > > The following method in src/db-bdb/bdb-collections.lisp is the > culprit: > > (defmethod cursor-prev ((cursor bdb-cursor)) > (if (cursor-initialized-p cursor) > (let ((sc (get-con (cursor-btree cursor)))) > (with-buffer-streams (key-buf value-buf) > (multiple-value-bind (key val) > (db-cursor-move-buffered (cursor-handle cursor) > key-buf value-buf :prev t) > (if (and key (= (buffer-read-oid key) (cursor-oid cursor))) > (values t (deserialize key sc) > (deserialize val sc)) > (setf (cursor-initialized-p cursor) nil)))) > (cursor-last cursor)))) > > That "cursor-last" call at the end looks very suspicious, and indeed > removing it solves the problem. But I don't know where it belongs > or if > it can be safely removed. Any ideas? > > Cheers, > > Chris Laux > > > Chris Laux wrote: >> Hi again, >> >> I seem to have stumbled across another bug: the from-end argument >> doesn't seem to be working, map-btree always calls with the same >> (presumably last) entry when it is set. >> >> (defun test-btree-from-end (btree from-end) >> (let (acc >> (n 0)) >> (map-btree #'(lambda (k v) >> (push (list k v) acc) >> (when (= (incf n) 4) >> (return-from test-btree-from-end acc))) >> btree :from-end from-end))) >> >>> (test-btree-from-end ** nil) >> ((3398012969 #) (3398012898 #> 226751>) >> (3398012815 #) (3398012772 #> 225621>)) >> >>> (test-btree-from-end *** t) >> ((3398231130 #) (3398231130 #> 552110>) >> (3398231130 #) (3398231130 #> 552110>)) >> >> My setup is BerkeleyDB 4.5.20 with Elephant 0.9.0 with SBCL 1.0.4 on >> Linux/x86-64. >> >> Thanks, >> >> Chris Laux >> > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Sat Sep 8 20:15:46 2007 From: read at robertlread.net (Robert L. Read) Date: Sat, 08 Sep 2007 15:15:46 -0500 Subject: [elephant-devel] map-btree from-end arg not working? In-Reply-To: References: <46E2C603.2030003@terraminds.com> <46E2F7EE.3030306@terraminds.com> Message-ID: <1189282546.9502.489.camel@penguin.yourdomain.com> Thanks for looking at that, Ian. The extensive test suite which the original authors started and Ian and I have improved gives us a lot of leverage here; if the change keeps us green on all tests then we are probably safe. On Sat, 2007-09-08 at 15:49 -0400, Ian Eslick wrote: > Let me look at the history for that file. I can't remember if I > fixed some of this on the working branch (darcs) or back during the > 0.9.0 release. There are a bunch of subtle interactions that can > disturb each other, but that does look like a problem doesn't it? > > Ian > > > On Sep 8, 2007, at 3:28 PM, Chris Laux wrote: > > > So I had a look at it myself and it turned out to be much easier > > than I > > had expected, which is what kept me from trying in the first place. > > > > The following method in src/db-bdb/bdb-collections.lisp is the > > culprit: > > > > (defmethod cursor-prev ((cursor bdb-cursor)) > > (if (cursor-initialized-p cursor) > > (let ((sc (get-con (cursor-btree cursor)))) > > (with-buffer-streams (key-buf value-buf) > > (multiple-value-bind (key val) > > (db-cursor-move-buffered (cursor-handle cursor) > > key-buf value-buf :prev t) > > (if (and key (= (buffer-read-oid key) (cursor-oid cursor))) > > (values t (deserialize key sc) > > (deserialize val sc)) > > (setf (cursor-initialized-p cursor) nil)))) > > (cursor-last cursor)))) > > > > That "cursor-last" call at the end looks very suspicious, and indeed > > removing it solves the problem. But I don't know where it belongs > > or if > > it can be safely removed. Any ideas? > > > > Cheers, > > > > Chris Laux > > > > > > Chris Laux wrote: > >> Hi again, > >> > >> I seem to have stumbled across another bug: the from-end argument > >> doesn't seem to be working, map-btree always calls with the same > >> (presumably last) entry when it is set. > >> > >> (defun test-btree-from-end (btree from-end) > >> (let (acc > >> (n 0)) > >> (map-btree #'(lambda (k v) > >> (push (list k v) acc) > >> (when (= (incf n) 4) > >> (return-from test-btree-from-end acc))) > >> btree :from-end from-end))) > >> > >>> (test-btree-from-end ** nil) > >> ((3398012969 #) (3398012898 # >> 226751>) > >> (3398012815 #) (3398012772 # >> 225621>)) > >> > >>> (test-btree-from-end *** t) > >> ((3398231130 #) (3398231130 # >> 552110>) > >> (3398231130 #) (3398231130 # >> 552110>)) > >> > >> My setup is BerkeleyDB 4.5.20 with Elephant 0.9.0 with SBCL 1.0.4 on > >> Linux/x86-64. > >> > >> Thanks, > >> > >> Chris Laux > >> > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From chris at terraminds.com Thu Sep 13 13:26:54 2007 From: chris at terraminds.com (Chris Laux) Date: Thu, 13 Sep 2007 15:26:54 +0200 Subject: [elephant-devel] Garbage collection problem Message-ID: <46E93A9E.5010009@terraminds.com> Hi all, while using Elephant I've noticed that garbage collections now take up to 10 minutes with the (current model) CPU at 100% load!? This even happens when I end all "client" threads and manually invoke a full GC despite low memory use. I've never seen anything like it before. Does anyone have an idea what could be going on? My setup is BerkeleyDB 4.5.20 with Elephant 0.9.0 with SBCL 1.0.4 on Linux/x86-64. Thanks, Chris Laux From nowhere.man at levallois.eu.org Fri Sep 14 10:53:27 2007 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Fri, 14 Sep 2007 12:53:27 +0200 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <46E93A9E.5010009@terraminds.com> References: <46E93A9E.5010009@terraminds.com> Message-ID: <20070914105327.GF7987@(none)> Scribit Chris Laux dies 13/09/2007 hora 15:26: > while using Elephant I've noticed that garbage collections now take up > to 10 minutes with the (current model) CPU at 100% load!? Could you give a small testcase that demonstrates this? Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From chris at terraminds.com Tue Sep 25 17:33:42 2007 From: chris at terraminds.com (Chris Laux) Date: Tue, 25 Sep 2007 19:33:42 +0200 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <20070914105327.GF7987@(none)> References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> Message-ID: <46F94676.8050101@terraminds.com> Pierre THIERRY wrote: >> while using Elephant I've noticed that garbage collections now take up >> to 10 minutes with the (current model) CPU at 100% load!? > > Could you give a small testcase that demonstrates this? Sorry about the late reply. I haven't yet managed to reproduce the behaviour outside of the production system. I don't think a small test case would produce this, but that it is produced by a fairly large database (GBs) with >>1000 BTrees that was queried by >10 parallel threads. Cheers, Chris Laux From nowhere.man at levallois.eu.org Tue Sep 25 19:22:39 2007 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 25 Sep 2007 21:22:39 +0200 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <46F94676.8050101@terraminds.com> References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> <46F94676.8050101@terraminds.com> Message-ID: <20070925192239.GC8000@(none)> Scribit Chris Laux dies 25/09/2007 hora 19:33: > > Could you give a small testcase that demonstrates this? > [...] I don't think a small test case would produce this, but that it > is produced by a fairly large database (GBs) with >>1000 BTrees that > was queried by >10 parallel threads. Could you give a testcase where a huge DB is created beforehand? Curiously, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Tue Sep 25 22:59:43 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 25 Sep 2007 18:59:43 -0400 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <46F94676.8050101@terraminds.com> References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> <46F94676.8050101@terraminds.com> Message-ID: How about just giving us a profile of memory usage? I don't know what Lisp your are using, but Allegro, for example, has a (prof:with- profiling (:type :space) ...) that gives you a breakdown of memory allocation by function. This would be very helpful. While a great deal of attention has been paid to efficiency, Some operations on some lisps may be consing (bignum processing) and none of us have gone through the system to ensure we've caught all those cases. A likely candidate is the serializer, but I'm sure there could be other corner cases. Ian On Sep 25, 2007, at 1:33 PM, Chris Laux wrote: > Pierre THIERRY wrote: >>> while using Elephant I've noticed that garbage collections now >>> take up >>> to 10 minutes with the (current model) CPU at 100% load!? >> >> Could you give a small testcase that demonstrates this? > > Sorry about the late reply. > > I haven't yet managed to reproduce the behaviour outside of the > production system. I don't think a small test case would produce this, > but that it is produced by a fairly large database (GBs) with >>1000 > BTrees that was queried by >10 parallel threads. > > Cheers, > > Chris Laux > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From chris at terraminds.com Thu Sep 27 13:39:09 2007 From: chris at terraminds.com (Chris Laux) Date: Thu, 27 Sep 2007 15:39:09 +0200 Subject: [elephant-devel] Garbage collection problem In-Reply-To: References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> <46F94676.8050101@terraminds.com> Message-ID: <46FBB27D.7090202@terraminds.com> > How about just giving us a profile of memory usage? The profile is at the bottom of this message, in sbcl's sb-sprof format. If you can't interpret the numbers let me know and I'll dig up the explanation from the web. I'm not really sure how to interpret the results, and it might be buggy, because asking the profiler to report causes the same (similar?) condition as garbage collection, that is the system hangs for minutes then works on as if nothing had happened (added later: or never recovers again...) > A likely candidate is the serializer, but I'm sure there could be other > corner cases. It occurred to me that I should just describe what I'm doing, maybe there's an obvious stupidity on my side. I'm indexing short messages to be queries by a text search engine. A main BTree maps from words to (sub-)BTrees that in turn map from ranking (e.g. time, for recency) to (persistent) result object. To answer a query, I then look up the (sub-)BTrees for each word in the query and map over their elements until I have enough to fill a result page with the intersection of their results. This is probably a rather extreme use of the DB, for example creating so many persistent instances from the store might be the problem for the GC? Is there a "nicer" way of doing something like this? E.g. mapping only to id's, so as not to reconstruct an instance if all I'm going to do with is is "intersect" it's occurrance between lists of messages? Cheers, Chris [NB: the order of the two output tables is reversed for ease of use] Number of samples: 390 Sample interval: 4 regions (approximately 32 kB) Total sampling amount: 1560 regions (approximately 12480 kB) Number of cycles: 0 Self Total Cumul Nr Count % Count % Count % Function ------------------------------------------------------------------------ 1 81 20.8 81 20.8 81 20.8 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM 2 73 18.7 73 18.7 154 39.5 (SB-PCL::FAST-METHOD ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") "#<...>" . "#<...>")) 3 57 14.6 57 14.6 211 54.1 ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE 4 56 14.4 56 14.4 267 68.5 ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM 5 50 12.8 50 12.8 317 81.3 FIND-PACKAGE 6 35 9.0 35 9.0 352 90.3 ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR 7 16 4.1 16 4.1 368 94.4 ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR 8 12 3.1 12 3.1 380 97.4 ELEPHANT::GET-CACHED-INSTANCE 9 6 1.5 6 1.5 386 99.0 (LAMBDA (K V)) 10 1 0.3 1 0.3 387 99.2 LIST 11 1 0.3 1 0.3 388 99.5 ELEPHANT::MAKE-TRANSACTION-RECORD 12 1 0.3 1 0.3 389 99.7 (SB-PCL::FAST-METHOD MAKE-CURSOR (DB-BDB::BDB-BTREE)) 13 1 0.3 1 0.3 390 100.0 SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT 14 0 0.0 244 62.6 390 100.0 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) 15 0 0.0 307 78.7 390 100.0 ELEPHANT-SERIALIZER2::DESERIALIZE 16 0 0.0 378 96.9 390 100.0 (SB-PCL::FAST-METHOD CURSOR-PREV (DB-BDB::BDB-CURSOR)) 17 0 0.0 387 99.2 390 100.0 "Unknown component: #x100F827230" 18 0 0.0 387 99.2 390 100.0 (LAMBDA NIL) 19 0 0.0 390 100.0 390 100.0 (SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . "#<...>")) 20 0 0.0 390 100.0 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T "#<...>" . "#<...>")) 21 0 0.0 390 100.0 390 100.0 GET-BTREE-SUBSEQ 22 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>"))) 23 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>"))) 24 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>") "#<...>" . "#<...>")) 25 0 0.0 390 100.0 390 100.0 ANSWER-QUERY 26 0 0.0 390 100.0 390 100.0 PRODUCE-FEED 27 0 0.0 390 100.0 390 100.0 HUNCHENTOOT::PROCESS-REQUEST 28 0 0.0 390 100.0 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION 29 0 0.0 390 100.0 390 100.0 "Unknown component: #x1000677630" 30 0 0.0 390 100.0 390 100.0 "foreign function call_into_lisp" 31 0 0.0 390 100.0 390 100.0 "foreign function new_thread_trampoline" 32 0 0.0 29 7.4 390 100.0 SB-INT:%FIND-PACKAGE-OR-LOSE 33 0 0.0 29 7.4 390 100.0 SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE 34 0 0.0 29 7.4 390 100.0 INTERN 35 0 0.0 121 31.0 390 100.0 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) 36 0 0.0 121 31.0 390 100.0 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION :AROUND (T)) 37 0 0.0 34 8.7 390 100.0 (FLET #:CLEANUP-FUN-279) 38 0 0.0 22 5.6 390 100.0 (FLET #:CLEANUP-FUN-1505) 39 0 0.0 21 5.4 390 100.0 TRANSLATE-AND-INTERN-SYMBOL 40 0 0.0 1 0.3 390 100.0 PACKAGE-NAME 41 0 0.0 2 0.5 390 100.0 (SB-PCL::FAST-METHOD CURSOR-LAST (DB-BDB::BDB-CURSOR)) 42 0 0.0 1 0.3 390 100.0 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE ------------------------------------------------------------------------ Callers Total. Function Count % Count % Callees ------------------------------------------------------------------------ 30 7.7 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] 51 13.1 (SB-PCL::FAST-METHOD CURSOR-PREV (DB-BDB::BDB-CURSOR)) [16] 81 20.8 81 20.8 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM [1] ------------------------------------------------------------------------ 73 18.7 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] 73 18.7 73 18.7 (SB-PCL::FAST-METHOD ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") "#<...>" . "#<...>")) [2] ------------------------------------------------------------------------ 57 14.6 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] 57 14.6 57 14.6 ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE [3] ------------------------------------------------------------------------ 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] 34 8.7 (FLET #:CLEANUP-FUN-279) [37] 56 14.4 56 14.4 ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM [4] ------------------------------------------------------------------------ 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] 29 7.4 SB-INT:%FIND-PACKAGE-OR-LOSE [32] 50 12.8 50 12.8 FIND-PACKAGE [5] ------------------------------------------------------------------------ 35 9.0 ELEPHANT-SERIALIZER2::DESERIALIZE [15] 35 9.0 35 9.0 ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR [6] ------------------------------------------------------------------------ 16 4.1 ELEPHANT-SERIALIZER2::DESERIALIZE [15] 16 4.1 16 4.1 ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR [7] ------------------------------------------------------------------------ 12 3.1 ELEPHANT-SERIALIZER2::DESERIALIZE [15] 12 3.1 12 3.1 ELEPHANT::GET-CACHED-INSTANCE [8] ------------------------------------------------------------------------ 6 1.5 "Unknown component: #x100F827230" [17] 6 1.5 6 1.5 (LAMBDA (K V)) [9] ------------------------------------------------------------------------ 1 0.3 (SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . "#<...>")) [19] 1 0.3 1 0.3 LIST [10] ------------------------------------------------------------------------ 1 0.3 (SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . "#<...>")) [19] 1 0.3 1 0.3 ELEPHANT::MAKE-TRANSACTION-RECORD [11] ------------------------------------------------------------------------ 1 0.3 "Unknown component: #x100F827230" [17] 1 0.3 1 0.3 (SB-PCL::FAST-METHOD MAKE-CURSOR (DB-BDB::BDB-BTREE)) [12] ------------------------------------------------------------------------ 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE [42] 1 0.3 1 0.3 SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT [13] ------------------------------------------------------------------------ 244 62.6 ELEPHANT-SERIALIZER2::DESERIALIZE [15] 1 0.3 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] 0 0.0 244 62.6 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] 121 31.0 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] 29 7.4 INTERN [34] 1 0.3 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] 73 18.7 (SB-PCL::FAST-METHOD ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") "#<...>" . "#<...>")) [2] ------------------------------------------------------------------------ 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST (DB-BDB::BDB-CURSOR)) [41] 305 78.2 (SB-PCL::FAST-METHOD CURSOR-PREV (DB-BDB::BDB-CURSOR)) [16] 0 0.0 307 78.7 ELEPHANT-SERIALIZER2::DESERIALIZE [15] 16 4.1 ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR [7] 12 3.1 ELEPHANT::GET-CACHED-INSTANCE [8] 35 9.0 ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR [6] 244 62.6 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] ------------------------------------------------------------------------ 378 96.9 "Unknown component: #x100F827230" [17] 0 0.0 378 96.9 (SB-PCL::FAST-METHOD CURSOR-PREV (DB-BDB::BDB-CURSOR)) [16] 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] 51 13.1 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM [1] 305 78.2 ELEPHANT-SERIALIZER2::DESERIALIZE [15] ------------------------------------------------------------------------ 387 99.2 (LAMBDA NIL) [18] 0 0.0 387 99.2 "Unknown component: #x100F827230" [17] 1 0.3 (SB-PCL::FAST-METHOD MAKE-CURSOR (DB-BDB::BDB-BTREE)) [12] 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST (DB-BDB::BDB-CURSOR)) [41] 6 1.5 (LAMBDA (K V)) [9] 378 96.9 (SB-PCL::FAST-METHOD CURSOR-PREV (DB-BDB::BDB-CURSOR)) [16] ------------------------------------------------------------------------ 387 99.2 (SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . "#<...>")) [19] 0 0.0 387 99.2 (LAMBDA NIL) [18] 387 99.2 "Unknown component: #x100F827230" [17] ------------------------------------------------------------------------ 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T "#<...>" . "#<...>")) [20] 0 0.0 390 100.0 (SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . "#<...>")) [19] 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE [42] 1 0.3 ELEPHANT::MAKE-TRANSACTION-RECORD [11] 1 0.3 LIST [10] 387 99.2 (LAMBDA NIL) [18] ------------------------------------------------------------------------ 390 100.0 GET-BTREE-SUBSEQ [21] 0 0.0 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T "#<...>" . "#<...>")) [20] 390 100.0 (SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . "#<...>")) [19] ------------------------------------------------------------------------ 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>"))) [22] 0 0.0 390 100.0 GET-BTREE-SUBSEQ [21] 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T "#<...>" . "#<...>")) [20] ------------------------------------------------------------------------ 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>"))) [23] 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>"))) [22] 390 100.0 GET-BTREE-SUBSEQ [21] ------------------------------------------------------------------------ 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>") "#<...>" . "#<...>")) [24] 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>"))) [23] 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>"))) [22] ------------------------------------------------------------------------ 390 100.0 ANSWER-QUERY [25] 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>") "#<...>" . "#<...>")) [24] 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>"))) [23] ------------------------------------------------------------------------ 390 100.0 PRODUCE-FEED [26] 0 0.0 390 100.0 ANSWER-QUERY [25] 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>") "#<...>" . "#<...>")) [24] ------------------------------------------------------------------------ 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] 0 0.0 390 100.0 PRODUCE-FEED [26] 390 100.0 ANSWER-QUERY [25] ------------------------------------------------------------------------ 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] 0 0.0 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] 390 100.0 PRODUCE-FEED [26] ------------------------------------------------------------------------ 390 100.0 "Unknown component: #x1000677630" [29] 0 0.0 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] ------------------------------------------------------------------------ 390 100.0 "foreign function call_into_lisp" [30] 0 0.0 390 100.0 "Unknown component: #x1000677630" [29] 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] ------------------------------------------------------------------------ 390 100.0 "foreign function new_thread_trampoline" [31] 0 0.0 390 100.0 "foreign function call_into_lisp" [30] 390 100.0 "Unknown component: #x1000677630" [29] ------------------------------------------------------------------------ 0 0.0 390 100.0 "foreign function new_thread_trampoline" [31] 390 100.0 "foreign function call_into_lisp" [30] ------------------------------------------------------------------------ 1 0.3 PACKAGE-NAME [40] 28 7.2 SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE [33] 0 0.0 29 7.4 SB-INT:%FIND-PACKAGE-OR-LOSE [32] 29 7.4 FIND-PACKAGE [5] ------------------------------------------------------------------------ 29 7.4 INTERN [34] 0 0.0 29 7.4 SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE [33] 1 0.3 PACKAGE-NAME [40] 28 7.2 SB-INT:%FIND-PACKAGE-OR-LOSE [32] ------------------------------------------------------------------------ 29 7.4 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] 0 0.0 29 7.4 INTERN [34] 29 7.4 SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE [33] ------------------------------------------------------------------------ 121 31.0 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] 0 0.0 121 31.0 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] 30 7.7 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM [1] 34 8.7 (FLET #:CLEANUP-FUN-279) [37] 57 14.6 ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE [3] ------------------------------------------------------------------------ 121 31.0 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] 0 0.0 121 31.0 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] 121 31.0 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] ------------------------------------------------------------------------ 34 8.7 (SB-PCL::FAST-METHOD ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] 0 0.0 34 8.7 (FLET #:CLEANUP-FUN-279) [37] 34 8.7 ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM [4] ------------------------------------------------------------------------ 22 5.6 (SB-PCL::FAST-METHOD CURSOR-PREV (DB-BDB::BDB-CURSOR)) [16] 0 0.0 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] 22 5.6 ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM [4] ------------------------------------------------------------------------ 21 5.4 (LABELS ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] 0 0.0 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] 21 5.4 FIND-PACKAGE [5] ------------------------------------------------------------------------ 1 0.3 SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE [33] 0 0.0 1 0.3 PACKAGE-NAME [40] 1 0.3 SB-INT:%FIND-PACKAGE-OR-LOSE [32] ------------------------------------------------------------------------ 2 0.5 "Unknown component: #x100F827230" [17] 0 0.0 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST (DB-BDB::BDB-CURSOR)) [41] 2 0.5 ELEPHANT-SERIALIZER2::DESERIALIZE [15] ------------------------------------------------------------------------ 1 0.3 (SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . "#<...>")) [19] 0 0.0 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE [42] 1 0.3 SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT [13] ------------------------------------------------------------------------ From eslick at csail.mit.edu Thu Sep 27 13:58:50 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 27 Sep 2007 09:58:50 -0400 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <46FBB27D.7090202@terraminds.com> References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> <46F94676.8050101@terraminds.com> <46FBB27D.7090202@terraminds.com> Message-ID: <1334A2A7-9F0B-474C-9BED-CDA2874F4DEC@csail.mit.edu> This was (sb-sprof:with-profiling (:mode :alloc ...) ...) ?? On Sep 27, 2007, at 9:39 AM, Chris Laux wrote: >> How about just giving us a profile of memory usage? > > The profile is at the bottom of this message, in sbcl's sb-sprof > format. > If you can't interpret the numbers let me know and I'll dig up the > explanation from the web. I'm not really sure how to interpret the > results, and it might be buggy, because asking the profiler to report > causes the same (similar?) condition as garbage collection, that is > the > system hangs for minutes then works on as if nothing had happened > (added > later: or never recovers again...) > > >> A likely candidate is the serializer, but I'm sure there could be >> other >> corner cases. > > It occurred to me that I should just describe what I'm doing, maybe > there's an obvious stupidity on my side. I'm indexing short > messages to > be queries by a text search engine. > > A main BTree maps from words to (sub-)BTrees that in turn map from > ranking (e.g. time, for recency) to (persistent) result object. To > answer a query, I then look up the (sub-)BTrees for each word in the > query and map over their elements until I have enough to fill a result > page with the intersection of their results. > > This is probably a rather extreme use of the DB, for example > creating so > many persistent instances from the store might be the problem for the > GC? Is there a "nicer" way of doing something like this? E.g. mapping > only to id's, so as not to reconstruct an instance if all I'm going to > do with is is "intersect" it's occurrance between lists of messages? > > Cheers, > > Chris > > > [NB: the order of the two output tables is reversed for ease of use] > > > Number of samples: 390 > Sample interval: 4 regions (approximately 32 kB) > Total sampling amount: 1560 regions (approximately 12480 kB) > Number of cycles: 0 > > > Self Total Cumul > Nr Count % Count % Count % Function > ---------------------------------------------------------------------- > -- > 1 81 20.8 81 20.8 81 20.8 > ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM > 2 73 18.7 73 18.7 154 39.5 (SB-PCL::FAST-METHOD > ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") > "#<...>" > . "#<...>")) > 3 57 14.6 57 14.6 211 54.1 > ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE > 4 56 14.4 56 14.4 267 68.5 > ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM > 5 50 12.8 50 12.8 317 81.3 FIND-PACKAGE > 6 35 9.0 35 9.0 352 90.3 > ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR > 7 16 4.1 16 4.1 368 94.4 > ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR > 8 12 3.1 12 3.1 380 97.4 ELEPHANT::GET-CACHED- > INSTANCE > 9 6 1.5 6 1.5 386 99.0 (LAMBDA (K V)) > 10 1 0.3 1 0.3 387 99.2 LIST > 11 1 0.3 1 0.3 388 99.5 > ELEPHANT::MAKE-TRANSACTION-RECORD > 12 1 0.3 1 0.3 389 99.7 (SB-PCL::FAST-METHOD > MAKE-CURSOR (DB-BDB::BDB-BTREE)) > 13 1 0.3 1 0.3 390 100.0 > SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT > 14 0 0.0 244 62.6 390 100.0 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) > 15 0 0.0 307 78.7 390 100.0 > ELEPHANT-SERIALIZER2::DESERIALIZE > 16 0 0.0 378 96.9 390 100.0 (SB-PCL::FAST-METHOD > CURSOR-PREV (DB-BDB::BDB-CURSOR)) > 17 0 0.0 387 99.2 390 100.0 "Unknown component: > #x100F827230" > 18 0 0.0 387 99.2 390 100.0 (LAMBDA NIL) > 19 0 0.0 390 100.0 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) > 20 0 0.0 390 100.0 390 100.0 (SB-PCL::FAST-METHOD > MAP-BTREE (T "#<...>" . "#<...>")) > 21 0 0.0 390 100.0 390 100.0 GET-BTREE-SUBSEQ > 22 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL > ("#<...>" > . "#<...>"))) > 23 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL > ("#<...>" > . "#<...>"))) > 24 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL > ("#<...>" > . "#<...>") "#<...>" . "#<...>")) > 25 0 0.0 390 100.0 390 100.0 ANSWER-QUERY > 26 0 0.0 390 100.0 390 100.0 PRODUCE-FEED > 27 0 0.0 390 100.0 390 100.0 HUNCHENTOOT::PROCESS- > REQUEST > 28 0 0.0 390 100.0 390 100.0 HUNCHENTOOT::PROCESS- > CONNECTION > 29 0 0.0 390 100.0 390 100.0 "Unknown component: > #x1000677630" > 30 0 0.0 390 100.0 390 100.0 "foreign function > call_into_lisp" > 31 0 0.0 390 100.0 390 100.0 "foreign function > new_thread_trampoline" > 32 0 0.0 29 7.4 390 100.0 SB-INT:%FIND-PACKAGE-OR- > LOSE > 33 0 0.0 29 7.4 390 100.0 > SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE > 34 0 0.0 29 7.4 390 100.0 INTERN > 35 0 0.0 121 31.0 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) > 36 0 0.0 121 31.0 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION :AROUND (T)) > 37 0 0.0 34 8.7 390 100.0 (FLET #:CLEANUP-FUN-279) > 38 0 0.0 22 5.6 390 100.0 (FLET #:CLEANUP-FUN-1505) > 39 0 0.0 21 5.4 390 100.0 TRANSLATE-AND-INTERN- > SYMBOL > 40 0 0.0 1 0.3 390 100.0 PACKAGE-NAME > 41 0 0.0 2 0.5 390 100.0 (SB-PCL::FAST-METHOD > CURSOR-LAST (DB-BDB::BDB-CURSOR)) > 42 0 0.0 1 0.3 390 100.0 > SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE > ---------------------------------------------------------------------- > -- > > Callers > Total. Function > Count % Count % Callees > ---------------------------------------------------------------------- > -- > 30 7.7 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > 51 13.1 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > 81 20.8 81 20.8 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM [1] > ---------------------------------------------------------------------- > -- > 73 18.7 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 73 18.7 73 18.7 (SB-PCL::FAST-METHOD > ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") > "#<...>" > . "#<...>")) [2] > ---------------------------------------------------------------------- > -- > 57 14.6 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > 57 14.6 57 14.6 ELEPHANT::DESERIALIZE-DATABASE-VERSION- > VALUE [3] > ---------------------------------------------------------------------- > -- > 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] > 34 8.7 (FLET #:CLEANUP-FUN-279) [37] > 56 14.4 56 14.4 ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM [4] > ---------------------------------------------------------------------- > -- > 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] > 29 7.4 SB-INT:%FIND-PACKAGE-OR-LOSE [32] > 50 12.8 50 12.8 FIND-PACKAGE [5] > ---------------------------------------------------------------------- > -- > 35 9.0 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 35 9.0 35 9.0 ELEPHANT-SERIALIZER2::GET-CIRCULARITY- > VECTOR [6] > ---------------------------------------------------------------------- > -- > 16 4.1 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 16 4.1 16 4.1 > ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR [7] > ---------------------------------------------------------------------- > -- > 12 3.1 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 12 3.1 12 3.1 ELEPHANT::GET-CACHED-INSTANCE [8] > ---------------------------------------------------------------------- > -- > 6 1.5 "Unknown component: #x100F827230" [17] > 6 1.5 6 1.5 (LAMBDA (K V)) [9] > ---------------------------------------------------------------------- > -- > 1 0.3 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 1 0.3 1 0.3 LIST [10] > ---------------------------------------------------------------------- > -- > 1 0.3 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 1 0.3 1 0.3 ELEPHANT::MAKE-TRANSACTION-RECORD [11] > ---------------------------------------------------------------------- > -- > 1 0.3 "Unknown component: #x100F827230" [17] > 1 0.3 1 0.3 (SB-PCL::FAST-METHOD MAKE-CURSOR > (DB-BDB::BDB-BTREE)) [12] > ---------------------------------------------------------------------- > -- > 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE > [42] > 1 0.3 1 0.3 SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT [13] > ---------------------------------------------------------------------- > -- > 244 62.6 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 1 0.3 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 0 0.0 244 62.6 (LABELS ELEPHANT-SERIALIZER2::% > DESERIALIZE) [14] > 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] > 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] > 29 7.4 INTERN [34] > 1 0.3 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 73 18.7 (SB-PCL::FAST-METHOD > ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") > "#<...>" > . "#<...>")) [2] > ---------------------------------------------------------------------- > -- > 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST > (DB-BDB::BDB-CURSOR)) [41] > 305 78.2 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > 0 0.0 307 78.7 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 16 4.1 > ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR [7] > 12 3.1 ELEPHANT::GET-CACHED-INSTANCE [8] > 35 9.0 > ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR [6] > 244 62.6 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > ---------------------------------------------------------------------- > -- > 378 96.9 "Unknown component: #x100F827230" [17] > 0 0.0 378 96.9 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] > 51 13.1 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM > [1] > 305 78.2 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > ---------------------------------------------------------------------- > -- > 387 99.2 (LAMBDA NIL) [18] > 0 0.0 387 99.2 "Unknown component: #x100F827230" [17] > 1 0.3 (SB-PCL::FAST-METHOD MAKE-CURSOR > (DB-BDB::BDB-BTREE)) [12] > 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST > (DB-BDB::BDB-CURSOR)) [41] > 6 1.5 (LAMBDA (K V)) [9] > 378 96.9 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > ---------------------------------------------------------------------- > -- > 387 99.2 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 0 0.0 387 99.2 (LAMBDA NIL) [18] > 387 99.2 "Unknown component: #x100F827230" [17] > ---------------------------------------------------------------------- > -- > 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T > "#<...>" . "#<...>")) [20] > 0 0.0 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE > [42] > 1 0.3 ELEPHANT::MAKE-TRANSACTION-RECORD [11] > 1 0.3 LIST [10] > 387 99.2 (LAMBDA NIL) [18] > ---------------------------------------------------------------------- > -- > 390 100.0 GET-BTREE-SUBSEQ [21] > 0 0.0 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T > "#<...>" . > "#<...>")) [20] > 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > ---------------------------------------------------------------------- > -- > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [22] > 0 0.0 390 100.0 GET-BTREE-SUBSEQ [21] > 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T > "#<...>" . "#<...>")) [20] > ---------------------------------------------------------------------- > -- > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [23] > 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [22] > 390 100.0 GET-BTREE-SUBSEQ [21] > ---------------------------------------------------------------------- > -- > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>") > "#<...>" . "#<...>")) [24] > 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [23] > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [22] > ---------------------------------------------------------------------- > -- > 390 100.0 ANSWER-QUERY [25] > 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>") > "#<...>" . "#<...>")) [24] > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [23] > ---------------------------------------------------------------------- > -- > 390 100.0 PRODUCE-FEED [26] > 0 0.0 390 100.0 ANSWER-QUERY [25] > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>") > "#<...>" . "#<...>")) [24] > ---------------------------------------------------------------------- > -- > 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] > 0 0.0 390 100.0 PRODUCE-FEED [26] > 390 100.0 ANSWER-QUERY [25] > ---------------------------------------------------------------------- > -- > 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] > 0 0.0 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] > 390 100.0 PRODUCE-FEED [26] > ---------------------------------------------------------------------- > -- > 390 100.0 "Unknown component: #x1000677630" [29] > 0 0.0 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] > 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] > ---------------------------------------------------------------------- > -- > 390 100.0 "foreign function call_into_lisp" [30] > 0 0.0 390 100.0 "Unknown component: #x1000677630" [29] > 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] > ---------------------------------------------------------------------- > -- > 390 100.0 "foreign function > new_thread_trampoline" [31] > 0 0.0 390 100.0 "foreign function call_into_lisp" [30] > 390 100.0 "Unknown component: #x1000677630" [29] > ---------------------------------------------------------------------- > -- > 0 0.0 390 100.0 "foreign function > new_thread_trampoline" [31] > 390 100.0 "foreign function call_into_lisp" [30] > ---------------------------------------------------------------------- > -- > 1 0.3 PACKAGE-NAME [40] > 28 7.2 SB-INT:FIND-UNDELETED-PACKAGE-OR- > LOSE [33] > 0 0.0 29 7.4 SB-INT:%FIND-PACKAGE-OR-LOSE [32] > 29 7.4 FIND-PACKAGE [5] > ---------------------------------------------------------------------- > -- > 29 7.4 INTERN [34] > 0 0.0 29 7.4 SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE [33] > 1 0.3 PACKAGE-NAME [40] > 28 7.2 SB-INT:%FIND-PACKAGE-OR-LOSE [32] > ---------------------------------------------------------------------- > -- > 29 7.4 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 0 0.0 29 7.4 INTERN [34] > 29 7.4 SB-INT:FIND-UNDELETED-PACKAGE-OR- > LOSE [33] > ---------------------------------------------------------------------- > -- > 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] > 0 0.0 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > 30 7.7 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM > [1] > 34 8.7 (FLET #:CLEANUP-FUN-279) [37] > 57 14.6 > ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE [3] > ---------------------------------------------------------------------- > -- > 121 31.0 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 0 0.0 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] > 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > ---------------------------------------------------------------------- > -- > 34 8.7 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > 0 0.0 34 8.7 (FLET #:CLEANUP-FUN-279) [37] > 34 8.7 ELEPHANT-MEMUTIL::RETURN-BUFFER- > STREAM [4] > ---------------------------------------------------------------------- > -- > 22 5.6 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > 0 0.0 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] > 22 5.6 ELEPHANT-MEMUTIL::RETURN-BUFFER- > STREAM [4] > ---------------------------------------------------------------------- > -- > 21 5.4 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 0 0.0 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] > 21 5.4 FIND-PACKAGE [5] > ---------------------------------------------------------------------- > -- > 1 0.3 SB-INT:FIND-UNDELETED-PACKAGE-OR- > LOSE [33] > 0 0.0 1 0.3 PACKAGE-NAME [40] > 1 0.3 SB-INT:%FIND-PACKAGE-OR-LOSE [32] > ---------------------------------------------------------------------- > -- > 2 0.5 "Unknown component: #x100F827230" [17] > 0 0.0 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST > (DB-BDB::BDB-CURSOR)) [41] > 2 0.5 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > ---------------------------------------------------------------------- > -- > 1 0.3 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 0 0.0 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE [42] > 1 0.3 SB-KERNEL::%MAKE-INSTANCE-WITH- > LAYOUT [13] > ---------------------------------------------------------------------- > -- > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Thu Sep 27 14:36:06 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 27 Sep 2007 10:36:06 -0400 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <46FBB27D.7090202@terraminds.com> References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> <46F94676.8050101@terraminds.com> <46FBB27D.7090202@terraminds.com> Message-ID: <466D394A-FD55-42F3-9253-D8E48C58D05F@csail.mit.edu> On Sep 27, 2007, at 9:39 AM, Chris Laux wrote: >> How about just giving us a profile of memory usage? > > The profile is at the bottom of this message, in sbcl's sb-sprof > format. > If you can't interpret the numbers let me know and I'll dig up the > explanation from the web. I'm not really sure how to interpret the > results, and it might be buggy, because asking the profiler to report > causes the same (similar?) condition as garbage collection, that is > the > system hangs for minutes then works on as if nothing had happened > (added > later: or never recovers again...) It looks like this comes from the initial run of your system, so there is a bunch of space allocated there that should only be allocated once so it's not representative of what is going on later. (Just to be sure, it might be interesting to look at elephant- memutil::*buffer-streams* and the sizes of all the buffer streams therein (apply #'+ (map 'list #'elephant-memutil::buffer-stream-size elephant-memutil::*buffer-streams*)) and see how many bytes are allocated to the serializer buffers.) Some suspects: - If you are reading a ton of strings from BDB, then you'll be allocated tons of space for each one as they have to go to/from foreign space. If you are reading tons of instances you'll be allocating space for the object header. - If you are doing a significant amount of deserialization with lots of threads than you should know that each deserialization requires a call to (with-lock ...) to ensure that the shared pool of buffer streams is thread safe (a problem with elephant < 0.9). This could conceivably cause a lockup if there are lots of small deserializations happening concurrently across threads mapping over the same Btrees. Are you sure it's GC that's eating all the time, or non-lisp CPU time in general? My guess is that you are deserializing much more data than you need during your join due to the map function reading the actual values of the tree into memory as you walk them. If you have a large, common word with little intersection in your final set then you'll spend alot of time reading in and throwing away that data. Although it breaks the abstraction barrier, using IDs will be a definite gain. You'd just make that second BTree pairs of word- freq / obj-oid. Then you use the OID and object type to grab the object directly from elephant: (elephant::get-cached-instance oid classname) Now using and throwing away persistent objects also means that you are filling a weak hash table (acting as cache) with objects that are then being reclaimed and this may be more expensive under SBCL than simple allocation. Your per-word btrees are likely to be reused query-to-query, but your result strings much less so. Instead, you may want to use your ID to directly grab the strings of interest and then, if you need to annotate them with more information, This kind of application (inverse document indexing) using join queries is good fodder for making a contrib to elephant as I'm sure others would want to do it. I'm not sure the elephant architecture, as it stands, is the best way to do this though. You might be better off, performance wise, doing this in a C full-text indexing system and wrapping an interface to it. Then you only need to allocate space when you read in the result strings. A potential feature for which there are already hooks is maintaining your own pool of lisp strings to deserialize into rather than relying on the GC. If you did a query to a BTree and provided a hint containing the string storage that might also relieve your memory congestion. This wouldn't show up until we get around to releasing the patches and features on the 0.9.1 tree and I can't predict when that will be. In summary, every read you make from elephant allocates memory for the read value unless it is a persistent object and hits in the weak hash table. Objects are not that big, but the MOP means quite a bit of work goes on to allocate them and the memory is non-trival (i.e. bigger than a cons). You want to engineer the system so you are only reading the large objects you want, and are letting elephant do the sorted lookups internally. Sadly, we don't have proper join support due to the way we use the internal BDB Btree structure so there isn't a high performance way to do it. However, using IDs should buy you quite a bit. Regards, Ian > >> A likely candidate is the serializer, but I'm sure there could be >> other >> corner cases. > > It occurred to me that I should just describe what I'm doing, maybe > there's an obvious stupidity on my side. I'm indexing short > messages to > be queries by a text search engine. > > A main BTree maps from words to (sub-)BTrees that in turn map from > ranking (e.g. time, for recency) to (persistent) result object. To > answer a query, I then look up the (sub-)BTrees for each word in the > query and map over their elements until I have enough to fill a result > page with the intersection of their results. > > This is probably a rather extreme use of the DB, for example > creating so > many persistent instances from the store might be the problem for the > GC? Is there a "nicer" way of doing something like this? E.g. mapping > only to id's, so as not to reconstruct an instance if all I'm going to > do with is is "intersect" it's occurrance between lists of messages? > > Cheers, > > Chris > > > [NB: the order of the two output tables is reversed for ease of use] > > > Number of samples: 390 > Sample interval: 4 regions (approximately 32 kB) > Total sampling amount: 1560 regions (approximately 12480 kB) > Number of cycles: 0 > > > Self Total Cumul > Nr Count % Count % Count % Function > ---------------------------------------------------------------------- > -- > 1 81 20.8 81 20.8 81 20.8 > ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM > 2 73 18.7 73 18.7 154 39.5 (SB-PCL::FAST-METHOD > ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") > "#<...>" > . "#<...>")) > 3 57 14.6 57 14.6 211 54.1 > ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE > 4 56 14.4 56 14.4 267 68.5 > ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM > 5 50 12.8 50 12.8 317 81.3 FIND-PACKAGE > 6 35 9.0 35 9.0 352 90.3 > ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR > 7 16 4.1 16 4.1 368 94.4 > ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR > 8 12 3.1 12 3.1 380 97.4 ELEPHANT::GET-CACHED- > INSTANCE > 9 6 1.5 6 1.5 386 99.0 (LAMBDA (K V)) > 10 1 0.3 1 0.3 387 99.2 LIST > 11 1 0.3 1 0.3 388 99.5 > ELEPHANT::MAKE-TRANSACTION-RECORD > 12 1 0.3 1 0.3 389 99.7 (SB-PCL::FAST-METHOD > MAKE-CURSOR (DB-BDB::BDB-BTREE)) > 13 1 0.3 1 0.3 390 100.0 > SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT > 14 0 0.0 244 62.6 390 100.0 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) > 15 0 0.0 307 78.7 390 100.0 > ELEPHANT-SERIALIZER2::DESERIALIZE > 16 0 0.0 378 96.9 390 100.0 (SB-PCL::FAST-METHOD > CURSOR-PREV (DB-BDB::BDB-CURSOR)) > 17 0 0.0 387 99.2 390 100.0 "Unknown component: > #x100F827230" > 18 0 0.0 387 99.2 390 100.0 (LAMBDA NIL) > 19 0 0.0 390 100.0 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) > 20 0 0.0 390 100.0 390 100.0 (SB-PCL::FAST-METHOD > MAP-BTREE (T "#<...>" . "#<...>")) > 21 0 0.0 390 100.0 390 100.0 GET-BTREE-SUBSEQ > 22 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL > ("#<...>" > . "#<...>"))) > 23 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL > ("#<...>" > . "#<...>"))) > 24 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL > ("#<...>" > . "#<...>") "#<...>" . "#<...>")) > 25 0 0.0 390 100.0 390 100.0 ANSWER-QUERY > 26 0 0.0 390 100.0 390 100.0 PRODUCE-FEED > 27 0 0.0 390 100.0 390 100.0 HUNCHENTOOT::PROCESS- > REQUEST > 28 0 0.0 390 100.0 390 100.0 HUNCHENTOOT::PROCESS- > CONNECTION > 29 0 0.0 390 100.0 390 100.0 "Unknown component: > #x1000677630" > 30 0 0.0 390 100.0 390 100.0 "foreign function > call_into_lisp" > 31 0 0.0 390 100.0 390 100.0 "foreign function > new_thread_trampoline" > 32 0 0.0 29 7.4 390 100.0 SB-INT:%FIND-PACKAGE-OR- > LOSE > 33 0 0.0 29 7.4 390 100.0 > SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE > 34 0 0.0 29 7.4 390 100.0 INTERN > 35 0 0.0 121 31.0 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) > 36 0 0.0 121 31.0 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION :AROUND (T)) > 37 0 0.0 34 8.7 390 100.0 (FLET #:CLEANUP-FUN-279) > 38 0 0.0 22 5.6 390 100.0 (FLET #:CLEANUP-FUN-1505) > 39 0 0.0 21 5.4 390 100.0 TRANSLATE-AND-INTERN- > SYMBOL > 40 0 0.0 1 0.3 390 100.0 PACKAGE-NAME > 41 0 0.0 2 0.5 390 100.0 (SB-PCL::FAST-METHOD > CURSOR-LAST (DB-BDB::BDB-CURSOR)) > 42 0 0.0 1 0.3 390 100.0 > SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE > ---------------------------------------------------------------------- > -- > > Callers > Total. Function > Count % Count % Callees > ---------------------------------------------------------------------- > -- > 30 7.7 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > 51 13.1 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > 81 20.8 81 20.8 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM [1] > ---------------------------------------------------------------------- > -- > 73 18.7 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 73 18.7 73 18.7 (SB-PCL::FAST-METHOD > ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") > "#<...>" > . "#<...>")) [2] > ---------------------------------------------------------------------- > -- > 57 14.6 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > 57 14.6 57 14.6 ELEPHANT::DESERIALIZE-DATABASE-VERSION- > VALUE [3] > ---------------------------------------------------------------------- > -- > 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] > 34 8.7 (FLET #:CLEANUP-FUN-279) [37] > 56 14.4 56 14.4 ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM [4] > ---------------------------------------------------------------------- > -- > 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] > 29 7.4 SB-INT:%FIND-PACKAGE-OR-LOSE [32] > 50 12.8 50 12.8 FIND-PACKAGE [5] > ---------------------------------------------------------------------- > -- > 35 9.0 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 35 9.0 35 9.0 ELEPHANT-SERIALIZER2::GET-CIRCULARITY- > VECTOR [6] > ---------------------------------------------------------------------- > -- > 16 4.1 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 16 4.1 16 4.1 > ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR [7] > ---------------------------------------------------------------------- > -- > 12 3.1 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 12 3.1 12 3.1 ELEPHANT::GET-CACHED-INSTANCE [8] > ---------------------------------------------------------------------- > -- > 6 1.5 "Unknown component: #x100F827230" [17] > 6 1.5 6 1.5 (LAMBDA (K V)) [9] > ---------------------------------------------------------------------- > -- > 1 0.3 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 1 0.3 1 0.3 LIST [10] > ---------------------------------------------------------------------- > -- > 1 0.3 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 1 0.3 1 0.3 ELEPHANT::MAKE-TRANSACTION-RECORD [11] > ---------------------------------------------------------------------- > -- > 1 0.3 "Unknown component: #x100F827230" [17] > 1 0.3 1 0.3 (SB-PCL::FAST-METHOD MAKE-CURSOR > (DB-BDB::BDB-BTREE)) [12] > ---------------------------------------------------------------------- > -- > 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE > [42] > 1 0.3 1 0.3 SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT [13] > ---------------------------------------------------------------------- > -- > 244 62.6 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 1 0.3 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 0 0.0 244 62.6 (LABELS ELEPHANT-SERIALIZER2::% > DESERIALIZE) [14] > 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] > 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] > 29 7.4 INTERN [34] > 1 0.3 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 73 18.7 (SB-PCL::FAST-METHOD > ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>") > "#<...>" > . "#<...>")) [2] > ---------------------------------------------------------------------- > -- > 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST > (DB-BDB::BDB-CURSOR)) [41] > 305 78.2 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > 0 0.0 307 78.7 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > 16 4.1 > ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR [7] > 12 3.1 ELEPHANT::GET-CACHED-INSTANCE [8] > 35 9.0 > ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR [6] > 244 62.6 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > ---------------------------------------------------------------------- > -- > 378 96.9 "Unknown component: #x100F827230" [17] > 0 0.0 378 96.9 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] > 51 13.1 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM > [1] > 305 78.2 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > ---------------------------------------------------------------------- > -- > 387 99.2 (LAMBDA NIL) [18] > 0 0.0 387 99.2 "Unknown component: #x100F827230" [17] > 1 0.3 (SB-PCL::FAST-METHOD MAKE-CURSOR > (DB-BDB::BDB-BTREE)) [12] > 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST > (DB-BDB::BDB-CURSOR)) [41] > 6 1.5 (LAMBDA (K V)) [9] > 378 96.9 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > ---------------------------------------------------------------------- > -- > 387 99.2 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 0 0.0 387 99.2 (LAMBDA NIL) [18] > 387 99.2 "Unknown component: #x100F827230" [17] > ---------------------------------------------------------------------- > -- > 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T > "#<...>" . "#<...>")) [20] > 0 0.0 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE > [42] > 1 0.3 ELEPHANT::MAKE-TRANSACTION-RECORD [11] > 1 0.3 LIST [10] > 387 99.2 (LAMBDA NIL) [18] > ---------------------------------------------------------------------- > -- > 390 100.0 GET-BTREE-SUBSEQ [21] > 0 0.0 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T > "#<...>" . > "#<...>")) [20] > 390 100.0 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > ---------------------------------------------------------------------- > -- > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [22] > 0 0.0 390 100.0 GET-BTREE-SUBSEQ [21] > 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T > "#<...>" . "#<...>")) [20] > ---------------------------------------------------------------------- > -- > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [23] > 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [22] > 390 100.0 GET-BTREE-SUBSEQ [21] > ---------------------------------------------------------------------- > -- > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>") > "#<...>" . "#<...>")) [24] > 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [23] > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [22] > ---------------------------------------------------------------------- > -- > 390 100.0 ANSWER-QUERY [25] > 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>") > "#<...>" . "#<...>")) [24] > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>"))) [23] > ---------------------------------------------------------------------- > -- > 390 100.0 PRODUCE-FEED [26] > 0 0.0 390 100.0 ANSWER-QUERY [25] > 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . > "#<...>") > "#<...>" . "#<...>")) [24] > ---------------------------------------------------------------------- > -- > 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] > 0 0.0 390 100.0 PRODUCE-FEED [26] > 390 100.0 ANSWER-QUERY [25] > ---------------------------------------------------------------------- > -- > 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] > 0 0.0 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] > 390 100.0 PRODUCE-FEED [26] > ---------------------------------------------------------------------- > -- > 390 100.0 "Unknown component: #x1000677630" [29] > 0 0.0 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] > 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27] > ---------------------------------------------------------------------- > -- > 390 100.0 "foreign function call_into_lisp" [30] > 0 0.0 390 100.0 "Unknown component: #x1000677630" [29] > 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28] > ---------------------------------------------------------------------- > -- > 390 100.0 "foreign function > new_thread_trampoline" [31] > 0 0.0 390 100.0 "foreign function call_into_lisp" [30] > 390 100.0 "Unknown component: #x1000677630" [29] > ---------------------------------------------------------------------- > -- > 0 0.0 390 100.0 "foreign function > new_thread_trampoline" [31] > 390 100.0 "foreign function call_into_lisp" [30] > ---------------------------------------------------------------------- > -- > 1 0.3 PACKAGE-NAME [40] > 28 7.2 SB-INT:FIND-UNDELETED-PACKAGE-OR- > LOSE [33] > 0 0.0 29 7.4 SB-INT:%FIND-PACKAGE-OR-LOSE [32] > 29 7.4 FIND-PACKAGE [5] > ---------------------------------------------------------------------- > -- > 29 7.4 INTERN [34] > 0 0.0 29 7.4 SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE [33] > 1 0.3 PACKAGE-NAME [40] > 28 7.2 SB-INT:%FIND-PACKAGE-OR-LOSE [32] > ---------------------------------------------------------------------- > -- > 29 7.4 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 0 0.0 29 7.4 INTERN [34] > 29 7.4 SB-INT:FIND-UNDELETED-PACKAGE-OR- > LOSE [33] > ---------------------------------------------------------------------- > -- > 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] > 0 0.0 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > 30 7.7 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM > [1] > 34 8.7 (FLET #:CLEANUP-FUN-279) [37] > 57 14.6 > ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE [3] > ---------------------------------------------------------------------- > -- > 121 31.0 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 0 0.0 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION :AROUND (T)) [36] > 121 31.0 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > ---------------------------------------------------------------------- > -- > 34 8.7 (SB-PCL::FAST-METHOD > ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35] > 0 0.0 34 8.7 (FLET #:CLEANUP-FUN-279) [37] > 34 8.7 ELEPHANT-MEMUTIL::RETURN-BUFFER- > STREAM [4] > ---------------------------------------------------------------------- > -- > 22 5.6 (SB-PCL::FAST-METHOD CURSOR-PREV > (DB-BDB::BDB-CURSOR)) [16] > 0 0.0 22 5.6 (FLET #:CLEANUP-FUN-1505) [38] > 22 5.6 ELEPHANT-MEMUTIL::RETURN-BUFFER- > STREAM [4] > ---------------------------------------------------------------------- > -- > 21 5.4 (LABELS > ELEPHANT-SERIALIZER2::%DESERIALIZE) [14] > 0 0.0 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39] > 21 5.4 FIND-PACKAGE [5] > ---------------------------------------------------------------------- > -- > 1 0.3 SB-INT:FIND-UNDELETED-PACKAGE-OR- > LOSE [33] > 0 0.0 1 0.3 PACKAGE-NAME [40] > 1 0.3 SB-INT:%FIND-PACKAGE-OR-LOSE [32] > ---------------------------------------------------------------------- > -- > 2 0.5 "Unknown component: #x100F827230" [17] > 0 0.0 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST > (DB-BDB::BDB-CURSOR)) [41] > 2 0.5 ELEPHANT-SERIALIZER2::DESERIALIZE [15] > ---------------------------------------------------------------------- > -- > 1 0.3 (SB-PCL::FAST-METHOD > ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" . > "#<...>")) [19] > 0 0.0 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE [42] > 1 0.3 SB-KERNEL::%MAKE-INSTANCE-WITH- > LAYOUT [13] > ---------------------------------------------------------------------- > -- > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From chris at terraminds.com Thu Sep 27 14:37:07 2007 From: chris at terraminds.com (Chris Laux) Date: Thu, 27 Sep 2007 16:37:07 +0200 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <1334A2A7-9F0B-474C-9BED-CDA2874F4DEC@csail.mit.edu> References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> <46F94676.8050101@terraminds.com> <46FBB27D.7090202@terraminds.com> <1334A2A7-9F0B-474C-9BED-CDA2874F4DEC@csail.mit.edu> Message-ID: <46FBC013.4090505@terraminds.com> > This was (sb-sprof:with-profiling (:mode :alloc ...) ...) ?? Yes. I exchanged the order of the two tables, as noted below. And the line formatting is messed up at bit by email. What's wrong with it? >> [NB: the order of the two output tables is reversed for ease of use] SBCL version is 1.0.4 Chris From chris at terraminds.com Thu Sep 27 15:46:14 2007 From: chris at terraminds.com (Chris Laux) Date: Thu, 27 Sep 2007 17:46:14 +0200 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <466D394A-FD55-42F3-9253-D8E48C58D05F@csail.mit.edu> References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> <46F94676.8050101@terraminds.com> <46FBB27D.7090202@terraminds.com> <466D394A-FD55-42F3-9253-D8E48C58D05F@csail.mit.edu> Message-ID: <46FBD046.3000701@terraminds.com> Thanks Ian, that is some good analysis. > - If you are doing a significant amount of deserialization with lots of > threads than you should know that each deserialization requires a call > to (with-lock ...) to ensure that the shared pool of buffer streams is > thread safe (a problem with elephant < 0.9). This could conceivably > cause a lockup if there are lots of small deserializations happening > concurrently across threads mapping over the same Btrees. I had a vague suspicion of something like that, but only looked at transactions. I guess I would have to modify elephant to allow me to do the locking to solve such a problem. > Are you sure > it's GC that's eating all the time, or non-lisp CPU time in general? Well, the 99% CPU is reported for the sbcl process. I only know that manually invoking a gc will trigger the problem. > Although it breaks the abstraction barrier, using IDs will be a definite > gain. You'd just make that second BTree pairs of word-freq / obj-oid. > Then you use the OID and object type to grab the object directly from > elephant: (elephant::get-cached-instance oid classname) I have also been considering doing away with the second layer of BTrees, and using my own, more "linear" structures. Not sure what that could look like exactly though. > You might be better off, performance > wise, doing this in a C full-text indexing system and wrapping an > interface to it. I hadn't thought of that yet. Can you recommend any? Anyway, I guess I was asking for trouble a bit with my setup. I'm not sure how I'll proceed yet, but if I stick to the two-level BTree setup and use id's I know what to look out for. Thanks again, Chris From eslick at csail.mit.edu Thu Sep 27 15:57:38 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 27 Sep 2007 11:57:38 -0400 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <46FBD046.3000701@terraminds.com> References: <46E93A9E.5010009@terraminds.com> <20070914105327.GF7987@(none)> <46F94676.8050101@terraminds.com> <46FBB27D.7090202@terraminds.com> <466D394A-FD55-42F3-9253-D8E48C58D05F@csail.mit.edu> <46FBD046.3000701@terraminds.com> Message-ID: On Sep 27, 2007, at 11:46 AM, Chris Laux wrote: > Thanks Ian, that is some good analysis. > >> - If you are doing a significant amount of deserialization with >> lots of >> threads than you should know that each deserialization requires a >> call >> to (with-lock ...) to ensure that the shared pool of buffer >> streams is >> thread safe (a problem with elephant < 0.9). This could conceivably >> cause a lockup if there are lots of small deserializations happening >> concurrently across threads mapping over the same Btrees. > > I had a vague suspicion of something like that, but only looked at > transactions. I guess I would have to modify elephant to allow me > to do > the locking to solve such a problem. By lockup I really meant bottleneck rather than deadlock. Elephant really should be thread-safe now but it's always possible there is some weird case we haven't seen yet. >> Are you sure >> it's GC that's eating all the time, or non-lisp CPU time in general? > > Well, the 99% CPU is reported for the sbcl process. I only know that > manually invoking a gc will trigger the problem. > >> Although it breaks the abstraction barrier, using IDs will be a >> definite >> gain. You'd just make that second BTree pairs of word-freq / obj- >> oid. >> Then you use the OID and object type to grab the object directly from >> elephant: (elephant::get-cached-instance oid classname) > > I have also been considering doing away with the second layer of > BTrees, > and using my own, more "linear" structures. Not sure what that could > look like exactly though. Updates are the real problem and you'd have to load the entire 2nd level data structure to do any processing on it. >> You might be better off, performance >> wise, doing this in a C full-text indexing system and wrapping an >> interface to it. > > I hadn't thought of that yet. Can you recommend any? > > Anyway, I guess I was asking for trouble a bit with my setup. I'm not > sure how I'll proceed yet, but if I stick to the two-level BTree setup > and use id's I know what to look out for. I'd suggest you try this and see if it helps if the overhead isn't too insane. Ian > Thanks again, > > Chris > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From midfield at gmail.com Thu Sep 27 15:59:58 2007 From: midfield at gmail.com (Ben) Date: Thu, 27 Sep 2007 08:59:58 -0700 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <46FBD046.3000701@terraminds.com> References: <46E93A9E.5010009@terraminds.com> <46F94676.8050101@terraminds.com> <46FBB27D.7090202@terraminds.com> <466D394A-FD55-42F3-9253-D8E48C58D05F@csail.mit.edu> <46FBD046.3000701@terraminds.com> Message-ID: <9157df230709270859r365faa07k7ae9c8dea6a99bcc@mail.gmail.com> > > You might be better off, performance > > wise, doing this in a C full-text indexing system and wrapping an > > interface to it. > > I hadn't thought of that yet. Can you recommend any? lucene is good, solar is a web service packaging of it, and there is a port for common lisp (montezuma), ruby (ferret) and perl (kinosearch). B From eslick at csail.mit.edu Thu Sep 27 16:51:21 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 27 Sep 2007 12:51:21 -0400 Subject: [elephant-devel] Garbage collection problem In-Reply-To: <9157df230709270859r365faa07k7ae9c8dea6a99bcc@mail.gmail.com> References: <46E93A9E.5010009@terraminds.com> <46F94676.8050101@terraminds.com> <46FBB27D.7090202@terraminds.com> <466D394A-FD55-42F3-9253-D8E48C58D05F@csail.mit.edu> <46FBD046.3000701@terraminds.com> <9157df230709270859r365faa07k7ae9c8dea6a99bcc@mail.gmail.com> Message-ID: <3770B4AA-0B21-4612-BFF8-F7D936C5BF9F@csail.mit.edu> On Sep 27, 2007, at 11:59 AM, Ben wrote: >>> You might be better off, performance >>> wise, doing this in a C full-text indexing system and wrapping an >>> interface to it. >> >> I hadn't thought of that yet. Can you recommend any? Elephant's use of BDB is designed for certain kind of operations and access patterns. Implementing a full text index means you are adding a data structure abstraction on top of another one that wasn't designed for it. The resulting awkward interaction can have significant performance penalties. A low level data structure explicitly designed for large-scale, fast read access inverse indexing can outperform an 'interpreted' one significantly. > lucene is good, solar is a web service packaging of it, and there is a > port for common lisp (montezuma), ruby (ferret) and perl (kinosearch). > > B > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel