From mb at bese.it Mon Oct 2 11:35:21 2006 From: mb at bese.it (Marco Baringer) Date: Mon, 02 Oct 2006 13:35:21 +0200 Subject: [elephant-devel] elephant64 Message-ID: i'm currently trying to get elephant to run on sbcl x86-64. i've patched memutil and sleepycat.lisp heavily to remove the assumption that an int is equivalent to a (signed 32), most of db.h is defined in terms of uint32_t anyway so this wasn't a big problem. i'm currently able to open a db store and most of the serializer tests pass (except those related to btree objects). 1) now i've got some problems related to the berkeleydb backend (i think) which i can't seem to figure out :( anything btree related fails with the cryptic error: arithmetic error FLOATING-POINT-OVERFLOW signalled this even happens if i just open, close and then attempt to reopen the test db. anybody seen this before? 2) in sleepycat.lisp the c function db_get_raw is defined twice, once as %db-get-key-buffered and once as %db-get-buffered. is this intentional? p.s. - i'm running elephant 0.6.0 since that's what i was using before and i know it works. i'll upgrade to cvs if neccessary but i'd like to avoid dealing wth api changes (assuming there have been some). p.p.s - elephant is a sweet piece of software :) -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen From eslick at csail.mit.edu Mon Oct 2 12:40:05 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 02 Oct 2006 08:40:05 -0400 Subject: [elephant-devel] elephant64 In-Reply-To: References: Message-ID: <452108A5.8060404@csail.mit.edu> Can you get us temporary access to your 64-bit machine to do some experimentation? We had another user go through this same problem on an AMD awhile back and it's hard for Robert and I to test because neither of us have a 64-bit machine available. #2 - The double definition is intentional - it's two different lisp ways to call the same C function, one is general for any generic value and one uses c-strings only. It avoids the general overhead of buffer-streams. I assume you're using BDB-4.3? #1 - I haven't seen the fp overflow problem before, however. A couple of thoughts. The system does read some values from the DB after open. You might trace the db-bdb/bdb-controller function to see where it's failing - that might help us have a better clue as to the source. Is it failing opening the environment, a particular table, or in reading those values? It's also possible that the serializer has a 32-bit assumption hidden in it (although most of that should be in memutil). I've got a serializer rewrite that fixes anything like that, but it won't be ready to commit for awhile. Ian Marco Baringer wrote: > i'm currently trying to get elephant to run on sbcl x86-64. i've > patched memutil and sleepycat.lisp heavily to remove the assumption > that an int is equivalent to a (signed 32), most of db.h is defined in > terms of uint32_t anyway so this wasn't a big problem. i'm currently > able to open a db store and most of the serializer tests pass (except > those related to btree objects). > > 1) now i've got some problems related to the berkeleydb backend (i > think) which i can't seem to figure out :( anything btree related > fails with the cryptic error: > > arithmetic error FLOATING-POINT-OVERFLOW signalled > > this even happens if i just open, close and then attempt to reopen > the test db. anybody seen this before? > > 2) in sleepycat.lisp the c function db_get_raw is defined twice, once > as %db-get-key-buffered and once as %db-get-buffered. is this > intentional? > > p.s. - i'm running elephant 0.6.0 since that's what i was using before > and i know it works. i'll upgrade to cvs if neccessary but i'd like to > avoid dealing wth api changes (assuming there have been some). > > p.p.s - elephant is a sweet piece of software :) > From eslick at csail.mit.edu Mon Oct 2 12:42:01 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 02 Oct 2006 08:42:01 -0400 Subject: [elephant-devel] elephant64 In-Reply-To: References: Message-ID: <45210919.2030407@csail.mit.edu> FYI - The biggest change in the current CVS is a simple update to use BDB 4.4 instead of 4.3. I don't believe we have yet promoted any API changes yet. Ian Marco Baringer wrote: > i'm currently trying to get elephant to run on sbcl x86-64. i've > patched memutil and sleepycat.lisp heavily to remove the assumption > that an int is equivalent to a (signed 32), most of db.h is defined in > terms of uint32_t anyway so this wasn't a big problem. i'm currently > able to open a db store and most of the serializer tests pass (except > those related to btree objects). > > 1) now i've got some problems related to the berkeleydb backend (i > think) which i can't seem to figure out :( anything btree related > fails with the cryptic error: > > arithmetic error FLOATING-POINT-OVERFLOW signalled > > this even happens if i just open, close and then attempt to reopen > the test db. anybody seen this before? > > 2) in sleepycat.lisp the c function db_get_raw is defined twice, once > as %db-get-key-buffered and once as %db-get-buffered. is this > intentional? > > p.s. - i'm running elephant 0.6.0 since that's what i was using before > and i know it works. i'll upgrade to cvs if neccessary but i'd like to > avoid dealing wth api changes (assuming there have been some). > > p.p.s - elephant is a sweet piece of software :) > From mb at bese.it Mon Oct 2 15:15:48 2006 From: mb at bese.it (Marco Baringer) Date: Mon, 02 Oct 2006 17:15:48 +0200 Subject: [elephant-devel] elephant64 In-Reply-To: <452108A5.8060404@csail.mit.edu> (Ian Eslick's message of "Mon, 02 Oct 2006 08:40:05 -0400") References: <452108A5.8060404@csail.mit.edu> Message-ID: Ian Eslick writes: > #1 - I haven't seen the fp overflow problem before, however. > > A couple of thoughts. > > The system does read some values from the DB after open. You might > trace the db-bdb/bdb-controller function to see where it's failing - > that might help us have a better clue as to the source. Is it failing > opening the environment, a particular table, or in reading those values? apparently it's failing while calling ele::get-value. i tried some simple tests and this: CL-USER> (make-instance 'bdb-tree) # CL-USER> (setf (get-value 5 *) 5) hangs. attempting to interrupt it and examine the backtrace doesn't provide any useful info. i traced a bunch of functions in the get-value method and it appears to block when calling db-get-key-buffered. thanks, -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen From mb at bese.it Mon Oct 2 15:45:31 2006 From: mb at bese.it (Marco Baringer) Date: Mon, 02 Oct 2006 17:45:31 +0200 Subject: [elephant-devel] elephant64 In-Reply-To: <45212E44.8090109@csail.mit.edu> (Ian Eslick's message of "Mon, 02 Oct 2006 11:20:36 -0400") References: <452108A5.8060404@csail.mit.edu> <45212E44.8090109@csail.mit.edu> Message-ID: Ian Eslick writes: > I've sometimes noticed that there is an odd blockage and that running > "db_deadlock -a o -t 0.1" on the database resolves it. It's possible > it's a locking problem caused by the way that transactions are allocated > in the low level lisp code. Try opening the DB, running db_deadlock, > then rerun your get-value experiment. cool. that worked....the first time, then i got my standard floating point error again :( -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen From malinibhandaru at yahoo.com Tue Oct 3 20:34:38 2006 From: malinibhandaru at yahoo.com (Malini Bhandaru) Date: Tue, 3 Oct 2006 13:34:38 -0700 (PDT) Subject: [elephant-devel] Problems create sleepycat.fasl Message-ID: <20061003203438.91546.qmail@web52405.mail.yahoo.com> Hello All! Sorry, I could not find a FAQ or past mailings to see if anyone ran into the same issues. I am a new comer to sleepycat, elephant, and a rusty lisper. Environment: SBCL on linux BerkeleyDB 4.4 installed in /usr/local with a happy symbolic link to the same /usr/local/BerkeleyDB. I installed "elephant-0.6.0" in my home directory and modified 1) config.lisp to point to /usr/local/BerkeleyDB (as opposed to 4.3 version) 2) Makefile as above 3) made symbolic links to all elephant all .asd files t in my .sbcl/systems 4) grabbed cl-base64.asd and rt.asd from CLiki and put them in .sbcl/systems (testmigration.fasl, package.fasl are created successfully) Now, whether I attempt to run the tests or jump straight in .. there seems to be a problem with my sleepycat.fasl. This occurs when attempting to load "ele-bdb". If I attempt to continue in slime, the problem just bites later such as "sleepycat::db-env-create" is undefined or something else. Much thanks malini -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Tue Oct 3 21:05:03 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 03 Oct 2006 17:05:03 -0400 Subject: [elephant-devel] Problems create sleepycat.fasl In-Reply-To: <20061003203438.91546.qmail@web52405.mail.yahoo.com> References: <20061003203438.91546.qmail@web52405.mail.yahoo.com> Message-ID: <4522D07F.1020204@csail.mit.edu> The problem opening environments is caused by the fact that elephant-0.6.0 relies on Berkeley DB 4.3 and will not work with version 4.4. You can try using the CVS version which is 4.4 compatible, but there are likely to be some problems in CVS as checkins to it are not as rigorously tested. I'd recommend you install 4.3 and try again. Let us know if you have more questions! Ian Malini Bhandaru wrote: > Hello All! > > Sorry, I could not find a FAQ or past mailings to see if anyone ran > into the same issues. I am a new comer to sleepycat, elephant, and a > rusty lisper. > > Environment: > SBCL on linux > BerkeleyDB 4.4 installed in /usr/local with a happy symbolic link to > the same /usr/local/BerkeleyDB. > > I installed "elephant-0.6.0" in my home directory and modified > 1) config.lisp to point to /usr/local/BerkeleyDB (as opposed to 4.3 > version) > 2) Makefile as above > 3) made symbolic links to all elephant all .asd files t in my > .sbcl/systems > 4) grabbed cl-base64.asd and rt.asd from CLiki and put them in > .sbcl/systems > > (testmigration.fasl, package.fasl are created successfully) > Now, whether I attempt to run the tests or jump straight in .. there > seems to be a problem with my sleepycat.fasl. This occurs when > attempting to load "ele-bdb". If I attempt to continue in slime, the > problem just bites later such as "sleepycat::db-env-create" is > undefined or something else. > > > Much thanks > malini > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From malinibhandaru at yahoo.com Wed Oct 4 03:23:30 2006 From: malinibhandaru at yahoo.com (Malini Bhandaru) Date: Tue, 3 Oct 2006 20:23:30 -0700 (PDT) Subject: [elephant-devel] Problems create sleepycat.fasl In-Reply-To: <4522D07F.1020204@csail.mit.edu> Message-ID: <20061004032330.58937.qmail@web52412.mail.yahoo.com> Hello Ian and Others, I removed the /usr/local/BerkeleyDB.4.4 and installed 4.3. Untarred freshly elephant0.6.0, added symbolic links for .asd files in .sbcl/systems but my woes continue. straight from the elephant tutorial I am trying: (asdf:operate 'asdf:load-op :elephant) (use-package "ELE") (setf *testbdb-spec* '(:BDB "/home/ray/elephant/tests/testdb/")) --- fine till here --- (open-store *testbdb-spec*) my lisp seems to hang after ; compiling (DEFVAR *ERRNO-BUFFER* ...) ; compiling (DEFMACRO WRAP-ERRNO ...) ; compiling (DEFMACRO FLAGS ...) at which point 40948 2006-10-03 23:00 sleepycat.fasl and the other fasls are not constructed. Do I have to set anything with LD-LIBRARY-PATH .. or any other environment thing? or could i have a ghost of BerkeleyDB4.4 (is that possible after a rm -rf of the /usr/local/B.. directory .. not by my reading of the install actions). Thank you once again. malini Ian Eslick wrote: The problem opening environments is caused by the fact that elephant-0.6.0 relies on Berkeley DB 4.3 and will not work with version 4.4. You can try using the CVS version which is 4.4 compatible, but there are likely to be some problems in CVS as checkins to it are not as rigorously tested. I'd recommend you install 4.3 and try again. Let us know if you have more questions! Ian Malini Bhandaru wrote: > Hello All! > > Sorry, I could not find a FAQ or past mailings to see if anyone ran > into the same issues. I am a new comer to sleepycat, elephant, and a > rusty lisper. > > Environment: > SBCL on linux > BerkeleyDB 4.4 installed in /usr/local with a happy symbolic link to > the same /usr/local/BerkeleyDB. > > I installed "elephant-0.6.0" in my home directory and modified > 1) config.lisp to point to /usr/local/BerkeleyDB (as opposed to 4.3 > version) > 2) Makefile as above > 3) made symbolic links to all elephant all .asd files t in my > .sbcl/systems > 4) grabbed cl-base64.asd and rt.asd from CLiki and put them in > .sbcl/systems > > (testmigration.fasl, package.fasl are created successfully) > Now, whether I attempt to run the tests or jump straight in .. there > seems to be a problem with my sleepycat.fasl. This occurs when > attempting to load "ele-bdb". If I attempt to continue in slime, the > problem just bites later such as "sleepycat::db-env-create" is > undefined or something else. > > > Much thanks > malini > > > > > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From pinterface at gmail.com Wed Oct 4 06:44:57 2006 From: pinterface at gmail.com (pinterface) Date: Wed, 4 Oct 2006 06:44:57 -0000 Subject: [elephant-devel] Re: Problems create sleepycat.fasl Message-ID: <013b01c6e780$9c18d0e0$3100a8c0@trinity> (Sorry if this reply appears out of thread. Ran into some difficulties with gmane, so I'm posting to the list directly.) "Malini Bhandaru" wrote in message news:20061004032330.58937.qmail at web52412.mail.yahoo.com... > Hello Ian and Others, > > I removed the /usr/local/BerkeleyDB.4.4 and installed 4.3. > Untarred freshly elephant0.6.0, added symbolic links for .asd files > in .sbcl/systems but my woes continue. > > ... > > my lisp seems to hang after > ; compiling (DEFVAR *ERRNO-BUFFER* ...) > ; compiling (DEFMACRO WRAP-ERRNO ...) > ; compiling (DEFMACRO FLAGS ...) This reminds me of a problem I had[1]. Actually, I think it's the same problem, because if memory serves compiles hung at the same point for me. If you're using emacs+slime, check the *inferior-lisp* buffer; if you've been thrown into SBCL's LDB debugger, and I suspect you have, that's where it'll be. You wouldn't happen to be using a threaded SBCL (:sb-thread in *features*), would you? sb-threads and libpthread.so don't play well together for me, and I'm guessing they aren't playing nicely for you, either. I think Ian was going to look into the cause of that, but I don't know if he's had a chance yet. The two options I can think of are: 1. Switch to an unthreaded SBCL 2. Disable loading of libpthreads in src/db-bdb/sleepycat.lisp Both have worked for me (cvs elephant and BDB4.4), but I have yet to do anything particularly taxing with elephant, so maybe it'll blow up later. -pinterface [1] http://common-lisp.net/pipermail/elephant-devel/2006-September/000527.html Though for some reason I provided almost no useful information to hunting down the cause. Whoops. From malinibhandaru at yahoo.com Wed Oct 4 15:33:29 2006 From: malinibhandaru at yahoo.com (Malini Bhandaru) Date: Wed, 4 Oct 2006 08:33:29 -0700 (PDT) Subject: [elephant-devel] Re: Problems create sleepycat.fasl In-Reply-To: <013b01c6e780$9c18d0e0$3100a8c0@trinity> Message-ID: <20061004153329.37526.qmail@web52412.mail.yahoo.com> pinterface Thou are correct! It had to do with threads and the *inferior-lisp* message was just as you predicted. I have to dig some more .. but in my environment sbcl is threaded and elephant::*sleepycat-pthreads-path* points to /lib/tls/libpthread.so .. when I commented out that section of code in sleepycat.lisp .. everything compiled till the end and I could put a value in the db and retrieve. I'll investigate some more and see how to work around this conflict. Thank you so much! regards, malini pinterface wrote: (Sorry if this reply appears out of thread. Ran into some difficulties with gmane, so I'm posting to the list directly.) "Malini Bhandaru" wrote in message news:20061004032330.58937.qmail at web52412.mail.yahoo.com... > Hello Ian and Others, > > I removed the /usr/local/BerkeleyDB.4.4 and installed 4.3. > Untarred freshly elephant0.6.0, added symbolic links for .asd files > in .sbcl/systems but my woes continue. > > ... > > my lisp seems to hang after > ; compiling (DEFVAR *ERRNO-BUFFER* ...) > ; compiling (DEFMACRO WRAP-ERRNO ...) > ; compiling (DEFMACRO FLAGS ...) This reminds me of a problem I had[1]. Actually, I think it's the same problem, because if memory serves compiles hung at the same point for me. If you're using emacs+slime, check the *inferior-lisp* buffer; if you've been thrown into SBCL's LDB debugger, and I suspect you have, that's where it'll be. You wouldn't happen to be using a threaded SBCL (:sb-thread in *features*), would you? sb-threads and libpthread.so don't play well together for me, and I'm guessing they aren't playing nicely for you, either. I think Ian was going to look into the cause of that, but I don't know if he's had a chance yet. The two options I can think of are: 1. Switch to an unthreaded SBCL 2. Disable loading of libpthreads in src/db-bdb/sleepycat.lisp Both have worked for me (cvs elephant and BDB4.4), but I have yet to do anything particularly taxing with elephant, so maybe it'll blow up later. -pinterface [1] http://common-lisp.net/pipermail/elephant-devel/2006-September/000527.html Though for some reason I provided almost no useful information to hunting down the cause. Whoops. _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From reddaly at gmail.com Thu Oct 5 08:23:12 2006 From: reddaly at gmail.com (Red Daly) Date: Thu, 05 Oct 2006 01:23:12 -0700 Subject: [elephant-devel] trouble getting going (with open-store, ...) Message-ID: <1160036592.9944.18.camel@localhost> I have been trying to get Elephant up and running for a few days with little luck. I have tried CMUCL and SBCL, with different unsuccessful outcomes. During my latest attempt on SBCL, I seem to be able to call the following two lines without errors: (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-bdb) I then run the tests... (asdf:operate 'asdf:load-op :elephant-tests) (in-package :ele-tests) (setf *default-spec* "bdb") (do-backend-tests) do-backend-tests returns NIL.. which I interpret to be success? however, I am unable to open a store according to the instructions in the two tutorials on the common-lisp.net project site. Here is an example attempt: * (open-store "store") debugger invoked on a SIMPLE-ERROR in thread #: The assertion (CONSP ELEPHANT::SPEC) failed. Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. restarts (invokable by number or by possibly-abbreviated name): 0: [CONTINUE] Retry assertion. 1: [ABORT ] Exit debugger, returning to top level. (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) 0] backtrace 0: (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) 1: (OPEN-STORE "store" :RECOVER NIL :RECOVER-FATAL NIL :THREAD T) 2: (SB-INT:EVAL-IN-LEXENV (OPEN-STORE "store") #) 3: (SB-EXT:INTERACTIVE-EVAL (OPEN-STORE "store")) 4: (SB-IMPL::REPL-FUN NIL) 5: ((LAMBDA ())) 6: ((LAMBDA ())) 7: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) 8: (SB-IMPL::TOPLEVEL-REPL NIL) 9: (SB-IMPL::TOPLEVEL-INIT) 10: ((LABELS SB-IMPL::RESTART-LISP)) I would appreciate any help in getting elephant up and running. I am running SBCL 0.9.8, ubuntu dapper, and bdb 4.3 (apt-installable version). I will gladly supply more information to help determine the error of my ways. I can't wait to try out elephant! Thanks sincerely, red daly From eslick at csail.mit.edu Thu Oct 5 13:01:09 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 05 Oct 2006 09:01:09 -0400 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <1160036592.9944.18.camel@localhost> References: <1160036592.9944.18.camel@localhost> Message-ID: <45250215.7020809@csail.mit.edu> I'm sorry if the website hasn't been updated (Robert, can you check this?), but the tutorial should be updated to reflect the new (as of 0.6.0) open-store protocol, which requires specification of the type of backend you'll be using. A specification is a list of two elements; a backend type specification [:bdb, :clsql] and backend-specific store information: (open-store '(:bdb "/home/user/db/testdb")) (open-store '(:clsql (:sqlite3 "sqlite3-test.db")) (open-store '(:clsql (:postgresql "localhost.localdomain" "test" "postgres" ""))) The tests should print a long list of test names, then return T. To run the tests you need to follow the comments in elephant-tests.lisp: ;; ;; GUIDE TO TESTING ;; ;; 1) Set *default-spec* to the above spec of your choice ;; 2) Call (do-backend-tests) to test the standard API ;; 3) To test migration: (do-migration-tests *default-spec* ) inserting a second ;; spec, typically a bdb spec or create another instance of a sql db depending on ;; your configuration ;; 4) A backend is green if it passes do-backend-tests and can succesfully be ;; used as spec1 or spec2 argument in the migration test ;; To set *default-spec* you need to do something like: (setq *default-spec* *testbdb-spec*) or (setq *default-spec* *testsqlite3-spec*) Can you list the various problems you had trying to get the system to run? We're working on useability for the next release and want to fix as many of the inconveniences as possible. Thank you, Ian Red Daly wrote: > I have been trying to get Elephant up and running for a few days with > little luck. I have tried CMUCL and SBCL, with different unsuccessful > outcomes. During my latest attempt on SBCL, I seem to be able to call > the following two lines without errors: > (asdf:operate 'asdf:load-op :elephant) > (asdf:operate 'asdf:load-op :ele-bdb) > > I then run the tests... > (asdf:operate 'asdf:load-op :elephant-tests) > (in-package :ele-tests) > (setf *default-spec* "bdb") > (do-backend-tests) > > do-backend-tests returns NIL.. which I interpret to be success? > > > however, I am unable to open a store according to the instructions in > the two tutorials on the common-lisp.net project site. Here is an > example attempt: > > * (open-store "store") > > debugger invoked on a SIMPLE-ERROR in thread > #: > The assertion (CONSP ELEPHANT::SPEC) failed. > > Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. > > restarts (invokable by number or by possibly-abbreviated name): > 0: [CONTINUE] Retry assertion. > 1: [ABORT ] Exit debugger, returning to top level. > > (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > 0] backtrace > > 0: (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > 1: (OPEN-STORE "store" :RECOVER NIL :RECOVER-FATAL NIL :THREAD T) > 2: (SB-INT:EVAL-IN-LEXENV (OPEN-STORE "store") #) > 3: (SB-EXT:INTERACTIVE-EVAL (OPEN-STORE "store")) > 4: (SB-IMPL::REPL-FUN NIL) > 5: ((LAMBDA ())) > 6: ((LAMBDA ())) > 7: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) > 8: (SB-IMPL::TOPLEVEL-REPL NIL) > 9: (SB-IMPL::TOPLEVEL-INIT) > 10: ((LABELS SB-IMPL::RESTART-LISP)) > > > I would appreciate any help in getting elephant up and running. I am > running SBCL 0.9.8, ubuntu dapper, and bdb 4.3 (apt-installable > version). I will gladly supply more information to help determine the > error of my ways. I can't wait to try out elephant! > > Thanks sincerely, > red daly > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From ml13 at onlinehome.de Thu Oct 5 14:03:26 2006 From: ml13 at onlinehome.de (Kilian Sprotte) Date: Thu, 5 Oct 2006 16:03:26 +0200 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <45250215.7020809@csail.mit.edu> References: <1160036592.9944.18.camel@localhost> <45250215.7020809@csail.mit.edu> Message-ID: <1B109030-433B-4AAB-8783-B167C22F73BE@onlinehome.de> Am 05.10.2006 um 15:01 schrieb Ian Eslick: > Can you list the various problems you had trying to get the system to > run? We're working on useability for the next release and want to fix > as many of the inconveniences as possible. I can help with testing on OSX sbcl and openmcl. At first, I would like to concentrate on the bdb backend. I haven't actually been using elephant for a while, so I will start with a fresh installation and tell you what happens. I will take the cvs version and db44 is that correct? (sbcl and openmcl 32bit on osx 10.4 db44 installed via fink) I will also tell you exactly what I need to comment/uncomment in the makefile or somewhere else - it would be wonderful, if this wasn't necessary at all, just taking the tarball and it works, even on a mac - if this cannot be reduced to zero, maybe it would be a good idea to put up some comments on the website categorized by the various platforms. Isn't there an attempt underway to get rid of the makefile entirely? (sorry, haven't really been following recent developments, as I should...) Greetings, Kilian From eslick at csail.mit.edu Thu Oct 5 14:21:42 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 05 Oct 2006 10:21:42 -0400 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <1B109030-433B-4AAB-8783-B167C22F73BE@onlinehome.de> References: <1160036592.9944.18.camel@localhost> <45250215.7020809@csail.mit.edu> <1B109030-433B-4AAB-8783-B167C22F73BE@onlinehome.de> Message-ID: <452514F6.8000706@csail.mit.edu> There are some patches pending for CVS which should remove the makefile step. I'm open to suggestions on the right way to provide parameters to the build process for the C libraries and the load process for Lisp (where are the foreign BDB libs and local libs). We've tried to make reasonable defaults for the various platforms, but if your BDB or SQL libraries are not in the usual place then customization is needed. There may be a few problems with CVS (although I use it daily) that haven't been tracked down yet so if you run into non-configuration related problems, please revert to using 0.6.0 and BDB 4.3. Thanks for the help! The next two weeks are busy for me, and then I hope to get back to hacking properly on Elephant. Ian Kilian Sprotte wrote: > Am 05.10.2006 um 15:01 schrieb Ian Eslick: > >> Can you list the various problems you had trying to get the system to >> run? We're working on useability for the next release and want to fix >> as many of the inconveniences as possible. > > I can help with testing on OSX sbcl and openmcl. At first, I would > like to concentrate on the bdb backend. I haven't actually been using > elephant for a while, so I will start with a fresh installation and > tell you what happens. > > I will take the cvs version and db44 is that correct? > (sbcl and openmcl 32bit on osx 10.4 db44 installed via fink) > > I will also tell you exactly what I need to comment/uncomment in the > makefile or somewhere else - it would be wonderful, if this wasn't > necessary at all, just taking the tarball and it works, even on a mac > - if this cannot be reduced to zero, maybe it would be a good idea to > put up some comments on the website categorized by the various platforms. > > Isn't there an attempt underway to get rid of the makefile entirely? > (sorry, haven't really been following recent developments, as I > should...) > > Greetings, > > Kilian > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From ml13 at onlinehome.de Thu Oct 5 14:49:41 2006 From: ml13 at onlinehome.de (Kilian Sprotte) Date: Thu, 5 Oct 2006 16:49:41 +0200 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <452514F6.8000706@csail.mit.edu> References: <1160036592.9944.18.camel@localhost> <45250215.7020809@csail.mit.edu> <1B109030-433B-4AAB-8783-B167C22F73BE@onlinehome.de> <452514F6.8000706@csail.mit.edu> Message-ID: Am 05.10.2006 um 16:21 schrieb Ian Eslick: > Thanks for the help! The next two weeks are busy for me, and then I > hope to get back to hacking properly on Elephant. Thats good, so I can prepare this slowly.... :) Basically, I have good news: With some minor inconveniences, I got sbcl running and all the *testbdb-spec* tests passed in the first run. Now (after restarting), I seem to be stuck on prepares-sleepycat, test-seq1, test-seq2, cleansup-sleepycat. But Ill repeat this being more careful. For openmcl, there will be a little more needed, but I think, there will be quick solutions. Lets get together in 2 weeks, then! Cheers, Kilian From read at robertlread.net Thu Oct 5 15:13:02 2006 From: read at robertlread.net (Robert L. Read) Date: Thu, 05 Oct 2006 10:13:02 -0500 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <1160036592.9944.18.camel@localhost> References: <1160036592.9944.18.camel@localhost> Message-ID: <1160061183.2473.672.camel@localhost.localdomain> On Thu, 2006-10-05 at 01:23 -0700, Red Daly wrote: > > > however, I am unable to open a store according to the instructions in > the two tutorials on the common-lisp.net project site. Here is an > example attempt: > > * (open-store "store") Dear Team, This syntax is (very) obsolete, as Ian has pointed out. I don't think this example occurs in the current tutorials for elephant itself (here: http://common- lisp.net/project/elephant/doc/), but does occur in the "with-araneida" tutorial: http://common- lisp.net/project/elephant/with-araneida/ That tutorial is about two years out of date; I think it is still valuable for showing an example use of Elephant for a small but meaningful application. It is not really within my power in the short term to repair it (although Ian and I have discussed creating a contemporary version of something similar, and we would welcome a good "example app" from this list.) However, it is terrible that it leaves unmarked examples that are out of date and wasted your time. I'm sorry about that and will go clearly indicate the obsolete nature of that tutorial. (If you find a bad example in the elephant tutorial, please inform me --- I ran all of those examples by hand before the 0.6.0 release, but I have an extraordinary ability to step in a bucket.) > > debugger invoked on a SIMPLE-ERROR in thread > #: > The assertion (CONSP ELEPHANT::SPEC) failed. > > Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. > > restarts (invokable by number or by possibly-abbreviated name): > 0: [CONTINUE] Retry assertion. > 1: [ABORT ] Exit debugger, returning to top level. > > (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > 0] backtrace > > 0: (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > 1: (OPEN-STORE "store" :RECOVER NIL :RECOVER-FATAL NIL :THREAD T) > 2: (SB-INT:EVAL-IN-LEXENV (OPEN-STORE "store") #) > 3: (SB-EXT:INTERACTIVE-EVAL (OPEN-STORE "store")) > 4: (SB-IMPL::REPL-FUN NIL) > 5: ((LAMBDA ())) > 6: ((LAMBDA ())) > 7: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) > 8: (SB-IMPL::TOPLEVEL-REPL NIL) > 9: (SB-IMPL::TOPLEVEL-INIT) > 10: ((LABELS SB-IMPL::RESTART-LISP)) > > > I would appreciate any help in getting elephant up and running. I am > running SBCL 0.9.8, ubuntu dapper, and bdb 4.3 (apt-installable > version). I will gladly supply more information to help determine the > error of my ways. I can't wait to try out elephant! > > Thanks sincerely, > red daly > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ml13 at onlinehome.de Thu Oct 5 15:24:09 2006 From: ml13 at onlinehome.de (Kilian Sprotte) Date: Thu, 5 Oct 2006 17:24:09 +0200 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <1160061183.2473.672.camel@localhost.localdomain> References: <1160036592.9944.18.camel@localhost> <1160061183.2473.672.camel@localhost.localdomain> Message-ID: Hi, why don't you just take off or correct the Example Usage part of the project homepage? Its probably just this line CL-USER> (open-store "/home/ben/testdb"). People might be tempted to look there first then into the tutorial. Bye, Kilian Am 05.10.2006 um 17:13 schrieb Robert L. Read: > On Thu, 2006-10-05 at 01:23 -0700, Red Daly wrote: >> >> however, I am unable to open a store according to the instructions in >> the two tutorials on the common-lisp.net project site. Here is an >> example attempt: >> >> * (open-store "store") > Dear Team, > This syntax is (very) obsolete, as Ian has pointed out. I > don't think this example occurs in > the current tutorials for elephant itself (here: http://common- > lisp.net/project/elephant/doc/), but > does occur in the "with-araneida" tutorial: http://common-lisp.net/ > project/elephant/with-araneida/ > That tutorial is about two years out of date; I think it is still > valuable for showing an example > use of Elephant for a small but meaningful application. It is not > really within my power in the > short term to repair it (although Ian and I have discussed creating > a contemporary version of > something similar, and we would welcome a good "example app" from > this list.) > However, it is terrible that it leaves unmarked examples that > are out of date and wasted > your time. I'm sorry about that and will go clearly indicate the > obsolete nature of that tutorial. > (If you find a bad example in the elephant tutorial, please inform > me --- I ran all of those examples > by hand before the 0.6.0 release, but I have an extraordinary > ability to step in a bucket.) > > >> debugger invoked on a SIMPLE-ERROR in thread >> #: >> The assertion (CONSP ELEPHANT::SPEC) failed. >> >> Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. >> >> restarts (invokable by number or by possibly-abbreviated name): >> 0: [CONTINUE] Retry assertion. >> 1: [ABORT ] Exit debugger, returning to top level. >> >> (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) >> 0] backtrace >> >> 0: (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) >> 1: (OPEN-STORE "store" :RECOVER NIL :RECOVER-FATAL NIL :THREAD T) >> 2: (SB-INT:EVAL-IN-LEXENV (OPEN-STORE "store") #) >> 3: (SB-EXT:INTERACTIVE-EVAL (OPEN-STORE "store")) >> 4: (SB-IMPL::REPL-FUN NIL) >> 5: ((LAMBDA ())) >> 6: ((LAMBDA ())) >> 7: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) >> 8: (SB-IMPL::TOPLEVEL-REPL NIL) >> 9: (SB-IMPL::TOPLEVEL-INIT) >> 10: ((LABELS SB-IMPL::RESTART-LISP)) >> >> >> I would appreciate any help in getting elephant up and running. I am >> running SBCL 0.9.8, ubuntu dapper, and bdb 4.3 (apt-installable >> version). I will gladly supply more information to help determine >> the >> error of my ways. I can't wait to try out elephant! >> >> Thanks sincerely, >> red daly >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Thu Oct 5 15:34:08 2006 From: read at robertlread.net (Robert L. Read) Date: Thu, 05 Oct 2006 10:34:08 -0500 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: References: <1160036592.9944.18.camel@localhost> <1160061183.2473.672.camel@localhost.localdomain> Message-ID: <1160062449.2473.682.camel@localhost.localdomain> Arggh! I realized that write before I got this email. I have now removed it; we should use the texinfo documentation as the master documentation. I'll fix it, thanks! On Thu, 2006-10-05 at 17:24 +0200, Kilian Sprotte wrote: > Hi, > > > why don't you just take off or correct the Example Usage part of the > project homepage? > > > Its probably just this line CL-USER> (open-store "/home/ben/testdb"). > > > People might be tempted to look there first then into the tutorial. > > > Bye, > > > Kilian > > > Am 05.10.2006 um 17:13 schrieb Robert L. Read: > > > > On Thu, 2006-10-05 at 01:23 -0700, Red Daly wrote: > > > > > however, I am unable to open a store according to the instructions in > > > the two tutorials on the common-lisp.net project site. Here is an > > > example attempt: > > > > > > * (open-store "store") > > > > Dear Team, > > This syntax is (very) obsolete, as Ian has pointed out. I don't > > think this example occurs in > > the current tutorials for elephant itself (here: http://common- > > lisp.net/project/elephant/doc/), but > > does occur in the "with-araneida" tutorial: http://common- > > lisp.net/project/elephant/with-araneida/ > > That tutorial is about two years out of date; I think it is still > > valuable for showing an example > > use of Elephant for a small but meaningful application. It is not > > really within my power in the > > short term to repair it (although Ian and I have discussed creating > > a contemporary version of > > something similar, and we would welcome a good "example app" from > > this list.) > > However, it is terrible that it leaves unmarked examples that > > are out of date and wasted > > your time. I'm sorry about that and will go clearly indicate the > > obsolete nature of that tutorial. > > (If you find a bad example in the elephant tutorial, please inform > > me --- I ran all of those examples > > by hand before the 0.6.0 release, but I have an extraordinary > > ability to step in a bucket.) > > > > > > > > > debugger invoked on a SIMPLE-ERROR in thread > > > #: > > > The assertion (CONSP ELEPHANT::SPEC) failed. > > > > > > Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. > > > > > > restarts (invokable by number or by possibly-abbreviated name): > > > 0: [CONTINUE] Retry assertion. > > > 1: [ABORT ] Exit debugger, returning to top level. > > > > > > (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > > > 0] backtrace > > > > > > 0: (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > > > 1: (OPEN-STORE "store" :RECOVER NIL :RECOVER-FATAL NIL :THREAD T) > > > 2: (SB-INT:EVAL-IN-LEXENV (OPEN-STORE "store") #) > > > 3: (SB-EXT:INTERACTIVE-EVAL (OPEN-STORE "store")) > > > 4: (SB-IMPL::REPL-FUN NIL) > > > 5: ((LAMBDA ())) > > > 6: ((LAMBDA ())) > > > 7: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) > > > 8: (SB-IMPL::TOPLEVEL-REPL NIL) > > > 9: (SB-IMPL::TOPLEVEL-INIT) > > > 10: ((LABELS SB-IMPL::RESTART-LISP)) > > > > > > > > > I would appreciate any help in getting elephant up and running. I am > > > running SBCL 0.9.8, ubuntu dapper, and bdb 4.3 (apt-installable > > > version). I will gladly supply more information to help determine the > > > error of my ways. I can't wait to try out elephant! > > > > > > Thanks sincerely, > > > red daly > > > > > > _______________________________________________ > > > elephant-devel site list > > > elephant-devel at common-lisp.net > > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From reddaly at gmail.com Thu Oct 5 19:17:06 2006 From: reddaly at gmail.com (Red Daly) Date: Thu, 05 Oct 2006 12:17:06 -0700 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <45250215.7020809@csail.mit.edu> References: <1160036592.9944.18.camel@localhost> <45250215.7020809@csail.mit.edu> Message-ID: <1160075826.9944.42.camel@localhost> Thank you for the quick and helpful response :). With your corrections, all tests passed and I seem to be able to open a functioning store! However, life is not perfect yet. The main problem I seem to be facing right now is when I attempt to load elephant using my slime environment (when calling (asdf:operate 'asdf:load-op :ele-bdb)). For some reason, I lose my lisp connection. A sample transcript of a common slime session can be found here: http://paste.lisp.org/display/27406 This is the point of failure of a similar session: (let ((store-spec '(:BDB "store"))) (ele:open-store store-spec)) ; loading system definition from ; /home/red/code/edufacebook/elephant-0.6.0/ele-bdb.asd into ; # ; registering # as ELE-BDB --- Lisp connection closed unexpectedly: exited abnormally with code 256 This is perhaps a slime/sbcl/other bug, but so far I haven't seen anything like it, and since slime is so popular (and useful) perhaps others have seen this bug or know what to do about it. In the raw sbcl REPL I do not seem to have a problem, but only when I attempt to load ele-bdb. If you would like, I can post the problems I had with CMUCL when I have more time. (off the top of my head, I remember package locking problems and also a problem loading cl-base64). at the moment I am more concerned with getting a working system in SBCL with Slime, but I would like to help make elephant better however I can. -red daly On Thu, 2006-10-05 at 09:01 -0400, Ian Eslick wrote: > I'm sorry if the website hasn't been updated (Robert, can you check > this?), but the tutorial should be updated to reflect the new (as of > 0.6.0) open-store protocol, which requires specification of the type of > backend you'll be using. A specification is a list of two elements; a > backend type specification [:bdb, :clsql] and backend-specific store > information: > > (open-store '(:bdb "/home/user/db/testdb")) > (open-store '(:clsql (:sqlite3 "sqlite3-test.db")) > (open-store '(:clsql (:postgresql "localhost.localdomain" "test" > "postgres" ""))) > > The tests should print a long list of test names, then return T. > To run the tests you need to follow the comments in elephant-tests.lisp: > > ;; > ;; GUIDE TO TESTING > ;; > ;; 1) Set *default-spec* to the above spec of your choice > ;; 2) Call (do-backend-tests) to test the standard API > ;; 3) To test migration: (do-migration-tests *default-spec* > ) inserting a second > ;; spec, typically a bdb spec or create another instance of a sql db > depending on > ;; your configuration > ;; 4) A backend is green if it passes do-backend-tests and can > succesfully be > ;; used as spec1 or spec2 argument in the migration test > ;; > > To set *default-spec* you need to do something like: > > (setq *default-spec* *testbdb-spec*) > > or > > (setq *default-spec* *testsqlite3-spec*) > > Can you list the various problems you had trying to get the system to > run? We're working on useability for the next release and want to fix > as many of the inconveniences as possible. > > Thank you, > Ian > > Red Daly wrote: > > I have been trying to get Elephant up and running for a few days with > > little luck. I have tried CMUCL and SBCL, with different unsuccessful > > outcomes. During my latest attempt on SBCL, I seem to be able to call > > the following two lines without errors: > > (asdf:operate 'asdf:load-op :elephant) > > (asdf:operate 'asdf:load-op :ele-bdb) > > > > I then run the tests... > > (asdf:operate 'asdf:load-op :elephant-tests) > > (in-package :ele-tests) > > (setf *default-spec* "bdb") > > (do-backend-tests) > > > > do-backend-tests returns NIL.. which I interpret to be success? > > > > > > however, I am unable to open a store according to the instructions in > > the two tutorials on the common-lisp.net project site. Here is an > > example attempt: > > > > * (open-store "store") > > > > debugger invoked on a SIMPLE-ERROR in thread > > #: > > The assertion (CONSP ELEPHANT::SPEC) failed. > > > > Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. > > > > restarts (invokable by number or by possibly-abbreviated name): > > 0: [CONTINUE] Retry assertion. > > 1: [ABORT ] Exit debugger, returning to top level. > > > > (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > > 0] backtrace > > > > 0: (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > > 1: (OPEN-STORE "store" :RECOVER NIL :RECOVER-FATAL NIL :THREAD T) > > 2: (SB-INT:EVAL-IN-LEXENV (OPEN-STORE "store") #) > > 3: (SB-EXT:INTERACTIVE-EVAL (OPEN-STORE "store")) > > 4: (SB-IMPL::REPL-FUN NIL) > > 5: ((LAMBDA ())) > > 6: ((LAMBDA ())) > > 7: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) > > 8: (SB-IMPL::TOPLEVEL-REPL NIL) > > 9: (SB-IMPL::TOPLEVEL-INIT) > > 10: ((LABELS SB-IMPL::RESTART-LISP)) > > > > > > I would appreciate any help in getting elephant up and running. I am > > running SBCL 0.9.8, ubuntu dapper, and bdb 4.3 (apt-installable > > version). I will gladly supply more information to help determine the > > error of my ways. I can't wait to try out elephant! > > > > Thanks sincerely, > > red daly > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > From eslick at csail.mit.edu Thu Oct 5 22:39:22 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 05 Oct 2006 18:39:22 -0400 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <1160075826.9944.42.camel@localhost> References: <1160036592.9944.18.camel@localhost> <45250215.7020809@csail.mit.edu> <1160075826.9944.42.camel@localhost> Message-ID: <4525899A.9070801@csail.mit.edu> Could be some odd case in asdf. When you open a store, the appropriate asdf file is called to load the backend code if it isn't loaded. Try loading the :elephant system instead of :ele-bdb, then doing open-store and see what happens. I'll test this on SBCL later tonight. Ian Red Daly wrote: > Thank you for the quick and helpful response :). > > With your corrections, all tests passed and I seem to be able to open a > functioning store! However, life is not perfect yet. The main problem > I seem to be facing right now is when I attempt to load elephant using > my slime environment (when calling (asdf:operate > 'asdf:load-op :ele-bdb)). For some reason, I lose my lisp connection. > A sample transcript of a common slime session can be found here: > http://paste.lisp.org/display/27406 > > This is the point of failure of a similar session: > (let ((store-spec '(:BDB "store"))) > (ele:open-store store-spec)) > ; loading system definition from > ; /home/red/code/edufacebook/elephant-0.6.0/ele-bdb.asd into > ; # > ; registering # as ELE-BDB > --- > Lisp connection closed unexpectedly: exited abnormally with code 256 > > This is perhaps a slime/sbcl/other bug, but so far I haven't seen > anything like it, and since slime is so popular (and useful) perhaps > others have seen this bug or know what to do about it. > > In the raw sbcl REPL I do not seem to have a problem, but only when I > attempt to load ele-bdb. > > If you would like, I can post the problems I had with CMUCL when I have > more time. (off the top of my head, I remember package locking problems > and also a problem loading cl-base64). at the moment I am more > concerned with getting a working system in SBCL with Slime, but I would > like to help make elephant better however I can. > > -red daly > > > On Thu, 2006-10-05 at 09:01 -0400, Ian Eslick wrote: > >> I'm sorry if the website hasn't been updated (Robert, can you check >> this?), but the tutorial should be updated to reflect the new (as of >> 0.6.0) open-store protocol, which requires specification of the type of >> backend you'll be using. A specification is a list of two elements; a >> backend type specification [:bdb, :clsql] and backend-specific store >> information: >> >> (open-store '(:bdb "/home/user/db/testdb")) >> (open-store '(:clsql (:sqlite3 "sqlite3-test.db")) >> (open-store '(:clsql (:postgresql "localhost.localdomain" "test" >> "postgres" ""))) >> >> The tests should print a long list of test names, then return T. >> To run the tests you need to follow the comments in elephant-tests.lisp: >> >> ;; >> ;; GUIDE TO TESTING >> ;; >> ;; 1) Set *default-spec* to the above spec of your choice >> ;; 2) Call (do-backend-tests) to test the standard API >> ;; 3) To test migration: (do-migration-tests *default-spec* >> ) inserting a second >> ;; spec, typically a bdb spec or create another instance of a sql db >> depending on >> ;; your configuration >> ;; 4) A backend is green if it passes do-backend-tests and can >> succesfully be >> ;; used as spec1 or spec2 argument in the migration test >> ;; >> >> To set *default-spec* you need to do something like: >> >> (setq *default-spec* *testbdb-spec*) >> >> or >> >> (setq *default-spec* *testsqlite3-spec*) >> >> Can you list the various problems you had trying to get the system to >> run? We're working on useability for the next release and want to fix >> as many of the inconveniences as possible. >> >> Thank you, >> Ian >> >> Red Daly wrote: >> >>> I have been trying to get Elephant up and running for a few days with >>> little luck. I have tried CMUCL and SBCL, with different unsuccessful >>> outcomes. During my latest attempt on SBCL, I seem to be able to call >>> the following two lines without errors: >>> (asdf:operate 'asdf:load-op :elephant) >>> (asdf:operate 'asdf:load-op :ele-bdb) >>> >>> I then run the tests... >>> (asdf:operate 'asdf:load-op :elephant-tests) >>> (in-package :ele-tests) >>> (setf *default-spec* "bdb") >>> (do-backend-tests) >>> >>> do-backend-tests returns NIL.. which I interpret to be success? >>> >>> >>> however, I am unable to open a store according to the instructions in >>> the two tutorials on the common-lisp.net project site. Here is an >>> example attempt: >>> >>> * (open-store "store") >>> >>> debugger invoked on a SIMPLE-ERROR in thread >>> #: >>> The assertion (CONSP ELEPHANT::SPEC) failed. >>> >>> Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. >>> >>> restarts (invokable by number or by possibly-abbreviated name): >>> 0: [CONTINUE] Retry assertion. >>> 1: [ABORT ] Exit debugger, returning to top level. >>> >>> (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) >>> 0] backtrace >>> >>> 0: (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) >>> 1: (OPEN-STORE "store" :RECOVER NIL :RECOVER-FATAL NIL :THREAD T) >>> 2: (SB-INT:EVAL-IN-LEXENV (OPEN-STORE "store") #) >>> 3: (SB-EXT:INTERACTIVE-EVAL (OPEN-STORE "store")) >>> 4: (SB-IMPL::REPL-FUN NIL) >>> 5: ((LAMBDA ())) >>> 6: ((LAMBDA ())) >>> 7: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) >>> 8: (SB-IMPL::TOPLEVEL-REPL NIL) >>> 9: (SB-IMPL::TOPLEVEL-INIT) >>> 10: ((LABELS SB-IMPL::RESTART-LISP)) >>> >>> >>> I would appreciate any help in getting elephant up and running. I am >>> running SBCL 0.9.8, ubuntu dapper, and bdb 4.3 (apt-installable >>> version). I will gladly supply more information to help determine the >>> error of my ways. I can't wait to try out elephant! >>> >>> Thanks sincerely, >>> red daly >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From reddaly at gmail.com Thu Oct 5 23:52:11 2006 From: reddaly at gmail.com (Red Daly) Date: Thu, 05 Oct 2006 16:52:11 -0700 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <4525899A.9070801@csail.mit.edu> References: <1160036592.9944.18.camel@localhost> <45250215.7020809@csail.mit.edu> <1160075826.9944.42.camel@localhost> <4525899A.9070801@csail.mit.edu> Message-ID: <1160092331.9944.68.camel@localhost> I am not sure if I follow entirely. perhaps you can elaborate if I seem to have misunderstood. I think the following program just loads the :elephant system and then calls open-store. Is this what you mean? (asdf:operate 'asdf:load-op :elephant) (let ((store-spec '(:BDB "store"))) (elephant:open-store store-spec)) This code causes a the same sort of SBCL disconnect (and exit code 256). Strangely I only see these errors in Slime. Since I am a lisp novice, I do not know much about how to debug these problems myself. thanks again, red daly P.S. I have also posted this problem on the slime-devel mailing list in case they have any insights/resolutions. On Thu, 2006-10-05 at 18:39 -0400, Ian Eslick wrote: > Could be some odd case in asdf. When you open a store, the appropriate > asdf file is called to load the backend code if it isn't loaded. Try > loading the :elephant system instead of :ele-bdb, then doing open-store > and see what happens. > > I'll test this on SBCL later tonight. > > Ian > > Red Daly wrote: > > Thank you for the quick and helpful response :). > > > > With your corrections, all tests passed and I seem to be able to open a > > functioning store! However, life is not perfect yet. The main problem > > I seem to be facing right now is when I attempt to load elephant using > > my slime environment (when calling (asdf:operate > > 'asdf:load-op :ele-bdb)). For some reason, I lose my lisp connection. > > A sample transcript of a common slime session can be found here: > > http://paste.lisp.org/display/27406 > > > > This is the point of failure of a similar session: > > (let ((store-spec '(:BDB "store"))) > > (ele:open-store store-spec)) > > ; loading system definition from > > ; /home/red/code/edufacebook/elephant-0.6.0/ele-bdb.asd into > > ; # > > ; registering # as ELE-BDB > > --- > > Lisp connection closed unexpectedly: exited abnormally with code 256 > > > > This is perhaps a slime/sbcl/other bug, but so far I haven't seen > > anything like it, and since slime is so popular (and useful) perhaps > > others have seen this bug or know what to do about it. > > > > In the raw sbcl REPL I do not seem to have a problem, but only when I > > attempt to load ele-bdb. > > > > If you would like, I can post the problems I had with CMUCL when I have > > more time. (off the top of my head, I remember package locking problems > > and also a problem loading cl-base64). at the moment I am more > > concerned with getting a working system in SBCL with Slime, but I would > > like to help make elephant better however I can. > > > > -red daly > > > > > > On Thu, 2006-10-05 at 09:01 -0400, Ian Eslick wrote: > > > >> I'm sorry if the website hasn't been updated (Robert, can you check > >> this?), but the tutorial should be updated to reflect the new (as of > >> 0.6.0) open-store protocol, which requires specification of the type of > >> backend you'll be using. A specification is a list of two elements; a > >> backend type specification [:bdb, :clsql] and backend-specific store > >> information: > >> > >> (open-store '(:bdb "/home/user/db/testdb")) > >> (open-store '(:clsql (:sqlite3 "sqlite3-test.db")) > >> (open-store '(:clsql (:postgresql "localhost.localdomain" "test" > >> "postgres" ""))) > >> > >> The tests should print a long list of test names, then return T. > >> To run the tests you need to follow the comments in elephant-tests.lisp: > >> > >> ;; > >> ;; GUIDE TO TESTING > >> ;; > >> ;; 1) Set *default-spec* to the above spec of your choice > >> ;; 2) Call (do-backend-tests) to test the standard API > >> ;; 3) To test migration: (do-migration-tests *default-spec* > >> ) inserting a second > >> ;; spec, typically a bdb spec or create another instance of a sql db > >> depending on > >> ;; your configuration > >> ;; 4) A backend is green if it passes do-backend-tests and can > >> succesfully be > >> ;; used as spec1 or spec2 argument in the migration test > >> ;; > >> > >> To set *default-spec* you need to do something like: > >> > >> (setq *default-spec* *testbdb-spec*) > >> > >> or > >> > >> (setq *default-spec* *testsqlite3-spec*) > >> > >> Can you list the various problems you had trying to get the system to > >> run? We're working on useability for the next release and want to fix > >> as many of the inconveniences as possible. > >> > >> Thank you, > >> Ian > >> > >> Red Daly wrote: > >> > >>> I have been trying to get Elephant up and running for a few days with > >>> little luck. I have tried CMUCL and SBCL, with different unsuccessful > >>> outcomes. During my latest attempt on SBCL, I seem to be able to call > >>> the following two lines without errors: > >>> (asdf:operate 'asdf:load-op :elephant) > >>> (asdf:operate 'asdf:load-op :ele-bdb) > >>> > >>> I then run the tests... > >>> (asdf:operate 'asdf:load-op :elephant-tests) > >>> (in-package :ele-tests) > >>> (setf *default-spec* "bdb") > >>> (do-backend-tests) > >>> > >>> do-backend-tests returns NIL.. which I interpret to be success? > >>> > >>> > >>> however, I am unable to open a store according to the instructions in > >>> the two tutorials on the common-lisp.net project site. Here is an > >>> example attempt: > >>> > >>> * (open-store "store") > >>> > >>> debugger invoked on a SIMPLE-ERROR in thread > >>> #: > >>> The assertion (CONSP ELEPHANT::SPEC) failed. > >>> > >>> Type HELP for debugger help, or (SB-EXT:QUIT) to exit from SBCL. > >>> > >>> restarts (invokable by number or by possibly-abbreviated name): > >>> 0: [CONTINUE] Retry assertion. > >>> 1: [ABORT ] Exit debugger, returning to top level. > >>> > >>> (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > >>> 0] backtrace > >>> > >>> 0: (SB-KERNEL:ASSERT-ERROR (CONSP ELEPHANT::SPEC) NIL NIL) > >>> 1: (OPEN-STORE "store" :RECOVER NIL :RECOVER-FATAL NIL :THREAD T) > >>> 2: (SB-INT:EVAL-IN-LEXENV (OPEN-STORE "store") #) > >>> 3: (SB-EXT:INTERACTIVE-EVAL (OPEN-STORE "store")) > >>> 4: (SB-IMPL::REPL-FUN NIL) > >>> 5: ((LAMBDA ())) > >>> 6: ((LAMBDA ())) > >>> 7: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) > >>> 8: (SB-IMPL::TOPLEVEL-REPL NIL) > >>> 9: (SB-IMPL::TOPLEVEL-INIT) > >>> 10: ((LABELS SB-IMPL::RESTART-LISP)) > >>> > >>> > >>> I would appreciate any help in getting elephant up and running. I am > >>> running SBCL 0.9.8, ubuntu dapper, and bdb 4.3 (apt-installable > >>> version). I will gladly supply more information to help determine the > >>> error of my ways. I can't wait to try out elephant! > >>> > >>> Thanks sincerely, > >>> red daly > >>> > >>> _______________________________________________ > >>> elephant-devel site list > >>> elephant-devel at common-lisp.net > >>> http://common-lisp.net/mailman/listinfo/elephant-devel > >>> > >>> > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > From read at robertlread.net Fri Oct 6 00:11:23 2006 From: read at robertlread.net (Robert L. Read) Date: Thu, 05 Oct 2006 19:11:23 -0500 Subject: [elephant-devel] trouble getting going (with open-store, ...) In-Reply-To: <1160092331.9944.68.camel@localhost> References: <1160036592.9944.18.camel@localhost> <45250215.7020809@csail.mit.edu> <1160075826.9944.42.camel@localhost> <4525899A.9070801@csail.mit.edu> <1160092331.9944.68.camel@localhost> Message-ID: <1160093484.2473.708.camel@localhost.localdomain> On Thu, 2006-10-05 at 16:52 -0700, Red Daly wrote: > I am not sure if I follow entirely. perhaps you can elaborate if I > seem > to have misunderstood. I think the following program just loads > the :elephant system and then calls open-store. Is this what you > mean? > > (asdf:operate 'asdf:load-op :elephant) > (let ((store-spec '(:BDB "store"))) > (elephant:open-store store-spec)) > > This code causes a the same sort of SBCL disconnect (and exit code > 256). > Strangely I only see these errors in Slime. Since I am a lisp novice, > I > do not know much about how to debug these problems myself. > > thanks again, > red daly > > P.S. I have also posted this problem on the slime-devel mailing list > in > case they have any insights/resolutions. The slime people should know better than we. However, make sure your version of slime is up-to-date, or at least known to work with your LISP. Also, utf-8 issues can freak slime out; that is a possible source of trouble, but I don't think it has anything to do with Elephant. -------------- next part -------------- An HTML attachment was scrubbed... URL: From reddaly at gmail.com Tue Oct 10 07:46:01 2006 From: reddaly at gmail.com (Red Daly) Date: Tue, 10 Oct 2006 00:46:01 -0700 Subject: [elephant-devel] working with many millions of objects Message-ID: <452B4FB9.3010906@gmail.com> I will be running experiments in informatics and modeling in the future that may contain (tens or hundreds of) millions of objects. Given the ease of use of elephant so far, it would be great to use it as the persistent store and avoid creating too many custom data structures. I have recently run up against some performance bottlenecks when using elephant to work with very large datasets (in the hundreds of millions of objects). Using SleepyCat, I am able to import data very quickly with a DB_CONFIG file with the following contents: set_lk_max_locks 500000 set_lk_max_objects 500000 set_lk_max_lockers 500000 set_cachesize 1 0 0 I can import data very quickly until the 1 gb cache is too small to allow complete in-memory access to the database. at this point it seems that disk IO makes additional writes happen much slower. (I have also tried increasing the 1 gb cache size, but the database fails to open if it is too large--e.g. 2 gbs. I have 1.25 gb physical memory and 4 gb swap, so the constraint seems to be physical memory.) the max_lock, etc. lines allow transactions to contain hundreds of thousands of individual locks, limiting the transaction throughput bottleneck What are the technical restrictions on writing several million objects to the datastore? Is it feasible to create a batch import feature to allow large datasets to be imported using reasonable amounts of memory for a desktop computer? I hope this email is at least amusing! Thanks again, red daly From read at robertlread.net Tue Oct 10 12:53:16 2006 From: read at robertlread.net (Robert L. Read) Date: Tue, 10 Oct 2006 07:53:16 -0500 Subject: [elephant-devel] working with many millions of objects In-Reply-To: <452B4FB9.3010906@gmail.com> References: <452B4FB9.3010906@gmail.com> Message-ID: <1160484796.2473.941.camel@localhost.localdomain> Yes, it's amusing. In my own work I use the Postgres backend; I know very little about SleepyCat. It seems to me that this is more of a SleepyCat issue, then an Elephant issue. Perhaps you should ask the SleepyCat list? Are you importing things into SleepyCat directly in the correct serialization format that they can be read by Elephant? If so, I assume it is just a question of solving the SleepyCat problems. An alternative would be to use the SQL-based backend. However, I doubt this will solve your problem, since at present we (well, I wrote it) use a very inefficient serialization scheme for the SQL-based backend that base64 encodes everything. This had the advantage that it makes it work trouble-free with different database backends, but could clearly be improved upon. However, it is more than efficient enough for all my work, and at present nobody is clamoring to have it improved. Is your problem importing the data or using it once it is imported? It's hard for me to imagine a problem so large that even the import time is a problem --- suppose it takes 24 hours --- can you not afford to pay that? A drastic measure and potentially expensive measure would be to switch to a 64-bit architecture with a huge memory. I intend to do that when forced by performance issues in my own work. On Tue, 2006-10-10 at 00:46 -0700, Red Daly wrote: > I will be running experiments in informatics and modeling in the future > that may contain (tens or hundreds of) millions of objects. Given the > ease of use of elephant so far, it would be great to use it as the > persistent store and avoid creating too many custom data structures. > > I have recently run up against some performance bottlenecks when using > elephant to work with very large datasets (in the hundreds of millions > of objects). Using SleepyCat, I am able to import data very quickly > with a DB_CONFIG file with the following contents: > > set_lk_max_locks 500000 > set_lk_max_objects 500000 > set_lk_max_lockers 500000 > set_cachesize 1 0 0 > > I can import data very quickly until the 1 gb cache is too small to > allow complete in-memory access to the database. at this point it seems > that disk IO makes additional writes happen much slower. (I have also > tried increasing the 1 gb cache size, but the database fails to open if > it is too large--e.g. 2 gbs. I have 1.25 gb physical memory and 4 gb > swap, so the constraint seems to be physical memory.) the max_lock, > etc. lines allow transactions to contain hundreds of thousands of > individual locks, limiting the transaction throughput bottleneck > > What are the technical restrictions on writing several million objects > to the datastore? Is it feasible to create a batch import feature to > allow large datasets to be imported using reasonable amounts of memory > for a desktop computer? > > I hope this email is at least amusing! > > Thanks again, > red daly > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From nowhere.man at levallois.eu.org Wed Oct 11 14:23:42 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Wed, 11 Oct 2006 16:23:42 +0200 Subject: [elephant-devel] Elephant and inheritance Message-ID: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> I'm happily using Elephant in a Web-based project right now, but I'm getting annoyed by the behoviour of Elephant WRT inheritance. The problem is that only direct slots of the class with the persistent metaclass as metaclass are made persistent. If I have the following classes: (defclass foo () ((bar))) (defclass sub-foo (foo) ((baz)) (:metaclass ele:persistent-metaclass)) Only the baz slot will be persistent. Is it impossible to gain advantage of both inheritance and persistence, or am I missing something? Curiously, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From eslick at csail.mit.edu Wed Oct 11 15:29:52 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 11 Oct 2006 11:29:52 -0400 Subject: [elephant-devel] Elephant and inheritance In-Reply-To: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> References: <20061011142342.GI7422@bateleur.arcanes.fr.eu.org> Message-ID: <452D0DF0.1070809@csail.mit.edu> Well it's a bit of a design trade off. If you have a non-persistent base class, should the semantics of the slot storage change based on whether it's included in a persistent subclass? Currently the policy is simple; persistence is determined by annotations on direct slots and not effective slots. Thus you can get functional inheritance, but have to manually specify if you want to persist an inherited non-persistent subclass. To get persistent behavior for non-persistent base classes you need to explicitly tell the CLOS protocol that a slot is persistent and the only way to do that today is to include the base class slots directly in the superclass. You'll still get the base class generic function dispatch, but reads/writes will go to the persistent store instead of memory. (defclass foo () ((myfoo) (bar))) (defclass sub-foo (foo) ((bar) (baz)) (:metaclass ele:persistent-metaclass)) Thus myfoo is transient and bar, baz are persistent. (defmethod mutate-bar ((obj foo) value) (setf (slot-value obj 'bar) value)) This will set the persistent value of bar for objects of sub-foo, but set the in-memory value of bar if the object is of type 'foo'. It is possible to do this automatically by modifying the Elephant metaclass protocol, perhaps with a class option or a new metaclass 'inheriting-persistent-metaclass' which defaults to making all subclass slots persistent...then to override this behavior you'd special case subclass slot names by specifying transient in a shadowing subclass definition. (defclass foo () ((myfoo) (bar))) (defclass sub-foo (foo) ((baz) (myfoo :transient t)) (:persist-subclass-slots t) (:metaclass ele:persistent-metaclass)) This would have identical behavior to the first example, but you don't have to make myfoo transient (it would then be given a persistent type for sub-foo). Now what happens if we inherit from sub-foo above? (defclass sub-sub-foo (sub-foo) ((qux)) (:metaclass ele:persistent-metaclass)) ;; protocol will assert an error if subclass of persistent isn't persistent In this case the normal CLOS machinery will pick up the non-direct slots from subclasses and the bar and myfoo slots will be non-transient. There may be a way to override this to search for subclass specifications of :persist-subclass-slots and then go ahead and promote :transient effective slots to :persistent but all the cases and interactions can start to become hairy. I think the currently policy is the simplest and least bug prone, even though it forces the increased verboseness of the first example. Thoughts? Ian Pierre THIERRY wrote: > I'm happily using Elephant in a Web-based project right now, but I'm > getting annoyed by the behoviour of Elephant WRT inheritance. The > problem is that only direct slots of the class with the persistent > metaclass as metaclass are made persistent. > > If I have the following classes: > > (defclass foo () > ((bar))) > > (defclass sub-foo (foo) > ((baz)) > (:metaclass ele:persistent-metaclass)) > > Only the baz slot will be persistent. Is it impossible to gain advantage > of both inheritance and persistence, or am I missing something? > > Curiously, > Nowhere man > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From reddaly at gmail.com Thu Oct 12 00:57:14 2006 From: reddaly at gmail.com (Red Daly) Date: Wed, 11 Oct 2006 17:57:14 -0700 Subject: [elephant-devel] working with many millions of objects In-Reply-To: <1160484796.2473.941.camel@localhost.localdomain> References: <452B4FB9.3010906@gmail.com> <1160484796.2473.941.camel@localhost.localdomain> Message-ID: <452D92EA.1030800@gmail.com> I was importing into sleepycat using standard elephant routines. I am not aware of an 'import mode' for sleepycat, but I will look into that when I have a chance. Another consideration using sleepycat is that using BTrees with a large working set demands large amounts of memory relative to a Hash representation. I am unfamiliar with the internals of elephant and sleepycat, but it feels like the basic access method is restricting performance, which seems to be described here: http://www.sleepycat.com/docs/gsg/C/accessmethods.html#BTreeVSHash My problem so far has been importing the data, which goes very fast until sleepycat requires extensive disk access. The in-memory rate is reasonable and would complete in a few hours. However, once disk operations begin the import speed suggests it would take many days to complete. I have yet to perform extensive benchmarks, but I estimate the instantiation rate shifts from 1800 persistent class instantiations /second to 120 / s. here are the two approaches that I hypothesize may help performance. I am admittedly unaware of innards of the two systems in question, so you expert developers will know best. If either sounds appropriate or you envision another possibility for allowing this kind of scaling, I will look into implementing such a system. 1. decreasing the size of the working set is one possibility for decreasing run-time memory requirements and disk access. I'm not sure how the concept of a 'working set' translates from the sleepycat world to the elephant world, but perhaps you do. 2. using a Hash instead of a BTree in the primary database? I am unsure what this means for elephant. In the mean time I will depart from the every-class-is-persistent approach and also use more traditional data structures. Thanks again, Red Daly Robert L. Read wrote: > Yes, it's amusing. > > In my own work I use the Postgres backend; I know very little about > SleepyCat. It seems > to me that this is more of a SleepyCat issue, then an Elephant issue. > Perhaps you should > ask the SleepyCat list? > > Are you importing things into SleepyCat directly in the correct > serialization format that > they can be read by Elephant? If so, I assume it is just a question > of solving the SleepyCat > problems. > > An alternative would be to use the SQL-based backend. However, I > doubt this will solve > your problem, since at present we (well, I wrote it) use a very > inefficient serialization scheme > for the SQL-based backend that base64 encodes everything. This had > the advantage that > it makes it work trouble-free with different database backends, but > could clearly be improved upon. > However, it is more than efficient enough for all my work, and at > present nobody is clamoring > to have it improved. > > Is your problem importing the data or using it once it is imported? > It's hard for me to imagine > a problem so large that even the import time is a problem --- suppose > it takes 24 hours --- can > you not afford to pay that? > > A drastic measure and potentially expensive measure would be to switch > to a 64-bit architecture > with a huge memory. I intend to do that when forced by performance > issues in my own work. > > > > On Tue, 2006-10-10 at 00:46 -0700, Red Daly wrote: >> I will be running experiments in informatics and modeling in the future >> that may contain (tens or hundreds of) millions of objects. Given the >> ease of use of elephant so far, it would be great to use it as the >> persistent store and avoid creating too many custom data structures. >> >> I have recently run up against some performance bottlenecks when using >> elephant to work with very large datasets (in the hundreds of millions >> of objects). Using SleepyCat, I am able to import data very quickly >> with a DB_CONFIG file with the following contents: >> >> set_lk_max_locks 500000 >> set_lk_max_objects 500000 >> set_lk_max_lockers 500000 >> set_cachesize 1 0 0 >> >> I can import data very quickly until the 1 gb cache is too small to >> allow complete in-memory access to the database. at this point it seems >> that disk IO makes additional writes happen much slower. (I have also >> tried increasing the 1 gb cache size, but the database fails to open if >> it is too large--e.g. 2 gbs. I have 1.25 gb physical memory and 4 gb >> swap, so the constraint seems to be physical memory.) the max_lock, >> etc. lines allow transactions to contain hundreds of thousands of >> individual locks, limiting the transaction throughput bottleneck >> >> What are the technical restrictions on writing several million objects >> to the datastore? Is it feasible to create a batch import feature to >> allow large datasets to be imported using reasonable amounts of memory >> for a desktop computer? >> >> I hope this email is at least amusing! >> >> Thanks again, >> red daly >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Thu Oct 12 01:14:36 2006 From: read at robertlread.net (Robert L. Read) Date: Wed, 11 Oct 2006 20:14:36 -0500 Subject: [elephant-devel] working with many millions of objects In-Reply-To: <452D92EA.1030800@gmail.com> References: <452B4FB9.3010906@gmail.com> <1160484796.2473.941.camel@localhost.localdomain> <452D92EA.1030800@gmail.com> Message-ID: <1160615677.5176.71.camel@localhost.localdomain> On Wed, 2006-10-11 at 17:57 -0700, Red Daly wrote: > I was importing into sleepycat using standard elephant routines. I am > not aware of an 'import mode' for sleepycat, but I will look into that > when I have a chance. I do not think there is anyway to import things directly and to deal with Elephants serialization at the same time. I think this is a dead end. > Another consideration using sleepycat is that > using BTrees with a large working set demands large amounts of memory > relative to a Hash representation. I am unfamiliar with the internals > of elephant and sleepycat, but it feels like the basic access method is > restricting performance, which seems to be described here: > http://www.sleepycat.com/docs/gsg/C/accessmethods.html#BTreeVSHash > > My problem so far has been importing the data, which goes very fast > until sleepycat requires extensive disk access. The in-memory rate is > reasonable and would complete in a few hours. However, once disk > operations begin the import speed suggests it would take many days to > complete. I have yet to perform extensive benchmarks, but I estimate > the instantiation rate shifts from 1800 persistent class instantiations > /second to 120 / s. > > here are the two approaches that I hypothesize may help performance. I > am admittedly unaware of innards of the two systems in question, so you > expert developers will know best. If either sounds appropriate or you > envision another possibility for allowing this kind of scaling, I will > look into implementing such a system. > > 1. decreasing the size of the working set is one possibility for > decreasing run-time memory requirements and disk access. I'm not sure > how the concept of a 'working set' translates from the sleepycat world > to the elephant world, but perhaps you do. Elephant keeps very little in memory (unless you used my DCM module in the contrib directory.) So I have to admit I don't quite understand why any import operation would suddenly shift to more disk access. Now, doing a simple import of a lot of data WOULD tend to leave objects in memory that could not be garbage collected, if you weren't careful. That could be solved by either making SURE you don't keep a reference to the objects or by doing your import in large batches (restarting LISP processes in between times). But I am just guesssing --- do you feel comfortable posting your code, or pseudocode for how you do the import? > > 2. using a Hash instead of a BTree in the primary database? I am > unsure what this means for elephant. I don't think that you make progress with that. Elephant depends very deeply on the BTree, and, as far as I know, so does Sleepycat. > > In the mean time I will depart from the every-class-is-persistent > approach and also use more traditional data structures. > > Thanks again, > Red Daly > > > > Robert L. Read wrote: > > Yes, it's amusing. > > > > In my own work I use the Postgres backend; I know very little about > > SleepyCat. It seems > > to me that this is more of a SleepyCat issue, then an Elephant issue. > > Perhaps you should > > ask the SleepyCat list? > > > > Are you importing things into SleepyCat directly in the correct > > serialization format that > > they can be read by Elephant? If so, I assume it is just a question > > of solving the SleepyCat > > problems. > > > > An alternative would be to use the SQL-based backend. However, I > > doubt this will solve > > your problem, since at present we (well, I wrote it) use a very > > inefficient serialization scheme > > for the SQL-based backend that base64 encodes everything. This had > > the advantage that > > it makes it work trouble-free with different database backends, but > > could clearly be improved upon. > > However, it is more than efficient enough for all my work, and at > > present nobody is clamoring > > to have it improved. > > > > Is your problem importing the data or using it once it is imported? > > It's hard for me to imagine > > a problem so large that even the import time is a problem --- suppose > > it takes 24 hours --- can > > you not afford to pay that? > > > > A drastic measure and potentially expensive measure would be to switch > > to a 64-bit architecture > > with a huge memory. I intend to do that when forced by performance > > issues in my own work. > > > > > > > > On Tue, 2006-10-10 at 00:46 -0700, Red Daly wrote: > >> I will be running experiments in informatics and modeling in the future > >> that may contain (tens or hundreds of) millions of objects. Given the > >> ease of use of elephant so far, it would be great to use it as the > >> persistent store and avoid creating too many custom data structures. > >> > >> I have recently run up against some performance bottlenecks when using > >> elephant to work with very large datasets (in the hundreds of millions > >> of objects). Using SleepyCat, I am able to import data very quickly > >> with a DB_CONFIG file with the following contents: > >> > >> set_lk_max_locks 500000 > >> set_lk_max_objects 500000 > >> set_lk_max_lockers 500000 > >> set_cachesize 1 0 0 > >> > >> I can import data very quickly until the 1 gb cache is too small to > >> allow complete in-memory access to the database. at this point it seems > >> that disk IO makes additional writes happen much slower. (I have also > >> tried increasing the 1 gb cache size, but the database fails to open if > >> it is too large--e.g. 2 gbs. I have 1.25 gb physical memory and 4 gb > >> swap, so the constraint seems to be physical memory.) the max_lock, > >> etc. lines allow transactions to contain hundreds of thousands of > >> individual locks, limiting the transaction throughput bottleneck > >> > >> What are the technical restrictions on writing several million objects > >> to the datastore? Is it feasible to create a batch import feature to > >> allow large datasets to be imported using reasonable amounts of memory > >> for a desktop computer? > >> > >> I hope this email is at least amusing! > >> > >> Thanks again, > >> red daly > >> _______________________________________________ > >> elephant-devel site list > >> elephant-devel at common-lisp.net > >> http://common-lisp.net/mailman/listinfo/elephant-devel > >> > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Thu Oct 12 01:12:48 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 11 Oct 2006 21:12:48 -0400 Subject: [elephant-devel] working with many millions of objects In-Reply-To: <452D92EA.1030800@gmail.com> References: <452B4FB9.3010906@gmail.com> <1160484796.2473.941.camel@localhost.localdomain> <452D92EA.1030800@gmail.com> Message-ID: <452D9690.6030509@csail.mit.edu> Can you tell me a little bit about what the import operations look like - that is to say how many objects are created per-transaction, how many slots per created object, etc? Things are only cached in-memory during a transaction. To ensure ACID properties (unless you've turned off synchronization) every transaction should flush to disk just prior to completion. It sounds almost like you're doing a giant transaction, or perhaps I have the scale wrong and it's the BTree cached index memory that is eating up all your working memory. The concept of a working set, is the number of distinct 'pages' touched during a transaction (or set of transactions) In elephant every unique slot access will hit a different page, but every slot access that is nearby in the BTree index will or may share storage. However import by it's very nature is a linear operation, there is (roughly) no locality as every record is new - so you'll be allocating lots of new pages and re-balancing the BTrees quite a bit. Until I have a better sense of how you are using transactions it's harder to be more helpful. My own DB is about 6GB but I've built it up over a long time with alot of large records. Thanks, Ian Red Daly wrote: > I was importing into sleepycat using standard elephant routines. I am > not aware of an 'import mode' for sleepycat, but I will look into that > when I have a chance. Another consideration using sleepycat is that > using BTrees with a large working set demands large amounts of memory > relative to a Hash representation. I am unfamiliar with the internals > of elephant and sleepycat, but it feels like the basic access method > is restricting performance, which seems to be described here: > http://www.sleepycat.com/docs/gsg/C/accessmethods.html#BTreeVSHash > > My problem so far has been importing the data, which goes very fast > until sleepycat requires extensive disk access. The in-memory rate is > reasonable and would complete in a few hours. However, once disk > operations begin the import speed suggests it would take many days to > complete. I have yet to perform extensive benchmarks, but I estimate > the instantiation rate shifts from 1800 persistent class > instantiations /second to 120 / s. > > here are the two approaches that I hypothesize may help performance. > I am admittedly unaware of innards of the two systems in question, so > you expert developers will know best. If either sounds appropriate or > you envision another possibility for allowing this kind of scaling, I > will look into implementing such a system. > > 1. decreasing the size of the working set is one possibility for > decreasing run-time memory requirements and disk access. I'm not sure > how the concept of a 'working set' translates from the sleepycat world > to the elephant world, but perhaps you do. > > 2. using a Hash instead of a BTree in the primary database? I am > unsure what this means for elephant. > > In the mean time I will depart from the every-class-is-persistent > approach and also use more traditional data structures. > > Thanks again, > Red Daly > > > > Robert L. Read wrote: >> Yes, it's amusing. >> >> In my own work I use the Postgres backend; I know very little about >> SleepyCat. It seems >> to me that this is more of a SleepyCat issue, then an Elephant >> issue. Perhaps you should >> ask the SleepyCat list? >> >> Are you importing things into SleepyCat directly in the correct >> serialization format that >> they can be read by Elephant? If so, I assume it is just a question >> of solving the SleepyCat >> problems. >> >> An alternative would be to use the SQL-based backend. However, I >> doubt this will solve >> your problem, since at present we (well, I wrote it) use a very >> inefficient serialization scheme >> for the SQL-based backend that base64 encodes everything. This had >> the advantage that >> it makes it work trouble-free with different database backends, but >> could clearly be improved upon. >> However, it is more than efficient enough for all my work, and at >> present nobody is clamoring >> to have it improved. >> >> Is your problem importing the data or using it once it is imported? >> It's hard for me to imagine >> a problem so large that even the import time is a problem --- suppose >> it takes 24 hours --- can >> you not afford to pay that? >> >> A drastic measure and potentially expensive measure would be to >> switch to a 64-bit architecture >> with a huge memory. I intend to do that when forced by performance >> issues in my own work. >> >> >> >> On Tue, 2006-10-10 at 00:46 -0700, Red Daly wrote: >>> I will be running experiments in informatics and modeling in the >>> future that may contain (tens or hundreds of) millions of objects. >>> Given the ease of use of elephant so far, it would be great to use >>> it as the persistent store and avoid creating too many custom data >>> structures. >>> >>> I have recently run up against some performance bottlenecks when >>> using elephant to work with very large datasets (in the hundreds of >>> millions of objects). Using SleepyCat, I am able to import data >>> very quickly with a DB_CONFIG file with the following contents: >>> >>> set_lk_max_locks 500000 >>> set_lk_max_objects 500000 >>> set_lk_max_lockers 500000 >>> set_cachesize 1 0 0 >>> >>> I can import data very quickly until the 1 gb cache is too small to >>> allow complete in-memory access to the database. at this point it >>> seems that disk IO makes additional writes happen much slower. (I >>> have also tried increasing the 1 gb cache size, but the database >>> fails to open if it is too large--e.g. 2 gbs. I have 1.25 gb >>> physical memory and 4 gb swap, so the constraint seems to be >>> physical memory.) the max_lock, etc. lines allow transactions to >>> contain hundreds of thousands of individual locks, limiting the >>> transaction throughput bottleneck >>> >>> What are the technical restrictions on writing several million >>> objects to the datastore? Is it feasible to create a batch import >>> feature to allow large datasets to be imported using reasonable >>> amounts of memory for a desktop computer? >>> >>> I hope this email is at least amusing! >>> >>> Thanks again, >>> red daly >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Thu Oct 12 01:18:16 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 11 Oct 2006 21:18:16 -0400 Subject: [elephant-devel] working with many millions of objects In-Reply-To: <1160615677.5176.71.camel@localhost.localdomain> References: <452B4FB9.3010906@gmail.com> <1160484796.2473.941.camel@localhost.localdomain> <452D92EA.1030800@gmail.com> <1160615677.5176.71.camel@localhost.localdomain> Message-ID: <452D97D8.8040200@csail.mit.edu> I don't think that performance of the base Elephant is good enough to justify an every-object-is-persistent model. It's just not seamless enough or efficient enough yet. DCM takes a good step in the performance direction for most applications (single lisp images) and I'm hoping that some of the changes I'd like to get into 0.7.0 or 0.8.0 integrate the DCM ideas into the persistent object protocol. >> 2. using a Hash instead of a BTree in the primary database? I am >> unsure what this means for elephant. >> > I don't think that you make progress with that. Elephant depends very > deeply on the BTree, > and, as far as I know, so does Sleepycat. Actually BDB has four record types of which BTree is one. However, I believe it would be a fairly major rewrite as I believe I recall that the C API for Hash access is quite different and most of elephant would have to be adjusted. The tradeoff boils down to this: - Every BTree access requires log(N) page accesses, if you have high locality than N or N-1 of these is cached and you average O(1) with a lower constant than Hash accesses - Every Hash access requires log(1) page accesses, but each access has higher overhead and for large hashes there is no way to ensure disk access locality other than repeated hits. So if you are using alot of indexing and accessing pages in some sorted order, you'll win with BTrees - but if you are doing random accesses you'll trash the cache by an extra log(N) factor. Ian From gk1 at four-four.com Thu Oct 26 08:13:56 2006 From: gk1 at four-four.com (George Khouri) Date: Thu, 26 Oct 2006 01:13:56 -0700 Subject: [elephant-devel] Elephant 0.6.0 on OpenMCL 1.x Message-ID: Friends, I'm new to this list and I couldn't find a thread on this subject. I've been using Elephant 0.2.1/Berkeley 4.2 on several versions of OpenMCL up to 1.1-pre-060720 successfully for some time now. Elephant 0.2.1/Berkeley 4.2 no longer work with the latest pre-release of OpenMCL 1.1 (061024). I've updated to Elephant 0.6.0/Berkeley 4.3 from the tarballs but can't get Elephant to work with any openMCL 1.x versions (perhaps it doesn't). Berkeley 4.3 and Elephant 0.6.0 compile fine but the tests don't run because open-store complains that any :bdb spec it is passed is an invalid type. Sample session: ********* Welcome to OpenMCL Version 1.0 (DarwinPPC32)! ? (asdf:oos 'asdf:load-op :elephant) ; loading system definition from /src/lisp/lisp-systems/elephant.asd into # ; loading system definition from /src/lisp/lisp-systems/uffi.asd into # ; registering # as UFFI ; registering # as ELEPHANT ; loading system definition from /src/lisp/lisp-systems/cl-base64.asd into # ; registering # as CL-BASE64 ; registering # as CL-BASE64-TESTS NIL ? (ele:open-store '(:bdb "/tmp/gk/")) ; loading system definition from /src/lisp/lisp-systems/ele-bdb.asd into # ; registering # as ELE-BDB > Error in process listener(1): The value (:BDB "/tmp/gk/"), derived from the initarg :SPEC, can not be used to set the value of the slot ELEPHANT::SPEC in #, because it is not of type (OR PATHNAME (VECTOR CHARACTER)). > While executing: CCL::%SHARED-INITIALIZE > Type :POP to abort. ********** I get the same error when running the tests. I also tried getting the latest Elephant sources from cvs but the sleepycat lib in the latest cvs won't compile: --> src/db-bdb/libsleepycat.c:568: error: 'struct __db' has no member named 'compact' I'm using OSX 10.4.7 PPC. Thanks for any help. -------- George Khouri gk1 at four-four.com From eslick at csail.mit.edu Thu Oct 26 08:35:00 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 26 Oct 2006 18:35:00 +1000 Subject: [elephant-devel] Elephant 0.6.0 on OpenMCL 1.x In-Reply-To: References: Message-ID: <45407334.3090202@csail.mit.edu> This sounds like a slot type declaration problem. I've had these before in OpenMCL. What is the type of a base string in OpenMCL? i.e. (type-of "/tmp/gk/") You might comment out the type declaration for that slot in src/db-bdb/bdb-controller.lisp and try again. Let me know how it goes. Thanks, Ian George Khouri wrote: > Friends, > I'm new to this list and I couldn't find a thread on this subject. I've been using Elephant 0.2.1/Berkeley 4.2 on several versions of OpenMCL up to 1.1-pre-060720 successfully for some time now. Elephant 0.2.1/Berkeley 4.2 no longer work with the latest pre-release of OpenMCL 1.1 (061024). I've updated to Elephant 0.6.0/Berkeley 4.3 from the tarballs but can't get Elephant to work with any openMCL 1.x versions (perhaps it doesn't). Berkeley 4.3 and Elephant 0.6.0 compile fine but the tests don't run because open-store complains that any :bdb spec it is passed is an invalid type. > > Sample session: > ********* > Welcome to OpenMCL Version 1.0 (DarwinPPC32)! > ? (asdf:oos 'asdf:load-op :elephant) > ; loading system definition from /src/lisp/lisp-systems/elephant.asd into # > ; loading system definition from /src/lisp/lisp-systems/uffi.asd into # > ; registering # as UFFI > ; registering # as ELEPHANT > ; loading system definition from /src/lisp/lisp-systems/cl-base64.asd into # > ; registering # as CL-BASE64 > ; registering # as CL-BASE64-TESTS > NIL > ? (ele:open-store '(:bdb "/tmp/gk/")) > ; loading system definition from /src/lisp/lisp-systems/ele-bdb.asd into # > ; registering # as ELE-BDB > >> Error in process listener(1): The value (:BDB "/tmp/gk/"), derived from the initarg :SPEC, can not be used to set the value of the slot ELEPHANT::SPEC in #, because it is not of type (OR PATHNAME (VECTOR CHARACTER)). >> While executing: CCL::%SHARED-INITIALIZE >> Type :POP to abort. >> > ********** > I get the same error when running the tests. > I also tried getting the latest Elephant sources from cvs but the sleepycat lib in the latest cvs won't compile: > --> src/db-bdb/libsleepycat.c:568: error: 'struct __db' has no member named 'compact' > I'm using OSX 10.4.7 PPC. > > Thanks for any help. > > -------- > George Khouri > gk1 at four-four.com > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From gk1 at four-four.com Thu Oct 26 18:49:20 2006 From: gk1 at four-four.com (George Khouri) Date: Thu, 26 Oct 2006 11:49:20 -0700 Subject: [elephant-devel] Elephant 0.6.0 on OpenMCL 1.x In-Reply-To: <45407334.3090202@csail.mit.edu> Message-ID: Ian, >This sounds like a slot type declaration problem. I've had these before >in OpenMCL. What is the type of a base string in OpenMCL? i.e. >(type-of "/tmp/gk/") (type-of "/tmp/gk/") ==> (SIMPLE-ARRAY CHARACTER (8)) > >You might comment out the type declaration for that slot in >src/db-bdb/bdb-controller.lisp and try again. Let me know how it goes. That did it (actually the spec slot is defined in the parent class in src/elephant/controller.lisp) I've run the tests in OpenMCL 1.0 and 1.1-pre-060720 and they pass. Elephant 0.6.0/Berkeley4.3 doesn't work with latest pre-release, OpenMCL 1.1-pre-061024. I'll report on that when I get a minute - i suspect it has to do with handling unicode. Thanks, George >George Khouri wrote: >> Friends, >> I'm new to this list and I couldn't find a thread on this subject. I've been >using Elephant 0.2.1/Berkeley 4.2 on several versions of OpenMCL up to >1.1-pre-060720 successfully for some time now. Elephant 0.2.1/Berkeley 4.2 no >longer work with the latest pre-release of OpenMCL 1.1 (061024). I've updated >to Elephant 0.6.0/Berkeley 4.3 from the tarballs but can't get Elephant to work >with any openMCL 1.x versions (perhaps it doesn't). Berkeley 4.3 and Elephant >0.6.0 compile fine but the tests don't run because open-store complains that any >:bdb spec it is passed is an invalid type. >> >> Sample session: >> ********* >> Welcome to OpenMCL Version 1.0 (DarwinPPC32)! >> ? (asdf:oos 'asdf:load-op :elephant) >> ; loading system definition from /src/lisp/lisp-systems/elephant.asd into ># >> ; loading system definition from /src/lisp/lisp-systems/uffi.asd into ># >> ; registering # as UFFI >> ; registering # as ELEPHANT >> ; loading system definition from /src/lisp/lisp-systems/cl-base64.asd into ># >> ; registering # as CL-BASE64 >> ; registering # as CL-BASE64-TESTS >> NIL >> ? (ele:open-store '(:bdb "/tmp/gk/")) >> ; loading system definition from /src/lisp/lisp-systems/ele-bdb.asd into ># >> ; registering # as ELE-BDB >> >>> Error in process listener(1): The value (:BDB "/tmp/gk/"), derived from the >initarg :SPEC, can not be used to set the value of the slot ELEPHANT::SPEC in >#, because it is not of type (OR >PATHNAME (VECTOR CHARACTER)). >>> While executing: CCL::%SHARED-INITIALIZE >>> Type :POP to abort. >>> >> ********** >> I get the same error when running the tests. >> I also tried getting the latest Elephant sources from cvs but the sleepycat >lib in the latest cvs won't compile: >> --> src/db-bdb/libsleepycat.c:568: error: 'struct __db' has no member named >'compact' >> I'm using OSX 10.4.7 PPC. >> >> Thanks for any help. >> >> -------- >> George Khouri >> gk1 at four-four.com From joubert at joubster.com Fri Oct 27 02:31:43 2006 From: joubert at joubster.com (Joubert Nel) Date: Thu, 26 Oct 2006 22:31:43 -0400 Subject: [elephant-devel] (get-from-root "key") results in "Heap exhausted" error Message-ID: <200610262231.47237.joubert@joubster.com> Hi, I've just started using Elephant and have run into a "Heap exhausted" error message when I try to retrieve a key's value from the root. I am doing: (add-to-root "first" "joubert") (get-from-root "first") After the 2nd expression I get the results as per the attachment. I'm running with the following config: Elephant 0.6.0 cl-base64 3.3.2 uffi 1.5.17 BerkeleyDB 4.3 SBCL 0.9.17 SUSE Linux 10.1 AMD64. Any pointers? Regards Joubert -------------- next part -------------- Heap exhausted: 8522182656 bytes available, 455266533424 requested. PROCEED WITH CAUTION! [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8522182656 455266533424) 1: ("foreign function: call_into_lisp") 2: ("foreign function: funcall2") 3: ("foreign function: gc_heap_exhausted_error_or_lose") 4: ("foreign function: gc_find_freeish_pages") 5: ("foreign function: gc_alloc_large") 6: ("foreign function: alloc_tramp") 7: ((LABELS %DESERIALIZE) #) 8: ((SB-PCL::FAST-METHOD GET-VALUE (T SLEEPYCAT::BDB-BTREE)) # # # #) 9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (GET-FROM-ROOT "first") #) 10: (SWANK::EVAL-REGION "(get-from-root \"first\") " T) 11: ((LAMBDA NIL)) 12: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-SYNTAX-HOOKS (T)) # # #) 13: (SWANK::CALL-WITH-BUFFER-SYNTAX #) 14: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(get-from-root \"first\") ") #) 15: ((LAMBDA NIL)) 16: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 17: ((LAMBDA NIL)) 18: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 19: (SWANK::CALL-WITH-REDIRECTED-IO # #) 20: (SWANK::CALL-WITH-CONNECTION # #) 21: (SWANK::HANDLE-REQUEST #) 22: ((LAMBDA NIL)) 23: ((LAMBDA NIL)) 24: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 25: (SWANK::CALL-WITH-REDIRECTED-IO # #) 26: (SWANK::CALL-WITH-CONNECTION # #) 27: (SWANK::CALL-WITH-BINDINGS NIL #) 28: ((LAMBDA NIL)) 29: ("foreign function: call_into_lisp") 30: ("foreign function: new_thread_trampoline") -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available URL: From joubert at joubster.com Sun Oct 29 22:12:52 2006 From: joubert at joubster.com (Joubert Nel) Date: Sun, 29 Oct 2006 18:12:52 -0400 Subject: [elephant-devel] (get-from-root "key") results in "Heap exhausted" error Message-ID: <200610291712.52671.joubert@joubster.com> Hi, I've just started using Elephant and have run into a "Heap exhausted" error message when I try to retrieve a key's value from the root. I am doing: (add-to-root "first" "joubert") (get-from-root "first") After the 2nd expression I get the results as per the attachment. I'm running with the following config: Elephant 0.6.0 cl-base64 3.3.2 uffi 1.5.17 BerkeleyDB 4.3 SBCL 0.9.17 SUSE Linux 10.1 AMD64. I presume this is related to my earlier post from today and something to do with running on x86_64. Any pointers? Regards Joubert -------------- next part -------------- Heap exhausted: 8522182656 bytes available, 455266533424 requested. PROCEED WITH CAUTION! [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8522182656 455266533424) 1: ("foreign function: call_into_lisp") 2: ("foreign function: funcall2") 3: ("foreign function: gc_heap_exhausted_error_or_lose") 4: ("foreign function: gc_find_freeish_pages") 5: ("foreign function: gc_alloc_large") 6: ("foreign function: alloc_tramp") 7: ((LABELS %DESERIALIZE) #) 8: ((SB-PCL::FAST-METHOD GET-VALUE (T SLEEPYCAT::BDB-BTREE)) # # # #) 9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (GET-FROM-ROOT "first") #) 10: (SWANK::EVAL-REGION "(get-from-root \"first\") " T) 11: ((LAMBDA NIL)) 12: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-SYNTAX-HOOKS (T)) # # #) 13: (SWANK::CALL-WITH-BUFFER-SYNTAX #) 14: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(get-from-root \"first\") ") #) 15: ((LAMBDA NIL)) 16: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 17: ((LAMBDA NIL)) 18: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 19: (SWANK::CALL-WITH-REDIRECTED-IO # #) 20: (SWANK::CALL-WITH-CONNECTION # #) 21: (SWANK::HANDLE-REQUEST #) 22: ((LAMBDA NIL)) 23: ((LAMBDA NIL)) 24: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 25: (SWANK::CALL-WITH-REDIRECTED-IO # #) 26: (SWANK::CALL-WITH-CONNECTION # #) 27: (SWANK::CALL-WITH-BINDINGS NIL #) 28: ((LAMBDA NIL)) 29: ("foreign function: call_into_lisp") 30: ("foreign function: new_thread_trampoline") From joubert at joubster.com Sun Oct 29 22:24:25 2006 From: joubert at joubster.com (Joubert Nel) Date: Sun, 29 Oct 2006 18:24:25 -0400 Subject: [elephant-devel] Fwd: (open-store) (close-store) (open-store) fails - running x86_64 on SBCL Message-ID: <200610291724.25424.joubert@joubster.com> Hi, I've noticed that when I call (open-store *my-storage*), then (close-store), then (open-store *my-storage*) again, my LISP simply hangs. From within SLIME/Emacs I don't get any feedback but when I run directly in SBCL I get the following error message after the 2nd (open-store): "fatal error encountered in SBCL. GC invariant lost, file "gencgc.c", line 4553". Attached is the backtrace. I'm running with the following config: Elephant 0.6.0 cl-base64 3.3.2 uffi 1.5.17 BerkeleyDB 4.3 SBCL 0.9.18 (also tried with 0.9.17) SUSE Linux 10.1 AMD64. I have started reading the archived threads and notice that this may be something related to running on x86_64? Any ideas? Regards Joubert ------------------------------------------------------- -------------- next part -------------- Heap exhausted: 8522182656 bytes available, 455266533424 requested. PROCEED WITH CAUTION! [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8522182656 455266533424) 1: ("foreign function: call_into_lisp") 2: ("foreign function: funcall2") 3: ("foreign function: gc_heap_exhausted_error_or_lose") 4: ("foreign function: gc_find_freeish_pages") 5: ("foreign function: gc_alloc_large") 6: ("foreign function: alloc_tramp") 7: ((LABELS %DESERIALIZE) #) 8: ((SB-PCL::FAST-METHOD GET-VALUE (T SLEEPYCAT::BDB-BTREE)) # # # #) 9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (GET-FROM-ROOT "first") #) 10: (SWANK::EVAL-REGION "(get-from-root \"first\") " T) 11: ((LAMBDA NIL)) 12: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-SYNTAX-HOOKS (T)) # # #) 13: (SWANK::CALL-WITH-BUFFER-SYNTAX #) 14: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(get-from-root \"first\") ") #) 15: ((LAMBDA NIL)) 16: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 17: ((LAMBDA NIL)) 18: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 19: (SWANK::CALL-WITH-REDIRECTED-IO # #) 20: (SWANK::CALL-WITH-CONNECTION # #) 21: (SWANK::HANDLE-REQUEST #) 22: ((LAMBDA NIL)) 23: ((LAMBDA NIL)) 24: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 25: (SWANK::CALL-WITH-REDIRECTED-IO # #) 26: (SWANK::CALL-WITH-CONNECTION # #) 27: (SWANK::CALL-WITH-BINDINGS NIL #) 28: ((LAMBDA NIL)) 29: ("foreign function: call_into_lisp") 30: ("foreign function: new_thread_trampoline") From joubert at joubster.com Sun Oct 29 21:38:45 2006 From: joubert at joubster.com (Joubert Nel) Date: Sun, 29 Oct 2006 17:38:45 -0400 Subject: [elephant-devel] (open-store) (close-store) (open-store) fails Message-ID: <200610291638.45595.joubert@joubster.com> Hi, I've noticed that when I call (open-store *my-storage*), then (close-store), then (open-store *my-storage*) again, my LISP simply hangs. From within SLIME/Emacs I don't get any feedback but when I run directly in SBCL I get the following error message after the 2nd (open-store): "fatal error encountered in SBCL. GC invariant lost, file "gencgc.c", line 4553". Attached are two screenshots - one from SBCL and one from SLIME/Emacs. I'm running with the following config: Elephant 0.6.0 cl-base64 3.3.2 uffi 1.5.17 BerkeleyDB 4.3 SBCL 0.9.17 SUSE Linux 10.1 AMD64. I have started reading the archived threads and notice that this may be something related to running on x86_64? Any ideas? Regards Joubert -------------- next part -------------- Heap exhausted: 8522182656 bytes available, 455266533424 requested. PROCEED WITH CAUTION! [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8522182656 455266533424) 1: ("foreign function: call_into_lisp") 2: ("foreign function: funcall2") 3: ("foreign function: gc_heap_exhausted_error_or_lose") 4: ("foreign function: gc_find_freeish_pages") 5: ("foreign function: gc_alloc_large") 6: ("foreign function: alloc_tramp") 7: ((LABELS %DESERIALIZE) #) 8: ((SB-PCL::FAST-METHOD GET-VALUE (T SLEEPYCAT::BDB-BTREE)) # # # #) 9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (GET-FROM-ROOT "first") #) 10: (SWANK::EVAL-REGION "(get-from-root \"first\") " T) 11: ((LAMBDA NIL)) 12: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-SYNTAX-HOOKS (T)) # # #) 13: (SWANK::CALL-WITH-BUFFER-SYNTAX #) 14: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(get-from-root \"first\") ") #) 15: ((LAMBDA NIL)) 16: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 17: ((LAMBDA NIL)) 18: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 19: (SWANK::CALL-WITH-REDIRECTED-IO # #) 20: (SWANK::CALL-WITH-CONNECTION # #) 21: (SWANK::HANDLE-REQUEST #) 22: ((LAMBDA NIL)) 23: ((LAMBDA NIL)) 24: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 25: (SWANK::CALL-WITH-REDIRECTED-IO # #) 26: (SWANK::CALL-WITH-CONNECTION # #) 27: (SWANK::CALL-WITH-BINDINGS NIL #) 28: ((LAMBDA NIL)) 29: ("foreign function: call_into_lisp") 30: ("foreign function: new_thread_trampoline") -------------- next part -------------- A non-text attachment was scrubbed... Name: elephant-open-store-fails-in-sbcl.png Type: image/png Size: 47391 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: elephant-open-store-fails.png Type: image/png Size: 10215 bytes Desc: not available URL: From joubert at joubster.com Sun Oct 29 22:15:59 2006 From: joubert at joubster.com (Joubert Nel) Date: Sun, 29 Oct 2006 18:15:59 -0400 Subject: [elephant-devel] (open-store) (close-store) (open-store) fails - running x86_64 on SBCL Message-ID: <200610291715.59236.joubert@joubster.com> Hi, I've noticed that when I call (open-store *my-storage*), then (close-store), then (open-store *my-storage*) again, my LISP simply hangs. From within SLIME/Emacs I don't get any feedback but when I run directly in SBCL I get the following error message after the 2nd (open-store): "fatal error encountered in SBCL. GC invariant lost, file "gencgc.c", line 4553". Attached are two screenshots - one from SBCL and one from SLIME/Emacs. I'm running with the following config: Elephant 0.6.0 cl-base64 3.3.2 uffi 1.5.17 BerkeleyDB 4.3 SBCL 0.9.17 SUSE Linux 10.1 AMD64. I have started reading the archived threads and notice that this may be something related to running on x86_64? Any ideas? Regards Joubert -------------- next part -------------- Heap exhausted: 8522182656 bytes available, 455266533424 requested. PROCEED WITH CAUTION! [Condition of type SB-KERNEL::HEAP-EXHAUSTED-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-KERNEL::HEAP-EXHAUSTED-ERROR 8522182656 455266533424) 1: ("foreign function: call_into_lisp") 2: ("foreign function: funcall2") 3: ("foreign function: gc_heap_exhausted_error_or_lose") 4: ("foreign function: gc_find_freeish_pages") 5: ("foreign function: gc_alloc_large") 6: ("foreign function: alloc_tramp") 7: ((LABELS %DESERIALIZE) #) 8: ((SB-PCL::FAST-METHOD GET-VALUE (T SLEEPYCAT::BDB-BTREE)) # # # #) 9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (GET-FROM-ROOT "first") #) 10: (SWANK::EVAL-REGION "(get-from-root \"first\") " T) 11: ((LAMBDA NIL)) 12: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-SYNTAX-HOOKS (T)) # # #) 13: (SWANK::CALL-WITH-BUFFER-SYNTAX #) 14: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(get-from-root \"first\") ") #) 15: ((LAMBDA NIL)) 16: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 17: ((LAMBDA NIL)) 18: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 19: (SWANK::CALL-WITH-REDIRECTED-IO # #) 20: (SWANK::CALL-WITH-CONNECTION # #) 21: (SWANK::HANDLE-REQUEST #) 22: ((LAMBDA NIL)) 23: ((LAMBDA NIL)) 24: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 25: (SWANK::CALL-WITH-REDIRECTED-IO # #) 26: (SWANK::CALL-WITH-CONNECTION # #) 27: (SWANK::CALL-WITH-BINDINGS NIL #) 28: ((LAMBDA NIL)) 29: ("foreign function: call_into_lisp") 30: ("foreign function: new_thread_trampoline") -------------- next part -------------- A non-text attachment was scrubbed... Name: elephant-open-store-fails-in-sbcl.png Type: image/png Size: 47391 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: elephant-open-store-fails.png Type: image/png Size: 10215 bytes Desc: not available URL: From mb at bese.it Mon Oct 30 10:38:49 2006 From: mb at bese.it (Marco Baringer) Date: Mon, 30 Oct 2006 11:38:49 +0100 Subject: [elephant-devel] Re: (open-store) (close-store) (open-store) fails - running x86_64 on SBCL References: <200610291715.59236.joubert@joubster.com> Message-ID: Joubert Nel writes: > I have started reading the archived threads and notice that this may be > something related to running on x86_64? it's due to a bug in the serializer and ffi assuming ints are 32bits. i have a patched version of elephant, which avoids 64bit datums, it runs but it doesn't take advantage of the larger int size. -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen From joubert at joubster.com Mon Oct 30 23:48:50 2006 From: joubert at joubster.com (Joubert Nel) Date: Mon, 30 Oct 2006 18:48:50 -0500 Subject: [elephant-devel] Re: (open-store) (close-store) (open-store) fails - running x86_64 on SBCL In-Reply-To: References: <200610291715.59236.joubert@joubster.com> Message-ID: <200610301848.50484.joubert@joubster.com> On Monday 30 October 2006 05:38, Marco Baringer wrote: > Joubert Nel writes: > > I have started reading the archived threads and notice that this may be > > something related to running on x86_64? > > it's due to a bug in the serializer and ffi assuming ints are > 32bits. i have a patched version of elephant, which avoids 64bit > datums, it runs but it doesn't take advantage of the larger int size. Marco - is this patch available for download? I'd be interested to see the source code for all the changes necessary. Joubert From nowhere.man at levallois.eu.org Tue Oct 31 13:43:44 2006 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Tue, 31 Oct 2006 14:43:44 +0100 Subject: [elephant-devel] Concurrency problem? Message-ID: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> I may have a problem with concurrency and elephant. I'm developing a web application using SBCL and Araneida (with it's threaded listener). Whereas araneida by irself doesn't seem to have problems dealing with many requests at a time (I tested it with apache's benckmark tool, with thousands of request, with from 1 to 16 concurrent requests to see how it reacts), my application just collapses when 3 requests come in a very short time. I had an error I did not copied, which was about a persistent slot of an object being unbound (but I know it should be bound when accessed at this time), and when trying to access one more time my app, I end up with this in the debugger: While accessing database # with expression "SELECT value FROM keyvalue WHERE ((clctn_id = 0) and (key = 'FSAAAAAyAAAANAAAACAAAABUAAAASQAAAFQAAABSAAAARQAAAA=='))": Error NIL / has occurred. [Condition of type CLSQL-SYS:SQL-DATABASE-DATA-ERROR] The database itself doesn't seem corrupted, as restarting the application from scratch, with the same store, works well. For the moment, I had to shield my app behind an HTTP cache so that requests are basicaly served to the user by the cache instead of my app... Concurrently, Nowhere man -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From mb at bese.it Tue Oct 31 14:33:40 2006 From: mb at bese.it (Marco Baringer) Date: Tue, 31 Oct 2006 15:33:40 +0100 Subject: [elephant-devel] Re: (open-store) (close-store) (open-store) fails - running x86_64 on SBCL References: <200610291715.59236.joubert@joubster.com> <200610301848.50484.joubert@joubster.com> Message-ID: Joubert Nel writes: > Marco - is this patch available for download? I'd be interested to see the > source code for all the changes necessary. http://www.bese.it/~segv/elephant-0.6.MARCO_RETRY.tar.bz2 sorry, i don't have the time to create a patch. most of the modifications went to the serializer and to to the ffi (i added a uint32 type and used that instead of integer). -- -Marco Ring the bells that still can ring. Forget your perfect offering. There is a crack in everything. That's how the light gets in. -Leonard Cohen From read at robertlread.net Tue Oct 31 14:54:42 2006 From: read at robertlread.net (Robert L. Read) Date: Tue, 31 Oct 2006 08:54:42 -0600 Subject: [elephant-devel] Concurrency problem? In-Reply-To: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> References: <20061031134344.GU7585@bateleur.arcanes.fr.eu.org> Message-ID: <1162306483.14560.285.camel@localhost.localdomain> I have had similar problems. My own solution has been to put mutual exclusion locks into DCM. Since my own app only accesses the database through DCM, this works for me; I think it would be inconvenient but not impossible if you are using Elephant directly. DCM is a "director-based" system that is in the contrib directory, that aims more toward a prevalence based system. There are/were at least three problems that have to be dealt with: 1) The serializer is definitely not thread-safe. In the next release, we need to improve this, but for the time being I have added mutexes (these are not checked in yet.) 2) CLSQL is not threadsafe unless you use a separate connection in each thread. Elephant's basic use of CLSQL reuses the connections, and so you can have big problems even at the CLSQL level in using Elephant in that way. So, based on what you have described, I can suggest a few solutions, neither of which are great: 1) If your access to Elephant is in some relatively narrow place, that is, you can find a few pieces of code where all of the elephant requests occur, you can put mutexes around those. (Code to define and use these mutexes is included at the end of this message. It defines a macro, "defemthodex", which is just like defmethod but takes an extra argument used to define the mutex. However, this is HIGHLY SBCL specific, and would not be trivial to adapt to anything other than DCM --- this is (in additio to being busy) why I have not put the thread-safe version of DCM in the contrib directory yet. It is, however, an example of how to use mutexes in SBCL, which you can easily copy.) 2) Depending on the demands of your application, you might be able to put in a smaller number of higher-level mutexes (that is, that operate around larger bodies of code), and still serve pages sufficiently rapidly. 3) If you use BDB, it will seem better, but the current version of Elephant is still not thread-safe, because of the serializer, although it will take you a lot longer to hit a problem than under CLSQL. I think really Ian and I, and whoever else is interested, need to plan on solving this problem in the next major release. To my knoledge, this would mean: 1) Writing a small connection-pool manager that is threadsafe in Elephant, 2) Making a reentrant serializer. (I know Ian is planning this already - AMEN!) 3) I'd be happy to publish my current version of DCM to anyone who wants it, but it is SBCL specific, and I personally don't know how to do mutexes under any other LISP. * * * I know that may not be terribly helpful ---- but let's continue this discussion until we work out a solution. I am extraordinarily busy launching my business, but am committed to at least managing patches provided by others or getting out of the way if someone else (such as Ian) wants to be the official maintainer. (defvar *dcm-mutexes* (make-hash-table :test 'equal)) (defvar *a-mutex* (sb-thread::make-mutex :name "my lock")) (defun insure-mutex (name) (let ((mtx (gethash name *dcm-mutexes*)) ) (or mtx (setf (gethash name *dcm-mutexes*) (sb-thread:make- mutex :name name))) ) ) ;; This assumes that the the variable "dir" is being defined and that we can can ;; create (defmacro defmethodex (mname dir args &body body) `(defmethod ,mname ,(cons dir args) ;; (format t "Thread ~A running ~%" sb-thread::*current-thread*) (sb-thread:with-mutex ((insure-mutex (format nil "mutex-~A" ,(car dir)))) ;; (format t "Thread ~A got the lock~%" sb-thread::*current- thread*) (let ((ret , at body)) ;; (format t "Thread ~A dropping lock~%" sb-thread::*current- thread*) ret ) ) ) ) On Tue, 2006-10-31 at 14:43 +0100, Pierre THIERRY wrote: > I may have a problem with concurrency and elephant. I'm developing a web > application using SBCL and Araneida (with it's threaded listener). > Whereas araneida by irself doesn't seem to have problems dealing with > many requests at a time (I tested it with apache's benckmark tool, with > thousands of request, with from 1 to 16 concurrent requests to see how > it reacts), my application just collapses when 3 requests come in a very > short time. > > I had an error I did not copied, which was about a persistent slot of an > object being unbound (but I know it should be bound when accessed at > this time), and when trying to access one more time my app, I end up > with this in the debugger: > > While accessing database # > with expression "SELECT value FROM keyvalue WHERE ((clctn_id = 0) and (key = 'FSAAAAAyAAAANAAAACAAAABUAAAASQAAAFQAAABSAAAARQAAAA=='))": > Error NIL / > has occurred. > [Condition of type CLSQL-SYS:SQL-DATABASE-DATA-ERROR] > > The database itself doesn't seem corrupted, as restarting the > application from scratch, with the same store, works well. For the > moment, I had to shield my app behind an HTTP cache so that requests are > basicaly served to the user by the cache instead of my app... > > Concurrently, > Nowhere man > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: