From fungsin.lui at gmail.com Sat May 5 06:30:18 2007 From: fungsin.lui at gmail.com (Lui Fungsin) Date: Fri, 4 May 2007 23:30:18 -0700 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> Message-ID: <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> Hi Ian, Congrats on the 0.9 RC1 release. Good news : the MOP error is gone with elephant 0.9 Now I can successfully do all these: (asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-bdb) (asdf:operate 'asdf:load-op :elephant-tests) (in-package :ele-tests) (setf *default-spec* *testbdb-spec*) Bad news is that I still can't get elephant to work out of the box: ELE-TESTS 8 > (do-backend-tests) lispworks-5-0-0-x86-linux: symbol lookup error: src/db-bdb/libberkeley-db.so: undefined symbol: db_env_create ;; lispworks exits abnormally So I spent sometime to diagnose this. Here're my findings: I went through the asdf compilation log and found that this looks suspicious: $ gcc -L/usr/local/BerkeleyDB.4.5/lib/ -I/usr/local/BerkeleyDB.4.5/include/ -shared -fPIC -Wall -O2 -g src/db-bdb/libberkeley-db.c -o src/db-bdb/libberkeley-db.so -lm $ ldd src/db-bdb/libberkeley-db.so linux-gate.so.1 => (0x009fb000) libm.so.6 => /lib/libm.so.6 (0x00110000) libc.so.6 => /lib/libc.so.6 (0x00db5000) /lib/ld-linux.so.2 (0x0085f000) I then recompiled the file with a -ldb flag $ gcc -L/usr/local/BerkeleyDB.4.5/lib/ -I/usr/local/BerkeleyDB.4.5/include/ -shared -fPIC -Wall -O2 -g src/db-bdb/libberkeley-db.c -o src/db-bdb/libberkeley-db.so -lm -ldb $ export LD_LIBRARY_PATH=/usr/local/BerkeleyDB.4.5/lib/ $ ldd src/db-bdb/libberkeley-db.so linux-gate.so.1 => (0x00d71000) libm.so.6 => /lib/libm.so.6 (0x00d43000) libdb-4.5.so => /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so (0x00918000) libc.so.6 => /lib/libc.so.6 (0x00b43000) /lib/ld-linux.so.2 (0x0085f000) libpthread.so.0 => /lib/libpthread.so.0 (0x00f9c000) After that lispworks successfully ran (do-backend-tests) and passed all the tests. How do you get it to work without these tweaks? fungsin From eslick at csail.mit.edu Sat May 5 10:59:25 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 5 May 2007 06:59:25 -0400 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> Message-ID: <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> I noticed that the .asd file has a line: #-(or linux) (list (format nil "-l~A" (get-db-name c))))) To keep linux builds from adding the library link command ( get-db- name will extract the db-4.5 or db45 name of the BDB library that you provide in the configuration ). I don't remember why this was put in and am not a regular user of linux. If you remove the reader conditional, does it work for you? Ian On May 5, 2007, at 2:30 AM, Lui Fungsin wrote: > Hi Ian, > > Congrats on the 0.9 RC1 release. > > Good news : the MOP error is gone with elephant 0.9 > > Now I can successfully do all these: > > (asdf:operate 'asdf:load-op :elephant) > (asdf:operate 'asdf:load-op :ele-bdb) > (asdf:operate 'asdf:load-op :elephant-tests) > (in-package :ele-tests) > (setf *default-spec* *testbdb-spec*) > > Bad news is that I still can't get elephant to work out of the box: > > ELE-TESTS 8 > (do-backend-tests) > lispworks-5-0-0-x86-linux: symbol lookup error: > src/db-bdb/libberkeley-db.so: undefined symbol: db_env_create > ;; lispworks exits abnormally > > > So I spent sometime to diagnose this. Here're my findings: > > I went through the asdf compilation log and found that this looks > suspicious: > > $ gcc -L/usr/local/BerkeleyDB.4.5/lib/ > -I/usr/local/BerkeleyDB.4.5/include/ -shared -fPIC -Wall -O2 -g > src/db-bdb/libberkeley-db.c -o src/db-bdb/libberkeley-db.so -lm > > $ ldd src/db-bdb/libberkeley-db.so > linux-gate.so.1 => (0x009fb000) > libm.so.6 => /lib/libm.so.6 (0x00110000) > libc.so.6 => /lib/libc.so.6 (0x00db5000) > /lib/ld-linux.so.2 (0x0085f000) > > I then recompiled the file with a -ldb flag > > $ gcc -L/usr/local/BerkeleyDB.4.5/lib/ > -I/usr/local/BerkeleyDB.4.5/include/ -shared -fPIC -Wall -O2 -g > src/db-bdb/libberkeley-db.c -o src/db-bdb/libberkeley-db.so -lm -ldb > > $ export LD_LIBRARY_PATH=/usr/local/BerkeleyDB.4.5/lib/ > > $ ldd src/db-bdb/libberkeley-db.so > linux-gate.so.1 => (0x00d71000) > libm.so.6 => /lib/libm.so.6 (0x00d43000) > libdb-4.5.so => /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so > (0x00918000) > libc.so.6 => /lib/libc.so.6 (0x00b43000) > /lib/ld-linux.so.2 (0x0085f000) > libpthread.so.0 => /lib/libpthread.so.0 (0x00f9c000) > > After that lispworks successfully ran (do-backend-tests) and passed > all the tests. > > How do you get it to work without these tweaks? > > fungsin > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From fungsin.lui at gmail.com Sat May 5 19:45:50 2007 From: fungsin.lui at gmail.com (Lui Fungsin) Date: Sat, 5 May 2007 12:45:50 -0700 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> Message-ID: <3990b5930705051245g68b63cafu9aea784d193e2b0c@mail.gmail.com> On 5/5/07, Ian Eslick wrote: > If you remove the reader conditional, does it > work for you? Yes, that fixed the problem. I also verified that this change doesn't break SBCL by removing all the fasl and so files and rerun the bdb test. > I don't remember why this was put in and am not a > regular user of linux. That's where a comment would be useful ;-) Thanks for spending the time to get it work on lispworks linux. fungsin From emailmac at gmail.com Sat May 5 21:56:06 2007 From: emailmac at gmail.com (Mac Chan) Date: Sat, 5 May 2007 14:56:06 -0700 Subject: [elephant-devel] (pushnew :elephant *features*) Message-ID: <4877ae640705051456x5584498apb4b564b244b687d0@mail.gmail.com> Hi, Could you add (pushnew :elephant *features*) somewhere in the source? This way I can easily comment out a block of code when elephant is not available. Thanks. (defun start () #+elephant (progn (ensure-directories-exist (second *blog-store-spec*)) (open-store *blog-store-spec*)) ......... .........) From eslick at csail.mit.edu Sat May 5 22:20:54 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 5 May 2007 18:20:54 -0400 Subject: [elephant-devel] (pushnew :elephant *features*) In-Reply-To: <4877ae640705051456x5584498apb4b564b244b687d0@mail.gmail.com> References: <4877ae640705051456x5584498apb4b564b244b687d0@mail.gmail.com> Message-ID: <9D4A9066-7A46-4EBC-A25B-ACD2167C239E@csail.mit.edu> Mac, I'll add that to the next incremental release (0.9.1). Can you submit a ticket on the Trac db so I remember? Ian On May 5, 2007, at 5:56 PM, Mac Chan wrote: > Hi, > > Could you add (pushnew :elephant *features*) somewhere in the source? > > This way I can easily comment out a block of code when elephant is not > available. > > Thanks. > > (defun start () > #+elephant > (progn > (ensure-directories-exist (second *blog-store-spec*)) > (open-store *blog-store-spec*)) > ......... > .........) > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Sat May 5 22:22:31 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 5 May 2007 18:22:31 -0400 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: <3990b5930705051245g68b63cafu9aea784d193e2b0c@mail.gmail.com> References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> <3990b5930705051245g68b63cafu9aea784d193e2b0c@mail.gmail.com> Message-ID: What linux distribution are you using? My SBCL/Linux setup worked both with and without the -ldb-4.5 option. I think I, or someone else, put that in because it was causing problems on some other linux build. Ian On May 5, 2007, at 3:45 PM, Lui Fungsin wrote: > On 5/5/07, Ian Eslick wrote: >> If you remove the reader conditional, does it >> work for you? > > Yes, that fixed the problem. I also verified that this change doesn't > break SBCL by removing all the fasl and so files and rerun the bdb > test. > >> I don't remember why this was put in and am not a >> regular user of linux. > > That's where a comment would be useful ;-) > > Thanks for spending the time to get it work on lispworks linux. > > fungsin > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Sat May 5 22:24:03 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 5 May 2007 18:24:03 -0400 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> <3990b5930705051245g68b63cafu9aea784d193e2b0c@mail.gmail.com> Message-ID: PS - I've had several people load the current HEAD and run on linux/ sbcl with the same positive result. So far you are the only failure. I'll test Lispworks under Linux later and see if I can reproduce the problem, but it worked fine for me before I released RC1. Do you have your library path setup correctly in your my- config.sexp? Ian On May 5, 2007, at 6:22 PM, Ian Eslick wrote: > What linux distribution are you using? My SBCL/Linux setup worked > both with and without the > -ldb-4.5 option. I think I, or someone else, put that in because > it was causing problems on some other linux build. > > Ian > > On May 5, 2007, at 3:45 PM, Lui Fungsin wrote: > >> On 5/5/07, Ian Eslick wrote: >>> If you remove the reader conditional, does it >>> work for you? >> >> Yes, that fixed the problem. I also verified that this change doesn't >> break SBCL by removing all the fasl and so files and rerun the bdb >> test. >> >>> I don't remember why this was put in and am not a >>> regular user of linux. >> >> That's where a comment would be useful ;-) >> >> Thanks for spending the time to get it work on lispworks linux. >> >> fungsin >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From emailmac at gmail.com Sat May 5 22:30:54 2007 From: emailmac at gmail.com (Mac Chan) Date: Sat, 5 May 2007 15:30:54 -0700 Subject: [elephant-devel] export #:db-error ? Message-ID: <4877ae640705051530x20f25acr5913fd02fce24369@mail.gmail.com> I noticed a warning "DB-ERROR is not a valid type" when I compile the blog-example. (handler-case (with-transaction () (make-instance 'blog-entry :date date :title title :text text)) (db-error () nil)) It so happens that I still have the old elephant source code (I simply untar the new code into the same directory) When I do a grep on all the source I find that in the original src/sleepycat.lisp it exports a bunch of symbol which includes #:db-error Is this symbol supposed to be exported? Thanks. From emailmac at gmail.com Sat May 5 23:41:34 2007 From: emailmac at gmail.com (Mac Chan) Date: Sat, 5 May 2007 16:41:34 -0700 Subject: [elephant-devel] Re: export #:db-error ? In-Reply-To: <4877ae640705051530x20f25acr5913fd02fce24369@mail.gmail.com> References: <4877ae640705051530x20f25acr5913fd02fce24369@mail.gmail.com> Message-ID: <4877ae640705051641s299c6660t5496234ec35b68a7@mail.gmail.com> So in the current source code DB-ERROR is wrap around with (eval-when (:compile-toplevel :load-toplevel)) I guess the question is whether the application programmer need to guard against database error when using any of the db operation api. Thanks. On 5/5/07, Mac Chan wrote: > I noticed a warning "DB-ERROR is not a valid type" when I compile the > blog-example. > > (handler-case > (with-transaction () > (make-instance 'blog-entry > :date date > :title title > :text text)) > (db-error () nil)) > > It so happens that I still have the old elephant source code (I simply > untar the new code into the same directory) > > When I do a grep on all the source I find that in the original > src/sleepycat.lisp it exports a bunch of symbol which includes > #:db-error > > Is this symbol supposed to be exported? > > Thanks. > From eslick at csail.mit.edu Sat May 5 23:45:36 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 5 May 2007 19:45:36 -0400 Subject: [elephant-devel] export #:db-error ? In-Reply-To: <4877ae640705051530x20f25acr5913fd02fce24369@mail.gmail.com> References: <4877ae640705051530x20f25acr5913fd02fce24369@mail.gmail.com> Message-ID: Good eye. There is an outstanding design issue which is reviewing the condition/error architecture and trying to clean it up. This is one example of what needs to be done. I've exported this from the elephant package in the 0.9.1 source tree. The examples have not been revisited for 0.9 so it is a good idea to go in and clean them up for 0.9.1. I've submitted a reminder ticket for that. Ian On May 5, 2007, at 6:30 PM, Mac Chan wrote: > I noticed a warning "DB-ERROR is not a valid type" when I compile the > blog-example. > > (handler-case > (with-transaction () > (make-instance 'blog-entry > :date date > :title title > :text text)) > (db-error () nil)) > > It so happens that I still have the old elephant source code (I simply > untar the new code into the same directory) > > When I do a grep on all the source I find that in the original > src/sleepycat.lisp it exports a bunch of symbol which includes > #:db-error > > Is this symbol supposed to be exported? > > Thanks. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Sat May 5 23:48:45 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 5 May 2007 19:48:45 -0400 Subject: [elephant-devel] Re: export #:db-error ? In-Reply-To: <4877ae640705051641s299c6660t5496234ec35b68a7@mail.gmail.com> References: <4877ae640705051530x20f25acr5913fd02fce24369@mail.gmail.com> <4877ae640705051641s299c6660t5496234ec35b68a7@mail.gmail.com> Message-ID: db-error is a general condition which says there was an internal error in the db (something that is not likely the fault of the user). I've started adding other conditions that are problem- specific and refer to things the user has done in error. As part of cleanup, they should inherit from a common source so you can inhibit / handle all elephant conditions as necessary. Please add comments to an existing ticket or a new tickets to describe your thoughts about what should be done here. Ian On May 5, 2007, at 7:41 PM, Mac Chan wrote: > So in the current source code DB-ERROR is wrap around with (eval-when > (:compile-toplevel :load-toplevel)) > > I guess the question is whether the application programmer need to > guard against database error when using any of the db operation api. > > Thanks. > > On 5/5/07, Mac Chan wrote: >> I noticed a warning "DB-ERROR is not a valid type" when I compile the >> blog-example. >> >> (handler-case >> (with-transaction () >> (make-instance 'blog-entry >> :date date >> :title title >> :text text)) >> (db-error () nil)) >> >> It so happens that I still have the old elephant source code (I >> simply >> untar the new code into the same directory) >> >> When I do a grep on all the source I find that in the original >> src/sleepycat.lisp it exports a bunch of symbol which includes >> #:db-error >> >> Is this symbol supposed to be exported? >> >> Thanks. >> > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From fungsin.lui at gmail.com Sun May 6 00:07:25 2007 From: fungsin.lui at gmail.com (Lui Fungsin) Date: Sat, 5 May 2007 17:07:25 -0700 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> <3990b5930705051245g68b63cafu9aea784d193e2b0c@mail.gmail.com> Message-ID: <3990b5930705051707r2f4a5bfdk5042c01b6c57a4e9@mail.gmail.com> On 5/5/07, Ian Eslick wrote: > PS - I've had several people load the current HEAD and run on linux/ > sbcl with the same positive result. So far you are the only > failure. I also don't have any problem with SBCL at all. I'm running fedora core 6, SBCL 1.0.3 and lispworks 5.0.2. BTW, I found out that the MOP error only happens with lispworks 5.0.1. I should have checked lispwork's website first, it turns out that 5.0.2 has been out for a while. Sorry for the false alarm. > I'll test Lispworks under Linux later and see if I can > reproduce the problem, but it worked fine for me before I released > RC1. Do you have your library path setup correctly in your my- > config.sexp? I use henrik's install-bdb.sh script to install the latest bdb and it creates my-config.sexp automatically. Remember without the -ldb flag, ldd shows that $ ldd src/db-bdb/libberkeley-db.so linux-gate.so.1 => (0x009fb000) libm.so.6 => /lib/libm.so.6 (0x00110000) libc.so.6 => /lib/libc.so.6 (0x00db5000) /lib/ld-linux.so.2 (0x0085f000) > (asdf:operate 'asdf:load-op :elephant) > (asdf:operate 'asdf:load-op :ele-bdb) The elephant.asd loads both (uffi:load-foreign-library "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so" :module "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so") and (uffi:load-foreign-library "src/db-bdb/libberkeley-db.so" :module "src/db-bdb/libberkeley-db.so") To verify that libdb is loaded, I type the following in the REPL > (uffi:def-function ("db_strerror" test-db-strerror) ((error :int)) :returning :cstring) > (test-db-strerror 1) "Operation not permitted" Now in src/db-bdb/libberkeley-db.c there's a wrapper function db_strerr which call the above db_strerror. If I tried this in the REPL > (%db-strerror 1) lispworks-5-0-0-x86-linux: symbol lookup error: src/db-bdb/libberkeley-db.so: undefined symbol: db_strerror So even though both "*.so" are loaded with load-foreign-library, libberkeley-db.so cannot access functions in libdb-4.5.so Now with the -ldb flag $ ldd src/db-bdb/libberkeley-db.so linux-gate.so.1 => (0x00d71000) libm.so.6 => /lib/libm.so.6 (0x00d43000) libdb-4.5.so => /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so (0x00918000) libc.so.6 => /lib/libc.so.6 (0x00b43000) /lib/ld-linux.so.2 (0x0085f000) libpthread.so.0 => /lib/libpthread.so.0 (0x00f9c000) By the time libberkeley-db.so is loaded, it should pull in libdb-4.5.so automatically, and the above test will succeed. I really don't have much knowledge of how SBCL/lispworks implement their FFI load-library function, but it seems that for SBCL, merely using load-foreign-library is enough. Someone might have a clue of what's going on. From MAILER-DAEMON at common-lisp.net Mon May 7 11:24:18 2007 From: MAILER-DAEMON at common-lisp.net (Returned mail) Date: Mon, 7 May 2007 13:24:18 +0200 Subject: [elephant-devel] [SPAM] Returned mail: Data format error Message-ID: <20070507112418.A0D0B2823B@common-lisp.net> An embedded and charset-unspecified text was scrubbed... Name: not available URL: -------------- next part -------------- An embedded message was scrubbed... From: "Returned mail" Subject: Returned mail: Data format error Date: Mon, 7 May 2007 13:24:18 +0200 Size: 41199 URL: From lists at infoway.net Mon May 7 18:28:48 2007 From: lists at infoway.net (lists at infoway.net) Date: Mon, 7 May 2007 14:28:48 -0400 (EDT) Subject: [elephant-devel] Problem loading latest CVS version on SBCL MacBook Intel Message-ID: <38833.192.168.1.71.1178562528.webmail@192.168.1.71> I just downloaded the latest CVS version and elephant is not loading properly. The error message I get is: (defun pprev-dup-hack (cur key) " ; compiling GO ; compiling BACK ; compiling ONE ; compiling STEP ; compiling IN ; compiling A ; compiling DUPLICATE ; compiling SET; compilation aborted because of fatal error: ; READ failure in COMPILE-FILE: ; READER-ERROR at 19745 (line 559, column 39) on #: ; comma not inside a backquote ; /Users/dev/lisp/elephant/src/elephant/collections.fasl written ; compilation finished in 0:00:00 WARNING: COMPILE-FILE warned while performing # on #. The original function in collections.lisp (line 558) looks like this: (defun pprev-dup-hack (cur key) "Go back one step in a duplicate set, returns nil if previous element is a different key. More efficient than the current default implementation of cursor-pprev-dup" (multiple-value-bind (exists? skey value pkey) (cursor-pprev cur) (when (lisp-compare-equal key skey) (values exists? key value pkey)))) I changed the last line to: (values exists? skey value pkey)))) and it now seems to load. However, now I'm getting this after running all tests: 10 out of 131 total tests failed: MAP-INDEXED-INDEX, MAP-INDEX-FROM-END, INDEXING-BASIC, INDEXING-RANGE, INDEXING-SLOT-MAKUNBOUND, INDEXING-WIPE-INDEX, INDEXING-RECONNECT-DB, INDEXING-CHANGE-CLASS, INDEXING-REDEF-CLASS, INDEXING-TIMING. Whereas before, I was not getting any test failures. Is this due to the typo in collections.lisp or something else? Thanks, Daniel From lists at infoway.net Mon May 7 18:33:48 2007 From: lists at infoway.net (lists at infoway.net) Date: Mon, 7 May 2007 14:33:48 -0400 (EDT) Subject: [elephant-devel] Problem loading latest CVS version on SBCL MacBook Intel Message-ID: <51402.192.168.1.70.1178562828.webmail@192.168.1.70> I just realized that what I "corrected" was not necessarily a typo. At this point, simply ignore my "fix", however, if anyone could shed some light as to why it's breaking I would appreciate it. I will try to debug it myself as well. Thanks, Daniel On Mon, May 7, 2007 2:28 pm, lists at infoway.net said: > I just downloaded the latest CVS version and elephant is not loading properly. > > The error message I get is: > > (defun pprev-dup-hack (cur key) > " > ; compiling GO > ; compiling BACK > ; compiling ONE > ; compiling STEP > ; compiling IN > ; compiling A > ; compiling DUPLICATE > ; compiling SET; compilation aborted because of fatal error: > ; READ failure in COMPILE-FILE: > ; READER-ERROR at 19745 (line 559, column 39) on > # /Users/dev/lisp/elephant/src/elephant/collections.lisp" > {1239E0D9}>: > ; comma not inside a backquote > > ; /Users/dev/lisp/elephant/src/elephant/collections.fasl written > ; compilation finished in 0:00:00 > WARNING: > COMPILE-FILE warned while performing # on > #. > > > The original function in collections.lisp (line 558) looks like this: > > (defun pprev-dup-hack (cur key) > "Go back one step in a duplicate set, returns nil > if previous element is a different key. More efficient than > the current default implementation of cursor-pprev-dup" > (multiple-value-bind (exists? skey value pkey) > (cursor-pprev cur) > (when (lisp-compare-equal key skey) > (values exists? key value pkey)))) > > > I changed the last line to: > (values exists? skey value pkey)))) > > and it now seems to load. However, now I'm getting this after running all tests: > > 10 out of 131 total tests failed: MAP-INDEXED-INDEX, MAP-INDEX-FROM-END, > INDEXING-BASIC, INDEXING-RANGE, INDEXING-SLOT-MAKUNBOUND, > INDEXING-WIPE-INDEX, INDEXING-RECONNECT-DB, INDEXING-CHANGE-CLASS, > INDEXING-REDEF-CLASS, INDEXING-TIMING. > > Whereas before, I was not getting any test failures. Is this due to the typo in > collections.lisp or something else? > > Thanks, > Daniel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From eslick at csail.mit.edu Mon May 7 18:39:24 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 7 May 2007 14:39:24 -0400 Subject: [elephant-devel] Problem loading latest CVS version on SBCL MacBook Intel In-Reply-To: <38833.192.168.1.71.1178562528.webmail@192.168.1.71> References: <38833.192.168.1.71.1178562528.webmail@192.168.1.71> Message-ID: <5D0F65A3-ECA2-4F59-81B9-A15A768A68A5@csail.mit.edu> This was my error. I was sloppy in promoting a documentation clarification. This should now be fixed and all tests pass for me locally. Let me know if you still have problems with the latest CVS HEAD. Ian On May 7, 2007, at 2:28 PM, lists at infoway.net wrote: > I just downloaded the latest CVS version and elephant is not > loading properly. > > The error message I get is: > > (defun pprev-dup-hack (cur key) > " > ; compiling GO > ; compiling BACK > ; compiling ONE > ; compiling STEP > ; compiling IN > ; compiling A > ; compiling DUPLICATE > ; compiling SET; compilation aborted because of fatal error: > ; READ failure in COMPILE-FILE: > ; READER-ERROR at 19745 (line 559, column 39) on > # collections.lisp" {1239E0D9}>: > ; comma not inside a backquote > > ; /Users/dev/lisp/elephant/src/elephant/collections.fasl written > ; compilation finished in 0:00:00 > WARNING: > COMPILE-FILE warned while performing # > on > #. > > > The original function in collections.lisp (line 558) looks like this: > > (defun pprev-dup-hack (cur key) > "Go back one step in a duplicate set, returns nil > if previous element is a different key. More efficient than > the current default implementation of cursor-pprev-dup" > (multiple-value-bind (exists? skey value pkey) > (cursor-pprev cur) > (when (lisp-compare-equal key skey) > (values exists? key value pkey)))) > > > I changed the last line to: > (values exists? skey value pkey)))) > > and it now seems to load. However, now I'm getting this after > running all tests: > > 10 out of 131 total tests failed: MAP-INDEXED-INDEX, MAP-INDEX-FROM- > END, > INDEXING-BASIC, INDEXING-RANGE, INDEXING-SLOT-MAKUNBOUND, > INDEXING-WIPE-INDEX, INDEXING-RECONNECT-DB, INDEXING-CHANGE-CLASS, > INDEXING-REDEF-CLASS, INDEXING-TIMING. > > Whereas before, I was not getting any test failures. Is this due to > the typo in collections.lisp or something else? > > Thanks, > Daniel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Tue May 8 00:11:37 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 7 May 2007 20:11:37 -0400 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: <3990b5930705051707r2f4a5bfdk5042c01b6c57a4e9@mail.gmail.com> References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> <3990b5930705051245g68b63cafu9aea784d193e2b0c@mail.gmail.com> <3990b5930705051707r2f4a5bfdk5042c01b6c57a4e9@mail.gmail.com> Message-ID: <3A38E62B-F168-430F-B434-153140820385@csail.mit.edu> Elephant users, I promoted a change that enables the -ldb option on linux. The db extension is computed as follows on Mac and Linux: If you provide a pointer to the library such as /usr/lib/libdb45.so. The build script in ele-bdb.asd extracts the name "libdb45" and returns the substring starting at offset 3: "db45" This will result in "-ldb45" being passed to gcc. This will not work if a system does not have the lib prefix on their library name. Please let me know if we need to handle additional cases. I'm aiming to re-test the recent patches and formally announce 0.9 next Monday, so test now or forever squelch your protests. I am willing to support minor patches to 0.9 through the release of 1.0 to keep the system stable, but any major new features or significant bug fixes will be done on our 0.9.x darcs tree on common-lisp.net. For the record, the patches to RC1 thus far are: Functional: - fix typo in elephant.asd - remove linux conditional on -l link option for gcc builds - Make the elephant code version 0.9.0 instead of 0.6.1 - Update the upgrade version map to accommodate 0.6.0 => 0.9.0 databases Aesthetic: - update parameter names for bdb performance tuning in my-config.sexp - fix typos in manual - document collect option (and fix bug in that promotion) Ian On May 5, 2007, at 8:07 PM, Lui Fungsin wrote: > On 5/5/07, Ian Eslick wrote: >> PS - I've had several people load the current HEAD and run on linux/ >> sbcl with the same positive result. So far you are the only >> failure. > > I also don't have any problem with SBCL at all. > > I'm running fedora core 6, SBCL 1.0.3 and lispworks 5.0.2. > > BTW, I found out that the MOP error only happens with lispworks 5.0.1. > > I should have checked lispwork's website first, it turns out that > 5.0.2 has been out for a while. > > Sorry for the false alarm. > >> I'll test Lispworks under Linux later and see if I can >> reproduce the problem, but it worked fine for me before I released >> RC1. Do you have your library path setup correctly in your my- >> config.sexp? > > I use henrik's install-bdb.sh script to install the latest bdb and it > creates my-config.sexp automatically. > > Remember without the -ldb flag, ldd shows that > > $ ldd src/db-bdb/libberkeley-db.so > linux-gate.so.1 => (0x009fb000) > libm.so.6 => /lib/libm.so.6 (0x00110000) > libc.so.6 => /lib/libc.so.6 (0x00db5000) > /lib/ld-linux.so.2 (0x0085f000) > >> (asdf:operate 'asdf:load-op :elephant) >> (asdf:operate 'asdf:load-op :ele-bdb) > > The elephant.asd loads both > > (uffi:load-foreign-library > "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so" :module > "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so") > > and > > (uffi:load-foreign-library "src/db-bdb/libberkeley-db.so" :module > "src/db-bdb/libberkeley-db.so") > > To verify that libdb is loaded, I type the following in the REPL > >> (uffi:def-function ("db_strerror" test-db-strerror) > ((error :int)) > :returning :cstring) > >> (test-db-strerror 1) > "Operation not permitted" > > Now in src/db-bdb/libberkeley-db.c there's a wrapper function > db_strerr which call the above db_strerror. > > If I tried this in the REPL >> (%db-strerror 1) > > lispworks-5-0-0-x86-linux: symbol lookup error: > src/db-bdb/libberkeley-db.so: undefined symbol: db_strerror > > So even though both "*.so" are loaded with load-foreign-library, > libberkeley-db.so cannot access functions in libdb-4.5.so > > Now with the -ldb flag > > $ ldd src/db-bdb/libberkeley-db.so > linux-gate.so.1 => (0x00d71000) > libm.so.6 => /lib/libm.so.6 (0x00d43000) > libdb-4.5.so => /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so > (0x00918000) > libc.so.6 => /lib/libc.so.6 (0x00b43000) > /lib/ld-linux.so.2 (0x0085f000) > libpthread.so.0 => /lib/libpthread.so.0 (0x00f9c000) > > By the time libberkeley-db.so is loaded, it should pull in > libdb-4.5.so automatically, and the above test will succeed. > > I really don't have much knowledge of how SBCL/lispworks implement > their FFI load-library function, but it seems that for SBCL, merely > using load-foreign-library is enough. Someone might have a clue of > what's going on. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Tue May 8 07:01:37 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Tue, 08 May 2007 09:01:37 +0200 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: <3A38E62B-F168-430F-B434-153140820385@csail.mit.edu> References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> <3990b5930705051245g68b63cafu9aea784d193e2b0c@mail.gmail.com> <3990b5930705051707r2f4a5bfdk5042c01b6c57a4e9@mail.gmail.com> <3A38E62B-F168-430F-B434-153140820385@csail.mit.edu> Message-ID: <1178607698.1893.36.camel@trinidad> On Mon, 2007-05-07 at 20:11 -0400, Ian Eslick wrote: > > For the record, the patches to RC1 thus far are: > > Functional: > - fix typo in elephant.asd > - remove linux conditional on -l link option for gcc builds > - Make the elephant code version 0.9.0 instead of 0.6.1 > - Update the upgrade version map to accommodate 0.6.0 => 0.9.0 databases > > Aesthetic: > - update parameter names for bdb performance tuning in my-config.sexp > - fix typos in manual > - document collect option (and fix bug in that promotion) Oops, I forgot to tell you about a small change I needed to make to make elephant RC work on AMD64. Sorry.. In config.sexp, the path /opt/local is something I guess for Macs only. Also db_deadlock is named db_deadlock without 45 on the Berkeley DB version I have (from the official source), and the lib file is .so and not .dylib So I propose a #linux version of sbcl and allegro for config.sexp (see below). I guess it is necessary for 0.9.0 for all linux users. /Henrik #+(and (or sbcl allegro) (not mswindows) (not windows) (not linux)) ((:compiler . :gcc) (:berkeley-db-include-dir . "/opt/local/include/db45/") (:berkeley-db-lib-dir . "/opt/local/lib/db45/") (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") (:berkeley-db-cachesize . 20971520) (:berkeley-db-map-degree2 . t) (:clsql-lib-paths . nil) (:prebuilt-libraries . nil)) #+(and (or sbcl allegro) linux) ((:compiler . :gcc) (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so") (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") (:berkeley-db-cachesize . 20971520) (:berkeley-db-map-degree2 . t) (:clsql-lib-paths . nil) (:prebuilt-libraries . nil)) From eslick at csail.mit.edu Tue May 8 11:26:26 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 8 May 2007 07:26:26 -0400 Subject: [elephant-devel] CVS HEAD w/ lispworks 5.0.1 linux In-Reply-To: <1178607698.1893.36.camel@trinidad> References: <3990b5930704211241o742aad76r3d2dd19c60658aad@mail.gmail.com> <3990b5930704230028m61ea4e25ve05a70937cecc2e8@mail.gmail.com> <3990b5930704230948p7718ef80ocbb38cee7e48bac7@mail.gmail.com> <07E314B5-3757-43AE-8268-A3968BC2BCC4@csail.mit.edu> <3990b5930705042330hf2747ffid157a6495ef47779@mail.gmail.com> <10E22567-8785-4BE4-BD73-CF877D39247B@csail.mit.edu> <3990b5930705051245g68b63cafu9aea784d193e2b0c@mail.gmail.com> <3990b5930705051707r2f4a5bfdk5042c01b6c57a4e9@mail.gmail.com> <3A38E62B-F168-430F-B434-153140820385@csail.mit.edu> <1178607698.1893.36.camel@trinidad> Message-ID: <6FD3ACB3-5E18-4A58-910F-FC346006EEF9@csail.mit.edu> Henrik, I don't see this as a patch needed in RC1. I'm happy to add more default configurations as examples in the development tree, but it is expected/required that users will customize my-config.sexp to be appropriate for their systems (see Installation chapter) as Lisp does not have enough information to determine where the libs are definitively. For example, the /opt/local entry is using DarwinPorts on the Mac, but if I installed from source my Mac would have /usr/local/ BerkeleyDB.4.5 as the BDB root. Some linux systems have the libs pre- installed (i.e. via apt-get on Debian) so would use paths /usr/include /usr/lib /usr/lib/libdb.4.5.so /usr/bin/db45_deadlock which wouldn't work for the linux build you sent. Now we could consider providing a list of canonical locations, using the platform & lisp as a hint, and write a routine that searches those locations for the lib and include files prompt the user to fix their my-config.sexp only if it failed. I prefer to make the user do one set of configuration steps in one location so they think through their options, but I'm open to an alternative approach. My thinking is that if we load the wrong lib by accident the error messages are likely to be low-level failures that are very confusing. I think some of the recent list traffic has demonstrated this. Ian On May 8, 2007, at 3:01 AM, Henrik Hjelte wrote: > On Mon, 2007-05-07 at 20:11 -0400, Ian Eslick wrote: >> >> For the record, the patches to RC1 thus far are: >> >> Functional: >> - fix typo in elephant.asd >> - remove linux conditional on -l link option for gcc builds >> - Make the elephant code version 0.9.0 instead of 0.6.1 >> - Update the upgrade version map to accommodate 0.6.0 => 0.9.0 >> databases >> >> Aesthetic: >> - update parameter names for bdb performance tuning in my-config.sexp >> - fix typos in manual >> - document collect option (and fix bug in that promotion) > > Oops, I forgot to tell you about a small change I needed to make to > make > elephant RC work on AMD64. Sorry.. > > In config.sexp, the path /opt/local is something I guess for Macs > only. > Also db_deadlock is named db_deadlock without 45 on the Berkeley DB > version I have (from the official source), and the lib file is .so and > not .dylib > > So I propose a #linux version of sbcl and allegro for config.sexp (see > below). I guess it is necessary for 0.9.0 for all linux users. > > /Henrik > > > #+(and (or sbcl allegro) (not mswindows) (not windows) (not linux)) > ((:compiler . :gcc) > (:berkeley-db-include-dir . "/opt/local/include/db45/") > (:berkeley-db-lib-dir . "/opt/local/lib/db45/") > (:berkeley-db-lib . "/opt/local/lib/db45/libdb-4.5.dylib") > (:berkeley-db-deadlock . "/opt/local/bin/db45_deadlock") > (:berkeley-db-cachesize . 20971520) > (:berkeley-db-map-degree2 . t) > (:clsql-lib-paths . nil) > (:prebuilt-libraries . nil)) > > #+(and (or sbcl allegro) linux) > ((:compiler . :gcc) > (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.5/include/") > (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.5/lib/") > (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so") > (:berkeley-db-deadlock . "/usr/local/BerkeleyDB.4.5/bin/db_deadlock") > (:berkeley-db-cachesize . 20971520) > (:berkeley-db-map-degree2 . t) > (:clsql-lib-paths . nil) > (:prebuilt-libraries . nil)) > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Sat May 12 14:46:02 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Sat, 12 May 2007 16:46:02 +0200 Subject: [elephant-devel] db-postmodern update Message-ID: <1178981162.1893.107.camel@trinidad> I have made some changes to the postmodern postgresql backend. The darcs repo is now based on the official elephant darcs repo, so patches can be moved between them easily. The postmodern backend passes all "official" tests, but has not been extensively tested with real data yet. The db-postmodern tree can be found here: http://common-lisp.net/project/grand-prix/darcs/elephant /Henrik Hjelte From gorbatovsky at gmail.com Wed May 16 08:27:26 2007 From: gorbatovsky at gmail.com (Dmitry V. Gorbatovsky) Date: Wed, 16 May 2007 11:27:26 +0300 Subject: [elephant-devel] multithreading Message-ID: <200705161127.26716.dg@midasitech.com> Hello and thanks for glorious project. I would like to ask for very short (few lines) example of using 'elephant' in multiple threads on sbcl/linux. Or maybe just link to where I may find one. Beforehand gratefull. Dmitry From eslick at csail.mit.edu Wed May 16 21:38:18 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 16 May 2007 17:38:18 -0400 Subject: [elephant-devel] multithreading In-Reply-To: <200705161127.26716.dg@midasitech.com> References: <200705161127.26716.dg@midasitech.com> Message-ID: Actually these days it's quite easy. If you have a function that acts as a handler for, say, a url request and can be invoked in different threads all you need to do is: All threads share a single, open store-controller object. Each transaction is started and committed/aborted in the same thread. The first one is easy, just use a global variable and open the store controller when the program starts up: (defvar *my-db-spec* '(:BDB "/user/me/db/")) (defvar *my-db-store* nil) (defun startup () (setf *my-db-store* (open-store *my-db-spec*))) Then a good policy to avoid transactions across threads is: 1) Always use with-transaction 2) Never spawn a thread from within the dynamic extent of the with- transaction body form(s) Pretty simple! I'm sorry we don't have a good canned example, however I've attached a hutchentoot-based blog example that is current and that I believe would run fine in multiple threads, although I haven't tested it. Regards, Ian -------------- next part -------------- A non-text attachment was scrubbed... Name: hunchentoot.blog.lisp Type: application/octet-stream Size: 8770 bytes Desc: not available URL: -------------- next part -------------- On May 16, 2007, at 4:27 AM, Dmitry V. Gorbatovsky wrote: > Hello and thanks for glorious project. > I would like to ask for very short (few lines) example > of using 'elephant' in multiple threads on sbcl/linux. > Or maybe just link to where I may find one. > > Beforehand gratefull. > Dmitry > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From gorbatovsky at gmail.com Thu May 17 00:29:41 2007 From: gorbatovsky at gmail.com (Dmitry V. Gorbatovsky) Date: Thu, 17 May 2007 03:29:41 +0300 Subject: [elephant-devel] multithreading In-Reply-To: References: <200705161127.26716.dg@midasitech.com> Message-ID: <200705170329.41263.dg@midasitech.com> Thank you. Regards, Dmitry From woei at yandex.ru Wed May 23 18:56:52 2007 From: woei at yandex.ru (woei at yandex.ru) Date: Wed, 23 May 2007 22:56:52 +0400 Subject: [elephant-devel] bdb licencing Message-ID: <200705232256.52391.woei@yandex.ru> I have read http://www.oracle.com/technology/software/products/berkeley-db/htdocs/licensing.html and http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html Am I right that I can't use elephant+bdb in closed source commercial application without purchasing licence? From ctdean at sokitomi.com Wed May 23 19:45:27 2007 From: ctdean at sokitomi.com (Chris Dean) Date: Wed, 23 May 2007 12:45:27 -0700 Subject: [elephant-devel] bdb licencing In-Reply-To: <200705232256.52391.woei@yandex.ru> (woei@yandex.ru's message of "Wed, 23 May 2007 22:56:52 +0400") References: <200705232256.52391.woei@yandex.ru> Message-ID: > Am I right that I can't use elephant+bdb in closed source commercial > application without purchasing licence? IANAL, but my reading is that if you ship a closed source product you need a license for Berkeley DB. If you have a service/web site (like Google, Yahoo, etc) my reading is that you do not need a license for Berkeley DB. Cheers, Chris Dean From eslick at csail.mit.edu Wed May 23 19:45:17 2007 From: eslick at csail.mit.edu (=?UTF-8?B?SWFuIEVzbGljaw==?=) Date: Wed, 23 May 2007 19:45:17 +0000 Subject: [elephant-devel] bdb licencing In-Reply-To: <200705232256.52391.woei@yandex.ru> References: <200705232256.52391.woei@yandex.ru> Message-ID: <1414109557-1179949558-cardhu_blackberry.rim.net-495334213-@bxe053-cell01.bisx.prod.on.blackberry> Yes, that is my understanding and one of the primary motivations for the SQL backend and forthcoming lisp backends. -Ian Sent via BlackBerry from T-Mobile -----Original Message----- From: woei at yandex.ru Date: Wed, 23 May 2007 22:56:52 To:elephant-devel at common-lisp.net Subject: [elephant-devel] bdb licencing I have read http://www.oracle.com/technology/software/products/berkeley-db/htdocs/licensing.html and http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html Am I right that I can't use elephant+bdb in closed source commercial application without purchasing licence? _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Wed May 23 20:06:14 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 23 May 2007 16:06:14 -0400 Subject: [elephant-devel] bdb licencing In-Reply-To: References: <200705232256.52391.woei@yandex.ru> Message-ID: On May 23, 2007, at 3:45 PM, Chris Dean wrote: > >> Am I right that I can't use elephant+bdb in closed source commercial >> application without purchasing licence? > > IANAL, but my reading is that if you ship a closed source product you > need a license for Berkeley DB. If you have a service/web site (like > Google, Yahoo, etc) my reading is that you do not need a license for > Berkeley DB. I think that Sleepycat, now Oracle clarified that distinction and that any public-facing for-profit website needs a license. Robert may have more to say on this topic. Ian From ctdean at sokitomi.com Wed May 23 22:05:07 2007 From: ctdean at sokitomi.com (Chris Dean) Date: Wed, 23 May 2007 15:05:07 -0700 Subject: [elephant-devel] bdb licencing In-Reply-To: (Ian Eslick's message of "Wed, 23 May 2007 16:06:14 -0400") References: <200705232256.52391.woei@yandex.ru> Message-ID: Ian Eslick writes: > I think that Sleepycat, now Oracle clarified that distinction and > that any public-facing for-profit website needs a license. Robert > may have more to say on this topic. Very interesting, do you know of a reference for that does clarify that distinction? Cheers, Chris Dean From read at robertlread.net Wed May 23 22:28:41 2007 From: read at robertlread.net (Robert L. Read) Date: Wed, 23 May 2007 17:28:41 -0500 Subject: [elephant-devel] bdb licencing In-Reply-To: References: <200705232256.52391.woei@yandex.ru> Message-ID: <1179959322.414.1110.camel@localhost.localdomain> I agree with Ian. Previously, one definitely required a license for any public-facing commercial website. I have not researched any change that Oracle may or may not have made. On Wed, 2007-05-23 at 16:06 -0400, Ian Eslick wrote: > On May 23, 2007, at 3:45 PM, Chris Dean wrote: > > > > >> Am I right that I can't use elephant+bdb in closed source commercial > >> application without purchasing licence? > > > > IANAL, but my reading is that if you ship a closed source product you > > need a license for Berkeley DB. If you have a service/web site (like > > Google, Yahoo, etc) my reading is that you do not need a license for > > Berkeley DB. > > I think that Sleepycat, now Oracle clarified that distinction and > that any public-facing for-profit website needs a license. Robert > may have more to say on this topic. > > Ian > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ctdean at sokitomi.com Wed May 23 22:29:35 2007 From: ctdean at sokitomi.com (Chris Dean) Date: Wed, 23 May 2007 15:29:35 -0700 Subject: [elephant-devel] bdb licencing In-Reply-To: <1179959322.414.1110.camel@localhost.localdomain> (Robert L. Read's message of "Wed, 23 May 2007 17:28:41 -0500") References: <200705232256.52391.woei@yandex.ru> <1179959322.414.1110.camel@localhost.localdomain> Message-ID: "Robert L. Read" writes: > I agree with Ian. Previously, one definitely required a license for > any public-facing commercial website. I have not researched any > change that Oracle may or may not have made. Wow, news to me. Do you know of a web page or email archive that confirms that? I did find http://www.winterspeak.com/columns/102901.html from 2001 which does seem to say the opposite: The restrictions apply only to people who actually ship Berkeley DB. That's the action that requires either payment or release of source code. Building a Web service on top of Berkeley DB and making it available via HTTP doesn't require payment or release of code. Cheers, Chris Dean From joubert at joubster.com Wed May 23 23:10:54 2007 From: joubert at joubster.com (Joubert Nel) Date: Wed, 23 May 2007 19:10:54 -0400 Subject: [elephant-devel] bdb licencing In-Reply-To: <1179959322.414.1110.camel@localhost.localdomain> References: <200705232256.52391.woei@yandex.ru> <1179959322.414.1110.camel@localhost.localdomain> Message-ID: <1179961854.27985.8.camel@joubster> On Wed, 2007-05-23 at 18:28, Robert L. Read wrote: > I agree with Ian. Previously, one definitely required a license for > any public-facing commercial website. > I have not researched any change that Oracle may or may not have made. Reading the Oracle licensing page (http://www.oracle.com/technology/software/products/berkeley-db/htdocs/licensing.html): 1) Whether the application is commercial or not is immaterial to whether you need to license. 2) The restriction is that if you "redistribute" an application you need a license, unless your application is distributed with source code (i.e. "open source") 3) They specifically state that you don't need a license if your application is not distributed to others. Legally, the clarification then needs to be around what constitutes "distribution". In the Q&A section of the licensing page they give 2 lame examples of what would not be considered "distribution", but unfortunately, I don't see any specific reference as to whether a public-facing website would be considered "distribution of the application". My (limited) legal knowledge would say that a public-facing website does not constitute distribution of the application. However, if you host the application on at a hosting company, this may be argued to be "distributed" to this party. Joubert PS: there is an e-mail address on this licensing page; perhaps someone wants to submit this question? > > > On Wed, 2007-05-23 at 16:06 -0400, Ian Eslick wrote: > > On May 23, 2007, at 3:45 PM, Chris Dean wrote: > > > > > > > >> Am I right that I can't use elephant+bdb in closed source commercial > > >> application without purchasing licence? > > > > > > IANAL, but my reading is that if you ship a closed source product you > > > need a license for Berkeley DB. If you have a service/web site (like > > > Google, Yahoo, etc) my reading is that you do not need a license for > > > Berkeley DB. > > > > I think that Sleepycat, now Oracle clarified that distinction and > > that any public-facing for-profit website needs a license. Robert > > may have more to say on this topic. > > > > Ian > > > > From read at robertlread.net Thu May 24 01:05:39 2007 From: read at robertlread.net (Robert L. Read) Date: Wed, 23 May 2007 20:05:39 -0500 Subject: [elephant-devel] bdb licencing In-Reply-To: <1179961854.27985.8.camel@joubster> References: <200705232256.52391.woei@yandex.ru> <1179959322.414.1110.camel@localhost.localdomain> <1179961854.27985.8.camel@joubster> Message-ID: <1179968739.414.1116.camel@localhost.localdomain> Thanks for pointing this out. Looking at the actual license, it seems clear to me that they have adapted a "BSD-style" license, which is quite a change from the sleepycat license (which very explicitly mentioned a public-facing website.) I will send a note to berkeleydb-info_us at oracle.com. In the absence of other information, I think this would be interpreted as NOT applying to a website; as Joubert Nel points out, however, this issue has always revolved around the definition of "redistribute". I will forward the response here. On Wed, 2007-05-23 at 19:10 -0400, Joubert Nel wrote: > On Wed, 2007-05-23 at 18:28, Robert L. Read wrote: > > I agree with Ian. Previously, one definitely required a license for > > any public-facing commercial website. > > I have not researched any change that Oracle may or may not have made. > > Reading the Oracle licensing page > (http://www.oracle.com/technology/software/products/berkeley-db/htdocs/licensing.html): > > 1) Whether the application is commercial or not is immaterial to whether > you need to license. > 2) The restriction is that if you "redistribute" an application you need > a license, unless your application is distributed with source code (i.e. > "open source") > 3) They specifically state that you don't need a license if your > application is not distributed to others. > > Legally, the clarification then needs to be around what constitutes > "distribution". In the Q&A section of the licensing page they give 2 > lame examples of what would not be considered "distribution", but > unfortunately, I don't see any specific reference as to whether a > public-facing website would be considered "distribution of the > application". > > My (limited) legal knowledge would say that a public-facing website does > not constitute distribution of the application. However, if you host the > application on at a hosting company, this may be argued to be > "distributed" to this party. > > Joubert > > PS: there is an e-mail address on this licensing page; perhaps someone > wants to submit this question? > > > > > > > On Wed, 2007-05-23 at 16:06 -0400, Ian Eslick wrote: > > > On May 23, 2007, at 3:45 PM, Chris Dean wrote: > > > > > > > > > > >> Am I right that I can't use elephant+bdb in closed source commercial > > > >> application without purchasing licence? > > > > > > > > IANAL, but my reading is that if you ship a closed source product you > > > > need a license for Berkeley DB. If you have a service/web site (like > > > > Google, Yahoo, etc) my reading is that you do not need a license for > > > > Berkeley DB. > > > > > > I think that Sleepycat, now Oracle clarified that distinction and > > > that any public-facing for-profit website needs a license. Robert > > > may have more to say on this topic. > > > > > > Ian > > > > > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ctdean at sokitomi.com Thu May 24 01:09:41 2007 From: ctdean at sokitomi.com (Chris Dean) Date: Wed, 23 May 2007 18:09:41 -0700 Subject: [elephant-devel] bdb licencing In-Reply-To: <1179961854.27985.8.camel@joubster> (Joubert Nel's message of "Wed, 23 May 2007 19:10:54 -0400") References: <200705232256.52391.woei@yandex.ru> <1179959322.414.1110.camel@localhost.localdomain> <1179961854.27985.8.camel@joubster> Message-ID: Joubert Nel writes: > Reading the Oracle licensing page ... > 3) They specifically state that you don't need a license if your > application is not distributed to others. > > Legally, the clarification then needs to be around what constitutes > "distribution". Exactly. That is the document I read and it does seem to hinge on the definition of "distribution". > My (limited) legal knowledge would say that a public-facing website does > not constitute distribution of the application. The previously mention 2001 article by the then CEO of Sleepycat seems to agree with that interpretation. Cheers, Chris Dean From eslick at csail.mit.edu Thu May 24 01:19:29 2007 From: eslick at csail.mit.edu (=?UTF-8?B?SWFuIEVzbGljaw==?=) Date: Thu, 24 May 2007 01:19:29 +0000 Subject: [elephant-devel] bdb licencing In-Reply-To: References: <200705232256.52391.woei@yandex.ru><1179959322.414.1110.camel@localhost.localdomain><1179961854.27985.8.camel@joubster> Message-ID: <1947888541-1179969523-cardhu_blackberry.rim.net-1684500674-@bxe041-cell01.bisx.prod.on.blackberry> That sounds fine, I didn't research the commercial license extensively. I think I was thinking of the other problem for commercial sites...isn't it the case that the BDB GPL requires that a public-facing website based on BDB make available all the code that is linked with it? A commercial license would allow you to bypass this GPL restriction so you would need a license unless you wanted to make your source available. I'd be very happy to be wrong about this so I invite counterarguments! Ian Ian Sent via BlackBerry from T-Mobile -----Original Message----- From: Chris Dean Date: Wed, 23 May 2007 18:09:41 To:Elephant bugs and development Subject: Re: [elephant-devel] bdb licencing Joubert Nel writes: > Reading the Oracle licensing page ... > 3) They specifically state that you don't need a license if your > application is not distributed to others. > > Legally, the clarification then needs to be around what constitutes > "distribution". Exactly. That is the document I read and it does seem to hinge on the definition of "distribution". > My (limited) legal knowledge would say that a public-facing website does > not constitute distribution of the application. The previously mention 2001 article by the then CEO of Sleepycat seems to agree with that interpretation. Cheers, Chris Dean _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From ctdean at sokitomi.com Thu May 24 02:08:51 2007 From: ctdean at sokitomi.com (Chris Dean) Date: Wed, 23 May 2007 19:08:51 -0700 Subject: [elephant-devel] bdb licencing In-Reply-To: <1947888541-1179969523-cardhu_blackberry.rim.net-1684500674-@bxe041-cell01.bisx.prod.on.blackberry> (Ian Eslick's message of "Thu, 24 May 2007 01:19:29 +0000") References: <200705232256.52391.woei@yandex.ru> <1179959322.414.1110.camel@localhost.localdomain> <1179961854.27985.8.camel@joubster> <1947888541-1179969523-cardhu_blackberry.rim.net-1684500674-@bxe041-cell01.bisx.prod.on.blackberry> Message-ID: "Ian Eslick" writes: > isn't it the case that the BDB GPL requires that a public-facing > website based on BDB make available all the code that is linked with > it? A commercial license would allow you to bypass this GPL > restriction so you would need a license unless you wanted to make > your source available. Well, Berkeley DB isn't under the GPL per se but that's not really the important point. Even Version 2 of the GPL (which is what everyone uses) has no clause about releasing source code for a service/website. It's only shipped products (not services) that must publish their source if they are using GPL licensed software. Version 3 of the GPL does have some clauses around service/website usage but that's not yet finalized. Cheers, Chris Dean From read at robertlread.net Thu May 24 02:22:58 2007 From: read at robertlread.net (Robert L. Read) Date: Wed, 23 May 2007 21:22:58 -0500 Subject: [elephant-devel] bdb licencing In-Reply-To: <1947888541-1179969523-cardhu_blackberry.rim.net-1684500674-@bxe041-cell01.bisx.prod.on.blackberry> References: <200705232256.52391.woei@yandex.ru> <1179959322.414.1110.camel@localhost.localdomain> <1179961854.27985.8.camel@joubster> <1947888541-1179969523-cardhu_blackberry.rim.net-1684500674-@bxe041-cell01.bisx.prod.on.blackberry> Message-ID: <1179973378.414.1127.camel@localhost.localdomain> But the Oracle BDB now appears to be under a "BSD-style" license; in what way is it GPLed? On Thu, 2007-05-24 at 01:19 +0000, Ian Eslick wrote: > That sounds fine, I didn't research the commercial license extensively. I think I was thinking of the other problem for commercial sites...isn't it the case that the BDB GPL requires that a public-facing website based on BDB make available all the code that is linked with it? A commercial license would allow you to bypass this GPL restriction so you would need a license unless you wanted to make your source available. I'd be very happy to be wrong about this so I invite counterarguments! > > Ian > > Ian > > Sent via BlackBerry from T-Mobile > > -----Original Message----- > From: Chris Dean > Date: Wed, 23 May 2007 18:09:41 > To:Elephant bugs and development > Subject: Re: [elephant-devel] bdb licencing > > > Joubert Nel writes: > > Reading the Oracle licensing page > ... > > 3) They specifically state that you don't need a license if your > > application is not distributed to others. > > > > Legally, the clarification then needs to be around what constitutes > > "distribution". > > Exactly. That is the document I read and it does seem to hinge on the > definition of "distribution". > > > My (limited) legal knowledge would say that a public-facing website does > > not constitute distribution of the application. > > The previously mention 2001 article by the then CEO of Sleepycat seems > to agree with that interpretation. > > Cheers, > Chris Dean > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From woei at yandex.ru Thu May 24 20:30:42 2007 From: woei at yandex.ru (woei at yandex.ru) Date: Fri, 25 May 2007 00:30:42 +0400 Subject: [elephant-devel] bdb licencing In-Reply-To: <1414109557-1179949558-cardhu_blackberry.rim.net-495334213-@bxe053-cell01.bisx.prod.on.blackberry> References: <200705232256.52391.woei@yandex.ru> <1414109557-1179949558-cardhu_blackberry.rim.net-495334213-@bxe053-cell01.bisx.prod.on.blackberry> Message-ID: <200705250030.42499.woei@yandex.ru> ? ????????? ?? 23 ??? 2007 23:45 Ian Eslick ???????(a): > Yes, that is my understanding and one of the primary motivations for the > SQL backend and forthcoming lisp backends. -Ian Sent via BlackBerry from > T-Mobile Thanks I am switching to sql backend :) From lists at infoway.net Thu May 24 22:08:46 2007 From: lists at infoway.net (lists at infoway.net) Date: Thu, 24 May 2007 18:08:46 -0400 (EDT) Subject: [elephant-devel] Understanding real-world use of Elephant Message-ID: <34651.192.168.1.70.1180044526.webmail@192.168.1.70> Hi all, I'm still on my quest to learn to effectively use Elephant. Although documentation is not so abundant, I've gotten a pretty good start with the available documentation. However, I don't have such a strong background on using ODBs and mainly come from the SQL world. So, just for curiousity's sake, I read the tutorial for AllegroCache which tries to show the "proper" way to use AllegroCache in real-world systems (http://www.franz.com/products/allegrocache/docs/acachetutorial.pdf). Well, conceptwise, I followed everything in that tutorial and could pretty much relate everything to Elephant. However, the truth is that I thought there was a tremendous "overhead" in using AllegroCache when dealing with multiple threads. The reality is that the applications we are thinking on using Elephant for are web-based applications served by multiple web servers, so the practicality of the tutorial was more inline to our objective use of Elephant. So, I figured that both systems (Elephant and AllegroCache) have more or less the same usage for practical purposes. But, none of the Elephant documentation I've read on Elephant has even referred to the things mentioned in the tutorial when dealing with multiple threads. I have read on the list that Elephant is thread safe, but am wondering if anyone could help me understand how Elephant would be different from AllegroCache. For example, do I still need to handle connection pools and thread locks and all of that in order to do simple multi-threaded requests? Not that I'm looking for an easy way out, but coming from the world of simply "using" a SQL-database and being presented with the needs of having to incorporate a lot of "logic" that the database server would've handled for you into your application seems a bit overwhelming at first. I don't know if it has anything to do with the way AllegroCache is architected (e.g. things like isolation between connections, etc) or other factors I'm not aware of. Again, I only read the tutorial in trying to learn more how to "apply the concepts" of ODB so I could understand them better. Any comments would be greatly appreciated. Thanks, Daniel From read at robertlread.net Thu May 24 23:46:54 2007 From: read at robertlread.net (Robert L. Read) Date: Thu, 24 May 2007 18:46:54 -0500 Subject: [elephant-devel] Understanding real-world use of Elephant In-Reply-To: <34651.192.168.1.70.1180044526.webmail@192.168.1.70> References: <34651.192.168.1.70.1180044526.webmail@192.168.1.70> Message-ID: <1180050414.5313.81.camel@localhost.localdomain> I would have to disagree about the documentation for Elephant not being abundant---Ian has written a 118 page manual. Nonetheless, you are correct that the use of Elephant in a multi- threaded webserver environment is not heavily documented. Ian and I have discussed the need for a killer "example app" and eagerly await someone contributing one. I'm a little rusty on some of this, but if you will agree to take my opinion about skeptically, I will describe my thoughts on the subject. My own application, Konsenti, is a multi-threaded web application. In the first place, one has to go back to basics a little bit. Whenever you have concurrency, you have to have concurrency control. Personally, I think to think of this at the object level, but I know it is now common to think of it at the "database level". You are generally correct that if you are using SQL (or BDB, for that matter) as a database and you keep ALL of your state in the database, then you can generally rely on the concurrency control of those systems to serialize everything and not allow two threads to interfere with each other. However, almost ANY application will have to think about concurrency; if your are SQL-orieneted, you will do this by defining "transactions". If you define your transactions wrong, you will have concurrency errors, even though the database serializes transactions perfectly. For example, generally, since the Web forces us into "page based" or "form based" interaction (which javascript, CSS and Ajax promptly complicate), one can generally think of a web applications as "one-page turn == one transaction". But even that might not be true---you could take 10 page turns to submit an order, and the order must be atomic--- that is, you either order the thing with 10 pages of specifications, or you don't order it. A half-order is a corrupt order. Elephant has the "with-tranaction" macro. This is really the best way to use Elephant in most circumstances --- but even then, if you are keeping ANYTHING in memory, whether to cache it for speed or because you don't want to put it in the database (session-based information would be a typical example), you may still have to "serialize" access to that state. That unavoidable means that you have to understand some sort of mutual exclusion lock; unfortunately, these are not 100% standard across different lisps. However, most lisps do provide a basic "with-mutex" facility. I use this in the DCM code (in the contrib directory) to serialize access to a "director", which is very similar to a version of Ian's persistent classes, but based on a "keep it all in memory and push writes to the DB" model (that is, Prevalence.) So before we can really answer you question deeply, I think we would have to understand more about what you are doing. If you will forgive me over-simplifying things, if: 1) You can legitimately think of every page turn as a transaction, and 2) You keep all of the meaningful state in the Elephant DB, and 3) You wrap your basic "process-a-page" function in "with-transaction", then you won't have a concurrency control problem. That is a completely appropriate style for certain relatively simple web-applications; for other kinds of web-applications it would be very constraining and slow --- but one should never optimize for speed before it is necessary. I don't know if that's a useful answer --- if you can refine your question, I will try again. On Thu, 2007-05-24 at 18:08 -0400, lists at infoway.net wrote: > Hi all, > > I'm still on my quest to learn to effectively use Elephant. Although documentation is not so abundant, I've gotten a pretty good start with the available documentation. > > However, I don't have such a strong background on using ODBs and mainly come from the SQL world. So, just for curiousity's sake, I read the tutorial for AllegroCache which tries to show the "proper" way to use AllegroCache in real-world systems (http://www.franz.com/products/allegrocache/docs/acachetutorial.pdf). > > Well, conceptwise, I followed everything in that tutorial and could pretty much relate everything to Elephant. However, the truth is that I thought there was a tremendous "overhead" in using AllegroCache when dealing with multiple threads. The reality is that the applications we are thinking on using Elephant for are web-based applications served by multiple web servers, so the practicality of the tutorial was more inline to our objective use of Elephant. > > So, I figured that both systems (Elephant and AllegroCache) have more or less the same usage for practical purposes. But, none of the Elephant documentation I've read on Elephant has even referred to the things mentioned in the tutorial when dealing with multiple threads. I have read on the list that Elephant is thread safe, but am wondering if anyone could help me understand how Elephant would be different from AllegroCache. > > For example, do I still need to handle connection pools and thread locks and all of that in order to do simple multi-threaded requests? Not that I'm looking for an easy way out, but coming from the world of simply "using" a SQL-database and being presented with the needs of having to incorporate a lot of "logic" that the database server would've handled for you into your application seems a bit overwhelming at first. > > I don't know if it has anything to do with the way AllegroCache is architected (e.g. things like isolation between connections, etc) or other factors I'm not aware of. Again, I only read the tutorial in trying to learn more how to "apply the concepts" of ODB so I could understand them better. > > Any comments would be greatly appreciated. > > Thanks, > Daniel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Fri May 25 01:16:36 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 24 May 2007 21:16:36 -0400 Subject: [elephant-devel] Understanding real-world use of Elephant In-Reply-To: <1180050414.5313.81.camel@localhost.localdomain> References: <34651.192.168.1.70.1180044526.webmail@192.168.1.70> <1180050414.5313.81.camel@localhost.localdomain> Message-ID: <4AF716B5-22F9-4BFF-A448-E41A8AEA39A7@csail.mit.edu> As a user of Elephant, you really shouldn't have to worry too much about threading so long as you follow the simple rules laid out in the manual under "multi-threading". I think you are trying to understand how we make this possible since it seems harder from your read of the acache interface. A simple conceptual model is that each thread has its own transaction. If these transactions are executing concurrently, the BDB library or SQL server logs the side read dependencies and the side effecting writes until a commit or abort. On abort, you throw away the log. On a commit, one transaction at a time writes their side effects and cancels any transaction that has read or written one of the entries written by the committing transaction. Thread isolation is guaranteed by a policy for using the BDB library or SQL server. Calls to these libraries are re-entrant. The current transaction ID (only used by one thread) determines where the reads and writes are logged (this is a little more complex for SQL, but handled by the data store and transparent to the user). The only other thing we need to do is make sure that the parts of elephant shared across threads are themselves protected by Lisp locks. Most of this is the store-controller and some data structures used to initialize a new store controller. If you stick to the user contract in the documentation, you shouldn't have to worry further about interactions of multiple threads (other than the usual considerations if you have shared lisp variables instead of shared persistent objects). Please let me know what would need to be clarified in the documentation to make this clearer, or ask pointed questions if the above was not a sufficient explanation. After all, the user is always right! Thank you, Ian On May 24, 2007, at 7:46 PM, Robert L. Read wrote: > I would have to disagree about the documentation for Elephant not > being abundant---Ian has written a 118 page manual. > > Nonetheless, you are correct that the use of Elephant in a multi- > threaded webserver environment is not heavily documented. Ian and > I have discussed the need for a killer "example app" and eagerly > await someone contributing one. > > I'm a little rusty on some of this, but if you will agree to take > my opinion about skeptically, I will describe my thoughts on the > subject. My own application, Konsenti, is a multi-threaded web > application. > > In the first place, one has to go back to basics a little bit. > Whenever you have concurrency, you have to have concurrency > control. Personally, I think to think of this at the object level, > but I know it is now common to think of it at the "database > level". You are generally correct that if you are using SQL (or > BDB, for that matter) as a database and you keep ALL of your state > in the database, then you can generally rely on the concurrency > control of those systems to serialize everything and not allow two > threads to interfere with each other. However, almost ANY > application will have to think about concurrency; if your are SQL- > orieneted, you will do this by defining "transactions". If you > define your transactions wrong, you will have concurrency errors, > even though the database serializes transactions perfectly. > > For example, generally, since the Web forces us into "page based" > or "form based" interaction (which javascript, CSS and Ajax > promptly complicate), one can generally think of a web applications > as "one-page turn == one transaction". But even that might not be > true---you could take 10 page turns to submit an order, and the > order must be atomic---that is, you either order the thing with 10 > pages of specifications, or you don't order it. A half-order is a > corrupt order. > > Elephant has the "with-tranaction" macro. This is really the best > way to use Elephant in most circumstances --- but even then, if you > are keeping ANYTHING in memory, whether to cache it for speed or > because you don't want to put it in the database (session-based > information would be a typical example), you may still have to > "serialize" access to that state. That unavoidable means that you > have to understand some sort of mutual exclusion lock; > unfortunately, these are not 100% standard across different lisps. > However, most lisps do provide a basic "with-mutex" facility. I > use this in the DCM code (in the contrib directory) to serialize > access to a "director", which is very similar to a version of Ian's > persistent classes, but based on a "keep it all in memory and push > writes to the DB" model (that is, Prevalence.) > > So before we can really answer you question deeply, I think we > would have to understand more about what you are doing. > > If you will forgive me over-simplifying things, if: > 1) You can legitimately think of every page turn as a transaction, and > 2) You keep all of the meaningful state in the Elephant DB, and > 3) You wrap your basic "process-a-page" function in "with- > transaction", > > then you won't have a concurrency control problem. > > That is a completely appropriate style for certain relatively > simple web-applications; for other kinds of web-applications it > would be very constraining and slow --- but one should never > optimize for speed before it is necessary. > > I don't know if that's a useful answer --- if you can refine your > question, I will try again. > > > > > On Thu, 2007-05-24 at 18:08 -0400, lists at infoway.net wrote: >> Hi all, I'm still on my quest to learn to effectively use >> Elephant. Although documentation is not so abundant, I've gotten a >> pretty good start with the available documentation. However, I >> don't have such a strong background on using ODBs and mainly come >> from the SQL world. So, just for curiousity's sake, I read the >> tutorial for AllegroCache which tries to show the "proper" way to >> use AllegroCache in real-world systems (http://www.franz.com/ >> products/allegrocache/docs/acachetutorial.pdf). Well, conceptwise, >> I followed everything in that tutorial and could pretty much >> relate everything to Elephant. However, the truth is that I >> thought there was a tremendous "overhead" in using AllegroCache >> when dealing with multiple threads. The reality is that the >> applications we are thinking on using Elephant for are web-based >> applications served by multiple web servers, so the practicality >> of the tutorial was more inline to our objective use of Elephant. >> So, I figured that both systems (Elephant and AllegroCache) have >> more or less the same usage for practical purposes. But, none of >> the Elephant documentation I've read on Elephant has even referred >> to the things mentioned in the tutorial when dealing with multiple >> threads. I have read on the list that Elephant is thread safe, but >> am wondering if anyone could help me understand how Elephant would >> be different from AllegroCache. For example, do I still need to >> handle connection pools and thread locks and all of that in order >> to do simple multi-threaded requests? Not that I'm looking for an >> easy way out, but coming from the world of simply "using" a SQL- >> database and being presented with the needs of having to >> incorporate a lot of "logic" that the database server would've >> handled for you into your application seems a bit overwhelming at >> first. I don't know if it has anything to do with the way >> AllegroCache is architected (e.g. things like isolation between >> connections, etc) or other factors I'm not aware of. Again, I only >> read the tutorial in trying to learn more how to "apply the >> concepts" of ODB so I could understand them better. Any comments >> would be greatly appreciated. Thanks, Daniel >> _______________________________________________ elephant-devel >> site list elephant-devel at common-lisp.net http://common-lisp.net/ >> mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Fri May 25 10:33:59 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 25 May 2007 12:33:59 +0200 Subject: [elephant-devel] Understanding real-world use of Elephant In-Reply-To: <34651.192.168.1.70.1180044526.webmail@192.168.1.70> References: <34651.192.168.1.70.1180044526.webmail@192.168.1.70> Message-ID: <1180089239.7967.37.camel@trinidad> I'm in the same situation, hoping to have several web servers connected to one store, and this is the setup I will use. The reason for multithreading in a web app as I see it is mostly to make use of several processors, which is not something solved with the allegrocache paper if I remember it right, since allegro multithreading only makes use of one processor/core. SBCL could do this, but to avoid possible unidentified troubles I instead start several single-threaded processes on the same machine, and use a proxy with sticky sessions (pound) to distribute requests between several processes. That also has the advantage that the processes need not be on the same physical machine, so it should be scalable. I think that SQL databases are a safer bet than Berkeley DB for having several processes on different machines talking to the same store, so I will have one instance of postgresql running on a server with scsi raid 10 and lots of ram. I have done some stress testing with this solution and am happy with it so far, it is a bit memory hungry though. Good luck, Henrik Hjelte From nowhere.man at levallois.eu.org Fri May 25 18:39:17 2007 From: nowhere.man at levallois.eu.org (Pierre THIERRY) Date: Fri, 25 May 2007 20:39:17 +0200 Subject: [elephant-devel] ECOOP 2007 Message-ID: <20070525183917.GC24354@bateleur.arcanes.fr.eu.org> Hi all, I mentioned the Lisp workshop in ECOOP 2007, and my intention to try and write a paper for it. Sadly, I did find time to do so before the may 13 deadline. I was very disappointed, so when they extended the deadline to may 31, I took my bibliography and began to write. I will co-author the paper with my brother, a Ph.D. student in Strasbourg[1], but I think I'll need, if not help, at least your input about the future plans of Elephant (about MOP semantics and features that make Elephant closer to an OODB). [1] https://dpt-info.u-strasbg.fr/~thierry/ I've setup a Mercurial repository for my paper[2]. [2] http://arcanes.fr.eu.org/~pierre/2007/05/ecoop/ Here is the current abstract: The data model of an application, the nature and format of data stored accross executions of the application, is typically a very rigid part of its early specification, even when prototyping, and changing it after code has been written that rely on it can prove quite expensive and error-prone. A MOP-based persistence library can bring to the data model the dynamic nature that Lisp shows with respect to code and data stored in a running Lisp image. This enables to extend the easy prototyping way of development to the storage of data. Lately, Pierre -- nowhere.man at levallois.eu.org OpenPGP 0xD9D50D8A -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From lists at infoway.net Fri May 25 20:32:35 2007 From: lists at infoway.net (lists at infoway.net) Date: Fri, 25 May 2007 16:32:35 -0400 (EDT) Subject: [elephant-devel] Understanding real-world use of Elephant Message-ID: <37737.192.168.1.71.1180125155.webmail@192.168.1.71> Hello Ian, Robert, and Henrik I'll try to comment based on the responses received from the three of you in this single thread so as to minimize the posts. Before proceeding, let me just clarify that I am only interested in using the BDB backend. > I would have to disagree about the documentation for Elephant not being > abundant---Ian has written a 118 page manual. > > Nonetheless, you are correct that the use of Elephant in a multi- > threaded webserver environment is not heavily documented. Ian and I > have discussed the need for a killer "example app" and eagerly await > someone contributing one. First of all, I want to apologize if my comment came across the wrong way. I know that Ian (and whoever else has been contributing) has done a superb job at enhancing Elephant's documentation. It definitely has come a long way. I first had a bit of difficulty finding the latest documentation since I couldn't find it online. I then learned that it came in the doc directory and that you had to "make" it. Anyway, it is great! As far as multi-threaded webserver environment, I know there was a section about it in the doc (section 6.5) but, as you said, it's not very elaborate. >> However, I don't have such a strong background on using ODBs and mainly come from >> the SQL world. So, just for curiousity's sake, I read the tutorial for >> AllegroCache which tries to show the "proper" way to use AllegroCache in >> real-world systems >> (http://www.franz.com/products/allegrocache/docs/acachetutorial.pdf). I'd like to clarify my comment above. Because I read several AllegroCache documents, I misreferenced the document I really wanted to reference. The document in question is title "AllegroCache with Web Servers" and can be found here: http://www.franz.com/products/allegrocache/docs/acweb.pdf > In the first place, one has to go back to basics a little bit. Whenever > you have concurrency, you have to have concurrency control. Personally, > I think to think of this at the object level, but I know it is now > common to think of it at the "database level". You are generally > correct that if you are using SQL (or BDB, for that matter) as a > database and you keep ALL of your state in the database, then you can > generally rely on the concurrency control of those systems to serialize > everything and not allow two threads to interfere with each other. > However, almost ANY application will have to think about concurrency; if > your are SQL-orieneted, you will do this by defining "transactions". If > you define your transactions wrong, you will have concurrency errors, > even though the database serializes transactions perfectly. > > For example, generally, since the Web forces us into "page based" or > "form based" interaction (which javascript, CSS and Ajax promptly > complicate), one can generally think of a web applications as "one-page > turn == one transaction". But even that might not be true---you could > take 10 page turns to submit an order, and the order must be atomic--- > that is, you either order the thing with 10 pages of specifications, or > you don't order it. A half-order is a corrupt order. I agree with you that, in general, when dealing with web applications involving multiple clients and servers, you have to have concurrency control. How much do you have to have in your own application vs how much does the "database framework" offer is, in my opinion, a good question. Making reference to the Allegro document, it says "In AllegroCache a program opens a connection to a database and that connection can only be used by one thread at a time." Then, as you read the document and focus on their client-server model, they present sample code that uses "thread-safe" connection pools, with a macro named with-client-connectionwith-client-connection. "This macro retrieves a connection from the pool. If no connection is available it will create a new connection but it will create no more than *connection?count* connections. If all connections are created and a new connection is needed the requesting thread will be put to sleep until a connection is returned to the pool." The macro is not the problem, since I could "think of" this macro as something like Elephant's with-transaction. The problem, and the overhead I was referring to in my original post is that, to perform a basic operation such as to update a hash table value, they write the function as this: (defun set-password-for-pool (user new-password) (with-client-connection poolobj (with-transaction-restart nil (setf (map-value (or (poolobj-map poolobj) (setf (poolobj-map poolobj) (retrieve-from-index 'ac-map 'ac-map-name "password"))) user) new-password) (commit)))) As you can see, there is some, possibly, unnecessary overhead in the fact that you are getting a connection from the pool and then obtaining a handle to the "password" hash table before anything can be set. The reason, as I understand it, they do this is because since each connection handle works independently in each thread, each connection has to maintain a separate handle to each persistent object class and their solution involves storing in the poolobj structure a handle to the connection and a handle to the hash table. So, if this was a more complex application, involving n persistent classes with m persistent attributes per class, the overhead of writing all this is significant. Assuming we follow the Elephant recommendation in section 2.9.3 where actions should be reduced to minimal operations and nest them with with-transaction/ensure-transaction, I would have to write, potentially, 2*n*m defuns (getter/setter) for all the attributes with all the code to fetch and cache the handles to the connection and to the respective n persistent class. > Elephant has the "with-tranaction" macro. This is really the best way > to use Elephant in most circumstances --- but even then, if you are > keeping ANYTHING in memory, whether to cache it for speed or because you > don't want to put it in the database (session-based information would be > a typical example), you may still have to "serialize" access to that > state. That unavoidable means that you have to understand some sort of > mutual exclusion lock; unfortunately, these are not 100% standard across > different lisps. However, most lisps do provide a basic "with-mutex" > facility. I use this in the DCM code (in the contrib directory) to > serialize access to a "director", which is very similar to a version of > Ian's persistent classes, but based on a "keep it all in memory and push > writes to the DB" model (that is, Prevalence.) The idea I have is to rely on the persistent data instead of in-memory data. Once I get this going, I may decide to improve performance with in-memory caches, or anything else. Just want to get the concept going in a stable and scaleable format. > If you will forgive me over-simplifying things, if: > 1) You can legitimately think of every page turn as a transaction, and > 2) You keep all of the meaningful state in the Elephant DB, and > 3) You wrap your basic "process-a-page" function in "with-transaction", > > then you won't have a concurrency control problem. > > That is a completely appropriate style for certain relatively simple > web-applications; for other kinds of web-applications it would be very > constraining and slow --- but one should never optimize for speed before > it is necessary. I don't mind the over-simplification as long as I understand it :), and I do. However, thinking back to the AllegroCache document, from what I understood, they basically take a handle to the connection, perform the operation, and then release the connection. If this was a multi-page web operation, it seems that their recommendation would not be most appropriate, IMHO, but then again, I don't know. In your recommendation, if I had a order entry system with multiple pages to be completed before committing the order, I could understand wrapping the whole thing with with-transaction. However, wouldn't that present a possible problem locking resources and leaving it up to the human user to complete the process before committing or rolling back the transaction? > As a user of Elephant, you really shouldn't have to worry too much > about threading so long as you follow the simple rules laid out in > the manual under "multi-threading". I think you are trying to > understand how we make this possible since it seems harder from your > read of the acache interface. You may be right. However, thinking more about this whole thing and from my understanding of Elephant and what I understood of AllegroCache, I may be trying to compare apples and oranges. They may be similar systems, but I don't know if it makes justice comparing Elephant with AllegroCache client-server model. If I understand it correctly (now), the current implementation of Elephant is more similar to AllegroCache stand alone (non-client-server) model. So, each web process that accesses Elephant can do so seamlessly with the standard *store-controller* (assuming a single store controller) and not have to deal with having to manage connection pools and all that. Keeping this in mind, I would also assume that in Elephant, I don't have to keep a handle to each persistent class for each connection. Maybe this is what confused me and maybe I shouldn't be reading AllegroCache's documentation :) > A simple conceptual model is that each thread has its own > transaction. If these transactions are executing concurrently, the > BDB library or SQL server logs the side read dependencies and the > side effecting writes until a commit or abort. On abort, you throw > away the log. On a commit, one transaction at a time writes their > side effects and cancels any transaction that has read or written one > of the entries written by the committing transaction. > > Thread isolation is guaranteed by a policy for using the BDB library > or SQL server. Calls to these libraries are re-entrant. The current > transaction ID (only used by one thread) determines where the reads > and writes are logged (this is a little more complex for SQL, but > handled by the data store and transparent to the user). I guess this goes back to what I just commented on the fact that each web thread/request will use the connection in place in the Lisp VM and not have to deal with establishing a new connection (I could have checks to make sure that if the store controller is not opened, I could open it, but once it's opened, I "shouldn't" have to worry too much about it). Right? > The only other thing we need to do is make sure that the parts of > elephant shared across threads are themselves protected by Lisp > locks. Most of this is the store-controller and some data structures > used to initialize a new store controller. As an end-user application developer, do I need to worry about this or should I expect the Elephant framework to handle it? > If you stick to the user contract in the documentation, you shouldn't > have to worry further about interactions of multiple threads (other > than the usual considerations if you have shared lisp variables > instead of shared persistent objects). I would assume you are referring to my own application shared variables and not Elephant-related variables, right? > I think that SQL databases are a safer bet than Berkeley DB > for having several processes on different machines talking to the same > store, so I will have one instance of postgresql running on a server > with scsi raid 10 and lots of ram. Henrik, would you mind elaborating more on this? Why would SQL databases be safer than the BDB stores? I know they are handled by separate processes, potentially, on separate machines, so in essence, they are independent of your application. However, isn't BDB designed just to tackle that using an application library instead of a separate process? Overall, and being this my first experience with Lisp and ODBs, I really like Elephant. After reading some of AllegroCache's documentation, I would still prefer using Elephant. Maybe I'm trying to see deeper than I need to. Maybe I just need to see more samples of real-world applications. I would love to contribute sample applications to the project so as to make it clearer and easier for others to learn, but I guess, I have to learn it myself first. Code-wise, I think I have grasped the whole thing. However, since I currently have no ability to test anything to a larger scale, I'm trying to understand what it would take for an application that uses Elephant to work in a large scale system (both hardware and software). Thanks again for everyone of your comments. They did in fact help me and am sure you follow up comments will further help me even more. Now, while you guys digest this and reply to my post, I will go back and read the updated Elephant manual :) Thanks, Daniel From eslick at csail.mit.edu Fri May 25 21:29:17 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 25 May 2007 17:29:17 -0400 Subject: [elephant-devel] Understanding real-world use of Elephant In-Reply-To: <37737.192.168.1.71.1180125155.webmail@192.168.1.71> References: <37737.192.168.1.71.1180125155.webmail@192.168.1.71> Message-ID: <96C936A2-835C-4EC8-9C7E-1317A5267806@csail.mit.edu> On May 25, 2007, at 4:32 PM, lists at infoway.net wrote: > Hello Ian, Robert, and Henrik > > I'll try to comment based on the responses received from the three > of you in this single thread so as to minimize the posts. Before > proceeding, let me just clarify that I am only interested in using > the BDB backend. > >> I would have to disagree about the documentation for Elephant not >> being >> abundant---Ian has written a 118 page manual. >> >> Nonetheless, you are correct that the use of Elephant in a multi- >> threaded webserver environment is not heavily documented. Ian and I >> have discussed the need for a killer "example app" and eagerly await >> someone contributing one. > > First of all, I want to apologize if my comment came across the > wrong way. I know that Ian (and whoever else has been contributing) > has done a superb job at enhancing Elephant's documentation. It > definitely has come a long way. I first had a bit of difficulty > finding the latest documentation since I couldn't find it online. I > then learned that it came in the doc directory and that you had to > "make" it. Anyway, it is great! Documentation has always been available online so only developers updating the web site or editing the documentation will need to 'make' them. The new site has documentation at: http://common- lisp.net/project/elephant/documentation.html. This page can be reached by clicking the "documentation" link which you can find in the leftmost column of the home page. You can jump directly to the latest online texinfo style HTML by clicking 'Online Docs' in the upper right hand corner of the home page. Do you know what caused you to miss these links to documentation? Was there anything confusing about our site that we could fix? > As far as multi-threaded webserver environment, I know there was a > section about it in the doc (section 6.5) but, as you said, it's > not very elaborate. Read 4.10, 4.11 and 4.13. 6.5 needs more work to server as a proper example. Section 6 mostly has placeholders at present. I'll see about expanding on 4.10 and 6.5 as time allows. > >>> However, I don't have such a strong background on using ODBs and >>> mainly come from >>> the SQL world. So, just for curiousity's sake, I read the >>> tutorial for >>> AllegroCache which tries to show the "proper" way to use >>> AllegroCache in >>> real-world systems >>> (http://www.franz.com/products/allegrocache/docs/ >>> acachetutorial.pdf). > > I'd like to clarify my comment above. Because I read several > AllegroCache documents, I misreferenced the document I really > wanted to reference. > > The document in question is title "AllegroCache with Web Servers" > and can be found here: http://www.franz.com/products/allegrocache/ > docs/acweb.pdf As you comment below, reading the acache document created a great deal of confusion! Please ignore it. While the object and class interfaces are similar, the system implications and usage models can be very different so as you comment, you are comparing apples and oranges. > >> In the first place, one has to go back to basics a little bit. >> Whenever >> you have concurrency, you have to have concurrency control. >> Personally, >> I think to think of this at the object level, but I know it is now >> common to think of it at the "database level". You are generally >> correct that if you are using SQL (or BDB, for that matter) as a >> database and you keep ALL of your state in the database, then you can >> generally rely on the concurrency control of those systems to >> serialize >> everything and not allow two threads to interfere with each other. >> However, almost ANY application will have to think about >> concurrency; if >> your are SQL-orieneted, you will do this by defining >> "transactions". If >> you define your transactions wrong, you will have concurrency errors, >> even though the database serializes transactions perfectly. >> >> For example, generally, since the Web forces us into "page based" or >> "form based" interaction (which javascript, CSS and Ajax promptly >> complicate), one can generally think of a web applications as "one- >> page >> turn == one transaction". But even that might not be true---you >> could >> take 10 page turns to submit an order, and the order must be >> atomic--- >> that is, you either order the thing with 10 pages of >> specifications, or >> you don't order it. A half-order is a corrupt order. > > I agree with you that, in general, when dealing with web > applications involving multiple clients and servers, you have to > have concurrency control. How much do you have to have in your own > application vs how much does the "database framework" offer is, in > my opinion, a good question. > > Making reference to the Allegro document, it says "In AllegroCache > a program opens a connection to a database and that connection can > only be used by one thread at a time." Then, as you read the > document and focus on their client-server model, they present > sample code that uses "thread-safe" connection pools, with a macro > named with-client-connectionwith-client-connection. "This macro > retrieves a connection from the pool. If no connection is > available it will create a new connection but it > will create no more than *connection count* connections. If all > connections are created and a new connection is needed the > requesting thread will be put to sleep until a connection is > returned to the pool." > > The macro is not the problem, since I could "think of" this macro > as something like Elephant's with-transaction. The problem, and the > overhead I was referring to in my original post is that, to perform > a basic operation such as to update a hash table value, they write > the function as this: > > (defun set-password-for-pool (user new-password) > (with-client-connection poolobj > (with-transaction-restart nil > (setf (map-value (or (poolobj-map poolobj) > (setf (poolobj-map poolobj) > (retrieve-from-index 'ac-map > 'ac-map-name > "password"))) > user) > new-password) > (commit)))) > > As you can see, there is some, possibly, unnecessary overhead in > the fact that you are getting a connection from the pool and then > obtaining a handle to the "password" hash table before anything can > be set. The reason, as I understand it, they do this is because > since each connection handle works independently in each thread, > each connection has to maintain a separate handle to each > persistent object class and their solution involves storing in the > poolobj structure a handle to the connection and a handle to the > hash table. > > So, if this was a more complex application, involving n persistent > classes with m persistent attributes per class, the overhead of > writing all this is significant. Assuming we follow the Elephant > recommendation in section 2.9.3 where actions should be reduced to > minimal operations and nest them with with-transaction/ensure- > transaction, I would have to write, potentially, 2*n*m defuns > (getter/setter) for all the attributes with all the code to fetch > and cache the handles to the connection and to the respective n > persistent class. > >> Elephant has the "with-tranaction" macro. This is really the best >> way >> to use Elephant in most circumstances --- but even then, if you are >> keeping ANYTHING in memory, whether to cache it for speed or >> because you >> don't want to put it in the database (session-based information >> would be >> a typical example), you may still have to "serialize" access to that >> state. That unavoidable means that you have to understand some >> sort of >> mutual exclusion lock; unfortunately, these are not 100% standard >> across >> different lisps. However, most lisps do provide a basic "with-mutex" >> facility. I use this in the DCM code (in the contrib directory) to >> serialize access to a "director", which is very similar to a >> version of >> Ian's persistent classes, but based on a "keep it all in memory >> and push >> writes to the DB" model (that is, Prevalence.) > > The idea I have is to rely on the persistent data instead of in- > memory data. Once I get this going, I may decide to improve > performance with in-memory caches, or anything else. Just want to > get the concept going in a stable and scaleable format. > >> If you will forgive me over-simplifying things, if: >> 1) You can legitimately think of every page turn as a transaction, >> and >> 2) You keep all of the meaningful state in the Elephant DB, and >> 3) You wrap your basic "process-a-page" function in "with- >> transaction", >> >> then you won't have a concurrency control problem. >> >> That is a completely appropriate style for certain relatively simple >> web-applications; for other kinds of web-applications it would be >> very >> constraining and slow --- but one should never optimize for speed >> before >> it is necessary. > > I don't mind the over-simplification as long as I understand it :), > and I do. However, thinking back to the AllegroCache document, from > what I understood, they basically take a handle to the connection, > perform the operation, and then release the connection. If this was > a multi-page web operation, it seems that their recommendation > would not be most appropriate, IMHO, but then again, I don't know. Connections and handles are completely different in elephant, acache docs are not helpful. > In your recommendation, if I had a order entry system with multiple > pages to be completed before committing the order, I could > understand wrapping the whole thing with with-transaction. However, > wouldn't that present a possible problem locking resources and > leaving it up to the human user to complete the process before > committing or rolling back the transaction? There are lots of ways to think about this. One is that you keep track of the ongoing session using in-memory objects unique to the session. When you need to manipulate a database (to submit an order, a blog entry, etc) the handler for the 'submit' action uses with-transaction to take the data from the in- memory session object and commit it to the database (an entry in a per-user btree, adding a new instance to a class, etc). If you need session history or want to maintain ongoing state, make this session object a persistent object instead. Then each post or get action in the session is logged so you can recover if the user goes away for awhile, or there is a server error. You will eventually fill up your disk with sessions (in the absence of GC) so you need to either drop the session objects when you are done with them or use a separate store for session objects and periodically delete and recreate it. We still need a clean model for online GC of persistent objects to avoid explicit reclamation. As for contention, with-transaction will retry the transaction code so if you have a POST handler you can do something like: (defun handle-post-1 (with-post-data (send-response-page (with-transaction () )))) This way the update can robustly handle contention while the user only sees the final page that results from the update to the persistent object for that user/session. If there is a real problem and the process fails, you can wrap (send-response-page ...) with a 'handler-case' form that sends a server error page with a link to restart the transaction (perhaps with the session object so the form entries are properly initialized on the retry) if the transaction cannot be committed. Failing transactions signal 'transaction-retry-count-exceeded. If you are using BDB make sure that db_deadlock is running. (Either with the :deadlock-detect keyword option or by running an external process (if using multiple lisp processes). >> As a user of Elephant, you really shouldn't have to worry too much >> about threading so long as you follow the simple rules laid out in >> the manual under "multi-threading". I think you are trying to >> understand how we make this possible since it seems harder from your >> read of the acache interface. > > You may be right. However, thinking more about this whole thing and > from my understanding of Elephant and what I understood of > AllegroCache, I may be trying to compare apples and oranges. They > may be similar systems, but I don't know if it makes justice > comparing Elephant with AllegroCache client-server model. If I > understand it correctly (now), the current implementation of > Elephant is more similar to AllegroCache stand alone (non-client- > server) model. So, each web process that accesses Elephant can do > so seamlessly with the standard *store-controller* (assuming a > single store controller) and not have to deal with having to manage > connection pools and all that. Keeping this in mind, I would also > assume that in Elephant, I don't have to keep a handle to each > persistent class for each connection. Maybe this is what confused > me and maybe I shouldn't be reading AllegroCache's documentation :) Correct and correct. We implement the physical storage persistent classes _very_ differently than acache and trying to compare the system implications of using each is likely to be more confusing than helpful. Don't think of them as the same kind of system, they are two different systems optimizing different aspects of the common problem of implementing persistent classes. >> A simple conceptual model is that each thread has its own >> transaction. If these transactions are executing concurrently, the >> BDB library or SQL server logs the side read dependencies and the >> side effecting writes until a commit or abort. On abort, you throw >> away the log. On a commit, one transaction at a time writes their >> side effects and cancels any transaction that has read or written one >> of the entries written by the committing transaction. >> >> Thread isolation is guaranteed by a policy for using the BDB library >> or SQL server. Calls to these libraries are re-entrant. The current >> transaction ID (only used by one thread) determines where the reads >> and writes are logged (this is a little more complex for SQL, but >> handled by the data store and transparent to the user). > > I guess this goes back to what I just commented on the fact that > each web thread/request will use the connection in place in the > Lisp VM and not have to deal with establishing a new connection (I > could have checks to make sure that if the store controller is not > opened, I could open it, but once it's opened, I "shouldn't" have > to worry too much about it). Right? Correct. Elephant maintains a connection to a BDB session which maintains an open file of the underlying logs and database files. This is shared among threads because BDB is re-entrant and transaction ids are used to provide isolation in the presence of concurrency. > >> The only other thing we need to do is make sure that the parts of >> elephant shared across threads are themselves protected by Lisp >> locks. Most of this is the store-controller and some data structures >> used to initialize a new store controller. > > As an end-user application developer, do I need to worry about this > or should I expect the Elephant framework to handle it? Elephant handles it, sorry if I confused the issue but I thought you were trying to understand how elephant implements thread safety. > >> If you stick to the user contract in the documentation, you shouldn't >> have to worry further about interactions of multiple threads (other >> than the usual considerations if you have shared lisp variables >> instead of shared persistent objects). > > I would assume you are referring to my own application shared > variables and not Elephant-related variables, right? Yes >> I think that SQL databases are a safer bet than Berkeley DB >> for having several processes on different machines talking to the >> same >> store, so I will have one instance of postgresql running on a server >> with scsi raid 10 and lots of ram. > > Henrik, would you mind elaborating more on this? Why would SQL > databases be safer than the BDB stores? I know they are handled by > separate processes, potentially, on separate machines, so in > essence, they are independent of your application. However, isn't > BDB designed just to tackle that using an application library > instead of a separate process? The problem he is trying to solve is scaling computational power by using multiple CPUs and multiple servers. This is doable with Elephant so long as each independent lisp image is using the same data store spec. However, if you have two machines Berkeley DB in its normal mode will not work correctly as it's locking facilities require shared memory between all processes sharing a given disk database. So the multiple-CPU problem is solved by using N lisp processes for N CPUs with shared memory. However the multiple server problem requires a common server that all web servers can talk to. This is easier to setup with SQL than to write your own server on top of BDB. > Overall, and being this my first experience with Lisp and ODBs, I > really like Elephant. After reading some of AllegroCache's > documentation, I would still prefer using Elephant. Maybe I'm > trying to see deeper than I need to. Maybe I just need to see more > samples of real-world applications. I would love to contribute > sample applications to the project so as to make it clearer and > easier for others to learn, but I guess, I have to learn it myself > first. Code-wise, I think I have grasped the whole thing. However, > since I currently have no ability to test anything to a larger > scale, I'm trying to understand what it would take for an > application that uses Elephant to work in a large scale system > (both hardware and software). The biggest issue in scaling is when you think your application needs to be larger than a single server. Elephant is great for single server applications. When you scale to multiple servers it is because you are talking about high hundreds to thousands of concurrent sessions instead of dozens. That kind of traffic likely requires a highly reliable substrate and I'm not sure Elephant is sufficiently hardened that I could recommend it for that kind of use. Unless, of course, you want to pave new ground with it in which case I think Elephant can get there. > Thanks again for everyone of your comments. They did in fact help > me and am sure you follow up comments will further help me even > more. Now, while you guys digest this and reply to my post, I will > go back and read the updated Elephant manual :) > > Thanks, > Daniel Good luck, when you figure all this out a detailed summary of the primary things that confused you would be helpful in improving the documentation. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Sun May 27 10:08:18 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Sun, 27 May 2007 12:08:18 +0200 Subject: [elephant-devel] Understanding real-world use of Elephant In-Reply-To: <37737.192.168.1.71.1180125155.webmail@192.168.1.71> References: <37737.192.168.1.71.1180125155.webmail@192.168.1.71> Message-ID: <1180260498.7967.110.camel@trinidad> On Fri, 2007-05-25 at 16:32 -0400, lists at infoway.net wrote: > > > I think that SQL databases are a safer bet than Berkeley DB > > for having several processes on different machines talking to the same > > store, so I will have one instance of postgresql running on a server > > with scsi raid 10 and lots of ram. > > Henrik, would you mind elaborating more on this? Why would SQL databases be safer than the BDB stores? I know they are handled by separate processes, potentially, on separate machines, so in essence, they are independent of your application. However, isn't BDB designed just to tackle that using an application library instead of a separate process? I'm perhaps guided by feeling rather than theory. I have experience with sql databases and trust them completely with my data, I don't know BDB enough to do that. Also as Ian said they might be better when several machines are involved. And with postgresql I can choose between different replication methods so I can have a clustered solution if I want to. Maybe I can with BDB also, I don't know. Also, I have made some testing and evaluation. I intended to code tests for the different Lisp ODB to be able to see their merits, and I made the project http://common-lisp.net/project/grand-prix for that purpose. However, since several ODB failed on the first test I made up my mind early. The simple test is to see if several instances of Lisp can write to the same store at the same time. Rucksack can't and with elephant-BDB I had also problems. Ian tried to help me solve them, but the fact that I had to redesign the code a bit to make it work means I don't think BDB is as "safe" as the sql backends. All sql based backends (perec, clsql,postmodern) worked. If you come up with some nice performance test you might consider adding it to grand-prix, that was my idea with the project anyway. I have also done some stress testing with my web-app. BDB is fastest, but the sql backends do good enough for me. Actually I found a performance problem with and ODB from a well known Lisp vendor that would take too long time doing commits. But they immediately started to fix it, so it is probably solved by now. But I no longer have a license to check it out. Henrik Hjelte From eslick at csail.mit.edu Sun May 27 12:03:13 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 27 May 2007 08:03:13 -0400 Subject: [elephant-devel] Understanding real-world use of Elephant In-Reply-To: <1180260498.7967.110.camel@trinidad> References: <37737.192.168.1.71.1180125155.webmail@192.168.1.71> <1180260498.7967.110.camel@trinidad> Message-ID: <22BD3398-E310-41A1-BDAD-D5D392543000@csail.mit.edu> On May 27, 2007, at 6:08 AM, Henrik Hjelte wrote: > On Fri, 2007-05-25 at 16:32 -0400, lists at infoway.net wrote: >> >>> I think that SQL databases are a safer bet than Berkeley DB >>> for having several processes on different machines talking to the >>> same >>> store, so I will have one instance of postgresql running on a server >>> with scsi raid 10 and lots of ram. >> >> Henrik, would you mind elaborating more on this? Why would SQL >> databases be safer than the BDB stores? I know they are handled by >> separate processes, potentially, on separate machines, so in >> essence, they are independent of your application. However, isn't >> BDB designed just to tackle that using an application library >> instead of a separate process? > > I'm perhaps guided by feeling rather than theory. I have experience > with > sql databases and trust them completely with my data, I don't know BDB > enough to do that. Also as Ian said they might be better when several > machines are involved. And with postgresql I can choose between > different replication methods so I can have a clustered solution if I > want to. Maybe I can with BDB also, I don't know. > > Also, I have made some testing and evaluation. I intended to code > tests > for the different Lisp ODB to be able to see their merits, and I > made the > project http://common-lisp.net/project/grand-prix for that purpose. > However, since several ODB failed on the first test I made up my > mind early. The simple test is to see if several instances of Lisp can > write to the same store at the same time. Rucksack can't and with > elephant-BDB > I had also problems. Ian tried to help me solve them, but the fact > that I had to > redesign the code a bit to make it work means I don't think BDB is > as "safe" as the To be fair to BDB, it is a lower level interface so the Elephant designer, or user, has to take responsibility for configuring it properly for different situations. In exchange, you get better speed. SQL by default accommodates all these usage modes and so is safer in that it is harder to shoot yourself in the foot! > sql backends. All sql based backends (perec, clsql,postmodern) worked. > If you come up with some nice performance test you might consider > adding it > to grand-prix, that was my idea with the project anyway. > > I have also done some stress testing with my web-app. BDB is > fastest, but the > sql backends do good enough for me. Actually I found a performance > problem with > and ODB from a well known Lisp vendor that would take too long time > doing commits. > But they immediately started to fix it, so it is probably solved by > now. But I > no longer have a license to check it out. > > Henrik Hjelte > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From joubert at joubster.com Sun May 27 20:23:43 2007 From: joubert at joubster.com (Joubert Nel) Date: Sun, 27 May 2007 16:23:43 -0400 Subject: [elephant-devel] Storing objects in the root when don't care about the key Message-ID: <1180297423.29976.13.camel@joubster> Hi - I have a persistent class (i.e. :metaclass persistent-metaclass) defined and am adding a whole bunch of objects that are instances of this class to the root of my Elephant store. In order to retrieve a list of them I do: (map-class fn 'my-class-name) My question is: Currently I'm using the OID of the object as its key when I (add-to-root) because this is the only slot in my class that is guaranteed to be unique. (from a use-case point of view I don't care what the key is). I'm doing this to avoid overhead in generating more esoteric unique keys such as UUID's. Are there any reasons why I should NOT use OID as the key? Joubert From read at robertlread.net Mon May 28 03:46:00 2007 From: read at robertlread.net (Robert L. Read) Date: Sun, 27 May 2007 22:46:00 -0500 Subject: [elephant-devel] Storing objects in the root when don't care about the key In-Reply-To: <1180297423.29976.13.camel@joubster> References: <1180297423.29976.13.camel@joubster> Message-ID: <1180323961.5313.158.camel@localhost.localdomain> On Sun, 2007-05-27 at 16:23 -0400, Joubert Nel wrote: > Hi - > > I have a persistent class (i.e. :metaclass persistent-metaclass) defined > and am adding a whole bunch of objects that are instances of this class > to the root of my Elephant store. > > In order to retrieve a list of them I do: > (map-class fn 'my-class-name) > > My question is: > Currently I'm using the OID of the object as its key when I > (add-to-root) because this is the only slot in my class that is > guaranteed to be unique. (from a use-case point of view I don't care > what the key is). I'm afraid this may betray a lack of clarity in the documentation. If you are using a persistent-metclass, you probably DON'T want to call add-to-root on it. In general, if you are using Ian's persistent classes, there is no reason to explicitly call add-to-root---- I think you will find that "map-class" above let's you get to every class instance quite easily without calling add-to-root. The act calling "make-instance" on such a class is sufficient to place the instance in the DB, I think. > > I'm doing this to avoid overhead in generating more esoteric unique keys > such as UUID's. > > Are there any reasons why I should NOT use OID as the key? > > Joubert > > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From joubert at joubster.com Mon May 28 04:36:26 2007 From: joubert at joubster.com (Joubert Nel) Date: Mon, 28 May 2007 00:36:26 -0400 Subject: [elephant-devel] Storing objects in the root when don't care about the key / add same key to root more than once In-Reply-To: <1180323961.5313.158.camel@localhost.localdomain> References: <1180297423.29976.13.camel@joubster> <1180323961.5313.158.camel@localhost.localdomain> Message-ID: <1180326985.78.11.camel@joubster> On Sun, 2007-05-27 at 23:46, Robert L. Read wrote: > On Sun, 2007-05-27 at 16:23 -0400, Joubert Nel wrote: > > My question is: > > Currently I'm using the OID of the object as its key when I > > (add-to-root) because this is the only slot in my class that is > > guaranteed to be unique. (from a use-case point of view I don't care > > what the key is). > I'm afraid this may betray a lack of clarity in the documentation. If > you are using a persistent-metclass, > you probably DON'T want to call add-to-root on it. Thanks for your note, Robert. You're right; the examples in the documentation seem to imply that it is usual to (add-to-root), even for persistent classes (e.g. example code on p.41). > > In general, if you are using Ian's persistent classes, there is no > reason to explicitly call add-to-root---- > I think you will find that "map-class" above let's you get to every > class instance quite easily without > calling add-to-root. > > The act calling "make-instance" on such a class is sufficient to place > the instance in the DB, I think. Thanks for confirming - I suspected as much and am now no longer doing an add-to-root of persistent objects. On another note: In my experimentation I added several objects to the root, all with the same key. All these objects get added but there is the following curiosity: - I can do a map-class to get them all, BUT - map-root will only return one key/value pair for a given key (even when there are multiple objects stored against the same key) I'm wondering whether a design decision was made to allow addition of the same key more than once? Thanks for a great library. Joubert From gorbatovsky at gmail.com Mon May 28 09:05:25 2007 From: gorbatovsky at gmail.com (Dmitry V. Gorbatovsky) Date: Mon, 28 May 2007 12:05:25 +0300 Subject: [elephant-devel] Storing objects in the root when don't care about the key / add same key to root more than once In-Reply-To: <1180326985.78.11.camel@joubster> References: <1180297423.29976.13.camel@joubster> <1180323961.5313.158.camel@localhost.localdomain> <1180326985.78.11.camel@joubster> Message-ID: <200705281205.25694.dg@midasitech.com> On Monday 28 May 2007 07:36, Joubert Nel wrote: > On Sun, 2007-05-27 at 23:46, Robert L. Read wrote: > > On Sun, 2007-05-27 at 16:23 -0400, Joubert Nel wrote: > On another note: > In my experimentation I added several objects to the root, all with the > same key. > All these objects get added but there is the following curiosity: > - I can do a map-class to get them all, BUT > - map-root will only return one key/value pair for a given key (even > when there are multiple objects stored against the same key) > > I'm wondering whether a design decision was made to allow addition of > the same key more than once? > Thanks for a great library. > > Joubert > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel Nop , as far as I understand, map-class operate on OID and type information and returns all persistent objects , while root 'hash-table' keep only last key-value pair. Regards , Dmitry From eslick at csail.mit.edu Mon May 28 12:51:13 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 28 May 2007 08:51:13 -0400 Subject: [elephant-devel] Storing objects in the root when don't care about the key / add same key to root more than once In-Reply-To: <200705281205.25694.dg@midasitech.com> References: <1180297423.29976.13.camel@joubster> <1180323961.5313.158.camel@localhost.localdomain> <1180326985.78.11.camel@joubster> <200705281205.25694.dg@midasitech.com> Message-ID: <64753CD9-4B1E-495A-BAE8-5529DA2D5D27@csail.mit.edu> A few clarifications: Standard persistent classes are not saved by default. To get access to a class via the map-class operator the class or a slot of the class must be declared as :indexed. If a slot is indexed, then that value can be used as a key for the object via the get-instances-by- value interface. Moreover, you can have duplicate slot values and get-instances-by-value will return all the objects having that slot value. Think of the root more like a hash table (single key-value pair) instead of a set or collection. The simple get-value interface would be more complex if it could return multiple values. It is poor coding practice to use system-assigned OIDs. In future versions of elephant OIDs may change over time due to GC, auto recovery of oid space, etc. It's sort of like using the physical address of an object in a C++ program. There is a legitimate way to use OIDs outside of elephant, but we haven't yet defined an interface for users to do so (for example you might want to use OIDs to create many-to-many maps or efficient persistent sets) Ian On May 28, 2007, at 5:05 AM, Dmitry V. Gorbatovsky wrote: > On Monday 28 May 2007 07:36, Joubert Nel wrote: >> On Sun, 2007-05-27 at 23:46, Robert L. Read wrote: >>> On Sun, 2007-05-27 at 16:23 -0400, Joubert Nel wrote: > >> On another note: >> In my experimentation I added several objects to the root, all >> with the >> same key. >> All these objects get added but there is the following curiosity: >> - I can do a map-class to get them all, BUT >> - map-root will only return one key/value pair for a given key (even >> when there are multiple objects stored against the same key) >> >> I'm wondering whether a design decision was made to allow >> addition of >> the same key more than once? >> Thanks for a great library. >> >> Joubert >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > Nop , as far as I understand, map-class operate on OID and type > information > and returns all persistent objects , while root 'hash-table' keep > only last > key-value pair. > > Regards , Dmitry > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From joubert at joubster.com Mon May 28 13:26:30 2007 From: joubert at joubster.com (Joubert Nel) Date: Mon, 28 May 2007 09:26:30 -0400 Subject: [elephant-devel] Storing objects in the root when don't care about the key / add same key to root more than once In-Reply-To: <64753CD9-4B1E-495A-BAE8-5529DA2D5D27@csail.mit.edu> References: <1180297423.29976.13.camel@joubster> <1180323961.5313.158.camel@localhost.localdomain> <1180326985.78.11.camel@joubster> <200705281205.25694.dg@midasitech.com> <64753CD9-4B1E-495A-BAE8-5529DA2D5D27@csail.mit.edu> Message-ID: <1180358790.386.23.camel@joubster> On Mon, 2007-05-28 at 08:51, Ian Eslick wrote: > A few clarifications: > > Standard persistent classes are not saved by default. To get access Ian, when are objects that are instanced from persistent classes saved? I simply call (make-instance) on the following class definition: (defclass risk () ((title :accessor title :initarg :title :initform "")) (:metaclass elephant:persistent-metaclass) (:index t)) the resulting object is immediately saved/persisted by Elephant... > to a class via the map-class operator the class or a slot of the > class must be declared as :indexed. If a slot is indexed, then that > value can be used as a key for the object via the get-instances-by- > value interface. Moreover, you can have duplicate slot values and > get-instances-by-value will return all the objects having that slot > value. Understood - this is what I got from the documentation as well. > > Think of the root more like a hash table (single key-value pair) > instead of a set or collection. The simple get-value interface would > be more complex if it could return multiple values. My impression until I saw how persistent objects behave was that everything you want to store via Elephant had to be "rooted". > > It is poor coding practice to use system-assigned OIDs. In future > versions of elephant OIDs may change over time due to GC, auto > recovery of oid space, etc. It's sort of like using the physical > address of an object in a C++ program. There is a legitimate way to > use OIDs outside of elephant, but we haven't yet defined an interface > for users to do so (for example you might want to use OIDs to create > many-to-many maps or efficient persistent sets) Thanks for this insight. Many-to-many maps and sets are exactly what I'm trying to achieve. I found a newly released UUID package for Lisp - http://www.cl-user.net/asp/libs/uuid I'm going to explore using this instead of OIDs. Joubert From eslick at csail.mit.edu Mon May 28 13:54:28 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 28 May 2007 09:54:28 -0400 Subject: [elephant-devel] Storing objects in the root when don't care about the key / add same key to root more than once In-Reply-To: <1180358790.386.23.camel@joubster> References: <1180297423.29976.13.camel@joubster> <1180323961.5313.158.camel@localhost.localdomain> <1180326985.78.11.camel@joubster> <200705281205.25694.dg@midasitech.com> <64753CD9-4B1E-495A-BAE8-5529DA2D5D27@csail.mit.edu> <1180358790.386.23.camel@joubster> Message-ID: On May 28, 2007, at 9:26 AM, Joubert Nel wrote: > On Mon, 2007-05-28 at 08:51, Ian Eslick wrote: >> A few clarifications: >> >> Standard persistent classes are not saved by default. To get access > > Ian, when are objects that are instanced from persistent classes > saved? > I simply call (make-instance) on the following class definition: > > (defclass risk () > ((title :accessor title :initarg :title :initform "")) > (:metaclass elephant:persistent-metaclass) > (:index t)) > > the resulting object is immediately saved/persisted by Elephant... I don't want to force users to take on the indexing overhead for all persistent classes. The addition of the class :index option is what ensures that it will be saved. This option is added by default if any slot is indexed. > >> to a class via the map-class operator the class or a slot of the >> class must be declared as :indexed. If a slot is indexed, then that >> value can be used as a key for the object via the get-instances-by- >> value interface. Moreover, you can have duplicate slot values and >> get-instances-by-value will return all the objects having that slot >> value. > > Understood - this is what I got from the documentation as well. > >> >> Think of the root more like a hash table (single key-value pair) >> instead of a set or collection. The simple get-value interface would >> be more complex if it could return multiple values. > > My impression until I saw how persistent objects behave was that > everything you want to store via Elephant had to be "rooted". Did you read through the tutorial in the new manual? I thought I put hints into the section on roots that you can store via indexed classes and then the details should have been covered in the indexing part of the tutorial. Do you have suggestions on how we can clarify this point? >> >> It is poor coding practice to use system-assigned OIDs. In future >> versions of elephant OIDs may change over time due to GC, auto >> recovery of oid space, etc. It's sort of like using the physical >> address of an object in a C++ program. There is a legitimate way to >> use OIDs outside of elephant, but we haven't yet defined an interface >> for users to do so (for example you might want to use OIDs to create >> many-to-many maps or efficient persistent sets) > > Thanks for this insight. Many-to-many maps and sets are exactly > what I'm > trying to achieve. > I found a newly released UUID package for Lisp - > http://www.cl-user.net/asp/libs/uuid > I'm going to explore using this instead of OIDs. We would be interested in a solution that we could integrate as a contrib or as a standard component of elephant. I wrote a many-to- many map awhile back that was just a btree where the key was the source OID and the targets were a vector of OIDs (assuming that the cross product was populated). If you wanted to be able to to do the reverse you just create two of these btrees and ensure that they are always synchronized. I'm sure there are more efficient solutions, but this is relatively simple. There is already a persistent set class (PSET), but you don't want to be doing alot of operations on persistent sets because you can't reclaim the intermediate products. There should be a tutorial section on PSETs that talks about this. > Joubert > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From joubert at joubster.com Mon May 28 14:48:40 2007 From: joubert at joubster.com (Joubert Nel) Date: Mon, 28 May 2007 10:48:40 -0400 Subject: [elephant-devel] Storing objects in the root when don't care about the key / add same key to root more than once In-Reply-To: References: <1180297423.29976.13.camel@joubster> <1180323961.5313.158.camel@localhost.localdomain> <1180326985.78.11.camel@joubster> <200705281205.25694.dg@midasitech.com> <64753CD9-4B1E-495A-BAE8-5529DA2D5D27@csail.mit.edu> <1180358790.386.23.camel@joubster> Message-ID: <1180363720.512.46.camel@joubster> On Mon, 2007-05-28 at 09:54, Ian Eslick wrote: > On May 28, 2007, at 9:26 AM, Joubert Nel wrote: > > My impression until I saw how persistent objects behave was that > > everything you want to store via Elephant had to be "rooted". > > Did you read through the tutorial in the new manual? I thought I put > hints into the section on roots that you can store via indexed > classes and then the details should have been covered in the indexing > part of the tutorial. Do you have suggestions on how we can clarify > this point? Re-reading section 2.8, starting on p.12, I now see that it is stated "An indexed class stores every instance of the class that is created, ensuring that every object is automatically persisted between sessions." And the example that follows contains no (add-to-root) so I should have picked up on this. I think because all the other examples included (add-to-root) my mind was hopping along that one path! If you think it is warranted an explicit statement can be made somewhere (perhaps secion 2.5 where Persistent Classes are introduced, or section 2.8) that with Indexed Persistent Classes no (add-to-root) is necessary, in contrast to non-indexed Persistent Classes and other objects. Not crucial, though. > > >> > >> It is poor coding practice to use system-assigned OIDs. In future > >> versions of elephant OIDs may change over time due to GC, auto > >> recovery of oid space, etc. It's sort of like using the physical > >> address of an object in a C++ program. There is a legitimate way to > >> use OIDs outside of elephant, but we haven't yet defined an interface > >> for users to do so (for example you might want to use OIDs to create > >> many-to-many maps or efficient persistent sets) > > > > Thanks for this insight. Many-to-many maps and sets are exactly > > what I'm > > trying to achieve. > > I found a newly released UUID package for Lisp - > > http://www.cl-user.net/asp/libs/uuid > > I'm going to explore using this instead of OIDs. > > We would be interested in a solution that we could integrate as a > contrib or as a standard component of elephant. I wrote a many-to- > many map awhile back that was just a btree where the key was the > source OID and the targets were a vector of OIDs (assuming that the In light of your recommendation against using OIDs I'm now investigating using UUIDs. Once I have a useful generalized solution I'll share my experience/thoughts. Joubert > cross product was populated). If you wanted to be able to to do the > reverse you just create two of these btrees and ensure that they are > always synchronized. I'm sure there are more efficient solutions, > but this is relatively simple. > > There is already a persistent set class (PSET), but you don't want to > be doing alot of operations on persistent sets because you can't > reclaim the intermediate products. There should be a tutorial > section on PSETs that talks about this. > From reddaly at stanford.edu Mon May 28 19:35:48 2007 From: reddaly at stanford.edu (Red Daly) Date: Mon, 28 May 2007 12:35:48 -0700 Subject: [elephant-devel] Storing objects in the root when don't care about the key / add same key to root more than once In-Reply-To: <64753CD9-4B1E-495A-BAE8-5529DA2D5D27@csail.mit.edu> References: <1180297423.29976.13.camel@joubster> <1180323961.5313.158.camel@localhost.localdomain> <1180326985.78.11.camel@joubster> <200705281205.25694.dg@midasitech.com> <64753CD9-4B1E-495A-BAE8-5529DA2D5D27@csail.mit.edu> Message-ID: <465B2F14.3000604@stanford.edu> Ian Eslick wrote: > A few clarifications: > > It is poor coding practice to use system-assigned OIDs. In future > versions of elephant OIDs may change over time due to GC, auto > recovery of oid space, etc. It's sort of like using the physical > address of an object in a C++ program. There is a legitimate way to > use OIDs outside of elephant, but we haven't yet defined an interface > for users to do so (for example you might want to use OIDs to create > many-to-many maps or efficient persistent sets) When can we expect this legitimate interface? And if it is not expected soon, do you have a favorite hack? I have wanted foreign references to persistent objects in the past, and I suspect other users run into situations, too. > > Ian Red From eslick at csail.mit.edu Mon May 28 22:11:09 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 28 May 2007 18:11:09 -0400 Subject: [elephant-devel] Storing objects in the root when don't care about the key / add same key to root more than once In-Reply-To: <465B2F14.3000604@stanford.edu> References: <1180297423.29976.13.camel@joubster> <1180323961.5313.158.camel@localhost.localdomain> <1180326985.78.11.camel@joubster> <200705281205.25694.dg@midasitech.com> <64753CD9-4B1E-495A-BAE8-5529DA2D5D27@csail.mit.edu> <465B2F14.3000604@stanford.edu> Message-ID: <6EF282C3-9883-47E7-9608-466F531B4665@csail.mit.edu> Here is a proposal for approaching this problem. Currently the only problem with depending on foreign uses of OIDs is migration. There is an interface of sorts already, although I'm sure it could be improved or cleaned up with a little bit of work. If future features (i.e. online GC) add any other limitations on OID use, we can define the interface to that then. Today, to support migrating a custom data structure that uses OIDs instead of persistent object references, you need to write a method on migrate that ensures the following: 1) Every object referenced by an OID in your data structure has been copied to the target (see below for how to do this yourself) 2) Use elephant::retrieve-copied-object to get the new object and extract its OID 3) Write the new OIDs into your data structure replacing the old. This can be done cleanly by iterating over your set of OIDs doing something like this. (defun copy-and-get-oid (old-store new-store old-oid class) (let ((old (make-instance class :from-oid old-oid))) (unless (elephant::object-was-copied-p old) (migrate new-store old)) (elephant::oid (retrieve-copied-object old-store old)))) This means, however, that your data structure has to know the class of the target OID apriori. This will be improved in the next release or two by the addition of an OID->Class map in the store. I want to make sure the performance impact is not too bad before I implement this, so it may be some time. Let me know if you have any questions about this. Cheers, Ian On May 28, 2007, at 3:35 PM, Red Daly wrote: > Ian Eslick wrote: >> A few clarifications: >> >> It is poor coding practice to use system-assigned OIDs. In future >> versions of elephant OIDs may change over time due to GC, auto >> recovery of oid space, etc. It's sort oft like using the physical >> address of an object in a C++ program. There is a legitimate way >> to use OIDs outside of elephant, but we haven't yet defined an >> interface for users to do so (for example you might want to use >> OIDs to create many-to-many maps or efficient persistent sets) > > When can we expect this legitimate interface? And if it is not > expected soon, do you have a favorite hack? I have wanted foreign > references to persistent objects in ttaihe past, and I suspect > other users run into situations, too. >> >> Ian > > Red > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel