From leslie.polzer at gmx.net Fri Feb 1 10:48:53 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 1 Feb 2008 11:48:53 +0100 (CET) Subject: [elephant-devel] Small tutorial fixes Message-ID: <61438.88.73.202.44.1201862933.squirrel@mail.stardawn.org> Section 2.8 has plain wrong example code and doesn't specify the equality relation employed. Patch attached. Leslie -- My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Fri Feb 1 10:51:48 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 1 Feb 2008 11:51:48 +0100 (CET) Subject: [elephant-devel] Small tutorial fixes In-Reply-To: <61438.88.73.202.44.1201862933.squirrel@mail.stardawn.org> References: <61438.88.73.202.44.1201862933.squirrel@mail.stardawn.org> Message-ID: <61959.88.73.202.44.1201863108.squirrel@mail.stardawn.org> > Patch attached. Alright, so I lied here. But it's attached with this one. Leslie -------------- next part -------------- A non-text attachment was scrubbed... Name: tutorial.diff Type: text/x-patch Size: 992 bytes Desc: not available URL: From read at robertlread.net Sat Feb 2 02:14:01 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 01 Feb 2008 20:14:01 -0600 Subject: [elephant-devel] Small tutorial fixes In-Reply-To: <61959.88.73.202.44.1201863108.squirrel@mail.stardawn.org> References: <61438.88.73.202.44.1201862933.squirrel@mail.stardawn.org> <61959.88.73.202.44.1201863108.squirrel@mail.stardawn.org> Message-ID: <1201918441.9786.204.camel@penguin.yourdomain.com> Thank you. I will apply it by Monday. On Fri, 2008-02-01 at 11:51 +0100, Leslie P. Polzer wrote: > > Patch attached. > > Alright, so I lied here. But it's attached with this one. > > Leslie > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From Blog at common-lisp.net Sun Feb 3 07:41:00 2008 From: Blog at common-lisp.net (Blog at common-lisp.net) Date: 02 Feb 2008 23:41:00 -0800 Subject: [elephant-devel] How would you like to have your ad on 2 Million Websites ? Message-ID: <20080202234059.38AA7887EC3BFC23@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From read at robertlread.net Sun Feb 3 16:19:35 2008 From: read at robertlread.net (Robert L. Read) Date: Sun, 03 Feb 2008 10:19:35 -0600 Subject: [elephant-devel] Small tutorial fixes In-Reply-To: <61959.88.73.202.44.1201863108.squirrel@mail.stardawn.org> References: <61438.88.73.202.44.1201862933.squirrel@mail.stardawn.org> <61959.88.73.202.44.1201863108.squirrel@mail.stardawn.org> Message-ID: <1202055575.9786.253.camel@penguin.yourdomain.com> I have applied this patch. Next time, if you could uses the "darcs" diffing system rather than cvs, it would tend to make things a little easier. On Fri, 2008-02-01 at 11:51 +0100, Leslie P. Polzer wrote: > > Patch attached. > > Alright, so I lied here. But it's attached with this one. > > Leslie > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Sun Feb 3 17:16:32 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 3 Feb 2008 18:16:32 +0100 (CET) Subject: [elephant-devel] Small tutorial fixes In-Reply-To: <1202055575.9786.253.camel@penguin.yourdomain.com> References: <61438.88.73.202.44.1201862933.squirrel@mail.stardawn.org> <61959.88.73.202.44.1201863108.squirrel@mail.stardawn.org> <1202055575.9786.253.camel@penguin.yourdomain.com> Message-ID: <62288.88.73.202.142.1202058992.squirrel@mail.stardawn.org> > Next time, if you could uses the "darcs" diffing system rather than cvs, > it would tend to make things a little easier. Gladly. Do you mean "darcs diff" without the -u option? Leslie -- My personal blog: http://blog.viridian-project.de/ From Feed at common-lisp.net Tue Feb 5 08:15:29 2008 From: Feed at common-lisp.net (Feed at common-lisp.net) Date: 05 Feb 2008 00:15:29 -0800 Subject: [elephant-devel] Receive hundreds of targeted hits to your website every day from the links in the feeds! Message-ID: <20080205001529.42274631FECBBC26@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From leslie.polzer at gmx.net Mon Feb 4 19:01:30 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 4 Feb 2008 20:01:30 +0100 (CET) Subject: [elephant-devel] Another tiny documentation fix Message-ID: <62464.88.73.230.17.1202151690.squirrel@mail.stardawn.org> Amends OPEN-STORE/CLOSE-STORE docs. I hope the patch format is alright, now. If not, slap me again. Leslie -------------- next part -------------- A non-text attachment was scrubbed... Name: controller-doc-fix.diff Type: text/x-patch Size: 1156 bytes Desc: not available URL: From Auto at common-lisp.net Thu Feb 7 07:43:58 2008 From: Auto at common-lisp.net (Auto at common-lisp.net) Date: 06 Feb 2008 23:43:58 -0800 Subject: [elephant-devel] Got a Website? - You need MORE TRAFFIC Message-ID: <20080206234358.B3C0DF8B2CBCF455@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From Hit-Booster at common-lisp.net Sat Feb 9 06:30:16 2008 From: Hit-Booster at common-lisp.net (Hit-Booster at common-lisp.net) Date: 08 Feb 2008 22:30:16 -0800 Subject: [elephant-devel] How would you like unlimited hits to your website 15 minutes from now ? Message-ID: <20080208223016.3F9A264CE646E35D@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From read at robertlread.net Sun Feb 10 16:08:00 2008 From: read at robertlread.net (Robert L. Read) Date: Sun, 10 Feb 2008 10:08:00 -0600 Subject: [elephant-devel] Another tiny documentation fix In-Reply-To: <62464.88.73.230.17.1202151690.squirrel@mail.stardawn.org> References: <62464.88.73.230.17.1202151690.squirrel@mail.stardawn.org> Message-ID: <1202659680.9786.697.camel@penguin.yourdomain.com> I have applied this diff, thank you. It appears from the post that you are computing it against an "old" and new" repository, presumably local copies that you have made. This prevents me from being able to apply the patches directly. Although there may be circumstances in which you need to do that, in general if you have a patch we want to compute it as a difference between your local version and the "official" repository, which is the url on common-lisp.net. Darcs fully expects to work over the internet against such a repository. So what I do is just: 1) Type in and test and my change as appropriate. 2) Darcs record (a local operation) 3) Darcs send -o deleteme-later.diff (and that should just work, if not you might have to add at the end: http://www.common-lisp.net/project/elephant/darcs/elephant Now the file deleteme-later.diff is ready to be mailed to me. What I then do is ftp it onto the server, log in, and do: darcs apply deleteme-later.diff. On Mon, 2008-02-04 at 20:01 +0100, Leslie P. Polzer wrote: > Amends OPEN-STORE/CLOSE-STORE docs. > > I hope the patch format is alright, now. > If not, slap me again. > > Leslie > _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From Blog at common-lisp.net Mon Feb 11 09:08:50 2008 From: Blog at common-lisp.net (Blog at common-lisp.net) Date: 11 Feb 2008 01:08:50 -0800 Subject: [elephant-devel] How would you like to have your ad on 2 Million Websites ? Message-ID: <20080211010850.88A08044D0156281@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From leslie.polzer at gmx.net Sun Feb 10 17:13:25 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 10 Feb 2008 18:13:25 +0100 (CET) Subject: [elephant-devel] Another tiny documentation fix In-Reply-To: <1202659680.9786.697.camel@penguin.yourdomain.com> References: <62464.88.73.230.17.1202151690.squirrel@mail.stardawn.org> <1202659680.9786.697.camel@penguin.yourdomain.com> Message-ID: <61616.88.73.192.221.1202663605.squirrel@mail.stardawn.org> > It appears from the post that you are computing it against an "old" and > new" repository, presumably local copies that you have made. This > prevents me from being able to apply the patches directly. I was using darcs diff which automatically generated this. > So what I do is just: Thanks! I'll do it this way in the future. Leslie -- My personal blog: http://blog.viridian-project.de/ From Feed at common-lisp.net Wed Feb 13 11:38:43 2008 From: Feed at common-lisp.net (Feed at common-lisp.net) Date: 13 Feb 2008 03:38:43 -0800 Subject: [elephant-devel] Receive hundreds of targeted hits to your website every day from the links in the feeds! Message-ID: <20080213033843.87B359EE0E42CF59@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From ranga at mmsindia.com Wed Feb 13 04:58:00 2008 From: ranga at mmsindia.com (Rangarajan Krishnamoorthy) Date: Wed, 13 Feb 2008 10:28:00 +0530 Subject: [elephant-devel] QDBM Support Message-ID: <000f01c86dfd$065a16e0$0201a8c0@COM> Hi, I recently purchased LispWorks for Windows. Downloaded Elephant and was able to make it work with BDB. Thanks (and congrats!) for such a nice package. I have heard that QDBM is much better than BDB in terms of performance and does not have the same licensing issues (there are royalty payments for embeding BDB in an application). Do you have any idea of supporting QDBM or another similar backend? I saw that you plan to develop a native implementation. That is great news, but when will that happen? I am unable to use Allegro Common Lisp (and ACache) because of HUGE royalties, so I am looking at Elephant. Regards, Rangarajan -------------- next part -------------- An HTML attachment was scrubbed... URL: From leslie.polzer at gmx.net Wed Feb 13 08:41:30 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 13 Feb 2008 09:41:30 +0100 (CET) Subject: [elephant-devel] QDBM Support In-Reply-To: <000f01c86dfd$065a16e0$0201a8c0@COM> References: <000f01c86dfd$065a16e0$0201a8c0@COM> Message-ID: <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> > I recently purchased LispWorks for Windows. Downloaded Elephant and was able to make it > work with BDB. Thanks (and congrats!) for such a nice package. I have heard that QDBM is > much better than BDB in terms of performance and does not have the same licensing issues > (there are royalty payments for embeding BDB in an application). IANAL, but the licensing FAQ has a case that can be made be analogous to Elephant: ?Do I have to pay for a Berkeley DB license to use it in my Perl or Python scripts? No, you may use the Berkeley DB open source license at no cost. The Berkeley DB open source license requires that software that uses Berkeley DB be freely redistributable. In the case of Perl or Python, that software is Perl or Python, and not your scripts. Any scripts you write are your property, including scripts that make use of Berkeley DB. None of the Perl, Python or Berkeley DB licenses place any restrictions on what you may do with them.?[1] Also, when we talk about QDBM, I must ask: do you know of its successor, Tokyo Cabinet[2]? Leslie [1] http://www.oracle.com/technology/software/products/berkeley-db/htdocs/licensing.html [2] http://tokyocabinet.sourceforge.net/ -- My personal blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Wed Feb 13 14:39:41 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 13 Feb 2008 09:39:41 -0500 Subject: [elephant-devel] QDBM Support In-Reply-To: <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> Message-ID: Technicalities aside, isn't the spirit of that license essentially: "if you make money off BDB, we should too". So SVN is a product that is free, BDB is too. I also thought that commercial web sites using BDB as a store were intended to be covered too - that seems to be the community concensus. It's the definition of linking that is usually troublesome with Lisp and licensing. If we look at it like Python as you intend below, then only the Lisp image itself need be open source (i.e. SBCL). However Lisp is also compiled and dynamically linked with the image, which tends to make us look a little bit more like a dynamically linked program that GNU, in general, considers part of the application (i.e. contaminated) Has anyone ever really ruled on this point in the Lisp community? This is in large part the reason for the LLGPL approach that Franz takes. The GPL technically says that a GPL library contaminates the whole lisp image, including application code. This is why I advocated the switch of the Elephant code license from GPL to LLPGL. Actually, Franz might have some light to shed on this (they clearly had to pay for AllegroStore, but they might understand the licensing issues better). Ian On Feb 13, 2008, at 3:41 AM, Leslie P. Polzer wrote: > >> I recently purchased LispWorks for Windows. Downloaded Elephant and >> was able to make it >> work with BDB. Thanks (and congrats!) for such a nice package. I >> have heard that QDBM is >> much better than BDB in terms of performance and does not have the >> same licensing issues >> (there are royalty payments for embeding BDB in an application). > > IANAL, but the licensing FAQ has a case that can be made be > analogous to Elephant: > > ?Do I have to pay for a Berkeley DB license to use it in my Perl or > Python scripts? > > No, you may use the Berkeley DB open source license at no cost. The > Berkeley DB open > source license requires that software that uses Berkeley DB be > freely redistributable. > In the case of Perl or Python, that software is Perl or Python, and > not your scripts. > Any scripts you write are your property, including scripts that make > use of Berkeley DB. > None of the Perl, Python or Berkeley DB licenses place any > restrictions on what you may > do with them.?[1] > > Also, when we talk about QDBM, I must ask: do you know of its > successor, Tokyo Cabinet[2]? > > Leslie > > > [1] http://www.oracle.com/technology/software/products/berkeley-db/htdocs/licensing.html > [2] http://tokyocabinet.sourceforge.net/ > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Wed Feb 13 14:40:21 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 13 Feb 2008 09:40:21 -0500 Subject: [elephant-devel] QDBM Support In-Reply-To: <000f01c86dfd$065a16e0$0201a8c0@COM> References: <000f01c86dfd$065a16e0$0201a8c0@COM> Message-ID: <259B87DB-1E81-4D3A-AC38-4760499FF6FD@media.mit.edu> Does QDBM have inverted indices that work against their main BTrees? The Odium API looks like me like it is specialized on text data. You can do range queries over B-trees with several of QDBM's interfaces, but it doesn't look like the system links to the value indexed by a primary key from one BTree to another... That would require a separate BTree lookup for each secondary index lookup rather than a simple pointer chase. I'm sure this could be done, but I don't know what the end-to-end performance implications are, how easy it would be to do, etc. I suspect that the BDB implementation is a good template, so if you wanted to take on this problem I'd be happy to help. Henrik did a pretty good job of doing his own data store implementation without much input from us, so it is possible! :) Ian On Feb 12, 2008, at 11:58 PM, Rangarajan Krishnamoorthy wrote: > Hi, > I recently purchased LispWorks for Windows. Downloaded Elephant and > was able to make it work with BDB. Thanks (and congrats!) for such a > nice package. I have heard that QDBM is much better than BDB in > terms of performance and does not have the same licensing issues > (there are royalty payments for embeding BDB in an application). Do > you have any idea of supporting QDBM or another similar backend? I > saw that you plan to develop a native implementation. That is great > news, but when will that happen? > > I am unable to use Allegro Common Lisp (and ACache) because of HUGE > royalties, so I am looking at Elephant. > > Regards, > Rangarajan > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Wed Feb 13 15:11:16 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Wed, 13 Feb 2008 16:11:16 +0100 Subject: [elephant-devel] QDBM Support In-Reply-To: <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> I had never heard of this project, but I it seems that Tokyo Cabinet describes itself as fast, has transactions and can handle multiple clients which is good. And it has a tcp/ip interface and protocol so you wouldn't even need uffi/cffi to interface it from Lisp. Tokyo cabinet seems to map to the bdb model good, so it should probably be easier to do an interface than the sql interfaces. One observation though, do we need yet another backend at this time, there are other things to fix first on my personal wishlist. Henrik From eslick at media.mit.edu Wed Feb 13 15:41:38 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 13 Feb 2008 10:41:38 -0500 Subject: [elephant-devel] QDBM Support In-Reply-To: <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> Message-ID: In general, I'm with Henrik on this. I'd rather see us get Elephant to a reasonable degree of feature completeness before we start to add more non-lisp datastore functionality. You can use postmodern for licensing purposes and BDB for performance. The answer to all of this, I think, is having a native lisp version that has BDB's performance and no licensing restrictions. Then supporting the other two becomes: Postmodern for a higher degree of reliability as well as for distributed systems and BDB for legacy reasons. I have a pretty good idea in my head of what an all-lisp backend requires and having one would lay to rest all of these discussions of bringing up "yet another backend". Edi Weitz and I discussed collaborating on this, but unfortunately he had some other projects that took priority. Is there a small critical mass of people out there that care enough about this that they'd be willing to contribute to such a project? I don't have the time to do it on my own, but if we broke it up into small projects over the next handful of months, I don't think it's a ton of work. I can put in a solid chunk of integration work in mid to late April. So what is involved? The tricky problems I've discovered so far are: - An efficient model of BTree-like storage for Elephant 1) BDB-like paged data + explicit page cache + operations over fields 2) Something more customized? - Efficient pointer-based indexing (BTtree plus ptrs to data in main BTree) - Performing sorting and searching on serialized data rather than having to deserialize to sort as in the clsql backend (required to do BTree insertions) - Transaction/logging architecture; how to store transaction data, track conflicts, etc. (at lisp layer, in page cache ala BDB?) multi-thread and multi-process safe? - locking to enable transactions on all 3 platforms; multi-process safe? (Is there a free library that has a C library that does this already, I think having a simple library that compensates for some of the missing features in lisps is fine) Some additional considerations: - Do we add support for persistent heap garbage collection? - Do we want to add supports for large persistent sets? - Do we want a server mode for N:1 distributed transactions? This is by no means a trivial design, but I think if we sketched out the architecture there are a set of subsystems that could be made somewhat independent: - BTrees and disk storage - Database maintenance ops: (reconstruct DB from log files, dump db, optimize, etc) - transaction support and logging - low-level locking library - online garbage collection Cheers, Ian On Feb 13, 2008, at 10:11 AM, Henrik Hjelte wrote: > I had never heard of this project, but I it seems that Tokyo Cabinet > describes itself as fast, has transactions and can handle multiple > clients which is good. And it has a tcp/ip interface and protocol so > you wouldn't even need uffi/cffi to interface it from Lisp. Tokyo > cabinet seems to map to the bdb model good, so it should probably be > easier to do an interface than the sql interfaces. One observation > though, do we need yet another backend at this time, there are other > things to fix first on my personal wishlist. > > Henrik > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From enaeher at gmail.com Wed Feb 13 16:35:57 2008 From: enaeher at gmail.com (Eli Naeher) Date: Wed, 13 Feb 2008 10:35:57 -0600 Subject: [elephant-devel] notify_btree_update type error Message-ID: Hello, I apologize if I've overlooked something obvious, but is the current darcs Elephant with the current darcs Postmodern backend working with Postgres 8.3? I've been getting the error "Database error 42883: function notify_btree_update(integer, bigint) does not exist" when I try to store anything, and am trying to ascertain whether this is related to the Postgres version or something else. Thanks, --Eli From killerstorm at newmail.ru Wed Feb 13 18:18:29 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Wed, 13 Feb 2008 20:18:29 +0200 Subject: [elephant-devel] Re: QDBM Support References: <000f01c86dfd$065a16e0$0201a8c0@COM><61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> Message-ID: IE> It's the definition of linking that is usually troublesome with Lisp IE> and licensing. If we look at it like Python as you intend below, then IE> only the Lisp image itself need be open source (i.e. SBCL). However IE> Lisp is also compiled and dynamically linked with the image, which IE> tends to make us look a little bit more like a dynamically linked IE> program that GNU, in general, considers part of the application (i.e. IE> contaminated) heh, that's interesting.. it's legal to use BDB from CLISP (since technically CLISP doesn't differ from Perl or Python), but illegal from SBCL? From midfield at gmail.com Wed Feb 13 18:33:50 2008 From: midfield at gmail.com (Ben) Date: Wed, 13 Feb 2008 10:33:50 -0800 Subject: [elephant-devel] QDBM Support In-Reply-To: References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> Message-ID: <9157df230802131033j14cb712an983fbca1d21139eb@mail.gmail.com> one (perhaps insane) idea to make an all-lisp backend easier to implement was to leverage the underlying file system ala ZODB directory storage, since the file system is probably using B-trees anyways. there are fairly good architecture docs on http://dirstorage.sourceforge.net/ tokyo cabinet looks good too. b On Feb 13, 2008 7:41 AM, Ian Eslick wrote: > In general, I'm with Henrik on this. I'd rather see us get Elephant > to a reasonable degree of feature completeness before we start to add > more non-lisp datastore functionality. You can use postmodern for > licensing purposes and BDB for performance. > > The answer to all of this, I think, is having a native lisp version > that has BDB's performance and no licensing restrictions. Then > supporting the other two becomes: Postmodern for a higher degree of > reliability as well as for distributed systems and BDB for legacy > reasons. > > I have a pretty good idea in my head of what an all-lisp backend > requires and having one would lay to rest all of these discussions of > bringing up "yet another backend". Edi Weitz and I discussed > collaborating on this, but unfortunately he had some other projects > that took priority. > > Is there a small critical mass of people out there that care enough > about this that they'd be willing to contribute to such a project? I > don't have the time to do it on my own, but if we broke it up into > small projects over the next handful of months, I don't think it's a > ton of work. I can put in a solid chunk of integration work in mid to > late April. > > So what is involved? > > The tricky problems I've discovered so far are: > - An efficient model of BTree-like storage for Elephant > 1) BDB-like paged data + explicit page cache + operations over fields > 2) Something more customized? > - Efficient pointer-based indexing (BTtree plus ptrs to data in main > BTree) > - Performing sorting and searching on serialized data rather than > having to > deserialize to sort as in the clsql backend (required to do BTree > insertions) > - Transaction/logging architecture; how to store transaction data, > track conflicts, etc. > (at lisp layer, in page cache ala BDB?) > multi-thread and multi-process safe? > - locking to enable transactions on all 3 platforms; multi-process safe? > (Is there a free library that has a C library that does this already, > I think having a simple library that compensates for some of the > missing > features in lisps is fine) > > Some additional considerations: > - Do we add support for persistent heap garbage collection? > - Do we want to add supports for large persistent sets? > - Do we want a server mode for N:1 distributed transactions? > > This is by no means a trivial design, but I think if we sketched out > the architecture there are a set of subsystems that could be made > somewhat independent: > - BTrees and disk storage > - Database maintenance ops: (reconstruct DB from log files, dump db, > optimize, etc) > - transaction support and logging > - low-level locking library > - online garbage collection > > Cheers, > Ian > > > On Feb 13, 2008, at 10:11 AM, Henrik Hjelte wrote: > > > I had never heard of this project, but I it seems that Tokyo Cabinet > > describes itself as fast, has transactions and can handle multiple > > clients which is good. And it has a tcp/ip interface and protocol so > > you wouldn't even need uffi/cffi to interface it from Lisp. Tokyo > > cabinet seems to map to the bdb model good, so it should probably be > > easier to do an interface than the sql interfaces. One observation > > though, do we need yet another backend at this time, there are other > > things to fix first on my personal wishlist. > > > > Henrik > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From killerstorm at newmail.ru Wed Feb 13 18:45:19 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Wed, 13 Feb 2008 20:45:19 +0200 Subject: [elephant-devel] Re: notify_btree_update type error References: Message-ID: EN> I apologize if I've overlooked something obvious, but is the current EN> darcs Elephant with the current darcs Postmodern backend working with EN> Postgres 8.3? i suspect nobody tested it. i'm not sure if i'll be able to test it in near future, because it's not yet in Debian packages.. EN> I've been getting the error "Database error 42883: function EN> notify_btree_update(integer, bigint) does not exist" when I try to EN> store anything, and am trying to ascertain whether this is related to EN> the Postgres version or something else. this could happen if you've created store with "old code" (three monthes ago or so) and use it with a new code. is that possible? function notify_btree_update is created when creating new store with ele-global-sync-cache support (maybe something gone wrong with new version of PostgreSQL..) you can do this manually via PostgreSQL command line: CREATE FUNCTION notify_btree_update (id integer, the_key text) RETURNS void AS $$ BEGIN END; $$ LANGUAGE plpgsql; or you can disable ele-global-sync-cache -- comment out (pushnew :ele-global-sync-cache cl::*features*) in pm-sql.lisp and recompile the stuff. From enaeher at gmail.com Wed Feb 13 20:03:13 2008 From: enaeher at gmail.com (Eli Naeher) Date: Wed, 13 Feb 2008 14:03:13 -0600 Subject: [elephant-devel] Re: notify_btree_update type error In-Reply-To: References: Message-ID: On Feb 13, 2008 12:45 PM, Alex Mizrahi wrote: > this could happen if you've created store with "old code" (three monthes ago > or so) and use it with a new code. > is that possible? It looks like this is what happened. I did create a new database to connect to as a new store when I switched to the newer Elephant, but wasn't really thinking about Postgres function definitions being global across databases. Thanks for you help, --Eli From eslick at media.mit.edu Wed Feb 13 21:06:07 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 13 Feb 2008 16:06:07 -0500 Subject: [elephant-devel] QDBM Support In-Reply-To: <9157df230802131033j14cb712an983fbca1d21139eb@mail.gmail.com> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <9157df230802131033j14cb712an983fbca1d21139eb@mail.gmail.com> Message-ID: <9BB7F71E-6879-4861-85C5-A3560237E884@media.mit.edu> Re: Tokyo Cabinet The API is nearly identical to BDB's. I think a TC version of the datastore would be pretty easy to do. The only way it makes sense to me to do this is to deprecate the BDB data store as of the next major release. Any thoughts on this? Depreciation: Speaking of which, what is the thinking on the CL-SQL store? With postmodern, there is a SQL interface - do we want to maintain CL-SQL long term? It does cover more SQL backends, including the easy-to-use SQLite, but it's another fork to maintain. I'm agnostic, but I thought I'd toss that out for Robert to comment on. Re: DirectoryStorage That's essentially the idea behind Rucksack, if I recall correctly. Finding a way to leverage the existing filesystem is a good idea. That may be a way to bypass the low-level locking, Btree design, paging and caching issues in the near term and perhaps the long term. Does anyone have a sense of the performance implications of this vs. BDB? Some potential issues: - Lots of file handles being created and destroyed (i.e. When walking an index, to get a primary value, you have to open and close each object's file) - Lots of open file handles! - Efficient index implementation - Secondary indices - We have to update whole objects on each commit, not just slot values as today. This may or may not matter given that an object usually lives on one BDB page and locking in BDB is done at the page level... - I think we still need a C interface to a POSIX function to lock a file explicitly; does Windows have a similar interface? Ian On Feb 13, 2008, at 1:33 PM, Ben wrote: > one (perhaps insane) idea to make an all-lisp backend easier to > implement was to leverage the underlying file system ala ZODB > directory storage, since the file system is probably using B-trees > anyways. there are fairly good architecture docs on > > http://dirstorage.sourceforge.net/ > > tokyo cabinet looks good too. > > b > > On Feb 13, 2008 7:41 AM, Ian Eslick wrote: >> In general, I'm with Henrik on this. I'd rather see us get Elephant >> to a reasonable degree of feature completeness before we start to add >> more non-lisp datastore functionality. You can use postmodern for >> licensing purposes and BDB for performance. >> >> The answer to all of this, I think, is having a native lisp version >> that has BDB's performance and no licensing restrictions. Then >> supporting the other two becomes: Postmodern for a higher degree of >> reliability as well as for distributed systems and BDB for legacy >> reasons. >> >> I have a pretty good idea in my head of what an all-lisp backend >> requires and having one would lay to rest all of these discussions of >> bringing up "yet another backend". Edi Weitz and I discussed >> collaborating on this, but unfortunately he had some other projects >> that took priority. >> >> Is there a small critical mass of people out there that care enough >> about this that they'd be willing to contribute to such a project? I >> don't have the time to do it on my own, but if we broke it up into >> small projects over the next handful of months, I don't think it's a >> ton of work. I can put in a solid chunk of integration work in mid >> to >> late April. >> >> So what is involved? >> >> The tricky problems I've discovered so far are: >> - An efficient model of BTree-like storage for Elephant >> 1) BDB-like paged data + explicit page cache + operations over >> fields >> 2) Something more customized? >> - Efficient pointer-based indexing (BTtree plus ptrs to data in main >> BTree) >> - Performing sorting and searching on serialized data rather than >> having to >> deserialize to sort as in the clsql backend (required to do BTree >> insertions) >> - Transaction/logging architecture; how to store transaction data, >> track conflicts, etc. >> (at lisp layer, in page cache ala BDB?) >> multi-thread and multi-process safe? >> - locking to enable transactions on all 3 platforms; multi-process >> safe? >> (Is there a free library that has a C library that does this >> already, >> I think having a simple library that compensates for some of the >> missing >> features in lisps is fine) >> >> Some additional considerations: >> - Do we add support for persistent heap garbage collection? >> - Do we want to add supports for large persistent sets? >> - Do we want a server mode for N:1 distributed transactions? >> >> This is by no means a trivial design, but I think if we sketched out >> the architecture there are a set of subsystems that could be made >> somewhat independent: >> - BTrees and disk storage >> - Database maintenance ops: (reconstruct DB from log files, dump db, >> optimize, etc) >> - transaction support and logging >> - low-level locking library >> - online garbage collection >> >> Cheers, >> Ian >> >> >> On Feb 13, 2008, at 10:11 AM, Henrik Hjelte wrote: >> >>> I had never heard of this project, but I it seems that Tokyo Cabinet >>> describes itself as fast, has transactions and can handle multiple >>> clients which is good. And it has a tcp/ip interface and protocol so >>> you wouldn't even need uffi/cffi to interface it from Lisp. Tokyo >>> cabinet seems to map to the bdb model good, so it should probably be >>> easier to do an interface than the sql interfaces. One observation >>> though, do we need yet another backend at this time, there are other >>> things to fix first on my personal wishlist. >>> >>> Henrik >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Wed Feb 13 21:37:20 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 13 Feb 2008 16:37:20 -0500 Subject: [elephant-devel] QDBM Support In-Reply-To: <9BB7F71E-6879-4861-85C5-A3560237E884@media.mit.edu> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <9157df230802131033j14cb712an983fbca1d21139eb@mail.gmail.com> <9BB7F71E-6879-4861-85C5-A3560237E884@media.mit.edu> Message-ID: <381DE414-7B9F-4960-8BA0-1DA418C8FC53@media.mit.edu> Also, from what I can tell, it doesn't run on Windows yet so that is another consideration... On Feb 13, 2008, at 4:06 PM, Ian Eslick wrote: > Re: Tokyo Cabinet > > The API is nearly identical to BDB's. I think a TC version of the > datastore would be pretty easy to do. The only way it makes sense > to me to do this is to deprecate the BDB data store as of the next > major release. Any thoughts on this? > From henrik at evahjelte.com Wed Feb 13 22:41:29 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Wed, 13 Feb 2008 23:41:29 +0100 Subject: [elephant-devel] QDBM Support In-Reply-To: <9157df230802131033j14cb712an983fbca1d21139eb@mail.gmail.com> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <9157df230802131033j14cb712an983fbca1d21139eb@mail.gmail.com> Message-ID: <50e8e4f60802131441v3b8cabd9teb0b049ae19c685a@mail.gmail.com> On using the filesystem for storage. I once made an experimental backend for rucksack that used files (one per object). I tried several filesystems including ReiserFS (which uses B+trees), but the performance was not good. So, I think it is no accident that the dirstorage page does not mention performance anywhere on the features, except fast startup and shutdown. /Henrik From jla415 at gmail.com Wed Feb 13 22:50:16 2008 From: jla415 at gmail.com (Jason Anderson) Date: Wed, 13 Feb 2008 14:50:16 -0800 Subject: [elephant-devel] QDBM Support In-Reply-To: <50e8e4f60802131441v3b8cabd9teb0b049ae19c685a@mail.gmail.com> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <9157df230802131033j14cb712an983fbca1d21139eb@mail.gmail.com> <50e8e4f60802131441v3b8cabd9teb0b049ae19c685a@mail.gmail.com> Message-ID: <5c2dc04a0802131450l6a7e9af7j32d24798a3524f17@mail.gmail.com> I don't claim to be an expert and I don't exactly remember the details but historically the linux/*bsd filesystems didn't like it when you had tens of thousands or more files in a single directory... using a deep directory structure such as /c/d/f/cdf383d3.dat helped alleviate the problem somewhat but I think there was still an issue with the amount of memory used just caching all the inodes and such and extreme slowness trying to delete many small files the newer filesystems may fare better but they didn't exist back in those days that I was exploring this sort of thing On Feb 13, 2008 2:41 PM, Henrik Hjelte wrote: > On using the filesystem for storage. I once made an experimental > backend for rucksack that used files (one per object). I tried several > filesystems including ReiserFS (which uses B+trees), but the > performance was not good. So, I think it is no accident that the > dirstorage page does not mention performance anywhere on the features, > except fast startup and shutdown. > > /Henrik > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From henrik at evahjelte.com Wed Feb 13 23:31:20 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 14 Feb 2008 00:31:20 +0100 Subject: [elephant-devel] QDBM Support In-Reply-To: References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> Message-ID: <50e8e4f60802131531j59178cb5jf215f95db74a7c00@mail.gmail.com> About the Lisp backend project, it is an interesting project and I really would like to help, but I don't really know if I have a lot of time during the coming months. I will try to help as much as I can though. On my CV is that I have spent some days to find a subtle bug in the the rucksack btree implementation. Also I have written some braindump notes of my own ideas of similar projects. About transactions, I think the rucksack model with multiple concurrent object versions is nice. Another idea from rucksack, having the btrees implemented as lisp objects that are serialized and deserialized just like other objects, is that an idea worth considering? It probably makes the implementation a little bit easier, but I am speculating that it may give worse performance. About garbage collection. If it is necessary at the beginning, I would like a design that it should be done by a second independent process so it doesn't degrade performance for the client program. Great idea this, Thanks, /Henrik From read at robertlread.net Thu Feb 14 02:54:27 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 13 Feb 2008 20:54:27 -0600 Subject: [elephant-devel] QDBM Support In-Reply-To: <9BB7F71E-6879-4861-85C5-A3560237E884@media.mit.edu> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <9157df230802131033j14cb712an983fbca1d21139eb@mail.gmail.com> <9BB7F71E-6879-4861-85C5-A3560237E884@media.mit.edu> Message-ID: <1202957667.6267.60.camel@penguin.yourdomain.com> On Wed, 2008-02-13 at 16:06 -0500, Ian Eslick wrote: > Re: Tokyo Cabinet > > The API is nearly identical to BDB's. I think a TC version of the > datastore would be pretty easy to do. The only way it makes sense > to > me to do this is to deprecate the BDB data store as of the next > major > release. Any thoughts on this? I would ask the original poster if we have strong reason to believe BDB is not fast for his original purposes. I certainly don't mind considering implementing QDBM, but if a particular user wants that before they have shown that BDB is not fast enough for their purposes, it seems to me that this is a form of "gold-plating". I personally would rather work on clean schema evolution than a QDBM backend; but if someone else wants to do the work I will support it. > > Depreciation: > > Speaking of which, what is the thinking on the CL-SQL store? With > postmodern, there is a SQL interface - do we want to maintain CL-SQL > long term? It does cover more SQL backends, including the > easy-to-use > SQLite, but it's another fork to maintain. I'm agnostic, but I > thought I'd toss that out for Robert to comment on. > I think the CL-SQL is worth preserving at present for three reasons. The SQLite support is the first; secondly fact that it has helped find bugs in the postmodern backend (though only as a reference implementation), and finally the fact that it is the shortest path to MySQL, Oracle, or SQLServer integration. That is, if someone wanted to run Elephant or Oracle, or more likely MySQL, in just a few days they could get the CL-SQL backend to do that. It would of course have the problems that Henrik and Alex improved upon in the postmodern backend, and which Ian has written about, which are basically that it is slow and might be very slow in some usage patterns. I think the time to deprecate the CL-SQL backend might come when we actually implement a feature which it makes harder to support. For example, a LISP-Native backend is orthogonal to the CL-SQL backend, but implementing a more abstract "Set/Ordered Set" model as we have discussed might not be. So as soon as it actually becomes a pain to support it, we can consider it. (I would love to know who is using SQLite 3 right now, or even how many total users we have at all.) From read at robertlread.net Thu Feb 14 03:42:14 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 13 Feb 2008 21:42:14 -0600 Subject: [elephant-devel] QDBM Support In-Reply-To: <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> Message-ID: <1202960534.6267.76.camel@penguin.yourdomain.com> I am not a lawyer either, but I believe there is some confusion. You correctly give a link to the most recent versions of BDB, which is released by Oracle under quite different conditions than the original sleepycat license. By my reading of this license, http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html 1) whether your purpose is commercial or not has no relevance. 2) If you were to redistribute a binary or source that uses BDB, you would have to redistribute the code that you wrote in source form along with it. There is some question as to whether presenting a website constitutes "redistribution", but I think it does not. Therefore, in contrast to some of the things mentioned in this thread, it seems to me that: 1) You can build a website against BDB without paying for it. 2) If you ship, release, or distribute a product, for free or for money, that uses BDB (I would interpret that to mean "uses in any way"), you must either distribute the source code or purchase some other license from Oracle. This is especially confusing because in fact I wrote the CL-SQL backend (some 3 years ago) not to avoid THIS license, but to avoid the SLEEPYCAT license. After the CL-SQL backend was a part of Elephant, Oracle bought Sleepycat and released BDB under this current license. In my non-professional opinion, the original purpose I had, of presenting a website with Elephant without purchasing BDB, has now evaporated. So it goes. That work still has a bit of value, since it enabled the postmodern backend and might help other integrations, and someone could very well wish to redistribute a product with Elephant. However, from the point of view of hosting a website, I think one can currently use BDB. After I wrote the CL-SQL backend, Ian Eslick cleaned things up a great deal and then implemented the class-based persistence (as opposed to the raw-btree model that was present from the beginning.) Then Alex and Henrik wrote the postmodern backend, which is much better than the CL-SQL backend. Although a lot of this is subjective, I think the fact that we can operate against multiple backends remains a great strength, and that will be true even after a LISP-native backend is implemented. On Wed, 2008-02-13 at 09:41 +0100, Leslie P. Polzer wrote: > > I recently purchased LispWorks for Windows. Downloaded Elephant and was able to make it > > work with BDB. Thanks (and congrats!) for such a nice package. I have heard that QDBM is > > much better than BDB in terms of performance and does not have the same licensing issues > > (there are royalty payments for embeding BDB in an application). > > IANAL, but the licensing FAQ has a case that can be made be analogous to Elephant: > > ?Do I have to pay for a Berkeley DB license to use it in my Perl or Python scripts? > > No, you may use the Berkeley DB open source license at no cost. The Berkeley DB open > source license requires that software that uses Berkeley DB be freely redistributable. > In the case of Perl or Python, that software is Perl or Python, and not your scripts. > Any scripts you write are your property, including scripts that make use of Berkeley DB. > None of the Perl, Python or Berkeley DB licenses place any restrictions on what you may > do with them.?[1] > > Also, when we talk about QDBM, I must ask: do you know of its successor, Tokyo Cabinet[2]? > > Leslie > > > [1] http://www.oracle.com/technology/software/products/berkeley-db/htdocs/licensing.html > [2] http://tokyocabinet.sourceforge.net/ > From read at robertlread.net Thu Feb 14 04:08:04 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 13 Feb 2008 22:08:04 -0600 Subject: [elephant-devel] QDBM Support In-Reply-To: References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> Message-ID: <1202962084.6267.91.camel@penguin.yourdomain.com> On Wed, 2008-02-13 at 10:41 -0500, Ian Eslick wrote: > The answer to all of this, I think, is having a native lisp version > that has BDB's performance and no licensing restrictions. Then > supporting the other two becomes: Postmodern for a higher degree of > reliability as well as for distributed systems and BDB for legacy > reasons. > > I have a pretty good idea in my head of what an all-lisp backend > requires and having one would lay to rest all of these discussions > of > bringing up "yet another backend". Edi Weitz and I discussed > collaborating on this, but unfortunately he had some other projects > that took priority. > > Is there a small critical mass of people out there that care enough > about this that they'd be willing to contribute to such a project? > I > don't have the time to do it on my own, but if we broke it up into > small projects over the next handful of months, I don't think it's a > ton of work. I can put in a solid chunk of integration work in mid > to > late April. > I completely agree with Ian about the value of a LISP-native backend. However, I can not personally offer to help with this. I have in fact abandoned my business plans for the time being and taken a normal job. Moreover, since I did the "schema evolution" system that we used in the Java application for Hire.com a while back, I feel more comfortable working on that than on the LISP native backend, although I think both are wonderful and challenging problems. The excellent set of automated tests produced by the original authors of Elephant (Andrew Blumberg and Ben Lee) and strengthened by Ian and myself and others since then remain the greatest asset in undertaking the LISP-Native backend. I know that many of you understand most of the technical challenges in bringing up a Native LISP backend better than I do. However, let me ask the question that my acquaintance Kent Beck always asks: What is the simplest thing that could possible work? By which I mean, is there any value in making a very simple LISP native backend? Forget locking, transactions, efficiency, and all that other ocean-boiling stuff that we all know will be needed for an enterprise application. Can anybody build a Native-Lisp backend in a weekend? If so, we would have an excellent "spike" solution that would inform further efforts, and furthermore we would have an out-of-the-box solution for demoing Elephant that would require ZERO additional software installations. This would be very useful, even if there are performance and reliability limits to that backend. From ranga at mmsindia.com Thu Feb 14 05:05:47 2008 From: ranga at mmsindia.com (Rangarajan Krishnamoorthy) Date: Thu, 14 Feb 2008 10:35:47 +0530 Subject: [elephant-devel] QDBM Support References: <000f01c86dfd$065a16e0$0201a8c0@COM><61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org><50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <1202962084.6267.91.camel@penguin.yourdomain.com> Message-ID: <000d01c86ec7$47122d60$0201a8c0@COM> Robert, I am the one who started this thread, so let me clarify. The background: I am starting to work on a .NET-based application (for Windows platforms) that might benefit by Lisp integration. I purchased Lispworks Enterprise in this context. When I searched for a lisp persistence library, I was told to consider Elephant. I tried it and it seems to work fine for me. Elephant documentation advises that BDB is the best performing backend among the backends it supports. But it appears that I cannot just "buy" BDB by opting for a one-time initial payment. I have to pay runtime royalties. This is different from other libraries that I have used so far. Although there may be legitimate reasons why Oracle does it this way, I am more comfortable with one-time payment. So the problem I am facing is that I would love to use Elephant in my work, but BDB's licensing is the concern. I found out that QDBM is another popular DB implementation which does not have this licensing issue, so posted the question to this list. It is not just that QDBM outperforms BDB (I don't know this). Please note that I am not against paying for the library that I use, but just that I am not in favor of royalties. Coming to your suggestion that it would be wise to build a very simple, native lisp backend without the "frills", I am for it! Regards, Rangarajan ----- Original Message ----- From: "Robert L. Read" To: "Elephant bugs and development" Sent: Thursday, February 14, 2008 9:38 AM Subject: Re: [elephant-devel] QDBM Support > On Wed, 2008-02-13 at 10:41 -0500, Ian Eslick wrote: >> The answer to all of this, I think, is having a native lisp version >> that has BDB's performance and no licensing restrictions. Then >> supporting the other two becomes: Postmodern for a higher degree of >> reliability as well as for distributed systems and BDB for legacy >> reasons. >> >> I have a pretty good idea in my head of what an all-lisp backend >> requires and having one would lay to rest all of these discussions >> of >> bringing up "yet another backend". Edi Weitz and I discussed >> collaborating on this, but unfortunately he had some other projects >> that took priority. >> >> Is there a small critical mass of people out there that care enough >> about this that they'd be willing to contribute to such a project? >> I >> don't have the time to do it on my own, but if we broke it up into >> small projects over the next handful of months, I don't think it's a >> ton of work. I can put in a solid chunk of integration work in mid >> to >> late April. >> > > I completely agree with Ian about the value of a LISP-native backend. > > However, I can not personally offer to help with this. I have in fact > abandoned my business plans for the time being and taken a normal job. > Moreover, since I did the "schema evolution" system that we used in the > Java application for Hire.com a while back, I feel more comfortable > working on that than on the LISP native backend, although I think both > are wonderful and challenging problems. > > The excellent set of automated tests produced by the original authors of > Elephant (Andrew Blumberg and Ben Lee) and strengthened by Ian and > myself and others since then remain the greatest asset in undertaking > the LISP-Native backend. > > I know that many of you understand most of the technical challenges in > bringing up a Native LISP backend better than I do. However, let me ask > the question that my acquaintance Kent Beck always asks: > > What is the simplest thing that could possible work? > > By which I mean, is there any value in making a very simple LISP native > backend? Forget locking, transactions, efficiency, and all that other > ocean-boiling stuff that we all know will be needed for an enterprise > application. Can anybody build a Native-Lisp backend in a weekend? > > If so, we would have an excellent "spike" solution that would inform > further efforts, and furthermore we would have an out-of-the-box > solution for demoing Elephant that would require ZERO additional > software installations. This would be very useful, even if there are > performance and reliability limits to that backend. > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Thu Feb 14 05:31:39 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 13 Feb 2008 23:31:39 -0600 Subject: [elephant-devel] QDBM Support In-Reply-To: <000d01c86ec7$47122d60$0201a8c0@COM> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <1202962084.6267.91.camel@penguin.yourdomain.com> <000d01c86ec7$47122d60$0201a8c0@COM> Message-ID: <1202967099.6267.108.camel@penguin.yourdomain.com> On Thu, 2008-02-14 at 10:35 +0530, Rangarajan Krishnamoorthy wrote: > Robert, > I am the one who started this thread, so let me clarify. > > The background: I am starting to work on a .NET-based application (for > Windows platforms) that might benefit by Lisp integration. I purchased > Lispworks Enterprise in this context. When I searched for a lisp persistence > library, I was told to consider Elephant. I tried it and it seems to work > fine for me. Elephant documentation advises that BDB is the best performing > backend among the backends it supports. But it appears that I cannot just > "buy" BDB by opting for a one-time initial payment. I have to pay runtime > royalties. This is different from other libraries that I have used so far. > Although there may be legitimate reasons why Oracle does it this way, I am > more comfortable with one-time payment. So the problem I am facing is that I > would love to use Elephant in my work, but BDB's licensing is the concern. I > found out that QDBM is another popular DB implementation which does not have > this licensing issue, so posted the question to this list. It is not just > that QDBM outperforms BDB (I don't know this). Please note that I am not > against paying for the library that I use, but just that I am not in favor > of royalties. Thanks for the clarification --- I got tangled in the thread. However, this allows me to rephrase my question: Can you consider using the postmodern backend (which is an interface to the PostGres database)? PostGres is free. The Postmodern backend is probably 3 to 10 times slower than BDB, and might be even worse in some circumstances. However, it is entirely possible that 10 times very fast is still fast enough for whatever operation you intend. It is unfortunate that to test this you will have to install PostGres and do a very, very small amount of db administration on PostGres in order to execute Elephant against it. With experience this should only take a few hours. I share your attitude toward royalties in this matter, which is mainly the reason I wrote the CL-SQL-based backend. I know only what you have stated about your application, but one possibility, if you have no better alternative to Elephant, is to use the postmodern backend now, during your development, and be prepared to either switch to BDB by paying for it later, or to hope for (and perhaps help) an effort to either implement QDBM or a LISP-native backend. Allow me to ask if you have really carefully considered the BDB license: http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html ...and concluded that you would have to pay a runtime royalty for your specific application? (As an aside: I have not studied the other persistence solutions deeply. I suspect that using CL-SQL directly (not through Elephant) is one of the most common and well-tested mechanisms. This is an Object-Relational Mapper (ORM) similar to Hibernate or Ibatis (although a bit less evolved, I think.) This approach would require you to do a great deal more work than Elephant does, since you have to create tables and consciously map your classes into those tables and be prepared to deal with schema evolution when you change the design of your object space. I think Elephant is much more convenient, but using CL-SQL directly is an option.) > > Coming to your suggestion that it would be wise to build a very simple, > native lisp backend without the "frills", I am for it! > > Regards, > Rangarajan > > ----- Original Message ----- > From: "Robert L. Read" > To: "Elephant bugs and development" > Sent: Thursday, February 14, 2008 9:38 AM > Subject: Re: [elephant-devel] QDBM Support > > > > On Wed, 2008-02-13 at 10:41 -0500, Ian Eslick wrote: > >> The answer to all of this, I think, is having a native lisp version > >> that has BDB's performance and no licensing restrictions. Then > >> supporting the other two becomes: Postmodern for a higher degree of > >> reliability as well as for distributed systems and BDB for legacy > >> reasons. > >> > >> I have a pretty good idea in my head of what an all-lisp backend > >> requires and having one would lay to rest all of these discussions > >> of > >> bringing up "yet another backend". Edi Weitz and I discussed > >> collaborating on this, but unfortunately he had some other projects > >> that took priority. > >> > >> Is there a small critical mass of people out there that care enough > >> about this that they'd be willing to contribute to such a project? > >> I > >> don't have the time to do it on my own, but if we broke it up into > >> small projects over the next handful of months, I don't think it's a > >> ton of work. I can put in a solid chunk of integration work in mid > >> to > >> late April. > >> > > > > I completely agree with Ian about the value of a LISP-native backend. > > > > However, I can not personally offer to help with this. I have in fact > > abandoned my business plans for the time being and taken a normal job. > > Moreover, since I did the "schema evolution" system that we used in the > > Java application for Hire.com a while back, I feel more comfortable > > working on that than on the LISP native backend, although I think both > > are wonderful and challenging problems. > > > > The excellent set of automated tests produced by the original authors of > > Elephant (Andrew Blumberg and Ben Lee) and strengthened by Ian and > > myself and others since then remain the greatest asset in undertaking > > the LISP-Native backend. > > > > I know that many of you understand most of the technical challenges in > > bringing up a Native LISP backend better than I do. However, let me ask > > the question that my acquaintance Kent Beck always asks: > > > > What is the simplest thing that could possible work? > > > > By which I mean, is there any value in making a very simple LISP native > > backend? Forget locking, transactions, efficiency, and all that other > > ocean-boiling stuff that we all know will be needed for an enterprise > > application. Can anybody build a Native-Lisp backend in a weekend? > > > > If so, we would have an excellent "spike" solution that would inform > > further efforts, and furthermore we would have an out-of-the-box > > solution for demoing Elephant that would require ZERO additional > > software installations. This would be very useful, even if there are > > performance and reliability limits to that backend. > > > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From ranga at mmsindia.com Thu Feb 14 07:16:28 2008 From: ranga at mmsindia.com (Rangarajan Krishnamoorthy) Date: Thu, 14 Feb 2008 12:46:28 +0530 Subject: [elephant-devel] QDBM Support References: <000f01c86dfd$065a16e0$0201a8c0@COM><61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org><50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com><1202962084.6267.91.camel@penguin.yourdomain.com><000d01c86ec7$47122d60$0201a8c0@COM> <1202967099.6267.108.camel@penguin.yourdomain.com> Message-ID: <000501c86ed9$88a718f0$0201a8c0@COM> Thanks for the detailed suggestions. I haven't thought about Postgres, but your suggestion sounds good. Does Postgres require any admin-type task by end users of my application? I hope not, because my intended audience is medical Doctors! I am in contact with Oracle India regarding the license and they have confirmed that I have to pay royalties. Although the royalty appears quite reasonable, looks like I have to be an "Oracle Partner" to distribute BDB under this licensing clause. There is an annual fee to be an OPN, which equals about 100 license royalty payments per year! Before I bought LispWorks I was keen to use Allegro CL/Allegro Cache, but found that the royalties there are astronomical, so decided to switch loyalty to LW. Let us see how this goes. For a product that cannot cost more than $500, and might not sell in huge numbers, this seems like too much trouble! Thanks for your advice. Regards, Rangarajan ----- Original Message ----- From: "Robert L. Read" To: "Elephant bugs and development" Sent: Thursday, February 14, 2008 11:01 AM Subject: Re: [elephant-devel] QDBM Support > On Thu, 2008-02-14 at 10:35 +0530, Rangarajan Krishnamoorthy wrote: >> Robert, >> I am the one who started this thread, so let me clarify. >> >> The background: I am starting to work on a .NET-based application (for >> Windows platforms) that might benefit by Lisp integration. I purchased >> Lispworks Enterprise in this context. When I searched for a lisp >> persistence >> library, I was told to consider Elephant. I tried it and it seems to work >> fine for me. Elephant documentation advises that BDB is the best >> performing >> backend among the backends it supports. But it appears that I cannot just >> "buy" BDB by opting for a one-time initial payment. I have to pay runtime >> royalties. This is different from other libraries that I have used so >> far. >> Although there may be legitimate reasons why Oracle does it this way, I >> am >> more comfortable with one-time payment. So the problem I am facing is >> that I >> would love to use Elephant in my work, but BDB's licensing is the >> concern. I >> found out that QDBM is another popular DB implementation which does not >> have >> this licensing issue, so posted the question to this list. It is not just >> that QDBM outperforms BDB (I don't know this). Please note that I am not >> against paying for the library that I use, but just that I am not in >> favor >> of royalties. > > Thanks for the clarification --- I got tangled in the thread. > > However, this allows me to rephrase my question: Can you consider using > the postmodern backend (which is an interface to the PostGres database)? > > PostGres is free. The Postmodern backend is probably 3 to 10 times > slower than BDB, and might be even worse in some circumstances. > However, it is entirely possible that 10 times very fast is still fast > enough for whatever operation you intend. > > It is unfortunate that to test this you will have to install PostGres > and do a very, very small amount of db administration on PostGres in > order to execute Elephant against it. With experience this should only > take a few hours. > > I share your attitude toward royalties in this matter, which is mainly > the reason I wrote the CL-SQL-based backend. > > I know only what you have stated about your application, but one > possibility, if you have no better alternative to Elephant, is to use > the postmodern backend now, during your development, and be prepared to > either switch to BDB by paying for it later, or to hope for (and perhaps > help) an effort to either implement QDBM or a LISP-native backend. > > Allow me to ask if you have really carefully considered the BDB license: > > http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html > > ...and concluded that you would have to pay a runtime royalty for your > specific application? > > (As an aside: I have not studied the other persistence solutions > deeply. I suspect that using CL-SQL directly (not through Elephant) is > one of the most common and well-tested mechanisms. This is an > Object-Relational Mapper (ORM) similar to Hibernate or Ibatis (although > a bit less evolved, I think.) This approach would require you to do a > great deal more work than Elephant does, since you have to create tables > and consciously map your classes into those tables and be prepared to > deal with schema evolution when you change the design of your object > space. I think Elephant is much more convenient, but using CL-SQL > directly is an option.) > > > > > > >> >> Coming to your suggestion that it would be wise to build a very simple, >> native lisp backend without the "frills", I am for it! >> >> Regards, >> Rangarajan >> >> ----- Original Message ----- >> From: "Robert L. Read" >> To: "Elephant bugs and development" >> Sent: Thursday, February 14, 2008 9:38 AM >> Subject: Re: [elephant-devel] QDBM Support >> >> >> > On Wed, 2008-02-13 at 10:41 -0500, Ian Eslick wrote: >> >> The answer to all of this, I think, is having a native lisp version >> >> that has BDB's performance and no licensing restrictions. Then >> >> supporting the other two becomes: Postmodern for a higher degree of >> >> reliability as well as for distributed systems and BDB for legacy >> >> reasons. >> >> >> >> I have a pretty good idea in my head of what an all-lisp backend >> >> requires and having one would lay to rest all of these discussions >> >> of >> >> bringing up "yet another backend". Edi Weitz and I discussed >> >> collaborating on this, but unfortunately he had some other projects >> >> that took priority. >> >> >> >> Is there a small critical mass of people out there that care enough >> >> about this that they'd be willing to contribute to such a project? >> >> I >> >> don't have the time to do it on my own, but if we broke it up into >> >> small projects over the next handful of months, I don't think it's a >> >> ton of work. I can put in a solid chunk of integration work in mid >> >> to >> >> late April. >> >> >> > >> > I completely agree with Ian about the value of a LISP-native backend. >> > >> > However, I can not personally offer to help with this. I have in fact >> > abandoned my business plans for the time being and taken a normal job. >> > Moreover, since I did the "schema evolution" system that we used in the >> > Java application for Hire.com a while back, I feel more comfortable >> > working on that than on the LISP native backend, although I think both >> > are wonderful and challenging problems. >> > >> > The excellent set of automated tests produced by the original authors >> > of >> > Elephant (Andrew Blumberg and Ben Lee) and strengthened by Ian and >> > myself and others since then remain the greatest asset in undertaking >> > the LISP-Native backend. >> > >> > I know that many of you understand most of the technical challenges in >> > bringing up a Native LISP backend better than I do. However, let me >> > ask >> > the question that my acquaintance Kent Beck always asks: >> > >> > What is the simplest thing that could possible work? >> > >> > By which I mean, is there any value in making a very simple LISP native >> > backend? Forget locking, transactions, efficiency, and all that other >> > ocean-boiling stuff that we all know will be needed for an enterprise >> > application. Can anybody build a Native-Lisp backend in a weekend? >> > >> > If so, we would have an excellent "spike" solution that would inform >> > further efforts, and furthermore we would have an out-of-the-box >> > solution for demoing Elephant that would require ZERO additional >> > software installations. This would be very useful, even if there are >> > performance and reliability limits to that backend. >> > >> > >> > _______________________________________________ >> > elephant-devel site list >> > elephant-devel at common-lisp.net >> > http://common-lisp.net/mailman/listinfo/elephant-devel >> >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From Instant at common-lisp.net Fri Feb 15 16:02:45 2008 From: Instant at common-lisp.net (Instant at common-lisp.net) Date: 15 Feb 2008 08:02:45 -0800 Subject: [elephant-devel] Can you afford to lose 300, 000 potential customers per year ? Message-ID: <20080215080241.D8FD1411560208DF@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From read at robertlread.net Fri Feb 15 01:54:46 2008 From: read at robertlread.net (Robert L. Read) Date: Thu, 14 Feb 2008 19:54:46 -0600 Subject: [elephant-devel] QDBM Support In-Reply-To: <000501c86ed9$88a718f0$0201a8c0@COM> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <1202962084.6267.91.camel@penguin.yourdomain.com> <000d01c86ec7$47122d60$0201a8c0@COM> <1202967099.6267.108.camel@penguin.yourdomain.com> <000501c86ed9$88a718f0$0201a8c0@COM> Message-ID: <1203040486.6267.128.camel@penguin.yourdomain.com> In order to deploy with Postgres you will have to write a script to create the db with the correct name. That is all that you will have to do, in order to have a system that technically runs. There may be long-term administrative tasks, such as executing "vacuum" to clean up indexes on a periodic basis. You might be able to code this into your application. I don't think it would ever keep something from working, but it might slow things down after months of use. I can see now that a Native-LISP backend would really be a blessing for your intended usage. Postgres has some installation set up which is required; you will have to work out a way to script this if you are selling it as an embedded database. It is possible that SQLite3 might even be better for you; I am not terribly familiar with it. I know that is is much simpler than PostGres and faster, though less "industrial" in its feature set. On Thu, 2008-02-14 at 12:46 +0530, Rangarajan Krishnamoorthy wrote: > Thanks for the detailed suggestions. > > I haven't thought about Postgres, but your suggestion sounds good. Does > Postgres require any admin-type task by end users of my application? I hope > not, because my intended audience is medical Doctors! > > I am in contact with Oracle India regarding the license and they have > confirmed that I have to pay royalties. Although the royalty appears quite > reasonable, looks like I have to be an "Oracle Partner" to distribute BDB > under this licensing clause. There is an annual fee to be an OPN, which > equals about 100 license royalty payments per year! > > Before I bought LispWorks I was keen to use Allegro CL/Allegro Cache, but > found that the royalties there are astronomical, so decided to switch > loyalty to LW. Let us see how this goes. > > For a product that cannot cost more than $500, and might not sell in huge > numbers, this seems like too much trouble! > > Thanks for your advice. > Regards, > Rangarajan > ----- Original Message ----- > From: "Robert L. Read" > To: "Elephant bugs and development" > Sent: Thursday, February 14, 2008 11:01 AM > Subject: Re: [elephant-devel] QDBM Support > > > > On Thu, 2008-02-14 at 10:35 +0530, Rangarajan Krishnamoorthy wrote: > >> Robert, > >> I am the one who started this thread, so let me clarify. > >> > >> The background: I am starting to work on a .NET-based application (for > >> Windows platforms) that might benefit by Lisp integration. I purchased > >> Lispworks Enterprise in this context. When I searched for a lisp > >> persistence > >> library, I was told to consider Elephant. I tried it and it seems to work > >> fine for me. Elephant documentation advises that BDB is the best > >> performing > >> backend among the backends it supports. But it appears that I cannot just > >> "buy" BDB by opting for a one-time initial payment. I have to pay runtime > >> royalties. This is different from other libraries that I have used so > >> far. > >> Although there may be legitimate reasons why Oracle does it this way, I > >> am > >> more comfortable with one-time payment. So the problem I am facing is > >> that I > >> would love to use Elephant in my work, but BDB's licensing is the > >> concern. I > >> found out that QDBM is another popular DB implementation which does not > >> have > >> this licensing issue, so posted the question to this list. It is not just > >> that QDBM outperforms BDB (I don't know this). Please note that I am not > >> against paying for the library that I use, but just that I am not in > >> favor > >> of royalties. > > > > Thanks for the clarification --- I got tangled in the thread. > > > > However, this allows me to rephrase my question: Can you consider using > > the postmodern backend (which is an interface to the PostGres database)? > > > > PostGres is free. The Postmodern backend is probably 3 to 10 times > > slower than BDB, and might be even worse in some circumstances. > > However, it is entirely possible that 10 times very fast is still fast > > enough for whatever operation you intend. > > > > It is unfortunate that to test this you will have to install PostGres > > and do a very, very small amount of db administration on PostGres in > > order to execute Elephant against it. With experience this should only > > take a few hours. > > > > I share your attitude toward royalties in this matter, which is mainly > > the reason I wrote the CL-SQL-based backend. > > > > I know only what you have stated about your application, but one > > possibility, if you have no better alternative to Elephant, is to use > > the postmodern backend now, during your development, and be prepared to > > either switch to BDB by paying for it later, or to hope for (and perhaps > > help) an effort to either implement QDBM or a LISP-native backend. > > > > Allow me to ask if you have really carefully considered the BDB license: > > > > http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html > > > > ...and concluded that you would have to pay a runtime royalty for your > > specific application? > > > > (As an aside: I have not studied the other persistence solutions > > deeply. I suspect that using CL-SQL directly (not through Elephant) is > > one of the most common and well-tested mechanisms. This is an > > Object-Relational Mapper (ORM) similar to Hibernate or Ibatis (although > > a bit less evolved, I think.) This approach would require you to do a > > great deal more work than Elephant does, since you have to create tables > > and consciously map your classes into those tables and be prepared to > > deal with schema evolution when you change the design of your object > > space. I think Elephant is much more convenient, but using CL-SQL > > directly is an option.) > > > > > > > > > > > > > >> > >> Coming to your suggestion that it would be wise to build a very simple, > >> native lisp backend without the "frills", I am for it! > >> > >> Regards, > >> Rangarajan > >> > >> ----- Original Message ----- > >> From: "Robert L. Read" > >> To: "Elephant bugs and development" > >> Sent: Thursday, February 14, 2008 9:38 AM > >> Subject: Re: [elephant-devel] QDBM Support > >> > >> > >> > On Wed, 2008-02-13 at 10:41 -0500, Ian Eslick wrote: > >> >> The answer to all of this, I think, is having a native lisp version > >> >> that has BDB's performance and no licensing restrictions. Then > >> >> supporting the other two becomes: Postmodern for a higher degree of > >> >> reliability as well as for distributed systems and BDB for legacy > >> >> reasons. > >> >> > >> >> I have a pretty good idea in my head of what an all-lisp backend > >> >> requires and having one would lay to rest all of these discussions > >> >> of > >> >> bringing up "yet another backend". Edi Weitz and I discussed > >> >> collaborating on this, but unfortunately he had some other projects > >> >> that took priority. > >> >> > >> >> Is there a small critical mass of people out there that care enough > >> >> about this that they'd be willing to contribute to such a project? > >> >> I > >> >> don't have the time to do it on my own, but if we broke it up into > >> >> small projects over the next handful of months, I don't think it's a > >> >> ton of work. I can put in a solid chunk of integration work in mid > >> >> to > >> >> late April. > >> >> > >> > > >> > I completely agree with Ian about the value of a LISP-native backend. > >> > > >> > However, I can not personally offer to help with this. I have in fact > >> > abandoned my business plans for the time being and taken a normal job. > >> > Moreover, since I did the "schema evolution" system that we used in the > >> > Java application for Hire.com a while back, I feel more comfortable > >> > working on that than on the LISP native backend, although I think both > >> > are wonderful and challenging problems. > >> > > >> > The excellent set of automated tests produced by the original authors > >> > of > >> > Elephant (Andrew Blumberg and Ben Lee) and strengthened by Ian and > >> > myself and others since then remain the greatest asset in undertaking > >> > the LISP-Native backend. > >> > > >> > I know that many of you understand most of the technical challenges in > >> > bringing up a Native LISP backend better than I do. However, let me > >> > ask > >> > the question that my acquaintance Kent Beck always asks: > >> > > >> > What is the simplest thing that could possible work? > >> > > >> > By which I mean, is there any value in making a very simple LISP native > >> > backend? Forget locking, transactions, efficiency, and all that other > >> > ocean-boiling stuff that we all know will be needed for an enterprise > >> > application. Can anybody build a Native-Lisp backend in a weekend? > >> > > >> > If so, we would have an excellent "spike" solution that would inform > >> > further efforts, and furthermore we would have an out-of-the-box > >> > solution for demoing Elephant that would require ZERO additional > >> > software installations. This would be very useful, even if there are > >> > performance and reliability limits to that backend. > >> > > >> > > >> > _______________________________________________ > >> > elephant-devel site list > >> > elephant-devel at common-lisp.net > >> > http://common-lisp.net/mailman/listinfo/elephant-devel > >> > >> > >> _______________________________________________ > >> elephant-devel site list > >> elephant-devel at common-lisp.net > >> http://common-lisp.net/mailman/listinfo/elephant-devel > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From desoi at pgedit.com Fri Feb 15 03:34:04 2008 From: desoi at pgedit.com (John DeSoi) Date: Thu, 14 Feb 2008 22:34:04 -0500 Subject: [elephant-devel] QDBM Support In-Reply-To: <1203040486.6267.128.camel@penguin.yourdomain.com> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <1202962084.6267.91.camel@penguin.yourdomain.com> <000d01c86ec7$47122d60$0201a8c0@COM> <1202967099.6267.108.camel@penguin.yourdomain.com> <000501c86ed9$88a718f0$0201a8c0@COM> <1203040486.6267.128.camel@penguin.yourdomain.com> Message-ID: <2CB0E561-3FA1-4150-B47F-C49BC6CF035D@pgedit.com> On Feb 14, 2008, at 8:54 PM, Robert L. Read wrote: > In order to deploy with Postgres you will have to write a script to > create the db with the correct name. That is all that you will have > to > do, in order to have a system that technically runs. Even if PostgreSQL is already on the target system, it is a bit more complicated than that. You have to create the database cluster (initdb utility) before you can create any databases. It can be done, but PostgreSQL is designed from the ground up to be a server database, not an embedded database. > > There may be long-term administrative tasks, such as executing > "vacuum" > to clean up indexes on a periodic basis. You might be able to code > this > into your application. I don't think it would ever keep something > from > working, but it might slow things down after months of use. All relatively recent versions of PostgreSQL have auto vacuum enabled, so this is generally not something you'll have to worry about. > > It is possible that SQLite3 might even be better for you; I am not > terribly familiar with it. I know that is is much simpler than > PostGres > and faster, though less "industrial" in its feature set. Much simpler, but faster is debatable. SQLite seems like it would be the ideal database for this project since it is easy to embed in an application and has no license hassles. I don't really know anything about BDB, but I'm surprised the performance of a properly indexed SQL database can't get close to it. If the performance with SQLite is poor, has it been verified to be only a SQLite issue and not something related to CLSQL? John DeSoi, Ph.D. From read at robertlread.net Fri Feb 15 03:41:34 2008 From: read at robertlread.net (Robert L. Read) Date: Thu, 14 Feb 2008 21:41:34 -0600 Subject: [elephant-devel] QDBM Support In-Reply-To: <2CB0E561-3FA1-4150-B47F-C49BC6CF035D@pgedit.com> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org> <50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com> <1202962084.6267.91.camel@penguin.yourdomain.com> <000d01c86ec7$47122d60$0201a8c0@COM> <1202967099.6267.108.camel@penguin.yourdomain.com> <000501c86ed9$88a718f0$0201a8c0@COM> <1203040486.6267.128.camel@penguin.yourdomain.com> <2CB0E561-3FA1-4150-B47F-C49BC6CF035D@pgedit.com> Message-ID: <1203046894.6267.135.camel@penguin.yourdomain.com> On Thu, 2008-02-14 at 22:34 -0500, John DeSoi wrote: > SQLite seems like it would be the ideal database for this project > since it is easy to embed in an application and has no license > hassles. I don't really know anything about BDB, but I'm surprised > the > performance of a properly indexed SQL database can't get close to > it. > If the performance with SQLite is poor, has it been verified to be > only a SQLite issue and not something related to CLSQL? > > I think SQLite is faster than PostGres on our test suite. It is my understanding that this is perhaps possible because it provides much weaker concurrency control. I wouldn't say that the CL-SQL backend on top of Postgres or SQLite3 has really been studied for performance. In writing it, my primary goal was to make the tests work as the did for BDB. This of course is valuable for migrating from one backend to the other. Since that time, Ian and I have agreed that in fact the tests are little too strong, and that we should allow some uncertainty in certain results which are currently demanded by the tests but are really just an artifact of the way BDB works. However, the work to change the tests has not been done, and the basic CL-SQL system that tries so hard to imitate BDB (mostly in the ordering of indices) has not been changed either. From killerstorm at newmail.ru Fri Feb 15 08:00:18 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Fri, 15 Feb 2008 10:00:18 +0200 Subject: [elephant-devel] Re: QDBM Support References: <000f01c86dfd$065a16e0$0201a8c0@COM><61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org><50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com><1202962084.6267.91.camel@penguin.yourdomain.com><000d01c86ec7$47122d60$0201a8c0@COM><1202967099.6267.108.camel@penguin.yourdomain.com><000501c86ed9$88a718f0$0201a8c0@COM><1203040486.6267.128.camel@penguin.yourdomain.com> <2CB0E561-3FA1-4150-B47F-C49BC6CF035D@pgedit.com> Message-ID: JD> hassles. I don't really know anything about BDB, but I'm surprised the JD> performance of a properly indexed SQL database can't get close to it. BDB works directly with files, but SQL backend working with PostgreSQL or database like this needs to send queries over network -- that has significant overhead, something like millisecond or so. this practically means you cannot make more than few thousands queries per second, that is quite prohibitive. this problem does not exist when you use SQL database directly -- people write complex queries with it and retrieve exactly what do they want in one pass. but we have to do query on each slot access, that's much worse.. in db-postmodern backend we've implemented btree lisp-side cache for this -- it can reduce query count greatly, but of course it works only on some workloads. JD> If the performance with SQLite is poor, has it been verified to be JD> only a SQLite issue and not something related to CLSQL? OTOH SQLite does not have this problem. that's why it can be much faster than PostgreSQL. however, as far as i remember, elephant's code that works with CLSQL works in extremely non-optimal way in some cases -- to retrieve single value from index via cursor, it retrieves all contents of index, sorts it, and retrieves that single value. but if application does not need range queries, this might work just fine, since some time ago we've optimized get-instance-by-value to get value with direct query to btree. so i'd advice to do benchmarking before coming to conclusions. if db-clsql + sqlite will work fine for given size of data base and queries, why should one need something else? From killerstorm at newmail.ru Fri Feb 15 08:10:43 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Fri, 15 Feb 2008 10:10:43 +0200 Subject: [elephant-devel] Re: QDBM Support References: <000f01c86dfd$065a16e0$0201a8c0@COM><61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org><50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com><1202962084.6267.91.camel@penguin.yourdomain.com><000d01c86ec7$47122d60$0201a8c0@COM><1202967099.6267.108.camel@penguin.yourdomain.com><000501c86ed9$88a718f0$0201a8c0@COM><1203040486.6267.128.camel@penguin.yourdomain.com><2CB0E561-3FA1-4150-B47F-C49BC6CF035D@pgedit.com> <1203046894.6267.135.camel@penguin.yourdomain.com> Message-ID: RLR> I think SQLite is faster than PostGres on our test suite. It is my RLR> understanding that this is perhaps possible because it provides much RLR> weaker concurrency control. i don't think concurrency control is in effect here. on postgresql side all is very fast! PostgreSQL could be slower because it has networking overhead. or because it had bugs in cursor implementation. i've re-implemented cursors, and performance is much smoother now. but this patches didn't go into main repository yet -- somehow you've ignored patches that Henrik have sent on 08.01.2008. i think it doesn't have sence to apply them now, though -- Henrik already has updated patches that work better (that famous instances reinitialization stuff got fixed), so, i think, one day he'll post them to the mailing list, and you can apply them :) From henrik at evahjelte.com Fri Feb 15 09:29:08 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 15 Feb 2008 10:29:08 +0100 Subject: [elephant-devel] Re: QDBM Support In-Reply-To: References: <000f01c86dfd$065a16e0$0201a8c0@COM> <1202962084.6267.91.camel@penguin.yourdomain.com> <000d01c86ec7$47122d60$0201a8c0@COM> <1202967099.6267.108.camel@penguin.yourdomain.com> <000501c86ed9$88a718f0$0201a8c0@COM> <1203040486.6267.128.camel@penguin.yourdomain.com> <2CB0E561-3FA1-4150-B47F-C49BC6CF035D@pgedit.com> <1203046894.6267.135.camel@penguin.yourdomain.com> Message-ID: <50e8e4f60802150129s5944ec0fgde3596611aae961@mail.gmail.com> On Fri, Feb 15, 2008 at 9:10 AM, Alex Mizrahi wrote: > i think it doesn't have sence to apply them now, though -- Henrik already > has updated patches that work better (that famous instances reinitialization > stuff got fixed), so, i think, one day he'll post them to the mailing list, > and you can apply them :) Ok, these patches have finally passed the "works on my computer" certification process, so here they are. /Henrik -------------- next part -------------- A non-text attachment was scrubbed... Name: patch-collection.darcs Type: application/octet-stream Size: 57555 bytes Desc: not available URL: From read at robertlread.net Fri Feb 15 13:31:59 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 15 Feb 2008 07:31:59 -0600 Subject: [elephant-devel] Re: QDBM Support In-Reply-To: <50e8e4f60802150129s5944ec0fgde3596611aae961@mail.gmail.com> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <1202962084.6267.91.camel@penguin.yourdomain.com> <000d01c86ec7$47122d60$0201a8c0@COM> <1202967099.6267.108.camel@penguin.yourdomain.com> <000501c86ed9$88a718f0$0201a8c0@COM> <1203040486.6267.128.camel@penguin.yourdomain.com> <2CB0E561-3FA1-4150-B47F-C49BC6CF035D@pgedit.com> <1203046894.6267.135.camel@penguin.yourdomain.com> <50e8e4f60802150129s5944ec0fgde3596611aae961@mail.gmail.com> Message-ID: <1203082319.6267.147.camel@penguin.yourdomain.com> On Fri, 2008-02-15 at 10:29 +0100, Henrik Hjelte wrote: > On Fri, Feb 15, 2008 at 9:10 AM, Alex Mizrahi wrote: > > i think it doesn't have sence to apply them now, though -- Henrik already > > has updated patches that work better (that famous instances reinitialization > > stuff got fixed), so, i think, one day he'll post them to the mailing list, > > and you can apply them :) > > Ok, these patches have finally passed the "works on my computer" > certification process, so here they are. OK, thanks, forgive me if I missed a previous message about this. > > /Henrik > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Fri Feb 15 14:03:36 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 15 Feb 2008 08:03:36 -0600 Subject: [elephant-devel] Re: QDBM Support In-Reply-To: <50e8e4f60802150129s5944ec0fgde3596611aae961@mail.gmail.com> References: <000f01c86dfd$065a16e0$0201a8c0@COM> <1202962084.6267.91.camel@penguin.yourdomain.com> <000d01c86ec7$47122d60$0201a8c0@COM> <1202967099.6267.108.camel@penguin.yourdomain.com> <000501c86ed9$88a718f0$0201a8c0@COM> <1203040486.6267.128.camel@penguin.yourdomain.com> <2CB0E561-3FA1-4150-B47F-C49BC6CF035D@pgedit.com> <1203046894.6267.135.camel@penguin.yourdomain.com> <50e8e4f60802150129s5944ec0fgde3596611aae961@mail.gmail.com> Message-ID: <1203084216.6267.152.camel@penguin.yourdomain.com> OK, I think I have applied those patches correctly. Perhaps you could do a "darcs pull" or a "darcs diff" against the repo to make sure I did it correctly; I still find darcs a little confusing, and my private repo was not in sync with the offical one. However, I was green with your changes in my private repository. I have not code reviewed them, but it seems best to trust the tests and to get them into the repo in any case. Good Work! On Fri, 2008-02-15 at 10:29 +0100, Henrik Hjelte wrote: > On Fri, Feb 15, 2008 at 9:10 AM, Alex Mizrahi wrote: > > i think it doesn't have sence to apply them now, though -- Henrik already > > has updated patches that work better (that famous instances reinitialization > > stuff got fixed), so, i think, one day he'll post them to the mailing list, > > and you can apply them :) > > Ok, these patches have finally passed the "works on my computer" > certification process, so here they are. > > /Henrik > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Fri Feb 15 14:17:46 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 15 Feb 2008 09:17:46 -0500 Subject: [elephant-devel] QDBM Support In-Reply-To: <000501c86ed9$88a718f0$0201a8c0@COM> References: <000f01c86dfd$065a16e0$0201a8c0@COM><61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org><50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com><1202962084.6267.91.camel@penguin.yourdomain.com><000d01c86ec7$47122d60$0201a8c0@COM> <1202967099.6267.108.camel@penguin.yourdomain.com> <000501c86ed9$88a718f0$0201a8c0@COM> Message-ID: For now, SQLite3 is the best way to avoid royalties and have an easily deployable solution. As Alex said, given the way we are using the SQL engine to store data, the performance isn't so great compared to BDB because we're simulating BTrees on top of SQL. If you want a direct map to a SQL DB, you're better off using CL-SQL directly (the query system is probably richer, but I find it more of a pain to manage, particularly the schema evolution. If QDBM supported Ming on Windows, I'd be happy to port it, but unless the solution works on all platforms I'm less motivated. Ian On Feb 14, 2008, at 2:16 AM, Rangarajan Krishnamoorthy wrote: > Thanks for the detailed suggestions. > > I haven't thought about Postgres, but your suggestion sounds good. > Does Postgres require any admin-type task by end users of my > application? I hope not, because my intended audience is medical > Doctors! > > I am in contact with Oracle India regarding the license and they > have confirmed that I have to pay royalties. Although the royalty > appears quite reasonable, looks like I have to be an "Oracle > Partner" to distribute BDB under this licensing clause. There is an > annual fee to be an OPN, which equals about 100 license royalty > payments per year! > > Before I bought LispWorks I was keen to use Allegro CL/Allegro > Cache, but found that the royalties there are astronomical, so > decided to switch loyalty to LW. Let us see how this goes. > > For a product that cannot cost more than $500, and might not sell in > huge numbers, this seems like too much trouble! > > Thanks for your advice. > Regards, > Rangarajan > ----- Original Message ----- From: "Robert L. Read" > > To: "Elephant bugs and development" > Sent: Thursday, February 14, 2008 11:01 AM > Subject: Re: [elephant-devel] QDBM Support > > >> On Thu, 2008-02-14 at 10:35 +0530, Rangarajan Krishnamoorthy wrote: >>> Robert, >>> I am the one who started this thread, so let me clarify. >>> >>> The background: I am starting to work on a .NET-based application >>> (for >>> Windows platforms) that might benefit by Lisp integration. I >>> purchased >>> Lispworks Enterprise in this context. When I searched for a lisp >>> persistence >>> library, I was told to consider Elephant. I tried it and it seems >>> to work >>> fine for me. Elephant documentation advises that BDB is the best >>> performing >>> backend among the backends it supports. But it appears that I >>> cannot just >>> "buy" BDB by opting for a one-time initial payment. I have to pay >>> runtime >>> royalties. This is different from other libraries that I have used >>> so far. >>> Although there may be legitimate reasons why Oracle does it this >>> way, I am >>> more comfortable with one-time payment. So the problem I am facing >>> is that I >>> would love to use Elephant in my work, but BDB's licensing is the >>> concern. I >>> found out that QDBM is another popular DB implementation which >>> does not have >>> this licensing issue, so posted the question to this list. It is >>> not just >>> that QDBM outperforms BDB (I don't know this). Please note that I >>> am not >>> against paying for the library that I use, but just that I am not >>> in favor >>> of royalties. >> >> Thanks for the clarification --- I got tangled in the thread. >> >> However, this allows me to rephrase my question: Can you consider >> using >> the postmodern backend (which is an interface to the PostGres >> database)? >> >> PostGres is free. The Postmodern backend is probably 3 to 10 times >> slower than BDB, and might be even worse in some circumstances. >> However, it is entirely possible that 10 times very fast is still >> fast >> enough for whatever operation you intend. >> >> It is unfortunate that to test this you will have to install PostGres >> and do a very, very small amount of db administration on PostGres in >> order to execute Elephant against it. With experience this should >> only >> take a few hours. >> >> I share your attitude toward royalties in this matter, which is >> mainly >> the reason I wrote the CL-SQL-based backend. >> >> I know only what you have stated about your application, but one >> possibility, if you have no better alternative to Elephant, is to use >> the postmodern backend now, during your development, and be >> prepared to >> either switch to BDB by paying for it later, or to hope for (and >> perhaps >> help) an effort to either implement QDBM or a LISP-native backend. >> >> Allow me to ask if you have really carefully considered the BDB >> license: >> >> http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html >> >> ...and concluded that you would have to pay a runtime royalty for >> your >> specific application? >> >> (As an aside: I have not studied the other persistence solutions >> deeply. I suspect that using CL-SQL directly (not through >> Elephant) is >> one of the most common and well-tested mechanisms. This is an >> Object-Relational Mapper (ORM) similar to Hibernate or Ibatis >> (although >> a bit less evolved, I think.) This approach would require you to >> do a >> great deal more work than Elephant does, since you have to create >> tables >> and consciously map your classes into those tables and be prepared to >> deal with schema evolution when you change the design of your object >> space. I think Elephant is much more convenient, but using CL-SQL >> directly is an option.) >> >> >> >> >> >> >>> >>> Coming to your suggestion that it would be wise to build a very >>> simple, >>> native lisp backend without the "frills", I am for it! >>> >>> Regards, >>> Rangarajan >>> >>> ----- Original Message ----- From: "Robert L. Read" >> > >>> To: "Elephant bugs and development" >>> Sent: Thursday, February 14, 2008 9:38 AM >>> Subject: Re: [elephant-devel] QDBM Support >>> >>> >>> > On Wed, 2008-02-13 at 10:41 -0500, Ian Eslick wrote: >>> >> The answer to all of this, I think, is having a native lisp >>> version >>> >> that has BDB's performance and no licensing restrictions. Then >>> >> supporting the other two becomes: Postmodern for a higher >>> degree of >>> >> reliability as well as for distributed systems and BDB for legacy >>> >> reasons. >>> >> >>> >> I have a pretty good idea in my head of what an all-lisp backend >>> >> requires and having one would lay to rest all of these >>> discussions >>> >> of >>> >> bringing up "yet another backend". Edi Weitz and I discussed >>> >> collaborating on this, but unfortunately he had some other >>> projects >>> >> that took priority. >>> >> >>> >> Is there a small critical mass of people out there that care >>> enough >>> >> about this that they'd be willing to contribute to such a >>> project? >>> >> I >>> >> don't have the time to do it on my own, but if we broke it up >>> into >>> >> small projects over the next handful of months, I don't think >>> it's a >>> >> ton of work. I can put in a solid chunk of integration work in >>> mid >>> >> to >>> >> late April. >>> >> >>> > >>> > I completely agree with Ian about the value of a LISP-native >>> backend. >>> > >>> > However, I can not personally offer to help with this. I have >>> in fact >>> > abandoned my business plans for the time being and taken a >>> normal job. >>> > Moreover, since I did the "schema evolution" system that we used >>> in the >>> > Java application for Hire.com a while back, I feel more >>> comfortable >>> > working on that than on the LISP native backend, although I >>> think both >>> > are wonderful and challenging problems. >>> > >>> > The excellent set of automated tests produced by the original >>> authors > of >>> > Elephant (Andrew Blumberg and Ben Lee) and strengthened by Ian and >>> > myself and others since then remain the greatest asset in >>> undertaking >>> > the LISP-Native backend. >>> > >>> > I know that many of you understand most of the technical >>> challenges in >>> > bringing up a Native LISP backend better than I do. However, >>> let me > ask >>> > the question that my acquaintance Kent Beck always asks: >>> > >>> > What is the simplest thing that could possible work? >>> > >>> > By which I mean, is there any value in making a very simple LISP >>> native >>> > backend? Forget locking, transactions, efficiency, and all that >>> other >>> > ocean-boiling stuff that we all know will be needed for an >>> enterprise >>> > application. Can anybody build a Native-Lisp backend in a >>> weekend? >>> > >>> > If so, we would have an excellent "spike" solution that would >>> inform >>> > further efforts, and furthermore we would have an out-of-the-box >>> > solution for demoing Elephant that would require ZERO additional >>> > software installations. This would be very useful, even if >>> there are >>> > performance and reliability limits to that backend. >>> > >>> > >>> > _______________________________________________ >>> > elephant-devel site list >>> > elephant-devel at common-lisp.net >>> > http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From ranga at mmsindia.com Fri Feb 15 15:01:07 2008 From: ranga at mmsindia.com (Rangarajan Krishnamoorthy) Date: Fri, 15 Feb 2008 20:31:07 +0530 Subject: [elephant-devel] QDBM Support References: <000f01c86dfd$065a16e0$0201a8c0@COM><61396.88.73.222.183.1202892090.squirrel@mail.stardawn.org><50e8e4f60802130711h4ed13735h3c65609a4848a2e@mail.gmail.com><1202962084.6267.91.camel@penguin.yourdomain.com><000d01c86ec7$47122d60$0201a8c0@COM><1202967099.6267.108.camel@penguin.yourdomain.com><000501c86ed9$88a718f0$0201a8c0@COM> Message-ID: <000701c86fe3$99bb2fe0$0501a8c0@kb5554caa7e384> Thanks for the advice. I agree with you that QDBM support is probably not justified at this point. Regards, Rangarajan ----- Original Message ----- From: "Ian Eslick" To: "Elephant bugs and development" Sent: Friday, February 15, 2008 7:47 PM Subject: Re: [elephant-devel] QDBM Support > For now, SQLite3 is the best way to avoid royalties and have an easily > deployable solution. As Alex said, given the way we are using the SQL > engine to store data, the performance isn't so great compared to BDB > because we're simulating BTrees on top of SQL. If you want a direct map > to a SQL DB, you're better off using CL-SQL directly (the query system is > probably richer, but I find it more of a pain to manage, particularly the > schema evolution. > > If QDBM supported Ming on Windows, I'd be happy to port it, but unless > the solution works on all platforms I'm less motivated. > > Ian > > > On Feb 14, 2008, at 2:16 AM, Rangarajan Krishnamoorthy wrote: > >> Thanks for the detailed suggestions. >> >> I haven't thought about Postgres, but your suggestion sounds good. Does >> Postgres require any admin-type task by end users of my application? I >> hope not, because my intended audience is medical Doctors! >> >> I am in contact with Oracle India regarding the license and they have >> confirmed that I have to pay royalties. Although the royalty appears >> quite reasonable, looks like I have to be an "Oracle Partner" to >> distribute BDB under this licensing clause. There is an annual fee to be >> an OPN, which equals about 100 license royalty payments per year! >> >> Before I bought LispWorks I was keen to use Allegro CL/Allegro Cache, >> but found that the royalties there are astronomical, so decided to >> switch loyalty to LW. Let us see how this goes. >> >> For a product that cannot cost more than $500, and might not sell in >> huge numbers, this seems like too much trouble! >> >> Thanks for your advice. >> Regards, >> Rangarajan >> ----- Original Message ----- From: "Robert L. Read" > > >> To: "Elephant bugs and development" >> Sent: Thursday, February 14, 2008 11:01 AM >> Subject: Re: [elephant-devel] QDBM Support >> >> >>> On Thu, 2008-02-14 at 10:35 +0530, Rangarajan Krishnamoorthy wrote: >>>> Robert, >>>> I am the one who started this thread, so let me clarify. >>>> >>>> The background: I am starting to work on a .NET-based application (for >>>> Windows platforms) that might benefit by Lisp integration. I purchased >>>> Lispworks Enterprise in this context. When I searched for a lisp >>>> persistence >>>> library, I was told to consider Elephant. I tried it and it seems to >>>> work >>>> fine for me. Elephant documentation advises that BDB is the best >>>> performing >>>> backend among the backends it supports. But it appears that I cannot >>>> just >>>> "buy" BDB by opting for a one-time initial payment. I have to pay >>>> runtime >>>> royalties. This is different from other libraries that I have used so >>>> far. >>>> Although there may be legitimate reasons why Oracle does it this way, >>>> I am >>>> more comfortable with one-time payment. So the problem I am facing is >>>> that I >>>> would love to use Elephant in my work, but BDB's licensing is the >>>> concern. I >>>> found out that QDBM is another popular DB implementation which does >>>> not have >>>> this licensing issue, so posted the question to this list. It is not >>>> just >>>> that QDBM outperforms BDB (I don't know this). Please note that I am >>>> not >>>> against paying for the library that I use, but just that I am not in >>>> favor >>>> of royalties. >>> >>> Thanks for the clarification --- I got tangled in the thread. >>> >>> However, this allows me to rephrase my question: Can you consider >>> using >>> the postmodern backend (which is an interface to the PostGres >>> database)? >>> >>> PostGres is free. The Postmodern backend is probably 3 to 10 times >>> slower than BDB, and might be even worse in some circumstances. >>> However, it is entirely possible that 10 times very fast is still fast >>> enough for whatever operation you intend. >>> >>> It is unfortunate that to test this you will have to install PostGres >>> and do a very, very small amount of db administration on PostGres in >>> order to execute Elephant against it. With experience this should only >>> take a few hours. >>> >>> I share your attitude toward royalties in this matter, which is mainly >>> the reason I wrote the CL-SQL-based backend. >>> >>> I know only what you have stated about your application, but one >>> possibility, if you have no better alternative to Elephant, is to use >>> the postmodern backend now, during your development, and be prepared to >>> either switch to BDB by paying for it later, or to hope for (and >>> perhaps >>> help) an effort to either implement QDBM or a LISP-native backend. >>> >>> Allow me to ask if you have really carefully considered the BDB >>> license: >>> >>> http://www.oracle.com/technology/software/products/berkeley-db/htdocs/oslicense.html >>> >>> ...and concluded that you would have to pay a runtime royalty for your >>> specific application? >>> >>> (As an aside: I have not studied the other persistence solutions >>> deeply. I suspect that using CL-SQL directly (not through Elephant) is >>> one of the most common and well-tested mechanisms. This is an >>> Object-Relational Mapper (ORM) similar to Hibernate or Ibatis (although >>> a bit less evolved, I think.) This approach would require you to do a >>> great deal more work than Elephant does, since you have to create >>> tables >>> and consciously map your classes into those tables and be prepared to >>> deal with schema evolution when you change the design of your object >>> space. I think Elephant is much more convenient, but using CL-SQL >>> directly is an option.) >>> >>> >>> >>> >>> >>> >>>> >>>> Coming to your suggestion that it would be wise to build a very >>>> simple, >>>> native lisp backend without the "frills", I am for it! >>>> >>>> Regards, >>>> Rangarajan >>>> >>>> ----- Original Message ----- From: "Robert L. Read" >>>> >>> > >>>> To: "Elephant bugs and development" >>>> Sent: Thursday, February 14, 2008 9:38 AM >>>> Subject: Re: [elephant-devel] QDBM Support >>>> >>>> >>>> > On Wed, 2008-02-13 at 10:41 -0500, Ian Eslick wrote: >>>> >> The answer to all of this, I think, is having a native lisp >>>> version >>>> >> that has BDB's performance and no licensing restrictions. Then >>>> >> supporting the other two becomes: Postmodern for a higher >>>> degree of >>>> >> reliability as well as for distributed systems and BDB for legacy >>>> >> reasons. >>>> >> >>>> >> I have a pretty good idea in my head of what an all-lisp backend >>>> >> requires and having one would lay to rest all of these >>>> discussions >>>> >> of >>>> >> bringing up "yet another backend". Edi Weitz and I discussed >>>> >> collaborating on this, but unfortunately he had some other >>>> projects >>>> >> that took priority. >>>> >> >>>> >> Is there a small critical mass of people out there that care >>>> enough >>>> >> about this that they'd be willing to contribute to such a >>>> project? >>>> >> I >>>> >> don't have the time to do it on my own, but if we broke it up >>>> into >>>> >> small projects over the next handful of months, I don't think >>>> it's a >>>> >> ton of work. I can put in a solid chunk of integration work in >>>> mid >>>> >> to >>>> >> late April. >>>> >> >>>> > >>>> > I completely agree with Ian about the value of a LISP-native >>>> backend. >>>> > >>>> > However, I can not personally offer to help with this. I have >>>> in fact >>>> > abandoned my business plans for the time being and taken a >>>> normal job. >>>> > Moreover, since I did the "schema evolution" system that we used >>>> in the >>>> > Java application for Hire.com a while back, I feel more >>>> comfortable >>>> > working on that than on the LISP native backend, although I >>>> think both >>>> > are wonderful and challenging problems. >>>> > >>>> > The excellent set of automated tests produced by the original >>>> authors > of >>>> > Elephant (Andrew Blumberg and Ben Lee) and strengthened by Ian and >>>> > myself and others since then remain the greatest asset in >>>> undertaking >>>> > the LISP-Native backend. >>>> > >>>> > I know that many of you understand most of the technical >>>> challenges in >>>> > bringing up a Native LISP backend better than I do. However, >>>> let me > ask >>>> > the question that my acquaintance Kent Beck always asks: >>>> > >>>> > What is the simplest thing that could possible work? >>>> > >>>> > By which I mean, is there any value in making a very simple LISP >>>> native >>>> > backend? Forget locking, transactions, efficiency, and all that >>>> other >>>> > ocean-boiling stuff that we all know will be needed for an >>>> enterprise >>>> > application. Can anybody build a Native-Lisp backend in a >>>> weekend? >>>> > >>>> > If so, we would have an excellent "spike" solution that would >>>> inform >>>> > further efforts, and furthermore we would have an out-of-the-box >>>> > solution for demoing Elephant that would require ZERO additional >>>> > software installations. This would be very useful, even if >>>> there are >>>> > performance and reliability limits to that backend. >>>> > >>>> > >>>> > _______________________________________________ >>>> > elephant-devel site list >>>> > elephant-devel at common-lisp.net >>>> > http://common-lisp.net/mailman/listinfo/elephant-devel >>>> >>>> >>>> _______________________________________________ >>>> elephant-devel site list >>>> elephant-devel at common-lisp.net >>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Sat Feb 16 09:22:26 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 16 Feb 2008 11:22:26 +0200 Subject: [elephant-devel] Re: notify_btree_update type error References: Message-ID: EN> I apologize if I've overlooked something obvious, but is the current EN> darcs Elephant with the current darcs Postmodern backend working with EN> Postgres 8.3? EN> I've been getting the error "Database error 42883: function EN> notify_btree_update(integer, bigint) does not exist" when I try to EN> store anything, and am trying to ascertain whether this is related to EN> the Postgres version or something else. i've got PostgreSQL 8.3 for testing, indeed there is a problem -- they have disabled implicit cast from integer to text. here is trivial patch to fix this issue: { hunk ./src/db-postmodern/pm-btree.lisp 122 -(format nil "PERFORM notify_btree_update(~a, the_key);" (oid bt)) +(format nil "PERFORM notify_btree_update(~a, the_key::text);" (oid bt)) hunk ./src/db-postmodern/pm-btree.lisp 140 - (register-query bt 'notify-update (format nil "select notify_btree_update(~a, $1)" (oid bt)))) + (register-query bt 'notify-update (format nil "select notify_btree_update(~a, $1::text)" (oid bt)))) } i've also attached patch in darcs format that is easy to apply. i suspect type casting might have effect on when global-sync-cache is turned on (it's not by default), i need to investigate it further.. thanks for reporting a problem! begin 666 t.patch M"DYE=R!P871C:&5S. at H*6V1B+7!O"!F;W(@ M<&G)A:&E 9VUA:6PN8V]M*BHR,# X,#(Q M-C Y,#8S.5T@>PIH=6YK("XO5]B=')E95]U M<&1A=&4H?F$L('1H95]K97DI.R()*&]I9"!B="DI"BLH9F]R;6%T(&YI;" B M4$521D]232!N;W1I9GE?8G1R965?=7!D871E*'YA+"!T:&5?:V5Y.CIT97AT M*3LB"2AO:60 at 8G0I*0IH=6YK("XO2UU<&1A=&4@*&9O'0Z M"@I;;6]V960 at 8V%C:&4M:6YS=&%N8V4@:6YT;R!I;FET:6%L+7!E"YM:7IR86AI0&=M86EL+F-O;2HJ,C P.# Q,C Q-#(T M,S8*(&)E8V%U"YM:7IR86AI0&=M86EL+F-O;2HJ,C P.# Q,38R,C(T,#5=( I; M9&(M<&]S=&UO9&5R;CH@<&TM8G1R964@:6YI=&EA;&EZ871I;VX at 9FEX960* M86QE>"YM:7IR86AI0&=M86EL+F-O;2HJ,C P.# Q,38R,C(S,39=( I;"YM:7IR86AI0&=M M86EL+F-O;2HJ,C P.# Q,38R,C Q,SA=( I;1FEX(&EN7!A2UF;W)M"G-R;W-S0&-O;6UO;BUL:7-P+FYE M="HJ,C P.# Q,3,Q-S,U-#==( I;9&]C=6UE;G1A=&EO;B!T>7!E(&9I> IR M96%D0')O8F5R=&QR96%D+FYE="HJ,C P.# Q,3$Q-3$Q,C1=( I;1FEX('1H M92!UGIA(#QV87)U>GIA0&=M86EL+F-O;3XJ*C(P M,#2!C;VUP87)I"YM M:7IR86AI0&=M86EL+F-O;2HJ,C P.# Q,#6YC:"!I"!I;G-T86YC92!D97-EF%T:6]N('!R;W1O8V]L"G-R;W-S0&-O;6UO;BUL M:7-P+FYE="HJ,C P-S$R,30Q-#$Y,SA=( I;9&(M<&]S=&UO9&5R;B!F:7@@ M;6%P+6EN9&5X(&]P=&EM:7IA=&EO;B!B=6<*2&5NG)A:&E M9VUA:6PN8V]M*BHR,# W,3(Q-3$Y,3 at P-5T@"EMD8BUP;W-T;6]D97)N.B!O M<'1I;6EZ960 at 9F]R;2US;&]T+6ME>2!F;W(@<&5RG)A:&E 9VUA:6PN8V]M*BHR,# W,3(P-S(P,#@S-0H@ M:70@=7-E&%M<&QE('5P9&%T90IA;&5X+FUI>G)A:&E M9VUA:6PN8V]M*BHR,# W,3(P-S(P,#8S,%T@"EMD8BUP;W-T;6]D97)N.B!O M<'1I;6EZ960@;6%P+6EN9&5X(&9OG)A:&E 9VUA:6PN8V]M*BHR,# W,3(P-S$Y-30P,ET@"EM&:7@@=&\@9G)O M;2UE;F0@=')A=F5R"!I M;7!L96UE;G1A=&EO;@IE2UV86QU90IE2!A(&-O;6EN9R!F96%T=7)E(&9R;VT at 26%N+"!B=70* M(')I9VAT(&YO=R!I="!B"!Q=6EC:R!F:7@*2&5N"X*=&IG M0'!E;G1A"!S;VUE('1E M7!O"F5S;&EC M:T!C;VUM;VXM;&ES<"YN970J*C(P,#"!S:6=N M86QI;F<@=&5S="!T;R!B:6YD(&5R&5S('1O('-Q;"!C=7)S;W)S(&%N9"!R M96UO=FEN9R!A('1E2!A8V-I9&5N="!U;F1E&X@:&%N9&QI;F<@:6UP;&5M96YT871I;VX*97-L:6-K M0&-O;6UO;BUL:7-P+FYE="HJ,C P-S$P,C0P,C4R,#5=( I;*",Q."D at 4')E M;&EM:6YA2!W;W)K(&]N("@C-#@I"F5S;&EC:T!C;VUM;VXM;&ES<"YN970J M*C(P,#"!M;W)E('1E IE&EN M9SL@=&5S=', at 87)E(&=R965N"F5S;&EC:T!C;VUM;VXM;&ES<"YN970J*C(P M,#7,*97-L:6-K0&-O;6UO;BUL:7-P+FYE M="HJ,C P-S$P,C(Q.30Q-#E=( I;1FEX(&-H87)A8W1E&5D(&UO<"!T97-T(&1E<&5N9&5N8VEE"!T;R!M:7)R;W(@ M,"XY<#$*97-L:6-K0&-O;6UO;BUL:7-P+FYE="HJ,C P-S$P,C(Q,S4X-#@* M( H at 1F]R9V]T('1O('!U&5D M(&EN(&UE"!&:79E04T@=&5S="!D97!E M;F1E;F-I97,L('-O;64 at 06QL96=R;R!I2!T97-T:6YG"FEE"!M87 M;&5G86-Y+6YA;65S(&)U9R!F;W(@;G5L;"!C87-E"FEE M&5S(&9O2!O2!C;VYV M96YI96YC92X*2!I;F5L96=A;G0@=&5S="P*(&)U="!I M="!I&5S(&EN('!M+6-A8VAE"F%L97 at N;6EZ6YC+6-A8VAE"F%L M97 at N;6EZ&5D"F%L97 at N;6EZ M"!T>7!E(&1E M8VQA"YM:7IR M86AI0&=M86EL+F-O;2HJ,C P-S Y,#0Q,C,W,C%=( I;:6YT97)N('1O('!R M;W!E2!D969A=6QT"DAE;G)I:R!(:F5L=&4\:&5NF%T:6]N(&-O9&4*2&5N"!T:&%T('-H;W5L9"!B92!S;VQV960@ M870@ I( M96YR:6L at 2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S W,C4R M,C Q,35=( I;='EP;R!F:7@*5&EE'1U M"!F;W(@<')O8FQE;2!W:71H(&UA<"UI;F1E> I(96YR:6L at 2&IE;'1E M/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S W,C(P-C$T,#$*(&5L97!H M86YT(&%P<&%R96YT;'D@2X@ M5&AI2X*72 *6W-O;64@;6]R92!B87-I8R!I;F1E>&EN9R!T97-T&EN9RUB87-I8R!T97-T"!T97-T&5D M+6)TF%T:6]N17)R;W)S"G)R96%D0&-O;6UO;BUL:7-P+FYE="HJ M,C P-S W,3 R,#$X,C8*(%1H:7,@:7, at 86X@871T96UP="!T;R!S=7)V:79E M('-E"!I&EN9RX*72 *6V1B+7!O2!T97-T('!S970*72 *6VUA:V4 at 36%K M969I;&4 at 9&]C=6UE;G1A=&EO;B!B=6EL9"!O;B!S8F-L"DAE;G)I:R!(:F5L M=&4@/&AH:F5L=&5 8V]M;6]N+6QIF5R*0II97-L:6-K0&-O;6UO;BUL:7-P+FYE="HJ,C P-S U,#4P-#,U M-#5=( I;1FEX(&)TF5D(&1A=&%B87-E(&5R'!O2!K M97D@;6ES2!C=7)S;W(@;W!S"FEE2!R97!O An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From read at robertlread.net Sun Feb 17 22:46:19 2008 From: read at robertlread.net (Robert L. Read) Date: Sun, 17 Feb 2008 16:46:19 -0600 Subject: [elephant-devel] Re: notify_btree_update type error In-Reply-To: References: Message-ID: <1203288380.21443.123.camel@penguin.yourdomain.com> I was green with this patch, although I am using PostGres 8.1, I think. I have committed it to the official repository. On Sat, 2008-02-16 at 11:22 +0200, Alex Mizrahi wrote: > EN> I apologize if I've overlooked something obvious, but is the current > EN> darcs Elephant with the current darcs Postmodern backend working with > EN> Postgres 8.3? > EN> I've been getting the error "Database error 42883: function > EN> notify_btree_update(integer, bigint) does not exist" when I try to > EN> store anything, and am trying to ascertain whether this is related to > EN> the Postgres version or something else. > > i've got PostgreSQL 8.3 for testing, indeed there is a problem -- they have > disabled implicit cast from integer to text. > here is trivial patch to fix this issue: > > { > hunk ./src/db-postmodern/pm-btree.lisp 122 > -(format nil "PERFORM notify_btree_update(~a, the_key);" (oid bt)) > +(format nil "PERFORM notify_btree_update(~a, the_key::text);" (oid bt)) > hunk ./src/db-postmodern/pm-btree.lisp 140 > - (register-query bt 'notify-update (format nil "select > notify_btree_update(~a, $1)" (oid bt)))) > + (register-query bt 'notify-update (format nil "select > notify_btree_update(~a, $1::text)" (oid bt)))) > } > > i've also attached patch in darcs format that is easy to apply. > > i suspect type casting might have effect on when global-sync-cache is turned > on (it's not by default), i need to investigate it further.. > > thanks for reporting a problem! > > > begin 666 t.patch > M"DYE=R!P871C:&5S. at H*6V1B+7!O"!F;W(@ > M<&G)A:&E 9VUA:6PN8V]M*BHR,# X,#(Q > M-C Y,#8S.5T@>PIH=6YK("XO M;&ES<" Q,C(*+2AF;W)M870@;FEL(")015)&3U)-(&YO=&EF>5]B=')E95]U > M<&1A=&4H?F$L('1H95]K97DI.R()*&]I9"!B="DI"BLH9F]R;6%T(&YI;" B > M4$521D]232!N;W1I9GE?8G1R965?=7!D871E*'YA+"!T:&5?:V5Y.CIT97AT > M*3LB"2AO:60 at 8G0I*0IH=6YK("XO M964N;&ES<" Q-# *+2 @*')E9VES=&5R+7%U97)Y(&)T("=N;W1I9GDM=7!D > M871E("AF;W)M870@;FEL(")S96QE8W0@;F]T:69Y7V)T M82P@)#$I(B H;VED(&)T*2DI*0HK(" H M=&EF>2UU<&1A=&4@*&9O M=7!D871E*'YA+" D,3HZ=&5X="DB("AO:60 at 8G0I*2DI"GT*"D-O;G1E>'0Z > M"@I;;6]V960 at 8V%C:&4M:6YS=&%N8V4@:6YT;R!I;FET:6%L+7!E M;G0M"YM:7IR86AI0&=M86EL+F-O;2HJ,C P.# Q,C Q-#(T > M,S8*(&)E8V%U M;F-E(&]T:&5R=VES90I=( I;86-C97-S;W(@;F%M92!I;B!T97-T M9V4*86QE>"YM:7IR86AI0&=M86EL+F-O;2HJ,C P.# Q,38R,C(T,#5=( I; > M9&(M<&]S=&UO9&5R;CH@<&TM8G1R964@:6YI=&EA;&EZ871I;VX at 9FEX960* > M86QE>"YM:7IR86AI0&=M86EL+F-O;2HJ,C P.# Q,38R,C(S,39=( I; M"YM:7IR86AI0&=M > M86EL+F-O;2HJ,C P.# Q,38R,C Q,SA=( I;1FEX(&EN M:6%L:7IA=&EO;B!T;R!B>7!A M M:V5Y=V]R9"UA8V-E M,#$Q,S$W,S8Q- at H@86QL;W=S(&QI M"EMF=6YC=&EO;BUC86QL+6ME>2UF;W)M"G-R;W-S0&-O;6UO;BUL:7-P+FYE > M="HJ,C P.# Q,3,Q-S,U-#==( I;9&]C=6UE;G1A=&EO;B!T>7!E(&9I> IR > M96%D0')O8F5R=&QR96%D+FYE="HJ,C P.# Q,3$Q-3$Q,C1=( I;1FEX('1H > M92!U M=&ELGIA(#QV87)U>GIA0&=M86EL+F-O;3XJ*C(P > M,# M M87)E;B=T(&5X<&]R=&5D(&EN('1H92!L871E M+ at H@"B!&:7@@:70 at 861D:6YG('1H92 Z.B!W:&5N(&%P<')O<')I871E"B * > M72 *6V1B+6)D8B!B=6=F:7 at Z('=H96X at 8F1B(&ME>2!C;VUP87)I M;7!A M87IE;FYI:V]V0&=M86EL+F-O;2HJ,C P-S$R,S Q-#$P-35=( I;9&(M<&]S > M=&UO9&5R;CH@"YM > M:7IR86AI0&=M86EL+F-O;2HJ,C P.# Q,# M92!C=7)S;W(@=F5R M= I=( I;8W5R M9&5R;@I(96YR:6L at 2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHM,C P > M-S$Q,C0Q-C,W,#%=( I;9&(M<&]S=&UO9&5R;B!R96UO=F5D('!O M='D@;V8@=7-I;F<@3DE,(&%S(&$@:V5Y(&EN(&)T M;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S$Q,C0Q-C,X,CA=( I; > M8W5R M96YR:6L at 2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S$Q,C0Q > M-C,W,#%=( I;16YS=7)E('-E="UD8BUS>6YC:"!I M92!P M,C$T,30U,#0Q72 *6T9I>"!I;G-T86YC92!D97-E M8GEP87-S(&EN:71I86QI>F%T:6]N('!R;W1O8V]L"G-R;W-S0&-O;6UO;BUL > M:7-P+FYE="HJ,C P-S$R,30Q-#$Y,SA=( I;9&(M<&]S=&UO9&5R;B!F:7@@ > M;6%P+6EN9&5X(&]P=&EM:7IA=&EO;B!B=6<*2&5N M M=&UO9&5R;CH at 8W5RG)A:&E > M9VUA:6PN8V]M*BHR,# W,3(Q-3$Y,3 at P-5T@"EMD8BUP;W-T;6]D97)N.B!O > M<'1I;6EZ960 at 9F]R;2US;&]T+6ME>2!F;W(@<&5R M861E<@IA;&5X+FUI>G)A:&E 9VUA:6PN8V]M*BHR,# W,3(P-S(P,#@S-0H@ > M:70@=7-E M(&EM<&QE;65N=&%T:6]N(&ET)W,@;&5S M;W-T;6]D97)N.B!S;6%L;"!E>&%M<&QE('5P9&%T90IA;&5X+FUI>G)A:&E > M9VUA:6PN8V]M*BHR,# W,3(P-S(P,#8S,%T@"EMD8BUP;W-T;6]D97)N.B!O > M<'1I;6EZ960@;6%P+6EN9&5X(&9O M>G)A:&E 9VUA:6PN8V]M*BHR,# W,3(P-S$Y-30P,ET@"EM&:7@@=&\@9G)O > M;2UE;F0@=')A=F5R M;BUL:7-P+FYE="HJ,C P-S$Q,S R,C,U,C1=( I;3F5W(&UA<"UI;F1E>"!I > M;7!L96UE;G1A=&EO;@IE M,#(R,C8R,%T@"EM#:&5A<&5R(&=E="UI;G-T86YC92UB>2UV86QU90IE M8VM 8V]M;6]N+6QI M(&$@;&ET=&QE(&-O;7!I;&5R('=A M=&4\:&5N M96UO=F4@:VEN9"UH:6YT M M,#0V"B!02!A(&-O;6EN9R!F96%T=7)E(&9R;VT at 26%N+"!B=70* > M(')I9VAT(&YO=R!I="!B M+6EN9&5X"B!A;F0@=&AU M9F]R(&YO=RX*72 *6U1!1R!%3$502$%.5"TP+3DM,0II97-L:6-K0&-O;6UO > M;BUL:7-P+FYE="HJ,C P-S$Q,38Q-3,V,S1=( I;1FEX97,@=&\@96YA8FQE > M('1H92!D;V-S('1O(&)U:6QD("AO;B!/4R!8("\@4T)#3"!U M92<@:6X at 96QE<&AA;G0O9&]C*0IE M,# W,3$P-#(P-#@P,ET@"EMA(&QI='1L92!C;VUM96YT('5P9&%T90I(96YR > M:6L at 2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S$Q,#8P.# R > M-3E=( I;<&]S=&UO9&5R;B!R96UO=F5D('5G;'DM9FEX(&9R;VT@<&TM8G1R > M964M:6YD97@@"DAE;G)I:R!(:F5L=&4\:&5N M*BHR,# W,3$P-C X,#(Q- at H@86YD(&UA9&4 at 8VAA M;V1E9" H M9F]R('-E M+F-O;3XJ*C(P,# M;'5D92!H:6YT M93QH96YR:6M 979A:&IE;'1E+F-O;3XJ*C(P,# M+7!O M;VYD87)Y+6-U M8V]M/BHJ,C P-S$Q,#$Q,# W,#!=( I;<&]S=&UO9&5R;B!R96UO=F4@;V)S > M;VQE=&4 at 8V]M;65N="!A8F]U="!W96%K('1A8FQE M/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S$P,S$P,C,S,3A=( I;<&]S > M=&UO9&5R;B!T97AI;F9O(&9I;&4*2&5N M:&IE;'1E+F-O;3XJ*C(P,# M=7!D871E('1H92!U9VQY(&UA<"UI;F1E>"!Q=6EC:R!F:7@*2&5N M96QT93QH96YR:6M 979A:&IE;'1E+F-O;3XJ*C(P,# M6V1B+7!O M M:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S$P,S Q.#$Q-31=( I;4&]S=&UO9&5R > M;B!B86-K96YD.B!C;VYN96-T:6]N('-P96,@;F]W(&%C8V5P=', at .G!O M:V5Y=V]R9"!A2!T:&4@<&]R="X at 4VEM:6QA > M"X*=&IG > M0'!E;G1A"!S;VUE('1E > M M+6QI7!O"F5S;&EC > M:T!C;VUM;VXM;&ES<"YN970J*C(P,#"!S:6=N > M86QI;F<@=&5S="!T;R!B:6YD(&5R M9VEV96X@;&ES< IE M,C4U,UT@"EM0871C:"!T;R!U M9&EC871E&5S('1O('-Q;"!C=7)S;W)S(&%N9"!R > M96UO=FEN9R!A('1E2!A8V-I9&5N="!U;F1E M1$(@9'5E('1O('1H92!I;F%B:6QI='D@=&\@8V]M<&%R92!S=&%N9&%R9"!O > M8FIE8W1S"F5S;&EC:T!C;VUM;VXM;&ES<"YN970J*C(P,# M72 *6T9I>"!B=6=S('1H870@ M('-U:71E.R!S;VUE('1E M:6X@=&AE(%-13"!T>&X@:&%N9&QI;F<@:6UP;&5M96YT871I;VX*97-L:6-K > M0&-O;6UO;BUL:7-P+FYE="HJ,C P-S$P,C0P,C4R,#5=( I;*",Q."D at 4')E > M;&EM:6YA M M:6UI;F%R>2!W;W)K(&]N("@C-#@I"F5S;&EC:T!C;VUM;VXM;&ES<"YN970J > M*C(P,# M87!P:6YG.R!A9&0@=&5S=',[(&9I>"!M;W)E('1E M.R!F:7@@8G5G(&EN(&UA<"UI;F1E> IE M*BHR,# W,3 R,S S,3 at S,5T@"ELH(S$Y*2!&:7AE9"!I;F-R96UE;G0 at 8W5R > M M-S$P,C,P,#0S-39=( I;1FEX(&)U9W,@:6X@ M('1E&EN > M9SL@=&5S=', at 87)E(&=R965N"F5S;&EC:T!C;VUM;VXM;&ES<"YN970J*C(P > M,# M:6YS=&%N8V4 at 961I=',@:6X at 8VAA;F=E+6-L87-S(&%N9"!R961E9FEN92UC > M;&%S M;VXM;&ES<"YN970J*C(P,#"!A(&1E9F%U;'1S > M(&)U9R!I;B!M86YU86P@=')A;G-A8W1I;VX@:&%N9&QI;F<*97-L:6-K0&-O > M;6UO;BUL:7-P+FYE="HJ,C P-S$P,C(R,S4X-35=( I;1FEX960 at 82!B=6<@ > M:6X at 8W5R M8G1R964 at 8W5R M;VXM;&ES<"YN970J*C(P,# M:&%R86-T97)S(&%S(&EN9&5X(&ME>7,*97-L:6-K0&-O;6UO;BUL:7-P+FYE > M="HJ,C P-S$P,C(Q.30Q-#E=( I;1FEX(&-H87)A8W1E M(&EN($)$0B!D871A('-T;W)E(&%N9"!L:7-P+6-O;7!A M"F5S;&EC:T!C;VUM;VXM;&ES<"YN970J*C(P,# M>&5D(&UO<"!T97-T(&1E<&5N9&5N8VEE M;VX*97-L:6-K0&-O;6UO;BUL:7-P+FYE="HJ,C P-S$P,C(Q-# T,SA=( I; > M1FEX(&QI"!T;R!M:7)R;W(@ > M,"XY<#$*97-L:6-K0&-O;6UO;BUL:7-P+FYE="HJ,C P-S$P,C(Q,S4X-#@* > M( H at 1F]R9V]T('1O('!U M(&9U;F-T:6]N&5D > M(&EN(&UE"!&:79E04T@=&5S="!D97!E > M;F1E;F-I97,L('-O;64 at 06QL96=R;R!I M M;&ES<"YN970J*C(P,# M9FQI8W1S(&)E='=E96X at 97-L:6-K('=O M;6]D97)N(&)R86YC: IE M.3$V,#,S,5T@"EM-;W-T(')E8V5N="!E9&ET&5S > M(&%N9"!Q=65R>2!T97-T:6YG"FEE M,# W,3 Q.3$U,S at U,%T@"EM!9&0@=&5S="!F;W(@=6YI8V]D92!V86QU97,@ > M:6X@ M,C P-S V,C M92!W:71H($-64R!F;W(@ M+6QI M,#$T-3-=( I;0VQE86YU<"!A;F0 at 97AP;W)T(&EN M M:6-K0&-O;6UO;BUL:7-P+FYE="HJ,C P-S U,#DP,#$S,C==( I;57!D871E > M(&5L97!H86YT(&-O9&4@=F5R M;BUL:7-P+FYE="HJ,C P-S U,#DP,#$R,39=( I;0VQE86YU<"!P97)S:7-T > M96YT(&]B:F5C="!P M M;&4 at 875G;65N=&%T:6]N(&]F(&1E8G5G9VEN9R!M;V1E;"!I;B!D97-E M;&EZ97(*:65S;&EC:T!C;VUM;VXM;&ES<"YN970J*C(P,# M72 *6T9I>"!M87 M;&5G86-Y+6YA;65S(&)U9R!F;W(@;G5L;"!C87-E"FEE > M M86QE;F-E(&9I>&5S(&9O M M,3-=( I;5&5S="!D=7!L:6-A=&4@;W!E M9&EN9R!O;B!P2!O M;VUM;VXM;&ES<"YN970J*C(P,#2!C;VYV > M96YI96YC92X* M-5T@"EM#;&5A;FEN9R!U<"!S;VUE('1Y<&4 at 9&5C;&%R871I;VYS"G)R96%D > M0&-O;6UO;BUL:7-P+FYE="HJ,C P-S Y,3$Q-34Y,CA=( I;4V]M96AO=R!T > M:&ES('=A M;B!I;B!T:&4@"G)R96%D0&-O;6UO;BUL:7-P+FYE="HJ,C P-S Y,3$Q-34W > M,30*(&-U M86-T=6%L;'D at 97AE M8VEP;&EN92X@($ET(&ES(&$@=F5R>2!I;F5L96=A;G0@=&5S="P*(&)U="!I > M="!I M8FQE+7-Y;F,M8V%C:&4@;6]R92!E9F9I8VEE;G0 at 86YD('-A9F4*2&5N M($AJ96QT93QH96YR:6M 979A:&IE;'1E+F-O;3XJ*C(P,# M72 *6V1B+7!O M8W5R M,# W,#DR,3 U,S$Q,UT@"EMA9&1E9"!S:"!S8W)I<'0 at 9F]R(&9L=7-H:6YG > M(&QO9W,@ M,#DU.# V72 *6V9I>&5S(&EN('!M+6-A8VAE"F%L97 at N;6EZ M;"YC;VTJ*C(P,#6YC+6-A8VAE"F%L > M97 at N;6EZ M;W-T;6]D97)N26UP M*BHR,# W,#@R,C(P,#4R-%T@"EMU;BUD:7-A8FQE9"!I;G-T86YC92!C86-H > M:6YG"F%L97 at N;6EZ M6W1X;B!B=')E92!V86QU92!C86-H90IA;&5X+FUI>G)A:&E 9VUA:6PN8V]M > M*BHR,# W,#DQ,C$Y,34U,UT@"EMP;2UB=')E92!M86ME+7!L<&=S<6PM:6YS > M97)T+W5P9&%T92!D=7!L:6-A=&5S(&AA;F1L:6YG(&9I>&5D"F%L97 at N;6EZ > M"!T>7!E(&1E > M8VQA"YM:7IR > M86AI0&=M86EL+F-O;2HJ,C P-S Y,#0Q,C,W,C%=( I;:6YT97)N('1O('!R > M;W!E M:4!G;6%I;"YC;VTJ*C(P,# M M"DAE;G)I:R!(:F5L=&4\:&5N M,S$U,SDQ-%T@"EMD;RUT97-T+7-P96,@:G5M<',@:6YT;R!D96)U9V=E M>2!D969A=6QT"DAE;G)I:R!(:F5L=&4\:&5N M*BHR,# W,#@R,S$T-3 M<&]S=&UO9&5R;G, at 8V]N;F5C=&EO;B!P;V]L:6YG"DAE;G)I:R!(:F5L=&4\ > M:&5N M;W-T;6]D97)N(')E;6]V92!M96%N:6YG;&5S M2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S X,C,P.30W,35= > M( I;9&(M<&]S=&UO9&5R;B!R96YA;64@=VET:"UC;VYN('=I=&@M<&]S=&UO > M9&5R;BUC;VYN"DAE;G)I:R!(:F5L=&4\:&5N M*BHR,# W,#@R,S Y,30U-ET@"EMD8BUP;W-T;6]D97)N(&-R96%T92UL86YG > M=6%G92!O;B!I;FET:6%L:7IA=&EO;B H4F]B97)T($PN(%)E860I"DAE;G)I > M:R!(:F5L=&4\:&5N M-%T@"EMD8BUP;W-T;6]D97)N(&EG;F]R92UE M86YD;&5R+6-A M;3XJ*C(P,# M(&EN:71I86QI>F%T:6]N(&-O9&4*2&5N M:&IE;'1E+F-O;3XJ*C(P,# M8G5G9FEX('=I=&@@=')A;G-A8W1I;VX@:&%N9&QI;F<*2&5N M93QH96YR:6M 979A:&IE;'1E+F-O;3XJ*C(P,# M+7!O M=F%H:F5L=&4N8V]M/BHJ,C P-S X,C(P,C V,3==( I;9&(M<&]S=&UO9&5R > M;CH at 97AE8W5T92UT M M.#(P,C Q-3 X72 *6V1B+7!O M M-S X,30P.3 T,3-=( I;9&(M<&]S=&UO9&5R;B!B=6=F:7@@86=A:6X@;6%P > M+6EN9&5X('!A=&-H(&ES(&%L=V%Y M96YR:6M 979A:&IE;'1E+F-O;3XJ*C(P,# M M=&4\:&5N M8BUP;W-T;6]D97)N(&EG;F]R92UE M M.# X,#DS.3 X"B!!;B!U9VQY(&9I>"!T:&%T('-H;W5L9"!B92!S;VQV960@ > M870@ M"DAE;G)I:R!(:F5L=&4\:&5N M,3 W,S S,5T@"EMD8BUP;W-T;6]D97)N(&UA:V4 at 8VAA M9F%U;'0*2&5N M,# M+6%N9"UV87)S(&-H86YG960@=&\@=VET:"UV87)S"DAE;G)I:R!(:F5L=&4\ > M:&5N M;W-T;6]D97)N(&-U I( > M96YR:6L at 2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S W,C4R > M,C Q,35=( I;='EP;R!F:7@*5&EE M( I;8G5G9FEX(&-U M;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S W,C(Q-# Y,CE=( I; > M9&(M<&]S=&UO9&5R;B!S;VUE(')E9F%C=&]R:6YG"DAE;G)I:R!(:F5L=&4\ > M:&5N M;W-T;6]D97)N(&UI M:6M 979A:&IE;'1E+F-O;3XJ*C(P,# M;V1E M=F%H:F5L=&4N8V]M/BHJ,C P-S W,C(P-S0Q,39=( I;;6%K92!A(&9I>'1U > M M86AJ96QT92YC;VT^*BHR,# W,# M(&9I>"!F;W(@<')O8FQE;2!W:71H(&UA<"UI;F1E> I(96YR:6L at 2&IE;'1E > M/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S W,C(P-C$T,#$*(&5L97!H > M86YT(&%P<&%R96YT;'D@ M M:7-P+6-O;7!A M2P at 8G5T(&YO="!F;W(@=&AE"B!P;W-T > M;6]D97)N(&1E M('-O M86YT(&]R(&EM<&QE;65N="!D8BUP;W-T;6]D97)N"B!D:69F97)E;G1L>2X@ > M5&AI M;'D*('=H96X@=7-I;F<@=&AE(&1B+7!O M('!R971T>2X*72 *6W-O;64@;6]R92!B87-I8R!I;F1E>&EN9R!T97-T M96YR:6L at 2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P-S W,C(P > M-C W,S-=( I;=&5S=&-A M:F5L=&4\:&5N M"EMD8BUP;W-T;6]D97)N(&YE=R!I;7!L96UE;G1A=&EO;B!O9B!C=7)S;W(M > M M,# M=')I;F=S+ at H@5V]R:W,@=VET:"!T97-T8V%S97, at 8G5T(&YE961S(&-L96%N > M:6YG('5P+BX*72 *6W1E M"DAE;G)I:R!(:F5L=&4\:&5N M,3$X,#4S-PH at 9F%I;',@;VX@<&]S=&UO9&5R;BP@=V]R:W,@=VET:"!B9&(@ > M86YD(&-L&EN9RUB87-I8R!T97-T M;W(@8V]M<&QE=&5N97-S"DAE;G)I:R!(:F5L=&4\:&5N M92YC;VT^*BHR,# W,# M<"UI;F1E>"!T97-T M8V]M/BHJ,C P-S W,C$Q-3,V,39=( I; M;65R9V5D('1O(&]N92!T97-T"DAE;G)I:R!(:F5L=&4\:&5N M96QT92YC;VT^*BHR,# W,# M;65N=&5D(&-O9&4*72 *6W1E M=F5A;0I(96YR:6L at 2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ,C P > M-S W,C$Q-#4V,S-=( I;9FEV96%M(&UA:V4 at 82!D969A=6QT('1E M92!E;&5P:&%N="UT97-T M=&4N8V]M/BHJ,C P-S W,C$Q-#4U,S-=( I;5&5S="!F M;F=E9"!T;R!&:79E04T*2&5N M+F-O;3XJ*C(P,# M M0&5V86AJ96QT92YC;VT^*BHR,# W,#&5D > M+6)T M, H@(" @(" @(" @(" @(" @(" @(" @(" @(" @(" @(" @(" @(" @(" @ > M(" @(" @(" @(" @(" @( I=( I;<&]S=&UO9&5R;B!D:7-A8FQE(&-A8VAE > M"DAE;G)I:R!(:F5L=&4\:&5N M-#$V,30T,PH at 0F5C875S92!I="!M87D at 8V%U M9&EF9F5R96YT('!R;V-E M72 *6U!U='1I;F=);F1E>$]N360U8V]L=6UN"G)R96%D0&-O;6UO;BUL:7-P > M+FYE="HJ,C P-S W,3 R,#(P-3D*($D@:&%V96XG="!G;W1T96X at 2&5N M)W, at 87!P M"B H86YD(&UY('1E M=6UN('1H870@ M=F5397)I86QI>F%T:6]N17)R;W)S"G)R96%D0&-O;6UO;BUL:7-P+FYE="HJ > M,C P-S W,3 R,#$X,C8*(%1H:7,@:7, at 86X@871T96UP="!T;R!S=7)V:79E > M('-E M<&5R M=&%B87-E('-I;F-E(&%R;W5N9" P+C,I+"!A;F0@=&AI M M6]U M8F%S92!G M0&-O;6UO;BUL:7-P+FYE="HJ,C P-S W,3 R,#$V,CD*(%1H:7,@:7, at 82!T > M97-T(&]F(&AO=R!B:6<@82!K97D at 8V%N(&=E="X@($ET('=A M8V5D('=H96X*(&1E8G5G9VEN9R!T:&4@<&]S=&UO9&5R;B!S='5F9BX@(%5N > M9F]R='5N871E;'DL('1H97)E(&ES(&$@;&EM:70L"B!W:&EC:"!W92!S:&]U > M;&0@=V]R:R!T;R!O=F5R8V]M92X*72 *6V1B+7!O M;&5A;FEN9PI(96YR:6L at 2&IE;'1E/&AE;G)I:T!E=F%H:F5L=&4N8V]M/BHJ > M,C P-S W,#DQ.3$S,3)=( I;1FEX960 at 4F]B97)T M92!B;&]B M,C P-S W,#DQ.3 W,3D*($)E8W5A=7-E('1H92!B;V(@8V]L=6UB(&EN(&)L > M;V(@:&%D(&$@=6YI<75E(&EN9&5X+ at H@3F]W('1H92!I;F1E>"!I M(&UD-2!V86QU92!O9B!T:&4 at 8F]B+ at I=( I; M=6)S971S('=O M86AJ96QT92YC;VT^*BHR,# W,# M('-E;F0 at 8FEN87)Y(&1A=&$@:6YS=&5A9"!O9B!B87-E-C0*2&5N M96QT93QH96YR:6M 979A:&IE;'1E+F-O;3XJ*C(P,# M6V1B+7!O M(#8T(&)I="!I;G1E9V5R"DAE;G)I:R!(:F5L=&4\:&5N M92YC;VT^*BHR,# W,# M="!S=')I;F=S(&%S(&]B:F5C=',N"DAE;G)I:R!(:F5L=&4\:&5N M86AJ96QT92YC;VT^*BHR,# W,# M1&%T86)A M965D M,2\S(&]F(&$@8G5F9F5R('!A9V4 at 8V%N;F]T(&)E(&EN9&5X960N"B!#;VYS > M:61E M86QU92P@;W(@=7-E(&9U;&P@=&5X="!I;F1E>&EN9RX*72 *6V1B+7!O M;V1E M:F5L=&4N8V]M/BHJ,C P-S W,#8P,S,S-#5=( I;=7!D871E9"!I;G-T86QL > M+6)D8BYS:"!I;B!H96YR:6LO8V]N=')I8 at I(96YR:6L at 2&IE;'1E(#QH:&IE > M;'1E0&-O;6UO;BUL:7-P+FYE=#XJ*C(P,# M+65X96,M<')E<&%R960@ M0&5V86AJ96QT92YC;VT^*BHR,# W,# M(&EN=&5G M,30Q,3%=( I;<&]S=&UO9&5R;B!T97-T M/&AH:F5L=&5 8V]M;6]N+6QI M9&(M<&]S=&UO9&5R;B!B=6<@9FEX(&9O M86P*2&5N M,# W,#4Q,3$W,#8T.%T@"EMT97-T8V]L;&5C=&EO;G,L(&-U M:6]U M/&AH:F5L=&5 8V]M;6]N+6QI M8F5C875S92!O9B!L;V]P(&9O M M:71E('=H870@=&AE('9A;'5E(&9O M M97-T960N"ET@"EMP;W-T;6]D97)N(&-U M8FQE('1O(&UO=F4@=&\@;F5X="!K97D*2&5N M94!C;VUM;VXM;&ES<"YN970^*BHR,# W,#4Q,3 W,#$U.5T@"EMU<&=R861E > M+6)T M:&IE;'1E0&-O;6UO;BUL:7-P+FYE=#XJ*C(P,# M:"!F:7AE9"!A(&)U9R!S:&]W;B!B>2!T97-T('!S970*72 *6VUA:V4 at 36%K > M969I;&4 at 9&]C=6UE;G1A=&EO;B!B=6EL9"!O;B!S8F-L"DAE;G)I:R!(:F5L > M=&4@/&AH:F5L=&5 8V]M;6]N+6QI M( I;;6%K92!D8BUP;W-T;6]D97)N(&-O;7!I;&4 at 86YD(&QO860*2&5N M($AJ96QT92 \:&AJ96QT94!C;VUM;VXM;&ES<"YN970^*BHR,# W,#4Q,# Y > M,S0R,5T@"EMI;FET:6%L(&EM<&]R="!D8BUP;W-T;6]D97)N"DAE;G)I:R!( > M:F5L=&4@/&AH:F5L=&5 8V]M;6]N+6QI M,#1=( I;4')E=F%L96YC92!B86-K96YD($)4 M=&%T:6]N.R!A;&P at 8G5T(#$Q('1E M86QI>F5R*0II97-L:6-K0&-O;6UO;BUL:7-P+FYE="HJ,C P-S U,#4P-#,U > M-#5=( I;1FEX(&)T M=&EO;@II97-L:6-K0&-O;6UO;BUL:7-P+FYE="HJ,C P-S U,# M( I;16YA8FQE('-U8F-L87-S(&]V97)R:61E(&]F(&-O;G1R;VQL97(@;V)J > M96-T(&-A8VAE"FEE M,C8P.%T@"EM!9&0@ M('1H92!D97-E MF5D(&1A=&%B87-E(&5R M9VYA;"!I'!O M+FYE="HJ,C P-S U,#4R,C$R,#A=( I;0G5G(&9I>"!F;W(@<')I;6%R>2!K > M97D@;6ES M970J*C(P,# M8V]N9&%R>2!C=7)S;W(@;W!S"FEE M,# W,#4P,C$Y,3(Q.%T@"EM297-O;'9E(%)%041-12!#;VYF;&EC= II97-L > M:6-K0&-O;6UO;BUL:7-P+FYE="HJ,C P-S U,#(Q.#$W,#==( I;4')E=F%L > M96YC92!S=&]R92!B87-I8R!O<',@*R!"5')E97,@*R!S;VUE(&EN9&5X("L@ > M M8V]M;6]N+6QI M<&]R="!O9B!#5E,@,"XY+C$@4D,Q"FEE M*BHR,# W,#0S,#(R-# R-ET@"EM%;7!T>2!R97!O M8V]M;6]N+6QI M92!H87-H. at HV,6(U96,R,C8V.6)D-V,P,S$R,F%E.#-F86(U96,T,#(R,#EF > $9#-B"@`` > ` > end > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From Hit-Booster at common-lisp.net Tue Feb 19 17:05:27 2008 From: Hit-Booster at common-lisp.net (Hit-Booster at common-lisp.net) Date: 19 Feb 2008 09:05:27 -0800 Subject: [elephant-devel] How would you like unlimited hits to your website 15 minutes from now ? Message-ID: <20080219090527.D0FC90C48B62F521@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From leslie.polzer at gmx.net Wed Feb 20 15:49:56 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 20 Feb 2008 16:49:56 +0100 (CET) Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE Message-ID: <61659.88.73.215.222.1203522596.squirrel@mail.stardawn.org> This concerns indexed classes. Observe: (asdf:oos 'asdf:load-op 'elephant) (defpackage #:ele-test (:use :cl :elephant)) (in-package :ele-test) (defvar *sc2* nil) (defvar item nil) (defpclass myclass () ((testslot :accessor testslot :initarg testslot :index t))) (open-store '(:BDB "/tmp/db1") :recover t) (setf *sc2* (open-store '(:BDB "/tmp/db2") :recover t)) (let ((*store-controller* *sc2*)) (setf item (make-instance 'myclass))) (setf (testslot item) 5) ; ==> Attempted to write object # with home store ; # into store # (close-store) (close-store *sc2*) Intuitively, the SLOT-VALUE writer should have figured out the correct store controller on its own, by simply looking up ITEM's home controller. Instead, one has to specify it explicitly: (let ((*store-controller* *sc2*)) (setf (testslot item) 5)) ; works Is there any sensible reason for this behaviour? Is my suspicion correct that multi-store operation is a point that isn't covered well by the unit tests and experience? Leslie From leslie.polzer at gmx.net Wed Feb 20 16:36:41 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 20 Feb 2008 17:36:41 +0100 (CET) Subject: [elephant-devel] Fix for Unicode2 serializer Message-ID: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> I don't know whether it was a change in Elephant, SBCL or even somewhere else that introduced problems with the Unicode serializer, but the fix is very small (changed two calls to SCHAR to CHAR) and should probably be applied ASAP. I'm not sure why the code is discerning between STRING and SIMPLE-STRING anyway. Does anyone know? Leslie -- My personal blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: unicode2.patch Type: text/x-patch Size: 4014 bytes Desc: not available URL: From killerstorm at newmail.ru Wed Feb 20 22:14:11 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 21 Feb 2008 00:14:11 +0200 Subject: [elephant-devel] txn Message-ID: hello i'm now going to improve transaction handling in db-postmodern. first of all, i'm going to use "serializable" isolation level instead of default "read commited". this means: transaction can fail because of concurrent slot updates. with default isolation level concurrent updates were OK, last value persisted. so with default isolation level operations like INCF could be just ignored, that's not safe in some cases. another difference is that with default isolation level it was possible to read data that was commited after txn started -- this artefact is also fixed. they say there should be no performance impact, but now we should be ready to actually retry transaction body (actually deadlocks were possible before, but since they are infrequent we ignored them). first of all, some aspects of execute-transaction implementation: * in BDB backend retries count is actually "tries count", so retries = 1 means "no retries". i find it counter-logical: there is always one first try, and then N retries. 0 = retries means that there still should be one TRY, but no REtries. i'm not only one interpreting parameter in this way, so please fix this in bdb implementation. * transaction-retry-count-exceeded condition does not have slot for original cause. that is weird. application catches retry-count-exceeded. but what in a hell happened?? i can extend this condition for db-postmodern, but i think it will be better if all backends would handle it in this way. * for a case retries = 0 i think it would be better to throw original condition rather than transaction-retry-count-exceeded, since there were no retries.. also this can help debugging in some way. however, i found it's problematic to integrate txn retrying in complex application having side effects (i.e. web framework). at same time using low level primitives in this case is ugly. i think it's possible to make it much more flexible with single callback function, that is called when transaction is about to be restarted. so applications has chance to: * log error, so later it would be possible to diagnose performance problems etc. * do cleanup of side-effects * implement some other restarting mechanism this will be called retry-cleanup-fn key parameter. additionally, i've found some problems with "separatist" error handlers i.e.: (ele:with-transaction () (handler-case (work-with-database) (error () (pring "we have a no-go"))) if work-with-database produced some error, with-transaction will not able to know about this and thus could commit erroneous data. of course this is a problem of application, but sometimes such handlers are installed by web framework or dictated by architecture in some way. and it will be nice to be able to deal with such cases. i have some ideas how to handle this: nested with-transaction instead of simply calling function (because SQL does not have nested transactions) could detect abnormal control flow or conditions signaled and inform parent transaction about this -- so it can rollback or retry. but i'm not going to implement this right now. other extensions in db-postmodern's execute-transaction: * always-rollback parameter that turns off auto-commit. could be useful if you're doing test and do not want results in database.. * execute-transaction creates restart named db-postmodern::retry-transaction, so application can programmatically force retry. comments appreciated. with best regards, Alex 'killerstorm' Mizrahi. From eslick at media.mit.edu Wed Feb 20 22:32:43 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 20 Feb 2008 17:32:43 -0500 Subject: [elephant-devel] Fix for Unicode2 serializer In-Reply-To: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> References: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> Message-ID: On Feb 20, 2008, at 11:36 AM, Leslie P. Polzer wrote: > > I don't know whether it was a change in Elephant, SBCL or even > somewhere > else that introduced problems with the Unicode serializer, but the fix > is very small (changed two calls to SCHAR to CHAR) and should probably > be applied ASAP. > > I'm not sure why the code is discerning between STRING and SIMPLE- > STRING > anyway. Does anyone know? I think that some lisps differentiated between 8-bit arrays of standard ASCII text (simple string) and string which is a more general type encompassing multi-byte and UTF8 unicode. Since we don't byte copy simple strings anymore (do we?) it may be an artifact of the earlier serializer implementation's optimization. > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Wed Feb 20 22:39:19 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 20 Feb 2008 17:39:19 -0500 Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE In-Reply-To: <61659.88.73.215.222.1203522596.squirrel@mail.stardawn.org> References: <61659.88.73.215.222.1203522596.squirrel@mail.stardawn.org> Message-ID: I just looked at the code and it appears to call 'get-con' in all the right places. The code certainly is structured to do what you suggest (get controller from the instance slot). I'll chase this down tonight. Perhaps some recent change created a corner case that the tests don't catch, although this seems like a pretty significant hole. FYI - Most of the multi-store ops are in the migration tests and I don't think people run those separate tests religiously. Perhaps we should make the standard tests default to running migration between two stores of the same type. The migration tests are a little slow so maybe there's a keyword option to turn them off. Ian On Feb 20, 2008, at 10:49 AM, Leslie P. Polzer wrote: > > This concerns indexed classes. > > Observe: > > > (asdf:oos 'asdf:load-op 'elephant) > > (defpackage #:ele-test (:use :cl :elephant)) > > (in-package :ele-test) > > (defvar *sc2* nil) > (defvar item nil) > > (defpclass myclass () > ((testslot :accessor testslot :initarg testslot :index t))) > > (open-store '(:BDB "/tmp/db1") :recover t) > (setf *sc2* (open-store '(:BDB "/tmp/db2") :recover t)) > > (let ((*store-controller* *sc2*)) > (setf item (make-instance 'myclass))) > > (setf (testslot item) 5) > ; ==> Attempted to write object # with home store > ; # into store # CONTROLLER /tmp/db1> > > > (close-store) > (close-store *sc2*) > > > Intuitively, the SLOT-VALUE writer should have figured out the > correct store controller on its own, by simply looking up ITEM's > home controller. Instead, one has to specify it explicitly: > > (let ((*store-controller* *sc2*)) > (setf (testslot item) 5)) ; works > > Is there any sensible reason for this behaviour? > Is my suspicion correct that multi-store operation is a point that > isn't > covered well by the unit tests and experience? > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Wed Feb 20 22:43:42 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 20 Feb 2008 17:43:42 -0500 Subject: [elephant-devel] txn In-Reply-To: References: Message-ID: <8712CF6D-01FC-484E-A307-567BF34428DA@media.mit.edu> I'm not sure I understand this part? Do you mean if you have an internal error which does not complete the intended transaction but doesn't throw that through the 'with-transaction' statement then you get bad data? It seems that you may be expecting too much of the database. If you are protecting complex web transactions that may take many different paths, some correct and some not, there really is no way to resolve all this from a context created at a higher point in the stack. We allow nesting to make default transaction behavior easy - the outer transaction determines the commit unit. Do you want an option to catch an unexpected internal nesting? I'll look at the rest of this tonight and comment. Cheers, Ian On Feb 20, 2008, at 5:14 PM, Alex Mizrahi wrote: > additionally, i've found some problems with "separatist" error > handlers > i.e.: > > (ele:with-transaction () > (handler-case (work-with-database) > (error () (pring "we have a no-go"))) > > if work-with-database produced some error, with-transaction will not > able to > know about this and thus could commit erroneous data. > > of course this is a problem of application, but sometimes such > handlers are > installed by web framework or dictated by architecture in some way. > and it > will be nice to be able to deal with such cases. From lists at infoway.net Thu Feb 21 01:29:34 2008 From: lists at infoway.net (lists at infoway.net) Date: Wed, 20 Feb 2008 20:29:34 -0500 Subject: [elephant-devel] Elephant BDB Tests Fail on OS X Leopard, SBCL 1.0.14 Message-ID: <3D3FE2FF-EB06-4E45-ACEB-2FFFBE68035A@infoway.net> Hi all, I downloaded the latest elephant this morning and ran the BDB tests, one of which failed. I haven't really looked much into it because I'm traveling. However, just wanted to share this with you in case you were aware (or not) and I might have missed something. Thanks, Waldo **** SBCL: SBCL was compiled from source enabling threads. This is SBCL 1.0.14, an implementation of ANSI Common Lisp. **** SLIME: ELE-TESTS> (do-backend-tests) Attempting to load libmemutil.dylib... Loaded /Users/waldo/dev/lisp/elephant/src/memutil/libmemutil.dylib STYLE-WARNING: redefining COPY-BUFS in DEFUN STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining READ-INT32 in DEFUN STYLE-WARNING: redefining READ-INT64 in DEFUN STYLE-WARNING: redefining READ-UINT32 in DEFUN STYLE-WARNING: redefining READ-UINT64 in DEFUN STYLE-WARNING: redefining READ-FLOAT in DEFUN STYLE-WARNING: redefining READ-DOUBLE in DEFUN STYLE-WARNING: redefining WRITE-INT32 in DEFUN STYLE-WARNING: redefining WRITE-INT64 in DEFUN STYLE-WARNING: redefining WRITE-UINT32 in DEFUN STYLE-WARNING: redefining WRITE-UINT64 in DEFUN STYLE-WARNING: redefining WRITE-FLOAT in DEFUN STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN ; compiling file "/Users/waldo/dev/lisp/elephant/tests/ testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): ; compiling (IN-PACKAGE :ELE-TESTS) ; compiling (DEFVAR ENV) ; compiling (DEFVAR DB) ; compiling (DEFUN PREPARE-BDB ...) ; compiling (DEFTEST PREPARES-BDB ...) ; compiling (DEFUN TEST-SEQUENCE1 ...) ; compiling (DEFTEST TEST-SEQ1 ...) ; compiling (DEFUN TEST-SEQUENCE2 ...) ; compiling (DEFTEST TEST-SEQ2 ...) ; compiling (DEFUN CLEANUP-BDB ...) ; compiling (DEFTEST CLEANSUP-BDB ...) ; /Users/waldo/dev/lisp/elephant/tests/testbdb.fasl written ; compilation finished in 0:00:00 Doing 132 pending tests of 132 tests total. FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS COMPLEXES BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- CONSTRUCT CIRCULAR PERSISTENT; in: LAMBDA NIL ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM REDEFCLASS MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES-SLOT2 MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET-RANGE SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM-INDEX3 DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST CUR- DEL1 INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC INDEXING-CLASS-OPT INDEXING-INHERIT Test INDEXING-RANGE failed Form: (PROGN (DEFCLASS IDX-FOUR NIL ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 :INDEX T)) (:METACLASS PERSISTENT-METACLASS)) (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) (DEFCLASS IDX-FOUR NIL ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 :INDEX T)) (:METACLASS PERSISTENT-METACLASS)) (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 VAL)) (WITH-TRANSACTION NIL (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 5 6 10))) (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) Expected values: T T T Actual value: #. INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS Ranged get of 10/700 objects = Linear: 0.435 sec Indexed: 0.01 sec INDEXING-TIMING ; in: LAMBDA NIL ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring REMOVE-ELEMENT ; in: LAMBDA NIL ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-BASIC ; in: LAMBDA NIL ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-BTREE ; in: LAMBDA NIL ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-IDX-BTREE ; in: LAMBDA NIL ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-PCLASS ; in: LAMBDA NIL ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-MULT-PCLASS ; in: LAMBDA NIL ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) ; ==> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) ; ; caught WARNING: ; undefined variable: *TEST-SPEC-SECONDARY* ; ; caught WARNING: ; This variable is undefined: ; *TEST-SPEC-SECONDARY* ; ; compilation unit finished ; caught 2 WARNING conditions Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 TEST-SEQ2 CLEANSUP-BDB 1 out of 132 total tests failed: INDEXING-RANGE.NIL From lists at infoway.net Thu Feb 21 01:44:15 2008 From: lists at infoway.net (lists at infoway.net) Date: Wed, 20 Feb 2008 20:44:15 -0500 Subject: [elephant-devel] BDB vs postmodern Message-ID: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> Hi all, I don't mean to start a war here or put any work down. However, I just needed some clarification/direction into which way the data stores work is going. From the documentation I've read, the BDB data store only supports BDB version 4.5 (or up to 4.5), so further work done in 4.6 and above is not leveraged. I don't know if there is any effort to continue evolving the BDB data store. Any comments? Also, from what I've been reading on the list, it seems the postmodern data store is more active. Granted is a more recent data store and may need more work to get to where the BDB data store has gotten after all this time (just speculation since I haven't really used the postmodern data store). However, from what I've read, the postmodern seems to use Postgres as the backend but not in a Object-Relational mode but rather as plain storing of "btree pages". So, in essence, what I understand is that operating on this data store requires elephant to read/write "pages" or "btrees" in Postgres and once in memory, they may be treated similar to how the BDB engine works (don't mean to mock postmodern with my lack of knowledge). So, putting licensing issues aside, what is the real difference/ advantage of one data store over the other? In a recent email I read by Alex, he mentions he's going to work on improving performance on the postmodern data store. So, even after that improves and performance is matched with that of BDB, is there an advantage of one data store over the other? Since Postgres does allow for features such as replication, clustering, and fail-over with multiple active simultaneous client connections, does this mean that I could have multiple (separate) lisp clients using elephant connecting to a separate Postgres cluster with no concurrency issues? Thanks, Waldo From eslick at media.mit.edu Thu Feb 21 02:09:54 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 20 Feb 2008 21:09:54 -0500 Subject: [elephant-devel] BDB vs postmodern In-Reply-To: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> Message-ID: On Feb 20, 2008, at 8:44 PM, lists at infoway.net wrote: > Hi all, > > I don't mean to start a war here or put any work down. However, I > just needed some clarification/direction into which way the data > stores work is going. One challenge is that the current implementation is setup to only work with one BDB version at a time. We tend not to exploit much of the new functionality available in later 4.x versions,a although we benefit from performance, stability and other general improvements. > From the documentation I've read, the BDB data store only supports > BDB version 4.5 (or up to 4.5), so further work done in 4.6 and > above is not leveraged. I don't know if there is any effort to > continue evolving the BDB data store. Any comments? Because we don't want to push people to upgrade each time BDB releases a new incremental, and because we don't keep close tabs on the BDB release schedule, we haven't been aggressively upgrading BDB's compatibility. Most Linux implementations allow you to keep multiple versions for awhile, and we haven't had any community pushback on this lazy upgrade policy. Also I'm the only lead developer with intimate knowledge of the db-bdb data store. While I support bug fixes, and will occasionally work on features, I'm not very active right now. > Also, from what I've been reading on the list, it seems the > postmodern data store is more active. Granted is a more recent data > store and may need more work to get to where the BDB data store has > gotten after all this time (just speculation since I haven't really > used the postmodern data store). However, from what I've read, the > postmodern seems to use Postgres as the backend but not in a Object- > Relational mode but rather as plain storing of "btree pages". So, in > essence, what I understand is that operating on this data store > requires elephant to read/write "pages" or "btrees" in Postgres and > once in memory, they may be treated similar to how the BDB engine > works (don't mean to mock postmodern with my lack of knowledge). That's essentially correct, although the postmodern developers can probably provide more insight. > So, putting licensing issues aside, what is the real difference/ > advantage of one data store over the other? In a recent email I read > by Alex, he mentions he's going to work on improving performance on > the postmodern data store. So, even after that improves and > performance is matched with that of BDB, is there an advantage of > one data store over the other? BDB is the fastest Postmodern is about 4x slower (is this still true) CL-SQL on Postgresql is about another 2x slower yet CL-SQL on SQLite3 is even slower (another 2x?), although there is speculation that this is artificial but we haven't investigated. > Since Postgres does allow for features such as replication, > clustering, and fail-over with multiple active simultaneous client > connections, does this mean that I could have multiple (separate) > lisp clients using elephant connecting to a separate Postgres > cluster with no concurrency issues? Yes. You can do this on BDB, but only on the same system as it relies on shared memory locks between processes. This helps for multi-CPU systems (one lisp process per CPU) but not for distributing Elephant across > Thanks, > Waldo > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Thu Feb 21 02:38:41 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 20 Feb 2008 21:38:41 -0500 Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE In-Reply-To: <61659.88.73.215.222.1203522596.squirrel@mail.stardawn.org> References: <61659.88.73.215.222.1203522596.squirrel@mail.stardawn.org> Message-ID: Ok, this is an architectural issue that wasn't completely thought out. Currently class indexing effectively limits instances of a class to a specific store. The first time an instance is saved, the current *store-controller* is used to create the class index tying the class to that store. I have a local patch which changes this behavior to create a class index for each store's objects. For example, in your test code, if you call get-instances-by-class in the context of *sc1*, you'll get nothing, in the context of *sc2* you'll get the instance you created in *sc2*. Each store has it's own local set of class indices. The problem here is that there are quite a few assumptions baked into class indexing that ties a given class to a given store: - The class object has an index cache which can only point to one store at a time. - The list of indexed slots in-memory can be synchronized to a store's version when the class is first used (to avoid losing track of existing indices of objects); two stores with two different versions of indexed slots would create very complex behavior - What if one store is created with one definition of a class, but another store is created with different defintion? If the classes are both indexed and have the same class name, one store's data may be overwritten by the definition created for another. The solution is probably to ensure that the class behaves differently in the context of a given store, but then I imagine we'd have to keep a per-store list of indexed slots and dispatch writes based on those differences. I'm concerned there are other areas where the policy choice becomes difficult. My inclination is to assert that if you want to index persistent objects in multiple stores, you should plan for that explicitly and not use the class indexing mechanism. I'm willing to be argued in the other direction, but I'm not terribly motivated to chase down all the possible corner cases at this point. Ian On Feb 20, 2008, at 10:49 AM, Leslie P. Polzer wrote: > > This concerns indexed classes. > > Observe: > > > (asdf:oos 'asdf:load-op 'elephant) > > (defpackage #:ele-test (:use :cl :elephant)) > > (in-package :ele-test) > > (defvar *sc2* nil) > (defvar item nil) > > (defpclass myclass () > ((testslot :accessor testslot :initarg testslot :index t))) > > (open-store '(:BDB "/tmp/db1") :recover t) > (setf *sc2* (open-store '(:BDB "/tmp/db2") :recover t)) > > (let ((*store-controller* *sc2*)) > (setf item (make-instance 'myclass))) > > (setf (testslot item) 5) > ; ==> Attempted to write object # with home store > ; # into store # CONTROLLER /tmp/db1> > > > (close-store) > (close-store *sc2*) > > > Intuitively, the SLOT-VALUE writer should have figured out the > correct store controller on its own, by simply looking up ITEM's > home controller. Instead, one has to specify it explicitly: > > (let ((*store-controller* *sc2*)) > (setf (testslot item) 5)) ; works > > Is there any sensible reason for this behaviour? > Is my suspicion correct that multi-store operation is a point that > isn't > covered well by the unit tests and experience? > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Thu Feb 21 02:47:35 2008 From: lists at infoway.net (lists at infoway.net) Date: Wed, 20 Feb 2008 21:47:35 -0500 Subject: [elephant-devel] BDB vs postmodern In-Reply-To: References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> Message-ID: <449E9A14-A4D2-4250-B920-A99DDBE2796D@infoway.net> Well, this is certainly interesting, since this would allow me to decouple the storage system from the lisp environment allowing the possibility of setting up a cluster of lisp machines to handle application logic. Isn't there a way to achieve this on BDB? We prefer to deploy our systems on clusters of "inexpensive" machines in order to leverage hardware failures and it seems that scalability of BDB/Lisp applications is achieved by scaling a single machine. Now, postmodern being 4x slower could be an issue. However, how does that compare to a regular CL-SQL and relational queries is a different story. If postmodern is about the same speed as hitting Postgres with CL-SQL and just using plain SQL instead of elephant, at the end of the day, that's the performance our users are getting anyway :) Thanks, Waldo >> >> Since Postgres does allow for features such as replication, >> clustering, and fail-over with multiple active simultaneous client >> connections, does this mean that I could have multiple (separate) >> lisp clients using elephant connecting to a separate Postgres >> cluster with no concurrency issues? > > Yes. > > You can do this on BDB, but only on the same system as it relies on > shared memory locks between processes. This helps for multi-CPU > systems (one lisp process per CPU) but not for distributing Elephant > across > >> Thanks, >> Waldo >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at media.mit.edu Thu Feb 21 03:05:11 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 20 Feb 2008 22:05:11 -0500 Subject: [elephant-devel] BDB vs postmodern In-Reply-To: <449E9A14-A4D2-4250-B920-A99DDBE2796D@infoway.net> References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> <449E9A14-A4D2-4250-B920-A99DDBE2796D@infoway.net> Message-ID: <86E12430-DEF0-4975-BE45-05EF5580148E@media.mit.edu> I doubt that Elephant on postmodern is going to be faster than using CL-SQL to do direct ORM against Postgresql. Despite the great work that Henrik and others have done with Postmodern, there are too many layers of abstraction in the architecture to overcome. The real point of using Elephant is ease of use, rapid prototyping and the programming API, not absolute performance. I do suspect that raw performance (using BDB) may be comparable or better to any SQL/ORM solution for some simple access patterns, but for complex queries and joins SQL is probably the better way to go. We have already seen various comments on the list about common features of SQL databases that Elephant is missing (like getting a COUNT without generating all the objects). As for using BDB to do replication and distributed transactions - it's possible but no one has tried it. Elephant would need some serious modifications as the transaction protocol is different and I think that you'd have to build your own Global Transaction Manager either in Lisp or as an additional C daemon. It's also possible that there are architectural or performance issues in using Postgresql across multiple machines of which I am unaware - hopefully one of the PG experts will comment. Ian On Feb 20, 2008, at 9:47 PM, lists at infoway.net wrote: > Well, this is certainly interesting, since this would allow me to > decouple the storage system from the lisp environment allowing the > possibility of setting up a cluster of lisp machines to handle > application logic. Isn't there a way to achieve this on BDB? > > We prefer to deploy our systems on clusters of "inexpensive" > machines in order to leverage hardware failures and it seems that > scalability of BDB/Lisp applications is achieved by scaling a single > machine. > > Now, postmodern being 4x slower could be an issue. However, how does > that compare to a regular CL-SQL and relational queries is a > different story. If postmodern is about the same speed as hitting > Postgres with CL-SQL and just using plain SQL instead of elephant, > at the end of the day, that's the performance our users are getting > anyway :) > > Thanks, > Waldo > >>> >>> Since Postgres does allow for features such as replication, >>> clustering, and fail-over with multiple active simultaneous client >>> connections, does this mean that I could have multiple (separate) >>> lisp clients using elephant connecting to a separate Postgres >>> cluster with no concurrency issues? >> >> Yes. >> >> You can do this on BDB, but only on the same system as it relies on >> shared memory locks between processes. This helps for multi-CPU >> systems (one lisp process per CPU) but not for distributing >> Elephant across >> >>> Thanks, >>> Waldo >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Thu Feb 21 03:08:43 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 20 Feb 2008 22:08:43 -0500 Subject: [elephant-devel] txn In-Reply-To: References: Message-ID: On Feb 20, 2008, at 5:14 PM, Alex Mizrahi wrote: > hello > > i'm now going to improve transaction handling in db-postmodern. Great! We should make sure this doesn't change the API, or if it does to create tracking changes in the BDB and CL-SQL stores. > first of all, i'm going to use "serializable" isolation level > instead of > default "read commited". this means: transaction can fail because of > concurrent slot updates. This is the BDB default. We have keyword options to :read-uncommitted (see bdb-transactions.lisp) or the documentation for other options. > with default isolation level concurrent updates were OK, last value > persisted. > so with default isolation level operations like INCF could be just > ignored, > that's not safe in some cases. > > another difference is that with default isolation level it was > possible to > read data that was commited after txn started -- this artefact is also > fixed. > > they say there should be no performance impact, but now we should be > ready > to actually retry transaction body (actually deadlocks were possible > before, > but since they are infrequent we ignored them). > > first of all, some aspects of execute-transaction implementation: > > * in BDB backend retries count is actually "tries count", so retries > = 1 > means "no retries". > > i find it counter-logical: there is always one first try, and then N > retries. > 0 = retries means that there still should be one TRY, but no > REtries. > > i'm not only one interpreting parameter in this way, so please fix > this > in bdb implementation. I just checked in a patch to correct this, good eye! > * transaction-retry-count-exceeded condition does not have slot for > original cause. that is weird. > application catches retry-count-exceeded. but what in a hell > happened?? > i can extend this condition for db-postmodern, but i think it will > be > better if all backends would handle it in this way. > > * for a case retries = 0 i think it would be better to throw original > condition rather than transaction-retry-count-exceeded, > since there were no retries.. also this can help debugging in some > way. > In the db-bdb store, transaction-retry-count-exceeded only happens when the non-local exist from the transaction closure is a DEADLOCK or a LOCK_NOT_GRANTED. Any other non-local exit results in the transaction being aborted and the error being returned. > > however, i found it's problematic to integrate txn retrying in complex > application having side effects (i.e. web framework). > at same time using low level primitives in this case is ugly. > > i think it's possible to make it much more flexible with single > callback > function, that is called when transaction is about to be restarted. > so applications has chance to: > * log error, so later it would be possible to diagnose performance > problems > etc. > * do cleanup of side-effects > * implement some other restarting mechanism > > this will be called retry-cleanup-fn key parameter. Great! Let me know when you have it, what assumptions you're making, and I'll update db-bdb to match. > additionally, i've found some problems with "separatist" error > handlers > i.e.: > > (ele:with-transaction () > (handler-case (work-with-database) > (error () (pring "we have a no-go"))) > > if work-with-database produced some error, with-transaction will not > able to > know about this and thus could commit erroneous data. > > of course this is a problem of application, but sometimes such > handlers are > installed by web framework or dictated by architecture in some way. > and it > will be nice to be able to deal with such cases. > > i have some ideas how to handle this: nested with-transaction > instead of > simply calling function (because SQL does not have nested > transactions) > could detect abnormal control flow or conditions signaled and inform > parent > transaction about this -- so it can rollback or retry. but i'm not > going to > implement this right now. > Too much magic and too much trying to do the work for the developers can make the system more complicated and actually make the developer's life harder, because architectural errors are hidden until they become really complicated. I'm hesitant about the above due to this principle. > other extensions in db-postmodern's execute-transaction: > * always-rollback parameter that turns off auto-commit. could be > useful if > you're doing test and do not want results in database.. What would the usage model be? I don't know how that makes sense since it means you always have an implicit, active transaction. The easiest way to do this today is to explicitly start a transaction (controller-start-transaction *sc*) and then do your tests and then call (controller-abort-transaction *sc* txnid) when you're done -- as long as the tests don't blow out your transaction cache. > > * execute-transaction creates restart named > db-postmodern::retry-transaction, so application can > programmatically force > retry. > That's a good idea. I assume this has an implicit abort tied to it, or do you need to abort and then retry? > comments appreciated. > > with best regards, Alex 'killerstorm' Mizrahi. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Thu Feb 21 03:55:52 2008 From: lists at infoway.net (lists at infoway.net) Date: Wed, 20 Feb 2008 22:55:52 -0500 Subject: [elephant-devel] BDB vs postmodern In-Reply-To: <86E12430-DEF0-4975-BE45-05EF5580148E@media.mit.edu> References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> <449E9A14-A4D2-4250-B920-A99DDBE2796D@infoway.net> <86E12430-DEF0-4975-BE45-05EF5580148E@media.mit.edu> Message-ID: Ian, Thanks for the great feedback. I understand your point about the postmodern performance issue. Hopefully that is something that could be overcome in the future. I do happen to prefer BDB for some reason and I like the whole elephant system in general for the reasons you mention and others. However, I'm a bit confused about your comment on BDB vs SQL/ORM performance. I would certainly hope that using Elephant with BDB would be faster, for most practical cases, than SQL/ORM, as long as the application is properly designed to leverage Elephant and BDB's architecture. Now, the part that confused me was that of "complex queries and joins". I was certainly under the impression that properly designing your object model could be more beneficial than the relation model and joins of the SQL world. It might have not been directly mentioned on the Elephant project, but has certainly been mentioned by the folks of AllegroCache and, in essence, the two projects seem to have a lot in common. One of the things that AllegroCache has that I haven't seen in Elephant is the "oid" keyword parameter to many of their functions. As per their documentation: oid - if true return the object id instead of the object. So, this could be used as a way to speed up certain queries in Elephant, such as getting a COUNT without having to generate the entire object. It is true about the architectural performances of Postgres. I'm personally not very familiar with it (I'm more of a MySQL person) but I know that MySQL does have their issues when it comes to clustering, replication, etc. As for the replication and distributed transactions in BDB, well, I don't know. I have a feeling that's more in the BDB side of the world rather than in Elephant, but I'm not very knowledgeable on that either. My impression is that BDB is designed to work in single systems rather than in a distributed environment and it has been the job of the DBMSs to leverage BDB and enhance it with all those other bells and whistles. If that's the case, then your point is correct and I would anticipate there would be an enormous amount of work to be done in Elephant to support that. Maybe one day. Thanks again, Waldo On Feb 20, 2008, at 10:05 PM, Ian Eslick wrote: > I doubt that Elephant on postmodern is going to be faster than using > CL-SQL to do direct ORM against Postgresql. Despite the great work > that Henrik and others have done with Postmodern, there are too many > layers of abstraction in the architecture to overcome. The real > point of using Elephant is ease of use, rapid prototyping and the > programming API, not absolute performance. > > I do suspect that raw performance (using BDB) may be comparable or > better to any SQL/ORM solution for some simple access patterns, but > for complex queries and joins SQL is probably the better way to go. > We have already seen various comments on the list about common > features of SQL databases that Elephant is missing (like getting a > COUNT without generating all the objects). > > > As for using BDB to do replication and distributed transactions - > it's possible but no one has tried it. Elephant would need some > serious modifications as the transaction protocol is different and I > think that you'd have to build your own Global Transaction Manager > either in Lisp or as an additional C daemon. > > It's also possible that there are architectural or performance > issues in using Postgresql across multiple machines of which I am > unaware - hopefully one of the PG experts will comment. > > Ian > > On Feb 20, 2008, at 9:47 PM, lists at infoway.net wrote: > >> Well, this is certainly interesting, since this would allow me to >> decouple the storage system from the lisp environment allowing the >> possibility of setting up a cluster of lisp machines to handle >> application logic. Isn't there a way to achieve this on BDB? >> >> We prefer to deploy our systems on clusters of "inexpensive" >> machines in order to leverage hardware failures and it seems that >> scalability of BDB/Lisp applications is achieved by scaling a >> single machine. >> >> Now, postmodern being 4x slower could be an issue. However, how >> does that compare to a regular CL-SQL and relational queries is a >> different story. If postmodern is about the same speed as hitting >> Postgres with CL-SQL and just using plain SQL instead of elephant, >> at the end of the day, that's the performance our users are getting >> anyway :) >> >> Thanks, >> Waldo >> >>>> >>>> Since Postgres does allow for features such as replication, >>>> clustering, and fail-over with multiple active simultaneous >>>> client connections, does this mean that I could have multiple >>>> (separate) lisp clients using elephant connecting to a separate >>>> Postgres cluster with no concurrency issues? >>> >>> Yes. >>> >>> You can do this on BDB, but only on the same system as it relies >>> on shared memory locks between processes. This helps for multi- >>> CPU systems (one lisp process per CPU) but not for distributing >>> Elephant across >>> >>>> Thanks, >>>> Waldo >>>> _______________________________________________ >>>> elephant-devel site list >>>> elephant-devel at common-lisp.net >>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Thu Feb 21 03:58:39 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 20 Feb 2008 21:58:39 -0600 Subject: [elephant-devel] Fix for Unicode2 serializer In-Reply-To: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> References: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> Message-ID: <1203566319.21443.305.camel@penguin.yourdomain.com> Dear Leslie, Thanks for this patch. However, I wonder if it possible for you to turn whatever made you notice it into a test-case? It is possible that this bug exists on the LISP you are using and not mine, but in any case we should use each such bug as an opportunity to expand the suite of automated tests. As you will see, we have a tiny number of tests around unicode issues, which arose out of my using some Esperanto stuff. However, I am sure our tests are not sufficient. The ideal thing would be to write a test, see that it fails without our patch, and then that it and all the other relevant tests are green with the patch in place. On Wed, 2008-02-20 at 17:36 +0100, Leslie P. Polzer wrote: > I don't know whether it was a change in Elephant, SBCL or even somewhere > else that introduced problems with the Unicode serializer, but the fix > is very small (changed two calls to SCHAR to CHAR) and should probably > be applied ASAP. > > I'm not sure why the code is discerning between STRING and SIMPLE-STRING > anyway. Does anyone know? > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Thu Feb 21 04:39:46 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 20 Feb 2008 23:39:46 -0500 Subject: [elephant-devel] BDB vs postmodern In-Reply-To: References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> <449E9A14-A4D2-4250-B920-A99DDBE2796D@infoway.net> <86E12430-DEF0-4975-BE45-05EF5580148E@media.mit.edu> Message-ID: <1EBF8274-EF85-475D-A884-CA341C26F772@media.mit.edu> On Feb 20, 2008, at 10:55 PM, lists at infoway.net wrote: > Ian, > > Thanks for the great feedback. > > I understand your point about the postmodern performance issue. > Hopefully that is something that could be overcome in the future. > Probably not entirely. Especially if you're talking over a socket. We're implementing BTrees on top of Tables on top of BTrees, you can't get around that overhead. > I do happen to prefer BDB for some reason and I like the whole > elephant system in general for the reasons you mention and others. > Of course! ORM is really annoying for rapid development, especially if you are evolving a running system. > However, I'm a bit confused about your comment on BDB vs SQL/ORM > performance. I would certainly hope that using Elephant with BDB > would be faster, for most practical cases, than SQL/ORM, as long as > the application is properly designed to leverage Elephant and BDB's > architecture. Now, the part that confused me was that of "complex > queries and joins". I was certainly under the impression that > properly designing your object model could be more beneficial than > the relation model and joins of the SQL world. It might have not > been directly mentioned on the Elephant project, but has certainly > been mentioned by the folks of AllegroCache and, in essence, the two > projects seem to have a lot in common. > It depends on your query and how much work you're willing to do on the Elephant side to replicate for your query types all the functionality that you get from a well-implemented SQL query engine. > One of the things that AllegroCache has that I haven't seen in > Elephant is the "oid" keyword parameter to many of their functions. > As per their documentation: oid - if true return the object id > instead of the object. So, this could be used as a way to speed up > certain queries in Elephant, such as getting a COUNT without having > to generate the entire object. > The serializer currently can't do this, but I don't think it's too hard. There are hooks and vaporware plans for an Elephant query infrastructure that would make this easier for most users, but alas time caught up to me before I was able to really push through the bulk of the work. > It is true about the architectural performances of Postgres. I'm > personally not very familiar with it (I'm more of a MySQL person) > but I know that MySQL does have their issues when it comes to > clustering, replication, etc. > Postgresql is great and while MySQL has come a long way in recent years, I still think PG is the superior platform (stability, performance, suitability for complex applications, etc). > As for the replication and distributed transactions in BDB, well, I > don't know. I have a feeling that's more in the BDB side of the > world rather than in Elephant, but I'm not very knowledgeable on > that either. My impression is that BDB is designed to work in single > systems rather than in a distributed environment and it has been the > job of the DBMSs to leverage BDB and enhance it with all those other > bells and whistles. If that's the case, then your point is correct > and I would anticipate there would be an enormous amount of work to > be done in Elephant to support that. Maybe one day. > > Thanks again, > Waldo > > On Feb 20, 2008, at 10:05 PM, Ian Eslick wrote: > >> I doubt that Elephant on postmodern is going to be faster than >> using CL-SQL to do direct ORM against Postgresql. Despite the >> great work that Henrik and others have done with Postmodern, there >> are too many layers of abstraction in the architecture to >> overcome. The real point of using Elephant is ease of use, rapid >> prototyping and the programming API, not absolute performance. >> >> I do suspect that raw performance (using BDB) may be comparable or >> better to any SQL/ORM solution for some simple access patterns, but >> for complex queries and joins SQL is probably the better way to >> go. We have already seen various comments on the list about common >> features of SQL databases that Elephant is missing (like getting a >> COUNT without generating all the objects). >> >> >> As for using BDB to do replication and distributed transactions - >> it's possible but no one has tried it. Elephant would need some >> serious modifications as the transaction protocol is different and >> I think that you'd have to build your own Global Transaction >> Manager either in Lisp or as an additional C daemon. >> >> It's also possible that there are architectural or performance >> issues in using Postgresql across multiple machines of which I am >> unaware - hopefully one of the PG experts will comment. >> >> Ian >> >> On Feb 20, 2008, at 9:47 PM, lists at infoway.net wrote: >> >>> Well, this is certainly interesting, since this would allow me to >>> decouple the storage system from the lisp environment allowing the >>> possibility of setting up a cluster of lisp machines to handle >>> application logic. Isn't there a way to achieve this on BDB? >>> >>> We prefer to deploy our systems on clusters of "inexpensive" >>> machines in order to leverage hardware failures and it seems that >>> scalability of BDB/Lisp applications is achieved by scaling a >>> single machine. >>> >>> Now, postmodern being 4x slower could be an issue. However, how >>> does that compare to a regular CL-SQL and relational queries is a >>> different story. If postmodern is about the same speed as hitting >>> Postgres with CL-SQL and just using plain SQL instead of elephant, >>> at the end of the day, that's the performance our users are >>> getting anyway :) >>> >>> Thanks, >>> Waldo >>> >>>>> >>>>> Since Postgres does allow for features such as replication, >>>>> clustering, and fail-over with multiple active simultaneous >>>>> client connections, does this mean that I could have multiple >>>>> (separate) lisp clients using elephant connecting to a separate >>>>> Postgres cluster with no concurrency issues? >>>> >>>> Yes. >>>> >>>> You can do this on BDB, but only on the same system as it relies >>>> on shared memory locks between processes. This helps for multi- >>>> CPU systems (one lisp process per CPU) but not for distributing >>>> Elephant across >>>> >>>>> Thanks, >>>>> Waldo >>>>> _______________________________________________ >>>>> elephant-devel site list >>>>> elephant-devel at common-lisp.net >>>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>>> >>>> _______________________________________________ >>>> elephant-devel site list >>>> elephant-devel at common-lisp.net >>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Thu Feb 21 05:22:17 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 21 Feb 2008 00:22:17 -0500 Subject: [elephant-devel] Elephant BDB Tests Fail on OS X Leopard, SBCL 1.0.14 In-Reply-To: <3D3FE2FF-EB06-4E45-ACEB-2FFFBE68035A@infoway.net> References: <3D3FE2FF-EB06-4E45-ACEB-2FFFBE68035A@infoway.net> Message-ID: <317AC4EB-C31F-4805-890C-D8399CCB4606@media.mit.edu> Hmmm...I can't reproduce this. Anyone else? Did you clean the tests/testdb directory before running the test? While we try to keep the tests idempotent, sometimes a pre-existing database (esp. from a prior version of the code) can create problems. Ian On Feb 20, 2008, at 8:29 PM, lists at infoway.net wrote: > Hi all, > > I downloaded the latest elephant this morning and ran the BDB tests, > one of which failed. I haven't really looked much into it because > I'm traveling. However, just wanted to share this with you in case > you were aware (or not) and I might have missed something. > > Thanks, > Waldo > > **** SBCL: > SBCL was compiled from source enabling threads. > This is SBCL 1.0.14, an implementation of ANSI Common Lisp. > > **** SLIME: > ELE-TESTS> (do-backend-tests) > Attempting to load libmemutil.dylib... > Loaded /Users/waldo/dev/lisp/elephant/src/memutil/libmemutil.dylib > STYLE-WARNING: redefining COPY-BUFS in DEFUN > STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining READ-INT32 in DEFUN > STYLE-WARNING: redefining READ-INT64 in DEFUN > STYLE-WARNING: redefining READ-UINT32 in DEFUN > STYLE-WARNING: redefining READ-UINT64 in DEFUN > STYLE-WARNING: redefining READ-FLOAT in DEFUN > STYLE-WARNING: redefining READ-DOUBLE in DEFUN > STYLE-WARNING: redefining WRITE-INT32 in DEFUN > STYLE-WARNING: redefining WRITE-INT64 in DEFUN > STYLE-WARNING: redefining WRITE-UINT32 in DEFUN > STYLE-WARNING: redefining WRITE-UINT64 in DEFUN > STYLE-WARNING: redefining WRITE-FLOAT in DEFUN > STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN > STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN > STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN > STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN > STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN > STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN > STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN > STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN > STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN > STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN > STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN > STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN > STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN > STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN > STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN > STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN > STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN > STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN > STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN > ; compiling file "/Users/waldo/dev/lisp/elephant/tests/ > testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): > ; compiling (IN-PACKAGE :ELE-TESTS) > ; compiling (DEFVAR ENV) > ; compiling (DEFVAR DB) > ; compiling (DEFUN PREPARE-BDB ...) > ; compiling (DEFTEST PREPARES-BDB ...) > ; compiling (DEFUN TEST-SEQUENCE1 ...) > ; compiling (DEFTEST TEST-SEQ1 ...) > ; compiling (DEFUN TEST-SEQUENCE2 ...) > ; compiling (DEFTEST TEST-SEQ2 ...) > ; compiling (DEFUN CLEANUP-BDB ...) > ; compiling (DEFTEST CLEANSUP-BDB ...) > > ; /Users/waldo/dev/lisp/elephant/tests/testbdb.fasl written > ; compilation finished in 0:00:00 > Doing 132 pending tests of 132 tests total. > FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM > WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS > COMPLEXES > BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES > HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP > TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- > CONSTRUCT > CIRCULAR PERSISTENT; in: LAMBDA NIL > ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION > UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 > NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS > BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS > INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM > REDEFCLASS > MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE > TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE > INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET > SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 > REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 > NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES-SLOT2 > MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET-RANGE > SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV > MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM-INDEX3 > DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST CUR- > DEL1 > INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 > GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 > ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET > DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC > INDEXING-CLASS-OPT INDEXING-INHERIT > Test INDEXING-RANGE failed > Form: (PROGN > (DEFCLASS IDX-FOUR NIL > ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 > :INDEX T)) > (:METACLASS PERSISTENT-METACLASS)) > (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) > (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) > (DEFCLASS IDX-FOUR NIL > ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 > :INDEX T)) > (:METACLASS PERSISTENT-METACLASS)) > (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 VAL)) > (WITH-TRANSACTION NIL > (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 5 6 > 10))) > (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) > (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) > (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) > (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) > (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) > (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) > Expected values: T > T > T > Actual value: #. > INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB > INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS > Ranged get of 10/700 objects = Linear: 0.435 sec Indexed: 0.01 sec > INDEXING-TIMING > ; in: LAMBDA NIL > ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring REMOVE-ELEMENT > ; in: LAMBDA NIL > ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-BASIC > ; in: LAMBDA NIL > ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-BTREE > ; in: LAMBDA NIL > ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-IDX-BTREE > ; in: LAMBDA NIL > ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-PCLASS > ; in: LAMBDA NIL > ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-MULT-PCLASS > ; in: LAMBDA NIL > ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > ; ==> > ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > ; > ; caught WARNING: > ; undefined variable: *TEST-SPEC-SECONDARY* > > ; > ; caught WARNING: > ; This variable is undefined: > ; *TEST-SPEC-SECONDARY* > ; > ; compilation unit finished > ; caught 2 WARNING conditions > > Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 > TEST-SEQ2 CLEANSUP-BDB > 1 out of 132 total tests failed: INDEXING-RANGE.NIL > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Thu Feb 21 11:12:03 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 21 Feb 2008 13:12:03 +0200 Subject: [elephant-devel] Re: BDB vs postmodern References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> Message-ID: > So, putting licensing issues aside, what is the real difference/ advantage > of one data store over the other? In a recent email I read by Alex, he > mentions he's going to work on improving performance on the postmodern > data store. So, even after that improves and performance is matched with > that of BDB, is there an advantage of one data store over the other? matching performance of BDB is not in postmodern backend design goals, and I don't think it's possible. having storage engine separate from application introduces significant communtication overhead -- with each query taking 0.1-1 ms, we simply cannot do more than 1000-10000 queries per second from a single client. elephant reads/writes slots individually, so this means we cannot have more than 1000-10000 slots operations per second. that sounds quite prohibitive. lR> Since Postgres does allow for features such as replication, lR> clustering, to my knowledge it doesn't work good enough, so probably you'll be limited to having only one active DB machine lR> and fail-over with multiple active simultaneous client connections, lR> does this mean that I could have multiple (separate) lisp clients using lR> elephant connecting to a separate Postgres cluster with no concurrency lR> issues? yes, that is an idea behind postmodern backend -- it was made to allow scaling to multiple client, multiple machines.. as i understand, at the time db-postmodern was created, other backends did not work correctly even if multiple processes on same machine connect to single DB. at least, in grand-prix test suite (that tries to work with DB in two processes simultaneously) there is a remark: "Currently (jan 07) is expected to fail." for elephant/bdb engine. besides multiple clients, there are some other benefits of using PostgreSQL as a backend -- for example, it doesn't suffer from locking issues due to MVCC approach: readers do not block writers, and it never says out-of-memory when you read too much. however, we've found scalability problems mentioned above -- Lisp application using persistent objects as if they were local does too many slot accesses, reading same data many times etc. the solution seems to be trivial: implement caching on a client side. simple solution is to cache data within single transaction. complex solution is to cache data accross transaction, tracking changes and invalidating stale cache entries automatically. we've implemented both options. so now it's possible to have all slots cached on a Lisp side. as long as there are much more reads than writes (which is common for web applications, for example), strain on a database is significantly decreased. all index queries (select object by value or by range) still work via database, but typically there's much more slot queries than index queries. but now we have performance not directly comparable with BDB -- it depends on usage patterns now. for one kind of applications caching might work very well and we'll get performance that is better than BDB's one. on other kind of applications caching could be just additional overhead, and postmodern's performance will suck. also keep in mind that indeed postmodern backend is "younger" and it can still have some glitches. From henrik at evahjelte.com Thu Feb 21 12:44:00 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 21 Feb 2008 13:44:00 +0100 Subject: [elephant-devel] BDB vs postmodern In-Reply-To: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> Message-ID: <50e8e4f60802210444g56290e54g9b3be7d92aeee3b3@mail.gmail.com> On Thu, Feb 21, 2008 at 2:44 AM, lists at infoway.net wrote: > Hi all, > > I don't mean to start a war here or put any work down. However, I just > needed some clarification/direction into which way the data stores > work is going. > Since Postgres does allow for features such as replication, > clustering, and fail-over with multiple active simultaneous client > connections, does this mean that I could have multiple (separate) lisp > clients using elephant connecting to a separate Postgres cluster with > no concurrency issues? It is going in the scalable direction I hope. Since we have a web startup in the consumer-segment, we will hopefully need a scalable and fast solution. If the postmodern backend can't take it, we can make a new backend, perhaps made in Lisp all the way. If elephant itself can't take it we will need a new persistence solution, but then we will provide a migration opportunity. The postmodern backend is optimized for use from several processes, that might be a cluster of several web servers connected to a singe database engine. If you want to do this you have to choose between sql, perec or elephant-sql or elephant-postmodern. I experienced performance problems with AllegroCache for this usecase, but Franz said that they were going to fix it, so it probably works as well. When elephant-postmodern is not scalable enough, it is probably possible to implement a solution that works with clusters of database servers as well. I have briefly looked into it at some time. The little tricky part is that the postmodern backend creates a new table for each btree, so the database schema evolves. Some of the replication solutions I looked at had a slight problem with that. But, since we have control of when a new table is created, we have a spot to trigger schema updates across the cluster, so it is doable. About performance measurements, since there is no good "real world" testcase of performance it is difficult to say. Simply counting the time for the testsuite surely does not measure the effects of caching. But for a singe user app BDB wins easily. > Now, the part that confused me was that of "complex > queries and joins". I was certainly under the impression that properly > designing your object model could be more beneficial than the relation > model and joins of the SQL world. It might have not been directly > mentioned on the Elephant project, but has certainly been mentioned by > the folks of AllegroCache and, in essence, the two projects seem to > have a lot in common. It is a difficult question to say. It depends on the use-case. One thing that is good about the sql model is that the query engine can optimize the query for you, and it might know more of the runtime environment than you do as a programmer when hand-optimizing the queries. And in the usual implementation you save roundtrips of data between server and client. On the other hand, direct pointer access through an object database might give great performance in some cases. I am sometimes thinking of ways to marry the two approaches, create a relational query api to an object oriented database. Or rather make a new type of relational database embedded in Lisp, with classes as a datatype (domain), and being able to call methods on the objects in queries. This is inspired by the writings of C.J Date. This model might even be more true to the mathematical relational model than current SQL databases. But that is something for the future, and to make it you would probably build it on top of a persistence layer like elephant. > One of the things that AllegroCache has that I haven't seen in > Elephant is the "oid" keyword parameter to many of their functions. As > per their documentation: oid - if true return the object id instead of > the object. So, this could be used as a way to speed up certain > queries in Elephant, such as getting a COUNT without having to > generate the entire object. Yes, it would be particularly useful for doing homemade joins. If you could have a more advanced query-api doing joins on the server, that would save even more roundtrips. /Henrik From eslick at media.mit.edu Thu Feb 21 12:52:59 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 21 Feb 2008 07:52:59 -0500 Subject: [elephant-devel] Re: BDB vs postmodern In-Reply-To: References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> Message-ID: > the solution seems to be trivial: implement caching on a client > side. simple > solution is to cache data within single transaction. complex > solution is to > cache data accross transaction, tracking changes and invalidating > stale > cache entries automatically. > > we've implemented both options. so now it's possible to have all slots > cached on a Lisp side. as long as there are much more reads than > writes > (which is common for web applications, for example), strain on a > database is > significantly decreased. all index queries (select object by value > or by > range) still work via database, but typically there's much more slot > queries > than index queries. > Is this caching something that we could import into BDB? There are certainly occasions for caching slot values as well as object references. Robert and I once talked about adding a slot policy for write-through caching but never got around to it... Ian From killerstorm at newmail.ru Thu Feb 21 13:08:02 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 21 Feb 2008 15:08:02 +0200 Subject: [elephant-devel] Re: BDB vs postmodern References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> Message-ID: IE> Is this caching something that we could import into BDB? it is pretty easy to add caching inside one transaction or accross transactions assuming there is only one instance (and one thread) working with database. we can have this part implemented in the core and shared among backends.. cache that works accross transactions and supports multiple concurrent instances connected requires tracking changes (for examle, on btree level) and storing them in the database -- that requires considerable amount of work and is backend-specific. given that it should have much less effect on BDB, probably it's not worth implementing. From leslie.polzer at gmx.net Thu Feb 21 13:41:24 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Thu, 21 Feb 2008 14:41:24 +0100 (CET) Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE Message-ID: <62536.88.73.249.123.1203601284.squirrel@mail.stardawn.org> I'm not sure I understood your message correctly. But see below. > Ok, this is an architectural issue that wasn't completely thought > out. Currently class indexing effectively limits instances of a class > to a specific store. The first time an instance is saved, the current > *store-controller* is used to create the class index tying the class > to that store. I do not have a problem with that. Actually, it's desired behaviour since the indexed objects of a certain class need to belong to two different stores in my setup. > > I have a local patch which changes this behavior to create a class > index for each store's objects. For example, in your test code, if > you call get-instances-by-class in the context of *sc1*, you'll get > nothing, in the context of *sc2* you'll get the instance you created > in *sc2*. Each store has it's own local set of class indices. Isn't this the current behaviour? > I'm concerned there are other areas where the policy choice becomes > difficult. My inclination is to assert that if you want to index > persistent objects in multiple stores, you should plan for that > explicitly and not use the class indexing mechanism. I still don't understand why there's a problem with having SETF SLOT-VALUE figure out the correct controller... Leslie From desoi at pgedit.com Thu Feb 21 14:35:04 2008 From: desoi at pgedit.com (John DeSoi) Date: Thu, 21 Feb 2008 09:35:04 -0500 Subject: [elephant-devel] Re: BDB vs postmodern In-Reply-To: References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> Message-ID: <34409281-2B79-483A-992A-AE65B67C7594@pgedit.com> On Feb 21, 2008, at 6:12 AM, Alex Mizrahi wrote: > elephant reads/writes slots individually, so this means we cannot > have more > than 1000-10000 slots operations per second. that sounds quite > prohibitive. So with an SQL engine this translates to one SELECT/INSERT/UPDATE per slot value read or write? Are you using transactions and prepared queries with PostgreSQL? That might help some, but as you say it probably won't scale up very well. John DeSoi, Ph.D. From henrik at evahjelte.com Thu Feb 21 14:50:49 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 21 Feb 2008 15:50:49 +0100 Subject: [elephant-devel] Re: BDB vs postmodern In-Reply-To: <34409281-2B79-483A-992A-AE65B67C7594@pgedit.com> References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> <34409281-2B79-483A-992A-AE65B67C7594@pgedit.com> Message-ID: <50e8e4f60802210650n28bdab1btb1c148bd53f844c5@mail.gmail.com> On Thu, Feb 21, 2008 at 3:35 PM, John DeSoi wrote: > > On Feb 21, 2008, at 6:12 AM, Alex Mizrahi wrote: > > > elephant reads/writes slots individually, so this means we cannot > > have more > > than 1000-10000 slots operations per second. that sounds quite > > prohibitive. > > So with an SQL engine this translates to one SELECT/INSERT/UPDATE per > slot value read or write? Are you using transactions and prepared > queries with PostgreSQL? That might help some, but as you say it > probably won't scale up very well We use prepared queries for select and a stored procedure for inserts and updates, within transactions. Tables are indexed. As explained, the bottleneck is probably communication overhead and context switching rather than database performance. /Henrik From eslick at media.mit.edu Thu Feb 21 15:29:06 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 21 Feb 2008 10:29:06 -0500 Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE In-Reply-To: <62536.88.73.249.123.1203601284.squirrel@mail.stardawn.org> References: <62536.88.73.249.123.1203601284.squirrel@mail.stardawn.org> Message-ID: <5DF3719A-FB72-4201-83C2-B995F53D2905@media.mit.edu> On Feb 21, 2008, at 8:41 AM, Leslie P. Polzer wrote: > > I'm not sure I understood your message correctly. But see below. > >> Ok, this is an architectural issue that wasn't completely thought >> out. Currently class indexing effectively limits instances of a >> class >> to a specific store. The first time an instance is saved, the >> current >> *store-controller* is used to create the class index tying the class >> to that store. > > I do not have a problem with that. > Actually, it's desired behaviour since the indexed objects of a > certain > class need to belong to two different stores in my setup. Do you need to do indexing operations across stores? i.e. run a cursor over all objects regardless of which store they are in? Ideally, what semantics would you like to see? > > >> >> I have a local patch which changes this behavior to create a class >> index for each store's objects. For example, in your test code, if >> you call get-instances-by-class in the context of *sc1*, you'll get >> nothing, in the context of *sc2* you'll get the instance you created >> in *sc2*. Each store has it's own local set of class indices. > > Isn't this the current behaviour? > Nope, the system only creates one class index independent of which store controller it's in, so you can only map indexed class instances to one store. > >> I'm concerned there are other areas where the policy choice becomes >> difficult. My inclination is to assert that if you want to index >> persistent objects in multiple stores, you should plan for that >> explicitly and not use the class indexing mechanism. > > I still don't understand why there's a problem with having SETF SLOT- > VALUE > figure out the correct controller... > There isn't a problem doing this. My concern is that this then creates the potential for other, less visible problems. e.g. you create indexed instances in store 1, the class index associated with store one indexes all those instances. When you call get-instances-by-class in the context of store 2, however, all the objects from store 1 are now missing. The semantics of get-instance- by-class/value/range are all incorrect when you have instances stored in multiple store controllers. Using multiple stores starts to break the model of a simple global namespace assumed by the current class indexing mechanism. Rather than contort the class indexing mechanism to accommodate multiple store operations, my preference is that users write their own indexing mechanism that has the semantics they care about. > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Thu Feb 21 16:11:12 2008 From: lists at infoway.net (lists at infoway.net) Date: Thu, 21 Feb 2008 11:11:12 -0500 Subject: [elephant-devel] Elephant BDB Tests Fail on OS X Leopard, SBCL 1.0.14 In-Reply-To: <317AC4EB-C31F-4805-890C-D8399CCB4606@media.mit.edu> References: <3D3FE2FF-EB06-4E45-ACEB-2FFFBE68035A@infoway.net> <317AC4EB-C31F-4805-890C-D8399CCB4606@media.mit.edu> Message-ID: <17ADEBAC-DD92-47A0-B71C-B102CBBDAA12@infoway.net> Hi Ian, I erased the entire elephant directory and re-downloaded the CSV repository again just now. The contents of the tests/testdb were just the README file and the CVS directory. I ran the tests and got the exact same result. I then erased the tests/testdb and then re-created a new blank one. Ran the tests and got the same results :( Thanks, - Waldo On Feb 21, 2008, at 12:22 AM, Ian Eslick wrote: > Hmmm...I can't reproduce this. Anyone else? > > Did you clean the tests/testdb directory before running the test? > While we try to keep the tests idempotent, sometimes a pre-existing > database (esp. from a prior version of the code) can create problems. > > Ian > > On Feb 20, 2008, at 8:29 PM, lists at infoway.net wrote: > >> Hi all, >> >> I downloaded the latest elephant this morning and ran the BDB >> tests, one of which failed. I haven't really looked much into it >> because I'm traveling. However, just wanted to share this with you >> in case you were aware (or not) and I might have missed something. >> >> Thanks, >> Waldo >> >> **** SBCL: >> SBCL was compiled from source enabling threads. >> This is SBCL 1.0.14, an implementation of ANSI Common Lisp. >> >> **** SLIME: >> ELE-TESTS> (do-backend-tests) >> Attempting to load libmemutil.dylib... >> Loaded /Users/waldo/dev/lisp/elephant/src/memutil/libmemutil.dylib >> STYLE-WARNING: redefining COPY-BUFS in DEFUN >> STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining READ-INT32 in DEFUN >> STYLE-WARNING: redefining READ-INT64 in DEFUN >> STYLE-WARNING: redefining READ-UINT32 in DEFUN >> STYLE-WARNING: redefining READ-UINT64 in DEFUN >> STYLE-WARNING: redefining READ-FLOAT in DEFUN >> STYLE-WARNING: redefining READ-DOUBLE in DEFUN >> STYLE-WARNING: redefining WRITE-INT32 in DEFUN >> STYLE-WARNING: redefining WRITE-INT64 in DEFUN >> STYLE-WARNING: redefining WRITE-UINT32 in DEFUN >> STYLE-WARNING: redefining WRITE-UINT64 in DEFUN >> STYLE-WARNING: redefining WRITE-FLOAT in DEFUN >> STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN >> STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN >> STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN >> STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN >> STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN >> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN >> STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN >> STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN >> STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN >> STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN >> ; compiling file "/Users/waldo/dev/lisp/elephant/tests/ >> testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): >> ; compiling (IN-PACKAGE :ELE-TESTS) >> ; compiling (DEFVAR ENV) >> ; compiling (DEFVAR DB) >> ; compiling (DEFUN PREPARE-BDB ...) >> ; compiling (DEFTEST PREPARES-BDB ...) >> ; compiling (DEFUN TEST-SEQUENCE1 ...) >> ; compiling (DEFTEST TEST-SEQ1 ...) >> ; compiling (DEFUN TEST-SEQUENCE2 ...) >> ; compiling (DEFTEST TEST-SEQ2 ...) >> ; compiling (DEFUN CLEANUP-BDB ...) >> ; compiling (DEFTEST CLEANSUP-BDB ...) >> >> ; /Users/waldo/dev/lisp/elephant/tests/testbdb.fasl written >> ; compilation finished in 0:00:00 >> Doing 132 pending tests of 132 tests total. >> FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM >> WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS >> COMPLEXES >> BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES >> HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP >> TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- >> CONSTRUCT >> CIRCULAR PERSISTENT; in: LAMBDA NIL >> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION >> UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 >> NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS >> BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS >> INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM >> REDEFCLASS >> MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE >> TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE >> INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET >> SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 >> REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 >> NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES- >> SLOT2 >> MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET- >> RANGE >> SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV >> MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM-INDEX3 >> DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST >> CUR-DEL1 >> INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 >> GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 >> ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET >> DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC >> INDEXING-CLASS-OPT INDEXING-INHERIT >> Test INDEXING-RANGE failed >> Form: (PROGN >> (DEFCLASS IDX-FOUR NIL >> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 >> :INDEX T)) >> (:METACLASS PERSISTENT-METACLASS)) >> (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) >> (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) >> (DEFCLASS IDX-FOUR NIL >> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 >> :INDEX T)) >> (:METACLASS PERSISTENT-METACLASS)) >> (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 VAL)) >> (WITH-TRANSACTION NIL >> (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 5 6 >> 10))) >> (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) >> (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) >> (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) >> (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) >> (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) >> (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) >> Expected values: T >> T >> T >> Actual value: #. >> INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB >> INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS >> Ranged get of 10/700 objects = Linear: 0.435 sec Indexed: 0.01 sec >> INDEXING-TIMING >> ; in: LAMBDA NIL >> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring REMOVE-ELEMENT >> ; in: LAMBDA NIL >> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-BASIC >> ; in: LAMBDA NIL >> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-BTREE >> ; in: LAMBDA NIL >> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-IDX-BTREE >> ; in: LAMBDA NIL >> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-PCLASS >> ; in: LAMBDA NIL >> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-MULT-PCLASS >> ; in: LAMBDA NIL >> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >> ; ==> >> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >> ; >> ; caught WARNING: >> ; undefined variable: *TEST-SPEC-SECONDARY* >> >> ; >> ; caught WARNING: >> ; This variable is undefined: >> ; *TEST-SPEC-SECONDARY* >> ; >> ; compilation unit finished >> ; caught 2 WARNING conditions >> >> Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 >> TEST-SEQ2 CLEANSUP-BDB >> 1 out of 132 total tests failed: INDEXING-RANGE.NIL >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Thu Feb 21 16:15:48 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 21 Feb 2008 11:15:48 -0500 Subject: [elephant-devel] Elephant BDB Tests Fail on OS X Leopard, SBCL 1.0.14 In-Reply-To: <17ADEBAC-DD92-47A0-B71C-B102CBBDAA12@infoway.net> References: <3D3FE2FF-EB06-4E45-ACEB-2FFFBE68035A@infoway.net> <317AC4EB-C31F-4805-890C-D8399CCB4606@media.mit.edu> <17ADEBAC-DD92-47A0-B71C-B102CBBDAA12@infoway.net> Message-ID: Robert, Can you see if you can reproduce this. I can't get back to it now for a couple of days. Waldo, if you don't hear anything from us in a few days just ping us again. Thanks, Ian On Feb 21, 2008, at 11:11 AM, lists at infoway.net wrote: > Hi Ian, > > I erased the entire elephant directory and re-downloaded the CSV > repository again just now. The contents of the tests/testdb were > just the README file and the CVS directory. I ran the tests and got > the exact same result. > > I then erased the tests/testdb and then re-created a new blank one. > Ran the tests and got the same results :( > > Thanks, > - Waldo > > On Feb 21, 2008, at 12:22 AM, Ian Eslick wrote: > >> Hmmm...I can't reproduce this. Anyone else? >> >> Did you clean the tests/testdb directory before running the test? >> While we try to keep the tests idempotent, sometimes a pre-existing >> database (esp. from a prior version of the code) can create problems. >> >> Ian >> >> On Feb 20, 2008, at 8:29 PM, lists at infoway.net wrote: >> >>> Hi all, >>> >>> I downloaded the latest elephant this morning and ran the BDB >>> tests, one of which failed. I haven't really looked much into it >>> because I'm traveling. However, just wanted to share this with you >>> in case you were aware (or not) and I might have missed something. >>> >>> Thanks, >>> Waldo >>> >>> **** SBCL: >>> SBCL was compiled from source enabling threads. >>> This is SBCL 1.0.14, an implementation of ANSI Common Lisp. >>> >>> **** SLIME: >>> ELE-TESTS> (do-backend-tests) >>> Attempting to load libmemutil.dylib... >>> Loaded /Users/waldo/dev/lisp/elephant/src/memutil/libmemutil.dylib >>> STYLE-WARNING: redefining COPY-BUFS in DEFUN >>> STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining READ-INT32 in DEFUN >>> STYLE-WARNING: redefining READ-INT64 in DEFUN >>> STYLE-WARNING: redefining READ-UINT32 in DEFUN >>> STYLE-WARNING: redefining READ-UINT64 in DEFUN >>> STYLE-WARNING: redefining READ-FLOAT in DEFUN >>> STYLE-WARNING: redefining READ-DOUBLE in DEFUN >>> STYLE-WARNING: redefining WRITE-INT32 in DEFUN >>> STYLE-WARNING: redefining WRITE-INT64 in DEFUN >>> STYLE-WARNING: redefining WRITE-UINT32 in DEFUN >>> STYLE-WARNING: redefining WRITE-UINT64 in DEFUN >>> STYLE-WARNING: redefining WRITE-FLOAT in DEFUN >>> STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN >>> STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN >>> STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN >>> STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN >>> STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN >>> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN >>> STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN >>> STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN >>> STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN >>> ; compiling file "/Users/waldo/dev/lisp/elephant/tests/ >>> testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): >>> ; compiling (IN-PACKAGE :ELE-TESTS) >>> ; compiling (DEFVAR ENV) >>> ; compiling (DEFVAR DB) >>> ; compiling (DEFUN PREPARE-BDB ...) >>> ; compiling (DEFTEST PREPARES-BDB ...) >>> ; compiling (DEFUN TEST-SEQUENCE1 ...) >>> ; compiling (DEFTEST TEST-SEQ1 ...) >>> ; compiling (DEFUN TEST-SEQUENCE2 ...) >>> ; compiling (DEFTEST TEST-SEQ2 ...) >>> ; compiling (DEFUN CLEANUP-BDB ...) >>> ; compiling (DEFTEST CLEANSUP-BDB ...) >>> >>> ; /Users/waldo/dev/lisp/elephant/tests/testbdb.fasl written >>> ; compilation finished in 0:00:00 >>> Doing 132 pending tests of 132 tests total. >>> FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM >>> WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS >>> COMPLEXES >>> BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES >>> HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP >>> TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- >>> CONSTRUCT >>> CIRCULAR PERSISTENT; in: LAMBDA NIL >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION >>> UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 >>> NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS >>> BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS >>> INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM >>> REDEFCLASS >>> MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE >>> TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE >>> INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET >>> SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 >>> REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 >>> NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES- >>> SLOT2 >>> MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET- >>> RANGE >>> SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV >>> MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM-INDEX3 >>> DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST >>> CUR-DEL1 >>> INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 >>> GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 >>> ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET >>> DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC >>> INDEXING-CLASS-OPT INDEXING-INHERIT >>> Test INDEXING-RANGE failed >>> Form: (PROGN >>> (DEFCLASS IDX-FOUR NIL >>> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 >>> :INDEX T)) >>> (:METACLASS PERSISTENT-METACLASS)) >>> (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) >>> (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) >>> (DEFCLASS IDX-FOUR NIL >>> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 >>> :INDEX T)) >>> (:METACLASS PERSISTENT-METACLASS)) >>> (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 VAL)) >>> (WITH-TRANSACTION NIL >>> (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 5 6 >>> 10))) >>> (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) >>> (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) >>> (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) >>> (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) >>> (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) >>> (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) >>> Expected values: T >>> T >>> T >>> Actual value: #. >>> INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB >>> INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS >>> Ranged get of 10/700 objects = Linear: 0.435 sec Indexed: 0.01 sec >>> INDEXING-TIMING >>> ; in: LAMBDA NIL >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring REMOVE-ELEMENT >>> ; in: LAMBDA NIL >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-BASIC >>> ; in: LAMBDA NIL >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-BTREE >>> ; in: LAMBDA NIL >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-IDX-BTREE >>> ; in: LAMBDA NIL >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-PCLASS >>> ; in: LAMBDA NIL >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-MULT-PCLASS >>> ; in: LAMBDA NIL >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) >>> ; ==> >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) >>> ; >>> ; caught WARNING: >>> ; undefined variable: *TEST-SPEC-SECONDARY* >>> >>> ; >>> ; caught WARNING: >>> ; This variable is undefined: >>> ; *TEST-SPEC-SECONDARY* >>> ; >>> ; compilation unit finished >>> ; caught 2 WARNING conditions >>> >>> Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 >>> TEST-SEQ2 CLEANSUP-BDB >>> 1 out of 132 total tests failed: INDEXING-RANGE.NIL >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Thu Feb 21 19:29:14 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 21 Feb 2008 21:29:14 +0200 Subject: [elephant-devel] Re: BDB vs postmodern References: <2CBADED6-F5F1-4FCC-91CE-8FAE08E59663@infoway.net> <34409281-2B79-483A-992A-AE65B67C7594@pgedit.com> Message-ID: ??>> elephant reads/writes slots individually, so this means we cannot ??>> have more ??>> than 1000-10000 slots operations per second. that sounds quite ??>> prohibitive. JD> So with an SQL engine this translates to one SELECT/INSERT/UPDATE per JD> slot value read or write? something like that JD> Are you using transactions and prepared queries with PostgreSQL? of course. i've done some benchmarking on socket level -- simple [perl] client that connects to local echo service and repeatedly sends 10 bytes to socket and reads them back (simulating request/response). on Pentium D 3Ghz (dual core) it was able to do 30000 requests per second. on AMD Athlon 2Ghz x64 dual core -- 15000 requests per seconds. this effectively limits of what can be done via sockets, no matter how efficient protocol and code will be. of course in real world we need to spend some time processing data. on my machine PostgreSQL was showing times like 0.1 ms for simple selects, so it's unlikely it can do more than 10000 requests. i didn't benchmark BDB myself, but here's a document from Oracle: http://www.oracle.com/technology/products/berkeley-db/pdf/berkeley-db-perf.pdf they mention performance like 100,000-1,000,000 operations per second on commodity hardware, that's orders of magnitude faster than what we can achieve via sockets. reasons for such difference in performance are also mentioned in this document: "Client/Server architecture increases complexity and slows down performance because applications must cross a process boundary and often a network link to access data." that's what i was saying.. From read at robertlread.net Thu Feb 21 22:38:16 2008 From: read at robertlread.net (Robert L. Read) Date: Thu, 21 Feb 2008 16:38:16 -0600 Subject: [elephant-devel] Elephant BDB Tests Fail on OS X Leopard, SBCL 1.0.14 In-Reply-To: References: <3D3FE2FF-EB06-4E45-ACEB-2FFFBE68035A@infoway.net> <317AC4EB-C31F-4805-890C-D8399CCB4606@media.mit.edu> <17ADEBAC-DD92-47A0-B71C-B102CBBDAA12@infoway.net> Message-ID: <1203633496.21443.344.camel@penguin.yourdomain.com> No, I can't test this, because I don't have BDB installed on my x86_64. On Thu, 2008-02-21 at 11:15 -0500, Ian Eslick wrote: > Robert, > > Can you see if you can reproduce this. I can't get back to it now for > a couple of days. > > Waldo, if you don't hear anything from us in a few days just ping us > again. > > Thanks, > Ian > > On Feb 21, 2008, at 11:11 AM, lists at infoway.net wrote: > > > Hi Ian, > > > > I erased the entire elephant directory and re-downloaded the CSV > > repository again just now. The contents of the tests/testdb were > > just the README file and the CVS directory. I ran the tests and got > > the exact same result. > > > > I then erased the tests/testdb and then re-created a new blank one. > > Ran the tests and got the same results :( > > > > Thanks, > > - Waldo > > > > On Feb 21, 2008, at 12:22 AM, Ian Eslick wrote: > > > >> Hmmm...I can't reproduce this. Anyone else? > >> > >> Did you clean the tests/testdb directory before running the test? > >> While we try to keep the tests idempotent, sometimes a pre-existing > >> database (esp. from a prior version of the code) can create problems. > >> > >> Ian > >> > >> On Feb 20, 2008, at 8:29 PM, lists at infoway.net wrote: > >> > >>> Hi all, > >>> > >>> I downloaded the latest elephant this morning and ran the BDB > >>> tests, one of which failed. I haven't really looked much into it > >>> because I'm traveling. However, just wanted to share this with you > >>> in case you were aware (or not) and I might have missed something. > >>> > >>> Thanks, > >>> Waldo > >>> > >>> **** SBCL: > >>> SBCL was compiled from source enabling threads. > >>> This is SBCL 1.0.14, an implementation of ANSI Common Lisp. > >>> > >>> **** SLIME: > >>> ELE-TESTS> (do-backend-tests) > >>> Attempting to load libmemutil.dylib... > >>> Loaded /Users/waldo/dev/lisp/elephant/src/memutil/libmemutil.dylib > >>> STYLE-WARNING: redefining COPY-BUFS in DEFUN > >>> STYLE-WARNING: redefining MAKE-BUFFER-STREAM in DEFUN > >>> STYLE-WARNING: redefining GRAB-BUFFER-STREAM in DEFUN > >>> STYLE-WARNING: redefining RETURN-BUFFER-STREAM in DEFUN > >>> STYLE-WARNING: redefining READ-INT32 in DEFUN > >>> STYLE-WARNING: redefining READ-INT64 in DEFUN > >>> STYLE-WARNING: redefining READ-UINT32 in DEFUN > >>> STYLE-WARNING: redefining READ-UINT64 in DEFUN > >>> STYLE-WARNING: redefining READ-FLOAT in DEFUN > >>> STYLE-WARNING: redefining READ-DOUBLE in DEFUN > >>> STYLE-WARNING: redefining WRITE-INT32 in DEFUN > >>> STYLE-WARNING: redefining WRITE-INT64 in DEFUN > >>> STYLE-WARNING: redefining WRITE-UINT32 in DEFUN > >>> STYLE-WARNING: redefining WRITE-UINT64 in DEFUN > >>> STYLE-WARNING: redefining WRITE-FLOAT in DEFUN > >>> STYLE-WARNING: redefining WRITE-DOUBLE in DEFUN > >>> STYLE-WARNING: redefining OFFSET-CHAR-POINTER in DEFUN > >>> STYLE-WARNING: redefining %COPY-STR-TO-BUF in DEFUN > >>> STYLE-WARNING: redefining COPY-STR-TO-BUF in DEFUN > >>> STYLE-WARNING: redefining PROCESS-STRUCT-SLOT-DEFS in DEFUN > >>> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM in DEFUN > >>> STYLE-WARNING: redefining RESIZE-BUFFER-STREAM-NO-COPY in DEFUN > >>> STYLE-WARNING: redefining RESET-BUFFER-STREAM in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-BYTE in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-INT32 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT32 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-INT64 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT64 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-FLOAT in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-DOUBLE in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-STRING in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-BYTE in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-BYTE-VECTOR in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-BYTE-VECTOR in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-TO-ARRAY-OFFSET in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-FROM-ARRAY-OFFSET in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-OID in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-OID in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-INT in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-INT in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-UINT in DEFUN > >>> STYLE-WARNING: redefining BUFFER-WRITE-UINT in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM32 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-INT32 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-UINT32 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-FIXNUM64 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-INT64 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-UINT64 in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-FLOAT in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-DOUBLE in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-UCS1-STRING in DEFUN > >>> STYLE-WARNING: redefining BUFFER-READ-UCS4-STRING in DEFUN > >>> STYLE-WARNING: redefining LITTLE-ENDIAN-P in DEFUN > >>> ; compiling file "/Users/waldo/dev/lisp/elephant/tests/ > >>> testbdb.lisp" (written 20 FEB 2007 10:48:14 AM): > >>> ; compiling (IN-PACKAGE :ELE-TESTS) > >>> ; compiling (DEFVAR ENV) > >>> ; compiling (DEFVAR DB) > >>> ; compiling (DEFUN PREPARE-BDB ...) > >>> ; compiling (DEFTEST PREPARES-BDB ...) > >>> ; compiling (DEFUN TEST-SEQUENCE1 ...) > >>> ; compiling (DEFTEST TEST-SEQ1 ...) > >>> ; compiling (DEFUN TEST-SEQUENCE2 ...) > >>> ; compiling (DEFTEST TEST-SEQ2 ...) > >>> ; compiling (DEFUN CLEANUP-BDB ...) > >>> ; compiling (DEFTEST CLEANSUP-BDB ...) > >>> > >>> ; /Users/waldo/dev/lisp/elephant/tests/testbdb.fasl written > >>> ; compilation finished in 0:00:00 > >>> Doing 132 pending tests of 132 tests total. > >>> FIXNUMS FIXNUM-TYPE-1 READ-32-BIT-FIXNUM READ-64-BIT-FIXNUM > >>> WRITE-32-BIT-FIXNUM WRITE-64-BIT-FIXNUM BIGNUMS FLOATS RATIONALS > >>> COMPLEXES > >>> BASE-STRINGS STRINGS HARD-STRINGS SYMBOLS CHARS PATHNAMES CONSES > >>> HASH-TABLES-1 HASH-TABLES-2 ARRAYS-1 ARRAYS-2 TEST-DEEP-EQUALP > >>> TEST-SERIALIZATION-UNICODE-SLOT OBJECTS STRUCTS STRUCT-NON-STD- > >>> CONSTRUCT > >>> CIRCULAR PERSISTENT; in: LAMBDA NIL > >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > >>> ; ==> > >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > >>> ; > >>> ; caught WARNING: > >>> ; undefined variable: *TEST-SPEC-SECONDARY* > >>> > >>> ; > >>> ; caught WARNING: > >>> ; This variable is undefined: > >>> ; *TEST-SPEC-SECONDARY* > >>> ; > >>> ; compilation unit finished > >>> ; caught 2 WARNING conditions > >>> > >>> Second store spec missing: ignoring CROSS-STORE-REFERENCE-CONDITION > >>> UNINDEXED-CLASS-CONDITION NON-TRANSIENT-CLASS-SLOT-1 > >>> NON-TRANSIENT-CLASS-SLOT-2 TRANSIENT-CLASS-SLOT CLASS-DEFINERS > >>> BAD-INHERITENCE MIXES MIXES-RIGHT-SLOTS INHERIT INHERIT-RIGHT-SLOTS > >>> INITFORM-CLASSES INITFORM-TEST INITARG-TEST NO-EVAL-INITFORM > >>> REDEFCLASS > >>> MAKUNBOUND UPDATE-CLASS CHANGE-CLASS CHANGE-CLASS3 BASICPERSISTENCE > >>> TESTOID BTREE-MAKE BTREE-PUT BTREE-GET REMOVE-KV REMOVED MAP-BTREE > >>> INDEXED-BTREE-MAKE ADD-INDICES TEST-INDICES INDEXED-PUT INDEXED-GET > >>> SIMPLE-SLOT-GET INDEXED-GET-FROM-SLOT1 INDEXED-GET-FROM-SLOT2 > >>> REMOVE-KV-INDEXED NO-KEY-NOR-INDICES REMOVE-KV-FROM-SLOT1 > >>> NO-KEY-NOR-INDICES-SLOT1 REMOVE-KV-FROM-SLOT2 NO-KEY-NOR-INDICES- > >>> SLOT2 > >>> MAP-INDEXED GET-FIRST GET-FIRST2 GET-LAST GET-LAST2 SET SET2 SET- > >>> RANGE > >>> SET-RANGE2 MAP-INDEXED-INDEX MAP-INDEX-FROM-END REM-KV REM-IDEXKV > >>> MAKE-INDEXED2 ADD-INDICES2 PUT-INDEXED2 GET-INDEXED2 GET-FROM-INDEX3 > >>> DUP-TEST NODUP-TEST PREV-NODUP-TEST PNODUP-TEST PPREV-NODUP-TEST > >>> CUR-DEL1 > >>> INDEXED-DELETE TEST-DELETED INDEXED-DELETE2 TEST-DELETED2 CUR-DEL2 > >>> GET-BOTH PGET-BOTH PGET-BOTH-RANGE PCURSOR NEWINDEX PCURSOR2 > >>> ADD-GET-REMOVE ADD-GET-REMOVE-SYMBOL EXISTSP PSET > >>> DISABLE-CLASS-INDEXING-TEST INDEXING-BASIC-TRIVIAL INDEXING-BASIC > >>> INDEXING-CLASS-OPT INDEXING-INHERIT > >>> Test INDEXING-RANGE failed > >>> Form: (PROGN > >>> (DEFCLASS IDX-FOUR NIL > >>> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 > >>> :INDEX T)) > >>> (:METACLASS PERSISTENT-METACLASS)) > >>> (DISABLE-CLASS-INDEXING 'IDX-FOUR :ERRORP NIL) > >>> (SETF (FIND-CLASS 'IDX-FOUR NIL) NIL) > >>> (DEFCLASS IDX-FOUR NIL > >>> ((SLOT1 :INITARG :SLOT1 :INITFORM 1 :ACCESSOR SLOT1 > >>> :INDEX T)) > >>> (:METACLASS PERSISTENT-METACLASS)) > >>> (DEFUN MAKE-IDX-FOUR (VAL) (MAKE-INSTANCE 'IDX-FOUR :SLOT1 VAL)) > >>> (WITH-TRANSACTION NIL > >>> (MAPC #'MAKE-IDX-FOUR '(1 1 1 2 2 4 5 5 5 6 > >>> 10))) > >>> (LET ((X1 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 2 6)) > >>> (X2 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 0 2)) > >>> (X3 (GET-INSTANCES-BY-RANGE 'IDX-FOUR 'SLOT1 6 15))) > >>> (VALUES (EQUAL (MAPCAR #'SLOT1 X1) '(2 2 4 5 5 5 6)) > >>> (EQUAL (MAPCAR #'SLOT1 X2) '(1 1 1 2 2)) > >>> (EQUAL (MAPCAR #'SLOT1 X3) '(6 10))))) > >>> Expected values: T > >>> T > >>> T > >>> Actual value: #. > >>> INDEXING-SLOT-MAKUNBOUND INDEXING-WIPE-INDEX INDEXING-RECONNECT-DB > >>> INDEXING-CHANGE-CLASS INDEXING-REDEF-CLASS > >>> Ranged get of 10/700 objects = Linear: 0.435 sec Indexed: 0.01 sec > >>> INDEXING-TIMING > >>> ; in: LAMBDA NIL > >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > >>> ; ==> > >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > >>> ; > >>> ; caught WARNING: > >>> ; undefined variable: *TEST-SPEC-SECONDARY* > >>> > >>> ; > >>> ; caught WARNING: > >>> ; This variable is undefined: > >>> ; *TEST-SPEC-SECONDARY* > >>> ; > >>> ; compilation unit finished > >>> ; caught 2 WARNING conditions > >>> > >>> Single store mode: ignoring REMOVE-ELEMENT > >>> ; in: LAMBDA NIL > >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > >>> ; ==> > >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > >>> ; > >>> ; caught WARNING: > >>> ; undefined variable: *TEST-SPEC-SECONDARY* > >>> > >>> ; > >>> ; caught WARNING: > >>> ; This variable is undefined: > >>> ; *TEST-SPEC-SECONDARY* > >>> ; > >>> ; compilation unit finished > >>> ; caught 2 WARNING conditions > >>> > >>> Single store mode: ignoring MIGRATE-BASIC > >>> ; in: LAMBDA NIL > >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > >>> ; ==> > >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > >>> ; > >>> ; caught WARNING: > >>> ; undefined variable: *TEST-SPEC-SECONDARY* > >>> > >>> ; > >>> ; caught WARNING: > >>> ; This variable is undefined: > >>> ; *TEST-SPEC-SECONDARY* > >>> ; > >>> ; compilation unit finished > >>> ; caught 2 WARNING conditions > >>> > >>> Single store mode: ignoring MIGRATE-BTREE > >>> ; in: LAMBDA NIL > >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > >>> ; ==> > >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > >>> ; > >>> ; caught WARNING: > >>> ; undefined variable: *TEST-SPEC-SECONDARY* > >>> > >>> ; > >>> ; caught WARNING: > >>> ; This variable is undefined: > >>> ; *TEST-SPEC-SECONDARY* > >>> ; > >>> ; compilation unit finished > >>> ; caught 2 WARNING conditions > >>> > >>> Single store mode: ignoring MIGRATE-IDX-BTREE > >>> ; in: LAMBDA NIL > >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > >>> ; ==> > >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > >>> ; > >>> ; caught WARNING: > >>> ; undefined variable: *TEST-SPEC-SECONDARY* > >>> > >>> ; > >>> ; caught WARNING: > >>> ; This variable is undefined: > >>> ; *TEST-SPEC-SECONDARY* > >>> ; > >>> ; compilation unit finished > >>> ; caught 2 WARNING conditions > >>> > >>> Single store mode: ignoring MIGRATE-PCLASS > >>> ; in: LAMBDA NIL > >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > >>> ; ==> > >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > >>> ; > >>> ; caught WARNING: > >>> ; undefined variable: *TEST-SPEC-SECONDARY* > >>> > >>> ; > >>> ; caught WARNING: > >>> ; This variable is undefined: > >>> ; *TEST-SPEC-SECONDARY* > >>> ; > >>> ; compilation unit finished > >>> ; caught 2 WARNING conditions > >>> > >>> Single store mode: ignoring MIGRATE-MULT-PCLASS > >>> ; in: LAMBDA NIL > >>> ; (NULL ELEPHANT-TESTS::*TEST-SPEC-SECONDARY*) > >>> ; ==> > >>> ; (IF ELEPHANT-TESTS::*TEST-SPEC-SECONDARY* NIL T) > >>> ; > >>> ; caught WARNING: > >>> ; undefined variable: *TEST-SPEC-SECONDARY* > >>> > >>> ; > >>> ; caught WARNING: > >>> ; This variable is undefined: > >>> ; *TEST-SPEC-SECONDARY* > >>> ; > >>> ; compilation unit finished > >>> ; caught 2 WARNING conditions > >>> > >>> Single store mode: ignoring MIGRATE-IPCLASS PREPARES-BDB TEST-SEQ1 > >>> TEST-SEQ2 CLEANSUP-BDB > >>> 1 out of 132 total tests failed: INDEXING-RANGE.NIL > >>> > >>> _______________________________________________ > >>> elephant-devel site list > >>> elephant-devel at common-lisp.net > >>> http://common-lisp.net/mailman/listinfo/elephant-devel > >> > >> _______________________________________________ > >> elephant-devel site list > >> elephant-devel at common-lisp.net > >> http://common-lisp.net/mailman/listinfo/elephant-devel > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Thu Feb 21 22:51:24 2008 From: read at robertlread.net (Robert L. Read) Date: Thu, 21 Feb 2008 16:51:24 -0600 Subject: [elephant-devel] Elephant BDB Tests Fail on OS X Leopard, SBCL 1.0.14 In-Reply-To: References: <3D3FE2FF-EB06-4E45-ACEB-2FFFBE68035A@infoway.net> <317AC4EB-C31F-4805-890C-D8399CCB4606@media.mit.edu> <17ADEBAC-DD92-47A0-B71C-B102CBBDAA12@infoway.net> Message-ID: <1203634284.21443.350.camel@penguin.yourdomain.com> Should you be using the darcs system? The CVS tree is about 6 months out of date. I think you should either use one of the release tarballs or use darcs to get the latest version. On Thu, 2008-02-21 at 11:15 -0500, Ian Eslick wrote: > > > > I erased the entire elephant directory and re-downloaded the CSV > > repository again just now. The contents of the tests/testdb were > > just the README file and the CVS directory. I ran the tests and > got > > the exact same result. > > From lists at infoway.net Fri Feb 22 03:13:52 2008 From: lists at infoway.net (lists at infoway.net) Date: Thu, 21 Feb 2008 22:13:52 -0500 Subject: [elephant-devel] Elephant BDB Tests Fail on OS X Leopard, SBCL 1.0.14 In-Reply-To: <1203634284.21443.350.camel@penguin.yourdomain.com> References: <3D3FE2FF-EB06-4E45-ACEB-2FFFBE68035A@infoway.net> <317AC4EB-C31F-4805-890C-D8399CCB4606@media.mit.edu> <17ADEBAC-DD92-47A0-B71C-B102CBBDAA12@infoway.net> <1203634284.21443.350.camel@penguin.yourdomain.com> Message-ID: <02A59D83-E2C1-4CA5-9156-EC50A34155D1@infoway.net> I guess the problem was that I was using the CVS version. I installed the darcs version and all tests were good. One comment I would make is that the documentation does not really say or recommend to use the darcs version. Additionally, the documentation does not say that fiveam and arnesi are required libraries. I'm a happy camper now. Will delve into this now. Thanks, Waldo On Feb 21, 2008, at 5:51 PM, Robert L. Read wrote: > Should you be using the darcs system? The CVS tree is about 6 months > out of date. I think you should either use one of the release > tarballs > or use darcs to get the latest version. > > On Thu, 2008-02-21 at 11:15 -0500, Ian Eslick wrote: >>> >>> I erased the entire elephant directory and re-downloaded the CSV >>> repository again just now. The contents of the tests/testdb were >>> just the README file and the CVS directory. I ran the tests and >> got >>> the exact same result. >>> > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Fri Feb 22 09:05:14 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 22 Feb 2008 10:05:14 +0100 (CET) Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE In-Reply-To: <5DF3719A-FB72-4201-83C2-B995F53D2905@media.mit.edu> References: <62536.88.73.249.123.1203601284.squirrel@mail.stardawn.org> <5DF3719A-FB72-4201-83C2-B995F53D2905@media.mit.edu> Message-ID: <62526.88.73.219.154.1203671114.squirrel@mail.stardawn.org> > Do you need to do indexing operations across stores? i.e. run a > cursor over all objects regardless of which store they are in? No, I don't do this at the moment. I might later, so let's postpone that point. > Ideally, what semantics would you like to see? Right now all I want is ?do what I mean? behaviour for SETF SLOT-VALUE. AFAICS the current behaviour for this operation is that Elephant attempts to stuff the object into the current sc instead of using the objects home controller. Which seems not exactly sensible to me. > Nope, the system only creates one class index independent of which > store controller it's in, so you can only map indexed class instances > to one store. I'm really having difficuly correlating what you say here to my experience. I can happily have separate sets of indexed objects with separate store controllers, or am at least deluded in a convincing way :) So I suppose it's my own dumbness again that's preventing me from understanding correctly what you say here. What you say here is that there's only one index for all controllers, but otherwhere you state that the problem is separation of controller indexes... I'm confused. > e.g. you create indexed instances in store 1, the class index > associated with store one indexes all those instances. When you call > get-instances-by-class in the context of store 2, however, all the > objects from store 1 are now missing. The semantics of get-instance- > by-class/value/range are all incorrect when you have instances stored > in multiple store controllers. It makes good sense for me, since I'm used to the model of store controllers as objects with a very high grade of separation. So when I change the sc, everything is different. And this fits my current project well, too. > Using multiple stores starts to break the model of a simple global > namespace assumed by the current class indexing mechanism. Rather > than contort the class indexing mechanism to accommodate multiple > store operations, my preference is that users write their own indexing > mechanism that has the semantics they care about. It depends on what you intend here. If the aim is global indexing across controllers, let's have it your way with user-based indexing. It makes sense. But if we talk about the thing as I see it, i.e. indexing separate for all controllers, let's just fix the system so it behaves well in this context. Leslie -- My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Fri Feb 22 09:13:49 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 22 Feb 2008 10:13:49 +0100 (CET) Subject: [elephant-devel] Fix for Unicode2 serializer In-Reply-To: <1203566319.21443.305.camel@penguin.yourdomain.com> References: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> <1203566319.21443.305.camel@penguin.yourdomain.com> Message-ID: <63028.88.73.219.154.1203671629.squirrel@mail.stardawn.org> > However, I wonder if it possible for you to turn whatever made you > notice it into a test-case? It is possible that this bug exists on the > LISP you are using and not mine, but in any case we should use each such > bug as an opportunity to expand the suite of automated tests. It was hard, but I managed to add one. It won't work on all implementations since the spec doesn't guarantee a way to generate a non-simple string. It's very plain to see that the old code was wrong. In a branch where VAR was guaranteed to be a STRING it used SCHAR, which is only defined for SIMPLE-STRING (a subtype of STRING). I wonder how it could get away with that for so long. > As you will see, we have a tiny number of tests around unicode issues, > which arose out of my using some Esperanto stuff. However, I am sure > our tests are not sufficient. It would appear so :) > The ideal thing would be to write a test, see that it fails without our > patch, and then that it and all the other relevant tests are green with > the patch in place. I don't know how other Lisps handle this string stuff, but on SBCL (1.0.14, probably earlier versions, too) the new test will fail. Note that if you have a Lisp where the test doesn't fail (despite incorrect use of SCHAR), you won't have any problems with the erroneous code, either. Now, attached are three patch files containing the following: * the new test case * the fix * a refactoring of the UTF{16,32}LE serializers If you want multiple revisions in one file in the future, please say so. Leslie -- My personal blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: STRING-test.patch Type: text/x-patch Size: 3859 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: unicode2.patch Type: text/x-patch Size: 4006 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: refactor-unicode-serializers.patch Type: text/x-patch Size: 7019 bytes Desc: not available URL: From eslick at media.mit.edu Fri Feb 22 12:35:08 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 22 Feb 2008 07:35:08 -0500 Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE In-Reply-To: <62526.88.73.219.154.1203671114.squirrel@mail.stardawn.org> References: <62536.88.73.249.123.1203601284.squirrel@mail.stardawn.org> <5DF3719A-FB72-4201-83C2-B995F53D2905@media.mit.edu> <62526.88.73.219.154.1203671114.squirrel@mail.stardawn.org> Message-ID: <6B3C8885-00DC-443E-A073-D3639B1B6518@media.mit.edu> The problem lies in an implementation choice. It can be changed, but it's not as easy as it sounds. 1) The class object caches the class index reference. It currently can only maintain a single cached reference. (from metaclasses.lisp) (defclass persistent-metaclass (standard-class) ((%persistent-slots :accessor %persistent-slots) (%indexed-slots :accessor %indexed-slots) (%index-cache :accessor %index-cache)) (:documentation "Metaclass for persistent classes. Use this metaclass to define persistent classes. All slots are persistent by default; use the :transient flag otherwise. Slots can also be indexed for by-value retrieval.")) 2) An indexed slot writer (in classindex.lisp) is called during writes which looks up the class index using find-class-index (defmethod indexed-slot-writer ((class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition) new-value) "Anything that side effects a persistent-object slot should call this to keep the dependant indices in synch. Only classes with derived indices need to update on writes to non-indexed slots. This is a side effect of user-managed indices in Elephant - a necessity because we allow arbitrary lisp expressions to determine index value so without bi-directional pointers, the indices cannot automatically update a changed indexed value in derived slots" (let ((slot-name (slot-definition-name slot-def)) (oid (oid instance)) (con (get-con instance))) (declare (type fixnum oid)) (if (no-indexing-needed? class instance slot-def oid) (persistent-slot-writer con new-value instance slot-name) (let ((class-idx (find-class-index class))) (ensure-transaction (:store-controller con) (when (get-value oid class-idx) (remove-kv oid class-idx)) (persistent-slot-writer con new-value instance slot-name) (setf (get-value oid class-idx) instance)))))) 3) find-class-index (classindex.lisp) looks in the class cache, defined above, to avoid doing an additional btree lookup on-disk for every write operation. (defmethod find-class-index ((class persistent-metaclass) &key (sc *store-controller*) (errorp t)) (ensure-finalized class) (if (not (indexed class)) (when errorp (signal-class-not-indexed class)) (if (class-index-cached? class sc) (%index-cache class) ;; we've got a cached reference, just return it (multiple-value-bind (btree found) (get-value (class-name class) (controller-class-root sc)) (if found (cache-existing-class-index class btree sc) (cache-new-class-index class sc)))))) 4) Now we could easily make this cache slot be an alist and dispatch on the store controller, but there is another wrinkle. The system tries to be smart about not losing references to index data if the user happens to change the class definition between connections. In general there is a configurable facility for defining what happens when the indexed slot list of the class and the database disagree. (defun cache-existing-class-index (class btree sc) "If we have a persistent index already, assign, synchronize & return it" (let ((method (determine-synch-method class))) (setf (%index-cache class) btree) (synchronize-class-to-store class :sc sc :method method) btree)) ;; from classindex-utils.lisp (defun synchronize-class-to-store (class &key (sc *store-controller*) (method *default-indexed-class-synch-policy*)) (let ((slot-records (compute-class-and-ele-status class sc)) (rule-set (cdr (assoc method *synchronize-rules*)))) (apply-synch-rules class slot-records rule-set))) If you can come up with a reasonable policy for N-way disagreement, then the rest of the changes probably aren't too hard. Perhaps this could be configurable. If a store controller has already connected to the class, then you just add any indices that are missing in the new database. Remember, though, that the consequence of adding an index is that the whole index has to be constructed so for classes with lots of instances, this can take awhile. If this doesn't make sense, you should try skimming the code in metaclasses.lisp, classindex.lisp, and classindex-utils.lisp. Ian On Feb 22, 2008, at 4:05 AM, Leslie P. Polzer wrote: > > >> Do you need to do indexing operations across stores? i.e. run a >> cursor over all objects regardless of which store they are in? > > No, I don't do this at the moment. I might later, so let's postpone > that point. > > >> Ideally, what semantics would you like to see? > > Right now all I want is ?do what I mean? behaviour for SETF SLOT- > VALUE. > AFAICS the current behaviour for this operation is that Elephant > attempts > to stuff the object into the current sc instead of using the objects > home controller. Which seems not exactly sensible to me. > > >> Nope, the system only creates one class index independent of which >> store controller it's in, so you can only map indexed class instances >> to one store. > > I'm really having difficuly correlating what you say here to my > experience. I can happily have separate sets of indexed objects > with separate store controllers, or am at least deluded in a > convincing way :) > > So I suppose it's my own dumbness again that's preventing me > from understanding correctly what you say here. > > What you say here is that there's only one index for all controllers, > but otherwhere you state that the problem is separation of > controller indexes... I'm confused. > > >> e.g. you create indexed instances in store 1, the class index >> associated with store one indexes all those instances. When you call >> get-instances-by-class in the context of store 2, however, all the >> objects from store 1 are now missing. The semantics of get-instance- >> by-class/value/range are all incorrect when you have instances stored >> in multiple store controllers. > > It makes good sense for me, since I'm used to the model of store > controllers as objects with a very high grade of separation. > So when I change the sc, everything is different. > And this fits my current project well, too. > > >> Using multiple stores starts to break the model of a simple global >> namespace assumed by the current class indexing mechanism. Rather >> than contort the class indexing mechanism to accommodate multiple >> store operations, my preference is that users write their own >> indexing >> mechanism that has the semantics they care about. > > It depends on what you intend here. > > If the aim is global indexing across controllers, let's have it your > way with user-based indexing. It makes sense. > > But if we talk about the thing as I see it, i.e. indexing separate > for all controllers, let's just fix the system so it behaves well > in this context. > > Leslie > > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Fri Feb 22 20:55:58 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 22 Feb 2008 15:55:58 -0500 Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE In-Reply-To: <6B3C8885-00DC-443E-A073-D3639B1B6518@media.mit.edu> References: <62536.88.73.249.123.1203601284.squirrel@mail.stardawn.org> <5DF3719A-FB72-4201-83C2-B995F53D2905@media.mit.edu> <62526.88.73.219.154.1203671114.squirrel@mail.stardawn.org> <6B3C8885-00DC-443E-A073-D3639B1B6518@media.mit.edu> Message-ID: > Right now all I want is ?do what I mean? behaviour for SETF SLOT- > VALUE. > AFAICS the current behaviour for this operation is that Elephant > attempts > to stuff the object into the current sc instead of using the objects > home controller. Which seems not exactly sensible to me. > Ok, I think I can clarify a couple of points (especially now that I've caught up on sleep! :) What we're arguing about is the correct default behavior. I think you're talking me around on supporting this functionality, however I want to make sure that you have to explicitly enable it. The reason for this is that allowing objects to have more than one home store means that the semantics of other functions has to change, such as get-instances-by-class which now becomes get-instances-by- class-in-current-store-controller. I'd like users to realize that this change is happening when it does. If a user unintentionally creates an object in a second store, I don't want that error to be invisible. Here is a proposal: 1) Writes to objects will always use that object's home store (as requested!) 2) The persistent-metaclass would be extended to support multiple class indexes; one for each store controller instances exist in. 3) We add an option to enable multi-store operation. Unless multi- store operation is enabled, writing an object to a new store will result in a condition which asks if the user really intended to do this with a reference to the manual section they should read. 4) When multi-store mode is on, class+database synchronization is disabled. This means that any indexes for a given class will be ignored if the class definition doesn't explicitly reference them (this just means that the index storage will be maintained, even if you choose to unindex a slot in the defpclass) 5) To fix the abstraction problems, we might want to add a facility when a store is opened to scan the data store's master list of class indexes and explicitly add the class index to the set of cached class indexes for that class. This way the user can rely on the class knowing about all the stores that reference an object. We could then implement 'the right thing' for get-instances-by-class, etc. by doing multiple map-indexes and merging the results. 6) Add a chapter to the manual expanding on the implications for storing indexed classes in multiple stores I'll do # 1-4 now since it doesn't really effect anyone but you and maintains the current behavior for anyone else. I can't sign up for #5-6. This proposal should do the right thing for the common cases you want (esp. if we do #5), but make sure that people think about what they're doing before they do it and don't shoot themselves in the foot accidentally by creating objects that disappear. The manual should talk about what parts of the class indexing abstraction will change. For example index-based queries of all objects or objects by value refer only to the objects in the current store, not to all instances of the class that may exist. The user can continue and disable the warning for the current session. I could explain why the current behavior is what it is, but I'd probably just confuse things further! Regards, Ian From read at robertlread.net Sat Feb 23 00:25:27 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 22 Feb 2008 18:25:27 -0600 Subject: [elephant-devel] Fix for Unicode2 serializer In-Reply-To: <63028.88.73.219.154.1203671629.squirrel@mail.stardawn.org> References: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> <1203566319.21443.305.camel@penguin.yourdomain.com> <63028.88.73.219.154.1203671629.squirrel@mail.stardawn.org> Message-ID: <1203726327.21443.414.camel@penguin.yourdomain.com> Thanks! I'm about to leave town, but should be able to review and commit this by next Wednesday. On Fri, 2008-02-22 at 10:13 +0100, Leslie P. Polzer wrote: > > However, I wonder if it possible for you to turn whatever made you > > notice it into a test-case? It is possible that this bug exists on the > > LISP you are using and not mine, but in any case we should use each such > > bug as an opportunity to expand the suite of automated tests. > > It was hard, but I managed to add one. It won't work on all implementations > since the spec doesn't guarantee a way to generate a non-simple string. > > It's very plain to see that the old code was wrong. In a branch where VAR > was guaranteed to be a STRING it used SCHAR, which is only defined for > SIMPLE-STRING (a subtype of STRING). I wonder how it could get away > with that for so long. > > > > As you will see, we have a tiny number of tests around unicode issues, > > which arose out of my using some Esperanto stuff. However, I am sure > > our tests are not sufficient. > > It would appear so :) > > > > The ideal thing would be to write a test, see that it fails without our > > patch, and then that it and all the other relevant tests are green with > > the patch in place. > > I don't know how other Lisps handle this string stuff, but on SBCL (1.0.14, > probably earlier versions, too) the new test will fail. Note that if you > have a Lisp where the test doesn't fail (despite incorrect use of SCHAR), > you won't have any problems with the erroneous code, either. > > Now, attached are three patch files containing the following: > > * the new test case > > * the fix > > * a refactoring of the UTF{16,32}LE serializers > > If you want multiple revisions in one file in the future, please say so. > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Sat Feb 23 09:03:26 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 23 Feb 2008 10:03:26 +0100 (CET) Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE In-Reply-To: References: <62536.88.73.249.123.1203601284.squirrel@mail.stardawn.org> <5DF3719A-FB72-4201-83C2-B995F53D2905@media.mit.edu> <62526.88.73.219.154.1203671114.squirrel@mail.stardawn.org> <6B3C8885-00DC-443E-A073-D3639B1B6518@media.mit.edu> Message-ID: <61759.88.73.235.79.1203757406.squirrel@mail.stardawn.org> > 2) The persistent-metaclass would be extended to support multiple > class indexes; one for each store controller instances exist in. I suppose that's the architectural issue you were talking about, but which I didn't fully grasp because I haven't looked at the code. > I'll do # 1-4 now since it doesn't really effect anyone but you and > maintains the current behavior for anyone else. I can't sign up for > #5-6. I'd do #5 and #6, in principle, but this issue is not really big enough an itch for me to scratch it right now. Perhaps it would be best to archive this discussion (or the results of it) in a ticket and then do it later. Leslie -- My personal blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Sat Feb 23 14:34:04 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sat, 23 Feb 2008 09:34:04 -0500 Subject: [elephant-devel] Cross-references and SETF SLOT-VALUE In-Reply-To: <61759.88.73.235.79.1203757406.squirrel@mail.stardawn.org> References: <62536.88.73.249.123.1203601284.squirrel@mail.stardawn.org> <5DF3719A-FB72-4201-83C2-B995F53D2905@media.mit.edu> <62526.88.73.219.154.1203671114.squirrel@mail.stardawn.org> <6B3C8885-00DC-443E-A073-D3639B1B6518@media.mit.edu> <61759.88.73.235.79.1203757406.squirrel@mail.stardawn.org> Message-ID: On Feb 23, 2008, at 4:03 AM, Leslie P. Polzer wrote: > >> 2) The persistent-metaclass would be extended to support multiple >> class indexes; one for each store controller instances exist in. > > I suppose that's the architectural issue you were talking about, > but which I didn't fully grasp because I haven't looked at the code. > There are some secondary architectural issues that are more complicated and involve how the system keeps track of mismatches between a data store and a class definition, how class re-definition is handled to add/drop indices, etc. >> I'll do # 1-4 now since it doesn't really effect anyone but you and >> maintains the current behavior for anyone else. I can't sign up for >> #5-6. > > I'd do #5 and #6, in principle, but this issue is not really big > enough an itch for me to scratch it right now. Great! I've implemented #1-3 already and #4 is easy. I am just testing them now. If you turn off multi-store support everything passes, just a few snafus in the multi-store case. > Perhaps it would be best to archive this discussion (or the results > of it) in a ticket and then do it later. I'll take care of that when I check in the new code. > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Sat Feb 23 18:50:52 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sat, 23 Feb 2008 13:50:52 -0500 Subject: [elephant-devel] Multi-store indexed classes Message-ID: I just pushed a patch that enables multi-store class indexing as Leslie requested with the design tradeoffs from my last e-mail. Still needed for 0.9.2: documentation, specific tests, better error messages when doing cross store indexing when *enable-multi-store- indexing* isn't set. I'll add a note to the Trac feature request list to enable the standard interfaces (get-instances-by-xxx) to work across all instances across all open stores. Ian From leslie.polzer at gmx.net Sun Feb 24 19:54:10 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 24 Feb 2008 20:54:10 +0100 (CET) Subject: [elephant-devel] SQLite locking error / BDB multi-image support Message-ID: <65095.88.73.214.84.1203882850.squirrel@mail.stardawn.org> I just got this: While accessing database # with expression "UPDATE KEYVALUE SET KEY = 'DysAAAANCQYAAABQT1RJT04JBgAAAE1ZU1RJQw==',CLCTN_ID = 28,VALUE = 'DQkBAAAAVAkLAAA AQ09NTU9OLUxJU1A=' WHERE ((CLCTN_ID = 28) AND (KEY = 'DysAAAANCQYAAABQT1RJT04JBgAAAE1ZU1RJQw=='))": Error 5 / database is locked has occurred. It seems to be a transient problem, since I got away error-free with repeating the action... Also the BDB backend instantly complains about a corrupted database as soon as I access a DB with another Lisp image. Seems I have to resort to Postgres... All backtraces available on request. Leslie -- My personal blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Sun Feb 24 22:38:31 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sun, 24 Feb 2008 17:38:31 -0500 Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: <65095.88.73.214.84.1203882850.squirrel@mail.stardawn.org> References: <65095.88.73.214.84.1203882850.squirrel@mail.stardawn.org> Message-ID: <8AB7FBBC-C2BF-4609-9CCD-F21A1DAC759B@media.mit.edu> I just opened a fresh DB from both an Allegro image and an SBCL image with no problems, set some data in one and read it from the other. This fails for you? I believe other people are using multiple images on multi-processor machines without incident so this may be a configuration or environment issue. Why don't you send over stack traces and a more in-depth description of your setup. Ian PS - Are you trying to run this on your local disk, or over a networked filesystem? On Feb 24, 2008, at 2:54 PM, Leslie P. Polzer wrote: > > I just got this: > > While accessing database # world.sqlite3 OPEN > {D5AEC99}> > with expression "UPDATE KEYVALUE SET KEY = > 'DysAAAANCQYAAABQT1RJT04JBgAAAE1ZU1RJQw==',CLCTN_ID = 28,VALUE = > 'DQkBAAAAVAkLAAA > AQ09NTU9OLUxJU1A=' WHERE ((CLCTN_ID = 28) AND (KEY = > 'DysAAAANCQYAAABQT1RJT04JBgAAAE1ZU1RJQw=='))": > Error 5 / database is locked > has occurred. > > It seems to be a transient problem, since I got away error-free with > repeating > the action... > > Also the BDB backend instantly complains about a corrupted database > as soon > as I access a DB with another Lisp image. > > Seems I have to resort to Postgres... > > All backtraces available on request. > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Mon Feb 25 07:19:50 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 25 Feb 2008 08:19:50 +0100 (CET) Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: <8AB7FBBC-C2BF-4609-9CCD-F21A1DAC759B@media.mit.edu> References: <65095.88.73.214.84.1203882850.squirrel@mail.stardawn.org> <8AB7FBBC-C2BF-4609-9CCD-F21A1DAC759B@media.mit.edu> Message-ID: <63400.88.73.228.215.1203923990.squirrel@mail.stardawn.org> > Why don't you send over stack traces and a more in-depth description > of your setup. Here's the SQLite story. I'm using SQLite3 version 3.5.6 (apparently compiled with thread safety), CLSQL 4.0.1, SBCL 1.0.14 and the latest Elephant on an ext3 file system, kernel 2.6.24. The error occurs on my development machine with close to zero load and only two processes writing quickly with minimum time in between. The backtrace is attached. You can also see the two requests in close succession before it. The first request removes an object from a PSET, the second adds it again to the same PSET (although this detail probably doesn't matter since the error is occuring on the SQLite layer). I have also found that the SQLite backend of the Trac project had a bunch of similar errors in previous versions[1]. The error occurs with roughly 50% probability. > PS - Are you trying to run this on your local disk, or over a > networked filesystem? Everything is local. Leslie [1] http://www.google.de/search?hl=en&q=+site:trac.edgewall.org+trac+sqlite+locking+error -------------- next part -------------- A non-text attachment was scrubbed... Name: sqlite-error-bt.log Type: text/x-log Size: 5306 bytes Desc: not available URL: From leslie.polzer at gmx.net Mon Feb 25 07:32:00 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 25 Feb 2008 08:32:00 +0100 (CET) Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: <8AB7FBBC-C2BF-4609-9CCD-F21A1DAC759B@media.mit.edu> References: <65095.88.73.214.84.1203882850.squirrel@mail.stardawn.org> <8AB7FBBC-C2BF-4609-9CCD-F21A1DAC759B@media.mit.edu> Message-ID: <64136.88.73.228.215.1203924720.squirrel@mail.stardawn.org> > Why don't you send over stack traces and a more in-depth description > of your setup. Here's the BDB case. Underlying setup is the same as in the SQLite scenario. Backtrace inline at the bottom of this message (irrelevant parts cut). I have a game engine and an administration backend, which are running in a separate SBCL image each. Each one works fine in isolation, but as soon as another opens the databases, the process started earlier throws the backtrace below when it sends the next database request. The new process can access the database without problems, though... Leslie Berkeley DB error: DB_RUNRECOVERY: Fatal error, run database recovery 0: (BACKTRACE 536870911 #) 1: (HUNCHENTOOT:GET-BACKTRACE #) 2: ((LAMBDA (COND)) #) 3: ((LAMBDA (COND)) #) 4: (SIGNAL #) 5: (ERROR ELEPHANT:BDB-DB-ERROR) 6: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER T)) # # # #) 7: ((SB-PCL::FAST-METHOD ELEPHANT:MAP-BTREE (T ELEPHANT:BTREE)) # # # #) 8: (MYSTIC:MAP-CLASS* # MYSTIC:PLAYER) Note: MAP-CLASS* is defined as (defun map-class* (fn class) (loop for class in (cons class (moptilities:subclasses class)) do (map-class fn class))) From leslie.polzer at gmx.net Mon Feb 25 07:49:28 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 25 Feb 2008 08:49:28 +0100 (CET) Subject: [elephant-devel] Automate "create language plpgsql;" Message-ID: <64775.88.73.228.215.1203925768.squirrel@mail.stardawn.org> At the moment this has to be done for each Postgres database used by Elephant. Any objections on doing this automatically if needed? Leslie From henrik at evahjelte.com Mon Feb 25 08:56:49 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Mon, 25 Feb 2008 09:56:49 +0100 Subject: [elephant-devel] Automate "create language plpgsql;" In-Reply-To: <64775.88.73.228.215.1203925768.squirrel@mail.stardawn.org> References: <64775.88.73.228.215.1203925768.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60802250056o67d2de85tc139314b005858e1@mail.gmail.com> Yes, because it doesn't work for postgres 8.2 in all cases, probably because of security settings. I know we have been trying that before, and there was at least a reason it was left out... But if you can make it work under a check for postgres 8.3 that is great.. / Henrik 2008/2/25, Leslie P. Polzer : > > At the moment this has to be done for each Postgres database used > by Elephant. Any objections on doing this automatically if needed? > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -- Henrik Hjelte henrik.hjelte at stix.to +46703993945 http://stix.to L?stmakarg 18-20 (IQube) Box 7438 S-103 91 Stockholm Sweden From killerstorm at newmail.ru Mon Feb 25 09:00:40 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Mon, 25 Feb 2008 11:00:40 +0200 Subject: [elephant-devel] Re: Automate "create language plpgsql;" References: <64775.88.73.228.215.1203925768.squirrel@mail.stardawn.org> Message-ID: LPP> At the moment this has to be done for each Postgres database used LPP> by Elephant. Any objections on doing this automatically if needed? we have this: (defun init-stored-procedures (con) (safe-ignore-postgres-error (con) (cl-postgres:exec-query con "CREATE LANGUAGE plpgsql;")) but it fails: alex at debetch:~$ psql -c "create language plpgsql" elepm ERROR: must be superuser to create procedural language no big deal, i have a script that re-creates database: #!/bin/bash dropdb elepm createdb elepm psql -c "grant all on database elepm to alex" psql -c "create language plpgsql" elepm i execute it as postgres via sudo -u. From desoi at pgedit.com Mon Feb 25 13:21:12 2008 From: desoi at pgedit.com (John DeSoi) Date: Mon, 25 Feb 2008 08:21:12 -0500 Subject: [elephant-devel] Re: Automate "create language plpgsql;" In-Reply-To: References: <64775.88.73.228.215.1203925768.squirrel@mail.stardawn.org> Message-ID: On Feb 25, 2008, at 4:00 AM, Alex Mizrahi wrote: > alex at debetch:~$ psql -c "create language plpgsql" elepm > ERROR: must be superuser to create procedural language I think this is changed in 8.3 -- you don't have to be superuser to create a trusted language. John DeSoi, Ph.D. From leslie.polzer at gmx.net Mon Feb 25 13:22:09 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 25 Feb 2008 14:22:09 +0100 (CET) Subject: [elephant-devel] SQLite locking error / BDB multi-image support Message-ID: <61108.88.73.228.215.1203945729.squirrel@mail.stardawn.org> > To isolate the problem, can you verify the base case, connect two > images to a fresh db? For BDB: (asdf:oos 'asdf:load-op 'elephant) (defpackage #:ele-test (:use :cl :elephant)) (in-package :ele-test) (defpclass myclass () ((testslot :accessor testslot :initarg testslot :index t))) (open-store '(:BDB "/tmp/db1") :recover t) (setf item (make-instance 'myclass)) (setf (testslot item) 5) I load this in two images. When I then try to access TESTSLOT in the former, I get the mentioned error. I'll try to make up something for the SQLite case, too. FYI, Postmodern doesn't show any problems with my setup. Leslie From leslie.polzer at gmx.net Mon Feb 25 13:34:29 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 25 Feb 2008 14:34:29 +0100 (CET) Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: <31CE2F99-A847-4053-8A74-80E8E4A4DF71@media.mit.edu> References: <65095.88.73.214.84.1203882850.squirrel@mail.stardawn.org> <8AB7FBBC-C2BF-4609-9CCD-F21A1DAC759B@media.mit.edu> <64136.88.73.228.215.1203924720.squirrel@mail.stardawn.org> <31CE2F99-A847-4053-8A74-80E8E4A4DF71@media.mit.edu> Message-ID: <62070.88.73.228.215.1203946469.squirrel@mail.stardawn.org> > To isolate the problem, can you verify the base case, connect two > images to a fresh db? Here's one for SQLite: (asdf:oos 'asdf:load-op 'elephant) (defpackage #:ele-test (:use :cl :elephant)) (in-package :ele-test) (defpclass myclass () ((testslot :accessor testslot :initarg testslot :index t))) (open-store '(:CLSQL (:SQLITE3 "/tmp/db1.sqlite"))) (setf item (make-instance 'myclass)) (dotimes (i 100000) (write-char #\*)(SB-INT:FLUSH-STANDARD-OUTPUT-STREAMS) (setf (testslot item) 5)) The second image failed already at MAKE-INSTANCE. Leslie From read at robertlread.net Mon Feb 25 13:54:28 2008 From: read at robertlread.net (Robert L. Read) Date: Mon, 25 Feb 2008 07:54:28 -0600 Subject: [elephant-devel] Fix for Unicode2 serializer In-Reply-To: <63028.88.73.219.154.1203671629.squirrel@mail.stardawn.org> References: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> <1203566319.21443.305.camel@penguin.yourdomain.com> <63028.88.73.219.154.1203671629.squirrel@mail.stardawn.org> Message-ID: <1203947668.21443.535.camel@penguin.yourdomain.com> Dear Leslie, In attempting to apply the second of your three patches (unicode2.patch), darcs says: [read at penguin elephant]$ darcs apply ../unicode2.patch darcs failed: Patch bundle failed hash! This probably means that the patch has been corrupted by a mailer. The most likely culprit is CRLF newlines. However, the hunks: hunk ./src/elephant/unicode2.lisp 131 - (let ((code (char-code (schar string i)))) + (let ((code (char-code (char string i)))) hunk ./src/elephant/unicode2.lisp 171 - (let ((code (char-code (schar string i)))) + (let ((code (char-code (char string i)))) Don't seem to match the version of unicode2.lisp that I have, which seems to be the official repository version. It is possible that you have not recently done a "darcs pull" against the official repository (http://www.common-lisp.net/project/elephant/darcs/elephant))? Whatever the cause of the problem, I am happy to make the fixes by hand. If would prefer to figure out how we are incorrectly using darcs, but you can also just send me a "context" diff and I will make the changes by hand. On Fri, 2008-02-22 at 10:13 +0100, Leslie P. Polzer wrote: > > However, I wonder if it possible for you to turn whatever made you > > notice it into a test-case? It is possible that this bug exists on the > > LISP you are using and not mine, but in any case we should use each such > > bug as an opportunity to expand the suite of automated tests. > > It was hard, but I managed to add one. It won't work on all implementations > since the spec doesn't guarantee a way to generate a non-simple string. > > It's very plain to see that the old code was wrong. In a branch where VAR > was guaranteed to be a STRING it used SCHAR, which is only defined for > SIMPLE-STRING (a subtype of STRING). I wonder how it could get away > with that for so long. > > > > As you will see, we have a tiny number of tests around unicode issues, > > which arose out of my using some Esperanto stuff. However, I am sure > > our tests are not sufficient. > > It would appear so :) > > > > The ideal thing would be to write a test, see that it fails without our > > patch, and then that it and all the other relevant tests are green with > > the patch in place. > > I don't know how other Lisps handle this string stuff, but on SBCL (1.0.14, > probably earlier versions, too) the new test will fail. Note that if you > have a Lisp where the test doesn't fail (despite incorrect use of SCHAR), > you won't have any problems with the erroneous code, either. > > Now, attached are three patch files containing the following: > > * the new test case > > * the fix > > * a refactoring of the UTF{16,32}LE serializers > > If you want multiple revisions in one file in the future, please say so. > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Mon Feb 25 14:14:34 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 25 Feb 2008 15:14:34 +0100 (CET) Subject: [elephant-devel] Fix for Unicode2 serializer In-Reply-To: <1203947668.21443.535.camel@penguin.yourdomain.com> References: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> <1203566319.21443.305.camel@penguin.yourdomain.com> <63028.88.73.219.154.1203671629.squirrel@mail.stardawn.org> <1203947668.21443.535.camel@penguin.yourdomain.com> Message-ID: <63287.88.73.228.215.1203948874.squirrel@mail.stardawn.org> Dear Robert, > hunk ./src/elephant/unicode2.lisp 131 > - (let ((code (char-code (schar string i)))) > + (let ((code (char-code (char string i)))) > hunk ./src/elephant/unicode2.lisp 171 > - (let ((code (char-code (schar string i)))) > + (let ((code (char-code (char string i)))) > > Don't seem to match the version of unicode2.lisp that I have, which > seems to be the official repository version. I just downloaded this file from the repository, and line 131 says: (let ((code (char-code (schar string i)))) but should read (with my fix): (let ((code (char-code (char string i)))) Same for line 171. I wonder what's wrong? > It is possible that you have not recently done a "darcs pull" against > the official repository > (http://www.common-lisp.net/project/elephant/darcs/elephant))? I might have missed the two patches Ian applied on the 21st. Attached are two updated patches. The refactoring patch includes the hunk above implicitly. If you want to apply the fix separately, you know what to do... > Whatever the cause of the problem, I am happy to make the fixes by hand. Thanks, I appreciate it :) Leslie -------------- next part -------------- A non-text attachment was scrubbed... Name: refactor-unicode-serializers.patch Type: text/x-patch Size: 7311 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: STRING-test.patch Type: text/x-patch Size: 4151 bytes Desc: not available URL: From leslie.polzer at gmx.net Mon Feb 25 16:30:33 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 25 Feb 2008 17:30:33 +0100 (CET) Subject: [elephant-devel] Glitch in classes.lisp Message-ID: <62687.88.73.228.215.1203957033.squirrel@mail.stardawn.org> Ian, the change introduced by you here: @@ -62,8 +62,7 @@ (persistent-object (find-class 'persistent-object)) (not-already-persistent (loop for superclass in direct-superclasses never (eq (class-of superclass) persistent-metaclass)))) - (when index - (update-indexed-record class nil :class-indexed t)) + (setf (%indexed-class class) index) (if (and (not (eq class persistent-object)) not-already-persistent) (apply #'call-next-method class slot-names :direct-superclasses (append direct-superclasses (list persistent-object)) args) just caused my Lisp to throw a warning about The slot %INDEXED-CLASS is unbound in the object #. Leslie From leslie.polzer at gmx.net Mon Feb 25 16:35:38 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 25 Feb 2008 17:35:38 +0100 (CET) Subject: [elephant-devel] Glitch in classes.lisp In-Reply-To: <62687.88.73.228.215.1203957033.squirrel@mail.stardawn.org> References: <62687.88.73.228.215.1203957033.squirrel@mail.stardawn.org> Message-ID: <62711.88.73.228.215.1203957338.squirrel@mail.stardawn.org> > just caused my Lisp to throw a warning about > > The slot %INDEXED-CLASS is unbound in the object # PERSISTENT-OBJECT>. That doesn't occur when starting cleanly without FASLs, so it seems to be a case of faulty dependency resolution or specification. But when I open my store now, I get debugger invoked on a UNBOUND-SLOT in thread #: (A UNBOUND-SLOT was caught when trying to print *DEBUG-CONDITION* when entering the debugger. Printing was aborted and the UNBOUND-SLOT was stored in SB-DEBUG::*NESTED-DEBUG-CONDITION*.) (CELL-ERROR-NAME SB-DEBUG::*NESTED-DEBUG-CONDITION*) = DB-POSTMODERN::DBTABLE Leslie -- My personal blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Mon Feb 25 22:35:24 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 25 Feb 2008 17:35:24 -0500 Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: <61108.88.73.228.215.1203945729.squirrel@mail.stardawn.org> References: <61108.88.73.228.215.1203945729.squirrel@mail.stardawn.org> Message-ID: <884B245F-EBB5-4A78-915D-9C5A45231349@media.mit.edu> I just pushed a patch that should automatically manage two processes connecting to the same environment. You typically have to coordinate who is the master user and who is the slave so that you don't try to recover an environment that is already in use by another process. This causes that first process to fail, I believe, because it discovers it's pointers into the environment are 'corrupted'. I don't fully understand all the BDB particulars here, but I'm short on time to fix these things. Let me know if this works for you. The short story is that BDB is unlikely to 'just work' for a real system using multiple processes. You have to run deadlock detection (probably in a parallel process), you have to think about how to coordinate processes to clean up in the event of an error, you may have to sequence your startup, you have to think about how parallel use of the DB effects performance. No free lunches here. Anyway, for the details of why I added the patch see info on DB_REGISTER option to DBENV->open (db-env-open in lisp via open- controller in bdb-controller which is called from open-store - I've made DB_REGISTER the default). Berkeley DB considerations when using multiple processes: http://www.oracle.com/technology/documentation/berkeley-db/db/api_c/env_open.html http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/app.html Some discussions of multi-processor use of BDB: http://forums.oracle.com/forums/thread.jspa?messageID=1751065 http://forums.oracle.com/forums/thread.jspa?messageID=1436656 I'm not going to be able to track down the CL-SQL/SQLite issues anytime soon. Ian On Feb 25, 2008, at 8:22 AM, Leslie P. Polzer wrote: > >> To isolate the problem, can you verify the base case, connect two >> images to a fresh db? > > For BDB: > > (asdf:oos 'asdf:load-op 'elephant) > > (defpackage #:ele-test (:use :cl :elephant)) > > (in-package :ele-test) > > > (defpclass myclass () > ((testslot :accessor testslot :initarg testslot :index t))) > > (open-store '(:BDB "/tmp/db1") :recover t) > > (setf item (make-instance 'myclass)) > > (setf (testslot item) 5) > > > I load this in two images. When I then try to access TESTSLOT in the > former, I get the mentioned error. > > I'll try to make up something for the SQLite case, too. > > FYI, Postmodern doesn't show any problems with my setup. > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Tue Feb 26 04:03:45 2008 From: read at robertlread.net (Robert L. Read) Date: Mon, 25 Feb 2008 22:03:45 -0600 Subject: [elephant-devel] Fix for Unicode2 serializer In-Reply-To: <63287.88.73.228.215.1203948874.squirrel@mail.stardawn.org> References: <64232.88.73.215.222.1203525401.squirrel@mail.stardawn.org> <1203566319.21443.305.camel@penguin.yourdomain.com> <63028.88.73.219.154.1203671629.squirrel@mail.stardawn.org> <1203947668.21443.535.camel@penguin.yourdomain.com> <63287.88.73.228.215.1203948874.squirrel@mail.stardawn.org> Message-ID: <1203998625.21443.567.camel@penguin.yourdomain.com> On Mon, 2008-02-25 at 15:14 +0100, Leslie P. Polzer wrote: > Same for line 171. I wonder what's wrong? > I don't know why the patch didn't hash, but I was simply mistaken in not seeing that 171 is the same; I must have gone to the wrong line number or something. I apologize for the confusion. I have applied your patches to the main repository, after then tested green against postmodern. From read at robertlread.net Tue Feb 26 04:23:41 2008 From: read at robertlread.net (Robert L. Read) Date: Mon, 25 Feb 2008 22:23:41 -0600 Subject: [elephant-devel] Time for the 4.6 patch..... Message-ID: <1203999821.21443.579.camel@penguin.yourdomain.com> Dear Team, On Dec. 30th, Anton Kazennikov submitted the attached patch to make Elephant work with BDB 4.6. (THANKS!) I twiddled my thumbs about it, in part because I didn't know how many people were ready for 4.6. It now seems that the time is ripe to make the move from 4.5 to 4.6 in the main repository. Anton's patch stimulated some discussion about how to allow one to configure for different versions, however, I suggest that we simply move to 4.6 as the default now, unless someone volunteers to make the BDB version configurable. I now have BDB working on my x86_64, and am green under BDB 4.6 with this patch. Moving to 4.6 will mean that if you do an update of the latest repository (which will NOT be an official release for some months, I think), and you use BDB, you would have to upgrade your BDB to 4.6. If you object to this move, please reply by Thursday. Until then, if you want to use 4.6, you may want to use the attached patch (a darcs patch). -------------- next part -------------- A non-text attachment was scrubbed... Name: BDB-4.6.patch Type: text/x-patch Size: 18483 bytes Desc: not available URL: From leslie.polzer at gmx.net Tue Feb 26 08:50:33 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 26 Feb 2008 09:50:33 +0100 (CET) Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: <884B245F-EBB5-4A78-915D-9C5A45231349@media.mit.edu> References: <61108.88.73.228.215.1203945729.squirrel@mail.stardawn.org> <884B245F-EBB5-4A78-915D-9C5A45231349@media.mit.edu> Message-ID: <62922.88.73.196.49.1204015833.squirrel@mail.stardawn.org> > I don't fully understand all the BDB particulars here, but I'm short > on time to fix these things. Let me know if this works for you. The > short story is that BDB is unlikely to 'just work' for a real system > using multiple processes. You have to run deadlock detection > (probably in a parallel process), you have to think about how to > coordinate processes to clean up in the event of an error, you may > have to sequence your startup, you have to think about how parallel > use of the DB effects performance. No free lunches here. Ewww... guess I'll leave it at Postmodern then for the time being. It's good to defer this stuff to a separate high-level software package. I'll check out later whether your changes work. > Anyway, for the details of why I added the patch see info on > DB_REGISTER option to DBENV->open (db-env-open in lisp via open- > controller in bdb-controller which is called from open-store - I've > made DB_REGISTER the default). I see. An easy fix. > I'm not going to be able to track down the CL-SQL/SQLite issues > anytime soon. Understandable. As I see it, one would have to fiddle with timeouts and flags, and probably add some coordination code, too. And most likely a bunch of this would have to happen at the CLSQL level. I might look into it when I get a round tuit. Thanks! Leslie From leslie.polzer at gmx.net Tue Feb 26 12:43:43 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 26 Feb 2008 13:43:43 +0100 (CET) Subject: [elephant-devel] Tutorial revision Message-ID: <62962.88.73.196.49.1204029823.squirrel@mail.stardawn.org> About a month ago I sent a patch for the tutorial, stating that get-instance(s)-by-value uses EQUALP. Today I noticed that this is not correct, and in fact the used equality relation will be determined by the backend. Attached patch changes it to reflect this. Slightly related to this... it would definitely make sense to make this user-defined (at the cost of some performance, it seems) and add things like :LIMIT and :INCLUDE-SUBCLASSES-P to the MAP-* family resp. to the indexed object collectors. I'd volunteer for the latter two, but I'm not sure whether it would really make sense to integrate this now. After all, the upcoming query interface would make those additions obsolete, wouldn't it? Leslie -------------- next part -------------- A non-text attachment was scrubbed... Name: tutorial-equalp.patch Type: text/x-patch Size: 4385 bytes Desc: not available URL: From eslick at media.mit.edu Tue Feb 26 15:18:28 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 26 Feb 2008 10:18:28 -0500 Subject: [elephant-devel] Time for the 4.6 patch..... In-Reply-To: <1203999821.21443.579.camel@penguin.yourdomain.com> References: <1203999821.21443.579.camel@penguin.yourdomain.com> Message-ID: <3EEA3DE8-2199-408F-81EA-47D4426AA12F@media.mit.edu> Hi there, I'm tired of these BDB upgrade issues! :) I implemented a quick hack that makes the BDB version configurable via a my-config.sexp option. This patch builds on Anton's patch by putting BDB version-specific constants into their own package and importing them at load time in the db-bdb package based on the my- config.sexp option :berkeley-db-version Current versions supported are "4.5" and "4.6". This patch, which I have committed, passes my BDB tests on "4.5" so it doesn't break anything. It needs to be tested on 4.6, but that support is is based on Anton's patch and I verified that "4.6" symbols get loaded properly so we should be good to go. Please let me know if there are any snafus. Ian On Feb 25, 2008, at 11:23 PM, Robert L. Read wrote: > Dear Team, > On Dec. 30th, Anton Kazennikov submitted the attached patch to make > Elephant work with BDB 4.6. (THANKS!) I twiddled my thumbs about it, > in part because I didn't know how many people were ready for 4.6. > > It now seems that the time is ripe to make the move from 4.5 to 4.6 > in > the main repository. Anton's patch stimulated some discussion about > how > to allow one to configure for different versions, however, I suggest > that we simply move to 4.6 as the default now, unless someone > volunteers > to make the BDB version configurable. > > I now have BDB working on my x86_64, and am green under BDB 4.6 with > this patch. > > Moving to 4.6 will mean that if you do an update of the latest > repository (which will NOT be an official release for some months, I > think), and you use BDB, you would have to upgrade your BDB to 4.6. > > If you object to this move, please reply by Thursday. > > Until then, if you want to use 4.6, you may want to use the attached > patch (a darcs patch). > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Tue Feb 26 17:43:51 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 26 Feb 2008 18:43:51 +0100 (CET) Subject: [elephant-devel] Next issue, this time with Postmodern backend Message-ID: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> Internal Server Error Database error 53300: connection limit exceeded for non-superusers 0: (BACKTRACE 536870911 #) 1: (HUNCHENTOOT:GET-BACKTRACE #) 2: ((LAMBDA (COND)) #) 3: ((LAMBDA (COND)) #) 4: (SIGNAL #) 5: (ERROR CL-POSTGRES:DATABASE-ERROR) 6: (CL-POSTGRES::GET-ERROR #) 7: (CL-POSTGRES::GET-ERROR #) 8: (CL-POSTGRES::AUTHENTICATE # "mystic" "mystic" "mystic-items") 9: (CL-POSTGRES::INITIATE-CONNECTION #) 10: (CL-POSTGRES:OPEN-DATABASE "mystic-items" "mystic" "mystic" "127.0.0.1" 5433) 11: ((FLET SB-THREAD::WITH-MUTEX-THUNK)) 12: (SB-UNIX::CALL-WITH-LOCAL-INTERRUPTS # T) 13: ((FLET SB-UNIX::WITHOUT-INTERRUPTS-THUNK) T) 14: ((FLET SB-UNIX::RUN-WITHOUT-INTERRUPTS)) 15: (SB-UNIX::CALL-WITHOUT-INTERRUPTS #) 16: (SB-THREAD::CALL-WITH-MUTEX # #S(SB-THREAD:MUTEX :NAME NIL :%OWNER # :STATE 1) # T) 17: ((SB-PCL::FAST-METHOD DB-POSTMODERN::CONTROLLER-CONNECTION-FOR-THREAD (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER)) # # #) 18: ((SB-PCL::FAST-METHOD ELEPHANT::PERSISTENT-SLOT-BOUNDP (DB-POSTMODERN::POSTMODERN-STORE-CONTROLLER T T)) # # # # MYSTIC::BLOCKING-TEXT) 19: ((SB-PCL::FAST-METHOD MYSTIC:CLONE (MYSTIC:WORK)) # # #) 20: (MYSTIC::INSTANTIATE-WORK # 720) [...] Right after: (sb-thread:list-all-threads) (# #) Incredible? What's going on here? I'm subjecting this to a one-person load. All the PG workers pile up idly until the limit is hit. There must be something fishy either with PostgreSQL 8 (tested 8.2.6, 8.1.11) or Postmodern. I tend to put it on the latter, because GRAND-PRIX with CLSQL doesn't show any problems, nor do any workers accumulate. Transactions seem to be fine; I myself didn't run any, and the Elephant ones seem to be neatly delimited, having a matching COMMIT for each BEGIN. My first wild guess is that Postmodern does not close its connections in some cases. I suppose others are using the Postmodern backend with PG8? What are your results? You can check idle workers easily with something like "ps aux | grep postgres" Leslie From leslie.polzer at gmx.net Tue Feb 26 18:06:27 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 26 Feb 2008 19:06:27 +0100 (CET) Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> Message-ID: <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> > There must be something fishy either with PostgreSQL 8 (tested 8.2.6, 8.1.11) > or Postmodern. I tend to put it on the latter, because GRAND-PRIX with CLSQL > doesn't show any problems, nor do any workers accumulate. Update: with CLSQL and my application, it's the same thing. But I can't see where I'm doing anything that would cause this behaviour. Leslie From eslick at media.mit.edu Tue Feb 26 18:27:54 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 26 Feb 2008 13:27:54 -0500 Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> Message-ID: <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> Sounds like you're opening to many connections to the db. How many threads do you have? Each thread opens it's own socket to the server so if you have 100 threads, you have 100 connections to the server. This is probably because the client library is not thread-safe. But I'm speculating. You'll have to get input from Alex or Henrik on this and whether there is an easy solution. Ian On Feb 26, 2008, at 1:06 PM, Leslie P. Polzer wrote: > >> There must be something fishy either with PostgreSQL 8 (tested >> 8.2.6, 8.1.11) >> or Postmodern. I tend to put it on the latter, because GRAND-PRIX >> with CLSQL >> doesn't show any problems, nor do any workers accumulate. > > Update: with CLSQL and my application, it's the same thing. > But I can't see where I'm doing anything that would cause this > behaviour. > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Tue Feb 26 18:38:29 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 26 Feb 2008 19:38:29 +0100 (CET) Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> Message-ID: <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> > Sounds like you're opening to many connections to the db. How many > threads do you have? Each thread opens it's own socket to the server > so if you have 100 threads, you have 100 connections to the server. Well, like I pointed out in my first mail, the threads are very few. They make peak to five or a bunch more, but most of that won't access the database. The main problem is that the idle workers don't get reaped by PG... Leslie -- My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Tue Feb 26 18:42:44 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 26 Feb 2008 19:42:44 +0100 (CET) Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: <884B245F-EBB5-4A78-915D-9C5A45231349@media.mit.edu> References: <61108.88.73.228.215.1203945729.squirrel@mail.stardawn.org> <884B245F-EBB5-4A78-915D-9C5A45231349@media.mit.edu> Message-ID: <63253.88.73.196.49.1204051364.squirrel@mail.stardawn.org> > I just pushed a patch that should automatically manage two processes > connecting to the same environment. You typically have to coordinate > who is the master user and who is the slave so that you don't try to > recover an environment that is already in use by another process. > This causes that first process to fail, I believe, because it > discovers it's pointers into the environment are 'corrupted'. Doesn't do it for me. Opening two stores is fine, and I believe even reading. But writing messes the other process up. Without attempting recovery, everything seems to be fine. I must live with that for now, since I ran out of backend choices... Leslie From eslick at media.mit.edu Tue Feb 26 19:07:11 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 26 Feb 2008 14:07:11 -0500 Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: <63253.88.73.196.49.1204051364.squirrel@mail.stardawn.org> References: <61108.88.73.228.215.1203945729.squirrel@mail.stardawn.org> <884B245F-EBB5-4A78-915D-9C5A45231349@media.mit.edu> <63253.88.73.196.49.1204051364.squirrel@mail.stardawn.org> Message-ID: Hmmm...I believe that only the first to connect should attempt recovery. Much of these issues are BDB restrictions that one has to live with if you want to do more than the basic thing with them. It would be worthwhile for you to read up on BDB if you want to do the multi- process implementation. I don't know of anyone who has fully charted this territory and elephant certainly doesn't have specific features to help you deal (although you can launch a background db_deadlock process to free deadlocked threads/processes with an argument to open- store on your master process). Using the described patch I was able to have two writers, one from an Allegro image and one from an SBCL image both hammering (reading/ writing) on the same set of objects similar to your example. By mess up the other process I assume you are running into a crash of some kind and not data corruption (i.e. transaction design) issues Ian PS - Make sure you force a rebuild anytime you update. I've found asdf doesn't always get it right. On Feb 26, 2008, at 1:42 PM, Leslie P. Polzer wrote: > >> I just pushed a patch that should automatically manage two processes >> connecting to the same environment. You typically have to coordinate >> who is the master user and who is the slave so that you don't try to >> recover an environment that is already in use by another process. >> This causes that first process to fail, I believe, because it >> discovers it's pointers into the environment are 'corrupted'. > > Doesn't do it for me. Opening two stores is fine, and I believe even > reading. But writing messes the other process up. > > Without attempting recovery, everything seems to be fine. > I must live with that for now, since I ran out of backend choices... > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Tue Feb 26 19:18:02 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 26 Feb 2008 14:18:02 -0500 Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> Message-ID: <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> Ah, you have alot of short-lived threads? A quick look at pm- controller.lisp seems to indicate that pm-postmodern does not close connections when the thread closes. Perhaps you can add a function to do this explicitly or better, simply scan through active threads and cull any connections for which the opening thread has been terminated. If you do this only when a new connection is being opened, and the thread count is reasonable, this should be cheap. I think there are some lisp-independent thread abstractions in elephant-utils. If not, we might import one for (get- all-threads) and some unique identifier per thread. Check out pm-controller.lisp for details. Ian On Feb 26, 2008, at 1:38 PM, Leslie P. Polzer wrote: > >> Sounds like you're opening to many connections to the db. How many >> threads do you have? Each thread opens it's own socket to the server >> so if you have 100 threads, you have 100 connections to the server. > > Well, like I pointed out in my first mail, the threads are very few. > They make peak to five or a bunch more, but most of that won't access > the database. The main problem is that the idle workers don't get > reaped by PG... > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Tue Feb 26 19:27:44 2008 From: lists at infoway.net (lists at infoway.net) Date: Tue, 26 Feb 2008 14:27:44 -0500 Subject: [elephant-devel] Time for the 4.6 patch..... In-Reply-To: <3EEA3DE8-2199-408F-81EA-47D4426AA12F@media.mit.edu> References: <1203999821.21443.579.camel@penguin.yourdomain.com> <3EEA3DE8-2199-408F-81EA-47D4426AA12F@media.mit.edu> Message-ID: <4B0F8535-FCCD-4D71-A56D-3DD12280D89F@infoway.net> I just downloaded the latest Elephant and the latest BDB (4.6.21) on OS X Leopard with SBCL 1.0.14 with threads Got a compilation issue when ASDF loading Elephant: ... ; file: /Users/waldo/dev/lisp/elephant/src/elephant/classes.lisp ; in: DEFUN MAKE-PERSISTENT-SLOT-BOUNDP ; (LAMBDA (ELEPHANT::INSTANCE) ; (DECLARE (TYPE ELEPHANT:PERSISTENT-OBJECT ELEPHANT::INSTANCE)) ; (ELEPHANT::PERSISTENT-SLOT-BOUNDP (ELEPHANT::GET-CON ELEPHANT::INSTANCE) ; ELEPHANT::INSTANCE ELEPHANT::NAME)) ; --> FUNCTION LOCALLY SB-C::%FUNCALL MULTIPLE-VALUE-BIND LET UNLESS COND ; --> IF NOT IF ; ==> ; (TYPEP #:G861 'ELEPHANT:PERSISTENT-OBJECT) ; ; note: can't open-code test of unknown type PERSISTENT-OBJECT ; compiling (DEFMETHOD INITIALIZE-INTERNAL-SLOT-FUNCTIONS ...) ; /Users/waldo/dev/lisp/elephant/src/elephant/classes.fasl written ; compilation finished in 0:00:01 WARNING: COMPILE-FILE warned while performing # on #. STYLE-WARNING: Implicitly creating new generic function RECREATE-INSTANCE-USING- CLASS. The slot ELEPHANT::%INDEXED-CLASS is unbound in the object #. [Condition of type UNBOUND-SLOT] Restarts: 0: [RETRY] Retry performing # on #. 1: [ACCEPT] Continue, treating # on # as having been successful. 2: [ABORT] Return to SLIME's top level. 3: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-PCL::REAL-ADD-METHOD # # NIL) 1: (SB-PCL::REAL-ADD-NAMED-METHOD ELEPHANT::RECREATE-INSTANCE NIL (#) (ELEPHANT::INSTANCE &REST ELEPHANT::ARGS &KEY ELEPHANT::FROM-OID (ELEPHANT::SC ELEPHANT:*STORE-CONTROLLER*))) 2: (SB-PCL::LOAD-DEFMETHOD-INTERNAL STANDARD-METHOD ELEPHANT::RECREATE-INSTANCE NIL (#) (ELEPHANT::INSTANCE &REST ELEPHANT::ARGS &KEY ELEPHANT::FROM-OID (ELEPHANT::SC ELEPHANT:*STORE-CONTROLLER*)) (:FUNCTION # SB-PCL::PLIST (:ARG-INFO (1 . T))) #S(SB-C:DEFINITION-SOURCE-LOCATION ..)) 3: (SB-FASL::FOP-FUNCALL) 4: (SB-FASL::LOAD-FASL-GROUP #) 5: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK)) 6: (SB-UNIX::CALL-WITH-LOCAL-INTERRUPTS To which I hit 0: RETRY and then it continued just fine. It then ran all tests (420) clean. Great job! Thanks, Waldo On Feb 26, 2008, at 10:18 AM, Ian Eslick wrote: > Hi there, > > I'm tired of these BDB upgrade issues! :) > > I implemented a quick hack that makes the BDB version configurable > via a my-config.sexp option. This patch builds on Anton's patch by > putting BDB version-specific constants into their own package and > importing them at load time in the db-bdb package based on the my- > config.sexp option :berkeley-db-version > > Current versions supported are "4.5" and "4.6". This patch, which I > have committed, passes my BDB tests on "4.5" so it doesn't break > anything. It needs to be tested on 4.6, but that support is is > based on Anton's patch and I verified that "4.6" symbols get loaded > properly so we should be good to go. > > Please let me know if there are any snafus. > > Ian > > On Feb 25, 2008, at 11:23 PM, Robert L. Read wrote: > >> Dear Team, >> On Dec. 30th, Anton Kazennikov submitted the attached patch to make >> Elephant work with BDB 4.6. (THANKS!) I twiddled my thumbs about >> it, >> in part because I didn't know how many people were ready for 4.6. >> >> It now seems that the time is ripe to make the move from 4.5 to >> 4.6 in >> the main repository. Anton's patch stimulated some discussion >> about how >> to allow one to configure for different versions, however, I suggest >> that we simply move to 4.6 as the default now, unless someone >> volunteers >> to make the BDB version configurable. >> >> I now have BDB working on my x86_64, and am green under BDB 4.6 with >> this patch. >> >> Moving to 4.6 will mean that if you do an update of the latest >> repository (which will NOT be an official release for some months, I >> think), and you use BDB, you would have to upgrade your BDB to 4.6. >> >> If you object to this move, please reply by Thursday. >> >> Until then, if you want to use 4.6, you may want to use the attached >> patch (a darcs patch). >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Tue Feb 26 19:41:51 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 26 Feb 2008 20:41:51 +0100 (CET) Subject: [elephant-devel] Time for the 4.6 patch..... In-Reply-To: <4B0F8535-FCCD-4D71-A56D-3DD12280D89F@infoway.net> References: <1203999821.21443.579.camel@penguin.yourdomain.com> <3EEA3DE8-2199-408F-81EA-47D4426AA12F@media.mit.edu> <4B0F8535-FCCD-4D71-A56D-3DD12280D89F@infoway.net> Message-ID: <61541.88.73.196.49.1204054911.squirrel@mail.stardawn.org> > The slot ELEPHANT::%INDEXED-CLASS is unbound in the object > #. > [Condition of type UNBOUND-SLOT] That's unrelated to the db4.6 upgrade. Leslie From lists at infoway.net Tue Feb 26 20:00:37 2008 From: lists at infoway.net (lists at infoway.net) Date: Tue, 26 Feb 2008 15:00:37 -0500 Subject: [elephant-devel] Time for the 4.6 patch..... In-Reply-To: <61541.88.73.196.49.1204054911.squirrel@mail.stardawn.org> References: <1203999821.21443.579.camel@penguin.yourdomain.com> <3EEA3DE8-2199-408F-81EA-47D4426AA12F@media.mit.edu> <4B0F8535-FCCD-4D71-A56D-3DD12280D89F@infoway.net> <61541.88.73.196.49.1204054911.squirrel@mail.stardawn.org> Message-ID: I thought so, but I wasn't getting it with the last version of Elephant I downloaded on 2/21. Maybe there were other changes made since then that's causing this. - Waldo On Feb 26, 2008, at 2:41 PM, Leslie P. Polzer wrote: > >> The slot ELEPHANT::%INDEXED-CLASS is unbound in the object >> #. >> [Condition of type UNBOUND-SLOT] > > That's unrelated to the db4.6 upgrade. > > Leslie > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Tue Feb 26 20:28:33 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 26 Feb 2008 15:28:33 -0500 Subject: [elephant-devel] Time for the 4.6 patch..... In-Reply-To: References: <1203999821.21443.579.camel@penguin.yourdomain.com> <3EEA3DE8-2199-408F-81EA-47D4426AA12F@media.mit.edu> <4B0F8535-FCCD-4D71-A56D-3DD12280D89F@infoway.net> <61541.88.73.196.49.1204054911.squirrel@mail.stardawn.org> Message-ID: Would you mind trying again? I may have missed a 'check for unbound slot' case and believe I've patched it. -Ian On Feb 26, 2008, at 3:00 PM, lists at infoway.net wrote: > I thought so, but I wasn't getting it with the last version of > Elephant I downloaded on 2/21. Maybe there were other changes made > since then that's causing this. > > - Waldo > > On Feb 26, 2008, at 2:41 PM, Leslie P. Polzer wrote: > >> >>> The slot ELEPHANT::%INDEXED-CLASS is unbound in the object >>> #. >>> [Condition of type UNBOUND-SLOT] >> >> That's unrelated to the db4.6 upgrade. >> >> Leslie >> >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Tue Feb 26 23:13:42 2008 From: lists at infoway.net (lists at infoway.net) Date: Tue, 26 Feb 2008 18:13:42 -0500 Subject: [elephant-devel] Time for the 4.6 patch..... In-Reply-To: References: <1203999821.21443.579.camel@penguin.yourdomain.com> <3EEA3DE8-2199-408F-81EA-47D4426AA12F@media.mit.edu> <4B0F8535-FCCD-4D71-A56D-3DD12280D89F@infoway.net> <61541.88.73.196.49.1204054911.squirrel@mail.stardawn.org> Message-ID: <77BFCD83-8CEA-4BEF-AC99-D1EFE7E774C4@infoway.net> It's all green lights! Thanks, Waldo On Feb 26, 2008, at 3:28 PM, Ian Eslick wrote: > Would you mind trying again? I may have missed a 'check for unbound > slot' case and believe I've patched it. -Ian > > On Feb 26, 2008, at 3:00 PM, lists at infoway.net wrote: > >> I thought so, but I wasn't getting it with the last version of >> Elephant I downloaded on 2/21. Maybe there were other changes made >> since then that's causing this. >> >> - Waldo >> >> On Feb 26, 2008, at 2:41 PM, Leslie P. Polzer wrote: >> >>> >>>> The slot ELEPHANT::%INDEXED-CLASS is unbound in the object >>>> #. >>>> [Condition of type UNBOUND-SLOT] >>> >>> That's unrelated to the db4.6 upgrade. >>> >>> Leslie >>> >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Wed Feb 27 13:36:46 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 27 Feb 2008 14:36:46 +0100 (CET) Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> Message-ID: <61819.88.73.234.22.1204119406.squirrel@mail.stardawn.org> > Ah, you have alot of short-lived threads? A quick look at pm- > controller.lisp seems to indicate that pm-postmodern does not close > connections when the thread closes. Err... amazing that no one else experienced issues with this. After all not only people using short-lived threads will have their connections piling up. AFAICS the connections only get closed on CLOSE-STORE, which is independent from thread running time. Here's a remedy: (defun pm-reap (sc) (maphash (lambda (thread bookkeeper) (let ((alive-p (sb-thread:thread-alive-p thread))) (unless alive-p (cl-postgres:close-database (car bookkeeper)) (remhash thread (controller-db-table sc))))) (controller-db-table sc))) I'm not sure how to integrate this best, though. The threading stuff is not portable (do we have Bordeaux as dependency? Guess not), so others here would have to provide that stuff for their Lisps. Leslie From leslie.polzer at gmx.net Wed Feb 27 13:40:33 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 27 Feb 2008 14:40:33 +0100 (CET) Subject: [elephant-devel] SQLite locking error / BDB multi-image support In-Reply-To: References: <61108.88.73.228.215.1203945729.squirrel@mail.stardawn.org> <884B245F-EBB5-4A78-915D-9C5A45231349@media.mit.edu> <63253.88.73.196.49.1204051364.squirrel@mail.stardawn.org> Message-ID: <61886.88.73.234.22.1204119633.squirrel@mail.stardawn.org> > Hmmm...I believe that only the first to connect should attempt recovery. I'm not so sure about that (in fact I think DB_REGISTER is only there so you can specify DB_RECOVER for every process), but it doesn't really matter for me as long as it works. The only solution for me therefore is having only one process perform recovery, if needed, and make sure this one is run first. Yuk. I'm fond of a Tokyo Cabinet backend. > Much of these issues are BDB restrictions that one has to live with if > you want to do more than the basic thing with them. It would be > worthwhile for you to read up on BDB if you want to do the multi- > process implementation. I don't know of anyone who has fully charted > this territory and elephant certainly doesn't have specific features > to help you deal (although you can launch a background db_deadlock > process to free deadlocked threads/processes with an argument to open- > store on your master process). Yeah, I got bitten by that one afterwards. Amazing how fast a deadlock will occur when I don't run the detector... > By mess up the other process I assume you are running into a crash of > some kind and not data corruption (i.e. transaction design) issues Yes, correct. Thanks for all the ideas. Leslie From eslick at media.mit.edu Wed Feb 27 14:40:41 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 27 Feb 2008 09:40:41 -0500 Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> Message-ID: <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> Elephant-utils has some of the basic thread stuff we need - only need to import one or two operators. If we add much more we'll add bordeaux as a dependency. Do you want to wrap this up in a bow so we can commit it? (Did you test by adding this to the connection creation routine in pm-controller?) Ian Sent from my BlackBerry On Feb 26, 2008, at 2:18 PM, Ian Eslick wrote: > Ah, you have alot of short-lived threads? A quick look at pm- > controller.lisp seems to indicate that pm-postmodern does not close > connections when the thread closes. > > Perhaps you can add a function to do this explicitly or better, > simply scan through active threads and cull any connections for > which the opening thread has been terminated. If you do this only > when a new connection is being opened, and the thread count is > reasonable, this should be cheap. I think there are some lisp- > independent thread abstractions in elephant-utils. If not, we might > import one for (get-all-threads) and some unique identifier per > thread. > > Check out pm-controller.lisp for details. > > Ian > > On Feb 26, 2008, at 1:38 PM, Leslie P. Polzer wrote: > >> >>> Sounds like you're opening to many connections to the db. How many >>> threads do you have? Each thread opens it's own socket to the >>> server >>> so if you have 100 threads, you have 100 connections to the server. >> >> Well, like I pointed out in my first mail, the threads are very few. >> They make peak to five or a bunch more, but most of that won't access >> the database. The main problem is that the idle workers don't get >> reaped by PG... >> >> Leslie >> >> -- >> My personal blog: http://blog.viridian-project.de/ >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Wed Feb 27 15:07:45 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 27 Feb 2008 16:07:45 +0100 (CET) Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> Message-ID: <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> > Elephant-utils has some of the basic thread stuff we need - only need > to import one or two operators. If we add much more we'll add > bordeaux as a dependency. Do you want to wrap this up in a bow so we > can commit it? (Did you test by adding this to the connection > creation routine in pm-controller?) Here's the patch. For SBCL it does the intended thing, others will have to fill in their stuff. It's straightforward and works for me. Later it would probably be nice to actually re-use existing connections. Leslie -------------- next part -------------- A non-text attachment was scrubbed... Name: dbpm-reaper.patch Type: text/x-patch Size: 5597 bytes Desc: not available URL: From eslick at media.mit.edu Wed Feb 27 17:08:53 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 27 Feb 2008 12:08:53 -0500 Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> Message-ID: <9E55ADEE-2534-4506-A97C-29025FDB81C5@media.mit.edu> You know, I think people haven't run into this problem because may web servers maintain and reuse pools of worker threads - thus the # of connections = max concurrent threads rather than # of sessions/requests. Definitely good to have this in there: dangling sockets/file refs are bad! Ian On Feb 27, 2008, at 10:07 AM, Leslie P. Polzer wrote: >> Elephant-utils has some of the basic thread stuff we need - only need >> to import one or two operators. If we add much more we'll add >> bordeaux as a dependency. Do you want to wrap this up in a bow so we >> can commit it? (Did you test by adding this to the connection >> creation routine in pm-controller?) > > Here's the patch. For SBCL it does the intended thing, others will > have to fill in their stuff. > > It's straightforward and works for me. Later it would probably be > nice to actually re-use existing connections. > > Leslie reaper.patch>_______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Wed Feb 27 17:17:26 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 27 Feb 2008 18:17:26 +0100 (CET) Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <9E55ADEE-2534-4506-A97C-29025FDB81C5@media.mit.edu> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> <9E55ADEE-2534-4506-A97C-29025FDB81C5@media.mit.edu> Message-ID: <61168.88.73.234.22.1204132646.squirrel@mail.stardawn.org> > You know, I think people haven't run into this problem because may web > servers maintain and reuse pools of worker threads - thus the # of > connections = max concurrent threads rather than # of sessions/requests. Hunchentoot seems to spawn a new worker on every incoming request, or at least gets rid of idle pool workers very quickly. > Definitely good to have this in there: dangling sockets/file refs are > bad! CLSQL suffers from the same problem, I skimmed its code. The same fix can be applied there, I'll probably patch it later. Leslie From read at robertlread.net Thu Feb 28 03:47:32 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 27 Feb 2008 21:47:32 -0600 Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> Message-ID: <1204170452.21443.716.camel@penguin.yourdomain.com> Dear Leslie, Thank you for this patch. I will try to test it by this weekend, unless Ian wants to. However, if you will forgive me being dogmatic, it would be even better if this had a test case with it. For example, I don't think we have any good examples of producing lots of threads and whacking Elephant with them simultaneously. Such a test would both inspire confidence, and show people how Elephant can be used. I'm imagining something that spins up 100 simultaneous threads and does something to elephant from within each thread. Following the mantra "red, green, refactor", it would be lovely to see things seize up and then see that your patch fixes it. It would also be a starting point for other concurrency related tests in the future. (It might be useful for the CL-SQL guys, as well.) Would it take you a long time to produce a test that demonstrated the failure? I don't want to abuse your excellent work by asking more of you, but every bug is an opportunity for a new and deeper test. On Wed, 2008-02-27 at 16:07 +0100, Leslie P. Polzer wrote: > > Elephant-utils has some of the basic thread stuff we need - only need > > to import one or two operators. If we add much more we'll add > > bordeaux as a dependency. Do you want to wrap this up in a bow so we > > can commit it? (Did you test by adding this to the connection > > creation routine in pm-controller?) > > Here's the patch. For SBCL it does the intended thing, others will > have to fill in their stuff. > > It's straightforward and works for me. Later it would probably be > nice to actually re-use existing connections. > > Leslie > _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Thu Feb 28 08:45:14 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Thu, 28 Feb 2008 09:45:14 +0100 (CET) Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <1204170452.21443.716.camel@penguin.yourdomain.com> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> <1204170452.21443.716.camel@penguin.yourdomain.com> Message-ID: <62147.88.73.207.127.1204188314.squirrel@mail.stardawn.org> Dear Robert, you must really be a fan of TDD :) I'll see what I can do. Fortunately, the thread tests should be trivial to implement, as opposed to multi-process tests (but we have this in GRAND-PRIX). To make the tests easily accessible as a confidence-inspiring device for the everyday developer, a TEST-OP should be added to the main ASD file as well. Leslie From leslie.polzer at gmx.net Thu Feb 28 15:32:02 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Thu, 28 Feb 2008 16:32:02 +0100 (CET) Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <1204170452.21443.716.camel@penguin.yourdomain.com> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> <1204170452.21443.716.camel@penguin.yourdomain.com> Message-ID: <61984.88.73.207.127.1204212722.squirrel@mail.stardawn.org> > I'm imagining something that spins up 100 simultaneous threads and does > something to elephant from within each thread. Following the mantra > "red, green, refactor", it would be lovely to see things seize up and > then see that your patch fixes it. It would also be a starting point > for other concurrency related tests in the future. (It might be useful > for the CL-SQL guys, as well.) Note that this test case doesn't match the problem with Postgres which my patch resolves. The problem is not a high number of threads accessing the database, but a lot of dangling connections that do not get closed when the thread terminates. Although it would be most sensible to have the test you proposed, I'd rather make up the test case that actually fits here first and then maybe implement the other test. Leslie From eslick at media.mit.edu Thu Feb 28 17:14:08 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 28 Feb 2008 12:14:08 -0500 Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <61984.88.73.207.127.1204212722.squirrel@mail.stardawn.org> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> <1204170452.21443.716.camel@penguin.yourdomain.com> <61984.88.73.207.127.1204212722.squirrel@mail.stardawn.org> Message-ID: <5FBCE6A1-5199-41E2-BCA9-28A6FC8B6BEA@media.mit.edu> Sounds like we want to add bordeax threads to the elephant dependency list so we have a proper interface to threads for testing and some of the other stuff (thread-alive-p) that we currently hack up in the elephant utils. Any objections? Ian On Feb 28, 2008, at 10:32 AM, Leslie P. Polzer wrote: > >> I'm imagining something that spins up 100 simultaneous threads and >> does >> something to elephant from within each thread. Following the mantra >> "red, green, refactor", it would be lovely to see things seize up and >> then see that your patch fixes it. It would also be a starting point >> for other concurrency related tests in the future. (It might be >> useful >> for the CL-SQL guys, as well.) > > Note that this test case doesn't match the problem with Postgres which > my patch resolves. The problem is not a high number of threads > accessing > the database, but a lot of dangling connections that do not get closed > when the thread terminates. > > Although it would be most sensible to have the test you proposed, > I'd rather make up the test case that actually fits here first > and then maybe implement the other test. > > Leslie > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Fri Feb 29 03:26:32 2008 From: read at robertlread.net (Robert L. Read) Date: Thu, 28 Feb 2008 21:26:32 -0600 Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <62147.88.73.207.127.1204188314.squirrel@mail.stardawn.org> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> <1204170452.21443.716.camel@penguin.yourdomain.com> <62147.88.73.207.127.1204188314.squirrel@mail.stardawn.org> Message-ID: <1204255592.21443.781.camel@penguin.yourdomain.com> On Thu, 2008-02-28 at 09:45 +0100, Leslie P. Polzer wrote: > Dear Robert, > > you must really be a fan of TDD :) Yes. I have had the pleasure of pair programming with Kent Beck himself. But more to the point, the test suite in Elephant is one of our strongest assets. Where it is weak (like in threading) it should be shored up. > > I'll see what I can do. Fortunately, the thread tests should be trivial > to implement, as opposed to multi-process tests (but we have this > in GRAND-PRIX). > > To make the tests easily accessible as a confidence-inspiring device > for the everyday developer, a TEST-OP should be added to the main > ASD file as well. > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Fri Feb 29 03:34:57 2008 From: read at robertlread.net (Robert L. Read) Date: Thu, 28 Feb 2008 21:34:57 -0600 Subject: [elephant-devel] Next issue, this time with Postmodern backend In-Reply-To: <61984.88.73.207.127.1204212722.squirrel@mail.stardawn.org> References: <64275.88.73.196.49.1204047831.squirrel@mail.stardawn.org> <62502.88.73.196.49.1204049187.squirrel@mail.stardawn.org> <102E79B9-4C4A-4CAB-A4E2-A43B832E40D4@media.mit.edu> <62863.88.73.196.49.1204051109.squirrel@mail.stardawn.org> <8847320F-A9DD-48D6-8188-EA74D6149BC6@media.mit.edu> <21F659CC-26FA-44BA-8462-8FF4962A87A3@media.mit.edu> <64659.88.73.234.22.1204124865.squirrel@mail.stardawn.org> <1204170452.21443.716.camel@penguin.yourdomain.com> <61984.88.73.207.127.1204212722.squirrel@mail.stardawn.org> Message-ID: <1204256097.21443.785.camel@penguin.yourdomain.com> On Thu, 2008-02-28 at 16:32 +0100, Leslie P. Polzer wrote: > > I'm imagining something that spins up 100 simultaneous threads and does > > something to elephant from within each thread. Following the mantra > > "red, green, refactor", it would be lovely to see things seize up and > > then see that your patch fixes it. It would also be a starting point > > for other concurrency related tests in the future. (It might be useful > > for the CL-SQL guys, as well.) > > Note that this test case doesn't match the problem with Postgres which > my patch resolves. The problem is not a high number of threads accessing > the database, but a lot of dangling connections that do not get closed > when the thread terminates. > > Although it would be most sensible to have the test you proposed, > I'd rather make up the test case that actually fits here first > and then maybe implement the other test. Of course! Anyone writing a tested patch is always right! > > Leslie > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Fri Feb 29 10:45:27 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 29 Feb 2008 11:45:27 +0100 (CET) Subject: [elephant-devel] Backend musings Message-ID: <63182.88.73.244.33.1204281927.squirrel@mail.stardawn.org> I have been thinking about new backend possibilities, and would like to know your opinion. Tokyo Cabinet would require us to write a FFI for it. I don't know what it does to prevent deadlocks; probably nothing, as I have looked at its B tree code and have found just mutexes without any special handling whatsoever. Using Tokyo Tyrant, a server interface to a TC db, would have the same locking problems (if there are any, I'm going to ask the author for more clarification), a slight performance hit and possibly a less convoluted glueing code than the FFI solution. Plus, one could offload the storage to another machine, of course. Then there would be the possibility of using the B+Trees provided by CL-CONTAINERS. This would be a Lisp base upon which one could build upon, layering transactions and file storage above. What do you think? Leslie From eslick at media.mit.edu Fri Feb 29 13:48:30 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 29 Feb 2008 08:48:30 -0500 Subject: [elephant-devel] Backend musings In-Reply-To: <63182.88.73.244.33.1204281927.squirrel@mail.stardawn.org> References: <63182.88.73.244.33.1204281927.squirrel@mail.stardawn.org> Message-ID: It strikes me that Tokyo Cabinet's only major benefit is have a better licensing model. Getting it to the same state as BDB would be a significant project (and it doesn't support Windows, I believe). Mirroring Robert's sentiments, I think we'd be far better off spending limited developer cycles on a lisp-only solution running that solves both licensing, integration, performance, usability and other headaches experienced with our ffi or socket based solutions. An elephant data store has to: - Support persistent slot writes - Expose a persistent key-value store, indexed key-value store and iterator/cursor interfaces over them - Support a transaction model that guarantees the ACID properties - Work correctly in a threading model - Ideally would also work in a multi-process, shared-memory scenario I think the big design decisions for a lisp data store include: - Backing-store model (prevalence-style logging, binary paging, Rucksack style maps, etc) - Multi-threaded transaction support (page/object locking vs. per-txn replication) - Multi-process support? It would also be nice, eventually, to have a lisp server model where we can launch a lisp instance on the same or a remote machine to serve access to the underlying store. My latest thinking is that we can spend disk space to save time/ complexity and assume that we have lots of main memory available. This can allow us to avoid messing with fixed-sized pages and locking. So an initial approach might be the following: - BTree-style nodes and slot value writes are duplicated in a per- transaction memory log. (btree ops are compressed into edit operations, but the altered page is also duplicated for later reads) - Any reads to these subsystems first look up data in this transaction cache - We serialize commit ops and cancel active transactions that have read or written data written by the commiting transaction. - Thus, on commit, we just write to the log file, update the btree pages, and write to the slot store We can make serialization of btree nodes easier by just serializing the lisp structure and having bins of differently sized disk regions to write them to rather than maintaining fixed sized pages. Later, we could choose to enhance the system by converting to a low-level read/ write into the C byte stream that we fetch from files. Just a few thoughts... Ian PS - I don't see B+Trees in CL-CONTAINERS. Is it in there with another name? Also, we need a B+Tree that actually is mapped to a disk file rather than into memory as seems to be the case with most of cl-containers On Feb 29, 2008, at 5:45 AM, Leslie P. Polzer wrote: > > I have been thinking about new backend possibilities, and > would like to know your opinion. > > Tokyo Cabinet would require us to write a FFI for it. > I don't know what it does to prevent deadlocks; probably > nothing, as I have looked at its B tree code and have > found just mutexes without any special handling > whatsoever. > > iI'm going to ask the author for more clarification), > a slight performance hit and possibly a less convoluted > glueing code than the FFI solution. Plus, one could > offload the storage to another machine, of course. > > Then there would be the possibility of using the > B+Trees provided by CL-CONTAINERS. This would > be a Lisp base upon which one could build upon, > layering transactions and file storage above. > > What do you think? > > Leslie > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Fri Feb 29 18:53:32 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 29 Feb 2008 19:53:32 +0100 (CET) Subject: [elephant-devel] Backend musings In-Reply-To: References: <63182.88.73.244.33.1204281927.squirrel@mail.stardawn.org> Message-ID: <63570.88.73.244.33.1204311212.squirrel@mail.stardawn.org> > It strikes me that Tokyo Cabinet's only major benefit is have a better > licensing model. Getting it to the same state as BDB would be a > significant project (and it doesn't support Windows, I believe). I thought that TC might have a better deadlock handling (after all there are other concurrency models that don't have that problem), but I might have been mistaken. > Mirroring Robert's sentiments, I think we'd be far better off spending > limited developer cycles on a lisp-only solution running that solves > both licensing, integration, performance, usability and other > headaches experienced with our ffi or socket based solutions. Guess I'm with you both here. > - Support persistent slot writes Isn't this trivial with the current serializer architecture? > - Expose a persistent key-value store, indexed key-value store and > iterator/cursor interfaces over them Comes for free with every B(+)-tree, I guess. > - Support a transaction model that guarantees the ACID properties > - Work correctly in a threading model > - Ideally would also work in a multi-process, shared-memory scenario > > I think the big design decisions for a lisp data store include: > - Backing-store model (prevalence-style logging, binary paging, > Rucksack style maps, etc) > > - Multi-threaded transaction support (page/object locking vs. per-txn > replication) OL seems more simple and therefore better to me. I don't know what per-txn replication is, and neither anything about the backing store methods you mentioned. > - Multi-process support? ...should definitely be included. My model of having separate processes for frontend and admin backend is not too uncommon. > My latest thinking is that we can spend disk space to save time/ > complexity and assume that we have lots of main memory available. > This can allow us to avoid messing with fixed-sized pages and locking. Yup. > - We serialize commit ops and cancel active transactions that have > read or written data What is meant by "serializing" here? > We can make serialization of btree nodes easier by just serializing > the lisp structure and having bins of differently sized disk regions > to write them to rather than maintaining fixed sized pages. Later, we > could choose to enhance the system by converting to a low-level read/ > write into the C byte stream that we fetch from files. Err... "C byte stream"? > PS - I don't see B+Trees in CL-CONTAINERS. Is it in there with > another name? It should be the QUAD-TREE for all I know. > Also, we need a B+Tree that actually is mapped to a > disk file rather than into memory as seems to be the case with most of > cl-containers Yes. I'm not sure how easily extensible the CL-CONTAINERS tree is, but it might be a good base to start with. Leslie From henrik at evahjelte.com Fri Feb 29 19:31:47 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 29 Feb 2008 20:31:47 +0100 Subject: [elephant-devel] Backend musings In-Reply-To: References: <63182.88.73.244.33.1204281927.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60802291131y73f06ba7od46f78e5995651dd@mail.gmail.com> On Fri, Feb 29, 2008 at 2:48 PM, Ian Eslick wrote: > An elephant data store has to: > - Support persistent slot writes > - Expose a persistent key-value store, indexed key-value store and > iterator/cursor interfaces over them > - Support a transaction model that guarantees the ACID properties > - Work correctly in a threading model > - Ideally would also work in a multi-process, shared-memory scenario To me the last point is is very important. Even more important than multithreading, and you will probably get multithreading support at the same time if you implement it correctly. A database that can't be used from several computers is not much of a database. It can be tricky, but should be doable. If you think conceptually dividing client and server code, and make a generic interface between these parts. Then you can have different ways of implementing the client-server communication protocol. For a single process application, the client-server protocol can of course be simple function calls between the server code and the client code in the same lisp process. Then for a multi process setup you can probably want different ways of communicating. One "simple" and straightforward socket interface that works on all platforms, then perhaps you (at least I will) want to add another performance optimized communication protocol that might be less generic. From the start, what need to be done is to mentally see the different client and server parts, and make sure no state (variables and memory areas) are shared between them. Perhaps have an eye on the design so if possible trying to keep the number of roundtrips betwen client and server down, Then, it probably becomes easy to add a basic simple client-server socket solution. As an extra bonus, having a clear client/server mindset from the start might make the design clearer and better. > > I think the big design decisions for a lisp data store include: > - Backing-store model (prevalence-style logging, binary paging, > Rucksack style maps, etc) > - Multi-threaded transaction support (page/object locking vs. per-txn > replication) > - Multi-process support? Strong vote for... > It would also be nice, eventually, to have a lisp server model where > we can launch a lisp instance on the same or a remote machine to serve > access to the underlying store. See my comments above. > My latest thinking is that we can spend disk space to save time/ > complexity and assume that we have lots of main memory available. > This can allow us to avoid messing with fixed-sized pages and locking. Yes, don't optimize these things too early. > So an initial approach might be the following: > - BTree-style nodes and slot value writes are duplicated in a per- > transaction memory log. > (btree ops are compressed into edit operations, but the altered > page is also duplicated for later reads) > - Any reads to these subsystems first look up data in this transaction > cache > - We serialize commit ops and cancel active transactions that have > read or written data > written by the commiting transaction. > - Thus, on commit, we just write to the log file, update the btree > pages, and write to the slot store In this model, isn't there a risk of problems: Transaction A starts, the world is in a coherent state and you want to get data from different b-trees that are related. Transaction B starts, does his things quick and writes new things to the database. But now, if transaction A wants to read some data that B has updated, it will suddently get the brand new fresh data from transaction B, which was not intended since A wants to read all data as the world looked from the beginning. Contrasted by a version numbered transaction model that don't overwrite data (like Rucksack does), transaction A can always get coherent data since B doesn't overwrite the db, it just write a new version. Am I right, is this a problem? > - Thus, on commit, we just write to the log file, update the btree > pages, and write to the slot store If we do a client/server solution, these kind of things can be performance optimized perhaps by splitting it into different subprocesses. The server can maybe just update the log file, save data in its cache, then return OK/Pending to the client, so the client can continue. Then the server process can continue doing the rest of the work until it is finished. This could be done with a multithreaded design as well. But, we can leave it for later and start out with something simple. But it could be something to think about in the design. > > We can make serialization of btree nodes easier by just serializing > the lisp structure and having bins of differently sized disk regions > to write them to rather than maintaining fixed sized pages. Later, we > could choose to enhance the system by converting to a low-level read/ > write into the C byte stream that we fetch from files. Agree. Better hunt performance later. It is not obvious what is the most limiting factor for performance from the start. About other B+tree implementations, there is one in rucksack as well, memory based but a clean design that can be adapted for disk usage. It is also not astrophysics to make one from scratch, or translate from one of the many implementations for other languages. About project management, how do we discuss the design and what needs to be done in the best way? Mails like these? The only problem with that is that it might be difficult to contribute to the discussion when you don't have the time to respond to these mails, so a design document or wiki could also be good because you can contribute more when you have time rather than when the discussion is ongoing on the list. Or is the mailing list enough, I am not sure... /Henrik From lists at infoway.net Fri Feb 29 19:47:41 2008 From: lists at infoway.net (lists at infoway.net) Date: Fri, 29 Feb 2008 14:47:41 -0500 Subject: [elephant-devel] Representational Question Message-ID: Hi all, As I'm further exploring more and more things to do in Elephant and Lisp, I think we're ready to start migrating some of our RoR apps over, if not just as an exercise, we'll someday migrate them to production. Since we all have a very strong and hard-headed background on MySQL and relational models, it's been extremely difficult for us to migrate away from that mentality and think of objects and some of Elephant's terminology such as class indexes, which kind of confuse us into thinking that a class index allows us to look at a set of objects in a similar way as a MySQL table. I've read and seen in the src the beginning efforts to building a query system into Elephant. That would be great and as our efforts approach that phase, we hope to contribute to it. So, in this email, first I will ask for advise as to how to best represent the structure of our objects/classes and indices in Elephant in order to ultimately be able to query the data. Again, I'm not going to ask for the querying strategy (just yet) but ultimately, we will need to be able to answer queries like this. Obviously I don't expect anyone to give me the full representation of this, but any advise/ hints as to best represent them will help greatly. We have a database with many related tables. For simplicity purposes, we'll describe a simplified scenario. We have a table with people information (e.g. first,last names, date of birth, and gender). We have a linked table with each person's addresses (multiple addresses in case they moved. Each address is timestamped so the most recent address is the current address). Then, each person may be subscribed to one or more health insurance plans, and so there is a table linking each person to one or more health insurance plans (and a table that defines the health insurance plans) Now, each person may select up to N preferred medical offices where they would like to receive treatment. Again, there is a table that links the person with one or more medical office. Needless to say, there is a table of medical offices. Each medical office is also linked to a timestamped address table, where the most recent address is the current one (in the event the office moves). To further expand on the issue, each office has one or more doctors rendering services, so there is a table that links the offices to the doctors, and of course, there is a table of doctors that contains basic information, such as fname, lname, and gender. Last, but not least, a doctor may be specialized in multiple areas, so there is a table that links doctors to all the specialties they have been certified on, and thus there is yet another table that lists all possible specialties. Now, assuming I was able to explain the scenario correctly, we then have users asking the system for information such as: "List all people (subscribers), who are male and live in zip code 33012 who are contracted under Health Insurance Plan A that have selected (as their preferred medical office) medical offices with male cardiologists that work within 10 miles of 33012 zip code or in MIAMI- DADE county and whose office names contain the sequence of letters 'HEAL'" The way we see it, the concept of tables disappears and so do the tables that provide many-to-many joins. So, we end up with some classes such as "Person" which contains a reference to a list of "Address" objects, and a list of preferred "Medical-Office" objects, where each Medical-Office object has a list of Doctor objects and each Doctor has a list of Specialty objects, etc, etc. Now, we assume that each of these classes will need to maintain multiple indices, such as the Person class being index on first name, last name, dob, gender, among others. The Address class indexed on zip code, county name, among others, and so on and so forth. The querying is one problem. The data representation is another. We think it's clear that we should have, as an example, a Person class. However, the representation of the links between a Person and its Addresses or Medical-Offices is not 100% clear. If we represent them as a slot in the Person class, where this slot would be a List or a set of references to the Address class, then in order for us to query on those, means that we always need to fetch all objects in those slots in order to apply any search criteria, which seems like a bottleneck. If that was the solution, I assume we could implement logic such that Addresses are pushed into the list, so that the most recent address is in the CAR, so we wouldn't necessarily need to read the entire list of Addresses for each member, but just fetch the CAR of the slot. Now, onto the second question. One of the other requirements we have is that we need to keep an audit log of data changes. The way we do it in RoR is relatively simple. We fetch an object from the DB and present it on the browser. When the user submits, we fetch another fresh copy from the DB and if the timestamps are the same (meaning no one else changed the record) we compare changes to the object's attributes (slots). If there are any differences, we save the changes (we're trying to avoid unnecessary trips to the DB) and if the changes are saved successfully, we write a log of ONLY the attributes that were changed (which is pretty trivial in Ruby). From what we've read in Elephant's manual, this seems harder because we don't want to work directly off the Elephant object but a memory copy while the user takes his/her time in the browser and after submitting, we would take the changes and commit them to the Elephant object. Makes me think that we would need to classes for each object (one with and one without the persistent metaclass). The other problem would be how to "easily" have two objects introspect themselves and spit out the slots that changed between the two. Are we looking at this incorrectly? Any advise would be greatly appreciated. Thanks, Waldo From eslick at media.mit.edu Fri Feb 29 19:48:24 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 29 Feb 2008 14:48:24 -0500 Subject: [elephant-devel] Backend musings In-Reply-To: <50e8e4f60802291131y73f06ba7od46f78e5995651dd@mail.gmail.com> References: <63182.88.73.244.33.1204281927.squirrel@mail.stardawn.org> <50e8e4f60802291131y73f06ba7od46f78e5995651dd@mail.gmail.com> Message-ID: <403E14DF-979A-42D4-9EBA-E3869E5A1D32@media.mit.edu> On Feb 29, 2008, at 2:31 PM, Henrik Hjelte wrote: > On Fri, Feb 29, 2008 at 2:48 PM, Ian Eslick > wrote: >> An elephant data store has to: >> - Support persistent slot writes >> - Expose a persistent key-value store, indexed key-value store and >> iterator/cursor interfaces over them >> - Support a transaction model that guarantees the ACID properties >> - Work correctly in a threading model >> - Ideally would also work in a multi-process, shared-memory scenario > > To me the last point is is very important. Even more important than > multithreading, and you will probably get multithreading support at > the same time if you implement it correctly. A database that can't be > used from several computers is not much of a database. It can be > tricky, but should be doable. > > If you think conceptually dividing client and server code, and make a > generic interface between these parts. Then you can have different > ways of implementing the client-server communication protocol. For a > single process application, the client-server protocol can of course > be simple function calls between the server code and the client code > in the same lisp process. Then for a multi process setup you can > probably want different ways of communicating. One "simple" and > straightforward socket interface that works on all platforms, then > perhaps you (at least I will) want to add another performance > optimized communication protocol that might be less generic. From the > start, what need to be done is to mentally see the different client > and server parts, and make sure no state (variables and memory areas) > are shared between them. Perhaps have an eye on the design so if > possible trying to keep the number of roundtrips betwen client and > server down, Then, it probably becomes easy to add a basic simple > client-server socket solution. > > As an extra bonus, having a clear client/server mindset from the start > might make the design clearer and better. What would be the goal of having a lisp-only client/server database vs. using the existing postmodern model or cl-perec or one of the other ORM solutions that exists? Wouldn't the socket protocol tends to swamp any server-side performance benefits? Do you how locking is handled and conflicts detected in something like Postgresql? >> >> I think the big design decisions for a lisp data store include: >> - Backing-store model (prevalence-style logging, binary paging, >> Rucksack style maps, etc) >> - Multi-threaded transaction support (page/object locking vs. per-txn >> replication) >> - Multi-process support? > > Strong vote for... > >> It would also be nice, eventually, to have a lisp server model where >> we can launch a lisp instance on the same or a remote machine to >> serve >> access to the underlying store. > > See my comments above. > >> My latest thinking is that we can spend disk space to save time/ >> complexity and assume that we have lots of main memory available. >> This can allow us to avoid messing with fixed-sized pages and >> locking. > > Yes, don't optimize these things too early. > >> So an initial approach might be the following: >> - BTree-style nodes and slot value writes are duplicated in a per- >> transaction memory log. >> (btree ops are compressed into edit operations, but the altered >> page is also duplicated for later reads) >> - Any reads to these subsystems first look up data in this >> transaction >> cache >> - We serialize commit ops and cancel active transactions that have >> read or written data >> written by the commiting transaction. >> - Thus, on commit, we just write to the log file, update the btree >> pages, and write to the slot store > > In this model, isn't there a risk of problems: > Transaction A starts, the world is in a coherent state and you want to > get data from different b-trees that are related. > Transaction B starts, does his things quick and writes new things to > the database. > But now, if transaction A wants to read some data that B has updated, > it will suddently get the brand new fresh data from transaction B, > which was not intended since A wants to read all data as the world > looked from the beginning. Transaction B doesn't write to the database until it's committing, at which time transaction A is aborted because B wrote over data that A read. I'm describing something very similar to versioning, except not at the object duplicate-on-write model which doesn't fit with the current BDB architecture. > Contrasted by a version numbered transaction model that don't > overwrite data (like Rucksack does), transaction A can always get > coherent data since B doesn't overwrite the db, it just write a new > version. Versioning, or duplication, makes sense while the transactions are occurring. If there is a conflict between A and B then in the versioned case you need to decide which version will be the surviving version for a following transaction C. > Am I right, is this a problem? > >> - Thus, on commit, we just write to the log file, update the btree >> pages, and write to the slot store > > If we do a client/server solution, these kind of things can be > performance optimized perhaps by splitting it into different > subprocesses. > The server can maybe just update the log file, save data in its cache, > then return OK/Pending to the client, so the client can continue. Then > the server process can continue doing the rest of the work until it is > finished. This could be done with a multithreaded design as well. But, > we can leave it for later and start out with something simple. But it > could be something to think about in the design. >> >> We can make serialization of btree nodes easier by just serializing >> the lisp structure and having bins of differently sized disk regions >> to write them to rather than maintaining fixed sized pages. Later, >> we >> could choose to enhance the system by converting to a low-level read/ >> write into the C byte stream that we fetch from files. > > Agree. Better hunt performance later. It is not obvious what is the > most limiting factor for performance from the start. > > About other B+tree implementations, there is one in rucksack as well, > memory based but a clean design that can be adapted for disk usage. It > is also not astrophysics to make one from scratch, or translate from > one of the many implementations for other languages. > > About project management, how do we discuss the design and what needs > to be done in the best way? Mails like these? The only problem with > that is that it might be difficult to contribute to the discussion > when you don't have the time to respond to these mails, so a design > document or wiki could also be good because you can contribute more > when you have time rather than when the discussion is ongoing on the > list. Or is the mailing list enough, I am not sure... I think if there is a critical mass of people interested, then we should start a design document on the Trac wiki. I think all you need is a common-lisp.net account and be giving a Trac password. > /Henrik > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Fri Feb 29 20:36:58 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 29 Feb 2008 15:36:58 -0500 Subject: [elephant-devel] Representational Question In-Reply-To: References: Message-ID: <6A96C939-ECF2-4FC6-B12A-106B3AF64CE5@media.mit.edu> Hi Waldo, Why do you want to migrate to Elephant for production and not stick with something like CL-SQL or cl-perec on top of a relational database so you get all the facilities that you're familiar with? Also, please don't expect a query system anytime soon. Finishing it is not in my critical path right now and no one else has stepped up and volunteered to lead or help with it. As for your query problem, I think the SQL solution for queries like that is likely to be faster in the end than putting this into Elephant. Elephant is not intended or designed to support efficient relational operations. That's what relational DBs are for! :) Wait until the next big update to elephant before you go too far down this road, I'm hoping that some new features I'm planning at least make this a little bit easier. For your second question, if you are going to save/store the objects in bulk, you can just use standard classes. Then you can have a transaction to fetch/diff/write the composite object to ensure atomicity of updates. This diff would also produce your log. However that means that you lose the indexing capability of persistent objects. Ian On Feb 29, 2008, at 2:47 PM, lists at infoway.net wrote: > Hi all, > > As I'm further exploring more and more things to do in Elephant and > Lisp, I think we're ready to start migrating some of our RoR apps > over, if not just as an exercise, we'll someday migrate them to > production. > > Since we all have a very strong and hard-headed background on MySQL > and relational models, it's been extremely difficult for us to > migrate away from that mentality and think of objects and some of > Elephant's terminology such as class indexes, which kind of confuse > us into thinking that a class index allows us to look at a set of > objects in a similar way as a MySQL table. > > I've read and seen in the src the beginning efforts to building a > query system into Elephant. That would be great and as our efforts > approach that phase, we hope to contribute to it. > > So, in this email, first I will ask for advise as to how to best > represent the structure of our objects/classes and indices in > Elephant in order to ultimately be able to query the data. Again, > I'm not going to ask for the querying strategy (just yet) but > ultimately, we will need to be able to answer queries like this. > Obviously I don't expect anyone to give me the full representation > of this, but any advise/hints as to best represent them will help > greatly. > > We have a database with many related tables. For simplicity > purposes, we'll describe a simplified scenario. We have a table with > people information (e.g. first,last names, date of birth, and > gender). We have a linked table with each person's addresses > (multiple addresses in case they moved. Each address is timestamped > so the most recent address is the current address). Then, each > person may be subscribed to one or more health insurance plans, and > so there is a table linking each person to one or more health > insurance plans (and a table that defines the health insurance plans) > > Now, each person may select up to N preferred medical offices where > they would like to receive treatment. Again, there is a table that > links the person with one or more medical office. Needless to say, > there is a table of medical offices. Each medical office is also > linked to a timestamped address table, where the most recent address > is the current one (in the event the office moves). To further > expand on the issue, each office has one or more doctors rendering > services, so there is a table that links the offices to the doctors, > and of course, there is a table of doctors that contains basic > information, such as fname, lname, and gender. Last, but not least, > a doctor may be specialized in multiple areas, so there is a table > that links doctors to all the specialties they have been certified > on, and thus there is yet another table that lists all possible > specialties. > > Now, assuming I was able to explain the scenario correctly, we then > have users asking the system for information such as: > > "List all people (subscribers), who are male and live in zip code > 33012 who are contracted under Health Insurance Plan A that have > selected (as their preferred medical office) medical offices with > male cardiologists that work within 10 miles of 33012 zip code or in > MIAMI-DADE county and whose office names contain the sequence of > letters 'HEAL'" > > The way we see it, the concept of tables disappears and so do the > tables that provide many-to-many joins. So, we end up with some > classes such as "Person" which contains a reference to a list of > "Address" objects, and a list of preferred "Medical-Office" objects, > where each Medical-Office object has a list of Doctor objects and > each Doctor has a list of Specialty objects, etc, etc. > > Now, we assume that each of these classes will need to maintain > multiple indices, such as the Person class being index on first > name, last name, dob, gender, among others. The Address class > indexed on zip code, county name, among others, and so on and so > forth. > > The querying is one problem. The data representation is another. We > think it's clear that we should have, as an example, a Person class. > However, the representation of the links between a Person and its > Addresses or Medical-Offices is not 100% clear. If we represent them > as a slot in the Person class, where this slot would be a List or a > set of references to the Address class, then in order for us to > query on those, means that we always need to fetch all objects in > those slots in order to apply any search criteria, which seems like > a bottleneck. If that was the solution, I assume we could implement > logic such that Addresses are pushed into the list, so that the most > recent address is in the CAR, so we wouldn't necessarily need to > read the entire list of Addresses for each member, but just fetch > the CAR of the slot. > > Now, onto the second question. One of the other requirements we have > is that we need to keep an audit log of data changes. The way we do > it in RoR is relatively simple. We fetch an object from the DB and > present it on the browser. When the user submits, we fetch another > fresh copy from the DB and if the timestamps are the same (meaning > no one else changed the record) we compare changes to the object's > attributes (slots). If there are any differences, we save the > changes (we're trying to avoid unnecessary trips to the DB) and if > the changes are saved successfully, we write a log of ONLY the > attributes that were changed (which is pretty trivial in Ruby). > > From what we've read in Elephant's manual, this seems harder because > we don't want to work directly off the Elephant object but a memory > copy while the user takes his/her time in the browser and after > submitting, we would take the changes and commit them to the > Elephant object. Makes me think that we would need to classes for > each object (one with and one without the persistent metaclass). The > other problem would be how to "easily" have two objects introspect > themselves and spit out the slots that changed between the two. > > Are we looking at this incorrectly? Any advise would be greatly > appreciated. > > Thanks, > Waldo > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lists at infoway.net Fri Feb 29 23:50:31 2008 From: lists at infoway.net (lists at infoway.net) Date: Fri, 29 Feb 2008 18:50:31 -0500 Subject: [elephant-devel] Representational Question In-Reply-To: <6A96C939-ECF2-4FC6-B12A-106B3AF64CE5@media.mit.edu> References: <6A96C939-ECF2-4FC6-B12A-106B3AF64CE5@media.mit.edu> Message-ID: <1B1F38CE-9239-4083-80B6-685318384C28@infoway.net> Hi Ian, Thanks for the prompt response. I know the querying facility is not necessarily a priority at this time, but will someday become a reality :) To tell you the truth, we haven't really had any direct experience with Elephant in production or larger-scale type projects. However, we do feel that the whole concept of object prevalence given the complexity of the overall data model would make Elephant a more appropriate framework than continuing the relational path (maybe we're just wrong and Elephant is not best suited for this at all). As it is, we currently need to do a lot of work to maintain all the data relations and integrity in the current system and hopefully working only with the object models would make things easier and more "maintainable". Granted, I agree that at this moment, it's a lot easier to formulate those queries in SQL, but I'd like to at least be able to setup a parallel model and migrate data over so we could compare performance (we're not even going to talk about the complexity/ difficulty of querying in Elephant, since we know that at this stage, it is much more complex than SQL queries). I hope I'm not wrong, but definitely your opinion is worth more since you (et al) know a lot more about this than us. As for the second question, the answer is no. The objects would not be stored in bulk. The idea is to keep an audit log of user-initiated changes on individual entities (e.g. changing a Person's address, or correcting a name, or assigning a health insurance plan, etc). Thanks, Waldo On Feb 29, 2008, at 3:36 PM, Ian Eslick wrote: > Hi Waldo, > > Why do you want to migrate to Elephant for production and not stick > with something like CL-SQL or cl-perec on top of a relational > database so you get all the facilities that you're familiar with? > > Also, please don't expect a query system anytime soon. Finishing it > is not in my critical path right now and no one else has stepped up > and volunteered to lead or help with it. > > As for your query problem, I think the SQL solution for queries like > that is likely to be faster in the end than putting this into > Elephant. Elephant is not intended or designed to support efficient > relational operations. That's what relational DBs are for! :) > > Wait until the next big update to elephant before you go too far > down this road, I'm hoping that some new features I'm planning at > least make this a little bit easier. > > For your second question, if you are going to save/store the objects > in bulk, you can just use standard classes. Then you can have a > transaction to fetch/diff/write the composite object to ensure > atomicity of updates. This diff would also produce your log. > However that means that you lose the indexing capability of > persistent objects. > > Ian > > On Feb 29, 2008, at 2:47 PM, lists at infoway.net wrote: > >> Hi all, >> >> As I'm further exploring more and more things to do in Elephant and >> Lisp, I think we're ready to start migrating some of our RoR apps >> over, if not just as an exercise, we'll someday migrate them to >> production. >> >> Since we all have a very strong and hard-headed background on MySQL >> and relational models, it's been extremely difficult for us to >> migrate away from that mentality and think of objects and some of >> Elephant's terminology such as class indexes, which kind of confuse >> us into thinking that a class index allows us to look at a set of >> objects in a similar way as a MySQL table. >> >> I've read and seen in the src the beginning efforts to building a >> query system into Elephant. That would be great and as our efforts >> approach that phase, we hope to contribute to it. >> >> So, in this email, first I will ask for advise as to how to best >> represent the structure of our objects/classes and indices in >> Elephant in order to ultimately be able to query the data. Again, >> I'm not going to ask for the querying strategy (just yet) but >> ultimately, we will need to be able to answer queries like this. >> Obviously I don't expect anyone to give me the full representation >> of this, but any advise/hints as to best represent them will help >> greatly. >> >> We have a database with many related tables. For simplicity >> purposes, we'll describe a simplified scenario. We have a table >> with people information (e.g. first,last names, date of birth, and >> gender). We have a linked table with each person's addresses >> (multiple addresses in case they moved. Each address is timestamped >> so the most recent address is the current address). Then, each >> person may be subscribed to one or more health insurance plans, and >> so there is a table linking each person to one or more health >> insurance plans (and a table that defines the health insurance plans) >> >> Now, each person may select up to N preferred medical offices where >> they would like to receive treatment. Again, there is a table that >> links the person with one or more medical office. Needless to say, >> there is a table of medical offices. Each medical office is also >> linked to a timestamped address table, where the most recent >> address is the current one (in the event the office moves). To >> further expand on the issue, each office has one or more doctors >> rendering services, so there is a table that links the offices to >> the doctors, and of course, there is a table of doctors that >> contains basic information, such as fname, lname, and gender. Last, >> but not least, a doctor may be specialized in multiple areas, so >> there is a table that links doctors to all the specialties they >> have been certified on, and thus there is yet another table that >> lists all possible specialties. >> >> Now, assuming I was able to explain the scenario correctly, we then >> have users asking the system for information such as: >> >> "List all people (subscribers), who are male and live in zip code >> 33012 who are contracted under Health Insurance Plan A that have >> selected (as their preferred medical office) medical offices with >> male cardiologists that work within 10 miles of 33012 zip code or >> in MIAMI-DADE county and whose office names contain the sequence of >> letters 'HEAL'" >> >> The way we see it, the concept of tables disappears and so do the >> tables that provide many-to-many joins. So, we end up with some >> classes such as "Person" which contains a reference to a list of >> "Address" objects, and a list of preferred "Medical-Office" >> objects, where each Medical-Office object has a list of Doctor >> objects and each Doctor has a list of Specialty objects, etc, etc. >> >> Now, we assume that each of these classes will need to maintain >> multiple indices, such as the Person class being index on first >> name, last name, dob, gender, among others. The Address class >> indexed on zip code, county name, among others, and so on and so >> forth. >> >> The querying is one problem. The data representation is another. We >> think it's clear that we should have, as an example, a Person >> class. However, the representation of the links between a Person >> and its Addresses or Medical-Offices is not 100% clear. If we >> represent them as a slot in the Person class, where this slot would >> be a List or a set of references to the Address class, then in >> order for us to query on those, means that we always need to fetch >> all objects in those slots in order to apply any search criteria, >> which seems like a bottleneck. If that was the solution, I assume >> we could implement logic such that Addresses are pushed into the >> list, so that the most recent address is in the CAR, so we wouldn't >> necessarily need to read the entire list of Addresses for each >> member, but just fetch the CAR of the slot. >> >> Now, onto the second question. One of the other requirements we >> have is that we need to keep an audit log of data changes. The way >> we do it in RoR is relatively simple. We fetch an object from the >> DB and present it on the browser. When the user submits, we fetch >> another fresh copy from the DB and if the timestamps are the same >> (meaning no one else changed the record) we compare changes to the >> object's attributes (slots). If there are any differences, we save >> the changes (we're trying to avoid unnecessary trips to the DB) and >> if the changes are saved successfully, we write a log of ONLY the >> attributes that were changed (which is pretty trivial in Ruby). >> >> From what we've read in Elephant's manual, this seems harder >> because we don't want to work directly off the Elephant object but >> a memory copy while the user takes his/her time in the browser and >> after submitting, we would take the changes and commit them to the >> Elephant object. Makes me think that we would need to classes for >> each object (one with and one without the persistent metaclass). >> The other problem would be how to "easily" have two objects >> introspect themselves and spit out the slots that changed between >> the two. >> >> Are we looking at this incorrectly? Any advise would be greatly >> appreciated. >> >> Thanks, >> Waldo >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel