From reddaly at gmail.com Thu Oct 2 06:57:26 2008 From: reddaly at gmail.com (Red Daly) Date: Wed, 1 Oct 2008 23:57:26 -0700 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: <97A107961C0C4D6A9572A164163C898A@killer> References: <48D8F262.3080903@saphor.de> <48DA2438.6050802@saphor.de> <20578A9D-A932-4AAF-BE6C-071CB6BA8D20@media.mit.edu> <48DA8CBC.5040707@saphor.de> <97A107961C0C4D6A9572A164163C898A@killer> Message-ID: Unfortunately I am still getting allocation errors even after increasing the cache size to 1.2 gigabytes and eliminating all transactions from my application. I have included db_stat output--the only alarming aspect is the section about mutexes. Do I need more memory dedicated to storing mutexes? 90 megabytes seems sufficient to me. (see attachment) here's some output from the lisp side: *** glibc detected *** /usr/local/bin/sbcl: malloc(): memory corruption: 0x000000000065c910 *** ======= Backtrace: ========= /lib/libc.so.6[0x7f5dfa799a14] /lib/libc.so.6(__libc_malloc+0x90)[0x7f5dfa79b360] /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so(__os_malloc+0x6c)[0x7f5df704bf2c] /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so(__os_calloc+0x2a)[0x7f5df704c06a] /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so (db_env_create+0x2d)[0x7f5df701cb7d] /home/red/lib/cl/site/elephant/src/db-bdb/libberkeley-db.so(db_env_cr+0x16)[0x7f5df6d692c6] [0x100491019b] Any idea what's going wrong here? I'd like to debug it a little more (maybe see an additional error log from bdb), but I can't quite figure out what to do. Right now all I'm getting is the error code and its message from elephant. This is with elephant 0.9.1 and bdb 4.5.20. Would a patch of elephant to support the 4.7 bdb be in order? Thanks, Red Daly -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: db-stat-results Type: application/octet-stream Size: 5603 bytes Desc: not available URL: From eslick at media.mit.edu Thu Oct 2 11:19:05 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 2 Oct 2008 07:19:05 -0400 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: References: <48D8F262.3080903@saphor.de> <48DA2438.6050802@saphor.de> <20578A9D-A932-4AAF-BE6C-071CB6BA8D20@media.mit.edu> <48DA8CBC.5040707@saphor.de> <97A107961C0C4D6A9572A164163C898A@killer> Message-ID: Well, based on the stat output it appears that you don't have any resource issues so something else must be going on. Can you break this down to a test case? Also, have you tried running this with elephant-unstable? Thanks, Ian On Oct 2, 2008, at 2:57 AM, Red Daly wrote: > Unfortunately I am still getting allocation errors even after > increasing the cache size to 1.2 gigabytes and eliminating all > transactions from my application. I have included db_stat output-- > the only alarming aspect is the section about mutexes. Do I need > more memory dedicated to storing mutexes? 90 megabytes seems > sufficient to me. (see attachment) > > here's some output from the lisp side: > *** glibc detected *** /usr/local/bin/sbcl: malloc(): memory > corruption: 0x000000000065c910 *** > ======= Backtrace: ========= > /lib/libc.so.6[0x7f5dfa799a14] > /lib/libc.so.6(__libc_malloc+0x90)[0x7f5dfa79b360] > /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so(__os_malloc+0x6c) > [0x7f5df704bf2c] > /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so(__os_calloc+0x2a) > [0x7f5df704c06a] > /usr/local/BerkeleyDB.4.5/lib/libdb-4.5.so(db_env_create+0x2d) > [0x7f5df701cb7d] > /home/red/lib/cl/site/elephant/src/db-bdb/libberkeley-db.so(db_env_cr > +0x16)[0x7f5df6d692c6] > [0x100491019b] > > Any idea what's going wrong here? I'd like to debug it a little > more (maybe see an additional error log from bdb), but I can't quite > figure out what to do. Right now all I'm getting is the error code > and its message from elephant. > > This is with elephant 0.9.1 and bdb 4.5.20. Would a patch of > elephant to support the 4.7 bdb be in order? > > Thanks, > > Red Daly > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From reddaly at gmail.com Wed Oct 15 20:05:50 2008 From: reddaly at gmail.com (Red Daly) Date: Wed, 15 Oct 2008 13:05:50 -0700 Subject: [elephant-devel] BDB hanging with many stalled threads Message-ID: Every day or so my web server is hanging due to an Elephant/BDB issue. I believe the BDB documentation has a fix for the problem that I'm working to implement. The symptoms are as follows: The server uses 99% CPU as hundreds of threads continue to run. It appears that each is trying to access objects in the database, but these database operations are all blocking. A look at db_stat shows that there are 300+ active transactions, and a look at sb-thread:list-all-threads seems to confirm this. When I attempt to run a database query (listing objects of a certain class) from the REPL, that operation blocks as well. Strangely, when I attempt an (ele:get-from-root :question-number) I get an error: There is no applicable method for the generic function # when called with arguments (:QUESTION-NUMBER NIL). [Condition of type SIMPLE-ERROR] Restarts: 0: [RETRY] Retry SLIME REPL evaluation request. 1: [ABORT] Return to SLIME's top level. 2: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: ((SB-PCL::FAST-METHOD NO-APPLICABLE-METHOD (T)) # # #) 2: (SWANK::EVAL-REGION "(ele:get-from-root I took at look at the BDB FAQ and the following item seems relevant: A transactional database environment is hanging, and no threads of control are making progress. The most common cause of this failure is a thread of control exiting unexpectedly, while holding a Berkeley DB mutex or a read/write logical database lock. If a thread of control exits holding a data structure mutex, other threads of control will likely lock up fairly quickly, queued behind the mutex. If a thread of control exits holding a logical database lock, other threads of control may lock up over a long period of time, as they will not be blocked until they attempt to acquire the specific page for which a lock is not available. See the "Deadlock debugging" section of the Berkeley DB Reference Guide for more information on debugging deadlocks. Whenever a thread of control exits m4_db holding a mutex or logical lock, the failure must be resolved. See the "Handling failure in Transactional Data Store applications" section of the Berkeley DB Reference Guide for more information. Finally, the Berkeley DB API is not re-entrant, and it is usually unsafe for signal handlers to call the Berkeley DB methods. See the "Signal handling" section of the Berkeley DB Reference Guide for more information. --- The solution to this problem seems to be to use DB_ENV->failchk to occasionally check for threads that have terminated without closing locks or mutexes. However, I'm not sure how this should ever occur given the UNWIND-PROTECT clauses in the current elephant system. What do you all make of this situation? The next step to resolving this issue requires several changes. The FFI for DB_ENV->failchk, set_thread_id, set_isalive, and set_thread_count must be implemented and set up to correctly deal with Lisp threads. This seems somewhat hairy to get working on all implementations and OSes. Unfortunately this issue crops up fairly frequently for me. Has anyone else run into it? -Red -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Thu Oct 16 07:43:41 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Thu, 16 Oct 2008 09:43:41 +0200 (CEST) Subject: [elephant-devel] BDB hanging with many stalled threads In-Reply-To: References: Message-ID: <63086.88.73.215.209.1224143021.squirrel@mail.stardawn.org> > The symptoms are as follows: The server uses 99% CPU as hundreds of threads > continue to run. It appears that each is trying to access objects in the > database, but these database operations are all blocking. Sounds like a deadlock scenario to me. "Fixed" in unstable. For stable, use the external deadlock detector: see DB-BDB::START-DEADLOCK-DETECTOR and the DEADLOCK-DETECT arg to OPEN-STORE. Be sure to set the correct path to your db_deadlock binary in my-config.sexp. Leslie From klists at saphor.de Fri Oct 10 15:46:00 2008 From: klists at saphor.de (Marc) Date: Fri, 10 Oct 2008 17:46:00 +0200 Subject: [elephant-devel] Citing Elephant Message-ID: <48EF78B8.7070209@saphor.de> Hi Ian! Does the group have any preferred way for citing elephant in academic papers? Best regards, Marc From reddaly at gmail.com Fri Oct 17 19:14:42 2008 From: reddaly at gmail.com (Red Daly) Date: Fri, 17 Oct 2008 12:14:42 -0700 Subject: [elephant-devel] BDB hanging with many stalled threads In-Reply-To: <63086.88.73.215.209.1224143021.squirrel@mail.stardawn.org> References: <63086.88.73.215.209.1224143021.squirrel@mail.stardawn.org> Message-ID: Unfortunately the standalone deadlock utility provided by BDB does not seem to resolve this. I just tried running it with all these hung threads and somehow the result was a database that needs recovery. db_stat also indicates the number of deadlocks is 0. This seems to indicate that the problem is not caused by deadlocks. What do you think? Red On Thu, Oct 16, 2008 at 12:43 AM, Leslie P. Polzer wrote: > > > The symptoms are as follows: The server uses 99% CPU as hundreds of > threads > > continue to run. It appears that each is trying to access objects in the > > database, but these database operations are all blocking. > > Sounds like a deadlock scenario to me. > > "Fixed" in unstable. > > For stable, use the external deadlock detector: see > DB-BDB::START-DEADLOCK-DETECTOR and the DEADLOCK-DETECT > arg to OPEN-STORE. Be sure to set the correct path to > your db_deadlock binary in my-config.sexp. > > Leslie > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Sat Oct 18 08:27:35 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sat, 18 Oct 2008 10:27:35 +0200 (CEST) Subject: [elephant-devel] BDB hanging with many stalled threads In-Reply-To: References: <63086.88.73.215.209.1224143021.squirrel@mail.stardawn.org> Message-ID: <61901.88.73.196.243.1224318455.squirrel@mail.stardawn.org> > Unfortunately the standalone deadlock utility provided by BDB does not seem > to resolve this. I just tried running it with all these hung threads and > somehow the result was a database that needs recovery. db_stat also > indicates the number of deadlocks is 0. > > This seems to indicate that the problem is not caused by deadlocks. What do > you think? Let's make sure. After opening the store controller run (db-bdb::db-env-set-lock-detect (elephant::controller-environment *store-controller*) db-bdb::DB_LOCK_DEFAULT) and check whether this still occurs. Leslie From elliottslaughter at gmail.com Mon Oct 20 18:21:18 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Mon, 20 Oct 2008 11:21:18 -0700 Subject: [elephant-devel] UFFI/win32 problems (was: Re: Class Heirarchies and Queries) In-Reply-To: <63687.88.73.240.40.1222331731.squirrel@mail.stardawn.org> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> <42c0ab790809241153o124371a2x7588db30e379be42@mail.gmail.com> <63687.88.73.240.40.1222331731.squirrel@mail.stardawn.org> Message-ID: <42c0ab790810201121p602e4ec9weaa9143fe6eb541a@mail.gmail.com> On Thu, Sep 25, 2008 at 1:35 AM, Leslie P. Polzer wrote: > > > I have attached a bundle with three patches against elephant-unstable: > one > > correcting a typo in a comment in elephant.asd, one fixing the alien > problem > > above, and one with the minimal changes I needed to get elephant running > on > > win32/SBCL. > > Thanks, that's good code. Is there some sort of patch submission process I'm missing here? These patches don't seem to have made there way into elephant-unstable yet. Thanks. -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Mon Oct 20 19:16:31 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Mon, 20 Oct 2008 21:16:31 +0200 (CEST) Subject: [elephant-devel] Patch integration (was: Re: UFFI/win32 problems) In-Reply-To: <42c0ab790810201121p602e4ec9weaa9143fe6eb541a@mail.gmail.com> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> <42c0ab790809241153o124371a2x7588db30e379be42@mail.gmail.com> <63687.88.73.240.40.1222331731.squirrel@mail.stardawn.org> <42c0ab790810201121p602e4ec9weaa9143fe6eb541a@mail.gmail.com> Message-ID: <63925.88.73.223.40.1224530191.squirrel@mail.stardawn.org> > Is there some sort of patch submission process I'm missing here? These > patches don't seem to have made there way into elephant-unstable yet. It's the process itself that is, err, missing. I'm going to sort this out within the next few days and push your patch. Sorry for the delay. Leslie From elliottslaughter at gmail.com Mon Oct 20 22:10:57 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Mon, 20 Oct 2008 15:10:57 -0700 Subject: [elephant-devel] Patch integration (was: Re: UFFI/win32 problems) In-Reply-To: <63925.88.73.223.40.1224530191.squirrel@mail.stardawn.org> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> <42c0ab790809241153o124371a2x7588db30e379be42@mail.gmail.com> <63687.88.73.240.40.1222331731.squirrel@mail.stardawn.org> <42c0ab790810201121p602e4ec9weaa9143fe6eb541a@mail.gmail.com> <63925.88.73.223.40.1224530191.squirrel@mail.stardawn.org> Message-ID: <42c0ab790810201510y6d963d0eo407e2886bca51be0@mail.gmail.com> On Mon, Oct 20, 2008 at 12:16 PM, Leslie P. Polzer wrote: > > > Is there some sort of patch submission process I'm missing here? These > > patches don't seem to have made there way into elephant-unstable yet. > > It's the process itself that is, err, missing. > > I'm going to sort this out within the next few days and push > your patch. Ok. Here's another patch (which depends on my earlier patches) which creates basic support for compiling C sources from win32/SBCL (that is, :prebuilt-libraries nil). -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: basic-support-for-__prebuilt_libraries-nil_-under-win32_sbcl_.dpatch Type: application/octet-stream Size: 18003 bytes Desc: not available URL: From rsynnott at gmail.com Tue Oct 21 17:19:34 2008 From: rsynnott at gmail.com (Robert Synnott) Date: Tue, 21 Oct 2008 18:19:34 +0100 Subject: [elephant-devel] Backend and data retrieval questions Message-ID: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> Hi, I'm using Elephant for a website backend. First, since I need to use multiple web application machines, I'm using the Postmodern backend. I assume there's no sane way to share a BDB db between multiple machines? My BDB knowledge more or less caps out at using it as a key/value store from C years ago, I'm afraid. Also, I want to do paginated data, so, for instance, I might, given an object A with index B, want to display the first ten items matching B = whatever. What's the recommended (ideally efficient) way to do this? How about searching via multiple indexes? Thank you Rob -- Robert Synnott http://myblog.rsynnott.com MSN: rsynnott at gmail.com Jabber: rsynnott at gmail.com From killerstorm at newmail.ru Tue Oct 21 20:37:17 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 21 Oct 2008 23:37:17 +0300 Subject: [elephant-devel] Backend and data retrieval questions References: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> Message-ID: <9F8A4929955A4D57821828378CBF1348@killer> RS> First, since I need to use multiple web application machines, I'm RS> using the Postmodern backend. I assume there's no sane way to share a RS> BDB db between multiple machines? i think so too RS> Also, I want to do paginated data, so, for instance, I might, given an RS> object A with index B, want to display the first ten items matching B RS> = whatever. What's the recommended (ideally efficient) way to do this? using cursor API. to get first ten items, first do cursor-pset, then cursor-pnext 9 times. to get second ten items, you need to know OID of last object of a first batch, (so you need to record it somewhere when you do first query). then you call (cursor-get-both index whatever last-oid). that should place cursor on last item of previous batch. then call cursor-pnext 10 times to get next 10 items. and so on. if you want prev/next buttons, you just need to record oids of first and last objects. the only problem with pagination is that there is no efficient way to calculate total amount of items, this cursor stuff should work in theory, but i did not test it, so if you have problems please notify me. or if it actually works post a success story :) RS> How about searching via multiple indexes? do you mean combining results of multiple queries into one ordered (and paginated) list? this is also possible with cursor API, but code will be somewhat hairy, i think.. From reddaly at gmail.com Wed Oct 22 00:37:23 2008 From: reddaly at gmail.com (Red Daly) Date: Tue, 21 Oct 2008 17:37:23 -0700 Subject: [elephant-devel] Backend and data retrieval questions In-Reply-To: <9F8A4929955A4D57821828378CBF1348@killer> References: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> <9F8A4929955A4D57821828378CBF1348@killer> Message-ID: On Tue, Oct 21, 2008 at 1:37 PM, Alex Mizrahi wrote: > RS> First, since I need to use multiple web application machines, I'm > RS> using the Postmodern backend. I assume there's no sane way to share a > RS> BDB db between multiple machines? > > i think so too Berkeley DB supports "replicated" database environments with many read-only environments and one environment responsible for all the writing. (It is not centralized, however, so when the write-enabled DB goes down it will be automatically replaced.) All the data exists on each instance of the database, so this does not spread the pain of storing tons of data. There is also the problem of hitting the write wall since one machine handles all writing. Fortunately these problems only affect extremely large applications. BDB does not provide any code to handle the networking required to get this replicated mode working. You have to write a few functions to do that yourself (presumably in C unless Lisp can handle FFI callbacks, but I don't think it can for most lisps). It should be fairly trivial networking code, however, and somebody may have already written it out there on the web. http://www.oracle.com/technology/documentation/berkeley-db/db/ref/cam/intro.html enjoy elephant :) -Red -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at media.mit.edu Wed Oct 22 03:12:23 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 21 Oct 2008 23:12:23 -0400 Subject: [elephant-devel] Citing Elephant In-Reply-To: <48EF78B8.7070209@saphor.de> References: <48EF78B8.7070209@saphor.de> Message-ID: <84189CBE-9CB2-4975-BF3F-221DDF4FA9C0@media.mit.edu> If no one has answered this, I'd suggest the website as a canonical reference. I don't think there are any actual pubs, although I'm minded to write a paper for the '09 lisp conference at MIT. (Any contributor who is interested in joining this, particularly Robert, should let me know). We would of course want to have a 1.0 in place by publication time... :) Ian On Oct 10, 2008, at 11:46 AM, Marc wrote: > Hi Ian! > > Does the group have any preferred way for citing elephant in academic > papers? > > Best regards, > > Marc > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From sky at viridian-project.de Wed Oct 22 07:46:15 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Wed, 22 Oct 2008 09:46:15 +0200 (CEST) Subject: [elephant-devel] Backend and data retrieval questions In-Reply-To: <9F8A4929955A4D57821828378CBF1348@killer> References: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> <9F8A4929955A4D57821828378CBF1348@killer> Message-ID: <62357.88.73.253.66.1224661575.squirrel@mail.stardawn.org> > using cursor API. to get first ten items, first do cursor-pset, then > cursor-pnext > 9 times. I'd use the higher-level MAP-CLASS instead: (defun first-ten (classname) ; untested (let ((i 1) result) (map-class (lambda (o) (when (> i 10) (return-from first-ten result)) (push o result)) classname) result)) That doesn't work if you want to start somewhere in the middle, though. -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From sky at viridian-project.de Wed Oct 22 08:10:01 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Wed, 22 Oct 2008 10:10:01 +0200 (CEST) Subject: [elephant-devel] Ditching Darcs Message-ID: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> At least one revision from stable produces a non-trivial merge in unstable. That would be a good opportunity to change revision control systems. If no one objects I'd do it with Mercurial and then upload the resulting repository to Bitbucket. Leslie From sky at viridian-project.de Wed Oct 22 08:36:42 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Wed, 22 Oct 2008 10:36:42 +0200 Subject: [elephant-devel] Backend and data retrieval questions In-Reply-To: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> References: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> Message-ID: <20081022083642.GA16337@viridian-project.de> Dear Rob, On Tue, Oct 21, 2008 at 06:19:34PM +0100, Robert Synnott wrote: > First, since I need to use multiple web application machines, I'm > using the Postmodern backend. I assume there's no sane way to share a > BDB db between multiple machines? My BDB knowledge more or less caps > out at using it as a key/value store from C years ago, I'm afraid. You basically have two options: * Use the Postmodern store with the old branch of Elephant (and be prepared to stay with it unless you make it work with the latest changes). * Develop or find a BDB server and integrate it into Elephant. > How about searching via multiple indexes? Try concatenated secondary indices. Leslie > From henrik at evahjelte.com Wed Oct 22 12:45:53 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Wed, 22 Oct 2008 14:45:53 +0200 Subject: [elephant-devel] Ditching Darcs In-Reply-To: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60810220545x622d5fe8qba243f8afe144951@mail.gmail.com> On Wed, Oct 22, 2008 at 10:10 AM, Leslie P. Polzer wrote: > > At least one revision from stable produces a non-trivial > merge in unstable. > > That would be a good opportunity to change revision control > systems. > > If no one objects I'd do it with Mercurial and then upload > the resulting repository to Bitbucket. > I would prefer that we wait with changing revision control system. Darcs 2 has solved most of darcs problems, and the darcs developers are working hard with improving the performance and with windows support. For example, there is an upcoming darcs hacking sprint in three locations in a week or so. And I am working on a possibly great new Lisp cpan killer that is based on darcs 2. It will have these features: Automated testing of Lisp-projects cross os and cross implementation. Based on Darcs 2. Support for cvs, git, mercurial and tarballs as "imports" using tailor. Easy install of all lisp libraries. Easy update of lisp libraries. Possibility to have several forks of the same library locally. You can generate one file ("project-state") that describes the exact patches that a project uses, including dependencies. This file can be used as a "last known good" file, so if a patch to project B breaks project A (detected by automated testing), you can still install Project A. It is just that the last patch to Project B will not be used for project A. But you can still upgrade your local Project B to the latest version. I am almost finished for a first release, but I am only hacking on it for max 40 mins each day (on the bus). So you can call it vapourware for a while.. /Henrik Hjelte -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsynnott at gmail.com Wed Oct 22 17:45:12 2008 From: rsynnott at gmail.com (Robert Synnott) Date: Wed, 22 Oct 2008 18:45:12 +0100 Subject: [elephant-devel] Backend and data retrieval questions In-Reply-To: <20081022083642.GA16337@viridian-project.de> References: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> <20081022083642.GA16337@viridian-project.de> Message-ID: <24f203480810221045n1e04728ate3f7fbccf65a54f0@mail.gmail.com> Thanks :) I ended up going with nasty derived indices for the time being for my multiple-index-query problem. It strikes me that it would be useful to have a relatively high-level server with a BDB store, that knew about the same objects as the clients, and could have methods acting on those object sent to it for speed, stored-procedure-like. Sadly, I certainly don't have the time to do this right now. :) Rob From eslick at media.mit.edu Wed Oct 22 18:48:34 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 22 Oct 2008 14:48:34 -0400 Subject: [elephant-devel] Backend and data retrieval questions In-Reply-To: <24f203480810221045n1e04728ate3f7fbccf65a54f0@mail.gmail.com> References: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> <20081022083642.GA16337@viridian-project.de> <24f203480810221045n1e04728ate3f7fbccf65a54f0@mail.gmail.com> Message-ID: <10BBA073-C95C-494A-A369-C3A3797998AF@media.mit.edu> As a storage server or as an rpc? It would actually be pretty complex for persistent objects unless you had replication implemented (which no one to my knowledge has done with BDB+Elephant) because in the most straightforward implementation you would have to have the same persistent ids and slot values on both machines to pass persistent references back and forth and to avoid network traffic on slot access. You could do something like Robert's DCM where you 'checked out' objects from the central server and could do RPC calls back to update or to transform a given object. This usage model might not be as hard to implement. Ian On Oct 22, 2008, at 1:45 PM, Robert Synnott wrote: > Thanks :) > I ended up going with nasty derived indices for the time being for my > multiple-index-query problem. > > It strikes me that it would be useful to have a relatively high-level > server with a BDB store, that knew about the same objects as the > clients, and could have methods acting on those object sent to it for > speed, stored-procedure-like. Sadly, I certainly don't have the time > to do this right now. :) > Rob > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Wed Oct 22 18:58:39 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Wed, 22 Oct 2008 21:58:39 +0300 Subject: [elephant-devel] Ditching Darcs References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> Message-ID: LPP> At least one revision from stable produces a non-trivial merge in unstable. what does this mean? LPP> If no one objects I'd do it with Mercurial and then upload the resulting repository to Bitbucket. why do you think mercurial would be free of problems? why not git? what if next year some new uber-cool DVCS will appear -- will we have to move to it? using bitbucket has its advantages, but IMHO not big enough to ditch current darcs repos. From killerstorm at newmail.ru Wed Oct 22 18:59:29 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Wed, 22 Oct 2008 21:59:29 +0300 Subject: [elephant-devel] Backend and data retrieval questions References: <24f203480810211019k7ba43ddcm3abdc60c88f87232@mail.gmail.com> <20081022083642.GA16337@viridian-project.de> Message-ID: ??>> First, since I need to use multiple web application machines, I'm using the Postmodern backend. I assume there's no sane way to share a BDB db between multiple machines? My BDB knowledge more or less caps out at using it as a key/value store from C years ago, I'm afraid. LPP> You basically have two options: * Use the Postmodern store with the old branch of Elephant (and be prepared to stay with it unless you make it work with the latest changes). * Develop or find a BDB server and integrate it into Elephant. third option would be to fix few postmodern glitches in a new branch. iirc it was almost working right last time i've checked From elliottslaughter at gmail.com Wed Oct 22 20:15:20 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Wed, 22 Oct 2008 13:15:20 -0700 Subject: [elephant-devel] Ditching Darcs In-Reply-To: <50e8e4f60810220545x622d5fe8qba243f8afe144951@mail.gmail.com> References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <50e8e4f60810220545x622d5fe8qba243f8afe144951@mail.gmail.com> Message-ID: <42c0ab790810221315l4669746of779f20536dac44@mail.gmail.com> 2008/10/22 Henrik Hjelte > > > On Wed, Oct 22, 2008 at 10:10 AM, Leslie P. Polzer < > sky at viridian-project.de> wrote: > >> >> At least one revision from stable produces a non-trivial >> merge in unstable. >> >> That would be a good opportunity to change revision control >> systems. >> >> If no one objects I'd do it with Mercurial and then upload >> the resulting repository to Bitbucket. >> > > I would prefer that we wait with changing revision control system. > > Darcs 2 has solved most of darcs problems, and the darcs > developers are working hard with improving the performance and > with windows support. > I've been bugging the cl.net admins to upgrade to darcs 2 for a couple of months now. Maybe if people here leveraged their support, they might finally do something about that upgrade. Currently, for my own projects on cl.net, I have a personal installation of darcs 2 in ~/bin (with a modified ~/.profile to add this directory to PATH), and my repos are all darcs 2 format. It works fairly well, although because of some other configuration issues I can't push directly (that is, on cl.net, bash doesn't read ~/.bashrc on startup, so I can't alter PATH for noninteractive logins). And I am working on a possibly great new Lisp cpan killer that is > based on darcs 2. > Or maybe your new site will solve this issue. Either way, I think darcs 2 is a viable option. It will have these features: > > Automated testing of Lisp-projects cross os and cross implementation. > Based on Darcs 2. > Support for cvs, git, mercurial and tarballs as "imports" using tailor. > Easy install of all lisp libraries. > Easy update of lisp libraries. > Possibility to have several forks of the same library locally. > > You can generate one file ("project-state") that describes the exact > patches that a project uses, including dependencies. This file can be used > as a "last known good" file, so if a patch to project B breaks project A > (detected by automated testing), you can still install Project A. It is just > that the last patch to Project B will not be used for project A. But you can > still upgrade your local Project B to the latest version. > > I am almost finished for a first release, but I am only hacking on it for > max 40 mins each day (on the bus). So you can call it vapourware for a > while.. > > /Henrik Hjelte > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Thu Oct 23 00:23:42 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 22 Oct 2008 19:23:42 -0500 Subject: [elephant-devel] Citing Elephant In-Reply-To: <84189CBE-9CB2-4975-BF3F-221DDF4FA9C0@media.mit.edu> References: <48EF78B8.7070209@saphor.de> <84189CBE-9CB2-4975-BF3F-221DDF4FA9C0@media.mit.edu> Message-ID: <1224721422.21512.1220.camel@penguin.yourdomain.com> Yes, I'm happy to write something. On Tue, 2008-10-21 at 23:12 -0400, Ian Eslick wrote: > If no one has answered this, I'd suggest the website as a canonical > reference. I don't think there are any actual pubs, although I'm > minded to write a paper for the '09 lisp conference at MIT. (Any > contributor who is interested in joining this, particularly Robert, > should let me know). > > We would of course want to have a 1.0 in place by publication time... :) > > Ian > > > On Oct 10, 2008, at 11:46 AM, Marc wrote: > > > Hi Ian! > > > > Does the group have any preferred way for citing elephant in academic > > papers? > > > > Best regards, > > > > Marc > > > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Thu Oct 23 01:15:48 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 22 Oct 2008 20:15:48 -0500 Subject: [elephant-devel] Ditching Darcs In-Reply-To: References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> Message-ID: <1224724548.21512.1232.camel@penguin.yourdomain.com> I tend to agree with Alex. I don't think we need to optimize our use of source control. I know nothing about Mercurial, having used only CVS, subversion, and Darcs but I think getting 1.0 done should be our focus. On Wed, 2008-10-22 at 21:58 +0300, Alex Mizrahi wrote: > LPP> At least one revision from stable produces a non-trivial > merge in unstable. > > > what does this mean? > > LPP> If no one objects I'd do it with Mercurial and then upload > the resulting repository to Bitbucket. > > > why do you think mercurial would be free of problems? why not git? > what if next year some new uber-cool DVCS will appear -- will we have to > move to it? > > using bitbucket has its advantages, but IMHO not big enough to ditch current > darcs repos. > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From sky at viridian-project.de Thu Oct 23 09:01:05 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Thu, 23 Oct 2008 11:01:05 +0200 (CEST) Subject: [elephant-devel] Ditching Darcs In-Reply-To: <1224724548.21512.1232.camel@penguin.yourdomain.com> References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <1224724548.21512.1232.camel@penguin.yourdomain.com> Message-ID: <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> >> LPP> At least one revision from stable produces a non-trivial >> merge in unstable. >> >> >> what does this mean? It means I have to manually wade through at least two conflicting revisions. >> why do you think mercurial would be free of problems? why not git? >> what if next year some new uber-cool DVCS will appear -- will we have to >> move to it? The issue here isn't really "hg/git/ubercool-dvcs is better" but more like "darcs sucks". For example it drives me nuts that it doesn't give patches a unique ID apart from the commit message. But it's not that big an issue, of course. If you'd all rather stay with Darcs that's fine by me. There are way worse VCS after all. :) Leslie From reddaly at gmail.com Thu Oct 23 09:10:25 2008 From: reddaly at gmail.com (Red Daly) Date: Thu, 23 Oct 2008 02:10:25 -0700 Subject: [elephant-devel] BDB hanging with many stalled threads In-Reply-To: <61901.88.73.196.243.1224318455.squirrel@mail.stardawn.org> References: <63086.88.73.215.209.1224143021.squirrel@mail.stardawn.org> <61901.88.73.196.243.1224318455.squirrel@mail.stardawn.org> Message-ID: This appears to have resolved the problem. The server has been running for the past several days with no database crashes. The only unusual behavior is an occasional condition being raised that looks like this: attempt to THROW to a tag that does not exist: DB-BDB::TRANSACTION [Condition of type SB-INT:SIMPLE-CONTROL-ERROR] Restarts: 0: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: ... 10: (SIGNAL #)[:EXTERNAL] 11: (ERROR SB-INT:SIMPLE-CONTROL-ERROR)[:EXTERNAL] 12: (SB-KERNEL::UNSEEN-THROW-TAG-ERROR-HANDLER ..) 13: (SB-KERNEL::UNSEEN-THROW-TAG-ERROR-HANDLER ..)[:EXTERNAL] 14: (SB-KERNEL:INTERNAL-ERROR #.(SB-SYS:INT-SAP #X7F3DDF523000) #) 15: ("foreign function: #x41D8E2") 16: ("foreign function: #x40B02E") 17: ("foreign function: #x40FBF0") 18: ("no debug information for frame") 19: (SLOT-VALUE # QT::MUSIC-LIBRARY-ID) Thanks for the fix. This remaining issue does not seem to happen too much so I can live with it. It simply belongs in the documentary record. Red On Sat, Oct 18, 2008 at 1:27 AM, Leslie P. Polzer wrote: > > > Unfortunately the standalone deadlock utility provided by BDB does not > seem > > to resolve this. I just tried running it with all these hung threads and > > somehow the result was a database that needs recovery. db_stat also > > indicates the number of deadlocks is 0. > > > > This seems to indicate that the problem is not caused by deadlocks. What > do > > you think? > > Let's make sure. After opening the store controller run > > (db-bdb::db-env-set-lock-detect > (elephant::controller-environment *store-controller*) > db-bdb::DB_LOCK_DEFAULT) > > and check whether this still occurs. > > Leslie > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From henrik at evahjelte.com Thu Oct 23 10:13:52 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 23 Oct 2008 12:13:52 +0200 Subject: [elephant-devel] Ditching Darcs In-Reply-To: <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60810230313w3a3957f1vb27754644c0fdc7a@mail.gmail.com> Slightly off-topic but I want to defend poor little darcs.. On Thu, Oct 23, 2008 at 11:01 AM, Leslie P. Polzer wrote: > > > The issue here isn't really "hg/git/ubercool-dvcs is better" but > more like "darcs sucks". > > For example it drives me nuts that it doesn't give patches a > unique ID apart from the commit message. Darcs gives patches a hash value to identify them, try: darcs changes --last 1 --xml-output And you can select a single patch for example like this: darcs pull --match 'hash 20080215135054-c77ce-d150ea6c60a911dc70c051bd238897f4b2bb3c2d' /Henrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From killerstorm at newmail.ru Thu Oct 23 11:21:10 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 23 Oct 2008 14:21:10 +0300 Subject: [elephant-devel] Ditching Darcs References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> Message-ID: LPP>>>> At least one revision from stable produces a non-trivial ??>>> merge in unstable. ??>>> ??>>> what does this mean? LPP> It means I have to manually wade through at least two conflicting LPP> revisions. and what? i had some really painful 3-way merges in git, does that mean that git sucks too? LPP> For example it drives me nuts that it doesn't give patches a LPP> unique ID apart from the commit message. huh? $ darcs changes --xml-output MAP-INVERTED-INDEX-1 test fixed hash looks pretty unique for me From sky at viridian-project.de Thu Oct 23 12:38:44 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Thu, 23 Oct 2008 14:38:44 +0200 (CEST) Subject: [elephant-devel] Ditching Darcs In-Reply-To: References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> Message-ID: <61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org> > and what? i had some really painful 3-way merges in git, does that mean > that git sucks too? No, it rather means "the merge bothers me like hell, but at least I can use a tool that doesn't get in my way." > hash looks pretty unique for me Make that "hash easily findable by the casual user" then. Anyway, I never meant to argue about it to that extent. No need to get all excited about the topic, it's not worth it. :) Leslie From sky at viridian-project.de Thu Oct 23 14:41:03 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Thu, 23 Oct 2008 16:41:03 +0200 (CEST) Subject: [elephant-devel] Patches applied Message-ID: <62808.88.73.226.69.1224772863.squirrel@mail.stardawn.org> Elliott, I applied your patches to elephant-unstable. Additionally I'm going to merge unstable and stable when I find the time. Leslie From elliottslaughter at gmail.com Thu Oct 23 18:36:52 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Thu, 23 Oct 2008 11:36:52 -0700 Subject: [elephant-devel] Patches applied In-Reply-To: <62808.88.73.226.69.1224772863.squirrel@mail.stardawn.org> References: <62808.88.73.226.69.1224772863.squirrel@mail.stardawn.org> Message-ID: <42c0ab790810231136m9f2d7fm36b3f46b9c9e1751@mail.gmail.com> On Thu, Oct 23, 2008 at 7:41 AM, Leslie P. Polzer wrote: > > Elliott, > > I applied your patches to elephant-unstable. Thanks! -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Thu Oct 23 23:32:35 2008 From: read at robertlread.net (Robert L. Read) Date: Thu, 23 Oct 2008 18:32:35 -0500 Subject: [elephant-devel] Ditching Darcs In-Reply-To: <61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org> References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> <61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org> Message-ID: <1224804755.21512.1305.camel@penguin.yourdomain.com> Actually, I think it is necessary to have such conversations, and they are bound to sound more heated in email than they would over coffee; but we are a distributed team, and have to spend time in such communication costs. The most important thing is that this little community is making progress, for which we can thank Leslie and Alex and others for their code, as well as their willingness to try to make decisions about how the project should be managed. It's fair to say that I'm not providing much leadership this year, so the y'all and Ian will have to do it. On Thu, 2008-10-23 at 14:38 +0200, Leslie P. Polzer wrote: > Anyway, I never meant to argue about it to that extent. > > No need to get all excited about the topic, it's not worth > it. :) > > Leslie From sky at viridian-project.de Fri Oct 24 07:11:48 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Fri, 24 Oct 2008 09:11:48 +0200 (CEST) Subject: [elephant-devel] Ditching Darcs In-Reply-To: <1224804755.21512.1305.camel@penguin.yourdomain.com> References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> <61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org> <1224804755.21512.1305.camel@penguin.yourdomain.com> Message-ID: <63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> > The most important thing is that this little community is making > progress, for which we can thank Leslie and Alex and others for their > code, as well as their willingness to try to make decisions about how > the project should be managed. I think it's important right now to get 092 out as an interim release. We have to show the rest of the tiny Lisp universe that the Elephant is still moving. The only major problem with it is that the Postmodern backend hasn't kept up with the schema evolution changes. There were still some deca of failing tests when I last tried it. Maybe it makes sense to have two releases: 091.1 for all who wish to use Postmodern (a snapshot of the current stable) and 092 for all who are content with CLSQL or BDB. > It's fair to say that I'm not providing much leadership this year, so > the y'all and Ian will have to do it. What are your plans for the future, Robert? Leslie From klists at saphor.de Fri Oct 24 12:37:54 2008 From: klists at saphor.de (Marc) Date: Fri, 24 Oct 2008 14:37:54 +0200 Subject: [elephant-devel] Citing Elephant In-Reply-To: <84189CBE-9CB2-4975-BF3F-221DDF4FA9C0@media.mit.edu> References: <48EF78B8.7070209@saphor.de> <84189CBE-9CB2-4975-BF3F-221DDF4FA9C0@media.mit.edu> Message-ID: <4901C1A2.7000904@saphor.de> Ian Eslick wrote: > If no one has answered this, I'd suggest the website as a canonical > reference. I don't think there are any actual pubs, although I'm > minded to write a paper for the '09 lisp conference at MIT. (Any > contributor who is interested in joining this, particularly Robert, > should let me know). > > We would of course want to have a 1.0 in place by publication time... :) > thanks! For future publications building on elephant we'll just cite the Web page. Best regards, Marc > Oct 10, 2008, at 11:46 AM, Marc wrote: > > >> Hi Ian! >> >> Does the group have any preferred way for citing elephant in academic >> papers? >> >> Best regards, >> >> Marc >> >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > From read at robertlread.net Sat Oct 25 01:35:47 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 24 Oct 2008 20:35:47 -0500 Subject: [elephant-devel] Ditching Darcs --- and other plans In-Reply-To: <63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> <61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org> <1224804755.21512.1305.camel@penguin.yourdomain.com> <63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> Message-ID: <1224898547.21512.1408.camel@penguin.yourdomain.com> On Fri, 2008-10-24 at 09:11 +0200, Leslie P. Polzer wrote: > The only major problem with it is that the Postmodern backend > hasn't kept up with the schema evolution changes. I did a bunch of work back in the spring to bring postmodern up-to-date, and there were just a few failing tests at that time---I'm not sure what else has happened. > > There were still some deca of failing tests when I last tried > it. > > Maybe it makes sense to have two releases: 091.1 for all > who wish to use Postmodern (a snapshot of the current stable) > and 092 for all who are content with CLSQL or BDB. > > > > It's fair to say that I'm not providing much leadership this year, so > > the y'all and Ian will have to do it. > > What are your plans for the future, Robert? Well, Ian has been the de facto leader for the last year or more. I'm happy to resign as the official maintain in favor of him, or whomever he appoints. One of the problems that I've had is that Ian has taken things in a different direction than I would have. I of course did not and do not criticize this, because, after all, he was doing most of the work, and I mainly kept it working with CL-SQL, and helped some with the postmodern stuff. Basically, I think Ian's "defpclass" stuff is great, and a great improvement over my bare-bones DCM approach. The additions of "associations" I find a lot more dubious. The schema evolution stuff that Ian spearheaded is the seed of something really great and unique (in ANY persistence solution), but until it is more complete, in ways that I think I have written about here, it doesn't really provide a sufficiently compelling schema evolution management system for people to want to switch from Hibernate clones to Elephant. I am currently working on developing another LISP project, a "natural diff." I hope to create a common-lisp project around it soon. At the moment this seems more compelling for me personally to be working on than Elephant. Personally, I would like to implement a native-LISP Btree basis for Elephant, and I would like to build on Ian's schema versioning to make what I personally envision as a really game-changing schema evolution system. However, both of these are relatively low priorities to me. Since I ran out of money for the business I was building that required Elephant and have been forced to take a 40-hour a week job that takes 60-hours a week, I have been fairly useless to this project, although I remain an enthusiastic fan of its potential. When my financial situation changes, I will be an active user again, and, in whatever form y'all find helpful, a contributor. From ludwig at fh-worms.de Tue Oct 28 10:35:05 2008 From: ludwig at fh-worms.de (Christoph Ludwig) Date: Tue, 28 Oct 2008 11:35:05 +0100 Subject: [elephant-devel] Elephant backend performance characteristics Message-ID: <20081028103505.GB14842@castellio.ztt.fh-worms.de> Hi, I'd like to know if people on the list have experience with the performance characteristics of the available backends for elephant. I am not asking for detailed measurements and performance differences of few percent but typical usage patterns where you can predict that one backend will likely outperform the others. Background: Our application that makes heavy use of elephant/bdb and the association feature in the unstable branch is too slow by at least one order of magnitude. We obviously need to profile it first before we start any refactoring or changing the elephant configuration, and most likely we will find that our application code used for the data import uses an inefficient algorithm. But it might be helpful for interpreting the profile data when we have some background knowledge about the performance characteristics one can expect from elephant and its various backends. Thanks Christoph -- FH Worms - University of Applied Sciences Fachbereich Informatik / Telekommunikation Erenburgerstr. 19, 67549 Worms, Germany From sky at viridian-project.de Tue Oct 28 11:40:36 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 28 Oct 2008 12:40:36 +0100 (CET) Subject: [elephant-devel] Elephant backend performance characteristics In-Reply-To: <20081028103505.GB14842@castellio.ztt.fh-worms.de> References: <20081028103505.GB14842@castellio.ztt.fh-worms.de> Message-ID: <63588.88.73.226.2.1225194036.squirrel@mail.stardawn.org> > Background: Our application that makes heavy use of elephant/bdb and the > association feature in the unstable branch is too slow by at least one order > of magnitude. We obviously need to profile it first before we start any > refactoring or changing the elephant configuration, and most likely we will > find that our application code used for the data import uses an inefficient > algorithm. But it might be helpful for interpreting the profile data when we > have some background knowledge about the performance characteristics one can > expect from elephant and its various backends. BDB is the fastest backend currently available. Postmodern is about half as fast and CLSQL lags three or four times behind. There are several areas here where a slowdown could occur: your algorithm, the association code (maybe Ian knows more about this) or maybe it's just the natural slowdown of Elephant's disk-backed reads and writes. Maybe using cached slots would help you, but that's work in progress. Leslie From killerstorm at newmail.ru Tue Oct 28 12:43:44 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 28 Oct 2008 14:43:44 +0200 Subject: [elephant-devel] Elephant backend performance characteristics References: <20081028103505.GB14842@castellio.ztt.fh-worms.de> <63588.88.73.226.2.1225194036.squirrel@mail.stardawn.org> Message-ID: LPP> BDB is the fastest backend currently available. Postmodern LPP> is about half as fast a little clarification -- it is not like there is a constant slowness factor. PostgreSQL storage itself is pretty fast , and probably in some aspects it is even better than BDB storage. but there is pretty high communication overhead associated with each operation, so often postmodern is a lot slower than bdb. also storage was modelled after BDB API, and thus it is pretty hard to implement same semantics on top of SQL database, so some stuff is implemented in very inefficient manner. From killerstorm at newmail.ru Tue Oct 28 13:23:31 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 28 Oct 2008 15:23:31 +0200 Subject: [elephant-devel] Ditching Darcs References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com><62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org><61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org><1224804755.21512.1305.camel@penguin.yourdomain.com> <63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> Message-ID: LPP> The only major problem with it is that the Postmodern backend LPP> hasn't kept up with the schema evolution changes. hm, what do you mean? i thought it works reasonably well, as there are just a handful of glitches to left resolve, but i won't call that "a major problem". or am i missing something? i've just updated repo elephant-unstable and ran tests with postmodern (i have few local changes but i don't think they make difference): Did 462 checks. Pass: 460 (99%) Skip: 1 ( 0%) Fail: 1 ( 0%) Failure Details: -------------------------------- SIMPLE-EXPLICIT-ASSOC-SETUP []: Unexpected Error: # There is no class named PJASSOC... -------------------------------- Skip Details: SIMPLE-EXPLICIT-ASSOC []: Dependencies not satisfied. actually it seems to be quite chaotic -- i have different set of errors each time i run a test suite. but i guess that is either a problem with tests or a problem with my SBCL, because i also get problems with BDB backend. heh, running it one more time left no errors: Did 462 checks. Pass: 462 (100%) Skip: 0 ( 0%) Fail: 0 ( 0%) really there's some BS with tests.. LPP> There were still some deca of failing tests when I last tried LPP> it. and no problems with BDB and CLSQL, if you run it on a clean DB, and afterwards on less clean? LPP> Maybe it makes sense to have two releases: 091.1 for all LPP> who wish to use Postmodern (a snapshot of the current stable) LPP> and 092 for all who are content with CLSQL or BDB. yep, as for me that makes sense.. From eslick at media.mit.edu Tue Oct 28 13:13:52 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 28 Oct 2008 09:13:52 -0400 Subject: [elephant-devel] Elephant backend performance characteristics In-Reply-To: <63588.88.73.226.2.1225194036.squirrel@mail.stardawn.org> References: <20081028103505.GB14842@castellio.ztt.fh-worms.de> <63588.88.73.226.2.1225194036.squirrel@mail.stardawn.org> Message-ID: <3FCEE757-C1A6-47E3-89E6-B963DF8C3CC0@media.mit.edu> Unfortunately performance is one thing of several problems that violate the persistent object abstraction. The first big thing to sanity check is that a set of operations is wrapped in a transaction. This avoids disk syncs after every primitive operation which can speed things up tremendously. Currently each primitive elephant op takes place in its own transaction. For example the slot write to an indexed slot is in a transaction with the index updates associated with it. If you write a set of indexed slots then each slot write is a separate transaction and this can become really expensive. We often see this kind of performance issue reported on the list for data import procedures where it's the most obvious. I highly recommend not using cached slots for the time being as they don't interact well with transactions at the present time. Associations can be considered to be of beta quality. Ian On Oct 28, 2008, at 7:40 AM, Leslie P. Polzer wrote: > >> Background: Our application that makes heavy use of elephant/bdb >> and the >> association feature in the unstable branch is too slow by at least >> one order >> of magnitude. We obviously need to profile it first before we start >> any >> refactoring or changing the elephant configuration, and most likely >> we will >> find that our application code used for the data import uses an >> inefficient >> algorithm. But it might be helpful for interpreting the profile >> data when we >> have some background knowledge about the performance >> characteristics one can >> expect from elephant and its various backends. > > BDB is the fastest backend currently available. Postmodern > is about half as fast and CLSQL lags three or four times > behind. > > There are several areas here where a slowdown could occur: > your algorithm, the association code (maybe Ian knows more > about this) or maybe it's just the natural slowdown of > Elephant's disk-backed reads and writes. > > Maybe using cached slots would help you, but that's > work in progress. > > Leslie > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Tue Oct 28 13:51:59 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 28 Oct 2008 09:51:59 -0400 Subject: [elephant-devel] Ditching Darcs In-Reply-To: References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> <61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org> <1224804755.21512.1305.camel@penguin.yourdomain.com> <63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> Message-ID: <5F0DC9C7-B3CB-40D8-8DC5-00A9484B3431@media.mit.edu> Historically we've had problems with tests being non-idempotent. I usually wipe the dbs between runs to ensure that hidden interactions don't break test assumptions. Ian On Oct 28, 2008, at 9:23 AM, Alex Mizrahi wrote: > LPP> The only major problem with it is that the Postmodern backend > LPP> hasn't kept up with the schema evolution changes. > > hm, what do you mean? i thought it works reasonably well, as there > are just a handful of glitches to left resolve, but i won't call > that "a > major problem". > or am i missing something? > > i've just updated repo elephant-unstable and ran tests with postmodern > (i have few local changes but i don't think they make difference): > > Did 462 checks. > Pass: 460 (99%) > Skip: 1 ( 0%) > Fail: 1 ( 0%) > > Failure Details: > -------------------------------- > SIMPLE-EXPLICIT-ASSOC-SETUP []: > Unexpected Error: # > There is no class named PJASSOC... > -------------------------------- > > Skip Details: > SIMPLE-EXPLICIT-ASSOC []: > Dependencies not satisfied. > > actually it seems to be quite chaotic -- i have different set of > errors > each time i run a test suite. but i guess that is either a problem > with tests or a problem with my SBCL, because i also get problems > with BDB backend. > > heh, running it one more time left no errors: > > Did 462 checks. > Pass: 462 (100%) > Skip: 0 ( 0%) > Fail: 0 ( 0%) > > really there's some BS with tests.. > > > > LPP> There were still some deca of failing tests when I last tried > LPP> it. > > and no problems with BDB and CLSQL, if you run it on a clean DB, > and afterwards on less clean? > > LPP> Maybe it makes sense to have two releases: 091.1 for all > LPP> who wish to use Postmodern (a snapshot of the current stable) > LPP> and 092 for all who are content with CLSQL or BDB. > > yep, as for me that makes sense.. > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From ludwig at Fh-Worms.DE Tue Oct 28 13:16:38 2008 From: ludwig at Fh-Worms.DE (Christoph Ludwig) Date: Tue, 28 Oct 2008 14:16:38 +0100 Subject: [elephant-devel] Elephant backend performance characteristics In-Reply-To: <63588.88.73.226.2.1225194036.squirrel@mail.stardawn.org> References: <20081028103505.GB14842@castellio.ztt.fh-worms.de> <63588.88.73.226.2.1225194036.squirrel@mail.stardawn.org> Message-ID: <20081028131637.GE14842@castellio.ztt.fh-worms.de> On Tue, Oct 28, 2008 at 12:40:36PM +0100, Leslie P. Polzer wrote: > > > Background: Our application that makes heavy use of elephant/bdb and the > > association feature in the unstable branch is too slow by at least one order > > of magnitude. We obviously need to profile it first before we start any > > refactoring or changing the elephant configuration, and most likely we will > > find that our application code used for the data import uses an inefficient > > algorithm. But it might be helpful for interpreting the profile data when we > > have some background knowledge about the performance characteristics one can > > expect from elephant and its various backends. > > BDB is the fastest backend currently available. Postmodern > is about half as fast and CLSQL lags three or four times > behind. > > There are several areas here where a slowdown could occur: > your algorithm, the association code (maybe Ian knows more > about this) or maybe it's just the natural slowdown of > Elephant's disk-backed reads and writes. > > Maybe using cached slots would help you, but that's > work in progress. Thanks. Do these estimates apply both to read and write operations? Our current grieve is with the bulk import of data; but in a production environment, queries will be much more common. Regards Christoph -- FH Worms - University of Applied Sciences Fachbereich Informatik / Telekommunikation Erenburgerstr. 19, 67549 Worms, Germany From killerstorm at newmail.ru Tue Oct 28 15:05:21 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 28 Oct 2008 17:05:21 +0200 Subject: [elephant-devel] Ditching Darcs References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com><62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org><61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org><1224804755.21512.1305.camel@penguin.yourdomain.com><63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> <5F0DC9C7-B3CB-40D8-8DC5-00A9484B3431@media.mit.edu> Message-ID: IE> Historically we've had problems with tests being non-idempotent. I IE> usually wipe the dbs between runs to ensure that hidden interactions IE> don't break test assumptions. i have errors in a completely clean environment with a clean store: Did 455 checks. Pass: 449 (98%) Skip: 2 ( 0%) Fail: 4 ( 0%) Failure Details: -------------------------------- SIMPLE-SLOT-ASSOC-SETUP []: Unexpected Error: # There is no class named PERSON... -------------------------------- -------------------------------- SIMPLE-EXPLICIT-ASSOC-SETUP []: Unexpected Error: # There is no class named PERSON... -------------------------------- -------------------------------- INDEXING-TIMING []: Unexpected Error: # There is no class named STRESS-NORMAL... -------------------------------- -------------------------------- INDEXING-REDEF-CLASS []: Unexpected Error: # There is no class named IDX-EIGHT... -------------------------------- Skip Details: SIMPLE-SLOT-ASSOC []: Dependencies not satisfied. SIMPLE-EXPLICIT-ASSOC []: Dependencies not satisfied. From sky at viridian-project.de Tue Oct 28 14:45:17 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 28 Oct 2008 15:45:17 +0100 (CET) Subject: [elephant-devel] Ditching Darcs In-Reply-To: References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com><62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org><61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org><1224804755.21512.1305.camel@penguin.yourdomain.com> <63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> Message-ID: <63473.88.73.226.2.1225205117.squirrel@mail.stardawn.org> > i've just updated repo elephant-unstable and ran tests with postmodern > (i have few local changes but i don't think they make difference): > > Did 462 checks. > Pass: 460 (99%) > Skip: 1 ( 0%) > Fail: 1 ( 0%) That's great! :) I must've missed something or unintentionally used a db with older stuff already in it. How about the NIL problem with secondary indices, you tried to tackle that? Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From rsynnott at gmail.com Tue Oct 28 15:44:51 2008 From: rsynnott at gmail.com (Robert Synnott) Date: Tue, 28 Oct 2008 15:44:51 +0000 Subject: [elephant-devel] Ditching Darcs In-Reply-To: References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> <61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org> <1224804755.21512.1305.camel@penguin.yourdomain.com> <63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> <5F0DC9C7-B3CB-40D8-8DC5-00A9484B3431@media.mit.edu> Message-ID: <24f203480810280844q5c4a25b2nadb515775ef8500a@mail.gmail.com> On this subject, is there a particular branch that I should be using for postmodern? I've noticed occasional issues where it'll complain that a table already exists, generally after a non-elephant error has occurred within a transaction. Rob From killerstorm at newmail.ru Tue Oct 28 17:52:53 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 28 Oct 2008 19:52:53 +0200 Subject: [elephant-devel] Ditching Darcs References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com><62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org><61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org><1224804755.21512.1305.camel@penguin.yourdomain.com><63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> <63473.88.73.226.2.1225205117.squirrel@mail.stardawn.org> Message-ID: LPP> I must've missed something or unintentionally used a db LPP> with older stuff already in it. yep, old format database could be the case if you've got it broken at start. but it seems our current tests are broken, so you get always get some error on the first run regardless of backend used. LPP> How about the NIL problem with secondary indices, you tried LPP> to tackle that? not really, so far :( i thought some more about approach with NULLs and wrote down queries and cases needed to handle NULLs -- it appears to be more realistic now than it was before, but still pretty hairy, so i've abandoned this for a while.. for example, cursor-next uses query WHERE (qi > $1) OR ((qi = $1) AND (value > $2)) and cursor-prev uses query WHERE (qi < $1) OR ((qi = $1) AND (value < $2)) being similar, they are made via template "WHERE (qi ~a $1) OR ((qi = $1) AND (value ~a $2))" now. but with nulls that would be four different queries: cursor-next: key is null: (qi IS NULL) and (value > $2) key is not null: (qi > $1) OR (qi IS NULL) OR ((qi = 1) AND (value > $2) cursor-prev key is null: (qi IS NOT NULL) OR ((qi IS NULL) AND (value < $2)) key is not null: (qi < $1) OR ((qi = $1) AND (value < $2)) and taking into account other cursor operations, instead of 3 query templates we now have something like 12 different queries now, and i see any pattern how they can be merged :( or maybe it makes sense to ditch templated query generations and just write these conditions manually From reddaly at gmail.com Tue Oct 28 17:30:22 2008 From: reddaly at gmail.com (Red Daly) Date: Tue, 28 Oct 2008 10:30:22 -0700 Subject: [elephant-devel] Lisp Backend Architecture Message-ID: I am hoping to clarify some of the prior discussions[1] about the native Lisp backend for Elephant and propose a basic architecture. Hopefully we can modularize development so if somebody wants to hack for a few days on the backend he can avoid being overwhelmed. Multiprocess support: What features do the major lisps have for locking and concurrency? What features are we missing from C/linux--what does BDB do in this regard that is hard to do in Lisp? I do not know too much about implementing efficient multi-threaded lisp programs where there is a lot of contention. Distribution: Are others interested in making the backend distributed? I prefer the design to allow adding an efficient distributed architecture at a later date. Custom indexing: A lot of discussion has gone into the best implementation practice for BTrees. But what about other indexing structures? Multidimensional indexing is relevant to my project now because we have spacial information (longitude latitude pairs) we would like to query. Right now I am using a kd-tree with nodes made persistent by elephant, but usually this sort of index would be implemented by the database itself. I imagine multi-dimensional indexing could improve queries, too. There are other indexing needs as well. Document-level search is another common case. I imagine an efficient search library is beyond our capacity, but how could we make the database suited to implement search on top of the thing? BTrees are not everything, unless I am missing something. I propose the following basic architecture for the backend: Logging package with generic undo/redo logging support. It would be customizable but it would not depend on other code from the Lisp backend. A database could plug into the log by implementing undo, redo, and serialization functions. By modularizing logging, it should make it easier to hack on it without being familiar with the rest of the backend. It will also be reusable, for what it's worth. A persistent heap. Beneath indices and data storage would be a heap-on-disk layer with transaction support. The heap would have methods for reading, writing, and locking sequences of bytes. It would also handle The persistent heap alone would qualify as a database and might be useful as a basis for other DB projects or experiments. Cachine would probably not be implemented at this level, though that would be easiest. It may be possible to implement the transactional heavy lifting for the persistent heap and make it extensible enough to be used throughout. B-Trees. A MOP-based implementation of B-Trees may be customizable enough to plug into the the rest of the system with a few additional method declarations for the persistent, transactional version. Specialized methods would access the persistent heap for reads and writes. Locking of the sort done by BDB (level 2 etc.) could probably be accomplished in these methods specialized for our persistent B-Tree. Bkd-Trees and other indexing structures could be implemented similarly to B-Trees. Caching. B-Tree nodes or other lispy objects should be cached as opposed to byte arrays. Caching of B-Tree nodes requires additional multiprocess code, so not all the transactional magic happens in the persistent heap layer. I'm a little fuzzy on exactly how this will work, but it sounds reasonable enough. The DB user would interact mostly with the B-Tree and other indexing structures, and with transactions. How does this sound for a basic architecture? The multiprocess details are not fleshed out so your comments are appreciated. I have attached my basic implementation of a logger and persistent heap to make this clearer. Best regards, Red [1] The most lengthy discussion I can find is here: http://common-lisp.net/pipermail/elephant-devel/2008-May/004108.html The trac has a page on the Lisp backend: http://trac.common-lisp.net/elephant/wiki/LispDataStore -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: persistent-heap.tar.gz Type: application/x-gzip Size: 7163 bytes Desc: not available URL: From reddaly at gmail.com Tue Oct 28 18:21:52 2008 From: reddaly at gmail.com (Red Daly) Date: Tue, 28 Oct 2008 11:21:52 -0700 Subject: [elephant-devel] Lisp Backend Architecture In-Reply-To: References: Message-ID: I believe the code I attached was scrubbed. Here is a link instead: http://iodb.org/static/persistent-heap/ Red On Tue, Oct 28, 2008 at 10:30 AM, Red Daly wrote: > I am hoping to clarify some of the prior discussions[1] about the native > Lisp backend for Elephant and propose a basic architecture. Hopefully we > can modularize development so if somebody wants to hack for a few days on > the backend he can avoid being overwhelmed. > > Multiprocess support: What features do the major lisps have for locking and > concurrency? What features are we missing from C/linux--what does BDB do in > this regard that is hard to do in Lisp? I do not know too much about > implementing efficient multi-threaded lisp programs where there is a lot of > contention. > > Distribution: Are others interested in making the backend distributed? I > prefer the design to allow adding an efficient distributed architecture at a > later date. > > Custom indexing: A lot of discussion has gone into the best implementation > practice for BTrees. But what about other indexing structures? > Multidimensional indexing is relevant to my project now because we have > spacial information (longitude latitude pairs) we would like to query. > Right now I am using a kd-tree with nodes made persistent by elephant, but > usually this sort of index would be implemented by the database itself. I > imagine multi-dimensional indexing could improve queries, too. > > There are other indexing needs as well. Document-level search is another > common case. I imagine an efficient search library is beyond our capacity, > but how could we make the database suited to implement search on top of the > thing? BTrees are not everything, unless I am missing something. > > > I propose the following basic architecture for the backend: > > Logging package with generic undo/redo logging support. It would be > customizable but it would not depend on other code from the Lisp backend. A > database could plug into the log by implementing undo, redo, and > serialization functions. By modularizing logging, it should make it easier > to hack on it without being familiar with the rest of the backend. It will > also be reusable, for what it's worth. > > A persistent heap. Beneath indices and data storage would be a > heap-on-disk layer with transaction support. The heap would have methods > for reading, writing, and locking sequences of bytes. It would also handle > The persistent heap alone would qualify as a database and might be useful > as a basis for other DB projects or experiments. Cachine would probably not > be implemented at this level, though that would be easiest. It may be > possible to implement the transactional heavy lifting for the persistent > heap and make it extensible enough to be used throughout. > > B-Trees. A MOP-based implementation of B-Trees may be customizable enough > to plug into the the rest of the system with a few additional method > declarations for the persistent, transactional version. Specialized methods > would access the persistent heap for reads and writes. Locking of the sort > done by BDB (level 2 etc.) could probably be accomplished in these methods > specialized for our persistent B-Tree. > > Bkd-Trees and other indexing structures could be implemented similarly to > B-Trees. > > Caching. B-Tree nodes or other lispy objects should be cached as opposed > to byte arrays. Caching of B-Tree nodes requires additional multiprocess > code, so not all the transactional magic happens in the persistent heap > layer. I'm a little fuzzy on exactly how this will work, but it sounds > reasonable enough. > > The DB user would interact mostly with the B-Tree and other indexing > structures, and with transactions. > > > How does this sound for a basic architecture? The multiprocess details are > not fleshed out so your comments are appreciated. I have attached my basic > implementation of a logger and persistent heap to make this clearer. > > > Best regards, > > Red > > > [1] The most lengthy discussion I can find is here: > http://common-lisp.net/pipermail/elephant-devel/2008-May/004108.html > The trac has a page on the Lisp backend: > http://trac.common-lisp.net/elephant/wiki/LispDataStore > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Tue Oct 28 18:40:07 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 28 Oct 2008 19:40:07 +0100 (CET) Subject: [elephant-devel] Lisp Backend Architecture In-Reply-To: References: Message-ID: <64474.88.73.226.2.1225219207.squirrel@mail.stardawn.org> > I am hoping to clarify some of the prior discussions[1] about the native > Lisp backend for Elephant and propose a basic architecture. Hopefully we > can modularize development so if somebody wants to hack for a few days on > the backend he can avoid being overwhelmed. Glenn Tarcea and me are currently drafting a Skiplist-based key-value storage in Lisp. This will be generic enough on the top-level to plug in a btree later if needed. You're welcome to join the effort if you wish. > Multiprocess support: What features do the major lisps have for locking and > concurrency? What features are we missing from C/linux--what does BDB do in > this regard that is hard to do in Lisp? I do not know too much about > implementing efficient multi-threaded lisp programs where there is a lot of > contention. My guess is that the most important thing is optimizing the data structure for partial locking. There are also a lot of scientific papers on database concurrency, none of which I have read so far. I'm busy enough getting a single-thread solution up first. ;) > Distribution: Are others interested in making the backend distributed? I > prefer the design to allow adding an efficient distributed architecture at a > later date. Definitely. Considerations? Where do we need to take care? > Custom indexing: A lot of discussion has gone into the best implementation > practice for BTrees. But what about other indexing structures? See above, Skiplist. > Multidimensional indexing You're losing me right here, so it would be good if you joined our skiplist effort by drafting an API that will support both traditional and multidimensional indexing approaches. > There are other indexing needs as well. Document-level search is another > common case. I imagine an efficient search library is beyond our capacity, > but how could we make the database suited to implement search on top of the > thing? You mean full-text indexing like offered by Montezuma? > Logging package with generic undo/redo logging support. It would be > customizable but it would not depend on other code from the Lisp backend. A > database could plug into the log by implementing undo, redo, and > serialization functions. By modularizing logging, it should make it easier > to hack on it without being familiar with the rest of the backend. It will > also be reusable, for what it's worth. Ideally I'd like the major parts to be generic enough to make a library of them. > A persistent heap. Beneath indices and data storage would be a heap-on-disk > layer with transaction support. The heap would have methods for reading, > writing, and locking sequences of bytes. It would also handle The > persistent heap alone would qualify as a database and might be useful as a > basis for other DB projects or experiments. Cachine would probably not be > implemented at this level, though that would be easiest. It may be possible > to implement the transactional heavy lifting for the persistent heap and > make it extensible enough to be used throughout. So you also want to collect the garbage at this level? > B-Trees. A MOP-based implementation of B-Trees may be customizable enough > to plug into the the rest of the system with a few additional method > declarations for the persistent, transactional version. Specialized methods > would access the persistent heap for reads and writes. Locking of the sort > done by BDB (level 2 etc.) could probably be accomplished in these methods > specialized for our persistent B-Tree. > > Bkd-Trees and other indexing structures could be implemented similarly to > B-Trees. Yes, but let's strive for a common API to abstract key-value storage. > How does this sound for a basic architecture? It's fine, but I'd like to emphasize development of a rudimentary on-disk data structure first. After that we can add generic transactions, establish an API for the underlying binary heap and so on. What do you think? Leslie From sky at viridian-project.de Tue Oct 28 18:50:46 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 28 Oct 2008 19:50:46 +0100 (CET) Subject: [elephant-devel] Ditching Darcs In-Reply-To: References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com><62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org><61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org><1224804755.21512.1305.camel@penguin.yourdomain.com><63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> <63473.88.73.226.2.1225205117.squirrel@mail.stardawn.org> Message-ID: <64898.88.73.226.2.1225219846.squirrel@mail.stardawn.org> > and taking into account other cursor operations, instead of 3 query > templates we now have something like 12 different queries now, and > i see any pattern how they can be merged :( > or maybe it makes sense to ditch templated query generations and just write > these conditions manually I took a look at it. CURSOR-FETCH-QUERY is complicated enough as it is, so let's reduce it to some common primitives as far as possible and implement our NULL cases manually. Leslie From sky at viridian-project.de Tue Oct 28 19:01:40 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 28 Oct 2008 20:01:40 +0100 (CET) Subject: [elephant-devel] Ditching Darcs In-Reply-To: <24f203480810280844q5c4a25b2nadb515775ef8500a@mail.gmail.com> References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org> <1224724548.21512.1232.camel@penguin.yourdomain.com> <62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org> <61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org> <1224804755.21512.1305.camel@penguin.yourdomain.com> <63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org> <5F0DC9C7-B3CB-40D8-8DC5-00A9484B3431@media.mit.edu> <24f203480810280844q5c4a25b2nadb515775ef8500a@mail.gmail.com> Message-ID: <61317.88.73.226.2.1225220500.squirrel@mail.stardawn.org> > On this subject, is there a particular branch that I should be using > for postmodern? I've noticed occasional issues where it'll complain > that a table already exists, generally after a non-elephant error has > occurred within a transaction. What branch are you using? 0.9.1, stable or unstable? 0.9.1 has a few bugs that were fixed. Leslie From midfield at gmail.com Tue Oct 28 18:26:44 2008 From: midfield at gmail.com (Ben) Date: Tue, 28 Oct 2008 11:26:44 -0700 Subject: [elephant-devel] elephant-devel Digest, Vol 5, Issue 12 In-Reply-To: References: Message-ID: <9157df230810281126u602ef4f2k2729a491f83fac9@mail.gmail.com> i don't know if you guys have looked at cache-oblivious b-trees (and other data structures) but they seem like the new hotness. http://supertech.csail.mit.edu/cacheObliviousBTree.html b On Tue, Oct 28, 2008 at 11:21 AM, wrote: > Send elephant-devel mailing list submissions to > elephant-devel at common-lisp.net > > To subscribe or unsubscribe via the World Wide Web, visit > http://common-lisp.net/cgi-bin/mailman/listinfo/elephant-devel > or, via email, send a message with subject or body 'help' to > elephant-devel-request at common-lisp.net > > You can reach the person managing the list at > elephant-devel-owner at common-lisp.net > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of elephant-devel digest..." > > > Today's Topics: > > 1. Re: Ditching Darcs (Robert Synnott) > 2. Re: Ditching Darcs (Alex Mizrahi) > 3. Lisp Backend Architecture (Red Daly) > 4. Re: Lisp Backend Architecture (Red Daly) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 28 Oct 2008 15:44:51 +0000 > From: "Robert Synnott" > Subject: Re: [elephant-devel] Ditching Darcs > To: "Elephant bugs and development" > Message-ID: > <24f203480810280844q5c4a25b2nadb515775ef8500a at mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On this subject, is there a particular branch that I should be using > for postmodern? I've noticed occasional issues where it'll complain > that a table already exists, generally after a non-elephant error has > occurred within a transaction. > Rob > > > > ------------------------------ > > Message: 2 > Date: Tue, 28 Oct 2008 19:52:53 +0200 > From: "Alex Mizrahi" > Subject: Re: [elephant-devel] Ditching Darcs > To: elephant-devel at common-lisp.net > Message-ID: > > LPP> I must've missed something or unintentionally used a db > LPP> with older stuff already in it. > > yep, old format database could be the case if you've got it broken > at start. > > but it seems our current tests are broken, so you get always get some > error on the first run regardless of backend used. > > LPP> How about the NIL problem with secondary indices, you tried > LPP> to tackle that? > > not really, so far :( > > i thought some more about approach with NULLs and wrote down > queries and cases needed to handle NULLs -- it appears to be > more realistic now than it was before, but still pretty hairy, so i've > abandoned this for a while.. > > for example, cursor-next uses query > > WHERE (qi > $1) OR ((qi = $1) AND (value > $2)) > > and cursor-prev uses query > > WHERE (qi < $1) OR ((qi = $1) AND (value < $2)) > > being similar, they are made via template "WHERE (qi ~a $1) OR ((qi = $1) > AND (value ~a $2))" now. > > but with nulls that would be four different queries: > > cursor-next: > key is null: (qi IS NULL) and (value > $2) > key is not null: (qi > $1) OR (qi IS NULL) OR ((qi = 1) AND (value > $2) > cursor-prev > key is null: (qi IS NOT NULL) OR ((qi IS NULL) AND (value < $2)) > key is not null: (qi < $1) OR ((qi = $1) AND (value < $2)) > > and taking into account other cursor operations, instead of 3 query > templates > we now have something like 12 different queries now, and i see any pattern > how they can be merged :( > or maybe it makes sense to ditch templated query generations and just write > these conditions manually > > > > > > > ------------------------------ > > Message: 3 > Date: Tue, 28 Oct 2008 10:30:22 -0700 > From: "Red Daly" > Subject: [elephant-devel] Lisp Backend Architecture > To: "Elephant bugs and development" > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > I am hoping to clarify some of the prior discussions[1] about the native > Lisp backend for Elephant and propose a basic architecture. Hopefully we > can modularize development so if somebody wants to hack for a few days on > the backend he can avoid being overwhelmed. > > Multiprocess support: What features do the major lisps have for locking and > concurrency? What features are we missing from C/linux--what does BDB do in > this regard that is hard to do in Lisp? I do not know too much about > implementing efficient multi-threaded lisp programs where there is a lot of > contention. > > Distribution: Are others interested in making the backend distributed? I > prefer the design to allow adding an efficient distributed architecture at a > later date. > > Custom indexing: A lot of discussion has gone into the best implementation > practice for BTrees. But what about other indexing structures? > Multidimensional indexing is relevant to my project now because we have > spacial information (longitude latitude pairs) we would like to query. > Right now I am using a kd-tree with nodes made persistent by elephant, but > usually this sort of index would be implemented by the database itself. I > imagine multi-dimensional indexing could improve queries, too. > > There are other indexing needs as well. Document-level search is another > common case. I imagine an efficient search library is beyond our capacity, > but how could we make the database suited to implement search on top of the > thing? BTrees are not everything, unless I am missing something. > > > I propose the following basic architecture for the backend: > > Logging package with generic undo/redo logging support. It would be > customizable but it would not depend on other code from the Lisp backend. A > database could plug into the log by implementing undo, redo, and > serialization functions. By modularizing logging, it should make it easier > to hack on it without being familiar with the rest of the backend. It will > also be reusable, for what it's worth. > > A persistent heap. Beneath indices and data storage would be a heap-on-disk > layer with transaction support. The heap would have methods for reading, > writing, and locking sequences of bytes. It would also handle The > persistent heap alone would qualify as a database and might be useful as a > basis for other DB projects or experiments. Cachine would probably not be > implemented at this level, though that would be easiest. It may be possible > to implement the transactional heavy lifting for the persistent heap and > make it extensible enough to be used throughout. > > B-Trees. A MOP-based implementation of B-Trees may be customizable enough > to plug into the the rest of the system with a few additional method > declarations for the persistent, transactional version. Specialized methods > would access the persistent heap for reads and writes. Locking of the sort > done by BDB (level 2 etc.) could probably be accomplished in these methods > specialized for our persistent B-Tree. > > Bkd-Trees and other indexing structures could be implemented similarly to > B-Trees. > > Caching. B-Tree nodes or other lispy objects should be cached as opposed to > byte arrays. Caching of B-Tree nodes requires additional multiprocess code, > so not all the transactional magic happens in the persistent heap layer. > I'm a little fuzzy on exactly how this will work, but it sounds reasonable > enough. > > The DB user would interact mostly with the B-Tree and other indexing > structures, and with transactions. > > > How does this sound for a basic architecture? The multiprocess details are > not fleshed out so your comments are appreciated. I have attached my basic > implementation of a logger and persistent heap to make this clearer. > > > Best regards, > > Red > > > [1] The most lengthy discussion I can find is here: > http://common-lisp.net/pipermail/elephant-devel/2008-May/004108.html > The trac has a page on the Lisp backend: > http://trac.common-lisp.net/elephant/wiki/LispDataStore > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://common-lisp.net/pipermail/elephant-devel/attachments/20081028/d566669c/attachment-0001.html > -------------- next part -------------- > A non-text attachment was scrubbed... > Name: persistent-heap.tar.gz > Type: application/x-gzip > Size: 7163 bytes > Desc: not available > Url : http://common-lisp.net/pipermail/elephant-devel/attachments/20081028/d566669c/attachment-0001.bin > > ------------------------------ > > Message: 4 > Date: Tue, 28 Oct 2008 11:21:52 -0700 > From: "Red Daly" > Subject: Re: [elephant-devel] Lisp Backend Architecture > To: "Elephant bugs and development" > Message-ID: > > Content-Type: text/plain; charset="iso-8859-1" > > I believe the code I attached was scrubbed. Here is a link instead: > http://iodb.org/static/persistent-heap/ > > Red > > On Tue, Oct 28, 2008 at 10:30 AM, Red Daly wrote: > >> I am hoping to clarify some of the prior discussions[1] about the native >> Lisp backend for Elephant and propose a basic architecture. Hopefully we >> can modularize development so if somebody wants to hack for a few days on >> the backend he can avoid being overwhelmed. >> >> Multiprocess support: What features do the major lisps have for locking and >> concurrency? What features are we missing from C/linux--what does BDB do in >> this regard that is hard to do in Lisp? I do not know too much about >> implementing efficient multi-threaded lisp programs where there is a lot of >> contention. >> >> Distribution: Are others interested in making the backend distributed? I >> prefer the design to allow adding an efficient distributed architecture at a >> later date. >> >> Custom indexing: A lot of discussion has gone into the best implementation >> practice for BTrees. But what about other indexing structures? >> Multidimensional indexing is relevant to my project now because we have >> spacial information (longitude latitude pairs) we would like to query. >> Right now I am using a kd-tree with nodes made persistent by elephant, but >> usually this sort of index would be implemented by the database itself. I >> imagine multi-dimensional indexing could improve queries, too. >> >> There are other indexing needs as well. Document-level search is another >> common case. I imagine an efficient search library is beyond our capacity, >> but how could we make the database suited to implement search on top of the >> thing? BTrees are not everything, unless I am missing something. >> >> >> I propose the following basic architecture for the backend: >> >> Logging package with generic undo/redo logging support. It would be >> customizable but it would not depend on other code from the Lisp backend. A >> database could plug into the log by implementing undo, redo, and >> serialization functions. By modularizing logging, it should make it easier >> to hack on it without being familiar with the rest of the backend. It will >> also be reusable, for what it's worth. >> >> A persistent heap. Beneath indices and data storage would be a >> heap-on-disk layer with transaction support. The heap would have methods >> for reading, writing, and locking sequences of bytes. It would also handle >> The persistent heap alone would qualify as a database and might be useful >> as a basis for other DB projects or experiments. Cachine would probably not >> be implemented at this level, though that would be easiest. It may be >> possible to implement the transactional heavy lifting for the persistent >> heap and make it extensible enough to be used throughout. >> >> B-Trees. A MOP-based implementation of B-Trees may be customizable enough >> to plug into the the rest of the system with a few additional method >> declarations for the persistent, transactional version. Specialized methods >> would access the persistent heap for reads and writes. Locking of the sort >> done by BDB (level 2 etc.) could probably be accomplished in these methods >> specialized for our persistent B-Tree. >> >> Bkd-Trees and other indexing structures could be implemented similarly to >> B-Trees. >> >> Caching. B-Tree nodes or other lispy objects should be cached as opposed >> to byte arrays. Caching of B-Tree nodes requires additional multiprocess >> code, so not all the transactional magic happens in the persistent heap >> layer. I'm a little fuzzy on exactly how this will work, but it sounds >> reasonable enough. >> >> The DB user would interact mostly with the B-Tree and other indexing >> structures, and with transactions. >> >> >> How does this sound for a basic architecture? The multiprocess details are >> not fleshed out so your comments are appreciated. I have attached my basic >> implementation of a logger and persistent heap to make this clearer. >> >> >> Best regards, >> >> Red >> >> >> [1] The most lengthy discussion I can find is here: >> http://common-lisp.net/pipermail/elephant-devel/2008-May/004108.html >> The trac has a page on the Lisp backend: >> http://trac.common-lisp.net/elephant/wiki/LispDataStore >> > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: http://common-lisp.net/pipermail/elephant-devel/attachments/20081028/75c72940/attachment.html > > ------------------------------ > > _______________________________________________ > elephant-devel mailing list > elephant-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/elephant-devel > > > End of elephant-devel Digest, Vol 5, Issue 12 > ********************************************* > From killerstorm at newmail.ru Tue Oct 28 19:38:04 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 28 Oct 2008 21:38:04 +0200 Subject: [elephant-devel] Ditching Darcs References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com><62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org><61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org><1224804755.21512.1305.camel@penguin.yourdomain.com><63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org><5F0DC9C7-B3CB-40D8-8DC5-00A9484B3431@media.mit.edu> <24f203480810280844q5c4a25b2nadb515775ef8500a@mail.gmail.com> Message-ID: RS> On this subject, is there a particular branch that I should be using RS> for postmodern? first of all, definitely a version from darcs rather than a 0.9.1 release. release version has lots of bugs. as for stable/unstable branches, there shouldn't be a big difference as most stuff is same and new one was just extended a bit to handle new features. on the other hand, new one is almost not tested -- except with automated tests we have. so, use at your own risk.. RS> I've noticed occasional issues where it'll complain that a table RS> already exists, generally after a non-elephant error has RS> occurred within a transaction. this might be a known problem -- when you touch index for the first time in a transaction, and that transaction fails, then in-memory index cache gets desynchronized with stuff in database. this is not a postmodern-specific issue, but a problem of index cache organization. if this gets annoying, there are some workarounds: * for a production environment, you can just ensure that you "touch" all indices, classes etc. in a safe environment without errors. then if you'll have errors in runtime it won't lead to erroneous behaviour. * in a testing environment where bugs in code are expected, you can add this option to your config sexp: (:enable-multi-store-indexing . t) and reopen store for each transaction. i use this for testing and it seems to be working fine, but i suspect it might leak some memory, so this approach is not good for a production environment. but i'm not sure that this known problem should produce "table already exist" error, could be some other bug. if you want us to investigate it please send full error description, a backtrace, and maybe some idea what transactions were doing From killerstorm at newmail.ru Tue Oct 28 20:02:29 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 28 Oct 2008 22:02:29 +0200 Subject: [elephant-devel] Ditching Darcs References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com><62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org><61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org><1224804755.21512.1305.camel@penguin.yourdomain.com><63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org><63473.88.73.226.2.1225205117.squirrel@mail.stardawn.org> <64898.88.73.226.2.1225219846.squirrel@mail.stardawn.org> Message-ID: ??>> and taking into account other cursor operations, instead of 3 query ??>> templates we now have something like 12 different queries now, and ??>> i see any pattern how they can be merged :( ??>> or maybe it makes sense to ditch templated query generations and just ??>> write these conditions manually LPP> I took a look at it. CURSOR-FETCH-QUERY is complicated enough as it LPP> is, so let's reduce it to some common primitives as far as possible LPP> and implement our NULL cases manually. it seems only it's where-generating part needs to be replaced (as well as parameters value-compare and key-compare), or you have some other restructuring in mind? here's the whole table btw: cursor-next (duplicates allowed) key is null: (qi IS NULL) and (value > $2) key is not null: (qi > $1) OR (qi IS NULL) OR ((qi = $1) AND (value > $2) cursor-prev (duplicates allowed) key is null: (qi IS NOT NULL) OR ((qi IS NULL) AND (value < $2)) key is not null: (qi < $1) OR ((qi = $1) AND (value < $2)) cursor-next-nodup: key is null: do nothing key is not null: (qi > $1) OR (qi is NULL) cursor-prev-nodup: key is null: (qi IS NOT NULL) key is not null: (qi < $1) cursor-set-range: key is null: (qi IS NULL) key is not null: (qi >= $1) OR (qi IS NULL) cursor-get-both-range: key is null: (qi IS NULL) AND (value >= $2) key is not null: (qi = $1) AND (value >= $2) i don't see some clear patter here, so we can just explicitly pass where part to cursor-fetch-query function. also i have some concerns that postgres might fail to optimize bigger conditionals in where.. From sky at viridian-project.de Wed Oct 29 09:55:20 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Wed, 29 Oct 2008 10:55:20 +0100 (CET) Subject: [elephant-devel] Ditching Darcs In-Reply-To: References: <62900.88.73.253.66.1224663001.squirrel@mail.stardawn.org><1224724548.21512.1232.camel@penguin.yourdomain.com><62630.88.73.226.69.1224752465.squirrel@mail.stardawn.org><61605.88.73.226.69.1224765524.squirrel@mail.stardawn.org><1224804755.21512.1305.camel@penguin.yourdomain.com><63201.88.73.215.90.1224832308.squirrel@mail.stardawn.org><63473.88.73.226.2.1225205117.squirrel@mail.stardawn.org> <64898.88.73.226.2.1225219846.squirrel@mail.stardawn.org> Message-ID: <63304.88.73.195.210.1225274120.squirrel@mail.stardawn.org> > LPP> I took a look at it. CURSOR-FETCH-QUERY is complicated enough as it > LPP> is, so let's reduce it to some common primitives as far as possible > LPP> and implement our NULL cases manually. > > it seems only it's where-generating part needs to be replaced (as well as > parameters value-compare and key-compare), > or you have some other restructuring in mind? That's exactly what I had in mind. > i don't see some clear patter here, so we can just explicitly pass where > part to cursor-fetch-query function. Yes. > also i have some concerns that postgres might fail to optimize bigger > conditionals in where.. You might be right here, but I suppose that this type of WHERE clause with NULL checks might be optimized. We can compare benchmarks to be sure. Leslie