From leslie.polzer at gmx.net Mon Sep 1 12:41:38 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 1 Sep 2008 14:41:38 +0200 (CEST) Subject: [elephant-devel] Fix for class redefinition schema bug Message-ID: <61240.88.73.253.235.1220272898.squirrel@mail.stardawn.org> Problem: when a class schema is introduced its db schema doesn't get saved. Proposal: --- old-unstable/src/elephant/classes.lisp 2008-09-01 14:37:20.462677601 +0200 +++ new-unstable/src/elephant/classes.lisp 2008-09-01 14:37:20.462677601 +0200 @@ -92,16 +92,18 @@ (defmethod finalize-inheritance :after ((instance persistent-metaclass)) "Constructs the metaclass schema when the class hierarchy is valid" (let* ((old-schema (%class-schema instance)) (new-schema (compute-schema instance))) ;; Update schema chain (setf (schema-predecessor new-schema) old-schema) (setf (%class-schema instance) new-schema) + (when *store-controller* + (lookup-schema *store-controller* instance)) ; ensure db schema ;; Cleanup some slot values (let ((idx-state (%class-indexing instance))) (when (consp idx-state) (setf (%class-indexing instance) (first idx-state)))) ;; Compute derived index triggers (awhen (derived-index-slot-defs instance) (compute-derived-index-triggers instance it)) ;; Synchronize instances to new schemas Issue: What happens when the store controller isn't open at class definition time? Is this a problem? Should we perhaps walk all persistent classes and ensure that they have db schemas whenever a store controller is opened? Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Mon Sep 8 18:31:22 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 8 Sep 2008 20:31:22 +0200 (CEST) Subject: [elephant-devel] [PATCH] Ensure schema on FINALIZE-INHERITANCE Message-ID: <64684.88.73.246.154.1220898682.squirrel@mail.stardawn.org> [this message comes late because it bounced at the first time] Hinted at this a few days ago in an earlier message. This patch is for the "unstable" version of Elephant and fixes the on-off test errors. All tests run cleanly now with BDB, even when run multiple times starting with a fresh or existing db. I'm not quite content that the border case is filtered by subtypep of 'btree, but I don't see any other way and it works fine as it is. Cheers, Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: ensure-schema-on-finalize-inheritance.darcspatch Type: application/octet-stream Size: 16260 bytes Desc: not available URL: From leslie.polzer at gmx.net Tue Sep 9 13:32:05 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 9 Sep 2008 15:32:05 +0200 (CEST) Subject: [elephant-devel] Re: [PATCH] Ensure schema on FINALIZE-INHERITANCE Message-ID: <64931.88.73.233.242.1220967125.squirrel@mail.stardawn.org> [...and another bounced mail] Boolean mix-up in previous patch, updated patch attached. -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: ensure-schema-on-finalize-inheritance.darcspatch Type: application/octet-stream Size: 16266 bytes Desc: not available URL: From leslie.polzer at gmx.net Tue Sep 9 19:35:15 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Tue, 9 Sep 2008 21:35:15 +0200 (CEST) Subject: [elephant-devel] Support for db4.7 Message-ID: <63513.88.73.233.242.1220988915.squirrel@mail.stardawn.org> Darcs patch attached; passes all tests. Darcs context parent is the FINALIZE-INHERITANCE patch I posted before. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: db47.darcspatch Type: application/octet-stream Size: 17940 bytes Desc: not available URL: From read at robertlread.net Thu Sep 11 02:42:50 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 10 Sep 2008 21:42:50 -0500 Subject: [elephant-devel] Support for db4.7 In-Reply-To: <63513.88.73.233.242.1220988915.squirrel@mail.stardawn.org> References: <63513.88.73.233.242.1220988915.squirrel@mail.stardawn.org> Message-ID: <1221100970.14403.1906.camel@penguin.yourdomain.com> Dear Leslie, I have applied your two most recent patches to elephant-unstable; they seemed to pass the tests under BDB for me as well. Thanks for your contribution; let me know if I didn't get it quiet right. On Tue, 2008-09-09 at 21:35 +0200, Leslie P. Polzer wrote: > Darcs patch attached; passes all tests. > > Darcs context parent is the FINALIZE-INHERITANCE patch I posted > before. > > Leslie > > -- > LinkedIn Profile: http://www.linkedin.com/in/polzer > Xing Profile: https://www.xing.com/profile/LeslieP_Polzer > Blog: http://blog.viridian-project.de/ > > > -- > LinkedIn Profile: http://www.linkedin.com/in/polzer > Xing Profile: https://www.xing.com/profile/LeslieP_Polzer > Blog: http://blog.viridian-project.de/ > _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Thu Sep 11 08:27:46 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Thu, 11 Sep 2008 10:27:46 +0200 Subject: [elephant-devel] Support for db4.7 In-Reply-To: <1221109266.10480.31.camel@blackbox> References: <1221109266.10480.31.camel@blackbox> Message-ID: <20080911082746.18260@gmx.net> > I have applied your two most recent patches to elephant-unstable; they > seemed to pass the tests under BDB for me as well. Thanks for your > contribution; let me know if I didn't get it quiet right. Thanks a lot for jumping in, Robert; looks fine so far. :) I have noticed corner cases in the FINALIZE-INHERITANCE stuff that require more changes, but the patch is fine so far. Leslie -- GMX startet ShortView.de. Hier findest Du Leute mit Deinen Interessen! Jetzt dabei sein: http://www.shortview.de/wasistshortview.php?mc=sv_ext_mf at gmx From elliottslaughter at gmail.com Fri Sep 12 23:36:22 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Fri, 12 Sep 2008 16:36:22 -0700 Subject: [elephant-devel] Elephant on Windows/SBCL? Message-ID: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> Hi, I would like to port a project of mine which currently uses Allegro/ACache to Elephant. Installing and loading Elephant 0.9.1 on Windows using Allegro 8.1, the precompiled dlls, and the default config.sexpr worked fine, but an identical configuration in SBCL 1.0.13 failed with the following error: ; compiling file "C:\\Program Files\\asdf\\elephant\\src\\utils\\os.lisp" (written 12 SEP 2008 03:09:13 PM): ; compiling (IN-PACKAGE :ELEPHANT-UTILS); compilation aborted because of fatal error: ; READ failure in COMPILE-FILE: ; SB-INT:SIMPLE-READER-PACKAGE-ERROR at 885 (line 27, column 27) on #: ; package "SB-POSIX" not found debugger invoked on a SB-INT:SIMPLE-FILE-ERROR: couldn't remove "C:\\Program Files\\asdf\\elephant\\src\\utils\\os.fasl": Permission denied Which is caused by using sb-posix:chdir in in-directory because sb-posix doesn't exist in win32 builds of SBCL. I can't seem to find any quick fixes, but http://brainrack.wordpress.com/2008/05/29/running-sbcl-on-windows/ might help. How much work would it be to get Elephant to work on Windows/SBCL? Do you need threads or anything else that just isn't available for Windows on SBCL? Or is it just that no one happens to test that combination of platforms? Thanks. -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Sat Sep 13 03:07:44 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 12 Sep 2008 22:07:44 -0500 Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> Message-ID: <1221275264.14403.2148.camel@penguin.yourdomain.com> I think it largely that no one tests on that platform. However, I personally don't know what the state of readiness of any lisp on Windows is. Elephant itself is moderate in its use of outside systems, such as CL-SQL and BDB. Someone who already had some experience porting to a Windows platform could answer this better --- or maybe we even have a user who has done it. If all of the pieces are in place, I would guess it would be about a three-day task, but it could be more, and someone with experience might do it faster. I would be happy to answer your questions about Elephant as it is, but I don't plan to start running windows. On Fri, 2008-09-12 at 16:36 -0700, Elliott Slaughter wrote: > Hi, > > > I would like to port a project of mine which currently uses > Allegro/ACache to Elephant. Installing and loading Elephant 0.9.1 on > Windows using Allegro 8.1, the precompiled dlls, and the default > config.sexpr worked fine, but an identical configuration in SBCL > 1.0.13 failed with the following error: > > > ; compiling file "C:\\Program Files\\asdf\\elephant\\src\\utils\ > \os.lisp" (written 12 SEP 2008 03:09:13 PM): > ; compiling (IN-PACKAGE :ELEPHANT-UTILS); compilation aborted because > of fatal error: > ; READ failure in COMPILE-FILE: > ; SB-INT:SIMPLE-READER-PACKAGE-ERROR at 885 (line 27, column 27) > on # \\utils\\os.lisp" {AB2CC49}>: > ; package "SB-POSIX" not found > > > debugger invoked on a SB-INT:SIMPLE-FILE-ERROR: > couldn't remove "C:\\Program Files\\asdf\\elephant\\src\\utils\ > \os.fasl": > Permission denied > > > Which is caused by using sb-posix:chdir in in-directory because > sb-posix doesn't exist in win32 builds of SBCL. I can't seem to find > any quick fixes, > but http://brainrack.wordpress.com/2008/05/29/running-sbcl-on-windows/ might help. > > > How much work would it be to get Elephant to work on Windows/SBCL? Do > you need threads or anything else that just isn't available for > Windows on SBCL? Or is it just that no one happens to test that > combination of platforms? > > > Thanks. > > > -- > Elliott Slaughter > > "Any road followed precisely to its end leads precisely nowhere." - > Frank Herbert > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From sky at viridian-project.de Sat Sep 13 08:49:33 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sat, 13 Sep 2008 10:49:33 +0200 (CEST) Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> Message-ID: <64125.88.73.238.75.1221295773.squirrel@mail.stardawn.org> > Which is caused by using sb-posix:chdir in in-directory because sb-posix > doesn't exist in win32 builds of SBCL. I can't seem to find any quick fixes, > but http://brainrack.wordpress.com/2008/05/29/running-sbcl-on-windows/ might > help. Yes, those alternative implementations seem to be just what you need. > How much work would it be to get Elephant to work on Windows/SBCL? Do you > need threads or anything else that just isn't available for Windows on SBCL? Threads shouldn't be required. The UFFI homepage doesn't mention SBCL/Win32 as a working platform. Not sure if that really means anything, though. > Or is it just that no one happens to test that combination of platforms? I'm not doing any development on win32 if I can avoid it. I'd be willing to support your efforts to the best I can, though, even if it means setting up SBCL under win32. Happy hacking, Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Sat Sep 13 09:12:43 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sat, 13 Sep 2008 05:12:43 -0400 Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <1221275264.14403.2148.camel@penguin.yourdomain.com> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> <1221275264.14403.2148.camel@penguin.yourdomain.com> Message-ID: <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> Elephant should work with Allegro on Win 32, it was my original development platform for Elephant, although I haven't tried it in some time. An earlier version of Elephant worked on Lispworks/Windows although I think I recall some problems with the latest version of the unstable tree. My impression from the last time I tried it was that SBCL was insufficiently mature on Win32 to be a reasonable configuration to target. Has this changed? Is it getting more use? I would check UFFI support first - for running with BDB that's likely to be the hairy part. Replacing the sb-posix calls for Windows shouldn't be too hard as Robert mentions. Threads aren't needed for simple apps; I'm assuming you aren't going to be running this under a lisp web server which seems to be the most common use of Elephant. Ian On Sep 12, 2008, at 11:07 PM, Robert L. Read wrote: > I think it largely that no one tests on that platform. > > However, I personally don't know what the state of readiness of any > lisp > on Windows is. Elephant itself is moderate in its use of outside > systems, such as CL-SQL and BDB. > > Someone who already had some experience porting to a Windows platform > could answer this better --- or maybe we even have a user who has done > it. > > If all of the pieces are in place, I would guess it would be about a > three-day task, but it could be more, and someone with experience > might > do it faster. > > I would be happy to answer your questions about Elephant as it is, > but I > don't plan to start running windows. > > > On Fri, 2008-09-12 at 16:36 -0700, Elliott Slaughter wrote: >> Hi, >> >> >> I would like to port a project of mine which currently uses >> Allegro/ACache to Elephant. Installing and loading Elephant 0.9.1 on >> Windows using Allegro 8.1, the precompiled dlls, and the default >> config.sexpr worked fine, but an identical configuration in SBCL >> 1.0.13 failed with the following error: >> >> >> ; compiling file "C:\\Program Files\\asdf\\elephant\\src\\utils\ >> \os.lisp" (written 12 SEP 2008 03:09:13 PM): >> ; compiling (IN-PACKAGE :ELEPHANT-UTILS); compilation aborted because >> of fatal error: >> ; READ failure in COMPILE-FILE: >> ; SB-INT:SIMPLE-READER-PACKAGE-ERROR at 885 (line 27, column 27) >> on #> \src >> \\utils\\os.lisp" {AB2CC49}>: >> ; package "SB-POSIX" not found >> >> >> debugger invoked on a SB-INT:SIMPLE-FILE-ERROR: >> couldn't remove "C:\\Program Files\\asdf\\elephant\\src\\utils\ >> \os.fasl": >> Permission denied >> >> >> Which is caused by using sb-posix:chdir in in-directory because >> sb-posix doesn't exist in win32 builds of SBCL. I can't seem to find >> any quick fixes, >> but http://brainrack.wordpress.com/2008/05/29/running-sbcl-on-windows/ >> might help. >> >> >> How much work would it be to get Elephant to work on Windows/SBCL? Do >> you need threads or anything else that just isn't available for >> Windows on SBCL? Or is it just that no one happens to test that >> combination of platforms? >> >> >> Thanks. >> >> >> -- >> Elliott Slaughter >> >> "Any road followed precisely to its end leads precisely nowhere." - >> Frank Herbert >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From elliottslaughter at gmail.com Sat Sep 13 21:15:02 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Sat, 13 Sep 2008 14:15:02 -0700 Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> <1221275264.14403.2148.camel@penguin.yourdomain.com> <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> Message-ID: <42c0ab790809131415s5d64a5bel1012a33252e9552a@mail.gmail.com> On Sat, Sep 13, 2008 at 2:12 AM, Ian Eslick wrote: > Elephant should work with Allegro on Win 32, it was my original development > platform for Elephant, although I haven't tried it in some time. An earlier > version of Elephant worked on Lispworks/Windows although I think I recall > some problems with the latest version of the unstable tree. Ok. > My impression from the last time I tried it was that SBCL was insufficiently > mature on Win32 to be a reasonable configuration to target. Has this > changed? Is it getting more use? I don't know exactly what qualifies as "more use", but I believe that every SBCL release since about 1.0 builds cleanly on win32, and I don't generally crash it in the sorts of things I work on... although it isn't quite stable enough to remove the "Your Kitten of Death awaits!" warning. > I would check UFFI support first - for running with BDB that's likely to be > the hairy part. Replacing the sb-posix calls for Windows shouldn't be too > hard as Robert mentions. Threads aren't needed for simple apps; I'm > assuming you aren't going to be running this under a lisp web server which > seems to be the most common use of Elephant. UFFI seems to load fine under SBCL/win32, although the test suite fails at compiling gethostname.lisp. (I'm not sure if that is significant or not... it looks like the tests also fail under Allegro/win32, although not in the same way.) My project is actually a video game where I am trying to use persistence in the physical simulation. I don't know if Elephant is actually fast enough to be used for this, but I have had some success with ACache, so I'd like to try :-) -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert From killerstorm at newmail.ru Sat Sep 13 21:24:35 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 14 Sep 2008 00:24:35 +0300 Subject: [elephant-devel] Elephant on Windows/SBCL? References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com><1221275264.14403.2148.camel@penguin.yourdomain.com><1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> <42c0ab790809131415s5d64a5bel1012a33252e9552a@mail.gmail.com> Message-ID: <4478B25746014E80ACF14D69F84BF8D7@killer> ??>> My impression from the last time I tried it was that SBCL was ??>> insufficiently mature on Win32 to be a reasonable configuration to ??>> target. Has this changed? Is it getting more use? ES> I don't know exactly what qualifies as "more use", but I believe that ES> every SBCL release since about 1.0 builds cleanly on win32, and I ES> don't generally crash it in the sorts of things I work on... it might work on your machine, but fail on neighbor machine, because SBCL depends on allocating huge continuous chunk of address space, and address space availability depends on dlls loaded etc. so IMHO SBCL/Win32 is not a good/stable platform, and i'm afraid it will never be such. From elliottslaughter at gmail.com Sun Sep 14 05:43:58 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Sat, 13 Sep 2008 22:43:58 -0700 Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> <1221275264.14403.2148.camel@penguin.yourdomain.com> <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> Message-ID: <42c0ab790809132243y17be2d14t1a3f303d12c7a747@mail.gmail.com> On Sat, Sep 13, 2008 at 2:12 AM, Ian Eslick wrote: > I would check UFFI support first - for running with BDB that's likely to be > the hairy part. Alternatively, would it be possible to use CFFI instead (with the UFFI compatibility layer if necessary)? I know CFFI works with SBCL/win32, and other currently unsupported implementations (e.g. CLISP). -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert From sky at viridian-project.de Sun Sep 14 08:57:07 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sun, 14 Sep 2008 10:57:07 +0200 (CEST) Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <42c0ab790809132243y17be2d14t1a3f303d12c7a747@mail.gmail.com> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> <1221275264.14403.2148.camel@penguin.yourdomain.com> <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> <42c0ab790809132243y17be2d14t1a3f303d12c7a747@mail.gmail.com> Message-ID: <63718.88.73.230.60.1221382627.squirrel@mail.stardawn.org> > Alternatively, would it be possible to use CFFI instead (with the UFFI compatibility layer if necessary)? I know CFFI works with SBCL/win32, and other currently unsupported implementations (e.g. CLISP). I tried this once but ran into problems with memutil (IIRC). At this point there are several possibilities: a) fix CFFI's UFFI emulation b) write a memutil version that uses CFFI instead of UFFI c) write conditionals for the problematic parts that use CFFI instead of CFFI's UFFI emulation d) write a memutil version that uses CL exclusively Not sure how involved a) would be. Better ask the CFFI guys. b) seems nice but may be a bit of work if you don't know much about FFI stuff. c) might be the best route in terms of time and speed. Again, ask the CFFI guys. Version d) is probably slower esp. with BDB, but it might be a quick and clean drop-in. So talk a bit to the regulars of cffi-devel and they might be able to tell us something useful. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From sky at viridian-project.de Sun Sep 14 09:00:00 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sun, 14 Sep 2008 11:00:00 +0200 (CEST) Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <42c0ab790809131415s5d64a5bel1012a33252e9552a@mail.gmail.com> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> <1221275264.14403.2148.camel@penguin.yourdomain.com> <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> <42c0ab790809131415s5d64a5bel1012a33252e9552a@mail.gmail.com> Message-ID: <63726.88.73.230.60.1221382800.squirrel@mail.stardawn.org> > UFFI seems to load fine under SBCL/win32, although the test suite > fails at compiling gethostname.lisp. (I'm not sure if that is > significant or not... it looks like the tests also fail under > Allegro/win32, although not in the same way.) Not sure whether win32 has got gethostname at all, I believe it's a UNIX thing (unistd.h). So that's probably no indicator for certain doom. :) > My project is actually a video game where I am trying to use > persistence in the physical simulation. I don't know if Elephant is > actually fast enough to be used for this, but I have had some success > with ACache, so I'd like to try :-) I haven't had a chance to try ACache but I suppose Elephant with BDB should be on par. Ian might know more about that. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Sun Sep 14 12:14:59 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sun, 14 Sep 2008 13:14:59 +0100 Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <42c0ab790809132243y17be2d14t1a3f303d12c7a747@mail.gmail.com> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> <1221275264.14403.2148.camel@penguin.yourdomain.com> <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> <42c0ab790809132243y17be2d14t1a3f303d12c7a747@mail.gmail.com> Message-ID: <0D54F30F-B0BE-4A35-9C01-EFC6DC299AB8@media.mit.edu> I've tried to port to CFFI, and I think someone else has too, but I don't think anyone has ever figured out and resolved the errors we've seen with it. Ian On Sep 14, 2008, at 6:43 AM, Elliott Slaughter wrote: > On Sat, Sep 13, 2008 at 2:12 AM, Ian Eslick > wrote: >> I would check UFFI support first - for running with BDB that's >> likely to be >> the hairy part. > > Alternatively, would it be possible to use CFFI instead (with the UFFI > compatibility layer if necessary)? I know CFFI works with SBCL/win32, > and other currently unsupported implementations (e.g. CLISP). > > -- > Elliott Slaughter > > "Any road followed precisely to its end leads precisely nowhere." - > Frank Herbert > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Sun Sep 14 13:54:00 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sun, 14 Sep 2008 14:54:00 +0100 Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <63726.88.73.230.60.1221382800.squirrel@mail.stardawn.org> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> <1221275264.14403.2148.camel@penguin.yourdomain.com> <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> <42c0ab790809131415s5d64a5bel1012a33252e9552a@mail.gmail.com> <63726.88.73.230.60.1221382800.squirrel@mail.stardawn.org> Message-ID: >> My project is actually a video game where I am trying to use >> persistence in the physical simulation. I don't know if Elephant is >> actually fast enough to be used for this, but I have had some success >> with ACache, so I'd like to try :-) > > I haven't had a chance to try ACache but I suppose Elephant with BDB > should be on par. Ian might know more about that. > > Leslie I haven't done a performance comparison of the two systems. I suspect that Acache is probably faster at least for certain tasks. Disk and memory layout as well as the transaction architecture have large impacts on what works well and doesn't and they've optimized for certain operations (such as ops over RDF triples). Also it's tightly integrated into the Allegro lisp/compiler/IO layer, doesn't have multiple backends so suffers from less indirection, etc. It would be interesting to get some data if you have it; I'd like to do a lisp-only version of Elephant someday and ACache is close enough that it might make sense to borrow from the architecture to get some of the same performance that they've managed to produce. Cheers, Ian From elliottslaughter at gmail.com Mon Sep 15 05:10:21 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Sun, 14 Sep 2008 22:10:21 -0700 Subject: [elephant-devel] Elephant on Windows/SBCL? In-Reply-To: <63726.88.73.230.60.1221382800.squirrel@mail.stardawn.org> References: <42c0ab790809121636s1c4ea025u35e61b0d8184650b@mail.gmail.com> <1221275264.14403.2148.camel@penguin.yourdomain.com> <1F2C4FC9-4957-4179-86C8-3B5A85F81292@media.mit.edu> <42c0ab790809131415s5d64a5bel1012a33252e9552a@mail.gmail.com> <63726.88.73.230.60.1221382800.squirrel@mail.stardawn.org> Message-ID: <42c0ab790809142210r7e73b156j7387ed90fd122951@mail.gmail.com> On Sun, Sep 14, 2008 at 2:00 AM, Leslie P. Polzer wrote: > > > UFFI seems to load fine under SBCL/win32, although the test suite > > fails at compiling gethostname.lisp. (I'm not sure if that is > > significant or not... it looks like the tests also fail under > > Allegro/win32, although not in the same way.) > > Not sure whether win32 has got gethostname at all, I believe > it's a UNIX thing (unistd.h). > > So that's probably no indicator for certain doom. :) Ok, here's my first attempt at hacking Elephant to work with SBCL/win32. (I'm hoping UFFI with SBCL/win32 will just work... I'll try the CFFI alternatives otherwise.) First, I tried using the code from http://brainrack.wordpress.com/2008/05/29/running-sbcl-on-windows/ to deal with not having sb-posix:chdir available. For some reason this resulted in failing to compile libmemutil.c (even though my-config.sexpr contains "(:prebuilt-libraries . t)" which should prevent even attempting to compile the file). Then I noticed all the Windows conditionals were of the form #+(or mswindows windows), which needed to be changed to #+(or mswindows windows win32) to work under SBCL/win32. These changes seem to allow compiling and loading elephant.asd (albeit with 4 warnings, 11 style warnings and 501 notes). I even managed to (open-store '(:bdb "/path/to/some/db/")) and do some basic operations, again with many warnings. This probably isn't anywhere near stable, but at least it looks like it could be made to work. -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From elliottslaughter at gmail.com Tue Sep 16 00:31:50 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Mon, 15 Sep 2008 17:31:50 -0700 Subject: [elephant-devel] Class Heirarchies and Queries Message-ID: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> Section 2.10 of the manual mentions that you can "query a base class of type people to get subclass instances such as employee, manager, consultant, etc". I happen to need to do exactly that, but I can't actually find the appropriate section in the manual. Thanks. -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Tue Sep 16 09:28:58 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 16 Sep 2008 11:28:58 +0200 (CEST) Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> Message-ID: <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> > Section > 2.10 of the manual mentions that you can "query a base class of type > people to get subclass instances such as employee, manager, > consultant, etc". > I happen to need to do exactly that, but I can't actually find the > appropriate section in the manual. I don't know where that section is (if it exists at all), but you can use something like (untested, haven't got the code at hand): (defun get-instances-by-class* (classname) (flatten (loop for c in (mop:class-direct-subclasses (find-class classname)) collect (get-instances-by-class classname)))) Hope that helps! I think an appropriate keyword arg (:include-sublasses-p t) should be added to those functions. Care to submit a patch for it sometime? :) If not then I'll do it eventually. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From killerstorm at newmail.ru Tue Sep 16 11:22:50 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 16 Sep 2008 14:22:50 +0300 Subject: [elephant-devel] Class Heirarchies and Queries References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> Message-ID: <3237650F6D014CF19571172A4E5B386C@killer> ??>> Section ??>> 2.10 of the manual mentions that you can "query a base class of type ??>> people to get subclass instances such as employee, manager, ??>> consultant, etc". ??>> I happen to need to do exactly that, but I can't actually find the ??>> appropriate section in the manual. LPP> I don't know where that section is (if it exists at all), there is no such thing in "old" version for sure, i dunno about "new version" though. definitely this is an important thing, without that queries suck. while it's fairly easy to implement it for get-instances-by-class, things are more tricky for "-by-range" queries because they need to be sorted, and even more tricky for "map-*" level. as i've mentioned once, i have sort of "combined cursors" implementation, with it we can support this "subclass aggregation" on low level map-* queries, so get-instances-* will get it almost automatically, and stuff will be properly ordered etc. so do you people think that inheritance support on map level is a right thing? From sky at viridian-project.de Tue Sep 16 11:53:17 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 16 Sep 2008 13:53:17 +0200 (CEST) Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <3237650F6D014CF19571172A4E5B386C@killer> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> <3237650F6D014CF19571172A4E5B386C@killer> Message-ID: <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> > so do you people think that inheritance support on map level is a right > thing? Yes, definitely. Got a code draft? -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Tue Sep 16 14:20:27 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 16 Sep 2008 10:20:27 -0400 Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> Message-ID: <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> There is an :inherit slot keyword option for indexed slots. It causes subclasses of the defining class to share the index of the base class. You can inhibit subclass sharing by re-defining the slot in the base class. I found a bug in unstable and patched it so this works again. Calling map-inverted-index (and the top-level API fn's get-instances- by-value/range) from the base or subclasses will return instances for all classes that share that base index. We could add some refinements such as mapping from a subclass filters out any superclass instances during the traversal. We can easily add a keyword arg to map-class and get-instances-by- class to return all subclass instances as well. Ian On Sep 16, 2008, at 7:53 AM, Leslie P. Polzer wrote: > >> so do you people think that inheritance support on map level is a >> right >> thing? > > Yes, definitely. > > Got a code draft? > > -- > LinkedIn Profile: http://www.linkedin.com/in/polzer > Xing Profile: https://www.xing.com/profile/LeslieP_Polzer > Blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From elliottslaughter at gmail.com Tue Sep 16 19:03:09 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Tue, 16 Sep 2008 12:03:09 -0700 Subject: [elephant-devel] Elephant Dependency on Bordeaux Threads Message-ID: <42c0ab790809161203h1900d7ay9426da7850bc24b5@mail.gmail.com> Hi, I just upgraded my Elephant installation from 0.9.1 to the newest on darcs, and I am wondering why Elephant has added a dependency on Bordeaux Threads? I don't think Elephant is inherently multi threaded, and Elephant/BDB still loads fine when the dependency is commented out. Admittedly, that makes the tests fail, but if you only need threads for testing, why not just move the dependency into the elephant-tests system definition? -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From elliottslaughter at gmail.com Wed Sep 17 05:02:55 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Tue, 16 Sep 2008 22:02:55 -0700 Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> Message-ID: <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> On Tue, Sep 16, 2008 at 7:20 AM, Ian Eslick wrote: > There is an :inherit slot keyword option for indexed slots. It causes > subclasses of the defining class to share the index of the base class. You > can inhibit subclass sharing by re-defining the slot in the base class. I > found a bug in unstable and patched it so this works again. This looks like what I need. Unfortunately I can't seem to compile the necessary C files to run elephant-unstable. Even when I copy the "gcc ..." command into cygwin and compile manually, I still get this when I try to use the resultant dll file: $ gcc -mno-cygwin -mwindows -Wall -c -std=c99 "c:\Program Files\asdf\elephant-unstable\src\memutil\libmemutil.c" -o "c:\Program Files\asdf\elephant-unstable\src\memutil\libmemutil.dll" Error opening shared object "c:\\Program Files\\asdf\\elephant-unstable\\src\\memutil\\libmemutil.dll": 5. [Condition of type SIMPLE-ERROR] Can anyone who works regularly with Windows give me advice on compiling these dependencies? Thanks. Calling map-inverted-index (and the top-level API fn's > get-instances-by-value/range) from the base or subclasses will return > instances for all classes that share that base index. We could add some > refinements such as mapping from a subclass filters out any superclass > instances during the traversal. > > We can easily add a keyword arg to map-class and get-instances-by-class to > return all subclass instances as well. > These might be useful as well, but I won't need them immediately (and I've got more pressing concerns anyways). -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From killerstorm at newmail.ru Wed Sep 17 07:53:46 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Wed, 17 Sep 2008 10:53:46 +0300 Subject: [elephant-devel] Elephant Dependency on Bordeaux Threads References: <42c0ab790809161203h1900d7ay9426da7850bc24b5@mail.gmail.com> Message-ID: <19031828491C406AA04A25AF16FE6ED5@killer> > I don't think Elephant is inherently multi threaded, and Elephant/BDB Elephant/pgsql uses thread API to reap orphaned connections. > why not just move the dependency into the elephant-tests system definition? yep we could make it more fine-grained, i think, but what is a problem with bordeaux-threads anyway? iirc it just works via dummy functions if threads are not supported by implementation, so it's totally safe From killerstorm at newmail.ru Wed Sep 17 07:53:46 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Wed, 17 Sep 2008 10:53:46 +0300 Subject: [elephant-devel] Elephant Dependency on Bordeaux Threads References: <42c0ab790809161203h1900d7ay9426da7850bc24b5@mail.gmail.com> Message-ID: <19031828491C406AA04A25AF16FE6ED5@killer> > I don't think Elephant is inherently multi threaded, and Elephant/BDB Elephant/pgsql uses thread API to reap orphaned connections. > why not just move the dependency into the elephant-tests system definition? yep we could make it more fine-grained, i think, but what is a problem with bordeaux-threads anyway? iirc it just works via dummy functions if threads are not supported by implementation, so it's totally safe From sky at viridian-project.de Wed Sep 17 08:31:51 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Wed, 17 Sep 2008 10:31:51 +0200 (CEST) Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> Message-ID: <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> > Error opening shared object "c:\\Program > Files\\asdf\\elephant-unstable\\src\\memutil\\libmemutil.dll": > 5. > [Condition of type SIMPLE-ERROR] > > Can anyone who works regularly with Windows give me advice on compiling > these dependencies? Thanks. I don't, but got a backtrace? -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From elliottslaughter at gmail.com Wed Sep 17 17:26:27 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Wed, 17 Sep 2008 10:26:27 -0700 Subject: [elephant-devel] Elephant Dependency on Bordeaux Threads In-Reply-To: <19031828491C406AA04A25AF16FE6ED5@killer> References: <42c0ab790809161203h1900d7ay9426da7850bc24b5@mail.gmail.com> <19031828491C406AA04A25AF16FE6ED5@killer> Message-ID: <42c0ab790809171026n76e7aa61jdc4b0551bec3c0b8@mail.gmail.com> On Wed, Sep 17, 2008 at 12:53 AM, Alex Mizrahi wrote: > > > why not just move the dependency into the elephant-tests system > definition? > > yep we could make it more fine-grained, i think, but what is a problem with > bordeaux-threads anyway? iirc it just works via dummy functions if > threads are not supported by implementation, so it's totally safe It's mainly just an annoyance for me, since I already had a package nicknamed :bt, which bordeaux-threads interferes with. But even ignoring that, I wouldn't want to enforce a dependency on something that isn't actually necessary. -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From elliottslaughter at gmail.com Wed Sep 17 22:30:22 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Wed, 17 Sep 2008 15:30:22 -0700 Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> Message-ID: <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> On Wed, Sep 17, 2008 at 1:31 AM, Leslie P. Polzer wrote: > > > Error opening shared object "c:\\Program > > Files\\asdf\\elephant-unstable\\src\\memutil\\libmemutil.dll": > > 5. > > [Condition of type SIMPLE-ERROR] > > > > Can anyone who works regularly with Windows give me advice on compiling > > these dependencies? Thanks. > > I don't, but got a backtrace? > Actually, that was just because I didn't know how to compile a dll properly. With the help of the "cygwin compile script" comment in elephant.asd, I think I have compiled the dlls properly now. (At least they work properly with Allegro CL now.) But in SBCL/win32 I still get an error when attempting to load using my manually compiled libberkeley-db.dll. * (asdf:oos 'asdf:load-op :elephant) ... * (asdf:oos 'asdf:load-op :ele-bdb) ... Loading C:/Program Files/Oracle/Berkeley DB 4.5.20/bin/libdb45.dll Attempting to load libberkeley-db.dll... Loaded C:\Program Files\asdf\elephant-unstable\libberkeley-db.dll debugger invoked on a UNDEFINED-ALIEN-ERROR: Undefined alien: "db_env_get_max_objects" Restarts: 0: [RETRY] Retry performing # on #. 1: [ACCEPT] Continue, treating # on # as having been successful. 2: [ABORT] Return to SLIME's top level. 3: [CLOSE-CONNECTION] Close SLIME connection 4: [ABORT] Exit debugger, returning to top level. Backtrace: 0: (SB-SYS:ENSURE-DYNAMIC-FOREIGN-SYMBOL-ADDRESS "db_env_get_max_objects" #) 1: (SB-IMPL::LINK-FOREIGN-SYMBOL "db_env_get_max_objects" NIL) 2: ((FLET SB-THREAD::WITH-RECURSIVE-SPINLOCK-THUNK)) 3: (SB-IMPL::ENSURE-FOREIGN-SYMBOL-LINKAGE "db_env_get_max_objects" NIL) 4: (SB-SYS:FOREIGN-SYMBOL-ADDRESS "db_env_get_max_objects" NIL) 5: (SB-FASL::FOP-FOREIGN-FIXUP) 6: (SB-FASL::LOAD-FASL-GROUP #) 7: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK)) 8: (SB-FASL::LOAD-AS-FASL # NIL #) 9: (SB-FASL::%LOAD #) 10: (SB-FASL::%LOAD #P"c:\\Program Files\\asdf\\elephant-unstable\\src\\db-bdb\\berkeley-db.fasl") 11: (LOAD #P"c:\\Program Files\\asdf\\elephant-unstable\\src\\db-bdb\\berkeley-db.fasl") 12: ((SB-PCL::FAST-METHOD ASDF:PERFORM (ASDF:LOAD-OP ASDF:CL-SOURCE-FILE)) # # # #) 13: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. SB-PCL::.ARG1.)) # # # #) 14: ((LAMBDA NIL)) 15: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK)) 16: (SB-C::%WITH-COMPILATION-UNIT #) 17: (ASDF:OPERATE ASDF:LOAD-OP :ELE-BDB) 18: (SB-INT:SIMPLE-EVAL-IN-LEXENV (ASDF:OOS (QUOTE ASDF:LOAD-OP) :ELE-BDB) #) 19: (SWANK::EVAL-REGION "(asdf:oos 'asdf:load-op :ele-bdb) ") 20: ((LAMBDA NIL)) 21: (SWANK::TRACK-PACKAGE #) 22: (SWANK::CALL-WITH-BUFFER-SYNTAX #) 23: (SWANK::REPL-EVAL "(asdf:oos 'asdf:load-op :ele-bdb) ") 24: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(asdf:oos 'asdf:load-op :ele-bdb) ") #) 25: ((LAMBDA NIL)) 26: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) # #) 27: ((LAMBDA NIL)) 28: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) # #) 29: (SWANK::CALL-WITH-REDIRECTED-IO # #) 30: (SWANK::CALL-WITH-CONNECTION # #) 31: (SWANK::HANDLE-REQUEST #) 32: (SWANK::SIMPLE-SERVE-REQUESTS #) 33: (SWANK::SERVE-CONNECTION # NIL NIL :ISO-8859-1) 34: (SWANK::SETUP-SERVER 0 # NIL NIL :ISO-8859-1) 35: (SB-INT:SIMPLE-EVAL-IN-LEXENV (FUNCALL (READ-FROM-STRING "swank:start-server") "c:/DOCUME~1/ELLIOT~1/LOCALS~1/Temp/slime.2152" :CODING-SYSTEM "iso-latin-1-unix") #) 36: (SB-INT:SIMPLE-EVAL-IN-LEXENV (PROGN (LOAD "c:/Program Files/Emacs/slime/swank-loader.lisp" :VERBOSE T) (FUNCALL (READ-FROM-STRING "swank-loader:init")) (FUNCALL (READ-FROM-STRING "swank:start-server") "c:/DOCUME~1/ELLIOT~1/LOCALS~1/Temp/slime.2152" :CODING-SYSTEM "iso-latin-1-unix")) #) 37: (INTERACTIVE-EVAL (PROGN (LOAD "c:/Program Files/Emacs/slime/swank-loader.lisp" :VERBOSE T) (FUNCALL (READ-FROM-STRING "swank-loader:init")) (FUNCALL (READ-FROM-STRING "swank:start-server") "c:/DOCUME~1/ELLIOT~1/LOCALS~1/Temp/slime.2152" :CODING-SYSTEM "iso-latin-1-unix"))) 38: (SB-IMPL::REPL-FUN NIL) 39: (SB-IMPL::REPL-FUN NIL) 40: ((LAMBDA NIL)) 41: ((LAMBDA NIL)) 42: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) 43: (SB-IMPL::TOPLEVEL-REPL NIL) 44: (SB-IMPL::TOPLEVEL-INIT) 45: ((LABELS SB-IMPL::RESTART-LISP)) 46: ("foreign function: #x411365") 47: ("foreign function: #x40A152") -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian.dnsalias.net Thu Sep 18 08:01:30 2008 From: sky at viridian.dnsalias.net (Leslie P. Polzer) Date: Thu, 18 Sep 2008 10:01:30 +0200 Subject: [elephant-devel] Elephant Dependency on Bordeaux Threads In-Reply-To: <42c0ab790809171026n76e7aa61jdc4b0551bec3c0b8@mail.gmail.com> References: <42c0ab790809161203h1900d7ay9426da7850bc24b5@mail.gmail.com> <19031828491C406AA04A25AF16FE6ED5@killer> <42c0ab790809171026n76e7aa61jdc4b0551bec3c0b8@mail.gmail.com> Message-ID: <20080918080130.GA10053@viridian-project.de> On Wed, Sep 17, 2008 at 10:26:27AM -0700, Elliott Slaughter wrote: > But even ignoring that, I wouldn't want to enforce a dependency on something > that isn't actually necessary. It makes sense to move the dependency to ele-tests and ele-postmodern. Leslie From sky at viridian-project.de Thu Sep 18 08:39:12 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Thu, 18 Sep 2008 10:39:12 +0200 (CEST) Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> Message-ID: <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> > debugger invoked on a UNDEFINED-ALIEN-ERROR: > Undefined alien: "db_env_get_max_objects" Can you confirm that UFFI on win32/SBCL works in other cases? Can you find the alien by using SBCL's FFI directly? Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From killerstorm at newmail.ru Thu Sep 18 11:10:57 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 18 Sep 2008 14:10:57 +0300 Subject: [elephant-devel] Class Heirarchies and Queries References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com><62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org><3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> Message-ID: <55921E40BEE448119697676A68DDBDB3@killer> ??>> so do you people think that inheritance support on map level is a ??>> right thing? LPP> Yes, definitely. LPP> Got a code draft? yep, sort of.. but i think Ian's approach is "good enough" since rarely people need both superclass and subclass lookups to be fast enough. but i'll probably post combined cursors as contrib (some time later..) maybe we can use them if we'll change our mind on this issue or for some other stuff From sky at viridian-project.de Thu Sep 18 14:47:26 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Thu, 18 Sep 2008 16:47:26 +0200 (CEST) Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <55921E40BEE448119697676A68DDBDB3@killer> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com><62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org><3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <55921E40BEE448119697676A68DDBDB3@killer> Message-ID: <65259.84.157.2.239.1221749246.squirrel@mail.stardawn.org> > but i'll probably post combined cursors as contrib (some time later..) > maybe we can use them if we'll change our mind on this issue or > for some other stuff Yes, please! -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From reddaly at gmail.com Mon Sep 22 02:25:10 2008 From: reddaly at gmail.com (Red Daly) Date: Sun, 21 Sep 2008 19:25:10 -0700 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> Message-ID: I have recently run into "Cannot allocate memory" problems with elephant on a production server. Unfortunately when one transaction is too large, it seems to blow the database until a manual recover is done. The occasional failure is slightly worrisome but it the whole database requiring a manual recover from one extra-large transaction is a scary thought for a live application with thousands of users. Why does the memory allocation failure sour the whole database instead of aborting a single transaction? I think that elephant should try to encapsulate this failure, recovering the database or whatever is necessary to make the store usable for the next transaction. Best, Red Daly On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov wrote: > On Jan 4, 2008 2:54 AM, Ian Eslick wrote: > > Hi Victor, > > > > Sounds like your transaction is blowing out the shared memory > > allocated by Berkeley DB to store dirty pages. This is caused by > > transactions that are too large; putting an entire file of data could > > well accomplish this. (We really should change the error message to > > be more informative in these cases). > > > > Try pushing with-transaction into the loop in import-movie as follows: > > Thanks for your suggestion, Ian - the problem was solved once I've > moved with-transaction inside the collect-rating-info. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Mon Sep 22 08:43:21 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Mon, 22 Sep 2008 10:43:21 +0200 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> Message-ID: <20080922084321.GB23936@viridian-project.de> On Sun, Sep 21, 2008 at 07:25:10PM -0700, Red Daly wrote: > > Why does the memory allocation failure sour the whole database instead of > aborting a single transaction? I think that elephant should try to > encapsulate this failure, recovering the database or whatever is necessary > to make the store usable for the next transaction. I'm willing to look into this, but can you give me a hint (or better yet full code) on how to reproduce this? That'd save some work for me. Until then, can't you divide the txn into smaller pieces? Leslie From sky at viridian-project.de Mon Sep 22 10:49:43 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Mon, 22 Sep 2008 12:49:43 +0200 (CEST) Subject: [elephant-devel] Additional fix for REDEFINITION-INITFORM Message-ID: <63905.88.73.235.179.1222080583.squirrel@mail.stardawn.org> Fixes test redefinition-initform across Lisp restarts. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: redefinition-initform_2.darcspatch Type: application/octet-stream Size: 16732 bytes Desc: not available URL: From sky at viridian-project.de Mon Sep 22 11:45:15 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Mon, 22 Sep 2008 13:45:15 +0200 (CEST) Subject: [elephant-devel] Elephant Dependency on Bordeaux Threads In-Reply-To: <20080918080130.GA10053@viridian-project.de> References: <42c0ab790809161203h1900d7ay9426da7850bc24b5@mail.gmail.com> <19031828491C406AA04A25AF16FE6ED5@killer> <42c0ab790809171026n76e7aa61jdc4b0551bec3c0b8@mail.gmail.com> <20080918080130.GA10053@viridian-project.de> Message-ID: <62187.88.73.235.179.1222083915.squirrel@mail.stardawn.org> > On Wed, Sep 17, 2008 at 10:26:27AM -0700, Elliott Slaughter wrote: >> But even ignoring that, I wouldn't want to enforce a dependency on something >> that isn't actually necessary. > > It makes sense to move the dependency to ele-tests and > ele-postmodern. The current state seems to be: * 0.9.1 has bt dep on elephant and possibly others * 0.9.2 has bt dep on ele-tests Elliott, do you really want to use 0.9.1? If not I'm just going to add the bt dep to ele-postmodern and everything should be fine. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Mon Sep 22 12:18:32 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 22 Sep 2008 08:18:32 -0400 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> Message-ID: You could be running out of cache or locks. I believe there are now parameters in config.sexp you can set to raise the default limits. The lack of robustness to allocation failures is a problem with Berkeley DB. Unfortunately, running recovery isn't a trivial process. You have to guarantee that all other threads have released any Berkeley DB resources (abort all active transactions) and don't try to request any more (meaning no persistent slot reads/writes for Elephant) so you essentially need to get inside the state of each process, abort any transactions, and then suspend each thread. This isn't something that you can canonicalize inside the Elephant library. Chalk this up as another reason to someday implement a lisp-only version of the library! Ian On Sep 21, 2008, at 10:25 PM, Red Daly wrote: > I have recently run into "Cannot allocate memory" problems with > elephant on a production server. Unfortunately when one transaction > is too large, it seems to blow the database until a manual recover > is done. > > The occasional failure is slightly worrisome but it the whole > database requiring a manual recover from one extra-large transaction > is a scary thought for a live application with thousands of users. > > Why does the memory allocation failure sour the whole database > instead of aborting a single transaction? I think that elephant > should try to encapsulate this failure, recovering the database or > whatever is necessary to make the store usable for the next > transaction. > > Best, > Red Daly > > > On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov > wrote: > On Jan 4, 2008 2:54 AM, Ian Eslick wrote: > > Hi Victor, > > > > Sounds like your transaction is blowing out the shared memory > > allocated by Berkeley DB to store dirty pages. This is caused by > > transactions that are too large; putting an entire file of data > could > > well accomplish this. (We really should change the error message to > > be more informative in these cases). > > > > Try pushing with-transaction into the loop in import-movie as > follows: > > Thanks for your suggestion, Ian - the problem was solved once I've > moved with-transaction inside the collect-rating-info. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From elliottslaughter at gmail.com Mon Sep 22 18:59:36 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Mon, 22 Sep 2008 11:59:36 -0700 Subject: [elephant-devel] Elephant Dependency on Bordeaux Threads In-Reply-To: <62187.88.73.235.179.1222083915.squirrel@mail.stardawn.org> References: <42c0ab790809161203h1900d7ay9426da7850bc24b5@mail.gmail.com> <19031828491C406AA04A25AF16FE6ED5@killer> <42c0ab790809171026n76e7aa61jdc4b0551bec3c0b8@mail.gmail.com> <20080918080130.GA10053@viridian-project.de> <62187.88.73.235.179.1222083915.squirrel@mail.stardawn.org> Message-ID: <42c0ab790809221159l73393d46l94406074629b8d3a@mail.gmail.com> On Mon, Sep 22, 2008 at 4:45 AM, Leslie P. Polzer wrote: > > > On Wed, Sep 17, 2008 at 10:26:27AM -0700, Elliott Slaughter wrote: > >> But even ignoring that, I wouldn't want to enforce a dependency on > something > >> that isn't actually necessary. > > > > It makes sense to move the dependency to ele-tests and > > ele-postmodern. > > The current state seems to be: > > * 0.9.1 has bt dep on elephant and possibly others > > * 0.9.2 has bt dep on ele-tests > > Elliott, do you really want to use 0.9.1? I don't believe the :inherit option for indexed slots is available in 0.9.1, so I am currently using the elephant-unstable repo. If not I'm just going to add the bt dep to ele-postmodern > and everything should be fine. Sounds good. > > > Leslie > > -- > LinkedIn Profile: http://www.linkedin.com/in/polzer > Xing Profile: https://www.xing.com/profile/LeslieP_Polzer > Blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Mon Sep 22 19:21:26 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Mon, 22 Sep 2008 21:21:26 +0200 (CEST) Subject: [elephant-devel] Elephant Dependency on Bordeaux Threads In-Reply-To: <42c0ab790809221159l73393d46l94406074629b8d3a@mail.gmail.com> References: <42c0ab790809161203h1900d7ay9426da7850bc24b5@mail.gmail.com> <19031828491C406AA04A25AF16FE6ED5@killer> <42c0ab790809171026n76e7aa61jdc4b0551bec3c0b8@mail.gmail.com> <20080918080130.GA10053@viridian-project.de> <62187.88.73.235.179.1222083915.squirrel@mail.stardawn.org> <42c0ab790809221159l73393d46l94406074629b8d3a@mail.gmail.com> Message-ID: <63224.88.73.235.179.1222111286.squirrel@mail.stardawn.org> > If not I'm just going to add the bt dep to ele-postmodern >> and everything should be fine. > > > Sounds good. Here it comes, plus some minor non-code changes fixed along the way. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: bt-dep_misc.darcspatch Type: application/octet-stream Size: 17121 bytes Desc: not available URL: From elliottslaughter at gmail.com Mon Sep 22 22:01:06 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Mon, 22 Sep 2008 15:01:06 -0700 Subject: [elephant-devel] Class Heirarchies and Queries In-Reply-To: <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> Message-ID: <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> On Thu, Sep 18, 2008 at 1:39 AM, Leslie P. Polzer wrote: > > > debugger invoked on a UNDEFINED-ALIEN-ERROR: > > Undefined alien: "db_env_get_max_objects" > > Can you confirm that UFFI on win32/SBCL works in other cases? I can't confirm that win32/SBCL is an "officially supported" platform for UFFI, but I can say that Elephant 0.9.1 worked just fine after minimal changes. Can you find the alien by using SBCL's FFI directly? I am a little confused: there doesn't seem to actually be a function called "db_env_get_max_objects" in libberkeley-db.c (although I do see "db_env_get_lk_max_objects"), but ele-bdb.asd loads fine under linux(x86-64)/SBCL and win32/Allegro. Where is "db_env_get_max_objects" actually coming from? Thanks. -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Tue Sep 23 08:28:50 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 23 Sep 2008 10:28:50 +0200 (CEST) Subject: UFFI/win32 problems (was: Re: [elephant-devel] Class Heirarchies and Queries) In-Reply-To: <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <62591.84.157.46.71.1221557338.squirrel@mail.stardawn.org> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> Message-ID: <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> > I am a little confused: there doesn't seem to actually be a function called > "db_env_get_max_objects" in libberkeley-db.c (although I do see > "db_env_get_lk_max_objects"), but ele-bdb.asd loads fine under > linux(x86-64)/SBCL and win32/Allegro. Where is "db_env_get_max_objects" > actually coming from? Err, yes. Good question. How about removing this alien? # Following line added by [Add max lock configuration to BDB # eslick at common-lisp.net**20080516185155] (def-function ("db_env_get_max_objects" %db-env-get-max-objects) Doesn't seem to be used by other code. Perhaps win32/SBCL DLL loading is eager and attempts to resolve this symbol right away? Just guessing here, though. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From sky at viridian-project.de Tue Sep 23 12:51:36 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 23 Sep 2008 14:51:36 +0200 (CEST) Subject: [elephant-devel] [PATCH] Proper deadlock detection for BDB Message-ID: <62442.88.73.220.117.1222174296.squirrel@mail.stardawn.org> This new approach ditches the usage of the external binary and introduces two strategies: * automatic deadlock detection on every conflict (default) * deadlock detection via a separate thread. Like the old approach but doesn't fiddle around with external binaries. Doesn't introduce the Bordeaux dep until this functionality is requested. Most people should be fine with the first method; it's also the fastest. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: bdb-deadlock-detect.darcspatch Type: application/octet-stream Size: 20462 bytes Desc: not available URL: From sky at viridian-project.de Tue Sep 23 13:13:27 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 23 Sep 2008 15:13:27 +0200 (CEST) Subject: [elephant-devel] [PATCH] Proper deadlock detection for BDB In-Reply-To: <62442.88.73.220.117.1222174296.squirrel@mail.stardawn.org> References: <62442.88.73.220.117.1222174296.squirrel@mail.stardawn.org> Message-ID: <63666.88.73.220.117.1222175607.squirrel@mail.stardawn.org> Fixed a small glitch. Updated version attached. -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: bdb-deadlock-detect.darcspatch Type: application/octet-stream Size: 20463 bytes Desc: not available URL: From eslick at media.mit.edu Tue Sep 23 13:17:50 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 23 Sep 2008 09:17:50 -0400 Subject: [elephant-devel] [PATCH] Proper deadlock detection for BDB In-Reply-To: <63666.88.73.220.117.1222175607.squirrel@mail.stardawn.org> References: <62442.88.73.220.117.1222174296.squirrel@mail.stardawn.org> <63666.88.73.220.117.1222175607.squirrel@mail.stardawn.org> Message-ID: Does this require BDB 4.7? As I recall, conflict detection used to require a separate process to run correctly but there was a change in 4.6 or 4.7 to allow you to do this in the same process. Also, isn't it true that a true deadlock in a given thread will cause the process to lock up if it's not using OS threads (like Allegro which will freeze up entirely when a C call fails to return)? SBCL shouldn't have this problem. Neither of these solutions will work, so we should keep around the external process model for these settings. This used to drive me nuts when I was developing Elephant on Allegro. I can dig back into this to refresh my memory unless you are confident you're up to speed. Thanks, Ian On Sep 23, 2008, at 9:13 AM, Leslie P. Polzer wrote: > Fixed a small glitch. > Updated version attached. > > -- > LinkedIn Profile: http://www.linkedin.com/in/polzer > Xing Profile: https://www.xing.com/profile/LeslieP_Polzer > Blog: http://blog.viridian-project.de/ detect.darcspatch>_______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From sky at viridian-project.de Tue Sep 23 14:40:37 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 23 Sep 2008 16:40:37 +0200 (CEST) Subject: [elephant-devel] [PATCH] Proper deadlock detection for BDB In-Reply-To: References: <62442.88.73.220.117.1222174296.squirrel@mail.stardawn.org> <63666.88.73.220.117.1222175607.squirrel@mail.stardawn.org> Message-ID: <63371.88.73.220.117.1222180837.squirrel@mail.stardawn.org> > Does this require BDB 4.7? As I recall, conflict detection used to > require a separate process to run correctly but there was a change in > 4.6 or 4.7 to allow you to do this in the same process. The function in question is DB_ENV->set_lk_detect, which is available at least since 4.5 (just verified). I have only tested this with 4.7 though, so we might want to put the patch on hold until then. > I can dig back into this to refresh my memory unless you are confident > you're up to speed. No, that's fine. Everything you wrote makes sense. I'm going to test the other BDB versions and probably give Allegro a go to check on cooperative mt. It might also make sense to offer the external db_deadlock solution as an option. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From sky at viridian-project.de Tue Sep 23 15:51:11 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 23 Sep 2008 17:51:11 +0200 (CEST) Subject: [elephant-devel] [PATCH] Proper deadlock detection for BDB In-Reply-To: <63371.88.73.220.117.1222180837.squirrel@mail.stardawn.org> References: <62442.88.73.220.117.1222174296.squirrel@mail.stardawn.org> <63666.88.73.220.117.1222175607.squirrel@mail.stardawn.org> <63371.88.73.220.117.1222180837.squirrel@mail.stardawn.org> Message-ID: <61976.88.73.220.117.1222185071.squirrel@mail.stardawn.org> > I have only tested this with 4.7 though, so we might want > to put the patch on hold until then. Works with 4.5 and 4.6, too. Do you think we still need the external db_deadlock? Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Tue Sep 23 15:53:03 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 23 Sep 2008 11:53:03 -0400 Subject: [elephant-devel] [PATCH] Proper deadlock detection for BDB In-Reply-To: <61976.88.73.220.117.1222185071.squirrel@mail.stardawn.org> References: <62442.88.73.220.117.1222174296.squirrel@mail.stardawn.org> <63666.88.73.220.117.1222175607.squirrel@mail.stardawn.org> <63371.88.73.220.117.1222180837.squirrel@mail.stardawn.org> <61976.88.73.220.117.1222185071.squirrel@mail.stardawn.org> Message-ID: <4A3E4C2D-12D2-4DE4-AC92-9944D6B62F74@media.mit.edu> It can't hurt to leave it there... Ian On Sep 23, 2008, at 11:51 AM, Leslie P. Polzer wrote: > >> I have only tested this with 4.7 though, so we might want >> to put the patch on hold until then. > > Works with 4.5 and 4.6, too. > > Do you think we still need the external db_deadlock? > > Leslie > > -- > LinkedIn Profile: http://www.linkedin.com/in/polzer > Xing Profile: https://www.xing.com/profile/LeslieP_Polzer > Blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From elliottslaughter at gmail.com Tue Sep 23 16:09:29 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Tue, 23 Sep 2008 09:09:29 -0700 Subject: UFFI/win32 problems (was: Re: [elephant-devel] Class Heirarchies and Queries) In-Reply-To: <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> Message-ID: <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> On Tue, Sep 23, 2008 at 1:28 AM, Leslie P. Polzer wrote: > > > I am a little confused: there doesn't seem to actually be a function > called > > "db_env_get_max_objects" in libberkeley-db.c (although I do see > > "db_env_get_lk_max_objects"), but ele-bdb.asd loads fine under > > linux(x86-64)/SBCL and win32/Allegro. Where is "db_env_get_max_objects" > > actually coming from? > > Err, yes. Good question. How about removing this alien? > > # Following line added by [Add max lock configuration to BDB > # eslick at common-lisp.net**20080516185155] > (def-function ("db_env_get_max_objects" %db-env-get-max-objects) > > Doesn't seem to be used by other code. Yeah, removing that form and the one below (wrap-errno db-env-get-max-objects ...) allows ele-bdb to load fine and seems to fix the problem. Thanks. Perhaps win32/SBCL DLL loading is eager and attempts to resolve this > symbol right away? Do other lisps not do this? Just guessing here, though. > > Leslie > > -- > LinkedIn Profile: http://www.linkedin.com/in/polzer > Xing Profile: https://www.xing.com/profile/LeslieP_Polzer > Blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From klists at saphor.de Tue Sep 23 13:42:58 2008 From: klists at saphor.de (Marc) Date: Tue, 23 Sep 2008 15:42:58 +0200 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> Message-ID: <48D8F262.3080903@saphor.de> Ian Eslick wrote: > You could be running out of cache or locks. I believe there are now > parameters in config.sexp you can set to raise the default limits. > The lack of robustness to allocation failures is a problem with > Berkeley DB. > > Unfortunately, running recovery isn't a trivial process. You have to > guarantee that all other threads have released any Berkeley DB > resources (abort all active transactions) and don't try to request any > more (meaning no persistent slot reads/writes for Elephant) so you > essentially need to get inside the state of each process, abort any > transactions, and then suspend each thread. This isn't something that > you can canonicalize inside the Elephant library. > > Chalk this up as another reason to someday implement a lisp-only > version of the library! Indeed, that'd be a dream. For us, at least, this prevents us from seriously using transactions at all in elephant. We do use them to speed up some bulk inserts when we know that the number of inserts in one go won't be too big, but not for larger logical transactions where they'd really be called for. Raising the limits at best slightly lessens the pain, but is not really a solution. Best regards, Marc > > On Sep 21, 2008, at 10:25 PM, Red Daly wrote: > >> I have recently run into "Cannot allocate memory" problems with >> elephant on a production server. Unfortunately when one transaction >> is too large, it seems to blow the database until a manual recover is >> done. >> >> The occasional failure is slightly worrisome but it the whole >> database requiring a manual recover from one extra-large transaction >> is a scary thought for a live application with thousands of users. >> >> Why does the memory allocation failure sour the whole database >> instead of aborting a single transaction? I think that elephant >> should try to encapsulate this failure, recovering the database or >> whatever is necessary to make the store usable for the next transaction. >> >> Best, >> Red Daly >> >> >> On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov >> wrote: >> On Jan 4, 2008 2:54 AM, Ian Eslick wrote: >> > Hi Victor, >> > >> > Sounds like your transaction is blowing out the shared memory >> > allocated by Berkeley DB to store dirty pages. This is caused by >> > transactions that are too large; putting an entire file of data could >> > well accomplish this. (We really should change the error message to >> > be more informative in these cases). >> > >> > Try pushing with-transaction into the loop in import-movie as follows: >> >> Thanks for your suggestion, Ian - the problem was solved once I've >> moved with-transaction inside the collect-rating-info. >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From eslick at media.mit.edu Tue Sep 23 18:38:11 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 23 Sep 2008 14:38:11 -0400 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: <48D8F262.3080903@saphor.de> References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> <48D8F262.3080903@saphor.de> Message-ID: Comments on original "out of memory problems" - Running out of disk space for logs can also be a problem as well as transaction size - The max # of locks needed will grow with the log of the DB size and linearly with the number of concurrent transactions. You can run db_stat to see what your resource usage is. - When bulk loading be sure to pull the data from a file into memory before running the transaction if there might be any concurrent transactions on that DB. (think about what happens when you have to restart the transaction and the DB state is reset but the stream & loop variables in memory are not). > but not for larger logical transactions where they'd > really be called for. Raising the limits at best slightly lessens the > pain, but is not really a solution. If raising the limits isn't a solution, then your application is probably mismatched with Elephant/BDB. There might be some artificial reasons you are seeing these problems. 1) If you have a large DB with deep indices and highly concurrent transactions, then you may start hit lock limits as described above. You should be able to set these limits high enough that you'll never hit problems. Fortunately increasing locks is a good solution. max- concurrent-sessions * max-objects-per-session * 2 + 20% is a good number for max locks. You can use db_stat to see what kind of locks- per-session numbers you're seeing and an analysis of your application should tell you the max-concurrent-sessions. Max concurrent transactions is upper-bounded by your machine's processing capacity and max # of locks is proportional to the logarithm of the number of objects which will grow very slowly so can be given practical upper bounds. (Set lock sizes in config.sexp - :berkeley-db-max-locks & :berkeley-db- max-objects) 2) If you really are blowing out the BDB data cache in normal operation (vs. bulk loads) then your application is either not using transactions efficiently or is highly mismatched to any transactional DB whether that be BDB or some future lisp implementation. If a logical transaction results in writing or modifying 10's of megabytes of data, then no transactional DB will ever work for you. This is just physics. To get ACID properties you have to be able to stage all side effects in memory to ensure that they are written as an atomic unit. If you don't have enough memory to perform this staging, then you can't implement a transaction. There is a special case where you are manipulating large data objects such as full-res images or uploaded files. I use a hybrid solution for these where BDB keeps pointers to disk files which store just the large items. If you can handle potential inconsistency between the on- disk storage and the DB, then you can write the file to disk, keep a pointer to it and do the bookkeeping in the DB inside a single transaction. You'll run into serializer limits before you run into cache size limits for large data items anyway. The following may be obvious, but it's easy to get complacent and forget that Elephant is backed up by a transactional DB and that you can't really ignore those implications for real-world applications. - The concept behind the transaction is to combine small DB operations into a single atomic chunk. Sometimes you can wrap a small set of such chunks for performance. You shouldn't be wrapping lots of unrelated operations into a single transaction. This is better for avoiding concurrency conflicts as well. For example, all composite operations inside Elephant use transactions to ensure consistency of slot and index values. Slot writes to indexed slots wrap both the write and the index updates into a single transaction. Otherwise every sub-operation ends up in its own little transaction and you can get inconsistent indices if you have an error in between the slot write and index update. - One downside of a transactional store on BDB is every little transaction results in a set of pages being flushed to disk (any Btree index pages and the leaf pages with data). Flushing pages to disk waits for a write-done from the driver; this can be very expensive! (10's of ms if I remember correctly). Flushing one page takes about the same time as flushing a few dozen pages so you can save significant run time by wrapping your leaf operations in a transaction - plus all the usual ACID properties. In my web application (using weblocks) I wrap every ajax callback on the server side in one big transaction since I know apriori that operations within these callbacks aren't going to blow out 20M of cache or 1000's of locks. Typically I update a couple of objects and indices. This may be a few dozen 4k or 8k pages. Cheers, Ian On Sep 23, 2008, at 9:42 AM, Marc wrote: > Ian Eslick wrote: >> You could be running out of cache or locks. I believe there are now >> parameters in config.sexp you can set to raise the default limits. >> The lack of robustness to allocation failures is a problem with >> Berkeley DB. >> >> Unfortunately, running recovery isn't a trivial process. You have to >> guarantee that all other threads have released any Berkeley DB >> resources (abort all active transactions) and don't try to request >> any >> more (meaning no persistent slot reads/writes for Elephant) so you >> essentially need to get inside the state of each process, abort any >> transactions, and then suspend each thread. This isn't something >> that >> you can canonicalize inside the Elephant library. >> >> Chalk this up as another reason to someday implement a lisp-only >> version of the library! > Indeed, that'd be a dream. For us, at least, this prevents us from > seriously using transactions at all in elephant. We do use them to > speed > up some bulk inserts when we know that the number of inserts in one go > won't be too big, but not for larger logical transactions where they'd > really be called for. Raising the limits at best slightly lessens the > pain, but is not really a solution. > > Best regards, > > Marc >> >> On Sep 21, 2008, at 10:25 PM, Red Daly wrote: >> >>> I have recently run into "Cannot allocate memory" problems with >>> elephant on a production server. Unfortunately when one transaction >>> is too large, it seems to blow the database until a manual recover >>> is >>> done. >>> >>> The occasional failure is slightly worrisome but it the whole >>> database requiring a manual recover from one extra-large transaction >>> is a scary thought for a live application with thousands of users. >>> >>> Why does the memory allocation failure sour the whole database >>> instead of aborting a single transaction? I think that elephant >>> should try to encapsulate this failure, recovering the database or >>> whatever is necessary to make the store usable for the next >>> transaction. >>> >>> Best, >>> Red Daly >>> >>> >>> On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov >>> wrote: >>> On Jan 4, 2008 2:54 AM, Ian Eslick wrote: >>>> Hi Victor, >>>> >>>> Sounds like your transaction is blowing out the shared memory >>>> allocated by Berkeley DB to store dirty pages. This is caused by >>>> transactions that are too large; putting an entire file of data >>>> could >>>> well accomplish this. (We really should change the error message >>>> to >>>> be more informative in these cases). >>>> >>>> Try pushing with-transaction into the loop in import-movie as >>>> follows: >>> >>> Thanks for your suggestion, Ian - the problem was solved once I've >>> moved with-transaction inside the collect-rating-info. >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From sky at viridian-project.de Tue Sep 23 19:12:53 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 23 Sep 2008 21:12:53 +0200 (CEST) Subject: UFFI/win32 problems (was: Re: [elephant-devel] Class Heirarchies and Queries) In-Reply-To: <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <3237650F6D014CF19571172A4E5B386C@killer> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> Message-ID: <61659.88.73.220.117.1222197173.squirrel@mail.stardawn.org> > Perhaps win32/SBCL DLL loading is eager and attempts to resolve this >> symbol right away? > > > Do other lisps not do this? This is O/S-specific, I think. I think DLL loading is always eager while SO loading is not, something like that. So maybe SBCL FFI is missing some workaround that other Lisps have. FYI, the above function generates a style warning on SBCL/Linux. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From sky at viridian-project.de Tue Sep 23 19:40:07 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Tue, 23 Sep 2008 21:40:07 +0200 (CEST) Subject: [elephant-devel] [PATCH] Proper deadlock detection for BDB In-Reply-To: <4A3E4C2D-12D2-4DE4-AC92-9944D6B62F74@media.mit.edu> References: <62442.88.73.220.117.1222174296.squirrel@mail.stardawn.org> <63666.88.73.220.117.1222175607.squirrel@mail.stardawn.org> <63371.88.73.220.117.1222180837.squirrel@mail.stardawn.org> <61976.88.73.220.117.1222185071.squirrel@mail.stardawn.org> <4A3E4C2D-12D2-4DE4-AC92-9944D6B62F74@media.mit.edu> Message-ID: <62734.88.73.220.117.1222198807.squirrel@mail.stardawn.org> Now with old detector. The interface of START-DEADLOCK-DETECTOR is a bit confusing now with all the key parameters, but normally it need not be called by the user anyway. Suggestions for untangling this are appreciated in any case. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: bdb-deadlock-detect.darcspatch Type: application/octet-stream Size: 22314 bytes Desc: not available URL: From klists at saphor.de Wed Sep 24 11:27:52 2008 From: klists at saphor.de (Marc) Date: Wed, 24 Sep 2008 13:27:52 +0200 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> <48D8F262.3080903@saphor.de> Message-ID: <48DA2438.6050802@saphor.de> Ian Eslick wrote: >> but not for larger logical transactions where they'd >> really be called for. Raising the limits at best slightly lessens the >> pain, but is not really a solution. > > > If raising the limits isn't a solution, then your application is > probably mismatched with Elephant/BDB. There might be some artificial > reasons you are seeing these problems. > > 1) If you have a large DB with deep indices and highly concurrent > transactions, then you may start hit lock limits as described above. > You should be able to set these limits high enough that you'll never > hit problems. Fortunately increasing locks is a good solution. > max-concurrent-sessions * max-objects-per-session * 2 + 20% is a good > number for max locks. You can use db_stat to see what kind of > locks-per-session numbers you're seeing and an analysis of your > application should tell you the max-concurrent-sessions. the main problem for us is that we automatically import data files and data fragments, where one of them can easily result in the creation of thousands or even tens of thousands heavily cross-referencing objects, each with a number of index entries associated to it. The complexity of imports is not known in advance and can vary drastically from import to import. None of the objects is big in itself, but the number adds easily up to tens of thousands of inserts, updates or deletesin one logical transaction (concurrency is less of an issue for our imports and we could easily live with blocking the DB during the various imports). Since transactions of this size don't work with BDB as a backend, we currently do not use transactions for the import as a whole. We do use them for the import of single entries each consisting of few dozen or so objects to ensure consistency within an entry and to speed up insertions. > > Max concurrent transactions is upper-bounded by your machine's > processing capacity and max # of locks is proportional to the > logarithm of the number of objects which will grow very slowly so can > be given practical upper bounds. naive question, but why is the # of logs proportional to the logarithm of the number of objects to be inserted? Well, probably I'll have to check how elephant uses locks in its BTrees. > > (Set lock sizes in config.sexp - :berkeley-db-max-locks & > :berkeley-db-max-objects) > > > 2) If you really are blowing out the BDB data cache in normal > operation (vs. bulk loads) then your application is either not using > transactions efficiently or is highly mismatched to any transactional > DB whether that be BDB or some future lisp implementation. the problem is that for our application the "bulk load" is a perfectly normal and recurrent transaction. > > If a logical transaction results in writing or modifying 10's of > megabytes of data, then no transactional DB will ever work for you. > This is just physics. To get ACID properties you have to be able to > stage all side effects in memory to ensure that they are written as an > atomic unit. If you don't have enough memory to perform this staging, > then you can't implement a transaction. I'm a bit surprised about this. Most relational databases use journalling and / or rollback files to ensure that they can commit or rollback large transactions as one atomic unit. The procedure is quite neatly described in http://www.sqlite.org/atomiccommit.html and http://www.sqlite.org/tempfiles.html, but works AFAIK more or less the same on other RDMSs. I've just tried it out with sqlite which quite happily handles a transaction of 10 million inserts as an atomic unit both for commit and rollback ? to be sure, in this case at the price of an extended read / write lock on the DB and, of course, not in one physical write operation. Main memory hardly enters the equation there. Without journalling the various read levels would quite probably be even more challenging to implement with reasonable efficiency than they are now. > > > The following may be obvious, but it's easy to get complacent and > forget that Elephant is backed up by a transactional DB and that you > can't really ignore those implications for real-world applications. > > - The concept behind the transaction is to combine small DB operations > into a single atomic chunk. Sometimes you can wrap a small set of > such chunks for performance. You shouldn't be wrapping lots of > unrelated operations into a single transaction. This is better for > avoiding concurrency conflicts as well. > indeed, it is. However, the very point for us is that the objects are related and cross-reference each other often quite heavily and that we really should do is either commit or, in the case of errors, rollback an import in its entirety rather then end up with objects that point to nowhere. > For example, all composite operations inside Elephant use transactions > to ensure consistency of slot and index values. Slot writes to > indexed slots wrap both the write and the index updates into a single > transaction. Otherwise every sub-operation ends up in its own little > transaction and you can get inconsistent indices if you have an error > in between the slot write and index update. > > - One downside of a transactional store on BDB is every little > transaction results in a set of pages being flushed to disk (any Btree > index pages and the leaf pages with data). Flushing pages to disk > waits for a write-done from the driver; this can be very expensive! > (10's of ms if I remember correctly). Flushing one page takes about > the same time as flushing a few dozen pages so you can save > significant run time by wrapping your leaf operations in a transaction > - plus all the usual ACID properties. agreed. > In my web application (using weblocks) I wrap every ajax callback on > the server side in one big transaction since I know apriori that > operations within these callbacks aren't going to blow out 20M of > cache or 1000's of locks. Typically I update a couple of objects and > indices. This may be a few dozen 4k or 8k pages. for this type of interaction, a transaction, let alone a single callback will never be large. Best regards, Marc > > On Sep 23, 2008, at 9:42 AM, Marc wrote: > >> Ian Eslick wrote: >>> You could be running out of cache or locks. I believe there are now >>> parameters in config.sexp you can set to raise the default limits. >>> The lack of robustness to allocation failures is a problem with >>> Berkeley DB. >>> >>> Unfortunately, running recovery isn't a trivial process. You have to >>> guarantee that all other threads have released any Berkeley DB >>> resources (abort all active transactions) and don't try to request any >>> more (meaning no persistent slot reads/writes for Elephant) so you >>> essentially need to get inside the state of each process, abort any >>> transactions, and then suspend each thread. This isn't something that >>> you can canonicalize inside the Elephant library. >>> >>> Chalk this up as another reason to someday implement a lisp-only >>> version of the library! >> Indeed, that'd be a dream. For us, at least, this prevents us from >> seriously using transactions at all in elephant. We do use them to speed >> up some bulk inserts when we know that the number of inserts in one go >> won't be too big, but not for larger logical transactions where they'd >> really be called for. Raising the limits at best slightly lessens the >> pain, but is not really a solution. >> >> Best regards, >> >> Marc >>> >>> On Sep 21, 2008, at 10:25 PM, Red Daly wrote: >>> >>>> I have recently run into "Cannot allocate memory" problems with >>>> elephant on a production server. Unfortunately when one transaction >>>> is too large, it seems to blow the database until a manual recover is >>>> done. >>>> >>>> The occasional failure is slightly worrisome but it the whole >>>> database requiring a manual recover from one extra-large transaction >>>> is a scary thought for a live application with thousands of users. >>>> >>>> Why does the memory allocation failure sour the whole database >>>> instead of aborting a single transaction? I think that elephant >>>> should try to encapsulate this failure, recovering the database or >>>> whatever is necessary to make the store usable for the next >>>> transaction. >>>> >>>> Best, >>>> Red Daly >>>> >>>> >>>> On Sat, Jan 5, 2008 at 5:02 PM, Victor Kryukov >>>> wrote: >>>> On Jan 4, 2008 2:54 AM, Ian Eslick wrote: >>>>> Hi Victor, >>>>> >>>>> Sounds like your transaction is blowing out the shared memory >>>>> allocated by Berkeley DB to store dirty pages. This is caused by >>>>> transactions that are too large; putting an entire file of data could >>>>> well accomplish this. (We really should change the error message to >>>>> be more informative in these cases). >>>>> >>>>> Try pushing with-transaction into the loop in import-movie as >>>>> follows: >>>> >>>> Thanks for your suggestion, Ian - the problem was solved once I've >>>> moved with-transaction inside the collect-rating-info. >>>> _______________________________________________ >>>> elephant-devel site list >>>> elephant-devel at common-lisp.net >>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>>> >>>> _______________________________________________ >>>> elephant-devel site list >>>> elephant-devel at common-lisp.net >>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >> >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From eslick at media.mit.edu Wed Sep 24 12:45:17 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 24 Sep 2008 08:45:17 -0400 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: <48DA2438.6050802@saphor.de> References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> <48D8F262.3080903@saphor.de> <48DA2438.6050802@saphor.de> Message-ID: <20578A9D-A932-4AAF-BE6C-071CB6BA8D20@media.mit.edu> > the main problem for us is that we automatically import data files and > data fragments, where one of them can easily result in the creation of > thousands or even tens of thousands heavily cross-referencing objects, > each with a number of index entries associated to it. The complexity > of > imports is not known in advance and can vary drastically from import > to > import. > > None of the objects is big in itself, but the number adds easily up to > tens of thousands of inserts, updates or deletesin one logical > transaction (concurrency is less of an issue for our imports and we > could easily live with blocking the DB during the various imports). > Since transactions of this size don't work with BDB as a backend, we > currently do not use transactions for the import as a whole. We do use > them for the import of single entries each consisting of few dozen > or so > objects to ensure consistency within an entry and to speed up > insertions. Then it sounds like you're doing the right thing already, given the current constraints in Elephant. See some comments below. > >> >> Max concurrent transactions is upper-bounded by your machine's >> processing capacity and max # of locks is proportional to the >> logarithm of the number of objects which will grow very slowly so can >> be given practical upper bounds. > naive question, but why is the # of logs proportional to the logarithm > of the number of objects to be inserted? Well, probably I'll have to > check how elephant uses locks in its BTrees. This discussion is about the BDB backend, elephant makes no commitments about locking in the various stores. From BDB docs: "For Btree and Recno access methods, you will need one lock object per level of the database tree, at a minimum. (Unless keys are quite large with respect to the page size, neither Recno nor Btree database trees should ever be deeper than five levels.) Then, you will need one lock object for each leaf page of the database tree that will be simultaneously accessed." > I'm a bit surprised about this. Most relational databases use > journalling and / or rollback files to ensure that they can commit or > rollback large transactions as one atomic unit. The procedure is quite > neatly described in http://www.sqlite.org/atomiccommit.html and > http://www.sqlite.org/tempfiles.html, but works AFAIK more or less the > same on other RDMSs. Actually, as I understand it, SQlite makes all the edits in user memory rather than in a shared memory region so it's just easier to allocate the memory in that case. The tradeoff as you point out is serializing side-effecting transactions at the whole-DB level. Actually if you had shared locks you could probably still pull of the same rough model... BDB is tuned for highly concurrent, but independent, write transactions across multiple threads/processes. The shared memory region is used for sharing the fine-grained locks as well as for a shared cache so each process doesn't have to separately allocate memory and that you can have processes with various levels of read isolation. This unfortunately leads to the requirement to pre- allocate that shared memory region and your current dilemma. BDB does journaling for recovery/durability purposes, but if you shared memory gets mucked up by an allocation error, you need to flush everything (even the dirty stuff), recover from the last known good DB state and clean up any in-process transactions in the journal. > I've just tried it out with sqlite which quite happily handles a > transaction of 10 million inserts as an atomic unit both for commit > and > rollback ? to be sure, in this case at the price of an extended read / > write lock on the DB and, of course, not in one physical write > operation. Main memory hardly enters the equation there. Yep, this is definitely a sweet spot for SQLite. We actually might be able to configure BDB to behave similarly (tune it for single-writer, multiple-readers) as an optional setup to Elephant; but we'd have to look into this a bit as I'm not entirely sure BDB can meet your constraints here. I'm a little stale on all this these days... Thanks, Ian From elliottslaughter at gmail.com Wed Sep 24 18:53:09 2008 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Wed, 24 Sep 2008 11:53:09 -0700 Subject: UFFI/win32 problems (was: Re: [elephant-devel] Class Heirarchies and Queries) In-Reply-To: <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> Message-ID: <42c0ab790809241153o124371a2x7588db30e379be42@mail.gmail.com> On Tue, Sep 23, 2008 at 9:09 AM, Elliott Slaughter < elliottslaughter at gmail.com> wrote: > On Tue, Sep 23, 2008 at 1:28 AM, Leslie P. Polzer > wrote: > >> >> > I am a little confused: there doesn't seem to actually be a function >> called >> > "db_env_get_max_objects" in libberkeley-db.c (although I do see >> > "db_env_get_lk_max_objects"), but ele-bdb.asd loads fine under >> > linux(x86-64)/SBCL and win32/Allegro. Where is "db_env_get_max_objects" >> > actually coming from? >> >> Err, yes. Good question. How about removing this alien? >> >> # Following line added by [Add max lock configuration to BDB >> # eslick at common-lisp.net**20080516185155] >> (def-function ("db_env_get_max_objects" %db-env-get-max-objects) >> >> Doesn't seem to be used by other code. > > > Yeah, removing that form and the one below (wrap-errno > db-env-get-max-objects ...) allows ele-bdb to load fine and seems to fix the > problem. Thanks. > After looking at the code for db-set-env-get-max-objects, I believe the reference to "db_env_get_max_objects" should really have been "db_env_get_lk_max_objects". I have attached a bundle with three patches against elephant-unstable: one correcting a typo in a comment in elephant.asd, one fixing the alien problem above, and one with the minimal changes I needed to get elephant running on win32/SBCL. -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: minimal-support-for-sbcl_win32_.dpatch Type: application/octet-stream Size: 22061 bytes Desc: not available URL: From klists at saphor.de Wed Sep 24 18:53:48 2008 From: klists at saphor.de (Marc) Date: Wed, 24 Sep 2008 20:53:48 +0200 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: <20578A9D-A932-4AAF-BE6C-071CB6BA8D20@media.mit.edu> References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> <48D8F262.3080903@saphor.de> <48DA2438.6050802@saphor.de> <20578A9D-A932-4AAF-BE6C-071CB6BA8D20@media.mit.edu> Message-ID: <48DA8CBC.5040707@saphor.de> Ian Eslick wrote: > >> I'm a bit surprised about this. Most relational databases use >> journalling and / or rollback files to ensure that they can commit or >> rollback large transactions as one atomic unit. The procedure is quite >> neatly described in http://www.sqlite.org/atomiccommit.html and >> http://www.sqlite.org/tempfiles.html, but works AFAIK more or less the >> same on other RDMSs. > > Actually, as I understand it, SQlite makes all the edits in user > memory rather than in a shared memory region so it's just easier to > allocate the memory in that case. The tradeoff as you point out is > serializing side-effecting transactions at the whole-DB level. > Actually if you had shared locks you could probably still pull of the > same rough model... > > BDB is tuned for highly concurrent, but independent, write > transactions across multiple threads/processes. The shared memory > region is used for sharing the fine-grained locks as well as for a > shared cache so each process doesn't have to separately allocate > memory and that you can have processes with various levels of read > isolation. This unfortunately leads to the requirement to > pre-allocate that shared memory region and your current dilemma. > that sounds like the key for this specific mismatch of the BDB locking philosophy ? which for the scenarios you describe is perfect ? and our quite different needs. We can (well, have to) live with these restrictions for now. However, if there ever were a new lisp backend, I don't think it'd necessarily have to have precisely the same constraints. >> I've just tried it out with sqlite which quite happily handles a >> transaction of 10 million inserts as an atomic unit both for commit and >> rollback ? to be sure, in this case at the price of an extended read / >> write lock on the DB and, of course, not in one physical write >> operation. Main memory hardly enters the equation there. > > Yep, this is definitely a sweet spot for SQLite. > > We actually might be able to configure BDB to behave similarly (tune > it for single-writer, multiple-readers) as an optional setup to > Elephant; but we'd have to look into this a bit as I'm not entirely > sure BDB can meet your constraints here. I'm a little stale on all > this these days... > that'd be great, of course. Best regards, Marc From sky at viridian-project.de Thu Sep 25 08:35:31 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Thu, 25 Sep 2008 10:35:31 +0200 (CEST) Subject: UFFI/win32 problems (was: Re: [elephant-devel] Class Heirarchies and Queries) In-Reply-To: <42c0ab790809241153o124371a2x7588db30e379be42@mail.gmail.com> References: <42c0ab790809151731l52b36eb7i42b4cc9c23294f57@mail.gmail.com> <61103.84.157.46.71.1221565997.squirrel@mail.stardawn.org> <7239FF41-4CEA-4E2E-B7B5-4F20D29330DD@media.mit.edu> <42c0ab790809162202i58896ed3h8ad6d07c489676b@mail.gmail.com> <64227.84.157.46.71.1221640311.squirrel@mail.stardawn.org> <42c0ab790809171530m144a9905x8c6341eab394f987@mail.gmail.com> <64723.84.157.13.176.1221727152.squirrel@mail.stardawn.org> <42c0ab790809221501v5d06a294wb564e9a01bd58688@mail.gmail.com> <62098.88.73.220.117.1222158530.squirrel@mail.stardawn.org> <42c0ab790809230909s3d2cacb5he3d330e6fa270a51@mail.gmail.com> <42c0ab790809241153o124371a2x7588db30e379be42@mail.gmail.com> Message-ID: <63687.88.73.240.40.1222331731.squirrel@mail.stardawn.org> > I have attached a bundle with three patches against elephant-unstable: one > correcting a typo in a comment in elephant.asd, one fixing the alien problem > above, and one with the minimal changes I needed to get elephant running on > win32/SBCL. Thanks, that's good code. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From killerstorm at newmail.ru Thu Sep 25 11:45:40 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 25 Sep 2008 14:45:40 +0300 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> <48D8F262.3080903@saphor.de> <48DA2438.6050802@saphor.de><20578A9D-A932-4AAF-BE6C-071CB6BA8D20@media.mit.edu> <48DA8CBC.5040707@saphor.de> Message-ID: <97A107961C0C4D6A9572A164163C898A@killer> M> that sounds like the key for this specific mismatch of the BDB locking M> philosophy ? which for the scenarios you describe is perfect ? and our M> quite different needs. there is other sort of locking philosophy in BDB too -- so-called "snapshot isolation" (also known as multi-version concurrency control). check this: http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/read.html ---- Snapshot isolation also guarantees repeatable reads, but avoids read locks by using multiversion concurrency control (MVCC). This makes update operations more expensive, because they have to allocate space for new versions of pages in cache and make copies, but avoiding read locks can significantly increase throughput for many applications. Snapshot isolation is discussed in detail below. If the cache becomes full of page copies before the old copies can be discarded, additional I/O will occur as pages are written to temporary "freezer" files. ---- so it seems if you use snapshot isolation, you won't have "cannot allocate memory', but if cache is too small performance will be degraded due to additional I/O. but i haven't used it myself, so i dunno if it's really so. From killerstorm at newmail.ru Thu Sep 25 11:45:40 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 25 Sep 2008 14:45:40 +0300 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> <48D8F262.3080903@saphor.de> <48DA2438.6050802@saphor.de><20578A9D-A932-4AAF-BE6C-071CB6BA8D20@media.mit.edu> <48DA8CBC.5040707@saphor.de> Message-ID: <97A107961C0C4D6A9572A164163C898A@killer> M> that sounds like the key for this specific mismatch of the BDB locking M> philosophy ? which for the scenarios you describe is perfect ? and our M> quite different needs. there is other sort of locking philosophy in BDB too -- so-called "snapshot isolation" (also known as multi-version concurrency control). check this: http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/read.html ---- Snapshot isolation also guarantees repeatable reads, but avoids read locks by using multiversion concurrency control (MVCC). This makes update operations more expensive, because they have to allocate space for new versions of pages in cache and make copies, but avoiding read locks can significantly increase throughput for many applications. Snapshot isolation is discussed in detail below. If the cache becomes full of page copies before the old copies can be discarded, additional I/O will occur as pages are written to temporary "freezer" files. ---- so it seems if you use snapshot isolation, you won't have "cannot allocate memory', but if cache is too small performance will be degraded due to additional I/O. but i haven't used it myself, so i dunno if it's really so. From franks-muc at web.de Fri Sep 26 20:26:00 2008 From: franks-muc at web.de (Frank Schorr) Date: Fri, 26 Sep 2008 22:26:00 +0200 Subject: [elephant-devel] get-instance-by-value where value is a list Message-ID: <1334827425@web.de> (using ele version 0.9 on lispworks/win32) I'm storing a list in an indexed slot of a persistent object. Retrieving the object with get-instance-by-value, where value is #'equal to the stored list, does not seem to work. One could print the list to a string and use the string for indexing, but I believe there is a better "design pattern". How do you store (nested) lists in indexed slots ? Thank you for your help. Frank _______________________________________________________________________ Jetzt neu! Sch?tzen Sie Ihren PC mit McAfee und WEB.DE. 30 Tage kostenlos testen. http://www.pc-sicherheit.web.de/startseite/?mc=022220 From eslick at media.mit.edu Fri Sep 26 20:43:41 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 26 Sep 2008 16:43:41 -0400 Subject: [elephant-devel] get-instance-by-value where value is a list In-Reply-To: <1334827425@web.de> References: <1334827425@web.de> Message-ID: <5CE92A0A-92B3-4C2A-9F5D-E8A91C5FE7CA@media.mit.edu> Currently none of the stores support comparing lists. I think we could add this to the BDB backend, but it would require a change in the way that lists are serialized that wouldn't be backwards compatible. Even if we did it for BDB, I don't think we can do it for the postmodern or CL-SQL stores. Unfortunately strings are the only unbounded data item that can be correctly compared by all stores. Consequently, they are the only way to store lists in indexed slots and get the expected behavior. Think of this re-encoding as a means to generate a signature for a class of objects that is guaranteed to have the same total order as the original objects under lisp 'equal. Sometimes the elegance of the hack depends on how you look at it. :) Ian On Sep 26, 2008, at 4:26 PM, Frank Schorr wrote: > (using ele version 0.9 on lispworks/win32) > > I'm storing a list in an indexed slot of a persistent object. > Retrieving the object with > get-instance-by-value, where value is #'equal to the stored list, > does not seem to work. > > One could print the list to a string and use the string for > indexing, but I believe there > is a better "design pattern". > > How do you store (nested) lists in indexed slots ? > > > Thank you for your help. > > Frank > > _______________________________________________________________________ > Jetzt neu! Sch?tzen Sie Ihren PC mit McAfee und WEB.DE. 30 Tage > kostenlos testen. http://www.pc-sicherheit.web.de/startseite/? > mc=022220 > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From elephant-devel at common-lisp.net Sat Sep 27 09:55:06 2008 From: elephant-devel at common-lisp.net (VIAGRA INC) Date: Sat, 27 Sep 2008 05:55:06 -0400 (EDT) Subject: [elephant-devel] RE: SALE 89% OFF Message-ID: <20080927105508.13461.qmail@cpc1-lutn9-0-0-cust979.lutn.cable.ntl.com> An HTML attachment was scrubbed... URL: From killerstorm at newmail.ru Sat Sep 27 12:41:09 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 27 Sep 2008 15:41:09 +0300 Subject: [elephant-devel] get-instance-by-value where value is a list References: <1334827425@web.de> <5CE92A0A-92B3-4C2A-9F5D-E8A91C5FE7CA@media.mit.edu> Message-ID: <64C5CCE2E8844C35833BC2FC09AEC15C@killer> IE> Currently none of the stores support comparing lists. em, we do not support get-instances-by-range queries, but get-instance-by-value should be fine, as long as serialized representation of list is same. ELE-TESTS> (open-store *testbdb-spec*) # ELE-TESTS> (defpclass rsdrp () ((islot :accessor islot :initarg :islot :index t))) # ELE-TESTS> (make-instance 'rsdrp :islot (list 1 2)) # ELE-TESTS> (make-instance 'rsdrp :islot (list 3 4)) # ELE-TESTS> (make-instance 'rsdrp :islot (list 5 6)) # ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 5 6)) # T ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 1 2)) # T ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 3 4)) # T works in db-postmodern too, and i guess should work with clsql. so either Frank has problems with some special form of lists, or maybe he means that it's get-instances-by-range which is broken. From sky at viridian-project.de Sat Sep 27 13:44:30 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sat, 27 Sep 2008 15:44:30 +0200 (CEST) Subject: [elephant-devel] Cached slots Message-ID: <63346.88.73.243.149.1222523070.squirrel@mail.stardawn.org> Are these supposed to work, or is this unfinished stuff? -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From eslick at media.mit.edu Sat Sep 27 17:14:28 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sat, 27 Sep 2008 13:14:28 -0400 Subject: [elephant-devel] get-instance-by-value where value is a list In-Reply-To: <64C5CCE2E8844C35833BC2FC09AEC15C@killer> References: <1334827425@web.de> <5CE92A0A-92B3-4C2A-9F5D-E8A91C5FE7CA@media.mit.edu> <64C5CCE2E8844C35833BC2FC09AEC15C@killer> Message-ID: <44492737-3EAB-40DC-9F04-F94569A72364@media.mit.edu> Alex, you are quite right; compounds structures are compared on per- byte basis so identical serialized bitstreams compare correctly. There is no ordering mechanism, as Alex clarifies below. Ian On Sep 27, 2008, at 8:41 AM, Alex Mizrahi wrote: > IE> Currently none of the stores support comparing lists. > > em, we do not support get-instances-by-range queries, > but get-instance-by-value should be fine, as long as serialized > representation of list is same. > > ELE-TESTS> (open-store *testbdb-spec*) > # > > ELE-TESTS> (defpclass rsdrp () > ((islot :accessor islot :initarg :islot :index t))) > # > ELE-TESTS> (make-instance 'rsdrp :islot (list 1 2)) > # > ELE-TESTS> (make-instance 'rsdrp :islot (list 3 4)) > # > ELE-TESTS> (make-instance 'rsdrp :islot (list 5 6)) > # > ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 5 6)) > # > T > ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 1 2)) > # > T > ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 3 4)) > # > T > > works in db-postmodern too, and i guess should work with clsql. > > so either Frank has problems with some special form of lists, > or maybe he means that it's get-instances-by-range which is broken. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Sat Sep 27 17:51:32 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sat, 27 Sep 2008 13:51:32 -0400 Subject: [elephant-devel] Cached slots In-Reply-To: <63346.88.73.243.149.1222523070.squirrel@mail.stardawn.org> References: <63346.88.73.243.149.1222523070.squirrel@mail.stardawn.org> Message-ID: <735F2133-7C1F-4914-B963-B32A9BB2B1B1@media.mit.edu> Leslie, A few of the prior messages are proof that my memory is full of holes these days, so take this with a grain of salt! This is very unfinished stuff. Cached operations has been tested minimally interactively, but hasn't been verified via test suites nor used in any significant way. As best as I can remember, it should work fine in a "single-user" mode or in a read-only mode. The open problem for concurrent use is making cached reads transaction safe in the presence of writes (Alex pointed out some of the holes in the original scheme in earlier e-mails). Another issue is maintaining the semantics of index operations within a transaction, etc. This is going to be tricky to get right. There may be some stylized usage modes, but it will take some careful thought to make sure any of these work properly in the presence of writes and transaction restarts, either concurrent or single-writer. - Read-only: I can't see a problem with this - Single-writer: but only if reading stale data is safe... - Single-writer w/ guard: have a persistent 'locked' slot that acts as a guard and is touched at the beginning of a transaction by readers and written by writers at the beginning and end. So long as all transactions have idempotent memory side effects (which should always be the case), this should allow you to maintain transaction semantics for the single-writer case. I'm not confident this actually covers all cases though. If it does work, you should be able to write a macro to simplify this; something like: (with-guarded-instances (:reading instances :writing instances) &body body) should be sufficient. There is also a 'with-cached-instances' macro there which caches reads only during a transaction, which means that all slots are read at the beginning. A write through policy on slots will capture write locks appropriately. I'm also not sure if this covers all cases and will require some thought as well. I'm not up to getting the transaction semantics right anytime soon, but we should be able to write some macros that cleanly encapsulate any design patterns that we can verify. Ian On Sep 27, 2008, at 9:44 AM, Leslie P. Polzer wrote: > > Are these supposed to work, or is this unfinished stuff? > > -- > LinkedIn Profile: http://www.linkedin.com/in/polzer > Xing Profile: https://www.xing.com/profile/LeslieP_Polzer > Blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From sky at viridian-project.de Sun Sep 28 10:03:05 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sun, 28 Sep 2008 12:03:05 +0200 (CEST) Subject: [elephant-devel] 0.9.2 Message-ID: <61545.88.73.251.160.1222596185.squirrel@mail.stardawn.org> When all my patches have entered the repository it might be time to release 0.9.2 or at least a 0.9.2pre1. The state by then will be: * working db4.7, built-in deadlock handling * working schema evolution * associations mostly working * cached slots unfinished (reserved for post-0.9.2) * documentation could still need a bit of love, but is usable Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From sky at viridian-project.de Sun Sep 28 10:04:01 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sun, 28 Sep 2008 12:04:01 +0200 (CEST) Subject: [elephant-devel] Cursor semantics Message-ID: <61593.88.73.251.160.1222596241.squirrel@mail.stardawn.org> I noticed the cursor interface has changed, with cursor-next et al. returning an oid instead of the object. Is this intentional? Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From sky at viridian-project.de Sun Sep 28 10:10:40 2008 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sun, 28 Sep 2008 12:10:40 +0200 (CEST) Subject: [elephant-devel] Ready-to-run Elephant with BDB Message-ID: <61847.88.73.251.160.1222596640.squirrel@mail.stardawn.org> Getting Elephant set up with any store other than SQLite is an annoying thing for beginners. If we'd make it easier to use BDB out of the box we could just drop the CLSQL store in future versions and make Elephant more accessible for newcomers. Proposal: * include BDB headers in src/db-bdb. * include pre-built libdb-4.7.so or offer to download them * offer to download a BDB tarball, build it and configure Elephant to use it The last two could either be offered on first run or when a problem (lib not found, etc.) is detected. Leslie -- LinkedIn Profile: http://www.linkedin.com/in/polzer Xing Profile: https://www.xing.com/profile/LeslieP_Polzer Blog: http://blog.viridian-project.de/ From franks-muc at web.de Sun Sep 28 16:27:21 2008 From: franks-muc at web.de (Frank Schorr) Date: Sun, 28 Sep 2008 18:27:21 +0200 Subject: [elephant-devel] get-instance-by-value where value is a list Message-ID: <1337342083@web.de> get-instance-by-value returns nil to me in the example below. May be it is due to my configuration, which is ele 0.9 on lispworks/win32 with postmodern. printing the list to a string and storing the string and reading back is ok, however. Frank > -----Urspr?ngliche Nachricht----- > Von: "Alex Mizrahi" > Gesendet: 27.09.08 15:11:39 > An: "Elephant bugs and development" > Betreff: Re: [elephant-devel] get-instance-by-value where value is a list > IE> Currently none of the stores support comparing lists. > > em, we do not support get-instances-by-range queries, > but get-instance-by-value should be fine, as long as serialized > representation of list is same. > > ELE-TESTS> (open-store *testbdb-spec*) > # > > ELE-TESTS> (defpclass rsdrp () > ((islot :accessor islot :initarg :islot :index t))) > # > ELE-TESTS> (make-instance 'rsdrp :islot (list 1 2)) > # > ELE-TESTS> (make-instance 'rsdrp :islot (list 3 4)) > # > ELE-TESTS> (make-instance 'rsdrp :islot (list 5 6)) > # > ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 5 6)) > # > T > ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 1 2)) > # > T > ELE-TESTS> (get-instance-by-value 'rsdrp 'islot (list 3 4)) > # > T > > works in db-postmodern too, and i guess should work with clsql. > > so either Frank has problems with some special form of lists, > or maybe he means that it's get-instances-by-range which is broken. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > ____________________________________________________________________ Psssst! Schon vom neuen WEB.DE MultiMessenger geh?rt? Der kann`s mit allen: http://www.produkte.web.de/messenger/?did=3123 From killerstorm at newmail.ru Sun Sep 28 17:03:44 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 28 Sep 2008 20:03:44 +0300 Subject: [elephant-devel] get-instance-by-value where value is a list References: <1337342083@web.de> Message-ID: <2CCBFB7178CB4B10A9AF0469D2B549CB@killer> FS> May be it is due to my configuration, which is ele 0.9 on FS> lispworks/win32 with postmodern. yep i think in 0.9 postmodern backend was quite botched. i recommend using version from darcs, considerable amount of issues were fixed since then.