From neuss at kit.edu Fri Mar 5 08:49:53 2010 From: neuss at kit.edu (Nicolas Neuss) Date: Fri, 05 Mar 2010 09:49:53 +0100 Subject: [elephant-devel] deadlock/ Berkeley 4.8 DB Message-ID: <87y6i7gnke.fsf@ma-patru.mathematik.uni-karlsruhe.de> Hello, I would like to start using Elephant. Unfortunately, I do not know much about the Berkeley database system, so I am stuck just at the beginning. I would like to use the Berkeley libraries which come with my Debian/Lenny (which is version 4.8), and cannot find the db_deadlock binary there which is asked for in config.sexp. Does anyone know where this binary resides? Or if it is there at all in this version? Thank you, Nicolas From lukas.giessmann at hotmail.de Fri Mar 5 13:12:19 2010 From: lukas.giessmann at hotmail.de (=?iso-8859-1?Q?Lukas_Gie=DFmann?=) Date: Fri, 5 Mar 2010 14:12:19 +0100 Subject: [elephant-devel] get-instances-by-value and get-instances-by-class Message-ID: Hi list, in our application which uses elephant 1.0 with sbcl 1.0.29.11.debian on ubuntu we have a memory leak. After a lot of debugging I am sure to have found the responsible function. It seems that elephant:get-instances-by-value does not release the needed memory after the operation is done. I also have some debug-examples with this and other elephant functions and the needed memory: --------------------- (dotimes (i 1000000) (elephant:get-instances-by-value 'd::TopicIdentificationC 'd::uri "goethe")) ;--> memory that is not released after operation 29.0 MB (dotimes (i 1000000) (elephant:get-instance-by-value 'd::TopicIdentificationC 'd::uri "goethe")) ;--> memory that is not released after operation 0.1 MB (dotimes (i 1000000) (elephant:get-instances-by-class 'd::TopicIdentificationC)) ;--> memory that is not released after operation 58.9 MB --------------------- Is there any function or any method to relaese the memory after a call of elephant:get-instances-by-value or elephant:get-instances-by-class. Maybe there is any other work-around? Thanks in advance Lukas -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at media.mit.edu Fri Mar 5 15:39:55 2010 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 5 Mar 2010 07:39:55 -0800 Subject: [elephant-devel] get-instances-by-value and get-instances-by-class In-Reply-To: References: Message-ID: <923A00BF-1423-476C-A591-343927B1CDC2@media.mit.edu> Hi Lukas, How are you defining 'memory not released'? Is this a measure of the Lisp heap or the OS footprint? As I recall, when allocating space in the heap for new objects, SBCL will periodically ask the OS for more heap memory. Even if the objects are garbage collected later, the allocated memory will remain (SBCL never releases heap memory back to the OS even if it isn't using most of it). It also asks for new heap space in increasingly large chunks much like a hash table. The objects loaded into memory are stored in a weak hash table, and unless there's a bug in a recent SBCL, if you don't reference them elsewhere they should be garbage collected. You can use (gc) to make sure the GC has run and cleaned everything out and (room) to see how much memory is allocated for what kinds of objects. It's possible that some of this memory is the shared memory allocated by BDB - but that can is upper-bounded in the my-config.sexp file and should be static. Ian On Mar 5, 2010, at 5:12 AM, Lukas Gie?mann wrote: > Hi list, > > in our application which uses elephant 1.0 with sbcl 1.0.29.11.debian on ubuntu we have a memory leak. > After a lot of debugging I am sure to have found the responsible function. It seems that elephant:get-instances-by-value does not release the needed memory after the operation is done. > I also have some debug-examples with this and other elephant functions and the needed memory: > > --------------------- > (dotimes (i 1000000) > (elephant:get-instances-by-value 'd::TopicIdentificationC 'd::uri "goethe")) > ;--> memory that is not released after operation 29.0 MB > > > (dotimes (i 1000000) > (elephant:get-instance-by-value 'd::TopicIdentificationC 'd::uri "goethe")) > ;--> memory that is not released after operation 0.1 MB > > > (dotimes (i 1000000) > (elephant:get-instances-by-class 'd::TopicIdentificationC)) > ;--> memory that is not released after operation 58.9 MB > --------------------- > > Is there any function or any method to relaese the memory after a call of elephant:get-instances-by-value or elephant:get-instances-by-class. Maybe there is any other work-around? > > > Thanks in advance > > Lukas > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at media.mit.edu Fri Mar 5 15:41:33 2010 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 5 Mar 2010 07:41:33 -0800 Subject: [elephant-devel] deadlock/ Berkeley 4.8 DB In-Reply-To: <87y6i7gnke.fsf@ma-patru.mathematik.uni-karlsruhe.de> References: <87y6i7gnke.fsf@ma-patru.mathematik.uni-karlsruhe.de> Message-ID: I'm not sure if Elephant will play nicely with Berkeley DB 4.8 yet - we should probably fix this. For now, however, you can ignore the db_deadlock setting in my-config as it is not strictly required as of 4.7. Ian On Mar 5, 2010, at 12:49 AM, Nicolas Neuss wrote: > Hello, > > I would like to start using Elephant. Unfortunately, I do not know much > about the Berkeley database system, so I am stuck just at the beginning. > > I would like to use the Berkeley libraries which come with my > Debian/Lenny (which is version 4.8), and cannot find the db_deadlock > binary there which is asked for in config.sexp. > > Does anyone know where this binary resides? Or if it is there at all in > this version? > > Thank you, > Nicolas > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Fri Mar 5 17:13:42 2010 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Fri, 5 Mar 2010 19:13:42 +0200 Subject: [elephant-devel] get-instances-by-value andget-instances-by-class References: <923A00BF-1423-476C-A591-343927B1CDC2@media.mit.edu> Message-ID: IE> As I recall, when allocating space in the heap for new objects, SBCL will periodically IE> ask the OS for more heap memory. Even if the objects are garbage collected later, IE> the allocated memory will remain (SBCL never releases heap memory back to the IE> OS even if it isn't using most of it). It also asks for new heap space in increasingly IE> large chunks much like a hash table. SBCL allocates a memory chunk of a fixed size, but it is only reserved at the beginning -- it is a normal behaviour on Linux that memory is actually allocated only on the first write to a page. So, yes, memory usage (RSS -- Resident Set Size) grows after use, but it does not grow indefinitely. There is --dynamic-space-size option in SBCL to control heap size. It makes sense to set --dynamic-space-size to some reasonable value, and then watch if application dies of memory starvation. If it does not die, then it is not a leak. Maybe SBCL just does not feel a need to clean up weak hash table or some memory generations, and if it is within allocated heap, it has a right to keep them. IE> It's possible that some of this memory is the shared memory allocated by BDB - but that can is upper-bounded in the my-config.sexp file and should be static. It is also possible that there is a bug in Elephant, somewhere in FFI code or in functions which do foreign calls. Technically it is also possible that leak is caused by SBCL itself -- SBCL has a conservative GC, but it only is a problem in some rare situations, when you build very long lists, for example, and probably it's not a case here. --------------------- (dotimes (i 1000000) (elephant:get-instances-by-value 'd::TopicIdentificationC 'd::uri "goethe")) ;--> memory that is not released after operation 29.0 MB (dotimes (i 1000000) (elephant:get-instance-by-value 'd::TopicIdentificationC 'd::uri "goethe")) ;--> memory that is not released after operation 0.1 MB --------------------- This is pretty interesting, if indeed get-instances-by-value leak and get-instance-by-value does not, it should be easy to pinpoint the problem. How many instances does get-instances-by-value return? Is it a long list? From lukas.giessmann at hotmail.de Fri Mar 5 20:15:23 2010 From: lukas.giessmann at hotmail.de (=?iso-8859-1?B?THVrYXMgR2ll321hbm4=?=) Date: Fri, 5 Mar 2010 21:15:23 +0100 Subject: [elephant-devel] get-instances-by-value andget-instances-by-class In-Reply-To: References: , <923A00BF-1423-476C-A591-343927B1CDC2@media.mit.edu>, Message-ID: Thanks for you answers, it seems that there is a problem with sbcl and linux 64-bit. I noticed that this problems does not appear on my 32-bit systems only on the 64-bit systems. If I start sbcl with --dynamic-space-size there is no effect by allocating more memory than specified - therefor I used the os-information about memory-usage. But (room) shows always a smaller amount of allocated memory. With more calls of the examples I showed before the difference between (room) and the os-information grows. (gc) has no effect - (room) shows that (gc) released some memory, but the os still shows that there wasn't released anything. Best regards Lukas > To: elephant-devel at common-lisp.net > From: killerstorm at newmail.ru > Date: Fri, 5 Mar 2010 19:13:42 +0200 > Subject: Re: [elephant-devel] get-instances-by-value andget-instances-by-class > > IE> As I recall, when allocating space in the heap for new objects, SBCL > will periodically > IE> ask the OS for more heap memory. Even if the objects are garbage > collected later, > IE> the allocated memory will remain (SBCL never releases heap memory back > to the > IE> OS even if it isn't using most of it). It also asks for new heap space > in increasingly > IE> large chunks much like a hash table. > > SBCL allocates a memory chunk of a fixed size, but it is only reserved at > the beginning -- it is a normal behaviour on Linux that memory is actually > allocated only on the first write to a page. So, yes, memory usage (RSS -- > Resident Set Size) grows after use, but it does not grow indefinitely. There > is --dynamic-space-size option in SBCL to control heap size. > > It makes sense to set --dynamic-space-size to some reasonable value, and > then watch if application dies of memory starvation. > > If it does not die, then it is not a leak. Maybe SBCL just does not feel a > need to clean up weak hash table or some memory generations, and if it is > within allocated heap, it has a right to keep them. > > IE> It's possible that some of this memory is the shared memory allocated > by BDB - but that can is upper-bounded in the my-config.sexp file and should > be static. > > It is also possible that there is a bug in Elephant, somewhere in FFI code > or in functions which do foreign calls. > > Technically it is also possible that leak is caused by SBCL itself -- SBCL > has a conservative GC, but it only is a problem in some rare situations, > when you build very long lists, for example, and probably it's not a case > here. > > --------------------- > (dotimes (i 1000000) > (elephant:get-instances-by-value 'd::TopicIdentificationC 'd::uri > "goethe")) > ;--> memory that is not released after operation 29.0 MB > > > (dotimes (i 1000000) > (elephant:get-instance-by-value 'd::TopicIdentificationC 'd::uri > "goethe")) > ;--> memory that is not released after operation 0.1 MB > --------------------- > > This is pretty interesting, if indeed get-instances-by-value leak and > get-instance-by-value does not, it should be easy to pinpoint the problem. > > How many instances does get-instances-by-value return? Is it a long list? > > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel _________________________________________________________________ http://redirect.gimas.net/?n=M1003xWin7Geschenk2 Hol dir das Gratis-Geschenkpaket von Windows 7 f?r deinen PC ab! -------------- next part -------------- An HTML attachment was scrubbed... URL: From killerstorm at newmail.ru Sat Mar 6 17:52:35 2010 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 6 Mar 2010 19:52:35 +0200 Subject: [elephant-devel] get-instances-by-value andget-instances-by-class References: , <923A00BF-1423-476C-A591-343927B1CDC2@media.mit.edu>, Message-ID: <266865E983A14E2CAFE3E9B51E57C724@killer> hi > it seems that there is a problem with sbcl and linux 64-bit. > I noticed that this problems does not appear on my 32-bit systems only on the 64-bit systems. I've tested it on Debian 5.0 64-bit: SBCL 1.0.18, BDB 4.6, both stock from Debian packages. There is no leak neither for get-instances-by-value, nor ofr get-instances-by-class: CL-USER> (require 'ele-bdb) ... CL-USER> (use-package :elephant) T CL-USER> (open-store '(:bdb #p"mydb/")) # CL-USER> (defpclass bigj () ((slo1 :accessor slot1 :initarg :slot1 :index t))) # CL-USER> (make-instance 'bigj :slot1 "goethe") # CL-USER> (make-instance 'bigj :slot1 "goethe2") # CL-USER> (make-instance 'bigj :slot1 "goeth") # CL-USER> (dotimes (i 1000000) (get-instances-by-value 'bigj 'slo1 "goethe")) NIL CL-USER> (dotimes (i 1000000) (get-instances-by-class 'bigj)) NIL CL-USER> (dotimes (i 1000000) (get-instance-by-value 'bigj 'slo1 "goethe")) NIL Both before and after tests SBCL had resident set size of 100 MB, as measured by `top`. So I guess it is an error with a specific version of SBCL or BDB, so try different versions. From sky at viridian-project.de Sat Mar 6 18:50:46 2010 From: sky at viridian-project.de (Leslie P. Polzer) Date: Sat, 6 Mar 2010 19:50:46 +0100 (CET) Subject: [elephant-devel] deadlock/ Berkeley 4.8 DB In-Reply-To: References: <87y6i7gnke.fsf@ma-patru.mathematik.uni-karlsruhe.de> Message-ID: <4f7ffffd320803e9891a640ad8f61860.squirrel@mail.stardawn.org> Ian Eslick wrote: > I'm not sure if Elephant will play nicely with Berkeley DB 4.8 yet - we should probably > fix this. Yeah, 4.8 needs some adjustments. > For now, however, you can ignore the db_deadlock setting in my-config as it > is not strictly required as of 4.7. I'm in favor of removing this altogether. The default built-in deadlock mechanism should be fine for the vast majority of people. Leslie From lukas.giessmann at hotmail.de Sat Mar 6 20:28:54 2010 From: lukas.giessmann at hotmail.de (=?iso-8859-1?B?THVrYXMgR2ll321hbm4=?=) Date: Sat, 6 Mar 2010 21:28:54 +0100 Subject: [elephant-devel] get-instances-by-value andget-instances-by-class In-Reply-To: <266865E983A14E2CAFE3E9B51E57C724@killer> References: , <923A00BF-1423-476C-A591-343927B1CDC2@media.mit.edu>, <266865E983A14E2CAFE3E9B51E57C724@killer> Message-ID: Hi Alex, you' re right! I spent the entire day with debugging the problem on several systems. I think the problem is caused by the compilation of elephant. Concerning the compilation problem I wrote to the list some time ago( [elephant-devel] installing elephant 1.0 from repository). The last email of this thread is attached. Well I solved this problem - at least, currently it looks like I did - by removing all .fasl files and recompiling elephant directly in sbcl and not in slime. So the gcc-call does not fail when I call (asdf:operate 'asdf:load-op :elephant). Best regards Lukas -------------------------------------------------- From: "Alex Mizrahi" Sent: Saturday, March 06, 2010 6:52 PM To: "elephant-devel" ; Subject: Re: [elephant-devel]get-instances-by-valueandget-instances-by-class> hi > > > it seems that there is a problem with sbcl and linux 64-bit. > > I noticed that this problems does not appear on my 32-bit systems only > > on > the 64-bit systems. > > I've tested it on Debian 5.0 64-bit: SBCL 1.0.18, BDB 4.6, both stock from > Debian packages. > There is no leak neither for get-instances-by-value, nor ofr > get-instances-by-class: > > CL-USER> (require 'ele-bdb) > ... > CL-USER> (use-package :elephant) > T > CL-USER> (open-store '(:bdb #p"mydb/")) > # > CL-USER> (defpclass bigj () ((slo1 :accessor slot1 :initarg :slot1 :index > t))) > # > CL-USER> (make-instance 'bigj :slot1 "goethe") > # > CL-USER> (make-instance 'bigj :slot1 "goethe2") > # > CL-USER> (make-instance 'bigj :slot1 "goeth") > # > CL-USER> (dotimes (i 1000000) > (get-instances-by-value 'bigj 'slo1 "goethe")) > NIL > CL-USER> (dotimes (i 1000000) > (get-instances-by-class 'bigj)) > NIL > CL-USER> (dotimes (i 1000000) > (get-instance-by-value 'bigj 'slo1 "goethe")) > NIL > > Both before and after tests SBCL had resident set size of 100 MB, as > measured by `top`. > > So I guess it is an error with a specific version of SBCL or BDB, so try > different versions. > --Weitergeleitete E-Mail-Anlagen-- From: lukas.giessmann at hotmail.de To: elephant-devel at common-lisp.net Date: Tue, 15 Dec 2009 15:08:26 +0100 Subject: Re: [elephant-devel] installing elephant 1.0 from repository Hi Ian, thanks for you answer. The gcc build via asdf works fine in the latest revision - but the darcs warning is still there. At the moment it looks like it causes no problems. Best regards Lukas From: eslick at media.mit.edu Date: Mon, 14 Dec 2009 07:53:23 -0800 To: elephant-devel at common-lisp.net Subject: Re: [elephant-devel] installing elephant 1.0 from repository Hi Lukas, I haven't seen that particular darcs problem before, although I have seen the gcc build via asdf fail but work when done manually. This may be fixed in the latest head, actually. You might try pulling elephant-1.0 without the --tag option and see if that fixes it. Best,Ian On Dec 14, 2009, at 6:49 AM, Lukas Gie?mann wrote:Hi, I am wondering about my installation of elephant 1.0. I just tried to install elephant 1.0 directly from the darcs repository by the command "darcs gethttp://www.common-lisp.net/project/elephant/darcs/elephant-1.0" and/or "darcs get --tag=ELEPHANT-1-0-A2http://www.common-lisp.net/project/elephant/darcs/elephant-1.0". When the repository was downloaded on the system, darcs displayed the message "Warning: CRC errors found. These are probably harmless but should be repaired. See 'darcs gzcrcs --help' for more information." The commands "darcs gzcrcs --check" and "darcs gzcrcs --repair" had no effects. When I was loading elephant on sbcl with slime, it stopped to work at the command "gcc -L/usr/local/BerkeleyDB.4.5/lib/ -I/usr/local/BerkeleyDB.4.5/include/ -shared -march=x86-64 -fPIC -Wall -g -O2 -g /home/lukas/.sbcl/site/elephant-1.0/src/db-bdb/libberkeley-db.c -o /var/cache/common-lisp-controller/1000/sbcl/local/home/lukas/.sbcl/site/elephant-1.0/src/db-bdb/libberkeley-db.so -lm". My work-around was to execute the last command in the shell and to recompile elephant in slime/sbcl. I am not sure if this is a bug or if I did something wrong? Maybe there is someone with the same problem? Best regards Lukas Alle E-Mail-Adressen online auf einen Blick Ich will Hotmail!_______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel Alle E-Mail-Adressen online auf einen Blick Ich will Hotmail! _________________________________________________________________ http://redirect.gimas.net/?n=M1003xIMVideochat2 Treffe Freunde im Messenger Videochat! -------------- next part -------------- An HTML attachment was scrubbed... URL: From neuss at kit.edu Sun Mar 14 10:37:58 2010 From: neuss at kit.edu (Nicolas Neuss) Date: Sun, 14 Mar 2010 11:37:58 +0100 Subject: [elephant-devel] Unique key with Elephant Message-ID: <87634z429l.fsf@ma-patru.mathematik.uni-karlsruhe.de> Hello, as suggested in the following thread I am posting the wish here in the mailing list: http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/23e19ec769d2f894 I would be interested in that feature. Thanks, Nicolas From eslick at media.mit.edu Sun Mar 14 17:16:54 2010 From: eslick at media.mit.edu (Ian Eslick) Date: Sun, 14 Mar 2010 10:16:54 -0700 Subject: [elephant-devel] Unique key with Elephant In-Reply-To: <87634z429l.fsf@ma-patru.mathematik.uni-karlsruhe.de> References: <87634z429l.fsf@ma-patru.mathematik.uni-karlsruhe.de> Message-ID: <8815EC08-97E8-475B-885E-A1947F8348F6@media.mit.edu> I tend to agree with Alex on this one. Moreover, I'm not going to have time to make changes for awhile, but I'd be willing to review a patch. Should be easy to add an index lookup to shared-initialize and a check in indexed slot access as to whether you are changing the unique value. Can a unique value be changed to another unique value or must it always remain the same value? Ian On Mar 14, 2010, at 3:37 AM, Nicolas Neuss wrote: > Hello, > > as suggested in the following thread I am posting the wish here in the > mailing list: > > http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/23e19ec769d2f894 > > I would be interested in that feature. > > Thanks, > Nicolas > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Thu Mar 18 13:37:40 2010 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 18 Mar 2010 15:37:40 +0200 Subject: [elephant-devel] Unique key with Elephant References: <87634z429l.fsf@ma-patru.mathematik.uni-karlsruhe.de> <8815EC08-97E8-475B-885E-A1947F8348F6@media.mit.edu> Message-ID: IE> Should be easy to add an index lookup to shared-initialize and a check IE> in indexed slot access as to whether you are changing the unique value. Hmm, there is a problem with this and concurrency -- one thread does a check, it passes and it inserts a value, and another thread does the same thing. Now there are two entries with same key, despite both checks passed. Or will locks in BDB prevent this? In PostgreSQL they do not, unless there is unique index in database. From eslick at media.mit.edu Sat Mar 20 06:47:05 2010 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 19 Mar 2010 23:47:05 -0700 Subject: [elephant-devel] Unique key with Elephant In-Reply-To: References: <87634z429l.fsf@ma-patru.mathematik.uni-karlsruhe.de> <8815EC08-97E8-475B-885E-A1947F8348F6@media.mit.edu> Message-ID: <648D60AF-6065-47FE-8601-082B0ABC2420@media.mit.edu> I can't say I've thought deeply about this, but instance creation is wrapped in a transaction - so long as the side effects you are concerned about, including reads the store, they should be atomic due to BDB locking. On Mar 18, 2010, at 6:37 AM, Alex Mizrahi wrote: > IE> Should be easy to add an index lookup to shared-initialize and a check > IE> in indexed slot access as to whether you are changing the unique value. > > Hmm, there is a problem with this and concurrency -- one thread does a > check, it passes and it inserts a value, and another thread does the same > thing. > Now there are two entries with same key, despite both checks passed. > > Or will locks in BDB prevent this? > > In PostgreSQL they do not, unless there is unique index in database. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel