From lukas.georgieff at hotmail.com  Thu Aug  4 13:07:13 2011
From: lukas.georgieff at hotmail.com (Lukas Georgieff)
Date: Thu, 4 Aug 2011 15:07:13 +0200
Subject: [elephant-devel] memory problems with elephant in a multi-threaded
	sbcl-system
Message-ID: <SNT125-W135F31A48326CCC7FD4D6AF53D0@phx.gbl>


Hi list,

I've a problem with elephant and threading that drives me to despair. In the hope that anyone solved similar problems and may suggest a solution I am posting this thread.

I am running the latest darcs version of elephant on a 64 bit debian dual-core system on sbcl 1.0.40.0.debian with berkeley db 4.7.25.

The last days I ran a performance test on our system. Since the system is multi-threaded and we use transactions, we also use mutexes to synchronize the write and read operations (our mutex synchronisation is working!). The test I performed has only read the storage, i.e. clients started a TCP connection and invoked a read operation on the system. If more than 4 clients/threads read concurrently the storage, the memory/swap of the server system was fully reserved by the SBCL process and not released after the requests.
The requested data we use is persisted by using defpclass on several class definitions.

During my tests I made the following observations:
* the more intensive the algorithm and reading operations, the faster the memory is reserved, i.e. more reading operations lead to more memory usage
* running SBCL on one core (with taskset) leads to a common memory usage, i.e. constantly about 140 - 150 MB for the SBCL process
* in the past we made a similar test, that worked properly - but since this test, our data model has become more complex, i.e. we use more objects that are handled by each request


Sincerely,

Lukas
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20110804/635bf355/attachment.html>

From alex.mizrahi at gmail.com  Fri Aug 19 15:51:02 2011
From: alex.mizrahi at gmail.com (Alex Mizrahi)
Date: Fri, 19 Aug 2011 18:51:02 +0300
Subject: [elephant-devel] patches
Message-ID: <j2m0p7$iru$1@dough.gmane.org>

I've pushed both my patches and fix posted by Nikodemus.

These ones fix general elephant functionality, so code review is welcome:

   * default-pset workaround
   * removed unnecessary MOP manipulations for SBCL (as reported by N. 
Siivola)
   * fix dropping instance with derived index
   * derived index rebuilding, inherited index rebuilding
   * different handling of ensure-schema and compute-derived-index-triggers

These ones are specific to db-postmodern:

   * pset fix
   * pm-pset
   * in place upgrade support




From killerstorm at newmail.ru  Sun Aug 21 11:52:43 2011
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Sun, 21 Aug 2011 14:52:43 +0300
Subject: [elephant-devel] patches
References: <j2m0p7$iru$1@dough.gmane.org>
Message-ID: <j2qrhu$p6b$1@dough.gmane.org>

Current status:

1. SBCL 1.0.16:
    BDB -- all tests ok (on the first run)
    Postmodern -- about 350 tests pass, then it loses database connection in 
test-indices test and all subsequent tests fail.
         Also it says something about starting select when interrupts are 
off, maybe it is connected?
2. CCL 1.7:
    BDB -- fails to even start because of some fucked up problem with 
classes
    Postmodern -- pretty much same like with SBCL

Removing my patches doesn't fix problems with postmodern or CCL so it I 
guess it isn't a regression.

Here's what I get in CCL:

There is no applicable method for the generic function:
  #<STANDARD-GENERIC-FUNCTION MAP-BTREE #x187D3EDE>
when called with arguments:
  (#<Compiled-function (:INTERNAL
                        ELEPHANT::GET-DB-SCHEMAS) (Non-Global)  #x188E8B0E>
   #<BDB-BTREE-INDEX oid:0> :VALUE DB-BDB::BDB-BTREE-INDEX :COLLECT T)


ELE-TESTS> (describe (find-class 'db-bdb::bdb-btree-index))
...
CCL::PRECEDENCE-LIST: (#<PERSISTENT-METACLASS DB-BDB::BDB-BTREE-INDEX>
                       #<PERSISTENT-METACLASS PERSISTENT-OBJECT>
                       #<STANDARD-CLASS PERSISTENT>
                       #<STANDARD-CLASS STANDARD-OBJECT> #<BUILT-IN-CLASS 
T>)


So somehow INDEXED-BTREE is not not among superclasses.
BUT it is defined like this:

(defclass bdb-btree-index (btree-index bdb-btree)
  ()
  (:metaclass persistent-metaclass)
  (:documentation "A BDB-based BTree supports secondary indices."))

And when I re-define class it skips this problem and goes shows some 
different error...
 But how could it lose one of class's ancestors?? 





From raison at chatsubo.net  Tue Aug 23 04:54:40 2011
From: raison at chatsubo.net (Kevin Raison)
Date: Mon, 22 Aug 2011 21:54:40 -0700
Subject: [elephant-devel] SBCL errors
In-Reply-To: <CADow0CqMmvgTtk=_0Lk7PtFsry-PF0-khaXCSWevF-vVQnGokQ@mail.gmail.com>
References: <4E323FA6.2020000@chatsubo.net>
	<CADow0CqMmvgTtk=_0Lk7PtFsry-PF0-khaXCSWevF-vVQnGokQ@mail.gmail.com>
Message-ID: <4E533290.9050008@chatsubo.net>

I just pulled sbcl-1.0.51 and tried the elephant test suite again;  I 
get the same errors;  did the changes not make it into the newest sbcl 
release?

  Did 505 checks.
     Pass: 500 (99%)
     Skip: 1 ( 0%)
     Fail: 4 ( 0%)

  Failure Details:
  --------------------------------
  SIMPLE-EXPLICIT-ASSOC-SETUP []:
       Unexpected Error: #<SB-INT:BUG "~@<failed AVER: ~2I~_~A~:>" 
{1005033571}>
   failed AVER: NSLOTDS
This is probably a bug in SBCL itself. (Alternatively, SBCL might have been
corrupted by bad user code, e.g. by an undefined Lisp operation like
(FMAKUNBOUND 'COMPILE), or by stray pointers from alien code or from unsafe
Lisp code; or there might be a bug in the OS or hardware that SBCL is 
running
on.) If it seems to be a bug in SBCL itself, the maintainers would like 
to know
about it. Bug reports are welcome on the SBCL mailing lists, which you 
can find
at <http://sbcl.sourceforge.net/>...
  --------------------------------
  --------------------------------
  CHANGE-CLASS2 []:
       Unexpected Error: #<SB-INT:BUG "~@<failed AVER: ~2I~_~A~:>" 
{10033EE511}>
   failed AVER: NSLOTDS
This is probably a bug in SBCL itself. (Alternatively, SBCL might have been
corrupted by bad user code, e.g. by an undefined Lisp operation like
(FMAKUNBOUND 'COMPILE), or by stray pointers from alien code or from unsafe
Lisp code; or there might be a bug in the OS or hardware that SBCL is 
running
on.) If it seems to be a bug in SBCL itself, the maintainers would like 
to know
about it. Bug reports are welcome on the SBCL mailing lists, which you 
can find
at <http://sbcl.sourceforge.net/>...
  --------------------------------
  --------------------------------
  CHANGE-CLASS []:
       Unexpected Error: #<SB-INT:BUG "~@<failed AVER: ~2I~_~A~:>" 
{1002EC2591}>
   failed AVER: NSLOTDS
This is probably a bug in SBCL itself. (Alternatively, SBCL might have been
corrupted by bad user code, e.g. by an undefined Lisp operation like
(FMAKUNBOUND 'COMPILE), or by stray pointers from alien code or from unsafe
Lisp code; or there might be a bug in the OS or hardware that SBCL is 
running
on.) If it seems to be a bug in SBCL itself, the maintainers would like 
to know
about it. Bug reports are welcome on the SBCL mailing lists, which you 
can find
at <http://sbcl.sourceforge.net/>...
  --------------------------------
  --------------------------------
  UPDATE-CLASS []:
       Unexpected Error: #<SB-INT:BUG "~@<failed AVER: ~2I~_~A~:>" 
{1003DEA081}>
   failed AVER: NSLOTDS
This is probably a bug in SBCL itself. (Alternatively, SBCL might have been
corrupted by bad user code, e.g. by an undefined Lisp operation like
(FMAKUNBOUND 'COMPILE), or by stray pointers from alien code or from unsafe
Lisp code; or there might be a bug in the OS or hardware that SBCL is 
running
on.) If it seems to be a bug in SBCL itself, the maintainers would like 
to know
about it. Bug reports are welcome on the SBCL mailing lists, which you 
can find
at <http://sbcl.sourceforge.net/>...
  --------------------------------

  Skip Details:
  SIMPLE-EXPLICIT-ASSOC []:
      Dependencies not satisfied.



On 07/29/2011 03:52 AM, Nikodemus Siivola wrote:
> On 29 July 2011 08:05, Kevin Raison<raison at chatsubo.net>  wrote:
>
>>       Unexpected Error: #<SB-INT:BUG "~@<failed AVER: ~2I~_~A~:>"
>> {1002E81E41}>
>>   failed AVER: NSLOTDS
>
> This appears to be an SBCL regression from 1.0.47.25. (Plus and older
> bug.) Fixed on SBCL master.
>
> Cheers,
>
>   -- nikodemus
>



From raison at chatsubo.net  Tue Aug 23 14:41:34 2011
From: raison at chatsubo.net (Kevin Raison)
Date: Tue, 23 Aug 2011 07:41:34 -0700
Subject: [elephant-devel] SBCL errors
In-Reply-To: <CADow0CqVPJL=wmxW71NB7LQbSsw6zpj4F_BBrEXyL-8WexdGpA@mail.gmail.com>
References: <4E323FA6.2020000@chatsubo.net>
	<CADow0CqMmvgTtk=_0Lk7PtFsry-PF0-khaXCSWevF-vVQnGokQ@mail.gmail.com>
	<4E533290.9050008@chatsubo.net>
	<CADow0CqVPJL=wmxW71NB7LQbSsw6zpj4F_BBrEXyL-8WexdGpA@mail.gmail.com>
Message-ID: <4E53BC1E.4030702@chatsubo.net>

Nikodemus, you can do the following to test the BDB backend, assuming 
that you have the BDB libs installed and your my-config.sexp* setup:

(asdf:operate 'asdf:load-op :elephant)
(asdf:operate 'asdf:load-op :ele-bdb)
(asdf:operate 'asdf:load-op :elephant-tests)
(in-package "ELEPHANT-TESTS")
(setq *default-spec* *testbdb-spec*)
(do-backend-tests)

After each test, you will want to clear out the test repo by changing to 
the elephant tests directory and running a script:

cd /usr/local/lib/sbcl/site/elephant-1.0/tests
sh ./delscript.sh

Cheers,
Kevin


* my-config.sexp goes in the elephant distribution root, in my case 
/usr/local/lib/sbcl/site/elephant-1.0/my-config.sexp.  On Ubuntu 10.04, 
with libdb4.7, libdb4.7-dev and db4.7-util packages installed, 
my-config.sexp should look like:

((:compiler . :gcc)
  (:berkeley-db-version . "4.7")
  (:berkeley-db-include-dir . "/usr/include/")
  (:berkeley-db-lib-dir . "/usr/lib/")
  (:berkeley-db-lib . "/usr/lib/libdb-4.7.so")
  (:berkeley-db-deadlock . "/usr/bin/db4.7_deadlock")
  (:berkeley-db-cachesize . 20971520)
  (:berkeley-db-max-locks . 2000)
  (:berkeley-db-max-objects . 2000)
  (:berkeley-db-max-transactions . 1000)
  (:berkeley-db-map-degree2 . t)
  (:berkeley-db-mvcc . nil)
  (:warn-when-dropping-persistent-slots . nil)
  (:return-null-on-missing-instance . t)
  (:no-deserialization-package-found-action . :warn)
  (:clsql-lib-paths . nil)
  (:prebuilt-libraries . nil))



On 08/23/2011 03:58 AM, Nikodemus Siivola wrote:
> On 23 August 2011 07:54, Kevin Raison<raison at chatsubo.net>  wrote:
>
>> I just pulled sbcl-1.0.51 and tried the elephant test suite again;  I get
>> the same errors;  did the changes not make it into the newest sbcl release?
>
> They did.
>
> If someone can give me copy-pasteable-to-terminal/Slime instructions
> for running Elephants tests, I can take a second look. (It's
> non-obvious to me how to set up a store controller, and
> ASDF:TEST-SYSTEM doesn't do it for me, and FIVEAM:RUN<name-of-test>
> complains about it missing.)
>
> A snippet that reproduces the issue without depending on Elephant
> would be even better, but not required.
>
> Cheers,
>
>   -- nikodemus
>



From killerstorm at newmail.ru  Tue Aug 23 18:35:00 2011
From: killerstorm at newmail.ru (Alex Mizrahi)
Date: Tue, 23 Aug 2011 21:35:00 +0300
Subject: [elephant-devel] SBCL errors
References: <4E323FA6.2020000@chatsubo.net><CADow0CqMmvgTtk=_0Lk7PtFsry-PF0-khaXCSWevF-vVQnGokQ@mail.gmail.com><4E533290.9050008@chatsubo.net><CADow0CqVPJL=wmxW71NB7LQbSsw6zpj4F_BBrEXyL-8WexdGpA@mail.gmail.com>
	<4E53BC1E.4030702@chatsubo.net>
Message-ID: <j30rsq$8o8$1@dough.gmane.org>

 KR> Nikodemus, you can do the following to test the BDB backend, assuming 
 KR> that you have the BDB libs installed and your my-config.sexp* setup:

 KR> (asdf:operate 'asdf:load-op :elephant)
 KR> (asdf:operate 'asdf:load-op :ele-bdb)
 KR> (asdf:operate 'asdf:load-op :elephant-tests)
 KR> (in-package "ELEPHANT-TESTS")
 KR> (setq *default-spec* *testbdb-spec*)
 KR> (do-backend-tests)

Alternatively:

cd elephant-1.0/tests
sbcl --load BerkeleyDB-tests.lisp

(It is same thing, just saves typing.)