From mega at hotpop.com  Thu May 12 14:17:40 2005
From: mega at hotpop.com (=?iso-8859-1?q?G=E1bor_Melis?=)
Date: Thu, 12 May 2005 16:17:40 +0200
Subject: [elephant-devel] berkeley db bulk get
Message-ID: <200505121617.40682.mega@hotpop.com>

Do you guys think that the bulk get feature is worth supporting?

http://www.sleepycat.com/docs/ref/am_misc/get_bulk.html

I ran into a number of situations where the db or a huge btree must be scanned 
(gc, creating a site map with a link to user pages, joins) and it takes ages. 
Current thinking is that adding a buffer to the cursor itself may perform 
well:

(with-transaction (:degree-2 t)
  (with-btree-cursor (cursor my-btree :multiple t
                                      :multiple-buffer-size 1000000)
    (loop for (found? key user) = (multiple-value-list
                                    (cursor-next cursor))
          while found?
          collect user)))

The berkeley db doc says multiple get will not work on secondary indices so 
joins will not benefit :-(.

Any thoughts? Is there another way to speed up iteration? Maybe raise the page 
size?

Gabor


From ben at medianstrip.net  Tue May 24 20:42:11 2005
From: ben at medianstrip.net (Ben)
Date: Tue, 24 May 2005 16:42:11 -0400 (EDT)
Subject: [elephant-devel] berkeley db bulk get
In-Reply-To: <200505121617.40682.mega@hotpop.com>
References: <200505121617.40682.mega@hotpop.com>
Message-ID: <20050524164003.X22242@contarex.medianstrip.net>

there is preliminary support for bulk get in the C library.  i don't
think i've written the FFI bindings (or elephant interface) yet
though.

andrew and i are both occupied with other things as one can tell by
our radio silence.  i'm sorry about that.  i know there have been some
promises made about new releases, we hope to get to them.

B

On Thu, 12 May 2005, [iso-8859-1] G?bor Melis wrote:

> Do you guys think that the bulk get feature is worth supporting?
>
> http://www.sleepycat.com/docs/ref/am_misc/get_bulk.html
>
> I ran into a number of situations where the db or a huge btree must be scanned
> (gc, creating a site map with a link to user pages, joins) and it takes ages.
> Current thinking is that adding a buffer to the cursor itself may perform
> well:
>
> (with-transaction (:degree-2 t)
>  (with-btree-cursor (cursor my-btree :multiple t
>                                      :multiple-buffer-size 1000000)
>    (loop for (found? key user) = (multiple-value-list
>                                    (cursor-next cursor))
>          while found?
>          collect user)))
>
> The berkeley db doc says multiple get will not work on secondary indices so
> joins will not benefit :-(.
>
> Any thoughts? Is there another way to speed up iteration? Maybe raise the page
> size?
>
> Gabor
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>