[cl-store-devel] store size of float in cl-store

[szergling]

On 1/16/10, kishor saitwal <kishor_saitwal at yahoo.com> wrote:
> Hi there,
>
> I was trying to figure out why the size of the stored file for any
> float matrix is many times more than what it should be and hence
> started looking into the actual package.
>
> I realized that the integers take 8 bytes per integer as expected.
> However, due to the splitting of the float into significand, exponent,
> sign, etc. and saving those parts separately, one float value takes 51
> bytes of memory to save. Is there any way to assign 8 bytes per float
> in this package so that the size of the saved Matlisp matrices will not
> be more than expected for such real values?
>
> Within our save and restore mechanism, we need to save matrices that
> would easily contain more than 1e6 floating point numbers and the
> current mechanism makes the size of the saved sets large enough to be
> concerned about the overall save and restore mechanism. We would
> certainly like to find a way to save these big matrices in such a way
> that the size is within 8 bytes per float.
>
> Is there any way we can achieve this within the package without making
> any changes? If not, can anybody recommend changes to the package in
> order to achieve it?
>
> In general, I thought that it may be a good resource optimization if
> the original package would have an option to assign maximum 8 bytes to
> each float, if the user chooses to.
>
> Any suggestions would be appreciated.
>


Hi Kishor,

Two ideas:

Perhaps you can use a wrapper around your arrays to dispatch
to your own custom code, as shown in temp.lisp (warning, very
very casually tested only). That one only stores 32 bit, so maybe
you'll need to write a store-64-bit version (or call store-32-bit twice).

I wonder if (tidied up, debugged, unit tested with test cases, and
polished) this could go into cl-store itself, or as a contrib... I am
sorry that code is so rough (not even sure if it'll run on your end...),
but I've simply banged that up in two hours or so.

Alternatively, idea 2, one could investigate online compression of
the streams? Surely, this is an even more general alternative?
I haven't looked into this unfortunately...

Theam Yong Chew.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: temp.lisp
Type: application/octet-stream
Size: 4914 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/cl-store-devel/attachments/20100117/fb3a8ccb/attachment.obj>