From kishor_saitwal at yahoo.com Fri Jan 15 20:08:23 2010 From: kishor_saitwal at yahoo.com (kishor saitwal) Date: Fri, 15 Jan 2010 12:08:23 -0800 (PST) Subject: [cl-store-devel] store size of float in cl-store Message-ID: <242270.39209.qm@web53108.mail.re2.yahoo.com> Hi there, I was trying to figure out why the size of the stored file for any float matrix is many times more than what it should be and hence started looking into the actual package. I realized that the integers take 8 bytes per integer as expected. However, due to the splitting of the float into significand, exponent, sign, etc. and saving those parts separately, one float value takes 51 bytes of memory to save. Is there any way to assign 8 bytes per float in this package so that the size of the saved Matlisp matrices will not be more than expected for such real values? Within our save and restore mechanism, we need to save matrices that would easily contain more than 1e6 floating point numbers and the current mechanism makes the size of the saved sets large enough to be concerned about the overall save and restore mechanism. We would certainly like to find a way to save these big matrices in such a way that the size is within 8 bytes per float. Is there any way we can achieve this within the package without making any changes? If not, can anybody recommend changes to the package in order to achieve it? In general, I thought that it may be a good resource optimization if the original package would have an option to assign maximum 8 bytes to each float, if the user chooses to. Any suggestions would be appreciated. Thanks, Kishor -- Kishor Saitwal, PhD Computer Vision Scientist Behavioral Recognition Systems, Inc. 2100 West Loop South, 9th Floor Houston, TX 77027 Voice: (713) 590-5160 -------------- next part -------------- An HTML attachment was scrubbed... URL: From senatorzergling at gmail.com Sun Jan 17 03:08:27 2010 From: senatorzergling at gmail.com (szergling) Date: Sun, 17 Jan 2010 16:08:27 +1300 Subject: [cl-store-devel] store size of float in cl-store In-Reply-To: <242270.39209.qm@web53108.mail.re2.yahoo.com> References: <242270.39209.qm@web53108.mail.re2.yahoo.com> Message-ID: <8a4b88391001161908m3af8b8b5q13fcdbb601fb90b5@mail.gmail.com> On 1/16/10, kishor saitwal wrote: > Hi there, > > I was trying to figure out why the size of the stored file for any > float matrix is many times more than what it should be and hence > started looking into the actual package. > > I realized that the integers take 8 bytes per integer as expected. > However, due to the splitting of the float into significand, exponent, > sign, etc. and saving those parts separately, one float value takes 51 > bytes of memory to save. Is there any way to assign 8 bytes per float > in this package so that the size of the saved Matlisp matrices will not > be more than expected for such real values? > > Within our save and restore mechanism, we need to save matrices that > would easily contain more than 1e6 floating point numbers and the > current mechanism makes the size of the saved sets large enough to be > concerned about the overall save and restore mechanism. We would > certainly like to find a way to save these big matrices in such a way > that the size is within 8 bytes per float. > > Is there any way we can achieve this within the package without making > any changes? If not, can anybody recommend changes to the package in > order to achieve it? > > In general, I thought that it may be a good resource optimization if > the original package would have an option to assign maximum 8 bytes to > each float, if the user chooses to. > > Any suggestions would be appreciated. > Hi Kishor, Two ideas: Perhaps you can use a wrapper around your arrays to dispatch to your own custom code, as shown in temp.lisp (warning, very very casually tested only). That one only stores 32 bit, so maybe you'll need to write a store-64-bit version (or call store-32-bit twice). I wonder if (tidied up, debugged, unit tested with test cases, and polished) this could go into cl-store itself, or as a contrib... I am sorry that code is so rough (not even sure if it'll run on your end...), but I've simply banged that up in two hours or so. Alternatively, idea 2, one could investigate online compression of the streams? Surely, this is an even more general alternative? I haven't looked into this unfortunately... Theam Yong Chew. -------------- next part -------------- A non-text attachment was scrubbed... Name: temp.lisp Type: application/octet-stream Size: 4914 bytes Desc: not available URL: From zellerin at gmail.com Wed Jan 20 13:18:25 2010 From: zellerin at gmail.com (Tomas Zellerin) Date: Wed, 20 Jan 2010 14:18:25 +0100 Subject: [cl-store-devel] new clisps and make-closure, hash-table-test In-Reply-To: References: Message-ID: Hello, I started to try to use cl-store and encountered two small issues (both related to clisp) so far. I can live with it or work around, so take this just as in indication of possible problem areas that might or might not bother you. 1) new clisp (tested on 2.48 on Win, 2.46 on Linux) has no %make-closure, (but has make-closure - not sure if same). I do not store closures/functions so far, but the fact that the package sys is locked leads to continuable errors during asdf load. 2) hash tables stored in clisp cannot be restored back to other lisps due to the fact that (hash-table-test (make-hash-table :test 'eq)) returns ext:fasthash-eql there. I am not even sure that the cl-store package (or rather current backend) is intended to create cross-implementation data, but the other way (sbcl to clisp) worked just fine, which raised the expectations. Otherwise, it is nice and easy to use - good job. With regards, Tomas From demmeln at in.tum.de Fri Jan 29 16:22:44 2010 From: demmeln at in.tum.de (Nikolaus Demmel) Date: Fri, 29 Jan 2010 17:22:44 +0100 Subject: [cl-store-devel] Redefine Backend Bug (+ fix) Message-ID: <582CB890-F3DF-4876-9D2B-893B689E0432@in.tum.de> Hi, in REGISTER-BACKEND the "Redefine backend" restart doesn't work as advertised, since it does not register the new backend at all. Attached is a proposed patch to fix it. It would be great if someone could check the patch and maybe apply it. Best regards, Niko -------------- next part -------------- A non-text attachment was scrubbed... Name: redefine_backend_bugfix.patch Type: application/octet-stream Size: 2437 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4623 bytes Desc: not available URL: From rosssd at gmail.com Fri Jan 29 16:54:36 2010 From: rosssd at gmail.com (Sean Ross) Date: Fri, 29 Jan 2010 16:54:36 +0000 Subject: [cl-store-devel] Redefine Backend Bug (+ fix) In-Reply-To: <582CB890-F3DF-4876-9D2B-893B689E0432@in.tum.de> References: <582CB890-F3DF-4876-9D2B-893B689E0432@in.tum.de> Message-ID: <61460A2F-3958-44B8-881D-247AF1285C49@gmail.com> Thanks for the patch, I've applied it to the darcs repo. Sean. On 29 Jan 2010, at 16:22, Nikolaus Demmel wrote: > Hi, > > in REGISTER-BACKEND the "Redefine backend" restart doesn't work as advertised, since it does not register the new backend at all. Attached is a proposed patch to fix it. > > It would be great if someone could check the patch and maybe apply it. > > Best regards, > Niko > > _______________________________________________ > cl-store-devel mailing list > cl-store-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/cl-store-devel From demmeln at in.tum.de Fri Jan 29 17:37:00 2010 From: demmeln at in.tum.de (Nikolaus Demmel) Date: Fri, 29 Jan 2010 18:37:00 +0100 Subject: [cl-store-devel] Redefine Backend Bug (+ fix) In-Reply-To: <61460A2F-3958-44B8-881D-247AF1285C49@gmail.com> References: <582CB890-F3DF-4876-9D2B-893B689E0432@in.tum.de> <61460A2F-3958-44B8-881D-247AF1285C49@gmail.com> Message-ID: Wow, that was quick! Thanks, Niko Am 29.01.2010 um 17:54 schrieb Sean Ross: > Thanks for the patch, > > I've applied it to the darcs repo. > > Sean. > > > On 29 Jan 2010, at 16:22, Nikolaus Demmel wrote: > >> Hi, >> >> in REGISTER-BACKEND the "Redefine backend" restart doesn't work as advertised, since it does not register the new backend at all. Attached is a proposed patch to fix it. >> >> It would be great if someone could check the patch and maybe apply it. >> >> Best regards, >> Niko >> >> _______________________________________________ >> cl-store-devel mailing list >> cl-store-devel at common-lisp.net >> http://common-lisp.net/cgi-bin/mailman/listinfo/cl-store-devel > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4623 bytes Desc: not available URL: