From rpgoldman at sift.info  Sun Oct 14 20:52:42 2012
From: rpgoldman at sift.info (Robert Goldman)
Date: Sun, 14 Oct 2012 15:52:42 -0500
Subject: [cl-json-devel] Problem with cl-json tests
In-Reply-To: <CANejTzo9OXZ6mrposqjv2zAHPxrXKfzq-QRWN1SjsVKLVRUN7w@mail.gmail.com>
References: <CANejTzo9OXZ6mrposqjv2zAHPxrXKfzq-QRWN1SjsVKLVRUN7w@mail.gmail.com>
Message-ID: <507B261A.9050908@sift.info>

On 10/14/12 Oct 14 -5:25 AM, Juan Jose Garcia-Ripoll wrote:
> They seem to be executed as soon as one loads them. However, the fact
> that a variable is only set _after_ the test suite is loaded, with
> 
> (defmethod perform :after ((op load-op) (comp (eql (find-system
> :cl-json.test))))
>   (eval `(setf ,(intern (symbol-name '#:*json-test-files-path*) :json-test)
>                (merge-pathnames "t/" *cl-json-directory*))))
> 
> means that a whole bunch of tests fail because late ASDF seems to only
> perform load-op on the system _after_ it has loaded all its components.
> A better, more robust way of doing this, which does not depend on ASDF's
> loading rules is adding this
> 
> (eval `(setf ,(intern (symbol-name '#:*json-test-files-path*)
>      (or (find-package "JSON-TEST")
>  (make-package "JSON-TEST")))
>     (merge-pathnames "t/" *cl-json-directory*)))
> 
> or, even better, split cl-json.asd into cl-json.asd and cl-json-test.asd
> to avoid that this side-effect happens for every user that loads cl-json.
> 
> Juanjo
> 
> -- 
> Instituto de F?sica Fundamental, CSIC
> c/ Serrano, 113b, Madrid 28006 (Spain) 
> http://juanjose.garciaripoll.googlepages.com

Attached is a darcs patch file that provides a fix for this.  Instead of
doing this in the ASDF system definition file, I moved this into the
defvar (now DEFPARAMETER) for *JSON-TEST-FILES-PATH*.  Bumped the patch
number of the system.

I don't have permissions to apply this patch, though.

BTW, I find that two of the JSON-NUMBER tests fail on my copy of ECL (on
Mac OS X).  Do you by any chance have a suggested fix?

Cheers,
r

-------------- next part --------------
1 patch for repository http://common-lisp.net/project/cl-json/darcs/cl-json:

Sun Oct 14 15:38:47 CDT 2012  rpgoldman at sift.info
  * fix-test-after-method-bug-from-juanjo
  
  juanjo (Juan Jose Garcia-Ripoli) found a bug in the binding of the 
  *JSON-TEST-FILES-PATH* global.  Fixed by moving the initializer into
  the test code, and using the ASDF:SYSTEM-RELATIVE-PATHNAME function.
  
  At the same time, killed the testjson.lisp file, which does the testing
  as a side-effect of loading.  This is unnecessary because the testing
  is done in the PERFORM method of the TEST-OP.
  


New patches:

[fix-test-after-method-bug-from-juanjo
rpgoldman at sift.info**20121014203847
 Ignore-this: 12cec5052070f37a2f72e4e5b0587c32
 
 juanjo (Juan Jose Garcia-Ripoli) found a bug in the binding of the 
 *JSON-TEST-FILES-PATH* global.  Fixed by moving the initializer into
 the test code, and using the ASDF:SYSTEM-RELATIVE-PATHNAME function.
 
 At the same time, killed the testjson.lisp file, which does the testing
 as a side-effect of loading.  This is unnecessary because the testing
 is done in the PERFORM method of the TEST-OP.
 
] {
hunk ./cl-json.asd 23
 (defsystem :cl-json
   :name "cl-json"
   :description "JSON in Lisp. JSON (JavaScript Object Notation) is a lightweight data-interchange format."
-  :version "0.4.1"
+  :version "0.4.2"
   :maintainer "Henrik Hjelte <henrik at evahjelte.com>"
   :licence "MIT"
   :in-order-to ((test-op (test-op "cl-json.test")))
hunk ./cl-json.asd 45
   :in-order-to ((test-op (load-op "cl-json.test")))
   :components ((:module :t
                :components ((:file "package")
-                            (:file "testjson" :depends-on ("package" "testdecoder" "testencoder" "testmisc"))
+                            ;; (:file "testjson" :depends-on ("package" "testdecoder" "testencoder" "testmisc"))
                             (:file "testmisc" :depends-on ("package" "testdecoder" "testencoder"))
                             (:file "testdecoder" :depends-on ("package"))
                             (:file "testencoder" :depends-on ("package"))))))
hunk ./cl-json.asd 54
   (funcall (intern (symbol-name '#:run!) :it.bese.FiveAM)
            (intern (symbol-name '#:json) :json-test)))
 
-(defparameter *cl-json-directory* (make-pathname :directory (pathname-directory *load-truename*)))
+(defparameter *cl-json-directory*
+  (make-pathname :directory (pathname-directory *load-truename*)))
+
 
hunk ./cl-json.asd 58
-(defmethod perform :after ((op load-op) (comp (eql (find-system :cl-json.test))))
-  (eval `(setf ,(intern (symbol-name '#:*json-test-files-path*) :json-test)
-               (merge-pathnames "t/" *cl-json-directory*))))
 
 
 
hunk ./t/testdecoder.lisp 269
          (* 2.0 (expt 10.0 444)))))
 
 
-(defvar *json-test-files-path*)
+(defparameter *json-test-files-path*
+  (asdf:system-relative-pathname "cl-json.test" "t/"))
 
 (defun test-file (name)
   (make-pathname :name name :type "json" :defaults *json-test-files-path*))
}

Context:

[Update html page
henrik at evahjelte.com**20110613194142
 Ignore-this: 9c948326a48610d42bf3bce9e6ab9ed3
] 
[TAG 0.4.1
henrik at evahjelte.com**20110613193019
 Ignore-this: 924d35e78a9ff89c8c3c37272cddf8ea
] 
[Added test case to check for exponent markers in float encodings.
rpgoldman at sift.info**20110625151708] 
[Added conditional compilations to kill Clozure etypecase warnings.
rpgoldman at sift.info**20110625151607] 
[TAG before-json-rpc-2
Henrik Hjelte <henrik at evahjelte.com>**20100117215948
 Ignore-this: 6ec3790b4b95f6812d4b8e24946d1ab9
] 
[Fixed typecase in ENCODE-JSON-NUMBER.
rpgoldman at sift.info**20110625030416] 
[bump-to-0.4.1
rpgoldman at sift.info**20110625000531] 
[Refactoring of the rational test in ENCODE-JSON-NUMBER, per Boris Smilga.
rpgoldman at sift.info**20110624022244] 
[Add check for rationals to ENCODE-JSON-NUMBER, per Boris Smilga.
rpgoldman at sift.info**20110623144708] 
[Tweak to Boris Smilga's ENCODE-JSON-NUMBER fix.  Sort float subtypes from smallest to largest.
rpgoldman at sift.info**20110623010610] 
[Better fix to ENCODE-JSON-NUMBER from Boris Smilga.
rpgoldman at sift.info**20110623005733] 
[Fix to ENCODE-JSON-NUMBER to remove D characters that break the JSON readers.
rpgoldman at sift.info**20110622211339] 
[Killed some trailing whitespace.
rpgoldman at sift.info**20100115234144] 
[add-json-rpc-2.0-support
rpgoldman at sift.info**20100114024858
 
 A large number of changes that provide json-rpc-2.0 support to 
 cl-json.  Tests are included.
] 
[record-function-names-not-functions
rpgoldman at sift.info**20100114024541
 
   Originally, when one defined a function to be exported as a json-rpc
   function, the function object was cached in a hash table.  This made it
   very difficult to debug with the CL tracer, since the trace-encapsulated
   function wouldn't make it onto the function table.  However, this problem
   goes away with this patch, which caches the function /name/ (a symbol)
   instead of the function itself.  Since json-rpc functions are invoked
   with APPLY, nothing else need change.  Slightly less efficient, but
   JSON-RPC is never going to be that efficient anyway.
 
] 
[export-json-rpc-version-constants
rpgoldman at sift.info**20100113210938] 
[reorganized-json-rpc-encoding
rpgoldman at sift.info**20100110051821
 
 Reorganized JSON-RPC result encoding, using CLOS, to make it more flexible,
 along lines discussed on the mailing list.
 
 Also added /partial/ support for JSON-RPC 2.0.
 
 This support is not yet tested in the test suite.
] 
[fixed-allegro-reader-error
rpgoldman at sift.info**20100110051717] 
[added a TODO file with some ideas
Henrik Hjelte <henrik at evahjelte.com>**20100106132910
 Ignore-this: dc7bfcff6aa5dd8c7ecdd140069ff361
] 
[added comment in json-rpc how to make a service-method-decription file
Henrik Hjelte <henrik at evahjelte.com>**20100106132014
 Ignore-this: 47abf3061d43c8088614a40744ecd3b3
] 
[Added Robert Goldman to CONTRIBUTORS
Henrik Hjelte <henrik at evahjelte.com>**20100106113826
 Ignore-this: ad0a7b48de15b60315c519ea271ef6c5
] 
[trap-acl-overflow-on-read
rpgoldman at sift.info**20100102230711
 
 Allegro just raises a simple-error when it reads a floating point
 number that's too big.  So we have to catch simple errors when
 in parse-number on allegro.
] 
[catch-read-numeric-overflow-on-allegro
rpgoldman at sift.info**20100105201756
 
 On Allegro Common Lisp, when there's a numeric overflow in the reader,
 they don't signal a reader-error, but a simple-error (I have reported this
 and they may fix it in the future).  This patch makes use of information
 about that simple-error to catch the case in parse-number and treat it 
 the way parse-number overflows are handled on other lisps.
] 
[removed-allegro-from-some-tests
rpgoldman at sift.info**20100102221106
 
 Some tests don't work on allegro on macosx or on allegro simpliciter.
 These seem like limitations of allegro, so I have added features flags
 to mask them.
] 
[revise json file loading for tests
rpgoldman at sift.info**20100102214917
 
 With asdf-binary-locations on, as it will be in new versions of ASDF, 
 it is no longer possible to use load-pathname or load-truename to find
 files from loaded binaries.  In the general case, binaries may not be 
 located in the same directory as the source files from which they are 
 compiled.
 
 We avoid this problem by caching the value of *load-truename* inside
 code that is run in the .asd file, and then referring to it later on.
 
 The *load-truename* of the .asd file will remain constant.
] 
[add test op to cl-json
rpgoldman at sift.info**20100102211745
 
 Made it easier to test cl-json by adding an asdf test-op to the asdf
 system definition, and providing a PERFORM method for it.
 
 Now one can simply say (asdf:oos 'asdf:test-op :cl-json) to load
 the system and run the FiveAM based tests.
] 
[json-rpc-handle-notifications
rpgoldman at sift.info**20091230211756
 
 A couple of modifications to hande JSON-RPC notifications (like normal requests, but they
 don't expect a response).  
 
 Also made it possible to invoke a helper function of invoke-json with the pieces of the
 request.  This is necessary because you can't use the standard invoke-rpc if you've 
 already parsed the JSON message from the client.
 
] 
[Doc add info about the known bug json-bind-in-bind
Henrik Hjelte <henrik at evahjelte.com>**20091202132732
 Ignore-this: b92b3c9703303475dcbce97aff3dee09
] 
[comment in doc about new encoder
Henrik Hjelte <henrik at evahjelte.com>**20091202125416
 Ignore-this: f7692a5e60ad7d4ea71264847566341e
] 
[Feature to turn off clos
Henrik Hjelte**20091012232210
 Ignore-this: 5d9ff167ffa4ed5b91b0e934362def3e
] 
[testcase for a bug with nested json-bind
Henrik Hjelte <henrik at evahjelte.com>**20091009190902
 Ignore-this: 4905daaa94ef2e2604df5de6bb092c64
] 
[explict decoder can handle null values as well
Henrik Hjelte <henrik at evahjelte.com>**20091009190817
 Ignore-this: 9beb1705b45b3692776de2f3731112b7
] 
[Now you can select encoder for json-rpc, streaming guessing or explicit
Henrik Hjelte <henrik at evahjelte.com>**20091009145907
 Ignore-this: 360302b256b196d7186d7b1bbc6384d2
] 
[new encoder: the explicit encoder, the old one now named guessing-encoder.
Henrik Hjelte <henrik at evahjelte.com>**20091009032101
 Ignore-this: 555b36f352707e7baec21f2638f60d0b
 With testcases.
] 
[Don't use with-substitute-printed-representation-restart inside encode-json
Henrik Hjelte <henrik at evahjelte.com>**20091009021546
 Ignore-this: 50a860841d16fc5333f17b6392c006f6
 beacuse it establishes a new restart for every recursion. My top count
 was 8000+ restarts. Instead, export it and let the user use it outside any
 encode-json calls if he or she wants it.
] 
[allow embedded nil in encode-json-alist
Henrik Hjelte <henrik at evahjelte.com>**20091009015850
 Ignore-this: 1e809eead41e2f6bc8d1e3e490c80cfe
] 
[kill-yank decoder-performance-with simplifed-camel-case
Henrik Hjelte <henrik at evahjelte.com>**20090810191200
 Ignore-this: bbaab17c2f5c0d09fe96cde11e2b9e1b
] 
[docs added info about new things in version history
Henrik Hjelte <henrik at evahjelte.com>**20090810185953
 Ignore-this: 7979e4788b9459bcfe03dd506e4a8f6a
] 
[docs added info about simplified-camel-case-to-lisp
Henrik Hjelte <henrik at evahjelte.com>**20090810185705
 Ignore-this: b36c062ffe85417b49022e6965862d46
] 
[docs comment about safe-json-intern in Security considerations section
Henrik Hjelte <henrik at evahjelte.com>**20090810185540
 Ignore-this: 4b55104417e786b43eb5fc0146ad6636
] 
[added Leslie. P Polzer and Red Daly to contributors
Henrik Hjelte <henrik at evahjelte.com>**20090810175728
 Ignore-this: 8cf5258eb91b9f6e95f8636616255889
] 
[simplified-camel-case by Leslie P. Polzer
Henrik Hjelte <henrik at evahjelte.com>**20090810175353
 Ignore-this: 8d2d1d112d69b0abfe2194010da5cc35
] 
[safe-json-intern a version that does not intern.  
Henrik Hjelte <henrik at evahjelte.com>**20090810134132
 Ignore-this: 44ab030099e2258d7acb37b647911fe4
 The default json-intern is not safe (spotted by Red Daly). Interns of
 many unique symbols could potentially use a lot of memory.  An attack
 could exploit this by submitting something that is passed through
 cl-json that has many very large, unique symbols. This version is safe
 in that respect because it only allows symbols that already exists.
] 
[From Red Daly, *identifier-name-to-key*
Henrik Hjelte <henrik at evahjelte.com>**20090810130334
 Ignore-this: a4a08ed5960fb2417c584d6a86277291
] 
[Removed unused li from html doc
Henrik Hjelte <henrik at evahjelte.com>**20090525202150
 Ignore-this: cb45eef6e051c10793826ebadeb50544
] 
[TAG 0.4.0
henrik at evahjelte.com**20090525190005
 Ignore-this: cb4f0fc56b8bfd3d1e46dd9257ae4e3a
] 
Patch bundle hash:
60119b5472f4095ede5f9ea429af489cc7f76692

From henrik at evahjelte.com  Mon Oct 15 16:10:58 2012
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Mon, 15 Oct 2012 18:10:58 +0200
Subject: [cl-json-devel] Problem with cl-json tests
In-Reply-To: <507B261A.9050908@sift.info>
References: <CANejTzo9OXZ6mrposqjv2zAHPxrXKfzq-QRWN1SjsVKLVRUN7w@mail.gmail.com>
	<507B261A.9050908@sift.info>
Message-ID: <CACfvCrj0vd5peOSF2HEDU=xZ+rFUXpp4zZibtBhvNx9CsEEoyA@mail.gmail.com>

On Sun, Oct 14, 2012 at 10:52 PM, Robert Goldman <rpgoldman at sift.info>wrote:

> Attached is a darcs patch file that provides a fix for this.  Instead of
> doing this in the ASDF system definition file, I moved this into the
> defvar (now DEFPARAMETER) for *JSON-TEST-FILES-PATH*.  Bumped the patch
> number of the system.
>
> I don't have permissions to apply this patch, though.
>

 I've now applied it. Thanks to both of you!
Cheers, Henrik
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/cl-json-devel/attachments/20121015/18269c83/attachment.html>

From sven.emtell at doremir.com  Wed Oct 31 17:07:27 2012
From: sven.emtell at doremir.com (Sven Emtell)
Date: Wed, 31 Oct 2012 18:07:27 +0100
Subject: [cl-json-devel] cl-json depends on lisp reader?
Message-ID: <8F79A86E-279B-4FF9-9F7E-C159E73FA0B4@doremir.com>

Hi everyone!

I use cl-json with LispWorks 6.1.
When delivering an app, I am trying to minimize the size of it by setting delivery level 5.
The cl-json function "decode-json-from-string" does not seem to like this without explicitly keeping the Lisp Reader using the delivery keyword (:keep-lisp-reader).
Since the Lisp Reader is unnecessary for the rest of the app, I would like to leave it out when delivering.
Any thoughts on this?

PS. I will also send a similar message to the LispWorks list, if that is more appropriate.

Thanks!
Sven Emtell


From boris.smilga at gmail.com  Wed Oct 31 22:26:18 2012
From: boris.smilga at gmail.com (Boris Smilga)
Date: Thu, 1 Nov 2012 02:26:18 +0400
Subject: [cl-json-devel] cl-json depends on lisp reader?
In-Reply-To: <8F79A86E-279B-4FF9-9F7E-C159E73FA0B4@doremir.com>
References: <8F79A86E-279B-4FF9-9F7E-C159E73FA0B4@doremir.com>
Message-ID: <38DA2487-A5C4-4C29-AA9D-6CDC9F474811@gmail.com>

On Wed, Oct 31, 2012 at 5:07 PM, Sven Emtell  
<sven.emtell at doremir.com> wrote:
>
> I use cl-json with LispWorks 6.1.
> When delivering an app, I am trying to minimize the size of it by  
> setting delivery level 5.
> The cl-json function "decode-json-from-string" does not seem to  
> like this without explicitly keeping the Lisp Reader using the  
> delivery keyword (:keep-lisp-reader).
> Since the Lisp Reader is unnecessary for the rest of the app, I  
> would like to leave it out when delivering.
> Any thoughts on this?

The Lisp reader is used on just one occasion, namely, in the function  
PARSE-NUMBER in src/decoder.lisp (line 417ff., four uses of READ-FROM- 
STRING).  You can easily ditch it by supplying another parser for  
numbers.  For this end, you have three options: A) wait for someone  
to develop a replacement?without guarantees; B) write a replacement  
yourself, send a patch to the maintainers, and wait for it to be  
pushed to the repository; C) customize DECODE-JSON in your own code,  
by setting or binding the handlers for number token.  If you choose  
option C, I'd suggest you read the relevant chapter in the User Guide  
(http://common-lisp.net/project/cl-json/#DECODER-CUSTOMIZATION).  The  
handlers in question are represented by the variables *INTEGER- 
HANDLER* and *REAL-HANDLER*.

Sincerely,
  ? B. Smilga.




From sven.emtell at doremir.com  Wed Oct 31 23:08:16 2012
From: sven.emtell at doremir.com (Sven Emtell)
Date: Thu, 1 Nov 2012 00:08:16 +0100
Subject: [cl-json-devel] cl-json depends on lisp reader?
In-Reply-To: <38DA2487-A5C4-4C29-AA9D-6CDC9F474811@gmail.com>
References: <8F79A86E-279B-4FF9-9F7E-C159E73FA0B4@doremir.com>
	<38DA2487-A5C4-4C29-AA9D-6CDC9F474811@gmail.com>
Message-ID: <C5CFC7FB-546B-4E34-8881-AFC4710D61F4@doremir.com>

Thanks Boris!
I'll see what to do.
Best regards,
Sven

31 okt 2012 kl. 23:26 skrev Boris Smilga:

> On Wed, Oct 31, 2012 at 5:07 PM, Sven Emtell <sven.emtell at doremir.com> wrote:
>> 
>> I use cl-json with LispWorks 6.1.
>> When delivering an app, I am trying to minimize the size of it by setting delivery level 5.
>> The cl-json function "decode-json-from-string" does not seem to like this without explicitly keeping the Lisp Reader using the delivery keyword (:keep-lisp-reader).
>> Since the Lisp Reader is unnecessary for the rest of the app, I would like to leave it out when delivering.
>> Any thoughts on this?
> 
> The Lisp reader is used on just one occasion, namely, in the function PARSE-NUMBER in src/decoder.lisp (line 417ff., four uses of READ-FROM-STRING).  You can easily ditch it by supplying another parser for numbers.  For this end, you have three options: A) wait for someone to develop a replacement?without guarantees; B) write a replacement yourself, send a patch to the maintainers, and wait for it to be pushed to the repository; C) customize DECODE-JSON in your own code, by setting or binding the handlers for number token.  If you choose option C, I'd suggest you read the relevant chapter in the User Guide (http://common-lisp.net/project/cl-json/#DECODER-CUSTOMIZATION).  The handlers in question are represented by the variables *INTEGER-HANDLER* and *REAL-HANDLER*.
> 
> Sincerely,
> ? B. Smilga.
>