From tritchey at mac.com Thu May 14 04:17:54 2009 From: tritchey at mac.com (Timothy Ritchey) Date: Thu, 14 May 2009 00:17:54 -0400 Subject: [mel-base-devel] multipart messages and pop3 In-Reply-To: References: Message-ID: I've been able to pretty easily move files from a pop folder to a local maildir: (mel:move-folder folder tmp-folder) (mel:map-recent-messages (lambda (m) (format t "~a~%" (find-plain-text m))) tmp-folder) The map-recent-messages gets through all of the messages in new, and they are printed out okay, but then brings up the error below. As best I can tell, it is trying to move the messages to the cur directory. I am running a recent sbcl on OS X. Running map-messages gets me the same error, but with a slightly different backtrace. Does anyone have any ideas about what might be going on? Cheers, Tim R. The value "1242273934.8901_1.localhost" is not of type (SIMPLE-ARRAY BASE-CHAR (*)). [Condition of type TYPE-ERROR] Restarts: 0: [RETRY] Retry SLIME REPL evaluation request. 1: [ABORT] Return to SLIME's top level. 2: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: ((SB-PCL::FAST-METHOD MEL.INTERNAL:FLAGP-USING-FOLDER (MEL.FOLDERS.MAILDIR:MAILDIR-FOLDER T (EQL :RECENT))) ..) 1: ((SB-PCL::FAST-METHOD MEL.INTERNAL:UNMARK-MESSAGE-USING-FOLDER (MEL.FOLDERS.MAILDIR:MAILDIR-FOLDER T (EQL :RECENT))) ..) 2: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. SB-PCL::.ARG1. SB-PCL::.ARG2.)) ..) 3: (SB-IMPL::%MAP-FOR-EFFECT-ARITY-1 # (#)) 4: ((SB-PCL::FAST-METHOD MEL.PUBLIC:MAP-RECENT-MESSAGES :AROUND (T MEL.PUBLIC:BASIC-FOLDER)) ..) 5: (SB-INT:SIMPLE-EVAL-IN-LEXENV (CHECK-EMAIL) #) On Apr 16, 2009, at 9:52 AM, Jochen Schmidt wrote: > Am 16.04.2009 um 14:28 schrieb Timothy Ritchey : > >> I'm having an issue extracting the parts of messages, particularly >> when trying to work with pop3 folders. I am wondering if I am doing >> something wrong. I am trying to extract any text/plain, or text/plain >> alternative from emails. The following works when I pass it an imaps >> folder, but mel:parts hangs when I pass it a pop3 folder. When I >> break >> it, it is hanging out in compute-bodystructure-using-folder, waiting >> on a stream. >> >> Am I grabbing messages the right way? > > It could well be a bug - I've not tested it, but actually it may be > a better idea to fetch the messages from the POP3 server into a > local maildir folder and then use that for further processing. POP3 > is quite braindead and doesn't really lean to well accessing > messages randomly. Due to the lazy caching nature of mel-base there > would probably be multiple fetches of a single mail in POP3. > > What you try to do seems to be what mel-base does in FIND-VIEWABLE- > PART (see multiparts.lisp). > > ciao > Jochen > >> >> (defun find-plain-text (message) >> (let ((parts (mel:parts message)) >> (body "Unable to process email contents")) >> (dolist (p parts) >> (multiple-value-bind (a b) (mel:content-type p) >> (if (and (equal a :TEXT) (equal b :PLAIN)) >> (let ((b (mel:part-body-string p))) >> (handler-case >> (setf body (mel:decode-quoted-printable b)) >> (t () >> (setf body b)))) >> (progn >> (when (and (equal a :MULTIPART) (equal b :ALTERNATIVE)) >> (find-plain-text p)))))) >> body)) >> >> (defun check-email (folder) >> (let* ((messages (mel:messages folder))) >> (dolist (message messages) >> (format t "~a~%" (find-plain-text message))))) >> >> I can call the following on the same message from the pop3 folder, >> and >> get the raw contents fine. >> >> (defmethod body (message) >> (with-output-to-string (str) >> (with-open-stream (stream (mel:message-body-stream message)) >> (loop for c = (read-char stream nil nil) >> while c do (write-char c str))))) >> >> Any help would be appreciated. >> >> - T >> >> >> _______________________________________________ >> mel-base-devel mailing list >> mel-base-devel at common-lisp.net >> http://common-lisp.net/cgi-bin/mailman/listinfo/mel-base-devel > > -- > Jochen Schmidt > CRISPYLOGICS > Uhlandstr. 9, 90408 N?rnberg > > Fon: +49 (0)911 517 999-82 > Fax: +49 (0)911 517 999-83 > > mailto:info at crispylogics.com > http://www.crispylogics.com From tritchey at mac.com Thu May 14 14:38:51 2009 From: tritchey at mac.com (Timothy Ritchey) Date: Thu, 14 May 2009 10:38:51 -0400 Subject: [mel-base-devel] multipart messages and pop3 In-Reply-To: References: Message-ID: As an update, if I use map-messages on the maildir folder, I get the backtrace below. In map-messages there is a (declare (type simple-base- string file)) in the lambda. Why it is not happy with the filename, I am not sure: (mel.filesystem:map-directory (lambda (file) (declare (type simple-base-string file)) (let ((uid (uidify file))) (setf (gethash uid uid-cache) (cons :cur file)) (funcall fn (push-message file)))) (namestring (truename (current-mail folder)))))) map-recent-messages seems to run afoul in flagp-using-folder with the inlined uidify: (defun uidify (file) (declare (optimize (speed 3) (safety 0)) (type simple-base-string file)) (let ((uid-end (position #\: file :from-end t))) (if uid-end (subseq file 0 uid-end) file))) So from what I can tell, the type declaration and the filename the system is providing are not happy. I have no idea if it could be related, but I do have: ;; possibly controversial as a global default, but shipping a lisp ;; that dies trying to talk to slime is stupid, so: (set-language-environment "UTF-8") (setq slime-net-coding-system 'utf-8-unix) at the top of my .emacs file from clbuild. Cheers, Tim R. ----- The value "1242273504.8901_1.localhost" is not of type (SIMPLE-ARRAY BASE-CHAR (*)). [Condition of type TYPE-ERROR] Restarts: 0: [RETRY] Retry SLIME REPL evaluation request. 1: [ABORT] Return to SLIME's top level. 2: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: ((LAMBDA (MEL.FOLDERS.MAILDIR::FILE)) "1242273504.8901_1.localhost") 1: (MEL.FILESYSTEM:MAP-DIRECTORY # "/ private/tmp/test-maildir/cur/") 2: ((SB-PCL::FAST-METHOD MEL.PUBLIC:MAP-MESSAGES (T MEL.FOLDERS.MAILDIR:MAILDIR-FOLDER)) ..) 3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (CHECK-EMAIL) #) On May 14, 2009, at 12:17 AM, Timothy Ritchey wrote: > I've been able to pretty easily move files from a pop folder to a > local maildir: > > (mel:move-folder folder tmp-folder) > (mel:map-recent-messages > (lambda (m) (format t "~a~%" (find-plain-text m))) tmp-folder) > > The map-recent-messages gets through all of the messages in new, and > they are printed out okay, but then brings up the error below. As > best I can tell, it is trying to move the messages to the cur > directory. I am running a recent sbcl on OS X. Running map-messages > gets me the same error, but with a slightly different backtrace. > Does anyone have any ideas about what might be going on? > > Cheers, > Tim R. > > The value "1242273934.8901_1.localhost" > is not of type > (SIMPLE-ARRAY BASE-CHAR (*)). > [Condition of type TYPE-ERROR] > > Restarts: > 0: [RETRY] Retry SLIME REPL evaluation request. > 1: [ABORT] Return to SLIME's top level. > 2: [TERMINATE-THREAD] Terminate this thread (# thread" RUNNING {133154B9}>) > > Backtrace: > 0: ((SB-PCL::FAST-METHOD MEL.INTERNAL:FLAGP-USING-FOLDER > (MEL.FOLDERS.MAILDIR:MAILDIR-FOLDER T (EQL :RECENT))) ..) > 1: ((SB-PCL::FAST-METHOD MEL.INTERNAL:UNMARK-MESSAGE-USING-FOLDER > (MEL.FOLDERS.MAILDIR:MAILDIR-FOLDER T (EQL :RECENT))) ..) > 2: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. > SB-PCL::.ARG1. SB-PCL::.ARG2.)) ..) > 3: (SB-IMPL::%MAP-FOR-EFFECT-ARITY-1 # {124D2DBD}> (#)) > 4: ((SB-PCL::FAST-METHOD MEL.PUBLIC:MAP-RECENT-MESSAGES :AROUND (T > MEL.PUBLIC:BASIC-FOLDER)) ..) > 5: (SB-INT:SIMPLE-EVAL-IN-LEXENV (CHECK-EMAIL) #) From tritchey at mac.com Thu May 14 14:49:00 2009 From: tritchey at mac.com (Timothy Ritchey) Date: Thu, 14 May 2009 10:49:00 -0400 Subject: [mel-base-devel] multipart messages and pop3 In-Reply-To: References: Message-ID: Further update: removing the type declaration in uidify and map-messages fixed the issue. The compiler complains about not being able to optimize certain calls, but I can now check the messages without any problems. Cheers, Tim R. On May 14, 2009, at 10:38 AM, Timothy Ritchey wrote: > As an update, if I use map-messages on the maildir folder, I get the > backtrace below. In map-messages there is a (declare (type simple- > base- > string file)) in the lambda. Why it is not happy with the filename, I > am not sure: > > (mel.filesystem:map-directory > (lambda (file) > (declare (type simple-base-string file)) > (let ((uid (uidify file))) > (setf (gethash uid uid-cache) (cons :cur file)) > (funcall fn (push-message file)))) > (namestring (truename (current-mail folder)))))) > > map-recent-messages seems to run afoul in flagp-using-folder with the > inlined uidify: > > (defun uidify (file) > (declare (optimize (speed 3) (safety 0)) > (type simple-base-string file)) > (let ((uid-end (position #\: file :from-end t))) > (if uid-end > (subseq file 0 uid-end) > file))) > > So from what I can tell, the type declaration and the filename the > system is providing are not happy. I have no idea if it could be > related, but I do have: > > ;; possibly controversial as a global default, but shipping a lisp > ;; that dies trying to talk to slime is stupid, so: > (set-language-environment "UTF-8") > (setq slime-net-coding-system 'utf-8-unix) > > at the top of my .emacs file from clbuild. > > Cheers, > Tim R. > > > ----- > The value "1242273504.8901_1.localhost" > is not of type > (SIMPLE-ARRAY BASE-CHAR (*)). > [Condition of type TYPE-ERROR] > > Restarts: > 0: [RETRY] Retry SLIME REPL evaluation request. > 1: [ABORT] Return to SLIME's top level. > 2: [TERMINATE-THREAD] Terminate this thread (# thread" RUNNING {133305C9}>) > > Backtrace: > 0: ((LAMBDA (MEL.FOLDERS.MAILDIR::FILE)) > "1242273504.8901_1.localhost") > 1: (MEL.FILESYSTEM:MAP-DIRECTORY # "/ > private/tmp/test-maildir/cur/") > 2: ((SB-PCL::FAST-METHOD MEL.PUBLIC:MAP-MESSAGES (T > MEL.FOLDERS.MAILDIR:MAILDIR-FOLDER)) ..) > 3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (CHECK-EMAIL) #) > > On May 14, 2009, at 12:17 AM, Timothy Ritchey wrote: > >> I've been able to pretty easily move files from a pop folder to a >> local maildir: >> >> (mel:move-folder folder tmp-folder) >> (mel:map-recent-messages >> (lambda (m) (format t "~a~%" (find-plain-text m))) tmp-folder) >> >> The map-recent-messages gets through all of the messages in new, and >> they are printed out okay, but then brings up the error below. As >> best I can tell, it is trying to move the messages to the cur >> directory. I am running a recent sbcl on OS X. Running map-messages >> gets me the same error, but with a slightly different backtrace. >> Does anyone have any ideas about what might be going on? >> >> Cheers, >> Tim R. >> >> The value "1242273934.8901_1.localhost" >> is not of type >> (SIMPLE-ARRAY BASE-CHAR (*)). >> [Condition of type TYPE-ERROR] >> >> Restarts: >> 0: [RETRY] Retry SLIME REPL evaluation request. >> 1: [ABORT] Return to SLIME's top level. >> 2: [TERMINATE-THREAD] Terminate this thread (#> thread" RUNNING {133154B9}>) >> >> Backtrace: >> 0: ((SB-PCL::FAST-METHOD MEL.INTERNAL:FLAGP-USING-FOLDER >> (MEL.FOLDERS.MAILDIR:MAILDIR-FOLDER T (EQL :RECENT))) ..) >> 1: ((SB-PCL::FAST-METHOD MEL.INTERNAL:UNMARK-MESSAGE-USING-FOLDER >> (MEL.FOLDERS.MAILDIR:MAILDIR-FOLDER T (EQL :RECENT))) ..) >> 2: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. >> SB-PCL::.ARG1. SB-PCL::.ARG2.)) ..) >> 3: (SB-IMPL::%MAP-FOR-EFFECT-ARITY-1 #> {124D2DBD}> (#)) >> 4: ((SB-PCL::FAST-METHOD MEL.PUBLIC:MAP-RECENT-MESSAGES :AROUND (T >> MEL.PUBLIC:BASIC-FOLDER)) ..) >> 5: (SB-INT:SIMPLE-EVAL-IN-LEXENV (CHECK-EMAIL) #) > > > _______________________________________________ > mel-base-devel mailing list > mel-base-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/mel-base-devel From tritchey at mac.com Fri May 15 15:13:05 2009 From: tritchey at mac.com (Timothy Ritchey) Date: Fri, 15 May 2009 11:13:05 -0400 Subject: [mel-base-devel] maildir multipart Message-ID: If I do something like: (defun check-email () (let* ((folder (mel:make-imaps-folder :host "imap.gmail.com" :username "XXXXX" :password "XXXXX")) (messages (mel:messages folder))) (dolist (m messages) (format t "~a~%" (find-plain-text m))) folder)) I am able to find the plain text part of the messages using: (defun find-plain-text (message) (let ((parts (mel:parts message)) (body "Unable to process email contents")) (dolist (p parts) (multiple-value-bind (a b) (mel:content-type p) (if (and (equal a :TEXT) (equal b :PLAIN)) (let ((b (mel:part-body-string p)) (encoding (mel:content-transfer-encoding p))) (if (string-equal encoding "quoted-printable") (setf body (mel:decode-quoted-printable b)) (setf body b))) (progn (when (and (equal a :MULTIPART) (equal b :ALTERNATIVE)) (find-plain-text p)))))) body)) but if I try to move things over to a maildir, and run from there, my FIND-PLAIN-TEXT function doesn't work. Is the maildir message format different than a message in an imap folder? (defun check-email () (let* ((folder (mel:make-imaps-folder :host "imap.gmail.com" :username "XXXXX" :password "XXXXX")) (tmp-folder (mel:make-maildir-folder "/tmp/test-maildir/" :if-does- not-exist :create))) (mel:move-folder folder tmp-folder) (mel:map-messages (lambda (m) (let ((body (find-plain-text m))) (format t "~a~%" body))) tmp-folder))) From js at codeartist.org Sun May 17 21:03:20 2009 From: js at codeartist.org (Jochen Schmidt) Date: Sun, 17 May 2009 23:03:20 +0200 Subject: [mel-base-devel] maildir multipart In-Reply-To: References: Message-ID: <281FAFEE-A7E1-49CC-B97A-87D75A81912F@codeartist.org> Am 15.05.2009 um 17:13 schrieb Timothy Ritchey: > If I do something like: > > (defun check-email () > (let* ((folder (mel:make-imaps-folder :host > "imap.gmail.com" :username "XXXXX" :password "XXXXX")) > (messages (mel:messages folder))) > (dolist (m messages) > (format t "~a~%" (find-plain-text m))) folder)) > > I am able to find the plain text part of the messages using: > > (defun find-plain-text (message) > (let ((parts (mel:parts message)) > (body "Unable to process email contents")) > (dolist (p parts) > (multiple-value-bind (a b) (mel:content-type p) > (if (and (equal a :TEXT) (equal b :PLAIN)) > (let ((b (mel:part-body-string p)) > (encoding (mel:content-transfer-encoding p))) > (if (string-equal encoding "quoted-printable") > (setf body (mel:decode-quoted-printable b)) > (setf body b))) > (progn > (when (and (equal a :MULTIPART) (equal b :ALTERNATIVE)) > (find-plain-text p)))))) > body)) > > but if I try to move things over to a maildir, and run from there, my > FIND-PLAIN-TEXT function doesn't work. Is the maildir message format > different than a message in an imap folder? No the message format is the same (octet wise). One of the design principles of mel-base was to never modify the content of a message. (Trivia: I started mel-base because the "standard" python mail library modified my mails while fetching them). You can compared the file stored in the maildir folder with one saved using e.g. Apple Mail.app from the IMAP server - they ought to be identical. The problem here is that perhaps the message is interpreted differently when accessing its parts. IMAP is a cool thing - you get many things from it without much work. RFC2060 (IMAP) defines a thing called "Body Structure" which describes the structure of the message- body. If you use IMAP, mel-base can just query the server for the body structure - if not it will have to compute it by itself. It seems that the body structure of your message parsed by your server is different from the one parsed by mel-base. The folder protocol of mel-base has a GF called COMPUTE-BODYSTRUCTURE- USING-FOLDER. Which has two implemented methods: One for IMAP-Foldes and one for all others. The user-level function is called COMPUTE- BODYSTRUCTURE and can be called on a message object to get the body structure of it. I've tried it using my current snapshot here on some multipart message in my mail account. The IMAP gives: (((:TEXT :PLAIN (:CHARSET "us-ascii") NIL NIL :QUOTED-PRINTABLE 1638 45 NIL NIL NIL) (:APPLICATION :PGP-SIGNATURE (:NAME "signature.asc") NIL NIL :7BIT 205 NIL NIL NIL NIL) :SIGNED (:MICALG "pgp-sha1" :PROTOCOL "application/pgp-signature" :BOUNDARY "=-N3IPeK0F6Rq/H013D1bn") NIL NIL) (:TEXT :PLAIN (:CHARSET "us-ascii") NIL NIL :7BIT 165 4 NIL NIL NIL) :MIXED (:BOUNDARY "===============1035646608==") NIL NIL) And the copied message in the Maildir-folder: (((:TEXT :PLAIN NIL NIL NIL :QUOTED-PRINTABLE 1638 45 NIL NIL NIL) (:APPLICATION :PGP-SIGNATURE (:NAME "signature.asc") NIL NIL NIL 205 7 NIL NIL NIL) :SIGNED (:BOUNDARY "=-N3IPeK0F6Rq/H013D1bn" :PROTOCOL "application/pgp- signature" :MICALG "pgp-sha1") NIL NIL) (:TEXT :PLAIN (:CHARSET "us-ascii") NIL NIL :7BIT 165 4 NIL NIL NIL) :MIXED (:BOUNDARY "===============1035646608==") NIL NIL) There seem to be some differences here too. As of now - mel-base doesn't get the charset "us-ascii" of the first text/plain part. This is ok since charset us-ascii is implicitely us-ascii. The mail-file doesn't specify us-ascii on that part - its the IMAP server that adds it to the bodystructure to make it explicit. I could do so too - but it is not against the spec to let it implicit. I also fixed a bug with computation of octet and line-counts of the parts. I always counted the last empty line before the closing boundary - this is actually not part of the part (pardon the pun ;-) ). I've built a new release of mel-base (0.9-0) uploaded it to my server and registered it at CLiki (so that things like ASDF-INSTALL work). You can just try it - I've reimplemented the computation of the bodystructure so that it now gives the exact same result as my IMAP server. I've also reworked the character coding infrastructure (now using bivalent streams and flexi-streams were appropriate). To see whats wrong in your case, I would need a bit more information. Can you send the result of COMPUTE-BODYSTRUCTURE for IMAP and Maildir? ciao, Jochen -- Jochen Schmidt js at codeartist.org http://www.codeartist.org > > (defun check-email () > (let* ((folder (mel:make-imaps-folder :host > "imap.gmail.com" :username "XXXXX" :password "XXXXX")) > (tmp-folder (mel:make-maildir-folder "/tmp/test-maildir/" :if-does- > not-exist :create))) > (mel:move-folder folder tmp-folder) > (mel:map-messages > (lambda (m) > (let ((body (find-plain-text m))) > (format t "~a~%" body))) > tmp-folder))) > > > > _______________________________________________ > mel-base-devel mailing list > mel-base-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/mel-base-devel Jochen Schmidt js at codeartist.org http://www.codeartist.org