From jacobite1607 at gmail.com Thu Mar 1 15:56:24 2012 From: jacobite1607 at gmail.com (William P. Proffitt) Date: Thu, 01 Mar 2012 10:56:24 -0500 Subject: [cl-smtp-devel] Corrupted attachments In-Reply-To: <1330608272.2188.53.camel@ji-desktop> References: <1330608272.2188.53.camel@ji-desktop> Message-ID: Hello Jan, > Have you try to run the tests: (cl-smtp::run-tests) No I had not thanks for telling me. I just ran it and the following is what failed. run test: "rfc2231-encode-string-utf-8" failed: The assertion (STRING-EQUAL CL-SMTP::QSTR "UTF-8''%C3%B6%C3%BC%C3%A4%C3%96%C3%9C%C3%84%C3%9F") failed. run test: "rfc2045-q-encode-string-utf-8" failed: The assertion (STRING-EQUAL CL-SMTP::QSTR "=?UTF-8?Q?=C3=B6=C3=BC=C3=A4=C3=96=C3=9C=C3=84=C3=9F?=") failed. Would it be safe to assume that "cl-base64" is the issue? Regards, William From jan.idzikowski at knowledgetools.de Thu Mar 1 16:22:31 2012 From: jan.idzikowski at knowledgetools.de (Jan Idzikowski) Date: Thu, 01 Mar 2012 17:22:31 +0100 Subject: [cl-smtp-devel] Corrupted attachments In-Reply-To: References: <1330608272.2188.53.camel@ji-desktop> Message-ID: <1330618951.2188.63.camel@ji-desktop> Hello William, > run test: "rfc2231-encode-string-utf-8" > failed: The assertion (STRING-EQUAL CL-SMTP::QSTR > "UTF-8''%C3%B6%C3%BC%C3%A4%C3%96%C3%9C%C3%84%C3%9F") failed. > > run test: "rfc2045-q-encode-string-utf-8" > failed: The assertion (STRING-EQUAL CL-SMTP::QSTR > "=?UTF-8?Q?=C3=B6=C3=BC=C3=A4=C3=96=C3=9C=C3=84=C3=9F?=") failed. > these tests have not a problem with cl-base64, these functions use flexi-streams. But I need more information, can you send me a stack trace, or can you print the qstr variable in the test function bevor the assertion. > Would it be safe to assume that "cl-base64" is the issue? No, I could not say it. The problem is that I have no LispWorks, only Allegro and Sbcl. Thanks Jan From jacobite1607 at gmail.com Fri Mar 9 16:37:35 2012 From: jacobite1607 at gmail.com (William P. Proffitt) Date: Fri, 09 Mar 2012 11:37:35 -0500 Subject: [cl-smtp-devel] Corrupted attachments In-Reply-To: <1330618951.2188.63.camel@ji-desktop> References: <1330608272.2188.53.camel@ji-desktop> <1330618951.2188.63.camel@ji-desktop> Message-ID: I posted an example of this on the Lispworks group in case some else is seeing it, but feel it is more appropriate here. When sending any type of file it is randomly scrambling attachments. For example a 1000 line ASCII text file it scrambled the file in 3 places each time with what appears to be some random data, for example at around line 75, I see: This is a test of an email attachment. This is a test of an email attachment. This is a test of an email atta??V?B???F??2%06?2%06%12%07FW7B%06?b%06%16?%06V?%16??%06%17GF%166??V?B???F??2%06?2%06%12%07FW7B%06?b%06%16?%06V?%16??%06%17GF%166??V?B???F??2%06?2%06%12%07FW7B%06?b%06%16?%06V?%16??%06%17GF%166??V?B???F??2%06?2%06%12%07FW7B%06?b%06%16?%06V?%16??%06%17GF%166??V?B???F??2%06?2%06%12%07FW7B%06?b%06%16?%06V?%16??%06%17GF%166??V?B???F??2%06?2%06%12%07FW7B%06?b%06%16?%06V?%16??%06%17GF%166??V?B???F??2%06?2%06%12%07FW7B%06?b%06%16?%06V?%16??%06%17GF%166??V?B???F??2%06?2%06%12%07FW7B%06?b%06%16?%06V?%16??%06%17GF%166??V?B???F??2%06? This is a test of an email attachment. This is a test of an email attachment. I am not sure if this is a encoding issue or not, someone suggested it may be a C pointer gone wild. This did not appear to be and issue in LW 6.0 only since LW 6.1 came out. Any other LW 6.1 users seeing the same issue? Regards, William From jacobite1607 at gmail.com Fri Mar 9 20:42:43 2012 From: jacobite1607 at gmail.com (William P. Proffitt) Date: Fri, 09 Mar 2012 15:42:43 -0500 Subject: [cl-smtp-devel] Corrupted attachments In-Reply-To: References: <1330608272.2188.53.camel@ji-desktop> <1330618951.2188.63.camel@ji-desktop> Message-ID: This appears to be an issue with Outlook and decoding what is sent. When I read the email with other email viewers I have no issues, so obviously encoding works to some degree. When I try opening the email in Outlook 2003 or 2007 it is messed up, if I try opening the same email sent to GMail via CC or forwarded via Exchange, there is no problem with the attachments. William From jan.tolenaar at xs4all.nl Fri Mar 9 21:06:50 2012 From: jan.tolenaar at xs4all.nl (Jan Tolenaar) Date: Fri, 09 Mar 2012 22:06:50 +0100 Subject: [cl-smtp-devel] Corrupted attachments In-Reply-To: References: <1330608272.2188.53.camel@ji-desktop> <1330618951.2188.63.camel@ji-desktop> Message-ID: <4F5A70EA.3040602@xs4all.nl> On 09/03/2012 21:42, William P. Proffitt wrote: > This appears to be an issue with Outlook and decoding what is sent. > When I read the email with other email viewers I have no issues, so > obviously encoding works to some degree. When I try opening the email > in Outlook 2003 or 2007 it is messed up, if I try opening the same > email sent to GMail via CC or forwarded via Exchange, there is no > problem with the attachments. > > William > > _______________________________________________ > cl-smtp-devel mailing list > cl-smtp-devel at common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/cl-smtp-devel > I had a problem with attachments (using LW 6.0 on W7) and made the attached patch available to the cl-smtp mailing list where it was bounced (because I had no karma). Never got another reply. Perhaps the patch solves your problem. Jan -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: patch.txt URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: attachments.lisp URL: From ehuels at gmail.com Fri Mar 9 21:48:37 2012 From: ehuels at gmail.com (Erik Huelsmann) Date: Fri, 9 Mar 2012 22:48:37 +0100 Subject: [cl-smtp-devel] Fwd: Corrupted attachments In-Reply-To: <4F5A70EA.3040602@xs4all.nl> References: <1330608272.2188.53.camel@ji-desktop> <1330618951.2188.63.camel@ji-desktop> <4F5A70EA.3040602@xs4all.nl> Message-ID: ---------- Forwarded message ---------- From: Jan Tolenaar Date: Fri, Mar 9, 2012 at 10:06 PM Subject: Re: [cl-smtp-devel] Corrupted attachments To: "William P. Proffitt" Cc: cl-smtp-devel at common-lisp.net On 09/03/2012 21:42, William P. Proffitt wrote: > This appears to be an issue with Outlook and decoding what is sent. When I > read the email with other email viewers I have no issues, so obviously > encoding works to some degree. When I try opening the email in Outlook 2003 > or 2007 it is messed up, if I try opening the same email sent to GMail via > CC or forwarded via Exchange, there is no problem with the attachments. > > William > > ______________________________**_________________ > cl-smtp-devel mailing list > cl-smtp-devel at common-lisp.net > http://lists.common-lisp.net/**cgi-bin/mailman/listinfo/cl-**smtp-devel > > I had a problem with attachments (using LW 6.0 on W7) and made the attached patch available to the cl-smtp mailing list where it was bounced (because I had no karma). Never got another reply. Perhaps the patch solves your problem. Jan _______________________________________________ cl-smtp-devel mailing list cl-smtp-devel at common-lisp.net http://lists.common-lisp.net/cgi-bin/mailman/listinfo/cl-smtp-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- --- d:\asdf-install-dir\quicklisp\dists\quicklisp\software\cl-smtp-20101107-http\attachments.lisp 2011-05-24 16:49:26.599664000 +0200 +++ attachments.lisp 2011-06-29 18:31:43.010440900 +0200 @@ -58,6 +58,7 @@ content-transfer-encoding ;; 7 bit or 8 bit (include-blank-line? t)) (when boundary + (write-blank-line sock) (write-to-smtp sock (format nil "--~a" boundary))) (when content-type (write-to-smtp sock (format nil "Content-type: ~a" content-type))) -------------- next part -------------- ;;; -*- mode: Lisp -*- ;;; This file is part of CL-SMTP, the Lisp SMTP Client ;;; Copyright (C) 2004/2005/2006/2007/2008/2009/2010 Jan Idzikowski ;;; This library is free software; you can redistribute it and/or ;;; modify it under the terms of the Lisp Lesser General Public License ;;; (http://opensource.franz.com/preamble.html), known as the LLGPL. ;;; This library is distributed in the hope that it will be useful, ;;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;;; Lisp Lesser GNU General Public License for more details. ;;; File: attachments.lisp ;;; Description: encoding and transmitting login to include a mime attachment ;;; ;;; Contributed by Brian Sorg ;;; ;;; Thanks to David Cooper for make-random-boundary ;;; (in-package :cl-smtp) ;;; Addition to allow for sending mime attachments along with the smtp message ;;---- Initialize array of possible boundary characters to make start of attachments (defparameter *boundary-chars* (let* ((chars (list #\a #\b #\c #\d #\e #\f #\g #\h #\i #\j #\k #\l #\m #\n #\o #\p #\q #\r #\s #\t #\u #\v #\w #\x #\y #\z #\A #\B #\C #\D #\E #\F #\G #\H #\I #\J #\K #\L #\M #\N #\O #\P #\Q #\R #\S #\T #\U #\V #\W #\X #\Y #\Z #\0 #\1 #\2 #\3 #\4 #\5 #\6 #\7 #\8 #\9)) (arr (make-array (length chars)))) (dotimes (i (length chars) arr) (setf (aref arr i) (pop chars))))) (defun make-random-boundary (&optional (length 30) (boundary-chars *boundary-chars*)) (let ((boundary (make-string length)) (prefix "_---------_") (chars-length (length boundary-chars))) (dotimes (i length (concatenate 'string prefix boundary)) (setf (aref boundary i) (svref *boundary-chars* (random chars-length)))))) (defun generate-multipart-header (sock boundary &key (multipart-type "mixed")) (write-to-smtp sock (format nil "Content-type: multipart/~a;~%~tBoundary=\"~a\"" multipart-type boundary))) (defun generate-message-header (sock &key boundary ;; uniques string of character -- see make-random-boundary content-type ;; "text/plain; charset=ISO-8859-1" content-disposition ;; inline attachment content-transfer-encoding ;; 7 bit or 8 bit (include-blank-line? t)) (when boundary (write-blank-line sock) (write-to-smtp sock (format nil "--~a" boundary))) (when content-type (write-to-smtp sock (format nil "Content-type: ~a" content-type))) (when content-disposition (write-to-smtp sock (format nil "Content-Disposition: ~A" content-disposition))) (when content-transfer-encoding (write-to-smtp sock (format nil "Content-Transfer-Encoding: ~A" content-transfer-encoding))) (when include-blank-line? (write-blank-line sock))) (defun escape-rfc822-quoted-string (str) (with-output-to-string (s) (loop for c across str do (when (find (char-code c) '(10 13 92 34)) (write-char #\\ s)) (write-char c s)))) (defun rfc2231-encode-string (string &key (external-format :utf-8)) (with-output-to-string (s) (format s "~A''" (string-upcase (symbol-name external-format))) (loop for n across (string-to-octets string :external-format external-format) for c = (code-char n) do (cond ((or (char<= #\0 c #\9) (char<= #\a c #\z) (char<= #\A c #\Z) (find c "$-_.!*'()," :test #'char=)) (write-char c s)) ((char= c #\Space) (write-char #\+ s)) (t (format s "%~2,'0x" (char-code c))))))) (defun send-attachment-header (sock boundary attachment external-format) (let ((quoted-name (escape-rfc822-quoted-string (rfc2045-q-encode-string (attachment-name attachment) :external-format external-format))) (quoted-name* (escape-rfc822-quoted-string (rfc2231-encode-string (attachment-name attachment) :external-format external-format)))) (generate-message-header sock :boundary boundary :content-type (format nil "~A;~%~tname*=~A;~%~tname=~S" (attachment-mime-type attachment) quoted-name* quoted-name) :content-transfer-encoding "base64" :content-disposition (format nil "attachment; filename*=~A; filename=~S" quoted-name* quoted-name)))) (defun send-end-marker (sock boundary) ;; Note the -- at beginning and end of boundary is required (write-to-smtp sock (format nil "~%--~a--~%" boundary))) (defclass attachment () ((name :initarg :name :accessor attachment-name) (data-pathname :initarg :data-pathname :accessor attachment-data-pathname) (mime-type :initarg :mime-type :accessor attachment-mime-type))) (defun make-attachment (data-pathname &key (name (file-namestring data-pathname)) (mime-type (lookup-mime-type name))) (make-instance 'attachment :data-pathname data-pathname :name name :mime-type mime-type)) (defmethod attachment-name ((attachment pathname)) (file-namestring attachment)) (defmethod attachment-data-pathname ((attachment pathname)) attachment) (defmethod attachment-mime-type ((attachment pathname)) (lookup-mime-type (namestring attachment))) (defmethod attachment-name ((attachment string)) (file-namestring attachment)) (defmethod attachment-data-pathname ((attachment string)) attachment) (defmethod attachment-mime-type ((attachment string)) (lookup-mime-type attachment)) (defun send-attachment (sock attachment boundary buffer-size external-format) (send-attachment-header sock boundary attachment external-format) (base64-encode-file (attachment-data-pathname attachment) sock :buffer-size buffer-size)) (defun base64-encode-file (file-in sock &key (buffer-size 256) ;; in KB (wrap-at-column 70)) "Encodes the file contents given by file-in, which can be of any form appropriate to with-open-file, and write the base-64 encoded version to sock, which is a socket. Buffer-size, given in KB, controls how much of the file is processed and written to the socket at one time. A buffer-size of 0, processes the file all at once, regardless of its size. One will have to weigh the speed vs, memory consuption risks when chosing which way is best. Wrap-at-column controls where the encode string is divided for line breaks." (when (probe-file file-in) ;;-- open filein --------- (with-open-file (strm-in file-in :element-type '(unsigned-byte 8)) (let* ((;; convert buffer size given to bytes ;; or compute bytes based on file max-buffer-size (if (zerop buffer-size) (file-length strm-in) ;; Ensures 64 bit encoding is properly ;; divided so that filler ;; characters are not required between chunks (* 24 (truncate (/ (* buffer-size 1024) 24))))) (column-count 0) (eof? nil) (buffer (make-array max-buffer-size :element-type '(unsigned-byte 8)))) (loop (print-debug (format nil "~%Process attachment ~a~%" file-in)) (let* ((;; read a portion of the file into the buffer arrary and ;; returns the index where it stopped byte-count (dotimes (i max-buffer-size max-buffer-size) (let ((bchar (read-byte strm-in nil 'EOF))) (if (eql bchar 'EOF) (progn (setq eof? t) (return i)) (setf (aref buffer i) bchar)))))) (if (zerop buffer-size) ;; send file all at once to socket. #+allegro (write-string (excl:usb8-array-to-base64-string buffer wrap-at-column) sock) #-allegro (cl-base64:usb8-array-to-base64-stream buffer sock :columns wrap-at-column) ;; otherwise process file in chunks. ;; The extra encoded-string, ;; and its subseq functions are brute force methods ;; to properly handle the wrap-at-column feature ;; between buffers. ;; Not the most efficient way, ;; but it works and uses existing functions ;; in the cl-base64 package. (let* ((;; drops off extra elements that were not filled in in reading, this is important for lisp systems that default a value into ;; the array when it is created. -- ie Lispworks, SBCL trimmed-buffer (if eof? (subseq buffer 0 byte-count) buffer)) (encoded-string #+allegro (excl:usb8-array-to-base64-string trimmed-buffer) #-allegro (cl-base64:usb8-array-to-base64-string trimmed-buffer))) (loop for ch across encoded-string do (progn (write-char ch sock) (incf column-count) (when (= column-count wrap-at-column) (setq column-count 0) (write-char #\Newline sock)))))) (force-output sock) (print-debug (format nil "~% Eof is ~a~%" eof?)) (when (or (zerop buffer-size) eof?) (write-blank-line sock) (return)))))))) From jacobite1607 at gmail.com Fri Mar 9 22:03:19 2012 From: jacobite1607 at gmail.com (William P. Proffitt) Date: Fri, 09 Mar 2012 17:03:19 -0500 Subject: [cl-smtp-devel] Corrupted attachments In-Reply-To: <4F5A70EA.3040602@xs4all.nl> References: <1330608272.2188.53.camel@ji-desktop> <1330618951.2188.63.camel@ji-desktop> <4F5A70EA.3040602@xs4all.nl> Message-ID: > I had a problem with attachments (using LW 6.0 on W7) and made the > attached patch available to the cl-smtp mailing list where it was > bounced (because I had no karma). Never got another reply. > Perhaps the patch solves your problem. Hi Jan, Thanks for the reply. I am on W7 64bit. I tried the patch but it had no effect. Oddly, I don't remember seeing this on LW 6.0, perhaps because the size of the attachments were to small. William From jan.tolenaar at xs4all.nl Fri Mar 9 22:17:21 2012 From: jan.tolenaar at xs4all.nl (Jan Tolenaar) Date: Fri, 09 Mar 2012 23:17:21 +0100 Subject: [cl-smtp-devel] Corrupted attachments In-Reply-To: References: <1330608272.2188.53.camel@ji-desktop> <1330618951.2188.63.camel@ji-desktop> <4F5A70EA.3040602@xs4all.nl> Message-ID: <4F5A8171.9090603@xs4all.nl> On 09/03/2012 23:03, William P. Proffitt wrote: >> I had a problem with attachments (using LW 6.0 on W7) and made the >> attached patch available to the cl-smtp mailing list where it was >> bounced (because I had no karma). Never got another reply. >> Perhaps the patch solves your problem. > > Hi Jan, > > Thanks for the reply. I am on W7 64bit. I tried the patch but it had > no effect. Oddly, I don't remember seeing this on LW 6.0, perhaps > because the size of the attachments were to small. > > William > William, If I remember correctly, cl-smtp doesn't always write a blank line before the boundary string between two attachments. You might not notice an error but then your smtp host has a bug because it recognizes a boundary string that is not preceded by a blank line. Of course during development and testing I used a 'bad' smtp host. My client's smtp host was coded correctly and produced corrupted attachments. Another suggestion: usocket 0.5.5 (not on quicklisp but on its home site). Jan From jacobite1607 at gmail.com Sat Mar 10 15:58:30 2012 From: jacobite1607 at gmail.com (William Proffitt) Date: Sat, 10 Mar 2012 10:58:30 -0500 Subject: [cl-smtp-devel] Corrupted attachments In-Reply-To: <4F5A8171.9090603@xs4all.nl> References: <1330608272.2188.53.camel@ji-desktop> <1330618951.2188.63.camel@ji-desktop> <4F5A70EA.3040602@xs4all.nl> <4F5A8171.9090603@xs4all.nl> Message-ID: > If I remember correctly, cl-smtp doesn't always write a blank line > before the boundary string between two attachments. That's a good place to place to look thanks. > Another suggestion: usocket 0.5.5 (not on quicklisp but on its home > site). I already have 0.5.5 as usocket wouldn't install properly when LW6.1 came out. I let Chun know of the issue and he had a patch to me in literally minutes. -- Regards, William