Updates for release 1.1

diff --git a/debian/README.Debian b/debian/README.Debian
index a1e87392b0250cc4ec0dc3935474701b2b404658..a9a91654008aeea441977412c28b31d29eed4c47 100644 (file)
--- a/debian/README.Debian
+++ b/debian/README.Debian
@@ -1,7 +1,7 @@
 This library also requires the Common Lisp
 This library also requires the Common Lisp

-cl-store and zlib libraries. Currently, these
+cl-store, salza2, and chipz libraries. Currently, these

 libraries will have to be manually downloaded
 and installed as Debian does not have those
 libraries in its repository.
 
 libraries will have to be manually downloaded
 and installed as Debian does not have those
 libraries in its repository.
 

- -- Kevin M. Rosenberg <kmr@debian.org>, Fri,  1 Jul 2011 23:15:07 -0600
+ -- Kevin M. Rosenberg <kmr@debian.org>, Tue,  5 Jul 2011 08:14:42 -0600

diff --git a/debian/changelog b/debian/changelog
index a2c9c201a10d36de256ba6730e7251605ff32279..5b1f88232bff0cf4f1c303f70bf54373c91917c7 100644 (file)
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+cl-memstore (1.1.0-1) unstable; urgency=low
+
+  * New upstream
+
+ -- Kevin M. Rosenberg <kmr@debian.org>  Tue, 05 Jul 2011 08:17:22 -0600
+

 cl-memstore (1.0-1) unstable; urgency=low
 
   * Initial version
 cl-memstore (1.0-1) unstable; urgency=low
 
   * Initial version

diff --git a/debian/docs b/debian/docs
index 49d454d4afd2d7a957f47777248fd5398b35e4f3..84a799465185029b715709c0a3fccaa03b590b65 100644 (file)
--- a/debian/docs
+++ b/debian/docs
@@ -1,3 +1,4 @@
-README.md
+doc/memstore.html
+doc/memcache.html

 debian/README.Debian
 
 debian/README.Debian
 

diff --git a/doc/memcache.html b/doc/memcache.html
new file mode 100644 (file)
index 0000000..2352e11
--- /dev/null
+++ b/doc/memcache.html
@@ -0,0 +1,55 @@
+<h1>Memcache</h1>
+
+<p><em>Library for memcached protocol</em></p>
+
+<p>Author: Kevin Rosenberg <a href="&#109;&#x61;&#x69;l&#x74;&#x6F;:&#x6B;&#x65;&#118;&#x69;&#x6E;&#64;&#114;o&#x73;&#101;&#110;&#x62;&#101;&#114;&#x67;&#46;&#x6E;&#x65;&#116;">&#x6B;&#x65;&#118;&#x69;&#x6E;&#64;&#114;o&#x73;&#101;&#110;&#x62;&#101;&#114;&#x67;&#46;&#x6E;&#x65;&#116;</a>, based on the
+<code>cl-memcached</code> library by Abhijit 'quasi' Rao and 
+Chaitanya Gupta .</p>
+
+<p>Date Started: July 1, 2011</p>
+
+<h2>Overview</h2>
+
+<p>This package is based on the <code>cl-memcached</code> library.  It is
+substantially modified for use with the
+<a href="http://memstore.b9.com">memstore</a> library. The primary areas of
+additional functionality are:</p>
+
+<ul>
+<li><p>Support for flags field with get and set functions.
+This is required as memstore stores bit flags denoting
+how the data is serialized. That information is required
+to deserialize the data.</p></li>
+<li><p>Support for additional memcached functionality, such as
+the gets command for retrieving CAS identifiers. The CAS
+unique ID is used for the added <code>:cas</code> storage command.
+Other storage commands newly supported are <code>:append</code> and 
+<code>:prepend</code>.</p></li>
+<li><p>All communication now uses <code>mc-send-command</code> function with
+transparently supports writing strings with <code>write-byte</code>.
+This allows <code>usocket</code> to be used on other Lisp implementations
+besides AllegroCL. Because cl-memcached used <code>write-string</code>
+with usocket-stream, only AllegroCL was supported.
+By sending all data as (unsigned-byte 8), all Lisp implementions
+supported by <code>usocket</code> are now supported with <code>memcached</code>.</p></li>
+<li><p>Encapsulated reading and writing to socket stream to avoid
+handling \#return characters in high-level code.</p></li>
+<li><p>Changes to support the change in statistics fields with membase.
+Some fields were no longer present. Also, membase 1.7 has
+187 statistics fields versus the 20 fields supported in
+<code>cl-memcached</code>. New function <code>mc-get-stat</code> allows to retrieving
+any statistics field by name.</p></li>
+<li><p>More robust <code>print-object</code> functions to avoid errors if fields
+in statistics are not present.</p></li>
+<li><p>Removed compatibility functions in <code>compat.lisp</code> by using the
+<a href="http://gitpub.b9.com/kmrcl.git"><code>kmrcl</code></a> library to provide those
+functions as well as utilitizing other <code>kmrcl</code> functions to simplify
+code.</p></li>
+<li><p>Added functions to support all memcached API commands,
+such as <code>flush_all</code> and <code>version</code>.</p></li>
+<li><p>Support for the <code>moreply</code> command argument accepted by
+many commands.</p></li>
+<li><p>Support the <code>noreply</code> argument that many API commands accept.</p></li>
+<li><p>Write nearly the entire code base for improved
+clarity, robustness, and efficiency.</p></li>
+</ul>

diff --git a/doc/memstore.html b/doc/memstore.html
new file mode 100644 (file)
index 0000000..62fbbb2
--- /dev/null
+++ b/doc/memstore.html
@@ -0,0 +1,55 @@
+<h1>Memstore</h1>
+
+<p><em>A high-level interface for the memcached and membase servers</em></p>
+
+<p>Written by Kevin M. Rosenberg <a href="&#x6D;&#97;&#x69;&#108;&#x74;&#111;:&#x6B;&#x65;&#118;&#x69;&#x6E;&#64;&#114;&#x6F;&#115;&#x65;&#110;&#x62;e&#x72;&#103;&#x2E;&#110;&#101;&#116;">&#x6B;&#x65;&#118;&#x69;&#x6E;&#64;&#114;&#x6F;&#115;&#x65;&#110;&#x62;e&#x72;&#103;&#x2E;&#110;&#101;&#116;</a></p>
+
+<h2>Downloads</h2>
+
+<ul>
+<li>The web site for memstore is <a href="http://memstore.b9.com/">http://memstore.b9.com/</a></li>
+<li>A git repository is available at <a href="http://gitpub.b9.com/memstore.git">http://gitpub.b9.com/memstore.git</a></li>
+<li>Releases are available for download at <a href="http://files.b9.com/memstore/">http://files.b9.com/memstore/</a></li>
+</ul>
+
+<h2>Prerequisites</h2>
+
+<p>This library incorporates a heavily modified version of <code>cl-memcached</code>
+version 0.4.1. The primary need for the modification is to support the
+<code>flags</code> field supported by the memcached server. The flag field is
+used to store information required to deserialize the object data from
+memcached. In addition to adding the <code>flags</code> field, the library has
+been heavily refactored. For more information on the changes in
+memcache, see <a href="memcache.html">memcache</a>.</p>
+
+<p>This library also requires Common Lisp libraries of 
+<a href="http://gitpub.b9.com/kmrcl.git">kmrcl</a>,
+<a href="http://common-lisp.net/project/cl-store/">cl-store</a>,
+<a href="http://weitz.de/flexi-streams/">flexi-streams</a>,
+and <a href="http://common-lisp.net/project/zlib/">zlib</a>.</p>
+
+<h2>Overview</h2>
+
+<p>Memstore allows efficient storing of simple objects as well as
+easy storing of complex objects and optional compression.</p>
+
+<p>When storing an object, if the object is a string then that is
+directly written to the memcached server. For non-strings, an attempt
+to made to write the object to a string with <code>*print-readably*</code> bound
+to <code>t</code>. If that succeeds, then the string is converted to a vector of
+octets. If that fails, the <code>cl-store</code> is used to serialize that object
+to a vector of octets.</p>
+
+<p>Next, optional compression is applied to the octets. First, the
+<code>*compression-enabled*</code> flag is checked to see if compression is
+enabled. Next, the length of the objects is compared to
+<code>*compression-threshold*</code>. Only objects larger than
+<code>*compression-threshold*</code> will be compressed. For objects that qualify
+for compression, the size of the compressed object is compared to the
+length of the uncompressed object to decide if the object is shrunk
+enough to make the compression worthwhile.</p>
+
+<p>The <code>flags</code> parameter to cl-memcached stores whether cl-store or
+write-to-string is used to serialize the object and whether
+compression is applied. <code>mem-restore</code> uses those flags to determine
+how to reconstruct the object.</p>

diff --git a/memcache/memcache.lisp b/memcache/memcache.lisp
index c5593cbc7ee712a69529275104b20544a7ea4aa2..231065b5c461e6bd5afe9365c3f892a73e908cf3 100644 (file)
--- a/memcache/memcache.lisp
+++ b/memcache/memcache.lisp
@@ -34,7 +34,7 @@
   (print-unreadable-object (mc stream :type t :identity t)
     (format stream "~A on ~A:~A ~AMB"
             (when (slot-boundp mc 'name) (name mc))
   (print-unreadable-object (mc stream :type t :identity t)
     (format stream "~A on ~A:~A ~AMB"
             (when (slot-boundp mc 'name) (name mc))

-            (when (slot-boundp mc 'ip) (ip mc))
+            (when (slot-boundp mc 'host) (host mc))

             (when (slot-boundp mc 'port) (port mc))
             (when (and (slot-boundp mc 'memcached-server-storage-size)
                        (numberp (slot-value mc 'memcached-server-storage-size)))
             (when (slot-boundp mc 'port) (port mc))
             (when (and (slot-boundp mc 'memcached-server-storage-size)
                        (numberp (slot-value mc 'memcached-server-storage-size)))


@@ -53,10 +53,10 @@
        (setf (slot-value memcache 'memcached-server-storage-size) (mc-stats-limit-maxbytes stats))
        (setf (slot-value memcache 'memcached-server-storage-size) -1))))
 
        (setf (slot-value memcache 'memcached-server-storage-size) (mc-stats-limit-maxbytes stats))
        (setf (slot-value memcache 'memcached-server-storage-size) -1))))
 

-(defun make-memcache-instance (&key (ip "127.0.0.1") (port 11211)
+(defun make-memcache-instance (&key (host "127.0.0.1") (port 11211)

                                  (name "Memcache") (pool-size 5))
   "Creates an instance of class MEMCACHE which represents a memcached server."
                                  (name "Memcache") (pool-size 5))
   "Creates an instance of class MEMCACHE which represents a memcached server."

-  (make-instance 'memcache :name name :ip ip :port port :pool-size pool-size))
+  (make-instance 'memcache :name name :host host :port port :pool-size pool-size))

 
 
 (defmacro with-pool-maybe ((stream memcache use-pool) &body body)
 
 
 (defmacro with-pool-maybe ((stream memcache use-pool) &body body)


@@ -414,7 +414,7 @@ information about each slot"
   (mc-pool-grow memcache))
 
 (defun mc-make-pool-item (&key (memcache *memcache*))
   (mc-pool-grow memcache))
 
 (defun mc-make-pool-item (&key (memcache *memcache*))

-  (handler-case (usocket:socket-connect (ip memcache) (port memcache) :element-type '(unsigned-byte 8))
+  (handler-case (usocket:socket-connect (host memcache) (port memcache) :element-type '(unsigned-byte 8))

     (usocket:socket-error (e) (error 'memcached-server-unreachable :error e))
     (error (e) (error 'cannot-make-pool-object :error e))))
 
     (usocket:socket-error (e) (error 'memcached-server-unreachable :error e))
     (error (e) (error 'cannot-make-pool-object :error e))))
 

diff --git a/memcache/specials.lisp b/memcache/specials.lisp
index 2006f37dac0f01f922c58d602f0248cd1e46b165..5a0f9a878de6c83180980018cd1a5ef4bfd9d601 100644 (file)
--- a/memcache/specials.lisp
+++ b/memcache/specials.lisp
@@ -97,12 +97,12 @@ limit-maxbytes             mc-stats-limit-maxbytes           Number of bytes thi
     :reader name
     :type simple-string
     :documentation "Name of this Memcache instance")
     :reader name
     :type simple-string
     :documentation "Name of this Memcache instance")

-   (ip
-    :initarg :ip
+   (host
+    :initarg :host

     :initform "127.0.0.1"
     :initform "127.0.0.1"

-    :accessor ip
+    :accessor host

     :type simple-string
     :type simple-string

-    :documentation "The IP address of the Memcached server this instance represents")
+    :documentation "The host name of the Memcached server for this instance.")

    (port
     :initarg :port
     :initform 11211
    (port
     :initarg :port
     :initform 11211

diff --git a/src/compress.lisp b/src/compress.lisp
new file mode 100644 (file)
index 0000000..172ff6d
--- /dev/null
+++ b/src/compress.lisp
@@ -0,0 +1,33 @@
+;; -*- Mode: Common-Lisp -*-
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; FILE IDENTIFICATION
+;;
+;; Name:          compress.lisp
+;; Purpose:       Uses zlib to compress and uncompress vectors of octets
+;; Author:        Kevin Rosenberg
+;; Date Started:  July 2011
+;;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+(in-package #:memstore)
+
+;; KMR: zlib gives compression errors on random tests:
+;;    indices outside of input array size
+;; Using salza2 as default compressor
+
+(defun compress-zlib (data)
+  (zlib:compress data :fixed))
+
+(defun compress (data)
+  (let ((comp (salza2:compress-data
+               data
+               (make-instance 'salza2:zlib-compressor))))
+    (values comp (length comp))))
+
+;; KMR: zlib appears to works fine with salza2 zlib compressor, but
+;; moving to chipz as default decompressor as appears better supported
+(defun uncompress-zlib (data)
+  (zlib:uncompress data))
+
+(defun uncompress (data)
+  (chipz:decompress nil 'chipz:zlib data))