This library also requires the Common Lisp
-cl-store and zlib libraries. Currently, these
+cl-store, salza2, and chipz libraries. Currently, these
libraries will have to be manually downloaded
and installed as Debian does not have those
libraries in its repository.
- -- Kevin M. Rosenberg <kmr@debian.org>, Fri, 1 Jul 2011 23:15:07 -0600
+ -- Kevin M. Rosenberg <kmr@debian.org>, Tue, 5 Jul 2011 08:14:42 -0600
+cl-memstore (1.1.0-1) unstable; urgency=low
+
+ * New upstream
+
+ -- Kevin M. Rosenberg <kmr@debian.org> Tue, 05 Jul 2011 08:17:22 -0600
+
cl-memstore (1.0-1) unstable; urgency=low
* Initial version
-README.md
+doc/memstore.html
+doc/memcache.html
debian/README.Debian
--- /dev/null
+<h1>Memcache</h1>
+
+<p><em>Library for memcached protocol</em></p>
+
+<p>Author: Kevin Rosenberg <a href="mailto:kevin@rosenberg.net">kevin@rosenberg.net</a>, based on the
+<code>cl-memcached</code> library by Abhijit 'quasi' Rao and
+Chaitanya Gupta .</p>
+
+<p>Date Started: July 1, 2011</p>
+
+<h2>Overview</h2>
+
+<p>This package is based on the <code>cl-memcached</code> library. It is
+substantially modified for use with the
+<a href="http://memstore.b9.com">memstore</a> library. The primary areas of
+additional functionality are:</p>
+
+<ul>
+<li><p>Support for flags field with get and set functions.
+This is required as memstore stores bit flags denoting
+how the data is serialized. That information is required
+to deserialize the data.</p></li>
+<li><p>Support for additional memcached functionality, such as
+the gets command for retrieving CAS identifiers. The CAS
+unique ID is used for the added <code>:cas</code> storage command.
+Other storage commands newly supported are <code>:append</code> and
+<code>:prepend</code>.</p></li>
+<li><p>All communication now uses <code>mc-send-command</code> function with
+transparently supports writing strings with <code>write-byte</code>.
+This allows <code>usocket</code> to be used on other Lisp implementations
+besides AllegroCL. Because cl-memcached used <code>write-string</code>
+with usocket-stream, only AllegroCL was supported.
+By sending all data as (unsigned-byte 8), all Lisp implementions
+supported by <code>usocket</code> are now supported with <code>memcached</code>.</p></li>
+<li><p>Encapsulated reading and writing to socket stream to avoid
+handling \#return characters in high-level code.</p></li>
+<li><p>Changes to support the change in statistics fields with membase.
+Some fields were no longer present. Also, membase 1.7 has
+187 statistics fields versus the 20 fields supported in
+<code>cl-memcached</code>. New function <code>mc-get-stat</code> allows to retrieving
+any statistics field by name.</p></li>
+<li><p>More robust <code>print-object</code> functions to avoid errors if fields
+in statistics are not present.</p></li>
+<li><p>Removed compatibility functions in <code>compat.lisp</code> by using the
+<a href="http://gitpub.b9.com/kmrcl.git"><code>kmrcl</code></a> library to provide those
+functions as well as utilitizing other <code>kmrcl</code> functions to simplify
+code.</p></li>
+<li><p>Added functions to support all memcached API commands,
+such as <code>flush_all</code> and <code>version</code>.</p></li>
+<li><p>Support for the <code>moreply</code> command argument accepted by
+many commands.</p></li>
+<li><p>Support the <code>noreply</code> argument that many API commands accept.</p></li>
+<li><p>Write nearly the entire code base for improved
+clarity, robustness, and efficiency.</p></li>
+</ul>
--- /dev/null
+<h1>Memstore</h1>
+
+<p><em>A high-level interface for the memcached and membase servers</em></p>
+
+<p>Written by Kevin M. Rosenberg <a href="mailto:kevin@rosenberg.net">kevin@rosenberg.net</a></p>
+
+<h2>Downloads</h2>
+
+<ul>
+<li>The web site for memstore is <a href="http://memstore.b9.com/">http://memstore.b9.com/</a></li>
+<li>A git repository is available at <a href="http://gitpub.b9.com/memstore.git">http://gitpub.b9.com/memstore.git</a></li>
+<li>Releases are available for download at <a href="http://files.b9.com/memstore/">http://files.b9.com/memstore/</a></li>
+</ul>
+
+<h2>Prerequisites</h2>
+
+<p>This library incorporates a heavily modified version of <code>cl-memcached</code>
+version 0.4.1. The primary need for the modification is to support the
+<code>flags</code> field supported by the memcached server. The flag field is
+used to store information required to deserialize the object data from
+memcached. In addition to adding the <code>flags</code> field, the library has
+been heavily refactored. For more information on the changes in
+memcache, see <a href="memcache.html">memcache</a>.</p>
+
+<p>This library also requires Common Lisp libraries of
+<a href="http://gitpub.b9.com/kmrcl.git">kmrcl</a>,
+<a href="http://common-lisp.net/project/cl-store/">cl-store</a>,
+<a href="http://weitz.de/flexi-streams/">flexi-streams</a>,
+and <a href="http://common-lisp.net/project/zlib/">zlib</a>.</p>
+
+<h2>Overview</h2>
+
+<p>Memstore allows efficient storing of simple objects as well as
+easy storing of complex objects and optional compression.</p>
+
+<p>When storing an object, if the object is a string then that is
+directly written to the memcached server. For non-strings, an attempt
+to made to write the object to a string with <code>*print-readably*</code> bound
+to <code>t</code>. If that succeeds, then the string is converted to a vector of
+octets. If that fails, the <code>cl-store</code> is used to serialize that object
+to a vector of octets.</p>
+
+<p>Next, optional compression is applied to the octets. First, the
+<code>*compression-enabled*</code> flag is checked to see if compression is
+enabled. Next, the length of the objects is compared to
+<code>*compression-threshold*</code>. Only objects larger than
+<code>*compression-threshold*</code> will be compressed. For objects that qualify
+for compression, the size of the compressed object is compared to the
+length of the uncompressed object to decide if the object is shrunk
+enough to make the compression worthwhile.</p>
+
+<p>The <code>flags</code> parameter to cl-memcached stores whether cl-store or
+write-to-string is used to serialize the object and whether
+compression is applied. <code>mem-restore</code> uses those flags to determine
+how to reconstruct the object.</p>
(print-unreadable-object (mc stream :type t :identity t)
(format stream "~A on ~A:~A ~AMB"
(when (slot-boundp mc 'name) (name mc))
- (when (slot-boundp mc 'ip) (ip mc))
+ (when (slot-boundp mc 'host) (host mc))
(when (slot-boundp mc 'port) (port mc))
(when (and (slot-boundp mc 'memcached-server-storage-size)
(numberp (slot-value mc 'memcached-server-storage-size)))
(setf (slot-value memcache 'memcached-server-storage-size) (mc-stats-limit-maxbytes stats))
(setf (slot-value memcache 'memcached-server-storage-size) -1))))
-(defun make-memcache-instance (&key (ip "127.0.0.1") (port 11211)
+(defun make-memcache-instance (&key (host "127.0.0.1") (port 11211)
(name "Memcache") (pool-size 5))
"Creates an instance of class MEMCACHE which represents a memcached server."
- (make-instance 'memcache :name name :ip ip :port port :pool-size pool-size))
+ (make-instance 'memcache :name name :host host :port port :pool-size pool-size))
(defmacro with-pool-maybe ((stream memcache use-pool) &body body)
(mc-pool-grow memcache))
(defun mc-make-pool-item (&key (memcache *memcache*))
- (handler-case (usocket:socket-connect (ip memcache) (port memcache) :element-type '(unsigned-byte 8))
+ (handler-case (usocket:socket-connect (host memcache) (port memcache) :element-type '(unsigned-byte 8))
(usocket:socket-error (e) (error 'memcached-server-unreachable :error e))
(error (e) (error 'cannot-make-pool-object :error e))))
:reader name
:type simple-string
:documentation "Name of this Memcache instance")
- (ip
- :initarg :ip
+ (host
+ :initarg :host
:initform "127.0.0.1"
- :accessor ip
+ :accessor host
:type simple-string
- :documentation "The IP address of the Memcached server this instance represents")
+ :documentation "The host name of the Memcached server for this instance.")
(port
:initarg :port
:initform 11211
--- /dev/null
+;; -*- Mode: Common-Lisp -*-
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+;; FILE IDENTIFICATION
+;;
+;; Name: compress.lisp
+;; Purpose: Uses zlib to compress and uncompress vectors of octets
+;; Author: Kevin Rosenberg
+;; Date Started: July 2011
+;;
+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
+
+(in-package #:memstore)
+
+;; KMR: zlib gives compression errors on random tests:
+;; indices outside of input array size
+;; Using salza2 as default compressor
+
+(defun compress-zlib (data)
+ (zlib:compress data :fixed))
+
+(defun compress (data)
+ (let ((comp (salza2:compress-data
+ data
+ (make-instance 'salza2:zlib-compressor))))
+ (values comp (length comp))))
+
+;; KMR: zlib appears to works fine with salza2 zlib compressor, but
+;; moving to chipz as default decompressor as appears better supported
+(defun uncompress-zlib (data)
+ (zlib:uncompress data))
+
+(defun uncompress (data)
+ (chipz:decompress nil 'chipz:zlib data))
projects / memstore.git / commitdiff