From 91a4befe34e15ab6c4c74901b3a16ec370c5de02 Mon Sep 17 00:00:00 2001 From: "Kevin M. Rosenberg" Date: Tue, 5 Jul 2011 08:18:49 -0600 Subject: [PATCH] Updates for release 1.1 --- debian/README.Debian | 4 +-- debian/changelog | 6 +++++ debian/docs | 3 ++- doc/memcache.html | 55 ++++++++++++++++++++++++++++++++++++++++++ doc/memstore.html | 55 ++++++++++++++++++++++++++++++++++++++++++ memcache/memcache.lisp | 8 +++--- memcache/specials.lisp | 8 +++--- src/compress.lisp | 33 +++++++++++++++++++++++++ 8 files changed, 161 insertions(+), 11 deletions(-) create mode 100644 doc/memcache.html create mode 100644 doc/memstore.html create mode 100644 src/compress.lisp diff --git a/debian/README.Debian b/debian/README.Debian index a1e8739..a9a9165 100644 --- a/debian/README.Debian +++ b/debian/README.Debian @@ -1,7 +1,7 @@ This library also requires the Common Lisp -cl-store and zlib libraries. Currently, these +cl-store, salza2, and chipz libraries. Currently, these libraries will have to be manually downloaded and installed as Debian does not have those libraries in its repository. - -- Kevin M. Rosenberg , Fri, 1 Jul 2011 23:15:07 -0600 + -- Kevin M. Rosenberg , Tue, 5 Jul 2011 08:14:42 -0600 diff --git a/debian/changelog b/debian/changelog index a2c9c20..5b1f882 100644 --- a/debian/changelog +++ b/debian/changelog @@ -1,3 +1,9 @@ +cl-memstore (1.1.0-1) unstable; urgency=low + + * New upstream + + -- Kevin M. Rosenberg Tue, 05 Jul 2011 08:17:22 -0600 + cl-memstore (1.0-1) unstable; urgency=low * Initial version diff --git a/debian/docs b/debian/docs index 49d454d..84a7994 100644 --- a/debian/docs +++ b/debian/docs @@ -1,3 +1,4 @@ -README.md +doc/memstore.html +doc/memcache.html debian/README.Debian diff --git a/doc/memcache.html b/doc/memcache.html new file mode 100644 index 0000000..2352e11 --- /dev/null +++ b/doc/memcache.html @@ -0,0 +1,55 @@ +

Memcache

+ +

Library for memcached protocol

+ +

Author: Kevin Rosenberg kevin@rosenberg.net, based on the +cl-memcached library by Abhijit 'quasi' Rao and +Chaitanya Gupta .

+ +

Date Started: July 1, 2011

+ +

Overview

+ +

This package is based on the cl-memcached library. It is +substantially modified for use with the +memstore library. The primary areas of +additional functionality are:

+ +
    +

  • Support for flags field with get and set functions. +This is required as memstore stores bit flags denoting +how the data is serialized. That information is required +to deserialize the data.

    • +

  • Support for additional memcached functionality, such as +the gets command for retrieving CAS identifiers. The CAS +unique ID is used for the added :cas storage command. +Other storage commands newly supported are :append and +:prepend.

    • +

  • All communication now uses mc-send-command function with +transparently supports writing strings with write-byte. +This allows usocket to be used on other Lisp implementations +besides AllegroCL. Because cl-memcached used write-string +with usocket-stream, only AllegroCL was supported. +By sending all data as (unsigned-byte 8), all Lisp implementions +supported by usocket are now supported with memcached.

    • +

  • Encapsulated reading and writing to socket stream to avoid +handling \#return characters in high-level code.

    • +

  • Changes to support the change in statistics fields with membase. +Some fields were no longer present. Also, membase 1.7 has +187 statistics fields versus the 20 fields supported in +cl-memcached. New function mc-get-stat allows to retrieving +any statistics field by name.

    • +

  • More robust print-object functions to avoid errors if fields +in statistics are not present.

    • +

  • Removed compatibility functions in compat.lisp by using the +kmrcl library to provide those +functions as well as utilitizing other kmrcl functions to simplify +code.

    • +

  • Added functions to support all memcached API commands, +such as flush_all and version.

    • +

  • Support for the moreply command argument accepted by +many commands.

    • +

  • Support the noreply argument that many API commands accept.

    • +

  • Write nearly the entire code base for improved +clarity, robustness, and efficiency.

    • +
diff --git a/doc/memstore.html b/doc/memstore.html new file mode 100644 index 0000000..62fbbb2 --- /dev/null +++ b/doc/memstore.html @@ -0,0 +1,55 @@ +

Memstore

+ +

A high-level interface for the memcached and membase servers

+ +

Written by Kevin M. Rosenberg kevin@rosenberg.net

+ +

Downloads

+ + + +

Prerequisites

+ +

This library incorporates a heavily modified version of cl-memcached +version 0.4.1. The primary need for the modification is to support the +flags field supported by the memcached server. The flag field is +used to store information required to deserialize the object data from +memcached. In addition to adding the flags field, the library has +been heavily refactored. For more information on the changes in +memcache, see memcache.

+ +

This library also requires Common Lisp libraries of +kmrcl, +cl-store, +flexi-streams, +and zlib.

+ +

Overview

+ +

Memstore allows efficient storing of simple objects as well as +easy storing of complex objects and optional compression.

+ +

When storing an object, if the object is a string then that is +directly written to the memcached server. For non-strings, an attempt +to made to write the object to a string with *print-readably* bound +to t. If that succeeds, then the string is converted to a vector of +octets. If that fails, the cl-store is used to serialize that object +to a vector of octets.

+ +

Next, optional compression is applied to the octets. First, the +*compression-enabled* flag is checked to see if compression is +enabled. Next, the length of the objects is compared to +*compression-threshold*. Only objects larger than +*compression-threshold* will be compressed. For objects that qualify +for compression, the size of the compressed object is compared to the +length of the uncompressed object to decide if the object is shrunk +enough to make the compression worthwhile.

+ +

The flags parameter to cl-memcached stores whether cl-store or +write-to-string is used to serialize the object and whether +compression is applied. mem-restore uses those flags to determine +how to reconstruct the object.

diff --git a/memcache/memcache.lisp b/memcache/memcache.lisp index c5593cb..231065b 100644 --- a/memcache/memcache.lisp +++ b/memcache/memcache.lisp @@ -34,7 +34,7 @@ (print-unreadable-object (mc stream :type t :identity t) (format stream "~A on ~A:~A ~AMB" (when (slot-boundp mc 'name) (name mc)) - (when (slot-boundp mc 'ip) (ip mc)) + (when (slot-boundp mc 'host) (host mc)) (when (slot-boundp mc 'port) (port mc)) (when (and (slot-boundp mc 'memcached-server-storage-size) (numberp (slot-value mc 'memcached-server-storage-size))) @@ -53,10 +53,10 @@ (setf (slot-value memcache 'memcached-server-storage-size) (mc-stats-limit-maxbytes stats)) (setf (slot-value memcache 'memcached-server-storage-size) -1)))) -(defun make-memcache-instance (&key (ip "127.0.0.1") (port 11211) +(defun make-memcache-instance (&key (host "127.0.0.1") (port 11211) (name "Memcache") (pool-size 5)) "Creates an instance of class MEMCACHE which represents a memcached server." - (make-instance 'memcache :name name :ip ip :port port :pool-size pool-size)) + (make-instance 'memcache :name name :host host :port port :pool-size pool-size)) (defmacro with-pool-maybe ((stream memcache use-pool) &body body) @@ -414,7 +414,7 @@ information about each slot" (mc-pool-grow memcache)) (defun mc-make-pool-item (&key (memcache *memcache*)) - (handler-case (usocket:socket-connect (ip memcache) (port memcache) :element-type '(unsigned-byte 8)) + (handler-case (usocket:socket-connect (host memcache) (port memcache) :element-type '(unsigned-byte 8)) (usocket:socket-error (e) (error 'memcached-server-unreachable :error e)) (error (e) (error 'cannot-make-pool-object :error e)))) diff --git a/memcache/specials.lisp b/memcache/specials.lisp index 2006f37..5a0f9a8 100644 --- a/memcache/specials.lisp +++ b/memcache/specials.lisp @@ -97,12 +97,12 @@ limit-maxbytes mc-stats-limit-maxbytes Number of bytes thi :reader name :type simple-string :documentation "Name of this Memcache instance") - (ip - :initarg :ip + (host + :initarg :host :initform "127.0.0.1" - :accessor ip + :accessor host :type simple-string - :documentation "The IP address of the Memcached server this instance represents") + :documentation "The host name of the Memcached server for this instance.") (port :initarg :port :initform 11211 diff --git a/src/compress.lisp b/src/compress.lisp new file mode 100644 index 0000000..172ff6d --- /dev/null +++ b/src/compress.lisp @@ -0,0 +1,33 @@ +;; -*- Mode: Common-Lisp -*- +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; +;; FILE IDENTIFICATION +;; +;; Name: compress.lisp +;; Purpose: Uses zlib to compress and uncompress vectors of octets +;; Author: Kevin Rosenberg +;; Date Started: July 2011 +;; +;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; + +(in-package #:memstore) + +;; KMR: zlib gives compression errors on random tests: +;; indices outside of input array size +;; Using salza2 as default compressor + +(defun compress-zlib (data) + (zlib:compress data :fixed)) + +(defun compress (data) + (let ((comp (salza2:compress-data + data + (make-instance 'salza2:zlib-compressor)))) + (values comp (length comp)))) + +;; KMR: zlib appears to works fine with salza2 zlib compressor, but +;; moving to chipz as default decompressor as appears better supported +(defun uncompress-zlib (data) + (zlib:uncompress data)) + +(defun uncompress (data) + (chipz:decompress nil 'chipz:zlib data)) -- 2.34.1