Redis Collections

Redis Collections are a set of basic Python collections backed by Redis.

Redis is a great key-value storage. There is well-designed client for Python, but sometimes working with it seems to be too low-level. You just call methods with names of corresponding Redis commands. Such approach is great when dealing with cutting edge software tasks, but if you write just a simple app or command line script for your own, you might appreciate a bit different interface.

The aim of this library is to provide such interface when dealing with collections. Redis has support for several types: strings, hashes, lists, sets, sorted sets. Why not to bring them to Python in a pythonic way?

>>> from redis_collections import Dict
>>> d = Dict()
>>> d['answer'] = 42
>>> d
<redis_collections.Dict at fe267c1dde5d4f648e7bac836a0168fe {'answer': 42}>
>>> d.items()
[('answer', 42)]

In Redis you will see can see a hash structure under key fe267c1dde5d4f648e7bac836a0168fe. That structure stores a field and value that corresponds to {'answer': 42}. The value is pickled, because Redis can store only strings.

On the Python side, you can do most anything you can do with standard dict instances:

>>> d.update({'hasek': 39, 'jagr': 68})
>>> d
<redis_collections.Dict at fe267c1dde5d4f648e7bac836a0168fe {'answer': 42, 'jagr': 68, 'hasek': 39}>
>>> del d['answer']
>>> d
<redis_collections.Dict at fe267c1dde5d4f648e7bac836a0168fe {'jagr': 68, 'hasek': 39}>

“Write” operations atomically change data in Redis.

Installation

Current version is 0.2.1.

pip install redis-collections

Persistence

When creating the Dict object, your collection gets a unique Redis key. If you keep this key, you can summon your collection any time in the future:

>>> d.key
fe267c1dde5d4f648e7bac836a0168fe
>>> Dict(key='fe267c1dde5d4f648e7bac836a0168fe')
<redis_collections.Dict at fe267c1dde5d4f648e7bac836a0168fe {'jagr': 68, 'hasek': 39}>

In case you wish to wipe all its data, use clear() method, which is available to all collections provided by this library:

>>> d.clear()
>>> d.items()
[]

If I look to my Redis, key fe267c1dde5d4f648e7bac836a0168fe completely disappeared.

Note

If you provide your own key string, a collection will be successfully created. If there is no key corresponding in Redis, it will be created and initialized as an empty collection. This means you can set up your own way of assigning unique keys dependent on your other code. For example, by using IDs of records from your relational database you can have exactly one unique collection in Redis for every record from your SQL storage.

Redis Connection

By default, collections use a new Redis connection with its default values, which is highly inefficient, but needs no configuration. If you wish to use your own Redis instance, pass it in redis keyword argument:

>>> from redis import StrictRedis
>>> r = StrictRedis()
>>> d = Dict(redis=r)
>>> l = List(redis=r)  # using the same connection as Dict above

There are several operations between collections resulting into creation of new instances of Redis Collections. These new instances always use the same Redis connection as the original object:

>>> from redis import StrictRedis
>>> from redis_collections import List
>>> r = StrictRedis()
>>> l = List([1, 2], redis=r)
>>> l
<redis_collections.List at 196e407f8fc142728318a999ec821368 [1, 2]>
>>> l + [4, 5, 6]  # result is using the same connection
<redis_collections.List at 7790ef98639043c9abeacc80c2de0b93 [1, 2, 4, 5, 6]>

Synchronization

Storing a mutable object like a list in a Dict can lead to surprising behavior. Because of Python semantics, it’s impossible to automatically write to Redis when such an object is retrieved and modified.

>>> d = Dict({'key': [1, 2]})  # Store a mutable object
>>> d['key'].append(3)  # Retrieve and modify the object
>>> d['key']  # Retrieve the object from Redis again
[1, 2]

To work with such objects you may use a Dict with writeback enabled. This will keep a local cache that is flushed to Redis when the sync method is called.

>>> d = Dict({'key': [1, 2]}, writeback=True)
>>> d['key'].append(3)
>>> d['key']  # Modifications are retrieved from the cache
[1, 2, 3]
>>> d.sync()  # Flush cache to Redis

You may also use a with block to automatically call the sync method.

>>> with Dict({'key': [1, 2]}) as d:
...     d['key'].append(3)
>>> d['key']  # Changes were automatically synced
[1, 2, 3]

The writeback option is automatically enabled for DefaultDict objects.

Pickling

If you don’t like the standard way of data serialization made by pickle, you may override the _pickle and _unpickle methods of the collection classes. Using other serializers may limit the objects you can store or retrieve.

Known issues

  • Storing objects that have the same hash (such as the float 1.0 and the int 1) in a Set can lead to surprising behavior. They can both be retrieved, unlike with a native Python set. See issue 49.
  • Support for Python 3 is in progress. Please report any issues you find.

Philosophy

  • All operations are atomic.

    Warning

    If an operation has race conditions, it is a bug. Please, report it.

  • Redis Collections stick to API of the original data structures known from Python standard library. To have the same (expected) behaviour is considered to be more important than efficiency.

    Warning

    If a collection has the method you want to use, but it does not behave as the original built-in and it does not raise NotImplementedError, then it is a bug. Please, report it.

    If there is more efficient approach than the one complying with the model interface, new method exposing this approach should be introduced.

  • Cases where different than standard approach would lead to better efficiency are mentioned and highlighted in API documentation as notes. Known incompatibilities with the original API are marked as warnings.

  • Behavior of nested Redis Collections containing other Redis Collections is undefined. It is not recommended to create such structures. Use collection of keys instead.

API Documentation

Redis Collections are composed of only several classes. All items listed below are exposed as public API, so you can (and you should) import them directly from redis_collections package.

base

class redis_collections.base.RedisCollection(data=None, redis=None, key=None)[source]

Abstract class providing backend functionality for all the other Redis collections.

__init__(data=None, redis=None, key=None)[source]
Parameters:
  • data – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

clear()[source]

Completely cleares the collection’s data.

copy(key=None)[source]

Return a copy of the collection.

Parameters:key (string) – Key of the new collection. Defaults to auto-generated.
key = None

Redis key of the collection.

redis = None

Redis client instance. StrictRedis object with default connection settings is used if not set by __init__().

dicts

Collections based on dict interface.

class redis_collections.dicts.Dict(*args, **kwargs)[source]

Mutable mapping collection aiming to have the same API as the standard mapping type, dict. See dict for further details. The Redis implementation is based on the hash type.

Warning

In comparing with original dict type, Dict does not implement methods viewitems(), viewkeys(), and viewvalues().

Note

Some operations, which are usually not used so often, can be more efficient than their “popular” equivalents. For example, get() should be preffered over the classic d[key] approach.

__contains__(key)[source]

Return True if key is present, else False.

__delitem__(key)[source]

Remove d[key] from dictionary. Raises a KeyError() if key is not in the map.

__getitem__(key)[source]

Return the item of dictionary with key key. Raises a KeyError if key is not in the map.

If a subclass of Dict defines a method __missing__(), if the key key is not present, the d[key] operation calls that method with the key key as argument. The d[key] operation then returns or raises whatever is returned or raised by the __missing__(key) call if the key is not present.

__init__(*args, **kwargs)[source]

Breakes the original dict API, because there is no support for keyword syntax. The only single way to create Dict object is to pass iterable or mapping as the first argument.

Parameters:
  • data (iterable or mapping) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (bool) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.
  • writeback – If True keep a local cache of changes for storing modifications to mutable values. Changes will be written to Redis after calling the sync method.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

Warning

As mentioned, Dict does not support following initialization syntax: d = Dict(a=1, b=2)

__iter__(pipe=None)[source]

Return an iterator over the keys of the dictionary.

__len__()[source]

Return the number of items in the dictionary.

Note

Due to implementation on Redis side, this method is inefficient. The time taken is varies with the number of keys in stored.

__setitem__(key, value)[source]

Set d[key] to value.

classmethod fromkeys(seq, value=None, **kwargs)[source]

Create a new dictionary with keys from seq and values set to value.

Note

fromkeys() is a class method that returns a new dictionary. value defaults to None. It is possible to specify additional keyword arguments to be passed to __init__() of the new object.

getmany(*keys)[source]

Return the value for keys. If particular key is not in the dictionary, return None.

items()[source]

Return a copy of the dictionary’s list of (key, value) pairs.

iter()[source]

Return an iterator over the keys of the dictionary. This is a shortcut for iterkeys().

iteritems()[source]

Return an iterator over the dictionary’s (key, value) pairs.

iterkeys()[source]

Return an iterator over the dictionary’s keys.

itervalues()[source]

Return an iterator over the dictionary’s values.

keys()[source]

Return a copy of the dictionary’s list of keys.

pop(key, default=<missing value>)[source]

If key is in the dictionary, remove it and return its value, else return default. If default is not given and key is not in the dictionary, a KeyError is raised.

popitem()[source]

Remove and return an arbitrary (key, value) pair from the dictionary.

popitem() is useful to destructively iterate over a dictionary, as often used in set algorithms. If the dictionary is empty, calling popitem() raises a KeyError.

setdefault(key, default=None)[source]

If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

update(other=None, **kwargs)[source]

Update the dictionary with the key/value pairs from other, overwriting existing keys. Return None.

update() accepts either another dictionary object or an iterable of key/value pairs (as tuples or other iterables of length two). If keyword arguments are specified, the dictionary is then updated with those key/value pairs: d.update(red=1, blue=2).

values()[source]

Return a copy of the dictionary’s list of values.

class redis_collections.dicts.Counter(*args, **kwargs)[source]

Mutable mapping collection aiming to have the same API as collections.Counter. See Counter for further details. The Redis implementation is based on the hash type.

Warning

Not available in Python 2.6.

Warning

Note that this Counter does not implement methods viewitems(), viewkeys(), and viewvalues(), which are available in Python 2.7’s version.

__delitem__(key)[source]

Like dict.__delitem__(), but does not raise KeyError for missing values.

__init__(*args, **kwargs)[source]

Breakes the original Counter API, because there is no support for keyword syntax. The only single way to create Counter object is to pass iterable or mapping as the first argument. Iterable is expected to be a sequence of elements, not a sequence of (key, value) pairs.

Parameters:
  • data (iterable or mapping) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

Warning

As mentioned, Counter does not support following initialization syntax: c = Counter(a=1, b=2)

elements(n=None)[source]

Return an iterator over elements repeating each as many times as its count. Elements are returned in arbitrary order. If an element’s count is less than one, elements() will ignore it.

most_common(n=None)[source]

Return a list of the n most common elements and their counts from the most common to the least. If n is not specified, most_common() returns all elements in the counter. Elements with equal counts are ordered arbitrarily.

subtract(other=None, **kwargs)[source]

Elements are subtracted from an iterable or from another mapping (or counter). Like dict.update() but subtracts counts instead of replacing them.

update(other=None, **kwargs)[source]

Elements are counted from an iterable or added-in from another mapping (or counter). Like dict.update() but adds counts instead of replacing them. Also, the iterable is expected to be a sequence of elements, not a sequence of (key, value) pairs.

class redis_collections.dicts.DefaultDict(*args, **kwargs)[source]

Mutable mapping collection aiming to have the same API as collections.defaultdict. See `defaultdict <https://docs.python.org/2/library/collections.html`_ for further details. The Redis implementation is based on the hash type.

Warning

Note that this DefaultDict does not implement methods viewitems(), viewkeys(), and viewvalues(), which are available in Python 2.7’s version.

__init__(*args, **kwargs)[source]

Breakes the original defaultdict API, because there is no support for keyword syntax. The only single way to create defaultdict object is to pass an iterable or mapping as the second argument.

Parameters:
  • default_factory (callable or None) – Used to provide default values for missing keys.
  • data (iterable or mapping) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

Warning

As mentioned, DefaultDict does not support following initialization syntax: d = DefaultDict(None, a=1, b=2)

lists

Collections based on list interface.

class redis_collections.lists.List(*args, **kwargs)[source]

Mutable sequence collection aiming to have the same API as the standard sequence type, list. See list for further details. The Redis implementation is based on the list type.

Warning

In comparing with original list type, List does not implement methods sort() and reverse().

Note

Some operations, which are usually not used so often, can be more efficient than their “popular” equivalents. Some operations are shortened in their capabilities and can raise NotImplementedError for some inputs (e.g. most of the slicing functionality).

__add__(values)[source]

Returns concatenation of the list and given iterable. New List instance is returned.

__delitem__(index)[source]

Item of index is deleted.

Warning

Slicing is generally not supported. Only empty lists are accepted if the operation leads into trimming:

del l[2:]
del l[:2]
del l[:]
__getitem__(index)[source]

Returns item of sequence on index. Origin of indexes is 0. Accepts also slicing.

Note

Due to implementation on Redis side, l[index] is not very efficient operation. If possible, use get(). Slicing without steps is efficient. Steps are implemented only on Python side.

__init__(*args, **kwargs)[source]
Parameters:
  • data (iterable) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (bool) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.
  • writeback – If True keep a local cache of changes for storing modifications to mutable values. Changes will be written to Redis after calling the sync method.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

__iter__()[source]

Return an iterator over the sequence.

__len__()[source]

Length of the sequence.

__mul__(n)[source]

Returns n copies of the list, concatenated. New List instance is returned.

__reversed__()[source]

Returns iterator for the sequence in reversed order.

__setitem__(index, value)[source]

Item of index is replaced by value.

Warning

Slicing is generally not supported. Only empty lists are accepted if the operation leads into trimming:

l[2:] = []
l[:2] = []
l[:] = []
append(value)[source]

Insert value at end of list.

count(value)[source]

Returns number of occurences of value.

Note

Implemented only on Python side.

extend(values)[source]

values are appended at the end of the list. Any iterable is accepted.

get(index, default=None)[source]

Return the value for index if index is not out of range, else default. If default is not given, it defaults to None, so that this method never raises a IndexError.

Note

Due to implementation on Redis side, this method of retrieving items is more efficient than classic approach over using the __getitem__() protocol.

index(value, start=None, stop=None)[source]

Returns index of the first occurence of value.

If start or stop are provided, returns the smallest index such that s[index] == value and start <= index < stop.

insert(index, value)[source]

Insert value before index. Can only work with index == 0.

pop(index=-1)[source]

Item on index is removed and returned.

Warning

Only indexes 0 and -1 (default) are supported, otherwise NotImplementedError is raised.

remove(value)[source]

Remove the first occurence of value.

sets

Collections based on set interface.

class redis_collections.sets.Set(*args, **kwargs)[source]

Mutable set collection aiming to have the same API as the standard set type. See set for further details. The Redis implementation is based on the set type.

__and__(*args)[source]

Return a new set with elements common to the set and the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:type of the first operand

Note

The same behavior as at __sub__() applies.

__contains__(elem)[source]

Test for membership of elem in the set.

__iand__(*args)[source]

Update the set, keeping only elements found in it and the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:None

Note

The same behavior as at __isub__() applies.

__init__(*args, **kwargs)[source]
Parameters:
  • data (iterable) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

__ior__(*args)[source]

Update the set, adding elements from the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:None

Note

The same behavior as at __isub__() applies.

__isub__(*args)[source]

Update the set, removing elements found in other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:None

Note

If other is instance of Set, operation is performed completely in Redis. Otherwise it’s performed in Python and results are sent to Redis.

__iter__()[source]

Return an iterator over elements of the set.

__ixor__(*args)[source]

Update the set, keeping only elements found in either set, but not in both.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:None

Note

The same behavior as at __isub__() applies.

__len__()[source]

Return cardinality of the set.

__or__(*args)[source]

Return a new set with elements from the set and the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:type of the first operand

Note

The same behavior as at __sub__() applies.

__sub__(*args)[source]

Return a new set with elements in the set that are not in the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:type of the first operand

Note

If other is instance of Set, operation is performed completely in Redis. Otherwise it’s performed in Python and results are sent to Redis.

__xor__(*args)[source]

Update the set, keeping only elements found in either set, but not in both.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:type of the first operand

Note

The same behavior as at __sub__() applies.

add(elem)[source]

Add element elem to the set. Returns False if elem was already present in the set.

Return type:boolean

Warning

Original add() in set returns no value.

difference(*others, **kwargs)[source]

Return a new set with elements in the set that are not in the others.

Parameters:
  • others – Iterables, each one as a single positional argument.
  • return_cls – Keyword argument, type of result, defaults to the same type as collection (Set, if not inherited).
Return type:

Set or collection of type specified in return_cls argument

Note

If all others are Set instances, operation is performed completely in Redis. If return_cls is provided, operation is still performed in Redis, but results are sent back to Python and returned with corresponding type. All other combinations are performed in Python and results are sent to Redis. See examples:

s1 = Set([1, 2])
s2 = Set([2, 3])
s3 = set([2, 3])  # built-in set

# Redis (whole operation)
s1.difference(s2, s2, s2)  # = Set

# Python (operation) → Redis (new key with Set)
s1.difference(s3)  # = Set

# Python (operation) → Redis (new key with Set)
s1.difference(s2, s3, s2)  # = Set

# Redis (operation) → Python (type conversion)
s1.difference(s2, return_cls=set)  # = set

# Redis (operation) → Python (type conversion)
s1.difference(s2, return_cls=list)  # = list

# Redis (operation) → Python → Redis (new key with List)
s1.difference(s2, return_cls=List)  # = List
difference_update(*others)[source]

Update the set, removing elements found in others.

Parameters:others – Iterables, each one as a single positional argument.
Return type:None

Note

If all others are Set instances, operation is performed completely in Redis. Otherwise it’s performed in Python and results are sent to Redis. See examples:

s1 = Set([1, 2])
s2 = Set([2, 3])
s3 = set([2, 3])  # built-in set

# Redis (whole operation)
s1.difference_update(s2, s2)  # = None

# Python (operation) → Redis (update)
s1.difference(s3)  # = None

# Python (operation) → Redis (update)
s1.difference(s2, s3, s2)  # = None
discard(elem)[source]

Remove element elem from the set if it is present.

intersection(*others, **kwargs)[source]

Return a new set with elements common to the set and all others.

Parameters:
  • others – Iterables, each one as a single positional argument.
  • return_cls – Keyword argument, type of result, defaults to the same type as collection (Set, if not inherited).
Return type:

Set or collection of type specified in return_cls argument

Note

The same behavior as at difference() applies.

intersection_update(*others)[source]

Update the set, keeping only elements found in it and all others.

Parameters:others – Iterables, each one as a single positional argument.
Return type:None

Note

The same behavior as at difference_update() applies.

issubset(other)[source]

Test whether every element in the set is in other.

Parameters:other – Any kind of iterable.
Return type:boolean
issuperset(other)[source]

Test whether every element in other is in the set.

Parameters:other – Any kind of iterable.
Return type:boolean
pop()[source]

Remove and return an arbitrary element from the set. Raises KeyError if the set is empty.

random_sample(k=1)[source]

Return a k length list of unique elements chosen from the set. Elements are not removed. Similar to random.sample() function from standard library.

Parameters:k – Size of the sample, defaults to 1.
Return type:list

Note

Argument k is supported only for Redis of version 2.6 and higher.

remove(elem)[source]

Remove element elem from the set. Raises KeyError if elem is not contained in the set.

symmetric_difference(other, **kwargs)[source]

Return a new set with elements in either the set or other but not both.

Parameters:
  • others – Any kind of iterable.
  • return_cls – Keyword argument, type of result, defaults to the same type as collection (Set, if not inherited).
Return type:

Set or collection of type specified in return_cls argument

Note

The same behavior as at difference() applies.

symmetric_difference_update(other)[source]

Update the set, keeping only elements found in either set, but not in both.

Parameters:others – Any kind of iterable.
Return type:None

Note

A bit different behavior takes place in comparing with the one described at difference_update(). Operation is always performed in Redis, regardless the types given. If others are instances of Set, the performance should be better as no transfer of data is necessary at all.

union(*others, **kwargs)[source]

Return a new set with elements from the set and all others.

Parameters:
  • others – Iterables, each one as a single positional argument.
  • return_cls – Keyword argument, type of result, defaults to the same type as collection (Set, if not inherited).
Return type:

Set or collection of type specified in return_cls argument

Note

The same behavior as at difference() applies.

update(*others)[source]

Update the set, adding elements from all others.

Parameters:others – Iterables, each one as a single positional argument.
Return type:None

Note

The same behavior as at difference_update() applies.

Maintainers

License: ISC

© 2013-? Honza Javorek <mail@honzajavorek>

This work is licensed under ISC license.

Indices and tables