Overriden `prepare_query_value` breaks current SequenceField behavior

[cachemem]

I have a class with sequence field defined like this:

class A(Document):
    id = me.SequenceField(
        primary_key=True,
        value_decorator='{:x}'.format  # to hex
    )

In mongoengine 0.8 it worked like this --- sequence field produced new integer and value_decorator converted it to hex string and it was stored in database like this. And this converted hex string value was used everywhere. "Raw" integer sequence value was used only for generating new id.

A.objects.get(id='1ab')

In version 0.9 if i try to get object of class A like this I will get an error:

...
  File ".../lib/python2.7/site-packages/mongoengine/queryset/transform.py", line 94, in query
    value = field.prepare_query_value(op, value)
  File ".../lib/python2.7/site-packages/mongoengine/fields.py", line 1784, in prepare_query_value
    return self.value_decorator(value)
ValueError: Unknown format code 'x' for object of type 'unicode'

I understand what was the logic behind this. For default value_decorator value for SequenceField can be passed as both string ('10') or integer (10), but it doesn't work for specific decorators like these.

I don't know what is the best course of action: just don't use override if self.value_decorator is not equal to self.VALUE_DECORATOR or create specific option like "decorate_on_query". I'm asking an advice on that and then I'd like to create a pull request.

[MRigal]

I have difficulties to reflect deep enough on the problem, even if you describe it really precisely. Maybe you could try to submit a PR with the first proposal, which sounds the easiest, with nice test(s) and we could discuss it

[cachemem]

OK, will do as soon as I can.

[warvariuc]

Hi @MRigal
How to overcome this issue? Looks like the decorator is called not only when a new sequence number is generated, but also when getting the fields from the database.
This looks like a bug.
The docs say:

Use any callable as value_decorator to transform calculated counter into any value suitable for your needs, e.g. string or hexadecimal representation of the default integer counter value.

And current behavior doesn't match the docs. If it is correct, what function should be used in this example as value_decorator to have A.id as a hex string primary key?

class A(Document):
    id = me.SequenceField(
        primary_key=True,
        value_decorator='{:x}'.format  # to hex
    )

Because in this form it creates new objects ok, but fails to query them:

>>> cc=Course.objects.create(title='asdf', owner='sadf')

>>> cc
>>> Course(id='99f', title=u'asdf', owner=u'sadf', ...)

>>> Course.objects.get(owner='sadf')
>>> Course(id=u'99f', title=u'asdf', owner=u'sadf', ...)

>>> Course.objects.get(id='99f')

.../local/lib/python2.7/site-packages/mongoengine/queryset/base.py in get(self, *q_objs, **query)
    236 
    237         try:
--> 238             result = queryset.next()
    239         except StopIteration:
    240             msg = ("%s matching query does not exist."

.../local/lib/python2.7/site-packages/mongoengine/queryset/base.py in next(self)
   1381             raise StopIteration
   1382 
-> 1383         raw_doc = self._cursor.next()
   1384         if self._as_pymongo:
   1385             return self._get_as_pymongo(raw_doc)

.../local/lib/python2.7/site-packages/mongoengine/queryset/base.py in _cursor(self)
   1447         if self._cursor_obj is None:
   1448 
-> 1449             self._cursor_obj = self._collection.find(self._query,
   1450                                                      **self._cursor_args)
   1451             # Apply where clauses to cursor

.../local/lib/python2.7/site-packages/mongoengine/queryset/base.py in _query(self)
   1481     def _query(self):
   1482         if self._mongo_query is None:
-> 1483             self._mongo_query = self._query_obj.to_query(self._document)
   1484             if self._class_check and self._initial_query:
   1485                 if "_cls" in self._mongo_query:

.../local/lib/python2.7/site-packages/mongoengine/queryset/visitor.py in to_query(self, document)
     88     def to_query(self, document):
     89         query = self.accept(SimplificationVisitor())
---> 90         query = query.accept(QueryCompilerVisitor(document))
     91         return query
     92 

.../local/lib/python2.7/site-packages/mongoengine/queryset/visitor.py in accept(self, visitor)
    153 
    154     def accept(self, visitor):
--> 155         return visitor.visit_query(self)
    156 
    157     @property

.../local/lib/python2.7/site-packages/mongoengine/queryset/visitor.py in visit_query(self, query)
     76 
     77     def visit_query(self, query):
---> 78         return transform.query(self.document, **query.query)
     79 
     80 

.../local/lib/python2.7/site-packages/mongoengine/queryset/transform.py in query(_doc_cls, **query)
     93                         value = field
     94                 else:
---> 95                     value = field.prepare_query_value(op, value)
     96 
     97                     if isinstance(field, CachedReferenceField) and value:

.../local/lib/python2.7/site-packages/mongoengine/fields.py in prepare_query_value(self, op, value)
   1758         values passed as string, the base implementation returns the value as is.
   1759         """
-> 1760         return self.value_decorator(value)
   1761 
   1762     def to_python(self, value):

ValueError: Unknown format code 'x' for object of type 'str'

[warvariuc]

My workaround is:

    id = me.SequenceField(
        primary_key=True,
        value_decorator=lambda count: '{:x}'.format(count) if isinstance(count, (int, long)) else count  # to hex
    )

But it looks like there is a bug.