Mongoengine makes conflicting modifications to embedded documents

[bremac]

Some embedded document structures trip up delta tracking and cause saves to fail. For example, take the following script:

import mongoengine

class EmbeddedDoc(mongoengine.EmbeddedDocument):
    content = mongoengine.StringField()
    docs = mongoengine.ListField(mongoengine.EmbeddedDocumentField('EmbeddedDoc'))

class Doc(mongoengine.Document):
    docs = mongoengine.ListField(mongoengine.EmbeddedDocumentField(EmbeddedDoc))

mongoengine.connect('test')
Doc.objects.delete()

d = Doc()
d.docs = [EmbeddedDoc(content='1', docs=[])]
d.save()

d = Doc.objects[0]
e = EmbeddedDoc()
e.content = '2'
d.docs[0].docs.insert(0, e)
d.save()

The final call to save() fails with an error:

Traceback (most recent call last):
  File "break-mongoengine.py", line 24, in <module>
    d.save()
  File "/home/sm/Code/ma/venv/src/mongoengine/mongoengine/document.py", line 372, in save
    raise OperationError(message % six.text_type(err))
mongoengine.errors.OperationError: Could not save document (Cannot update 'docs.0.docs.0.content' and 'docs.0.docs' at the same time)

[wojcikstefan]

Thanks @bremac, looks like a bug indeed. Just curious, if you do append(e) instead of an insert(0, e), does it work as expected?

[bremac]

Nope, append does not work either.

[userlocalhost]

Hi,

I wanna challenge to solve this.
As a first step, I'll show the cause of this problem and tentative way to avoid it.

Because mongoengine simultaneously performs a plurality of update processes for the same field internally, an exception of pymongo.errors.OperationFailure is occur.
(Here is explained in detail about what this exception is and meaning of it.)

In this case, OperationFailure is occur because the following two update processes will be executed.

1. Set '2' to the content attribute of the hash which is contained in the first element of the docs list.
2. Set the hash which is set '2' in the content attribute to the first element of docs list.

You can avoid this problem by following. Then, the first update processing will be omitted.

--- /tmp/before	2017-03-21 12:07:03.000000000 +0900
+++ /tmp/after	2017-03-21 12:07:22.000000000 +0900
@@ -15,7 +15,6 @@
 d.save()
 
 d = Doc.objects[0]
-e = EmbeddedDoc()
-e.content = '2'
+e = EmbeddedDoc(content='2')
 d.docs[0].docs.insert(0, e)
 d.save()

I'll try to find a solution and make a PR.

Thank you.

[wojcikstefan]

Building on top of @bremac's example, the main issue is that setting a value during and after instantiation of the embedded doc produces different update docs, the latter causing an update conflict:

In [8]: d = Doc.objects[0]
   ...: e = EmbeddedDoc(content='2')
   ...: d.docs[0].docs.insert(0, e)
   ...: pprint.pprint(d._get_update_doc())
   ...:
{'$set': {u'docs.0.docs': [SON([('content', u'2'), ('docs', [])])]}}

In [9]: d = Doc.objects[0]
   ...: e = EmbeddedDoc()
   ...: e.content = '2'
   ...: d.docs[0].docs.insert(0, e)
   ...: pprint.pprint(d._get_update_doc())
   ...:
{'$set': {u'docs.0.docs': [SON([('content', u'2'), ('docs', [])])],
          'docs.0.docs.0.content': u'2'}}

[bptarpley]

I'm having this same issue with release 0.19.1. I have an embedded document as a mapfield inside another embedded document inside a mapfield, ie:

class File(EmbeddedDocument):
  path = StringField()

class Page(EmbeddedDocument):
  files = Mapfield(EmbeddedDocumentField(File))

class Doc(Document):
    pages = Mapfield(EmbeddedDocumentField(Page))

When attempting:

f = File()
f.path = '/some/path'
p = Page()
p.files['1'] = f
d = Doc.objects(id='someid')[0]
d.pages['1'] = p
d.save()

I get:

mongoengine.errors.OperationError: Could not save document (Cannot update 'pages.1.files.1' and 'pages.1.files.1.path' at the same time)

This was working in earlier releases... Please advise :)

[bptarpley]

FWIW, am temporarily working around this issue like so:

f = File()
f.path = '/some/path'
p = Page()
p.files['1'] = f
d = Doc.objects(id='someid')[0]
d.modify(**{'set__pages__1': p})

[harshmishraintg]

Is this still open as I am facing issue in 0.16.0