-
-
Notifications
You must be signed in to change notification settings - Fork 78
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allows working copy of the Portal #1823
base: main
Are you sure you want to change the base?
Conversation
@wesleybl thanks for creating this Pull Request and helping to improve Plone! TL;DR: Finish pushing changes, pass all other checks, then paste a comment:
To ensure that these changes do not break other parts of Plone, the Plone test suite matrix needs to pass, but it takes 30-60 min. Other CI checks are usually much faster and the Plone Jenkins resources are limited, so when done pushing changes and all other checks pass either start all Jenkins PR jobs yourself, or simply add the comment above in this PR to start all the jobs automatically. Happy hacking! |
I tried to do a checkout test of the Portal in this class:
The test is as follows: def test_workingcopy_checkout_checkin_portal(self):
# We create the working copy
response = self.api_session.post(
"/@workingcopy",
)
self.assertEqual(response.status_code, 201) But I am getting error 500. I went to debug and the error is not in the restapi code. But in Zope. The error occurs when trying to execute this commit: https://github.com/zopefoundation/Zope/blob/5.10/src/ZPublisher/WSGIPublisher.py#L187 The error is a *** transaction.interfaces.TransactionFailedError: An operation previously failed, with traceback:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/home/user/.buildout/eggs/cp312/waitress-2.1.2-py3.12.egg/waitress/task.py", line 84, in handler_thread
task.service()
File "/home/user/.buildout/eggs/cp312/waitress-2.1.2-py3.12.egg/waitress/channel.py", line 428, in service
task.service()
File "/home/user/.buildout/eggs/cp312/waitress-2.1.2-py3.12.egg/waitress/task.py", line 168, in service
self.execute()
File "/home/user/.buildout/eggs/cp312/waitress-2.1.2-py3.12.egg/waitress/task.py", line 434, in execute
app_iter = self.channel.server.application(environ, start_response)
File "/home/user/.buildout/eggs/cp312/WebTest-3.0.0-py3.12.egg/webtest/http.py", line 82, in wrapper
return self.test_app(environ, start_response)
File "/home/user/.buildout/eggs/cp312/Zope-5.10-py3.12.egg/ZPublisher/httpexceptions.py", line 30, in __call__
return self.application(environ, start_response)
File "/home/user/.buildout/eggs/cp312/Zope-5.10-py3.12.egg/ZPublisher/WSGIPublisher.py", line 390, in publish_module
with transaction_pubevents(request, response):
File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
next(self.gen)
File "/home/user/.buildout/eggs/cp312/Zope-5.10-py3.12.egg/ZPublisher/WSGIPublisher.py", line 187, in transaction_pubevents
tm.commit()
File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_manager.py", line 254, in commit
return self.manager.commit()
File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_manager.py", line 133, in commit
return self.get().commit()
File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_transaction.py", line 280, in commit
t, v, tb = self._saveAndGetCommitishError()
File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_transaction.py", line 273, in commit
self._commitResources()
File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_transaction.py", line 456, in _commitResources
raise v.with_traceback(tb)
File "/home/user/.buildout/eggs/cp312/transaction-4.0-py3.12.egg/transaction/_transaction.py", line 430, in _commitResources
rm.commit(self)
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/Connection.py", line 492, in commit
self._commit_savepoint(transaction)
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/Connection.py", line 1033, in _commit_savepoint
obj._p_estimated_size = len(data)
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/git/buildout.coredev/src/Products.CMFPlone/Products/CMFPlone/Portal.py", line 68, in __setattr__
if self._tree is not None and name in self:
^^^^^^^^^^
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/Connection.py", line 787, in setstate
p, serial = self._storage.load(oid)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/mvccadapter.py", line 164, in load
r = self._storage.loadBefore(oid, self._start)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/DemoStorage.py", line 229, in loadBefore
return self.base.loadBefore(oid, tid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/DemoStorage.py", line 229, in loadBefore
return self.base.loadBefore(oid, tid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/DemoStorage.py", line 229, in loadBefore
return self.base.loadBefore(oid, tid)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 2 more times]
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/utils.py", line 281, in __call__
return func(*args, **kw)
^^^^^^^^^^^^^^^^^
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/MappingStorage.py", line 171, in loadBefore
raise ZODB.POSException.POSKeyError(oid)
ZODB.POSException.POSKeyError: 0x2a33e463be38bb08 When Plone is running normally this error does not occur, only in testing. Could there be some missing test configuration? |
Wesley Barroso Lopes wrote at 2024-10-8 06:14 -0700:
...
File "/home/user/.buildout/eggs/cp312/ZODB-5.8.1-py3.12.egg/ZODB/MappingStorage.py", line 171, in loadBefore
raise ZODB.POSException.POSKeyError(oid)
ZODB.POSException.POSKeyError: 0x2a33e463be38bb08
You try to load an object state which is not there.
Your use of a `DemoStorage` (i.e. a RAM based storage) suggests
a test setup with few objects. The oid `0x2a33e463be38bb08`
is huge for such a setup.
I cannot tell you more. I likely would try to find out where
this oid comes from.
|
@d-maurer the functionality I'm implementing clones the portal with pickle. See: Usually the Portal is a large object. Could it be that the test configuration is not supporting the clone size? In real life, would I have to optimize the portal clone, to clone only the attributes that are really needed? |
Wesley Barroso Lopes wrote at 2024-10-8 08:12 -0700:
> The oid `0x2a33e463be38bb08` is huge for such a setup.
@d-maurer the functionality I'm implementing clones the portal with pickle. See:
https://github.com/plone/Products.CMFEditions/blob/c80bc31af46bff45fee2908878b1f01190fda8d8/Products/CMFEditions/ArchivistTool.py#L210-L229
Usually the Portal is a large object.
Your `oid` is huge; it indicates a setup much larger than a
typical Plone site.
Could it be that the test configuration is not supporting the clone size?
I would not expect a `POSKeyError` indicating a RAM (storage) shortage.
In real life, would I have to optimize the portal clone, to clone only the attributes that are really needed?
I am not sure that cloning a portal will work out of the box.
At least, the cloned catalog will contain entries refering to
the source (and not only entries refering to the clone).
They might be other problems. Keep fingers crossed.
|
@d-maurer there is already a treatment to prevent child objects of the Portal from being placed in the clone: The child objects are replaced by an object of the Checking the traceback, I noticed that the error occurs in the Portal class, in this line: https://github.com/plone/Products.CMFPlone/blob/6.0.13/Products/CMFPlone/Portal.py#L68 So I printed
In other words, the object that caused the error was this:
When we had the object:
the error did not occur. Does this give any clue as to what might be causing the error? |
Wesley Barroso Lopes wrote at 2024-10-8 12:16 -0700:
...
<Products.CMFPlone.Portal.PloneSite object at 0x7f697d4581a0 oid 0x1d86b25a16bc20b5 in <ZODB.Connection.Connection object at 0x7f697d4ddb80>>
OIDs (= "Object IDs") are assigned sequentially;
this means, an OID roughly gives a hint on how many objects
are in the ZODB (at least if objects have not been massively deleted).
The OID "0x1d86b25a16bc20b5"
corresponds to "2.127.583.973.977.366.709" or roughly
2 * 10^18.
I cannot believe that you really have a ZODB with so many objects.
I fear something is very wrong with your setup (and the OIDs
are wrong which would correspond to `POSKeyError`s).
...
When we had the object:
<PloneSite at plone>
This is the `repr` format for an acquisition wrapper;
this means, here you do not yet access the base persistent object
but the acquisition wrapper wrapping the object at location `plone`.
The `<Products.CMFPlone.Portal.PloneSite object at 0x7f697d4581a0 oid 0x1d86b25a16bc20b5 in <ZODB.Connection.Connection object at 0x7f697d4ddb80>`
follows the `repr` format for a persistent object (when
derived classes do not define `repr` differently).
This means that you access here the base portal object (no longer
acquisition wrapped).
If `pobj` is a persistent object (or an acquisition wrapper thereof),
you get its OID with `pobj._p_oid` and its ZODB connection with
`poid._p_jar`.
You can check whether a ZODB connection `conn` knows about an OID `oid`
with `conn[oid]`. This will give you the (unwrapped) object with `oid`
or a `POSKeyError` (if the object is not known).
|
7e2f686
to
1ab8ffa
Compare
I have been partially wrong: Looking more closely at your traceback: Checking the code, I see that |
I have created zopefoundation/ZODB#401 |
After changing the line, I get the error:
I did a
|
Wesley Barroso Lopes wrote at 2024-10-9 05:57 -0700:
> I suggest you replace line 135 of ZODB.DemoStorage (self._next_oid = random.randint(1, 1 << 62)) by self._next_oid = self.base._next_oid + 0x100000 and check whether this lets your test succeed.
After changing the line, I get the error:
```
AttributeError: 'MappingStorage' object has no attribute '_next_oid'. Did you mean: 'new_oid'?
```
My fault! I had not checked the base implementation.
Looking at `DemoStorage.new_oid`:
the `_next_oid` is just used as a potential OID candidate;
`new_oid` checks that it is currently not used; if it is,
then a new candidate is randomly chosen.
Thus, at least if the base storage is not independently changed
(unlikely in your situation), `DemoStorage`'s OID generation
should be safe.
Almost surely, my assumption was wrong that the huge OID is related to the
problem you have observed.
|
@d-maurer I debugged a bit and found out that this error occurs when we have a savepoint and a Plone Site object is involved. It occurs when the code enters here: https://github.com/zopefoundation/ZODB/blob/5.8.1/src/ZODB/Connection.py#L486-L494 However, the error only occurs in the test and when there is a portal object involved. Even in the test, this code works, if there is no modified Plone Site object. In the normal execution of an instance, the code does not enter the if, because https://github.com/zopefoundation/transaction/blob/4.0/src/transaction/_transaction.py#L618 So the savepoint is not actually created. But in the test we have resources and the savepoint code is executed. In fact, if I return after this line, the test works. Do we have a limitation of |
New hypothesis: The code cited above tries to update the object's size estimation ( I suggest you try the following:
by
Apparently the special |
@d-maurer This worked! Thanks! I'll try to create a PR to fix this. |
Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment)
Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment)
…ves (#4026) Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment) Co-authored-by: Rohan Shaw <[email protected]>
Branch: refs/heads/master Date: 2024-10-15T22:21:08-07:00 Author: Wesley Barroso Lopes (wesleybl) <[email protected]> Commit: plone/Products.CMFPlone@9a1a62b Avoid POSKeyError when commit occurs and we have savepoint that involves (#4026) Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment) Co-authored-by: Rohan Shaw <[email protected]> Files changed: A news/4026.bugfix M Products/CMFPlone/Portal.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@wesleybl Could you add a test?
David Glick wrote at 2024-10-30 13:45 -0700:
@davisagli commented on this pull request.
@wesleybl Could you add a test?
It is very difficult to create a reliable test for the original
szenario (portal copy).
One could create a test following the decription
in "zopefoundation/ZODB#402".
It would use a mockup class with a `__setattr__` read accessing
a (normal) instance attribute.
The test would instantiate a class instance, create a savepoint,
use the instance as attribute of an object stored inside the ZODB
and then `_p_deactivate` the instance. A following `commit`
should show the problem (because the instance state can only be loaded
from the temporary savepoint storage no longer available during
the savepoint commit).
If `__setattr__` is fixed to not access a normal attribute
for an assignment to `_p_estimated_size`, the error should go away.
The test would not directly verify the fixed `PortalSite.__setattr__`.
Instead, it would indirectly show that the approach used there is effective
for this kind of `savepoint` commit problem.
|
I meant a functional test that the new REST API endpoints in this PR are working as expected. |
@davisagli for the test to work, we first need a Plone release with: plone/Products.CMFPlone#4026 I need to backport this to Plone 6.0. |
***@***.*** wrote at 2024-10-30 22:06 +0100:
David Glick wrote at 2024-10-30 13:45 -0700:
***@***.*** Could you add a test?
It is very difficult to create a reliable test for the original
szenario (portal copy).
Maybe, I was too pessimistic.
One could create a test following the decription
in "zopefoundation/ZODB#402".
It would use a mockup class with a `__setattr__` read accessing
a (normal) instance attribute.
The test would instantiate a class instance, create a savepoint,
use the instance as attribute of an object stored inside the ZODB
and then `_p_deactivate` the instance. A following `commit`
should show the problem (because the instance state can only be loaded
from the temporary savepoint storage no longer available during
the savepoint commit).
If `__setattr__` is fixed to not access a normal attribute
for an assignment to `_p_estimated_size`, the error should go away.
We likely could use `PortalSite` as the mockup class
`_tree` is the (normal) attribute accessed in `__setattr__`).
|
David Glick wrote at 2024-10-30 14:09 -0700:
I meant a functional test that the new REST API endpoints in this PR are working as expected.
The former observations (works in production, fails in test)
show that special conditions are necessary for a failure.
Therefore, a test may indicate success even though the problem is still
there.
Based on my current understanding, I suppose that
during the test accidentally some objects are ghostified which
are not in production.
Therefore, I would say that a test should enforce that an essential
object has been ghostified.
|
Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment)
Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment)
@davisagli Should the github actions tests pass even if we don't have the package versions with the other PRs? Should we expect a new Plone version with the other PRs? |
Allows working copy services to be accessed in the Portal. Also returns working copy data in the Portal serialization.
1ab8ffa
to
27d4afb
Compare
…ves (#4043) Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment)
Branch: refs/heads/6.0.x Date: 2024-11-05T14:27:57-08:00 Author: Wesley Barroso Lopes (wesleybl) <[email protected]> Commit: plone/Products.CMFPlone@c2b14e3 Avoid POSKeyError when commit occurs and we have savepoint that involves (#4043) Plone Site When ZODB handles savepoint and we have changes in Plone Site at that savepoint, it changes the `_p_estimated_size` attribute of Plone Site. This is an assignment to one of the special persistency attributes (identified by an _p_ name prefix); it should happen without access to any other attributes of obj. But obj._tree is accessed in __setattr__ of PloneSite class and results in a ZODB load which apparently fails. See: plone/plone.restapi#1823 (comment) Files changed: A news/4043.bugfix M Products/CMFPlone/Portal.py
This is part of: plone/volto#5284
Allows working copy services to be accessed in the Portal. Also returns working copy data in the Portal serialization.
In Plone 6.0, this PR should be tested together with the following PRs:
plone/plone.app.iterate#128
plone/Products.CMFEditions#113
In Plone 6.1, the
plone.app.iterate
PR should be replaced with:plone/plone.app.iterate#130
📚 Documentation preview 📚: https://plonerestapi--1823.org.readthedocs.build/