Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Profile mem #678

Merged
merged 8 commits into from
Nov 23, 2022
Merged

Profile mem #678

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
3449cf7
ocrd process: simple memory usage measurement
kba Mar 4, 2021
0cfa612
fix profiler logging test
kba Mar 4, 2021
7e78a5d
ocrd process: memory_usage w/ multiprocess and include_children
kba Mar 4, 2021
f005897
Merge branch 'master' into profile-mem
bertsky Oct 12, 2022
342316e
Update ocrd/ocrd/processor/helpers.py
kba Nov 23, 2022
a103ac9
mem_usage: run process only once
kba Nov 23, 2022
4fdfbf3
Implement OCRD_PROFILE{_FILE} environemnt variable to configure CPU/m…
kba Nov 23, 2022
97a14b0
describe OCRD_PROFILE{,_FILE} in README
kba Nov 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion ocrd/ocrd/processor/helpers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@
import json
import inspect
from subprocess import run, PIPE
from memory_profiler import memory_usage
from sparklines import sparklines

from click import wrap_text
from ocrd_utils import getLogger
Expand Down Expand Up @@ -69,7 +71,13 @@ def run_processor(
log.debug("Processor instance %s (%s doing %s)", processor, name, otherrole)
t0_wall = perf_counter()
t0_cpu = process_time()
processor.process()
mem_usage = memory_usage(proc=processor.process, interval=.1, timeout=None, timestamps=True)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you wanna add multiprocess=True in there for processors that use multiprocessing or similar, or that delegate to shell calls. Perhaps also include_children=True (not sure about the meaning though – does the sum then count shared/CoW pages repeatedly?).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you wanna add multiprocess=True in there for processors that use multiprocessing or similar, or that delegate to shell calls.

I will.

Perhaps also include_children=True (not sure about the meaning though – does the sum then count shared/CoW pages repeatedly?).

            if include_children:
                mem +=  sum(_get_child_memory(process, meminfo_attr))

https://github.com/pythonprofilers/memory_profiler/blob/master/memory_profiler.py#L135-L136

Copy link
Collaborator

@bertsky bertsky Mar 4, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps also include_children=True (not sure about the meaning though – does the sum then count shared/CoW pages repeatedly?).

            if include_children:
                mem +=  sum(_get_child_memory(process, meminfo_attr))

I saw that, but the question remains: what is in that sum? (We probably don't want to count shared and CoW pages repeatedly. But it may be hard to calculate exactly. IIRC standard ps and top and even time do count repetitions, so they appear too large – don't "add up" – for multiprocessed programs, whereas smem gets it right. This tells me we are actually interested in the PSS, the proportional set size, rather than the RSS.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but PSS vs RSS is a more general problem/discussion. Any multiprocessing coder will know about that. And RSS is still a valuable information. So let's stick to your current implementation.

mem_usage_values = [mem for mem, _ in mem_usage]
mem_output = 'memory consumption: '
mem_output += ''.join(sparklines(mem_usage_values))
mem_output += '\n'
mem_output += 'max: %.2f MiB min: %.2f MiB' % (max(mem_usage_values), min(mem_usage_values))
logProfile.info(mem_output)
t1_wall = perf_counter() - t0_wall
t1_cpu = process_time() - t0_cpu
logProfile.info("Executing processor '%s' took %fs (wall) %fs (CPU)( [--input-file-grp='%s' --output-file-grp='%s' --parameter='%s']" % (
Expand Down
2 changes: 2 additions & 0 deletions ocrd/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,5 @@ Flask
jsonschema
pyyaml
Deprecated == 1.2.0
memory-profiler >= 0.58.0
sparklines >= 0.4.2