training gets very slow when visualizing plots on visdom

[ansabsheikh9]

Use this to open other questions or issues, and provide context here.
Hi,
I am using PyTorch to train my network and Visdom to visualize,
Each forward pass is taking almost .07 sec but when i start to visualize results on visdom, training gets very slow like 0.2 secs for each epoch, and also the visualization gets very heavy and slow to zoom in or out. Any Idea why it is happening.
Best Regards

[lvdmaaten]

Without additional context, it is hard to say anything meaningful. What are you trying to visualize, and why do you want to push a new visualization every 0.1 seconds? Where does your visdom server run and what is the interconnect with the client? Etc. etc. If, for instance, you're trying to make a scatter plot with 10 million points in it, this is completely expected behavior.

Please us this template to provide additional context: https://github.com/facebookresearch/visdom/blob/master/.github/ISSUE_TEMPLATE/bug-report.md

[flamz3d]

Without additional context, it is hard to say anything meaningful. What are you trying to visualize, and why do you want to push a new visualization every 0.1 seconds? Where does your visdom server run and what is the interconnect with the client? Etc. etc. If, for instance, you're trying to make a scatter plot with 10 million points in it, this is completely expected behavior.

Please us this template to provide additional context: https://github.com/facebookresearch/visdom/blob/master/.github/ISSUE_TEMPLATE/bug-report.md

We've been chasing down a huge performance bottleneck getting progressively worse as training goes on to this particular issue. To me this is showstopper in terms of adopting visdom. Of course one should not get a linear performance hit while training on the client (!) due to sending telemetry data to a server.

[JackUrb]

@flamz3d definitely agreed that this behavior is breaking for long-running applications. At the moment I'm involved in other projects that have taken priority over visdom, and haven't been able to return to clean up the old server code (#675). Some community members have started to try tracking down the cause of it in #640 and more recently #695 but nobody has actually run a profiler yet it seems.

The issue is probably associated specifically with the update code (according to #695), which points somewhat to this code block:
https://github.com/facebookresearch/visdom/blob/645c907e2ffa6a0488e06c071c0c8323c65ce61e/py/visdom/server.py#L1028
Which initially has me thinking that something was being retained in the actual update code:
https://github.com/facebookresearch/visdom/blob/645c907e2ffa6a0488e06c071c0c8323c65ce61e/py/visdom/server.py#L920
But could also potentially be a memory leak in jsonpatch.make_patch: https://github.com/facebookresearch/visdom/blob/645c907e2ffa6a0488e06c071c0c8323c65ce61e/py/visdom/server.py#L916

Most of the server code predates me though and I haven't been able to dig into it beyond my current work on #675.

You can try running on visdom version 0.1.8.6 to see if that solves the issue. If so, it probably points to a leak in jsonpatch or our implementation using it.