Is there a way to get the access log programatically?

[AsafMah]

Hey, I want to test my code when going through a proxy.
Proxy.py is a great solution, but I want in the end of the test to see the list of requests that actually went through the proxy.
I know I can setup a log file and then parse it, but that's not ideal.

Is there a way to get all of the requests that went through the proxy as a python list/object?

Thanks

[abhinavsingh]

Yes, you can get access to the logs programmatically. You need to write a plugin for it. Example, ProxyPoolPlugin utilizes on_access_log hook, see https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/plugin/proxy_pool.py#L198-L212

Also look at formal definition of on_access_log here

proxy.py/proxy/http/proxy/plugin.py

Lines 166 to 178 in d4449b8

	def on_access_log(self, context: Dict[str, Any]) -> Optional[Dict[str, Any]]:
	"""Use this method to override default access log format (see
	DEFAULT_HTTP_ACCESS_LOG_FORMAT and DEFAULT_HTTPS_ACCESS_LOG_FORMAT) and to
	add/update/modify/delete context for next plugin.on_access_log invocation.
	This is specially useful if a plugins want to provide extra context
	in the access log which may not available within other plugins' context or even
	in proxy.py core.
	Returns Log context or None. If plugin chooses to access log, they ideally
	must return None to prevent other plugin.on_access_log invocation.
	"""
	return context

Once you have a plugin with custom on_access_log handler, you can choose to do whatever you want within you Python code. Write to a database, on disk, in-memory data structure, anything.

Let me know how it goes for you.

[AsafMah]

After creating the plugin, how can I access the instance of the plugin from my proxy.Proxy object?

[abhinavsingh]

Instances of plugins are created for every request. There is no global instance of each plugin which you can access from outside. Your best bet is to add your logic within your plugin code. Example

Approach 1

Write each log line in a db/disk/memory, then when proxy.py exists, you can accumulate all those logs into a logical data structure that suites you.

with proxy.Proxy():
    sleep_loop()
assemble_logs()

This is technically nothing different than parsing the log file itself. After all, assemble_logs() can assemble from db, disk, memory or the log file itself.

Approach 2

If you are looking for a convenient data structure which can be updated on the fly, then you have 2 options:

1. Define your global data structures. Guard your data structure with locks when updating them from within the plugin
2. Or simply use something like Redis which provides inbuilt data structures like LIST etc

[AsafMah]

The problem is that the Plugin seems to run in another process, so the global data doesn't update.
Is there a way to force the proxy to not be in a new process (and not block)?

[abhinavsingh]

What you are asking is related to multiprocessing Python programming. You must read about it, specially how to share data between multiple processes. Then you can use that strategy within your plugins. This is how most proxy.py users achieve the same. Start here https://docs.python.org/3/library/multiprocessing.shared_memory.html

Is there a way to force the proxy to not be in a new process (and not block)?

No there is no magic way. You must just use Python idiomatic/semantics to achieve what you are after.

PS: In general having shared memory is simply a bad idea. You will slow down the proxy requests. If you are using proxy.py in testing environment, then thats not an issue. But in production environment, users will experience slightly longer latencies, because a plugin is trying to synchronize over shared memory.