Viewing multiple projects

[davidaustinm]

I am working on two PreTeXt projects (A and B) concurrently and viewing project A on port 8128. When I ask to view project B with "pretext view", it opens up project A on port 8128. When I ask to view project B on a different port with "pretext view -p 8201", it doesn't acknowledge the request and again opens A on 8128. However, I see B if I manually edit the URL to point to the requested port.

Trying "pretext view -r" again opens A except with the report:

warning: Failed to stop server.
warning: Port 8128 could not be used.
warning: Trying port 8129 instead.
Server will soon be available at http://localhost:8128
Opening browser for target web at http://localhost:8128/output/web

I am able to stop the server with "pretext view -s", which enables me to see B with "pretext view" on port 8128. But now I've lost project A.

If port 8128 is not in use, then the -p switch opens the project on the requested port.

I am working on ubuntu 22.04 in a python3.10 virtual environment with a fresh pretext installation.

[skiadas]

An interesting problem, looking through the code right now I think the relevant bits are around https://github.com/PreTeXtBook/pretext-cli/blob/03d0641495d728b5d56b9f9d322afbee3c98d1d3/pretext/cli.py#L764C1-L812C1 and

pretext-cli/pretext/utils.py

Line 688 in 03d0641

def active_server_port() -> t.Optional[int]:

It seems to me the system tries to reuse an existing server if it can find it, which would make multiple projects harder. It may be better to not try to reuse a server, and assume that a call to pretext view should start a new server? Perhaps there should be instead a flag if someone really wants to try to reuse an existing server.

But either way there is something wrong in the current code, as it starts a new server and gives you a message to that effect (trying a new port), but provides you with links to the old port.

[StevenClontz]

Something I have pushed for (and should probably implement myself) is a single server that hosts everything in the author's output directory, and then when you pretext view a particular target, it just provides the appropriate URL to that target.

I have three books in the same repo at TBIL.org and this affects me to.

[oscarlevin]

There is definitely a bug. I like the idea of having a single server, but I'm unsure how to implement this. That was sort of the idea that is in place now, at least to not keep starting new servers every time a user asks to view.

If there are different projects in different folders, then where is the root of the server? And how do we identify a server as started by pretext for a particular project?

[skiadas]

If there are different projects in different folders, then where is the root of the server? And how do we identify a server as started by pretext for a particular project?

I imagine that it would involve changing

pretext-cli/pretext/utils.py

Line 229 in 03d0641

def serve_forever(

and the custom RequestHandler there to instead keep track of a dictionary of hashes and directories, and use a SimpleHTTPRequestHandler for each directory, so maybe something like a dictionaryMappingDirectoriesToHashes and a dictionaryThatMapsHashesToRequestHandlers, and then a path like /dfgerftg/output/web would be routed to the handler matching the hash dfgerftg, and the process of pretext view would first look to find the server and start one if it doesn't find it, and then it would look at the dictionary to see if there is a hash for the current directory, or else create one, and also create a request handler for that directory. Actually we probably don't need two dictionaries. Here's what I envision:

• The first pretext view starts a background server process. I might consider calling it watch instead of view but probably too late for that.
• A new pretext stop command stops the server.
• A server's request handler keeps a dictionary of SimpleHTTPRequestHandler instances, one for each project directory, and keyed by a hash of the full path to the directory-to-be-served.
• On a pretext view the path's hash is computed and looked up, and if needed a new request handler launches.
• The routes to the various versions would look like: localhost:8128/<hashHere>/output/web or whatever. To produce and return the route all we really need is to hash the directory-to-be-served.

Haven't tested it, not sure simplerequesthandlers are meant to be used this way, or if there is an existing comborequesthandler doing something like that. But that's what I envision.

[skiadas]

I have a rough proof-of-concept on this, I should have it as a PR soonish, but it will probably need some work, and I have no idea how it will play in a codespace. The gist is this:

• A new pretext webserver command starts a webserver using a custom subclass of BaseHTTPServer and a custom subclass of SimpleHTTPHandler as a handler.
• This server listens on localhost:8128, with the port being customizable.
• The server has two special routes. The first is /quit which shuts the server down.
• The other special route is register?path=encodedPathToProjectDirectory. It then decodes the directory, computes its (sha256, probably truncated to 8 chars or something) hash, and stores it in a dictionary. It then returns the hash.
• Any other route is assumed to have the form /<hash>/relativePathInProjectDirectory. The custom handler grabs the hash, looks it up in the dictionary to find the corresponding directory, then sets self.directory to the correct value and sets the path to the correct value before delegating to the SimpleHTTPHandler. This way the SimpleHTTPHandler will operate on the appropriate directory each time.
• A new command pretext watch (probably needs a new name, or maybe to become part of view) makes the appropriate /register request in order to register the current directory with the webserver. It would basically take the place of a call to view in this setting.

So in this setting a normal operation would be:

• Run pretext webserver once in your system, from any pretext project directory. Only a single server ever needs to run, so can just use one specific port.
• Run pretext watch from any pretext project directory that you want the server to provide to you, and get back the link to use (or maybe opens it directly?).
• Repeated pretext watch calls from the same directory simply replace the dictionary entry, and give you back the same link.
• You can shut down the server by going to localhost:8128/quit, or we can probably make that another pretext command.

[oscarlevin]

This sounds really great, @skiadas. I'm looking forward to seeing your implementation, partly to learn more about this server stuff.

Do you think we can have all three of the webserver|watch|quitserver commands be just view? So pretext view looks on the default or specified port for whether it is a pretext server, if it is, do what you say for watch, if the port is available, does what you say for webserver, and if it is taken by another process, either quits or just picks the next best port. Then pretext view --stop or similar quits the server. Just trying to make this is user-friendly and backward compatible for non-techy folks as possible.

[skiadas]

Yeah I think it can definitely be integrated into view, especially if we add a few flags. I was just focused on getting the proof of concept down, and view was doing too many things that I wasn't prepared to understand, and I think most of it having to do with codespaces. And I'm not sure how this is supposed to play out in codespaces. I think the problem would be maintaining the current "local-server" functionality while adding the new "global-server" option. If there are a few local servers, and so a global server ends up on port 8132 say, then the local servers are shut down, then I try to start a global server again, it would find 8128 available and would start a second global server. I basically would like to have a more clear way to identify the port on which the global server is running, just stepping up one at a time until one is free feels a bit clunky. Good to get an initial port perhaps, but I would like to store it somewhere, and not sure where. As future calls to view need to know what port that global server is running on.

If we are able to use some configuration file on the user's system, a ~/.pretextrc or something, that might be a place to store such things. But not sure how cross-platform that is. So the answer I went with at the moment is to say that if you are using the new web-server then you are not using the old view system, and then we can count on that server being on port 8128 (and hope that no other application tries to use that port). So here is perhaps an option:

• On view --start-server we try to start a global-webserver on port 8128, first checking if a global-webserver exists there (can add a new hearbeat route for that). If the port is busy with something that is not the global-webserver, just end, warn the user, and maybe fall back to non-global behavior. If successful, also register the current directory with the global-webserver.
• On view --stop-server check if there is global-webserver on 8128 and if there is shut it down. Don't go digging around in other ports.
• If neither of those flags is present, fall back to normal view behavior.

[StevenClontz]

This is sounding slick, so I'll just note my lukewarm concern that it might be overengineered: why not just have a server for the project @output directory, so if I want to view a web target with default @output, I head to https://localhost:8128/web, and if I want print, I head to https://localhost:8128/print?

I can give one reason: if for some reason a target sets @output="../foo" and lives outside the project @output. Is that worth supporting?

[oscarlevin]

This approach would allow viewing of two or more distinct pretext projects on the same server. This will become more common as we allow for small documents that might be part of a larger project, so I think this is a case worth supporting.

[skiadas]

This is sounding slick, so I'll just note my lukewarm concern that it might be overengineered: why not just have a server for the project @output directory, so if I want to view a web target with default @output, I head to https://localhost:8128/web, and if I want print, I head to https://localhost:8128/print?

I can give one reason: if for some reason a target sets @output="../foo" and lives outside the project @output. Is that worth supporting?

Hm actually yeah I think directly hitting the output directory would also be more secure, and easy to achieve. No reason to serve the actual project directory, can simply serve the output directory.
But the key part of this is to support in effect all your PreText projects in your computer with one server running on one port. In fact I envision that opening up localhost:8128 will give you a list of all the projects on your computer that PreText knows about and you can pick one to view (in addition to direct links to it being possible).

[StevenClontz]

Having a single server to preview builds of different projects is not a pattern followed by any static site generator or web framework I'm familiar with. That's not to say that we shouldn't do it, but it's worth considering why they may or not, and why our use case might be different.

[skiadas]

Having a single server to preview builds of different projects is not a pattern followed by any static site generator or web framework I'm familiar with. That's not to say that we shouldn't do it, but it's worth considering why they may or not, and why our use case might be different.

I have to admit that beyond the fact that it is really cool, it is probably not something that is really needed in that form, and allowing a webserver to effectively serve arbitrary locations on the computer is probably not desired and likely the reason it's not really done. Perhaps as part of a different app to provide a GUI/Web supervisor/manager for all your PreText projects in your computer, but that's probably a different project than pretext-cli. So I'll put it in the freezer for now.

But I think that the seed of the idea can be used to solve the problem we are having, which is basically simply the fact that right now the way that we try to detect a running server is not really working. The current way is to basically to look for "any process named pretext that is a child of a process named pretext", and possibly compare a port if one was provided. First of all, this appears to not be working, at least in my local testing on my mac. When I look at the results of the psutil.process_iter() call, even with mutliple pretext view commands currently running, it does not show me any processes at all whose names are "pretext", let alone one whose parent is also called "pretext". The names appear to be simply inherited from the parent process, which is either the shell or a VSCode helper, in my case. So something is happening at least on macos that makes this heuristic not viable. We might be able to fix that by finding a way to explicitly set a name for our pretext process and its children. But even if it did work, it would return just the first server regardless of which project that server is related to, and also a view -s command would stop all servers, and not just the one corresponding to the current project. It also seems that the normal view fails to detect existing servers as it follows the same heuristic. So I think we should consider a different approach.

So here is my current idea:

1. We maintain a file under ~/.ptx, maybe called running_servers or something.
2. When a server for a project starts, it hashes the project directory and adds a line in the file containing the triple of: hash -- pid -- binding+port
3. To look for an existing server for a particular project, we compute its hash then look through the relevant line in ~/.ptx/running_servers to find the pid and binding/port for any server running for the current project, then check that pid/port to see if the server is still active.
4. To stop a server, we similarly look through that file for a server for the current project hash.
5. We can also have an option to stop all servers, by again going through that file and now looking at all entries and attempting to terminate all the relevant processes.

[Alex-Jordan]

Would it help (be simpler) to just have a project declare a port to use in the project.ptx file? Sorry if this has been proposed.

…

On Sat, Oct 26, 2024, 1:47 PM Haris Skiadas ***@***.***> wrote: Having a single server to preview builds of different projects is not a pattern followed by any static site generator or web framework I'm familiar with. That's not to say that we shouldn't do it, but it's worth considering why they may or not, and why our use case might be different. I have to admit that beyond the fact that it is really cool, it is probably not something that is really needed in that form, and allowing a webserver to effectively serve arbitrary locations on the computer is probably not desired and likely the reason it's not really done. Perhaps as part of a different app to provide a GUI/Web supervisor/manager for all your PreText projects in your computer, but that's probably a different project than pretext-cli. So I'll put it in the freezer for now. But I think that the seed of the idea can be used to solve the problem we are having, which is basically simply the fact that right now the way that we try to detect a running server is not really working. The current way is to basically to look for "any process named pretext that is a child of a process named pretext", and possibly compare a port if one was provided. First of all, this appears to not be working, at least in my local testing on my mac. When I look at the results of the psutil.process_iter() call, even with mutliple pretext view commands currently running, it does not show me any processes at all whose names are "pretext", let alone one whose parent is also called "pretext". The names appear to be simply inherited from the parent process, which is either the shell or a VSCode helper, in my case. So something is happening at least on macos that makes this heuristic not viable. We might be able to fix that by finding a way to explicitly set a name for our pretext process and its children. But even if it did work, it would return just the first server regardless of which project that server is related to, and also a view -s command would stop all servers, and not just the one corresponding to the current project. It also seems that the normal view fails to detect existing servers as it follows the same heuristic. So I think we should consider a different approach. So here is my current idea: 1. We maintain a file under ~/.ptx, maybe called running_servers or something. 2. When a server for a project starts, it hashes the project directory and adds a line in the file containing the triple of: hash -- pid -- binding+port 3. To look for an existing server for a particular project, we compute its hash then look through the relevant line in ~/.ptx/running_servers to find the pid and binding/port for any server running for the current project, then check that pid/port to see if the server is still active. 4. To stop a server, we similarly look through that file for a server for the current project hash. 5. We can also have an option to stop all servers, by again going through that file and now looking at all entries and attempting to terminate all the relevant processes. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/PreTeXtBook/pretext-cli/issues/848*issuecomment-2439731830__;Iw!!Ka_JY85zDv0FFw!h1MMxUWxhYbKPkgESqqF7K5D20xNjU1Xsy39EEfNw1FbdRH3Cs6wWhROBgu-A1grVgzkdVNhoTuiK5CIQMyTnu1Ng5A$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABEDOAH7V7NHLXWQALDV4ZLZ5P5URAVCNFSM6AAAAABQKGC35GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZZG4ZTCOBTGA__;!!Ka_JY85zDv0FFw!h1MMxUWxhYbKPkgESqqF7K5D20xNjU1Xsy39EEfNw1FbdRH3Cs6wWhROBgu-A1grVgzkdVNhoTuiK5CIQMyTr3wAuj8$> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[StevenClontz]

IMO project.ptx should be machine agnostic (specifying a port on one machine might conflict with another machine)

[skiadas]

Partial work towards this: #878