Summary of major tech debts

[viniarck]

This is just a summary of existing major tech debts on kytos-ng and which future potential epics to start discussing about them and address them. Major tech debts:

Front-end

1. Vue 2 needs to be migrated to Vue 3

• There's an ongoing effort current in progress

Back-end

0. @listen_to potential out of order events execution is getting cumbersome and tricky to maintain

• Alternative solutions: use @alisten_to or @listen_to(pool="dynamic") if you can afford, however the current MongoDB client with pymongo is sync.
• Refactor existing @listen_to as @alisten_to when there's no blocking dependency down (including threading.Lock instances, those need to be asyncio.Lock, although managing asyncio.Lock is much easier to reason about since all the yield/await points are explicit so certain non suspending that functions that get called might not even need to have a lock mid iteration), asyncio is not a silver bullet though, but compared with the threading use case it should be not only lighter in terms of resources but easier to maintain and reason.
• RabbitMQ can also help here but it's a large infrastructure dependency that would also need a ton of refactoring, trying to add a RabbitMQ-like message broker on Kytos-ng at the moment, would feel like a bazooka to kill a fly, sure it can also help with message persistence, but as of now when event handlers when implemented without race conditions will work reliably and we have a basic support for dead letter queues too.

1. Keep moving more towards asyncio, try to refactor existing synchronous threadpool based endpoints to async. This is related to 1, but for HTTP endpoints that front-end and other API clients use.

2. NApps aren't microservices and kytosd can't be stateful clustered easily without a massive refactoring that would take years. Kytos-ng can be categorized as a majestic monolith, which is great too. A Napp will run in a shared Python runtime, it's just a dynamic lodead "plugin", it's a Python package that gets loaded dynamically. There's room to simplify service-to-service / NApp-to-NApp communication in certain use cases that's a request is being used, concrete example: mef_eline when trying to find paths via pathfinder

• A little bit of context for future readers: Each NApp also doesn't have it's own database and neither it's own HTTP server, the uvicorn/starlette app instance is shared accross all NApps, so there's no full fault-tolerance in terms of process/HTTP/database. Also, many NApp depends on shared resources on core such as Interface, Switch and Link (on topology) and operation on these are using shared memory. The original intention was great to try to make a NApp as a microservice-like via HTTP for service to service communication, but at the moment we get all the downsides of this type of communication without much of the benefits. If one day this were to be a full blown real microservices architecture it could be done if it were to be business justifiable, but it would need a team much bigger, and all NApps to core communication via HTTP too (this is what uonos is doing with gRPC), and much harder and deployment orchestration, and since we don't have any scaling issues that can justify this, we have room to simplify service-to-service communication by not using HTTP to service-to-service and only use it for the front-end (or external 3rd party clients) and service-to-service/NApp-to-NApp via Python callables and/or via the event bus when it makes sense. We don't have any major bottleneck in production use cases, uvicorn/starlette theoretically handle dozens of thousands requests/sec, and MongoDB can scale horizontally also handling dozens of thousands of ops/sec.
• Potential simplification: start allowing NApps to expose/registry which callables either a def func(*args, **kwargs) or async def func(*args, **kwargs) that can be called by any other NApp with Pydantic model validations or trying to reuse validations from openapi-core. A NApp that needs to call other NApp would try to get whether or not the callable has been registered as start using it. There's also the fact that NApps get started in certain order, but long as you start to call it after you get a loaded event it should work without exceptions, and then with rate limiting in the future we can also plug rate limit here.

3. Kytos event bus is meant to be pub sub fanout. Such as when indeed a one or more NApps need to subscribe and subscribe for changes. It shouldn't not be primarily used for background 1-to-1 / channel like communication between two NApps when you need to avoid an HTTP request overhead unless the underlying resource indeeds only responds asynchronously after some time.

• Good usages of 1-to-1: sending flow mods since the error is asynchronously and depends on a switch response anyway.
• Good usages of 1-to-1: fire-and-forget use cases when you know the payload won't fail and when the handling NApp can also deal with errors and ensure an eventual execution.
• Non optimal usages of 1-to-1 events: trying to use a event for a function that can reply immediately or a func that finds network path like on pathfinder. This can work too, but it's a lot of extra code and leg work for virtually no benefit at all, not to mention asynchronously error handling on a different code path. (This can be solved with the proposed simplification on point 2)

4. Start support an additional MongoDB driver such as motor once it's been confirmed it's prod grade at least equal or superior performance (this is related to point 1 on this list).

5. Migrate to MongoDB 7, MongoDB 5 EOL in late 2024.

e2e testing

1. e2e test suite deserved a bit more of attention and review, xfail tests, reruns and overall house keeping.
2. automated on-demand stress tests to really try to catalyze reasonable scalability a bit more than used in prod. (This isn't actually tech debt, but the absence of it sometimes make it harder to quickly stress test certain changes or features that are being developed)
3. e2e upgrade db tests (using an old DB collection), including running DB specific upgrade scripts.

cc'ing @jab1982 for his information, and the core team @italovalcy @Ktmi @Alopalao @rmotitsuki. If you guys also have noticed any major paint point feel free to add in the comments, and have any other suggestion, as we can afford to try to mitigate certain tech, the first step is to start mapping just so they can be potentially considered and prioritized one day. Once they are, it'll be further categorized in a epic and further broken down into tasks.

[Ktmi]

I've actually been looking into async mongo specifically for tyring to move as much code over to async. As you've mentioned one option is motor. It uses very similar syntax to the existing pymongo driver, as it just an async wrapper that runs pymongo in a separate thread. Alternatively, we could wrap our existing calls to the db to run in an executor, and create a db threadpool. Personally, I would suggest the latter, as it reduces the area of code we would have to modify, and ensures maximum compatibility.

As for inter-napp communication, the direction I've been thinking of moving this is into a having generic references to objects or collections that we can then attempt to resolve actions on, similar to graphQL. For example, you want to add a flow to a table on a particular switch, you would ask the switch for the references to its tables, then use the table reference to insert the flow. At the top level, to access a reference you would use the NApp's reference, provided by the controller. Optionally you could create a wrapper class for the references which would automatically make the calls to the resolvers when trying to access an attribute.

[viniarck]

Alternatively, we could wrap our existing calls to the db to run in an executor, and create a db threadpool.

Yes, also being able to fully manage this pool and gracefully shuts it down can be great. We also use this pattern on start_shell_async, and it's definitely Pythonic and well supported. To be honest, I was also expecting motor to be a bit more asyncio first class citizen, specially now in 2024, but yes, probably for their business use case they wanted to have a thin wrapper async layer compatibility being good enough including other ones like tornado that used be popular back in the days. Let's keep this mind, and see if in the future motor will evolve, if by the time we prioritize this migration and it's still the same, then we might as well stick with pymongo in an executor managed by us. In the final result when developing this, let's also benchmark against motor to ensure that at least our implementation is on par, which should be based on their current implementation.

[viniarck]

At the top level, to access a reference you would use the NApp's reference, provided by the controller.

Yes, that'd be great. Allowing NApps to invoke the (exported) callables accordingly. It's certainly also desirable to have a fine control of what is allowed, of course in Python land, this is just a logical boundary, but at least also explicitly marking/decorating/or registrying and then accessing/getting this NApp reference/callable references becomes a bit more explicit and kind of controlled of what it's expected to be called by other NApps (which is also great for developers to also quickly figure out what they can call directly on other NApps).

Yes the idea is to "resolve" a callable, let's try to really make this as dead simple as possible, as it should be just a simple callable call anyway. Then we can also discuss if/how/when Pydantic or any other validation should be used. Good thing about this is that also for existing functionality via endpoints, where applicable we can extract a new func, make it "exportable" and then reuse it as on the API endpoints for front-end or other clients.

[viniarck]

Great suggestions, David.

[Alopalao]

Great summary. I was only aware of the out of order events since I hit it a couple of times with testing. callables is a great addition but are there any plans to automatically run NApps in order despite them being installed out of order?

[Ktmi]

NApps list their dependencies within the kytos.json file. It should be possible to build a dependency graph based on this, then preform a reverse topological sort to find the order which NApps should be loaded.

[viniarck]

Side note:

For future reference it was discussed on this issue #163, and kytos.json last updated time was reasonable enough and good compromise.

Speaking of which, a minor pain point relying on kytos.json stat.ST_MTIME is that when you bump kytos-ng versions then you need to retouch these files. There's room to potentially slighly improve here too. Let us know if you've encountered any other pain point though @Alopalao .