As a developer, I want to share code between projects

[Aboisier]

We could benefit from setting up Yarn Workspaces.

1. Ability to share code between projects (workspaces)
2. Define clear modules boundaries to prevent ending up with a monolith

There are other solutions such as Nx, but I believe Yarn would suit our needs better. It's a bit more lightweight.

Ref.: https://yarnpkg.com/features/workspaces

[Aboisier]

I'm trying to figure out the best way to structure our monorepo. I'll document my thought process in this issue. Here's a snapshot of my current reflections on the subject.

First of all, I think I'd rename a couple of things:

old name	new name	reason
stalker_ui	stalker_app	It will be move to a "frontend" folder, so it's implicit it's a UI
docs	stalker_docs	Just to be consistent with "stalker_app"
flow_manager	jobs	I think we should aim for this service to be responsible for jobs only. Authentication and users and such could be movers to another microservice eventually
cron_service	cron	It will be moved to a "backend" folder, so it's implicit it's a service

Monorepo structure guidelines

There are 3 main packages categories, some of which are subdivided into sub-categories.

• common: logic and interfaces which may be reused by both frontend and backend
• backend: backend stuff
  • service: some kind of microservice. May not be imported
  • client: models and clients to consume a micro-service. May be imported by backend/*/service and frontend/*
• frontend: contains frontend stuff. Frontend packages may not be imported by any package, except the ui_kit, which may be imported by any frontend.

Here's what this could look like. Each leaf represents a package.

packages
├── common
│   └── domain
├── backend
│   ├── jobs
│   │   ├── service
│   │   └── client
│   ├── cron
│   │   ├── service
│   │   └── client
│   └── orchestrator
│       └── service
└── frontend
    ├── ui_kit
    ├── stalker_app
    └── stalker_doc

Packages may be imported via the prefix @red-kite/* (except the backend/orchestrator/service, which is a C# package). For example, @red-kite/backend/jobs/client, or @red-kite/frontend/ui-kit

[lm-sec]

I agree with most of the renames, they make sense.

Isn't "jobs"'s name conflicting/confusing with the actual jobs'? This service would have to also manage subscriptions and findings, it's the whole workflow. I can't say that I have a much better name at hand, hence why it is currently named flow manager, which I agree is not the best either. I still think jobs is too precise though and it should be named from a step further.

We could name it stalker_api for now, in the backend folder. It would be more coherent with the frontend folder, and it's a proper name for the api part if/when we eventually split it. We later only have to find a name for the workflow micro service, which is a problem for future us.

I agree though that splitting the FM's service could be beneficial: easier scaling, less unnecessary scaling of lightly used parts of the code, smaller code base in service, splitting of concerns, etc. And there is also kind of a natural split in the logic already. It would have to be properly thought out, but it could be beneficial.

Do you see another queue between the API and the workflow split services? Would you wait until it is an issue, or would you tackle the potential problem right away? We could rethink the architecture as needed when we need to scale the FM? What do you think?

[Aboisier]

Isn't "jobs"'s name conflicting/confusing with the actual jobs'? This service would have to also manage subscriptions and findings, it's the whole workflow. I can't say that I have a much better name at hand, hence why it is currently named flow manager, which I agree is not the best either. I still think jobs is too precise though and it should be named from a step further.

We could name it stalker_api for now, in the backend folder. It would be more coherent with the frontend folder, and it's a proper name for the api part if/when we eventually split it. We later only have to find a name for the workflow micro service, which is a problem for future us.

My vision was "everything related to jobs", including subscriptions and such. Maybe "job_manager"? I'm not fond of suffixing with "_api", since those are all technically all APIs.

I agree though that splitting the FM's service could be beneficial: easier scaling, less unnecessary scaling of lightly used parts of the code, smaller code base in service, splitting of concerns, etc. And there is also kind of a natural split in the logic already. It would have to be properly thought out, but it could be beneficial.

Do you see another queue between the API and the workflow split services? Would you wait until it is an issue, or would you tackle the potential problem right away? We could rethink the architecture as needed when we need to scale the FM? What do you think?

I'd tackle this problem when we cross that bridge!