preview implementation of the Laminas ecosystem RFC

[Jurj-Bogdan]

Q	A
New Feature	yes
RFC	yes

Description

This PR is an initial implementation for the Laminas ecosystem RFC. There are still features I think should be implemented (i'll list a few at the end), but hopefully there's enough for a preview to help continue the conversation started in the RFC.

Current functionality

The laminas ecosystem:create-db and laminas ecosystem:seed-db commands are used to generate and update the list found on '/ecosystem' by using the user-provided data from ecosystem-packages.json.

As the consensus from the RFC is that packages must be installable via composer, the Packagist url is mandatory and used to make a single request per package, from which the displayed data is extracted.
If provided, the 'homepage' and 'githubUrl' data will be used to overwrite the data coming from Packagist, and the 'categories' are used as extra filtering options in the package listing.

The create-db command currently works similar to the blog version, regenerating the whole database file after each build, although i would change this to make it more future proof. I've thought of the seed-db command as a cron used to select packages not updated in X hours and update some of the relevant data - would something real-time (or closer to) be desired instead?

As for the listing page ('/ecosystem') I've left a low page size to make it easier to test the pagination with the few default packages. For now there's searching by package name and filtering by "tags" (keywords taken from packagist) and "categories" defined by the user in the json file.

what next?

• i'm aware of the failing psalm check, but seeing as this PR is subject to change, I'll make sure to clean that up once i have a better understanding of what is required from this feature
• maybe adding a github action to validate the ecosystem-packages.json file, to make it easier for the reviewer ?
• if the current approach is kept, i think changing the database creation command to only add the newest entries if the db already exists would be a good change
• design changes
• as both the blog search and maintenance status features provide json responses, this could be added to the ecosystem package list as well if desired
• expanding the instructions for adding a new package once the desired rules and approach is agreed on

[froschdesign]

@Jurj-Bogdan
What was the link to the preview?

[arhimede]

@Jurj-Bogdan What was the link to the preview?

Is not built yet, I need to do some git operations first

[froschdesign]

"laminas" does not make sense here, because we need to differentiate between integration in Mezzio-based and/or laminas-mvc-based applications. This means that the package can be used as a:

• module in a laminas-mvc-based application
• and/or via a config provider in a Mezzio-based application

Maybe the categories are not suitable for this and a separate entry or entries are required.

[Jurj-Bogdan]

At the moment the "categories" key is only used for filtering in the interface, so I added a few here and there so they'll be visible in the preview - once a new role is found for them, I can upgrade the functionality to fit the request of course.

Even if the current approach of "user defined keywords" is kept, some other guidelines could be defined for them if wanted.

[arhimede]

we need to brainstorm a bit about those possible categories.

[arhimede]

PREVIEW HERE:
https://preview-1-hy2vwsq-2ja7ciew2nbkm.us-2.platformsh.site/ecosystem/

[froschdesign]

My suggestions:

1. Use Bootstrap's cards and create 3 columns for large screens.
2. Maybe an image can be defined for a package entry, based on the Bootstrap icons or using the social media preview of a repository (Unfortunately I don't know if you can retrieve this from API.).
3. Remove the paginator.
4. Reduce the information of a package. For example, the number of watchers is not really necessary.

[Jurj-Bogdan]

A few details and questions about the latest commit:

1. The packagist api doesn't give access to the repository's social preview, so I've added a separate call to github's graphql api to only bring the image URL. A few issues here:

   • if a repository has no custom social preview, an image will be generated for it which will double some already available info (name, number of stars, example below), but I can't remove said information as I don't have a way to know what info is displayed in the preview (i.e. for a custom preview such as dotkernel api's that doesn't provide a name etc.)
     • to note is that the packagist api provides access to the repository owner's avatar, so if wanted this avatar could be used instead, so the separate github api call can be removed - however this means custom social previews would be ignored, it's your call
     • another option would be to add an optional "image" key to the json file, where the user would manually submit a URL to be displayed
       
       

2. Is this version of getting data from the api ("create-db" command run on build + "seed-db" command used as a cron for data refreshing every X hours) alright with you or would another way be desired? If it's OK i'd like to (and should) go ahead and clean up the code alongside any future changes.

   • In the initial commit, the "create ecosystem db" command worked similar to the blog one by deleting the existing db file and regenerating in on each build which will be problematic with large numbers of packages added. I've changed this so the existing database is kept and packages existing in the db are skipped when getting data from the api. This way will also ensure different "created/updated" dates for the cron to update packages in smaller batches.

3. about the data saved and displayed for each package:

   • i've kept the keywords taken from the package's composer.json and the user-defined keywords from the ecosystem-packages.json and am basically using them as filters, however they can clutter up the card in large numbers - do you think any of them should be removed/limited in number?
   • the Composer package type is now pulled from the API and displayed as it feels relevant, would filtering by type be a good addition?
   • similar to the above, i've added a user-defined "category" with the "skeleton", "integration" (or "wrapper" as mentioned in the original issue) and "tool" values available, which again seem to be a useful information to display and maybe to filter based on the intended use case - should I go ahead with this feature?

[arhimede]

Preview updated
https://preview-1-hy2vwsq-2ja7ciew2nbkm.us-2.platformsh.site/ecosystem/

[arhimede]

now the social preview is not looking good

We will try a variant to use as image or "social preview" only the github organisation logo .

[gsteel]

My 2 cents:

1. I think that using the GH social preview is probably the best plan - user avatars are a bit hit and miss and also less relevant - For example, the avatar for all the laminas packages is @weierophinney gravatar:
   
   Can we get away with no image at all as an alternative or does the layout get too boring? Also, what about libs hosted on GitLab/Codeberg etc?
2. Retaining the DB seems like a sane plan 👍
3. Usage of the composer package type is really worth-while - I don't think the concept of "type" should be extended beyond the ones composer offers, that said "category" is helpful, but I believe these should be small in number, predefined and packages should only appear in one category. The main problem I see here is: lots of "integrations" and a handful of "tools" and "skeletons" at which point category starts feeling less useful.
   I think that the main filtering should be done via user-supplied tags that are audited by us prior to merge, rather than relying on GH topics or composer keywords - this will be the main tool that will allow us to curate collections so that they are meaningful for users.

[gsteel]

Also, rather than "See on GitHub", perhaps just "Source" with the icon for the relevant source code hosting platform?

[weierophinney]

How are you/will you determine packages to include? I ask, because, for instance, the phly-simple-page package on there likely should not be (I've not updated it in 3 years, and am in fact planning on archiving all my ZF2 packages (even if I did update them for Laminas at some point).

[arhimede]

How are you/will you determine packages to include? I ask, because, for instance, the phly-simple-page package on there likely should not be (I've not updated it in 3 years, and am in fact planning on archiving all my ZF2 packages (even if I did update them for Laminas at some point).

All packages we have listed now are only test data.
In order to decide the layout and the logic of that page.
Will be subject of a future TSC meeting to decide the rules to include and remove a package from that Ecosystem page

[froschdesign]

1. You're going to hate me for this, but the term "ecosystem" is wrong here, because a few modules, packages or whatever you want to call them cannot be the ecosystem of Laminas. The listed packages/modules are a part of the ecosystem, but not the entire system.
   And as an end user, I would also not search for or suspect extensions and libraries from third parties under this name.
2. The amount of information can be greatly reduced, because licences, downloads and various links are too much. A link to the repository and - if available - to the documentation/homepage is sufficient. (Is the link to Packagist really helpful for a user? After all, it only redirects to the repository.)
3. If we also want to have packages/modules hosted on Gitlab or other platforms, then we need to rethink the information gathering process.
4. I would also like to point out again that we need a clear distinction between what can be used in an MVC application and what can be used in Mezzio.
   For example, LmcUser must be listed and it must be clear to the end user that this only works in a laminas-mvc based application.

[froschdesign]

A consistent layout would also be desirable, and the unnecessary spacing due to paragraphs is not necessary.

The description text at the end, then the problems are reduced. Example: https://getgrav.org/downloads/plugins

[arhimede]

1. You're going to hate me for this, but the term "ecosystem" is wrong here, because a few modules, packages or whatever you want to call them cannot be the ecosystem of Laminas. The listed packages/modules are a part of the ecosystem, but not the entire system.
   And as an end user, I would also not search for or suspect extensions and libraries from third parties under this name.

We can call it "directory" instead , even the word "ecosystem" is more catchy :-)
This will be a subject of a vote in TSC.

2. The amount of information can be greatly reduced, because licences, downloads and various links are too much. A link to the repository and - if available - to the documentation/homepage is sufficient. (Is the link to Packagist really helpful for a user?
   After all, it only redirects to the repository.)

Agree, now it is too much information.
@Jurj-Bogdan
One nice directory is the one for React Directory

3. If we also want to have packages/modules hosted on Gitlab or other platforms, then we need to rethink the information gathering process.

The information will be gathered only from Packagist, as I presume a mandatory condition for a package to be lsited here will be to be installable using Composer.

4. I would also like to point out again that we need a clear distinction between what can be used in an MVC application and what can be used in Mezzio.
   For example, LmcUser must be listed and it must be clear to the end user that this only works in a laminas-mvc based application.

@Jurj-Bogdan we need a Flag here, separate from "category".
MVC, Middleware, Mezzio, will see which one will be the default flags

[froschdesign]

By the way, this is boring: https://symfony.com/bundles 😉

One nice directory is the one for React Directory

Ah, this was the original inspiration for the layout.

[Jurj-Bogdan]

I have started tweaking the info such as the keywords from composer being taken out, but keeping the user defined ones (probably going to limit those to 5? as well). Will also provide a sort of flag as @arhimede mentioned, with preset values which should be used as filters as well (thinking of adding a navbar around the search input for these "preset filters" - category, type and the new "flag")

Thanks @froschdesign for the example, i'll work on a version with the social preview working similar to getgrav.org's images; that is unless @gsteel 's question of removing images altogether is preferred, of course.

[froschdesign]

Another example: https://astro.build/integrations/
Also interesting what information is required: https://docs.astro.build/en/reference/publish-to-npm/#integrations-library

[gsteel]

Adding relevant keywords to your composer package is a great idea - I wonder if the packagist api can handle that? We'd need a way of excluding packages too though.

[froschdesign]

We'd need a way of excluding packages too though.

The packages must be added manually, not automatically! Or have I missed something?

[gsteel]

I was just ruminating on being able to automate that - it's probably best to stick to the plan and add them manually as originally specified 👍

[froschdesign]

Adding must be done proactively and I'm not worried about that, because some people are already waiting for an official listing.
The next step would be a new series of blog articles under the name "Package/Module of the Month", or something similar.

[arhimede]

updated preview
https://preview-1-hy2vwsq-2ja7ciew2nbkm.us-2.platformsh.site/ecosystem/

[arhimede]

Preview updated
https://preview-1-hy2vwsq-2ja7ciew2nbkm.us-2.platformsh.site/

[Jurj-Bogdan]

a rundown of the latest update:

1. added package removal by deleting the entry from the ecosystem-packages.json file, so anytime a package is removed from the file it will also be removed from the database.

   • to note is that I haven't automated removal of packages in case they become abandoned after being added; abandoned packages are treated as follows:
   • any package that is abandoned before being added will be ignored by default (as "mezzio/mezzio-aurarouter" is right now in the file, kept there for testing purposes)
   • any package that is abandoned after being successfully added will be marked as abandoned by the cron and will display a warning on the listing page, but removal will have to be manual via commit
   • let me know if another option is preferable

2. psalm fixes

3. added an initial batch of unit tests

   • adding functional tests for the commands that write files will require mikey179/vfsstream or an alternative to mock creating/writing the db, would that be an issue?

[PowerKiKi]

I would remove images entirely, because they duplicate info, such as the most important one: the title. They are obviously sub-optimal for SEO. And they are hard to read because text is too small and contrast too poor, especially on a 770px wide screen.

Then we can get rid of the mouse over animation too, so that the card-text is entirely readable without having to move the mouse around (and avoid the "jumpy" effect it introduces).

If you want to keep some sort of image, then I would use the user/organization profile picture, which is guaranteed to have a more uniform layout, and provide a strong visual indicator of the library author(s).

In the preview, card-updated are all the same. If it will always be that way, then we don't need to repeat it everywhere. One mention somewhere on the page (header or footer ?) should be enough.

Similarly the "homepage" link is very often the same as the source link. In that case it brings no value, and should not be shown. In fact, I'd even suggest to never show the homepage link, because we already have two links (packagist and source) for basically the same thing.

[froschdesign]

Similarly the "homepage" link is very often the same as the source link. In that case it brings no value, and should not be shown. In fact, I'd even suggest to never show the homepage link, because we already have two links (packagist and source) for basically the same thing.

If the target is the same, then it really makes no sense.

---

@arhimede @Jurj-Bogdan
Individual notes, corrections and tasks listed here should be kept somewhere. Possibly as sub-issues.

[arhimede]

@arhimede @Jurj-Bogdan Individual notes, corrections and tasks listed here should be kept somewhere. Possibly as sub-issues.

Yup, we are about to update this PR tomorrow with some modifications and corrections
As i want to release this feature before jumping on implementing the new design