Discussion: Future of Phoniebox

[ZyanKLee]

To go deeper into the side-tracked topic about the code future on #763 I open this issue (as we are lacking an alternative communication method).

Yesterday I had a short look into the build, install and update scripts of LibreElec.tv - it left me barely any wiser after an hour reading through all the scripts, testing the build script and trying to figure how the heck the actual update of a LibreElec system works. (not the download but the flashing of the new version - what does it and where is that piece of code?)

Anyway: as far as I can see this project can be divided into some more or less separate sub-projects.

• Installer
• Webinterface (incl mpd-controller)
• Sound System (MPD / Mopidy)
• RFID-Controller
• Optional components
  • Streaming Services (Spotify, GMusic, etc)
  • Buttons / Rotary Encoders (USB/GPIO)
  • Soundcards (USB/GPIO)
  • Others

... did I miss something vital?

Questions:

• Is it ok to have all those parts in one repo or is a github organisation with multiple projects better?
• How can we organize the project without one of those parts constantly interfering with the others?
• What could a git workflow look like that allows us to maintain a stable version (currently: master branch) and development of new code / rewrites of existing code that can deviate in large parts from the stable version?
• Do we need another communication method beside GH Issues?
• With @MiczFlor as the sole maintainer of this project: how can we help him not to get burned up in the process? Is Micz trusting anyone enough to help him out with MR approvals and such?
• Is there a "masterplan" on what Micz / we envision for the phoniebox to look like in a year or so? (core-features, UI, addons, etc)
• Is the effort that would be needed for the definition of a formal plugin system worth the probably enhanced stability of the whole system?

[ZyanKLee]

I'd like to notify and invite at least the six most recent contributors (Micz has been mentioned in the post above) to this discussion:
@andreasbrett @s-martin @patrickweigelt @splitti @veloxidSchweiz @xn--nding-jua

[s-martin]

Questions:

• Is it ok to have all those parts in one repo or is a github organisation with multiple projects better?

I think one repo with sub directories for each component is fine. Phoniebox isn’t that big and I think different repos adds another level of complexity, which could be hard to integrate in a working product.

• How can we organize the project without one of those parts constantly interfering with the others?

Different sub directories with maybe directories for common stuff could work. Adding (automated) unit tests for each component could be added more easily as a separation in components forces clear interfaces.

• What could a git workflow look like that allows us to maintain a stable version (currently: master branch) and development of new code / rewrites of existing code that can deviate in large parts from the stable version?

I think the current workflow is fine in general. Alternatively we could use GitHub Flow, which is very lightweight. Master is always deployable and each feature/fix branches from master and after review/test is merged in master.

• Do we need another communication method beside GH Issues?

I think something like Gitter, Slack or similar could help for quick questions and discussions of ideas.

• With @MiczFlor as the sole maintainer of this project: how can we help him not to get burned up in the process? Is Micz trusting anyone enough to help him out with MR approvals and such?

I think, if we try to add more tests like @veloxidSchweiz has already started (and automate them with Github Actions), we become more confident that changes don’t break the product, so @MiczFlor doesn‘t need to review and test everything so thoroughly.

• Is there a "masterplan" on what Micz / we envision for the phoniebox to look like in a year or so? (core-features, UI, addons, etc)

Yes, I’m really interested what plans or ideas exist.

• Is the effort that would be needed for the definition of a formal plugin system worth the probably enhanced stability of the whole system?

Difficult question :)

[xn--nding-jua]

Hi,

because you asked me to give my opinion:

I think one repo with sub directories for each component is fine. Phoniebox isn’t that big and I think different repos adds another level of complexity, which could be hard to integrate in a working product.

I agree with @s-martin to keep the Phoniebox in one single Github-repo. In the past weeks I've created two pull-requests and was able to understand the functionality of the code within some evenings. If the code is splitted into several individual repositories, for people like me (with low/middle skills in php but some ideas) it could be harder to contribute to Phoniebox. Keep in mind that the project is focused on parents and childrens.

If a more complex version is nescesary, maybe the repo can be forked to a "PhonieBoxPro" with enhanced functionality? It may then be justified to create a distributed repository.

By the way: thank you for this wonderful project!

[MiczFlor]

Hi guys, this is the right thread, just at the right time. I took a little time to write up a few thoughts regarding Phoniebox. Possibly this should - eventually - become a small manifesto about the idea, progress, community and future of phoniebox?

I believe the few posts above already illustrate the nature of the Phoniebox community. Generally speaking: I am very much open to the idea of sharing responsibility, giving access to merge and plan together. From the past (years!) of experience, I have seen great minds and coders come and go. So all in one place is a good idea, to make sure the project actually has a future. (I am also happy to have a Phoniebox meetup of developers in Berlin, if there are enough around. Any interest?)

https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/Architecture_HowWeWork

Phoniebox consists of code, a lively community and inspiring designs

A Phoniebox...

1. ... is easy to install.
2. ... out of the box plays local audio files and can be controlled with RFID cards and through a web app.
3. ... is not complete until fitted into a unique, jaw-dropping and eye-opening box. Everyone builds an original.

How the Phoniebox community works

1. Phoniebox should inspire people to tinker with technology.
2. Contributors are welcome!
   • more than code: feature requests, filing bugs, engaging in the discussion are all contributions.
3. The Phoniebox code lives in one place.
   • making the future of Phoniebox independent from individual developers.
   • contributors come and go. And that's ok.
4. The discussion around the future of Phoniebox is open to all.
5. Sharing responsibility.
   • while the project is comparably small, not one person understands all the code there is.
   • peer reviews and merging pull requests are responsibilities to be shared with the community.
   • peer support keeps the project alive (and is working very well, thanks to the community).
6. Contributing means great responsibility: major rewrites and refactoring are only taken on if the one or more developers plan to see it through to the bitter end...

The Phoniebox code base

1. The Phoniebox code base is free and open (currently available on github).
   • Open meaning:
   1. the source code can be accessed and read by anyone who choses to do so.
   2. the code base is open to contributions by other developers to improve stability and add features.
2. the code base is open to interface with new hard- and software and services (soundcards, RFID readers, displays, smart homes, GPIO buttons, barcode scanners, Spotify, Google Music, ...).
3. Where possible, configuration beats hard coded solutions.
4. Configuration should not be more complex than a hard coded alternative script would be.
5. Phoniebox thrives towards a unique API (currently the shell scripts pretend to be that).
6. Phoniebox incorporates many languages (bash, php, python, js) if need be.
7. Phoniebox depends on APIs which are out of our control (e.g. Spotify integration).
8. The architecture, structure and coding convention of the code are open to improvements (naming conventions, folder structures, etc.).

[MiczFlor]

From my more abstract statement above, I deduce the following action points:

1. stability: the develop branch should become stable first, before we embark on reqrites and configuration
2. merging pull requests. Who would want to get access to that privilege - and if only temporary
3. configurable components: yes! In practical terms, I suggest that at least two or more people should commit themselves to drawing up an idea, suggesting it to the community and coding it :)
4. In order to keep Phoniebox in one place, improving documentation, structure, architecture would be good

... what did I miss?

[s-martin]

I'd like to add:
Testing and test environment.

Adding first unit tests was already done in #763 (can't wait until that is merged). This could be improved step by step, maybe even for integration tests with hardware (mockups).
In #738 docker files were added, so this will be a big help.

If we can write a wiki article how to set up a test environment on real hw or docker, this could help many people and improve overall test coverage.

[Flowdeeps]

I think one repo with sub directories for each component is fine. Phoniebox isn’t that big and I think different repos adds another level of complexity, which could be hard to integrate in a working product.

I agree with this. If you look at how Google organises something like the NSynth project which has four different very definitely interconnected but also very different components (hardware schematics, audio pipeline, interface software, firmware, and PCB…) it's all separate folders each with a README but a root level README giving the overall information.

https://github.com/googlecreativelab/open-nsynth-super/

[Flowdeeps]

6. Contributing means great responsibility: major rewrites and refactoring are only taken on if the one or more developers plan to see it through to the bitter end...

This is why you force contributors to make forks with feature branches and create pull requests based on their suggested changes. I've seen some quite correct suggestions about master being always deployable and only merging after manual and automated pipeline acceptance testing. 👍

[MiczFlor]

@s-martin great energy, a wiki article would be great. And how would you write Testing and test environment. into the above list. Here my suggestion:

https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/Architecture_HowWeWork

How the Phoniebox community works

...

2. Contributors are welcome!
   • more than code: feature requests, filing bugs, engaging in the discussion and testing are all contributions.
   • testing is a selfless and noble form of contribution.

...

The Phoniebox code base

8. running existing tests is a must do, writing new tests is for kudos.

[Flowdeeps]

@MiczFlor I would love to join a Phoniebox group. Have you considered starting a meetup? It might be a good fit for the Embedded Linux or Linux Audio groups that already exist here in Berlin and have an active userbase. I could potentially do an introduction to some local hardware folks if you're interested. Alternatively there are the always excellent C-Base and XHain hackspaces who might be interested in hosting an event in exchange for exposure and beer money through the door.

[splitti]

First of all: sorry for my late answer, my time in the early year is really expensive, but i look forward to invest the time i can give to this Project.

Questions:

• Is it ok to have all those parts in one repo or is a github organisation with multiple projects better?

The basic Installation should be one repo in my opinion. I would like to see more repos, that can be attached like modules, maybe also as an installer. I also think, that config files, which will be edited manually, and also Shortcuts, Music, Playlists and so on should be seperated from the basic github repo in the installation. That makes it much easier to get an update working... But this is only my point of view.

• How can we organize the project without one of those parts constantly interfering with the others?

First of all, we should think of a stable and actual repo. @MiczFlor this is in german, because i don't find the words in english: Das Projekt hier ist der Hammer und ich bin Dir dafür mega dankbar. Aber es ist in dieses Repo so viel rein geflossen, dass es sehr schwer zu überschauen ist. Nimm das nicht als Kritik, sondern eher als Verbesserungsvorschlag, dass man Modulbasiert verschiedene Repos anbietet.

• What could a git workflow look like that allows us to maintain a stable version (currently: master branch) and development of new code / rewrites of existing code that can deviate in large parts from the stable version?

The main problem is, that there must be a main developer and tester for each step. One problem is also, there are maybe only two contributers, but both checking in to develop. Maybe it is better to make a repo like develop_username_timestamp? I am not sure, this is not easy for @MiczFlor to handle.

• Do we need another communication method beside GH Issues?

If there were different developers that organize the developent: yes. At this stage, i think no, because contributers are coming and going. There a just a handful of developers here about months.

• With @MiczFlor as the sole maintainer of this project: how can we help him not to get burned up in the process? Is Micz trusting anyone enough to help him out with MR approvals and such?

Ask @MiczFlor

• Is there a "masterplan" on what Micz / we envision for the phoniebox to look like in a year or so? (core-features, UI, addons, etc)

This would be a nice plan! But you need main contributers always working for this project, not just for a couple of weeks.

• Is the effort that would be needed for the definition of a formal plugin system worth the probably enhanced stability of the whole system?

Do you have an example.

So, i orderd three more Pis, i will build three more boxes to test. I hope, the project goes on and i try helping as much as i can.

[veloxidSchweiz]

Hi,
Thanks @ZyanKLee for starting this discussion. I think this is a goo opportunity to go forward.
@MiczFlor: I like your 'HowWeWork', this would have helped me when I started/thought about working on that project.

Here some personal opinions from my experiences. I am not a full-stack developer (whatever that means), but I already worked in some software projects from different perspectives.
The company I am workin in, is building electronic devices which requires some quite advanced software.
What I learned there is that, first of all testing is one of the key components to have a code which is always deployable. This includes good unit tests but also a bunch of system tests.
We experienced having one mono-repo seems to be the best way to guarantee the testing the code. This does not mean that the code should be separated in different modules, The modules could, for example, be maintained in different folders. Setting up multiple build-chains/different test chains for each module seems to be an overkill for me and does not really simplify the workflow.
One important step I see is that one should think about the interfaces. I.e. how is the interface between the php code, the gpio control ,....
For the testing the deployment I like the idea of using docker and/or ansible.
Some more comments on testing:

• I think testing is one of the best possibilities to have a certain 'trust' in the changes you do. It helps to understand breaking changes, as this will typically be visible by a change in the test.
• As we are using hardware it is important that one has system tests. I personally try always to work without actual hardware and try to mock the behaviour, then when i have implemented the functionality i will validate it with actual hardware and adjust the unittests to actually represent the behaviour as good as possbile.

I am not an expert on running docker on RPis, but if there is possiblity to containerize the functionalities this would probably allow a very easy deployment.... One would run separate containers for each subject of concern and it would be quite easy to update single compontents... here i would suggest to have one for the html interace, one for the mpd (or one of his spotify sisters), one for the control of the RFID reader, one for the gpio handling (maybe one separate for the oled display).... By only 'mounting' the music- and setting-directory, the system is very likley to be good upgradable.
Even though docker is quite lightweight and does not cost many resources, I am not sure what would be the effect on systems based on RPiZero.... Such an idea is in my opinion a very nice option, especially as most communcation happens by mpc, bbut it would need some tests for evaluation.

This is only my personal view and I am open for such a discussion. Having such an open-source-project in which many people invest a marginal time of their free time is really great and I love this projects, If I can show my acknowledgement by contributing this is even better ;-)

[MiczFlor]

Thanks for the input, this is really cool.

Regarding the code repo: I agree with ...

• @s-martin ("I think one repo with sub directories for each component is fine.")
• @Flowdeeps ("it's all separate folders each with a README but a root level README giving the overall information.") and
• @veloxidSchweiz ("The modules could, for example, be maintained in different folders. Setting up multiple build-chains/different test chains for each module seems to be an overkill for me and does not really simplify the workflow.")

I know what it's been like to answer questions regarding other peoples forked repos who once posted a link to their file and then changed it a few weeks later to fit their specific, it didn't work for others and suddenly I had to take the heat and try to awake the contributor from the dead (who did great work, and meant to do good, and did). Then suddenly everybody was annoyed for no reason. I prefer people creating pull requests and merge to develop - then they want to be proud of their code, comment their code and I know it is tested (enough) even if I don't have the hardware (ok, sure @veloxidSchweiz this is not the type of testing you are talking about :)

... and would like to restructure the folders and rename the files to make it easier for new people to dive into the deep end. In terms of components or functionality, and following @ZyanKLee list in the first post, I think these are the "folders" we need - not sure how to name them... (what am I missing?)

• Installation
• Local audio files and folders (currently under shared, together with the shortcuts for RFID tags)
• Web App (currently htdocs Webinterface incl mpd-controller)
• Sound System configuration and howto (MPD / Mopidy)
• RFID-reader hardware
• GPIO buttons and rotary (USB/GPIO : WARNING that GPIO buttons might be taken by soundcards :)
• Soundcards (USB/GPIO : WARNING that GPIO buttons might be taken by buttons :)
• Displays (WARNING that GPIO buttons might be taken by buttons or soundcards :)
• External services (Streaming Services, Spotify, GMusic, etc)
• Bash scripts triggered by web app (php), RFID chip (python) or services (This currently(!), as I said before, functions like an API. everything eventually calls a bash script)
• Services (on startup, daemons .service files)

Any suggestions for naming and folder structure? I started this document in the wiki, please put suggestions there (the wiki might be the more collaborative editing space)
https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/Architecture_FoldersFiles

[ZyanKLee]

First of all: I'm glad this discussion was so well received and on time. :)
Thanks to all of you for all those great comments and ideas.

After @MiczFlor's latest post I thought about how to go on from here with his plan while avoiding any interruption to the installation of new Phonieboxes and came up with this idea:

First of all, I think that new github feature "projects" might come in very handy to organize the planned steps and workpackages without us cluttering up the main issues too much.
If @MiczFlor would create a project by the name of "reorga" (or something else along those lines), we can cut this massive undertaking in small manageable chunks and keep track of them in the project.

To also separate the code from the "legacy" system, I would suggest a new long-living branch with a similar name; i.e. "reorga".
Then master and develop could go on as they do with the maintenance of the current code. In the meantime reorga would be used as base-branch for everything related to the reorganization; with its own feature branches for each reorga ticket ( - ideally 😆 ).
After we are satisfied with its status, it will be merged to master (replacing everything in there) and develop will be refreshed from the new base as well.
New functionality would ideally only be added to the reorga - but I'm not sure if this is feasible as the reorga might run some time.

This is an attempt to illustrate the git flow:

The reorga branch should be set as protected - similar to develop and master to avoid commits being pushed directly.

[ddbkva]

maybe you can change the phoniebox software so that it can be integrated into the Logitech Media Center.
Squeezelite runs very well on a Raspberry pi.

[pabera]

It is an interesting idea to think about using Mopidy as a player only. It's already running when using Spotify and keeping both softwares in sync just increases the complexity. To be fair, this goes opposite to what the Phoniebox software initially had been made for.

There is still a lot of stuff the Phoniebox software entails.. but it might make sense to focus on this more.

[MiczFlor]

Hi

I want to give a little background, so we can move forward together. At the end of this post, a very top level idea of what I believe we need in place to make that bold move to mopidy.

@arne123
Regarding: where is the discussion happening? - we go with the flow :) and the general discussion would be here. Specific ideas, like the refactoring of the playout script should move to the discussions, like here: #1298

@pabera
Regarding mopidy as the standard player, you are right about pointing out a big inner conflict. My hope would be that the Phoniebox is always also a very low level entry project (like an IKEA kitchen: everybody can set it up, BUT experts can make it something very special and take it to a new level).

Phoniebox has been a fun project that has brought families together :) How so?

Phoniebox Classic

• plug and play, no soldering, local audio files
• I might be mistaken, but it seems more often dads with an IT background who have itching fingers at night during Elternzeit to apply their skills at something that is good for everybody in the family
• the minimal setup would be: install the Pi OS, plug all the USB stuff in, run the install script and control the functionality of Phoniebox as well as the playout via RFID cards. Plays local files, podcasts, web radio; control via Web APP over WiFi.
• the kids often -- at times with other family members -- work on the design of the box. I've gotten very heart warming mails about this part and it made me start the yearly calendar's to celebrate one of the beauties about Phoniebox: Everybody builds an original

Then there is the other Phoniebox

Phoniebox +

• Custome content provider: Spotify, Google Music,
• Custom hardware (GPIO, smart home integration, special RFID, LCD screens, ...) usually involves
  • tinkering with the code, coming to the forum for questions
  • soldering skills needed :)
• Still true: Everybody builds an original
• This is also where innovation is happening, where there are amazing community contributions to the code.
• From a developer's point of view: this is really how open source software can flourish and, like making a good stew, the more ingredients, the tastier the dinner.

The Phoniebox + makes me very humble, seeing the quality of code coming in and the innovative ideas. I guess everybody here knows the codebase enough and has possibly been around long enough to know the components folder where all these ideas and gadgets find their home. (Thanks to @s-martin again for doing such great work in supporting new community members to make their code contributions documented and standardised in a way that everybody benefits. Again, this is needed in an open source project. And Martin is doing a great job.)

The future of Phoniebox should provide both. And I believe it can provide both.

If you look through the issues filed here on GitHub, they are either newbies who are inspired and motivated, but fail to install the Classic version. And on the other end of the spectrum issues related to specific hardware, integrating Spotify, ... and mopidy, which has been a fountain of issues in the past. Especially Spotify together with mopidy has created headaches and frustration in threads in the past. Nobody went to capitalised messages yet, but you know that somebody is frustrated when they post questions at 9:17pm, 01:12am and 03:29am :)
At times this was also about the Spotify API which had changed. It's frustrating for us, too, because you can not control it, but you still get the blame :)

Phoniebox went through a major shift in the past: migrating the playout from vlc to mpd. If you want to see who was involved in that time, take a look at this extract from the contributor's timeline. From this past experience with Phoniebox, this is what I think would be needed:

• make the move to mopidy for version 3.0 (new branch) - which should be shipped in autumn 2021, so it's all stable for Christmas
• we need contributors who raise their hand and say: I am in, I'll be happy to take a part of the responsibility and stick around for six months to make that move. (new components can go to the new components folder, but this is heart surgery and once you are in, you can't leave the room :)
• we need to decide on the best way to develop. Docker might be the way to go? (@s-martin, what do you think?)
• we need to decide on name spaces and architecture so that everybody can work alone and the discussion is about the bigger picture, not the little details (possibly that's me overshooting the target?)
• we need to keep the Classic plug any play option in mind as the default. Like building a kitchen: it's operational when you have a sink and a cooker. Microwave, dish washer, espresso machine, Dunstabzugshaube, special LED lighting under the hanging cupboards, all that is where the fun and innovation is, but that's on top (if I am wrong about this hierarchy, please tell me, I want to believe it's possible :)
• Documentation is the essence of any open source project. Many people rightfully complained that the Phoniebox Wiki and the code base aren't in sync (enough). So if you want to contribute documentation based on experience, not code, you are as welcome as any developer!

What am I missing?

And who would raise their hand to say: "I am in"

[s-martin]

I'm in!

[pabera]

I appreciate that you have taken the time to summarize those thoughts. It made me laugh too. I am a dad with tech background, but my Elternzeit is yet to come .. all of it makes sense and I like the separation of Classic and +. Which must be kept as a recognition and tribute to the initial, great idea.

I help where I can. I want to say I am in, but I don’t want to give wrong expectations.

[arne123]

@MiczFlor I like to say thank you for the extensive background information and your thoughts
I absolutely agree with your points, especially the contributor and namespace /architecture part.
Same as @pabera I was struggling a bit to raise the hand to say yes I am in since I don't want to promise something I can not really keep. But I do like the project very much and therefor I'd like to give my best in contributing.

I totally agree that one of the major goals should be the a simple box as a low level entree project.
But I am not entirely convinced if these should be really 2 Versions where you need to make a hard decision at a very early point in time where not all consequences are clear.
I would think more about and feature based approach, which you can gradually increase (in complexity)
Also from technical perspective and my personal experience I know that it can be really painful to maintain 2 different versions of basically the same.

I hope more people will join this discussion. And say I am in, at least a bit.

In the past days I was a little bit looking around in the project, and I'd like to ask in what kind of state the code in python-phoniebox folder is. Was this already somehow running?
Does it make sense to continue from there or at least start over from there? Or is it kind of abandoned?

[Luegengladiator]

Thanks @MiczFlor and @s-martin for taking the project to the next level.
I would also like to vote for a modular extension.
You should avoid having to restart from scratch if you want to switch to Phoniebox+ after successfully working with Phoniebox.
Otherwise I like the idea, even if I have significantly fewer opportunities to participate in Python.

I would like to break a lance again for standardization.
Maybe you can now also think about packaging for a Raspbian repository?
Even if the package causes additional work, modularization using expansion packages could make life easier for the end user.

[MiczFlor]

Hi @pabera @arne123
Thanks for being in (a bit and possibly a bit more :)
I see now how my question raise your hand and say: "I am in" sounds really creepy :) I got carried away there by my own memories how far this project has travelled by the community.
If you are in, that mainly means that people like you for the good you do, collecting kudos for happy families (even more valuable in covid times) and at the end of the year, the Phoniebox calendar will mean a little more to you, when you think: "gee, the ideas that people come up with to wrap the ugly Pi in a cute box".
And being in means to have a say in architecture, namespaces, modularity and the like. Because you'll also be digging into the code after decisions are being made (and we actually have to figure out how to discuss and make decisions, if this is here in the issue section, under discussions on github or what other communication tool would be best for that).

[MiczFlor]

@Luegengladiator
we all agree: "You should avoid having to restart from scratch if you want to switch to Phoniebox+ after successfully working with Phoniebox" :) you are very right...

the idea of modularity is great, the idea of Pi packages is also great (but, see below).

Modularity will actually increase productivity, I assume.
Making and maintaining packages will decrease productivity, I assume.

a) It's an extra skillset that would really only make sense if there was dedicated developer for that task.
b) The entry level for code contributions would change quite a bit.

Regarding modularity (and standardization, of course) what do you have in mind?

[ZyanKLee]

Making packages would in best case be an automated task in a ci/cd system.

[Luegengladiator]

Hi @MiczFlor,

Professionally, I am on the road as an IT delivery manager in larger Linux environments and therefore familiar with the challenges of system operation. Even if that sounds a bit like "overkill" now. Which child wants to go without the box for 3 days and instead have a frustrated dad just because the update doesn't work as expected? I can see parallels here.

That's where I want to bring modularity into play.
I could imagine that there is a module

• phoniebox-base

which then the modules

• phoniebox-gpio,
• phoniebox external streaming (google, spotify, amazon, ...)
• phoniebox display
• ...

could reload. The personally used box can easily be assembled using modules and each module can be developed separately.
You can then still consider whether you want to support a packet manager. Personally, I would love to see this as you can simplify the tutorial by automatically installing necessary packages as a dependency and then automatically updating the box via system updates.

On the subject of standardization, there are defined rules within the project.
Your instructions and coding guidelines are the best examples.
I think the project has grown in the meantime so that it can step out of the "shadowy existence" of the user directory and integrate itself as a program into the respective Linux (currently Raspian).
Examples:
Optional software is usually under /opt
Logs in /var/log
PIDs in /var/run
(might be easier with package manager support)

This does not make handling the application any more difficult, but it also enables further customization options.
For example, it is not necessary to attach the entire storage space to /home, which may enable further combinations of uses, for example with retropie. Such a phonie game box would be able to take audio books and music and additionaly a game console with you. I've seen some Boxes with little Displays on the calendar. Perfect combination!
Conversely, there are beautiful retropies that look like an old Gameboy, which would work perfectly as a portable Phoniebox.
Especially with older children, I could easily imagine the combination.

To sum it up briefly:
So far, the Phoniebox has focused on a dedicated raspbian and has been completely oriented towards running perfectly on this.
Using the modules and standards described, it would be possible to address a significantly larger number of users who may see combinations of usage. Some use a Retropie and "only" want to expand it, others may use other project-driven raspberries and do not want to rebuild everything by hand. Last but not least, it would also be possible to use any Debian-based system as a Phoniebox.
The basic focus on the individual machine is correct and important as the origin of the idea! Many parents without background knowledge are able to easily build a working box (main use). However, the background knowledge of the end user or the associated parent does not matter to the software as it is built. The software runs just as well on any other Raspberry;)

[ZyanKLee]

I like the idea of modules that may be un/loaded based on the features one wishes to use.
Do you have any experience with such a software architecture in python? Or anyone else?

[ZyanKLee]

Hello everybody,

It's been a while, but the recent uptick in discussion here woke me from my slumber. At least a little. 😀

So first and foremost: thanks to Micz and Martin for their continued work on managing the project. Thanks to everyone for driving it further.

Since my time with the project there is a sore spot for me in this project. But that is not due to it being ugly nor bad code - those aspects are great.
But using PHP for the web interface in addition to python brings overhead that might not be necessary. Porting the web ui over to python would be my suggestion if you are willing to look into this?

[arne123]

In regards to php via python:
I did some (hobby)php website around ~10 years back, it was kind of company internal lunch ordering system.
I tried to convert it to an complete python approach a few years ago , but my feeling at that time was that if you are using python, your are more or less bound to an framework like flask or django which will have some additional things to learn before there will be an outcome.

Please correct me, but I still have the feeling that in terms of web development the php entry level is lower, and the integration with webservers like lighttp is much better.
Based on that my recommendation is, if there is someone around with some experience in that field, why not.
If not I think we should stay with php. Especially the parts with mixed html and php code have to be redone from scratch.

[pabera]

Using Python for web interfaces is like using monster truck tires for your every day car. Depending on where you have grown up, it might look weird but overall it works. PHP is the hated open source standard because it was the simplest way to run a server in 2000. If you'd ask me, I'd prefer JavaScript (and NodeJS) for both backend and frontend because you need it for the UI anyways. But what does it matter after all, it's just personal preferences.

If you think in components (in larger environments you might use the term Microservices), it does not matter if you use the same programming language for BE and FE or not. You'd chose the most suitable for each use case.

For the Phoniebox project, I believe Python is good choice for the BE. I would try to avoid both Bash and Python at the same time, though!

still have the feeling that in terms of web development the php entry level is lower

Using PHP in the FE is ok but there are leaner solutions today. It's easier to get started, but also easier to write spaghetti code (e.g. merging logic and presentation layers)

and the integration with webservers like lighttp is much better.

Not really :-) Running Python servers or NodeJS servers is equally easy today.

Based on that my recommendation is, if there is someone around with some experience in that field, why not.

I agree!

If not I think we should stay with php. Especially the parts with mixed html and php code have to be redone from scratch.

Even if we stick to PHP, this should be addressed ;-)

[MiczFlor]

I agree with you both. I don't believe Phoniebox will have an advantage of migrating the web app to python.

[arne123]

Dear all,

I did some more programming on idea level to advance my earlier proposed zmq IPC approach a bit.
Besides adding some python stuff (see later), I added a tool called pbc (the name is derived from mpc) to offer the possibility control the python-phoniebox (PhonieboxRpcServer) from command line.
Please have a look to components/cli_client/pbc.c in my repo, if you are interested
I did this in C to have a very lightweight and fast interface with nearly no further dependencies.

So there are now there possibilities to interact with the PhonieboxRpcServer (explained below):
scripts/phonie_access_objects.py for python
htdocs/api/zmq.php for php
components/cli_client/pbc.c for command line (or c if you extract the function)

So by now I implemented zmq in three environments and can confirm "zmq is like Sockets, just on steroids"
If you have done raw socket programming once, you will notice how easy it has become using zmq and how similar it is everywhere.

I like the modularization which @Luegengladiator is proposing.
If I have seen correctly so far there is not a yet a "real" python module, which you e.g. can install via pip (-e)
This is not my filed of experience, but I think it is worth to have a deeper look into here.
@Luegengladiator besides the modules you mentioned, do you have a more clear idea where you would start technical wise?

In terms of utilizing linux standard directories I just want to add /etc for configuration

Based on the python-phoniebox I tried to give some inputs into class structure here, but still have a lot of mind conflicts.
Originally I just wanted to base on the existing code base, but I could not really get my mind into that, a could not get into flow there.
So I ended up with taking some functional aspects out of that, but architectural wise I am going a different way now.
Starting from PhoniboxDaemon.py there are now classes in

PhonieboxRpcServer ZMQ Based API for Phonibox
PhonieboxVolume Alsa Control
PhonieboxPlayer MPD Control

which can just do very little at the moment changing alsa volume and reading mpd version is working.
(GPIO and RFID are set to None, since I am programming on my host pc at the moment.)

in PhonieboxNvManager I am just trying to sort my mind.

I know that the rest of the discussion is more abstract and this is very concrete deep down in some details,
just let me know your thoughts and comments.

Br.
Arne

[MiczFlor]

Thank you all for the input. Everything feels really important, interesting, relevant - and inspiring.

Going through this discussion, I am looking at everything (not as a programmer, but) as a product owner. And, most of you know this, the contributions from people like you exceeds my skill-set as a programmer by far :)

As a product owner and/or project manager, I do see the benefits in refactoring some core elements of the Phoniebox code. Also rewriting parts to python seems like a very good idea.

So if we look at the life cycle, the first generation of Phoniebox was a "stand alone" proof of concept (and not even called Phoniebox at the time). The second generation started when migrating to mpd (from vlc) and with that move, opening Phoniebox to Spotify and Google Music, therefore increasing the user base and with it the code contributions (a long list of hardware support as well as plenty of new features).

Now we are looking at the third generation coming up. The three components in the mix to make it future proof are:

• refactoring (architecture, modularity component-based separately maintained)
• efficiency (refactoring, migrating, abstraction, installation process)
• maintainability (community support, openness, documentation)

These are very common pain points for any software development as the product matures. Seems the Phoniebox was a good enough proof of concept to start with, because we are talking about the future. So there is a future! There is a community!

Below what I scraped from the discussion. I really like the C++ pbc idea. But it's also the typical kind of addition that increases efficiency, but only few community members would be skilled in C++ to help out and/or continue development. @arne123 do you think the little C++ application will be future proof forever? :)

And generally, I am weary that the idea of moving the installation process to a package manager will create extra overhead, no matter what package manager. Without having somebody "on staff" to look after this, the new releases on github will soon outshine the effort of once putting everything to a package manager - which will be quickly out of date.

What do the others think? Am I missing anything in the list? And since our ticketing system on github is very noisy with questions: what would be the best way to structure these ideas into tickets? @s-martin you've been starting milestones in github. Do you think that would be the way to go?

Thanks for the input, I would really love to see the third generation Phoniebox taking shape!

All the best, micz

Architecture

• integrate Phoniebox as a program into the respective Linux (currently Raspian). (@Luegengladiator)
  • /opt is where optional software is usually
  • /var/log for log files
  • /var/run for PIDs
  • /etc for configuration

Installation

• support a packet manager (@Luegengladiator)
• "real" python module, which you e.g. can install via pip (@arne123)

Player

• mopidy replaces mpd as a player daemon (@pabera)
• mopidy as player and JSON api dedicated options to interact with mopidy beside the normal functions like start, stop, pause, seek, ... (@StefanMinke)

Refactoring

• Migrate Bash to Pythonin the backend (@pabera)
• Refactor Web App PHP (e.g. separate application/business layer and presentation layer). (@pabera)
• Python is good choice for the backend (@pabera)

Rewriting

• C++ application pbc and the PhonieboxRpcServer (in python) (@arne123) More details here.
  • Examples:
  • scripts/phonie_access_objects.py for python
  • htdocs/api/zmq.php for php
  • components/cli_client/pbc.c for command line (or c if you extract the function)
  • Question: The C++ application pbc does not need to be changed with new Phoniebox features, if I see this correctly?
• JavaScript (and NodeJS) for both backend and frontend because you need it for the UI anyways(@pabera)
• Modularity and modules (@Luegengladiator) For example like:
  • phoniebox-base as the core application
  • phoniebox-gpio, phoniebox-streaming (google, spotify, amazon, ...), phoniebox-display
  • Question: What does modularity mean in actual code? How would you want to do this?

[s-martin]

I’m really glad that many new people are putting in new ideas and are already submitting PRs, because that means many people are investing time to improve Phoniebox.

I like many of the new ideas, my personal goal would be to keep the entry barrier low for new users which means to improve and simplify installation. During the last year there was a big refactoring of the install script, which improved the maintenance of the script a lot and we began to offer options during installation. But I think there’s still a lot of room for improvements.
In addition I want it for new developers as easy as possible, too. So new users can improve Phoniebox easily and provide PRs.

From my point of view we could try to refactor the bash scripts to Python, but I think that’s a major undertaking (there’s already a dead attempt in phoniebox-python). In addition playout_controls.sh is the backbone of Phoniebox, errors will affect all users. I think it could still be worth it, because when doing so we should try to improve the testability of the code and make the code base more robust to changes in the future.

second big issue I see is the Spotify edition, which obviously isn’t so error proof and e.g. @MiczFlor and me don’t use it personally, so helping others is more difficult.

and yes, i think we should structure the bug and improvement issues in milestones, so we get a better overview what needs to be done.

[arne123]

To answer @MiczFlor question: " do you think the little C++ application will be future proof forever? :)"
well everyone knows that software build ones never needs to be changed, since it always stable from the beginning on ;-)

At the moment this program is in a state of proof of concept, but I do think that as long as we keep the feature set low we can get this into stable operation for quite a while.
If some guys will support in defining the feature set and especially in testing, we can reduce the risk here.
e.g. I have not yet considered how results will be printed to the command line in a way which is easy to parse for scripting. Any proposals?

"Question: The C++ application pbc does not need to be changed with new Phoniebox features, if I see this correctly?"
Yes, this is correct, the interface should be quite generic, so new functional features will not effect that, as long as we stay in the scope of control commands. If at some point the wish arises e.g. to transfer files or larger amount of date it is a different story,

[damaev]

I'm not able to dig into all the contributing repositories, so please proof me wrong, but it looks to me like a clear setup for the main python file (hereafter 'main.py') is still missing?
Personally, I go for the idea of using events: Any module would then (probably only) be able to just notify events like 'Playback' with an attribute like 'pause'. The main.py would do the rest. The only thing the modules would be allowed to do is check some attributes.
For the module init: Any module would be initialized by the main python file (e.g. when in the config-file something like '[myModule]' appears, the pythonfile 'components/myModule/myModule.py' would be initialized). The modules would have to be based on some abstract class making sure they are all build up equally.
This should (hopefully) make it easy to integrate new modules. For the unusual event of a necessary interaction between modules they should probably be able to communicate with each other (on some basic level). So during initialization they should be listed by a searchabel name or something like that.

(Edit: I see there could be way more lines to highlight the all possibilities)

But thats just my idea of makeing such a huge project acessible to loads of modules while keeping the need for changes to main code as low as possible (when it would be finally working).

PS: Of course, it's possible to replace the event-notification by function-calls, but - as far as I know - anybody would be able to listen to the events and could then perform some tasks on their own.

[ghost]

Of course, it makes a lot of sense to have only one backend player with all the benefits of the mopidy API. But I am a bit weary when reading that mpd support will be dropped. However, I have mixed experiences with mopidy + spotify on Raspberry Pi Zeros and Pi 2s with minute-long boot-up times due to playlist updates and unresponsiveness up to the point of unusability. So, it should probably be ensured that things like Spotify integration can be disabled (or not even installed) to keep performance for local-only-mp3 Phonieboxes good on Pi Zeros. Did some of you make the same experience, or was that just me?

Another topic that could do with a going over is the way different RFID card readers are handled. Currenlty it needs manual copying/renaming of files and custom pin settings e.g. for the MFRC522 are only possible by editing scripts.

I am thinking of a concept where

• different readers are loaded as different modules (a single module to each reader).
• a readers.ini file is used to select the approriate reader and passes optional parameters to the reader module (much like the gpio_settings.ini)

That way new readers could be added but just adding a new module. That's it. Most of the current code interfacing with the readers themselves would stay the same. Only the interface to the daemon needs to be refactored. The major isssue here is not the coding, but the testing, since many differnt readers are involved they all need to be tested. I have at home a USB Reader, a RDM6300, a MFRC522, and a Adafriut PN532 and could cover quite a stretch of the different readers currenlty being used. If nobody has set their eyes on that task already, I would raise my hand an embark on that. What do you think?

[arne123]

Very good points,
I never had a deeper look in all reader files, just that one having the RDM6300.
I totally agree with your Opinion her, each individual reader class should be in a separate module.

Regarding configuration, I have the opinion that the number of configuration files should be reduced.
So my proposal would be that a top reader class has a parameter for the reader to choose, where this information is coming from should be the concern of the main module.

[s-martin]

The readers are on my list for a long time.

i also think we should have each reader separated (except basic USB reader)and should be in the components directory (where the docs and the config already are) and installed when needed.

[damaev]

I was thinking of an abstract class for the readers to define a plain setup for further contributions.
This way all readers would work identical to the system and they could be easily loaded during the start-up or likewise by an online config-change by just re/loading a different reader object based on the configuration.
But I'm lacking time. So there hasn't been a real contribution from my site yet.

Additionally, I would like to add (I haven't read all comments - lack of time - so it may be already said):
As the backend will, as I understand it, be narrowed down to one system (python or whatever), I hope object-oriented design is your plan, as the current scriptings-thing forces a lot of inelegant hacks, like the current develop-implementation of the stop-on-removal feature. There, in case of an inbox solution, the reader could then just trigger an rfid event and the card handling process would decide how to handle it, as it will be able to retrieve any information necessary, from the object (so hold and release action for e.g. playback -cars or a single events for system-cards).

[ghost]

@damaev: Good thoughts. Much of that stuff regarding the RFID readers is already implemented :-) It is currently work in progress in a draft pull request as I want to do more testing. Take a look here #1359 and some concepts here #1358. If you have any suggestions, let me know.

[pabera]

In addition playout_controls.sh is the backbone of Phoniebox, errors will affect all users

I am currently working on refactoring this piece to be individual bash files. I started the process before this discussion started, so I will try to finish it. It'll have 2 positive effects: 1) reaction speed will be a little faster and 2) separation of concern. With #2, we could transform each aspect to Python (or whatever else we come up with)

@MiczFlor Thanks for the credits above. I don't claim those ideas to be mine though. They seem to be right to me, but I don't have much experience yet with all this software.

I actually noticed what @ChisSoc said in the below comment as well ...

have mixed experiences with mopidy + spotify on Raspberry Pi Zeros and Pi 2s with minute-long boot-up times due to playlist updates and unresponsiveness up to the point of unusability

This is even true for Raspberry 3. I also have a lot of problems with it constantly rebooting (but I haven't dived deep into it yet, could be basically anything)

---

I believe there is a lot to discuss and think about. We are all locked down and all over places. Wouldn't it make sense to set up a video call and discuss some milestones, exchange some knowledge and create a backlog and maybe lay out a roadmap. From this backlog, current contributors and new joiners can pick up tasks and execute them without always rethinking the idea every month :-)

A nice side effect would be to get to know each other a little better.

Thoughts?

[MiczFlor]

@pabera asks: "Wouldn't it make sense to set up a video call and discuss some milestones"
I think that's a great idea. There has never been a "face to face" in the history of the Phoniebox(!). I had planned to do some meetup - which was around the same time that Corona took over the planet and the way we work.
Here some suggested time slots:
https://doodle.com/poll/csw7g6vwdvv4aemi?utm_source=poll&utm_medium=link
And here the link where we will meet:
https://meet.jit.si/Phoniebox

[MiczFlor]

Looks like we have a winner:

Apr, 12th
Mon 4:00 PM-5:00 PM (CET)

If you haven't made it to doodle yet, the link is in the message above. Unless there is a landslide shift towards another date, let's stick to the 12th!
Excited, micz

[MiczFlor]

pabera, ChisSoc, Lügengladiator, arne123 can make it that day. Could you please send me a mail to [email protected] - so I have your mail addresses in case there are any short term changes. Thank you.

[mimi-ac]

Hello everybody,
how was your conference yesterday? Are there any results concerning the next versions of phoniebox?
Greetings
Christian

[MiczFlor]

Hi @mimi-ac
the conference yesterday didn't feel much like a conference :) first time I realise that the term "video conference" really upgrades a meeting. In the end @ChisSoc @pabera @arne123 and I were present.
It was an hour long and it was mostly spent getting to know one another and sharing a bird's eye view on the pain points and pleasure points of the Phoniebox as a whole, meaning the code base, the "product" and also the community. I have meeting notes, but not shared them with the participants for feedback yet - and I am unsure there is anything concrete enough regarding milestones to share here already.
So we decided to have a second meeting and @arne123 created a new doodle here:
https://doodle.com/poll/dpm89348x6ivt5ct?utm_source=poll&utm_medium=link
This meeting should be an hour where we identify some "low hanging fruits" to work on. If you sniff around the issues, discussiona and pull requests, you will see what @ChisSoc @pabera @arne123 are currently working on. (there is also the list above in this discussion).
If you want to join, feel free and take a look at the doodle.