rAudio 20250422: Counting problem in "Latest"

[klassikmann]

@rern
General remark:
This is a minor issue for me. I can "live" without the "Latest" feature. But the root cause of the problem described below could affect other features also.

This thread is a continuation of an earlier thread starting with comment #1885 (reply in thread).

Observations:

• My library view is configured as

• The album count and view contains 4.122 entries. This the correct.
• "Latest" contains 4.120 entries, 12 less than the album count.
• If I add a new album the other day without "Append Latest list" the "Latest" count does not fall back to 1 as in former releases.
  This behaviour alone left the "Latest" function useless.
• New counts: "Album" 4.123, "Latest": 4.121 again 12 entries less.

Compare now both views:

1. Album View:

2. Latest View:

Observations:

• The first 4 entries ("Acoustic Guitars", alphabetically first entries) vanished.
• The first 7 entries of 12 of Haydn with "Ádám Fischer" vanished.
• The entry of Bartók with "Ádám Fischer" vanished.
• All remaining entries in album view and "Latest" view are identical.
• Therefore the 12 missing entries are identified.

I suspect as the "culprit" is the album artist starting with a diacritic: "Ádám Fischer".
My conjecture: There is a difference in the process of building the two list, my guess: something around "sort -u".

[bmw-biker]

I can confirm similar counting issue in Webradio.
Count is about double the real number of entries.
I remember the count being correct in 20250404.

[klassikmann]

@bmw-biker
I can't confirm the counting issue regarding webradios. I have 127 webradions and the count is 127.

[bmw-biker]

Sorry for the confusion...
After doing some reboots and testing some other stuff today Webradio count is correct again at my player.
I should have taken a screenshot 🤣
...
Oh wait - just happened again - I'll put that into another issue...

[hjfisch]

Also no problem with counting .

[klassikmann]

The problem I describe above stems from the use of non-ASCII characters, e.g. in "Ádám Fischer", which are encoded in UTF-8, the common character encoding standard. To put it simply, non-ASCII characters are encoded in a multi-byte scheme.
Consequence: An alphabetic sort doesn't always produce the expected result. A "-i" option (ignore-nonprinting) to the "sort" command could perhaps deliver better sums (I dont know).

[klassikmann]

rAudio 20250502: No change.

[rern]

Try:

• Disconnect all USB and NAS
  • Update Library > all counts - reset
• Add album 1 to SD
  • Update Library > Album - 1, Latest (none - initial update)
• Add album 2
  • Update Library > Album - 2, Latest - 1
• Add album 3
  • Update Library with Append > Album - 3, Latest - 2
• Add album 4
  • Update Library > Album - 4, Latest - 1

[klassikmann]

@rern
First, thank you for your hint to initialize the counters. It worked as described.

But ....
After removing again the few albums on the SD for the test and restore the NAS, the "latest" counter still was 0, as expected. After that I added a new album to the NAS. And I had the same situation as before:
Album: 4132
Latest: 4120

I'm quite convinced about the cause: as mentioned at the end in my OP:

I suspect as the "culprit" is the album artist starting with a diacritic: "Ádám Fischer".
My conjecture: There is a difference in the process of building the two list, my guess: something around "sort -u".

Details in my OP.

[klassikmann]

rAudio 20250627: Issue still unsolved.

As I wrote in my OP:

I suspect as the "culprit" is the album artist starting with a diacritic: "Ádám Fischer".

[klassikmann]

@Sagnasty
@rern
I think I found at least a workaround for the "Ádám Fischer" problem. I'll describe it tomorrow; hint: I'm working with locales.

[klassikmann]

@Sagnasty
@rern
To identify a possible bug in creating the set of LATEST albums the concept of "locale" (see https://wiki.archlinux.org/title/Locale how to deal with locales) is crucial. Let me start with an example sort four polish names "Zbigniew", "Ludmiła", "Żaneta", "Łukasz".

If no specific locale is set the sort command delivers an undesired result:
Ludmiła
Zbigniew
Łukasz
Żaneta

If a UTF-8 locale is set the sort command delivers the desired result:
Ludmiła
Łukasz
Zbigniew
Żaneta

Back to the determination of the LATEST count.
My library contains 4.183 albums today. I started the tests resetting the counter as proposed by rern in this thread.

If no special system locale is set (i.e. locale is "C" or "POSIX") the LATEST count is 4.171 as usual 12 less then the album count.
AFAIK: If locale is "C" or "POSIX", then ISO-8859-1 will be used as the filesystem charset and not UTF-8. As a consequence sorting will be performed according the old-fashioned ISO-8859-1 and the power of UNICODE/UTF-8 is unused.

Setting the system locale to LANG=de_DE.UTF-8 (Germany) or LANG=pl_PL.UTF-8 (Poland) or the well-known LANG=en_US.UTF-8 the LATEST count is 4.183 exactly the same as the album count.

I'll stay with LANG=de_DE.UTF-8 for the next weeks to see whether my argumentation is valid. I recommend other users to use their familar locale. Use the link to the archlinux wiki mentioned above.

As for today I see no bug in rAudio in counting LATEST albums (at least in my installation).

[klassikmann]

@Sagnasty
@rern
New results:

• After any further library update the LATEST count is no more identical to the album count but some hundreds smaller.
• I'm investigating the reasons. Up to now I found at least two scenarios where the library update is "forgetting" albums. I think the findings have an impact on rAudio 20250627 - Strange Latest count behavior #1971 by @Sagnasty
  OTOH the findings should help @rern where to look at his algorithm to determine the latest albums.

[klassikmann]

@Sagnasty
@rern

A week has passed in which I have been working intensively on the tag values of my library and the LATEST list.
What I have to say today is that my last enthusiasm has evaporated:

As for today I see no bug in rAudio in counting LATEST albums (at least in my installation).

Rather, I cannot recognize any regularities in the LASTEST list, which leads me to the following questions:

• What algorithm is used to calculate the LATEST list during a library update?
• What is the LATEST counter supposed to tell the user?
• What is the specification of the LATEST counter?

Let me present some examples from my library.

At the top level of my library there are 7 directories, simply described here as TD1 to TD7.

I'm beginning with an empty library, i.e. "album" and "latest" aren't present in /srv/http/data/mpd. Then I inserted one by one the subtrees TD1 to TD7 to the library and made a note of the counters for "album" and "latest" after an "Update all files" without the checkbox "Append Latest list".

I could not recognize any systematics in the figures, except for the correct album count 4191.

Then I built the difference of the files "album" and "latest" and found some interesting irregulaties (selection only).

1. My first example is best explained using classical music, but not restricted to.
   Consider your library contains three different interpretations of the symphonies of LvB.
   └── Beethoven, Ludwig van
   └── Symphonies
   ├── Orchestra1
   │   ├── Symphony No 1 C Major
   │   ├── Symphony No 2 D Major
   │   ├── ...
   │   └── Symphony No 9 D Minor
   ├── Orchestra2
   │   ├── Symphony No 1 C Major
   │   ├── Symphony No 2 D Major
   │   ├── ...
   │   └── Symphony No 9 D-Minor
   └── Orchestra3
   ├── Symphony No 1 C Major
   ├── Symphony No 2 D Major
   ├── ...
   └── Symphony No 9 D Minor
   In all sets you use Orchestra[123] as the value of the tags "Artist" and "Album Artist", in the first two sets the tag "Album Title" equals to "Ludwig van Beethoven: Symphony No ". But the third set is adding the opus number "Ludwig van Beethoven: Symphony No op. <nn". The third set is contained in the "latest" file, but not the first two. Why?

2. In my "latest" file the are no albums by "The Beatles", "The Rolling Stones" and other artists with a leading "The", but the albums by "The Who" are present.

I have found some other "unexplained phenomena" that are not so easy to describe. I now have the choice of simply not using LATEST or trying to help providing the originally intended function. But first, I need from @rern the definition of LATEST. My interest in LATEST is a function informing me about the newest (daily, weekly, ...) contributions to my library.

Cheers klassikmann

[rern]

Latest album list created by comparing:

prev=$( cut -c 4- /srv/http/data/mpd/albumbyartist-year )
new=$( mpc -f '[%albumartist%|%artist%]^^%date%^^%album%^^%file%' listall \
                    | awk -F'/[^/]*$' 'NF && !/^\^/ {print $1|"sort -u"}' )
latest=$( comm -23 --nocheck-order <( echo "$new" ) <( echo "$prev" ) )
echo
echo PREVIOUS
echo ---------------------------------------------
echo "$prev"
echo
echo NEW
echo ---------------------------------------------
echo "$new"
echo
echo LATEST
echo ---------------------------------------------
echo "$latest"

[klassikmann]

@rern
Thank you for the code snippets. Now I have a better understanding of the "LATEST" semantics.
"albumartist-year is of not of a special interest for me, in particular I'm using the tag "date" not as the date of publication by the record label but set the "date" tag as far as possible as the recording date. Such I'm avoiding the as "date" the n-th remastering of e.g. a jazz record of the sixties.

What I'm interesting in is to find the additions to the music library since the last n days. I'll open another thread (category "Tips") to share my efforts to achieve that goal.