Community Heat Pump Analytics

[kr0ner]

Ahoi everyone 😃

I was thinking a lot recently on how to continue with the project and what would bring the most benefit to all of us.
Tracking values locally is mostly working, but we can't really compare the data nicely.

So my idea is to allow users to include an analytics.yaml that sends anonymized data to a cloud MQTT server and visualize the data with grafana.
Transmission of the data would happen every 15 minutes for sensors and event based for switches, fans and numbers.
Each user would get a unique id (hash of the MAC address) that would allow them to select their own heat pump and compare with the community.

This way we could figure out outliers and further increase our efficiency.

Please let me know what you think :)

[RedAnt79]

i would definitely share my data

[Quotic]

I would also participate (y)

[kr0ner]

This is what I have so far: #263
and this is where the dashboard will show up:
http://oneesp32torulethemall.freeddns.org:3000/

There is also some metadata transmitted on each startup incl. the state (Bundesland) and the config that is used.
A friend pointed out, that the region would be useful to be able to get better comparison.

[Nebnenut]

The state is very large. Perhaps someone could voluntarily share more information?

It's impossible to trace the hash back to the GitHub user.

[kr0ner]

I can add a parameter to the yaml that allows to use city instead of state.

[aLimp86]

Great idea. You can only benefit from such statistic.

[Oggy512]

After checking the Code to verify what data will be sent (no offense 😅), I would also Share mine.

[kr0ner]

The malicious bitcoin mining code will be sideloaded 😅

Jokes aside... I never did sth. similar before, so the PR is just the best I could do. I will definitely add certificates and store the credentials differently.

[kr0ner]

The current code lacks the possibility to exclude entities. I'm not sure if White- or Blacklist should be used for that

[Schmidtjanroman]

That's super interesting, I'd like to be part of that

[kr0ner]

Yesterday I've added the support for blacklisting entities. Also I want to restrict access to the system, so that only the people who are contributing their data will be able to see the data from the others. Might change that later

[kr0ner]

Ok, blacklisting was the wrong direction. It will be whitelisting to not expose other entities by accident

[Nebnenut]

Do you already have an idea for sharing the data with other users who share their data? I have an idea: You configure your GitHub name/ID in ESPHome, and Grafana can use GitHub OAuth to verify whether that ID is also authorized to provide access to the analytics. This could be checked every x weeks or at every login.

[kr0ner]

My idea was to use a publically accesible grafana dashboard and to use the hashed MAC address (the client id) that is printed in the logs to reveal the data

[kr0ner]

If you want real auth, I'd need your help 😅 but let's keep it lean for now

[Nebnenut]

My idea was to use a publically accesible grafana dashboard and to use the hashed MAC address (the client id) that is printed in the logs to reveal the data

It would be quick to implement. Create a sensor with its hash to find its dashboard in Grafana.

You might even be able to create a direct link.

[Oggy512]

I checked the code of the analytics branch and have a suggestion:
I think it does make sense to make the values for state (or city or lat/long or whatever location item is used later) configurable by the user.
Right now, you're using ip-api.com, which reports my WAN IP is located in Bavaria, which isn't true. When I lookup my IP on whatismyip.com it says I'm located in Berlin, when using https://www.iplocation.net/ip-lookup (where one get different providers for IP lookups) , my IP is either in Berlin or Bavaria or Hesse or Rheinland-Pfalz or Saarland (which is actually the correct location), depending on the service.

What I'm trying to say: the IP lookup services are apparently not realiable, so I suggest to let the user configure his location, since most likely the location of a heatpump won't change that often ;-)

[kr0ner]

Ok, I was not aware, that you get such crazy random locations 😅 makes no sense to auto detect. I'll add a parameter 👍

[Oggy512]

Ok, I was not aware, that you get such crazy random locations 😅 makes no sense to auto detect. I'll add a parameter 👍

Yeah, those services are good for countries, i.e. for Geoblocking, but for granular locations they're too unreliable...

I suggest to let the user configure his location. I think using the postcode would reflect the geographical differences within a (large) state like Bavaria, but would still be "inaccurate" enough to respect data-privacy concerns.
You yould then use an API to lookup the state for displaying this in Grafana, i.e. https://www.openplzapi.org/
With something like the following call:

curl -X GET 'https://openplzapi.org/de/Localities?postalCode=66571' -H 'accept: text/json' | ConvertFrom-Json | ConvertTo-Json

[kr0ner]

Ok, I'm almost there ... all sensors created through templates will be whitelisted automatically by default for now.
Each sensor can be disabled by passing a var exposed : "false" to the template.

Other custom sensors will not be exposed.

Analytics can be enabled by removing the comment:

# analytics: !include { file: OneESP32ToRuleThemAll/yaml/features/analytics.yaml, vars: { district: '"Landkreis"' }} <-- remember to change the district

The unique ID needed to be able to see any data in Grafana can be found in the diagnostic section as a text-sensor

enter it in Grafana to be able to see your data. Sidenote ... what a horrible UI Grafana has xD should've just used a Home Assistant instance ^^

http://oneesp32torulethemall.freeddns.org:3000/

Sensors are published every 15 minutes, everything else is published event based.

Need to figure out how to present the data nicely

[Oggy512]

Hmm... you changed the location to district, but I can't find any validation? How to make sure, people use the "right" (=same) term? When I look at one of my neighbour districs for instance, I come up with at least 4 names for the same district:
Sankt Wendel
St. Wendel
St.Wendel
WND

That was one of the reasons, why I proposed the postalcode, since this is a uniqe value, which is easy for people to know and understand.
If you want to stick with districts, there are also uniqe ids for those, however I assume the majority of ppl are not aware of this...

You can find them here:
https://www.xrepository.de/details/urn:de:bund:destatis:bevoelkerungsstatistik:schluessel:kreis

or here:
https://codefor.de/projekte/da-ars-tool/

Even if you do not do a automatic value lookup/validation, at least a hint in the readme could be meaningful, to make sure, ppl are able to enter the correct location?

[kr0ner]

The postal code is too revealing. Nobody should be forced to share their location. The data is only to be used in Grafana to show the locations on the map. I can sanitize the input serverside

[kr0ner]

Final version so far: #263
Feel free to review, leave comments and try out. I will try to get some basic Grafana Dashboard up and running, so that we can visualize the data.

[Oggy512]

These are the outlets in the rooms, they "connect" directly to the hose in the wall, no filter, damper or something in between.

Here are the dampers, which we had initialy only:

Any now we have those dampers additionally:

[kr0ner]

except for the plastic hose diameter it looks pretty similar

[Sulik7761]

I have no issue with noise. I have this type:

[kr0ner]

Quick explanation also from my side again ... at some point I noticed that the ventilation is really loud. It was set to level 1 but sounded like level 3. So I tried to look at the filters and all this stuff with no result. Then I remembered that a friend of mine had frozen intake and ice blocking the air.

That was also clearly visible in the data. ZULUFT_IST went up by 100% in no time. After removing the ice, it went really silent again. So maybe you @Oggy512 can double check the intake and outlet again 🤷

[Sulik7761]

Yeah, I got this few time.
It happened during the freezing fog period.
I keep my two step close now :). And sometime go out removing ice 😅
The first time my neighbour looked at me weirdly 😆

[reneschuster]

I also updated to the latest analytics version from today and the first data has already been transferred.

[kr0ner]

I think there is sth. wrong with your code. Did you use the exact analytics yaml I created? Your ESP is pushing 10x more messages than the others and sensors seem to be updated individually instead of as a bulk every 15 minutes.

Edit: I've limited the topics that are allowed to be pushed to for your endpoint. The spam floods the db

[reneschuster]

I have incorporated the changes from the “introduce_analytics” branch into my version, as I have made some changes for my tests of the THZ 304 eco heat pump.

I created a diff on the file system and only the files src/property.h and yaml/thz_base.yaml differ. So far, I have only disabled properties that are not available on my heat pump. I also made a copy of the file yaml/thz404.yaml to yaml/thz304.yaml with a small change.

I have just reinstalled everything again. So feel free to check if anything has changed.

[kr0ner]

Weird, did the esp maybe restart? I saw frequent writes e.g. every 15 seconds to numbers or lueft_stufe settings which should be quite static 🤷‍♂️

Or maybe it is entirely my fault... on_value is executed whenever the heatpump sends data. God damn 🥲 I will implement a safe guard as I did for PROGRAMMSCHALTER earlier. Sorry for the mess

[reneschuster]

No restart of the ESP. If I can help in any way, let me know.

[reneschuster]

I have seen, that only at the beginning mqtt messages are in the log file and nothing later when messages will be sent.
Maybe some additional mqtt logging would also be good for troubleshooting and to see if all is working.

[aLimp86]

I updated as well, but I can't find my device id and I don't see any data on the website. Probably because I'm not logged in?

[kr0ner]

The change is still not merged to master.
device id appears as a new sensor in HA under diagnostik sektion at the bottom 😉

[kr0ner]

Since I'm using a free hosted VM, I need to pay attention to the traffic as there is only little budget. Everyone can join the game by uncommenting analytics.yaml but only the info topic is allowed to be published initially. I will get a notification and can whitelist the device id to allow pushing all sensors. Don't know if this makes sense though or if it will lead to lots of errors on the device until I enable all topics. Please let me know what you think or if you have a better approach.

And that was b*llshit ... only outbound traffic is limited.

[sascha-egerer]

Just joined the party. Thank you, once again :) Will be very interesting to see the data.

But somehow DNS Resolving does not work and i can't find the reason. Still investigating.

[21:48:21.793][W][mqtt:244]: Couldn't resolve IP address for 'oneesp32torulethemall.freeddns.org'
[21:48:21.800][W][mqtt:367]: Disconnected: Unknown
[21:48:21.913][I][mqtt:277]: Connecting
[21:48:22.154][W][component:430]: mqtt cleared Warning flag
[21:48:22.154][I][mqtt:318]: Connected
[21:48:22.257][W][component:547]: mqtt took a long time for an operation (104 ms)
[21:48:22.261][W][component:550]: Components should block for at most 30 m

[reneschuster]

[21:48:22.154][I][mqtt:318]: Connected

It should work, I also had the same messages and it was working.

[16:52:44.745][W][mqtt:244]: Couldn't resolve IP address for 'oneesp32torulethemall.freeddns.org'
[16:52:44.755][W][mqtt:367]: Disconnected: Unknown
...
[16:52:45.134][I][mqtt:277]: Connecting
...
[16:52:45.461][W][component:430]: mqtt cleared Warning flag
[16:52:45.466][I][mqtt:318]: Connected
[16:52:45.568][W][component:547]: mqtt took a long time for an operation (106 ms)
[16:52:45.571][W][component:550]: Components should block for at most 30 ms

[sascha-egerer]

ah ok... i was already confused by the "Connected" info. The log output is a bit weird sayin "could not resolve ip" and 0.3 seconds later it is connected... Anyways, then i'll just wait and see what happens.

[kr0ner]

the timeout is a bit low in ESPHome :/ cannot do anything about it

[Nebnenut]

Either I'm too stupid or my ESPhome is having real problems right now. I just can't get it to compile. I've loaded the fork version and adjusted the path in my YAML file.

I'll wait until it all goes into the master version. :/

https://pastebin.com/raw/YVbeUxrU

[Nebnenut]

Am I missing something? If I download the fork (https://github.com/kr0ner/OneESP32ToRuleThemAll/tree/introduce_analytics) and put it in a separate directory, all the changes are included, right?

Just adjust the path in the YAML file and you're done.

[kr0ner]

can you look at the util.h in there and check if it contains?

/**
 * @brief Singleton access to whitelisted entities that shall be included in analytics.
 */
std::unordered_set<esphome::EntityBase*>& getWhitelistedEntities() {
    static std::unordered_set<esphome::EntityBase*> entities;
    return entities;
}

/**
 * @brief Whitelist a set of entities that shall be included in analytics.
 */
template <typename... Args>
void addToWhitelist(Args... args) {
    (getWhitelistedEntities().insert(static_cast<esphome::EntityBase*>(args)), ...);
}

[Nebnenut]

Right side is the folder is the fork for analytics; left is the "master".

My YAML changes:

[kr0ner]

If you do it like that, you also need to go to core.yaml and adopt the path to the utils.h
I highly recommend to use git to switch between the branches. It is far less error prone 😉
There is only a few commands that you need to know and in the times of AI this can be easily solved

[Nebnenut]

Data arrived from Düsseldorf!

i pressed Compile at the same time as your reply.

[reneschuster]

Just updated to the latest version. You can take a look. ;-)

[kr0ner]

looks way better now :)

[Oggy512]

Feature Request (putting it here, instead of own issue?)

Can we include a diagnostic sensor which populates the latest MQTT push timestamp? Since MQTT pushes are not logged in the livelog, one does not have any indication, if analysis is working or not.
With such a sensor one could at least see, if the last transmit was recently?

[kr0ner]

pushed the changes

[Nebnenut]

How would you like to receive future requests? Via ticket (create issue categories)?

I also have a couple of ideas. One would be to add the type of heat pump to the firmware sensor. There are different heat pumps, and you can't compare them "firmware-wise."

I'd like to know who has version 5.5 and a newer firmware than me :)

[kr0ner]

the filtering based on heat pump variant will be added add some point ;) let me first get the grafana right for the stuff we have :D

[kr0ner]

How would you like to receive future requests? Via ticket (create issue categories)?

as an issue would be perfect if you already know exactly what you want e.g. propertyXY is misisng

everything else can happen here

[Oggy512]

Greate, just flashed and looks good :-)
Improvement: rename "Last Analytics Update" to "Analytics Last Update" and "Global Device ID" to "Analytics Global Device ID". So are all anayltics reletaed sensors together in the view.

[kr0ner]

Grafana seems to be finally working. Please tell me what you think of it

[sascha-egerer]

Pretty cool. Thanks a lot. But we need more participants now :)
Some stats look weird. It looks like some sensors are sending "0" values sometimes and this breaks some of the stats.
Example:
ABTAUEN_VERDAMPFER Analysis

[reneschuster]

That screenshot can't be ABTAUEN_VERDAMPFER, because ABTAUEN_VERDAMPFER has only on or off.

[kr0ner]

Your screenshot also looks like the version from yesterday 😉 ... by now the UI glitches should be fixed

[sascha-egerer]

Yes, look better know. I'm not sure which value it was. But due to this data I can finally prove to the heating engineer that one of the valves is broken. Thanks :)

[kr0ner]

Great to hear that :)

[aLimp86]

Now I've updated and included analytics. Did it work? Are you guys seeing my stats?
I now have a 'Analytics Global Device ID' entity. That is the one which must be written into Device ID in Grafana?
I still can't see any data of you all.

[sascha-egerer]

Yes you have to use this for the interface to see data. But if i understand correctly @kr0ner has to accept your id first.

ah ;)

[aLimp86]

ok, then I have to be more patient. Thank you. :-)

[reneschuster]

Now I see 6 heat pumps, so it should work.
It takes some time until your see your data and then also the community avg, min and max.

[kr0ner]

@aLimp86 you should be good 😃

[aLimp86]

Works

[kr0ner]

Although the Grafana-Dashboard is nice to have, what do you think ... should I create an outgoing topic to which I publish the community avg,min&max so that one can visualize it in Home Assistant?

[kr0ner]

Ok, so my plan to add the min, max, avg automatically as attributes to the sensors that are exposed to Home Assistant does not fly :-/ ESPHome does not support setting attributes. Nevertheless I've automated the registration process so that I no longer have to allow new heat pumps manually. Additionally you can now connect to the MQTT Server using you client id as both username and password. You have access to the community/heatpumps/aggregate/# topics, which publish the avg, min and max for each sensor every 15 minutes. You can bridge your mosquitto HA add-on with the oneesp32torulethemall.freeddns.org mqtt server using this config:

connection OneESP32ToRuleThemAll
address oneesp32torulethemall.freeddns.org
remote_username <client id>
remote_password <client id>

topic community/heatpumps/aggregate/# in

I've used this guide:
https://solar-assistant.io/help/home-assistant/broker-setup?srsltid=AfmBOopJ33LEAKMl0Cz8h0u2rYD9HF7MrRbTsMgNkI4E4pibmjylv2Fb

[Oggy512]

Not sure if that helps, or would be sort of an alternative , but if you setup a proper https certificate and allow iFrames on th grafana side, one could add the GF dashboard as external source/iFrame in HA.

[kr0ner]

reverse proxy and certs are in place ... https://oneesp32torulethemall.freeddns.org directly leads to the dashboard in kiosk mode 👍

[reneschuster]

Currently, 11 heat pumps are registered in Grafana, but only 7 are visible on the map. 😕

[kr0ner]

One forgot to change "Landkreis" to the real district and i haven't automated the mapping yet 😅😅

Edit: there are 8 actually :P two are from same district ^^

[RedAnt79]

One was pretty sure to have changed that value. One was surprised that this was not the case. One has corrected it now. 🙈

[kr0ner]

I've updated the UI and added the possibility to filter by heat pump type

[Schmidtjanroman]

Tried to join the analytics club today, my global ID entity is here, the log says:

The dashboard looks a bit empty, did I do something wrong?

[kr0ner]

@Schmidtjanroman the log is always there. ESPHome has a hardcoded timeout which will always be hit when connecting for the first time.
Can you try again? The data is there. Maybe at the time you've looked, there was no data sent. Takes 15 minutes