protocol for granting ssh access to node whisperers

[paidforby]

Currently, there is only one active member who is capable of pushing out patches to nodes on the people's open network, along with two formerly active members. This information is publicly available (but somewhat buried) here. I'd like to open a discussion on whether we should have this access, who should have it, and when do we use it. Is there a formal procedure for adding developer keys to nodes? How do we make sure end-users are aware of the fact that we have developer access?

Should completing the Node Whisperer Program grant you developer access to all nodes? Some nodes? What about exit nodes and extender nodes?

I'm interested in hearing what people think. I'd also, at least, like to create a standard patch to remove the two formerly active members from authorized_keys.

[wwwhtml]

@paidforby thank you for bringing this up.

About answering/giving my opinion about: "How do we make sure end-users are aware of the fact that we have developer access?" I believe we should be at front about this with all node-runners. Telling them would freak out some folks, but at the same time may also tell them that we are not hiding it from them. Ideally, at the time of the node-install/config node-runners could choose between choosing a box that says "Provide Automatic Access to PON-Developers (recommended)" or "Provide Manual Access to PON-Developers".

Also, I feel that node-runners would feel better y they could have access to a page where they can see the times and reasons why PON Developers connected to their nodes.

My 2 cents.

[paidforby]

@wwwhtml true it would scare/confuse some people, but it's important that they understand how things are working and educate themselves on security best practices. I'd like to see a lot of the education and information to (eventually) be presented through a "register your node" web portal. Some ideas for this,

1. It would present an "opt-in" to receive patches/updates from SudoMesh developers, with an update-to-date list of all certified node whisperers and their ssh keys. Node operators would then select the keys they would like to allow to have access, similar to when you add your keys to a digital ocean droplet.
2. The default selection would be "Other", with a text box next to it for the node operator to paste their public key.
3. It would also present a list of active SudoMesh exit nodes, from which you could select "preferred", "secondary", so on.
4. There would also be an "Other" option for exit node, where the node operator could add the IP address of their own exit node.

Ideally, nodes would fetch updates from a patch server (or from other nodes?) instead of having them pushed from the exit node. But until we have a method for doing that, pushing patches works.

Anyway, this sort of registration process seems like a good time to educate node operators about the mesh, it's risks, rewards, and possibilities.

[bennlich]

During the install at Doug's, I tried communicating the option to allow/disallow remote access. His response was something like, "Why would I want / not want to allow you to have ssh access to my node?" He was wondering what it really means that we have access--i.e. what can we screw up? I didn't really know how to answer this question.

My takeaway was, when we communicate this option, it'll be important to communicate what the ramifications of each choice are. E.g. what is a best/worst-case scenario if one of our admins goes rogue and wants to do unethical things. And what is a best/worst-case scenario if we are unable to log in to perform remote patches.

Should completing the Node Whisperer Program grant you developer access to all nodes? Some nodes?

I think we want a program for sharing knowledge and learning together about how the mesh hardware / firmware / software works and how to debug it--in my mind, this is the Node Whisperer Program.

I think we also want a process for granting people dev access to nodes so that they can push updates. I think this is something more in-depth. Maybe Node Whisperer++. Probably should require some demonstrated amount of time devoted to the project. I would think these would need to be really trusted individuals.

---

@wwwhtml

Also, I feel that node-runners would feel better y they could have access to a page where they can see the times and reasons why PON Developers connected to their nodes.

Mmmm, yeah. Would be cool if the admin panel of the node could render a list of notes left by node admins.

@paidforby I like what you've outlined for the web portal. Seems very clear and transparent to me.

---

How do we make sure end-users are aware of the fact that we have developer access?

Do we currently have an exit form, or something we give people after an install that highlights what worked, what didn't, what to expect? Maybe this would be an appropriate place to include copy about how to access the admin panel, and how security works.

[jhpoelen]

Just to add my 2 cents - while I appreciate the idea to educate folks on how mesh nodes work, I don't think it should be a justification for adding overly complex configuration. I would hope that default settings are sufficient to get things working and allow for keep nodes up-to-date. Also, I would advocate for assuming good intentions. The idea of a "rogue operator" sounds more like a sensational headline than a likely scenario to me. Wishing I could join for a lively discussion on this later today in person, but unfortunately, I am unable to attend to Tue meeting.

[bennlich]

The idea of a "rogue operator" sounds more like a sensational headline than a likely scenario to me.

Yeah. I think I just want to be able to answer the question: "Why would I not want you to have access to my node?" Am open to w/e ideas about how to start the conversation.

[jhpoelen]

If you don't trust PON to maintain the network, folks are free to start their own. It just takes a bunch of work to keep things running.

[bennlich]

And PON would be willing to do an install for someone to start their own network?

[jhpoelen]

I'd say that is up to volunteers and personal relationships. I'd totally help you setup your Benny's Open Network.

[jhpoelen]

Also, I think that setting up your own PON network is part of a node whisperer introduction, so it'd be happing a bunch anyway, 'cause we're hoping to get more and more folks whispering .

[jhpoelen]

Anyways, probably best to be discussed in person, which I can do in a week. Please don't wait for me though ; )

[gobengo]

I feel pretty strongly that we can do better than giving SSH access to every node-whisperer. I don't want SSH access. If that's what it is, I don't want it. It makes me a more attractive attack target, and increasingly so as the network becomes more valuable.

I will caveat the following: I will describe an envisioned future state, but acknowledge the need for intermediate milestones and SSH access for the immediate future.

I think the opposite of the above is a more ethical setup for stewards of a network whose users sometimes don't know any better: no ssh:

1. No SSH at all by default. Port 22 refuses connections. nmap scans should not show anything that an automated bot might use as a heuristic that it has found something to poke holes at.
2. There is a cronjob/similar that checks for new packages from a well-known package feed on peoplesopen.net domain, verifies provenance, and installs them
   • in modern IoT/embedded setups it seems you'd also want things like immutable upgrades so the thing isn't bricked during installation, and if an installation fails halfway through, you can restart and reboot from a backed-up partition.
3. Packages in this feed are governed by an accountable task force or working group. A straw man for this governance is:
   • there is a general assembly, e.g. of home node operators or the node whisperers
   • there is a 'security council' for the most trusted individuals in the community. People can be appointed to the security council by 2/3rds vote of general assembly. Members can be removed by majority of general assembly + security council
   • All new versions in the package feed must be approved by a consensus of the security council, and a vote should occur only after designating who is responsible for that release going out and being reverted afterward if needed. This is also a good time to justify why the release is needed, needed now vs later, etc. Proposed changes with signatures must be public for at least X (2?) days for community review before being deployed.
4. There is a #peplesopen-security hashtag mailing list or trello for security council efforts, and breakout meetings at least once a month to process issues/concerns so they do not get more than one month stale.

[jhpoelen]

@gobengo some comments / thoughts -

1. you mention "I will caveat the following: I will describe an envisioned future state, but acknowledge the need for intermediate milestones and SSH access for the immediate future" - given the rate of our development and involvement, I am concerned that the immediate future covers at least the coming 6-8 months. How do you propose to maintain the network up until then? Would you like to contribute to helping with this?

2. You mention that "I don't want SSH access. If that's what it is, I don't want it. It makes me a more attractive attack target, and increasingly so as the network becomes more valuable." - with a centralized patch server, isn't the "attack" target shifted to the group of individuals that have access to the patch server? Worse even, rather than having to access tens, hundreds or thousands of nodes to apply patches, you'd only have to change the package feed to update all of them at once.

I like the idea of mitigating risks of a network being hacked, a consensus decision model (not just 2/3 vote) and balancing that with the speediness of getting patches out. In addition to that, I am also interested in figuring out where we are today and getting a little closer to where we want to go tomorrow in a somewhat incremental way.

Again, this is probably best discussed in person.

[gobengo]

@jhpoelen

How do you propose to maintain the network up until then?

I can't think of any better interim solution than where we're at with SSHability.

with a centralized patch server, isn't the "attack" target shifted to the group of individuals that have access to the patch server?

Yes, but it seems harder to attack a group than any one of separate individuals.

Worse even, rather than having to access tens, hundreds or thousands of nodes to apply patches, you'd only have to change the package feed to update all of them at once.

2.1. Packages should be signed and updating node should verify integrity and authenticity (e.g. by requiring sigs of several hard-to-compromise-at-once keys)
2.2. I still think it's no worse than the status quo, where you can basically also can upgrade them all at once if I steal any one of the authorized_keys private keys. At least we can centralize our defenses in a single 'bastion' of the package feed server andpeer-reviewed package publishing process. Savvy or concerned node operators can run their own package feed mirror all without allowing SSH.

[gobengo]

The stakes of this are high enough for the sudomesh legal entity that it is probably worth spending <= $1000 on an embedded linux / security firm to make a formal recommendation.

At the very least I will continue to find existing publications and best practices docs that the group can point to as 'best practices' we did diligence on and followed.

[gobengo]

@jhpoelen

How do you propose to maintain the network up until then?

I guess I thought of one, but doesn't require much creativity: Start a bounty to incentivize getting it done sooner (and deployed). Or, ofc, hire an expert to make it their top priority.