Shell server adjustments

[ds_sonic_asif]

Incidentally, anyone who has taken an interest in the shell...how do you feel about the current environment, where the only processes you can see are your own?

Seems fine to me. I don't have a use case for needing to know about other users and other processes. The majority of my usage is occasionally feeding mini hosting via scp. Every blue moon or so I may ssh in to run some diagnostic from afar (e.g. nmap to see if I am unintentionally leaking access at home).

On the other hand, if the interactive ssh session was behaving oddly, I would probably run a handful of command line stats commands to see what was going on. It appears that df, vmstat, iostat, ifconfig, and probably others, are reporting system statistics.

[leondis]

Hi Scott:

Just wanted to say Thanks for getting the load avgs back down to a reasonable range. Login is no longer difficult.

As for not seeing others' processes, the only downside i can think of is that maybe unconsciously people think that they have the machine to themselves and are more likely to overload it because there's no sense that others are using it too. But other
than that i don't see a big deal one way or the other... though i used to "talk" to people i know sometimes when i saw they were online...

Thanks again

[scott]

Hi Scott:

Just wanted to say Thanks for getting the load avgs back down to a reasonable range. Login is no longer difficult.

As for not seeing others' processes, the only downside i can think of is that maybe unconsciously people think that they have the machine to themselves and are more likely to overload it because there's no sense that others are using it too. But other
than that i don't see a big deal one way or the other... though i used to "talk" to people i know sometimes when i saw they were online...

Thanks again

I've investigated what it would take to get talk running, it's a very retro thing that would be fun (even ytalk).

Load avg is creeping up again. I am monitoring it:



(That is on a tiny lilliput monitor sitting next to me at my desk.)

[apl]

Getting much worse again:
] uptime
13:41:56 up 2 days, 4:42, 0 users, load average: 4.79, 3.78, 3.51
] uptime
16:37:46 up 2 days, 7:38, 0 users, load average: 21.27, 21.67, 19.45

[scott]

Getting much worse again:
] uptime
13:41:56 up 2 days, 4:42, 0 users, load average: 4.79, 3.78, 3.51
] uptime
16:37:46 up 2 days, 7:38, 0 users, load average: 21.27, 21.67, 19.45

I've tracked it down to some user cron jobs and frequent (apparently scripted) remote commands.

Quick note[*], while on the subject: If you're ssh'ing in remote commands, please don't background them unless you have some kind of locking in place to prevent simultaneous execution. That's related to what's happening on the shell server, which has been stacking up a significant load average as it goes, since these user remote commands I just mentioned are being executed with '&' at the end of a long command line. That's legitimate in many cases, but you've got to use locking to prevent the next invocation of the remote command from messing up the previous...or the previous 2, or 3, or however many are getting stacked up and interfering with each other.

[*] "quick note" -- ch'yeah'right, that'll be the day

[apl]

Thanks Scott.
So basically, an unintentional DOS attack. Such are the risks of a shared server I guess.
Load has remained low since yesterday.
If the problem recurs, it looks like you might be able to set per-user CPU limits using systemd

[scott]

Okay, here's the skinny.

Kelsey wins the Internet for the day. He looked at the system and noticed there were a lot of mounts. That isn't unusual, since there are bind mounts galore in every person's chroot.

But this was a _lot_ of mounts. Way too many of them. We had xdg mounts over the top of xdg mounts. It was panic in the streets.

Anyway, we think we found the problem, should be (knock on wood) better.

-Scott

[scott]

https://imgur.com/fXUSyay

How is your Sunday morning going?

-Scott

[swerthei]

Has something changed in the shell server ssh config that would invalidate some ssh clients? I can't consistently connect via ssh to sh.sonic,.net anymore. It works from some clients and not others. The only common denominator I've found is that the clients that do not work are using ssh version "OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5" and the clients that work are using other versions of ssh, such as for example "OpenSSH_8.5p1, OpenSSL 1.1.1k 25 Mar 2021" and "OpenSSH_7.9p1, OpenSSL 1.1.1i-freebsd 8 Dec 2020". Here is a verbose output from a connection attempt that fails:

PS C:\Users\swerthei> ssh -v swerthei@sh.sonic.net
OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5
debug1: Connecting to sh.sonic.net [208.201.242.22] port 22.
debug1: Connection established.
debug1: identity file C:\\Users\\swerthei/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_ed25519-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_xmss type -1
debug1: key_load_public: No such file or directory
debug1: identity file C:\\Users\\swerthei/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_for_Windows_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to sh.sonic.net:22 as 'swerthei'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes128-ctr MAC: umac-128-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-128-etm@openssh.com compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:3Shjz0z7pf5EhaJuPaq4Dij92qFd34dRl9pbeNZAtWk
debug1: Host 'sh.sonic.net' is known and matches the ECDSA host key.
debug1: Found key in C:\\Users\\swerthei/.ssh/known_hosts:8
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: pubkey_prepare: ssh_get_authentication_socket: No such file or directory
Corrupted MAC on input.
ssh_dispatch_run_fatal: Connection to 208.201.242.22 port 22: message authentication code incorrect

[sls123]

Has something changed in the shell server ssh config that would invalidate some ssh clients? I can't consistently connect via ssh to sh.sonic,.net anymore. It works from some clients and not others. The only common denominator I've found is that the clients that do not work are using ssh version "OpenSSH_for_Windows_7.7p1, LibreSSL 2.6.5" and the clients that work are using other versions of ssh, such as for example "OpenSSH_8.5p1, OpenSSL 1.1.1k 25 Mar 2021" and "OpenSSH_7.9p1, OpenSSL 1.1.1i-freebsd 8 Dec 2020". Here is a verbose output from a connection attempt that fails:

I was unable to log in as well But then I went to members.sonic.net and saw that my credit card had expired and now overdue bill. I updated cc info, paid and was able to ssh and content to mail.sonic.net via browser. Now that I can log in, I find that my shell setting has been lost and chsh says permission denied.