New shell server transition

Advanced feature discussion, beta programs and unsupported "Labs" features.
316 posts Page 25 of 32
by gie » Wed May 23, 2018 5:21 pm
Still unchanged as of 5:20.
by sensei » Wed May 23, 2018 5:30 pm
same here
by scott » Wed May 23, 2018 5:37 pm
fw wrote:A few notes:

I'm definitely running afoul of some sort of idle timeout, and it's in not very many minutes (though I haven't yet measured the exact duration). This is with bash. I haven't yet changed any ssh settings, but according to the documentation, TCP keepalive is enabled by default, which should both keep the TCP connection from dropping and keep the NAT mapping from timing out (as long as the keepalive interval is shorter than the NAT timeout). I haven't tried fiddling with ServerAliveInterval yet.

There doesn't seem to be a way to disable the motd at login without also disabling the "last login" message (and possibly other messages of interest). The old server honored the .no-motd flag file, but the new one ignores it. Using .hushlogin is overly heavy-handed.
I can make it honor .no-motd -- will do.
fw wrote: The new server doesn't add ~/bin to PATH globally, which the old one did. Perhaps this was intentional, since binaries built for oldshell won't necessarily work (due to missing libraries), but it means that some tweaking of local startup scripts is needed to get that effect.
I'm surprised it doesn't do that by default, and can certainly add that.
fw wrote: The chroot containerization unfortunately separates multiple sessions from the same user. On oldshell, when something I'm running goes out to lunch, I can log in another session and kill the offending process, but that's not possible on the new server. It would be better if the containers could be per-userid rather than per-session, though I don't know how easy that would be to implement. A kludgy workaround might be to have special versions of ps and kill that could reach outside the chroot jail (while still being userid-constrained). And of course, if any real use is made of groups, containers would need to be per-group.
The chroots & process visibility _are_ per-user -- not per-session. (Unless I'm misunderstanding what the mount option to the /proc fs is doing.) Not sure what you are seeing, but I just verified this with a test, and I got the expected behavior -- you can see all of your processes all over the system.

Also, the chroot is shared -- there is no other place for it to be, I wrote the script that starts and stops the chroots.

-Scott
by scott » Wed May 23, 2018 5:59 pm
gie wrote:Still unchanged as of 5:20.
Just saw:

"PTY allocation request failed on channel 0"

Related, but not the same symptom.

I restarted the system to get this back into a known state.

Did an "su - gie", which worked. You should get _some_ output from connecting to the host, even when it is falling down. If it disconnects immediately, that points to your IP having ended up in denyhosts.

-Scott
by scott » Wed May 23, 2018 6:38 pm
Well, it's clear that we're going to have to move back to an earlier kernel for now. I'll do that later on this evening -- unless it needs a reboot before that, in which case I'll boot the old kernel that we ran on for quite a while with very little problem.

-Scott
by fw » Wed May 23, 2018 6:53 pm
fw wrote:A few notes:

I'm definitely running afoul of some sort of idle timeout, and it's in not very many minutes (though I haven't yet measured the exact duration). This is with bash. I haven't yet changed any ssh settings, but according to the documentation, TCP keepalive is enabled by default, which should both keep the TCP connection from dropping and keep the NAT mapping from timing out (as long as the keepalive interval is shorter than the NAT timeout). I haven't tried fiddling with ServerAliveInterval yet.
I investigated this further and determined that the ssh session times out after about five minutes of inactivity. Setting ServerAliveInterval on the client to 240 (four minutes) fixes it. I don't know what the equivalent is in other clients, but this isn't necessary on oldshell.
scott wrote:
fw wrote: The chroot containerization unfortunately separates multiple sessions from the same user. On oldshell, when something I'm running goes out to lunch, I can log in another session and kill the offending process, but that's not possible on the new server. It would be better if the containers could be per-userid rather than per-session, though I don't know how easy that would be to implement. A kludgy workaround might be to have special versions of ps and kill that could reach outside the chroot jail (while still being userid-constrained). And of course, if any real use is made of groups, containers would need to be per-group.
The chroots & process visibility _are_ per-user -- not per-session. (Unless I'm misunderstanding what the mount option to the /proc fs is doing.) Not sure what you are seeing, but I just verified this with a test, and I got the expected behavior -- you can see all of your processes all over the system.

Also, the chroot is shared -- there is no other place for it to be, I wrote the script that starts and stops the chroots.
My bad. I hadn't given ps the right options.

Regarding devpts - I've sometimes seen a large proliferation of devpts mounts (I think I counted 124 in one case). Maybe this is what systemd is trying to clean up.
by scott » Wed May 23, 2018 7:15 pm
fw wrote:
fw wrote:A few notes:

I'm definitely running afoul of some sort of idle timeout, and it's in not very many minutes (though I haven't yet measured the exact duration). This is with bash. I haven't yet changed any ssh settings, but according to the documentation, TCP keepalive is enabled by default, which should both keep the TCP connection from dropping and keep the NAT mapping from timing out (as long as the keepalive interval is shorter than the NAT timeout). I haven't tried fiddling with ServerAliveInterval yet.
I investigated this further and determined that the ssh session times out after about five minutes of inactivity. Setting ServerAliveInterval on the client to 240 (four minutes) fixes it. I don't know what the equivalent is in other clients, but this isn't necessary on oldshell.
scott wrote:
fw wrote: The chroot containerization unfortunately separates multiple sessions from the same user. On oldshell, when something I'm running goes out to lunch, I can log in another session and kill the offending process, but that's not possible on the new server. It would be better if the containers could be per-userid rather than per-session, though I don't know how easy that would be to implement. A kludgy workaround might be to have special versions of ps and kill that could reach outside the chroot jail (while still being userid-constrained). And of course, if any real use is made of groups, containers would need to be per-group.
The chroots & process visibility _are_ per-user -- not per-session. (Unless I'm misunderstanding what the mount option to the /proc fs is doing.) Not sure what you are seeing, but I just verified this with a test, and I got the expected behavior -- you can see all of your processes all over the system.

Also, the chroot is shared -- there is no other place for it to be, I wrote the script that starts and stops the chroots.
My bad. I hadn't given ps the right options.

Regarding devpts - I've sometimes seen a large proliferation of devpts mounts (I think I counted 124 in one case). Maybe this is what systemd is trying to clean up.
Those are actually "bind" mounts within the chroots. There are a few other bind mounts, too. Right now, they may pile up a bit, as I've "no-opped" the unmount script until I've satisfied myself that that isn't the cause of various API filesystems getting unmounted (/dev/pts being one of them).

-Scott
by coad » Wed May 23, 2018 10:51 pm
I'm still not able to login:

_____________________________________________
==== THIS IS THE NEW SHELL HOST ====
If you need to connect to the old shell host, please
connect to oldshell.sonic.net.
For assistance, please post to
viewtopic.php?f=13&t=5350
... or email shellmaster@sonic.net

*** Reboot scheduled for later on this evening (5/23/18) ***
====================================
/bin/tcsh: Permission denied
Connection to sh.sonic.net closed.

Is there something up with tcsh or is there something else going on? I have not gotten past this (no login from work or home) since the reboot at 1330 today. Hopefully the reboot later will fix this.

Thanks,

--paul
by casner » Wed May 23, 2018 10:58 pm
Logging in with tcsh worked for me just now.
by coad » Wed May 23, 2018 11:09 pm
Thanks for checking tcsh! Hopefully the reboot will fix whatever is keeping me from logging in.

--paul
316 posts Page 25 of 32

Who is online

In total there are 3 users online :: 0 registered, 0 hidden and 3 guests (based on users active over the past 5 minutes)
Most users ever online was 999 on Mon May 10, 2021 1:02 am

Users browsing this forum: No registered users and 3 guests