New shell server transition

[scott]

The home directly file system on the shell machine seems to have been eaten by a large monster. Not a huge crisis, no reason for me to call tech support on the phone, but someone needs to reach in there and clear the jam.

I thought we had that problem solved (we've run for weeks without a relapse after I made some changes). Looking into it now.

-Scott

[scott]

The home directly file system on the shell machine seems to have been eaten by a large monster. Not a huge crisis, no reason for me to call tech support on the phone, but someone needs to reach in there and clear the jam.

I thought we had that problem solved (we've run for weeks without a relapse after I made some changes). Looking into it now.

-Scott

It's this thing again:

Code: Select all

(mount.c:70): Messages from underlying mount program:
(mount.c:74): grantpt: Operation not permitted

The problem is the root /dev/pts is getting unmounted by...something. I'm starting to suspect systemd getting confused, because I have very strict checks about what gets unmounted when managing the chroots.

Much to my chagrin, I might have to swing the shell alias back to bolt, if I can't keep this from happening.

-Scott

[lr]

I'm starting to suspect systemd getting confused, ...

Systemd having bugs? Tell me it ain't so

That piece of the code is the largest [ earthenware container of expletive deleted ] ever let loose on the open source community. I feel very sorry for you having to deal with it. BTDT, got the T-shirt, and ran away as fast as my little feet carried me.

Let me ask a provocative question, not because I want you to answer it, but to make you think the pros and cons through (not here in public, on a comfortable chair at home, or with your colleagues): in retrospect, is your complicated ssh-filesystem and chroot thing really worth it? Given that it is at the edge of normal usage, and likely to tickle bugs that will be very hard to deal with? And another question (again, please don't answer it here in public if you don't feel like it, just think about it, and maybe discuss it over a beer with your boss): Would installing RHEL and getting paid support for it be a good idea? What if you could just take this particular problem, file a trouble ticket, and perhaps a miracle happens and Lennart gets an e-mail from his employer to fix it?

As usual: Thank you so much for keeping a really well working shell machine up and running. Its uptime record is exemplary, and it makes so many things just so convenient.

[utilika]

Where is this .ssh directory? Not in my Sonic home directory.

[pitb]

I can't run at commands. ls -A run under at gives only .history and .ssh directories and no files.

[lr]

Where is this .ssh directory? Not in my Sonic home directory.

That's where it should be. If I do "ls -a1" in my home directory, I see it:

# ls -a1
...
ssh/
...

The -a switch means: Show all files, even ones whose name begins with a dot. The -1 switch means: show files one line at a time, to make the output easier to read.

Even nicer is the output of the long format of ls:

# ls -alF | grep ssh
drwxr-xr-x. 1 lr user 4096 Jun 16 2017 .ssh/

The -l switch means longs format (including size, date and permissions), -F means decorate the file name with a trailing slash if it is a directory, and grepping for ssh shows just that one line. Note that the output of that is very different from looking explicitly into a directory: by default, ls shows the content of directories explicitly named on the command line:

# ls -alF .ssh
total 40
drwxr-xr-x. 1 lr user 4096 Jun 16 2017 ./
drwxr-xr-x. 1 lr user 16384 Apr 20 09:46 ../
-rw-------. 1 lr user 1935 Jun 16 2017 authorized_keys
...
-rw-r--r--. 1 lr user 5593 Apr 10 15:03 known_hosts

If you really don't have a .ssh directory (meaning: if "ls .ssh" in your home directory returns "file not found"), then you have to create one (with "mkdir .ssh"), and then populate it with an authorized_keys file. One thing that is important is that the authorized_keys file has to be well protected, otherwise ssh will deliberately ignore it. If you look at my directory listing above, I used "chmod 600 authorized_keys", meaning only the user can read and write it, and nobody else can. There may be different options, but using just the default protection will probably not work.

[lr]

I can't run at commands. ls -A run under at gives only .history and .ssh directories and no files.

Those are exactly the same symptoms I had yesterday, when the file system had leaped into the toilet and pulled the chain. Paging Scott, Scott to the white courtesy phone please ...

[utilika]

Is it correct that the .ssh directory should be in the /home/u/username directory? If so, it seems that I don’t have it, but that is because it has disappeared, apparently not through any action on my part.

What is there now is:

.bash_history
.forward
.rhosts
.spamcan-graymail
.spamcan-spamassassin-sql
.viminfo
public_cgi
public_ftp
public_html
tmp/

[lr]

Is it correct that the .ssh directory should be in the /home/u/username directory?

It should be in your home directory; you can find out what this is with "echo $HOME". It seems plausible that for you it should be /home/u/utilika, but check.

If it really has disappeared (strange), then you need to recreate it, and populate it with a suitable authorized_keys file. What should be in that file? The public key (one or more) of the ssh login you are coming from; typically those are created in the .ssh directory of the machine you log in from, and they are called id_xxx.pub (for suitable values of xxx used by different authentication mechanisms). If you have a single place you come from, then you can just copy that id_xxx.pub file to .ssh/authorized_keys on sonic. If you have multiple places you come from, you need to concatenate them. I personally edit the authorized_keys file by putting blank lines in to separate sections, and then add comments (marked with # in the first column) to remind myself which is which.

And remember to change the permissions of the newly created file. I think Sonic also maintains a wiki explaining how to do this, but instructions can be found all over the web. Just be warned: some of the instructions are old, and may still have older file names and authentication styles that are obsolete.

[utilika]

Thanks for all the details. The significant issue in this case seems to me to be whether a class of users has had their .ssh directories evaporate, or whether I am the only one. If it’s the former, then bulk restoration should be considered.