New shell server beta testing

Advanced feature discussion, beta programs and unsupported "Labs" features.
101 posts Page 1 of 11
by scott » Wed Jan 03, 2018 10:58 am
Hello!

So we've been working on a new shell server. Try it here:

sh.sonic.net

Fingerprint:

MD5:44:ef:ed:17:07:d1:a9:83:ab:aa:43:a3:00:77:60:7e
SHA256:2v/NEm+2VjOi7WV/W/BV42TG5D8rBYrHELsCVyEbciA

If you have shell access on Bolt, you should be able to log in to sh.sonic.net, the address of the new shell server. Both password and ssh public key logins should work -- so if you have a public key in your Bolt ~/.ssh/authorized_keys, it should also work on sh.sonic.net .

This is a beta test of a work in progress -- please, any and all feedback is very much appreciated.

A few caveats:

We would really like to redo the backend for our mail spools, and in order to do that, the format may be changing. Because of this, the unified interface to mail from sh.sonic.net is secure imap.

Pine is replaced with alpine. For help setting up alpine with imap, run the command "alpine_help.sh".

For mutt, you can either run imutt.sh -- or, alternately, run mutt_help.sh and follow the instructions there.

This is a bit of an experiment, where we are using sshfs mounts for home directories and web directories, all in a chrooted environment. Cron jobs should still work, as they automatically fire off one's chrooted environment.

Please give it a try and tell us what you think!

Thanks,

-Scott
by lr » Thu Jan 04, 2018 9:28 am
At first glance, looks good.

Just for clarification: You did not port crontab from the old system to the new one; that makes sense, since otherwise all cron jobs would run twice. I guess users with cron jobs will have to port them from old to new machine. I'll try that for a while, and move my cron jobs to the new machine (they are tiny and don't run often, just monitoring the network connection to our house).

I like the fact that a reasonably current Python version is now available; the old 2.3 on bolt was getting annoyingly ancient. Any reason you didn't bother installing python3 yet? I have a small number of python scripts on shell.sonic.net (mostly management tools for the web content which I run manually every few months, and a tiny cron job). While I'm not eager to convert all my python stuff at home and at work to python3 yet, once Sonic has a working python3 version, I can start that process.

The choice of chroot'ing each user is a pretty radical move. Sure, it gives better security (isolation between users). But a normal multi-user Unix system is already reasonably good, as long as users are knowledgeable about their file permissions, which should be assumed for folks logging into a multi-user system. I certainly hope that the Fuse-based ssh file system works well for you; if not, you'll have some extreme pain. I'm actually a file system and storage person, and using Fuse for production is certainly an unusual and gutsy choice. But good for you that you're trying it, and I won't get the slightest bit upset if the beta machine goes unavailable for a while when you tune and tweak the setup. If you get this to work smoothly, fabulous.

Question about the future: The old shell server was pretty much frozen (for many years), with no upgrades other than security fixes. That's one sensible way of running a system: build it well, and then leave it alone and don't tinker with it. Classic joke: to administer a computer you need a man and a dog: the man to feed the dog, and the dog to bite the man if he tries to touch the computer. There is also another way of running a system: keep it current, perform small upgrades all the time, and don't fall too far behind releases and new software. Clearly, the real world is in the middle between never upgrading anything and always staying current. Do you want to make a forecast of which camp the new server will fall in?

Most importantly: Big big kudos for keeping a well-managed shell machine available. It makes my task of administering the web content (as minimalist as it is) much easier, and it gives me a convenient place to store a few important files and perform a few important tasks, even when I can't get to my home server. Thank you.

Ralph
Linda and Ralph and John; 735 Sunset Ridge Road; Los Gatos, CA 95033; 408-395-1435
by scott » Thu Jan 04, 2018 3:03 pm
lr wrote:
At first glance, looks good.

Just for clarification: You did not port crontab from the old system to the new one; that makes sense, since otherwise all cron jobs would run twice. I guess users with cron jobs will have to port them from old to new machine. I'll try that for a while, and move my cron jobs to the new machine (they are tiny and don't run often, just monitoring the network connection to our house).


You're right, I didn't try to move over crontabs. I did a study of the issue, and determined that folks' crontabs are so varied that it would be a real bear to try to accommodate everyone, even if the environment hadn't changed so dramatically.

lr wrote:
I like the fact that a reasonably current Python version is now available; the old 2.3 on bolt was getting annoyingly ancient. Any reason you didn't bother installing python3 yet? I have a small number of python scripts on shell.sonic.net (mostly management tools for the web content which I run manually every few months, and a tiny cron job). While I'm not eager to convert all my python stuff at home and at work to python3 yet, once Sonic has a working python3 version, I can start that process.


The only reason I hadn't installed Python 3 yet: I didn't realize it was in the CentOS repos. It is installed now! (More on keeping this a repo-based system in a bit...)

lr wrote:
The choice of chroot'ing each user is a pretty radical move. Sure, it gives better security (isolation between users). But a normal multi-user Unix system is already reasonably good, as long as users are knowledgeable about their file permissions, which should be assumed for folks logging into a multi-user system. I certainly hope that the Fuse-based ssh file system works well for you; if not, you'll have some extreme pain. I'm actually a file system and storage person, and using Fuse for production is certainly an unusual and gutsy choice. But good for you that you're trying it, and I won't get the slightest bit upset if the beta machine goes unavailable for a while when you tune and tweak the setup. If you get this to work smoothly, fabulous.


I do hear you about the somewhat unorthodox way we've set up the chroots, as well as employing sshfs. I've spent months tweaking things, running torture tests, and so forth. While it isn't as performant as a native filesystem or NFS, it still seems to be robust in the face of some pretty horrendous hammering.

Naturally, if sshfs doesn't work out, we'll have to use NFS. We're hoping to avoid that, though -- currently, there are no NFS mounts on the system, and we'd like to keep it that way for security purposes.

Note that each chroot's /tmp is on the local SSD, so if you need some fast scratch space, that's the place to do it.

lr wrote:
Question about the future: The old shell server was pretty much frozen (for many years), with no upgrades other than security fixes. That's one sensible way of running a system: build it well, and then leave it alone and don't tinker with it. Classic joke: to administer a computer you need a man and a dog: the man to feed the dog, and the dog to bite the man if he tries to touch the computer. There is also another way of running a system: keep it current, perform small upgrades all the time, and don't fall too far behind releases and new software. Clearly, the real world is in the middle between never upgrading anything and always staying current. Do you want to make a forecast of which camp the new server will fall in?


When we originally built Bolt, package management was nowhere near as easy as it is today -- we didn't have yum! This system, though, will track CentOS, and we'll be keeping it up-to-date. (The same will be true for the new web cluster, but we had to build the shell server first.) Every attempt has been made to install packages from the repositories, rather than building them on our own. We also have some packages that we've modified on the system, and those are installed via RPM from our own repository.

lr wrote:
Most importantly: Big big kudos for keeping a well-managed shell machine available. It makes my task of administering the web content (as minimalist as it is) much easier, and it gives me a convenient place to store a few important files and perform a few important tasks, even when I can't get to my home server. Thank you.

Ralph


You're very welcome. :) We wanted to keep the shell server, but asked ourselves how we could make it more secure. In the (hopefully unlikely) event of a compromise, we want to reduce the exposure of customer data.

BTW, thank you for the heads-up about Python 3. If there is anything else that needs to be installed, please let me know.

-Scott
by Gypsy Baron » Fri Jan 05, 2018 9:14 am
I have been using PuTTY to access the shell and pine for my email for years. Many years, more than 20!

I have absolutely no idea as to how I would configure PuTTY to access this beta server or how to convert to using alpime.

What happens to all my mail boxes and saved mail when this all becomes 'live'.

I am often traveling and spend several weeks out of the country. I sure would not like to be 'cut off' from my Sonic email
when this changes and I am in Prague or Budapest! BTW, I dislike webmail with a passion !

Paul Strogen ( csardas@sonic.net)
by miken » Fri Jan 05, 2018 9:24 am
In Putty under your Sessions menu (should default to this when you open the program) you can add sh.sonic.net to the field that is labeled Saved Sessions and then hit the Save button. It will be added to your list of sessions and you can open it normally like you do shell.sonic.net. The first time you do so, you'll see a prompt verifying the fingerprint that Scott posted above. :)
Mike N.
Development Trainer
Sonic
by Gypsy Baron » Fri Jan 05, 2018 1:04 pm
Getting an error message when I attempt to load sh.sonic.net in PuTTY.

Here is a screenshot of the message:

http://www.sonic.net/~csardas/TEMP/PuTTY_Error.jpg

Paul
by miken » Fri Jan 05, 2018 1:49 pm
What version of Putty are you using? May need to update.
Mike N.
Development Trainer
Sonic
by Gypsy baron » Fri Jan 05, 2018 5:50 pm
PuTTY 0,53b I haven't touched it in 15-20 years.

I never update anything that is working and meeting my needs.

Paul
by scott » Sat Jan 06, 2018 4:03 pm
miken wrote:
What version of Putty are you using? May need to update.


Yes indeed -- we had another tester that got a similar error. Updating putty solved the problem.

-Scott
by scott » Sat Jan 06, 2018 4:11 pm
Gypsy Baron wrote:

What happens to all my mail boxes and saved mail when this all becomes 'live'.


They will all continue to be on the mail server, but accessable via secure imap. If you have any local folders (under ~/Mail, for example), they will still be available, as your home directory remains the same.

There are instructions on the new system for setting up alpine to use secure imap. Run "alpine_help.sh" when logged in to the new shell server to see them. They basically say:

* create a .pinerc
* set your inbox-path, like this:
inbox-path={imap.sonic.net:993/ssl/user=scott}

You can create a .pinerc by running alpine and changing
some configuration variable, such as 'full name'.


Gypsy Baron wrote:
I am often traveling and spend several weeks out of the country. I sure would not like to be 'cut off' from my Sonic email
when this changes and I am in Prague or Budapest! BTW, I dislike webmail with a passion !

Paul Strogen ( csardas@sonic.net)


There will certainly be a transition period while we have folks move to the new server. I'd certainly encourage you to try the new server out when you find the time. I can also help you get set up with alpine.

-Scott
101 posts Page 1 of 11

Who is online

In total there are 3 users online :: 0 registered, 0 hidden and 3 guests (based on users active over the past 5 minutes)
Most users ever online was 422 on Sat May 26, 2012 5:28 am

Users browsing this forum: No registered users and 3 guests