At first glance, looks good.
Just for clarification: You did not port crontab from the old system to the new one; that makes sense, since otherwise all cron jobs would run twice. I guess users with cron jobs will have to port them from old to new machine. I'll try that for a while, and move my cron jobs to the new machine (they are tiny and don't run often, just monitoring the network connection to our house).
You're right, I didn't try to move over crontabs. I did a study of the issue, and determined that folks' crontabs are so varied that it would be a real bear to try to accommodate everyone, even if the environment hadn't changed so dramatically.
I like the fact that a reasonably current Python version is now available; the old 2.3 on bolt was getting annoyingly ancient. Any reason you didn't bother installing python3 yet? I have a small number of python scripts on shell.sonic.net (mostly management tools for the web content which I run manually every few months, and a tiny cron job). While I'm not eager to convert all my python stuff at home and at work to python3 yet, once Sonic has a working python3 version, I can start that process.
The only reason I hadn't installed Python 3 yet: I didn't realize it was in the CentOS repos. It is installed now! (More on keeping this a repo-based system in a bit...)
The choice of chroot'ing each user is a pretty radical move. Sure, it gives better security (isolation between users). But a normal multi-user Unix system is already reasonably good, as long as users are knowledgeable about their file permissions, which should be assumed for folks logging into a multi-user system. I certainly hope that the Fuse-based ssh file system works well for you; if not, you'll have some extreme pain. I'm actually a file system and storage person, and using Fuse for production is certainly an unusual and gutsy choice. But good for you that you're trying it, and I won't get the slightest bit upset if the beta machine goes unavailable for a while when you tune and tweak the setup. If you get this to work smoothly, fabulous.
I do hear you about the somewhat unorthodox way we've set up the chroots, as well as employing sshfs. I've spent months tweaking things, running torture tests, and so forth. While it isn't as performant as a native filesystem or NFS, it still seems to be robust in the face of some pretty horrendous hammering.
Naturally, if sshfs doesn't work out, we'll have to use NFS. We're hoping to avoid that, though -- currently, there are no NFS mounts on the system, and we'd like to keep it that way for security purposes.
Note that each chroot's /tmp is on the local SSD, so if you need some fast scratch space, that's the place to do it.
Question about the future: The old shell server was pretty much frozen (for many years), with no upgrades other than security fixes. That's one sensible way of running a system: build it well, and then leave it alone and don't tinker with it. Classic joke: to administer a computer you need a man and a dog: the man to feed the dog, and the dog to bite the man if he tries to touch the computer. There is also another way of running a system: keep it current, perform small upgrades all the time, and don't fall too far behind releases and new software. Clearly, the real world is in the middle between never upgrading anything and always staying current. Do you want to make a forecast of which camp the new server will fall in?
When we originally built Bolt, package management was nowhere near as easy as it is today -- we didn't have yum! This system, though, will track CentOS, and we'll be keeping it up-to-date. (The same will be true for the new web cluster, but we had to build the shell server first.) Every attempt has been made to install packages from the repositories, rather than building them on our own. We also have some packages that we've modified on the system, and those are installed via RPM from our own repository.
Most importantly: Big big kudos for keeping a well-managed shell machine available. It makes my task of administering the web content (as minimalist as it is) much easier, and it gives me a convenient place to store a few important files and perform a few important tasks, even when I can't get to my home server. Thank you.
You're very welcome.
We wanted to keep the shell server, but asked ourselves how we could make it more secure. In the (hopefully unlikely) event of a compromise, we want to reduce the exposure of customer data.
BTW, thank you for the heads-up about Python 3. If there is anything else that needs to be installed, please let me know.