February 2010 - admins-discuss@lists.redbrick.dcu.ie

[Admin-discuss] Installing lucid on carbon
by Andrew Harford 27 Oct '10

27 Oct '10

On Thu, Jan 28, 2010 at 11:20:02AM +0000, Cian Brennan wrote: > On Thu, Jan 28, 2010 at 11:14:20AM +0000, Andrew Harford wrote: > > Hi, > > > > I think that after the big storage migration we should move the stuff on > > fast-storage to worf (faster-storage, if you like ;) ). > > > > Once this is done there should be nothing at all left running on carbon, > > we can do a clean install to get rid of anything left over from the > > various things what ran there. > > > > There's two reasons for this: > > > > * If carbon is to stay as a 'development machine' then people are going > > want the newest development tools and libaries. Hardy isn't getting any > > younger. Unless carbon has software that we can't or won't run on the > > hardy boxes it doesn't make much sense to develop anything there. > > > > * Eventually we will probably have to upgrade everything to lucid. Given > > that there will be no more features released on hardy I think it's > > unlikely that 'the admins of the future' will be happy to wait until > > 12.04 is stable to move away from hardy, so we may as well get an > > enviroment where we can test it out properly for a while. > > > I second this excellent recommendation, but advise that it be sent to > admin-discuss. Having looked more into this, lucid is not ready yet, but I think we should move to karmic, and keep carbon on the most recent ubuntu indefinatly. This is def a tradeoff in terms of stability and availablity but I think it makes sense in this case. a. -- Andrew Harford You do not just avoid the Suzuki Wagon R. You avoid it like you would avoid unprotected sex with an Ethiopian transvestite. --Jeremy Clarkson

6 9

[Admin-discuss] Murphy on Linux
by Andrew Martin 24 Mar '10

24 Mar '10

Hey, Last night I moved apache temporarily to deathray, installed Linux from a DVD on to the spare disk on Murphy, moved the web server there and ran some tests. In short, it worked perfectly, zero hitches. There was an issue with networking, but it turned out to be me noobing and configuring the wrong ehternet ports, it worked fine once I told it to use eth2 and 3, not 0 and 1. I installed apache with suphp, suexec, pubcookie, etc, all configured and running all of the vhosts, worked fine from a user point of view (even wiki logins worked, they never work). Tested a few sites (static ones like my own, PHP/database ones like gamessoc's, and some things under ~testwww that usually break badly configured servers), all worked. It was under the regular load of 2am (admittedly not much), plus 15 instances of a script on a few external servers making randomised requests to various pages drove load to about 12 with no noticeable side effects. Booted back into Solaris after about half an hour of this. The bugs on OpenBSD showed up after 5/10 minutes of apache being switched on, so we can assume Linux doesn't have the same problem. I'd suggest we switch Linux back on permanently next weekend. We can leave it on the spare disk for a few days (maybe a week), that gives us a window to find bugs, etc, and if there's anything serious we can jsut reboot back into Solaris at any time. After a week or so is up, assuming nothing goes wrong, we could then transfer Linux onto the main disks in a software raid 1 setup. A side effect of all this is that we now have a cold spare web server on Deathray :) If murphy catches fire or anything we can just configure the WWW IPs on deathray, send a gratuitous ARP and fire up it's apache to minimise downtime while we figure out what's wrong with $main_web_server. -Andrew

7 23

[Admin-discuss] Virtual Machines!
by Andrew Martin 16 Feb '10

16 Feb '10

Hey, After a good bit of work by a lot of people, I think we're (finally) able to offer members full virtual machines :) As the software to manage them is only about 3 weeks old and was written primarily by someone in a semi-asleep state, these (or specifically, the management interface) should be considered in a sort of pre-pre-pre-pre-alpha v0.00001-unstable state, but if anyone is interested in testing them, let us know and we'll set you up. We'd love to hear if people have any thoughts (or bug reports) on the whole setup, before we "officially" launch them in a few weeks (hopefully tech week). The management interface is at https://vmweb.redbrick.dcu.ie - the login details are not the same as your main Redbrick account, so don't enter your main password. As I said, let us know and we'll create you an account and a VM to play with. If anyone's interested, the source for the management interface is at http://hg.redbrick.dcu.ie/rbvm - if anyone sees any security holes (and I don't doubt they exist), let me know. VMs are given on the basis of trust. While they're fairly well firewalled from the main Redbrick network (and ISS's network), you still have the potential to screw up the VM network itself. We trust that users are generally well intentioned and won't do this, but if anyone does, we'll say mean things to them and delete their VM account, among other things. VMs have http/ftp access through the proxy at http://136.206.16.254:3128. The DNS server is 136.206.16.254. The mail relay is the same. (these services run on the physical machine cynic). The VMs all run on Daniel, a big tank of a machine we got last year for this purpose (8 core AMD Opteron 2356, 24GB RAM). Looks exactly like morpheus only with more lights. There are a bunch of ISOs uploaded to the system already for users to choose from, but if you want more, let us know and we'll see what we can do. Regards, -Andrew

2 1

[Admin-discuss] Admin training list of things to do
by Andrew Martin 16 Feb '10

16 Feb '10

Hey, It was brought up at a committee meeting earlier that we should put together some sort of list of things to cover for admin training. I didn't go to many of last years (only one :( ), but since we have a VM server this year (I'm a few hours away from something we can announce to users) we could have a much more practical set of tutorials this year. To begin with, I'd suggest (these need to be filled out a little): Scripting What it says on the tin, an hour or so on doing useful stuff with shell scripting. I think Andrew Harford wants to do this. He's sitting in front of a redbrick terminal right now so I'm sure he'll reply with his thoughts within a few seconds :) Server setup How to install a server (people attending could do this in a VM), usual post-setup tricks (any extra networking setups, fail2ban) etc. Services Setting up a DNS server, a mail server, maybe an IRC server, etc, on their VM. Web server Make people set up a web server with a few vhosts, pubcookie, SuPHP, etc on their VM. For vhosts, we can just have them add entries to their hosts files. Security Start off with a demo of what a buffer overflow in a piece of user code can do (give attendees a healthy paranoia and an irrational fear of C code), a run through of what pf can do and how to use it. Maybe a section with pointers on how to deal with getting rooted, or something. Anyone have input? -Andrew

6 7

[Admin-discuss] Murphy, again
by Andrew Martin 08 Feb '10

08 Feb '10

Hey, (sorry for the length of this mail) Last night while we were moving storage over we also updated Solaris on murphy (which took a stupid amount of time because of Solaris acting stupid, stupid OS). Afterwards we used the downtime window (and the spare disk in murphy) to give Linux another try, as it's been well over a year since we last gave up on it. It got a hell of a lot further then before with the netboot installer (from Debian Lenny), and the only thing that stopped it was an e1000e (network driver) bug that seems to affect a few architectures, not just sun4v or even sparc. This caused a bit of a problem for a network installer. Other then that, there wasn't so much as a warning from the system (I broke the installer out to a busybox shell and poked around looking at dmesg, etc). Before I figured out what the bug actually was, I also updated the system firmware (that's given as a catch-all bugfix by a few places), which seems to have fixed the DVD drive. I'd like to try this again with a DVD installer (and a USB stick in the USB port holding an updated ethernet driver). The spare disk means that if it goes wrong, and when we're finished testing, we can just boot back into Solaris (on the other two disks). If it installs, we install apache and set it up fully (I've done this more times then I can count in the last 2 years, I can have it up and running in an hour if the OS is alive), test it for an hour or 2 (maybe with some added load test scripts running at a few locations), see if everything's stable, and then reboot into solaris to decide what to do next (decide if it'd be worth fully moving back to Linux or not). This could be done at a quiet time (like, start at 11PM or something) so service impact would be minimal. It's been two years and despite best efforts, murphy with Solaris is still a complete mess, with two and a half package managers each of which half-work and an OS upgrade procedure that throws up new random problems every time we do it (last night, a bunch of services didn't come back, they're still not quite there). The ldap client needs the root password typed as a command-line option, ffs. I'd *really* like to at least know if we have an alternative OS option for that machine, as there's a major risk of the next set of admins either leaving it to stagnate and develop security holes over time (carbon and deathray two years ago being prime examples of this happening before), or throwing out a very powerful machine. Opinions? -Andrew

4 9