December 2008 - admins-discuss@lists.redbrick.dcu.ie

[Admin-discuss] Fwd: [rb-admins] Murphy Crash
by Andrew Harford 03 Jan '09

03 Jan '09

As planned, we began the migration to Murphy at 9pm last night. Apache itself seemed fine but after a few minutes the kernel seemed to die. We rebooted and went back to working on apache, but the kernel seemed to die again after a while, so www was moved back to deathray. This may be related to the load of nfs traffic, which is something we wouldn't have seen in testing. Either way, unless we can resolve the bug this pretty much screws our plans to run openbsd on this for apache :( andrew. ----- Forwarded message from Eoghan Cotter <johan(a)redbrick.dcu.ie> ----- From: Eoghan Cotter <johan(a)redbrick.dcu.ie> Date: Wed, 10 Dec 2008 02:16:25 +0000 To: admins(a)redbrick.dcu.ie Hey, Here's the trace and ps output from the debugger, ps output ddb{0}> ps PID PPID PGRP UID S FLAGS WAIT COMMAND 29881 28999 21463 6677 2 0x2004000 php-fastcgi 6120 21463 21463 576 3 0x2000180 poll httpd * 4835 3297 21463 32436 7 0x2004000 php-fastcgi 15130 17738 21463 0 3 0x2004080 piperd cronolog 17738 21463 21463 0 3 0x2004080 pause sh 3297 21463 21463 576 3 0x2000180 poll httpd 11115 21463 21463 576 3 0x2000180 poll httpd 21909 21463 21463 576 3 0x2000180 semwait httpd 23564 21463 21463 576 3 0x2000180 semwait httpd 10681 21463 21463 576 3 0x2000180 semwait httpd 18388 21463 21463 576 3 0x2000180 poll httpd 9691 25003 21463 32436 3 0x2004080 poll php-fastcgi 12091 21463 21463 576 3 0x2000180 poll httpd 19947 21463 21463 576 3 0x2000180 semwait httpd 12921 21463 21463 576 3 0x2000180 semwait httpd 28999 21463 21463 576 3 0x2000180 poll httpd 25003 21463 21463 576 3 0x2000181 poll httpd 652 22204 21463 0 3 0x2004080 piperd cronolog 22204 21463 21463 0 3 0x2004080 pause sh 10483 16299 10483 1001 3 0x2004082 ttyin bash 16299 17814 17814 1001 3 0x2000180 select sshd 17814 12650 17814 0 3 0x2004180 netio sshd 21988 26966 21988 0 3 0x2004082 ttyin bash 26966 29262 26966 101031 3 0x2004082 pause zsh 29262 20975 20975 101031 3 0x2000180 select sshd 20975 12650 20975 0 3 0x2004080 netio sshd 21626 31748 21626 101352 3 0x2004082 ttyin zsh 31748 25761 25761 101352 3 0x2000180 select sshd 25761 12650 25761 0 3 0x2004180 netio sshd 3022 26485 3022 0 3 0x2004082 ttyin bash 26485 6403 26485 101089 3 0x2004082 wait bash 6403 6172 6172 101089 3 0x2000180 select sshd 6172 12650 6172 0 3 0x2004080 netio sshd 22166 1 22166 0 3 0x2004082 ttyin getty 16543 1 16543 0 3 0x2000080 select cron 21463 1 21463 0 3 0x2000080 poll httpd 20491 1 20491 0 3 0x2000080 select ypbind 8743 1 8743 556 3 0x2000180 select nrpe 2046 1 2046 0 3 0x2000080 poll syslog-ng 6116 1 6116 521 3 0x2040180 select exim 23556 27092 27092 93 3 0x2000180 kqread ypldap 27092 1 27092 93 3 0x2000180 kqread ypldap 12650 1 12650 0 3 0x2000080 select sshd 5165 1 5165 0 3 0x2000180 select inetd 6186 0 0 0 3 0x2100280 nfsidl nfsio 4829 0 0 0 3 0x2100280 nfsidl nfsio 23194 0 0 0 3 0x2100280 nfsidl nfsio 15120 0 0 0 3 0x2100280 nfsidl nfsio 18433 1 18433 0 3 0x2000080 poll ntpd 4909 1 4909 83 3 0x2000180 poll ntpd 4048 5129 5129 0 3 0x2000080 nfsd nfsd 22795 5129 5129 0 3 0x2000080 nfsd nfsd 4388 5129 5129 0 3 0x2000080 nfsd nfsd 11035 5129 5129 0 3 0x2000080 nfsd nfsd 5129 1 5129 0 3 0x2000080 netcon nfsd 14928 1 14928 0 3 0x2000080 select mountd 81 1 81 28 3 0x2000180 poll portmap 46 0 0 0 3 0x2100200 bored crypto 45 0 0 0 3 0x2100200 aiodoned aiodoned 44 0 0 0 3 0x2100200 syncer update 43 0 0 0 3 0x2100200 cleaner cleaner 42 0 0 0 3 0x100200 reaper reaper 41 0 0 0 3 0x2100200 pgdaemon pagedaemon 40 0 0 0 3 0x2100200 rfwcond raid0 39 0 0 0 3 0x2100200 pftm pfpurge 38 0 0 0 3 0x2100200 usbevt usb1 37 0 0 0 3 0x2100200 usbtsk usbtask 36 0 0 0 3 0x2100200 usbevt usb0 35 0 0 0 7 0x100200 idle31 34 0 0 0 7 0x100200 idle30 33 0 0 0 7 0x100200 idle29 32 0 0 0 7 0x100200 idle28 31 0 0 0 7 0x100200 idle27 30 0 0 0 7 0x100200 idle26 29 0 0 0 7 0x100200 idle25 28 0 0 0 7 0x100200 idle24 27 0 0 0 7 0x100200 idle23 26 0 0 0 7 0x100200 idle22 25 0 0 0 7 0x100200 idle21 24 0 0 0 7 0x100200 idle20 23 0 0 0 7 0x100200 idle19 22 0 0 0 7 0x100200 idle18 21 0 0 0 7 0x100200 idle17 20 0 0 0 7 0x100200 idle16 19 0 0 0 7 0x100200 idle15 18 0 0 0 7 0x100200 idle14 17 0 0 0 7 0x100200 idle13 16 0 0 0 7 0x100200 idle12 15 0 0 0 7 0x100200 idle11 14 0 0 0 7 0x100200 idle10 13 0 0 0 7 0x100200 idle9 12 0 0 0 7 0x100200 idle8 11 0 0 0 7 0x100200 idle7 10 0 0 0 7 0x100200 idle6 9 0 0 0 7 0x100200 idle5 8 0 0 0 7 0x100200 idle4 7 0 0 0 7 0x100200 idle3 6 0 0 0 7 0x100200 idle2 5 0 0 0 7 0x100200 idle1 4 0 0 0 3 0x2100200 bored syswq 3 0 0 0 3 0x100200 idle0 2 0 0 0 3 0x2100200 kmalloc kmthread 1 0 1 0 3 0x2004080 wait init 0 -1 0 0 3 0x2080200 scheduler swapper ddb{0}> Trace Output: ddb{0}> trace text_access_fault(40083e87940, 9, 1c14008, 0, 0, 1810350) at text_access_fault+ 0x1d4 trapbase_sun4v(0, 0, 40083e87c00, 4efe3970, 4a, 4a) at trapbase_sun4v+0x979c end(0, 4001cdd45f8, 40083e87c00, 4001cdc7690, 0, 1810350) at 0x1c14000 dofilewrite(16, 1, 4001cdd45c0, 4efe3970, 4a, 4a) at dofilewrite+0x6c sys_write(9, 40083e87dd0, 40083e87dc0, 0, 0, 1810350) at sys_write+0x58 syscall(40083e87ed0, 4, 4d92db08, 4d92db0c, 0, 41ec6600) at syscall+0x13c softtrap(1, 4efe3970, 4a, 0, 0, 0) at softtrap+0x18c end(ffffffffffffffff, ffffffffffffffff, ffffffffffffffff, ffffffffffffffff, fff fffffffffffff, ffffffffffffffff) at 0x444947bc ddb{0}> Initial output: OpenBSD/sparc64 (murphy.redbrick.dcu.ie) (console) login: Dec 9 23:23:38 su: werdz to root on /dev/ttyp0 Dec 9 23:49:43 su: receive to root on /dev/ttyp2 panic: kernel text_access_fault: pc=1c14008 va=1c14000 kdb breakpoint at 142e1e0 Stopped at Debugger+0x4: nop RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! _______________________________________________ rb-admins mailing list rb-admins(a)lists.redbrick.dcu.ie http://lists.redbrick.dcu.ie/mailman/listinfo/rb-admins ----- End forwarded message ----- -- Andrew Harford System Administrator, DCU Networking Society Equipment Officer, Societies & Publications Committee Your own father said that artists use lies to tell the truth. Yes, I created a lie. But because you believed it, you found something true about yourself. --V

3 5

[Admin-discuss] Hey Changes
by Andrew Harford 26 Dec '08

26 Dec '08

Hey, I've added some logging stuff to hey so we can have huh working again. Since I don't actually know any C i'd appreciate it if a few people could take a look and make sure I haven't done anything crazy. http://www.redbrick.dcu.ie/~receive/hey_main.html (changes are marked in red) andrew. -- Andrew Harford System Administrator, DCU Networking Society Equipment Officer, Societies & Publications Committee Tomorrow will be the most beautiful day of Raymond K. Hessel's life. His breakfast will taste better than any meal you and I have ever tasted. --Tyler Durden

3 4

[Admin-discuss] Web server move
by Andrew Martin 09 Dec '08

09 Dec '08

Hi all, I'm just about to send an announce-redbrick on this but figured I should send another mail to ye to make sure it was seen. We're planning to move murphy back into production tomorrow (Tuesday the 9th) at 9PM. Work is expected to take a few hours, although the longest wait will be for ARP to time out - approximately 4 hours. Everything at the moment seems ready to go, performance is a bit on the low side but it certainly seems enough to handle the current load. This can be tuned later on anyway. The system is up to date (or it was last I looked) and everything that I can think of is working. We're using the prefork MPM. I gave the worker MPM a try but it did not seem to get on well with OpenBSD and 32 visible cores. suPHP also threw a hissy fit, as it did with solaris+mpm_worker. -Andrew

1 0

[Admin-discuss] Murphy, now with 100% more functionality
by Andrew Martin 03 Dec '08

03 Dec '08

Hi all, Anyone who read my blog entry the other day on the committee blog (http://www.redbrick.dcu.ie/~chair/blog/?p=51#more-51 ) will know that a few days ago we were on the verge of giving up on murphy as a web server. Then, a day or two later, one of us happened to stumble on the OpenBSD hardware list. Niagara (and specifically, the T2000 platform) was listed as supported. We decided to try it. Everything seems to work, damn near perfectly :) Most importantly, the OS boots cleanly, out of the box, with no random errors, on an up to date firmware version (which is more then Linux can say for itself). It also didn't make installing things a pain in the head, and it has an extremely well documented, relatively simple and painless update mechanism for both the OS itself and the userland tools (which is more than Solaris can say for itself). We also have a test installation of apache up and running. Version 2.2.10 from the apache website (the packaged version lacked suexec), suphp, pubcookie and a few other bits and pieces redbrick uses are all running nicely. Everything I've tried is working, at least. To test it yourself, add 136.206.15.14 www.redbrick.dcu.ie to your hosts file and poke around. PHP5, Perl, Python, etc are all packaged, as are most of their modules, so this can all be installed and kept up to date painlessly. It's using the prefork MPM - the threaded MPM doesn't get on well with SuPHP. We have the system authenticating off our LDAP directory using ypldap. OpenBSD lacks NSS and PAM support, but it can use a NIS server. Ypldap is a NIS to LDAP proxy. It had a few teething problems (took a while to figure out how to configure it), but it appears to be working perfectly now. The only issue is that only the primary ldap group + local groups will apply (secondary/tertiary ldap groups are ignored) - this is a limitation of ypldap, although I can't see it causing much of a problem, as there are little or no uses the webserver has for secondary groups. All that's left to be done now is to RAID the disks (there's a documented procedure for it online - we've already compiled support into the kernel), and do some performance testing on apache to see if it's delivering what it should be. What do ye all think? Anyone have any opinions on OpenBSD in general? Anyone have any reasons against it that we haven't considered here? -Andrew

6 15