Server Woes

I've always know that managing a server's no easy task and definitely not for the faint-hearted. Here's a really complicated situation that's really so frustrating for me that my stress level just keeps climbing.. At times I really feel like strangling someone's neck and twisting his head off... Yeah.. way too much stress....

Here's the situation. We have a major client who's has their own dedicated servers. Their entire hardware is worth over S$1.5 million. That includes routers, switches, firewalls, load balancers, servers, database servers, sms gateways and tons of other stuff I'm unaware of. These machines occupied 2 entire racks in the data centre. It is one of the most complicated configuration, network I've ever experienced. And the complexity's the least of my worries...

I'm undertaking a rather huge project to re-do this client's rewards program from Cold Fusion into PHP. This involves talking to other machines in the rack and stuff lie that. The timeline for this project was to be completed within a month and by now, we're nearly into the third month and still not even halfway there. I'm feeling the stress of the deadline and with the servers not even behaving properly, its driving me nuts.

My project's being developed on one machine. Let's call it Machine A. Machine A is a mirror of Machine B with the same hardware specs and setup. Both are running on RedHat. Machine B is currently hosting a single live website because we're still testing the stability of it.  It was going up and down for the last month until someone realised that its the Load Balancer that's killing the outside connection from this machine. That's a long story so I'll save it for next time. Anyhow, that gave me alot of trouble because I had to migrate the said website on and off Machine B until they could sort it out. Meanwhile, with the website off Machine B, the users can't stop complaining about how slow our backup server was (which is true).  Just on Monday, I move them back to Machine B and they haven't been complaining about the speed, so phew....

Ok back to Machine A. The person who setup Redhat and configures both Machines is an expert in Linux Security so from my expectation, he's someone who knows what he's doing and thus I should have confidence in him. Wait, I'm not saying I don't.. Its just that its incredibly hard to have to wait for said person to do certain changes while I waste hours of my time waiting before I can proceed. You see, said person's working full time and my boss hired him to manage the machines as a freelancer.

Just a few days ago, I realised I needed to grant some privilleges to a MySQL user account because the application I'm using needed to lock tables. I attempted to do this myself from my mysql access but I realised it was denied access. So I asked the server guy to create the access for me and grant the required privillege to the database user. He did this.. and I'm happy and continued with my work. The next day, I get these 'Lock tables access denied' errors again on the screen so I ask him about it. The first thing he asked me was, 'Didn't I fix this yesterday?'... I told him 'exactly' thouggh I was thinking , 'you obviously didn't fix this properly or somehow something else overwritten this settings'. Ok, so he took about 3 hours to fix this since he's at work.

Finally. and guess what? Same error today.. Its driving me insane.. Wtf...??? This should have been fixed and I even tested it after he told me he fix it, and it happened again the very next day... Obviously something's wrong here.

Next on my rant list is the server file permission. When I login via FTP to upload files, I know FTP user usually have a different set of permission, though I always think that FTP user and webserver user should belong in the same user groups, for the simple fact that we always have to use chmod or create directories. FTP user chmod a directory to 777 should mean webserver user can write files into the directory but this is not the case. My application threw out a series of nasty 'Permission Denied' errors,  'Unable To create files', 'mkdir(dir) fails' etc.

So I went back to server guy and told him to fix this. He then told me that I should chmod the directory to 777.... Didn't I already do this before I went to him? Anyhow, frustrated as I was, I took a deep breath and told him I already did it. So I politely suggested to him that he should perhaps check if the webserver user has permission to create/delete/mkdir on the web directories, especially directories created by the ftp user. He told me they are configured as I said.. Now strange, because I still keep getting all these file permisison errors..

So I scratched my head and he scratched his too. Look, as a programmer I can at most provide clues based on my own conclusion and experience what might be the problems but I wouldn't have a clue what kind of situation in the server can produce this kind of crap errors. If he had configured the webserver user as he said, I really don't understand why I'm still getting all these errors. I did search on google and forums and something call SELinux might be screwing the permissions up despite being set properly so I sent him the link. Actually I was thinking searching for solutions about server configuration should be his job. NOT mine! Regardless he told me he disabled SELinux and I tested the application again this morning.. Still the same.. Right now I'm totally clueless and I still think its the permission thing in the webserver user because that's all PHP tells me. 'Permission in 'dir' denied'. 'mkdir(dirname)' denied..

I have to stop work on my project because of this problem. I simply can't route around it and continue. Frustrating!!!