Charles Rowe dot com

I was surfing dzone and I came across two links regarding site monitoring. The first link was “using Twitter for server-monitoring” and the second link was, in a nutshell, making a PHP poor man’s ping without using Net_Ping.
Both of these methods have a little bit of a square peg in a round hole approach, but I definitely understand where they are coming from. My secret embarrassing project has been to build a scalable multi-server monitoring service using the bare minimum amount of utilities. I got pretty close, and I put it down to pursue more worthwhile items. I will eventually finish it with a little divine inspiration. Personally, the primary purpose of the project is to scratch the MacGyver itch when it arises.

The plan:
Have a continuously running monitor script which will check mail, database, and web services on a variable number of servers.
If something is wrong, the monitoring server is supposed to check with another server to confirm that a problem does exist. The second server will respond with the results, and the first server will act accordingly.

The challenge:
Do all this with a bare minimum of tools. I ended up using Bash and Netcat.

The results:
I got the mail and web monitoring script taken care of. To check the database, I would have a keyword pulled from the database and placed on a web page.

The hitch is when the monitoring script goes to alert the other server/monitoring script that something is wrong. Given Netcat isn’t secure, I have to verify the request is coming from a white-listed IP address. Lsof doesn’t work, because by the time I make the check, the connection is already terminated. Checking with Netstat works, but given the connection is listed for awhile, it would leave a loophole for someone to inject their own data without a need to pass the white-list process.

What follows are the tiny scripts that do the monitoring. You could easily throw in a
echo "uh, you broke it"|mail -s "Server Down" foo@bar.com to have these scripts actually DO something.

mailCheck.sh

#! /bin/bash
#requires netcat
# look for a valid quit to show that mail is paying attention

if [ "$stcode" == "221" ]
then
echo 1
else
echo 0


webCheck.sh

#! /bin/bash
#requires netcat
# do not use it on a page where it will match HTTP...my awk skills aren't that good
#URL is first argument
# page is second argument

echo -en "GET /$2 HTTP/1.1\nHost:$1\nConnection:Close\n\n\n\n" |nc $1 80|awk '/HTTP\/1.1/{print $2}'


I don’t know if these scripts are useful (yet). This was more a post of solidarity to those who enjoy the DIY approach
I just need to stress that if the site monitoring is a professional requirement, skip the fun and do it right.

This post could also be titled reason number 5,233,456 for why you shouldn’t develop on your production server.

The reasons why developers arent allowed on production servers:

1. To prevent circumvention of version control systems in the development environment
2. To prevent those moments of hubris that begin with words: “Oops a teeny bug, Let me fix that quickly before anyone notices”

We are interested in item #2. Basically, the more you futz around in a system, the greater the opportunity for a security vulnerability. Your production web server is not an IDE, and it is not a great idea to develop using it.

Whenever Apache encounters a file that it doesn’t have a directive for, it will serve up the contents of that file as text.
This is why servers have something similar to the below code in their httpd.conf:

<Files ~ "^\.ht">
Order allow,deny
Deny from all
Satisfy All
</Files>


This tells Apache to never serve up files who have an extension starting with “ht”. This is so that someone trying to view your .htaccess or .htpasswd files won’t be able to gain the sensitive information within those files.
This is a great start, but the list needs to be expanded upon. What happens when someone views your php script with the wrong extension? They get to see everything. Besides being able to look at possibly proprietary code, If your database password is included within the script you could be in for some trouble. By quickly trying the various common file extensions below on all the visible files on a website, an attacker can come up with some interesting results if the users have been editing and backing up on the go.

Tip #1: Block insecure file extensions from being served up
For brevity, I am only going to list the file matching line. You can apply the rest.
<Files ~ "\.tmp$">
<Files ~ "\.temp$">
<Files ~ "\.bak$">
<Files ~ "\.sav$">

This first group are all common file extensions people use as a poor man’s backup before committing a significant change.

<Files ~ "\.*~$"> - An Emacs backup file (this option can be turned off by the user)
<Files ~ "\.*#"> - A file that was in the process of being edited when something happened (such as a disconnect)
<Files ~ "\.save$"> - A Pico/Nano save file after a disconnect.

<Files ~ "\.old$"> - This is a typical file for when people want to release a new file version but they aren’t comfortable with removing the old one.

Blocking these files will take about 2 minutes, and it is a great safety net to insure that you aren’t exposing any confidential information. This leads us into Tip #2…

Tip #2: Protect your database connections
If someone is able to view your source code through exploit, vulnerability, or sheer accident, it is a good idea to have your database information stored in an include file. By doing this, your database and username are not exposed.

The best place to put this file is outside of your web accessible path

. If your script was in the web root, you could store your PHP $username and $password variables in a file below your web root:
include('../db.php/');
By doing this, even if someone knows exactly where your server credentials are stored, they are going to have a hard time getting to them because they are inaccessible by Apache.
Lastly, don’t use the root database account for your web scripts. I know everyone knows not to do this, but I keep consistently seeing it.
I feel better just reiterating that point.

As an essential requirement to your web development project, the business requirements document acts as a communication conduit between the developers and the project owner (customer). A well written BRD can protect the developers and the customers from miscommunication, inefficiency, and missed deadlines.

Put Simply: the BRD is a carefully worded, non-technical document that lists the features, requirements, and basic operations of an application. It is important to write the document so that your customer can clearly understand it. If applicable, use any of the customer’s specialized terminology to demonstrate clear lines of communication and an understanding of their needs.

If you have yet to use a BRD the important thing is to do your research and get your customer’s requirements on paper. The BRD is your shield against the consistent redesign of an application while it is in the process of development (known as feature creep). The consequences of consistent unplanned redesign are missed deadlines or budgets, and in the worst case, project failure.

Remember, the developer’s job is to serve the customer. The customer has the right, and sometimes the need, to change the scope of the project by modifying their requirements. This is fine. Nothing the developer does can ever be set in stone. If the customer is willing to pay the cost of a scope change, then it is something that has to be done. The BRD is not a wall intended to separate customer from developer, It is a means of bringing the two parties closer together so that they can accomplish the project as a team.

You will often have to go through many versions of a BRD until you are ready for development. Usually there are areas which are unclear and will require prototypes or interviews with a subject matter expert. This is perfectly normal. The goal of the business requirements document is to act as the back bone of the project. Make it clear to the customer that the business requirements document drives the project plan, which in turn sets the deadlines and cost of the project. When the customer understands that a modification of the business plan could push the project boundaries, then the developer has the best chance of avoiding impossible deadlines.

Click here for a rough example of a Business Requirements Document

For those who have never seen a BRD, the above document is an extremely basic example to give you a rough idea. Numbering a BRD in outline format is typical for large Business requirements. When making revisions, or using the business requirements for acceptance tests. It is much easier to say “item 8.4.2.1 needs modification” than trying to explain that a modification is required on pg. 9, 15^th line.

Welcome to my website.
First on the agenda is to clean up WordPress and get it functioning nicely.
After that I hope to write a few posts on web development fundamentals.