John's Technical Blog

Bin2Txt - Remove Non-printable Characters from a Text File with Java

2011-10-31T09:54:00.000-05:00

I recently was dealing with some DB2 "unload" (e.g., export) files that I wanted to parse and then load into Oracle. I found that the unload files use a lot of binary characters, which makes it very difficult to parse. I wrote the following Java class to convert the unprintable characters into a tilde (which is a character that does not occur in the data). This resulted in DB2 unload files that were parsable as fixed-width data files.
The main problem this approach does not attempt to solve is that the DB2 unload files save numeric fields as the actual value, not the digit equivalent (i.e., the number 84 is unloaded as the ASCII-equivalent "T", not "84"). This code obviously does not reference the DB2 "punch" (e.g., parse instruction) files, so it makes no attempt to parse the files into fields itself - that is a separate exercise in my case. BTW, if there is a good way to import these files into Oracle automatically, please let me know, as I have not been able to find a better solution.
This code is fairly generic, and can be used for other purposes beyond converting DB2 unload files, so if you have a need to replace non-printable characters in text files, you can start with this code base.

packagecom.threeleaf.bin2txt;

import java.io.File;

importjava.io.FileInputStream;

importjava.io.FileOutputStream;

import java.io.IOException;

import java.io.InputStream;

importjava.io.OutputStream;

/**

* Purpose is to read a file and replace non-printablecharacters with a given character.

* Specifically, I want to use this to make DB2unload files parsable with other applications so

* that the data can be imported into Oracle.

* @author John A. Marsh

* @since 2011-10-27

public final class Bin2Txt {

/**

* Run this class from the command line with:

* java Bin2Txt<pathAndFilename>.

* @param args

* the filename to convert

* @throws IOException

* Signals that an I/Oexception (e.g., file not found) has occurred.

public static void main (final String[] args) throws IOException {

final byte ASCII_SPACE = 32;

final byte ASCII_CR = 13;

final byte ASCII_LF = 10;

final byte ASCII_TILDE = 126;

try {

final File file = new File(args[0]);

final InputStreaminputStream = new FileInputStream(file);

final long fileLength =file.length();

* Array needs to be created withan int type, so need to check to ensure that file is

* not larger thanInteger.MAX_VALUE.

if (fileLength > Integer.MAX_VALUE) {

throw new IOException("File is toobig");

}

/* Create the byte array to hold the data*/

final byte[] bytes = new byte[(int) fileLength];

/* Read in the bytes */

int offset = 0;

int numRead = 0;

while (offset < bytes.length && (numRead= inputStream.read(bytes, offset, bytes.length - offset)) >= 0) {

offset += numRead;

}

/* Ensure all the bytes have been read in*/

if (offset < bytes.length) {

throw new IOException("Could notcompletely read file " + file.getName());

}

inputStream.close();

for (int i = 0; i < bytes.length; i++) {

if (bytes[i] ==ASCII_CR && bytes[i + 1] == ASCII_LF) {

* Preserve line breaks(carriage return + line feed) by skipping over them.

* Note that I don't checkfor end of file here because I already know my

* particular files willnever end with a CRLF.

i = i + 2;

}

if (bytes[i] <ASCII_SPACE || bytes[i] > ASCII_TILDE) {

/* Replace allnon-printable characters. */

bytes[i] = ASCII_TILDE;

}

/* Output file name will be the same as theinput, with ".out.txt" added to the end. */

final OutputStreamoutputStream = new FileOutputStream(args[0] + ".out.txt");

outputStream.write(bytes);

outputStream.close();

} catch (final ArrayIndexOutOfBoundsException e) {

* If no file was passed on thecommand line, this exception is generated. A message

* indicating how to the classshould be called is displayed.

System.out.println("Usage: javaBin2Txt filename\n");

}

Here is a batch file that will convert all the files in a given directory:

::Directory where Bin2Txt.class is located ::

cdC:\projects\workspace\bin2txt\bin\

::Put in directory where unload files are ::

for %%f in ("C:\projects\Database\Unloads\*.txt") do call java com.threeleaf.bin2txt.Bin2Txt%%f

Solved: Firefox Tab Icons Are Missing

2011-10-13T16:35:00.001-05:00

I have been frustrated over the last few weeks because I could not figure out why the icons suddenly stopped showing up in Firefox. I have scoured the web and tried several suggestions, but none of them worked for me. I finally got clued in that it might be one of my extensions, and sure enough, going to Help » Restart With Add-ons Disabled caused the icons to reappear. After a little trial and error I finally found that it was the Favicon Picker 2. I wanted the functionality, so I tried Favicon Picker 3, but it caused the same problem. I finally found that Bookmark Favicon Changer (https://addons.mozilla.org/en-US/firefox/addon/bookmark-favicon-changer/) allowed me to customize my bookmark icons without causing the tabs to go blank. So, if you find that the favicons in your tabs are absent, give it a try. As of this writing, I am using Firefox 7 and Bookmark Favicon Changer 1.54.

Interning String Literals In Java

2011-09-13T08:02:00.000-05:00

A coworker and I were discussing strings, and some of the things we had heard about how the Java compiler stores and retrieves them from memory. We were particularly wondering if constants, local variables, literals, and runtime generated strings were handled differently. I wrote the following JUnit 4 test to show how all these cases are handled (Java JDK 1.6.0_27):

/**

* Test string interning affects on literal and runtime strings.

public final class StringInternTest {

/**

* String intern test.

@Test

public void stringIntern () {

/* literals */

assertEquals("date", "date");

assertTrue("date" == "date");

/* Compare to an external constant */

assertEquals("date", Constants.DATE);

assertTrue("date" == Constants.DATE);

/* Compare locally defined strings. */

final String myDateString = "date";

assertEquals("date", myDateString);

assertTrue("date" == myDateString);

final String myDateString2 = "date";

assertEquals(myDateString, myDateString2);

assertTrue(myDateString == myDateString2);

assertEquals(Constants.DATE, myDateString);

assertEquals(Constants.DATE, myDateString2);

assertTrue(Constants.DATE == myDateString);

assertTrue(Constants.DATE == myDateString2);

/* Create new strings at runtime. */

final String runtime1 = new String("date");

final String runtime2 = new String("date");

assertEquals("date", runtime1);

assertEquals("date", runtime2);

assertEquals(Constants.DATE, runtime1);

assertEquals(Constants.DATE, runtime2);

assertEquals(runtime1, runtime2);

assertFalse(runtime1 == runtime2); // !!!

assertFalse("date" == runtime1); // !!!

assertFalse("date" == runtime2); // !!!

assertFalse(Constants.DATE == runtime1); // !!!

assertFalse(Constants.DATE == runtime2); // !!!

/* Intern the runtime strings. */

final String interned1 = runtime1.intern();

final String interned2 = runtime2.intern();

assertTrue(interned1 == interned2);

assertTrue("date" == interned1);

assertTrue("date" == interned2);

assertTrue(Constants.DATE == interned1);

assertTrue(Constants.DATE == interned2);

}

If you set a break point in this test and examine the variables you will find that Constants.DATE, myDateString, myDateString2, interned1, and interned2 all have the same internal object ID. I learned online that all string literals are supposed to be interned when the application is compiled, which accounts for the variables having the same ID.

Strings stored in runtime1 and runtime2 are each given a unique object ID when instantiated at runtime, and thus return a false when == is tried with any of the interned strings.

Some have suggested that manually interning is better because using == is much faster (5x) than equals() (because comparing object IDs is faster than comparing string lengths || characters). However, others dismiss this as a minor gain at best, and I can certainly see that using == might lead to some hard-to-find bugs if a non-interned string is compared to either an interned or another non-interned string. I also learned that main argument strings (args[]) are not interned.

Interned strings are stored in the PermGen (Permanent Generation) memory, and it is possible to fill up that space with strings if one is not careful.

References:

SOLUTION: "SQL driver not found org.apache.derby.jdbc.ClientDriver" When Using Oracle with Sonar and Maven

2011-09-08T09:35:00.000-05:00

I am new to Sonar and Maven, and have been perplexed for several hours with the "SQL driver not found org.apache.derby.jdbc.ClientDriver" error I was getting when running mvn sonar:sonar from the Windows command line after starting the Sonar server.

What is not clear in the instructions is that you must modify both <sonar path>\conf\sonar.properties and <maven path>\conf\settings.xml to get this to work. The Sonar properties tell the Sonar server what database driver to use in the Sonar server, and the Maven settings file tells Maven what database driver to use when it is running Sonar.

Here are the changes I made to connect to my local Oracle XE instance:

sonar.properties

...

sonar.jdbc.username:                       sonar
sonar.jdbc.password:                       sonar

...

# Comment the following lines to deactivate the default embedded database.
#sonar.jdbc.url:                            jdbc:derby://localhost:1527/sonar;create=true
#sonar.jdbc.driverClassName:                org.apache.derby.jdbc.ClientDriver
#sonar.jdbc.validationQuery:                values(1)

...

sonar.jdbc.url:                            jdbc:oracle:thin:@localhost:1521:xe
sonar.jdbc.driverClassName:                oracle.jdbc.driver.OracleDriver
sonar.jdbc.validationQuery:                select 1 from dual

settings.xml

<profiles>
  ...
  <profile>
    <id>sonar</id>
    <activation>
      <activeByDefault>true</activeByDefault>
    </activation>
    <properties>
      <sonar.jdbc.url>
      jdbc:oracle:thin:@localhost:1521:xe</sonar.jdbc.url>
      <sonar.jdbc.driver>
      oracle.jdbc.driver.OracleDriver</sonar.jdbc.driver>
      <sonar.jdbc.username>sonar</sonar.jdbc.username>
      <sonar.jdbc.password>sonar</sonar.jdbc.password>
      <sonar.host.url>http://localhost:9000</sonar.host.url>
    </properties>
  </profile>
</profiles>

SOLUTION: Modify PMD's ShortVariable Rule To Ignore ID Fields

2011-09-06T12:42:00.000-05:00

Several people online have asked about the problem of not being able to add exceptions to PMD's ShortVariable rule. Most often, it has been the desire to get PMD to ignore the case where the variable name is 'id'. I was not able to find a solution online, but I worked with the XPath until I came up with a working solution.

In Eclipse, go to Window » Preferences » PMD » Rules configuration » ShortVariable » Edit Rule...
Change the XPath field to:
//VariableDeclaratorId[(string-length(@Image) < 3) and (not (@Image='id'))]
[not(ancestor::ForInit)]
[not((ancestor::FormalParameter) and (ancestor::TryStatement))]
Click Apply » Confirm rebuild » Click Ok » Confirm rebuild
Restart Eclipse
Right-click on the project » PMD » Check code with PMD

You should then see the warning markers disappear on your id fields.

Can is_string() be used without an isset() for unset variables in PHP?

2009-12-19T10:59:00.006-05:00

The Short Answer

Yes, if there is the potential for an undeclared variable to be passed to functions like is_string(), then you must use the isset() function first.

Example:

if(isset($foo) && is_string($foo)) {
    // Do something with $foo
}

The Long Answer

I recently had an error occur in a 3rd party function I use on my web site. The code assumed that at a certain point that a variable would be set and a string function could be run against it. I wanted to modify the if statement to check to see if the variable was a string before continuing with the said block of code. What I did not know was whether or not the is_string function in PHP worked in the same way as the instanceOf operator in Java. In Java, you do not need to detect if the variable is null before the operation, because instanceOf has the built in ability to determine that a null is not an instance of anything.

The reason I wanted to test this is because the error resulted after a customer interaction with the site that I am not able to reproduce (this is intentional, because it involves translating a user's file, which I do not keep a copy of in accordance with my privacy statement).

Example Code

To test this I wrote a PHP page to determines what happens when a variable is in different states. I also expanded the testing to include is_array, is_bool, is_float, is_int, and is_object. I also include some custom error handling, so you can try this at home without filling up your error log.

To see this in action: Using is_string() and isset() Demo

<?php 
/* we will do our own error handling */
function jamErrorHandler($errno, $errmsg, $filename, $linenum, $vars) {
    $errortype = array (
        E_ERROR             => 'Error',
        E_WARNING           => 'Warning',
        E_PARSE             => 'Parsing Error',
        E_NOTICE            => 'Notice',
        E_CORE_ERROR        => 'Core Error',
        E_CORE_WARNING      => 'Core Warning',
        E_COMPILE_ERROR     => 'Compile Error',
        E_COMPILE_WARNING   => 'Compile Warning',
        E_USER_ERROR        => 'User Error',
        E_USER_WARNING      => 'User Warning',
        E_USER_NOTICE       => 'User Notice',
        E_ALL                => 'All Error',
        E_STRICT             => 'Strict Error',
        E_RECOVERABLE_ERROR    => 'Recoverable Error'
    );
    print '<strong>* ' . $errortype[$errno] . ': ' . $errmsg . ' *</strong><br />';
}
function jamExceptionHandler($exception){
    jamErrorHandler($exception->getCode(), $exception->getMessage(), $exception->getFile(), $exception->getLine(), $exception->getTrace());
}
set_error_handler('jamErrorHandler');
set_exception_handler('jamExceptionHandler');
print '<h2>$foo = 123</h2>';
$foo = 123;
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>$foo = 123.456</h2>';
$foo = 123.456;
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>$foo = \'123\'</h2>';
$foo = '123';
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>$foo = \'abc\'</h2>';
$foo = 'abc';
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>$foo = date(\'r\')</h2>';
$foo = date('r');
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>$foo = true</h2>';
$foo = true;
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>$foo = false</h2>';
$foo = false;
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>$foo = array(\'1\', \'a\')</h2>';
$foo = array('1', 'a');
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>$foo = jamObject()</h2>';
class jamObject {
    function test() {
        print 'test';
    }
}
$foo = new jamObject();
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
if(is_string($foo)) {
    print '<strong>$foo is a string.</strong><br />';
} else {
    print '$foo is not a string.<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
print '<h2>unset($foo)</h2>';
unset($foo);
if(isset($foo)) {
    print '<strong>$foo is set.</strong><br />';
} else {
    print '$foo is not set.<br />';
}
if(isset($foo) && is_string($foo)) {
    print '<strong>$foo isset and is a string.</strong><br />';
} else {
    if(isset($foo)){
        print '$foo isset, but is not a string.<br />';
    } else {
        print '$foo is not set, so is not string.<br />';
    }
}
try {
    if(is_string($foo)) { // Produces error!
        print '<strong>$foo is a string.</strong><br />';
    } else {
        print '$foo is not a string.<br />';
    }
} catch (Exception $e) {
    print '<strong>Error in is_string($foo)</strong>' . $e->getMessage() . '<br />';
}
if(is_float($foo)) {
    print '<strong>$foo is a float.</strong><br />';
} else {
    print '$foo is not a float.<br />';
}
if(is_int($foo)) {
    print '<strong>$foo is an integer.</strong><br />';
} else {
    print '$foo is not an integer.<br />';
}
if(is_bool($foo)) {
    print '<strong>$foo is a boolean.</strong><br />';
} else {
    print '$foo is not a boolean.<br />';
}
if(is_object($foo)) {
    print '<strong>$foo is an object.</strong><br />';
} else {
    print '$foo is not an object.<br />';
}
if(is_array($foo)) {
    print '<strong>$foo is an array.</strong><br />';
} else {
    print '$foo is not an array.<br />';
}
?>

ELIZA, the Computer Therapist Facebook application

2009-03-06T21:14:00.002-05:00

I have converted the classic (read: old) ELIZA application to run on Facebook. ELIZA is a Rogerian therapist parody originally written by Joseph Weizenbaum. This application is for entertainment purposes only. ELIZA is not human and is not a real therapist. If you have a session with ELIZA that you think was amusing or thought provoking, you can publish it on your profile.

Please give it a try at:
http://apps.facebook.com/eliza-therapist/

I would appreciate your feedback.

This is my first Facebook application. I have ideas for two more at the moment, but they are going to take more work than this one... I wanted to start with something simple :-).

If you are not on Facebook, I have also made an online version at ThreeLeaf.com:
http://www.threeleaf.com/freelance-work/demos/eliza/

Enjoy!
John

Gateway Extended Warranty... Definitely a Waste of Money… Maybe the Computer Too

2008-10-02T07:26:00.000-05:00

This falls into the category of "you should have known better." My first experience with a Gateway computer was as a graduate student at the University of Wyoming. The first thing I heard about these computers is that the motherboard had a design flaw, which resulted in the CPU occasionally popping out of its socket. Someone would then have to open the case and reseat it whenever this happened. I worked with a lot of Gateway computers as a PC technician at Research Triangle Institute. During my 3 year tenure in that position, RTI signed a purchase agreement with Gateway, and I received, installed, and repaired approximately 200 Gateway computers during that time. I do not have an exact count, but I am pretty sure nearly every one of those units needed repair at some point during that short period of time. The worse case was a batch of 26 computers whose hard drives all gave out within 4 months of being received.

Many years later, in 2005, I am looking at purchasing a new computer. I included Gateway in my list of options despite my past experiences with them, hoping that they may have finally learned to build reliable computers despite their long history of poor quality. When it came to decision time, Gateway was one of my final two contenders. I ended up going with Gateway because they offered more features for a lower price (not inexpensive... this was one of their top-of-the-line PCs). I purposely purchased the four year extended warranty, hedging my bet that if I did have problems, the PC would be covered. The customer service while I was at RTI typically was typically good, and they would often ship replacement parts to me within 2-3 business days, and after I had replaced them, I could send the old parts back. I am not naïve enough to think that "regular" customers would get the same level of support as corporate customers, but I was really unprepared for what my situation would be.

In June of this year (2008), the PC suddenly stopped working. It would not boot. It would not even POST. It was dead as a doornail with no apparent cause. After running through the online support tests, I eventually determined that it had to be a motherboard problem. Not a problem, I thought, I will put in a service request and get this fixed up. My first issue became what the four-year warranty actually covered. I was certain that I had purchased on-site service. My first several correspondents ranged from "your warranty expired" to "it only covers parts" to "even if you had on-site service, they would only replace the part, service representatives do not visit to diagnose the problem." My online receipt has a cryptic entry for the four year warranty, so it is quite possible that it does only cover replacement. The service people I corresponded with said I should check my receipt or order a replacement receipt. Unfortunately, even though I have all my manuals, I could not find my receipt for some reason. I did order a replacement, which involves both an online order form and an e-mail confirmation, but I did not receive a receipt, though it does not appear that they charged my card either. Good grief! It is hard for me to understand why a simple process that has a manual customer service follow-up can still fail.

Ok, so by this time I have been without my PC for about 3 weeks bickering with Gateway over the warranty and receipt. I decide it is not worth the hassle anymore. I purchased their $67 pre-paid shipping box (miraculously, that order went through) and sent my computer off. On August 8, I got a confirmation e-mail that Gateway received my PC, and I should expect a 3-5 day turnaround.

So I wait.

And wait.

Finally, after two weeks, I send in an e-mail asking for a status report. I get a message back indicating that they determined it was indeed a motherboard problem, and they did not have the part in stock, but it had been ordered. The computer should be ready in 3-5 more days.

So I wait.

After a week with no communication, I write again. Again, I am told that the repair is taking longer than expected, but it should be shipped in a couple of days.

So I wait.

After another week, I am pretty unhappy that I have no computer, and had no communication. I sent a more terse e-mail asking for an update and some kind of immediate solution, like an equivalent replacement. My request was returned with a profuse apology about the delay, and an assurance that they were doing everything they could. However, that obviously did not include a replacement or a loaner.

So I wait.

Again, after another week I send another exasperated e-mail in, requesting action, and a phone number where I can call someone. I received a reply simply stating that they were working on it. No phone number, of course.

So I wait.

Finally! A communication arrived that I did not have to initiate! They said they had finished the repair on my laptop, but needed my permission to wipe the hard drive to reinstall the operating system. I replied that I had sent in a PC, not a laptop, but if they were talking about the PC then yes, of course they could reinstall the operating system (I had actually sent it in with a blank drive, because I cannot lose what is on the hard drive). About three days later I get an e-mail stating that they were finished and were shipping the PC back to me. I should receive it in 3-5 days, unless I had purchased the gold plan, in which case I would receive it in one day (I do not think I had the gold plan, but I could wait 5 days if needed, and in my opinion it was pointless to include that note since it would have been too late to purchase it anyway) Hurray! … Or so I thought.

So I wait.

After another week goes by I send in a request for the tracking number, since I had not yet received the PC. They responded saying that the package had been received and included the tracking number. I go to the FedEx site and find that the destination was Temple, TX (I live in NC). So, I wrote back and said that they had shipped it to the wrong address, and that they needed to either recover the PC or send me an equivalent. I get a response back that the PC needed more repairs and was shipped to Temple, TX, where they contract out repairs. Actually, they said it was still "in transit," indicating to me that the really had no idea where the PC was.

So I wait.

It has now been another week since my last e-mail, and I am about to e-mail them again for a status update. I have no idea if Gateway really knows where my PC is, what state of repair it is in, nor do I know if I will ever get my PC or a replacement.

I have reminded them in several of my e-mails that I paid a lot for the PC and more for the extended warranty, but except for the one apologetic message, I have received no sympathy and no action to help resolve the problems. This level of service is completely abominable, and unacceptable. It certainly seems to me that if you purchase a four-year warranty that there is a built-in expectation that the seller would have some plan to deal with any and all repairs during that timeframe, even if it is just to send an equivalent computer back (I suspect an equivalent replacement today costs less than the price I paid for the warranty).

I am posting this blog entry so that my friends, family, and readers will reconsider any thoughts they might have of purchasing Gateway computers. As I stated at the beginning, I should have known better, but I will never make that mistake again, and I hope my readers will not either.

The onContextMenu Event Handler

2008-01-25T22:16:00.000-05:00

onContextMenu event does not fire on disabled fields

The onContextMenu event (right-click in Windows) is a proprietary handler in HTML, but is very useful for displaying custom context menus on web pages. For this particular article, I was investigating an application where the business owner wanted a context menu available on disabled fields. The problem specifically occurs because all click events are apparently ignored on disabled fields. Note that Opera apparently ignores the onContextMenu event completely, because it is not a W3C compliant attribute.

Regardless of the W3C non-compliance, the business still wanted this behavior (as far as we know everyone is using IE or FF). Try the following in your browser to see how it behaves.

<input id="input1" type="text" value="input1" oncontextmenu="alert('input1');return false;" />

Attempt 1: Nest it in a div

The first thing we tried was to wrap the input fields in a div element that had an onContextMenu attribute defined. We found that while this solved the problem in IE7, FF did not "bubble up" the event through the disabled input field. However, any text inside the div element did respond to the onContextMenu event. Try the following in your browser to see how it behaves.

<div oncontextmenu="alert('input3');return false;">Inside div, outside input3 field 

 <input id="input3" type="text" value="input3" />

</div>

Solution 1: Convert from disabled to read-only

After a little more research, I found another attribute named "readonly" that I had not used before. I tried it, and found that the onContextMenu event worked just fine in both IE7 and FF.

<input id="input5" type="text" value="input5" readonly="readonly" oncontextmenu="alert('input5');return false;" />

Solution 2: Make an external, clickable object

I definitely prefer W3C compliant code, however, so a compliant solution would have to involve an external, clickable element to trigger the event. The » symbol below has an onClick event, which works in IE, FF, and Opera.

<input id="input7" type="text" value="input7" disabled="disabled" /> <span onclick="alert('input7');return false;" style="cursor:pointer;">&raquo;</span>

There are some accessibility issues with this approach, of course. One might want to use a link instead of a span to underline the clickable element or choose a more obvious graphical symbol. There are other inherent accessibility issues with the entire context menu approach, but I will not go into those here.

Zip Code Distances

2007-04-04T15:14:00.000-05:00

I am currently involved in a project where the program I am writing is supposed to return a list of the 10 doctors nearest to a visitor's zip code. Note that this is not the same as doctors in a given radius (say 10 miles), but it does use the same basic formula. This article will be the first of a two part series and lays the groundwork for calculating what is nearest to a zip code. In the second part I intend to show how I integrate this logic with a list of doctors with addresses.

Below you will find the results of my experimental queries that compare the classic radius methodology with my own "10 nearest" methodology. My production program uses a UDB (DB2) database, and I thought of the performance contrasts between UDB and MySQL was interesting, so I am going to show all the comparisons below. Finally, I wanted to show what differences there were between the more accurate "Haversine Formula" as compared to the less intensive "Spherical Law of Cosines" when calculating the distance between two points on a sphere. These examples should give you plenty of real life code you can use in your own programs.

The company I work for subscribes to the zip code database (referred to here as ZipUSA) from http://www.zipdatafiles.com/data/ The data comes from the United States Postal Service and other sources, but I have occasionally found strange geocoding results. I will probably write a different article about that using a different application that I have written.

Within 10 miles

Haversine Formula

The following finds the zip codes within 10 miles of 27712 using the haversine (great circle) equation. The field names are based on the ZipUSA tables.

SELECT DISTINCT
destination.zip_code,
3956 * 2 * ASIN(SQRT(
 POWER(SIN((origin.lat - destination.lat) * 0.0174532925 / 2), 2) +
 COS(origin.lat * 0.0174532925) *
 COS(destination.lat * 0.0174532925) *
 POWER(SIN((origin.lng - destination.lng) * 0.0174532925 / 2), 2)
)) distance

FROM
tblZipUSA origin,
tblZipUSA destination

WHERE
origin.zip_code = '27712'
AND
3956 * 2 * ASIN(SQRT(
 POWER(SIN((origin.lat - destination.lat) * 0.0174532925 / 2), 2) +
 COS(origin.lat * 0.0174532925) *
 COS(destination.lat * 0.0174532925) *
 POWER(SIN((origin.lng - destination.lng) * 0.0174532925 / 2), 2)
)) < 10

ORDER BY
distance, zip_code

The ZipUSA database was set up with the same indexes in MySQL and UDB (DB2). Queries in MySQL were run in the phpAdmin tool. The UDB/DB2 queries were run in Quest.

MySQL

zip_code	distance
27712	0
27704	4.75326962990057
27705	5.34287681627437
27708	5.57383981806715
27503	5.58564401941337
27701	6.41245709075393
27706	6.58029004252293
27709	6.59449266010451
27702	6.61586916445661
27710	6.61586916445661
27711	6.61586916445661
27715	6.61586916445661
27717	6.61586916445661
27722	6.61586916445661
27707	8.57121741472298
27703	9.5828790270889
27278	9.58552637370736
27564	9.72570727493623
27509	9.80777888765073

Showing rows 0 - 18 (19 total, Query took 2.1139 sec)

DB2/UDB

zip_code	distance
27712	0
27704	4.75326962990023
27705	5.34287681627435
27708	5.57383981806714
27503	5.58564401941335
27701	6.41245709075397
27706	6.58029004252301
27709	6.5944926601046
27702	6.61586916445641
27710	6.61586916445641
27711	6.61586916445641
27715	6.61586916445641
27717	6.61586916445641
27722	6.61586916445641
27707	8.571217414723011
27703	9.582879027088699
27278	9.58552637370768
27564	9.725707274936291
27509	9.80777888765007

19 rows selected in 35.77 secs.

I was very shocked to have such a slow run time in UDB! I was expecting better performance from a corporate-level product. I also went back and checked all the indexes to verify they were on the same fields as were in my MySQL database. I do not have enough knowledge about UDB to explain why this disparity exists or how one might optimize the query to run better on this platform. I would not think one should have to do that.

Spherical Law of Cosines

SELECT DISTINCT
destination.zip_code,
3956 * ACOS(
 SIN(origin.lat * 0.0174532925) * SIN(destination.lat * 0.0174532925) +
 COS(origin.lat * 0.0174532925) * COS(destination.lat * 0.0174532925) *
 COS((destination.lng - origin.lng) * 0.0174532925)
) AS distance

FROM
tblZipUSA origin,
tblZipUSA destination

WHERE
origin.zip_code = '27712'
AND
3956 * ACOS(
 SIN(origin.lat * 0.0174532925) * SIN(destination.lat * 0.0174532925) +
 COS(origin.lat * 0.0174532925) * COS(destination.lat * 0.0174532925) *
 COS((destination.lng - origin.lng) * 0.0174532925)
) < 10

ORDER BY
distance, zip_code

MySQL

zip_code	distance
27712	0
27704	4.75326963012033
27705	5.34287681651191
27708	5.57383981817397
27503	5.58564401945863
27701	6.41245709087189
27706	6.58029004273494
27709	6.59449266019943
27702	6.61586916464669
27710	6.61586916464669
27711	6.61586916464669
27715	6.61586916464669
27717	6.61586916464669
27722	6.61586916464669
27707	8.57121741478871
27703	9.58287902708428
27278	9.58552637385399
27564	9.72570727482013
27509	9.80777888768795

Showing rows 0 - 18 (19 total, Query took 1.9229 sec)

DB2/UDB

zip_code	distance
27712	0
27704	4.75326963012037
27705	5.3428768165119
27708	5.57383981817396
27503	5.58564401945862
27701	6.41245709087191
27706	6.58029004247087
27709	6.59449266019946
27702	6.61586916438409
27710	6.61586916438409
27711	6.61586916438409
27715	6.61586916438409
27717	6.61586916438409
27722	6.61586916438409
27707	8.571217414991439
27703	9.58287902726561
27278	9.58552637385398
27564	9.72570727482014
27509	9.807778887510789

19 rows selected in 11.05 secs.

It is obvious that using the Spherical Law of Cosines equation is much faster than the Haversine Formula. MySQL shows a 5% improvement in speed and UDB shows a 53% performance improvement. It is also obvious that the distances calculated are very close. For these mile calculations, the first seven decimal places are the same. In most applications I have seen, the miles are rounded off to two decimal places, meaning that there is no reason to use the Haversine formula for distances as far apart as most "centers of zip codes" are. Haversine calculations are more suited to much closer calculations. Also, since zip code to zip code distances are only used for approximate distances, it would not make sense to nit-pick over the only-so-slightly more accurate results the Haversine formula gives. And even beyond that, accuracy depends a lot on the latitude and longitude data supplied. I have found that different sources have different geocoding information concerning the geographic center of zip codes.

10th closest

The next part of this experiment was to find the 10 nearest zip codes to the one given. The first step is finding out which zip ranks number 10. In this case it is simply a matter of choosing the 10th zip code from the list.

Haversine Formula

SELECT DISTINCT
destination.zip_code,
3956 * 2 * ASIN(SQRT(
 POWER(SIN((origin.lat - destination.lat) * 0.0174532925 / 2), 2) +
 COS(origin.lat * 0.0174532925) *
 COS(destination.lat * 0.0174532925) *
 POWER(SIN((origin.lng - destination.lng) * 0.0174532925 / 2), 2)
)) distance

FROM
tblZipUSA origin,
tblZipUSA destination

WHERE
origin.zip_code = '27712'

ORDER BY
distance, zip_code

LIMIT 10, 1

Zip Code	Distance
27711	6.61586916445661

Showing rows 0 - 0 (1 total, Query took 3.4441 sec)

Spherical Law of Cosines

SELECT DISTINCT
destination.zip_code,
3956 * ACOS(
 SIN(origin.lat * 0.0174532925) * SIN(destination.lat * 0.0174532925) +
 COS(origin.lat * 0.0174532925) * COS(destination.lat * 0.0174532925) *
 COS((destination.lng - origin.lng) * 0.0174532925)
) AS distance

FROM
tblZipUSA origin,
tblZipUSA destination

WHERE
origin.zip_code = '27712'

ORDER BY
distance, zip_code

LIMIT 10, 1

zip_code	distance
27711	6.61586916464669

Showing rows 0 - 0 (1 total, Query took 3.1833 sec)

10th Closest Without PO Box Zip Codes

If you have a stiuation where you will be working only with street addresses and can eliminate the zip codes, there is some performance improvement. In the ZipUSA tables, this is accomplished by placing fac_cd and zip_class in the WHERE clause. However, I soon found that 1) some of our doctors are using P.O. Boxes, and 2) some small towns only have post offices with PO Boxes. I ended up not using that further past this experiment.

SELECT DISTINCT
destination.zip_code,
(3956 * (2 * ASIN(SQRT(
 POWER(SIN(((origin.lat-destination.lat)*0.017453293)/2),2) +
 COS(origin.lat*0.017453293) *
 COS(destination.lat*0.017453293) *
 POWER(SIN(((origin.lng-destination.lng)*0.017453293)/2),2)
)))) distance

FROM
tblZipUSA origin,
tblZipUSA destination

WHERE
origin.zip_code='27712'
AND destination.fac_cd = 'P'
AND destination.zip_class != 'P'

ORDER BY distance
LIMIT 10,1

Zip Code	Distance
27703	9.58287927196898

Showing rows 0 - 0 (1 total, Query took 2.4226 sec)

Nearest 10

Now that we can find the 10th closest zip code, we can join the two types of queries together to find all the zip codes up to the closest 10th. As a pleasant side effect, if there is a tie for 10th place, all of those zip codes are included as well. This becomes particularly obvious in the case where several P.O. Box zip codes are served from the same post office, and thus have the same geocoding. Of course, it is possible that there will be two or more physically distinct zip codes that will be equidistant from a given reference zip code, but I can only imagine that would be a very rare case.

Haversine Formula

SELECT DISTINCT
destination.zip_code,
3956 * 2 * ASIN(SQRT(
 POWER(SIN((origin.lat - destination.lat) * 0.0174532925 / 2), 2) +
 COS(origin.lat * 0.0174532925) *
 COS(destination.lat * 0.0174532925) *
 POWER(SIN((origin.lng - destination.lng) * 0.0174532925 / 2), 2)
)) distance

FROM
tblZipUSA origin,
tblZipUSA destination,
(
SELECT DISTINCT
 destination1.zip_code,
 3956 * 2 * ASIN(SQRT(
  POWER(SIN((origin1.lat - destination1.lat) * 0.0174532925 / 2), 2) +
  COS(origin1.lat * 0.0174532925) *
  COS(destination1.lat * 0.0174532925) *
  POWER(SIN((origin1.lng - destination1.lng) * 0.0174532925 / 2), 2)
 )) distance

FROM
 tblZipUSA origin1,
 tblZipUSA destination1

WHERE
 origin1.zip_code = '27712'

ORDER BY
 distance, zip_code

LIMIT 10, 1
) zipDistance

WHERE
origin.zip_code = '27712'
AND
3956 * 2 * ASIN(SQRT(
 POWER(SIN((origin.lat - destination.lat) * 0.0174532925 / 2), 2) +
 COS(origin.lat * 0.0174532925) *
 COS(destination.lat * 0.0174532925) *
 POWER(SIN((origin.lng - destination.lng) * 0.0174532925 / 2), 2)
)) <= zipDistance.distance

ORDER BY
distance, zip_code

zip_code	distance
27712	0
27704	4.75326962990057
27705	5.34287681627437
27708	5.57383981806715
27503	5.58564401941337
27701	6.41245709075393
27706	6.58029004252293
27709	6.59449266010451
27702	6.61586916445661
27710	6.61586916445661
27711	6.61586916445661
27715	6.61586916445661
27717	6.61586916445661
27722	6.61586916445661

Showing rows 0 - 13 (14 total, Query took 5.6745 sec)

Spherical Law of Cosines

SELECT DISTINCT
destination.zip_code,
3956 * ACOS(
 SIN(origin.lat * 0.0174532925) * SIN(destination.lat * 0.0174532925) +
 COS(origin.lat * 0.0174532925) * COS(destination.lat * 0.0174532925) *
 COS((destination.lng - origin.lng) * 0.0174532925)
) AS distance

FROM
tblZipUSA origin,
tblZipUSA destination,
(
SELECT DISTINCT
 destination1.zip_code,
 3956 * ACOS(
  SIN(origin1.lat * 0.0174532925) * SIN(destination1.lat * 0.0174532925) +
  COS(origin1.lat * 0.0174532925) * COS(destination1.lat * 0.0174532925) *
  COS((destination1.lng - origin1.lng) * 0.0174532925)
 ) AS distance

FROM
 tblZipUSA origin1,
 tblZipUSA destination1

WHERE
 origin1.zip_code = '27712'

ORDER BY
 distance, zip_code

LIMIT 10, 1
) zipDistance
WHERE
origin.zip_code = '27712'
AND
3956 * ACOS(
 SIN(origin.lat * 0.0174532925) * SIN(destination.lat * 0.0174532925) +
 COS(origin.lat * 0.0174532925) * COS(destination.lat * 0.0174532925) *
 COS((destination.lng - origin.lng) * 0.0174532925)
)  <= zipDistance.distance

ORDER BY
distance, zip_code

zip_code	distance
27712	0
27704	4.75326963012033
27705	5.34287681651191
27708	5.57383981817397
27503	5.58564401945863
27701	6.41245709087189
27706	6.58029004273494
27709	6.59449266019943
27702	6.61586916464669
27710	6.61586916464669
27711	6.61586916464669
27715	6.61586916464669
27717	6.61586916464669
27722	6.61586916464669

Showing rows 0 - 13 (14 total, Query took 5.1353 sec)

So there you have it. In the second part of this series I will show how I use this same kind of logic to find the 10 closest locations from a database of addresses. Until then, enjoy!

On the Use of Captchas

2007-02-11T17:43:00.000-05:00

In an earlier post, PHP Form Signature I developed a bit of code to render form attacking bots harmless. On my own web sites I did expand the code to create a random hidden form field with the signature information, and on all the web sites I have implemented the signature methodology the obviously scripted form attacks were completely thwarted. At the same time, there is no usability impact since the mechanism works behind the scenes, completely hidden from the site visitor.

However, as I predicted, one of my forms was spammed by a bot that simply harvested the form signature and reposted it along with its spam payload. Looking at the contents of the spam I immediately recognized that it was coded to work in a blog. This was not a targeted attack by someone who ran across a form on the site and developed a script to try to exploit it. Instead, this was a bot that was programmed to roam the web looking for any form that has a text area in it, and post there hoping that it will show up as a comment in someone's blog. This kind of attacker doesn't care whether any particular form submission works or fails, knowing only that there are enough unprotected blogs and similar public forums that will instantly display anonymous posts.

This is exactly the kind of thing captchas were developed to prevent. However, as I have stated before, a captcha places a stumbling block in the way of innocent site visitors who truly do want to communicate. I have been on several sites that use a captcha on every form submission, which gets very annoying. The dilemma, then, is how one tells the difference between a robot and a human. After all, a robot can post HTTP header information to make it look like the POST is coming from an ordinary web browser.

The main difference, that I can tell, is that nearly none of my human correspondents place links in their form submissions, but every one of the spam posts I have recorded contain one or more links. So, I have added to my form validation sequence a check for a link reference in posts. If it detects one, it re-presents the form with a captcha and prompts the visitor to fill in the letters they see in the image. A robot will never see that page, of course, but neither will their form POST be completely processed. The human visitor, who has likely seen captchas before, may still be annoyed with this interruption, but are much more likely to fill in the field as requested in order to complete their correspondence. In this way, I, as the web developer, protect my server and e-mail box, while providing a normal user experience to most of the visitors who use my forms.

For the purposes of this example, I am using Ed Eliot's Visual and Audio PHP CAPTCHA Generation Class. The particular example I am giving also involves a form that posts to its same address ($_SERVER['SCRIPT_NAME']). This means that the model, or processing code, first checks to see if the request is a GET (automatically resulting in the form being presented) or a POST (which initiates the validation cycle).

At the beginning of the model section of code, then, before any processing takes place, I have the following lines of code:

$useCaptcha = 1; // flag to indicate captcha can be displayed if needed
$requireCaptcha = 0; // flag to indicate that a captcha condition has been met
require_once('php-captcha.inc.php'); // Ed Elliot's captcha class

Before I run the validation code, I fill in the $captchaTriggers array with any string that should cause a captcha to be displayed. The following will capture all links and images pasted in the form fields. You can add other rules, of course.

$captchaTriggers[] = 'http';
$captchaTriggers[] = '<img';

As part of my validation, I loop through the expected field names (I will likely cover details of my validation procedure in a later post, but this should give you the general idea). I check all the text fields for any of the captcha triggers. If one is found, then it's position is added to the $requireCaptcha variable.

if($_POST[$thisFieldName] > ''){
    for ($j=0; $j< sizeof($captchaTriggers); $j++) {
        $requireCaptcha += strpos(strtolower(' ' . $_POST[$thisFieldName]), strtolower($captchaTriggers[$j]));
    }
...

At the end of the validation phase, I have the following code. If any of the captcha trigger strings was found in the above code, the value of $requireCaptcha will be greater than zero, which is one of the two conditions that need to be met ($useCaptcha being the second). If the conditions are met, the code then checks to see if the $_POST['captchaCode'] has been set, and if it validates against Elliot's captcha class. The first pass through, the captchaCode variable would not be present, of course, which then leads to the errorMessages array being set. On subsequent passes through, invalid captcha codes would continue to fail while a valid one will complete the processing (unless there are other validation errors, of course).

if($requireCaptcha > 0 && isset($useCaptcha)){
    if(!isset($_POST['captchaCode'])) {
        $_POST['captchaCode'] = '';
    }
    if (!PhpCaptcha::Validate($_POST['captchaCode'])) {
        $errorMessages[] = "Please enter the letters you see in the graphic.";
        $errorFields[] = 'captchaCode';
    }
}

All of my validation checks use the $errorMessages array, so my check on whether or not there were any validation errors is the determining factor on whether or not to display the form for corrections. In the form, I do another check for $requireCaptcha to determine whether or not to display the captcha image and field.

<? if($requireCaptcha > 0) { ?>
        <div class="formRow">
            <div class="formLabel"><label for="captchaCode"><img src="/assets/images/visual-captcha.php" width="100" height="40" alt="Visual CAPTCHA" /></label></div>
            <div class="formField"><input type="text" id="captchaCode" name="captchaCode" value="" style="width:10em;" maxlength="10" accesskey="c" tabindex="5" /></div>
        </div>
<? } ?>

Since I have implemented this code, I have not received any robot generated spam. One possible improvement is to set a cookie in the client browser when a captcha challenge is successfully answered. The cookie would essentially say "this user has already proven that they are human, so you do not need to challenge them in the future."

Hope this helps you!

Dynamic Copyright Dates

2007-01-31T13:39:00.000-05:00

Copyright notices remind web site visitors that the content of a web page belongs to the web site owners and cannot be legally reproduced without permission (with some exceptions - search for other resources for more information). To fully protect the content of a site, most web site owners place the copyright notice in the footer of each page. It is normal to update the copyright year with the current year.

When most web sites are launched, they have only a few pages, and if they use includes, normally only have one footer file. It is quite easy to simply hard code the current year in the footer. However, as web sites grow larger, the maintenance can become more difficult if the copyright has been hard coded in every page or if more than one footer becomes necessary (for example if a web site uses multiple designs for different sections of the site or if different sections are written in different languages).

Rather than hard coding the year, it may be more efficient to dynamically generate the year in each footer. Below I have example snippets showing how this can be done in a few different languages. Feel free to add your own suggestions.

PHP

&copy; <? print date("Y"); ?>

ColdFusion

&copy; <cfoutput>#dateFormat(Now(), "yyyy")#</cfoutput>

JSP

<%@ page language="java" contentType="text/html" session="true" %>
<%@ taglib prefix="fmt" uri="http://java.sun.com/jsp/jstl/fmt" %>
<jsp:useBean id="now" class="java.util.Date" />
...
&copy; <fmt:formatDate value="${now}" pattern="yyyy" />

JavaScript

The following code can be used in static HTML pages or in place of any of the above. It requires no server side work, but it does mean that the browsing client must have JavaScript enabled.

&copy; <script type="text/javascript">
d=new Date();document.write(d.getFullYear());
</script>

PHP Form Signature

2006-12-29T19:31:00.000-05:00

In my previous post, I began developing the concept of creating a checksum for a form to help prevent spammers from abusing forms on web sites. A malicious spammer will first attempt to inject code into a form in an effort to send e-mails through the web site's host. Those attempts can usually be foiled by properly validating the form fields. However, a spammer may still inject their marketing materials into a form that appears to send an e-mail to someone, not really caring where that e-mail might wind up. If these e-mails are directed at a server administrator or the customer support staff, they can get very annoying, even if they are otherwise rendered harmless by the validation rules. The goal then is to try to ensure that a given form post if from a human being, and not from a robot. Visual and verbal "captchas" can be used, but that tends to annoy the human visitor, which we don't want to do.

The idea of using a checksum and a form timeout is one way a web developer might attempt to prevent a spammer from writing bots against forms. I am expanding on the idea of the checksum and will now call it a signature. This example is in PHP, and will work as-is, but I certainly encourage you to make your own variations.

In my previous post, I developed an example with two hidden fields which were used to validate the form post. In this example I do two things: 1) combine the timestamp and the hash to create one "signature" field, and 2) encrypt the timestamp to make it less obvious. These measures should make it more difficult for a robot to spoof posts to your forms. Note that with any encryption and hash scheme, it is possible to break the code, but the objective here is to make it so arduous that the spammer would rather look elsewhere for targets than your forms. My opinion is that it would probably be easier for someone to hack the server than this form validation technique, so one must not overlook hardening the rest of the server, of course.

Since you might use this code in several places, consider making it an include:

<?
// http://rc4crypt.devhome.org/
include_once 'cryptography.php';

function formSignatureCreate($formPassword){
    $formTime = array_sum(explode(' ', microtime()));
    $encryptedTime = encrypt($formTime, $formPassword);
    $hash = formSignatureHash($formTime, $formPassword . $encryptedTime);
    return $hash . $encryptedTime;
}

function formSignatureValidate($formSignature, $formPassword){
    $encryptedTime = substr($formSignature, 32);
    $formTime = decrypt($encryptedTime, $formPassword);
    $hash = formSignatureHash($formTime, $formPassword . $encryptedTime);
    if(!is_numeric($formTime)) {
        return false;
    }
    $currentTime = array_sum(explode(' ', microtime()));
    if($currentTime - $formTime > 1800) {
        return false;
    }
    if($_SERVER['HTTP_USER_AGENT'] == ''){
        return false;    
    }
    if($hash == substr($formSignature, 0, 32)) {
        return true;
    }
    return false;
}

function formSignatureHash($formTime, $formPassword){
    return md5($formTime . $_SERVER['REMOTE_ADDR'] . $_SERVER['HTTP_USER_AGENT'] . $_SERVER['SERVER_NAME'] . $formPassword);
}
?>

The encrypt and decrypt functions are wrappers I have placed around the rc4crypt code. You may use any encryption technique you want, of course, as I am only showing this as an example. These functions include a conversion of the encrypted string to base 64, so I do not have to deal with making the string web-friendly at this level.

For the time stamp ($formTime) , I have used the microtime() PHP function for simplicity, but you could use a formatted time as well, along with appropriate changes in the formSignatureValidate() function. This time stamp is encrypted with a password designated by the programmer. It is probably a best practice to have a unique password with each form, and since it is on the programming side, it can be a random sequence of characters.

The hash is generated in the formSignatureHash() function, which combines the following elements:

$formTime - The time the form was created, which allows us to validate that the form POST was submitted within a certain time frame.

REMOTE_ADDR - To tie the form post with the client's computer.

HTTP_USER_AGENT - To tie the form post with the client's browser.

SERVER_NAME - To tie the form to a given host server.

$formPassword - To tie the form POST to a given form.

Just to further confound any reverse-engineering, I combine the encrypted time with the given form password when the hash function is called.

Within the form, add a hidden field for the signature:

<? $signature = formSignatureCreate('thisFormPassword'); ?>
<input type="hidden" name="signature" value="<? print $signature; ?>" />

The way I have set this example up, a new signature would be created if other validation checks failed. This eliminates the need to validate the basic form (i.e., alpha-numeric) of the signature itself before printing it back into the form. We would not want someone attempting to inject code into an unvalidated signature variable!

In your validation section, add the signature check:

if(formSignatureValidate($signature, 'thisFormPassword')) {
    // other form validation
} else {
    // spawn error message
}

During validation, I make this my primary check. If the post does not pass the signature check, there is no reason to do any other validation, since it will always be rejected. In my error messaging system, I do not even display the input form again, forcing the visitor to click on a link to start the original form over. You may wish to restart a little more graciously with a blank form.

The formSignatureValidate() function first decrypts the time portion of the signature. In this example, the hash portion is always 32 characters long, so it is easy to pull the encrypted time off the end. Since I have used the microtime() function, it is easy to check to be sure the timestamp is 1) numeric and 2) within the 30 minute (1,800 second) time frame allowed.

I do an additional check on HTTP_USER_AGENT only because one of my sites was once attacked by a robot without a name. I don't bother checking during the signature creation phase, since it is really only important in the validation phase. If you know of specific rogue agents or IP addresses, those can be globally addressed in an .htaccess file or in the server configuration.

The final check is that the regenerated hash matches the hash at the beginning of the signature.

If all these checks pass, then it is considered a genuine post.

I have thought of a way to defeat this methodology. Someone could certainly develop a script to "screen scrape" the signature dynamically, but hopefully, a spammer will consider that too much effort. If the need arises, it is likely one could come up with a methodology to dynamically create the signature name as well. Time will tell. In the meantime, I hope this helps someone. I hope to develop a Java/JSP version of this in the near future.

Thwarting form spammers in ColdFusion

2006-11-30T17:23:00.000-05:00

As I have posted on before, spammers are more and more attempting to use form posting methods to send emails through unsuspecting web hosts. The most dangerous form, of course, are posts through which the spammer injects code that is run invisibly on the server. This approach can often be foiled by verifying form post data and properly escaping characters before they are displayed anywhere.

I have added validation to all the forms on the web site for the main company I work for, but one spammer is still attempting to use a form. However, the only destination is an administrative e-mail box, so the spammer has expended a lot of effort to effectively annoy only a few people. We have been receiving about 100 spam messages a day in the box, so further action was needed to stop this spammer and any other who might follow after.

One common solution has been the "captcha," a graphic with hard-to-read text that only the human brain can theoretically interpret and relay back to the server to prove the post did not originate from a robot. For our purposes, however, this approach is undesirable since it makes forms less usable. I wanted to come up with a verification process that is invisible to humans and would be (nearly?) impossible for a robot to reproduce.

My solution is to create a checksum value that would consist of data from the server, from the client, and from the form that could be matched up before and after the post to "prove" it to be authentic. Taking a page from our login guidelines, I thought it also appropriate to have forms time stamped, further hindering the likelihood that the checksum could be duplicated, and would certainly make it impossible for a "good" post to be reused at a later time.

This first solution is in ColdFusion since that is what this company uses the most right now. I plan to follow shortly with a PHP example followed by a Java/JSP example.

Creating the Checksum

The following code creates the checksum value. Any number of variables can be used to create the checksum, with the only requirement being that the values at the time of validation have to be the same as the time of posting.

This code uses the following four values:

dateTime - The time the form was created, which allows us to validate that the form POST was submitted within a certain time frame.

remoteAddress - The client machine's address, which allows us to validate that the form POST arrived from the same machine that requested the original form. Specifically, this would thwart spammers who use Trojans to hijack unprotected computers and make them spam zombies.

serverName - This is the web server's domain name in this example. If your servers are inside a firewall, and/or if you have multiple, load balanced servers with "sticky persistence" (i.e., a client is guaranteed to always communicate with one particular server during its session), you might use either the server's IP address or a global variable specific to a given server.

formName - On sites that I program, each form is given a unique name. Originally, this was for statistical usage purposes, but it can also come in handy here. This is essentially a "password" for the form data, preventing a given set of validation code from attempting to validate data from a different form.

Overall, this system attempt to guarantee that communication about one form is transferred between a known host and a known client within a known time span. Any detected violation prevents validation from proceeding. Hashing this data using a technique like MD5 creates a checksum that should be virtually impossible to reverse-engineer. Even though spammers are persistent, they are more likely to hack someone else's less protected site than tackle this problem.

<cfparam name="form.dateTime" default="#DateFormat(Now(),'yyyy-mm-dd')# #TimeFormat(Now(),'HH:mm:ss')#">
<cfset remoteAddress = cgi.REMOTE_ADDR>
<cfset serverName = cgi.SERVER_NAME>
<cfset formName = "formSomeName">
<cfset form.checksum = hash("#form.dateTime#~#formName#~#remoteAddress#~#serverName#")>

The next piece of code is added into the form. The only two pieces of information in clear text are the dateTime and checksum values (now that I think about it, these two values could actually be encrypted and given different names to further enhance security - I will most likely do that in further form work). All of the other variables live on the server, and the spammer would have to know all this, including your separators and the exact sequence of variables used, in order to forge the checksum value (not likely without inside help or an outright takeover of your server).

<input type="hidden" name="dateTime" value="#form.dateTime#" />

<input type="hidden" name="checksum" value="#form.checksum#" />

On the validation side, you want to add the following code at the top.

<cfif cgi.REQUEST_METHOD IS NOT "POST" OR NOT isDefined("form.checksum")>
    <cflocation url="/" addtoken="No">
</cfif>
<cfset remoteAddress = cgi.REMOTE_ADDR>
<cfset serverName = cgi.SERVER_NAME>
<cfset formName = "formSomeName">
<cfset checksumTest = hash("#form.dateTime#~#formName#~#remoteAddress#~#serverName#")> 
<cfif checksumTest IS NOT form.checksum>
        <cfset errorMessage = "The form data was corrupted. We are sorry, but we cannot send your request at this time.">
</cfif>
<cfif isDate(form.dateTime)>
    <cfif checksumTest IS form.checksum AND DateDiff("n", form.dateTime, now()) GT 30>
        <cfset errorMessage = "The form data has expired. We are sorry, but we cannot send your request at this time.">
    </cfif>
<cfelse>
    <cfset errorMessage = "The form date was corrupted. We are sorry, but we cannot send your request at this time.">
</cfif>

If the errorMessages is set, you most likely want to halt further validation to prevent any harmful code from affecting anything you have missed validating. At this point you are already aware that something is amiss! This should not be your only line of defense, however. You still want to validate every field to make sure it contains data of the right datatype, length, and is free from characters that could lead to other hacks.

CSS hack to differentiate between IE 6 and 7

2006-11-30T13:17:00.000-05:00

I discovered this quite by accident, and thought this might be helpful for those who find they need to set up different stylesheet rules for formatting in Internet Explorer 6 and IE 7. The crux of this solution is the CSS attribute selector (dentoned as square brackets [] in CSS). It turns out that IE6 does not understand attribute selectors, and, in fact, will ignore the entire associated rule (including rules with more than one selector).

To leverage this difference, two rules can be defined. A normal rule that will be rendered in IE6, and a duplicate rule containing an attribute selector that will be rendered in IE7. In the example below, I use the IE-specific "if" statement to show how you can further differentiate between IE and "other browsers." The solution below was tested in the Windows environment using IE6, IE7, Oprah 9, and Firefox 2.

If you use a different browser or environment, try the code out and let me know what you see.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>CSS hack to differentiate between IE 6 and 7</title>
    <meta name="author" content="John A. Marsh" />
    <meta name="copyright" content="&copy; 2006 ThreeLeaf.com" />
    <style type="text/css">
        P {
            background:green;
        }
    </style>
    <!--[if IE]>
        <style type="text/css">
            P {
                background:red;
            }
            P[],P {
                background:blue;
            }
        </style>
    <![endif]-->
</head>
<body>
    <p>Blue in IE 6, Red in IE 7, Green in non-IE browsers.</p>
</body>
</html>

JavaScript parent / child communication example

2006-10-30T15:52:00.000-05:00

In a test I was running, I needed a valid date of birth in mm/dd/yyyy format. Here is the solution I came up with:

${__javaScript(with(new Date()) dob = String(Math.floor(Math.random() * 12) + 101).substr(1) + '/' + String(Math.floor(Math.random() * 28) + 101).substr(1) + '/' + String(getFullYear() - Math.floor(Math.random() * 99) - 1), dateString)}

The plain JavaScript version is

with(new Date()) dob = String(Math.floor(Math.random() * 12) + 101).substr(1) + '/' + String(Math.floor(Math.random() * 28) + 101).substr(1) + '/' + String(getFullYear() - Math.floor(Math.random() * 99) -1 ); alert(dob);

The first part gives us a zero padded month from 01-12. The second part gives a zero padded day from 01-28 (I did not want to deal with leap years and end-of month guessing for this test). The final part subtracts 1 to 100 from the current year (guaranteeing no future dates in the current year). Quick and dirty -- Enjoy!

Struts Upgrade - Validation-rules.xml - Solution

2006-05-04T15:41:00.000-05:00

At the company where I work, we are upgrading our WebSphere servers from version 5 to 6. In the process, we are also updating our Java applications. Part of the application upgrades involve converting from Struts 1.1 to 1.2.8.

I have spent two days puzzled by the problem this caused with form bean validation. I was getting errors like this:

[5/3/06 9:46:11:643 EDT] 00000057 DynaValidator E org.apache.struts.validator.DynaValidatorForm validate org.apache.struts.validator.FieldChecks.validateMask(java.lang.Object, org.apache.commons.validator.ValidatorAction, org.apache.commons.validator.Field, org.apache.struts.action.ActionErrors, javax.servlet.http.HttpServletRequest)
org.apache.commons.validator.ValidatorException: org.apache.struts.validator.FieldChecks.validateMask(java.lang.Object, org.apache.commons.validator.ValidatorAction, org.apache.commons.validator.Field, org.apache.struts.action.ActionErrors, javax.servlet.http.HttpServletRequest)
at org.apache.commons.validator.ValidatorAction.loadValidationMethod(ValidatorAction.java:627)
at org.apache.commons.validator.ValidatorAction.executeValidationMethod(ValidatorAction.java:557)
at org.apache.commons.validator.Field.validateForRule(Field.java:827)
at org.apache.commons.validator.Field.validate(Field.java:906)

The error message gives no indication of what the actual problem is, and while I found several posts on the Internet of people who have had the problem, I have not seen any solutions yet. My best clue was from an article in the Apache Wiki.

The root cause is that the validation method's signature has changed. Another issue is that the old ActionError method has been depricated and replaced with ActionMessages. Rules in your validation-rules.xml file should now look like this:

<validator name="required"
classname="org.apache.struts.validator.FieldChecks"
method="validateRequired"
methodParams="java.lang.Object,
org.apache.commons.validator.ValidatorAction,
org.apache.commons.validator.Field,
org.apache.struts.action.ActionMessages,
org.apache.commons.validator.Validator,
javax.servlet.http.HttpServletRequest"
msg="errors.required"/>

Hope that helps someone

Exporting Outlook Contacts Folders - Solution

2006-04-24T19:42:00.000-05:00

A problem I have run across a couple of times, and have noticed that many others have is exporting Microsoft Outlook Contacts folders to a comma separated values (CSV) file. By default Outlook can only export one folder at a time. I have not found any software add-on yet that can do this either.

However, I have discovered by accident a way to do this using the Plaxo service. Plaxo comes as an optional install with the AOL Instant Messenger Triton application. I personally could not use Triton because it lacks Proxy settings or any way to change the communication port.

Anyway, when you use Plaxo, you can upload all your contacts to their web sites, and a fortunate side effect (depending on how seriously you take your privacy), it can help you discover your contacts' AIM addresses, and if they also are in Plaxo, will share whatever information the person allows to share.

One feature is that you can export your contact list. You are given the option to export all your contacts or any one folder of your choice. The download options are:

Microsoft Outlook (.CSV file)
Outlook Express (.CSV file)
Palm (.CSV file)
Netscape (.LDIF file)
Yahoo! (.CSV file)
vCard (.VCF file)

Hope this helps someone! Let me know if you have comments or questions. My main question is why can't Microsoft make this easier???

VBScript / Excel / SendKeys / Sleep - Solution

2006-03-08T17:53:00.000-05:00

This is a script I wrote to help me add IPTC information to over 1200 images. It contains several techniques you might be able to use in your own VBScripts.

' Based on a script to automate Excel with VBscript by Richard L. Mueller
' Demonstrates several additional functions including
' 1) Launching applications
' 2) using SendKeys to control application
' 3) useing Sleep to pause execution during the script
' 4) Arrays

' I wrote this script to help me update the IPTC data of images
' I had downloaded the data from MySQL and put it in an Excel spreadsheet
' Each line of the spreadsheet contains the file name and the desired IPTC information
' I leverage IrfanView to insert the IPTC information into each photograph
' This IPTC information is later read by the server using PHP when I upload the pictures to my photography web site

' You have a royalty-free right to use, modify, reproduce, and
' distribute this script file in any way you find useful, provided that
' you agree that the copyright owner above has no warranty, obligations,
' or liability for such use.

Option Explicit

Dim objExcel, strExcelPath, objSheet, intRow
dim fileName, pictureTitle, pictureDescription, keywords, categoryID, thisYear, originalDate, filesys

' Bind to Excel object.
On Error Resume Next
Set objExcel = CreateObject("Excel.Application")
If Err.Number <> 0 Then
  On Error GoTo 0
  WScript.Echo "Excel application not found."
  WScript.Quit
End If
On Error GoTo 0

strExcelPath = "C:\Temp\originals\tblPictures.xls"

' Open specified spreadsheet and select the first worksheet.
objExcel.WorkBooks.Open strExcelPath
Set objSheet = objExcel.ActiveWorkbook.Worksheets(1)

dim category(9)
category(1) = "Plants"
category(2) = "Animals"
category(3) = "Landscapes"
category(4) = "Textures"
category(5) = "Structures"
category(6) = "Miscellaneous"
category(7) = "John"
category(8) = "People"
category(9) = "Transportation"

dim WshShell
Set WshShell = WScript.CreateObject("WScript.Shell")
Set filesys = CreateObject("Scripting.FileSystemObject") 

'Get copyright symbol - VBScript does not have a way to do an alt-0169 combination

Dim charMap
charMap = WshShell.Run("CharMap")
WScript.Sleep 250
WshShell.AppActivate charMap
WScript.Sleep 250
WshShell.SendKeys "{TAB}{TAB}{RIGHT 3}{DOWN 5}~"
WScript.Sleep 250
WshShell.SendKeys "%F"
WScript.Sleep 250
WshShell.SendKeys "Times New Roman"
WScript.Sleep 250
WshShell.SendKeys "%C"
WScript.Sleep 250
WshShell.SendKeys "%{F4}"


' Iterate through the rows of the spreadsheet after the first, until the
' first blank entry in the first column. For each row, bind to the user
' specified in the first column and set attributes.
intRow = 2
Do While objSheet.Cells(intRow, 1).Value <> ""
    originalDate = Mid(objSheet.Cells(intRow, 1).Value, 4, 8)
    fileName = "C:\Temp\originals\" & objSheet.Cells(intRow, 1).Value
    if filesys.FileExists(fileName) Then 
        pictureTitle = objSheet.Cells(intRow, 2).Value
        categoryID = objSheet.Cells(intRow, 3).Value
        pictureDescription = objSheet.Cells(intRow, 4).Value
        keywords = objSheet.Cells(intRow, 5).Value
        ' On Error Resume Next
        
        
        thisYear = mid(originalDate, 1, 4)
        WshShell.Run ("C:\Programs\Graphics\I-View\i_view32.exe " & fileName)
        WScript.Sleep 250
        WshShell.AppActivate "IrfanView"
        WshShell.SendKeys "i", True
        WScript.Sleep 250
        WshShell.SendKeys "%i", True
        WScript.Sleep 250
        WshShell.SendKeys "^v " & thisYear & " John A. Marsh, all rights reserved" & "{TAB}", True
        WshShell.SendKeys replace(replace(pictureDescription, "(", "{(}"), ")", "{)}") & "{TAB}John A. Marsh{TAB}", True
        WshShell.SendKeys pictureTitle & "{TAB}", True
        WScript.Sleep 250
        'WshShell.SendKeys frmOptions.infoInstructions.Text, True
        WshShell.SendKeys "^{TAB}" & keywords, True
        WScript.Sleep 250
        WshShell.SendKeys "^{TAB}" & category(categoryID), True
        WScript.Sleep 250
        WshShell.SendKeys "^{TAB}John A. Marsh{TAB}Photographer / Owner{TAB}John A. Marsh{TAB}http://www.johnmarshphotography.com/", True
        WScript.Sleep 250
        WshShell.SendKeys "^{TAB}" & pictureTitle, True
        WScript.Sleep 250
        WshShell.SendKeys "{TAB}", True
        WshShell.SendKeys thisYear & "{TAB}" & Mid(originalDate, 5, 2) & "{TAB}" & Mid(originalDate, 7, 2) & "{TAB}{TAB}", True
        WScript.Sleep 250
        WshShell.SendKeys "Durham{TAB}North Carolina, NC{TAB}United States of America, USA{TAB}", True
        WshShell.SendKeys "{ENTER}", True
        WScript.Sleep 250
        WshShell.SendKeys "%o", True
        WScript.Sleep 250
        WshShell.SendKeys "%{F4}"
        WScript.Sleep 250
    end if
    intRow = intRow + 1
Loop

' Close workbook and quit Excel.
objExcel.ActiveWorkbook.Close
objExcel.Application.Quit

' Clean up.
Set objExcel = Nothing
Set objSheet = Nothing

WScript.Echo "Done"


Sub IrfanviewIPTC(fileName)

'Add IPTC data to image via SendKeys commands to IrfanView

'start application
'Dim appIrfanView
'appIrfanView = Shell("C:\Programs\Graphics\I-View\i_view32.exe " & fileName)

End Sub

JMeter Birthday Boundary Testing - Solution

2006-02-21T15:36:00.000-05:00

One of the applications I was testing today required a boundary test for people aged 65 years old. To do this, I set up two variables, one with a generated date exactly 65 years ago, and the other being 65 years old plus one day, thus giving an age of 64.

age64:

${__javaScript(with(new Date()) X = String(getMonth() + 101).substr(1) + "/" + String(getDate() + 101).substr(1) + "/" + String(getFullYear() - 65), dateString)}

age65:

${__javaScript(with(new Date()) X = String(getMonth() + 101).substr(1) + "/" + String(getDate() + 100).substr(1) + "/" + String(getFullYear() - 65), dateString)}

Note that this will fail on the last day of the month and the day before a leap day. But hey, you have been working hard all month ... take a day or two off! ;-)

Formatted Date in JMeter / JavaScript - Solution 2

2006-02-15T10:41:00.000-05:00

In another JMeter testing scenario, I wanted to set a user defined variable to a date string with the format "yyyymmddhhmmss". The solution below solved the problem of left-padding with zeros and getting all the formatting done in a single line.

${__javaScript(with(new Date()) X = getFullYear() + String(getMonth() + 101).substr(1) + String(getDate() + 100).substr(1) + String(getHours() + 100).substr(1) + String(getMinutes() + 100).substr(1) + String(getSeconds() + 100).substr(1), dateString)}

The uses of this kind of user variable include:

Knowing exactly when a test was run for auditing

Guaranteeing a unique string for things like new test usernames [note that I am writing regression tests which only run once. If you are running load tests, you will want to add String(getMilliseconds() + 1000).substr(1) to get unique string each pass]

Having a string you can key off of, for example to clean up database entries you have made

Boxes With Rounded Corners in CSS - Solution

2006-02-05T09:46:00.000-05:00

CSS Code

.textBox {
    background:#9cf url(/images/question-box.gif) left top no-repeat;
    /* Width attribute needed to fix behavior in Internet Explorer */
    /* width:100%; */
}
.textBox .right {
    background:url(/images/question-box-right.gif) right top no-repeat;
}
.textBox .bottom {
    background:url(/images/question-box-bottom.gif) left bottom no-repeat;
}
.textBox .corner {
    background:url(/images/question-box-corner.gif) right bottom no-repeat;
    padding:1em;
}

HTML Code

<div class="textBox"><div class="right"><div class="bottom"><div class="corner">
    <p>It is easy to create a resizable box with rounded corners using Cascading Style Sheets (CSS). Four nested &lt;div&gt; elements are required, each having a background image respectively set to the left-top corner (including the body), right edge, bottom edge, and right-bottom corner.</p>
</div></div></div></div>

See screenshots at http://www.threeleaf.com/freelance-work/demos/round-corners/ , along with some examples of why the workaround is necessary in Internet Exploer 6 and 7. My question: Why don't they fix that? If less funded browser companies like Firefox and Opera have come up with solutions, why can't Microsoft?

Formatted Date in JMeter / JavaScript - Solution

2006-02-05T09:41:00.000-05:00

I am using JMeter for regression testing, and needed a way to use the current date in mm/dd/yyyy format in a form post. Here is the solution I came up with using the JavaScript function:

${__javaScript(d = new Date; prefix=d.getMonth()<9?'0':'';X = prefix + String(d.getMonth()+1) + '/' + d.getDate() + '/' + d.getFullYear(), Dummy)}

Looks like date formatting in JavaScript is a perpetual problem, so I hope this is helpful.

Microsoft Word Clears the Clipboard - Solution

2006-02-05T09:36:00.000-05:00

I had been using Microsoft Word for years, when suddenly I had a strange problem where if I copied text to the clipboard, then opened Word, and then attempted to paste that nothing would happen . I have searched on the Web serveral times, but have not found a solution until now. Turns out I needed to disable a COM addin.

Thanks to Dopyiii (http://howtotroubleshoot.blogspot.com/#113035503245630446) for getting me on the right track.

His instructions for viewing COM addins:

Open Word and go to Tools > Customize > Commands tab
In the Categories section on the left, choose Tools
In the Commands column on the right, you'll see "COM Addins"
Grab this COM Addins button and drag it up and drop it off on your toolbar somewhere.
- It doesn't matter where - you'll delete it in a minute anyway.
- The quick way to delete this button (or to customize the toolbar) is to hold down Alt and click and drag the button into the document. When you let go, it'll disappear!
Close the Customize dialog
Now, click the new COM Addins button on your toolbar.
- Like the Templates and Addins dialog, you can enable or disable whatever you see here
- Remove the button if you like, or keep it there for nostalgic purposes.

I found that when I disabled Natural Voice Reader, I was able to paste after opening. Most people won't have NVR installed, of course, but this might also get you on the right track if it is a COM addin that is causing the problem. Let me know if this helps you!

Spammers can exploit unvalidated PHP forms

2006-02-05T09:26:00.000-05:00

This is a message I sent to the AOL postmaster on September 19, 2005:

I have recently figured out that an AOL member known as "homerragtime" has developed a method to exploit unvalidated form posts to send SPAM e-mail. For several months homerragtime was successful sending e-mail through my unvalidated forms, and I was perplexed at receiving so many rejected spams that appeared to have come from my own site.

About two months ago I finally began validating my forms, and began to find form postings like the following being captured by my validation process:

pictureID aorg@johnmarshphotography.com
comments aorg@johnmarshphotography.com
send aorg@johnmarshphotography.com Content-Type: multipart/mixed; boundary=\"===============1010877498==\" MIME-Version: 1.0 Subject: 90c4558b To: aorg@johnmarshphotography.com bcc: homerragtime@aol.com From: aorg@johnmarshphotography.com This is a multi-part message in MIME format. --===============1010877498== Content-Type: text/plain; charset=\"us-ascii\" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit wygqm --===============1010877498==--

The tip-off for me was seeing the BCC field. It appears homerragtime has a robot that crawls the web and posts these kinds of messages. If he gets one of the messages that he is BCCed on, he matches that with the number in his subject line, and then knows that the form can be exploited. I did a search for homerragtime, and see this same signature in many places.

Please research this further. I believe homerragtime is violating your company policy not to send spam, even though technically, he is sending the spam through other people's web sites.

A few more notes:

I have found this problem has occurred on all three of the web sites I maintain. They are all fixed now.
My sites are all in PHP, but it is quite possible that this exploit could be used in other languages.
Bottom line: validate your forms. This person was very aggressive at sending out spam through my sites, judging by the large number of rejected e-mails I was getting returned to me.

IE Toolbar Names Get Mixed Up

2006-02-05T09:21:00.000-05:00

I enjoy adding toolbars to Internet Explorer (currently have nine). I do not keep them all active all the time, of course. I only keep the Google toolbar open all the time and open others as needed. However, I have noticed a problem where when I right click on the toolbar area to open or close a toolbar, the names do not always correspond to the correct toolbar. For example, clicking on Yahoo! may open or close Google. Sometimes this can get very confusing when, say, four toolbars have switched names .

Anyone else seen this? I have not seen it documented on the Web, and I have not figured out a solution yet.

Formatted Date in JMeter / JavaScript - Solution

2006-01-31T20:27:00.000-05:00

I am using JMeter for regression testing, and needed a way to use the current date in mm/dd/yyyy format in a form post. Here is the solution I came up with using the JavaScript function:

${__javaScript(d = new Date; prefix=d.getMonth()<9?'0':'';X = prefix + String(d.getMonth()+1) + '/' + d.getDate() + '/' + d.getFullYear(), Dummy)}

Looks like date formatting in JavaScript is a perpetual problem, so I hope this is helpful.