network security appliance

one of the ideas i have for testing the capabilities of the sun fire t2000 server is to build a network security appliance. this would involve utilisation of the zones feature in solaris 10. this allows full virtualisation of sevrers on one machine, along with allocation of resources, such as network ports or physical cpus, to that instance. each instance is a separate, full version of the solaris operating environment, and is indistinguishable from a complete physical machine to any processes running in it. this makes it ideal for separating security critical functions like firewalls and intrusion detection systems from each other, while still allowing them to run on one server.


network security appliance diagram

in the above diagram you can see that i intend to virtualise six instances, four firewalls, one ids sensor and a management system. the t2000 has four gigabit ethernet ports, which would be assigned to each of the four networks, while inter-machine communication and intrusion detection would all be done using the virtual internal network. it will be simple to allocate at least one cpu to each machine, and the resource pooling commands available will allow some of the virtual machines to have extra cpus allocated, perhaps the internet-facing firewall and the ids sensor. the sun bigadmin site has some useful resources on zones, including the original usenix paper describing the implementation.

the software to be installed will all be open-source packages, most of which are de-facto industry standards. i will use squid as the outgoing web proxy, snort as the network ids and use native solaris networking for the firewall rules. i will need to determine a suitable console to administer the firewalls, but sguil will be used for ids command and control.

a useful test would be to determine the line-speed that the firewalls and the ids are capable of handling without dropping any packets, and the number of simultaneous outgoing connections that the proxy will allow, while the dmz also has web traffic being sent to it from the internet. i believe that the t2000 should be a good platform for this kind of appliance, due to the one-box approach that can be taken, while not having to compromise on cpu power available. i intend to set this environment up over the next week and produce some performace figures to try and validate this claim.

sun, sparc, security, t2000, network, solaris

hardening solaris ten

my first job on booting solaris 10 on hexagon, my sun fire t2000 system, was to harden the operating system. i want to make sure that the system is not going to be offering extraneous services to passers-by on the internet (even though everything but ssh will be firewalled off.) this will have the added bonus of stopping cpu being used unnecessarily. an initial portscan using the ubiquitous nmap utility revealed the following open ports:

robot$ nmap -p1-65535 -A hexagon

Interesting ports on hexagon (10.10.10.6):
(The 65514 ports scanned but not shown below are in state: closed)
PORT      STATE SERVICE          VERSION
21/tcp    open  ftp              Solaris ftpd
22/tcp    open  ssh              SunSSH 1.1 (protocol 2.0)
23/tcp    open  telnet
25/tcp    open  smtp             Sendmail 8.13.4+Sun/8.13.3
111/tcp   open  rpcbind           2-4 (rpc #100000)
513/tcp   open  login            Berkeley remote login service
514/tcp   open  tcpwrapped
587/tcp   open  smtp             Sendmail 8.13.4+Sun/8.13.3
898/tcp   open  http             Solaris management console server
4045/tcp  open  nlockmgr          1-4 (rpc #100021)
5987/tcp  open  unknown
5988/tcp  open  unknown
7100/tcp  open  font-service     Sun Solaris fs.auto
9010/tcp  open  tcpwrapped
22273/tcp open  wnn6?
32771/tcp open  status            1 (rpc #100024)
32772/tcp open  fmproduct         1 (rpc #1073741824)
32773/tcp open  rusersd           2-3 (rpc #100002)
32774/tcp open  ttdbserverd       1 (rpc #100083)
32777/tcp open  sometimes-rpc17?
32778/tcp open  dmispd            1 (rpc #300598)
32779/tcp open  snmpXdmid         1 (rpc #100249)
32795/tcp open  unknown
Service Info: OSs: Solaris, Unix, SunOS

Nmap finished: 1 IP address (1 host up) scanned in 1778.040 seconds

as you can see, there's a lot of unwanted access provided there. at least ssh is there by default, but we also have telnet and rlogin, the X11 font server, as well as all those RPC services... solaris 10 manages services with the svcxsxxx utilities, and i will use them to turn off telnetd and rlogin, as follows:

root@hexagon# svcadm disable svc:/network/telnet
root@hexagon# svcadm disable svc:/network/login:rlogin
root@hexagon# svcadm disable svc:/application/x11/xfs
root@hexagon# svcadm disable svc:/network/ftp:default
root@hexagon# svcadm disable svc:/network/rpc/rusers
root@hexagon# svcadm disable svc:/network/rpc/rstat
root@hexagon# svcadm disable svc:/network/shell:default

and we can also get rid of the packages that provide telnetd itself, since it is inherently insecure, and there is always potential access via telnet to the console over the ALOM network port. first, check what packages need removed, then remove them with the pkgrm utility:

root@hexagon# pkginfo | grep -i telnet
    SUNWtnetr      Telnet Server Daemon (Root)
    SUNWtnetc      Telnet Command (client)
    SUNWtnetd      Telnet Server Daemon (Usr)
root@hexagon# pkgrm SUNWtnetr SUNWtnetd

hopefully, this has given you an idea of how to do all this manually. i also downloaded the sun solaris security toolkit which has a lot of useful scripts to automate the hardening process. the file you require is SUNWjass-4.2.0.pkg.tar.Z and is only 600KB. you need to be registered with sun to download anything, but this is useful anyway, since you need an id to get the latest security patches, and also to access the sun update connection site.

root@hexagon# uncompress SUNWjass-4.2.0.pkg.tar.Z
root@hexagon# tar xf SUNWjass-4.2.0.pkg.tar
root@hexagon# pkgadd -d . SUNWjass       

Processing package instance <SUNWjass> from </root/install>

Solaris Security Toolkit 4.2.0(Solaris) 4.2.0
Copyright 2005 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Using </opt> as the package base directory.
## Processing package information.
## Processing system information.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

Installing Solaris Security Toolkit 4.2.0 as <SUNWjass>

## Installing part 1 of 1.
/opt/SUNWjass/Audit/disable-IIim.aud
/opt/SUNWjass/Audit/disable-ab2.aud
...etc...
/opt/SUNWjass/rules.SAMPLE
/opt/SUNWjass/sysidcfg <symbolic link>
[ verifying class <none> ]

Installation of <SUNWjass> was successful.

you'll notice that the package was loaded from /root/install. this is because i modify the root user to have a different home directory. often / is a shared home directory for other system accounts and daemon user ids, and it's never a good ide to have the root .profile and other dot-files there. moving home is relatively easy though:

root@hexagon# usermod -d /root root
root@hexagon# mkdir /root
root@hexagon# chmod 700 /root
root@hexagon# mv /.[a-zA-Z0-9]* /root/

and even all the existing dot-files get copied across. the jaas security toolkit has a large number of configurable options, which are documented in the reference manual. the security blueprints collection is also a good place to look for information. to secure your solaris system with the jaas tool, execute the hardening driver using the following command:

root@hexagon# /opt/SUNWjass/bin/jass-execute -d hardening.driver |
 tee jaas-hardening.log

which will lock down your system, and place a log of all output into jaas-hardening.txt. once this has completed, reboot to implement the changes. when you next login you will see that a security warning has been added:

|-----------------------------------------------------------------|
| This system is for the use of authorized users only.            |
| Individuals using this computer system without authority, or in |
| excess of their authority, are subject to having all of their   |
| activities on this system monitored and recorded by system      |
| personnel.                                                      |
|                                                                 |
| In the course of monitoring individuals improperly using this   |
| system, or in the course of system maintenance, the activities  |
| of authorized users may also be monitored.                      |
|                                                                 |
| Anyone using this system expressly consents to such monitoring  |
| and is advised that if such monitoring reveals possible         |
| evidence of criminal activity, system personnel may provide the |
| evidence of such monitoring to law enforcement officials.       |
|-----------------------------------------------------------------|

which should be modified to comply with local legal requirements. also, the passwords for any existing users will have been expired, and a much more stringent policy is now in place. if an nmap scan is run against the system now, you will see that most ports are closed, except ssh and one other that will be investigated later:

$ nmap -p 1-65535 -A hexagon | tee entries/hexagon.ports.03.txt

Interesting ports on hexagon (10.10.10.6):
(The 65533 ports scanned but not shown below are in state: closed)
PORT      STATE SERVICE VERSION
22/tcp    open  ssh     SunSSH 1.1 (protocol 2.0)
22273/tcp open  wnn6?

Nmap finished: 1 IP address (1 host up) scanned in 1814.355 seconds

next, i installed some useful extra utilities, from the sun freeware site. this has lots of GNU software compiled for SPARC on solaris 10, although you may want to check out sun's offerings from the solaris 10 companion dvd. sudo is a good replavement for the standard su program, curl makes retrieval of files from the internet simple and lsof lists all files that a process has . once you have downloaded them, installstallation follows the same basic pattern. this is how i installed the SMClsof and SFWsudo packages:

root@hexagon# gunzip lsof-4.76-sol10-sparc-local.gz
root@hexagon# ln -s /usr/sfw /usr/local
root@hexagon# pkgadd -d ./lsof-4.76-sol10-sparc-local

The following packages are available:
  1  SMClsof     lsof
                 (sparc) 4.76

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: all

Processing package instance <SMClsof>
from </root/install/lsof-4.76-sol10-sparc-local>

lsof(sparc) 4.76
Vic Abell
Using </usr/local> as the package base directory.
## Processing package information.
## Processing system information.
   2 package pathnames are already properly installed.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.

The following files are already installed on the system and are being
used by another package:
* /usr/local/doc
* /usr/local/man

* - conflict with a file which does not belong to any package.

Do you want to install these conflicting files [y,n,?,q] n

Do you want to continue with the installation of <SMClsof> [y,n,?] y
## Checking for setuid/setgid programs.

The following files are being installed with setuid and/or setgid
permissions:
 /usr/local/bin/lsof <setgid bin>

Do you want to install these as setuid/setgid files [y,n,?,q] y
## Processing package information.
## Processing system information.

Installing lsof as <SMClsof>

## Installing part 1 of 1.
/usr/local/bin/lsof
/usr/local/doc &glt;conflicting pathname not installed>
/usr/local/doc/lsof/00.README.FIRST
/usr/local/doc/lsof/00CREDITS
/usr/local/doc/lsof/00DCACHE
/usr/local/doc/lsof/00DIALECTS
/usr/local/doc/lsof/00DIST
/usr/local/doc/lsof/00FAQ
/usr/local/doc/lsof/00LSOF-L
/usr/local/doc/lsof/00MANIFEST
/usr/local/doc/lsof/00PORTING
/usr/local/doc/lsof/00QUICKSTART
/usr/local/doc/lsof/00README
/usr/local/doc/lsof/00TEST
/usr/local/doc/lsof/00XCONFIG
/usr/local/doc/lsof/lsof.man
/usr/local/man <conflicting pathname not installed>
/usr/local/man/man8/lsof.8
[ verifying class <none> ]

Installation of <SMClsof> was successful
root@hexagon# bzip2 -d SFWsudo.bz2
root@hexagon# ln -s /usr/sfw /opt/sfw
root@hexagon# pkgadd -d ./SFWsudo

The following packages are available:
  1  SFWsudo     Sudo - superuser do
                 (sparc) 1.6.8.5,REV=2005.01.05.17.49

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: all

Processing package instance <SFWsudo> from </root/install/SFWsudo>

Sudo - superuser do(sparc) 1.6.8.5,REV=2005.01.05.17.49

...and so on...

Installation of <SFWsudo> was successful.
root@hexagon# chmod u+s /usr/sfw/bin/sudo

important - notice that the sudo executable was not setuid root, and we had to change this after installation, to make it work! after this, the installs for any other packages will be very similar - use the above processes as a guide, just remember to check where in the filesystem things get installed, and either create symlinks or allow it as required. once sudo has been installed, you need to authorise users to have access to the root user. use the visudo command as root, and setup the sudoers file. i added the following line, which gives everyone in the sysadmin group root access:

%sysadmin       ALL=(ALL)       ALL

since we have lsof installed now, we can check what was holding the other port (22273/tcp) open. in the nmap output it is listed as wnn6? but we can check what process is using it with lsof and find as follows:

root@hexagon# lsof | grep -i wnn
jserver_m  741   root    3u  IPv4 0x600036e0100  0t0  TCP *:wnn6 (BOUND)
jserver_m  741   root    4u  IPv6 0x6000377f940  0t0  TCP *:wnn6 (LISTEN)
root@hexagon# find / -name "jserver_m" -print
/usr/lib/locale/ja/wnn/jserver_m

and we can see that it is a program called jserver_m that seems to have something to do with the japanese locale input method. i don't live in japan, or speak japanese, so this can be safely turned off. checking with svcs shows that it is started by init and can be disabled as follows:

root@hexagon# svcs | grep -i wnn
legacy_run     20:50:28 lrc:/etc/rc2_d/S94Wnn6
root@hexagon# /etc/init.d/Wnn6 stop
root@hexagon# mv /etc/rc2.d/S94Wnn6 /etc/rc2.d/_S94Wnn6.DISABLED

hopefully this has given you an idea of how best to approach hardening a solaris 10 system. the one thing not covered here is patching, which i will describe in another post. depending on how tightly you want things locked down initially, you can either manually turn off certain services or you can use sun's provided toolkit, and edit the default settings. this gives you a lot of flexibility, but i now have a system i feel safe about connecting to my router and assigning an IP address...

security, hardening, sun, solaris, sparc

plasma flickring

at the weekend, my friend alex came round, and we toook some amazing pictures of one of those little plasma-ball toys that barry has in his front room. they were taken with alex's digital nikon slr, and at shutter speeds varying from 1/10 to 2 seconds. they really look beautiful, and with a tripod and some preparation they could probably even be improved...! the thumbnails below link to the photo-set on flickr:

plasma, photos, pretty, ball

trees in a forest

this is a really nice photograph, taken by my friend rob. he's an amateur photographer, looking to make it professionally. he took this with a hasselblad medium format camera. it was shot on kodak film, and scanned with an imacon 949. adobe photoshop was used to adjust the gamma curves because the film used is hard to scan, although no other retouching or other editing was needed.

trees in a forest - copyright © 2006 robert phillips

for more technical details, or if you would like to get a print made, or see his other work, email him directly..

trees, photograph, forest, colour

niagara falls

i am currently running one of sun's new sun fire t2000 servers, as part of an evaluation and review programme. sun are allowing qualified individuals and companies to try the system for sixty (60) days before buying one. this can only be a good thing for sun, since it ought to get people who would not normally specify sun kit to have a look. as far as cost goes, the server retails at around usd 10K depending on configuration. this is actually pretty cheap for a system of this quality and power. think of it this way - how much would a 24-way PC system cost? and in a 2U form factor chassis as well?

the specification of my machine is listed as medium and has a niagara T1 processor. this is a six core ultra SPARC T1 cpu, each core of which runs at 1 GHz and has four 'coolthread' execution units, giving a total of twenty four (24) processors. the machine also has 8 gigabytes of ram and two 73 gigabyte serial attached SCSI (SAS) drives. the technology is known as coolthreads because the system only consumes seventy five watts (75W) at full load. this isn't the highest spec, either - it is possible to have t2000 configurations with eight T1 cores, running at 1.2 GHz, giving 32 coolthreads. note that there is no floating point processor in the T1, although the system does have a cryptographic accelerator built in.

so, when the box arrived yesterday, i unpacked it immediately...

the shipment consisted of: the server itself, a rack-mounting kit, two utp patch cords and two uk power cords. there is no real documentation shipped, just a small warranty booklet and a set of packing notes. it does, however, have a whole set of neat little diagrams on the top of the chassis explaining common maintenance tasks, like replacing fans or installing more ram modules. i downloaded the documentation from sun, and read the install guide first.

the sun fire t2000 'coolthreads' server

it turns out that on power being supplied initially, it will go into the lights-out management mode (ALOM) and stay there. this must be accessed via the serial management console, which is the only active port on the box as shipped.

to get into it i needed an RJ45 (sun) to DB9F (PC) null-modem cable. unfortunately, nothing of the kind came in the box. still, a trip to maplins and application of a soldering iron and a few hours later (yes, i'm that bad at soldering, and i had help!) a cable was ready. it turns out that this is what is commonly called a 'cisco console rollover cable' and they are almost always available on ebay. i enabled the network management port and booted into the open firmware ok prompt, and then into solaris. sun don't configure Solaris for you, although they do install it, however the configuration is as simple as setting IP address parameters and location details, so it didn't take long until i had a working, networked server. annd here is the proof:

adk@hexagon$ prtdiag
System Configuration:  Sun Microsystems  sun4v Sun Fire T200
System clock frequency: 200 MHz
Memory size: 8184 Megabytes

========================= CPUs =========================
                            CPU                 CPU  
Location     CPU   Freq     Implementation      Mask 
------------ ----- -------- ------------------- -----
MB/CMP0/P0       0 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P1       1 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P2       2 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P3       3 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P4       4 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P5       5 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P6       6 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P7       7 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P8       8 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P9       9 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P10     10 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P11     11 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P12     12 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P13     13 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P14     14 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P15     15 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P16     16 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P17     17 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P18     18 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P19     19 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P20     20 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P21     21 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P22     22 1000 MHz  SUNW,UltraSPARC-T1         
MB/CMP0/P23     23 1000 MHz  SUNW,UltraSPARC-T1         

as you can see, i have 24 cpus ready to do whatever i want. i have been waiting for this technology ever since i first read about it in 2004 so i have some pretty good ideas about how to utilise it. although sun seem to be promoting this as an enterprise class web and web application server (which it will perform fine as.) but, i think that it would make an excellent network security appliance. i intend to run an array of security applications and services to see how well it copes. this would include network IDS and IPS sensors and management servers, which can take advantage of the virtualisation technology available in solaris 10. also, some kind of all-in-one firewall and dmz protection device with deep packet inspection and virtualised ingress and egress firewalls, using all four gigabit ethernet ports. it also has crypto acceleration, which is ideal for several other security tasks.

my first job is to secure and harden the stock solaris 10 install that it came with. i have to turn off all the default services, such as telnetd and rlogin, only then will i be able to start thinking about allowing hexagon onto the internet, and doing something useful. more on this as i run the tests and build the environments to test them...

sun, server, sparc, review, install, security

wake me up!

i'm notoriously bad at getting up, and since i pawned my last ipod (don't ask...) i don't have any way of waking up to a selection of music in the morning. i decided this wouldn't do at all, and i was getting tired of the awful ring-tone my phone used as its alarm noise. now, my macintosh has itunes, and a set of nice loud speakers. howevber, i'm running OS X 10.3.9, so no automater for me. i do have a working knowledge of applescript though, and itunes is chock-full of applescript-awareness, so i decided to write a little script to work as an alarm clock.

the script is really very simple. to use it, open up the script editor, which lives in the /Applications/AppleScript/ folder. enter the following text, exactly as shown:

-- iWake
-- 
-- slowly raise itunes volume to wake up.
-- call from batch processing every morning
-- 
-- author: andrew kennedy
-- created: 02 march 2006 09:54
-- 
-- copyright (c) 2006 nevada systems

property wake : 30 -- time in minutes to wake up in
property vol : 100 -- volume setting
property step : 1 -- delay in seconds between volume changes

on run
 -- get current volume
 tell application "iTunes"
  set vol to sound volume
  set sound volume to 0
 end tell
 
 -- set wake time in minutes
 set step to (wake * 60) / vol
 
 -- start itunes
 tell application "iTunes"
  play
 end tell
 
 -- slowly raise the volume
 repeat with counter from 0 to vol by 1
  delay step
  tell application "iTunes"
   set sound volume to counter
  end tell
 end repeat
end run

you can test this script out by choosing compile and then, making sure iTunes is running but paused, press run at the top of the script editor window. what should happen is that the itunes volume will be reset to zero, and then start playing, while slowly raising the volume back to the original level over the next ten minutes. assuming you see the volume drop and iTunes start, you can (rather than wait ten minutes) just quit the iWake application, but make sure it's not the script editor.

now, save the whole thing as an application somewhere useful. i chose to put mine in ~/bin/iWake.app which is in my path. you will need a way to run your alarm clock, at whatever time in the morning you want woken up. i use the Unix cron daemon, which is part of the BSD package installation on OS X. go to the terminal, and run the command crontab -e and you will be presented with a blank editor window, probably vi. now, add the following text (to the end of the file if there is anything there already) and save it.

##
# adk cron entries
# modified 2006/03/04 -5h00
##
# wake up with itunes in the morning at 09h00
00 09 * * 1-5 osascript /Path/to/your/saved/iWake
# and at 10h30 weekends
30 10 * * 0,6 osascript /Path/to/your/saved/iWake

make sure that you replace the path after osascript with wherever you saved the script. if you're not sure how to use vi, paste the text into another editor and modify it there, then copy the whole thing to the clipboard and just press the following keys in order G o [command]-V [escape] :wq [enter] when vi appears, and you should be told crontab: installing new crontab when finished. for help on changing the times and days look at the crontab(5) man page.

you now get woken up gently by your favourite music. which is good. as an exercise for the reader, i would suggest modifying the script to choose a particular playlist, since this version just resumes whatever was playing when itunes was paused. next time, a sleep timer that drops the volume...

iTunes, AppleScript, cron, alarm, wake

MAKE.MONEY.FAST

i'm a (pretty good?) web programmer by now, and i've amassed a bunch of skillz over my time served in the industry. i started working on perl cgi scripts in 1992 when nobody knew what the web was, let alone perl or cgi. i then moved on to java in 1995 when it arrived, and tried my hand at javascript in the first browsers that supported it. i remember creating my first site with frames and javascript rollovers back then, because the client wanted something modern and flashy. i coded a netscape server api library that accessed a database over odbc on a dec alpha running nt 3.51 when such things were cutting edge. you get the idea, i've been doing web applications for a long time now, over ten years anyway.

so, as you might have seen from the development environment posts, i'm also fluent in the latest java and java enterprise apis, and the associated libraries. things like struts, jsps, servlets, mysql/jdbc and so on. i'm even able to turn my hand to php when the need arises. what, though, can i do with this hard-won knowledge? i believe the official MBA term would be to monetize my skill set or something like that. oh, and i don't want to have to go to an office for 0900, wear a suit or interact with people on a daily basis.

the answer turns out to be freelance bit-work. i've been working from home on a web application for a friend's small business, and i thought there must be a lot of people in his situation. he wanted some custom software, but couldn't pay the tens of thousands of pounds for a full-scale j2ee solution, with oracle, weblogic, and all that kind of heavyweight server-based junk. i started looking at the rent-a-coder site to see what it was like, and found that there were plenty of likely candidates.

the other three sites that i'm registered on as a developer are: get a coder, get a freelancer and script-lance. so far, i have two projects active on rent-a-coder, and several projects that have reached the shortlist stage on get a coder. i'm not sure about the other two sites, but i'm bidding on them at the moment and will see what comes up. one problem i've noticed is that a lot of indian, chinese and eastern european developers and teams of developers use these sites. they seem to be able to put in extremely low prices, which is the benefit of offshore outsourcing, i guess, but makes it hard for me to be competitive and profitable.

anyway, i'm working on two projects right now, and the buyers seem really friendly and have been pretty clear about what they wanted, and accepted my advice about what was and wasn't possible. the sites encourage communication using their message boards/forums so that disputes and arbitration when a disagreement occurs about scope can be resolved by referring to what each party actually said. when a project starts, your IM alias is given out, and this makes simple back and forth chat easier, but i have been summarising any decisions on the site so there is a permanent record. one thing to watch out for is people trying to get their college assignments and homework done on the cheap. i worked for my degree (well, a little) and i have big problems with someone trying to submit work that they just paid someone else to do.

another type of project to avoid is the 'clone' request. this usually involves a (probably teenager) asking for a clone of amazon/ebay/myspace/insert-commercial-site-here and offering the princely sum of, say, fifty dollars. i wonder if they can even comprehend the amount of money that a company like amazon spends on their e-commerce web service? avoid! with regard to payment for real projects, the site will escrow the full bid amount from the buyer at the start. this means i am sure i'll get paid at the end (assuming i deliver an acceptable product...)

something i'd like to have clarified is the position on open source libraries. i believe that the GNU LGPL (lesser GNU public license) allows me to sell software that links to libraries with that license. also, since i provide source code for my app and unmodified binaries (which have freely downloadable source anyway) for libraries i use, i interpret the apache ASL (apache source license 2.0) as allowing me to distribute, say, jakarta commons httpclient with my application. i'd REALLY like to get this properly clarified.

the ebay sdk and api download seems to get away with distributing apache axis (the web services api, more on this and the ebay development platform some other time) and a whole load of jakarta commons libraries, so they must think it's legal, and in this case, i'm going to redistribute the ebay sdk anyway, so the licensing issues are theirs. it's a grey area though, and i need to be careful. i don't want richard stallman coming round to my house with a bunch of the FSF hired goons!

i'm really pleased with my discovery of rent-a-coder work, and i'm pretty sure it's a good way for me to make money doing something i enjoy. so far, admittedly, i haven't won any bids on the other sites so i'll just have to keep bidding, but at least i'm going to be productive. i'll update with some more information about my interactions with get a coder, get a freelance and script-lance when they happen, and also report on the outcome of my current projects when i'm finished.

freelance, devenv, programming, rentacoder, software

fully organised

you may notice on my list of interesting links i have a pointer to a to-do list service. this is the rather splendid remember the milk site.

the site is one of the best to-do list managers on the web that i have come across. i have also looked at several others before making my mind up, and they each had their benefits. i mentioned tadalists from 37signals, and there is also voo2do. although these last two services provide task listings and management, both in a convenient AJAX-ified interface, where they fall down is integration.

anyway, i will go over the features and problems with all three services and explain my reasoning behind the choice of RTM as my to-do list manager in the following set of reviews:

reviews of free web-based to-do list managers

i will start with Ta-da Lists, which is the first service i tried. it gives you a password protected custom URL to access your lists, as a sub-domain off their site. for instance, mine would be http://grkvlt.tadalist.com/ which is easy to remember. the list functionality is fairly rudimentary, only allowing you to add tasks and attach notes. there is no concept of a repeating task or a due date. this functionality is available in thir commercial offering, backpack however. the user interface is simplicity itself, since there are very few functions, and the use of AJAX in-place editing makes adding and removing list items and lists incredibly easy. overall, the site design is very polished and attractive, and renders properly in all browsers that i tested it with.

the front page of your account shows you all the lists you have created, and these are marked with a bullet whose size indicates how many incomplete tasks are remaining in that list. from here you can add a new list or edit or delete existing lists. clicking on a list name takes you to the list of tasks. each task has an html form checkbox next to it, which will when checked, mark that item as completed immediately. an optional description can be added to a list to explain its purpose, and it is also possible to alter the order items are displayed, however that is as far as prioritising of individual items gets.

it is possible to share lists with other people, both publicly (read-only) and privately (full access). the person you are sharing with does not need a tadalists account, a special url is mailed to them for access. the other publishing options available are to email the list contents to your registered email address (only) and to export your list as an RSS feed.

to be fair, there are commercial offerings from 37signals, such as backapckit and bascamp hq, and they even have free trial versions available. these are more project management tools, and even the simplest, backpackit, just adds the ability to create notes and upload images, and doesn't improve on the todo list functions. basecamp is a much more complex product, and this actually gets in the way of managing simple lists, due to its concepts of milestones that must be assigned before being used.

although short, the FAQ list is a good resource, and there is also a page listing comments made about the application bu ysers, which should give you ideas on how to use it.

Ta-da Lists

provides a very simple task list manager, which is suitable for static checklists, very easy to use, static lists only, limited export capability, no reminders

three out of five cats preferred tadalists

Remember The Milk is a much more polished application in terms of features, however it should be noted that it is apparently still a 'BETA' product. this, in and of itself, doesn't seem to mean much these days. google are forever producing applications like GMail and Groups that never seem to leave beta status but are perfectly usable. what it does suggest is that the creators are activley fixing bugs and seeking to improve and add new features, which can only be a good thing. the site was started in august v2004, and is run by a team of just three people.

your lists are accessed via a subdirectory from the main site, after you have logged in, and if you choose the 'remember me' cookie, you will always be redirected to your tasks overview page when you access the site. for those that find typing the whole url a pain, it can also be accessed as http://rtmilk.com/. once at the overview page, it is possible to see all your tasks, or view them in groups. the groups can simply be labels for a list of tasks or a 'smart' list, based on a saved search you specify. these are all shown in a familiar tabbed interface. once a tab is selected, tasks can then be added to that list. initially, no options are set for the new tasks, but it is possible to set these up later. one small issue is that selecting individual tasks by clicking on them is confusing - the checkboxes for other tasks are not deselected, and it is sometimes unclear which selected task will be edited when you only want to alter details on one. the rest of the user interface is fairly easy to understand, and accessibility features like keyboard shortcuts (c.f. GMail) have been added.)

sample rememberthemilk screenshot

a task can have as many tags as required added to it, to act as category filters or keywords when searching or grouping. it can also have a due date (which can be entered in english ('next tuesday', 'tomorrow') or as a standard date and time. tasks can also be set to repeat, using similar text options, and an estimate of the time required can be added. selected tasks may be prioritised from nothing through three different levels, and these are shown as coloured highlights. in addition tasks with date information will be formatted differently if they are due or overdue. you may also add text notes to a task with additional information, apparently without limit!

the smart list feature works the same as smart playlists in iTunes, and allows you to create a virtual task list of entries that fulfil some set of criteria, specified as a boolean expression. this is where tags are useful for assigning tasks to groups. usefully, on the overview page there is a 'tag soup' box, showing all tags attached to your tasks, sized according to number of tasks, priority and due data, giving you a quick visual reminder of what is most important.

the lists can also be shared, both publicly and with chosen subscribers, similar to tadalists, and also exported as an Atom feed. the feed can contain just a particular list, or all your tasks, and all feeds are auto-discoverable by most modern browsers. the most useful (for me) publishing feature is the iCal integration. this exports either all tasks, or just a particular list, as a webcal subscription for use by iCal on OS X. this means that your remember the milk tasks are then available on iSync and to any other application on your local computer that can access iCal. the shared feeds, webcals and your task pages are all protected by secure password authentication, unless you choose to make some of them fully publicly available.

as i mentioned earlier, due dates and times can be set for tasks, and RTM uses this information to send out reminders. it will send both a daily summary of your tasks, and also remind you just before the start time. this is all fully configurable in the settings page. additionally, reminders can be sent using IM (various clients and networks) and by text message (SMS) to mobile phones, including t-mobile in the UK. this raises the usefulness of the service by an order of magnitude. finally, if you don't like this technological nonsense, there is a weekly planner page, designed to be printed out with empty boxes to tick, and lots os space to scribble on with low-tech pencils! also, if you need to set up a new task while away from the web, you can email a message to a personalised address, and it will be added toy your tasks inbox for later editing.

really, the fact that there's a huge number of help pages rather than one short FAQ page (tadalists, i'm looking at you) should tell you all you need to know about the feature set, and if you look at the blog or forums you can find out what ever new features are planned and anything else you need to know. the current feature list is incredibly comprehensive, and should cover everything you want to do, however.

Remember The Milk

provides an excellent, full-featured reminders and tasks service, many useful notification options and methods, useful smart grouping and searching functionality, accessible using iCal, simple to use with extensive help, extras like sms/im notification and contact lists.

five out of five cats preferred rememberthemilk

voo2do is the newest (i think) of the services i am reviewing, and was created by the authour for his personal use before making it publicly available. i discovered it while trying to find a better list service than tadalist, and was attracted to it because of the ajax type interface and the fact that it is project based, much like basecamp, but free and apparently simpler to use. the feature list has more in common with rememberthemilk, and tasks can be assigned priorities, due dates and have time estimates attached. however, they can also be allocated to a project. the project tasks can be managed by updating them with hours elapsed and setting current versus original duration estimates. these are used to track total time elapsed and remaining in a whole project composed of sub-tasks.

the task views are mainly arranged aroun due dates, and consist of historical (what you've done) and deadlines (what needs done this week). views, showing the tasks that fall due today, tomorrow etc. for each individual project are also available. the initial screen aggregates all of this information into a dashboard, showing all the high-priority tasks their dates with overdue items highlighted.

similarly to rememberthemilk, it is possible to email the server with a new task and have it added to a list. however, the area where voo2do excels over the other applications here is its programmability. it is the only application featured that allows REST web service access to all of its functions. this means that there are now a number of third party applications available that can poll your account for upcoming due tasks, or provide a windows GUI to add new tasks. the REST method of accessing a web service is also simple enough that it is possible to write basic scripts in shell, perl or python to access your task list. this makes voo2do potentially the most powerful application reviewed, although this is dependant on the userbase writing these useful add-ons and scripts, and we will need to wait and see whether this happens?

something that appears to be missing is automatic notifications, using any out-of-band mechanism. the only way to check task and project status is to log in, which is not always possible. the status diusplays are also very cluttered, with too many form controls visible, and lacking the smooth design that we have grown used to with most modern web applications. this mkay be a nescessity brought on by the complexity of the project and time tracking features, but a cleaner printout would have been easy to achieve with alternate stylesheets, rather than printing out the unnecessary screen chrome.

voo2do also makes use of keyboard shortcuts, and is very much a web 2.0 application, with new features being added all the time. there is a blog published with information and notes written by the author, and documentation on the API for programmers. however, i found it quite hard to get started using, mainly because new accounts are not initially populated with projects and it is unclear what the best way of implementing personal tasks are, since i don't usually assign shopping or tidying to a project. i can see this being useful to contracters or freelance workers who need to manage their time, and are willing to invest in setting things up. it should also appeal to anyone who wants to expand the features or integrate it into an existying project setup using the API.

voo2doo

a project focussed task manager with excellent REST API based extensibility. initially cumbersome and aesthetically lacking, but has some very good time tracking and management features not found anywhere else, but no reminder facilities.

three out of five cats preferred voo2do

HassleMe

after all these complex systems, sometimes the simplicity of a one-page web form is refreshing. HassleMe was first written as a script by the creators of mysociety to encourage them to update their development blog. this then mutated into a public site offering the same features. essentially it is a scheduled email reminder service, although not too scheduled, in case it gets too easy to ignore.

simply fill in the self explanatory (although there is a FAQ) form, and you will be reminded 'approximately' after every specified interval by an email with whatever you wanted hassled to do. the software took two afternoons to write, but does its one job perfectly, so there's little elso to say. it doesn't even require registration!

HassleMe

reminders via email at semi-unpredictable intervals

two out of five cats preferred hassleme

summary

in my opinion, rememberthemilk is the best of these applications, particularly if you are a Macintosh user, due to the excellent iCal integration. the options for task settings and searching are well thought out, and the system is easy to use and understand, as well as being the most professional looking of the four (sorry, HassleMe!). since all of these systems are free, though, i would suggest you grab an account at each of them, have a play and see what suits you best.

• sign up for Tadalist
• sign up for Remember The Milk
• sign up for voo2do
• sign up for HassleMe

and, if you think i'm wrong about rememberthemilk, or there are any other similar applications that i've missed, just add a comment and let me know...

ajax, web, tasks, gtd, lists, reviews

development environment (two)

(note: this is part two of a series on developing java enterprise applications. part one deals with the tools used to set up your development environments.)

the current project i'm working on requires a number of external libraries for its functionality. i use open source projects from apache's jakarta and the jakarta commons projects for many of these features, but there is a lot of good, quality OSS code around. the application i'm working on is a web based service, running under tomcat. i will go through each of the packages it depends on and explain why i chose and use them.

struts

apache struts is an MVC (model-view-controller pattern) framework. the current release version is 1.2.8 and it has been around for long enough that there are many good books available.

the struts project is currently undergoing a revision, and will be split into two frameworks, action (much like the current version) and shale (which will integrate with sun's JSF presentation layer) as well. currently, the 1.2.8 release is likely to remain useful and stable, and i would not recommend using any of the forked versions until they have more support and documentation avaialable.

in terms of books, the two i would recommend are both part of the o'reilly java series: jakarta struts cookbook by bill siggelkow and programming jakarta struts by chuck cavaness. the documentation and javadocs are available on the jakarta site.

struts implements the front controller pattern, and is mainly concerned with the C of MVC. the model and presentation layers are up to you. the controller is based on the concept of actions, which are invoked from the struts action servlet, usually by any URL ending in .do and are defined declaratively. in fact, the whole of the struts configuration is done this way, from a single (or multiple, if required) XML file. this specified the forms, their contents and the actions that process them, as well as any exceptions that may occur, message resources, plug-ins and forwards. forwards are struts mechanism for separating the presentation layer from the controller, and allow you to give a name to a a JSP or HTML file, that an action will then forward to. this way, you can change what the forward points to without re-coding your actions.

in general, struts is a very nice framework, particularly since you can mix and match any other model and presentation frameworks you want. many, many commercial sites use struts - look for the tell-tale .do extensions on pages that perform the business functions in the site. for example, Blogger uses this, as do Sony Austrailia and Yell to name just three.

torque

this was a difficult choice. i knew that i didn't want anything to do with real J2EE and the mindless tedium that is developing EJBs (ie writing tons of boilerplate template code for home, remote, whatever interfaces) and anyway i don't want to be running a heavyweight app-server. Resin (a lightwight J2EE container from Caucho Software) and JBoss (the de facto standard OSS app-server) are overkill. i didn't contemplate BEA WebLogic, IBM WebSphere, Oracle AS etc. for even a moment - they are all hugely expensive too, as well as requiring more cpu power than i or my client have available.

basically, torque is a spin-off from the turbine project, and now forms part of the Apache DB project, which is an attempt to produce a completely open-source database system by the apache organisation. this includes DDL (the data description language), an XML dialect for specifying relational database schemas. the torque system parses a DDL definitionj file, and produces a java object model. the model is provided as a set of SQL files used to create the database tables in whichever dialect of SQL you require, and a set of java source files implementing the model. the generation of the model and SQL is easily autyomated, and a build.xml ant file is provided that can be imported into your projects build process.

the objects created follow the JavaBeans standards, and come in two types. there are the actual model objects, which have get/set methods for the database columns in whatever table the object is represented by, and peer objects, which implement finder methods. the peers use torque criteria objects to specify which bits of the model to retrieve from the database and populate the model with, and can handle complex 1:n, 1:1 and m:n relationships. it is also possible to feed straight SQL to a criteria. the model and peer objects are defined as super classes, and a set of empty sub-classes extending them are made available for you to use in your project, and add any convenience methods you require.

generation and runtime configuration are accomplished by .properties files, which specify the dialect of SQL used, and a connection to a database. at runtime, the only thing required is to call Torque.init(properties-file) somewhere in your applications startup code. from then on, you can use the model and peer objects as desired, loading your data from the peer's finder methods, and calling .save() on the model objects to update them.

i particularly like torque because the only configuration required is the DDL, and the database connection details at runtime. the object model code is auto-generated, so you only need to write out your database definition once. for example, hibernate needs the database SQL to be written, the hibernate mapping configuration to be written in an XML file, and the model beans to be coded, which involves repeating the same list of fields three times, with different implementations, and hoping they all match. plus if you were to alter a table, those three files all need modified, with torque, just run the generator again after updating the DDL. spring just adds another configuration file representing the same data... (yes, i know things like xdoclet go some way towards solving this, but it's still complex, and just something else to learn.

i'm not sure how scalable torque is, but for a small webapp with a few users, i think it should be fine, and i believe turbine was designed as an enterprise class system, so you should be safe.

log4j

this is the best-of-breed standard java enterprise logging library. it is part of the Apache Logging project, available here. there are actually various implementations, in .NET, C++, Perl and several others, all using the same compatible API and output. configuring log4j can be problematic, since may j2ee containers and application servers also use it, and class-loader issues can prevent the library being loaded correctly. the onjava site has a good discussion of the problem, and it's solutions.

(note: this is not to be confused with commons logging, which is a super-api for many logging services. logging using log4j under the commons-logging framework will be covered later in this series.)

quartz

i need to have jobs that run, producing reports and emailing messages and information to users and administrators on a regular basis. quartz is developed by OpenSymphony and has been open sourced recently. it provides job/task scheduling for enterprise java systems, with facilities for triggers and schedules, including a cron-style trigger. this uses the same configuration options as the standard Unix crontab file, with a few extra selectors for ranges and last-n of month, third-m of month etc. i haven't added the scheduler to my current project, but this project appears to include everything i could ever need. there's also part of a soon to be published book, written by struts and j2ee developer and author chuck cavaness available for free download.

poi

an important part of a webapp for businesses is reporting. it's all very well to present data in HTML tables, but this is not easily exported into other applications for analysis. sometimes formatting your data as a comma-separated or tab-separated file is sufficient, but i prefer to output Microsoft Excel workbooks directly, and so do most users. the apache POI project provides apis to manipulate and generate DOC, XLS and PPT files. i used the HSSF (horrible spreadsheet format, by the way) routines to produce excel sheets with tabular data returned by an SQL query on a database. the code is extremely simple if all you want is numbers and text in boxes, and often that's all that the users need - they can pretty it up themselves. the api allows a lot more than this, though, and in my current project i'd like to take this a step further and start formatting the cells appropriately, and maybe even insert formulas and colours/lines to produce a sheet that can be printed out or used immediately.

related to the generation of MS Office documents is Adobe PDF file generation. there are a couple of options here that i have yet to fully explore. the iText library seems fairly basic, but there is a lot of good documentation and tutorials. for a more complex solution, XSL formatting objects may be the way to go. the Apache FOP project implements this.

the next installment of this series will deal with some more libraries, including many of the Jakarta Commons sub-projects. hopefully you found this information useful, but let me know if you think i missed anything or got something wrong...

programming, devenv, struts, torque, log4j, quartz, poi, java, j2ee

migration - new and improved

i've decided to change my weblog provider. the blog is now hosted by Blogger, a Google subsidiary, instead of livejournal. Blogger has a bit more of a 'serious' reputation than LJ (it doesn't keep asking about how i'm feeling, like an insecure teenage girl in the middle of a relationship crisis, for instance!) the customisation options also seem a lot better, and it has full API access via an Atom XML-RPC / HTTP interface.

the templates for page design are also much more flexible - the CSS / HTML is fully editable and i'm still tweaking the layout options. so far i'm very impressed with the functionality and customisation options. i expect to produce the same high quality random text that i always have, but with better fonts and prettier design.update your bookmarks for consonants to this URL: http://grkvlt.blogspot.com

i'll be continuing the development environment article here in the next few days...