2009
10.20

It has been a very long time since I posted anything so I thought I would share this small snippet.

Recently I had to extract some data which was stored as xml in a field within a mysql database table. To explain briefly, this data was actually saved as part of an XMPP (jabber) chat message by an openfire server. Openfire is a real time collaboration server which includes jabber functionality. To store the chat history Openfire records the chat message as an xml chunk in a field named body in a particular database table ofMucConversationLog. I needed to extract certain data that our system had communicated as a chat message through openfire, to do this I used a combination of regular expressions and string manipulation.

Ok, lets say I have messages that are either in format A or B as below:

<messageformata>
  <child1>some text</child1>
  <child2>some more text</child2>
</messageformata>

<messageformatb>
  <child1>some text</child1>
  <child2>some more text</child2>
</messageformatb>

Each row in the database can contain xml in the format of A or B and there are many rows. If I need to extract only the contents of the first child of message format a, I could use the following SQL code to do it (assuming I’ve created a temporary table called temptable to store these values):

INSERT INTO temptable (child1value)
SELECT
SUBSTRING(body, LOCATE(‘<child1>’, body)+8,
  LOCATE(‘</child1>’, body) LOCATE(‘<child1>’, body) 8)
FROM ofMucConversationLog WHERE body REGEXP ‘<messageformata>*’;

Unfortunately the REGEXP operator can only be used for testing true or false and so is probably only useful in a WHERE clause. There are no capturing options like in other languages (php, perl java etc.), however you can use it in combination with the substring and locate functions to get the job done, although it is admittedly tedious.

2009
09.23

After updating my debugger version of Adobe Flash Player to 10.0.32.18 I noticed that Mouse Events were no longer working correctly.

There appears to be a bug in the more recent flash player which appears to break the mouse events. It only appears to affect Macs, I am having the problem on both Safari 4.0.3 and FireFox 3.5.3. I made a test app below which adds a few mouse event listeners to the white box and informs you of the events being received. Try it out below and comment on your results.

Get Adobe Flash player

If you do not have a problem, do not upgrade your flash player or you will. This has been reported as a bug to Adobe here and here, more people are starting to notice it, hopefully Adobe will fix it soon.

Update 25/09/09

So far the results of people trying out the above test on a mac have been varied. The problem seems to be occuring in Snow Leopard 1.6, with most people FireFox being the only browser experiencing the bug. I have witnessed someone running Leopard with the same versions of FireFox and Safari as myself with no problems at all. Most people who are running Snow Leopard appear to only experience the bug in FireFox with safari working fine. However, as I mentioned above,  I am having the bug appear both in Safari and FireFox. I don’t know what is so different about my set up, apart from that I have installed the latest debugger version of the Flash Player and that my system is 32-bit only.

If you do try out the test above and want to report your results, please provide as much information as you can, what versions of browser you are using, what is your flash player version, what OS version, 32 or 64-bit etc? Some people think the problem is with FireFox but I am not so sure.

Update 06/10/09

Ok, I am pretty sure now that this problem is not to do with a flash player upgrade, it is to do with Snow Leopard!

Recently my hard disk died, I reinstalled Snow Leopard and then restored my files from a time machine backup, I did not restore my applications.

Upon reinstalling Flex Builder I noticed that my flash player was now at version 9, which gave me an opportunity to test if this bug was still happening and it is, both in Safari and FireFox for Flash Player 9,0,124,0 as well. This almost definitely means the problem is down to the operating system, either Apple needs to fix it or Adobe need to come up with a fix in Flash Player. I can not run my demo test app above using flash 9 as the demo is set to require 10, but in other test apps I am getting the same broken behaviour with mouse events being lost after loading the FileReference browser. Looks like we will need to put a warning for Snow Leopard users on our flash apps.

Update 22/11/09

Adobe Labs have recently released Flash Player 10.1, which has fixed the issue detailed above. I have tried out the new player, but while the problems above have been fixed, many others new problems are present. For example, it appears that key events are duplicated, every keystroke entered appears twice and filtering by file type does not appear to work with FileReference. I realise this is a pre-release version constantly being improved, so hopefully these will be fixed soon. Unfortunately there is no debugger version of the new player, which is quite strange since the people who are going most likely to try out a pre-release are developers who would require the debugging features.

So far it appears I am the only one that has experienced these problems with the latest player, I think this is somehow related to my system being 32 bit only. It appears that most of the development efforts into Flash on the Mac are on 64 bit support, which makes sense since Apple is shifting eveything to 64 bit. If you too try out the new beta flash player and experience other bugs please report them here and on the adobe bug reporting site. This new flash player update proves that an active Flash community can get flash bugs fixed quicker and help improve the flash platform for all. Thanks to all who commented on their issues experienced with the demo.

You can try out the new 10.1 player at the Adobe Labs site.

2009
09.14

Ok, in my previous post, which was a while ago now (I must make an effort to post more often), I wrote about installing FreeBSD and setting up your home router to give a fixed ip address for your server and forward some relevant ports on your router to make it accessible from outside of your local network. Up until now you should have a working FreeBSD server that you can ssh into using its local address. Your server so far does not do very much, lets install some software to make it a useful server.

Installing Apache

We are now going to install apache from the ports collection, remember I said the ports collection was important and that you definitely wanted to include it in the install process? The ports collection is a massive collection of free software, mostly source code which has been specifically ported to work on FreeBSD. You install a port by compiling the source code and copying the compiled binary files to the relevant places. Fortunately you don’t have to do this manually, the ports have a Makefile which will do most if not all the work for you.

Let’s start by installing the latest version of apache 2.2 which is located in /usr/ports/www/apache22, you will see there is a Makefile. You can install almost any port with a make file by just running make install clean, in some cases you can run make config to set any configuration options before actually compiling and installing, this applies here, run make config,  you will be presented with a window with some configuration options, you can change these if you want if you already know there are certain Apache requirements you want, otherwise accept the defaults, press the tab key so OK is selected and press enter. Then run make install clean, it will then go about downloading,  compiling the source code, install the files and any required dependencies. Make a cup of tea, this might take a while. Whenever you are installing software on your FreeBSD server it is always best to do it via the ports collection, firstly because you are getting the software that is most likely to work on your system. Secondly when you install a port, other ports will know about the installed software, thereby the installed port can be used by other ports as a software dependency because the system knows where those installed port programs should be. Thirdly the dependencies are all managed for you. If one port relies on some other software that is not installed, in most cases the make file will go about installing that port for you, in some cases, especially when installing an upgraded port you might run into problems you have to solve manually. For instance, when I tried upgrading apache, I ran into this problem:

Found libtool-1.5.26, but you need to upgrade to libtool>=2.2

The make process failed for apache because of this dependency that it couldn’t solve automatically, in this case I had to find out where the libtool port existed (/usr/ports/devel/libtool22) and install that first. You may run into this kind of problem along the way when administering your server but once you get used to installing ports you will find it a walk in the park,the internet is your friend if you run into any problems. The final reason why you should install from the ports collection is a combination of all three, it’s easier!

After apache has finished installing, you need to add a line in /etc/rc.conf so that apache will be started automatically when the computer first starts, as root edit rc.conf and add the following.

apache22_enable="YES"

Installing MySQL Server

Now lets install mysql server, there are a few different versions to choose from, I would recommend going with 5.1, you can try 6.0 if you want. Go to /usr/ports/databases/mysql51-server and type:

make install clean

This is going to take a very long time, longer than apache. When its finished, type rehash, this will refresh you shell path variables so that you will have access to the new binaries installed with mysql server.  After that, you can run the mysql install script which initialises mysql server. Just type mysql_install_db. To make mysql automatically start like we did for apache you need to the following to /etc/rc.conf:

mysql_enable="YES"

You will probably want to change the mysql root password, I think by default it doesn’t require a password and that is probably a bad thing. Just like in unix where the root account can do anything, so to in mysql the root account can do anything to any database on the system. To set a password use:

mysqladmin -u root password 'mypassword'

The quotes are important here. Now if you restart your machine you should have both apache and mysql running, but you don’t have to restart to start using them. In FreeBSD most software that runs as a long term process (i.e. a daemon) is stored in the rc.d directory and are avaible to start and stop. System daemons you will find in /etc/rc.d/, user installed programs such as mysql-server and apache are found in /usr/local/etc/rc.d/. These programs are actually known as rc scripts, many of them can be configured to start when the system starts up. They all usually have a common way to start and stop them, for instance you can do the following for mysql-server:

/usr/local/etc/rc.d/mysql-server start
/usr/local/etc/rc.d/mysql-server stop
/usr/local/etc/rc.d/mysql-server status

These are pretty self explanatory, the status option shows whether or not the daemon is running. These scripts start the processes making sure they run under the correct user (mysql runs under the user mysql, apache runs by default under the www user) and performing the necessary commands to start or stop the process. You can start and stop mysql server this way, but for apache there is a better command called apachectl. If you need to start apache you can run apachectl start, if you want to restart it (because you’ve made some config changes) its best to use apachectl graceful, this will keep any clients connected and do a safe restart.

Installing PHP 5

Now on to PHP and the php extensions, this port is located in /usr/ports/lang/php5. This has the same process, I’ll put all the steps in the one go:

make install clean
cd /usr/ports/lang/php5-extensions
make install clean

For the php extensions you can choose what you need, definitely enable mysql if you are planning to work with mysql and php.

Finally we have a few steps to perform on our apache config file to get php to work with it and to change which files our web server points to. As root edit the apache config file:

edit /usr/local/etc/apache22/httpd.conf

Change the document root to where ever you want, there are two places to change this. This is the default directory on the system that you want apache to serve on the web.

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/usr/local/www/apache22/data"

#
# This should be changed to whatever you set DocumentRoot to.
#
<Directory "/usr/local/www/apache22/data">

Ok, now we need to make sure php files are recognised by apache, do a search for DirectoryIndex in the file and add index.php before index.html as below. You can change this according to which file you want apache to give preference to, if you put index.php first then apache will look first for an index.php before index.html in a folder to serve.

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
<IfModule dir_module>
 DirectoryIndex index.php index.php3 index.html index.htm
</IfModule>

Now lets make sure that the php install process has added the php extension in apache. Make sure that this extension is in the exensions list, if not, add it.

LoadModule php5_module  libexec/apache22/libphp5.so

Nearly done, now add the two lines at the end of the file, which tell apache that php and phps files should be handled by php.

AddType application/x-httpd-php .php
AddType application/x-httpd-php-source .phps

That should be it, you now have a FAMP server! Now lets make sure you can get to it from outside.

Setting up Dynamic DNS

In order to get over the problem of your broadband provider assigning you ip addresses that can change at any time you use dynamic dns. This involves installing a program which periodically updates a server with your current IP address. I used dyndns for my home server. They are very good and charge nothing. The way it works is you can sign up for a free account with them, you get to pick a subdomain and one of their many domain names. For instance I chose chris.is-a-geek.net. You can then download their client program which periodically updates their servers with your current ip address, so that anyone that types in for instance, http://chris.is-a-geek.net will end up at my current ip address. Now luckily for me, my wireless router has a section in its admin interface for configuring dynamic dns and DynDNS.org (.org or .com its the same site) is a supported option. I just enabled the Dynamic DNS option on my router and entered my DynDNS account username and password. With this up and running it meant my router was now contactable via the domain chris.is-a-geek.net. Now together with the router port forwards that I mentioned in my previous post I now had a web server that is accessible via the internet, and you can too!

If your broadband router does not have an option for dynamic dns, don’t dispair! There is a port for a dynamic dns client on FreeBSD called ddclient, to get it working follow these steps:

cd /usr/ports/dns/ddclient
make install clean
cd /usr/local/etc
cp ddclient.conf.sample ddclient.conf
edit ddclient.conf

Do a search for a line that starts with use=web, remove the comment # to activate that line, it should say the following:

use=web, web=checkip.dyndns.org/, web-skip='IP Address'

Now uncomment the section for your dynamic dns provider and enter your account login details, domain name and save.
Remember I mentioned rc scripts, well this has one too, you can enable ddclient to start when the system starts by editing rc.conf and adding the following lines:

ddclient_enable="YES"
ddclient_flags="-daemon 600"

This tells ddclient to start on system start up and to update the dynamic dns provider with the current ip address every 600 seconds. You can change this value, don’t set it too low or your dynamic dns provider might get annoyed and block you, don’t set it too long either. As ddclient has an rc script you don’t have to wait to restart for it to work, just run as root:

/usr/local/etc/rc.d/ddclient.sh start

You should now have an accessible web server! You can point existing domains you own to your home server by adding a CNAME (canonical name) record in your domain dns records. For instance I have set up a CNAME dns record for the subdomain server.opencoder.co.uk to point to chris.is-a-geek.net (if you ping both of these you will see they resolve to the same ip address). This way you can point many domains to your home server and be able to run different websites for those domains by setting up apache virtual hosts (I won’t cover this here, maybe in a later post).

That’s it for now, there are many more things you can put on your server to make it even more useful. I may do more posts in the future covering a few of them. That’s enough of this for now, I need to get more techy and start writing about programming, anyway hope you found this info useful. By they way if you are interested in learning more about freebsd I’d highly recommend getting a copy of Absolute FreeBSD 2nd Edition by Michael W. Lucas and for instructions on using FreeBSD as a server, Building a server with FreeBSD 7 by Bryan J. Hong. There are also a few good posts about FreeBSD port management and installing FreeBSD with nice screenshots at my work colleague Luis’ blog http://www.lupomontero.com. We are working closely with Luis on an interesting project called StickyWorld, which involves posting stickynotes on work in realtime in a flash based ‘room’. I will no doubt be writing about that soon, until then, have fun with your server.

2009
08.26

For the first real post here (work, procrastination and thinking about what to write about ate up most of my time) I thought I’d write about what I did to set up a web server at home. I will try and go through all the important steps from the beginning and may split this into multiple posts if it gets a bit too big.

To follow this you will need a pc, connected by ethernet cable to a home broadband router that has DHCP enabled.

Why?

Ok, so why did I want to set up a web server at home in the first place? Well, it so happened that we (Slider Studio) had just aquired a dedicated server at work and in the near future we’d be running quite a lot of stuff from it. We also had set up a development server at work which we could mess about with and make sure everything worked as expected before we put it on our production server. Knowing that I had to work quite a bit with servers in the future, I thought it would be a good idea to get more experience and practice with them. So I looked around for a cheap pc, I wasn’t that really concerned about the spec, mainly that it wouldn’t take up too much space and wasn’t too noisy.

Hardware

At my local computer fair, I managed to get a really good deal on a second-hand compaq evo d510s base unit (only 48 quid!), pics below.

The specs weren’t that bad either:

  • 1.6 GHz processor
  • 256MB SD RAM
  • 20GB hard drive

It was in a pretty small compact case too, designed to be used flat on a desktop with perhaps a monitor on top. It later turned out to be pretty quiet, another bonus. I bought the machine and later upgraded the RAM to 1GB (2 x 512MB SD) and swapped the CD-ROM drive for a DVD drive (I needed a dvd drive to install FreeBSD from an install DVD). Finding the correct SD RAM for an old pc is not easy and certainly not as cheap as just buying any existing DDR memory. Most of what I could find on eBay was specifically for server rack machines, I had very strict requirements on the memory I needed, unbuffered, non-ECC, 266MHz  PC2100 etc. The machine had on-board graphics, which I didn’t really care about, the machine is only going to run a command line interface and I’m not doing any server side graphics intensive stuff.

OS

I decided to use FreeBSD 7.1 as the operating system because this was what our real production server was running and my aim was to replicate the set up at home. You can choose your own operating system, debian, ubuntu, whatever, just don’t bother installing a graphical windowing system, you shouldn’t need it. I would recommend FreeBSD, it’s very easy to install and the huge ports collection makes it super easy to install whatever software you need on the server. Plus, FreeBSD is used in many commercial servers and is very reliable. I will only be talking about setting up a FreeBSD server here.

Okay, on to actually installing now. First thing I did was get rid of the existing Windows XP installation, I did this by using a utilities CD to run the fdisk program and delete all partitions on the hard disk, create one primary partition and proceed to format that new partition. Actually it turns out I didn’t need to do this at all FreeBSD comes with its own fdisk-like program which does the same thing. So after partitioning and formatting the hard disk, I put in the FreeBSD 7.1 dvd (a burnt iso image you can get from here).

The first menu is about language and keyboard layout, just choose the correct one for you, the next screen is the installation type. Here you have a few options: Standard, Express, Custom + other options, I’d recommend just going with the standard install. It will then launch the installer, which is not the most user friendly installer, but don’t be scared! You should get a message talking about setting up the disk partition scheme, press OK for that, if you get a scary message talking about disk geometry don’t worry just press enter. When you get to the partitioning bit, just press A to use all of your disk space for FreeBSD with the default options for the partitions, then press q. Next you’ll get a menu about choosing the boot manager, just leave it on the default standard, we are just running FreeBSD on this system, nothing else. Next you get to choose how to partition the drive for the operating system folders. In FreeBSD the root folders such as /usr, /var and /bin are contained on separate partitions, you can create your own additional partitions if you want for instance if you wanted a partition mounted on /data for doing data backups, but I’d recommend just accepting the default by pressing A and then Q to quit. The default options give the most space to the /usr folder, which is useful because /usr is the location where the ports collection resides and the user accounts. Most of the space you will be using from your day to day server use will be in the /usr folder. The /var folder is commonly used for logs, /etc for system configuration and /bin for system binaries.

The next step is to choose the install source, I chose the install dvd, this later turned out not to work, my dvd drive or the disc may have been bad, blocks of data from the dvd could not be read. When the install failed I retried and this time chose to install from an FTP server. The next screen asks how much of FreeBSD to install, contrary to belief, you don’t need everything on here, just choose the User option, this gives you the basics and everything you need for your server. When it asks you if you want to install the ports collection say yes! You definitely want the ports collection, read on to find out why. After this you are taken back to the previous menu, press Exit, you will get one last chance to change your mind here, after that your drive is partitioned and formatted and the files copied.

When its finished you’ll get a chance to configure the network, say yes to this option (if you chose an ftp install you would have already gone through this). You will be given a list of network interfaces to choose from. Choose the ethernet one, my system is connected by ethernet directly to my broadband router. When it asks about using IPV6 configuration say no. When it asks if you want to try DHCP configuration you can say yes if your machine is connected to a DHCP enabled router. You will have to enter a hostname for your system a domain, IPv4 gateway and name server. Host is something you can choose to identify your server, it can be anything you like but don’t put any funny characters in. Domain should be a domain your server is part of. For me I didn’t have a domain yet, so I made one up, for the gateway and name server I entered the local  ip address of my router. For me this was 192.168.0.1, they are commonly this or 192.168.1.1.

After this you are asked about enabling services, for most of these you can say no, except for SSH which you definitely want to enable, I would recommend only enabling SSH, definitely don’t enable anonymous FTP. You then get asked to choose a time zone, whether to enable Linux mode, say no, set up a PS/2 mouse (if you have one), I ignored this. You are then asked about installing any additional packages, don’t install any at the moment we will do this later.

You will get a chance to create a user account say yes to this, creating a user account separate from the root account is a good idea, you should only use the root account only when you have to and to ssh into your server in the first place will require a user account. In the new user interface, enter the desired username under Login ID, leave the UID as it is, leave the group as it is, by default the group should be the same as the username. Enter your password carefully, you can only do it once here. Enter your full name, under member groups you can put in wheel in order for this user to be able to switch to root or do administrative tasks on the server. The home directory should be fine as it is, for the shell you may want to change the default /bin/sh to /bin/tcsh, there isn’t much difference that I’ve noticed apart from with tcsh you get the tab complete functionality when logged in via ssh, but not with sh. Once you have created the user you can set the root password for the system, you definitely do want to set a root password, enter this twice. Finally you go back to the main menu, select exit and your system will reboot and you should have a fully functioning FreeBSD system. Nearly done here, you just need to make sure the hostname you set for the machine resolves to itself, first lets edit the rc.conf file found in /etc. Log in to you server either at the machine or on another machine in your local network using ssh. Switch to the root user/super user by typing su then:

ee /etc/rc.conf

this file should have some key-value pairs, don’t worry too much about this file now, just make sure there is a key value pair

hostname="myhostname"

obviously replace myhostname with the hostname you set. Next edit the hosts file /etc/hosts and make sure that after 127.0.0.1 you have the localhost and your hostname, e.g.:

::1            localhost
127.0.0.1      localhost myhostname

Now if you run hostname from the command line you should get your hostname echoed back to you.

Router Set Up

In order to get remote access to your newly installed server you will need to perform some router configuration steps. Log into your router configuration, usually this done by opening a browser and going to 192.168.0.1 or 192.168.1.1, you will need the router login details. The first thing to do is set up a static ip address for your new server. For my router (which is a netgear sky broadband router), I did this by going to the Advanced tab -> LAN IP Setup a list of machines connected to the router was shown, I selected the one that looked like my server, entered the static ip address that I wanted (192.168.0.5) and gave the device a name (server). Your router configuration may be different to this and you may have to look at the manual for how to do it.

Next thing to do is set up some port forwards so we can access the server from outside of our home network (the internet). First port forward to set up is for SSH access. For my router this was under Security -> Firewall rules, I was given a list of services to choose from, I chose SSH (TCP/UDP:22), under the action I chose Allow Always,  for the destination IP address I entered the static ip address I had reserved for the server, 192.168.0.5. Under WAN users I selected Any and chose to log this always. If your router does not have ssh listed as a service, you can enter a custom service, just make sure the service is for port 22 using tcp and udp and that the destination address for this is the static IP you set for your server.

Finally you can do the same for http service (port 80) and https service (port 143). We haven’t got a web server yet so the http(s) port forwards won’t be doing anything yet, but it’s better to prepare for it now. You can test whether your port forwards are working by trying to get remote access to your server, first find out the public ip address your router has (this is the address that’s visible to the internet) then try and gain ssh access using your routers public IP address as the server location and the new user account you created on your server. For ssh on windows you can use the free PuTTY ssh client. If you could log in successfully then you now have remote-accessible server, it just can’t do much yet! If you couldn’t log in, try connecting to the server using the local ip address e.g. 192.168.0.5, if that works but the remote address does not, then you have a problem in you router port forwarding. Obviously most people’s public internet ip address will change from time to time, unless you have paid extra for a static one. So how our we supposed to get to our server when we’re away from home? More to the point how are we supposed to run a web site or more from a place  which IP address is changing all the time? The answer to this is Dynamic DNS, which I will leave until the next post, this is already way too long for a blog post, I’ll have to learn to keep things shorter for next time.

So look out for the next post where I’ll cover setting up dynamic dns for your server, installing Apache, MySQL and PHP (making your server a FAMP server)! Peace.

2009
08.05

Blog started

Hey everyone, it is finally here, set your bookmarks and subscribe to the feeds, tell your friends and wake your neighbours. Ok, well only if they happen to be coder geeks and only if they can be bothered with that sort of thing. Quite excited by this, my first blog. See the about page for reasons why I decided on starting one.

I’m still not decided with the theme so might play about with it. Expect some posts soon on technical related issues and things that might be useful for the code enthusiast, or maybe interesting to those that wonder what geeks gets up to (where are these interested people? please point them out to me).

Why is it after doing something like this, you tend to get stuck thinking about what to write about?