2014
08.06

Weekend Hackathon

Last weekend I entered a hackathon, the last of 4 hackathons as part of the Salesforce Summer of Hacks. This was not my first hackathon experience, but it was the first that lasted more than a day. Overall, it was a good if tiring experience and our team came in third place claiming the $1,500 prize. Our 5 member team Text Relay worked on a mobile app that allows deaf or hard of hearing (HOH) people to have a two-way phone conversation, where the call receiver hears speech and the deaf/HOH person sees text in the app. Here’s a link to the submissions page for the hackathon, where you can read about the submitted apps. If you want the blow-by-blow version, read on, apologies but it’s a bit of a mammoth post.

 

Salesforce Summer of Hacks

 

The Hackathon Experience

I arrived to start the experience on a Friday evening, greeted with friendly faces, food and drink and lots of goodies. The event was held at a venue in Hoxton Square, fairly close to the silicon roundabout, it was booked for the whole weekend and open at all times for those who wanted to code through the night (spoiler, I didn’t make it through a whole night coding). After everyone had arrived and put on their assigned t-shirts (Code Breakers, Pixel Pushers and Secret Weapons), the event was announced and proceedings explained. Then came the idea pitches, I can’t remember the exact number of idea pitches, but it was in the region of about 42. This was fairly strict on timing, about a minute each, so I believe all the pitches were done in the space of an hour. I made some notes of the interesting ideas, spotted some nice ideas, but also heard ones I knew wouldn’t nearly be possible to do in the space of a hackathon. One of the pitches I heard was from Andrew, a deaf man who pitched, with the assistance of a speaking BSL interpreter, an idea for a mobile app to help deaf people making calls to hearing people. When I first heard that I thought I must have misheard, but kept it in my mind and listened to the other pitches.

When pitches were over we had to vote for the ideas, each of us had 3 poker chips that we had to give away to the ideas we liked the best. I gave away 2 tokens to a couple of enthusiastic people and also gave one to Andrew. Voting over the winning ideas were announced, this meant out of the 42 or so pitches, 26 were chosen as winning ideas, now we had to form teams for the winning ideas. I spoke to a few people and had to turn down a few offers, I think by then I had made up my mind to join Andrew’s team. After all I had some experience working with phones using Twilio in a previous hackathon, so thought I may be of some use. I joined the team as the 4th member and we got round to talk about the idea. We were required to work on a mobile app that would operate within Salesforce1 the mobile app for Salesforce, from the documentation I read on the day, that meant a native app was out of the question, which was fine by me, not having done any native apps before. I explained that I had worked with Twilio before and knew that it could do recording and transcribing of calls, so that it might be possible, but I wasn’t sure if an active call could be maintained while a message was being transcribed. At this point it was getting quite late, so the others called it a night.

I stayed a bit later that night, I had brought my sleeping bag along, although in truth didn’t need to. I spent some of the night researching what was possible with Twilio and then going through a tutorial on the Salesforce1 platform. By 2:30am I was pretty tired and realised I wasn’t going to get any sleep there. Since it was a summer night and the building was quite hot, there were many fans and AC units brought in and it was quite noisy. I decided to head home, catch a few hours sleep and head back early the next morning. I hired one of the cycle hire bikes and rode to Bow, getting some late night food at McDonalds before getting an uber cab home (that was my first time user Uber and sure wont’ be the last).

After around 3 hours sleep at home I got up, ready and headed back to the hackathon for Saturday morning. We had a deadline of 10am to register our team and create the initial app submission. We could change this as much as we liked but the initial submission needed to be in by 10am. Following this there was a git commit challenge, which actually just meant that you had to a git commit within a specific hour I believe. We were given access to a github repo to do our commits. At this point we still hadn’t really progressed the app at all, having gone through most of the tutorial the previous night, I was beginning to understand Salesforce a bit better, but still not good enough. So our team took up the opportunity to attend the very informative workshop led by Christophe Coenraets, being held that morning to learn about the Salesforce1 platform.

Salesforce1

This was a much needed intro to the system, to summarise, Salesforce1 is a mobile app you can download from the app/play store, with a developer account you have access to a developer organisation, which can have users with access credentials. Using Salesforce, you have access to different types of objects, such as People, Accounts, Contacts, Tasks, with your developer account you are free to customise these objects, add custom fields, perform custom actions on these objects and even create your own. The system has a sort of CMS for admins, which allow you to create and customise objects, actions, layouts and more, right in the web application. Since salesforce apps are also available on mobile using the Salesforce1 app, you can create mobile applications this way. Salesforce1 acts like an app container and this was what we were required to use for the hackathon. I think of the salesforce admin a bit like the django admin but with the ability to also create and edit models and not just instances of those models.

Salesforce1 offers three ways to develop an app, one method is to just use the admin CMS to create your own objects, actions and page layouts. Another option is to create VisualForce pages, these are server side pages, essentially HTML with a bit of server side scripting in a language called Apex, which looks quite similar to Java. A third option is to use what’s called Force.com Canvas. This essentially acts like an iframe so you can pull in a web app hosted somewhere else, but are able to authenticate with the app, so it can request data on a user’s behalf using a REST API. You can use any combination of these three to create your salesforce1 app, I knew at that point that canvas was going to be the way to go. Since Twilio requires an accessible server urls to return twiml. After the very useful workshop I had a look through some of the code that was mentioned in workshop, then got to work on the app. Starting with the canvas app.

The App

The canvas app itself was built as a node express app, we needed to be able to be able to initiate a twilio phone call and provide accessible urls to respond with the correct twiml. We used the node-twilio npm module, which is great for initiating calls and generating the correct twiml, in a programmatic way. For those who don’t know twilio, it’s a telecommunication service that allows you to create custom phone and sms integrations. It allows you to purchase twilio telephone numbers which can make and receive calls, you can configure what happens when a call is received or can initiate a call programatically, which was great for our needs. Twilio uses the concept of twiml, which is a special markup (xml) language related to phone/text actions. For example, you can use a Dial tag to dial a number, Record to record the call or a section of the call. Most of the twiml verbs support action attributes, which basically means what should happen when the verb has completed. For instance the Record verb supports the action attribute, so when the recording has finished the action (which should be a url) is requested and the resulted twiml returned from that url is used to carry on the next stage of the call.

With this in mind I set about creating the first step, which is initiating a call and reading a message. The first step worked, using my hard coded message, adding a simple html page with a button, I was able to initiate a call to my mobile and hear the message being read out. With that first baby step working, I decided to focus more on getting the node canvas app integrated into the salesforce1 platform. Despite my best efforts this required assistance from Christophe, who had previously shown a working node canvas example in the workshop earlier that day. Once we finally had the canvas app loading within the Salesforce1 mobile app, the next step was to try and get some of the contact records so that our app would be able to call any contact we had created within Salesforce. I created a basic html page, which would render out the contacts in a basic list, slowly improving the styling until we had a contact listing with buttons to call the number for the stored contact.

With this initial Salesforce1 integration out of the way, we could focus on getting the real test of the phone call. Getting the call receiver to record a message and transcribe it. This was the part I wasn’t sure if twilio could do and maintain the call. I already knew about the Record verb and that transcription of a call was possible if kept under 2 minutes, but I wasn’t sure it the call could keep going. Looking up the documentation, it appears the record verb supports finishing the recording on a key press. It also supports transcribing and 2 callbacks, action which is called when the recording has finished and transcribeCallback when the transcription has finished. So all we needed to do was return the right twiml for action to maintain the call for the receiver and make sure the transcribeCallback records the message and displays it in the app for the user to see. From this point it was obvious that some sort of call log was needed, which needs to be available on the session, while the active call is going on. Rather that complicate things and trying to add session support and a database backend, I decided to keep things simple for the hackathon, the node server would keep everything held in memory and only support a single call at a time. With this is mind, I implemented a transcribeCallback handler to add the transcribed message to the call log. I also added another url endpoint to handle the action callback. This endpoint would be called when the person receiving the call has pressed a key to end the recording. To keep it simple, this endpoint just responded with TwiML to read out the message, thanks for your message, then end the call. At this stage I just wanted to check whether the transcription would work. In order to see the message, the browser would need to poll for any new messages in the call log, so I added another endpoint which just returns the call log in JSON format. After a lot more fiddling and at around 4pm we had the first part working. The app could call my mobile, I’d hear a message that I’m receiving a call from a deaf or hard of hearing person and that I should leave a message after the beep and press any key when I’ve finished. Then after about 10 seconds I’d see the transcribed message appear in the app.

At this point I was feeling much more confident that our task might be achievable. There is always a bit of stress on developers at hackathons, having a very tight deadline to show off something and with prizes at stake, it can feel like a lot of pressure. So I was very relieved at having at least something working. The rest of the evening progressed well and I had settled into a bit of a more productive coding flow. I added support for placing the call in a queue after the recorded message had been made (TwiML Enqueue verb), removing them from the queue after the app user had typed their text response and reading the text response. By 2am both sides of the call were complete, the app user can call someone, the speaking person leaves a message, the app user sees the transcribed message, responds with a text reply, the speaking person hears the typed reply, then the process loops. I left for home for some sleep feeling in a much better position with the status of our app.

The next morning I implemented a few more improvements, adding support for the app user to type their own initial message which is read out to the call receiver. Also ensured the caller is announced correctly, using the salesforce user full name. Being a force.com canvas app, we can have a signed request passed to our application, which includes information about the authenticated salesforce user. We also added support for voice gender, adding a select field for the app user to choose whether their text messages are read out in a male or female voice. Oddly I didn’t spot a method to work out gender from the salesforce user we had access to, so had to add a select field for the user to specify. Added some final UI styling improvements to make it look a bit nicer. Finally, I wanted to try and get one more salesforce integration in the app, that is to save the completed call log back to salesforce. However, by this time we were so close to the final submission deadline I wasn’t able to finish it in time.

Demo

Luckily enough, and I think it was to do with our teams preparation, our demo actually worked. We actually got a round of applause when the transcribed message appeared on the screen. And as far as I can tell it was transcribed correctly! All our practice tests leading up to the demo had many transcription failures, which was worrying but I attributed it to the background noise. Likewise we were able to submit the text reply and hear it on the phone. In the end our app won third prize, which was probably fair considering the level of salesforce integration wasn’t great. I felt if I had managed to finish the last bit of the app and be able to save the call log back into salesforce, we might have come away with a higher score and maybe taken second place. Having said that, the other winning apps were great ideas and had a better business case use for salesforce. Overall, the hackathon was a fun experience, I’d do another one, but not right away. I certainly enjoyed a good Sunday night sleep.

Text relay app, initiating a call

Text relay app, initiating a call

Text relay app, active call

Text relay app, active call

You can try making your own Salesforce1 mobile app by registering for a developer account. You can also sign up for a Twilio account to try out some of the phone call techniques described here, you’ll be given a trial number which you can use to try out the service, everything should work, except for transcription, which must be paid for. You’ll also hear a demo message in all your calls until you switch to a paid number.

2014
07.13

A very short warning, for those working with backbone views using event delegation. You may already know this, but another friendly notice can’t be bad. Be sure to avoid using focus, blur, change, submit and reset as your backbone view events. Whilst these may work in modern browsers, it will not work in older Internet Explorer.
For example the following will not work in IE 8:

var FormView = Backbone.View.extend({
events: {
‘submit’: ‘handleSubmit’
},
handleSubmit: function(e){
console.log("Form was submitted");
this.model.save();
return false;
}
});

The main backbone documentation does not seem to mention this rather important fact, although the annotated source does.

How to solve this for IE 8?

In short, I’m not sure. You could try listening for a click on the submit button instead, but that won’t work when the form is submitted by pressing enter on the keyboard. So what I have found works is after your view is rendered, use jQuery to listen to the submit event (this.$el.on(‘submit’ this.handleSubmit). In fact, in most cases I’m still using jQuery validate after the form is rendered. This has a submitHandler callback so you can use that instead.

I’ve seen plenty of examples of people using submit  in their views, including myself. Unfortunately I still need to support IE 8 in my work. I envy those of you who don’t.

2011
04.15

This is a quick post to let everyone aware that instanceof should be avoided like the plague in your Flex/ActionScript 3 projects. Coming from a Java development background I was quite familiar with using instanceof to test if a class is of a certain type, in Java this also matches interfaces, not so in ActionScript.

To illustrate the problem, here are some simple classes and an interface:

public class Vehicle {
    private var _name:String;

    public function Vehicle(name:String)
    {
        _name = name;
    }

    public function get name():String
    {
        return _name;
    }

    public function drive():void
    {
        trace("Vehicle named "+_name+" drove.");
    }
}

public interface IManualGears {

    function changeGear(gear:int):void;

}

public class ManualCar extends Vehicle implements IManualGears {

    public function ManualCar(name:String)
    {
        super(name);
    }

    public function changeGear(gear:int):void
    {
        trace("Vehicle named "+_name+" changed gear to "+gear);
    }

}

If I were to create a new instance of ManualCar and tested for the interface IManualGears using instanceof it would fail, as shown below;

var car:ManualCar = new ManualCar("Ford Escort");

var isVehicle:Boolean = (car instanceof Vehicle);  //isVehicle would be true

var isManualCar:Booelan = (car instanceof ManualCar);  //isManualCar would be true

var hasGears:Boolean = (car instanceof IManualGears);  //hasGears would be false

hasGears:Boolean = (car is IManualGears);  //hasGears would be true;

To be fair this is stated in the flex documentation, but it did catch me out. So the moral of the story is to always use is instead of instanceof.

2011
01.05

This is just a quick post to share a regular expression for a URL I had to come up with when needing to validate a URL in a Flex app. The code below is for Flex, but would only require a few minor changes for another language, the double backslash before the ? appears to be required for Flex, using a single backslash does not work, read more about that in this older post. This also contains what would be capturing brackets in other languages, I could have used non-capturing brackets but that would have made this already complicated example even more difficult to read.

linkValidator = new RegExpValidator();
linkValidator.expression = "(http(s)?:\/\/)?(([a-z]+[a-z0-9\-]*[.])?([a-z0-9]+[a-z0-9\-]*[.])+[a-z]{2,3}|localhost)(\/[a-z0-9_-]+[a-z0-9_ -]*)*\/?(\\?[a-z0-9_-]+=[a-z0-9 ‘,.-]*(&[a-z0-9_-]+=[a-z0-9 ‘,.-]*)*)?(#[a-z0-9/_-]*)?$";
linkValidator.noMatchError = resourceManager.getString("lang", "invalidURL");
linkValidator.flags = "i";
linkValidator.source = linkTextArea;
linkValidator.property = "text";
linkValidator.trigger = linkTextArea;
linkValidator.triggerEvent = Event.CHANGE;

I’ll break it down to the individual sections with a brief explaination.

//protocol and subdomain
(http(s)?:\/\/)?(([a-z]+[a-z0-9\-]*[.])?

The first part includes the protocol (http:// or https://), I am only dealing with web http urls here and it is optional in my app hence the ? at the end of the first group, the rest includes an optional subdomain which should start with one or more letters followed by zero or more letters/numbers/hyphens and a dot. This first subdomain and dot is also optional. So far this would match: [empty string] http:// https//www. https://ww2 etc.

//server hostname
([a-z0-9]+[a-z0-9\-]*[.])+[a-z]{2,3}|localhost)

This next part includes the rest of the web host, the first grouping (first enclosing brackets) specifies the start of the hostname or a further subdomain which must start with a letter or number followed by a dot (the dot as a character set is how to represent the dot in Flex, you might be able to just use \.). This can be repeated many times, but then should be followed by 2 or three characters. Alternatively the hostname localhost can be used instead, the extra closing bracket matches the additional opening one after the protocol. This section should match: www.example.com localhost example.com example.co.uk co.uk etc.

//web path
(\/[a-z0-9_-]+[a-z0-9_ -]*)*\/?

This next part consists of the optional path (directory from the web root), it starts with a forward slash and can be any number of letters, numbers, underscores, spaces or hyphens, but can not start with a space (you might need to backslash escape your hypen in a different language. The trailing backslash is also optional as is the entire path. This part should match: [empty string] / /directory /a/b/ etc.

//query string
(\\?[a-z0-9_-]+=[a-z0-9 ',.-]*(&[a-z0-9_-]+=[a-z0-9 ',.-]*)*)?

This part contains the optional query string part of the URL. Starting with a ? (may require only a single backslash in a different language), followed by the first parameter made up of one or more letters/numbers/underscores/hyphens the equals sign, followed by an optional parameter value made up of letters/numbers/spaces/apostrophes/commas/dots/hypens. This parameter=value part of the query string can be repeated several times after that but each extra parameter should be preceded with the ampersand (you would normally use just the & for this, but flex requires the encoded version). This section could match: [empty string] ?a= ?a=bc ?a=b&c=d&e=f etc.

//fragment
(#[a-z0-9/_-]*)?$

Finally the last part of the expression contains the optional url fragment (the part with a #). In my case I specified zero or more letters/numbers/forward slashes/underscores/hypens (flex does not require escaping the forward slash when it is included in a character set). Then the dollar sign specifies that there should not be anything else after this. This could match: [empty string] # #value #a/b/c etc.

I hope this is useful to those struggling to create their own URL regular expression matchers. Flex devs remember to double escape the ? for the query string part of the URL.

2010
11.12

This is a very short post which will probably become irrelevant by the time someone sees it, but if you are a PHP developer using eclipse you might have noticed that your compare viewer is not working. I ran into this a few times before bothering to do a bit of research and found this bug report.

So it looks like the bug has been fixed but has not yet been released. So for those who want their compare viewer working again and don’t want to read the bug report, the solution is go to Help – Install New Software – Add a new site (link below) and select PHP Development Tools (PDT) Runtime Feature, and the SDK if you want it. Heres the link:

http://download.eclipse.org/tools/pdt/updates/2.2/milestones
2010
07.24

Again another long time between posts, this one is partly a small tutorial and partly about a small issue I encountered and workaround. This post discusses form validation using a cool jQuery form validation plugin created by Jörn Zaefferer, which I highly recommend, you can download the plugin from here. The plugin allows you to quickly and easily validate your html forms using the power of jQuery, the basic principle is that you specify the rules for validation on the fields of your form and submitting the form will be disabled until all fields are valid, where validation rules are broken the user is informed with messages next to the invalid fields. I wouldn’t be surprised if this plugin became part of jQuery directly since so many people are using it already.

Lets have a look at a quick simple example, lets say I have a form with 3 input fields, name, email and website, we want to have the name and email fields required, we want to check the email is valid and because we are storing these fields in a database with a character limit we need to check the fields are not too long. Below is the html form code and the javascript for the validation.

<script type="text/javascript">
$(function(){
    $("#subscribe_test").validate({
        rules: {
            fullname: { required: true, maxlength: 64 },
            email: { maxlength: 32 },
            website: { maxlength: 32 }
        }
    });
});
</script>

<form id="subscribe_test" action="save_subscriber.php" method="POST">
  <table>
    <tbody>
      <tr>
        <td><label for="fullname">Full name: *</label></td>
        <td><input id="fullname" type="text" /></td>
      </tr>
      <tr>
        <td><label for="email">Email: *</label></td>
        <td><input id="email" class="email required" type="text" /></td>
      </tr>
      <tr>
        <td><label for="website">Website:</label></td>
        <td><input id="website" type="text" /></td>
      </tr>
      <tr>
        <td><input type="submit" /></td>
        <td></td>
      </tr>
    </tbody>
  </table>
</form>

In the above example you can see that you can apply the validation rules both in javascript and also using the predefined rule names as the css class for the input. E.g. the email input field is using classes email and required, the validation plugin will look for any class names on the inputs and apply the validation rule if there is one. I could have defined the email rule solely in the script but I wanted to show that there is a quicker way using classes. The working example for the above code is below.

You can also do remote validation. For example, lets say in our example we did not want someone with the same email address to be subscribed twice. You can use a special validation rule remote which points to a script which should output true or false based on the form input value. We will also give a special message for when the email address is already in use. The changed script code is then:

<script type="text/javascript">
 $(function(){
     $("#subscribe_test2").validate({ rules: {
         fullname: { required: true, maxlength: 64 },
         email: { maxlength: 32, remote: "/check_subscribed.php" },
         website: { maxlength: 32 }
     },
    messages: {
        email:{ remote: "Already subscribed" }
    } });
 });
</script>

The running example now will not allow you to enter the email test@example.com, for this example there is a script called check_subscribed.php which is hard coded to look for the POST email variable and return false if it equals test@example.com.

So that is all good everything seems to be validating fine. Now I will show you a small case to watch out for. Lets assume that the subscribe form needs to be submitted using ajax and only accepts a required email address which should be remotely validated. Below is the code and the working example, if you simply enter the already subscribed email test@example.com and submit you see it does not work properly, the form gets submitted even though the validation fails.

Update 29/09/2010 since updating this post and rechecking the code it appears, this actually does work. Perhaps the browsers have changed the order the javascript events occur, I certainly did not change anything, this was an issue previously.

Code and example below.

<script type="text/javascript">
jQuery(function($){
       $("#subscribe_test3").submit(function(e){
          if ($(this).valid()){
               $.ajax( {
                   type: "POST",
                   url: "/save_subscriber.php",
                   data: "email="+$("#subscribe_test3_email").val(),
                   success: function(data){
                        $("#submit_result").html("Ajax submit results:
"
+ data);
                   }
               });
          }
          return false;
    });
    $("#subscribe_test3").validate({ rules: {
         email: { maxlength: 32, remote: "/check_subscribed.php" }
     }, messages: {
        email: { remote: "Already subscribed"}
     }
     });
});
</script>

<form id="subscribe_test3" action="save_subscriber.php" method="POST">
  <table>
    <tbody>
      <tr>
        <td><label>Email: *</label></td>
        <td><input id="subscribe_test3_email" class="email required" name="email" type="text" /></td>
      </tr>
      <tr>
        <td><input type="submit" /></td>
      </tr>
    </tbody>
  </table>
</form>

The reason the above example did not work properly is because the remote validation checking is done asynchronously, as is the ajax request. In the above code, the form submit event binding function checks first if the form is valid before doing the ajax submit. The trouble is that the remote validation is done asynchronously. So the valid check is just returning true instead of false. In order to solve this the remote validation needs to be done synchronously instead (or async: false). Below is the relevant fix and the fixed example.

$("#subscribe_test3").validate({
  rules: {
    email: {
      maxlength: 32, remote: { url: "/check_subscribed.php", async: false }
    }
  },
  messages: {
    email: { remote: "Already subscribed"}
  }
});

Update 27/09/2010

It has been asked what the check_subscribed.php script did, so here it is below:

<?php
$email = $_REQUEST["email"];
if ($email == "test@example.com"){
    echo "false";
} else {
    echo "true";
}
?>

Update 01/11/2010

Someone has asked for the files used in these examples, so here is a zip file containing a html page and the necessary php scripts. The jquery and validation plugin are not included in the zip but are referenced as external scripts. This is a recommended way of including jQuery in your sites (but obviously a more up to date version).

2010
05.08

Very recently I decided to clean my home FreeBSD server  and start from scratch, meaning deleting everything and reinstalling FreeBSD. The fact that I happened to do this after completely messing up my ports system is purely coincidence… 😉

I took the time to try out the new FreeBSD release 8.0, so I downloaded the iso from the FreeBSD site, burned it and installed it on my home server (the server I’m referring to is in this post). After reinstalling all the ports I had on it previously (Apache, MySQL, PHP5, PHP5-extensions, Java 1.6, Openfire etc.), I decided to go one step further and try something I hadn’t done before, get email working.

DNS settings

To be able to receive email I needed a domain, now it happens that I purchased chrismcdonald.co.uk at some point last year, got it very cheap for 2 years from 123-reg. The domain wasn’t really doing anything and was just being used to forward to this blog (and still is), so I decided to get more use out of it and use a subdomain to point to my home server. I set the hostname of my FreeBSD server to server.chrismcdonald.co.uk by adding hostname=”server.chrismcdonald.co.uk” to /etc/rc.conf and added an entry in /etc/hosts so that it would resolve locally. So my hosts file now contained:

127.0.0.1               localhost server.chrismcdonald.co.uk

To get the domain to point to my home server I changed my dns records for chrismcdonald.co.uk and added a CNAME record pointing server.chrismcdonald.co.uk to my dynamic dns host chris.is-a-geek.net. I also added an MX record for the root domain which directs email to chrismcdonald.co.uk to chris.is-a-geek.net. Dynamic DNS allows users with a non-static IP (most home internet users) to use a host name to refer to your system’s ip address, which might change whenever you connect to your ISP. It works by having your home system contact the dynamic dns provider every so often to make sure the ip address mapping is kept up to date. I use dyndns.com, you can create a free account with them where you can create some free host names to use. You get to pick your own subdomain from any one of the domains they allow you to use, I chose the domain is-a-geek.net and my subdomain chris. My wireless router has built in support to work with dyndns.com and keep my host ip address updated, but if yours does not, there are some instructions also in my older post to get this working on FreeBSD.

Receiving mail

To ensure I could send and receive email past my wireless router, I added some port forwards for ports 25 (smtp), 465 (secure smtp) and 993 (secure imap), I already have existing port forwards set up for web and ssh access. Sendmail is configured by default to only allow sending of email and receiving mail only from localhost. To enable sendmail to receive mail externally I added this line to /etc/rc.conf and started sendmail again.

sendmail_enable="YES"

/etc/rc.d/sendmail stop
/etc/rc.d/sendmail start

OK so with the domain name sorted out and sendmail set to receive external mail I tried testing it out. I logged into another remote server, which was located outside of my local network on the internet. I used telnet to attempt to manually send an email. Below is the transcript, my commands are in bold, square brackets are for your server name and email.

telnet server.chrismcdonald.co.uk 25
Connected to chris.is-a-geek.net.
Escape character is '^]'.
220 server.chrismcdonald.co.uk ESMTP Sendmail 8.14.4/8.14.3; Sat, 8 May 2010
12:08:15 +0100 (BST)
HELO [remote server name]
250 server.chrismcdonald.co.uk Hello [remote server name and ip] , pleased
to meet you
MAIL FROM: [user@remoteserver]
250 2.1.0 [user@remoteserver]... Sender ok
RCPT TO: [user@server - in this case chris at chrismcdonald.co.uk]
550 5.7.1 [user@server]... Relaying denied
QUIT
221 2.0.0 server.chrismcdonald.co.uk closing connection

So it appeared that my server was not accepting email for my domain. To fix this I created the file /etc/mail/local-host-names and added the following lines, you would add your own hostnames.

chrismcdonald.co.uk
server.chrismcdonald.co.uk

Then when I tried again:

telnet server.chrismcdonald.co.uk 25
Connected to chris.is-a-geek.net.
Escape character is '^]'.
220 server.chrismcdonald.co.uk ESMTP Sendmail 8.14.4/8.14.3; Sat, 8 May 2010
12:08:15 +0100 (BST)
HELO [remote server name]
250 server.chrismcdonald.co.uk Hello [remote server name and ip], pleased
to meet you
MAIL FROM: [user@remoteserver]
250 2.1.0 [user@remoteserver]... Sender ok
RCPT TO: [user@server - in this case chris at chrismcdonald.co.uk]
550 5.7.1 [user@server]... Recipient ok
DATA
354 Enter mail, end with "." on a line by itself
Subject: test email again
Just testing
.
250 2.0.0 o48BLo6R001500 Message accepted for delivery
QUIT
221 2.0.0 server.chrismcdonald.co.uk closing connection

Sure enough I had the new email in my local mailbox (checked by running mail).

IMAP access

So now I could receive mail from outside, I looked around for a few guides to get started on setting up secure IMAP and secure SMTP sending. I needed a program that would provide IMAP access to mail and came across this blog post on Dovecot on freebsddiary.org, this is a great site with loads of tutorials and info on FreeBSD. I followed the steps in the guide to install Dovecot with some minor adjustments.

cd /usr/ports/mail/dovecot
make install clean

Below are the make options I used:

Dovecot make options

Dovecot make options

You can follow those steps from freebsddiary if you are starting from scratch. I ignored the certificate stuff because I already had a self signed one created, I also chose to use PAM authentication, which uses your user account password to authenticate. The important bits which I changed from /usr/local/etc/dovecot.conf are below, change them where they occur in the file:

protocols = imaps
ssl_cert_file = /etc/ssl/certs/server.chrismcdonald.co.uk.cert
ssl_key_file = /etc/ssl/private/server.chrismcdonald.co.uk.key
ssl_key_password = ******************
mail_location = mbox:~/mail/:INBOX=/var/mail/%u
protocol imap {
listen = *:143
ssl_listen = *:993
# Login executable location.
...
}
auth default {
  # Space separated list of wanted authentication mechanisms:
  #   plain login digest-md5 cram-md5 ntlm rpa apop anonymous gssapi otp skey
  #   gss-spnego
  # NOTE: See also disable_plaintext_auth setting.
  mechanisms = plain login
...
}

Then add the following to /etc/rc.conf and start dovecot.

dovecot_enable="YES"

/usr/local/etc/rc.d/dovecot start

Now I access my mail using imap securely over port 993. So far the server is able to receive mail, send mail (only from localhost) and has secure imap access to read mail, if this is suitable for what you need you can stop here. If you want to be able to send mail from another machine using your secure smtp read on.

SMTP sending

I found a couple more guides on having secure smtp authentication here and here. I did not follow those steps but borrowed some from both guides. The common setup appears to use cyrus sasl (Simple Authentication and Security Layer) for the authentication and to configure sendmail to use it. SASL SASL support is not build into sendmail by default, the guides said to recompile sendmail from its source /usr/src/usr.sbin/sendmail which I couldn’t find, I guess these things are all done using the ports system now, so here’s what I installed.

cd /usr/ports/security/cyrus-sasl2
make install clean
cd /usr/ports/security/cyrus-sasl2-saslauthd
make install clean
cd /usr/ports/mail/sendmail-sasl
make install clean

Then I needed to enable the sasl daemon by editing /etc/rc.conf and adding this line and starting the sasl daemon.

saslauthd_enable="YES"

/usr/local/etc/rc.d/saslauthd start

Finally sendmail needed to be configured to use secure smtp authentication, I also copied my certificate files that I generated a while back into /etc/mail/certs. I copied the default freebsd.mc and freebsd.submit.mc files located in /etc/mail to server.chrismcdonald.co.uk.mc and server.chrismcdonald.co.uk.submit.mc and edited server.chrismcdonald.co.uk.mc. Then following bits from both guides I added the following:

dnl set SASL options
TRUST_AUTH_MECH(`GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN')dnl
define(`confAUTH_MECHANISMS', `GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN')dnl
dnl Offer SMTP AUTH only after encryption (STARTTLS) has been negotiated
define(`confAUTH_OPTIONS',`p,y')dnl

define(`CERT_DIR', `/etc/mail/certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/server.chrismcdonald.co.uk.cert')dnl
define(`confSERVER_CERT', `CERT_DIR/server.chrismcdonald.co.uk.cert')dnl
define(`confSERVER_KEY', `CERT_DIR/server.chrismcdonald.co.uk.key')dnl
define(`confCLIENT_CERT', `CERT_DIR/server.chrismcdonald.co.uk.cert')dnl
define(`confCLIENT_KEY', `CERT_DIR/server.chrismcdonald.co.uk.key')dnl
DAEMON_OPTIONS(`Port=smtps, Name=TLSMTA, M=s')dnl

The lines starting with dnl are comments and those backticks ` are not typos, the opening quote is a backtick and the closing one a normal apostrophe. Finally after running the following in /etc/mail I had secure smtp authentication working:

make
make install
make restart

Well, that’s it for another post, sorry it was such a long one. I am buying a couple of real servers on eBay that wer e going pretty cheap, the kind they use in datacentres. I will probably have a go at getting internal dns working, so that I won’t be restricted to a single server working at home. If it works I will do another post on it.

2010
03.26

Today I came across an annoying problem involving the RegExpValidator class in Flex. In flex app for one of our projects StickyWorld, we allow users to upload images, pdfs, 3d models and most recently, reference YouTube movies inside a virtual room where people can comment on them, adding sticky notes in context. In our upload window, users can choose what type of media they want to add to the room, in the case of a YouTube video only the video id is required. To make it a bit more flexible we decided to allow the users to specify a url to a video as well as just the video id. The app needed to handle urls in several different formats, e.g.

  • http://www.youtube.com/watch?v=[videoid]
  • http://www.youtube.com/v/[videoid]
  • http://www.youtube.com/[userchannel]/[somepath]/[videoid]

Sounds like a job for a regular expression right? Since we also wanted to do some validation on the field to make sure we were given a valid YouTube video url or video id, it seemed that the RegExpValidator class was perfect for the job. However after adding the RegExpValidator in MXML I had a problem that took a good while to figure out what was wrong.

Below is the snippet of code containing the regular expression string, its use in the validator and a validation handler function, apologies for the wrapping of the regular expression, I have split it below where the main formats are separated by pipes.

[Bindable]
private var youtubeRegExp:String
"^(?:http:\/\/(?:www\.)?youtube\.com\/watch(?:\?|#!)v=(.\{11\})(?:&.*)?
|(.\{11\})
|http:\/\/(?:www\.)?youtube\.com\/(?:v|[A-Za-z0-9#\/_\-]*)\/(.\{11\}))$"
;

private function youtubeValid(ev:ValidationResultEvent):void
{
    if (ev.type == ValidationResultEvent.VALID) {
        for (var i:int = 0;
                i < ev.results[0].matchedSubstrings.length;
                i++
        ) {
             if (ev.results[0].matchedSubstrings[i] != null)
                  youtubeVidId = ev.results[0].matchedSubstrings[i];
        }
        txtName.text = youtubeVidId;
        btnSubmit.enabled = txtYouTube.text.length>0;
    } else {
         btnSubmit.enabled = false;
    }
}

<mx :RegExpValidator id="youtubeValidator"
    source="{txtYouTube}" property="text"
    expression="{youtubeRegExp}" valid="youtubeValid(event)"
    invalid="youtubeValid(event)"
    noMatchError="YouTube video url/id invalid"
    trigger="{txtYouTube}"/>

The problem was, nothing was coming out as valid, not even the videoid just by itself. So wrote a test using the regular expression, creating a RegExp object using the expression and testing some urls using the exec function and it appeared to be working fine, so why not with the validator? In the end it turned out to be that the problem was down to several characters needing to be escaped. The correct expression that worked is below:

youtubeRegExp =
"^(?:http:\/\/(?:www\.)?youtube\.com\/watch(?:\\?|#!)v=(.\{11\})(?:&;.*)?
|(.\{11\})
|http:\/\/(?:www\.)?youtube\.com\/(?:v|[A-Za-z0-9#\/_\-]*)\/(.\{11\}))$"
;

Since the regular expression used in RegExpValidator needs to be a String and the String is used as a bound property in MXML the curly brackets need to be be preceded with a backslash, because without them in MXML it means a data binding. That didn’t make much sense to me, since I was binding a String variable which contained the expression, but ok, I can kind of see the problem. What made even less sense was that I needed to double escape the ? when I actually wanted to include a literal ? in the expression. So I suppose since it is a string and I want to include the backslash character before the question mark I need to escape the backslash itself, leaving \\?, but then shouldn’t I have to do that for all the other times I need to include a backslash in the expression? Well it turns out the answer is no, I do not really understand it and only figured this out after a lot of debugging.

Another pitfall to avoid, if you are binding on a condition in MXML and you need to use logical operators, you will need to encode the logical AND && should be &amp;&amp;, the less than < should be &lt; or greater than signs > should be &gt;, logical OR || works fine. However in those cases you should really be binding to a function which returns the boolean result you are looking for.

Working with MXML is cool because you can create data-bindable UI components quickly and easily, but it really isn’t when you have to worry about silly issues like wondering why your regular expressions are not working.

2010
03.12

It has been another long time between posts, but here is as promised, but very delayed, the second part to this post.

Working with git repositories in trac

It is incredibly simple to use a git repository instead of svn with trac, you just need to have the right plugin installed and enabled. Once again I’ll give install details for FreeBSD and Ubuntu.

Firstly we need to install the trac git plugin. For FreeBSD there is as usual a dedicated port install for what we need. As of writing the plugin version is 0.11.0.2 r7034. As the root user run:

cd /usr/ports/www/trac-gitplugin
make install clean

Ubuntu users, as root or using sudo:

apt-get install trac-git

With the trac git plugin installed, all we need to do is create a new git repository if we haven’t got one already and create a new trac project pointing to the git repository (or edit the trac.ini config file of an existing trac project). So following from the FreeBSD logic in the previous post, lets create a new git repository in /usr/local/git as root.

cd /usr/local
mkdir git
cd git
mkdir MyGitProject
git init MyGitProject

The git repository above was created by the root user, if you are accessing the git repository over ssh and want to be able to push changes to it, you will need to make sure that the user has read and write access to the folder and everything in the .git folder. What I have done in the past is create a group on the system called developers and add all the user accounts to the group. Then change the group of the git repository to developers and give it group write access. You have to make sure the whole folder and all files are writable by this group or you will get errors trying to push to the repository. An alternative is to just create the git repository using one user account or change the owner of the whole repository to a specific user account and only use that user account to push and pull to the server.

With the repository created, we can now create a new trac project, this is the same steps as in the previous post but we change the repository type from svn to git.

cd /usr/local/trac
mkdir MyTracGitProject
trac-admin MyTracGitProject initenv

The trac-admin program runs in interactive mode and will ask you questions to set up your repository, example questions and responses are given below.

Project Name [My Project]> My Trac Git Project
Database connection string [sqlite:db/trac.db]>
Repository type [svn]> git
Path to repository [/path/to/repos]> /usr/local/git/MyGitProject/.git

Now we just need to edit the trac.ini, this is in the trac project folder, so in our case /usr/local/trac/MyTracGitProject/conf/trac.ini. Make sure that under the [trac] section repository_type is git and the repository_dir is correctly pointing to the .git folder. Also FreeBSD gets the location of the git binary wrong, under the [git] section change git_bin from /usr/bin/git to /usr/local/bin/git. Finally, add the following if your trac.ini file does not already have it.

[components]
tracext.git.* = enabled

Now we just need to change the owner of the new trac project to the web user and edit the apache config for the new trac project location. See the previous post for how to do that.

Adding Mylyn support for trac

Mylyn for Eclipse is great, if you are not using it already, you should be. It basically allows you to create tasks, those tasks you can activate and work on, whilst a task is activated it looks at what files you have open and have been working on. So that the next time you activate the task, you see only the files that are most relevant to the task, for more info on Mylyn go here. Mylyn is integrated in the more recent editions of Eclipse, but there is a Mylyn trac plugin that allows you to create and view tickets in your trac repository. How nice is that? If someone reports a new bug, you can see it appear right in your IDE, work on it and close it, all from within Eclipse.

To get the Mylyn support working in trac you need to install and enable the xml-rpc plugin. As usual FreeBSD has a port for it.

cd /usr/ports/www/trac-xmlrpc
make install clean

Ubuntu users can run:

easy_install -Z -U http://trac-hacks.org/svn/xmlrpcplugin/trunk # 0.11

At the end of the install process keep a look out for the location where the python egg was installed, we need to copy that file to our plugins folder in the trac project. This was /usr/local/lib/python2.6/site-packages/TracXMLRPC-1.0.6-py2.6.egg on FreeBSD.

Edit the trac.ini file, under the [component] section add:

tracrpc.* = enabled

Now when you go to the admin section of your trac site you should see the xml-rpc plugin available under Plugins.

Close and comment on trac tickets in your SVN commit messages

You can have trac close and add comments to your tickets simply by including certain text triggers in your SVN commit messages, such as fixes #231, refs bug:29. To get this working you need to include a special script in your svn repository location. First download this python script from the trac web site. Save it in a location which is easily accessible, on FreeBSD I put the script in the trac share folder /usr/local/share/trac.

Now go to your svn repository location in the hooks folder you should see a lot of files with a .tmpl extension. One of them is post-commit.tmpl, we need to copy this and edit it as follows:

cp post-commit.tmpl post-commit
edit post-commit

Ignore all the comments, at the end make sure the file reads as below:

TRAC_ENV="/usr/local/trac/[TracProjectName]"
REPOS="$1"
REV="$2"

/usr/local/bin/python /usr/local/share/trac/trac-post-commit.py -p "$TRAC_ENV" -r "$REV"
#mailer.py commit "$REPOS" "$REV" /path/to/mailer.conf

Replace [TracProjectName] with your trac project name or change the path if yours is different. I specified the full path to the python command, on freebsd when this shell script is triggered the script does not inherit you environment PATH, so absolute paths to commands need to be specified, at least on FreeBSD anyway. Change the path of the trac-post-commit.py file to wherever you saved it. I commented out the mailer.py script, this is another post-commit script that you can run which sends an email detailing the commit. This script should be shipped with your subversion installation, on FreeBSD it is in /usr/local/share/subversion/hook-scripts/mailer/mailer.py.

Finally we need to make sure that the python script is executable and owned by the web user like everything else in the svn repository.

chmod 755 post-commit
chown www post-commit

Now when you do your next commit to the repository try closing or referencing one of the trac tickets. You can refer to a ticket by prefixing the ticket number with #, ticket:, issue: or bug:. You can close a ticket using by using a command prefix close, closed, closes, fix, fixed, fixes before the ticket reference. To comment on a ticket but not close it use any of these command prefixes: addresses, re, references, refs or see.

You can customise these commands or ticket references, just edit the trac-post-commit.py file, the following is what you’ll need to change:

ticket_prefix = '(?:#|(?:ticket|issue|bug)[: ]?)'

    _supported_cmds = {'close':      '_cmdClose',
                       'closed':     '_cmdClose',
                       'closes':     '_cmdClose',
                       'fix':        '_cmdClose',
                       'fixed':      '_cmdClose',
                       'fixes':      '_cmdClose',
                       'addresses':  '_cmdRefs',
                       're':         '_cmdRefs',
                       'references': '_cmdRefs',
                       'refs':       '_cmdRefs',
                       'see':        '_cmdRefs'}

So, thats all for now. Apparently you can also do git post-commit hooks as well, but as I am not using git that much I haven’t tried it out yet. Happy coding with your svn/git trac repositories.

2010
01.08

It has been way too long since the last post, so here is something you might find useful.

A couple of months back we noticed that our commercially hosted svn and trac repositories were becoming quite unresponsive and slow, which held up development albeit only slightly. Time is money as they say, so we decided to cancel our commercial hosting and go for a local svn set up on our development server (just an old networked dell laptop, not very high specs at all). Our dev server is just a freebsd 7.1 install on an old dell laptop, which already had Apache, MySQL and PHP installed (FAMP server). For this guide you will need to have at least Apache installed (plus a few other port installs which I will detail below), but if you’re going to install Apache you might as well go all the way and have MySQL and PHP, see here and here posts on this. This guide is mainly for FreeBSD, although steps should be similar for Ubuntu, although your installs are very different, where appropriate I’ll put the ubuntu alternative installation steps.

Assuming you have already set up your FAMP server, next steps are as follows:

Install subversion

Use the make options as defined below. At the time of writing I am installing subversion 1.6.6

cd /usr/ports/devel/subversion
make install clean
Subversion make options

Subversion make options

This might take a while depending on how many dependencies need to be installed.

Install trac

Leave the options as they are or copy from below. At time of writing I am installing trac 0.11.5

cd /usr/ports/www/trac
make install clean
trac make options

trac make options

Install mod_python for apache

From /usr/ports/www/mod_python3. Trac runs on python, the trac website neds to be handled by python so we need to install this apache module for the trac website(s) to work

cd /usr/ports/www/mod_python3
make install clean

Apache needs to be set up to enable the new mod_python module, so edit the httpd.conf file located in /usr/local/etc/apache22/httpd.conf add the following to the end of the LoadModule section if it is not there already.

LoadModule python_module libexec/apache22/mod_python.so

Install Git (optional)

There is a trac plugin to work with git repositiories if you prefer that to svn. I will not go into git now, see my next post for integrating trac with a git repository.

cd /usr/ports/devel/git
make install clean
git make options

git make options

You do not need perforce or cvs support unless you want it. I have never used either, only subversion. You will not be able to install the GUI Tools without install the X11 windowing system. If you have already got X11 installed, then I would recommend installing the GUI tools as they can help with visualising the current branch on the development tree, but if not then do not bother as X11 takes AGES to install and will bloat your system with hundreds of dependencies.

Install the trac git plugin (optional)

You can install the trac git plugin if you want to work with git repositories rather than svn.

cd /usr/ports/www/trac-gitplugin
make install clean

Ubuntu install steps for the all of the above are below(they are a lot simpler and quicker! Ubuntu uses pre-compiled files binary files and automatically sets up apache with the relevant modules loaded):

sudo apt-get install subversion
sudo apt-get install libapache2-mod-python
sudo apt-get install libapache2-svn
sudo apt-get install trac
sudo apt-get install git-core
sudo apt-get install trac-git

Now that we have all the programs we need lets go on to set up an svn repository.

Create an SVN repository

First we need to decide where we are going to have our svn repository, where ever you decide to keep it, you will need to make sure it is readable (and writable if you want to make checkins) by the apache user. For this example I am storing all my svn repositories under one location, /usr/local/svn. Make sure you are root and run the following:

cd /usr/local
mkdir svn
cd svn
mkdir repositories
cd respositories
mkdir Test
svnadmin create Test
cd /usr/local/svn
htpasswd -c .passwd [Username]
[Enter password, and again]
chown -R www /usr/local/svn

Ok, the steps above we created a location where we are going to store our svn repositories, we created one respository called Test, then we created an apache basic authentication access file which the svn server will use to authenticate the users. Finally we changed the owner of the entire svn directory over to the apache web user. Ubuntu users can perform the same steps, feel free to choose a different location than /usr/local, this is common practice in a freebsd system.

Setup Apache to serve our SVN repository

Now we need to tell apache which directory contains our svn repository. As root create the following file by running:

touch /usr/local/etc/apache22/Includes/svn.conf

Ubuntu users:

touch /etc/apache2/conf.d/svn.conf

Then edit that file and save the following in it:

<Location /svn>
DAV svn
SVNParentPath /usr/local/svn/repositories
AuthType Basic
AuthName "Subversion access"
AuthUserFile /usr/local/svn/.passwd
Require valid-user
</Location>

Now if you restart your apache webserver (apachectl restart) and navigate to http://[server address]/svn/Test you should be asked for your username and password. Once you have logged in you should see your empty repository!

Create a trac repository

Now we will create a trac repository, I will follow the same logic as for the svn repositories and keep all the trac repositories under /usr/local/trac. As root again run:

cd /usr/local
mkdir trac
cd trac
mkdir Test
trac-admin Test initenv

trac-admin is an interactive program and you will be asked a series of questions to set up the trac repository. You will have to enter or accept the default value (by pressing return) at the following prompts:

Project Name [My Project]> Test
Database connection string [sqlite:db/trac.db]>
Repository type [svn]>
Path to repository [/path/to/repos]> /usr/local/svn/repositories/Test

When you have finished change the owner of the trac folder to the apache web user:

chown -R www /usr/local/trac

The steps are again the same for Ubuntu users.

Setup apache to serve our trac repository

Apache needs to serve the trac repository using the mod_python module we installed earlier. To set this up, create the configuration file by running the following as root:

touch /usr/local/etc/apache22/Includes/trac.conf

Ubuntu users:

touch /etc/apache2/conf.d/trac.conf

Save the following text in the new file:

<Location /trac/MyRepoName>
SetHandler mod_python
PythonHandler trac.web.modpython_frontend
PythonOption TracEnv /usr/local/trac/Test
PythonOption TracUriRoot /trac/Test
AuthType Basic
AuthName “trac access”
AuthUserFile /usr/local/svn/.passwd
Require valid-user
</Location>
<Location /trac/MyRepoName>
SetHandler mod_python
PythonHandler trac.web.modpython_frontend
PythonOption TracEnv /usr/local/trac/Test
PythonOption TracUriRoot /trac/Test
AuthType Basic
AuthName "trac access"
AuthUserFile /usr/local/svn/.passwd
Require valid-user
</Location>

Now if you restart apache again by running apachectl restart and navigate to http://[server address]/trac/Test you should be asked to authenticate and see your trac site!

Testing the SVN repository

To test the repository we will checkout a working copy, in your users home directory, try the following:

mkdir svnworkingcopy
cd svnworkingcopy
svn checkout http://[server address]/svn/Test . --username [username]
[Enter password]
svn mkdir trunk branches tags
svn commit -m "Checking in svn directory structure"

Now if you go back to the trac site and click the Timeline button you should see the first commit with the message. Click on Browse source to see the directory structure we just created. You now have everything you need to work with your trac and svn repository and to start adding and committing your project code.

Look out for the next post which will detail a few more steps to get more out of your trac server setup including:

  • Using a git repository instead by using the trac git plugin
  • A few extras steps that will help integrating your trac repository with the Eclipse Mylyn trac connector
  • A svn post-commit hook to automatically add comments or close your trac tickets, simply by referring to a ticket number in your svn commit messages