Last weekend I entered a hackathon, the last of 4 hackathons as part of the Salesforce Summer of Hacks. This was not my first hackathon experience, but it was the first that lasted more than a day. Overall, it was a good if tiring experience and our team came in third place claiming the $1,500 prize. Our 5 member team Text Relay worked on a mobile app that allows deaf or hard of hearing (HOH) people to have a two-way phone conversation, where the call receiver hears speech and the deaf/HOH person sees text in the app. Here’s a link to the submissions page for the hackathon, where you can read about the submitted apps. If you want the blow-by-blow version, read on, apologies but it’s a bit of a mammoth post.
The Hackathon Experience
I arrived to start the experience on a Friday evening, greeted with friendly faces, food and drink and lots of goodies. The event was held at a venue in Hoxton Square, fairly close to the silicon roundabout, it was booked for the whole weekend and open at all times for those who wanted to code through the night (spoiler, I didn’t make it through a whole night coding). After everyone had arrived and put on their assigned t-shirts (Code Breakers, Pixel Pushers and Secret Weapons), the event was announced and proceedings explained. Then came the idea pitches, I can’t remember the exact number of idea pitches, but it was in the region of about 42. This was fairly strict on timing, about a minute each, so I believe all the pitches were done in the space of an hour. I made some notes of the interesting ideas, spotted some nice ideas, but also heard ones I knew wouldn’t nearly be possible to do in the space of a hackathon. One of the pitches I heard was from Andrew, a deaf man who pitched, with the assistance of a speaking BSL interpreter, an idea for a mobile app to help deaf people making calls to hearing people. When I first heard that I thought I must have misheard, but kept it in my mind and listened to the other pitches.
When pitches were over we had to vote for the ideas, each of us had 3 poker chips that we had to give away to the ideas we liked the best. I gave away 2 tokens to a couple of enthusiastic people and also gave one to Andrew. Voting over the winning ideas were announced, this meant out of the 42 or so pitches, 26 were chosen as winning ideas, now we had to form teams for the winning ideas. I spoke to a few people and had to turn down a few offers, I think by then I had made up my mind to join Andrew’s team. After all I had some experience working with phones using Twilio in a previous hackathon, so thought I may be of some use. I joined the team as the 4th member and we got round to talk about the idea. We were required to work on a mobile app that would operate within Salesforce1 the mobile app for Salesforce, from the documentation I read on the day, that meant a native app was out of the question, which was fine by me, not having done any native apps before. I explained that I had worked with Twilio before and knew that it could do recording and transcribing of calls, so that it might be possible, but I wasn’t sure if an active call could be maintained while a message was being transcribed. At this point it was getting quite late, so the others called it a night.
I stayed a bit later that night, I had brought my sleeping bag along, although in truth didn’t need to. I spent some of the night researching what was possible with Twilio and then going through a tutorial on the Salesforce1 platform. By 2:30am I was pretty tired and realised I wasn’t going to get any sleep there. Since it was a summer night and the building was quite hot, there were many fans and AC units brought in and it was quite noisy. I decided to head home, catch a few hours sleep and head back early the next morning. I hired one of the cycle hire bikes and rode to Bow, getting some late night food at McDonalds before getting an uber cab home (that was my first time user Uber and sure wont’ be the last).
After around 3 hours sleep at home I got up, ready and headed back to the hackathon for Saturday morning. We had a deadline of 10am to register our team and create the initial app submission. We could change this as much as we liked but the initial submission needed to be in by 10am. Following this there was a git commit challenge, which actually just meant that you had to a git commit within a specific hour I believe. We were given access to a github repo to do our commits. At this point we still hadn’t really progressed the app at all, having gone through most of the tutorial the previous night, I was beginning to understand Salesforce a bit better, but still not good enough. So our team took up the opportunity to attend the very informative workshop led by Christophe Coenraets, being held that morning to learn about the Salesforce1 platform.
This was a much needed intro to the system, to summarise, Salesforce1 is a mobile app you can download from the app/play store, with a developer account you have access to a developer organisation, which can have users with access credentials. Using Salesforce, you have access to different types of objects, such as People, Accounts, Contacts, Tasks, with your developer account you are free to customise these objects, add custom fields, perform custom actions on these objects and even create your own. The system has a sort of CMS for admins, which allow you to create and customise objects, actions, layouts and more, right in the web application. Since salesforce apps are also available on mobile using the Salesforce1 app, you can create mobile applications this way. Salesforce1 acts like an app container and this was what we were required to use for the hackathon. I think of the salesforce admin a bit like the django admin but with the ability to also create and edit models and not just instances of those models.
Salesforce1 offers three ways to develop an app, one method is to just use the admin CMS to create your own objects, actions and page layouts. Another option is to create VisualForce pages, these are server side pages, essentially HTML with a bit of server side scripting in a language called Apex, which looks quite similar to Java. A third option is to use what’s called Force.com Canvas. This essentially acts like an iframe so you can pull in a web app hosted somewhere else, but are able to authenticate with the app, so it can request data on a user’s behalf using a REST API. You can use any combination of these three to create your salesforce1 app, I knew at that point that canvas was going to be the way to go. Since Twilio requires an accessible server urls to return twiml. After the very useful workshop I had a look through some of the code that was mentioned in workshop, then got to work on the app. Starting with the canvas app.
The canvas app itself was built as a node express app, we needed to be able to be able to initiate a twilio phone call and provide accessible urls to respond with the correct twiml. We used the node-twilio npm module, which is great for initiating calls and generating the correct twiml, in a programmatic way. For those who don’t know twilio, it’s a telecommunication service that allows you to create custom phone and sms integrations. It allows you to purchase twilio telephone numbers which can make and receive calls, you can configure what happens when a call is received or can initiate a call programatically, which was great for our needs. Twilio uses the concept of twiml, which is a special markup (xml) language related to phone/text actions. For example, you can use a Dial tag to dial a number, Record to record the call or a section of the call. Most of the twiml verbs support action attributes, which basically means what should happen when the verb has completed. For instance the Record verb supports the action attribute, so when the recording has finished the action (which should be a url) is requested and the resulted twiml returned from that url is used to carry on the next stage of the call.
With this in mind I set about creating the first step, which is initiating a call and reading a message. The first step worked, using my hard coded message, adding a simple html page with a button, I was able to initiate a call to my mobile and hear the message being read out. With that first baby step working, I decided to focus more on getting the node canvas app integrated into the salesforce1 platform. Despite my best efforts this required assistance from Christophe, who had previously shown a working node canvas example in the workshop earlier that day. Once we finally had the canvas app loading within the Salesforce1 mobile app, the next step was to try and get some of the contact records so that our app would be able to call any contact we had created within Salesforce. I created a basic html page, which would render out the contacts in a basic list, slowly improving the styling until we had a contact listing with buttons to call the number for the stored contact.
With this initial Salesforce1 integration out of the way, we could focus on getting the real test of the phone call. Getting the call receiver to record a message and transcribe it. This was the part I wasn’t sure if twilio could do and maintain the call. I already knew about the Record verb and that transcription of a call was possible if kept under 2 minutes, but I wasn’t sure it the call could keep going. Looking up the documentation, it appears the record verb supports finishing the recording on a key press. It also supports transcribing and 2 callbacks, action which is called when the recording has finished and transcribeCallback when the transcription has finished. So all we needed to do was return the right twiml for action to maintain the call for the receiver and make sure the transcribeCallback records the message and displays it in the app for the user to see. From this point it was obvious that some sort of call log was needed, which needs to be available on the session, while the active call is going on. Rather that complicate things and trying to add session support and a database backend, I decided to keep things simple for the hackathon, the node server would keep everything held in memory and only support a single call at a time. With this is mind, I implemented a transcribeCallback handler to add the transcribed message to the call log. I also added another url endpoint to handle the action callback. This endpoint would be called when the person receiving the call has pressed a key to end the recording. To keep it simple, this endpoint just responded with TwiML to read out the message, thanks for your message, then end the call. At this stage I just wanted to check whether the transcription would work. In order to see the message, the browser would need to poll for any new messages in the call log, so I added another endpoint which just returns the call log in JSON format. After a lot more fiddling and at around 4pm we had the first part working. The app could call my mobile, I’d hear a message that I’m receiving a call from a deaf or hard of hearing person and that I should leave a message after the beep and press any key when I’ve finished. Then after about 10 seconds I’d see the transcribed message appear in the app.
At this point I was feeling much more confident that our task might be achievable. There is always a bit of stress on developers at hackathons, having a very tight deadline to show off something and with prizes at stake, it can feel like a lot of pressure. So I was very relieved at having at least something working. The rest of the evening progressed well and I had settled into a bit of a more productive coding flow. I added support for placing the call in a queue after the recorded message had been made (TwiML Enqueue verb), removing them from the queue after the app user had typed their text response and reading the text response. By 2am both sides of the call were complete, the app user can call someone, the speaking person leaves a message, the app user sees the transcribed message, responds with a text reply, the speaking person hears the typed reply, then the process loops. I left for home for some sleep feeling in a much better position with the status of our app.
The next morning I implemented a few more improvements, adding support for the app user to type their own initial message which is read out to the call receiver. Also ensured the caller is announced correctly, using the salesforce user full name. Being a force.com canvas app, we can have a signed request passed to our application, which includes information about the authenticated salesforce user. We also added support for voice gender, adding a select field for the app user to choose whether their text messages are read out in a male or female voice. Oddly I didn’t spot a method to work out gender from the salesforce user we had access to, so had to add a select field for the user to specify. Added some final UI styling improvements to make it look a bit nicer. Finally, I wanted to try and get one more salesforce integration in the app, that is to save the completed call log back to salesforce. However, by this time we were so close to the final submission deadline I wasn’t able to finish it in time.
Luckily enough, and I think it was to do with our teams preparation, our demo actually worked. We actually got a round of applause when the transcribed message appeared on the screen. And as far as I can tell it was transcribed correctly! All our practice tests leading up to the demo had many transcription failures, which was worrying but I attributed it to the background noise. Likewise we were able to submit the text reply and hear it on the phone. In the end our app won third prize, which was probably fair considering the level of salesforce integration wasn’t great. I felt if I had managed to finish the last bit of the app and be able to save the call log back into salesforce, we might have come away with a higher score and maybe taken second place. Having said that, the other winning apps were great ideas and had a better business case use for salesforce. Overall, the hackathon was a fun experience, I’d do another one, but not right away. I certainly enjoyed a good Sunday night sleep.
You can try making your own Salesforce1 mobile app by registering for a developer account. You can also sign up for a Twilio account to try out some of the phone call techniques described here, you’ll be given a trial number which you can use to try out the service, everything should work, except for transcription, which must be paid for. You’ll also hear a demo message in all your calls until you switch to a paid number.