Friday, December 26, 2014

Time Zones on the Web

Dealing with time zones has come up several times in my career developing web applications. It is actually a VERY simple thing to take care of, but it can turn into an ongoing source of problems if you do it wrong.

When dealing with most point-in-time values (created at, updated at, expires at, scheduled for, etc), it's not necessary to store the time zone of the user.

The idea is to realize that the time zone is just a presentation/communication detail of an absolute concept: instant in time. Although, time zones are a very important thing to consider, neither the application nor the user actually care about time zone all that much.


Here are the 2 typical use cases:
  • The server. Needs an absolute way to store and compare instants-in-time reliably.
  • The user. Needs to see instants-in-time, always presented in his/her time zone.
We only need time zone for translating the instant-in-time as the server stores it, to a version the user expects. You can almost think of time zone as just a formatting detail. So, the solution:
  • The back-end only deals with datetimes in a single time zone: GMT.
  • The front-end only deals with datetimes in a single time zone: Local time (user) 
With that in mind, the way to take care of it is to push all time zone calculations to the border between front-end and back-end: The code directly talking to the web browser (the controller layer in most frameworks). Yes, I know. Technically it is still part of the backend, but let's forget that detail for now :/ It is the layer mostly concerned with the communication between the two. 

This keeps both the server and front end simple, as they only ever have to deal with one time zone. And it puts the time-zone code in one place: the controller layer.

Request-Response Process

Here is how the process should go: 
  • The browser obtains the time zone offset using Javascript and stores it in a cookie (or a header/queryparam sent with every subsequent request). 
After that, all requests to the server should follow this flow:
  1. The browser makes request to the server (includes tz-offset cookie automatically). Any datetime in the request is sent in the end-user's timezone (the one from the cookie), but the time zone is not included in the datetime (it's already on the cookie).  
  2. The server uses the time zone offset from the cookie to transform any datetimes found in the request to GMT. 
  3. From that point on, it's GMT all the way to the DB.
  4. When the server obtains data from the database, any datetimes retrieved are in GMT.
  5. As the server prepares the HTTP response, it uses the tz-offset value from the cookie to transform the datetimes in the response to the correct time zone for the user.
As an example, in the Play Framework when using Scala you typically create an implicit Format[DateTime] object to perform the serialization of a type to JSON (play-json using typeclasses). You can have this method take as an implicit parameter the actual HTTP request, from which it can read the cookie (or header/queryparam) with the time zone offset and serialize/parse the datetime accordingly. 

What about REST APIs that may be called by non-browser clients, you say? Well, it's not unheard of for them to also make use of cookies. But if that doesn't sound good to you, you can always include the time zone offset as an extra HTTP header or query string parameter instead of the cookie.


There are times where you DO need to keep the time zone and deal with it on the backend code AND possibly front end code: Broadcast times, office hours, airfare times, etc. The preceding advice does not apply to these types of times and you'll have to come up with an explicit way to handle them.

To summarize...

Here are some guidelines:
  • Set all servers (OS, Database, Web Server, etc) to GMT
  • In the database, store every datetime representing a point-in-time in GMT.
  • Write all your backend code assuming GMT.
  • Include end-user's tz-offset on all requests.
  • Don't burden the browser with time zone calculations. Once it reaches the browser, all times should be in the user's time zone already.
  • Recognize date/times that are not absolute instants-in-time and treat them different (office hours, broadcast times, etc). For these, you will need to keep track of the time zones in the database and your backend code will have to deal them explicitly.

Tuesday, September 2, 2014

Amazon AWS Async and Scala

Do you use Scala? how about AWS? Then you know that the Amazon AWS Java SDK comes with async variants of most of their methods. Unfortunately, they all return java Futures which are pretty useless. There's no easy or clean way to turn that java Future into a scala Future, since java doesn't provide the necessary methods. You can write some code to poll continuously, but that sort of negates the whole async thing.

Fortunately, their methods usually also include a version that takes an instance of AsyncHandler, which allows our code to remain truly async. But... what a pain... what we really want is a scala Future. One that you can map, flatMap, stick in a for-comprehension, etc. 

Lucky for us, there's a way to transform that AsyncHandler-taking-method into something more natural. And here it is:

class AwsAsyncPromiseHandler [R<:AmazonWebServiceRequest,T] (promise: Promise[T]) extends AsyncHandler [R,T] {
  def onError (ex: Exception) = promise failure ex
  def onSuccess (r: R, x: T) = promise success x
def awsToScala [R<:AmazonWebServiceRequest,T](fn: Function2[R,AsyncHandler[R,T],java.util.concurrent.Future[T]]): Function1[R,Future[T]] = {req =>
  val p = Promise[T]
  fn(req, new AwsAsyncPromiseHandler(p) )
This is an example of how you use it:
import com.gravitydev.awsutil.awsToScala
// regular amazon request (dynamodb for this example)
val req = new GetItemRequest()

// wrap the relevant '...Async' call with 'awsToScala' to convert the aws call from:
// someAwsMethod(request, callback) => unit 
// into a scala future
// type: Future[BatchGetItemResult]
val response = awsToScala(client.batchGetItemAsync)(req)

// do what you want with it now, asynchronously:
// response is a regular scala Future[BatchGetItemResult]
for (res <- response) yield {
You can copy it into your code, or if you want, you can get it from here:

Tuesday, July 23, 2013

Symmetry in REST Considered Harmful

When building web applications, it is often tempting to look for symmetry in the APIs for the creation and the retrieval of data. We are also nudged in that direction by the popularity of REST and its emphasis on treating data as resources. And by ORMs that encourage a single representation of an entity (the model) for CRUD operations.

Don't be tricked. That symmetry may exist on desktop (or device) applications where APIs are local, but isn't truly there for web applications. Give it up.

We are better off building APIs that are tailored to the specific act being performed. This goes back to the practice of making remote APIs coarse-grained (minimize chattiness, messages are expensive because of latency) and local APIs fine-grained (messages are cheap).

When we create/update entities, we need one representation that contains attributes (strings/numbers/etc) and references to other entities (ids). On the other hand, retrieval usually involves offering different shapes depending on what the needs of the client are (rendering a list, a show page), and responses usually embed a lot more data about related entities to prevent extra queries later (eager-fetching).

Naive ORMs that map tables to models are unsuitable for the task of serving data this way. We need tools that let you map data from the DB to the response in an more ad hoc way.

I see this capability in libraries like Anorm (even it has other issues), Play JSON (json-coast-to-coast), subset (mongodb library), etc. This was one of the guiding principles behind scoop (sql toolkit) and dynasty (dynamodb toolkit). Incidentally, a lot of it is powered by the concept of typeclasses.

Sunday, February 24, 2013

Scala Inferring Nothing

A lot of times i've run an issue with scala inference and type parameters:
abstract class Activity[T] {
  def apply (): T

object Count extends Activity[Int] {
  def apply (): Int = ...
// a method that needs to know both A and T
def run [T, A <: Activity[T]] (activity: A) = ...

Trying to use the method:
error: inferred type arguments [Nothing,Count.type] do not conform to method run's type parameter bounds [T,A <: Activity[T]]

But it works if you pass the type parameters explicitly:
A solution:
def run [T, A <: Activity[T]] (activity: A with Activity[T]) = ...

The rule seems to be that the compiler will usually infer only the parameters that appear directly in the parameter list.

Monday, January 16, 2012

Basic Scala HList

I've been trying to follow this excellent article:

But I couldn't quite make the jump from the part (a) to part (b).

To try to get to an intermediate mental step, I decided to try to implement an HList with just an append method. Here it is:

sealed trait HList {
  type Appended[L] <: HList
  def append[L](l: L) :Appended[L]

final case class HCons [H, T<:HList] (head: H, tail: T) extends HList {
  type Appended[L] = HCons[H, tail.Appended[L]]
  def ::[C](c: C): HCons[C, HCons[H, T]] = HCons(c, this)
  def append[L](l: L) = HCons(head, tail.append(l))

sealed class HNil extends HList {
  type Appended[L] = HCons[L, HNil]
  def ::[C](c: C): HCons[C, HNil] = HCons(c, this)
  def append[L](l: L): Appended[L] = l :: HNil
  override def toString = "HNil"
object HNil extends HNil

object test {
  def main (args:Array[String]) {
    val a = "Test" :: 24 :: HNil
    val b: HCons[String, HCons[Int, HCons[String, HNil]]] = a append "January"

Now that I got that working (some good practice, I suppose), I might be able to make sense of the original article.

Saturday, July 30, 2011

Eager Fetching and Mapping Entities with Scala

Note: This post assumes you are doing the sql-results to objects mapping manually. Which I find it is the best approach (we'll leave the reasons for another post).

Lets say we have an Employee entity that is related (in the database) to a few other entities.
case class Employee (firstName: String, lastName :String)
case class Department (name:String)
case class Project (name:String)

To model the relationship, we COULD create a field on the Employee entity that references the department and the project:
case class Employee (
  firstName :String, 
  lastName :String, 
  department :Department,  // <---
  project :Project         // <---
But when displaying a page it might turn out that we only need the employee info, not the department or project. When retrieving them from the database, we might choose to leave those null, but that will mean we'll get an NPE (at runtime of course: very bad) if we happen to reference one of the related entities on the page. We might also choose to use Option but that's not entirely accurate, since they are not "Optional" fields, we KNOW that for a particular page we WILL need, say, the Department to be there.

This is where traits and the ability to apply them at instantiation comes in handy.

As an example, let say I would like to display a list of employees with their respective departments. I can create traits that specify that an entity comes with a related entity:
trait HasDepartment {
  val department :Department
trait HasProject {
  val project :Project
Now, when rendering the table of employees with their departments, I can use the trait to specify the required entity:
def listEmployees (projects:List[Employee with HasDepartment]) = {
    {for (e <- employees) yield
        <td>{e.firstName} {e.lastName}</td>
The requirement only specifies that the Employee must have its related Department. If we try to display the related Project using {} we will get a compile-time error complaining about the missing "project" property right away (very good).

Now, how do we actually add the related Department to the Employee when querying the database? Like this:
// let's assume we are using some basic db wrapper, like anorm
val employees = => {
  new Employee ( row[String]("employee.first_name"), row[String]("employee.last_name") ) 
      with HasDepartment {
    val department = Department( row[String]("") )
It is now encoded on the type that this Employee entity has its related Department entity (eagerly fetched). You can now decide on a case by case basis when to include related entities, and have it checked by the compiler that you are only able to use them when they have been included.

As an extra example, here is the same code that actually requires that the Project entity also be there:
def listEmployees (projects:List[Employee with HasDepartment with HasProject]) = {
    {for (e <- employees) yield
        <td>{e.firstName} {e.lastName}</td>
And the code to map the employee from the database:
// let's assume we are using some basic db wrapper, like anorm
val employees = => {
  new Employee ( row[String]("employee.first_name"), row[String]("employee.last_name") ) 
      with HasDepartment with HasProject {
    val department = Department( row[String]("") )
    val project = Project( row[String]("") )

Tuesday, March 22, 2011

Play! Framework and Templating with Scala: XML Literals

As you know from the previous post, I have been developing using the Play! / Scala / Scalate / Squeryl combination as my platform. Everything has been working well, but I haven't been that happy with Scalate. I am not a fan of the syntax (ssp or any of the others), the fact that the play plugin is outdated (must compile from source), and that eclipse won't compile it and let me know of errors like it does for plain scala files.

This got me thinking, since Scala supports XML literals, there should be some way to take advantage of them for templating. I had considered it before but had dismissed the idea because:
  • It required some sort of compilation unit (class, object) for each template.
  • It was not designer friendly (see above).
  • I wasn't familiar enough with Scala to really try it.
Well, after some time has passed, it turns out I don't care about designer friendliness (not sure why I cared about it before, I've always gotten a PSD file that I end up turning into a templates myself) and I am more comfortable with scala now. So it was time to consider it again.

So I gave it a try and I am REALLY surprised at how well it works. And you get TONs of benefits: Compile time checks, auto-complete in the IDE, full power of scala, familiar syntax, etc.


Layouts were the first thing I needed. A way to create a widget that takes parameters (like html attributes) and that wraps around template code. So I started with an object to hold my layout templates. One method for each template. We can also leverage default values, multiple parameter lists, and by-name parameters to make the syntax for the client code extra nice:

object Templates {
  // a page template
  def page (title:String="Default title")(content: => xml.Elem) = {

  // a panel template, just as an example
  def panel (label:String="Some label")(content: => xml.Elem) = {
    <div class="panel">
      <div class="panel-label">{label}</div>

[Note: As you can see, the syntax highlighter in this blog is unfortunately not as sophisticated as the scala compiler when it comes to XML literals]

Now you can use these directly on your controllers:

import util.Templates._

object Application extends Controller {
  def index = 
    page(title="Welcome to my Page!") {
        <p>Some template markup</p>

        {panel(label="Dashboard panel")(
            Panel content

Loops and Ifs

Within your XML, you can use "for" and "if" as you would in plain scala:
{for (user <- users if user.isAdmin) yield
    {if (user.isAvailable) 
      <img src="available.png"/> 
      <img src="unavailable.png"/>

Intra-Application Links

Play! comes with a really cool way to go from a method-call-to-controller to a url (reverse routing), but there's no equivalent way to do that from scala. But it turns out it is really easy to create a method that will do that for you. I'll put it in a Utils object I can import on my pages:
import play.mvc.results._
object Utils {
    def Url (action: => Any) = new ScalaAction(action).actionDefinition.url    
We can now use it like this:
import util.Utils._ // import Url method and others

object Application extends Controller {
  def users = 
    page(title="Users page") {
        <a href={Url( Application.user(24) )}>Click to see user page</a>

  def user (userId:Int) = {

As you can see, XML literals can work great as the templating solution for a web app.