A good friend has said to me again and again over the years. "I don't see why I can't just serve up an entire web application right from the database using SQL." Granted not everyone loves T-SQL as much as he does.
We've all been been pushing more and more of the development process into the browser. I've talked about this in terms interactivity and putting the end user experience in the hands of front-end developers. Lately it strikes me that at bottom what we are doing is pushing the templating process into the client (and by templating I mean merging dynamic data with HTML markup).
The more projects I do this way the more I realize that there doesn't currently seem to be any standard or even really good way of doing client side templating. Recently I was doing a quick and dirty project that served up an html web page from inside of the CouchDB. Specifically I was doing this on the Cloudant hosted service, essentially giving my client a free place to run an application. I wound up using icanhazjs. I wound up having a template embedded in a script tag like so:
<script id="loans" type="text/html"><table id="template" bgcolor="gray" class="couch-grid"><tbody>{{#rows}}<tr bgcolor="white" class="data-row"><td>{{state}}</td><td>{{college}}</td>..{{/rows}}</tbody></table></script>
Then using the document.ready function to retrieve the data and merge the template.
$(document).ready(function(){
var $db = $.couch.db("loans_private");
var res = $db.view("app/by_state",{ success: function(data) {
var rows = [];
$.each(data.rows, function(i,o) {rows.push(o.value)} );
var html = ich.loans( {"rows": rows});
$('.container').html(html);
}, descending:true});
});
This all seems rather ass-backwards to me, especially having to wrap the template markup in a script tag. One can put the templates into external files, but either you wind up a big template for every html page, or you wind up with many small templates referenced from inside your HTMl page. javascript mvc works like this too, as does jquery templates which is no longer actively being developed. It seems to me that this breaks with the notion of the document which is still a guiding metaphor for the web. pure gets around this by working with existing markup and using jquery like DOM selectors to do the templating. It strikes me that there will be problems with naming conventions in the markup as designers and front end developers my have different ideas on how to use ids and classes. Plus we reproduce the Babel that we had on the server side with dozens or even hundreds of templating languages with huge or minor differences among them.
When I finally had a chance to look over the HTML 5 spec, I was keen to see what the authoritative solution would be. While I found many very good things in the spec: local storage, cross-domain messaging, "background threads"; I was very surprised to see no simple and universal way to handle templating. Did anyone else have the same experience? I for one would like to see a uniform templating language that worked naturally in the browser as soon as possible.
]]>A company operates their primary line of business application in a highly automated environment. One where it is as likely that the client for for the middle tier is a python or perl script as a browser based application. In the past they had made the mistake of creating a separate facade for each sort of client. PHP for the web application, perl and python libraries each hitting the database and reproducing any required business logic.
It's not hard to imagine the problems this led to. So the first thing I promised was that when we rewrote the app there would be a single interface no matter the client (including the UI). In 2009, it seemed reasonable to pursue a RESTful apporoach and indeed an extreme form of it since we decided that all interaction with the middle tier would be via http and JSON.
This approach had a number of results and consequences that I think are worth sharing in 2010.
By treating all clients equally, certain aspects of controller on the server dwindle, the part that knows knows anything about a particular presentation technology like Struts, Wicket or an asp page. The middle tier is a RESTFUL wrapper around EVERY application feature.
For .net applications we can use the WCF REST Starter Kit. For java applications there are a dozen excellent implementations of the JAX-RS (jsr311) specification. The point is that the service layer of my application stay essentially the same, no matter how we build the presentation layer.
As a manager and a developer, I think this is great. I'm always trying to make all my applications look the same no matter the technology stack. Using a common set of idioms makes it easier to support existing applications and to build new ones.
Rather than adding yet another layer of abstraction we make clear of a boundary between system components (what made this boundary ambiguous were the different types of "shim" code dealing with different presentation layer technologies).
What about the additional work needed to translate our domain objects to JSON, XML or some other client side format? JAX-RS transparently marshalls and unmarshalls several flavors of JSON and XML for my service layer. I write my service methods as I normally do, working with my domain objects. Clients can even negotiate the type of content returned by specifying an html encoding. True, marshalled domain classes have to be identified with the @XmlRootElement annotation and the REST URis are specified on the service interface (The WCF starter kit works in a similar fashion):
@Path("test")
@Produces("application/json"
@Produces("application/xml")
@Consumes("application/json")
public interface TestService {
@PUT
@Path("testput/{val}")
public void testPut(@PathParam("val")String val);
@GET
@Path("test-error/")
public void testError();
}
The service layer and the ways it is accessed is completely specified in one place and this seems like a very good thing to me. Java clients in the same JVM can call the service code directly. As an added bonus remote Java clients can also use a proxy system (you must have the source of the domain objects and the service layer interfaces) that allows for idiomatic and IDE assisted access to the services:
@Test
public void testStartPocessNoPipeline() throws Exception {
RegisterBuiltin.register(ResteasyProviderFactory.getInstance());
DeployService client = ProxyFactory.create(DeployService.class, "http://localhost:8080/ext-webapp/rs");
SkylineProcessWrapper pipelineresults1 = client.startAssetPipeline(UUID.randomUUID().toString(), "NONE",
"boss_animation", "something nice",new ArrayList<EntryWrapper>());
assertNotNull(pipelineresults1);
}
It's even better if I use grails. The jaxrs plugin automagically generates a RESTFUL service layer for all CRUD and listing, filtering, sorting behaviors for my domain objects, eliminating huge swaths of normally hand-coded boiler-plate service layer! The anti-scaffolding folks may be right when they complain that the CRUD screens produced by (G)rails aren't suitable for production apps, but I think the service layer generated in this fashion will be.
I think it becomes harder to write a good service layer though, since we really don't know exactly how the conversation with clients will go (this makes sense in the age of SOA and the mashup). We have to embrace an open-endedness which is difficult to do and that goes against many of our instincts as developers. And somtimes I wonder whether the RESTFUL grammar wll be rich enough to support all possible application semantics.
Test driven development (TDD) helped us. Our tests are the service layers first and best clients. We also prototyped a browser based interface using Ext JS. Our application is a work flow and job tracking system, and I think we managed to put together a middle tier that will support significant variation of functionality without needing to change very much. It helps that it's a wrapper around a mature and full-featured work flow library, JBoss jBPM. Wherever possible we pushed application variablity down into the work flow system.
I suspect that in coming years, the level of flexibility achieved in the service layer will be one of the key ways we distinguish developer excellence.
As the controller orchestrating the conversation with the user moves into the browser, client side coders will have to rise to the occasion by taking on the controller portion of MVC. They will always pass an XML or more likely a JSON model back and forth to the server via a RESTful interface. They may manipulate the DOM directly using javascript or use a rich client framework. I think their work will become more valuable, including monetarily, as they take on the job of fundamentally driving user interaction. Certainly they will be less frustrated and more productive as a single way of doing things they control takes the place of dozens of server side templating systems. But they will have to learn new tools techniques for orchestrating the conversation with users.
I've been very impressed with Ext JS, a client side RIA framework. It provides an excellent suite of UI components, but is even better at providing tools for putting the controller part of the application in the browser though its Ext.data.store construct and its ability to save state on the client. Plus, it's another one of those rare tools that just make sense, where you can try things and they just work.
I don't imagine swapping out presentation layers the same way as we sometimes swap databases for our Hibernate or JPA entities. In every case client code is going to have to know something about the business at hand. The users at the company I introduced in the beginning of this post are incredibly sophisticated. If a group of them doesn't like the way part of our web based interface works, they'll write their own. Our flagship human interface is written in flex and they are unlikely to replace it, but we've given them the tools to do so but in a safe and controlled manner. The innovations they come up will improve the application for everyone.
Certainly this "let a thousand flowers bloom" philsophy may not always be appropriate. But as we work this way, there'll be fewer and fewer times when a small change request for the way a client works will require rework all the way down into into bowels of an application. That's no small thing, especially in tightly controlled environments. This should include changes to the relational database, but that's the subject for another post dealing with the goodness of nosql databases.
]]>When I last posted, I was very excited about Apache Wicket as the way to develop line of business applications with a domain model, CRUD screens for maintaining the model, and in the most interesting cases, doing something else useful besides.
I still like Wicket. It has, as its web site says "a small conceptual surface area." It reminds me of python in that "You try something it usually just works." In many respects, though, Wicket seems to be at the wrong level of abstraction for the for the sorts of line of business applications described above. If your team is spending any time at all writing code to produce listing, filtering, and sorting behavior, not to mention creating CRUD screens and the back end logic for these operations, they are probably working at the wrong level of abstraction.
This is why I continue to move my clients away from relational databases whenever it makes sense to, and it seems to make sense to more and more. It's just harder and harder justify creating a bespoke database schema to support a content management system or line of business application when mega frameworks like Plone, SharePoint, Drupal or Joomla are available.
Nevertheless, sometimes you just have to write against a relational database. So I looked for a way forward among the frameworks that generate scaffolding for CRUD operations. I was productive, but uncomfortable with Rails. Ruby is swell, but there is a real lack of third party libraries as compared to Java. And so to the extent that an application might have to do anything besides maintain a domain model, the pickings might be slim. For example there are no more than two or three work flow engines available in Ruby. Whereas you can hardly count the full featured ones in Java. SOA is another area in which it is interesting to compare.
One way around this is with JRuby, running rails inside the JVM, which exposes the wealth of Java apis to Rails applications. But no one has pushed me down that path yet.
Recently I did a small project using Grails and was quite pleased. Grails uses groovy a dynamic language compatible with Java and is based on the proven technologies that I know and love well: Spring, Hibernate, SiteMesh, Maven, etc. It is configured via convention, but still produces all the artifacts I understand in case I need to override default behaviors.
Not only did Grails produce nice scaffolding code that was easy to customize to support some persnickety client demands, but it afforded several surprisingly nice features:
GORM which is a simplified wrapper on top of Hibernate, which let me define my domain objects and validation rules simply and elegantly:
class Wine {
static searchable=true;
Distributor distributor;
Importer importer;
Producer producer;
Region region;
Region subRegion;
String name;
String notes;
String internalNotes;
String vineyard;
Date dateCreated
Date lastUpdated
static constraints = {
name(blank:false)
vineyard(nullable:true,blank:false)
region()
subRegion(nullable:true)
producer()
importer()
distributor()
dateCreated()
lastUpdated()
internalNotes(nullable:true, maxSize:1500)
notes(nullable:true, maxSize:1500)
}
String toString(){
return name
}
}
That little bit of code gives me a domain object, database persistance, relations with it's dependent objects, validation, and lucene full text indexing.
Groovy gives me closures, functions as first class objects, easy creation of DSLs, and meta-programming. Adding a method to the Java string class is as easy as:
String.metaClass.intro = {len ->
return StringUtils.abbreviate(delegate, len) ?: ''
}
I get all the power of the Java ecosystem without the fustiness and lack of expressivity of the core language (no more getters and setters, ever!).
I still have the problem I complained about a few years ago: the problems that arise from mixing client side and server side markup in the same artifact. Whereas three years ago I thought "clients without code" was solution to this problem, whether using Wicket or something like GWT. I've come to realize that the problem was not having logic in the web page, but in the mixing of client and server side markup. In fact, for my last few projects I've been pushing most of the presenation layer logic into the web browser, taking advantage of javascript and JSON. This style of development already has acronyms: SOFEA (Service-Oriented Front-End Architecture) and SOUI (Service-Oriented User Interface).
In a future post I'll explain how this style of development has made my applications both simpler, more powerful, and even more flexible. Hopefully it won't take me three years to do.
]]>How many times does a simple change in presentation require getting a hold of the server side engineer to make the change even though it's trivial.
The problem is our primary artifact, the web page, consists of markup/code from three domains. The html/css markup that governs presentation, javascript code that can govern both client side interaction and server communication, and server side markup/code like jsp/asp/rhtml/velocity/freemarker.
Often this mix results in the most modern and up to date spaghetti code where it is difficult to determine what is happening and where (especially as an app evolves over time). Plus you have pages that won't render outside of a complex test environment, resulting in some of your team members flying blind when making changes.
After some initial resistance, a front end guy I work with recently picked up velocity templating. But I think this was an exceptional situation. To expect that he also pick up enough jsp, struts, tapestry,rhtml, etc., etc, etc. to render a model or at least not to &%@#-up my server side code, is neither reasonable or desirable (though it would make my life infinitely better on legacy projects). Carlos, are you listening?
To counter these tendencies, I've been moving towards a style of web development which aims at a clean separation of concerns between tiers/technologies/ and project roles. I call it "No code at all in the client!"
In this approach, the web tier contains nothing but html and css. The Wicket framework is a good example of this way of doing things. Here is the markup for a complex ajax page with lots of linked form elements:
<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" >
<wicket:panel>
<p>Grant Access</p>
<form wicket:id="accessForm" >
Select Department;<select wicket:id="selectedDepartment"></select><br/>
Select name:<select wicket:id="selectedUser"></select><br/>
<input type="submit" value="Add" wicket:id="ajax-button"/>
</form>
<p> The following accounts have been granted access to work on your behalf for your projects:</p>
<form wicket:id="grantedForm" >
<select wicket:id="grantedUsers"/>
<input type="submit" value="Remove" wicket:id="ajax-button"/>
</form>
<p>The following users have requested access to your account's projects, pending your approval.
Select whether to approve each account.</p>
<form wicket:id="requestedForm" >
<select wicket:id="requestedUsers"/>
<input type="submit" value="Approve Accces" wicket:id="approve"/>
<input type="submit" value="Deny Accces" wicket:id="deny"/>
</form>
</wicket:panel>
</html>
This page will render properly in any browser. Css styles be applied to it, and additional javascript functionality added, without any risk of breaking the server side interaction. The complete form functionality is specified server side, taken care of in java. Here's an example of adding a dropdown list to a form that updates the contents of another dropdown via AJAX when its selection changes:
departmentList.add(new AjaxFormComponentUpdatingBehavior("onchange") {
protected void aonUpdate(AjaxRequestTarget target) {
target.addComponent(UIHelpers.resetDropDown(userList,
userManager.fetchForDepartment ( (Department)departmentList.getModel().getObject())));
}
});
A more radical version of this approach is to generate all the output server side and display it on the client via embedded script tags (like in this example from the R2J feed from CNET:
<script src='http://i.i.com.com/cnwk.1d/html/export/r2j/News-19.html'></script>
If you follow the link, you'll notice that it returns a javascript snippet that renders an rss feed in html. Here's a part of it:
document.write('<div id="r2j-feed"><div id="r2j-title">
<a href="http://www.news.com/&tag=r2j">CNET News.com</a></div>
<div id="r2j-tdesc">Tech news and business reports by CNET News.com.
Focused on information technology, core topics include computers, hardware,
software, networking, and Internet media.</div>
<div id="r2j-head">...');
The HTML developer can style the emitted content and there's almost no danger at all of breaking the client-server interaction. The web page can be deployed on any web server (no need for Tomcat, Rails, or whatever) and it can get its dynamic data from any other server on the internet without the need to worry about cross-site scripting issues. Richer interactivity can be gotten by passing JSON messages back and forth with javascript. I've been increasingly using this style of development for clients with minimal IT expertise and infrastructure. If A full featured featured web app can be deployed on a $10 a month Go-Daddy account with the back end provided by one of my hosted servers or one in the EC2 cloud.
]]>Processor cycles are cheap, developer cycles are dear. Until you run the unit tests.
Love Ruby!
It's a little bit scary not being to lean on the compiler for things parameter type checking, but the Rubyistas assure me I'll get used to it. Being used to writing a lot of unit tests helps.
So I start my first Rails app. Loving Ruby! Happy to use the equals operator instead of endless getters and setters. Happy to use duck typing instead of extracting another interface when I want to treat things polymorphically. A little sad to have to learn yet another markup language for the HTML templates.
So I write some tests and I write some code (sometimes I do it the other way around).
Lo, as soon as I require '/../test_helper' and run my rake task, the testing harness takes nearly a full minute to initialize. This happens every time! In contrast, I have an asp.net app with 200 tests, half of which, regenerate the database schema between runs. Everything runs in well under a minute.
I've only see a few posts on the internet complaining about the Rails test fixtures. Using a statically typed language I'm used to running my unit test suite after almost every change to my code. With Ruby I only want to do this more often. A friend says that he likes Rails because it makes him do all the things he wouldn't do as a Perl hacker, separate concerns, write tests. Can it be that most Rails developers don't actually run their tests?
I found myself writing tests (designing my software) so as to avoid requiring '/../test_helper'. This doesn't seem good. What the language and Rails gave, the pokeyness of the test harness took away.
I see that there are ways around this: http://blog.jayfields.com/2006/06/ruby-on-rails-unit-tests.html. But us this something an average Rails developer will do? And eventually you are going to write tests that hit against the database, whether you call them unit tests or not.
It seems like a deal breaker to me, but no one else seems to have noticed.
]]>Maybe the problem is the embarrassment of riches. You can choose between a half dozen open source and proprietary frameworks for most everything a typical application needs to do: object relational mapping, creating web pages, consuming web services, templating. There's so much code out there that if you're writing a lot of your own, you're probably doing something wrong. But it certainly makes it difficult to come up with an acronym. In this post I'm going to describe a Java development stack that I think as productive as anything out there. As for an acronym, I'll probably proppose a algorithm for constructing them, but will leave the any serious acronym development work as an exercise the to the reader.
Spring and Hibernate have become truly indispensable to me in my development practice, enabling me to deliver ever more significant applications faster and with less and less effort. It seems like I'm in touch with the zeitgeist on this. I see scores of web-sites, blogs, and books touting "better, faster, lighter" Java. I even use them for my .net projects and there's more and more interest in spring and hibernate within that community as well.
At first I was skeptical about Spring. Inversion of control, composition, and declarative programming are things every competent developer eventually discovers as they enable you to write easily testable and extensible systems with a minimum of fuss. I guess I was jealous of my secret sauce (you can read about some of my reservations on the spring.net discussion list. However, in the last year I've been compelled by the excellent Hibernate and JDBC templates, easy to use annotation based transaction support, aspect oriented programming features, and now the ability to do asynchronous messaging. I can scarcely imagine creating a java application without it.
When circumstances allow, I'll choose Hibernate for object relational mapping. Hibernate just does its job well, especially with release 3 and the ability to directly annotate domain classes instead of creating xml mapping files. I needed little prompting to give up both entity beans and my home-grown persistence layer. Of course where there's no need to back a significant domain model with a relational database, Hibernate drops out of the equation as do any of the other java mapping tools I could have picked. Still, applications are so commonly backed by a database that an ORM component has to be considered as a core part of any java development stack. I think Hibernate has reached a tipping point in terms of popularity, to where it's the de facto standard for java ORM; much in the same was as Struts has been the standard for the web tier.
Most significant applications can benefit from a dose of asynchronicity. Nothing makes separation of concerns easier to implement on the back end. In addition to ensuring a reliable connection to external systems, JMS functions as an application service bus which, combined with the sort of workflows you can create with Spring, gives you the hooks you need to easily evolve an application over time.
There are many propprietary JMS implementations, but ActiveMQ is fast, free and supports clustering, peer networks, discovery, TCP, SSL, multicast, persistence, and XA transactions. It also integrates with scripting languages and crosses the .net divide via Open Wire.
Until recently I'd almost always recommend a full-blown J2EE server to my clients for two reasons. None of the open source JMS implementations seemed truly ready for prime time, so you'd use one that came with a proprietary server like WebSphere, WebLogic, or Oracle. Or you'd use JBoss. Plus it always seemed to me that message driven beans seemed like the most elegant wrapper around JMS (and the only part of the J2EE spec that seemed worth holding onto for my purposes). With the maturation of ActiveMQ and Spring 2.0 providing a nice wrapper around JMS we can eliminate the J2EE container for the vast majority of applications.
Why not use Tomcat? It's ubiquitous and free and perfornance is very good in the latest versions. If you don't like Tomcat, try Jetty.
This brings me to a crucial point. I develop primarily apps with a web gui, and if there's' one thing that's been horribly broken in the Java space, it's the web tier. I've dealt with the situation two ways. When I can, I run away from it and concentrate on server side coding where java seems to give me super powers. Still, as often as not, I'll inherit Struts application whose owners seem surprised when the only answer I can give them to an interminably long list of new features is "You can't get there from here, at least not easily." Even when I get to select the web tier technology I haven't found anything with the same power and elegance Spring and Hibernate provide me in their respective domains.
Things may have changed. A few weeks ago I stumbled on a development framework called Wicket. The hype sounded good. Wicket is "designed to handle one problem well: enabling component-oriented, programmatic manipulation of markup." It has built in support for AJAX controls and building JSR-168 portlets. The mailing lists are very active and it's becoming an Apache incubator project.
As a test I decided to port the web tier of a user preferences engine I've created from Struts-JSP to Wicket. The results were very encouraging. Its approach seems to be a cross between asp.net (with its code behind pages) and swing. Lines of code went from 1166 for the struts version to 409. What code there was in the Wicket version was more straight forward (primarily because it took a lot of work to translate the map backed forms I used in struts to the the data structures used by the engine). Here's an example of the code that renders a one of n multiple choice question as a group of radio buttons and that process the input:
public class OneOfNPanel extends Panel {
public OneOfNPanel(String arg0,QuestionnaireItem myData) {
super(arg0);
add(new FeedbackPanel("feedback"));
add(new Label("q.question", myData.getContents()));
add (new ResponseForm("form",myData));
}
private class ResponseForm extends Form
{
@SpringBean protected QuestionaireFacade2 myManager;
protected QuestionnaireItem myQuestion;
protected List<QuestionnaireAnswer> data;
private QuestionnaireResponseItem mySelection;
public ResponseForm(final String componentName,QuestionnaireItem anItem )
{
super(componentName);
data = anItem.getAnswers();
myQuestion = anItem;
mySelection = new QuestionnaireResponseItem();
List<QuestionnaireResponseItem> state = myManager.get(anItem.getId());
if (!state.isEmpty())
mySelection = state.get(0);
mySelection.setQuestionId(anItem.getId());
RadioGroup myGroup = new RadioGroup("group",new PropertyModel(mySelection,"responseId"));
add(myGroup);
myGroup.add(new ListView("list", data)
{
protected void populateItem( ListItem item)
{
QuestionnaireAnswer anItem= (QuestionnaireAnswer)item.getModelObject();
item.add(new Label("name", anItem.getName()));
item.add(new Radio("check", new Model(anItem.getId())));
}
});
}
public void onSubmit()
{
List<QuestionnaireResponseItem> questionAnwers = new ArrayList<QuestionnaireResponseItem>();
if (mySelection.getResponseId() != null)
questionAnwers.add(mySelection);
myManager.update(questionAnwers );
}
}
}
You have to be careful reading Wicket code, it leads you to make liberal use of anonymous inner classes (one of the ways it cuts down on the proliferation of artifacts in frameworks like Struts). In any case, I think my port was a pretty realistic exercise of the framework and I'm pretty stoked (and I'm not even a swing programmer). Next time I get to choose the presentation layer for a web app, I think I'll add Wicket to the stack.
I'm not saying the java stack is always the best solution. For simple CRUD applications backed by a database, whose schema I control Ruby on Rails seems ubeatable. In many situations I recommend eschewing a traditional database entirely. Zope/Plone works well for portal sites and uses the Zope object oriented data store. You can easily spin out new out new content types and get ui screens, persistence, workflows, permissions, etc for free. In a MS shop Sharepoint provides similar advantages.
But for significant applications, with a rich domain model, complex and variable workflows, complex integration requirements, and a need to evolve, Spring-Hibernate-ActiveMQ seem like an unbeatable base. If you're building a web then pile Wicket and Tomcat on top. Do whatever else needs doing by selecting from the incredible variety of Java libraries: for templating you can't go wrong with Velocity or FreeMarker, for XSLT try Xalan or Saxon.
So I might call an application built with Spring, Hibernate, ActiveMQ, Wicket, and running on Tomcat a SWATH application. But if I subsitute JettyI get WAIJS and so on. It seems impossible to get a handle on a stable acronym. All I can think of is to form the first or last "syllable" of our acronym from the Spring, Hibernate, ActiveMQ troika. So SHA or ASH, then to optionally pile additional letters on the front or the back pro re nata.
]]>