Agile Ajax

Grails and JSONP: How Easy is That?

For all of those Java developers casting longing glances at their buddies doing Rails development, there is hope. Grails, which has just celebrated its 1.0 release, is a Rails-like "convention over configuration" framework that aims to do for the Java world what Rails has done for Ruby. Instead of Ruby, the dynamic programming language of choice is Groovy, which compiles to bytecode (no Groovy to Java translation) and integrates smoothly with just about any Java code you may be using.

I'll stop hyping Groovy and Grails in general here; there's plenty of good informational stuff on the Grails and Groovy home pages. I will note that if you run off the tracks a little bit, you can find yourself reading through 500-odd line stack traces of Groovy, Spring and Hibernate -- there's room for some improvement there.

One of the many nice things about Grails is it's support for JSON and XML. Let me put together a simple example that shows off some of Grails' power.

 

HSE 1.1 - Hibernate, Spring and Echo2

Let a hundred flowers bloom, let one hundred schools of thought contend. -- Chairman Mao

Maybe it's not a good idea to have so many AJAX UI frameworks contending for our attention; it confuses users and has people sitting on the sidelines waiting for things to shake out. The same can not be said of AJAX application frameworks -- frameworks that provide you with the infrastructure to write full featured n-tier applications. In my opinion there aren't enough of those.

I've been watching the development of HSE -- Hibernate, Spring, Echo2 -- and it's already come together into something pretty useful. Right now it's more of a sample application that you can use as a starting point for developing your own application. If you haven't come across these various tools before, let me give you a quick overview of each of them in turn:

• Hibernate - an Object Relational Mapping tool that allows you to map Java Objects to RDBMS tables. There are a ton of these types of tools, both commercial and open source. Hibernate is so slick and featureful it's pretty much killed the commercial market here.
• Spring Framework - this is the one framework whose main purpose is to allow you to design better software. From their mission statement: OO design is more important than any implementation technology, such as J2EE. After using it, you'll wonder how you ever developed applications the old way. I strongly urge you to check this one out, even if you decide not to investigate HSE further.
• Echo2 - Java AJAX component GUI framework that allows you to write web applications the way you do Swing programs.

By combining these tools, the HSE framework already sports some useful features:

• User and group based authentication and entitlements/permissions.
• Input validation with AJAX alert dialogs.
• Sample UI for managing users, groups and permissions.

A short but promising list. I'd like to see it fleshed out a bit with things like a workflow facility, support for messaging and asynchronous processing (see Mule ESB for some ideas), and scripting (Groovy, anyone?). If you're a developer out there thinking of rolling your own AJAX framework, I'd like to encourage you to supress that urge and contribute to a GUI framework like Echo2 or an application framework like HSE. Don't reinvent the wheel when so many new widgets and tools are waiting to be invented.

HSE 1.1 - Hibernate, Spring and Echo2

Let a hundred flowers bloom, let one hundred schools of thought contend. -- Chairman Mao

Maybe it's not a good idea to have so many AJAX UI frameworks contending for our attention; it confuses users and has people sitting on the sidelines waiting for things to shake out. The same can not be said of AJAX application frameworks -- frameworks that provide you with the infrastructure to write full featured n-tier applications. In my opinion there aren't enough of those.

I've been watching the development of HSE -- Hibernate, Spring, Echo2 -- and it's already come together into something pretty useful. Right now it's more of a sample application that you can use as a starting point for developing your own application. If you haven't come across these various tools before, let me give you a quick overview of each of them in turn:

• Hibernate - an Object Relational Mapping tool that allows you to map Java Objects to RDBMS tables. There are a ton of these types of tools, both commercial and open source. Hibernate is so slick and featureful it's pretty much killed the commercial market here.
• Spring Framework - this is the one framework whose main purpose is to allow you to design better software. From their mission statement: OO design is more important than any implementation technology, such as J2EE. After using it, you'll wonder how you ever developed applications the old way. I strongly urge you to check this one out, even if you decide not to investigate HSE further.
• Echo2 - Java AJAX component GUI framework that allows you to write web applications the way you do Swing programs.

By combining these tools, the HSE framework already sports some useful features:

• User and group based authentication and entitlements/permissions.
• Input validation with AJAX alert dialogs.
• Sample UI for managing users, groups and permissions.

A short but promising list. I'd like to see it fleshed out a bit with things like a workflow facility, support for messaging and asynchronous processing (see Mule ESB for some ideas), and scripting (Groovy, anyone?). If you're a developer out there thinking of rolling your own AJAX framework, I'd like to encourage you to supress that urge and contribute to a GUI framework like Echo2 or an application framework like HSE. Don't reinvent the wheel when so many new widgets and tools are waiting to be invented.

OpenLaszlo Developments Afoot

One way that AJAX frameworks are going to be embraced is by providing lots of widgets. Another is by signing up other companies to develop applications using their framework. Nothing spells credibility than working software. To that end Laszlo Systems has announced that someone's going to be adding more oomph to their email software product built using OpenLaszlo:

Laszlo Systems, developer of OpenLaszlo, the leading advanced open source platform for building and deploying Ajax applications, and Goodmail Systems, creator of the CertifiedEmail system for trusted email, today announced that Laszlo Mail will support the recognition and presentation of CertifiedEmail messages. Offered for license to businesses and communications service providers, Laszlo Mail delivers the functionality and responsiveness of desktop email without requiring any client software installation. Now with CertifiedEmail, Laszlo Mail will have the added benefit of protecting users from phishing and other forms of email-based fraud.

Hmm, the former Flash product is now "the leading advanced open source platform for building and deploying Ajax applications." Last time I looked at their source tree it didn't do DHTML yet.

The Hazards of Exposing Business Logic on the Client

Via Ajaxian we get an object lesson in the dangers of exposing business logic in the browser:

Beau Hartshorne of Snipshot (formerly Pixoh) says "massive chunks" of Cellsea code are identical to Snipshot. "This is not an accidental inspiration. Check out the cropping code, the resizing code, and so on. We've also noticed that portions of their website are also stolen directly from ours ... We are contributors to MochiKit open source project. However, the code in question is proprietary and was taken directly from out site."

Can I say "I told you so?" I've blogged about the danger of Ajax and Leaky Business Logic before. What is the danger here and the lesson the be learned?

• Don't put your crown jewels in the browser. See if you can't lock a fair bit of your business logic on the server side by using a server-side component framework like Echo2 or ZK.
• If you're going to deploy meaningful applications in Javascript, do what people have been doing with script source code for decades: obfuscate. You can do something simple like renaming all of your variable to one or two character names, or you can use a code generation framework like GWT or Morfik that produce unreadable code from the beginning. Be forewarned that code generators can be vulnerable to decompilers -- in short, if someone knows how GWT generates it's code, you could possibly reverse the process and produce the original source code.
• Build in anti-theft mechanisms. This could be something as simple as a method that checks to see that the url the application is running on is the correct one, otherwise display a nasty message. You could make this as tricky and complicated as you like, all the way to encrypting big chunks of code with the website url and only decrypting them at runtime.
• Hold out for "byte compiled" Javascript.

A combination of all of these may be what we end up with. I'm not sure I'd want to run any script in my browser, though, that I can't understand. Still, you would hate to be the victim of code-theft as has apparently happened to Snipshot.

How similar are these two programs? Well, Cellsea offers a bunch more functions, but they do look very similar, both in terms of the UI and the underlying code. The original Snipshot is here and the knock off here. You be the judge.

If you are really hungry for a good AJAX image editing app, the best of the bunch of them may be Phixr, which gives you preview, the ability to marquee select for certain operations, etc. Slick, even if the UI is a bit jumpy.

Wanted: A Javascript Source Code Search Engine

 

I use quite a few search engines in my day-to-day work. I expect you do too. There's one that I use quite a bit to see what's happening in the Open Source world. No, it's not Freshmeat. It's Koders.

Koders.com allows you to search the source code of thousands of Open Source projects. It allows you to filter by language (and license). That's really very handy when your looking for coding examples. It also let's you see when and where someone is using a particular Javascript library in their project. Looking for Dojo and Scriptaculous, it is no surprise that Webwork and Rails show up.

Now endless hours of fun can be had with the Koders search engine, but what I'd really like is a search engine that allows me to search all of the Javascript on the web. It's out there, it's readable, it's searchable. Surely google or Yahoo or whoever already slurps in the Javascript in their crawls. Why not add a new google engine for searching Javascript?

To anyone looking for examples of Javascript or AJAX, the utility of such a search engine is self evident. It might make corporate IT managers a little nervous to have all of their logic flapping in the breeze. Let that be a lesson to not put your business logic in the UI.

If anyone know of such a search engine or of any hacks or tricks to make existing search engines give up their secrets, please let me know. Passing xmlhttprequest filetype:js into google gives a less than satisfying result.

Update 1: Building such a search engine shouldn't be all that hard. The pieces are already out there. See the Heritrix web crawler that the Internet Archive uses for it's work.

Steps to AJAX Mastery

No, I wouldn't call myself an AJAX master, but, in the words of Laurence Peter, "A man doesn't know what he knows until he knows what he doesn't know." I definitely know what I don't know. And I do know a little (I've been around long enough to have written LiveScript, so I'm old in Internet years if nothing else), so I thought I'd share with you all the Javascript, CSS, XHTML and XSL reading I've done (and haven't done). Maybe you can share your own favorites in the comments.

Yes I've been a big proponent of Component GUI's and writing just Java in developing AJAX applications. Stay in one language and don't mess with CSS, Javascript or XHTML. Reuse the components that others have already written. That way lies productivity. There's only one problem with this approach: who is going to write the widgets and the framework in the first place? Someone has to write Javascript and CSS and XHTML, otherwise nothing will get done. As Billy Preston sang, "nothing from nothing leaves nothing."

So if someone has to write all those widgets, why not you? You know a little Javascript and have written a XPath expression or two. And your girlfriend really liked the CSS you put together for her homepage. So what's stopping you? Well, the truth is that having a passing familiarity with Javascript and the other technologies that make up AJAX is just not enough. Not even close. Your knowledge of Java may actually hurt you in this regard. You have to know this stuff cold to make it sing and dance.

My approach to learning any programming discipline has been as follows:

1. Read books, the right ones.
2. Read code, lots of it.
3. Write code, lots of it.

The Right Books

Where to start? Picking up a tome like Ajax in Action seems like a good place to start, but it's really just a crash course in those fundamental technologies you'll need, sort of like skipping over Calculus to take Algebraic Topology as a freshman, or, for those non-math geeks, trying to get a job in Paris after only one lesson in French. You need a better grounding.

In my opinion, the right place to start is by understanding the browser. That's because Javascript has two parts to it: Core and Client-Side. Client-Side is all of the browser specific stuff, so if you try to learn Javascript without understanding the browser, it's just jibberish. Get to know all you can about CSS and XHTML and XSL before you tackle Javascript and you'll save yourself a lot of pain.

I recommend starting with XHTML first, then move on to CSS. (The problem with some of the better books, as you'll find, is that they're a bit long in the tooth, i.e. they have publication dates from as far back as 2001.)

None of the books that I learned on for HTML/XHTML are still relevant. The fifth edition of the O'Reilly book HTML & XHTML: The Definitive Guide is so out of date and focused on old browser quirks that it's a sheer waste of time. My instinct is to stay away from something like the CSS and XHTML combo Head First book. At 658 pages, you'll be well into next year by the time you've read it all. As you'll find out, you've got plenty of reading to do. Plus the whole Head First series is annoying to read. Still, I'm not sure I have any better suggestions. Until AJAX came along, browser technology was considered uninteresting, so nobody published in that category.

 

For CSS I'd recommend two books: Cascading Style Sheets: The Definitive Guide and CSS Mastery: Advanced Web Standards Solutions. The second is a more advanced text, so read it after working through the first. You may also want to supplement your reading with CSS examples from the CSS Zen Garden. Also check out Open Source Web Design for some 1600+ examples. Finally, Position is Everything is likely to have more recent updates on browser quirks than any book you are likely to buy.

For XML/XSL, I'd recommend an old text (originally a Wrox book, now put out by APress): Beginning XSLT. Get the first one, not the 2.0 book. The examples in this book are not for the browser, so you'll also need some more info on how to do it in Javascript. For this, look to Professional JavaScript for Web Developers. Chapter 15 is probably the best intro to the topic in print.

Finally we've arrived at Javascript, for which I have two recommendations. The latest edition of the JavaScript Bible by Danny Goodman is still the definitive work. The other book is the ancient JavaScript: The Definitive Guide by David Flanagan. The latest edition is from 2001, but it really is the only book that deals with Javascript the language.

Only now that you've swallowed these tomes whole are you ready to move on to the Ajax in Action book. All those bits of jargon that were confusing before are now old hat. This book will fill in some of the missing pieces around XHR that those other Javascript books left out.

Now I've left out lots of other books that are useful and should probably be required reading. You should probably bone up on OOAD, an Operating System or two, a server-side development language, networking, etc.. My intent here was to give you a minimal set of books, not an exhaustive one. Even the books that I have recommended could be considered to be exhausting if not exhaustive. Anyone keeping score will have noticed that these books clock in at over 5700 pages. That's a lot of reading. That huge page volume, however, is simply a reflection of the fact that you are learning four different languages and combining them all together. Maybe you don't have to read all of those page, but even half that number is a commitment to many nights at home. I wish I could recommend small 100 page books on each subject that are more than simple "Hello World!" pamphlets. If anyone is aware of any, please let me know.

Reading Code

Books will only tell you so much. To really get a feel for how things are done -- best practices, so to speak -- you have to read code. Preferably good code. Avoid generated code like that produced from Tibco GI or GWT. Stick to hand-coded Open Source stuff like Prototype, Scriptaculous, Dojo and the like. At this point you'll probably know enough to decide for yourself what to read. There's a good book on how to read code, but I'll spare you another tome.

Make sure to get some tools so you can read and debug the code. I personally like the Firebug for spelunking around the DOM and Javascript, but there's a whole list of other tools available over at ajaxpatterns.org.

Writing Code

Now, five years later, you are ready to start writing some code. Use what you've learned. Good luck.

AJAX Java IDE Support - Roundup

I've had it on my list of things to do for some time to round up all of the AJAX Java IDE's. That's what I've tried to do in this post. The category isn't all that developed, as it turns out -- though there are a few impressive tools like TibCo GI. Good tooling is often the last thing to come along for a new technology and AJAX is no exception. 

The tools below fall into two categores: those that are plugins for the Eclipse IDE and those that are standalone tools. Most of these IDE's tie you to a particular framework, which is kind of a taste of what life would be like if we had 10 different desktop operating systems for which to develop.

• EchoStudio 2 - Commercial. Eclipse plugin that allows you to build component trees, preview the UI, debug the application, etc. Not WYSIWYG, i.e. no drag and drop page layout.
• Tibco GI - Commercial. Free for development and publicly available web sites. Eats its own dogfood, i.e the IDE is implemented in itself and runs in IE. Is WYSIWIG and pretty slick.
• Google GWT - Free to use for personal and commercial purposes. As for IDE integration, there's mostly just an Eclipse project generator and a "hosted mode" runtime. Being able to debug Javascript as Java in an IDE has to count for something, though, which is why I've included it.
• Morfik WebOS AppsBuilder - Commercial. Freestanding IDE. Support several source languages including Pascal, Java, C# and VB. Drag-and-drop, WYSIWYG design. The behavior of the GUI designer is a little awkward. For example, right click doesn't give you the ability to cut and paste, etc., necessitating a roundtrip to the window's menu. Doesn't look like they have a whole lot of widgets in the evaluation version. A bunch of ther stuff thrown in, like DB integration, PDF reporting, etc.
• JoyiStar Juno - I wasn't able to get this to work for me (RAR file...I didn't try very hard).
  
• Eclipse ATF - Open Source Eclipse Plugin. Really more of a toolkit for building other AJAX IDE plugins. Starts by combining Dojo and Zimbra Toolkit. Very early in it's development. From the project docs: ATF enables support of DOM browsing and JavaScript debugging by using Mozilla XULrunner to embed the Mozilla browser component (Gecko) in the Eclipse framework.
• MyEclipse AJAX support - Commercial Eclipse Plugin. Mostly just Javascript editors and debuggers. Useful, but not at a high level of abstraction.
• Backbase - Commercial with a community edition. Eclipse plugin. Right now provides editing support for Backbase files. A WYSIWYG tool is under development.

So far, the only real WYSIWYG contenders out there are TibcoGI and Morfik. Everyone else is just providing at most widget trees and property editors with preview. (If I've missed any IDE's or have made any obvious errors, please let me know.)

IDE's are nice to have and can ease many a developer's introduction to AJAX, but it's no substitute for a good grounding in XHTML, CSS, Javascript and a fundamental understanding of software engineering. Don't think that these IDE's are going to make you a bullet proof developer. In the words of Niklaus Wirth, ...the woes of Software Engineering are not due to lack of tools, or proper management, but largely due to lack of sufficient technical competence.

Sprajax? Security Scanner for AJAX

I came across an open source AJAX security scanner called Sprajax. From the Denim Group's press release:

Sprajax is the first web security scanner developed specifically to scan AJAX web applications for security vulnerabilities. Denim Group, an IT consultancy specializing in web application security, recognized that there were no tools available on the market able to scan AJAX. AJAX allows web-based applications a higher degree of user-interactivity, a feature with growing popularity among developers.

Denim Group developed this innovative tool that will revolutionize security assessments by providing a more thorough diagnosis of security vulnerabilities within the AJAX code that other web security scanners are not designed to read. The software then produces a report of possible weaknesses for developers to remedy.

It's Open Source, so I decided to download it and see what the hype was about. First, the application is written in C#. Second, you will need SQL Server to get it to run as the results are stored in SQL Server and are pulled into reports using stored procedures. Finally, the application doesn't do all that much.

The application takes an application URL as a parameter and has three main functions.

• You can "footprint" the application, i.e. scan it for Javascript files and web services.
• You can "fuzz" the application, i.e. pass garbage into the web services to see if they will barf.
• You can display the results of the previous two functions.

Essentially, the footprint function spiders the application, detects the framework used, finds Javascript files included, and discovers web services used. The fuzz function uses those discovered web services and passes some garbage into them and sees if they returned error conditions. There are just a few problems with this tool, however:

• The tool only detects the Atlas framework. No Dojo, DWR, etc.
• It only detects SOAP web services used by the Atlas framework. No REST, no framework specific calls.
• It doesn't scan the Javascript files for XHR calls, another place to find calls back to the server.

The fuzz function only tests robustness and in a fairly limited way, i.e. will the app go belly up when you pass in garbage data. Typically, security scanning tools test for known exploits of web application frameworks, a much more useful approach.

Finally, consider that many AJAX frameworks make use of dynamic XHTML and Javascript to drive content and navigation. This tool just sucks in the content and looks for URL's. It wouldn't find even one link in an Echo2 application, for instance.

Overall, there isn't much here. It has the feel of something that a journeyman programmer put together over a weekend. I applaud the Denim Group for releasing an Open Source tool, but the hype in their statement is wildly inaccurate. You can likely write something in Perl in a few hours that would do all of this and more.

MVC and RIA - Learning From Desktop Apps

Whether you're writing AJAX applications using enhancements to existing webapplication frameworks or adopting the new component GUI approach, you as a developer are faced with a new granularity of events. Where before you had the monster postback, now you must respond to mouse clicks, key presses, menu selections, etc.

On top of this, with a single page interface, you may have many more controls on the page. This brings on the problem of context, i.e. that not all of these controls are appropriate for a particular context such as when a particular part of a table or list box is selected or when the focus is on a particular window.

The above just scratches the surface of the sorts of challenges that you'll discover writing RIA's with AJAX technologies. The good news is that there are already solutions in place since these are the sorts of problems that desktop GUI developers face all of the time. For example:

• Enabling and disabling menus and other controls can be done using the State pattern. A collection of state objects would be used to enable and disable the controls as appropriate.
• Many of the component GUI frameworks implement an Observer pattern for event management. UI events come in now in little bites instead of one huge request and can be consumed by listeners that wait on a specific component.
• MVC needs an upgrade when you're dealing with big heirarchies of components. Combining the Composite and Chain of Responsibility design patterns, you get the Heirarchical MVC (HMVC) pattern. (If you've worked with WebWork you may have run across the "Pull HMVC" pattern. That heirarchy refers to the model, not the whole MVC.)
• The Command pattern is often used to implement undo.

That's just a sampling of desktop GUI solutions you can make use of. What can you do to expand your solution space for RIA's? Read lots of code. Find as many good open-source desktop GUI's as you can and read their code. You'll find ways of doing things that don't resemble the typical webapp approach. Here are a few to get you started.

Resources:

• The Spring Rich-Client Project. Based on the Spring Framework. Used for buidling desktop GUI's. Read the code and get a sense for how to build your RIA framework. Good source for ideas

• HMVC framework and tutorial. View/Implementation independent. You can plug this into your own apps or just learn from it.

• The Scope Framework. The bad new is that this is a desktop GUI framework based on Swing that hasn't been updated since 2002. The good news is that this is a great place to read some code and learn how to develop component GUI applications.
• The HMVC Framework from Crionics. Again, no longer maintained, but a great place to read some code.
• Tutorial (in C#) on implementing undo with the Command pattern.
• A discussion of the Presentation-Abstraction-Control (PAC) architectural pattern. This one is just as old as the MVC pattern, but since the MVC pattern was intended for simple interfaces it was also simpler to implement. Thus MVC won out even in cases where PAC is more appropriate, such as with more complicated UI's.

Update 1: Oliver Steele has an excellent overview of MVC as it exists in the desktop, webapp and RIA worlds. He points out why non-RIA, i.e. traditional webapp programming is so tough:

That complexity includes the real world application of what are still research topics in academia: staged programming for multiple-target code generation (Mac IE, Windows IE, Netscape, Mozilla, Safari, Opera), process migration, and (usually manual) CPS conversion to maintain program state across pages.  No wonder web programming is hard!

Well worth a look.

Stomping out the Misconceptions

A reader pointed out this blog entry from Infoworld, Mercury: AJAX has its drawbacks. It's from the middle of April, but it is still worth responding.

"AJAX is incredible where people are starting to adopt it and it immediately causes a lot of problems because it's not very structured," said Rajesh Radhakrishnan, vice president of Application Delivery at Mercury. Several Mercury executives met with InfoWorld editors at Mercury offices in Mountain View, Calif. on Tuesday morning.

"We've seen tons and tons of problems," with AJAX, Radhakrishnan said. In testing for functionality and regression, Mercury has seen an increased number of regressions in AJAX, said Radhakrishnan.

As a workaround, Radhakrishnan suggests using AJAX for the cutting edge part of UI development, to enable interactions between the client and server in which the server is able to respond to client requests later. "For the rest of it, you don't really use AJAX,""Radhakrishnan said.

It is precisely using AJAX for the "cutting edge" parts of a UI that causes the problems. Calling an architectural principle like AJAX "not very structured" betrays an ignorance of the topic. It's like calling web service "not very structured" because people are using raw java.net sockets to do everything.

This post is from April of 2006, not 2005 when this sort of comment would have been excusable. Now there are several stable intermediate forms such as DWR that allow for fairly structured development of AJAX solutions on top of existing webapp frameworks. Further, there are already some more advanced forms such as Tibco GI, OpenLaszlo, ZK and Echo2 that allow for development of sophisticated desktop-type apps.

Mercury may be trying to discourage folks from developing AJAX apps until they've had a chance to update their testing software to keep pace. I suggest that they work harder on their next release instead.

Struts and the Tension Headache

I'm spoiled. This past week I had to go back and revisit a Struts application I developed back in the early days, before Hibernate, before EJB's. I found myself developing an anger headache as I bounced between the struts-config.xml, the web.xml, the jsp's, the home-brew ORM layer, and the various validators, actions and pieces of business logic. Even using an IDE and with well documented code, it was a real pain to make just a handful of modifications. Each time I made a change, it seemed that I had to modify three or four files to make it work. A little XML, a little JSP, a little Java, a little SQL. Oiy!

Why did this stuff make me so angry? Hadn't I designed my code in layers to avoid all this mess? Well, it was actually pretty painless for a struts application. The problem is that struts, and most frameworks, expose too much of the plumbing and force developers to repeat themselves in several places and in different languages and formats.

Why didn't I just accept that this was the way things were? Because I'd spent the last 7 days working on an AJAX newspaper interface using Echo2, and I hadn't spent one second writing HTML, Javascript, XML plumbing or SQL. Just Java. And now here I was back in the bad old world of the web worrying about sessions and postbacks, repeating myself over and over again in different languages.

Yes, things today aren't as bad as in the early days of struts. Hibernate and Spring have made things easy on the ORM front, so I write little or no SQL. Web application frameworks like Webwork and JSF are moving toward components and a higher level of abstraction. But the problem I have with their MVC is that all three really should be realized in Java, not a mix of different languages. That's the only way to stay DRY (Don't Repeat Yourself).

So, why isn't everyone embracing this new, pure Java (or C#, or Ruby, pick your poison) way of writing apps?

Paradigm Shift
People tend to overuse the term "Paradigm Shift", which was coined by Thomas Kuhn back in 1962 in his book The Structure of Scientific Revolutions. What does it mean, Paradigm Shift? It happens when a scientific theory can no longer explain the facts, i.e. when enough anomalies have cropped up to invalidate it. When this occurs, it isn't just the theory that is discarded but the whole worldview from which is comes. From the Wikipedia:

When enough significant anomalies have accrued against a current paradigm, the scientific discipline is thrown into a state of crisis, according to Kuhn. During this crisis, new ideas, perhaps ones previously discarded, are tried. Eventually a new paradigm is formed, which gains its own new followers, and an intellectual "battle" takes place between the followers of the new paradigm and the hold-outs of the old paradigm.

Yes, we're not fighting about Ptolemaic versus Copernican cosmology and if the latter meant that man was no longer the center of all creation. In our case the paradigm is not how which questions we ask and what experiments we do, but rather how we build web applications now that AJAX has come along. The worldview is not whether the planets revolve around the Earth or the Sun, but what exactly web applications can do. The battle is between those who want to build web applications with the old methods -- sort of a souped-up struts with AJAX -- or with the component GUI approach.

I for one am on the side of the component GUI's. I can't stand the tension headaches.

Ajax and Leaky Business Logic

I gave my presentation last evening to a welcoming Chi2 audience. The attendees were a good mixture of developers, designers and business folks. They sat through my rantings on why I thought Rich Interaction Applications (RIA) in AJAX should be developed with server-side component based GUI's, though, in a pinch, I could see the argument for client-side component based GUI's that use the back-end as a dumb data web service. Then I showed them what 150 lines of Java code could do in the Echo2 framework with my special "why would you do things this way" stock quote app. Lot's of dragging and dropping ensued.

I'll put the presentation and the demo up when I get a chance.

One of the more interesting and probing questions after the talk was about business logic leakage, i.e. how does a server-side component GUI (SGUI) keep you from leaking business logic? The answer is simple: with the SGUI, the only "logic" that makes it to the browser is "turn this area blue," "move this button over here," and "make this element draggable." While I suppose it is possible to leak business logic in this way, I'd have to say that the possibility is pretty remote and esoteric, akin to sending smoke signals.

In the pseudo hand-coded world of traditional webapp development, you are dependent on developers following good practices and not making mistakes, such as leaking the secret sauce in their Javascript validation logic. At least you hope they do. If the last 50 years of software development practice have taught us anything, it's that hope is not a plan.

There are two kinds of information leak -- what and how. "What" is your customer data, your orders, your accounts receivable, your employee database. The "how" is your secret sauce, the special way of pricing or quoting or doing that gives you a competitive advantage. Reports can leak the "what," and many businesses can survive a leak of quite a bit of "what," but leak the "how," your secret sauce that makes you money, and you're cooked. Javascript can leak the "how" if you let it.

Michael Mahemoff feels it's sometimes OK to include business logic in the Javascript:

I realise there are plenty of problems with Javascript business logic, and it’s certainly not always applicable. Those problems include:

• Security - Exposes business logic algorithms and enables them to be changed.

That first bit applies to most all valuable services that a company might provide. If all you're providing is data, then no business logic would be leaked. If you are performing some sort of business logic on the data, it would have to be something pretty trivial to still have value and not be a divulging of the crown jewels. Can a business consist of a bunch of ultra trivial, obvious logic and still be competitive? Probably not. In short, I can't imagine the circumstances where that first item wouldn't just preclude exposing most any business logic.

X Windows and Ajax: Two Kinds of Display Servers

X11 is a client/server display system that realizes user interfaces by sending asynchronous messages back and forth between a display server -- that manages a display, mouse and keyboard -- and clients such as xterms, Firefox, GIMP, etc. Some of the drawing, windowing, events and other operations that the X protocol supports look suspiciously like some of the things flying around in Ajax these days.

Now there are some substantial differences between X and Ajax. For one, X clients (the equivalent of the http server) open up persistent connections to the X server (the equivalent to the browser), while the 'Ajax server", aka the browser, has to poll the "Ajax clients" (the apps on the http server) for messages. X is typically used in a LAN rather than a WAN setting, for this reason. Persistent connections usually don't hold up very well in a WAN context. Also, there's no embedded scripting language like Javascript in X.

One definite similarity is that both X and Ajax don't have any widget standards. In X you've got several different widget sets -- Motif, Qt, Gtk, etc. -- while in Ajax you've got a few early frameworks with their own widgets.

So, what's to be learned from all of this? First, while user interfaces may well head in the component GUI/widget directions, we'll likely have quite a number of different widget sets in the long run. Second, there are a few design lessons to be learned from X Windows. For example, the creation of windows with a specific look and feel is controlled by a window manager abstract factory.

Some of these design solutions may not be appropriate for Ajax, but I think it's worth mining this trove of design solutions. Might we some day have an Ajax desktop that unifies multiple independent Ajax applications?

Update 1: Bill Scott makes a similar point (though much earlier -- more coffee, I'm sure).

Jetty 6's Continuation Mechanism for Ajax

I've touched on the topic of updates and asynchronous processing before. My preferred method of performing updates between the browser and server is via a polling mechanism that returns quickly. An alternative is to open up an XHR connection and keep it open to wait for a response (or a timeout). I don't like this method because it is wasteful in terms of sockets and threads; also, it is likely to stress stateful firewalls, load balancers, etc., and may break in lots of client environments.

Nevertheless, if you want to keep a connection open for notification initiated by the server, this is the way for now. And the Jetty 6 server has at least addressed the thread issue with Continuations.

Behind the scenes, Jetty has to be a bit sneaky to work around Java and the Servlet specification as there is no mechanism in Java to suspend a thread and then resume it later. The first time the request handler calls continuation.getEvent(timeoutMS) a RetryReqeuest runtime exception is thrown. This exception propogates out of all the request handling code and is caught by Jetty and handled specially. Instead of producing an error response, Jetty places the request on a timeout queue and returns the thread to the thread pool.

When the timeout expires, or if another thread calls continuation.resume(event) then the request is retried. This time, when continuation.getEvent(timeoutMS) is called, either the event is returned or null is returned to indicate a timeout. The request handler then produces a response as it normally would.

Sockets are still consumed, though. Hopefully the next servlet specification will address some of these issues. Until that time, this may be a good workaround.

Still, my preference is to keep everything except for the display logic on the server side, and that includes handling complex communication with async processing.

Update 1: ActiveMQ can make use of Jetty 6's continuation mechanism.

Update 2: Greg Wilkins has some more extensive thoughts on using Jetty 6 to scale Ajax apps.

Webtop - I Already Hate It

A reader emailed me this old article from CNN Money discussing the concept of the webtop, i.e. that

A killer app no longer requires hundreds of drones slaving away on millions of lines of code. Three or four engineers and a steady supply of Red Bull is all it takes to rapidly turn a midnight brainstorm into a website so hot it melts the servers.

What has changed is the way today's Web-based apps can run almost as seamlessly as programs used on the desktop, with embedded audio, video, and drag-and-drop ease of use. Behind this Web-desktop fusion are technologies like Ajax (asynchronous JavaScript and XML), Macromedia's Flash, and Ruby on Rails. We'll spare you the technical details; suffice it to say that these technologies are giving rise to a new webtop that may one day replace your suite of desktop applications.

I already hate the term "webtop," though I've used it myself in the past. The article goes on to discuss the various webapps that are starting to move into areas once reserved for desktop applications, then lists of a few noteworthy apps, such as 37signals campfire chat client.

The conclusion? We'll all be doing our word processing over the web from now on.

Now I like a little bit of breathless hype as much as the next guy, but this is over the top. Yes, webapps are going to change, but there are certain limitations to the medium that will make the webtop a much tamer place:

• Reliability and performance - why aren't most desktop apps in corporate environments served up as client/server apps? The technology is there; the kinks of client/server have mostly been worked out. License sharing could save tons of money. The benefits of reliable, centralized storage and ease of collaboration seem pretty obvious. Yet the most we see is networked storage of documents. The reason? Even on a corporate network, performance and reliability are not high enough to make client/server computing for desktop apps attractive.
• It's more than just CRUD on speed - even after the widespread introduction of the WIMP (Windows, Icons, Menus, Pointing device) environments in the 80's, it took a while for programs to mature beyond GUI versions of Lotus 1-2-3. We didn't see precursors to Photoshop until a few years after the introduction of the Mac. Most webapps today are still glorified CRUD (Create, Read, Update, Delete) engines. Writing the more substantial applications will take a good bit of work, not just a few cans of Red Bull (or Jolt!, sniff).
• Writely ain't Word - more like Wordpad. If Writely was out as a desktop app, it wouldn't get a second look. Yes, there is a need for a Word-compatible, easier, less bloated, free word processor, but they never seem to make it. Yes, a web based word processor that saves your work more frequently than an occasional submit is kinda cool. But the truly cool part is the collaboration feature of Writely, and honestly, there are other, better solutions for that.
• Can I Use it Offline? - You're not going to be online all the time. The moment you have desktop versions that can function independent of the server, is it still Ajax or a Javascript/Browser app with save capability?

This hype around Ajax is really starting to remind me of the first dotcom boom (See FauxJAX for a good sendup of this). I had a few venture capitalists back then asking me to look at business plans that added "the web" to their business models. My rule for evaluating these was always the same: what does the web add? Most times it didn't change anything; it was just another marketing channel. For others, like online classifieds, it removed distribution costs. For the social networking type businesses, it made it easier and less costly to jumpstart the network effect.

So, for those going gaga over Ajax, ask yourself, what does Ajax add? If you can't come up with a good answer, odds are it doesn't add a thing.

Upcoming Talk - Back to the Future: Component GUI's and Ajax

I'll be giving a talk at the Chi2 monthly meeting on Wednesday, April 26, 2006. The highlights are:

• Why web applications are they way they are and how things are changing
• First attempts: frameworks and toolkits that extend current practice
• How Ajax has turned the browser into a desktop
• Component-based GUI's and how they cut down on Ajax Development time
• Which frameworks get it right

For more information see here. If there is a podcast, we'll try to make that available along with the presentation.

Asynchronous Processing in Ajax Apps

Most web applications are still request driven, i.e. the browser makes a request and backend state is updated as a response to that request. There is some backend processing, but usually that processing is very losely coupled with the webapp, updating a backend data store.

Many more advanced apps make use of web services, MDB's, JMS, and other approaches to perform asynchronous processing. The need for asynchronous processing in webapps has been mostly confined to some special cases, requiring extreme scalability or performance. But now with the advent of Ajax, many new webapps are going to have to perform ongoing background processing in order to update front end displays. Some of these processes are going to exceed the time you want an XHR to be open. Or maybe you want to perform some backend processing tightly coupled to the session and UI, rather than loosely tied to the persistent store.

I've used open source tools like Mule, ActiveMQ and the like for some of these things. It does make life a bit easier when doing async processing, but moving to these new technologies will involve a bit of a learning curve for most webapp developers. Sorry, guys, the pain isn't over.

The Javascript Hackers Dilemna

Ajax has brought a new focus onto many parts of the browser, but none more so than Javascript. What started out as a rip and run embedded language called "Livescript" in Netscape Navigator has evolved into a rich language that supports sophisticated OO programming and design styles. But most Javascript written today remains primitive, poorly designed and implemented -- hack jobs, so to speak. The most you'll see on many commercial sites is the Javascript divided up into procedural utility libraries. Just look at the excellent book "JavaScript Anthology" (see a very illuminating review of it here). Most of the code in the book is very procedural; that seems to be the expectation of Javascript -- ignore most of its capabilities by writing little procedural snippets.

So here comes Ajax, along with tons of other tools and platforms that embed Javascript: OpenLaszlo (flash UI), Kettle (ETTL tool), and the various projects that embed Rhino, the Mozilla Projects Java-based Javascript interpreter. On their heels come an army of sophisticated and knowledgeable programmers who have different standards. These guys are relentlessly raising the bar on design and style in Javascript. The days of rip-and-run Javascript are over.

So, what's a Javascript Hacker to do? Maybe it's time for a second edition of Javascript Objects, a title that went nowhere in 1998.

Yahoo Mail, Ajax and Your Server

I recently did a little unscientific comparison between the old Yahoo mail and the new Beta Ajax interface. The idea was to strap a non-caching proxy in front of the browser for a day each, clear the cache and see how they compared.

I fully expected to see the increase in requests and a decrease in request size that everyone talked about. I did see the former, but not much of the latter, as it turns out. The average request size was smaller for the Ajax version, but not that much smaller.

While this test wasn't scientific -- I just used my yahoo mail account as I usually do throughout the day -- my educated guess is that the temptation to add lots of features to the new whiz-bang interface is the culprit.

Here, without further ado, are some of the facts and figures (count is number of requests. All others are in bytes):

Ajax Yahoo Mail

Count	9848
Average	3392.27
Median	543.5
90th Percentile	6593.2
25th Percentile	94
10th percentile	43
75th percentile	1604.75

Old Yahoo Mail

Count	7329
Average	4153.74
Median	195.5
90th Percentile	15052.4
25th Percentile	70
10th percentile	43
75th percentile	1880

What does this mean to you and your infrastructure requirements? Unless you exercise incredible discipline in pushing back on feature creep, you are likely to have Ajax apps that make more and bigger requests.

Remember when 1MB was more computer memory than anyone would ever need, or when car makers used the extra energy in those hybrid vehicles for greater fuel efficiency instead of more power of hybrid SUV's?

What the Commodore 64 can Teach Us About Ajax

Here's just a little sampling of articles from the November 1983 issue of COMPUTE!'s Gazette.

• Computer Graphics - The Age Of Electronic Art
• VIC Super Expander Graphics64
• Introduction To Custom Characters For VIC And 64
• VICreations: Animating With Custom Characters

It seems every issue would have at least two articles on how to twiddle the graphical bits on the C64 or VIC-20, articles on how to make sprites dance across the screen or writing your own video game or drawing program in 6502 assembler. You had similar articles in similar magazines with the introduction of the IBM PC.

With the advent of the Mac (yes, yes, Xerox did it first), all that bit twiddling disappeared. People started wondering more about how to get windows and menus and buttons to do what they wanted them to do, or even to start developing their own controls.

Fast forward almost 23 years and read just about any blog or magazine that features Ajax articles. The articles are almost universally about nuts and bolts, nitty-gritty details, i.e.:

• Writing a server-side Ajax handler
• Creating souped up popup windows with Javascript
• Implementing fancy form controls in Javascript
• Understanding the DOM

It's bit twiddling all over again. Perhaps it's time to write an article on how to do Sprite animation with Javascript.

Why Frameworks are Important

I've been beating the drum of component based GUI frameworks for Ajax for a while now. To understand why, I think it's useful to look at the evolution of the desktop GUI framework and understand what makes them so successful. Over at ACM's Queue publication, I came across an article that gives a nice overview of what an application framework does and the characteristics that make it successful. This following struck me as especially apropos with regard to most of the current Ajax frameworks.

A framework exhibits "inversion of control" at runtime via callbacks. These callbacks invoke the hook methods of application-defined components after the occurrence of an event, such as a mouse click or data arriving on a network connection. When an event occurs, the framework calls back to a virtual hook method in a preregistered application component, which then performs application-defined processing in response to the event. The hook methods in the components decouple the application software from the reusable framework software, which allows each to change independently as long as the interface signature and interaction protocols are not modified. Since frameworks exhibit inversion of control, they can simplify application design because the framework—rather than the application—runs the event loop to detect events, demultiplex events to event handlers, and dispatch hook methods on the handlers that process the events.

Inversion of Control, it's not just for Spring anymore. :-) Seriously, though, how many of the current Ajax toolkits -- I'm not sure that with the exception of appfuse it's correct to call them application frameworks -- give you IoC? I'll be watching to see if they make the transition to frameworks and will give credit where credit is due.

Echo2 a Cult?

Paul Browne has nice little survey post over at O'Reilly about Ajax frameworks. All's cool until I stumble across the following:

Echo2 - Evolution of original Echo Framework, can run in any Servlet container. Original has cult following, but doubt if it will become the number 1 web framework.

I hear you, Paul. I remember evaluating Echo1 a long while back and thinking to myself: "why would you want to build a web application the way you would a component based desktop GUI? The behavior is so different, the lack of control, layout, look and feel, etc., etc." My conclusion was that Echo1 was a nice design solution applied to the wrong problem.

Fast forward to today. I have DHTML, CSS, Javascript, XHR, etc. I have complete control over what shows up in the browser, anywhere, any time, anyhow. I've got people writing thousands of lines of rich client application code, business tier logic leaking into the browser, cat and dogs living together...you get the idea.

Now the Echo design solution makes plenty of sense. The simple report and forms beast of the early 90's has turned into a kind of GUI desktop. Now my reaction is "why wouldn't you want to write a rich client web app the way you would write a component based desktop GUI?" It's a proven design. Nobody twiddles the bits by hand anymore. And when the browser wars start up again, the standards start diverging, and your Ajax apps start breaking, then the problem starts resembling that of a cross platform GUI, i.e. isolating Ajax variations in browser specific peer classes, much like AWT.

You can easily imagine running the same application as a Swing desktop GUI or a browser embedded Flash app. That's if the design is right and makes use of what? That's right, browser or rendering layer specific peers.

Guess who has all that today? That's right, Echo2. Come on and drink the Kool-Aid.