asymptotic tight bound

Managing source code repositories

2011-08-25T23:04:00.000-07:00

There are few areas in software engineering that get programmers and managers as actively engaged as the source code management processes. Naturally, it has evolved rapidly over the last decade or so as we went from RCS and Visual Source Safe to Git/Mercurial and SVN.

As these tools have come and gone, so also have good and bad practices. In an effort to keep up with current practices, I came across this article on using Git for source code management.

What I like about this article is its simplicity and the ease with which you can survive a corporate software development environment for product engineering.

New WD for programmable HTTP caching and processing

2010-01-16T16:09:00.000-08:00

The programmable HTTP caching and processing standard that is advancing through W3C, called DataCache, has recently been updated. A new working draft is available. You can read the frequently updated, latest and greatest version or get to a stable WD of DataCache.

This version has several improvements over the previous working draft:

The spec extends the HTML5 ApplicationCache interface and add some behaviors to it related to programmability of caching.
There is no more a secure data cache concept that would be guarded by cookies.
This spec is heavily tied to the HTML5 Application Cache and refers to a lot of terms, algorithms, interfaces, and events defined there. It also modifies the networking model defined there.
The asynchronous interfaces are styled similar to Indexed Database.
I have removed the synchronous interface until we agree on the rest of the spec.
Security considerations are modified.
Mechanisms to inspect the metadata of application caches is provided.
A new mechanism to annotate an application cache with outbox type of metadata is provided so that user agents can provide a control panel to users for acting on outbox when network connectivity is reestablished.
New events on the cache are removed or moved to events on cache transactions.
The File API Blob interface is now used in the spec.
It is more richly hyperlinked and features improved IDL listings.

New WD for browser database standard

2009-12-29T23:12:00.000-08:00

At the start of a new year, I wish the best to you in 2010.

The browser database standard that is advancing through W3C, called Indexed Database API, has recently been updated. A new working draft will soon be available. You can read the frequently updated, latest and greatest version or get to a stable WD of Indexed Database.

(Please note Indexed Database used to be called WebSimpleDB originally.)

This version has several improvements over the previous working draft:

1. It has a smaller footprint in terms of features.
2. It is more specific in terms of supported data types.
3. It supports two modes of deadlock-free operation.
4. It offers concurrent iteration through multiple cursors for both synchronous and asynchronous clients.
5. It features a consistent API style for asynchronous clients.
6. It contains security considerations
7. It is more richly hyperlinked and features improved IDL listings.

This blog is moving to its own domain

2009-12-23T07:53:00.001-08:00

After much thought, I decided to finally host this blog at my own domain - blog.o-micron.com. Please adjust your subscriptions to the blog's new URL - http://blog.o-micron.com.

Use Cases for DataCache

2009-11-06T08:03:00.000-08:00

I discussed a set of use cases not well supported by HTML5 ApplicationCache with the HTML WG during a breakout session at TPAC. These use cases were the basis for AtomDB are being solved by the DataCache API.

Found out that OMA and AT&T had similar ideas too. Also learnt that these techniques could be merged with HTTP caching.

Hixie agreed these use cases covered the ones the GMail team had asked for but he favored waiting until after AppCache had stabilized and browser implemented other HTML5 stuff.

WebSimpleDB gets thumbs up from major browser vendors

2009-11-05T12:20:00.000-08:00

Microsoft and Mozilla declared their intention to advance WebSimpleDB and their preference for it over WebDatabase (which itself is getting renamed to WebSQLDatabase) and WebStorage.

WebDatabase will be frozen in terms of its current API and likely stay at the SQL dialect of SQLite 3. WebSimpleDB has taken on the mandate of the WG to create a small and general set of primitives for building various rich JS libraries. WebSimpleDB is heavily influenced by Oracle's experience with Berkeley DB.

Stay tuned here for more on this topic.

DataCache API now rewritten and ready for FPWD

2009-10-16T12:00:00.000-07:00

There was a wave of support for programmable HTTP caching when the first DataCache editor's draft was published. The main issue with the draft was the inability to derive the exact list of additional requirements on browsers compared with HTML5's AppCache. I went ahead and rewrote DataCache to make it easier to spot the differences. Here are the differences:

A data cache does not have fallback or online whitelist namespaces or foreign entries.
A data cache provides a monotonically increasing numeric version identifier.
A data cache may have an authorization cookie.
Every resource managed in a data cache may be configured to process certain (HTTP) protocol methods locally
Zero or more data caches are automatically associated with a cache host at its creation/load time. A secure data cache group is available to a cache host if its authorization cookie will be sent to an origin server for fetching that cache host [RFC2965].
The events checking and noupdate are not used. The events cached and updateready are merged in to a single event ready. Events downloading and progress are renamed as fetching and captured. There is no downloading update status for a data cache group.
No manifest is used. Instead an update transaction encapsulates a set of changes to the cache. An update transaction consists of any number of capture steps or release steps. An application can store a bag of bits independent of a network representation of that resource. This allows storing of off-line resources. The results are not visible until the transaction is committed.
Update transactions can be performed in workers but not in the background independently of applications.
It is possible to have only one online transaction but multiple off-line transactions are allowed.
It is possible to find out the resources added to or removed from cache starting at a certain version.
A data cache offers applications the ability to get the contents corresponding to a URI.
The navigator registers a scriptable HTTP interceptor that can off-line respond to arbitrary HTTP requests based on prior (data cache) configuration of the resources in those requests.
A new header is defined to by-pass data cache in request processing
A slightly different networking model is required to take into account interception.

I have a request to publish this new editor's draft as a FPWD and hopefully that will happen before TPAC with a view to discussing the draft during WebApps WG f2f meeting. Read this in conjunction with WebSimpleDB so you can create off-line Web apps that can serve HTTP resources locally, including even query on the off-line cache.

Now published - an alternative to SQL for database storage in Web browsers

2009-09-09T11:58:00.000-07:00

Many of us have raised serious issues with the WebDatabase API that provides SQL query access to a browser embedded database. Some of these are:

Use of SQL means that it will be next to impossible to get agreement on the exact dialect supported by independent implementations.
The API is extraordinarily complex that an ordinary programmer would have a hugely difficult time using it. This means more bugs will remain hidden for longer in the spec, and more bugs would be found in code using the API. The complexity arises out of a need to prevent delays when accessing a database from affecting the user interface. As a result, practically every call to the API results in an asynchronous callback.
The API requires a lock acquisition scheme that does allows extremely limited parallelism in data access. As a result, superior implementations cannot do well. Even though the intention is noble - avoid producing exceptions as a result of deadlocks - the result is delays and timeouts as well as higher level deadlocks, which the programmer still has to deal with.
The API has chosen to use SQL due to its well developed relational model. However, it doesn't want to provide an API that the typical database developer will be able to program against. This mythical "Web database" programmer receives undue importance in the design of the API.
Overly reliant on SQL, when the data model in browsers is that of objects. This forces everyone to develop layers to translate between rowsets and objects.

There is currently a deadlock that has arisen out of a number of browser vendors refusing to go along with WebDatabase (notably Mozilla and Microsoft) and a lack of an alternative. To break this deadlock, we have developed an alternative that will provide a highly efficient and functionally rich (although less richer than SQL), called WebSimpleDatabase.

This editor's draft of WebSimpleDB is now available from W3C. I am inviting feedback on this draft so that I can incorporate it before it goes to FPWD.

Embed XML document inside another XML document

2009-07-13T09:09:00.001-07:00

Ever considered the implications of embedding an XML document inside another XML document? Every new format developer, especially extension format developer has to consider the consequences of embedding an XML extension payload inside an XML envelope.

In plain terms, what are the consequences of doing the following:


<feed xmlns="http://www.w3.org/2005/Atom">
<id>...</id>
...
<entry>
 <id>...</id>
 ...
</entry>
...
</feed>

While we do it all the time with Atom, this is an example of a document embedded inside another - Atom entry inside Atom feed. There are a number of issues lurking under the surface that one has to consider before copying the entry bits in to the feed itself. In general, there are many issue to consider while embedding one XML document inside another.

These issues don't surface if you are managing all the bits yourself, i.e., you store the original bits of everything and normalize to a common set of encodings, namespaces, signatures, entities, and so on. However, if you are aggregating content from different publishers, then it's a different ball game. I started thinking about these issues as a result of Mark Nottingham's feedback about the Atom in-line I-D.

Here are some of the issues I could think about (and manifestations in the case of Atom):

Duplication of content (e.g., <atom:title>)
Inherited values for metadata (e.g., xml:lang, xml:base)
Character set and encoding (for XML and binary content)
Entity declarations
Namespace declarations
Digital signatures
Protocol metadata (ETag, Content-Length, Content-Type)

Did I miss some more? If so, please help me identify them. It will help in making the in-line I-D stronger and easier to implement and use.

Temper your 3G and iPhone expectations

2009-07-10T09:05:00.000-07:00

"Why do we need offline capabilities when today's phones are so savvy about using the network and the network is so good."

In a PCWorld article, Bill Snyder, veteran mobile industry journo, basically admonishes us for expecting too much from mobile devices. Here are a few excerpts:

But my first few days with the iPhone [...] made me realize how far we are from the mobile, Web-based nirvana so often promised. The technology and the infrastructure are simply not there yet. It's time to lower our expectations.

...

Our national infrastructure lags behind demand for high-speed services, and given the costs and the difficult economic conditions, we just have to wait until it catches up.

...

Batteries have gotten much better. The problem, though, is the ever increasing demands made upon them by new features. Take my iPhone: That big, bright screen sucks power like a baby quaffs milk. Use a few other features, and suddenly you're running on empty. It's a serious problem for engineers, and it's only going to get worse.

...

Battery technology, unlike CPU technology, does not dance to Moore's Law -- and it won't for some time.

There are two engineering alternatives for poor network and low battery life:

Hunker down and make designs for offline apps
Design seamless online/off-line apps

#1 is good old mobile app design. There is no reason to go back to that since networks are indeed better and users want to use the network more often. However, #2 is a smart evolution path while we wait for 4G.

Requirements for and components needed in real Web Storage

2009-06-24T15:59:00.000-07:00

Several times I have prodded people to provide requirements to WebStorage. Here's what I have gathered from use cases and scenarios people have suggested for WebStorage's scope:

Here are three sets of requirements as I see them:

Structured key-value pair storage and cursors

Store, remove, and retrieve individual data items
1. items identified by a string key
2. item values can be either null, object, or string
Allow duplicate values for the same key
Allow organization of items in to more than one database per application
Sequentially walk items in a cursor starting at a point identified by a key value
1. items should be presented in an increasing key value as determined by JavaScript's String comparison operator '=='
2. once the highest key value is reached, no more items are provided
3. if the matching key value is empty, the first key in the database is the starting point
4. can walk through items in reverse order
5. items obtained in the cursor are deletable - IOW, cursor is modifiable
Provide a sequence object in each database to generate a monotonically increasing sequence of numbers
A database belongs to a single storage unit.
Applications are able to open or create a storage unit.
Allow storage of multiple items to be committed as part of an atomic transaction
Offer READ_COMMITTED and READ_UNCOMMITTED cursors
Transaction scope is limited to operations on databases within the same storage unit.
All APIs are synchronous

HTTP Cache programming:

capture a resource that is used for satisfying GET & HEAD requests
1. given just its URL, fetch and durably store its representation
2. as a locally produced representation
3. over write an existing representation
retrieve or remove a resource's representation or check for its presence

HTTP interception:

Store the methods on resources that are locally intercepted
Choose among the following for a locally intercepted method
Pass the headers and body of the request, without jeopardizing security, to the programmed interceptor
Make the request to the server, relay the request and the response to the programmed interceptor, and return the response to the application

A JS library for synchronized Web storage would use all three components to process network requests locally using HTTP interception, respond to queries locally using the b-tree storage and the HTTP cache as well as store pending requests for forwarding to the server when it is reachable.

A pictorial representation of the layers involved in an implementation of CouchDB to run in a browser captures the contrast between the existing and the proposed breakdown of Web Storage.

The browser is then required to do the switching between off-line and on-line and between relay and intercept as I explain in the post on BITSY 0.5.

As I see it, these requirements subsume the requirements known to date for Ian Hickson's Web Storage draft. Were it not for the widespread use of LocalStorage, we could do away with it completely because the b-tree store is a more robust way of dealing with key-value pairs. However, there is no reason and, hopefully, little chance of the SQL portion making its way to the W3C hall of fame, I mean Rec.

In-lining extensions and hierarchy for Atom

2009-06-05T10:55:00.000-07:00

Linked resources are a real treasure in Atom. They allow you to connect, through hypertext, all kinds of things in to a useful content model. No wonder all sorts of Atom applications have invented custom or new standard link relations for Atom.

One challenge with linked resources, though, is that one has to make separate network requests to obtain their representations. While this is exactly the approach taken in HTML, there are some benefits to making representations of linked resources available in-line with the referring Atom document such as reduced network dependency and greater responsiveness.

Introducing an in-line representation technique for such situations, while making Atom as a format more useful, does nothing for standard Atom clients such as feed readers. These standard clients could, over time, provide better access to Atom extensions. For example, some feed readers do not provide adequate support for presenting categories. Others don't make available the "replies" link. A part of working with the Atom format is the opportunity of unimaginable extensibility and the disappointment of uptake and general utility.

In any case, we have revised the atom-hierarchy I-D and it is available in both text and HTML. There is a demand for separating the in-line parts from the hierarchy parts, and this 01 version keeps them apart inside a single I-D.

CMIS XVI: Ignore good hypertext practices at your own risk

2009-06-02T15:07:00.000-07:00

The spec lead for CMIS AtomPub extensions simply doesn't have the patience to get his protocol right. Feedback is either too minor for his current attention or pretty much useless for his needs. Let me give an example for each.

When asked about a number of optional RFC5023 things that are never referenced by CMIS, such as those in CMIS XIV: Can't do basic path manipulation? Don't blame AtomPub, Al takes refuge in the W-I-P tent.

On the other hand, Al Brown is dead set against using entry inside link elements for in-lining hierarchy representations. I am beginning to get that his opposition is often not well-intentioned and that the CMIS TC members are nodding their heads without fully realizing the consequences of doing so.

#1 does not ideally address CMIS' use case of how to represent a tree of atom entries in a single document but rather provide a generic pre-fetching/inlining mechanism.

This appears rather cavalier to me. But this would not be the time someone did so. Some of you would remember Dare Obasanjo and Yaron Goland griping about AtomPub's lack of support for "hierarchical data inlining" and Microsoft's justification for Web3S as a brand new publishing protocol for the Web , which spawned a discussion about whether Atom could deal with recursive structures such as a hierarchy. Joe Gregorio eloquently refuted claims that Atom syntax is not suitable for hierarchies by, essentially, advocating its hypertext mechanisms, which is exactly what the atom-hierarchy I-D is proposing.

Quoting James Snell about how he dealt with hierarchy in IBM Lotus Connections:

In reality, it turns out that hierarchy in APP is actually quite simple to achieve. The IBM Lotus Connections product includes a component called “Activities” which has a very hierarchical data structure. Every user has a set of collections, each of which has a set of entries, each of which can have any number of comments that can be nested to any depth. When I first designed the Atom Publishing Protocol API for activities, I created a top level collection of Activities. Every entry in this collection represents an Activity. Every activity is also a collection. I post entries and comments to the activity collection using the standardized Atom Threading Extension to provide the additional hierarchy. An Atom Entry can represent pretty much anything, including a collection of Atom entries.

At IBM we’ve look at pretty much all of these issues and found simple approaches to resolving them that fit perfectly well within the bounds of the Atom specs. We’re using APP for a lot of different things, not all of which fall neatly into the original use cases APP was intended to cover. Yes, we had to give certain aspects some careful thought but we generally could not find any showstoppers.

Microsoft subsequently rolled back their plans and gave up on Web3S and instead switched to using Atom as is.

If Al is right are right, perhaps the atom-syntax group, Microsoft, IBM, or Google would have defined the meaning of atom:entry inside atom:entry, which appears natural to Al, and included it in their APIs. I can only guess, having hung around atom-syntax and atom-protocol for the last 3 years, that in-lining an entire hierarchy was not a good hypertext-based protocol design approach, just like HTML does not inline all its CSS, images, JS, and embedded objects.

Perhaps all this evidence will make you see why the approach proposed in atom-hierarchy I-D is suitable for what you are trying to achieve. But perhaps CMIS is right in what it is doing. Quoting Joe from the earlier referenced post:

no one has found the one true hierarchy to rule them all

We'll leave it for the future to decide.

CMIS XV: Perils of embedding entry inside entry

2009-05-28T16:38:00.000-07:00

Recent updates to CMIS AtomPub binding suggest embedding atom:entry directly under atom:entry for in-lining hierarchically related entries. There are two bad consequences of this:

There is no way to differentiate an entry that has no child entries from the case where the server did not inline the nested entries.
There is no context in the inlined entry as to its relation to the parent entry. Conversely, it is wrong to appropriate the use of atom:entry inside atom:entry entirely for the purpose of expressing hierarchical relation, because several other possibilities for opportunistic in-ling exist, e.g., replies and via.
The repetition of multiple entries inside an entry means that the benefits of atom:feed as a listing container are lost.

Creating a hierarchy of documents and folders using hierarchy-ID

2009-05-28T10:54:00.000-07:00

You wouldn't be able to create images and HTML content in a local file system, and, say, store them in a CMIS system, and have all the hyperlinks work just fine. This appears to be a major problem to me as described in my previous post.

On the other hand, this would work just fine in AtomPub, provided the server did the right thing and followed the hierarchy-ID. Here's an example:

> GET /store;contents/

< 200 OK

< Content-Type: application/atom+xml;type=feed

< Content-Length: nnn

< <feed>

< <id>4b01ffb2-91da-4f80-a908-79b66d35db42</id>

< <title>Folder named "store"</title>

< <link rel="self" href="/store;contents/"/>

< <link rel="up" href="/store;metadata"/>

< <app:collection href="/store;contents/">

< <title>Folder named "store"</title>

< <app:accept ah:role="master">application/atom+xml;type=entry</app:accept>

< <app:accept ah:role="detail" alternate="multipart-related">image/*</app:accept>

< </app:collection>

< <updated>2009-01-01T10:00:00-07:00</updated>

< </feed>

> POST /store;contents/

> Content-Type: application/atom+xml;type=entry

> Content-Length: nnn

> Slug: images

> <entry>

> <title>Image collection for Web site</title>

> <content/>

> <updated>2009-05-26T14:48:31.859-07:00</updated>

> </entry>

< 201 Created

< Location: /store/images;metadata

< Content-Location: /store/images;metadata

< Content-Type: application/atom+xml;type=entry

< Content-Length: nnn

< <entry>

< <id>4b01ffb2-91da-4f80-a908-79b66d343b42</id>

< <title>Image collection for Web site</title>

< <link rel="edit" href="/store/images;metadata"/>

< <link rel="up" href="/store;metadata"/>

< <link rel="down" href="/store/images;contents/"/>

< <summary>Folder named "images"</summary>

< <updated>2009-05-26T14:48:31.859-07:00</updated>

< <published>2009-05-26T14:50:31.859-07:00</published>

< <app:edited>2009-05-26T14:50:31.859-07:00</app:edited>

< </entry>

> GET /store;contents/

< 200 OK

< Content-Type: application/atom+xml;type=feed

< Content-Length: nnn

< <feed>

< <id>4b01ffb2-91da-4f80-a908-79b66d35db42</id>

< <title>Folder named "store"</title>

< <link rel="self" href="/store;contents/"/>

< <link rel="up" href="/store;metadata"/>

< <app:collection href="/store;contents/">

< <title>Folder named "store"</title>

< <app:accept ah:role="master">application/atom+xml;type=entry</app:accept>

< <app:accept ah:role="detail" alternate="multipart-related">image/*</app:accept>

< </app:collection>

< <updated>2009-05-26T14:50:31.859-07:00</updated>

< <entry>

< <id>4b01ffb2-91da-4f80-a908-79b66d343b42</id>

< <title>Image collection for Web site</title>

< <link rel="edit" href="/store/images;metadata"/>

< <link rel="up" href="/store;metadata"/>

< <link rel="down" href="/store/images;contents/"/>

< <summary>Folder named "images"</summary>

< <updated>2009-05-26T14:48:31.859-07:00</updated>

< <published>2009-05-26T14:50:31.859-07:00</published>

< <app:edited>2009-05-26T14:50:31.859-07:00</app:edited>

< </entry>

< </feed>

> GET /store/images;contents/

< 200 OK

< Content-Type: application/atom+xml;type=feed

< Content-Length: nnn

< <feed>

< <id>4b01ffb2-91da-4f80-a908-79b66d35db42</id>

< <title>Folder named "images"</title>

< <link rel="self" href="/store/images;contents/"/>

< <link rel="up" href="/store/images;metadata"/>

< <app:collection href="/store/images;contents/">

< <title>Folder named "images"</title>

< <app:accept ah:role="master">application/atom+xml;type=entry</app:accept>

< <app:accept ah:role="detail" alternate="multipart-related">image/*</app:accept>

< </app:collection>

< <updated>2009-05-26T14:48:31.859-07:00</updated>

< </feed>

> POST /store/images;contents/

> Content-Type: multipart/related;

> boundary="===============1605871705==";

> type="application/atom+xml"

> Slug: logo.gif

> MIME-Version: 1.0

> Media Post

> --===============1605871705==

> Content-Type: application/atom+xml; charset="utf-8"

> MIME-Version: 1.0

> <?xml version="1.0"?>

> <entry xmlns="http://www.w3.org/2005/Atom">

> <title>Logo image</title>

> <updated>2005-10-07T17:17:08Z</updated>

> <author><name>Daffy</name></author>

> <summary type="text">

> A company logo

> </summary>

> <content src="cid:99334422@example.com"

> type="image/gif" />

> </entry>

> --===============1605871705==

> Content-Type: image/gif

> MIME-Version: 1.0

> Content-ID: <99334422@example.com>

> GIF89a...binary image data...

> --===============1605871705==--

< 201 Created

< Location: /store/images/logo.gif;metadata

< Content-Location: /store/images/logo.gif;metadata

< Content-Type: application/atom+xml;type=entry

< Content-Length: nnn

< <entry>

< <id>4b01ffb2-91da-4f80-a908-79c56d343b42</id>

< <title>Logo image</title>

< <link rel="edit" href="/store/images/logo.gif;metadata"/>

< <link rel="up" href="/store/images;metadata"/>

< <updated>2005-10-07T17:17:08Z</updated>

< <author><name>Daffy</name></author>

< <summary type="text">

< A company logo

< </summary>

< <content src="/store/images/logo.gif"

type="image/gif" />

< <updated>2009-05-26T14:58:31.859-07:00</updated>

< <published>2009-05-26T14:59:31.859-07:00</published>

< <app:edited>2009-05-26T14:59:31.859-07:00</app:edited>

< </entry>

CMIS XIV: Can't do basic path manipulation? Don't blame AtomPub

2009-05-28T10:34:00.001-07:00

David Nuscheler pointed out in an offline conversation that he can't do basic content management such as authoring an HTML document with links to images in the form <img src="http://www.blogger.com/images/logo.gif" /> using CMIS. He suspects it is due to AtomPub's approach to server-managed namespace.

To cut the long story short, he is right that CMIS AtomPub seems a lost chance to address this basic requirement. The CMIS AtomPub draft (revised after so much wrangling over AtomPub interoperability) passively works against the intent of AtomPub mechanisms to support the desired naming and path usage. Here's why I think so:

It says nothing about the desirability of supporting the AtomPub Slug header when creating new items (whether folders or documents).
It also says nothing about the desirability of hierarchical paths for a certain class of CMIS applications.
It suggests that all content be supplied out-of-line.
It promotes examples such as the following path names that bear no resemblance to the hierarchical arrangement of folders and documents.
http://cmisexample.oasis-open.org/rep1/8991ed69-adb9-4833-90a0-504d5338f006 (is the folder child) with the content of this document in http://cmisexample.oasis-open.org/rep1/4b01ffb2-91da-4f80-a908-79b66d35db42
http://cmisexample.oasis-open.org/rep1/151c2ae8-ea3d-4031-b331-08680ac3e607/3 (is the folder entry)

In another post I will illustrate how to perform a basic series of AtomPub operations to obtain the desired behavior.

WebDAV or AtomPub

2009-05-26T11:46:00.001-07:00

This question is often asked by content managers, whether blogs or otherwise. I often pull together a bunch of links from all over the Web to answer this question. So this time, I thought of summarizing it on my blog. Your comments to correct any mistakes are most welcome.

Here's my (meek) attempt to contrast and compare the two approaches:

WebDAV:

Adds 7 new HTTP methods
Natively understands folders as "collections"
Clients manipulate and control namespace
Clients can "parse" URIs to apply semantics of hierarchy
Supports locking of entire hierarchies
Natively supports relocating and copying document to a different folder
Does not separate modifiable resources from read-only view of resources

AtomPub:

No new HTTP methods
Natively understands listings as "feeds" and modifiable listings as "collections"
Servers control namespace, clients can make suggestions.
Clients can navigate to URIs identified in hypertext but not construct these URIs
Natively supports only optimistic concurrency control of single resources
Separates view-only resources from their underlying modifiable resources.

There is an interesting recent discussion comparing AtomPub and WebDAV in the context of Wiki APIs about the differences.

Bottom line is "use WebDAV if you can". AtomPub is a simpler technique, but it is also less feature rich. Of course, extensions and other such things will add back some of the features over time, if people like the lightweight foundation of AtomPub.

Splitting hierarchy I-D

2009-05-20T10:29:00.000-07:00

Based on the discussion of hierarchy concepts in Atom on the atom-protocol list as well as others interested in hierarchical relations in Atom, we have split out the hierarchical navigation and representation portions from the original hierarchy I-D into a new I-D titled atom-hierarchy. This was done with the intention of achieving consensus on the Atom syntax to be used for parent/child like navigation separately from how such resources are manipulated.

We will continue to edit the new I-D to incorporate new feedback. Both text and HTML versions of this I-D are available.

Also, given the discussion about discovery of AtomPub collections on the atom-protocol list , it is useful to have a clear path to take as well as guidance about which mechanisms to use under certain conditions. Therefore, we feel that a new RFC is needed to clarify the way in which collections ought to be identified.

Some of the material for this RFC is taken from the existing hierarchy I-D. The idea is that the best practices for collection discovery are best kept separate from new stuff.

We continue to edit the I-D to incorporate any feedback. Both text and HTMLversions of this I-D are available.

As a result of these two new submissions, the original hierarchy I-D is going to be significantly smaller in its scope. We will revise that I-D in the near future, in case you were wondering what would happen to it.

Write-through Web Caches and GMail

2009-05-18T16:48:00.000-07:00

Google I/O has a session on a generalized architecture for offline Web applications. Interestingly it is touting its use of a write-through cache as the basis for this new architecture as you can see from this session description:

Puzzled by all the new architectural choices possible when trying to build an offline-capable web application? So were we until we decided to design the newly launched Gmail Mobile Web for iPhone and Android's offline capabilities by analogy with microprocessor caches: offline via a portable write-through caching layer running on either HTML 5 or Gears databases.

We first demonstrated the utility of this architecture in June 2008 by building AtomDB and its recommendations for application architecture. See details in my original post titled: Mobile databases or write-through Web caches.

I couldn't be more amused as Gears was initially touting an application-specific switch for offline-ready Web apps, which would be an absolute nightmare for app developers and users. All the best to Google for marketing this new term, which we had coined.

CMIS XIII: Interoperability in AtomPub

2009-05-18T16:24:00.000-07:00

Sometimes I feel that the CMIS community feels compelled to interoperate at the AtomPub protocol level without knowing why they want to nor fully comprehending what it means to. Therefore, IMHO, despite best intentions they seem to be stumbling on their way to it.

AtomPub interoperability is elusive -- how many generic AtomPub clients talk to multiple servers constantly? Brian Smith thinks there are no standard AtomPub clients and there is no real interoperability in AtomPub.

Because an AtomPub server is allowed to do anything it wants in response to a
request, there's no way to write a useful AtomPub client without making many
assumptions about the server's behavior.

In doing so, he falls for the common fallacy in thinking that true interoperability means the client knows exactly what the server can and will do. This is the same fallacy that doomed the features draft after 12 revisions because it tried to specify the whole universe of server behavior interfaced through AtomPub.

Roy Fielding nicely explained what interoperability means in IETF and specifically in AtomPub was discussed long back in the atom-protocol list (h.t. Peter Keane). The key takeaway is this:

Interoperability means that client and server should be able to communicate and understand intentions and responses even if they don't understand the payload.

It is not trivial to define a standard AtomPub client, but I disagree that there are no standard AtomPub clients. We wrote one and it is called AtomDB. It works with standard and/or extended AtomPub servers. To take a few examples, we work with both blogging servers as well as with GData servers equally well.

CMIS XII: Must-understand signaling in app:accept

2009-05-18T13:37:00.000-07:00

Since rudimentary AtomPub clients may be unduly surprised when a sophisticated CMIS AtomPub server summarily rejects a POST request with a hard to understand response. So, it is best to forewarn such clients about the requirements of the server.

To do this, one would need to use a must-understand signaling mechanism in AtomPub, of which there is exactly ONE in standard AtomPub. The app:accept element's content signals to the client what type of content the server is willing to accept. If the request fails because the content does not meet the expectations of the server, a 415 response is generated. This means the rudimentary client can easily understand the sophisticated server. Plus, the sophisticated server now has a means of conveying additional information to the sophisticated client. And, best of all, no permission is needed from anyone at IETF, unlike the registration for "type", although an IANA registration for the parameter would be good to have.

Advertising entry POST constraints in AtomPub

2009-05-18T12:12:00.000-07:00

The main question of this post is:

How can a server advertise additional entry POST processing behavior beyond that specified in AtomPub so that a client is forewarned or better informed?

There are two kinds of AtomPub extensions:

some provide additional protocol behavior. For example, a multi-part accepting AtomPub server may wish to advertise that it can process media entry creation in a single HTTP request. Or a hierarchy managing AtomPub server may create new collections as a side effect of creating new entries.
some restrict the kind of Atom entries they accept. For example, a Google Finance server only accepts new portfolio or transaction entries. A CMIS server may disallow an entry that does not meet certain typing rules that control the server.

For the first kind of server, a new kind of client would be required to take advantage of the behavior.

It is the capriciousness of the second kind of server that I worry about and have blogged here and here. The basic problem is that a client would not know whether an AtomPub server will cooperate or not. Of course, there are very few guarantees in the AtomPub world, and co-operation is not one of them. However, to make a good citizen out of an extende AtomPub server would be an important thing to fret about. Don't you agree?

The main effect of the approach taken to address the above need is on standard AtomPub clients which are not limited to a single server's behavior. In other words, a specialized GData client will understand Google's extensions for multi-part POST, but is there a way this kind of behavior can be advertised and understood by the client. Is there a case where a client should desist from making POST requests if it encounters a server that is too restrictive to work with the rudimentary AtomPub client?

Solutions:

The main server behavior advertisement in AtomPub is the content of app:collection, i.e., app:accept and app:categories.

The basic question is whether, for standard AtomPub clients that don't understand any extensions, there is benefit to receiving better app:accept advertisements. In other words, is there some additional information about app:accept that will help clients better determine whether a request is likely to be accepted by the server. I know there is no must-understand extension mechanism for app:collection, so I don't know what would be the best alternative:

Use a new MIME type parameter on app:accept for the Atom entry content type
Use a new attribute on app:accept
Use an extension element in app:collection to identify CMIS requirements
Use the app:categories with a link to an AtomPub category document

CMIS XI: Why I would reorganize the AtomPub extensions

2009-05-14T23:20:00.000-07:00

I have been pretty aggressive in pointing out issues with CMIS's AtomPub bindings and personally do not like the way it has been written. Why? I feel it is loosely written, even for a pre-release draft, and it leaves too many interoperability issues arising from editorial standards that would be discovered rather late.

I have written in both the W3C style (see the BITSY proposal) and the IETF style (see the hierarchy-ID). I will be the first to admit that some of the technical details of my proposal might be in error. However, those errors got pointed out and corrected very early in the cycle. Nevertheless, I haven't gone through enough of these cycles myself to understand the benefits or shortcomings of either, I have a strong feeling that readers like the terseness of either specs as that creates tighter specs, as long as the concepts are well explained and examples used. There is way too much repetition in the current CMIS AtomPub draft to pass muster at my shop.

So instead of moaning and griping about the current CMIS AtomPub bindings draft, one would question - how would you do it instead? Although fortunately, no one has asked that question, I figured I better have an answer in case someone did.

But before sketching a proposal, let me state my goals of reorganizing the content:

It should illustrate CMIS's use of AtomPub since it is an AtomPub variant, not a brand new protocol.
It should avoid duplication of link relation values. The link relation "repository" is defined/explained identically at least 17 times plus once in the IANA registration. It is a different matter that this particular registration may never take place as "service" might replace "repository" or for some other reason. Other relations are not much better and may occur up to one dozen times. May be it is just me, but I fail to see how this style is of benefit to anyone. Ideally, one place where you describe the model, one place where you define it, and a registration. It should be pretty much an exercise in inference about where you may use which links.
It does not clearly state how interoperability is to be achieved with standard AtomPub clients, an issue I have highlighted here, here, here, and here.
Although AtomPub clearly defines Collection, Entry, Media Resource, and Feed resources and the standard operations that can be performed on each, there is a fair amount of duplication going on in the CMIS AtomPub bindings draft. Instead, the spec should only define any additional restrictions for each CMIS resource mapping to an AtomPub resource.
The draft should define the meanings of various special arguments used as query parameters in one place instead of scattering them all over the document and repeating the explanations a dozen or so times. As an example, the exact same text for "filter Argument" is repeated 13 times across the draft. The draft should spend far more time on encodings and allowed content than it currently does. If certain attributes have repository-specific syntaxes, this draft should not even introduce them.
The whole notion of allowable actions very likely could be collapsed in to the Allows HTTP header. I have not had the chance to fully understand that, but certainly some effort should be expended towards that direction.

If this is an early draft, then less is more. I wouldn't push a hard-to-review draft and disclaim reviewers comments saying that this is an early draft. Again, may be its only me, but I get the feeling that this draft would have a pretty hard time getting anywhere, if it were not OASIS.

CMIS X: Is CMIS a good AtomPub citizen?

2009-05-14T09:23:00.000-07:00

CMIS has consumed a lot of my blog space for a good reason. Hierarchy is quite important to CMIS and we (Colm Divilly and I) have spent a great deal of time on it. Despite having pointed out several issues in this blog, I can't beat the feeling that CMIS is determined to go its own way at the slightest hint of inconvenience.

Firstly, despite the futility of modeling Atom with an XSD, CMIS insists on it whether it is informational, mandatory, or illustrative for tooling. Additionally, when probed about AtomPub interoperability with the binding for CMIS, here's what the editor has to say:

The spec states it is repository specific what happens when a non atom entry media type or atom entry media type without cmis info is POST'ed to a collection. It suggests that the repository may create an instance of a document of a default type.

What this means is that whether a CMIS server works with a non-CMIS client is upto the server to decide. This is what I posted about in CMIS II: (C)MISsing good old AtomPub.

If you leave it to individual CMIS servers to decide whether they support non-CMIS AtomPub clients, then this is not true interoperability.

CMIS IX: Confused loyalties - CMIS or AtomPub

2009-05-13T13:00:00.001-07:00

In a previous post, I complained that I miss good old AtomPub when working with CMIS servers. One of the reasons is duplication of key information items of Atom in a CMIS property set. One of my colleagues reported an issue to the CMIS TC for CMIS properties that duplicate APP/ATOM information. Duplication produces two problems as any good DBA will tell you:

Wasted resources
Missed updates

Let me illustrate what would happen if the editor's proposal to resolve the issue (to which the TC agreed) (emphasis mine) were to be applied:

I'd like to keep the duplicated info. Originally we did remove the duplicated fields: id, createdby, etc. However, that made some properties special and could not be updated in the same way - by setting a cmis property field. Also, it became less obvious on how to search on those fields.

I do think an order of processing should be stated: cmis:property is authoritative if present, then atompub (or vice-versa).

The bandwidth cost of having that info in two places is not that significant in my opinion.

Now take the case where a vanilla AtomPub client is working with a CMIS server. It attempts to update an existing document entry and changes the content of the atom:entry/atom:updated field but does not change the atom:entry/atom:published andatom:entry/cmis:object/cmis:properties/cmis:propertyDateTime[cmis:name=UpdateDate]/cmis:value then, the server will silently ignore the update if CMIS comes first.

However, version 0.61c1 of the AtomPub bindings for CMIS states:

When POSTing an Atom Document, the atom fields take precedence over the CMIS property field for writeable properties. For example, atom:title will overwrite cmis:name

If this does make it to the final version of the spec, then all will be forgotten, although I do think that this little footnote deserves greater attention.

I hope the lessons from my analysis of CMIS prove useful for the numerous activities (e.g., SunCloud, GData, Netflix) currently underway to develop AtomPub interfaces to various applications.

Atom: Multiple links with same rel value

2009-05-13T10:39:00.000-07:00

The Atom syntax RFC stays away from specifying too many restrictions on the atom:link element, which is quite useful in a number of scenarios. Here are two that are of immediate value:

Atom neither prohibits content inside the link element nor ascribes any particular meaning to it. The hierarchy-ID uses this behavior to allow servers to inline the content of a particular kind of link, with rel=detail, inside the link element. This example has been updated to be valid by the RNG Atom schema. For example:
<entry xmlns="..." xmlns:h=".../"> <title>A master entry</title> <id>...</id> <updated>...</updated> <author><name>John Doe</name></author> <content>Some text.</content> <link rel="self" href="/collection/master1"/> <link rel="edit" href="/collection/master1"/> <link rel="detail" href="/collection/master1/collection"> <h:inline count="0"> <feed xmlns:app="..."> <id>...</id> <app:collection href="/collection/master1/collection"> <title>Detail collection</title> <app:accept h:role="detail"> application/atom+xml;type=entry </app:accept> </app:collection> <link rel="self" href=""/collection/master1/collection"/> </feed> </h:inline> </link> </entry>
The highlighted portion shows an empty detail feed inlined in the master entry. This saves a round-trip and invents minimal new markup.
Another use arises when a certain link is repeated a number of times. For example, CMIS shows a good number of such examples where a single folder entry can have children of four different kinds - policies, relationships, documents, and descendants. In another example, this time using Netflix REST API, a Netflix title entry can have children of 7 different kinds - awards, directors, cast, screen formats, episodes, seasons, languages. Each different kind leads to its own feed/collection. So instead of creating dozens of new application specific relation registrations that have no use in any applications besides the one that create them, just provide for a new attribute on atom:link elements to distinguish the kind of content in that link. Heck we could even use the atom's category term attribute for this. Here's an example demonstrating my point.
<entry xmlns="..." xmlns:h=".../"> <title>A folder entry</title> <id>...</id> <updated>...</updated> <author><name>John Doe</name></author> <content src="..."/> <link rel="self" href="/folder;metadata"/> <link rel="edit" href="/folder;metadata"/> <link rel="detail" type="application/atom+xml;type=feed" title="Folder children collection" href="/folder;contents/" h:kind="contents"> <h:inline count="0"/> </link> <link rel="detail" type="application/atom+xml;type=feed" title="Folder relationships collection" href="/folder;relationships/" h:kind="relationships"> <h:inline count="0"/> </link> <link rel="related" type="application/atom+xml;type=feed" title="Folder descendants feed" href="/folder;descendants/" h:kind="descendants"> <h:inline count="0"/> </link> <link rel="detail" type="application/atom+xml;type=feed" title="Folder policy collection" href="/folder;policies/" h:kind="policies"> <h:inline count="0"/> </link> </entry>
This approach does away with creating new IANA registrations and allows the API developer to control the semantics of the term associated with rel=detail links. Google already uses repeating links with the same rel value in their Google Documents API. They use rel="http://schemas.google.com/docs/2007#parent", but the point is that there is an alternative to inventing new rel values, especially when no new Atom data model semantics are involved.

UPDATE: Julian corrected me in the comments below. So the second example uses h:kind instead of Atom's term attribute.

UPDATE 2: Albert corrected me in the comments below. So the examples above are corrected and do not introduce Atom markup directly under atom:link elements.

CMIS VIII: Futile to model Atom using XSD

2009-05-12T16:58:00.000-07:00

The CMIS AtomPub bindings include a W3C schema for XML, aka XSD for Atom, in a file called atom.xsd. I wanted to warn potential users of that schema about the perils of validating against that schema.

First of all, Atom's syntax has been specified using RNG. There are programmatic Atom validators to ascertain whether an Atom document meets the requirements of RFC4287. There is a very good reason to not use XSD for modeling Atom - its open extension model.

If you have done any application programming with Atom formats, you would know that Atom allows foreign mark-up to be used anywhere (pretty much). That means you can arbitrarily interleave the elements from Atom's namespace with those from a foreign namespace. XSD does not have any mechanism to allow this. For example, in the document below, the highlighted mark up could occur anywhere except after the entry close tag:

<feed xmlns="http://www.w3.org/2005/Atom" xmlns:f="http://example.com"> <title>Example Feed</title>
<link href="http://example.org/"/>
<updated>2003-12-13T18:30:02Z</updated>
<author>
<name>John Doe</name>
</author>
<f:foreign>something<f:foreign>
<id>urn:uuid:60a76c80-d399-11d9-b93C-0003939e0af6</id>
<entry>
<title>Atom-Powered Robots Run Amok</title>
<link href="http://example.org/2003/12/13/atom03"/>
<id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
<updated>2003-12-13T18:30:02Z</updated>
<summary>Some text.</summary>
</entry>
</feed>

The only restriction on foreign markup is that it cannot occur directly under the feed element anywhere after the first entry element. I highlighted a problem in CMIS with this concept in a previous blog post that explains why hasMoreItems results in invalid Atom documents.

CMIS VII: hasMoreItems and paging

2009-05-12T16:43:00.000-07:00

In the example for folder children included with the AtomPub bindings for CMIS, I noticed a weird thing just before the end of the document:

    ...
  </ns3:entry>
  <ns1:hasMoreItems>false</ns1:hasMoreItems>
</ns3:feed>

Now Atom's liberal content model allows various kinds of foreign markup. However, Atom very carefully prevents any foreign markup from occurring at the end of a list of entries. In other words, it is illegal to put anything in a feed document after the first entry besides another entry element.

That means if you want to indicate that this feed document is the last in a series of pages containing the children of a folder, that foreign markup has to come before the first entry in the feed document.

Alternatively, just use Atom paging and archiving and be done with it. In fact, AtomPub has its own paging, if that meets the needs of CMIS (no reason why it can't).

CMIS VI: A case of unpredictable relations

2009-05-12T16:09:00.000-07:00

In CMIS AtomPub extensions, an entry may optionally specify two relations:

If this is a folder entry, it only specifies children link if it actually has children
If this is a folder or document entry, it only specifies parent link if a parent exists.

There is a basic problem with this approach. Suppose that a folder is empty, i.e., has no children. How can I find out the URL to which I can POST an entry to create a document? Ideally, this should be solved in the manner proposed by hierarchy-ID where a link with rel="detail" is automatically created for folder type of entries. Thereby, if a folder is empty, its detail link would produce a feed with no entries, but include enough information to describe that the collection that produced the feed, i.e., the folder resource, is willing to receive new entries and the URL at which a POST request will result in a new document entry being added to the folder would also be disclosed. Here's what I mean in terms of concrete syntax:

<entry>

  <title>Folder metadata</title>

<link rel="self" href="/a;metadata"/>

<link rel="edit" href="/a;metadata"/>

<link rel="children" href="/a/" title="document children collection"/>

<id>tag:example.org;2009:1</id>

...

</entry>

<feed>

  <title>Folder contents</title>

<link rel="self" href="/a/"/>

<link rel="parent" href="/a;metadata"/>

<id>tag:example.org;2009:1-1</id>

...

<app:collection href="/a/">

<title>Folder collection</title>

</app:collection>

</feed>

This mechanism works in all cases (for parent and children relations), is auto-discoverable and hence works beautifully with standard AtomPub clients.

One thing to notice here is that I am using a link to the parent folder entry from the folder children feed. Since CMIS allows the possibility that an implementation may or may not allow multiple parents, the optimization for a single parent system is that the parent link returns an entry as opposed to a feed with a single parent entry. This makes it possible for a client to know that it cannot create more parents.

On a related note, I found that the parent link is optional for document entries. What does it mean for a document to not have a parent? It sounds like another major headache for clients (especially those who are synchronizing for disconnected use) to know under what conditions certain entries will outlive their parent folders.

CMIS V: Semantics of multi-parent document entries

2009-05-12T15:45:00.000-07:00

First a disclaimer: I haven't read the CMIS domain model completely, but I think I understand the premise broadly.

CMIS allows a document to be hard-linked from several folders. Therefore the same document entry may occur in several folder children feeds. The CMIS AtomPub extension spec does not clarify whether the entry occurring in different folder feeds corresponding to a shared document is the exact same Atom entry. (Atom uniquely identifies entries using the atom:id element.)

Let's take an example.

<feed>

<link rel="self" href="/a/b"/>

<id>tag:example.org,2009:1</id>

<entry>

<id>tag:example.org,2009:2</id>

<link rel="edit" href="/a/1"/>

&content src="..." type="..."/>

...

</entry>

...

</feed>

<feed>

<link rel="self" href="/a/c"/>

<id>tag:example.org,2009:3</id>

<entry>

<id>tag:example.org,2009:2</id>

<link rel="edit" href="/a/1"/>

&content src="..." type="..."/>

...

</entry>

...

</feed>

If by some deduction we find out that both representations above are of collection feeds, then it is going to raise some questions for AtomPub. I have not seen two completely different collections claiming membership of the same entry.

In any case, another unaddressed issue with shared documents is what happens when one of the folders containing it is DELETEd. The behavior specified in Section 3.4.5.4.2.4 is silent about this matter. One of the problems of the CMIS model is that it extends Atom's semantic model and creates encumbrances across collections. AtomPub steers clear of this problem, but CMIS will need to address this issue as it defines a specific data model that includes this problem.

CMIS III: Gratuitous CMIS markup on Atom elements

2009-05-12T15:06:00.000-07:00

I never quite understood the need to enlist the help of cmis:id on atom:link elements, which is called for in Section 3.2 of AtomPub bindings for CMIS. There are prominent signals about its importance, but no justification of its needs in AtomPub link relations.

For example, here's what a atom:link element would look like if generated by a CMIS server:

<atom:link rel="repository" cmis:id="..." href="..." type="application/atomsvc+xml"/>

CMIS AtomPub extensions make the presence of cmis:id mandatory, which is yet another encumbrance on using existing AtomPub clients.

Even though this requirement looks silly, I was disappointed that the editors of the spec did not follow up with examples that showed the use of cmis:id. For example, the document entry example in the specification package does not use cmis:id on any of the links including for relations introduced by CMIS.

CMIS IV: Confusing feeds and collections

2009-05-12T14:49:00.001-07:00

CMIS AtomPub extension does not see any difference between feeds and collections, even though they are vastly different in Atom's data model. A feed is a representation of a resource whereas a collection is a resource and it defines the methods (and its semantics) and representations that can be exchanged with it.

CMIS provides a queer AtomPub service document that contains among other things:

A collection with cmis:collectionType="rootchildren" that is aimed at holding the folders and documents that are lodged in the root folder.
A second collection with cmis:collectionType="rootdescendants" that is aimed at holding all the folders and documents in that repository.

It is perfectly fine to have two such feeds, but, IMHO, it makes no sense to have two such collections. In fact it makes no sense to have any collection of descendants. There can only be a feed of descendants.

To draw an analogy to a simple file-system perspective, there is no such thing as adding a file to the recursive listing of a directory. You can only add a file to a directory.

Also, since a CMIS server is going to end up creating such a large number of collections, it makes sense to adopt a standard mechanism for discovering collections backing feeds and their metadata like how we describe in the hierarchy-ID.

CMIS I: AtomPub binding and AtomPub service documents

2009-05-12T12:22:00.000-07:00

This is the beginning of a new series of posts on the still-nascent AtomPub binding of CMIS. Although, this spec has been seen by over 20 companies (led by IBM, Microsoft, and EMC that are participating in its TC) over six months now, I doubt anyone involved in writing this spec has ever written an AtomPub server or client.

The section 3.4.2 of the CMIS AtomPub "binding" specification is quite illogical if you follow through these arguments:

A repository is represented by an Atom service document.
Within the Atom service document, each workspace maps to a single repository.

Firstly, there is no such thing as an Atom service document. The spec is referring to an AtomPub service document.

Secondly, if Repository ≊ Service Document and Service Document contains multiple workspaces, then how can each workspace ≊ Repository? May be the spec should have stated that the AtomPub service document corresponding to a CMIS repository MUST contain exactly one workspace.

Thirdly, the CMIS spec says nothing about the constraints on what kind of entries the server can accept into its various collections. This is information that is typically advertised in the AtomPub service document. Two things can be currently specified in this:

What MIME type of data is acceptable to the server for adding to the collection
What categories are acceptable to be used on entries being posted.

For a moment, disregard the second part since AtomPub's use of categories is the worst thing about RFC 5023. At least the CMIS spec should state how the first part is to be addressed. Here's what could be a decent CMIS service document:


<service xmlns="..." xmlns:cmis="...">
  <workspace>
    <collection cmis:collectionType="rootchildren">
      <atom:title atom="...">Root children collection</atom:title>
      <accept>application/atom+xml;type=entry</accept>
      <accept>text/*</accept>
      <accept>image/*</accept>
      ...
    <collection>...</collection>
  </workspace>
<service>

This example states that the collection can accept new entries (for example, creating new folders). Unless the CMIS spec requires this, there is no means to address introspection needs of AtomPub clients. The example above also shows how documents could be created as well that are either images or text.

CMIS TC has invited comments from the AtomPub community on this spec but the spec provides no examples of their use of AtomPub. If they went through this exercise, they would find a number of such illogical things that I am going to highlight in a multi-part blog series.

Stay tuned...

CMIS II: (C)MISsing good old AtomPub

2009-05-12T11:42:00.000-07:00

In my earlier post on AtomPub vs. CMIS, I expressed concern that new constraints on Atom syntax introduced by CMIS go so far as to rule out using a standard AtomPub client with a CMIS AtomPub server. In theory, one could say that adding a new required element to Atom's schema is a non-starter. However, if there is a desire for CMIS servers to support standard AtomPub clients, then the current spec is quite far from that goal.

Let's see the consequences of new constraints on the AtomPub and Atom syntax used by CMIS servers. A non-CMIS client will be unable to talk to CMIS even for use cases so simple as adding a document to a folder. Consider the following request to the root folder of a CMIS repository:

POST /folder
Content-Type: application/atom+xml
Content-Length: nnn

<entry xmlns="...">
  <id/>
  <content type="text">
    Can I live inside CMIS?
  </content>
  <updated>...</updated>
</entry>

Even though the CMIS AtomPub "binding" doesn't state this clearly, and is a fantastic example of how NOT to write a RESTful specification, in my understanding the above request will fail with a 400 status code (not clear which one). This is because:

CMIS doesn't state acceptable content in an AtomPub understandable manner anywhere.
Even if it were to advertise accepted content, a compliant CMIS AtomPub server could reject a POSTed entry if it did not meet CMIS expectations. Thus you could not expect the server to take the text from the atom:content element and save it as the document content. I am basing this on reading the following paragraph in Section 3.4.4.3.2.2:
- The behavior is repository specific when a non-(atom/cmis)-entry document is posted to a folder URI. For example, the repository MAY auto-create a document with a specific type (document) the client could edit. The repository MAY also throw an exception.
Nowhere does the document state what makes a valid CMIS/Atom entry document. As a saving grace, the specification package includes an example document entry. This example uses out-of-line content and an edit-media link element.
Let's assume that the server allows our entry through and creates a new document using it. Now the entry that is stored is full of duplicate information.
1. The content stream location is in two places: atom:entry/atom:content@src and atom:entry/atom:link[rel=stream]@href
2. The time when the entry was created is in two places: atom:entry/atom:published and atom:entry/cmis:object/cmis:properties/cmis:propertyDateTime[cmis:name=CreationDate]/cmis:value
3. Also repeated is information such as atom:entry/atom:id, atom:entry/atom:author and atom:entry/atom:updated

There is more material for many more posts. So stay tuned...

Discovering AtomPub Collections - best current practices

2009-05-11T09:10:00.000-07:00

When working with data synchronization using Atom feeds in AtomDB, we frequently encounter situations where we learn about a feed simply through its public URL. However, most feed documents do not provide any indication of whether new items can be added to them. (This is not because of a lack of existing standards but because existing standards are either not well known or are unsuitable.)

Some assume that if a feed is generated from some well known AtomPub server, then it must be modifiable. Of course, specialized AtomPub clients, such as for CMIS, dealing with specific feeds can always use out-of-band communication to figure this out. The problem is that standard AtomPub clients that are provided only a feed URL have no way of figuring out such assumptions.

There are two alternatives:

James Snell has suggested the use of "rel=service" but that tends to not be present on any of the feeds we see. It is mainly designed for use in HTML pages for discovering AtomPub services.
Section 8.3.5 of RFC5023 specifies the semantics of an app: collection element appearing as a child of atom:feed element. This mechanism is very useful to us for discovering whether a feed is modifiable and, if so, how it may be modified using AtomPub. It helps in situations where there are far too many feeds to be enumerated in a service document as well as where an implementation does not use AtomPub service documents. Lotus Connections uses this approach in its feeds (among other non-standard apporaches).

James Snell also suggests using app:collection inside of atom:entry elements. Lotus Connections does this in its feeds. However, per 8.3.5 of RFC 5023 there is no meaning ascribed to putting app:collection in atom:entry documents. So I am not sure it has any standard interpretation at the moment.

The downside of the first approach is that it requires a round-trip to find enough metadata about editing the feed. Another problem is that if collections are programmatically produced, the service/workspace/collection metaphor easily breaks down. This part of AtomPub is the least suitable for application feeds as evidenced by their absence in AtomPub usage within GData, OpenSocial, as well as Netflix APIs.

The downside of the second approach is that it is repeated in every feed document, and hence consumes more resources. Secondly, this approach is a little verbose and wouldn't be understood at all in HTML documents.

CMIS vs. AtomPub Hierarchy I-D

2009-05-08T09:57:00.000-07:00

Led by Microsoft, IBM, and EMC, OASIS has been working on a modern standard for content management systems, called CMIS. This group has started from a submission made by the above three vendors called CMIS v0.5.

This standard also includes a "binding for REST/Atom". In layman terms a binding is an adaptation of the abstract standard to a specific network protocol and format. Julian Reschke suggested that we hash out the overlap between CMIS AtomPub extensions and the AtomPub hierarchy-ID. Here's a summary of the most important areas of disconnect.

CMIS genuinely tries to provide a good AtomPub extension, but fails on at least three counts:

Link relations defined by CMIS cannot be used by a client that does not understand CMIS. So if you would like to have parent-child relations but don't want to put a cmis:id attribute (yes, that is a SHOULD level requirement), then you are out of luck. On the contrary, rel=master and rel=detail from hierarchy-ID do not impose any additional requirements.
There is no way to auto-discover the POST url for a feed. For example, you obtained a list of documents in a folder and now want to add another document to that folder. Without understanding rel=repository and cmis:collectionType, there is no way to do that. On the contrary, hierarchy-ID will be understood even by existing AtomPub clients, and these clients will be able to add another item next to the existing document.
The CMIS spec defines PUT and DELETE methods on an atom:feed type resource (yes, see Section 3.4.5.4 and 3.4.5.6). I would imagine that the CMIS spec would need to go to IETF for doing something like this because they are in effect extending the semantics of the application/atom+xml;type=feed MIME type. No such extraordinary extension is going on in hierarchy-ID and this ID is before the IETF for consideration.

Now, we have every intention of engaging the CMIS to help them understand the consequences of these issues, namely a monolith hanging off a lean protocol that can only be used for content management and improving the use of existing AtomPub metadata for doing simple things like discovering the URL for modifying a feed. Whether that will have any useful outcome is anyone's guess.

Hierarchy in AtomPub

2009-05-05T11:09:00.000-07:00

We have submitted the AtomPub hierarchy I-D to IETF for consideration along Standards Track. Although there is no working group looking at extensions to Atom or AtomPub, and there is not so good feeling about the future of AtomPub, there seems to be a very good use for Atom and AtomPub for document-oriented data.

Feedbooks is planning to use the hierarchy approach (hat tip Peter Keane) described in the I-D in its future versions, which means there will be greater usage experience to go around and improve future revisions of this I-D.

Getting to Offline Web data via BITSY

2009-04-27T13:51:00.000-07:00

There are probably two camps of developers interested in WebStorage - one that develop local applications and another that develop offline-ready applications. The first camp doesn't really care for the data that lives on the server; it is just a back up of the data that lives locally. The second camp has tons of data that is shared among users and was previously only available in a connected mode. The first camp probably doesn't deal with HTTP or REST abstractions for their data but the second one has to deal with the HTTP interface to data as well as situations in which offline access to that data is required.

For the second category, IMHO, it may be better to focus on getting the right HTTP abstractions emulated locally for offline purposes. If people are going to need to upgrade their browsers for off-line use, then might as well provide them one that will allow them to run an "embedded REST server" within the browser. I mean let people write their own server emulation logic in JavaScript using whatever storage that's available to them in the browser, whether it is key-value pair or SQLite or another, and serve offline requests over the same interface that the server does - HTTP and URL.

I have a proposal before the Webapps WG at W3C to consider adding JavaScript interceptors to enable various kinds of offline operations to take place without necessitating any standards about the SQL storage and access.

Ideally, combine my proposal with something Atom-like but more query-friendly, such as Shoji, which allows a rich set of queries using URI Templates. I haven't fully understood Shoji or verified that it would work well for specifying and processing queries that are in the 80% scenario, so I am going out on a limb there.

How is BITSY different (from HTML, Dojo Offline, Gears, WebStorage)?

2009-04-27T09:33:00.000-07:00

Just last week I released a new draft version of BITSY that dramatically reduced its scope to provide simple primitives that can be used with different formats and protocols.

That this continues to raise the basic question - how's it different from everything out there - is an interesting problem in and of itself. This question has been asked several times on the public-webapps ML (Maciej Stachowiak, Kris Zyp, Mike Smith, Jonas Sicking, and Anne van Kesteren) in a few different contexts. So I am trying to summarize the answers here for future reference.

Contrasting with ApplicationCache in HTML5

The HTML5 ApplicationCache is one specific use case of providing a durable, read-only offline store of HTTP resources inside the browser. The browser is itself a client of the offline cache and uses a specialized protocol to perform caching of application resources such as JPG, HTML, CSS, and JS files.

A special file format is used to convey the required resources and a special versioning scheme is used to move from one version of files to another in an atomic manner. That way you wouldn't have an old JS file and a new HTML page. It is critical for browsers to have this atomicity, but that is not necessarily the case with other applications running inside browsers. Their needs are more dynamic and their behavior in failure scenarios is more flexible than a browser's internal offline store.

ApplicationCache does not allow programmatic inclusion of items (dynamic entries were removed some time ago); all data capture in BITSY is through an API, i.e., as a dynamic entry
ApplicationCache does not secure one user's private resources from another; BITSY requires the presence of a specified cookie
ApplicationCache only responds to GET and HEAD requests; BITSY can respond to arbitrary HTTP requests
ApplicationCache does not allow an application to intercept any requests locally; BITSY allows application-defined JavaScript code to intercept requests locally
ApplicationCache uses its own data format for identifying items for local storage and exludes any other formats such as JSON and Atom; BITSY does not have any format limitations
ApplicationCache operates per its own refresh protocol and that excludes a different protocol, especially one that does not require all or nothing semantics for data versioning; BITSY has no protocol limitations.

Contrasting with Gears LocalServer

Gears is quite close to BITSY's primitives in terms of dynamic entries and capturing locally generated data. However, it is primarily a means of serving read-only requests for applications.

LocalServer only responds to GET and HEAD requests; BITSY can respond to arbitrary HTTP requests locally
LocalServer does not allow an application to intercept any requests locally; BITSY allows application-defined JavaScript code to intercept requests locally

Contrasting with Dojo's Offline Toolkit OfflineRestStore

For one, you will need to cede all data access logic to the library, in this case Dojo, in order to use offline serving. This would be true of any other library. However data is not always used in this way. Your HTTP caching choice should be independent of your data access library.

Dojo offline API provides a means of logging pending network requests and accessing data through a special API (that is different from page navigation, XHR, or other kinds of network requests originating in a browser); BITSY does not require any special API for data access because it is present in every HTTP access path.
Requests are queued for immediate delivery to the server, even if connectivity is available. This means that your normal path is always local, even if you didn't want to. BITSY's review policy means that the normal path is not affected when browser is connected.
Dojo switches between the network and local store based on the navigator.onLine property; BITSY can use any intelligence available about battery power, connection latency and bandwidth, location, and so on to decide whether to perform a network operation or satisfy a request locally. Heck, it can even pin the app request to the local store to get maximum responsiveness.
When using the OfflineRest Dojo API, an application cannot emulate the server. Using BITSY's interceptors, a complete server can be emulated inside the browser.
Dojo cannot store non-textual data such as images or video; BITSY allows capture of arbitrary server-hosted resources
OfflineRest can replay PUT, POST, and DELETE requests, provided they all do the thing Persevere understands. If you wanted to use a different semantic or if you need to use extended HTTP methods, you are out of luck.
Dojo's synchronization protocol is internal to its OfflineRest implementation. That makes it hard to resolve conflicts or adapt to your server's requirements. With BITSY, there are no limitations on how to synchronize. It might be a little daunting, but it means any library developer can take advantage and build a suitable sync library.

BITSY 0.5.0 - Develop seamlessly on-line/off-line Web applications

2009-04-24T15:13:00.000-07:00

We have been working for a while to make it possible to perform HTTP processing from within JavaScript code. This means that you can deal with off-line situations as well as improve responsiveness and availability of applications.

Now we have a new draft of BITSY, version 0.5.0, which explains exactly how you can seamlessly access data on-line and off-line using application-supplied interceptors.

This draft is based on the availability of pluggable protocol handlers available in Safari (NSURLProtocol), Internet Explorer's (URLMon), and Firefox (nsIHttpChannel and nsIHttpProtocolHandler). We have developed an implementation of this new BITSY draft for Safari and Firefox for Linux, MacOS, and Windows platforms as a browser plug-in.

In a major distinction from AtomDB, the implementation of BITSY 0.5.0 is completely independent of Atom and XML. This allows custom HTTP-based protocols to be combined with transparent off-line data access and manipulation. To test BITSY, we have developed JavaScript libraries for Atom Publishing Protocol based synchronization.

Now you can perform your own RESTful data synchronization or create and distribute libraries of RESTful data synchronization.

Limiting access to locally persistent data

2009-04-13T16:26:00.000-07:00

Sam Ruby worries about security of locally persistent data, for example, in WebStorage:

Having a Web Storage mechanism will enable people like me, who aren’t security experts, to design applications which capture more sensitive data which could be exposed with the right XSS attack.

A sane way to protect locally persistent data is to try and protect it the same way server resident data is protected. For example, if an authentication token is required in the form of a cookie to access protected parts of the site, the same token must be good to protect data that came from those protected parts.

Let's work through this example:

Some JavaScript requests URL X from its server. GET X
Server challenges the browser for auth. 401 WWW-Authenticate: Cookie form-action="/login?redirect=X"
Browser shows the form-action page. GET /login?redirect=X
User enters credentials and submits the form. POST /login/do
Server checks credentials and responds with the redirect and an auth token. 303 Set-Cookie: C=Y, Location =X
Browser redirects to the original location along with the cookie. GET X Cookie: C=Y

Now when the page asks browser for persistent storage, it should be able to identify the cookie used to secure the data stored locally. For example:

window.createDatabase('myDB', 'C').

The browser automatically uses the current value of the cookie 'C' to restrict access to 'myDB'. Next time, when I access 'myDB' by doing:

window.openDatabase('myDB', 'C')

then the browser verifies that the current value of cookie 'C' is the same as that recorded when 'myDB' was created. If it is not, the request would fail.

My proposal, in effect, is to ensure that data access can only take place if the authorization token that the server verified in order to provide the private data is, in fact, present at the time of accessing that same data.

This protects against scenarios where different directories are protected differently on the server. It also avoids asking the JS code any questions about the current value of a cookie and does not pass any cookie values to the JS code.

Of course, if the cookie goes bad after a while, the locally persistent data is also useless. Unless the local data is somehow flushed out to the server periodically, this could be a big deal. Otherwise, you can just refresh from the server using the current auth token and go from there.

Web Storage needs Free/Open/Simple Sync as well

2009-04-13T15:57:00.000-07:00

Persistent storage on Web clients, especially where augmenting data stored on server, MUST be friendly for synchronization. This is the golden rule for offline applications that we have learnt from 10 years of offline CRM applications in Siebel (now Oracle).

SQL has done nothing to make it easier to sync unless you are willing to pay $$ to license complex sync software. Why can't we make simpler sync software for Web? Well the answer to that question lies in our ability to simplify data access abstractions in Web applications. Sync/offline can be made easy, cheap, and automatic if the following are acceptable:

Identify data through URL
Operate on data with HTTP methods
Provide JavaScript interception on XHR (just like Apple provides Objective-C interception on NSURLRequest and URLMON provides Asynchronous Pluggable Protocol).

At Oracle, we have a working implementation of all three for Firefox 3 and Safari, which when combined with a generally acceptable query language can provide an easy to sync, secure, persistent storage solution. This approach is biased towards document-oriented data models, which REST and Web are already predisposed towards. Moreover, it does not require extensive synchronization software to be buried inside the database itself. Unlike CouchDB whose synchronization code lives alongside the db itself, synchronization in our implementation is performed in JavaScript thus giving as much or as little sync flexibility as required.

Oracle can make this technology available to anyone who likes this approach over the current DIY sync and at-your-own-risk SQL storage available in some browsers. Contact me if you would like to know more about this or get your hands on the technology or develop an alternative to the current storage mess in Web browsers.

To SQL or not to SQL - that is the question

2009-04-12T22:53:00.001-07:00

Members of the Web apps WG of W3C are debating what access model is appropriate for persistent storage on Web clients. HTML5 WD [1] used to include a section on "Structured client-side storage" until February this year. Then, after a long discussion on the scope of the draft, Ian Hickson carved out various pieces including persistent storage to their own separate spec drafts.

The current editor's draft of WebStorage continues the model from HTML5's previous working draft of SQL/relational access. This was intended to build upon the SQL support available in Mozilla as a non-standard component as well as the inclusion of SQL storage in Google Gears. Even the iPhone and BlackBerry browsers provide SQL storage inside Web browsers based on an a previous version of WebStorage.

In March, the Ian Hickson of the Web Storage draft made a request for publishing his draft as a FPWD to the Web apps WG whose charter includes "Offline APIs and Structured Storage for enabling local access to Web application resources when not connected to a network." This is at variance to the abstract of the current editor's draft of Web Storage which states that the spec includes an API for "a database system with a SQL frontend." (emphasis mine)

A discussion on the use of SQL in a data access API and its open-ended nature has been going on over at the public-webapps mailing list. The main pro- and con- points of the discussion are:

There is no definition of what constitutes SQL in the API.
The only two public implementations of this API - WebKit and Gears - and both are using SQLite, whose implementation doesn't conform to any public SQL standards.
Basing the standard on a limited SQLite syntax (and semantics) compromises interop
There is a need to consider other options for structured data management. Some of those named are OODB, jLINQ, CouchDB, Persevere. Among query techniques come - JSONQuery (similar to XQuery & XPath, but for JSON), and CouchDB's view mechanisms.
Providing access to multiple different database mechanisms via a single API does not solve the original problem that there is no standard vocabulary for any one of them.
The behavior of a simple operator such as equality test is different for different implementations.
Even though there is "no such thing as standard SQL", it is not clear that inventing a brand new query language and database model for Web clients' structured storage is better.
Since applications are beginning to use the SQL storage in browsers, there must be something good about it. So let's just hash out the details and get this done.
Since very few use an OODB, it would not meet the requirements of Web Storage
There's lots of competence and experience with SQL out there. And there's something to be said for the fact that SQL has proven itself as a usable language at this point. I doubt that we could design a language that cover as many use cases as well as SQL does. Though of course we might be able to still be able to follow the 80% rule if we design our own language.
SQLite wasn't the first browser-accessible DBMS, nor is it the most ubiquitous choice of target. IE's Jet database engine, which is the underlying engine for Access, would seem to be the most useful target specification.
There are strong arguments for not breaking existing content, of course, but there are also strong arguments for not having experimental implementations of early drafts completely dictate the standardization process.

My contribution to this discussion so far is:

Many CLI standards have been based on SQL 92, but they allow all kinds of extensions. Not only that, Web Storage is more than CLI, it also includes an embedded database.
the question in my mind is not whether an unanchored reference to SQL is fine as much as whether SQL is the right way (and for years to come, the only structured way) to think of Web application's (locally persistent) data.
There is no end to how much Oracle and various other companies shield developers using their platforms from using raw SQL. There are more reasons for that than I can list here, but suffice it to say that the Web Storage spec should consider techniques that are better matched to the Web as a data access platform - i.e., in terms of URLs and HTTP methods
building applications on WebbKit is fine, but anyone can rely on the SQL API, at this time, at their own risk. It does not justify why the SQL technology is superior to other choices, however preliminary they might be. Or you risk going down the path of EJB which only 'got persistence right' after 5 iterations and only when the public declared a revolt and completely gave up on J2EE persistence.
SQL as a given cannot be assumed to be the only acceptable query language for the data being managed. Problems of this model include -
- Problems caused by version mismatches between server and client schemas significantly affects decentralized evolution.
- The relational model does not allow data for which there is no schema placeholder, thus rejecting the principle of 'ignore what you don't understand'.
- Incompatibility between database ids and URLs means that applications live a schizophrenic life when it comes to identifying data.
- Incompatibility between database operations and HTTP methods means that atomicity and idempotence guarantees of HTTP are inapplicable to local data

Overriding default HTTP behavior in Mozilla

2009-02-23T17:28:00.000-08:00

Faced with Darin Fisher's puzzle of overriding the default HTTP behavior in Mozilla, I am coming up short on my implementation.

I have done all the programming and developed a component that implements nsIHttpProtocolHandler and is installed to the component registrar as offering MyHttpHandler (my custom handler) with the "@mozilla.org/network/protocol;1?name=http" contract ID. However, this handler doesn't seem to get picked at all.

Here's how I went about my custom component in a Firefox add-on (in native code):

Create a module using the routine module entry point. The entry point returns a list of components that includes MyHttpHandler's ClassID, ContractID, constructor, and registration routine
In the registration routine, add to the http-startup xpcom-startup category the name value pair for my MyHttpHandler and its contract ID
MyHttpHandler implements both nsIHttpProtocolHandler and nsIObserver interfaces
MyHttpHandler's Observe method handles xpcom-startup and http-startup events.
When it gets the http-startup xpcom-startup event, MyHttpHandler is registered to override the default HTTP handler. To do this, I use the subject that arrives on the eventNS_HTTP_PROTOCOLHANDLER_CID to get its service from XPCOM as the default handler and chain it to MyHttpHandler. Then I use MyHttpHandler's factory and register it as the factory for MyHttpHandler's Class ID and NS_HTTP_PROTOCOLHANDLER_CID contract ID.
The registration succeeds after the browser is initialized and I can confirm that from the logs.
Oh and yes, I have to ensure that I support the following interfaces in my class - nsIHttpProtocolHandler,nsIProxiedProtocolHandler, nsIProtocolHandler, nsIObserver

What may be missing? Does Mozilla not trust anyone else to write the HTTP behavior or is there something lacking in my code? You can follow this discussion on the Mozilla extension developer mailing list.

UPDATE: I found the source of the problem. I needed to respond to QueryInterface for the nsIProtocolHandler interface in addition to the nsIHttpProtocolHandler interface. Now, I am able to intercept all the HTTP scenarios encountered within Firefox including proxy, https, and, of course, caching.

Replacing the Http/Https protocol handlers and channels, of course, means impacting the browser big time, but the replacement works quite OK. The only concern is how my replacement will be impacted by future revisions to Mozilla's own nsHttpProtocolHandler and nsHttpChannel. That, though, is a story for another day.

UPDATE: I don't listen for http-startup anymore. Instead I do my work at xpcom-startup. There are other updates to the algorithm as well. So be sure to re-read this post to get an updated solution.

New I-D on Atom feed hierarchies

2008-10-31T15:13:00.000-07:00

We have been looking at issues around the publication and consumption of application feeds. Naturally, we come across data that is easily modeled as master-detail relations. A simple representation of this relation cannot currently be performed in any standard or even sane way that works entirely using hyperlinks.

A hierarchical master-detail relation of an Entry to a Feed implies the detail Feed is created when the master Entry is created and the Feed is removed when the Entry is removed. The Entry is called the "master entry" and the Feed is called "detail feed". This relationship allows a client to use AtomPub to create a new Collection by posting an Entry to an existing Collection.

Therefore, I co-wrote an I-D to support master-detail relations among feeds. A discussion was started on the Atom protocol mailing list. You are welcome to leave comments here as well.

Microsoft's take on off-line and data services

2008-10-30T16:52:00.000-07:00

Pablo Castro of Microsoft recently kicked off a discussion on off-line data services for Astoria. He is averse to caching and seamless online/off-line applications. If the data service is provided by Astoria, then there is no problem. However, if the data source is not Astoria-based then sync falls apart, and there is no chance of getting the full set of data locally in a way that guarantees that "all the data required for offline" is stored locally at sync time.

This is no different from the SOAP story, where everyone thought it would enable interoperable services only to find that the technologies were good enough only for intranet solutions built with the same tools and technologies on both sides of the interaction.

If you are interested in seeing advances made in the area of cache syncing standard AtomPub services (which by the way are way more interoperable than Astoria-only data sources), then you should check out the details of AtomDB. My blog argues for a seamless caching approach as Web-friendly as well as open to any standard AtomPub data source. So does the feed technology center on OTN http://oracle.com/technology/tech/feeds/ have details about AtomDB, a project I have been focused on for the last one year.

Non-hypertext "RESTful API"

2008-10-28T14:43:00.001-07:00

Roy Fielding gives a bunch of anti-patterns for applying REST to application interface design. One of them is:

A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations. [Failure here implies that clients are assuming a resource structure due to out-of band information, such as a domain-specific standard, which is the data-oriented equivalent to RPC's functional coupling].

Roy does not give any examples of this anti-pattern. However, I wanted to note an example that will help illustrate the way in which this anti-pattern manifests itself in interface design.

The Google Finance API is a purported RESTful API that constructs URLs inside JavaScript code for managing application state instead of using hyperlinks produced on servers.

Here's how: when creating a transaction for a position, the API defines a URL format to use.

http://finance.google.com/finance/feeds/default/portfolios/portfolioID/positions/TICKERID/transactions

This format is to be used in JavaScript code for preparing a request to submit to the server. The server does provide the URL to the position collection, so it is merely a matter of appending "transactions" to that URL. Still by putting that logic in JavaScript as opposed to a well-defined media type, this design choice fouls up on adherence to REST.

Mobile Monday Silicon Valley

2008-10-08T09:54:00.000-07:00

Yesterday, over 50 people attended Mobile Monday's meeting at Oracle HQ in Redwood City. We discussed data management for mobile applications and considered the use of feeds as data sources as well as the importance of seamless online-offline applications.

The session was very interactive and a large number of issues were discussed along the way. Discussion revolved around the following issues:

Security
- authorization
- encryption
- device loss
- holes in IE mobile browser
Trade-offs in application development vis-a-vis HTML5 storage
Fit for feature phones as opposed to smart phones
Suitability for graph structured data models in general
Support for JSON as opposed to XML feeds

Here's the slide deck I used for the presentation.

Netflix adopts a feed-style API

2008-10-01T11:45:00.001-07:00

Netflix joined the growing list of Atom feed-based APIs for Web-based applications by announcing their REST API Conventions.

Link expansion continues to be an interesting area of innovation in feed-based APIs aimed at reducing the number of requests involved in fetching data in bulk. Another area of innovation is authorization, and the increasing adoption of OAuth as the means of delegating access to third parties has made a good REST API within the reach of many service providers.

Finally, another interesting thing about the API is their emphasis on XML as the choice format, with Atom feeds providing much of the vocabulary such as categories and links. However, they retain POX formats for message exchange, which includes HTTP header information in response bodies. Hopefully, this is vestigial and will be treated as inferior to the feed format.

HTTP and Content Negotiation

2008-09-30T17:30:00.000-07:00

There are three levels of HTTP experts -

Who understand HTTP GET and POST (I'll call them naive)
Who understand uniform methods and conditional operations (I'll call them RESTafarians)
Who understand content negotiation (and its resulting design challenges), HTTP caching (and its industrial use) and idempotence (I'll call them REST masters)

I am certainly in the second category - RESTafarians and try to learn from the REST masters. So when I came across this message from Bill de hOra elucidating the impact of a content negotiation choice, I thought I would aggregate it on my blog.

any solution that depends on conneg needs to define a processing ladder of alternatives (eg, depending solely on Accept: won't fly in the part of the mobile web I currently work in, unfortunately).

Oracle OpenWorld Unconference

2008-09-30T15:38:00.000-07:00

Oracle's OpenWorld was a great opportunity to meet with developers building mobile applications and discussing their data management needs.

It was also an opportunity to do an unconference session around the rich Web application architecture and AtomDB. Those who came (about a dozen or so folks) were thinking how they could benefit from this kind of an architectural move.

Here's the slide deck from the Rich Web Applications unconference session (although the demos can't really be seen in this presentation).

Android, Chrome, and Gears - what's cooking?

2008-09-17T11:16:00.000-07:00

I have been following Google's emergence as a player of note in the applications area and respect them for driving innovations in application interfaces, browser standards, as well as mobile technologies.

The combination and evolution of Android, Chrome, and Gears is suggestive of a vision being laid out by Google about the Web as a platform for building useful applications. Certainly, this would be interest to those of us trying to bridge the gap between enterprise and useful Web.

Of course, Google has tried to improve browser stability with Chrome, but that is not the only reason why they built a Web browser. David Berlind's article on Information Week tries to justify the case that Gears is quite central to Google's browser story.

My guess is that Google sees an offline technology like Gears as being so fundamental to the future of Web applications, that it can't not be built into the browser. The folks at Mozilla (where Gears is an add-on t the browser), on the other hand, probably didn't see it that way.

My blog and Oracle's investment in AtomDB is proof of the fact that Oracle feels that the Web has a lot to offer, and for many applications a key missing piece is the lack of a suitable caching and proxy technology that works under the hood. This story is the best means of transitioning gradually from the current state of affairs in terms of browser depoloyment. You can write an application that works on conventional Web browsers like IE and Firefox which lack the Gears off-line capability and that work even better when Gears is available as either a plug-in or standard inside the browser.

Gives you more than one good reason to switch to Chrome as you get more of your coffee, er applications from the Web.

Oracle OpenWorld - Ajax Unconference

2008-09-17T11:08:00.000-07:00

I have created an unconference session at OpenWorld on Rich Web Applications. The idea is to meet like minded people who are developing Ajax applications for standard Web browsers and are interested in learning how feed technologies can help them standardize their APIs as well as improve the serviceability of these interfaces.

As you know, GData has long offered feed and Atom publishing protocol-based API as a means of retrieving and updating data in various Google applications through a RESTful protocol. Other applications also now offer similar interfaces, e.g., and OpenSocial. The preponderance of such application interfaces combined with their fairly high usage on the Web indicates a good match of this style with the application integration requirements.

Come to the unconference session to discuss your pet peeves about Ajax application data interfaces as well as hear about Oracle's technology proposals to help address some of these issues such as reliability, availability, and responsiveness.

Oracle is hosting Mobile Monday Silicon Valley next month

2008-09-17T10:28:00.000-07:00

Mobile Monday comes to Redwood City next month, as Oracle hosts mobile developers from all over the Bay Area. If you have attended this event before, then the new thing is that Oracle is now taking a renewed interest in mobile infrastructure. Oracle will talk about ways in which mobile Web applications can offer greater responsiveness and reliability, while at the same time tapping multiple sources of data on the Web.

If you haven't come to MoMo before, then you will find it to be a great networking event besides meaty technical topics and a serious discussion of issues faced by mobile developers.

Detailed announcement will follow shortly.

Mobile Web Megatrends

2008-09-17T10:22:00.001-07:00

Ajit Jaokar, a mobile die-hard, organized a small event on the emerging trends of Mobile Web. The event received a pretty good audience of infrastructure developers and Web enthusiasts ranging from browser vendors to ecosystem reviewers.

Of course, Oracle was there too. Our presentation highlighted our interest in making the mobile Web browser a competent environment for developing reliable and speedy applications, and the technology Oracle has developed, called AtomDB, to support this vision.

Information about AtomDB is now available on the Oracle Technology Network as part of its Feed Technology Center.

True character

2008-08-04T13:49:00.001-07:00

How many of us can see this applicable to ourselves? Khushwant Singh, a political and social commentator in India had very prescient words to say about character. I hazard a guess that even companies work the same way.

A person's true character is revealed in the way he behaves in times of crises when he believes his future is at stake. If he is afraid of consequences that may follow and compromises with his principles, he becomes a lesser person in people’s eyes as well as his own: he is unable to see his face in the mirror. On the other hand, if he sticks to his avowed beliefs, he may suffer adversity but people will admire him as an example of rectitude and he can hold his head high with pride.

Architectural Evoution of Commercial Computing

2008-07-31T15:53:00.002-07:00

Discussions about the evolution of styles of software architecture used in commercial computing often veer towards the usual cyclical story. However, it is not as simple as some would like to think of it.

Take a look at the following image of the major innovation in software architecture during the last six decades and see how each culminated in the pronouncement of a style as the pre-eminent one. Every time a style gained prominence, the next decade led to unlearning as new computer architecture, networks, and tools led to another new style's emergence.

Do you agree with the trend line here? Do you know of others that are better at telling this history?

Uniform data synchronization for mobile data access

2008-07-15T10:42:00.000-07:00

As I mentioned in a previous blog post, Mobile Monday Austin hosted a session on mobile data access allowing me an opportunity to present Oracle's work on AtomDB. Here are my slides used in the session titled "Mobile Data Access using AtomDB".

Among many interesting questions I heard, the most interesting one was "How do you perform local authorization/authentication?" I say this because like many other issues with local processing for greater responsiveness/higher availability this issue also forces us to transparently provide local behavior without imposing additional burden on servers or clients.

The answer to that is that we do not perform local authentication. However, once a principal is authenticated, applications inform AtomDB about the required authorization through the JavaScript control API for AtomDB called BITSY. Through this API, applications provide the HTTP Authorization or Cookie header used to ascertain whether a certain data access request is sufficiently authorized or not. AtomDB sends this header in requests to the server and records responses. Only if later the application uses the same header for the data access request will the recorded responses be provided to it. If there is a mismatch or a missing header, then the response is 401 Authorization Required.

Mobile Monday Austin in July

2008-06-27T09:41:00.000-07:00

I will be speaking at MobileMonday, Austin on July 14. C. Enrique Ortiz has invited Oracle to discuss challenges for mobile applications in the context of the rich Web. Here's a synopsis of the talk:

There is heavy latent demand for mobile applications and recent industry events have heightened interest in mobile Web applications. Mobile application developers face a number of inherent challenges of mobility such as form factor and user interaction. Besides these, they are also forced to deal with incidental issues such as platform fragmentation and unreliable connectivity. Various “Rich Internet Application” platforms are currently targeting these problems. Oracle, of these, is interested primarily in standards-based platforms, i.e., Web browsers and AJAX. Therefore, Oracle has developed a solution to the network unreliability problem in the context of Web browsers, Oracle’s AtomDB . This technology provides a transparent, local, read-write cache for application data that is synchronized with data sources using the same REST APIs used by AJAX applications to access their data. This talk will focus on the dual use of IETF’s Atom publishing protocol for application programming as well as data synchronization. He will also discuss areas of future work in off line data for Web applications.

Mobile databases or write-through Web caches

2008-06-27T09:34:00.000-07:00

Finally, Oracle is going public with the work we have been doing on Web data, specifically in the context of mobile devices. In the past, most developers, including Oracle, have developed full-fledged relational databases for mobile devices. Our group at Oracle has took a different approach due to the uncertain outlook on connectivity and the need for responsiveness and availability over consistency.

We have developed a tiny embedded database, rather a write-through cache for mobile Web browsers that provides transparent access to Web data for read-write operations. A technical paper on our approach describes AtomDB, the motivations for it, and the future work to make AtomDB a core component of the rich, mobile-ready Web. Here's an excerpt from that paper Going far without the bars:

Mobile application developers can’t avoid human computer interface challenges, limited network connectivity, and harried users, but they shouldn't have to deal with new data storage and communication models and synchronization issues. In this paper we explore how the AJAX application model and the Atom data model, widely used for on-line applications, can be combined to create seamless on-line/off-line model for mobile applications. We show how existing applications can be made robust to intermittent connectivity and limited battery capacity without any application changes. This is achieved through a local Web data proxy and synchronization architecture called BITSY. We illustrate this approach with two robust mobile Web applications, one existing and another new, running on AtomDB, an implementation of BITSY built as a browser plug-in.

Differences between S3 and Atompub

2008-06-03T10:51:00.000-07:00

Since both S3 and Atompub are considered REST style interfaces, people often wonder whether it matters which one of the two they are using.

The bottom line is - S3 is best suited for server-side programming and Atompub is best for client-side interaction. This does not mean that the converse is disallowed, but, at least in the case of S3, a number of issues will render it unsuitable.

I will highlight a few differences between S3 and Atompub:

Various Amazon proprietary headers, data formats, URL templates, and operations are required in S3's REST API. Atom does not require any new headers (one new optional header is defined in Atompub), and all the formats and operations are public standards in heavy use on the Internet.
S3 requires clients to mint URIs on their own following a template.

This means that the server cannot change without breaking clients.

This means that one user can easily overwrite another's data since they could each mint the same URLs when creating new data.

The S3 API does not provide a means of retrieving or changing metadata of objects.
S3 does not handle any concurrency - it allows arbitrary writes to win.

S3 and Atompub are at different levels of abstraction. S3 is suitable for server-side programming, but cannot be exposed to the entire world. Therefore, S3 is good for corporates to implement their applications internally. However, anyone interested in providing an API that an unaffiliated developer would use would prefer to employ Atompub.

Benefits of using Atompub over S3

No need to create new metadata vocabularies - just build on Atom and related efforts (such as Geo RSS, Dublin Core, etc.)
Standard dumb-proof techniques for retrieving and manipulating individual items and their metadata, operating on lists of items including paging and
Atompub uses HTTP conditional updates to control concurrent updates

UPDATE: A well-done slide deck by takemaru on practical Atompub servers explains how S3 is well suited for situations where the client of the API wishes to control the URI space, where metadata is simple and with the data.

Clamor for self-scaling databases for Web development

2008-04-09T10:56:00.000-07:00

Not that this is an easy problem, but awareness is half the problem solved, or something to the same effect.

This is what I would like from a Web database:

Require no work on the part of the application development or administrative orgs for

high availability
disaster recovery
scaling
geographical placement
Of course, availability guarantees are negotiable

Self-tuned query execution

Allow for run-time evolution of data structure/schema
based on analysis of the structure and queries being processed
Support querying of XML data and full text search, not necessarily XQuery - see GData API, Lucene query syntax
Subset of SQL only supported - GQL, Amazon's SimpleDB

Make database directly Web aware

Protect against DOS attacks
Protect against injection threats
Provide feeds and native support for Atom/WebDAV
Relations recorded as Web-style hyperlinks
Native support for synchronization to devices

Record level authorization

Using encrypted authorization keys
Works directly with HTTP authentication (or its extension)
Support for flexible authentication, OpenID, OAuth

Flexible transaction policies

Stale reads
Optimistic concurrency

Am I missing something important in this list?

As to those who have expressed a desire for such databases:

MegaData - Joe Gregorio
Database Requirements in the Age of Scalable Services - Adam Bosworth
Scalable Databases for Web applications - Salvatorre Sanfilippo

Entrepreneurs and society

2008-01-22T11:05:00.000-08:00

"Once an entrepreneur, an entrepreneur for life." No? Perhaps I am wrong in believing this but I have never been able to wean myself off the E-tap since I first co-founded a company in late 1999.

I constantly stumble in to interesting non-profit problems that it seems like an endless loop of social entrepreneurism. Not the enlightened and exalted form like that practiced by Muhammad Yusuf or Baba Amte. Anyway, I came across this interesting quote from G. B. Shaw on unreasonable people, which I can relate to quite well -

“The reasonable man adapts himself to the world, the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.”

Vive la unreasaonable!

SAP CRM for iPhone and drag-and-drop

2007-12-10T03:29:00.000-08:00

Apple, through its lack of an SDK for iPhone, has coerced software developers to treat the Web as a first class development platform for mobile device.

Last week SAP announced availability of its CRM software for the iPhone ahead of other platforms. Someone looking for details about breakthroughs that SAP made in its mobile software offering to adjust to the iPhone environment will be quite disappointed in this announcement. But a careful reading betrays the brute force approach most enterprise vendors of Web-based applications take -- bend the platforms to their whims. Their announcement claims

The iPhone software, part of a new sales-force automation suite from SAP, uses a Web-based interface with drag-and-drop tools ...

Apple went to great lengths to convey that drag-and-drop is not good for mobile devices. In fact, its own iPhone applications forgo drag-and-drop features. And yet, SAP says that is exactly what its iPhone applications are distinguished at!

No surprises here.

Theory of astronomical real estate markets

2007-09-20T12:14:00.000-07:00

I keep on wondering how income to house price ratios soared to 4.5 from a 40 year level of 3. The Economist points out how high housing prices might be the result of an extremely stable global economy over the last 3 decades.

Belief that the business cycle has been tamed for good helps explain why property prices in many rich countries have risen so high and why there has been such a willingness to take on debt at large multiples of income. A less volatile economy makes income streams more reliable and, goes the argument, justifies higher prices for all assets, including housing. A reduced fear of job losses means homebuyers in America, Britain and elsewhere have been content to take out huge home loans.

The reasoning seems to make sense. However, the true test of this theory is in its ability to explain effects of the credit market volatility on the future of US and the global real estate markets.

Graph structures and Atom

2007-06-21T14:50:00.000-07:00

If one is willing to interpret a relational database through the eyes of an application (family, really), Atom does indeed help. In more general sense a simple two level hierarchy (container, element (container, element()) can go quite far when using hyperlinks. However, if interpreting a relational schema without any knowledge of applications that use this schema, Atom won't be of much help, and you might be better off with something else like Web3S or RDF.

Note to myself: Atom pub still doesn't provide a comprehensive mechanism for synchronizing changes to these feeds across autonomous agents, but that's for another day.

Consider an application where data is richly interconnected, and at present, lives in relational databases. For example, drivers carry insurance (except those who like paying for tickets), and insurance covers vehicles. This data can be supplied as Atom feeds (both atompub and regular atom). So this three node graph can be represented as three separate collections one for each node, and three compound feeds one for each edge in the graph. A compound feed is one where the entries contain feeds in their content. Alternatively, one could use microformats and provide links, but then you'd require an XHTML processor to understand the relations.

Might I add that the additional bytes being generated for Atom feeds could be offset by the diminished traffic to a server because the data required is already sitting in an intermediary or the client, and may be synchronized efficiently. Whether you would be willing to work with potentially "stale" application data is still another discussion to have.

Changing the world?!

2007-05-30T02:23:00.000-07:00

Why would you listen to me? Why, I am just a sleep-deprived gadfly who could dream of nothing better. Adam Bosworth said this a couple of years ago at the MySQL User Conf 2005. In any case, I referred to this talk in one of my previous posts converging applications and the Web. (Surprisingly, there are plenty of references to Adam's talk but none really to his talk directly - the top results are, instead, everyone else's opinions of his talk.)

Joe Gregorio resonated with this idea when I discussed the future of AtomPub at the recent AtomPub Interop at Google. His post on MegaData also discusses how it is possible to achieve Web-level federation of data.

This space has only just begun to attract attention and the rest of this decade should produce reasonable replacements for the traditional client-server RDBMS + 3GL based applications we have lived with for the last three decades or so. Clearly a lot more is at stake than just blogs and news, social networks and photo sharing.

Are you smart enough?

2007-03-28T11:17:00.001-07:00

Paul Graham says:

If you don't think you're smart enough to start a startup doing something technically difficult, just write enterprise software. Enterprise software companies aren't technology companies, they're sales companies, and sales depends mostly on effort.

Email riot

2007-03-19T13:17:00.000-07:00

Have you ever heard of an email riot? I could only find one genuine use of it online. Nevertheless, let me explain to you what it means. For those with little time -

An email riot occurs when one mob of users tries to rush out of an administered mailing list while another mob wants to keep in on an interesting conversation, while no administrators are actually listening to their requests. The bigger the maling list, the more dangerous the riot becomes.

I just witnessed one and am completely bewildered that people scantly understand mail technology or protocol in this day and age. The riot started when I sent an email to a scarcely used, but a large mailing list. Of course, some people wrote to me "WAY TO GO MAN" and some others pleaded that I stop the impending mobs. But I knew nothing would work better than to not respond to any of the hundreds of messages that were going to follow.

Blackberry and PDA users were trying to rush out the door asking someone to "PLEASE REMOVE ME" from this mailing list. Others were pleading to keep the thread alive for the reach it afforded. Yet others were simply saying "Do what you want but I want to get more messages on this topic" All the while few realized that their messages were being sent to countless people as they chose to "Reply to All".

I guess people are so used to engaging in CYA email discussions that they tend to remove the Reply button from their tool bar in favor of the almighty "Reply to All". Not only that, they forget that no one on the administered mailing list is actually reading their messages. Its almost like a riot-prone area which has no government. No one pays attention to "Help me" or "Save my life".

Then there were people offering advice to add an email filter to keep the rogue email out of their in boxes. And there were others shouting at the former to keep their mouth shut to reduce the email traffic, all the while adding their own through the "Reply to All". The most amusing emails were ones asking everyone to stop responding to the rioters while heaping more pain to the entire mailing list:

THERE IS NO ONE WHO WILL "REMOVE" OR "UNSUBSCRIBE" YOU FROM THE ... EMAILS.
SO PLEASE DO NOT REQUEST THAT!!

THINK BEFORE YOU SEND:
THIS THREAD DIES OFF ONLY IF YOU JUST STOP REPLYING TO IT.

Yes, it's ironical that I am sending a message for this purpose. I'm sorry.

How long did it go on? Well it started on a Friday evening, and is still going on on a Monday morning, albeit the traffic is about one every hour.

Lazy JSR spec writers

2006-11-21T11:06:00.001-08:00

Can't believe the JSR 173 spec writers were so lazy!

When did you last feel like flagellating a JSR editor? Well, Chris Fry (BEA) and his team of JSR 173 expert group members definitely deserve this rich reward for a stupid mistake of copy-and-paste. You'd say we all do that some time or the other. In this case, the spec has produced a lingering result due to the copy-and-paste error.

Copy-and-paste

The StAX API uses the standard service provider initialization techniques and requires that users beseech the "standard" factories for an instance of various XML factory classes. Lazily enough, the spec editors copied a method to instantiate the InputFactory and used it ditto for instantiating the OutputFactory.

Only that they forgot to change the signature of this method, so it also returns an InputFactory.

So the code for:

public static XMLInputFactory newInstance(String factoryId,  
                                     ClassLoader classLoader)
throws FactoryConfigurationError {
return (XMLInputFactory) FactoryFinder.find(
 factoryId,
 "com.bea.xml.stream.MXParserFactory",
 classLoader);
}

was copied as:

public static XMLInputFactory newInstance(String factoryId,
                                     ClassLoader classLoader)
throws FactoryConfigurationError {
return (XMLInputFactory) FactoryFinder.find(
 factoryId,
 "com.bea.xml.stream.XMLOutputFactoryBase",
 classLoader);
}

Don't believe me? Then take a look for yourself.

Why am I complaining now?

I am working with this library in an OSGi run time, where I have to use this two-arg factory creation. But I can't quite do this with the existing StAX API and I don't see any existing fix for this BUG. Sun closed a bug filed for this in August 2005 saying that the spec needs to be changed and they can't do anything. Codehaus recorded a bug in September 2005 that has yet to be worked on. Seems like the Extreme Lab at Indiana is tracking this issue as well to make it a part of some maintenance release.

Are you working with an alternative for pull parsing that works better with large amounts of character data? Let me know...

The King (J2EE) is dead, long live the King

2006-11-10T06:14:00.000-08:00

I always considered middleware technologies to be ephemeral and favored explicit architectural thinking for long term stability. (My dissertation shows how different styles are essentially composable from small and simple primitives independent of middleware technologies.) Industrial technologies aka runaway trains, viz., CORBA, DCOM, and J2EE were each a rehash of DCE RPC. Neither introduced anything fundamentally superior to the other and was eventually bogged down by its own mess - a melange of services and factories, single points of failures, nasty recovery techniques, specialized and opaque serializations, and a lofty but misguided goal for portability.

While it may reflect my naivety, I am of the opinion that if a technology cannot be explained within a few hours, and if its evolution cannot be guaranteed over a decade, then its premature death is certain.

The only technology I know of that has survived for more than a decade, receives tremendous use, and shows no signs of aging or obsolescence is HTTP. Even SMTP could be considered to be in this league, except that it is plagued by problems of authentication and credibility even as our mailboxes are jammed with spam.

Now Richard Monson-Haefel writes of the impending demise of JEE and posts a preemptive obit of the pre-eminent enterprise stack - JEE 5. I saw this coming for the last year and a half and my move to XML feeds and REST was no coincidence. Some of you know that I have switched out of the Oracle middleware group to the Oracle database group a few months ago.

Two years ago when I finished at USC, I was raring to build simple software architectures for solving complex problems that would survive decades and still be held in high regard. I now finally have a shot at the goal with the first internal venture project funded at Oracle that campaigned for and am now heading. This project focuses on high throughput data synchronization at the Internet scale. This technology would form the underpinning of a resource-oriented database, which I considered the best alternative for petabyte sized databases, which are in line to become the norm in the SaaS world.

Editorial selection

2006-09-13T11:49:00.000-07:00

A very nicely written article on the design issues with the democratic editorial process being used widely on Web 2.0 discusses the inherent systemic flaws leading to overall poor quality of information.

Value of information
Novel and true information has more value than stale and misleading information. Big media editing process is inherently biased and produces a small community of information evaluators. On the other hand democratic, i.e., social editing often creates the least common denominator of information, not the same as novel and true information.

Approximation to optimal editing
The academic editorial process works quite effectively by producing blind votes, subjective review, and reviewer selection. The process usually advances the community and its agenda.

It is not quite optimal because it is slow, biased towards the community goals, requires a qualification for entry, and can be rigged in favor of certain groups.

Conclusion
Democracy unfortunately does not produce good information. Neither does capitalism. You ought to have a meritocracy to distribute valuable information and foster excellence. Would be great if someone could address some of these drawbacks of academic the editorial process and introduce it to mass communication.

XML feeds - format selection

2006-08-11T15:02:00.000-07:00

I am spending a fair amount of time these days thinking about middleware that will power the applications for the '20s and beyond. From the wave of metaphors used in the "Web 2.0" parlance, one dominant theme is the pervasive use of XML feeds to communicate information to consumers.

As Ray Ozzie claims, XML feeds are the equivalent of Unix pipes for the Internet. If Unix pipes provided much fuel for composite applications on the workstation, XML feeds are expected to provide the glue for Internet scale composite applications. In another article, Adam Bosworth claims that XML feeds will save the world.

There are mainly two flavors of XML feeds as I speak - RSS and Atom. I am sure you have read several articles comparing the two. Many such articles end up being technical comparisons for the sake of it, especially to promote Atom.

If you are thinking "Why does it matter how many formats there are or that I am using one and not the other?", read on.

Of course, anything can be computed (within Turing's incompleteness limits). Therefore, anything that is in RSS can be written as Atom and vice versa, correct? Turns out the answer is actually in the negative. To quote DeWitt Clinton the lead engineer of Amazon's A9 search:

the problem is that the RSS syndication format is that it is lossy

Hopefully, this statement will get your attention and suggest that you read the entire article. The lossiness is invisible when dealing with simple use cases of syndication. However, when you start aggregating feeds and introducing semi-structured information in the feed payload, the entire problem starts becoming more and more apparent.

converging applications and the Web

2006-08-09T13:04:00.000-07:00

Adam Bosworth paints a vision for the future of applications (and the Web) that far exceeds what I have been pushing for as a vision for business applications. Although this is an old post, with more than 60K of markup text, its relevance and applicability is quite durable. One of the best takeaways from this article is his definition of a Web services browser:

... is a browser that can access information published as XML messages by services, let the user interact in a rich and graceful way with this information or these services, but can run well in terms of interaction whether the user is online or offline.

In separate but loosely congurent ways, we are preparing for the convergence of large applications with the Web mainly as a response to

customers demanding greater flexibility in creating and organizing live data owned by a wide variety of sources, and
users increasingly craving for mobility from their desktops without being tearfully separated from their applications.

I am convinced that it is not a question of if but rather of when, applications will take on a greater similarity with the Web and the Web will resemble more of a read/write medium like traditional applications than a 1% contribute/99% consume medium that was the Web in its first decade and a half.

Of course, this requires that the Web (as in content and platform) evolve to increasingly favor a subscription model (think of Atom), a greater separation of content from information (think of JSON/XML/microformat), and a richer interaction model for end users (think of AJAX). A lot of this is tagged "Web 2.0" these days.