Extend batch protocol

[ilinkuo]

From the discussion on #152, this proposal aims to:

• extend batching to data and html requests
• support/prefer static manifests

The benefits, in addition to the benefits of a static manifest, are that server traffic subsequent to the initial manifest load is also greatly reduced.

1. AppConfig.manifestUrl [mandatory] becomes a static url for retrieving the static manifest.
2. AppConfig.batchUrl [optional] is a new url for retrieving optionally supported batch functionality.
3. F2 adds a new batch method with signature F2.batch(url, params, type, [delay], [callback], [errback]) and opens up the batch request mechanism to
   • Type can be manifest(default), combo, html, data. A manifest request is a static manifest request. If data and html are provided for a static manifest request, then these should be non-changing values (for example, html for a waiting icon, or sample data for a demo App, with real data protected by authentication on a data request) A combo request is the current mix of manifest plus dynamic html and data.
   • Like the existing registerApps, batches are grouped by having a common batchUrl.
   • The target server side batchUrl knows how to decode a batch request into its component requests. This isn't spelled out in the current spec, but I think F2 needs to explicitly spell this out.
   • The response is an array of static manifests, combination manifests, html, or data. Today, the response is an array of dynamic manifests only.
   • The handling of each element of the response array is by either a user specified callback handler, or a default handler for the type. The default handler for the type combo is just the current _loadApps(). Note how the _loadApps() takes an array of AppConfigs and an array of AppManifests which is just the response? Well, the handling of the combined response just takes an array of handers and the response's array of objects. Each AppConfig is easily made into a handler using something like var handler = new ComboManifestHandler(appConfig);
   • Batch requests are given a slight delay so that multiple requests have a chance of being batched.
   • An alternate signature for batch could use a promise instead of explicit callback and errback parameters.
4. The current batch loading of manifests is rewritten to use the new F2.batch() to do the actual work
5. AppConfig.batchUrl should in most cases, be a remembered value from having read in AppManifest.batchUrl (or via some out-of-band communication, as it is currently). The manifest server indicates whether or not it supports batch requests by providing AppManifest.batchUrl. If provided, AppManifest,batchUrl != AppManifest.manifestUrl because the latter points to a static resource. However, the Container may implement its own batch request aggregation irregardless of whether the App Server does so by providing an AppConfig.batchUrl. In this case, AppConfig.batchUrl != AppManifest.batchUrl.

[brianbaker]

Would the new F2.batch() be in addition to F2.registerApps() (thus far called F2.load() in v2)? If so, could a container developer circumvent the batching wishes of the app developer by not using F2.batch()?

I like the idea of a slight delay, as today container developers can circumvent the batch logic by calling F2.registerApps() multiple times instead of calling it once with an array.

[ilinkuo]

Would the new F2.batch() be in addition to F2.registerApps() (thus far called F2.load() in v2)? If so, could a container developer circumvent the batching wishes of the app developer by not using F2.batch()?

F2.batch() would be a lower level method than F2.registerApps(). Where F2.registerApps() today calls _loadApps(), it would call F2.batch() instead.

The problem of circumvention exists as long as _loadApps() is visible. For F2.batch(), you can make it harder by hiding the default handler for the types "manifest" and "combo" and make it possible only for the framework to call.

Nonetheless, circumvention of intent exists in the other direction as well, as I've deliberately allowed a mechanism where the Container implementer can implement a server-side batch service to batch requests even thought the App owner did not indicate the App was batchable.

I like the idea of a slight delay, as today container developers can circumvent the batch logic by calling F2.registerApps() multiple times instead of calling it once with an array.

This design is so that no explicit coordination is really needed for batching. Manifest requests, data requests, etc. can all be batched together in a single request.

[ilinkuo]

Actually, the discussion about the App developer's intent highlights the existing problem of out-of-band communication. There is no way currently for the Container to know that an app is capable of being batched. The enableBatch is in the AppConfig (owned by the Container), not in the AppManifest. How does the Container currently know that an App can be batched, and at what URL? it is a bit unfair to accuse the Container of circumventing the developer's intent when there's no way within the spec for that intent to be communicated. That's the out-of-band communication problem -- F2 should find a way to communicate this intent within the spec itself.

This proposal begins to address this issue by saying that the manifest must declare the App's batchability, and implies that the Container should update its AppConfig when it sees a manifest's batchUrl, but there should be a more general way, in my opinion.

[brianbaker]

I think today with v1 the general practice has been container developers and app developers working closely together - including app developers sending container developers what they want included in their AppConfig. The question of "how does the container currently know that an App can be batched" is no different than the question of "how does the container currently know the manifestUrl of an app" - its all about the two parties communicating what should be in the AppConfig.

That said, if we can work towards a model where the App can define whether or not it can be batched somewhere outside of the AppConfig - like you propose in the manifest - then that takes a little more responsibility off of the container and is one less talking point between container and app developers.

[markhealey]

I don't want to take us off topic too much...just a quick comment on communication.

it's all about the two parties communicating

Communication is definitely something we want to keep in mind as we discuss v2. We should chase down any opportunities to reduce the amount of required exchanges before getting started with F2. For example, we previously had conversations about Apps broadcasting an interface it expects for any Container-App communication and for F2 to provide an interface validator as it is the broker between Container and App.

[ilinkuo]

The question of "how does the container currently know that an App can be batched" is no different than the question of "how does the container currently know the manifestUrl of an app" - its all about the two parties communicating what should be in the AppConfig.

@brianbaker, on this point I vehemently disagree, and let me try to convince you that the two are really very different. I think that knowing the manifestUrl is like knowing the url of a website -- that's all you should need to know. Once you know the website's url, you just go there. A lot is happening under the scenes -- DNS resolution, content negotiation, caching, etc., but all that is taken care of by protocols in an automated fashion.

I think F2 should work in the same way. Once I know the manifest Url, I shouldn't have to negotiate additional issues on the AppConfig with the human App developer. Imagine browsing the web where I have to make explicit agreements with the owner of each website I go to. Or imagine if npm worked like that. Or if I had contact each email host owner of the person I'm emailing.

So, from my point of view, about the two human parties of App developer and Container developer, F2's goal ought to be about them not communicating. On the other hand, about the two machine parties of App and Container, F2's goal should be to enable the two to communicate and negotiate, and that there should be zero talking points between the App and Container developers.

For v2, I would like F2 to nail down the communicate part, with an eye towards adding the negotiation in the future.

[brianbaker]

Not really sure how there is anything to disagree with - I'm just speaking from past experience. With the current setup, when I wanted the container developer to make sure to batch the apps I asked them to do so (or I actually just sent them the AppConfig I wanted them to use). Also, I'm not at all opposed to reducing or eliminating the amount of communication between developers - I believe that's exactly what I was talking about in my second paragraph above.

Anyways, to get back on the issue at hand I think it would be great to see a gist, jsfiddle, plunker, etc. Things don't have to be functional, but it would be good to look at what a container developer or app developer would have to write in order to accomplish the suggestions. Speaking for myself, I can usually digest a proposed change a little easier by way of code samples versus written descriptions. I'd take a stab at it myself, but I'll admit I'm not following the type parameter or the difference (or not) between AppConfig.batchUrl and AppManifest.batchUrl.

[ilinkuo]

@brianbaker I'm glad we're on the same page. I'll try to come up with an example jsFiddle, but it'll probably be a week from now.