Lambda the Ultimate

As a method to quickly write test code for GUIs, the system seems apropos. Altough the idea isn't new, this may be the best impelementation for all I know.

However, I'd hate to see this expand to script generation for the purpose of performing user-orientated tasks. The user-interface, although shinier than ever, should really not be the target of non-user manipulation. Programs have not made any strides (save for Accessibility APIs) of becoming any more semantically orientated or self-descriptive, and this would take things in the wrong direction. Until that happens, I beleive that true dynamic compositionality, multiple/dynamically created GUIs, handling of other modalities, creating intellegent synergy between applications, etc., will not be possble, or at least will require unnecessary hand-jamming.

The last thing we need is another hypercard like phenomena but this time just by clicking around on existing apps - I can see the TODO-manager that's built by clicking around Excel for 10 mins now :)
Luckily the concept runs out of steam (as most visual aids do) when it comes to performing anything above modest complexity - after which it becomes worth learning how to program to the API.

But it is "nifty"...

My biggest gripe with newfangled ideas from MIT is that places like MIT reward how clever you are, and not how practical you are. My belief is that this is a direct result of how brilliant everyone there truly is -- million times smarter than me.

I don't know. To me, it seems like
a) throwback to screenscraping, with an extra level of indirection; this is the antithesis of the Web movement and thus puts us back in the '70s
b) lacks any strategy for "lifting" out the underlying service hydrating the UI

I've wrote a simple, pragmatic prototype for WPF and Silverlight that does b) and doesn't depend on a), but it's dependent on a SearchMonkey-like data interchange format. I believe this is not at all clever, but simply the right approach.

No disrespect to the researchers... this is highly clever... just not that practical {{in my humble opinion, only}}.

Edit:

One other comment

A computer user hoping to learn how to use an obscure feature of a computer program could use a screen shot of a GUI — say, the button that depicts a lasso in Adobe Photoshop — to search for related content on the web. In an experiment that allowed people to use the system over the web, the researchers found that the visual approach cut in half the time it took for users to find useful content.

This is only half the battle, though. Nirvana is being able to automatically produce screenshots of your application, in a particular state, for a particular user. If you take the stance that a GUI is fundamentally a dynamically federated, dynamically distributed system... then it's a mistake to assume a given user will see the same screenshot, and may even present an information secrecy problem.

Again, I am not meaning to disrespect Rob Miller, who always has great and fascinating ideas... this one just overlooks established thought on what to do and how to use hypermedia.

The Sikuli-Script idea for Drag Drop is actually a pretty cool hack :)


y=Find( Picture_1 )[0]
for x in find ( Picture_2 )
DragDrop (x, y)



def dragLeft(t)
dragDrop(t, [t.x - 200, t.y])
click ( Picture_3 )
click ( Picture_4 )
for t in find( Picture_5 )
dragLeft(t)


By the way, ignoring questions of scaling this approach (posed by Michael Robin and myself), Sikuli states in one of their videos that it lets you do anything you can do visually... I am curious about a practical challenge in most screenscrapers, such as Boston WorkStation scripting software used to screenscrape Meditech screens. With traditional screenscrapers, you cannot monitor what the "final" value is for something. For example, if a value is only known after a series of constraints have been solved, and those constrains continuously change, how do you monitor that value and then "finalize" it when the user clicks, say, "Done" or "Next"? For example, looking up patient information and then, context-sensitive, loading a web browser with a REST-based GET request for some additional information not provided by the closed-source, proprietary, non-JIT/MP composable Meditech screens.

Thus, I think the ability to do "anything" is overpromised. In order for Sikuli to actually do "anything" it would need to be able to interactively debug the application it is monitoring. This would allow it to trap into debug mode on the "Done" click, and then capture the "final" value. Sikuli thus needs "capture" and "seize" primitive commands to go beyond normal screenscraping.

Edit: Also, how do you support zoomable uis?

This is pretty cool... an interesting evolution of scripting. Besides the huge security holes of course... (I'm assuming if you can display a graphic on a users screen, you can change how Sikuli scripts would execute.)