Recently I joined a team developing an iPad application. The application had no existing automated tests, and the team was under a tight deadline to deliver on promised functionality. Faced with similar circumstances in the past I have advocated introducing automated UI-driven acceptance tests to establish a loose but broad test coverage net. After a brief investigation we decided that the available tools weren't quite what we needed, so we built our own 'Frankenstein's Monster' by plumbing together several existing open source libraries to create a light-weight UI automation framework for iPhone and iPad applications, called Frank.
How does Frank work?
In use Frank consists of two main parts. There is a small objective-C server which is compiled into the iOS application you want to automate. You then use a remote driver from within your automated testing framework of choice to send automation commands to the Frank server and interpret the responses sent back.
Currently the project includes a Cucumber driver. Creating additional drivers (for RSpec, NUnit or JBehave perhaps) is a pretty trivial task, as the Frank server provides just a few key commands.
I don't get it. Show me moving pictures!
Here is a brief screencast which demonstrates a Cucumber test suite exercising the functionality of a simple iPhone app via Frank. [Click Here to watch the screencast in a larger window]
You can see the Cucumber scenarios and then watch as the Frank driver automates the application in the iPhone simulator, restarting the application between each scenario.
Anything else?
Frank has additional functionality which I haven't gone into here. There is a javascript web app called Symbiote which is embedded within your iOS app itself. This allows you to interact with your iOS application as it's running, testing UI selection paths and inspecting the application's DOM in realtime. We also provide Cucumber steps for utility tasks such as recording a test run to video (great for debugging a failing acceptance test within a CI system), rotating the simulated device, and restarting the application between scenarios to ensure independent tests.
Where can I get it?
Frank (plus some more documentation) can be found on github right here.What's coming?
The next important step is including our automated acceptance tests within a Continuous Integration build. We intend to run Frank tests upon every checkin, much as you would run Selenium tests for a web application. A failing test will break the build, emailing the team with a video showing the failing test.
I'm also planning to blog about specific aspects of Frank in greater depth in coming weeks.
Please do check Frank out, and get in touch if you have any feedback, feature requests, bug reports, or anything else!
16 comments:
Hi Pete, interesting way of tackling the problem.
I saw an interesting presentation at this year's WWDC conference (on June 9th 2010) entitled, "UI Automation" by Mike Jurewitz of Developer Tools. This might be of use for you, for testing on iOS 4.0 and onward.
Please see:
UI Automation Reference Collection
-J
Hi pete,
Thanks for the article.
Does frank use uiautomation?
can ou show somecode?
Regards,
Hi Hari, Thanks for the interest.
No, Frank doesn't use UIAutomation. It uses an open source library called UISpec which provides similar capability to UIAutomation.
If you'd like to see how Frank is implemented, check out the github repo: http://github.com/moredip/Frank
You can also find example test scripts there too: http://github.com/moredip/Frank/blob/master/example/EmployeeAdmin/features/main.feature
Thank you for taking the time to blog about this!
I am new to automation on iOS and have been using Instruments to do so. Can you elaborate a bit more on the limitations you were hitting in Instruments that made you use Frank?
One reason we didn't use UIAutomation for Frank was because it wasn't available at the time :)
Aside from that, UIAutomation seems more targeted towards the record-and-playback style of functional testing. In my experience those type of functional tests are quite brittle. I prefer programmatic functional tests using tools like Cucumber, JBehave.
Frank is really innovative. Thanks for creating it!
btw, when I try to "Dump the current DOM" of the iphone app, the result is the following error:
HTTP Server: Error 405 - Method Not Allowed: OPTIONS
Any ideas why and what the solution would be?
Are there any plans to setup a Google group for Frank ?
Hi Rodney,
I set up a google group for questions about Frank: http://groups.google.com/group/frank-discuss
Please do sign up and ask away with any questions you have.
I installed and used Frank for some time, looked its code a little and made a little automation test case with it as well and I think it is a great tool among other possible choices. The thing that I wonder is that Frank will be a tool that can test application on the device itself rather than simulator...
Yes, it is possible to run Frank tests against a physical device. Two things are more challenging in that scenario. We haven't figured out a way to automate re-launching the app on the device, and you can't automate rotation of the device.
hmm i was trying to test my application against my testing IPAD but I just couldn't understand even howto set the environment variable for bundle path to show the application working on IPAD :(. Automating rotation does not have huge importance for me, automation on simulator for rotation is enough. For the other matter, I think an operating system wide mechanism like android adb library would be helpful for relaunching the application so even restarting IPAD can be tested(as we can test android mobile handsets for this matter)... But I don't know iOS has any library or protocol supporting these features, it is your speciality :)
Hi Pete,
This is a great idea!!
I am trying to set this up for iPhone and I can build and run the simulator. I just cannot connect to it from the browser via http://localhost:37265.
I also do not get the "Allow incoming connections" dialog.
What am I missing?
Thanks.
Srini
Hi Pete i'm biting into getting Frank to work on an example Calculator program. Most things work as intended right now. But...
The "Given I lanch the app" step implementation doesn't seem to work. I want to restart my apps every time to - as you indicated also - start with a clean sheet.
Right now if i start the simulator and run cucumber. It waits until i run the app, and then pushes the home button (like the step indicates). It then keeps waiting and doesn't restart the app like intended.
I'm working on Mac OS Lion, XCode 4.
Could you help me out on this one? (info@eekay.nl) thanks!
@EeKay:
I have the same problem with the "given i launch the app" today, running Lion with the new Xcode.
I fixed it by editing the launch_steps.rb, changing all instances of "4.2" to "4.3".
The new xcode apparently does now have build in iOS 4.2 support.
Is there a documentation how to run frank test on physical iPad device?
Thanks in advance!!!
AB
how to use it on physical device
Hi pete,
Thanks for the article.
but does this automation is supportable in xcode4.2
because when i am trying to do it's showing "Ping Failed".
can you tell me what's the problem?
Post a Comment