Automated app testing!?! Burn baby burn proposal ...

[cpg]

Hi, just checking to see what you guys think about this proposal ...

One of the reasons it's hard to release applications is that we don't know how well they are tested. This is going to be more complex as we work on new releases from Ubuntu and Fedora.

We rely on the testing team, however, we don't really have a sense of how much or how well apps are tested.

One possible solution is to automate things some, and we had been talking about this for a while. Today, however, we got around to doing some of this investigation and we have a proposal for a testing framework. Part of it is actually working today (with something called selenium). The idea is to call a script we call

Code: Select all

burn-baby-burn

which is partially written, however, there will be need to add at least one field to the app: the testing script

This field we're proposing is a "data structure" to drive application tests. The format is called YAML (a common ruby language - Yet Another Markup Language - YAML). For instance, we got Piwik running today in automation with this testing script:

Code: Select all

:test: - :page: - "Piwik \xE2\x80\xBA Login" - Open Source Web Analytics - :form: - - form_login - admin - - form_password - hda - loginform - :page: - "Piwik \xE2\x80\xBA Web Analytics Reports" - Sign out

What this says is that the app is tested by opening a browser on the app, checking one page, filling one form and checking another page.

First the app checks for this string as the <title> of the app, after the app's web page is brought up: "Piwik \xE2\x80\xBA Login". (excuse the weird characters, for some reason the &mraquo; is encoded like that when you copy and paste it).

The next thing the testing framework does is check for this string in the page: Open Source Web Analytics

Then it fills the form_login field with "admin" and the password field "hda", the the loginform form is submitted.

Then the next page is checked to have a title and a "Sign out" text.

This is not theory, this is working, which is kinda cool. Not sure how much it will vary once we start looking at more apps ... but ... bypassing that ... some questions:

• Is inputting this test script OK for you guys?
• How do we test these scripts by you guys?
• Can we somehow find some way (in the cloud? in some volunteers networks?) to host VMs to test these things easily?

[pjcrux]

* Is inputting this test script OK for you guys?

I'm ok with this...

* How do we test these scripts by you guys?

not quite sure I understand this question. Do we get the testers to use this script or how do we get the apps to be tested by this new script maybe?

I say the system does this. Or better yet we have the testmasters use this as one of the tools we use to determine if an app is ready for beta/live.

* Can we somehow find some way (in the cloud? in some volunteers networks?) to host VMs to test these things easily?

I know bk and Rampage537 use vm's as well as cpg. I don't know how to set one up. but this warrants further review. Let me think on this.