Your browser was unable to load all of the resources. They may have been blocked by your firewall, proxy or browser configuration.
Press Ctrl+F5 or Ctrl+Shift+R to have your browser try again.

Automating VirtualC64 - An interim report #11

#1

Over the last years, VirtualC64 has become a quite large and complex piece of software and the introduction of an automated regression test system is long overdue.

During the last two weeks, I invested some time to evaluate two well known macOS techniques for program automation, namely Apple's Open Scripting Architecture (OSA) and the Distributed Notification Center (DNC). This is ongoing work and the current implementation experimental. My major goal right now is to compare technologies and to find out which is best suited for building an automated testing environment. Maybe none of them is the right choice, we’ll see.

Here is a brief summary of my experience with OSA and NC:

  1. Apple's Open Scripting Architecture

Apple’s OSA is the technology where Applescript is build on. Using OSA might be the natural choice, because it was build for scripting apps and that is exactly what we want to do: scripting an app. Furthermore, OSA has the charm that it offers two frontends. First, there is AppleScript which of course has come into ages. Second, there is Java Script for Automation (JXA) wich was introduced with macOS Yosemite.

I managed to enable OSA in VirtualC64 and register a couple of commands that can be used to remote control the emulator. E.g., here is a script for running the Errata demo by magic hands:

// 
// JXA script for running the Errata demo
// 

// Connect to the currently running VirtualC64 instance.
c64 = Application('VirtualC64');

// Put the emulator into warp mode.
c64.configure({alwayswarp: 'on'});

// We don't want to see any user dialog window when loading programs.
c64.configure({automount: 'on'});

// Emulate a C64 II with PAL video output.
c64.configure({model: 'C64_II_PAL'}); 

// Reset and give the C64 some time to execute the Kernal boot code.
c64.reset();
delay(2);

// Drag in the Errata demo file
c64.dragin({path: '/Users/hoff/tmp/errata.prg'});

// After a while, hit space to get to the second demo screen.
delay(2);
c64.typetext(' ');

// After a while, hit space to get to the third demo screen.
delay(2);
c64.typetext(' ');
 
// Wait a couple of seconds and take a screenshot.
delay(10);
c64.takescreenshot({path: '/Users/hoff/tmp/errata.png'});

// That's it, folks. Show is over.
c64.documents[0].close({saving: "no"});
c64.quit();

Although OSA seems to be the natural choice for app automation, it has many drawbacks. The biggest is the aged API, which is no longer state of the art. My experience can be summed up like this: For OSA to work, everything, and I mean everything, has to be done right. If only a single error remains in the code or the XML interface description, nothing works. This wouldn't be a problem if the API generated reasonable error messages. However, this is not the case. I had to struggle with many meaningless error messages and most of the time I just had no idea how to proceed. In the end, I was forced to come up with a solution that doesn't use OSA in the way it was supposed to. A little background: If OSA is used the way it is intended, it exposes the data model of an app to the script writer. In my case, this means allowing the script writer access the C64 proxy object and its sub proxies. Unfortunately, I didn't get that far. Somewhere small details in my XML description (or somewhere else) must be wrong, so OSA persistently refuses to present my classes. At some point I gave up and registered a number of commands as a workaround instead of exposing the data model. These commands are used in the above script and are sufficient to control the emulator remotely. Although this approach works, it is unsatisfactory because OSA is not used as it should be.

I think it's time for Apple to come up with something new. JXA can surely remain as frontend language, but the underlying technology needs to be replaced. Apple's API, which goes back to OS 9, really looks like it was developed in the dark ages somewhere in Middle-earth.

  1. Distributed Notification Center (DNC)

After my unsatisfactory experience with OSA, I turned to Apple's Distributed Notification Center. From a software quality point of view, the use of the Notification Center should be penalized, as it is a kind of an invitation to the programmer to write unclean and unmaintainable code. This is at least true when NC is used within an application. Basically, it is a kind of anti software pattern. But since I only want to use Notification Center to communicate with an app from the outside, I was willing to make the pact with the devil.
From a practical point of view, using NC is surprisingly simple. The sender has to register some recipients in the default object of the DistributedNotificationCenter and the sender has to select this object and post something. As soon as the message name matches, the recipient's callback is invoked. With DNC, I was able to re-implement my OSA stuff in much less time. The good thing is that the test automation script can be written in Swift. Since I'm not a JavaScript expert (the script shown above is my very first JavaScript), I obviously find Swift more comfortable, yet.

With DNC the Errata demo can be executed automatically by the following Swift script:

//
// Swift script for running the Errata demo
//

// Put the emulator into warp mode.
configure(["VC64AlwaysWarp" : "on"]);

// We don't want to see any user dialog window when loading programs.
configure(["VC64AutoMount" : "on"]);

// Emulate a C64 II with PAL video output.
configure(["VC64HwModel" : "C64_II_PAL"]);

// Reset and give the C64 some time to execute the Kernal boot code.
reset();
sleep(2);

// Drag in the Errata demo file
dragIn("/Users/hoff/tmp/errata.prg");

// After a while, hit space to get to the second demo screen.
sleep(2);
typeText(" ");

// After a while, hit space to get to the third demo screen.
sleep(2);
typeText(" ");

// Wait a couple of seconds and take a screenshot.
sleep(10);
takeScreenshot("/Users/hoff/tmp/errata.png");

// That's it, folks. Show is over.
quit();

For everything to work, you need another small Swift script that actually sends the DNC messages:

import Foundation

let dc = DistributedNotificationCenter.default as! DistributedNotificationCenter

func reset() {
    dc.postNotificationName(Notification.Name("VC64Reset"),
                            object: nil, userInfo: nil, deliverImmediately: true)
}
    
func configure(_ args: [String : String]) {
    dc.postNotificationName(Notification.Name("VC64Configure"),
                            object: nil, userInfo: args, deliverImmediately: true)
}

func dragIn(_ path: String) {
    dc.postNotificationName(Notification.Name("VC64DragIn"),
                            object: nil, userInfo: ["VC64Path" : path], deliverImmediately: true)
}

func typeText(_ text: String) {
    dc.postNotificationName(Notification.Name("VC64TypeText"),
                            object: nil, userInfo: ["" : text], deliverImmediately: true)
}

func takeScreenshot(_ path: String) {
    dc.postNotificationName(Notification.Name("VC64TakeScreenshot"),
                            object: nil, userInfo: ["VC64Path" : path], deliverImmediately: true)
}

func quit() {
    dc.postNotificationName(Notification.Name("VC64Quit"),
                            object: nil, userInfo: nil, deliverImmediately: true)
}

Although the use of DNC is very simple compared to OSA, there are some limitations. First, it is not possible to send parameters (user info) to the recipient if the sender is a sandboxed app. In other words, a sandboxed sender can send a message, but cannot attach arguments. Although there are good security reasons for this, it is a bummer that eliminates DNC from the list of possible inter-process communication techniques in many real-life scenarios. Fortunately, this is not a problem in my case. I send notifications with parameters only in test scripts, and there is no reason to run test scripts in a sandbox.

Another (smaller) problem concerns Swift itself. So far it's not possible to import a Swift script into another Swift script, but that's exactly what I need to do: The errata.swift file must include the commands.swift file to work. Fortunately, this can be solved very easily. For example, you can combine both scripts to a common script before execution, which can itself be done with a single line of shell code. To run the test, just execute the following on the command line:

cat commands.swift errata.swift | swift -

As I said before, I'm not sure if scripting VirtualC64 is the right choice for building an automatic testing environment. Maybe it's better to write decent UI tests in Xcode. Unfortunately, adding UI tests to my existing project didn't work out of the box, but that's a different story...
But even if scripting VirtualC64 is not the right way to go, I learned a lot about interprocess communication in macOS in general, namely (medieval) OSA and (anti-pattern) DNC. I’m looking forward to macOS 10.15. Maybe, they give us something new at hand that eliminates the drawbacks of the existing technologies.

  • replies 0
  • views 179
  • likes 1