Comment by elevation | Hacker News Reader

Comment by elevation | Hacker News Reader

Parent Full thread

I test embedded devices with RF. You write most of your test drivers in python, RF simply orchestrates the tests and collects the results into a report, which works well in simple QA scenarios.

Robot lacks features you'd need to support larger-scale embedded test, such as with a device farm, where you need the concept of tests leasing resources. If you build a test stand with 10 testable devices and need to run a suite of 100 tests, it's ideal to run the tests in parallel as devices become available. Some test setups might require more than one instance of a testable device, or instances of more than one type of device (for example, to ensure that a V2 product can still interoperate with a V3 product.) Robot doesn't really support this.

While a feature to support this could be made extremely general (resource classes, instances, and leases) the RF developers have been uninterested in incorporating this aspect of test into their framework. The result is everyone who does even mid scale embedded device testing has to writes their own.

Another complaint about robot framework is that when you have an expensive setup like a 4 minute flashing operation, you don't want to repeat it more than necessary. So in a file, you might make the expensive setup a suite-level setup, followed by the tests cases that depend upon it. When this file grows, you might want to refactor it into a multi-file test suite in it's own sub-directory. However, these tests no longer share a suite scope (because robot's "suite scope" is actually a file scope" for legacy reasons) so in practice you may need to tolerate 3000 line files to avoid long setups.

Have you found a solution to mid-large scale embedded test setup? Could you provide some shallow insight into frameworks or other infrastructure used for embedded testing? I've previously been responsible for firmware testing small production volume devices in aerospace but have since moved to a high volume product with multiple active hardware revisions and no test infrastructure currently in place. It's a different beast to test now while trying to balance schedule and feature dev.

If you're trying to validate new software on known good hardware, then your goal at any scale is to insert a suite of embedded software acceptance tests into your development pipeline usually somewhere between build and release. CI infra is ideal for this, my counterparts all use Jenkins.

Robotframework is a great tool for expressing tests. For simple setups, Jenkins has an RF plugin to kick off tests.

The tricky part is managing many test resources. One jenkins server triggering robot tests on one bench PC with one embedded target is a great way to develop an initial test suite. But once your org gives assigns you more hardware, the tricky part is figuring out how to use it maximally (either optimizing for fastest time to completion or for least time idle of your dedicated test hardware.) This means the tests themselves can no longer "drive" the test resources directly, they need to lease them from a resource manager service.

Larger companies with nearly homogeneous hardware may get by with a basic priority queue. But some test setups with an L1 or L2 switch between them may support a DSL to allow tests to request resources that are physically or logically connected, and may need abstraction layers for common tasks such as "physically disconnect power" which the Resource Manager could implement as a SNMP command to a PDU or as a SCPI command to a DC supply -- the test itself should be declaring what has to happen, not how.

I've scoured the internet and plied counterparts in other industries for any example of commercially available prior art. Everyone I've talked to wrote their own. It took a week to produce something acceptable for my use case. But if the RF devs were to maintain a similar solution, I'd switch to that in a heartbeat.

> the RF developers have been uninterested in incorporating this aspect of test

Have they been asked? I had the same problem and had different solutions built in house with different levels of success

Pekka has previously expressed that test resource management should not be included in the core and is better implemented as separate keywords. One example would be the limited resource management that Pabot supports (basically a single global semaphore that tests can increment/decrement through a pair of keywords.)

The problem with 3-rd party keyword-based solutions is that the resource acquisition becomes a part of the test process and could a test report to look like "ran for 6 minutes, then failed" when what really happened is the actual test couldn't be run at all because resources weren't found. It would be nice to be able to queue tests based on the resources they'll need.

Also, another feature that's more difficult with custom implementations is static analysis of the test base. If resource requests were a declarative part of the RF language (much like tags) you could do things like "run all tests that exercise resource X" or "run all tests except those which require more than one Y". This is technically possible with a keyword library but requires an additional global lexing pass of the entire test base, and this is the kind of logic you don't want to decouple from RF and it's test discovery behavior.

Perhaps Pekka can be won over in the future.

Reply on news.ycombinator.com