Getting Help: When 'RTFM' is Not Enough

One of the great joys of working with complex equipment is the total unpredictability of the ways that it may choose not to work as expected.
Author:
Publish date:

One of the great joys of working with complex equipment is the total unpredictability of the ways that it may choose not to work as expected.

No factory-training program can prepare you for the spider that chose last night to build a web in the igniter circuitry of the HMI fixture that you fired up (quite literally as it turns out) this morning. No manual can hope to offer you a fix for the combined effects of a serious supply voltage sag combined with accidentally bumping all the buttons on the front of your intelligent dimmer rack, as you bent down to recover a dropped multitool.

Sooner or later, the problems facing you will prove to be insoluble given the time, tools, spares, test gear and expertise available to you at that moment.

A few decades ago I used to pride myself that I could fix pretty much every piece of gear in a lighting system with a pair of linesman's pliers, a small selection of screwdrivers, a soldering iron and a half-decent multimeter. Occasionally a more complex dimmer problem necessitated borrowing an oscilloscope for an hour or so from the white-coated guys in engineering maintenance. However, that was back in the dark ages of manual preset control desks, incandescent manually operated luminaires and analog phase-control SCR dimmers--when a lighting control network was just a very big bundle of low-voltage multicore cables.

The chances are that any one robotic luminaire in your rig today has more complex electronics than an entire control room from that era. Conversely, since the move to digital storage systems for audio and video, the lighting department now quite possibly maintains more mechanically complex equipment than just about any other. Even if you have enough inventory of any one fixture, dimmer or console family to justify sending someone off on a factory maintenance course, that level of training can't possibly prepare a technician for all of the bizarre range of behaviors possible when complex systems interact.

I'm not suggesting that factory maintenance courses are a waste of time. Far from it. I'm simply pointing out that no training course, maintenance manual, interactive video tutorial or DVD can possibly prepare any of us for some of the situations that we will eventually encounter. Even if we have outsourced our maintenance to technical specialists, sooner or later, they too will stumble up against the impenetrable interactions of a unique system configuration. And when it happens, you can bet that problem will be in your system, and that it will occur at the least convenient time. (Don't ever forget that Murphy's law is universal and, by definition, utterly predictable and unpredictable in its impact.)

There's a maxim in the subterranean world of techno-geeks that goes "When all else fails, Read The Flipping (or other F-word of choice) Manual," or "RTFM." However, most serious lighting geeks have already downloaded and read the manual from cover to cover before the equipment was even ordered. In that case, when all else fails, it really is time to seek help from a guru.

In the days before DMX512 set us free from having to source all of our equipment from a single supplier, we could have picked up a phone or sent a telex and (eventually) contacted the team that manufactured our system, then passed the problem on to them for analysis and solution.

HOLD, PLEASE

Despite the vast improvements in telecommunications technologies that now let us contact the support team via landline, cell, voice-over-IP, push-to-talk, e-mail, video conference, Web form, instant messaging, text, or fax, we are often further than ever from being able to get the help that we really need. It's quite likely that person who responds to our inquiry has little or no experience with using the equipment in real production situations.

When a system doesn't behave the way we think it ought to, there are an immense number of potential points of failure. Virtually every piece of equipment in a lighting system today is likely to have at least one processor on board, whether it's merely monitoring voltage, temperature and running hours, driving a dozen stepper motors, chatting to its neighbors on the network, or trying to figure out the CMY equivalent of Roscolux 57 Lavender.

Every new batch of devices to leave the factory is likely to contain an improved version of the controlling software or firmware, even if the hardware is identical to the previous batch. As you would hope, new software for many pieces of equipment is made available via Internet file download, just as soon as an upgrade or a fix is written.

However, not every piece of equipment gets every upgrade straight away, even if it's our policy to keep everything at the most current version. After all, most of us are not crazy or brave enough to upgrade the firmware in our moving lights, DMX merge box, dimmers or console, if we're part way through recording a series, and everything seems to be working just fine.

When a problem eventually arises, we are trying to isolate an anomaly in a system that consists of many complex devices, usually from a range of manufacturers and suppliers, and frequently with a variation in versions of control software. Even if, after days of working our way up through the hierarchy of the field service personnel who have "never seen a fault like this before," we are able to speak with the team that designed the equipment, we are likely to be disappointed. The designers are unlikely to know about the need that has arisen to use a different part in manufacturing, and the three attempts that were required to get the firmware right for the changed hardware. They're almost certainly going to tell you that they believe the fault is in some other part of your system, doubtlessly one not designed or manufactured by them.

For some equipment producers, this has been one of the few benefits of the DMX512 standard. If they can no longer lock you in to using their proprietary equipment exclusively, then at least they can try to blame someone else for problems arising from their equipment. Finger-pointing is one of the least acknowledged art forms to have come out of the great leap forward in electronics technology. Since the wide acceptance of DMX512, I have been "confidentially" informed by any number of equipment suppliers, that Brand X dimmers don't deal with full frame rate updates, that Console W doesn't put out properly formed data packets, that Brand Z doesn't produce detailed enough parameter information to produce proper personalities for its fixtures, that Brand Y merge boxes introduce too much latency, etc, etc. Very rarely does anyone volunteer the information that they have produced some buggy units, and that we shouldn't be using a particular version of their software or hardware.

P2P HELP

In circumstances such as these, where the complexity of interactions between devices leaves us without a definitive source of information to help us resolve our problems, the Internet comes into its own. Now while you probably checked the equipment manufacturers' and suppliers' support pages, and may even have visited their online forums, there is a much more powerful resource available--the whole of the lighting world.

Although a lot of money has been spent to give you the contrary impression, the overwhelming majority of lighting knowledge and experience is not held in factories or service departments in Shenzhen, Austin, Gothenburg, New York, Kirkaldy, Bergamo, Sydney, Los Angeles, Hammersmith or Ärhus. It's held by the people who make the gear work every day, despite the unusable placement of knobs and controls, the errors in manuals, the bugs in software, the undocumented changes to parts and the hype of marketing departments. In some cases, these real experts have given up even talking to the folks in factory support, who are increasingly being muzzled by paranoid corporate lawyers or anxious marketing executives.

Among the best of these resources are both Web forum sites that work as bulletin boards and mailing lists that work via e-mail servers. Having said that, most forums can send out e-mail updates and most mailing lists are now also available through a Web site interface.

One of the most active and lighting-focused resources is LightNetwork ( www.lightnetwork.com/ ), which has been around for more than eight years and offers a huge range of topic forums. The Cinematography mailing list ( www.cinematography.net ) has a dedicated lighting forum area, although some of the topics tend to be more film related. And even though it is ostensibly about show control, the Show Control mailing list (groups.yahoo.com/group/Show-Control/) deals extensively with lighting control issues, in particular control protocols.

Almost as old as the Internet itself, the Stagecraft mailing list ( www.theprices.net/lists/stagecraft/ ), covers a wide range of issues related to stagecraft and technical production, but has much lighting-related material.

In addition to being live interactive discussion platforms, all of these resources also have valuable archives, covering many years of previous exchanges. It's quite likely that you may find that your topic has been discussed before, and that the answer you need is just waiting for you to find.