First-Hand:Recollections of the development of the FoxTrax hockey puck tracking system

Personal Recollections of Rick Cavallaro, who was the project manager in the development of the FoxTrax Puck Tracking System. Currently Rick Cavallaro is Chief Scientist for Sportvision.

Origin of the Idea

In late 1994, David Hill, the President of Fox Sports, and Stan Honey, the Executive VP of Technology at New Corp, had started discussing the idea of building a puck tracking system for televised professional hockey. Fox, having just won the broadcast rights to NHL hockey, was eager to find ways to expand the television audience for ice hockey. David concluded that any new technology that could highlight the movement of the puck would attract a larger audience. Stan had argued that for $2 million he could put together a team to develop such a technology. News Corp owned Fox. Rupert Murdoch, who was the CEO of News Corp, also liked the idea and gave the approval to go ahead. In 1995, Stan started to put his development team in place.

It should probably be noted that Stan was pitching an idea to do virtual billboards to David. It was David that wanted to track the puck, but assumed it would be impossible to do. Stan explained that it could be done, but would be too costly. When David asked for an estimate, Stan made an educated guess and called it $2 million. Given that estimate, David wanted Stan to start right away. I believe they both talked to Murdoch before proceeding.

My entry into the project

Stan had been the CEO of Etak, a company that developed the first automobile navigation system. News Corp bought Etak at just about the time I joined the company. After the acquisition, Murdoch appointed Stan to Executive VP of Technology for News Corp. When Stan started to build his team for the puck tracking project, he looked to the engineering skills at Etak and the people he knew. I believe that he went to Marv White and asked, “Who should I talk to about managing this development?” Marv suggested my name; perhaps because I had already annoyed Marv enough that he wanted to pass me off. So Stan approached me and asked if I wanted to do this. I gave it a fair amount of thought, and talked to my wife about whether or not this was a good idea. At that time, I knew Marv very well. I’d been working for Marv and we were good friends. I had moved up in the ranks at Etak from Programmer to Senior Developer to Project Manager to Group Manager. It was a very comfortable situation and I felt that I had a future there. The project represented a new set of uncertainties. I wasn’t much of a sports fan. But I do very much enjoy participating in individual sports. I don’t watch sports on TV, even snowboarding which I love doing. Despite not being a sports fan, I saw that this project would produce a fundamental change in the way sports are broadcast. This idea of tracking a hockey puck had never been done before - not in this way. Not a live effect where one tracks the object of play, the puck, and put a highlight on it in real time.

So naturally I wondered, “What’s the likelihood of success? What will happen if I succeed? What if I fail? Will there still be a job for me at Etak, which I was enjoying at this time?” I had some sense of the risk involved in developing this technology. I understood that we didn’t yet know how we were going to go about doing it. I was also well aware of the tight schedule. The system had to be ready to go on the air for the All-Star game in January of 1996. I knew that if I agreed to take on the project, the next 11 months were going to be at a more intense pace than I was enjoying at Etak. So I had to think seriously about my decision. Where would it leave me in 11 months? In the end, the technical challenge of the project was to great to resist and I accepted the role as project manager and chief engineer. It was February of ’95 when I officially accepted.

Building the team

For this project, Stan Honey was the ringleader. He had hired me as Project Manager, but, in fact, he was the force that brought it together and kept us on track. Stan had been at SRI before he founded Etak, and he knew some very good people there. One of these guys was now at Vista Research, and he thought he’d be great for doing some of the infrared work. We were going to use infrared for tracking the cameras. And at the time we were going to use RF for tracking the puck, which is another story in itself. We asked the guys at Vista Research if they would be interested in taking on a component of tracking the broadcast camera motions using infrared.

This would involve putting slave cameras on top of the lenses of the broadcast cameras. And we would have a number of infrared beacons mounted at known locations in the arena. So depending on which beacons were seen, and where they were seen in the image of these slave cameras, we could compute where the broadcast camera was pointed. So they set out to take on this task for us - to track the broadcast camera motions. That would handle pan and tilt, but would not give us the zoom reading. That was another problem to solve. So we got Vista signed on to the project at that level.

In particular there were two guys at Vista, Phil Evans and Alan Burns. At Etak we had an engineer by the name of Alan Phillips, who was a really top notch EE guy. Alan was old school EE and really an analog guy more than anything. Alan had worked with Stan at SRI, and founded ETAK with Stan. Alan certainly knew digital electronics, but we were looking for analog skills for some of the stuff that we ultimately wanted to do with the puck. Our initial interests in his analog EE abilities, however, had little to do with the puck. Alan had since left Etak and had gone to work for a company called SK Communications. So we signed them up to do a portion of the project which evolved as well.

Stan also knew a fellow named Jud Heinzmann, who was a very skilled RF guy from SRI and was now an independent contractor. We signed Jud up to do the RF design for the electronics we would put in the puck. The plan was to use an FMCW system with an active repeater in the puck. We would have a number of radars around the arena to determine puck location. So Jud signed on and built the prototype that we initially worked with.

Finally, there was Tim Heidemann. Tim was at SGI and had done some very interesting work for the America’s Cup. He had developed a system called Sail Track, where they put GPS on the boats that were competing in the America’s Cup race and they modeled a virtual world, showing the sail boats, how they raced, where they were along the course, and even simulated things like the angle that the boats are heeling on tack. It was a pretty compelling effect. Tim was a software guy primarily, but he was very much a project guy. He was a guy that understood putting GPS on sail boats and, making the system work. So we talked to him, he was the hardest one to get signed up. We actually talked to SGI initially. Based on the Sail Track project, we tried to get SGI to take on the software component of this project. But SGI didn’t sign on.

So we pursued Tim Heidemann directly. Tim decided to form a company with this as his first contract. This would be his first and only contract out of the gate. He started a company named Shoreline Studios (although Stan always used “Tim Heidmann Intergalactic” as a place holder in the contracts until Tim settled on a company name). Shoreline Studios was initially dedicated to us and to this project. And, in fact, Shoreline Studios existed in Etak’s Building 2 across the street from Building 1 where I worked. They rented space from us and started Shoreline Studios. They later moved to Shoreline Boulevard, interestingly. So that was the core team at the outset.

When we first pulled the team together, probably five of us. Stan, Jud and, I, and another fellow, Brian Smart, who was at Etak and maybe one of the Vista guys all went down to the Shark Tank to see a hockey game.

Here we were, a bunch of guys that weren’t sports fans, hadn’t ever been to a hockey game, and didn’t know the first thing about television. I’d never seen how live sports broadcasts were produced. I wasn’t aware that sports are shot from a rolling studio. They had a semi-truck or several semi-trucks in some cases, that they would roll into the parking lot. They would blow the “expando” sides out to create a full studio. The whole truck is filled with racks of computers, monitors and broadcast gear. And they’re just sitting in the parking lot as a studio. It’s kind of astonishing to hear the level of excitement in that studio as the director is yelling for shots, directing the camera men, and 100 other things. And yet the final product is perfectly smooth and seamless. It just seems incongruous with the level of hectic activity in the trucks.So it was my first time, or probably any of ours, in a production truck to see what we were going to integrate with. And the first time I’d ever been in a hockey arena where I was looking at the ice and the cameras and the size and the scope of it and thinking, “Wow, we’re going to try to track a 3” puck in this area that I could fly a blimp in.” And I guess at that time, looking at it in that perspective I was just thinking, “Holy... What have I gotten myself into?” or “What have we gotten ourselves into?” So we were all there sorting all that out - poking around, the way we did on a lot of things, poking around in areas where we probably shouldn’t have - climbing up in the catwalks and saying, “Hey, could we put cameras up here?” And, going up to the camera operators, looking very closely at their cameras, tripods and tripod heads and saying, “Hey, can we instrument these? Could we put something inside the lens?” This first outing was educational and eye opening. At least for me, my sense of fear increased a bit, but we were ready to give it a try.

The project as a critically damped system

When I first heard about the project I wondered “Could it even be done.?” In fact the course of the project was amazing. It turned out to be a classic case, in my mind at least, of a critically damped system at every stage of the game; from the initial presentation - up to the day of the first game. The day before the system debuted at the All-Star game, Fox gave a press conference about how the system worked – and even at that point we all kept our fingers crossed. Every day we wondered “Can we get there? I don’t know. Right now we have no idea. Maybe it’s possible; maybe not. But we’re going to try.” And a month later we had identified some technologies, and, hoped that we could have said, “Oh, no. We can’t find a technology that we can put into the puck, that can track, and that will be robust or...” Or that we could say, “Wow, this is a slam dunk.” Well, of course it didn’t happen that way. We found a number of technologies that might work. Different ones had different advantages and disadvantages, and we selected the one that we thought was the most promising given both the practical and the technical considerations. And we thought “Okay, we’ll follow this path.” We simply didn’t have the time or resources to follow all paths that showed promise.

As I said before, at every stage, the project seemed critically damped. We would identify a technology and think, “Well, this could work.” And then two months later we would be doing a test in the field with a prototype of that technology and still asking “Will it work?” We would say to ourselves, “yeah, probably, but given the time we have left are we going to make it work? Can we get all these problems solved?” I think we always expected there’d come a point where we’d feel, “We’re hosed, it can’t be done” or “Okay, now we’re over the hump.”

But it always remained uncertain. For me, at least, it was a coin toss right up to the very last moment. Every day I kept thinking “Well, pretty soon we’re going to know yes or no.” But it just didn’t work out that way. There were long nights and some sleepless ones. We were living in the catwalks above the ice at various hockey arenas. Being there at 2:00 a.m. when there weren’t players on the ice and there weren’t people working below. It was, typically, when we had to do our work. And certainly in those days we worked whenever no one was there on the ice; everything else had higher priority.

<flvplayer>NHL_GLOWING_PUCK.flv|400|300</flvplayer>

The project was very challenging to say the least. There was a lot of pressure on the team. Fox Sports's over-the-top promotion leading up to he All-Star game, as shown in the video above, further raised expectations and the pressure. (Courtesy of Fox Sports, a Division of the Fox Broadcasting Company)

Radar Tracking technology: a possibility

Our first idea was to use an FMCW (frequency modulated continuous wave) radar system with a repeater in the puck. It’s a chirping system that gives the puck’s range from each of four radars positioned around the arena. This would give us 3-dimensional position fixes on the puck several times per second.To enhance the live video with the tracking data, we would need to know two things: which camera is tallied, i.e. which one is on air at any instant at any frame; and what is that camera looking at, and not just what is it looking at, but exactly how is it looking at it? In other words, what is its pan and tilt? Field of view? Distortion? etc. We had to map that video into our virtual world with a high degree of accuracy. The idea at that time was to track the puck with RF, using this radar repeater system, and to track the cameras using this idea of a slave camera on the lens of the broadcast camera. The lens on a broadcast camera is a $100,000 box about 3 feet long, and the slave camera is, basically a security camera. So our idea was to place these IR beacons positioned around the top of the glass that surrounds the ice, and the slave cameras mounted on the broadcast camera lens would see them. And, of course, even that had certain known shortcomings. What would we do in close up shots where no part of the glass was seen or none of these beacons were seen? Does one dead-reckon using inertial sensors in that case? But we really didn’t get that far. Essentially, that was the architecture that we went in assuming we were going to follow. Jud built us a radar unit and a repeater prototype that would go into the puck. The repeater was relatively expensive, relatively large, and relatively fragile. And it turns out, what probably none of us realized at the very outset, the pucks were more disposable than we had guessed. One might go through 10 or 15 pucks in a game. In an All-Star game it they might go through 30 pucks. And not because the batteries wore out, but because the puck gets a little nick, and the goalie sees that, and he tosses it out. Or one of the guys just sends it into the stands, which is less common now because they’ve got the nets up on the ends now to protect the fans. But, then many pucks just went into the stands. With an active repeater in them these pucks would cost $250 each. Going through 15 of these a game was an expensive proposition.So in our first approach we were testing IR beacons and sensors for the purpose of tracking the broadcast cameras, and RF for the purpose of tracking the puck. Those were essentially two independent tests that performed on the same day during our first working day at the arena. One thing we learned that day is that hockey arenas provide an awful environment for RF.

We were doing our testing at the San Jose arena at that time. It’s got a different name now, although I believe people still call it the Shark Tank. The San Jose Sharks were incredibly obliging to us and our project. Those guys were great and still are. Every now and then we have some reason to go and talk to those guys and ask them to put something on the ice, to do some test. They’re extremely helpful. It surprises me how many organizations, whether it was teams, leagues, venues, were willing to let us come in and muck around with their venue and their gear and even their players. But people were surprisingly obliging to us.

There were times when we practically lived at the San Jose arena. We’d be there for two weeks, spending most of the time living in the catwalks above the ice. We’d have all of our computers, work area, and everything up there. We would come down to the concourse or the suites when we were demonstrating to David Hill, or to some of the other executives, where we were on the project at any given time. Effectively all of our development took place at the Shark Tank in San Jose.

The initial test of the RF system was frankly scary. We saw that this repeater was bigger than we had hoped it would be. So rather than a half dollar buried in the puck, we were starting to think this was going to be a puck-shaped piece of electronics painted black. And that was scary to us. So we were asking Jud, “it’ll get smaller, right?” And he’d say, “Well, yeah, you know, it will, but...” And we tried to imagine how small it could get. “Well, the batteries aren’t going to get smaller”, we thought. We knew that one of the antennas has to be a certain minimum length”. So there was always the question of how small could it get. In addition to size there was also the issue of performance. The way that the ice attenuated the RF was also problematic. I’m not an RF guy – but my understanding is that it’s not just the reflections, but also the absorption and refraction by the ice of the RF that was making the return signal very weak when the puck was on the ice. Life got a little better, when the puck was a few inches above the ice.

There was also the problem that an arena is made of corner reflectors. It’s all I-beams and such, so when you turn the radar on and chirp it, what comes back at you is a hundred thousand chirps from every direction - and we’re looking for the puck in all that noise. So at that time we went back and looked at the data. Trying to identify the puck amongst all that clutter was simply not reliable enough. Jud deserves huge marks on all the work he did. The timeline was such that we thought maybe something could be done with this, maybe not. We’re going to pursue it, but we’re going to pursue other options at the same time. It was probably sometime in March that we did this first test on the ice. At that point it was just each of us going whole hog on our piece of the puzzle. Jud just knocked off, as quickly as he possibly could, a prototype radar. It was not for public consumption - just something that we could test in the arena. The Vista guys did some IR stuff that we could test there.

The IR stuff went pretty well. We said, “We can see those IR beacons in the slave cameras, but, this RF stuff is scaring us”. One could do lot of things to improve the radar approach but we also realized that there were limits. We certainly couldn’t cover all those I-beams with RF absorbent material. We simply couldn’t go to every hockey arena and do that. And we certainly were not going to convince them not to play on ice. These constraints were a given. We hadn’t yet given up hope on it, but we were pretty down at that test. That was a test where I learned a lot from Stan. Of course I learned a lot from Stan throughout this project.

It was at a point where I thought, “Well, I guess we’re shot. This is our technology and it doesn’t look so good.” And Stan said, “Well, you know, you encounter problems and you work through them.” I was surprised to learn how true that is, in how many different areas where what seems like absolutely fundamental problems with what you’re doing crop up, and you then say, “Well, we may even have to redefine the problem. Whatever we do we’re going to do something.” And that’s not to say that everything can be solved, but I was surprised how true it is that you just say we’re going to do something here, and you do.

Shift to Infrared Tracking Approach for Puck

At this point Stan said, “Wow, the IR technology is looking good. Maybe we ought to be tracking the puck with IR.” I said, “Well, what do you have in mind?” And Stan said, “These little IR LEDs,” (and these are exactly the LEDs that you see in your TV remote), “we’ll put those in the puck.” These LEDs emitted very near IR; something like 900 nanometers, just barely outside of the visible spectrum. I asked, “The puck? You know, Stan, they hit that at a hundred miles an hour. They whack it with sticks. These little LEDs aren’t going to last”. I thought he was crazy, and I’m sure I told him that. But Stan is an extraordinarily persuasive guy - and I was working for him. He convinced me that we should give it a try. He said “Let’s prototype a puck, and put some LED’s in it. We’ll see what we can do with it”.

In the Spring of 1995, we demonstrated the idea of mounting infrared emitters in a hockey puck to the executives at Fox Sports. The demo was more show than substance. Watching the puck glow brightly on a TV monitor had an impact. I remember quite vividly the collective silence that followed the demo. And then David Hill, in booming Australian voice, pierced the silence with, “This is F…ing, great!” Despite our efforts to be upfront and explain the limited meaning of the demo, they were sold.

I was lead engineer probably simply by virtue of the fact that I was with Etak/News Corp. The others were contracting for us. Instrumenting the camera became one of my subprojects, and instrumenting the puck became my other primary subproject. We ended up prototyping a puck with LEDs. At our second test we brought this puck along with the RF repeater for the radar system. Jud had actually worked through a number of issues, and improved it significantly by this point. One of the things he did was to add an artificial delay that basically let the arena ring down, and so what was left coming back a millisecond later was mostly the puck return. So it was much easier to find that amid the noise. He did a number of things that were clever and impressive. As a result we believed his technology probably could work; but by that point we were feeling pretty good about the IR approach. Of course this was only the second test, and there were still plenty of hurdles with the IR.

My initial concern with the IR was that the LED’s simply wouldn’t survive the abuse. Initially the repeater was to be embedded in the puck, and I thought it was to be relatively small. So I was contemplating armor-grade electronics that could take a beating. But these LEDs had to be exposed at the surface of the puck. We ended up using 20, and I had serious doubts that they would survive.

The coldness of the ice surface was not an issue in the detecting these LEDs. The potential problem was the hundreds of thousands of watts of lighting in the arenas. Oddly enough, in a modern arena all the hundreds of thousands of watts of mercury vapor lamps are not such a big deal. What turned out to be a big deal was the scoreboard. In many cases, they used incandescent lamps in them. Exit signs were also a problem. There would be this little dim 25-watt bulb in an exit sign on a concourse at the opposite side of the arena that was burning a hole in our IR cameras. And we’d look at the output and say, “What is this thing?” We would then look around the arena. Finally, after some effort it dawned on us, “It’s that fricking exit sign over there that I can’t see with the naked eye.” We ended up doing a whole host of things to contend with the various reflections and false targets. The ice, at the beginning of a period, is like a mirror and reflects everything – the exit signs, the scoreboard, goal lights, anything that was emitting IR energy. But later in the period, it becomes covered with shaved ice and is less reflective.

The “Electronic Puck”

Another great challenge in building this system was developing the puck itself. And it’s funny - we talk about the “electronic hockey puck” as if that is the system, but the puck is actually a teeny part of the system. But to the people at home who hear about the electronic puck, the puck “is” the system. The puck is a little bitty disposable piece of the system. But it’s obviously a critical piece. It was critical that this puck, full of electronics, behave like a puck when in play. Interestingly enough, the NHL didn’t seem to take us very seriously until a couple of weeks before the All-Star game. They didn’t seem to really believe that we were going to be showing up with electronic pucks, putting them on the ice, and asking them to use these pucks for the first time ever at the All-Star game.

I don’t think we really had their attention until about two weeks before the All-Star game. And all of a sudden they realized “ holy shit - this is going to happen”. At that point they told us to drop everything, and fly to Canada to brief them on the puck, the technology, and convince them it wouldn’t affect the game. The NHL is headquartered in New York, but my recollection is that we were meeting them in Canada. Now that we were getting close, they needed us to convince them that pucks were not going to blow apart and have electronics falling out. They needed us to convince them that to the players, it will feel like a puck and act like a traditional puck.

But at the time there was simply no way for us to do that. We were two weeks away from the All-Star game, still wondering if we could finish building a system to put on the ice. So we drafted another guy from Etak, a guy by the name of Terry O’Brien who did a lot of our mechanical work at Etak. Terry had also done the layout on the circuit board that goes into the puck. He had designed the battery can and a few of the components in the puck. So we drafted him, and designated him our puck expert to go and meet with the guys at the NHL. We had done everything we could to make sure that the puck was survivable - that it wasn’t going to break apart, and that it had exactly the same weight and rebound properties that an official NHL puck has. We had removed a lot of rubber from the puck to accommodate the electronics. And then we filled that back up with electronics and a compound that we developed that was basically a flexible epoxy mixed with some filler material to get the right rebound and weight.

We learned something astonishing at one of the very first tests where we had put some LEDs in a puck. We had drilled some holes in a puck, and installed LEDs. If I recall correctly, I don’t think we even installed batteries. This was a survivability test. We wanted to know if we could get LEDs in a puck and have them last for several minutes with these guys slapping them around.

So we threw it on the ice at a practice, and we had these guys slap it around. I knew it wasn’t exactly the right weight, but it was close. An official puck has to be between 5.5 and 6.0 ounces. But in reality the official pucks are 6 ounces - not 5.9 - the players know the difference. So this one I brought in was probably 5.85 or 5.9 ounces. They were slapping it around and playing hockey with it. And just as an afterthought I asked “Can you tell the difference between that puck and the others?” So the first guy, being a smart ass, said, “Oh yeah. This one won’t go in the goal.” But then he told us, “we can’t keep it on the ice. It’s too light. You whack the thing, and it’s in the air.” I was amazed. At the end of a stick, about 1/10 of an ounce difference, and they could absolutely feel it. So we did have to get the weight right, and we did. We got them to be, within our ability to measure them, 6 ounces. That’s what they needed to be. Despite the fact that the rules say 5.5 to 6 ounces, the official accepted puck is 6 ounces.

Early on one of the first things we did was to get every kind of adhesive we could think of to glue the puck halves together. It had to be flexible but it had to have an iron grip. It couldn’t tear apart. We tried a bunch of epoxies. We also tried 3M 5200 Marine Adhesive and a bunch of other things. For a week I had all of these things curing, sitting in cups of water in my office. Then I took them out, and started tearing them apart to see what would last and what wouldn’t. Once we had that, I was convinced that we could make a puck that wasn’t going to break apart during a game. This was absolutely essential. The electronics could fail, the tracking could fail, but we could not have a puck blow apart in the middle of game. Once we had identified the material that we would use to glue it together, I was convinced that we would be able to build a puck.

When Stan said we’ll glue these pucks, I probably thought he was out of his mind there, too. And that was another example where he was not only right, but that was what we ultimately. We also had spent a fair amount of time looking at molding electronics into the puck, but there were problems with that – primarily that the batteries wouldn’t withstand the heat. Of course we continued to work on a number of things with the puck. Initially, I was making them by hand. I was soldering components onto boards and gluing them together in my garage – for the prototype pucks. And then, a week or two before the first game we hired a young lady at Etak with whom I had worked closely for a long time to manufacture the pucks for the games. Unfortunately our process was far from perfect. So not every puck we built was acceptable.

There were a number of problems that we had with them at that time, but probably the one thing that continued to haunt us is that, not being sports fans we didn’t realize they froze the pucks before they used them. Obviously, they’re sitting on ice in the game, but, we hadn’t stopped to think that they freeze them before using them. So the batteries were going to be cold, and the puck’s rebound is very different when frozen.

We had developed a puck that had the same size and weight and rebound of a regulation puck, but our compound, truth be known, when it was frozen probably had more rebound by a slight amount than the standard puck. If we were ever to do this again, I have no doubt that we could match that, too. Of course there’s a lot of resistance to change. There were players that just didn’t like the puck. They claimed that it had more rebound. And perhaps it did. If so, it was small, but they’re professional players and perhaps they could tell. Of course we have a lot more experience with this sort of thing now. At that time we had zero.

The LEDs

In many ways Stan and I think very much alike. And in some ways we think differently. I’m just a more-is-better guy. I wanted more cameras in the cat-walks. I wanted more LEDs in the puck. More is just better. I wanted more current going through the LEDs. You can’t give me too much. One of the things the puck had to do was radiate infrared in all directions, because it couldn’t have blind spots when it was sliding on the ice, rolling on the ice, or tumbling. As a result we placed LEDs on all the puck surfaces. I think my level of comfort was a minimum of four LEDs seeing any given camera. Given the amount of energy emitted from a single LED, I felt four should be seen by each camera to have a reasonable safety margin. That led to our configuration of four on top, four on bottom, and 12 around the perimeter. We had cameras looking straight down, and we had other cameras looking from the sides. We ended up with ten cameras. Of course each LED also had to have a wide enough emission pattern to avoid creating blind spots.

As it happened, this was one of the few things that went our way. Typical LEDs have a dome on them. We actually mill off the dome of the LED so that it’s flush with the surface of the puck. And it turns out that that dome is a lens. The dome gives a much narrower emission pattern. So grinding it off, obviously produces the native, wider emission pattern, which is what we needed. That was one of the few things that wasn’t a trade-off. “Hey, we grind them off and they do what we want.” So we did that. To be honest I’m sure part of the reality is that we ended up with 20 LEDs in the puck because that was as many as we could fit. If I could have figured out a way to put 30 in there and get away with it, we’d probably have 30.This puck had to compete with all of the wattage of all of the lighting in the arena. The business of trying to find that amidst the background and clutter was a challenge at every corner. We ended up with a system that pulsed the LEDs in the puck. They pulsed for 1/8000 of a second about 30 times per second. They were running at 29.85 hertz nominally, although we intentionally varied them slightly about that point . By pulsing the puck for such a brief period we could drive the LED’s at more than 10 times their rated current. We synchronized the camera shutters to expose the CCD for that same 1/8000 of a second, thereby letting very little of the ambient light through. In this way, we squeeze all of the puck’s energy into that narrow time slot while blocking out most of the other background IR .

Challenge of Synchronizaton

Synchronization was another important aspect of the system. We had to choose between a free-running puck to which we would sync the cameras or a free-running camera network to which we could sync the puck. The latter would require putting additional electronics in the puck to detect the sync signal. We chose to have the puck free run. The subsystem used to detect the timing of the puck pulse ended up being a very significant portion of the project itself. Getting that right was possibly the biggest challenge we faced.The sync system didn’t have to know where the puck was, but it had to say, “I saw a pulse, and I know that pulse was the puck.” This would trigger the cameras to open their shutters. The puck had a resonator in it for timing. The resonator is a physical device. So when the players whack the puck at 100 miles an hour, it whacks the resonator. Typically this would change the puck’s pulsing frequency slightly. So the puck might be pulsing at 29.85 hertz and when it was whacked it might change it to 29.82 hertz. This shift presented the challenge of synchronizing to the puck’s frequency and then almost instantaneously syncing to a new frequency when the puck was hit. Additionally the cameras had to be modified by the manufacturer so their shutter frequency could do exactly the same.

The system would sync at the new frequency until it was hit hard enough to change the frequency again. Each time this happened the frequency might change by a few hundredths of a hertz – but that was far too great a change to ignore when the camera shutters were only opening for 1/8000th of a second. We were running a phase lock loop system. But it was not a simple phase lock loop which would have too much inertia to the puck frequency. The guys at Vista really get the credit for solving this problem.

LEDS used in puck and the packaging before being placed in the puck. (Courtesy of RIck Cavallaro, Sportvision)

NHL game puck. Transmitting openings on puck are visible (Courtesy of Rick, Cavallaro, Sportvision)

The Infrared Cameras

We started with an off-the-shelf security camera. We identified the one that was best suited for our needs, had the most sensitive array in the near infrared, and had the right features and a fast shutter speed. Nevertheless, their shuttering wants to be very constant. They want to be either free-running or gen-locked to a constant sync source. The need for the shutter timing to change abruptly became a non-trivial challenge for the manufacturer. We bought the cameras from Pulnix and we got them to do some fairly extensive, low level work on the cameras to allow us to dictate exactly when the shutter would open. So rather than saying, “Run at 29.85 hertz,” with each pulse of the puck, we’d tell the camera, “Open your shutter right now.” The folks at Pulnix went through a good deal of effort to deliver that to us. We now had the sync system with ten infrared cameras distributed through the cat-walks and at the broadcast camera level. Stan had suggested four cameras, so I said ten. Actually, I’m sure I said 20, but we settled on ten ultimately.

We had all agreed that our top priority was getting everything to work. Then, and only then, would we worry about paring down the price and figuring out how to be more efficient. We wanted to cover the ice with multiple views. Seeing the puck from a single camera would not be adequate. A single camera could give the puck’s 3D position if one knows that the puck is on the ice, but in hockey the puck often flies through the air. For this reason, at least two cameras needed to see the puck at all times. This allowed the system to intersect the lines of position to the puck from which it could calculate its 3D position whether the puck was on the ice or flying through the air. On top of that, there was the problem of occlusion. The puck spends a fair amount of time sliding right along the boards, right in the corner. Or worse yet, it’s right in the very corner and there are three guys fighting for it. By having the puck nominally appear in say 4 camera views, we significantly increased the chances of seeing it in at least two, even when it’s being occluded from some of the cameras by players fighting to control it.

With the 10 cameras, it was pretty darn rare to have the puck blocked. There were times that it would flicker for one reason or another, but with the ten infrared cameras and a little bit of interpolation we did reasonably well. We could miss a frame or two or three – but we used 22 frames of delay at that time. So home viewers were seeing the game 2/3rds of a second delayed versus live. Now, at home the viewer probably had a 3-second delay after the signal had gone through the back haul to the studio, where more delay is added, and then it’s gone through repeaters and distribution. So we added 2/3 of a second to that. But that gave us 22 frames to “look ahead” and interpolate if necessary. Mainly, we just tried to have enough LEDs in the puck and enough cameras to see it all the time.

There was nothing terribly analytical about our camera placement beyond walking around in the catwalks, walking around the broadcast camera level, trying different camera positions, and settling on where we could mount them, and how many we could reasonably mount. I felt pretty firm that we had to have some directly above in the catwalks, because the overhead view reduces occlusion by players and dasher boards immensely. We typically ended up with four in the catwalks and six on the broadcast camera level. If I were to do it again, I probably would put all 10 cameras in the catwalks for a number of different reasons. But like anything, it never was perfected and never will be.

Knowing about the broadcast camera

Interestingly it was actually that very first event we went to at the very start of the project where the group of us looked around at the ice, the catwalks, and the game itself to try and figure out what we were going to do. I poked around the broadcast camera a bit, and I thought: we can directly instrument this camera. Until that time, I don’t think any of us had seen a broadcast camera and lens up close. We had no sense as to how we might instrument it to collect pan, tilt, and zoom data.

We retrofitted the broadcast cameras with optical shaft encoders which gave us, direct readings of the camera’s pan and tilt. And then we tapped off of the electrical signals in the lens that gave us the voltage to indicate the zoom, the extender position, and the focal distance. We then created a mathematical model of the lens distortion, the motion of the front nodal point of the lens, and the kinematics of the tripod head. Of course, we had to model the distortion of the sensor cameras as well, the cameras that were looking at the puck. In the end we didn’t use slave cameras to track the motion of the broadcast cameras. We used IR to track the puck, but we abandoned it for tracking the broadcast cameras. Instead, we realized that if we could, track the broadcast cameras directly with instrumentation we wouldn’t have the problem of computing camera pose when none of the IR beacons are in view. So we ended up putting encoders on the tripod head and reading voltages directly from the lenses.

Interacting with the television crew

At that time all of the technology that we introduced was completely foreign to broadcasters. In the early days the broadcast technicians referred to us as “The Scientists” in the derogatory sense. They’d say “Aw, here come: the Scientists”. This was really code for “Oh, Christ, these people are going to get in the way of us doing the job we’re here to do.” I would say it took two or three years before the sports broadcast business recognized that we were part of the show. We were not just the people in the way of them doing their job. We were part of the team producing the broadcast the fans wanted to see. In the end we were smart enough to assign operators to each of the games that live and work, and eat and drink with the team - they become part of the team – but it was not immediate. They were also nervous about us messing with their cameras. The first time we took a $100,000 lens apart and started poking at it with screwdrivers and soldering irons some people got nervous. Fortunately, we never destroyed anything terribly expensive. There may have been a couple of embarrassing incidents, but we did not leave anything damaged, though there may have been times when we sweated into some expensive gear.

Despite their apprehension, the broadcast technicians had to cooperate with us. After all Rupert Murdoch and David Hill wanted this to happen. Equally important was Jerry Gepner who was a member of our team. Jerry was then Vice President of Field Operations for Fox Sports. He acted as our liaison with Fox Sports. Jerry was the guy that would work with us to make our technology ready for prime-time. We knew the lab side, and Jerry knew the broadcast side. Of course Jerry also had a tremendous amount of pull with everybody in the business. In a sense, people didn’t have to trust us at that time; because they trusted him.

Jerry literally started out as a roadie for Kiss, and worked his way up as a grip and had done every job in the trucks. He was now Vice President of Field Operations for Fox Sports. And if he called one of the trucking companies or one of the camera manufacturers and said, “I need a camera for three weeks,” we’d get a camera for three weeks. Without someone like Jerry in that position, we might have developed a system that works, but we wouldn’t have gotten it on air.

The need for a lot of computational power

For each of the infrared cameras that were in the rafters looking at the puck, there was a co-located computer. It was what we called the “Outpost“ computer. Those computers had a set of boards that were custom designed and built by Vista Research, one of our subcontractors, to take the output from the cameras and do the pre-processing to locate and characterize the bright spots. These were candidate targets which might be the puck. These computers employed a number of types of filter to minimize false positives. We would set a minimum brightness threshold, and eliminate regions that might show a return from a goal light, or any region that didn’t correspond to the ice. At that time we would sometimes use physical “filters” as well – we would position a cardboard baffle in front of the camera to block out portions of the scene (such as the scoreboard). This relieved the computer from even doing the computations for areas we knew would have a lot of false targets.

The Outpost computers would identify bright clusters of pixels as puck candidates and send the descriptions and screen locations of these clusters to the computers in the production truck over Cat-5 cables as an RS-422 signal. The description would include such things as the level of the brightest pixel, the total number of pixels, both the horizontal and vertical extent of the cluster, and of course the x-y location of the cluster in the camera frame. All of that data was then processed in the “Puck Truck” by two SGI computers – “Puck” and “Ice”.

The SGI computers took the data from each of the Outpost computers and computed all possible combinations of the lines of position from the sensor cameras that corresponded to puck candidates. Where they found a good intersection of two or more candidates, if that intersection was within the region of play, the computer would identify it as a likely 3D candidate for the puck location. It was possible however to have a bunch of vectors all pointing at something other than the puck; the light on top of the goal box right after they score a goal, the reflection of lights up in the rafters, or a reflection of the scoreboard for example. Spectral reflections from the ice would produce 3D candidate locations that were below the ice, so they could be discarded on that basis. Targets such as the goal lights were typically defined by the operators prior to the game using 3D filters. Such targets might be in a valid location in the region of play, but they were known false targets.

We also used the SGI computers to define 2D filters in each of the sensor camera frames. These filters would prevent us from considering pixel clusters that correspond to an exit sign or a portion of the Jumbotron for example. Thus we had both 2-D filters in hardware (cardboard baffles), 2-D filters in software, and 3-D filters in software. We also had the puck flashing so that all of its energy came in through a very narrow time slot, which in effect attenuated the ambient light sources by a factor of more than 250. We also used interference filters in front of each of our sensor camera lenses to eliminate any target that was outside our narrow acceptance spectrum in the near IR region. The idea of course was that the combination of all these spatial, temporal, and spectral filters would allow a puck powered by four coin cell batteries to compete with all the wattage of the arena lights, scoreboard lights, and other non-puck targets so we could identify the puck among all the clutter.

The Blue Glow

The puck typically was rendered with a blue glow at any given time, and with a blue streak when it was moving fast. The blue streak turned into a red streak when it went above some higher threshold speed., When it was moving fast, it would leave a streak that was some duration in time. So the faster the puck was going the longer its “tail” would be.And then when it was above some threshold speed, which we would set, the blue tail would become a red tail. These thresholds could be changed from game to game, or even during a game. David Hill would sometimes call us and say, “I’m not seeing enough red streaks. Let’s lower the threshold to 70, or 60 miles an hour.” So we’d tweak the thresholds, and would also tweak the colors, and other properties of the graphic effects. Of course our effects were pretty primitive back then. Looking back I wish the effect looked more refined, and perhaps more subtle, than it did. If we were ever to bring the system back, I think we would employ a lot of what we’ve learned over the years and improve it substantially. Despite its shortcomings, it was the system that gave us our start.

In our testing, we had just put a blue glow on the puck as an indicator that showed us that we were tracking. It was a diagnostic. And when we demo’d it to the Fox executives we told them that blue glow was just a placeholder. Fox assured us that their art department would develop a final look for the effect, but the one thing they were certain of is that it wouldn’t be the awful blue glow we showed them. In fact, David Hill used to refer to it as that god damn blue hedgehog. Oddly enough, in the end it somehow stayed a blue glow.

<flvplayer>GLOW PUCK FOOTAGE.flv|400|300</flvplayer>

Video of the puck tracking system in action during the All Star game. (Courtesy of Fox Sports, a Division of the Fox Broadcasting Company.

A month before the All-Star Game

It was terrifying. And the advertizing didn’t help. It was definitely over the top. Fox Sports was never known for its subtlety. I was pretty young and wet behind the ears at the time. I’m seeing commercials on TV that are saying this is the greatest innovation since the replay, or since color or something. They were playing ads for the upcoming All-Star game in which they had scientists in lab coats with chainsaws and Jacob’s ladders and shocking pucks and breathing life into them. It was funny, and it was fun, but at the same time it was intimidating.

But they were saying, “January 20th, tune in to Fox Sports for the greatest innovation in broadcast sports history,”. And of course I’m thinking, “Well, God, I hope so. We’ll see.” Three weeks before the NHL All Star game, we were trying to get our finishing touches on it and really we had only ever operated in San Jose Arena. That was where we lived so to speak. We really didn’t know yet what to expect at different venues. Clearly the locations where we could mount our IR cameras would vary, as would the height of the catwalks over the ice. Some arenas were relatively easy to run our cabling in, while others were very challenging. Most arenas had fairly modern lighting that produced relatively little IR, but we ran into an occasional venue that still used incandescent lighting – this was the biggest challenge. In fact we rented temporary lighting for one arena for the Stanley Cup playoffs to avoid anticipated difficulties with their incandescent lighting. After that, the venue immediately upgraded their permanent lighting.

Two weeks until All-Star

We tested the system one week before the All-Star game. It failed the test. One of the things we learned very late in the game is that fluorescent lights were not a big source of IR, but they were a pretty big problem for noise. As they warm up they would produce brief spikes of IR that interfered with our sync system. Learning this at such a late date was terrifying. The IR cameras weren’t seeing the puck, because their shutters weren’t opening at the right time. We struggled with the question of what to do at this point? We decided to go ahead and pack the system up and head to Jersey. The plan was to spend a week in N.J. testing at their facility before going on to Boston where the system would debut the following week. In N.J. we just plain failed to track the puck. We spent 16 to 18-hour days on the ice, and in the rafters. We left that test having only succeeded in tracking the Zamboni.

It was extraordinarily disheartening - and scary. We had some idea of what our problems were and some ideas of how to possibly solve them; but no time do try. It was time to move on to Boston for the system debut at the All-Star game. We figured we were better off learning the problems of that venue and tackling them, than to stick around N.J. any longer solving problems we might not encounter in Boston. Clearly we would have to address the N.J. problems, but that would have to wait. The lights in Boston were better, but being an All-Star game meant there were literally fireworks, strobes, and light shows.

This was a reasonably awful environment to be tracking the IR puck in. But it was doable. Making the changes we needed to make to see the puck was doable. We encountered a couple of places that were particularly challenging - Jersey was one of the worst places at that time for tracking the puck. Denver had incandescent lights, which we learned when we did a Stanley Cup series in there. Fox Sports, rented replacement lights. So they replaced the arena lighting with rented mercury vapors or halogens.At the Mile High Arena they replaced their lights as a result of us using replacement lighting. They saw what it looked like under proper lights, the players weren’t sweating, and the ice wasn’t melting. The incandescent lights were so awful that after Fox came in and rented these lights, they decided it was time to update.

Week of the All-Star Game

We’re in Boston. And we’re basically just scrambling to install the system and get everything working. Before arriving at the arena we put in an order for the wiring to be installed. We had a lot of Cat-5 to run up into the rafters and into the broadcast camera positions. But one thing that became an absolute constant; we’d show up to find that the cabling wasn’t installed, or it was installed to the wrong locations, or installed but not labeled, or worst of all – installed, and labeled, but just didn’t work. We spent many hours in those days at every venue “ringing out” the cabling. So we’re getting all our wires squared away and getting our camera mounting locations sorted out up in the rafters and at the broadcast camera. And then, of course, we were working with a new venue and a group of people that have never seen this system and they were somewhat dubious about it. We didn’t always know what we could and couldn’t get away with asking of the arena staff. Sometimes we asked permission, and at other times we asked for forgiveness. It was a learning experience for both sides.

Just before the All-Star game

Things were still “critically damped” so to speak. We’d just come off a week in Jersey where it didn’t work, and still didn’t know if it would work here. Probably two days before the game, Fox called a press conference. They had maybe a hundred members of the press there for a live demonstration with the puck and players on the ice. At this point we’re still warning David Hill that we’re not sure what’s going to happen. At this point the system was “kind of” working, but we didn’t know yet whether it would work well enough for the demo – or the game two days later. David is a sharp guy of course. He explained to the press “this is the greatest new thing, and if this works, it was my idea and it is the greatest invention since color TV. And if it doesn’t, it was an experiment.” But it worked well enough for the press conference. I think this is the first time we’d really seen the system operate in game conditions on the full ice. The demo just before the game worked. We felt some definite relief. But of course we didn’t exhale until we got through that first game.

We were very worried about the opening when they had dry ice, smoke, strobes and fireworks. Until all that cleared we were very worried about how well the sync system would work, and whether we could track the puck. We knew that it was nominally working, but fragile when everything was clear. We knew we were going to make some air. We didn’t know if we were going to be steady enough and robust enough to make air enough that it would be considered a success. Fortunately, it ended up being, as I understand it, the highest-rated hockey game - at least up to that time.

After the All-Star Game: Post Mortem

The All-Star Game proved to be a success. Even so, it wasn’t all roses and sunshine going forward. We continued to face new problems at many of the early games. Looking back, I’m proud of what we accomplished, but there were still plenty of tense times ahead at that point. At that time broadcasters weren’t really exposed to systems in an R&D phase. Traditionally, they would buy a deck or a camera, and it would be reasonably well tested and debugged by the time they got it. Even if one did fail, they had options. They could swap it with a camera from another position, or run with one less camera. They weren’t used to the complexity (and fragility) of a system like the one we were still fine-tuning. At that time they didn’t have computers in the signal path. The idea of a technology that wasn’t ready for prime time was a little foreign to them. For them it works or it doesn’t. And we said, “No, there are all levels of “works”. Sometimes you’ll see the highlight and sometimes you won’t. Or you see it, but it’s baubling around a little or it’s not completely accurate.”

After each game we’d have a post mortem of sorts. Maybe the game was a complete success, or maybe it was a partial success, or occasionally a complete, failure. But in any case, there was usually a long list of things that had to be fixed or improved by the following week. I’d say that went on, for the first season. The press coverage resulted in an inch thick pile of press clippings which Fox showed us two days later. They certainly got a ton of exposure. I don’t know how to tell how much was attributable to the puck, and how much was due to the press blitz they mounted over the puck.

In the original puck design we had a shock sensor to activate the puck. We designed and built a sensor that is not unlike a tilt switch in a pinball machine. It was a thin wire cantilevered inside of a tube so that when one slapped the puck down on the ice, as the referee does to start the game, it started a timer. And that timer gives you about 40 seconds of pulses. Anytime the puck gets hit hard enough to set that shock sensor off again, it resets the clock to start a new 40 second countdown. So it would time out in 40 seconds if it got thrown into the stands. But if it got hit again, the 40 seconds would keep renewing, and would keep the puck pulsing the entire time it was in play.

During transport the pucks could be turned on due to the inevitable bumps and shakes that happen in shipping. We built these big boxes with foam inserts for shipping the pucks. But the consistency of the shock sensors, which we were building by hand in our in our lab at Etak, wasn’t perfect. As a result, some of the pucks had been activated one or more times before being put on the ice. This is one reason we changed the puck design for subsequent years. Because we didn’t want to risk putting a dead puck on the ice we decided to embed an inductive coil in the puck which would be used to enable the electronics by RF before the puck could be activated by being hit. Prior to a game, we would take 30 pucks, and put them in a fixture that we made to enable them. The pucks had an extremely low current sleep mode they would remain in until they were activated. Once they’ve been enabled, the puck would behave in the normal way in response to the shock sensor.

The Reaction of the Fans

The real hard core fans were the ones that typically had a problem with the new effect. There were a lot of casual fans or non-fans that said, “Wow, that’s cool. Now I can see the puck and I can follow the game.” And even some of the fans recognized that “Hey, now I can watch it with my girlfriend. I don’t have to turn the game off, because now they can follow the puck.” But the interesting thing with the fans is that some of the fans just plain hated it, and there were plenty of them. Some of the fans said, “I hate everything about it - except I like the streak, because now I can see if there was an assist on a goal.”

You could see the streak bounced off of a skate or a stick or something like that. So they’d say, “I hate that damn glow, but I like the streak because that actually tells me something.” Or some of the fans that didn’t like the puck said “I hate that god damn streak, but I like the blue glow.” Or they’d say, “I hate the blue glow and the streak, but I like the x-ray effect when it’s between players at the boards.” We would still put a highlight and draw the puck in the highlight when it was hidden by the boards and players were fighting for it. So some fans would say, “Hey, now, I get something that I don’t get otherwise.” So even among some of the fans that hated it, some of them still had components that they liked. And interestingly, they didn’t all like and hate the same components.

The system was used for three seasons. But ultimately ESPN outbid Fox Sports for the broadcast rights to hockey. So it was used as long as Fox had the games.

The rights to the puck tracking technology that we developed belonged to Fox Sports. So it was highly unlikely that ESPN would use it, particularly since it had been introduced to a lot of criticism. There were sports writers that loved it; and there were sports writers that hated it, but ultimately there were enough hard-core fans that disliked it, that it was shrouded by this negative sentiment. Furthermore ESPN had made plenty of fun of the puck when Fox had the broadcast rights. Of course, they’re going to make fun of everything Fox does - the animated fighting robots and everything else. So they made fun of the puck. They couldn’t say, “gosh, we made fun of this puck for three years, but now we want to use it on our show.” But interestingly enough, ESPN has talked to us more than once about tracking the puck and tracking the players. I suspect they would go with a very different look than Fox. Of course Fox was not known for its understated approach. Fox tends to be over the top.

<flvplayer>GLOW PUCK SPOT 2.flv|400|300</flvplayer>

Video of Fox Sports's irreverent approach to promoting the puck tracking system. (Courtesy of Fox Sports, a Division of the Fox Broadcasting Company.

See also the article on Sportvision
See also the article Tracking the Ice Hockey Puck - FoxTrax (Glow Puck)