Oral-History:Robert M. Gray (1991)

About Robert M. Gray

Born in San Diego in 1943, Robert M. Gray possessed an early interest in digital communications. After completing his bachelor’s and master’s degrees at MIT, he chose to attend USC for his Ph.D. His dissertation dealt with rate distortion theory, or source coding for autoregressive processes, in an attempt to evaluate Shannon functions for sources with memory that had not been previously evaluated. Of theoretical importance, his dissertation disproved the claim that the Shannon results could only be obtained for memoryless sources. He was recruited directly from USC to work at Stanford. Gray has published a significant body of work, both individually and collaboratively including: developing one of the first examples of a universal code; developing a speech coding system; popularizing the algorithm for vector quantifier design, and later, its applications for images. His publications have received such recognition as the Information Theory Society paper prize. Gray has also been involved with IEEE Transactions both as an associate editor and editor-in-chief.

The interview concentrates on Gray's funding from the NSF and other sources. He briefly describes his areas of research, compares NSF to DOD and industrial funding, estimates the extent to which NSF funding has made a difference to his research, and evaluates the peer review process and program officers.

Also see Robert M. Gray's 1998 Oral History for a broader discussion of Gray's career.

About the Interview

ROBERT M. GRAY: An Interview Conducted by William Aspray, IEEE History Center, August 7, 1991

Interview #128 for the IEEE History Center, The Institute of Electrical and Electronics Engineers Inc.

Copyright Statement

This manuscript is being made available for research purposes only. All literary rights in the manuscript, including the right to publish, are reserved to the IEEE History Center. No part of the manuscript may be quoted for publication without the written permission of the Director of IEEE History Center.

Request for permission to quote for publication should be addressed to the IEEE History Center Oral History Program, Rutgers - the State University, 39 Union Street, New Brunswick, NJ 08901-8538 USA. It should include identification of the specific passages to be quoted, anticipated use of the passages, and identification of the user.

It is recommended that this oral history be cited as follows:

Robert M. Gray, an oral history conducted in 1991 by William Aspray, IEEE History Center, Rutgers University, New Brunswick, NJ, USA.

Interview

INTERVIEW: Robert M. Gray
INTERVIEWER: William Aspray
DATE: August 7, 1991
PLACE: Telephone Interview

Extent of NSF Funding

Aspray:

I have had a chance to see the e-mail message that you sent to my colleague Andy Goldstein and that has been most helpful, but I had a few questions that I would like to ask you about.

Gray:

Okay.

Aspray:

And do not hesitate to repeat anything that is in the e-mail message. Anything you give us would be most helpful to have.

Gray:

Okay.

Aspray:

I would like, if you would, to ask you what period of time you had grants from the National Science Foundation. Did you have one as soon as you came to Stanford in 1969?

Gray:

I had one probably the following year because it took some months to find out about writing proposals and submission. The first grant I applied for was one of the research initiation grants. I should add, though, that I had an NSF traineeship as a student, as a Ph.D. student, so in a sense my Ph.D. work was funded by the NSF before I actually had a formal grant.

Aspray:

Where was that work?

Gray:

At the University of Southern California.

Aspray:

Okay. And has the support been more or less continuous since?

Gray:

More or less. There have been breaks, I think the longest being a few months, but I have had more or less continuous support of, typically, a piece of my time and one student, although it has been in at least a couple of different technical areas.

Aspray:

What areas were they?

Gray:

The early work was mostly in Shannon information theory. Then it was for awhile more in actual data compression, in particular in speech compression. Then it turned to over-sampled analog to digital conversion, and most recently to image compression.

Shannon Theory and Data Compression

Aspray:

And can you tell me, for each of those four areas, what relation if any there was to the general computing field?

Gray:

The first area, the information theory, source coding, Shannon theory, whatever, is primarily mathematical theory, and the theoretical part did not involve computers except possibly for actually computing some of the performance measures for idealized communication systems. The latter part on speech compression and image compression was aimed at developing algorithms to actually do compression of speech and images. Those were highly computer intensive in that we had our learning sets, our databases on computers. We ran iterative optimization algorithms to design codes, we simulated those codes on computers and then used the computers to reconstruct the speech and image to either listen or see. Right now our image compression work is mostly medical image compression, and again that is intensively done on workstations and high resolution monitors.

Aspray:

As opposed to the tools that you use in doing the work, what about the impact or application of the research? Does that have application in the computing field?

Gray:

That is a little tough to answer. It has applications in a lot of areas where computers are used, in particular like medical imaging. In cases where computers are communicating with each other and you want to see images and store and send them. It certainly has application in image processing. Some of the people who do image compression are in fact computer scientists. It also has applications in multimedia high resolution monitors. For example, if you want a video window, that is still in relative infancy and involves quite a bit of compression. But does it have applications to computer science per se? That I do not know of.

Aspray:

Can you give me a bit of a better sense of what problem it was that you worked on, how that fit into maybe the larger community of research problems that were being worked on at about the same time or possibly later?

Gray:

Winging this for a more than 20-year career I cannot guarantee I will remember everything, and I will try to keep to key points.

Aspray:

That is fine.

Gray:

The early work on information theory, which took me pretty much into the middle 1970s to actually the early 1980s, when I was still moderately active. This was Shannon theory, it was proving coding theorems, that is, developing fundamental performance bounds for idealized communication systems. My work focused on the branch of Shannon theory called source coding, which is the mathematical name for data compression, that is, looking at efficient representations of information sources, what you do to those sources before you actually try and communicate them. Common examples are things like A to D conversion and speech coding, vocoding, image coding of various types that fit into the general area of information theory, which involved both channel coding, that is, error correction, and source coding. When I started in the area in my Ph.D. thesis, channel coding was by far the larger of those two disciplines. Source coding was relatively small, and I think source coding is still the minority partner but has grown. If you simply count papers on both theory and practice there is far more work going on in that area now than there used to be. But the work was definitely theoretical, the prime output was proving mathematical theorems, describing the optimal achievable performance in a communication system.

Aspray:

How does your work fit into that of the rest of the community? What accomplishments did your work have?

Gray:

I would say at the start I was one of the very few people working in source coding, although it was suggested to me by Irwin Jacobs, with whom I had done my masters thesis at the Massachusetts Institute of Technology. Since Shannon’s work, with the exception of some work by Gallagher and Berger, very little had been done in that area. So I think I helped popularize it. My initial work was extending existing results to more general models of communication systems and hence making the models better, less simplistic, less artificially simple. That involved extending coding theorems, extending the performance measures, evaluating and bounding the trade-offs between the performance measures of the communication system in particular between bit rate and fidelity or distortion. That was what most of my early work was involved in.

Later on it got not only into more general mathematical models of information sources and channels, but also into more general coding structures and showing that the Shannon results held for classes of codes in addition to those that he had considered, including things like nonlinear filters and various finite state algorithms. There were by the 1980s several more people developing similar results and variations on those results. By the late 1970s the field was much more crowded than it was in the late 1960s when I first started it.

Influential Papers

Aspray:

Were particular ones of your papers or particular results used extensively by the community?

Gray:

I think some of them were. I would have to go back and tell you exactly which ones. One of my early papers jointly with Lee Davisson won the IEEE Information Theory Group paper prize in the early or mid-1970s, and I think that had an influence on starting a bunch of work on more general sorts of channels. We had showed that some things that people had sort of abandoned, because they looked too complicated, were either solvable or at least almost solvable; that is, you can come up with useful bounds that were fairly simple. Now that is the first part.

The second main thing, and, here, let me lump together speech and image compression, because, since I first started that in probably around 1976 or 1977, that has been a common theme of my work. That was not so much doing theory for codes as trying to come up with algorithms, computer aided algorithms, to design codes that were good, hopefully optimal, but the best you could usually count on would be just good. The algorithms try to optimize but they are just usually incapable of actually finding a provably true optimum. There were two early papers, one joint with Linde and Buzo, where we came up with a basic design algorithm that subsequently turned out to be a lot like algorithms that had been around in statistics, in particular clustering algorithms, although there were some new ideas in that paper. There was another paper that also included Buzo, my brother A.H. Gray, Jr. and John Markel. The latter two had written a book on linear predictive coding of speech and were much more experts in the speech field.

Those two papers, I think, are probably my best known research papers, and they really started a lot of activity that is still continuing in an area now called vector quantization, [which is one approach to compressing data to very low bit rates] and, after lots of years of work, these ideas are used in most very low bit rate speech coding systems, in most speech recognition systems, and are incorporated into the standard vocoding systems. They are not nearly as ubiquitous in image compression, but they definitely, I think, made an impact. They have started a lot of work and they are a contender in that area. Those probably are the best known papers that I have written, although there was a tutorial paper called "Vector Quantization" in, I think, 1984 by IEEE ASSP Magazine (now named Signal Processing Magazine), which may be my most referred-to paper, simply because when people want to find out about it that is where they go.

Aspray:

Right.

Over-sampled Analog to Digital Conversion

Gray:

So, if I were to tell my boss which paper to look up in the citation index, seeing how much I am cited, those are probably the papers that are the best known and have probably generated the most work. Some returning to the theory, but mostly in code design. Lumping those two things together, the one remaining area is a newer area. That is, I first started in it maybe four years ago. That is the over-sampled analog to digital conversion work, in particular, looking at the theory of Sigma Delta modulation. And that is sort of an odd area. That is a very popular area in circuits and products and a very hot area from the engineering standpoint for very high resolution A to D conversion. But very little correct fundamental theory had been done for it early on, with some very minor exceptions. And again funded by the NSF, although this time by a different division, I think the Circuits and Systems division rather than the communications and networking.

I got involved with students in trying to develop theory that actually correctly and rigorously analyzed the behavior of the nonlinear difference equations that describe the ideal Sigma Delta modulators. That started off by just reproving results people thought were true anyway, but ended up in getting some fairly surprising results characterizing multistate Sigma Delta modulators and various other architectures. That work, I think, has had an initial impact in stirring up a lot of interest in doing theory. It has not yet had a practical impact, because we can still only analyze existing systems that are already a few years old, and we cannot analyze all of the ones that are out there. Some of them are just, they are not amenable to the techniques that we use. And the Circuits and Systems people, I think, like the work because it explains some of the bizarre behavior they are familiar with.

Others do not like the work because we have not told them how to fix it. It has not yet resulted in any improvements into the seat of the pants and engineering intuition design techniques, but it has warned, I think, a generation of circuit designers to mistrust the theory that they are initially taught. Because it is often wrong and incorrect. But the main impact of that is it makes them do a whole lot more simulations before they trust a result. That is, again, on the extreme theory side; I always have tried to have sort of the theory, which is kind of fun, and you can do with a computer for guessing and simulating, but it is mostly paper and pencil. The actual code design work, which is what most of my students do, because it makes them the most employable and gives them a broader range of signal processing and communications, not just information theory and coding; the latter is what I spend most of my time doing and what most of my students do.

Aspray:

I see.

Gray:

That, by the way, is not always funded by the NSF, although it will be shortly. At least one student or two half-students will be. The main work right now on that is funded by the National Cancer Institute of the NIH, because it is aimed specifically at medical imaging and evaluating or verifying the diagnostic accuracy of the coding schemes that we have produced from their ancient ancestors in information theory.

Non-NSF Funding Sources

Aspray:

Maybe this is an appropriate time to ask you about funding sources. In addition to the NSF over the years, who else have you gone to or who have you been able to go to, whether you chose to or not?

Gray:

The general comment that you probably hear from a lot of professors these days is that funding has been increasingly difficult through the years, and more and more of my time has been spent finding funding, primarily in order to support students. Because this is a very expensive university. During my first probably seven or eight years, funding was fairly plentiful. It was not a problem. In addition to the NSF, my main alternative sources were the Air Force and the Army research arms.

Aspray:

And how did their programs differ? Why would you choose to go to one rather than another?

Gray:

It was two reasons. I was invited to join an existing thing, and second, I liked the program monitor at the time, who was Joseph Bram. By the way, I should add, one of the things I liked about the NSF has been that most, not all, but most of the people I have dealt with have been very reasonable, trying very hard, and I respected the work they were doing. That was far less the case in my dealings with the Department of Defense. Bram was an exception. The Army, I do not remember how that happened. I think I was invited to submit, I did, we scored. They funded work around 1973 through 1975, and then, basically, with little warning, changed their priorities. The DOD was often changing its priorities, balancing between fundamental research and stuff that would be more Mansfield Amendment and immediately applicable. By 1976 or 1977 I just sort of got fed up with DOD funding and quit pursuing it.

Aspray:

Were there any conditions that were different, say when you had DOD funding as opposed to NSF funding?

Gray:

Not that I remember. The conditions that were different came later when I had industrial funding, and they would often carry the constraint that before we could publish anything it had to go through an approval procedure in the company. That is the only real constraint that has ever been imposed on me for contract research. After the DOD, there was sort of a break. There was, I guess it was Defense Advanced Research Projects Agency funding, which is still the DOD through the Navy, but it was a contract I inherited. Somebody left. Since I had lost other funding I was asked to take over that and did.

Aspray:

This was IPTO funding?

Gray:

It was more of a civic duty than my pursuing it. That was a distributed sensor network contract which had like six or seven faculty, including me, and I basically used it as an administrative job that I took on. Then, by the 1980s I got involved with getting some small industrial contracts, or gifts sometimes. They were typically aimed at providing computing equipment and paying for students, especially paying for students between other grants when I found myself without funding and had to finish up a student. It is also good for the equipment, because one difficulty with the NSF is, because the delays are so long it was always tough to get equipment that was current. You would propose something, but you would want it, and rather than wait a year and a half we would use unrestricted funds to buy it. And often the NSF grants would have the equipment removed from them. So it has really been industrial gifts that have provided me with the computing equipment that is unique to my own group.

Aspray:

Even avoiding the question about delay, were the grants large enough from the NSF to pay for the equipment?

Gray:

No. My grants, I think, were usually, if you compare it with the national average, better, but that was because Stanford’s overhead was higher. If you looked at what actually came in to faculty and students I would guess I was maybe slightly higher than typical because Stanford put so much pressure on us to get research offset during the academic year. But except for that difference, they were comparable. I rarely had the money for equipment. There were maybe one or two exceptions out of 20 years where I would get an extra $10,000 or something to put into workstations or micros.

NSF Bureaucracy

Aspray:

When you first started getting your money from the NSF, the organizational unit in computing was the Office of Computing Activities. Since then I guess there have been many different changes. But it was not clear to me whether your money came from the computing groups or the engineering groups at the NSF. Could you tell?

Gray:

Mine came mostly from the engineering groups. And it was the group that Elias Schutzman headed up for a very long time. I would have to go back and look up exactly what the name was, but it probably included the word communications in it.

Aspray:

What about more recently?

Gray:

Recently it has come from MIPS, which is part of the Circuits and Systems. It is now under John Cozzens, was under John Woods when I first gravitated in that direction. There does seem to be some overlap between those two groups.

Aspray:

Do you want to explain that?

Gray:

Both are interested in communications and signal processing. I think the group that Schutzman used to head, and now Aubrey Bush is in charge of, looks at a variety of network issues as well, which I do not think Cozzens’ group does. Cozzens’ group, on the other hand, funds a lot of actual circuit implementations and VLSI work. They are very interested in looking at communications and signal processing that is implementable, if not by the people doing the research, at least by others. Both do support various types of applied mathematics within the engineering division. But information theory, which was my background, is mostly in what is now Bush’s group, and I have no funding from that group and have not had funding from them for probably about four years now. It has all been from the MIPS and the Circuits and Systems.

Aspray:

Is there any more that you would care to say about the orientation of those programs at the foundation? For example, did you know that a proposal slanted in one direction or another direction would be more likely to be successful than in another direction?

Gray:

I am not sure exactly what you are asking.

Aspray:

Certain foundation programs have certain objectives and, just to take a hypothetical case, somebody might be interested in developing only theory oriented work. Or another group might be interested in something that would have potential industrial applications.

Gray:

In my early stage it was clear. I had only one choice. I sort of jumped groups later on because when I first applied to the MIPS I already had funding from the Information Theory and Communications and I thought the new stuff would be of interest to circuits people. And so I sent it to the circuits division, even though it was still theory. Also, it was not clear to me that it would fit in the Information Theory. Knowing those groups had different budgets and that they talked to each other, it seemed reasonable to target towards the group whose charter seemed to more directly include the work. So I think mostly I just took them at their word of what was the responsibility of those groups and submitted accordingly.

Since they do talk to each other, there have been times where it has gone to one group originally and ended up somewhere either because one group had funding and the other did not, or in one case because I was informally invited to submit some stuff, but not told that there was a deadline which I missed by a week. At that point they decided to make the deadline hard. So they basically passed work that would have probably been more appropriate for one group onto a different group, which eventually funded it, because I think they were trying to expand into that direction, and it seemed to me there was a little bit of competition going on there. But I am not going to question who decides to give us the funds for the work that has been proposed.

NSF Funding for Information Theory

Aspray:

Do you think that your case is typical of researchers in information theory and signal processing in terms of the interest of the foundation in supporting that area? Do most of your colleagues in the area get their support from the NSF?

Gray:

I think certainly in the information theory area most get it from the NSF, because that has been really the sole source that I know of that would fund fundamental research that is essentially applied mathematics to engineering problems. Because there is a limited amount of money in that area, there has always been a reasonable amount of competition. I think people younger than I have often had a hard time unless they really stood out. One advantage is that when I started in the area I chose there was almost nobody. So I had a chance to get started. And then it was an issue of being able to continue to do good work. For people starting fresh out of university I have often heard of difficulties getting funding or, even once they have sort of gotten a start, getting continuing funding. But it is tough for me to judge just what the real problems there are. I suspect sometimes it has been that what they were working on were simply not problems of sufficiently great interest to the peer reviewers, as well as simply more people were applying for more money as more universities pushed their junior faculty into trying to get research grants. So that had a lot to do with it as well. My major funding now comes from the NIH, and that grant is much more than the NSF grants that I have had.

Aspray:

By a factor of what?

Gray:

The trouble is, NIH deals with only direct costs. So I would guess probably computing the equivalent in direct costs, since the numbers I am familiar with are different, are probably like four or five times. On the other hand, my NSF grants have only me, and the NIH grant has three genuine PIs (Principal Investigators) and three sort of token PIs and more students. So there are simply more people on it. But the NSF to my knowledge does not give out multiple PI grants of that size, at least not very many, unless it is to a center.

Aspray:

That confirms my understanding as well.

Gray:

But all of the stuff we do for NIH had its origins in tiny little projects done for the NSF, and without the NSF they would have never happened.

Peer Review Process

Aspray:

You mentioned the peer review process before. Do you think that your reviews were fair? Did they choose good reviewers? Did people seem to understand what was going on?

Gray:

<flashmp3>128_-_gray_-_clip_1.mp3</flashmp3>

I would say usually. It is an imperfect system, but nobody has a better one, and there certainly have been ones that have irritated me, where I think they got it wrong. But I think only once have I actually lost a grant because I got a bad throw of the dice, and that fluke balanced by at least once where I thought I did not have much of a chance because I was trying to sell theory to circuit theorists and the reviews came out surprisingly knowledgeable and helpful and positive. So those probably cancel each other. So on the whole I would say the variance is about as much as one could hope for. And I do not know how they could do it better. I try very hard to write informative reviews. The more negative I am, the longer they are in order to justify it. What makes me angry is when I get a very negative review that is very short and absolutely unhelpful. But I have had NSF program monitors specifically state that even if you are going to be negative and if it is somebody young, do your best to provide helpful advice so the next time they will get it right. And I think that is a good attitude for the program monitor, and I have tried to respond in kind.

Aspray:

Have you had other kinds of NSF support for conferences or travel or whatever else?

Gray:

I have had no separate grants for travel because I have always pulled out whatever I had in my own grant and added unrestricted funds to make my trips. I may have been indirectly involved with sort of a group effort to go to the NSF to get, I think, some matching funds once for some computer equipment for the entire laboratory, but somebody else put that together and I just wrote a paragraph or something. I think that was a relatively modest thing basically to allow us to get matching funds from Sun and get a file server.

Aspray:

Has the NSF had any effect whatsoever in shaping the direction of your research program?

Gray:

I would say yes. I pay great attention to the peer reviews I get back on proposals, even if I do not agree with them. They often have something to say about things we should be looking at or incorrect trees that we are barking up. That has been the main influence, the comments sent back to me by peer reviewers. The times that I have had grant proposals turned down, I think I’ve had only one NSF proposal actually turned down, and that basically made me decide to change the emphasis and go elsewhere. In hindsight that was probably beneficial. That was a case when funding was really awful, the reviews were good but they were not good enough to make what was a terribly tight cut. And in hindsight that was probably a good thing to happen.

NSF Program Officers

Aspray:

Are the program officers you have dealt with knowledgeable enough about the area that they can provide some guidance in research direction?

Gray:

Usually. There have been exceptions which I chaffed a bit under and did not feel that they were sufficiently knowledgeable, but that was a definite minority. Most of the people I have worked with worked very hard and, I think, given the constraints that they had, they did a very good job of understanding a broad range of things and providing decent advice.

Value of NSF

Aspray:

I would like to give you a chance to make any additional comments or remarks that you care to make about the foundation.

Gray:

I do not know. I mean even when I have had a grant turned down, I felt very positive towards the organization. As it gets more difficult to find funding and as the stress level on assistant professors in particular seems to be growing with the years, I think the NSF has played an absolutely crucial role in keeping fundamental research going in universities. It has not made it easy and has not provided lots and lots of money, but it has provided a lot of small grants that have kept a student or two and a faculty member doing by and large interesting stuff, and I think it has done as good a job as could be expected deciding where to put the money. I think looking from my admittedly prejudiced viewpoint my only negative global remark would be having to do with the centers. And this is nothing new. I have heard these arguments lost before, but I always worry about putting lots of money into concentrated points in large operations where I fear much money goes into administration, very little gets divided amongst the small number of people, and you have no guarantee you are getting the best people in an area if you simply pick it by geography. So I have high praise for the individual PI; I would like to see maybe a few more joint grants with maybe two or three PIs, but I am not a fan of the centers that I have seen. But that is another conversation.

Aspray:

Okay.

Gray:

Thanks a lot.

Aspray:

Right. Bye.