Oral-History:Russell Mersereau

About Russell Mersereau

Merserau went to MIT for his undergraduate (Electrical Engineering, 1968), graduate, and post-doctoral years, from 1964 to 1975. He then went to work at Georgia Tech, where he has taught ever since. His Masters work was on statistical correlations on electrocardiographical cardiac data. He then took a course with Allen Oppenheim that propelled him into the field of digital signal processing. His doctoral work, on building multidimensional images from projections of two-dimensional objects, resulted in the dissertation, “The Reconstruction of Multidimensional Signals from their Projections.” As a post-doc, he wrote a Thompson Award-winning paper (1975) with Dan Dudgeon on multidimensional digital filter design. Dudgeon has tended to apply the work to sonar array processing, Merserau to medical imaging. Mersereau also collaborated with Wolfgang Mecklenbräuker and Tom Quatieri on multidimensional digital filters. At Georgia Tech Mersereau co-wrote a textbook with Dan Dudgeon on two dimensional digital signal processing. As noted in this interview, Mersereau's research has included hexagonal sampling, iterative signal restoration algorithms, image restoration, image modeling, two-stage multirate coding of color images, and video coding. Mersereau also describes his mentorship of graduate students researching a variety of topics.

This interview describes the growth of the Georgia Tech digital signal processing group (from Merserau, Ron Schafer, and Tom Barnwell to ten faculty), noting that the professors were more collaborative than most departments (partly for lack of graduate students and high-tech local industry to work with), and more focused on the group’s collective progress.  Mersereau describes innovation in the Georgia Tech digital signal processing curriculum. He notes the ballooning of publication that makes it harder to keep up with the field in general; he believes the Signal Processing Society performs an important function.

For further discussion of Mersereau's collaborations with Wolfgang Mecklenbräuker, see Wolfgang Mecklenbräuker Oral History.

About the Interview

RUSSELL MERSEREAU: An Interview Conducted by Frederik Nebeker, IEEE History Center, 6 October 1998

Interview #346 for the IEEE History Center, The Institute of Electrical and Electronics Engineers, Inc., and Rutgers, The State University of New Jersey

Copyright Statement

This manuscript is being made available for research purposes only. All literary rights in the manuscript, including the right to publish, are reserved to the IEEE History Center. No part of the manuscript may be quoted for publication without the written permission of the Director of IEEE History Center.

Request for permission to quote for publication should be addressed to the IEEE History Center Oral History Program, Rutgers - the State University, 39 Union Street, New Brunswick, NJ 08901-8538 USA. It should include identification of the specific passages to be quoted, anticipated use of the passages, and identification of the user.

It is recommended that this oral history be cited as follows:
Russell Mersereau, an oral history conducted in 1998 by Frederik Nebeker, IEEE History Center, Rutgers University, New Brunswick, NJ, USA.

Interview

Interview: Russell Mersereau
Interviewer: Frederik Nebeker
Date: 6 October 1998
Place: Chicago, Illinois

Childhood and education

Nebeker:

Could we start by hearing where and when you were born and just a little about your family?

Mersereau:

I was born in 1946. I was conceived right after my father got back from World War II. I was actually born in Cambridge, Massachusetts, but I didn’t grow up there. I grew up in southern New Hampshire, and was an only child.

Nebeker:

What did your father do?

Mersereau:

He was a small businessman. He had a couple of local stores and things that didn’t work out particularly well, and he eventually discovered that he was much more talented with figures and became an IRS agent. He did that until he retired.

Nebeker:

Were you attracted to science and technology as a young person?

Mersereau:

I had an uncle who was an engineer, a mechanical engineer it turns out, but he was an engineer. He had only one child, who was a daughter, so he took me on as his project of creating an engineer of the new generation. So when I was about four years old I was taken on a tour of MIT and told that this is where I’m supposed to go, and all these kinds of things.

Nebeker:

Sounds like it worked.

Mersereau:

Yes.

Nebeker:

And so you did take an interest in such matters and thought at an early age?

Mersereau:

Yes. That sounds like it was a little bit more of a definite interest than it probably really was. I mean, I had this lingering thing about being an engineer, and I did enjoy the science and things, but you know I considered medicine and some of the other kinds of things, as all kids do when growing up.

MIT

Undergraduate studies

Mersereau:

I applied to and got into MIT and went there in the fall of 1964.

Nebeker:

Why did you choose Electrical Engineering?

Mersereau:

I don’t know. Initially at least I didn’t have a strong feeling about what I wanted to be. I had chosen to go to MIT because I saw myself as a scientist or engineer and that was where you went for the broadest education. You had the most flexibility to move around and enough flexibility in case you didn’t want to become a scientist or an engineer. But I more or less eliminated those possibilities and in the first few weeks that I was there I considered several different potential majors, but I think at that moment I was following the crowd. The majority of the students went into electrical engineering and it did strike me as being a potentially interesting area. Initially, I went without a well-defined plan.

Nebeker:

Were there teachers that really influenced you in those years?

Mersereau:

There weren’t teachers that influenced me before I went into electrical engineering. When I got into electrical engineering there were teachers that convinced me that it was a great place to be. Amar Bose was probably the first of those.

Nebeker:

You took classes from him?

Mersereau:

My first course, 6.01, he was the instructor. Probably the best instructor I ever had.

Nebeker:

What sort of electrical engineering attracted you as you went through the curriculum?

Mersereau:

What appealed to me the most about things was what I would call the applied math side of electrical engineering. I didn’t enjoy the labs as much because I wasn’t really a circuit builder. But I was intrigued by communications. It was difficult at first finding my way in this area. When I was looking for my Master’s thesis I started talking with some people in the communications area, and although they were doing some important things, it just didn’t appeal to me. I ended up doing a Master’s thesis in a biomedical area. And it was then that I began talking with Oppenheim and started talking about signal processing which was what I was looking for.

Nebeker:

Did you take more than the usual amount of math as an undergraduate?

Mersereau:

No. At MIT you more or less are forced to take a certain amount of it, but I didn’t take an unusual amount. It certainly wasn’t a dual major.

Master's studies

Nebeker:

And after you finished your Bachelor’s degree you decided to get a Master’s?

Mersereau:

Yes. I actually originally had planned to leave MIT and I wanted to try a different school for the Master’s. I had an NSF fellowship to go to Stanford. But that was 1968 and the Vietnam War was on, and there was a mechanism by which I could get a deferment if I stayed at MIT. That option would not have been open to me if I changed schools, so I stayed. At MIT you could use your Master’s thesis to satisfy your Bachelor’s thesis requirement, and as far as the Draft Board was concerned I didn’t have a Bachelor’s degree. As far as MIT was concerned, I was a Master’s student. And so that provided a year of deferment. Then I got a teaching assistantship and my local Draft Board gave me a deferment as an occupational at that point.

Nebeker:

You said the Master’s thesis was in the biomedical field?

Mersereau:

The Master’s thesis was in the biomedical area. I did some statistical correlations on cardiac data with a professor named Peter Katona, who later left MIT and went to Case Western Reserve.

Nebeker:

How did that relate to electrical engineering?

Mersereau:

He was an electrical engineering professor, and basically it concerned electrocardiography. They had made some measurements at Massachusetts General Hospital on activity in the vagus nerve, as they were operating on some dogs. They were trying to monitor some blood pressure reflex activities. But because of the drug-induced state of a few of these dogs, they started getting some very irregular heartbeats. The question that I was trying to confront was, was the activity that they were monitoring in this nerve statistically different before these irregular heartbeats than it was at other times. It was a big task calling for statistical analysis. The one thing that led me toward picking this particular problem was that I always had computer anxiety. Partly because of a bad experience in a freshman programming course, I more or less avoided computers. But even in 1968 one could recognize that one had to overcome this because computers were definitely here to stay. So this particular problem was a real hindrance that I resolved by plunging into the middle of the pool and learning how to swim. This PDP-7 computer over at MGH is what we ran this data through.

Nebeker:

I see that you were in the co-op program.

Mersereau:

I was a co-op student, and I worked at a company that was near Boston. I was in a biomedical group that built cardiac assist devices.

Nebeker:

What did your work with them involve?

Mersereau:

It was primarily instrumentation. I was building circuitry for timing these devices and doing some modeling of the systems.

Nebeker:

Was that a good experience for you, the co-op program?

Mersereau:

The co-op experience was a great experience, because it put you to work doing things. I was treated like a real engineer, so that was good. Biomedical engineering at the time struck me as an area where I didn’t fit in. I liked the idea of working on problems that might be useful to people, but I didn’t see it as being as a viable area of research. It was probably a bit of shortsightedness on my part.

Nebeker:

How did you first run across Allen Oppenheim?

Mersereau:

Through Don Johnson. Don Johnson had worked with him at Lincoln Labs in the summer. I guess he had also worked with Tom Stockham and was excited about some of the things that they were doing. He said that Oppenheim was going to teach this DSP course using Gold and Rader. It was actually the first time that they taught it. Don was going to be the TA for it, although he was also a student in the class at the same time, and he said that I should take it. So I signed up to take it at that point.

Nebeker:

How did you come to know Don Johnson?

Mersereau:

Don Johnson and I were friends. He was in my class at MIT. We lived on the same floor of the same dorm, and he had a room that was a couple doors down the hall from mine.

Nebeker:

Can you tell me about that course?

Mersereau:

Well, we used Gold and Rader as a textbook. It was a unique teaching style. If you started to look the least bit dazed, Oppenheim had a tendency to call out your name. He would call out your name and ask you a rather pointed question.

Nebeker:

How large was the class?

Mersereau:

The class was fairly large. It filled one of the decent sized classrooms. It didn’t fill an auditorium, but there were probably 70 or 80 people in there. But they were all on their toes after the first few times that he pulled that off. Those are the most vivid things I remember about it. There was a certain amount of signal processing that I picked up and a certain amount that I didn’t due to the pace of the class. I remember a few of the specific homework problems that I had some difficulties with, and a couple that I solved successfully that other people had trouble with.

Nebeker:

What year did you take that course?

Mersereau:

That would have been in the fall of ‘69. Oppenheim had been at Lincoln Labs. He was an assistant professor at MIT and he had gone to Lincoln Labs for a couple of years while still retaining his MIT position, and then he came back to the campus.

Nebeker:

Did that course make you think that signal processing was a good area for you?

Mersereau:

Of course, I had also been talking to Oppenheim at that point about a research problem. He brought the problem of what has since come to be called the CAT scanning problem to my attention. This was the idea of building multidimensional images from projections of two-dimensional objects. He had seen some papers about it. Not specifically applied to X-rays, but instead to electron micrographs. Those papers had intrigued him. There was nobody in the signal processing group that was particularly involved with multidimensional problems at that time. There was Prof. Schrieber and Prof. Huang’s group upstairs who were doing some image processing. But they had come from slightly different directions.

Nebeker:

So the idea here was to use some of the same techniques taught in that course for this problem?

Mersereau:

Yes, it was a Fourier transform problem. Oppenheim had an approach that he thought would solve the problem that was based on doing some sampling and turning one of the key integrals into a convolution sum. He could then compute this with a DFT. That particular technique didn’t work, or at least I didn’t get it to work. But we stayed on the problem and attacked it in other ways. Oppenheim’s own work at the time dealt mostly with a lot of one-dimensional problems, and so I informed him of what I was doing and he gave me some suggestions. Still, most of the project’s solutions evolved from me.

Ph.D. studies

Nebeker:

When you completed the Bachelor’s and Master’s, you were admitted to the Ph.D. program and got a TA-ship?

Mersereau:

Right. There were the usual exams and things that had to be taken, and apparently I made it through those. So I went straight through. I was actually there for 11 years. There were three degrees in all that I received, and then I stayed for two and a half years as a post-doc.

Nebeker:

So Oppenheim was your advisor?

Mersereau:

Yes. Oppenheim was my thesis advisor for the Ph.D., and Ben Gold and Tom Huang were readers.

Nebeker:

Were there major hurdles along the way?

Mersereau:

Well, yes, there were hurdles along the way. Getting a Ph.D. is a serious self-evaluation process, and you go through this period of self-doubt. I went through one of those a couple years into the thing before useful results came in. You start asking yourself whether you can do it. I remember going to Oppenheim's house one evening. I guess I’d called him first and he told me to come over. My wife and I went out and were commiserating on the frustration of having to deal with this. He said that it was actually fairly natural, that he had had similar periods and that he had seen it in others. Some people didn’t emerge from them. They just left. While for others, that would be their low point, and then they’d pick up. And that he had some faith that I would be able to see things through. Actually it wasn’t that long after that that I got the first promising results. That definitely was the low point of the career. But in my own students I see in many cases those same dips. It seems to be quite common.

Nebeker:

What was the doctoral dissertation called, do you remember?

Mersereau:

Something like "The Reconstruction of Multidimensional Signals from their Projections."

Nebeker:

I see you have this 1973 article, “Recovering Multidimensional Signals from Projection.”

Mersereau:

Yes. That more or less summarized the work from the thesis.

Nebeker:

Was it well received?

Mersereau:

I don’t know.

Nebeker:

I mean, did people notice?

Mersereau:

It got some notice. In the Signal Processing Society it got some notice. In the actual computer-aided tomography community I think that it was less visible. Shortly after doing that, I decided to stay on as a postdoc. I wanted to branch out a little bit in my area so I got involved with more general problems in multidimensional signal processing. At the time there was some problems with multidimensional digital filter design. That led to a paper with Dan Dudgeon, which had a much more profound effect than the thesis.

Nebeker:

Was Dan Dudgeon a student there?

Mersereau:

Dan was another student of Oppenheim’s. He was about a year behind me, and Tom Huang was also a reader on his thesis. About that time Tom left MIT to go to Purdue and I stepped in as the reader because at that point I had my degree and I was there as a postdoc, a research associate. I got involved in Dan’s research, and Al Oppenheim was putting together a special issue of the proceedings on digital signal processing that came out in 1975. He was inviting contributors. He had a feeling for the kinds of things that he wanted to be in that issue and he invited me to put together one on two-dimensional digital filtering. I don’t know if that was actual charge, or if the charge was two-dimensional digital signal processing. But in any case, he asked me to put that in, and I asked Dan to do it with me. That paper got the Thompson Award for that year.

Nebeker:

Can you summarize what that work was?

Mersereau:

Oppenheim and Schafer’s textbook came out in 1974. It had this nice, elegant description of how one did signal processing, everything from sampling to linear systems and DFT computations, and we basically took the same topics in more or less the same order and showed how they all worked when you dealt with images and multidimensional signals. Therefore, it was the first tutorial exposition of some multidimensional signals. There were a few things in there that were a little bit less obvious from the one-dimensional case, some of which historically proved to be interesting and some not.

Nebeker:

What were other two-dimensional signals techniques that you had in mind that might have been useful?

Mersereau:

Dudgeon went off to work at Bolt, Beranek & Newman (BBN), and he did sonar array processing. So he was definitely looking at arrays and time frequency signals. I was more interested in the images and the medical sort of problems. So those were the two applications that drove us, but we had separate applications in that respect.

Nebeker:

Can you explain to me how ? You mentioned before that the Schrieber-Huang work was coming at image processing from a different direction. Can you explain that to a lay person?

Mersereau:

Well, maybe not with complete authority, because I wasn’t in the Schrieber and Huang group, but Schrieber’s background was in television, and so he definitely was coming from that point of view. Huang was doing more digital signal processing. In point of fact, Huang was doing a lot of the same sorts of things that I was, but I wasn’t as aware of it at the time. Oppenheim’s point of view had been more algorithm driven, looking at the FFT, looking at the z-transforms and some of these kinds of things. So it was a mathematics coming up, whereas the Schrieber group was looking at the application and coming down. And Tom Huang was the person that was really connecting them all together.

Postdoctoral appointment

Nebeker:

So you completed your Ph.D. and stayed on as a postdoc you said?

Mersereau:

Right. At the completion of the Ph.D., I thought that I really wanted to teach, but it wasn’t a great market for faculty positions at that time. Therefore, I did an interview in Flanagan’s group at Bell Labs, and there ironically I would have worked with Ron Schafer. I also had an offer to stay and to work in a group at Lincoln Labs. Then I had an opportunity to stay on at MIT as a postdoc, which is what I chose to do. I thought that I could do it for a couple of years and then if the market opened up for teaching I would do that.

Nebeker:

And was that to work with Oppenheim or somebody else in particular, or to do your own work?

Mersereau:

It was in Oppenheim’s group, and so I definitely was working with him in that respect. But he and I were not working on joint projects, other than jointly trying to keep the group together and keep it growing. I had my own ideas. I wanted to look at some of these more general multidimensional signal processing algorithms. So I did that. Al provided some funding support with some of his grants to make some of that possible.

Nebeker:

This desire that you had, was it driven more by math or by the possibilities for application?

Mersereau:

At the time it was more mathematics driven. From a professional point of view, I also recognized that I needed to publish. But I did have some ideas of things I wanted to try for specific kinds of applications, some of which evolved from the thesis. And of course MIT is a great place to get very talented students to get involved with things.

Collaborations with Wolfgang Mecklenbräuker

Nebeker:

I was interested to see that you did a few papers together with Wolfgang Mecklenbräuker.

Mersereau:

Yes.

Nebeker:

How did that come about?

Mersereau:

Wolfgang came to MIT on a sabbatical. He had a one year sabbatical. It went from January of 1975 until I guess January of 1976. I left in about September of that year. But during the eight months that he and I were both there, we had a marvelous collaboration. He was looking for some problems to get involved with. He had done some work Theo Claasen, on limit cycles and techniques for dealing with those. He wanted a little bit of a break from that particular problem. As people in the group go, I had a lot more free time than Oppenheim did. We talked about many things; one of the things we discussed was a problem that I was working on. I was trying to design some two-dimensional equiripple filters. We batted that around for a while.

There’s an approach that Jim McClellan had worked out in his thesis for transforming one-dimensional designs into the multidimensional, which was a very elegant and beautiful idea. We got looking at the problem of how could we design these. I thought this might be a framework that we could use for designing the multidimensional filters. We got into that work with a Master’s student name Tom Quatieri, who has gone on to do a lot of other things in other directions. He once processed multidimensional digital signals. That led to the idea that we could actually use this as a structure for implementing these filters, and some very efficient implementations that came out of it. I think it was actually Wolfgang that originally got the idea, but it was hard to tell because we were getting excited and we were always throwing ideas back and forth. Anyway, that led to a flurry of activity for eight months.

Nebeker:

Do you think those were influential papers?

Mersereau:

Yes, I think they were.

Nebeker:

I’m actually going to be talking to Mecklenbräuker in a few months, so I will have a chance to meet him.

Mersereau:

He went back to Philips, and then of course he went on to Vienna. He’s done a lot of other things. But I don’t think he’s done as much since then in the two-dimensional area. That was his introduction to it, but we maintained the collaboration. I spent six weeks last summer of ‘96 in Vienna in his lab.

Teaching and research environments

Nebeker:

So is there more in that postdoc period that we should cover?

Mersereau:

I did some teaching in the DSP course and in the year that I actually finished my thesis, Oppenheim was on sabbatical in France. This was the pre-e-mail era, so we had to mail drafts of theses back and forth. I actually oversaw the move from Building 20 into the new building.

Nebeker:

Now Building 20, which was the old radiation laboratory building.

Mersereau:

Right. One that I guess was built temporarily in the Second World War. It’s just now in the process of being replaced I think.

Nebeker:

How was that for Oppenheim’s group? The facility itself?

Mersereau:

The floors of Building 20 were uneven, not level and everything else, there was no air conditioning. In that period Oppenheim was more of a squatter. Ken Stevens had most of the space, and Oppenheim had an office there had some students, but we didn’t all get housed there. I, being a TA, had an office in another building. So I’d go over there to meet with him, and the computers were over there. That’s where we ended up working, on the same computers as Ken Steven’s students. We didn’t have much of a group identity until we moved into the new building. And that was a space that was separate from the speech group, for example, which was on a different floor.

Georgia Tech

Digital signal processing group

Nebeker:

I see. Then what came along after the postdoc?

Mersereau:

After the postdoc I went to Georgia Tech. I interviewed at a number of schools, and one of them that I visited was Georgia Tech. At Georgia Tech I again ran into Ron Schafer. In fact he was the reason that I had applied to Georgia Tech.

Nebeker:

You knew him from MIT?

Mersereau:

Well, no, he’d left MIT before I was involved with Oppenheim, but he was a recurring visitor, in part because he and Al were working on the book. At the Arden House workshops I also ran into him. I would have been working with him at Bell Labs and then the year after I interviewed at Bell Labs he left there and came to Georgia Tech. So I applied there, and a few other places as well. Ultimately, he was probably the main factor that led me to go there. I figured I couldn’t turn him down twice. He and I have both been there ever since.

Nebeker:

Tell me about the situation at Georgia Tech when you arrived. How long had Schafer been there?

Mersereau:

Schafer had been there one year. I came one year behind him. There was a third member of the group, Tom Barnwell, who was actually there first. He came in about 1970 or ‘71. He had MIT degrees as well, but in a different area. He had worked with Francis Lee, and at Georgia Tech he was trying to teach some courses in signal processing and trying to maintain the ancient computers to do it on. He was an assistant professor and drowning in that particular mode. Some people had given money for an endowed chair, and he used that to entice Schafer.

With Barnwell, Schafer and myself arriving at Georgia Tech, that was enough of a critical mass for us to start doing some things. We were very lucky at getting some early funding and getting some early equipment donations for improving our computing capability. He definitely was a major player, and particularly in getting the hardware we used.

Nebeker:

At that time it was probably one of the most significant digital signal processing groups.

Mersereau:

Yes. I mean Rice had a couple of people that did it. At MIT, its group had Jim McClellan after I left. Therefore Oppenheim and McClellan were a two-man DSP group. They did have the speech laboratory that involved Ken Stevens and they did have Schrieber’s activity in what was called the CIPG lab. But the image processing folks up there were many. I mean if you put all those people together who at Georgia Tech would have been called signal processors, there were more of them. But they didn’t function as a separate group. MIT was a little bit more fragmented.

Nebeker:

Did you, Schafer, and Barnwell work together very much?

Mersereau:

Yes. We didn’t have that much choice. At MIT, we’re blessed with hundreds of eager and good students. At Georgia Tech we had some good students and we had eager students, but we didn’t have hundreds of them. So we did a lot more collaborating among ourselves. For some of the projects and courses we had to pool our efforts. It’s always been a very cohesive team there. Whenever we’ve added faculty members to the group, it’s been with the understanding that these are people that we can work with.

Nebeker:

And just to get an overview on that, can you tell me how it’s gone in the couple of decades since then at Georgia Tech for signal processing? You say you added some people?

Mersereau:

Yes. The group now is about ten people. For the first few years there were the three of us. Tom’s first Ph.D. student was Panos Papamichalis, who has done things in the society. Ron Schafer’s was Steve Kay, who is now on the faculty at Rhode Island and my first Ph.D. student was Gary Shaw, who is at Lincoln Labs. About the time that the three of those were finishing up their degree, DSP was becoming a much more popular area with students everywhere. So there was a fair amount of student demand. The courses were getting fairly large; therefore we decided to add some additional faculty. One year we hired both Monty Hayes and Mark Clements, both of whom also came from MIT. As can be seen, we had a little bit of an inbreeding problem. Monty Hayes in fact had worked with Oppenheim. He actually had his thesis signed by Jae Lim, but he also was a product of that group. Jae Lim at that point had joined the group at MIT. I guess he finished his Ph.D. a couple of years after I left.

Nebeker:

In the years since a few more positions have been added?

Mersereau:

Right. Then we added a couple of faculty members that were our own. Mark Smith was one of them, and David Schwartz. David Schwartz left after a couple of years and went to work at Hughes heading a research effort out there. Mark Smith is still at Georgia Tech. The person whose life I’ve been most tangled with is probably Jim McClellan, who got his Ph.D. at about the same time as me. McClellan was working at Rice and I had gotten mine at MIT, and when we were looking for the limited positions that were out there at the time, we were basically both looking for the same positions. When I had the offer from Flanagan’s group at Bell Labs, if I had turned the offer down, McClellan was the one that would have received the offer. But I took so long to turn it down, that he despaired and took a position at Lincoln. Consequently, when I finally did turn down the position, Flanagan’s group had a hole for another year because he had gone to Lincoln.

When I left MIT and went to Georgia Tech, MIT hired McClellan as an assistant professor and he took over my old office, my own phone number, and worked with my old secretary. He stayed there for a few years, and later went to Schlumberger. But at some point about 11 years ago, I finally called him and said that I thought he’d probably been at Schlumberger long enough and that it was time for him to come back to an academic position. And so this person I had been competing with all my life came to Georgia Tech. It was a productive collaboration because of the productive competition.

I had done some work with Mecklenbräuker on ideas that came out of his thesis, and we’d developed these efficient structures, but we had a few difficulties with the way that we constructed it. He [McClellan] corrected it and published some papers with one of his own students. A couple of the students that I left behind that I’d done Master’s theses with did their doctorates with him, so our careers have been tangled a little bit. We talked him into coming to Georgia Tech. So now we’re on the same side, and we’ve added some other people more recently. Vijay Madesetti and Doug Williams came seven or eight years ago. I guess, Tong Zhon came from Virginia more recently. So we have fewer MIT people now and have more academic diversity.

Curriculum, signal processing education

Nebeker:

I seem to recall reading about Georgia Tech’s innovation in signal processing education and introducing it at a very early stage in the curriculum. Is that right?

Mersereau:

Right. That’s been more an activity of Schafer and McClellan trying to begin a computer engineering side of our program. We give degrees in both electrical engineering and computer engineering. The goal was to get the computer engineers, who could relate to algorithms and doing things on computers, to think about signals at a very early stage. It worked fairly well, and now we’re doing it with all the engineers. That’s the first course that they see. Of course the goal is two-fold. First of all it’s our belief that students in this generation are oriented more toward audio-visual mediums. These aren’t kids that have built ham radios anymore.

These kids are decompressing images and displaying images and capturing waveforms and splicing things on their computers. So it was our belief that they related to these activities a little bit more. Once they had a feeling for the kinds of things that they could do, then we’d go back and fill in the mathematical details of how you do it carefully. It’s still somewhat controversial and still being worked out, but Georgia Tech has at least put in place a system of doing that course. Now we’re changing some of the other courses, like teaching circuits after people already know about signal processing.

Program structure; industrial collaborations

Nebeker:

While we’re on the topic of the Georgia Tech program, are there other things that you would like to comment on? Maybe the graduate education there, or other aspects of the undergraduate program? I mean you’ve had a lot of students.

Mersereau:

I had a number of Ph.D. students. Some of them are very, very good and have done very nicely, but other schools have had good students too.

Nebeker:

I just wonder if there’s anything that you feel distinguishes the Georgia Tech program.

Mersereau:

Well, they work with great people. I mean every program has its own personality, their own way that they look at things, and we probably have got a little bit of that. It’s probably still pretty highly correlated to a lot of the way that we were trained, which is to say it’s highly correlated with the MIT type of approach. I guess the one thing that might distinguish us a little bit is that the group has always functioned a little like a large ant colony, with all of these ants contributing toward the common goal.

The students do a lot of interacting among themselves. Unlike other groups at Georgia Tech, we always thought it very important to have a geographic center, and so the faculty tended to be located in one portion of the building. Whereas other groups were oriented more by office space. Since we have moved off campus into another building, which has some issues of its own, but there as well space has been a primary consideration. We continue to be fairly successful in terms of getting a lot of equipment, particularly with Tom Barnwell. So we’ve never really been starved for facilities to do things.

Nebeker:

What about connections with industry? Have there been many of those?

Mersereau:

There have been some. Georgia Tech is, one of its frustrations, located in Atlanta, which is a big city, but it’s a technically starved city in some respects. It aspires to be creating a Silicon Valley, and although it has made a fair amount of progress over the 20 years since we’ve been there, it still does not have a great deal of technical industry. Georgia Tech has spawned a few companies, some of which have done nicely, but nowhere near as many as it probably would be capable of. A lot of its people have spawned nice companies in Silicon Valley and in suburban Washington and such places. And so there are some connections with industry. Texas Instruments has been very good in collaborating and there has been some joint research with some folks at TI, in particular with Barnwell in developing a particular speech coder, the MELP coder. Schafer has some strong connections with Hewlett-Packard, but it’s Hewlett-Packard in Palo Alto, not in Atlanta. Therefore, in terms of industrial collaborations, we probably have fewer than most schools. We do have some, and we’re always looking for more, but that hasn’t been one of our big success stories.

Filter design, two-dimensional digital signal processing

Nebeker:

So when you moved to Georgia Tech there in, what was it, ‘75 ?

Mersereau:

‘75.

Nebeker:

Do you remember what your research projects were at the time?

Mersereau:

Gary Shaw, who was the first student that I was involved with, was still concerned with problems of two-dimensional filter design. He particularly was concerned with some recursive filter designs in which we resolved some issues. In the end we decided that recursive filters weren’t what people wanted to do. At about that time, Dan and I started working on turning that 1975 Proceedings paper into a textbook.

Nebeker:

Where was Dan at this time?

Mersereau:

I believe when we started working on the textbook he was at, still at BBN, but when we finished it, he was at Lincoln. I know this because there were some questions about who owned the copyright, and he wanted it known to Lincoln that he started on this project already. So we were expanding that original paper into a more complete exposition of two-dimensional DSP. That took forever to write. The book came out in late 1983.

Nebeker:

What do you mean when you say forever?

Mersereau:

We started writing it probably in ‘78 or ‘79, but that’s a short time by J.D. Salinger’s standards.

Nebeker:

I’ve heard very good things about that book.

Hexagonal sampling

Mersereau:

I think it was the first one to really put down a lot of the formal aspects. In the process of doing it, there was one sampling theorem for signals that were sampled on more general lattices. That was fairly well known and there was an important paper written about that in the early ‘60s. I was including that material in the text, when I was talking about the sampling theorem, because it was really pretty basic. But then I was trying to write a section in there that said you can prove it’s more efficient than rectangular sampling for isotropically band limited signals. But why weren't people using hexagonal sampling? That was difficult because I didn’t have a good answer. I knew I had to answer the question, but I didn’t know what the answer was.

Mersereau:

It was possible, I surmised, that people weren’t using it, because nobody had sat down and worked out the algorithms for dealing with those kinds of signals. Later, when I got home I asked myself, is this something that’s fundamental, or is it that people have just never thought of trying it. In about a week I discovered that almost anything you wanted to do could be generalized to work with these lattices very well. In many cases there were some engineering advantages to doing that. With that, a simple question that I posed to myself in the back of the book, I developed into three years worth of research.

Nebeker:

Did some of that get into the book itself?

Mersereau:

Some of it got into the book itself, and some got into a couple of student theses that came along after the book, as well as a few more papers in the proceedings.

Publication and research

Nebeker:

I’m always curious when people attempt one of these early reviews of an area of technology, whether that process changes their thinking much or raises questions, and here is one example. What do you think?

Mersereau:

It does. To me, the best way to come up with ideas usually is when I start writing something, perhaps it’s my style, but I like what I write to be fairly complete. I try to pose these kinds of questions, and in many cases they lead to things. There is a danger however when you do something like that. When you pose one of these questions and think you’ve got this answer, sometimes you get all excited about what you’re doing and include a little bit too much of your own stuff that hasn’t really stood the test of time. That probably happened in that original Proceedings paper, and it probably happened in a few places in the book. But hopefully the pluses outnumber the minuses.

Nebeker:

Did you continue to work with Dan after that book?

Mersereau:

No, not really. He and I are still good friends and we’re starting to think about doing a revision of the book. The period when we worked together was really back when we were graduate students 24 years ago. Apart from that, we’ve never really been together other than meeting each other at conferences and things like that, or sharing Christmas cards.

Iterative signal restoration algorithm; image restoration

Nebeker:

Looking here at some of the other topics that your papers have dealt with, what about iterative signal restoration algorithms?

Mersereau:

That was some work that Ron and I did. Although we worked together on a lot of things, we haven’t actually published very many papers together. Iterative signal restoration algorithms were an interesting little problem, and we passed it on to one of his students, Aggelos Katsaggelos. He started on that problem and has continued to work on it off and on along with other things since then.

The problem came to mind from a book edited by Tom Huang. I’m not even sure what the collection was about, but it was Springer-Verlag Topics in Applied Physics book. There was a great paper in there by Frieden, in any case. It talked about some of the iterative schemes, and we became intrigued by it. That led to Katsaggelos' thesis and in the middle of that Jan Biemond came to Georgia Tech for a 3-month stay. Maybe it was a 6-month stay. Anyway, he came for a stay, and got involved with Aggelos on that particular problem. The three of us were three chiefs banging away at one poor Indian. But in time he did become a very good Indian, and has done some very nice work since then. Biemond later did some work on the problem. He had a background in doing image restoration, more from a Kalman filtering point of view.

Nebeker:

Was that a new topic for you, image restoration?

Mersereau:

Yes, that was pretty much the first time that Ron and I had gotten into it. We had a problem brought by somebody in the physics department. They had some gamma ray spectra that they wanted cleaned up, and we got that posed as a deconvolution problem. It was while we were scrambling for answers to that that we ran into this chapter in the Tom Huang book.

Nebeker:

Well it’s another case then of an application-driven piece of work.

Mersereau:

That one was application driven. Stan Reeves, a later student of mine, worked on some other aspects of that particular problem. We made some nice contributions to the problem. Others did too. I mean, it was a big topic, but it’s come along very, very nicely. Been some nice work down there.

One-dimensional signal processing; multi-dimensional image processing

Nebeker:

While I’m thinking about it, maybe I could get you to comment at this point on how separate image processing is from one dimensional signal processing. I mean, has there been a great deal of transfer or sharing of results? Or, is it more a story of techniques peculiar to images and other two or three-dimensional signals?

Mersereau:

We shared a lot of stuff. In the area of multidimensional image processing as related to computer vision there’s probably less of that. But that’s also work that I haven’t been as involved in. For problems such as coding and compression, restoration, and modeling, most of the image processers make their living by stealing results from the one-dimensional case and adjusting them to fit their problem. Almost anything that seems to work well in the one-dimensional case works pretty well in the multi-dimensional case. There are a few cases, transform coding for example, which was used on one-dimensional signals, but not as successfully as an alternative, was used with more success on images. It’s the basis of JPEG compression. But subband coding, which was a one-dimensional idea based on the old channel vocoder for speech modeling, worked very nicely on images and spawned all kinds of activities there. Pretty much the mathematics that underlies everything works very well once you figure out how to generalize it.

Nebeker:

So the picture of signal processing as this core of mathematical results in Fourier analysis, mixed with course development in different directions is what?

Mersereau:

The concepts are pretty much independent of the dimensionality of the signal. The actual details of the techniques will get in there and start to vary. There are a few places where the mathematics breaks down. Recursive filtering for example is fairly difficult to do in multi-dimensions, because we can’t factor polynomials, and that turns out to mean that we can’t easily deal with stability or structures for building recursive filters. It turns out, however, that the image processing applications basically don’t want recursive filters anyway. They have nasty phase responses and the finite size of the image contradicts some of the assumptions of things. So those few places where the models haven’t worked very well or the techniques haven’t worked very well can be discarded and not worried about. The basic concepts are usually there, and as I say, there are people, myself among them, that make a living by taking some of these one-dimensional techniques and seeing whether they’re applicable.

Nebeker:

So even today after the field is fairly mature, with the work being done by people with one-dimensional signals you might look to see if that could be adapted or somehow extended to higher dimensions?

Mersereau:

Right. Now there’s a fair amount of the early signal processing that was oriented towards speech processing. Speech processing is done with a particular model for the way that speech signals are created, but images are not created with that as the rule. Therefore some of the techniques, such as linear prediction, that had an amazing effect on the speech processing community, had much less of an impact on images. That is because the model just doesn’t fit the production mechanism. That’s a general property. Things aren’t always equally applicable because of constraints of the problem. But the mathematics isn’t the limiting factor.

Nebeker:

Is it fair? Or, is this the core of the theory, and then there is specialization for say speech or array processing? Areas where there’s less relevance and less ties from one type of signal processing to another.

Mersereau:

Right. I mean, in terms of one-dimensional signal processing, speech processing is a very special case. Audio processing is probably a more general case, and audio processing is probably more at the level of doing image processing. The equivalent of speech processing might be something like fingerprint processing, something where you’re talking about a very special class of images. Speech is simply a more specialized application than general sound processing or audio processing. Image processing as a group is also a higher level.

Nebeker:

A large share of the general audio processing work is relevant you say to image processing?

Mersereau:

A great deal of it is. It’s might be because the eye and ear are both made out of similar kinds of neurons ultimately, but a lot of the models that work pretty well for audio perception aren’t that far away from things that work pretty well for video and the still image.

Image modeling

Nebeker:

Now another area I know that you’ve done some work in is image modeling. Can you comment on that?

Mersereau:

Some of this is the ambiguity found when you have a model for the speech signal and it is limited. In the speech case you can do so much with it. A lot of people have tried to figure out if we come up with similar kinds of models for images. They are decomposing them using wavelets for example, which works nicely and where a lot of people have done some very nice work recently. Earlier models tried generalizing linear predictive one-dimensional models. These can work on occasions, but they can be frustrating. A general set of models is a bit more elusive. Still, they do help you.

Modeling an image is a key component of the restoration problem. It’s a key component of the compression problem, because you if you have a good model for the signal then you can do compression on it.

Students and research

Nebeker:

To pick another of your papers here, this two-stage multirate coding of color images. Is that some area of work that was important to you?

Mersereau:

That was some work that I did in conjunction with Mark Smith. The goal there was to try to exploit vector quantization in ways that might be useful for compressing signals. As a coder it worked reasonably well at the time. What I mean is that it was slightly better than what we thought was available at the time. It had its day in the sun for about a month and a half, or two months until somebody came along with one that was a little bit better.

That is the way it is in the coding business in many cases. Staying the king of the mountain is usually a very transitory thing. There are theses which claim that they are building something that’s better than what is out there. This is the kind of thing that a student can base their work on. Ultimately there is a more important class of theses which make more fundamental contributions that maybe somebody else has to build on before they become part of a major system. This particular work was more of trying to see if we could combine some things. So it was a hierarchical a coder in that it coded a signal with a vector quantizer and looked at the error. Then it tried to represent that with a cruder vector quantizer, and tried to improve things further after that.

Nebeker:

What about the subject of wavelets? Is that something that you have found useful?

Mersereau:

I think wavelets are very useful, but I’m not one that can really speak on them. Mark Smith, at Georgia Tech, has always been more of our wavelet sub-band person. I guess to avoid treading on his area and knowing that if somebody really wanted to work on those he would be the right person to work with, I devoted very little time to this. Some of my students do occasionally touch on it.

Nebeker:

What have been your main areas of work in the last, in the ‘90s say?

Mersereau:

A recent student did some interesting work on very low bit rate video coder using some hierarchical structures that we actually got a patent on. He’s a Vietnamese boat person, eventually got a Ph.D. at Georgia Tech.

Nebeker:

Is this F.H. Lin?

Mersereau:

No, his name was Truong. But he did some very, very clever thinking. I don’t know whether in the long run the techniques will surface as survivors, but they certainly displayed some beautiful thinking along the way. I still have students trying to figure out whether we can squeeze some more out of some of that work. I’ve gone back to the hexagonally sampled signals. Although I haven’t done much in wavelets, we have a student that’s looking at hexagonally represented wavelets. Why not, I say, it looks like the eye does it that way. Maybe we should try to understand it. The main thing that I produce is students, and one of the main questions that I’m always asking with respect to students is: what is an interesting problem that keeps them far enough away from the other students so that they have free reign?

Nebeker:

I see a couple of the topics here are actually biomedicine again. Returning to your early years how much work did you do there?

Mersereau:

I have some that have looked at some problems in that area. A recent student of mine, Ram Rao, not included in that list because he is a little bit older, was looking at some multidimensional processing. He was looking at video signals and audio signals at the same time and dealing with their mutual information. He was concerned with whether you can use the audio track to improve the video quality. If you know the words that the person is saying, can you perhaps make their lips move in a more natural way for example. Or, given that, can you improve the performance of a speech recognizer if there’s a video camera looking at the person and you have video signals. Does lip reading basically help an audio speech recognizer?

Another question that he made some preliminary stabs at was can you even synthesize a video talking head just given the audio track. And we have some students that are going to pick up on some of those problems and deal with them. I think that’s an emerging area and an exciting one.

IEEE Societies and publications

Nebeker:

A different issue I wanted to ask you about is the IEEE Society structure. Talk about the range of publications and the Transactions that are out there. It seems that most of your publications have been in the Signal Processing Society Transactions, later Image Processing Transactions. But you also had plenty in Circuits and Systems I see, and then scattering of other. Does it seem that it’s a fairly rational division of the subject areas, or are there big overlaps that make it troubling to know where to look or where to send papers?

Mersereau:

There certainly were overlaps, but I guess I don’t find them troubling. In my own mind, maybe there are prejudices, but I have identified certain journals as having certain attributes, certain areas where I’ll go looking for them. Some of the early work, for example, that I did on multidimensional filter design seemed to be more of what I thought related to Circuits and Systems rather than the Signal Processing Transactions. This despite the fact that the Signal Processing Transactions was doing some of that at that time.

Nebeker:

But it seems that the system is working satisfactorily?

Mersereau:

No. There’s a downside that I keep hearing about and that I experienced myself. That is, I have far more to read than I ever have time to read.

Nebeker:

What about the critique that I’ve heard that with the proliferation of journals outside and within IEEE that there’s been a fall in quality? That a lot more stuff is getting published and that maybe the quality isn’t as high as it was say 20 years ago?

Mersereau:

There’s a lot more being published. I certainly have heard the same comment. To some extent it probably is true. I suspect, however, that if you took a few copies of the Transactions from the last couple of months and ask people which ones of these they would keep, and recorded their votes, five years from then the votes probably wouldn’t line up. I’m not sure we can always spot a very clever idea in a very badly written paper. And to some extent by planting a lot of trees and making a lot of forests, we’re making it harder to find the individual gems, but the alternative of being overly critical has its dangers too. But we do publish more than people can screen. I have the benefit of having graduate students who can point out particularly important papers and do some pre-screening for me. I don’t know exactly how we solve that problem of finding flowers among the weeds.

Nebeker:

Well, in some fields there is a clearer hierarchy of people who try to publish in the top journal and that works as a screening process for you. I don’t know that that’s the case here in signal processing. Now that it appears, I understand, that there will be a Transactions in Multimedia. That’s going to be one more outlet for papers that might have been in image processing in some cases, and in some cases elsewhere.

Mersereau:

There certainly are more papers by any measure, but the other tendency that seems to be there is that papers now are longer than the ones that were being written 20 years ago. If you go back, articles published, famous ones in many cases, often were only three and four pages long. Now somebody has a fear that if they try to publish a three page paper it will be put back in the letters section somewhere. I think we probably do ourselves a disservice by in many cases getting pretty verbose and trying to say things. We were better at doing it a generation or two ago. I’m guilty of writing long papers along with everybody else, but I think there’s also that tendency.

Nebeker:

What about the Signal Processing Society itself? In your association with it over the years, has it seemed to do the things it should be doing?

Mersereau:

Yes. The main thing that the society should probably be doing, and it managed to do quite well, was just to bring people together in the conferences. They do that very nicely. I’m not altogether sure what other goals the society realistically aspires to. I don’t know that it functions as much as lobbying, these activities tend to be done by other boards.

Nebeker:

Well, one critique I have heard is that there hasn’t been as much involvement of the industry in the society, and this affects the immediate transfer of knowledge. Do you agree with this?

Mersereau:

That’s true. For many of the other societies it is true as well, though not for all. The Power Engineering Society has more of a connection with its industry. I don’t think that’s been a conscious design on the part of the society. It may be more a matter of the way that things have evolved.

Nebeker:

Really I’m just fishing for any ideas you have about how the society might be better or might have done things better.

Mersereau:

No, I don’t particularly have any suggestions. I’m sure when we spot some things that work we’ll follow up on them.

Nebeker:

Is there anything I haven’t asked about that you care to comment on?

Mersereau:

I don’t know. We’ve gone on for quite awhile on all kinds of different things. More than I thought I would have had to say. No, I think we’ve done a pretty good job.

Nebeker:

Thank you for the interview.