Oral-History:Bertram Raphael

About Bertram Raphael

Bertram Raphael was born in 1936 in New York City and is known for his contributions to artificial intelligence. He received his bachelor's degree in physics from the Rensselaer Polytechnic Institute in 1957 and his Ph.D. in mathematics from MIT in 1964, where he was a student of Marvin Minsky. He directed the Artificial Intelligence Center at SRI International from 1970 to 1973. He invented the A* Search Algorithm, contributed to the development of Shakey the Robot, co-founded the Journal of Artificial Intelligence, and authored The Thinking Computer: Mind Inside Matter (1976).

The interview concentrates on the funding for the Shakey Artificial Intelligence Project through the NSF, the Department of Defense and DARPA. Raphael describes the motivation for creating Shakey as an integrated robotic system, the differences between SRI's approach and that of MIT and Johns Hopkins, and further projects involving man-machine interactions and groupware.

About the Interview

BERTRAM RAPHAEL: An Interview Conducted by Andrew Goldstein, IEEE History Center, July 25, 1991

Interview #113 for the IEEE History Center, The Institute of Electrical and Electronics Engineers, Inc. and Rutgers, The State University of New Jersey

Copyright Statement

This manuscript is being made available for research purposes only. All literary rights in the manuscript, including the right to publish, are reserved to the IEEE History Center. No part of the manuscript may be quoted for publication without the written permission of the Director of IEEE History Center.

Request for permission to quote for publication should be addressed to the IEEE History Center Oral History Program, Rutgers - the State University, 39 Union Street, New Brunswick, NJ 08901-8538 USA. It should include identification of the specific passages to be quoted, anticipated use of the passages, and identification of the user.

It is recommended that this oral history be cited as follows:

Bertram Raphael, an oral history conducted in 1991 by Andrew Goldstein, IEEE History Center, Rutgers University, New Brunswick, NJ, USA.

Interview

INTERVIEW: Dr. Bertram Raphael
INTERVIEWER: Andrew Goldstein
DATE: July 25, 1991
PLACE: Telephone Interview

NSF Support for Shakey

Goldstein:

I think I described to you the purpose of our project. In this large study of the National Science Foundation and their role in computer science, we want to write about some of the research work that the foundation has supported. So what I'd like to do in this conservation is talk about your work, what led you to it, the nature of the work itself, and what some of its consequences have been. Let me tell you that I was looking through your book, The Thinking Computer, and I saw the descriptions you wrote of what was going on at SRI (Stanford Research Institute). How involved were you in the Shakey Artificial Intelligence Project?

Raphael:

I was the Director of the Shakey Project and head of the Artificial Intelligence Lab for awhile.

Goldstein:

What dates?

Raphael:

It must have been about 1967 or 1968. I was Project Leader in 1970 and 1971. I was head of the AI Lab probably from 1972 to 1974. I was one of the three or four key people in the project from the beginning.

Goldstein:

From our examination of the NSF Award, I see that you are receiving money from the National Science Foundation between 1971 and 1975. Is that accurate?

Raphael:

Sounds right.

Goldstein:

What were those awards for?

Raphael:

Do you have the titles of those?

Goldstein:

Yes.

Raphael:

I do not remember. We were receiving money from NSF, the Office of Naval Research and the Defense Advanced Research Projects Agency, and they were all for different components of related projects, so I do not remember exactly which was which.

Goldstein:

I see that in 1971 the NSF grant was for automatic theorem proving. That was a two-year grant. And then in 1973 you received two separate grants. One was $60,000 over one year for equipment needs for artificial intelligence.

Another one was for machine perception that was $39,000 for two years.

Raphael:

Those are each different subjects. I can tell you a little about each of them.

Goldstein:

Okay, go ahead.

Raphael:

<flashmp3>113_-_bertram_-_clip_1.mp3</flashmp3>

The automatic theorem proving was work on essentially machine reasoning. I think that was probably the project that lead to, or supported the development of what was called QA3 and QA4. Demonstration "question answering" programs showing how classical Logic Predicate Calculus could be used by a computer automatically and applied to everyday reasoning, its problems, and some of the results. The approaches that were taken there I would think are parts of the roots of the "Prologue" programming language, and some of what is currently called logic programming. The equipment needs probably were supplementing the Shakey Project.

Shakey was a robot controlled by a computer in the days before microcomputers. And it was operated by a combination of minis. It was a Digital Equipment Corp. PDP-10. We did most of the computing on the PDP-10 in the back room, and there was a PDP-11 or another front-end computer, and there were various devices to get sensory data into those machines. I believe that NSF provided support for some of the hardware glue, the custom interfaces, and ad hoc ways we had to tie together the different machines to get them talking to each other and to the sensors and motors.

One of the main areas of our work was how to get a computer to see. Enabling the robot to understand its environment through a visual sensor was one area of focus. Another was looking at aerial photographs, or interpreting scenes of real world pictures taken through the window. We developed a number of algorithms. We started with work from the MIT Lincoln Labs, Larry Roberts early work on vision, and looked at analyzing areas and regions rather than just lines and edges. The key issue that ran through the whole project was how do you relate formal computer analysis to the semantics of the real everyday world? Our machine perception work continued and evolved over many years at SRI and at Stanford and spin-offs. Some of the people in the group went to Schlumberger and had a major project there in their analysis of oil-well data and other kinds of real world information, like pattern-recognition programs.

Goldstein:

Who were some of the key collaborators?

Raphael:

Peter Hart and Marty Tenenbaum (who recently is the co-founder of a new company in Palo Alto, California). Tenenbaum was head of machine intelligence and perception work at Schlumberger Labs for many years. Hart was a founder of SYNtelligence. Recently, he moved on to Ricoh.

Goldstein:

You said that the NSF was supplying funds for the hardware glue. Were the grants that you requested that specific, or was there general support for this project? Would you turn to the NSF for some particular need and then to the ONR for some other need?

Raphael:

At that time you had to be fairly specific as to what piece of the project was going to result from the particular piece of funding. Nobody wanted to put a little bit of money into a big pot and not have a recognizable result of what was done with their money.

Goldstein:

So your budgets had to be specific?

Raphael:

Yes.

Goldstein:

How did you come to turn to the NSF for the mechanical glue for this hardware? Why did you turn to the NSF?

Raphael:

Now I am speculating. I really do not remember the particular grants. But the NSF was the place to go when you could not get funded anyplace else. The Defense Department generally wanted the proposal tied to some specific outcome or demonstration or result. The NSF was willing to provide funds for equipment needs and such to support the experimental environment to facilitate getting results in other areas. I expect it was easier to get money from the Dept. of Defense for developing a camera or a pattern-recognition algorithm for understanding a certain type of picture, as for communications interface hardware. That was a prerequisite for the actual applications.

Motivations for Shakey Project

Goldstein:

How did the Shakey Project get started? Did you find yourself at SRI with Hart and just begin to consider things you could do? Or was it an outside initiative?

Raphael:

That is a good question. That is going way back now. I was at SRI as a consultant in 1964 and then full time in 1965, after I received my Ph.D. in Artificial Intelligence, List Processing, and Problem Solving using high level programming approaches at the Massachusetts Institute of Technology in 1964. I taught a course in the "LISP" programming language and symbol manipulation in a short summer course at the University of California, Los Angeles in 1964, and a couple of the key people from SRI were there, including Charlie Rosen, who was head of the Learning Machine Group at SRI, at that point in pattern recognition. He was intrigued by the possibility of combining some of this higher level problem solving software with the more basic pattern recognition analysis of lines and dots on the screen. And he had already been thinking about robots; first as the possible application for pattern recognition in factories in the real world.

It was at that time a very futuristic use of computers. A robot operating in the real world has to do more than react to the sensory data. It has to make some decisions, and LISP and similar programming techniques could be the basis of that kind of capability. So Rosen hired me and I joined the group at SRI, and we thought up how to combine the work the SRI group was already doing in pattern recognition and pattern classification. They were sponsored by the Army Signal Corps at that point to do photo analysis, with their interest in how to extend that to machines and possible non-military applications, and in more basic Artificial Intelligence work. And SRI had another lab that we could work closely with that had a lot of mechanical expertise and mechanical engineering people that could actually build the machine to experiment with. Charlie and I and Dick Duda were in the group at that point, and John Munson, and Nils Nilsson, who is currently chairman of the Computer Science Department at Stanford. Nilsson is one of the key people in the group.

We got together and dreamed up the idea of a complete robot project. As I mention in my book there was some early work going on in Robotics at Johns Hopkins and a couple of other places. But Stanford and MIT and the main research labs were trying to do high quality vision or high quality manipulation in mechanical arm control. And we thought that, if you have all of the components to build on each other and to coordinate to work toward a more general intelligence, then maybe you do not have to be as good at seeing or as good at sensing touch or controlling motion.

Goldstein:

Were there industrial applications in mind? When you say that for industrial applications, the robot would have to do more than simple pattern recognition. Depending on the application, that may or may not be true. But it seems like Shakey was given capabilities that were not necessarily essential for an industrial function.

Raphael:

My guess is that Rosen had industrial applications in the back of his mind from the beginning, but what we really talked about was more basic research. Whether it was military or industrial applications, the issue was how do you integrate visual or physical sensing with manipulation affecting the environment and integrate all that with automated decision making. How big of a machine do you need? How complex is the program going to be?

Goldstein:

How simple its instructions could be, whether they would be in English and how vague they could be?

Raphael:

How are people going to communicate with the system? We thought the time was ripe. One interesting thing that happened about then was the SRI group working in pattern recognition and picture analysis had spent three or four years developing a machine with specialized hardware to do the picture processing. And just about the time I was joining the group, or within a year thereafter, they were able to simulate that whole machine on the newest Digital Equipment Corp. computer that they had gotten. And the simulation was not only much easier to change and experiment with, but it actually ran a little faster and was a lot more reliable than the hardware that they had been building. That is what triggered the thought that, "Gee, we do not have to build special-purpose machines anymore. The computers have reached the point where we can do the algorithms to operate all the sensors and integrate everything."

Goldstein:

When did you say that was developed? Because I can remember you writing in your book that a computer simulation of robots often overlooks certain physical factors, and indeed you do need to build robots.

Raphael:

Right. You need to do both. We need the real interaction to find out what are the errors and what kind of information can you get from the real world data. You need the machine to be fast enough and easy enough to program to use, so it can do simulations in the sense that the machine needs to do "what if" experiments in order to decide which way to roll the wheels or which way to reach out its arm. The machine should be thinking ahead, which is a simulation item. Anyway, about 1967 or 1968, we felt the machines had reached the point where they could do the integration and the simulations as well as control the hardware.

The other proposal went to the Defense Department, who had the most money at that point. And this was a big project; we were looking for about $300,000 to $500,000 a year to get started in order to put some funding into each of all of the different components that were needed. We needed to pull together pattern recognition and high level programmers and communications people and mechanical engineers. I went to Washington with Rosen, talked to a number of people and to Ruth Davis, who was then an undersecretary assistant in the Defense Department. She went back and reached into a file cabinet and said yes, she has two or three hundred thousand dollars eligible for this kind of work and would like to get it started. And that was one of the first projects funded by what became the Defense Advanced Research Projects Agency.

DOD and DARPA Support for Shakey

Goldstein:

So the Defense Department and then later DARPA was the principle funder?

Raphael:

Yes. The Office of Naval Research funded a lot of the vision, the pattern recognition side of it. We went to NSF for the longer range, more theoretical kinds of things, like the theorem proving and reasoning part. As you mentioned a couple of years later the NSF had a special program of hardware funding, for supporting the equipment needed as a basis for a variety of different computer research projects.

Goldstein:

Their facilities program was quite active.

Raphael:

That is probably what this sixty thousand-dollar equipment needs project tried to tap into.

Goldstein:

Can you recall the sort of equipment you were after from NSF?

Raphael:

No, I cannot.

Goldstein:

Did you discuss the potential military applications of this work with the Department of Defense and the ONR?

Raphael:

They were not requiring that. We were amused some years later when we had Shakey rolling around the laboratory and saw a demonstration, and there were some military people that came through, and a General asked, "Can you mount a 36 inch bayonet on it?" The people we were working with in the funding offices, like Ruth Davis and later Larry Roberts and Marv Dennicoff at the ONR, who is now with "Thinking Machines," were quite long-sighted. They were interested in building the basic computer science capability for the country, rather than trying to make any particular weapons.

Goldstein:

Did you have discussions that were as detailed with the NSF when you applied for NSF money as you did with DOD? For instance, you say you went over to the Department of Defense and talked with them about what you wanted to do. Was it similar with NSF?

Raphael:

Yes. The standard procedure for the SRI people looking for grants, and I think it was the most effective way of getting government grants, was to spend a week in Washington and visit the contract monitors and try to talk to the people who allocate the budgets. Once Congress has put money into some budget, there are a couple of layers of bureaucrats that make the decisions as to who gets how much. And we try to get to know the people making those decisions and convince them that we were better qualified or likely to produce more interesting results with the money than the competitors for that money.

Work Contrasted with MIT & Johns Hopkins'

Goldstein:

In what essential ways did the work of your group differ from the group at MIT or Johns Hopkins? Did you have any approaches that were unique to your effort?

Raphael:

Number one was the idea of going for overall integrated performance rather than highly tuned specialized performance. Vision projects, and what was called the robotic projects that evolved at both Stanford and MIT, were really aimed at high performance precision control of mechanisms. I think we were paying more attention to our stated goal. Suppose you do not have that good of a mechanism, you do not have that good of a camera, you do not have that big of computer, but you really want them to all to work together. What kind of underlying logical structure is needed?

Goldstein:

As a result, was the MIT work or Stanford work more focused on hardware and mechanical design, mechanical engineering? Or was it the computer control of those?

Raphael:

Although there were other projects in those places that were specific software projects also, but in each of the universities — A university AI lab typically consists of many separate master's and Ph.D. theses, each of which is a one-person effort, and we had ten or fifteen people who could collaborate with the constraint that what they do has to fit together with what everyone else does. They are a team effort coordinating the different components.

Automatic Theorem Proving and Prolog

Goldstein:

You say that your work in automatic theorem proving was a precursor to Prolog. What's the genealogy there?

Raphael:

It would follow the work. We had a couple of young new computer scientists that he had hired right out of Carnegie and Stanford. One is named Cordell Green, who developed the QA 3 system, and subsequently went on to found a company called The Kestrel Institute. He is probably today doing some of the leading work in the country on how to get computers to prove mathematical theorems. Bob Yates was a mathematician working with us. Green and Yates and I attended a lot of the early meetings that involved the American and the British Artificial Intelligence communities on logic programming, or theorem proving by computer. A lot of exchange of ideas. I think you would find the papers on QA 3 and QA 4 cited by Kowalsky, one of the originators of Prolog, and others in that area.

End of Shakey

Goldstein:

In the mid 1970s you (and tell me if I understand this correctly), after a successful computer simulation of the sort of systems you were actually building, you decided to abandon the robots.

Raphael:

The DARPA decided. There was a very trying period in which we were directed by the DARPA to get much more applied. And if we wanted that level of DARPA funding we had to really work on weapons systems or military equipment maintenance, something more specific and highly targeted than the early work. As I said in the early days, the DARPA seemed to see its charter as building the computer science competence of the country, and by 1971 or 1972 they were shifting; they were being directed by Congress, I guess, to show specific results with the military applications.

Goldstein:

What did you begin working on? Did you preserve the level of the DARPA funding?

Raphael:

No, I think it dropped some and it shifted probably into the vision projects, and the machine perception rather than robotics area. And we went full bore into industrial robotics. We started a project jointly funded by five to ten companies interested in assembly line robots. And that became largely a project of the commercial side of SRI with a little bit of DOD support for some components of it. I think the ONR continued to contribute to the machine vision part of it.

Goldstein:

Did that work benefit from the knowledge you accumulated working on Shakey, or was that a different line?

Raphael:

No, it was very much a direct descendent.

Goldstein:

What companies were you involved with?

Raphael:

Unimation Inc. was involved. At one point I believe Ford was a sponsor. I do not remember. I think that was about the time I was moving out of that area.

Goldstein:

What happens? Do they do research on a system, develop it, and then if it is successful, it is purchased by an industrial client?

Raphael:

They do a little more, but basically they do research on algorithms and sensory equipment and the conclusion of the research is usually a paper, maybe some hardware designs, that then belongs jointly or goes back to all of the sponsors of that research.

Goldstein:

So it would be unlikely to find, say, two auto manufacturers sponsoring the same research, because then there would be disputes about the ownership of the results?

Raphael:

Not necessarily. I think there have been a lot of changes on how technical work is protected. At least originally the idea was that they were sponsoring work at a university, except that the results would not be published openly. It was to be given equally to all of the sponsors and they would each use it in different ways.

Goldstein:

And you say you got out of that. What did you take up?

Man-Machine Systems and Groupware

Raphael:

I moved into a management job at SRI involved with man-machine systems. There were some problems in a laboratory that had previously been run by Doug Engelbart, the inventor of the mouse, doing experiments in man-computer, human-computer interaction group, what is now being called groupware: how you get a group of people to collaborate with a common database in a computer and share information and ideas. It was a project that had been going on parallel to the Shakey Artificial Intelligence Project, but was not paying its way in SRI, meaning it was not meeting SRI's financial needs. It was not getting enough sponsorship to the level of investments. And I had a one to two year job to manage that group and figure out how to fix it from SRI's point of view. The result of that was we sold the laboratory to Timeshare Incorporated with all of their ideas. And that has gone on and has continued, as Timeshare is now owned by McDonnell-Douglas.

Goldstein:

So Timeshare was responsible for funding it?

Raphael:

They bought the rights to all of the past developments of that group and hired most of the people in it, including Engelbart. He ended up developing it and making it a commercial product that they sold to a number of companies.

Goldstein:

Did your work on Shakey and the automatic theorem proving inject a new perspective into the work of this group?

Raphael:

I do not think anything Shakey had to do was relevant. I think the approach, the effectiveness at getting sponsorship helped some. I thought some of those folks made some contacts for them with sponsoring agencies to help over the transition until they sold the project. My perception was that they had made some basic errors. They were trying to do both fundamental research and applications. They were getting funding from the ONR and then the NSF and also from industry, with the idea that these two kinds of work would complement each other and the theoretical work would be applied to the business needs, and the business people would put some real world constraints and goals in place for the theoretical work. And what actually had happened was that the two sides were competing, or hurting each other, and the fact that there was this theoretical government sponsored work going on and the bugs were not being fixed in the real practical system they were trying to build. They were not paying enough attention to the customers if they wanted to make it a commercial success. And they could not really build a commercial system and do basic research on the same platform at the same time. And so we worked on defining the project a little better, and splitting it up so that there was a real practical application system of groupware and human-computer interaction software could be sold.

Level of NSF Support

Goldstein:

You said before that DARPA the military sponsors of research got a little more pragmatic in the mid 1970s. They wanted more concrete results. Would you say the same of the NSF? How did their support of artificial intelligence as a research effort hold up during the 1970s?

Raphael:

To my knowledge it held up pretty well. It stayed pretty uniform. But it always had been a very small part of the whole research picture.

Goldstein:

Compared to the military?

Raphael:

Yes.

Goldstein:

Did they rise up to fill in the gap left by contracting military funding then?

Raphael:

No.

Goldstein:

Just a steady level.

Raphael:

Yes, with a little bit of an increase in the more applied sides. They did some funding for applications of computer science separate from development of the basic science.

Goldstein:

Can you think of any really strong work in the 1970s that they were involved in supporting, whether it was your own or someone else's?

Raphael:

Not really.

Teamwork and Consensus at SRI

Goldstein:

Did you have any comments you want to make? Some highlights of your research career that you would like to elaborate on?

Raphael:

I think the key thing was the teamwork. The thing that was great about the SRI environment, was that it was not a matter of an individual researcher, and maybe his or her couple of students getting support and then doing something on their own, which is the way that the university research community is generally structured. Rather, there was a team of quite a number of quite excellent computer scientists that work together on bigger projects. And the collaboration with Nilsson, Duda, Hart and Tenenbaum.

Goldstein:

Were you as the director responsible for getting the grants from the NSF, say, and for that reason you were the principle investigator? Or did each of the people involved in the project seek funding independently?

Raphael:

Everybody did. Not independently, but we would coordinate it, and we would have meetings and decide by consensus how much effort we wanted to do in each technical area, and who was the best qualified person to lead that. Each of the senior scientists in the group would head proposal efforts and promotions. And it was almost a joke that I was the director of the group. Rosen was the head of the lab before me. And in probably my third year there as a young researcher, he called me in for annual review and asked me if I ever thought about managing and I said, "Sure, some day." And a couple of days later he called me back and said, "It's all set, we're trading jobs!" Really, the members of the group were all so strong both technically and with respect to each other that it was run by consensus, and the manager was just the person whose name went on some of the papers for coordination of SRI's administrative needs. But in terms of really directing the research, it was pretty even collaboration.

Goldstein:

And the support that you received from the NSF may not be limited to the amount that you as a PI (Principal Investigator) were getting? Is that right? I mean, someone else in the group may have been funded by the NSF for joint work?

Raphael:

Sure. There were times when we would have had two or three NSF grants. It might have been that I was PI or Cordell Green was PI on a theorem proving grant, and Marty Tenenbaum was on some aspect of machine perception, and Rosen or a mechanical engineer might have been on equipment foundation.

Goldstein:

Goodbye.

Raphael:

Bye.