Dr. William J. Dally
Chief Scientist and Sr. Vice President, NVIDIA Corporation
For contributions to Graphics Processing Unit (GPU) computing, as a Professor of Computer Science at Stanford and Chief Scientist of Nvidia Corporation, which led to the world’s fastest, most energy efficient supercomputers and enabled the current revolution in Artificial Intelligence (AI).
Dr. Bill Dally is Chief Scientist and Senior Vice President of Research at NVIDIA Corporation and an Adjunct Professor and former chair of Computer Science at Stanford University. Bill is currently working on developing hardware and software to accelerate demanding applications including machine learning, bioinformatics, and logical inference. He has a history of designing innovative and efficient experimental computing systems. He developed much of the technology used in high-performance computer networks – from the routing and flow control protocols down to the signaling and equalization methods used on the wire. His development of stream processing at Stanford led to GPU computing. GPUs now power the world’s fastest supercomputers and enable the current revolution in AI. He co-founded STAC, AVICI Systems, Velio Communications, and Stream Processors. While at Bell Labs Bill contributed to the BELLMAC32 microprocessor and designed the MARS hardware accelerator. At Caltech he designed the MOSSIM Simulation Engine and the Torus Routing Chip which pioneered network routing and flow control technology. At the Massachusetts Institute of Technology his group built the J-Machine and the M-Machine, experimental parallel computer systems that were the first to demonstrate many communication and synchronization methods. At Stanford University his group developed the Imagine processor, which introduced the concepts of stream processing and partitioned register organizations, the Merrimac streaming supercomputer, which led to GPU computing, the ELM low-power processor, the Darwin bioinformatics accelerator, and the Satin SAT solving accelerator. Bill is a Member of the National Academy of Engineering, a Fellow of the IEEE, a Fellow of the ACM, and a Fellow of the American Academy of Arts and Sciences. He has received the ACM Eckert-Mauchly Award, the IEEE Seymour Cray Award, the ACM Maurice Wilkes award, the IEEE-CS Charles Babbage Award, and the IPSJ FUNAI Achievement Award. He is a member of President Biden’s Council of Advisers on Science and Technology (PCAST). He currently leads projects on computer architecture, network architecture, circuit design, and programming systems. He has published over 250 papers in these areas, holds over 160 issued patents, and is an author of the textbooks: Digital Design: A Systems Approach, Digital Systems Engineering, and Principles and Practices of Interconnection Networks.
Hall of Fame Induction & Tribute Video
Hall of Fame Tribute Video
Dr. Bill Dally, Chief Scientist and Senior VP of Research at NVIDIA, was honored and inducted into the Silicon Valley Engineering Hall of Fame. The tribute video showcased the immense respect and admiration for Dr. Dally from various individuals, friends and colleagues, who have worked closely with him.
Jensen Huang, Founder and CEO of NVIDIA, expressed his delight in celebrating Dr. Dally’s induction, describing him as an extraordinary scientist, engineer, leader, and an amazing person deserving of this recognition.
David Luebke, Vice President of Graphics Research at NVIDIA, highlighted Dr. Dally’s exceptional intellect and ability to grasp complex concepts quickly. He admired how Dr. Dally would ask insightful questions that one wouldn’t expect until much later in a conversation.
Brian Kelleher, Senior Vice President of Hardware Engineering at NVIDIA, shared his initial impression of Dr. Dally, emphasizing his extensive technical interests and unparalleled depth of knowledge in the field.
Fei-Fei Li, Sequoia Professor of Computer Science & Co-Director of Stanford Institute for Human-Centered AI, Stanford University, acknowledged Dr. Dally’s transition from an academic scholar and world-class researcher to an industry leader. She commended his role in leading the AI revolution through advancements in both software and hardware.
Dave Patterson, Distinguished Engineer at Google & Pardee Professor of Computer Science, Emeritus, UC Berkeley, praised Dr. Dally’s boldness and willingness to take risks in designing computers. He noted that this fearlessness extended to his personal life, such as his passion for flying planes, even though he had experienced a plane crash.
Chris Malachowsky, co-founder and NVIDIA Fellow, highlighted the broad and profound influence of Dr. Dally, describing him as a prolific inventor. He shared an example of Dr. Dally’s remarkable dedication by mentioning how he designed a new type of respirator during the COVID-19 pandemic, showcasing his technical expertise and humanitarian efforts.
John Hennessy, Professor of Computer Science and Electrical Engineering at Stanford University, commended Dr. Dally’s work at NVIDIA, which complemented his contributions in academia. He acknowledged Dr. Dally’s leadership in developing GPU-focused machines and turning them into powerful general-purpose computers, aligning with the future of domain-specific computing.
Jensen Huang reiterated Dr. Dally’s significant contributions as NVIDIA’s Chief Scientist, particularly in the field of accelerated computing. He emphasized Dr. Dally’s expertise across various domains, including computer graphics, climate science, quantum computing, robotics, and AI. Jensen recognized Dr. Dally’s leadership in building NVIDIA’s world-class industry lab, where groundbreaking work is conducted by a team of 300 researchers worldwide.
Sophia Shao, Assistant Professor at UC Berkeley, expressed her appreciation for Dr. Dally’s encouragement of exploring innovative ideas and fostering inclusivity within the team. She praised his dedication to ensuring all team members, especially junior members, had their voices heard and were given opportunities to contribute to different aspects of projects.
Overall, the tribute video celebrated Dr. Bill Dally’s exceptional career, technical prowess, leadership, and humanitarian efforts. The quotes from various individuals highlighted his brilliance, boldness, and dedication to advancing the fields of computer science, AI, and accelerated computing. Dr. Dally’s induction into the Silicon Valley Engineering Hall of Fame was well-deserved and recognized as a significant achievement in his illustrious career.
Hall of Fame Acceptance Speech
Dr. Bill Dally
Chief Scientist and Sr. Vice President, NVIDIA Corporation
Dr. Bill Dally, Chief Scientist and Senior VP of Research at NVIDIA, delivered his acceptance speech. He expressed his gratitude for being inducted into the Silicon Valley Engineering Council Hall of Fame and acknowledged the esteemed individuals who have been inducted before him, such as Bill Hewlett and David Packard.
Dr. Dally emphasized that engineering is a team sport and credited several people who have played significant roles in his career. He thanked the leadership of the Silicon Valley Engineering Council, particularly SVEC President, Sharif Zadeh and Hall of Fame Committee Chairman, Dr. Fred Barez, for recognizing his contributions and organizing the Hall of Fame. He also expressed his appreciation for his parents, who instilled in him a love for science, engineering, and learning, as well as his family, including his wife Sharon and daughters Katie, Jenny, and Eliza, for their support throughout the years.
The development of stream processing and its evolution into GPU Computing were highlighted. Dr. Dally acknowledged Pat Hanrahan, his colleague at Stanford on the stream streaming supercomputer project, and the students who made pivotal contributions to the project. He emphasized the importance of teaching and mentoring, having supervised 56 PhD students and expressed gratitude to those who were present at his group’s alumni table. Dr. Dally recognized the contributions of individuals like John Nichols, John Danskin, Ian Buck, David Kirk, and others, who played key roles in translating stream processing into GPU Computing. He credited Jensen Huang, the CEO of NVIDIA, for his vision and investment in GPU Computing, which ultimately revolutionized the field and enabled modern deep learning.
Dr. Dally emphasized the significance of GPU Computing in the current revolution, enabling advancements in various fields. He noted that the algorithms and techniques used in deep neural networks have been around for decades, but the spark that ignited the revolution was having computers with sufficient power to train powerful networks on large datasets within a reasonable timeframe. He acknowledged the role of NVIDIA GPUs in fueling this revolution and mentioned the software developments, such as CUDA, that have made it easier for developers to run deep learning on NVIDIA GPUs.
The speech also addressed the focus on efficiency in GPU Computing. Dr. Dally mentioned the end of Moore’s Law and the need for real engineering to improve machine efficiency. He discussed the optimizations and innovations that have doubled the efficiency of GPUs in deep learning every year since 2012. He mentioned a prototype chip described at a previous symposium, which achieved a hundred teraflops per watt efficiency, a significant advancement compared to their latest GPU. Dr. Dally expressed confidence in achieving further advancements in efficiency and stated that there is much more to come.
Looking ahead, Dr. Dally expressed excitement about the application of stream processing principles and specialization to various computations beyond deep learning. He mentioned the addition of an accelerator for dynamic programming in the Hopper generation GPU, which is useful for accelerating bioinformatics computations. He envisioned the acceleration of other demanding computations, such as continuous and discrete optimization, through specialized hardware and accelerated computing.
In conclusion, Dr. Dally expressed his honor of being inducted into the Silicon Valley Engineering Hall of Fame. He emphasized that the recognized work was a result of a large team effort, involving faculty and students in stream processing research at Stanford and a dedicated team at NVIDIA. He highlighted the exciting time to be a computer engineer and expressed optimism about the future impact of accelerated computing in addressing demanding applications.
Acceptance Speech Transcript:
I am honored to be inducted into the SVEC Hall of Fame and to be in the company of such luminaries as Bill Hewlett and David Packard
I would like to thank many people who helped me on my journey to this point. I thank Sharif, Fred, and the leadership of the SVEC for recognizing me, and for administering the HOF.
My parents instilled a love of science, engineering and learning in me and taught me the value of hard work. My loving wife, Sharon, and daughters Katie, Liza, and Jenny, who are here tonight, supported me and were very tolerant of long hours spent on engineering projects.
The development of stream processing and its evolution to GPU computing was a large team effort. Pat Hanrahan was my partner on the streaming supercomputer project. Many of the students on the stream processor project made pivotal contributions.
Teaching, both in the classroom and by mentoring Ph.D. students has been the most important and most rewarding contribution of my career. Since I started as an assistant professor at MIT I have supervised 56 Ph.D. students. Several of them are here tonight. I thank them for coming and also for working with me during their time as a student. They have contributed energy, ideas, and fresh viewpoints to all of my research – on networks, circuits, and parallel computing – as well as the work on stream processing.
The translation of stream processing to GPU computing involved a large team of people. My longtime friend and ski buddy, John Nickolls invited me to give a seminar on stream processing at NVIDIA in 2002-3 and then had John Danskin hire me as a consultant to work on the architecture of NV50 (aka G80). Ian Buck, a student from our Stanford project, was hired in 2004 and used the experience gained at Stanford to develop the CUDA language. David Kirk and Wen-Mei Hwu taught “teachers” courses to spread the gospel of CUDA.
Jensen Huang, who is here tonight, had the vision to invest in GPU computing and to stick with it for many years before it became mainstream. The cost of supporting “CUDA everywhere” was not cheap, but ultimately changed the face of computing and enabled modern deep learning. GPU computing was the spark that ignited the current revolution in deep learning. The algorithms have been around since the 1980s. Large labeled data sets were available from the early 2000s. GPUs provided the missing ingredient of enough computing power to train a large model on a large dataset in a reasonable amount of time.
My pivot from scientific computing to deep learning started with a series of conversations I had with my Stanford Colleague Andrew Ng in 2010. Later – in 2011 – I recruited Bryan Catenzaro – then a programming system researcher – to reproduce Andrew’s “cat finding” result on GPUs – using 48 GPUs instead of 16,000 CPUs. The software that came out of that effort became CuDNN and laid the foundation of the software stack that makes NVIDIA GPUs the deep learning engine of choice.
Since GPUs became the engines of deep learning, our focus has been on efficiency. With Moore’s Law dead, we can no longer count on semiconductor technology to give us regular improvements in efficiency. It has taken optimized number representations, exploitation of sparsity, and specialized instructions to double GPU efficiency on inference every year since 2012, giving a 1000x increase over the past decade. We are working hard to come up with innovations that will continue this scaling. Some of these ideas were prototyped in a chip that we described at the VLSI Circuits Symposium last year that achieved 100 TOPS/W efficiency, about 10x a Hopper GPU. More is yet to come.
Going forward many of the principles of stream processing that led to GPU computing and the specialization that has fueled the growth in efficiency in AI can be applied to other computations. We added an accelerator for dynamic programming to Hopper to accelerate bioinformatics computations. In the future, we envision accelerators for other demanding computations such as continuous and discrete optimization.
To conclude, I am honored to be inducted into the SVEC hall of fame. The work for which I am being recognized was the result of a large team effort – many faculty and students participated in the stream processing research at Stanford and a large team at NVIDIA was involved in translating this research into GPU computing. It is a very exciting time to be a computer engineer. The future is bright with many more demanding applications waiting to be accelerated using the principles of stream processing.
