Toyota’s journey from Waterfall to Lean software development
Guess what. Toyota uses the waterfall method for software development – and now they’re trying to figure out how to go Lean.
Surprised? So was I!
My lean study tour to Japan in April 2009
One of the core tenets of Lean Thinking is Genchi Genbutsu – go and see for yourself. After years of experimenting with how to apply Lean principles in software development I decided to apply Genchi Genbutsu and go the source – visit Toyota and find out how they do it.
Despite all the books and articles written on Toyota Production System and Lean Thinking, very little has been published about their product development process and just about nothing about how they do software development.
So In april 2009 I was part of a small group that went on a “lean study tour” to Japan to visit Toyota and other companies in the Lean & Agile space. The group included several colleagues from Crisp, Tom and Mary Poppendieck, consultants from Best Brains in Denmark, and other mostly Scandinavian lean enthusiasts. I grew up in Japan so it was extra fun to visit the country again, this time wearing “lean glasses”.
We learned many interesting things, for example we met Katayama-san (chief engineer of several luxury and sports car models) and learned about Toyota’s product development process. But in this article I’m going to focus on the biggest surprise – how Toyota develops software.
Meet Satoshi Ishii, manager of automative sofware engineering dept.
We had the great honour of meeting Satoshi Ishii, manager of the embedded software division – i.e. the software that goes into the cars. His English was rather halting and I didn’t take detailed notes, so some of the quotes and conversations below are paraphrased.
First surprise was when he opened up by saying “I think you know more about Lean software development than we do”, and after that it just got more and more interesting.
All in all it I was impressed by Ishii-san and his presentation. He started with Toyota’s goal and vision (“we aim at the realization of sustainable mobility society”), then went into how software development plays into this, then described the problems they are having today and their strategy for solving them. We realized afterward that his presentation (and in fact all presentations we saw at Toyota) pretty followed the A3 problem solving format, confirming that this approach is deeply engrained in the Toyota culture.
Why Toyota needs to become an IT company
Toyota builds cars (duh). In the past that didn’t involve much software, and the little software that was needed was mostly developed by suppliers and embedded in isolated components. Toyota assembled them and didn’t much care about the software inside. But “The importance of automatic electronic control system has been increasing dramatically year by year” said Ishii-san.
A modern car is pretty much a computer on wheels! In a hybrid car about half of the development cost is software, it contains millions of lines of code as all the different subsystems have to integrate with each other. He mentioned that a Lexus contains 14 million lines of code, comparable to banking and airplane software systems.
Ishi-san concluded that “Therefore Toyota needs to become an IT company”.
Toyota’s current development process: Waterfall
Imagine our surprise when he pulled up this picture:
It was all there. The big V. Architects that don’t code. Distinct phases – all requirements to be complete and signed off before before coding, all code complete before testing, etc. He even called it a waterfall method.
We asked if he was happy with this way of developing. The answer was No. He told us about all kinds of problems they are having, most of them familiar to anyone that has experienced waterfall-style software development. There are countless case studies illustrating why the waterfall process just isn’t appropriate for software development. Even the original paper on the waterfall model in 1970 (”Managing the development of large software projects” by Dr Winston Royce) says “the implementation described above is risky and invites failure”.
Toyota is moving towards lean and agile software development
We asked Ishii-san if he had considered Agile software development. He was aware of Agile and liked the ideas, and said they will probably move in that direction. But they will do it in the Toyota Way – patiently and methodically, as Agile is not a goal in itself. I couldn’t agree stronger.
He said that “we are trying to learn how to apply TPS (= what we call Lean in the west) to software development”. Imagine the look on our faces. We came there to learn from what we thought would be the holy grail of lean software development, most of us were expecting to be dazzled and impressed.
He showed us this version of the Toyota house:
He said that they are currently at the bottom of the “house” (see the text highlighted in red, with numbers). Management, Process, and Making People. Without that, you can forget the other more technical aspects of lean such as Flow and Waste elimination.
He talked a lot about transparency – “How can you manage something that is invisible? How can we visualize software development? The key to success is mutual understanding between managers and engineers.”
He also talked a lot about engineer motivation and skills – that software is a creative process and motivation is key. “If engineers feel their own skills are improving by ‘visualization’, they can get high motivation for their work. The most important thing in the project is not only to develop software but also to cultivate engineers.”
One of the big impediments is their current software architecture. He didn’t get into details, but mentioned that they need to make significant changes in their architecture to enable Lean and Agile software development. My belief is that it is the other way around – that Lean and Agile software development provides a way to implement the architectural change in an iterative & incremental way.
He emphasized the importance of testing & fixing defects early. A defect found in the production phase is about 50 times more expensive than if it is found during prototyping. If the defect is found after production it will be about 1,000 – 10,000 times more expensive! I’ve seen other studies showing similar numbers. He showed some data on this, visualized in the bar chart below:
Reality turned out to be worse, a lot worse! As Toyota’s current problems with the Prius braking systems is costing them over $2 000 000 000 (2 billion!) to fix because of all the recalls and lost sales. “Toyota announced that a glitch with the software program that controls the vehicle’s anti-lock braking system was to blame".
Standardization and metrics – is Toyota going too far?
Most of Toyota’s ideas about how to do Lean software development resonated well with me. My feeling was that they are on the right track.
One thing bothered me though – the extreme focus on detailed metrics. I agree with the value of visualization, standardization, and data-driven process improvement – but only if used at a high level. My feeling was that Toyota was going to far. They say engineer motivation is critical, but how motivating is it to work in an organization that plans and measures everything you do – every line of code, every hour, every defect, how many minutes it takes to do an estimate, etc? I saw very thick and detailed process manuals about this – not only at Toyota, but at other Japanese companies that claimed to be implementing Lean software development.
The thought “ugh… micromanagement” popped into my head several times during the tour. Maybe it’s a cultural thing.
I’ve spent years helping IT companies implement this stuff in practice. I’ve found that in many cases both motivation and predictability improves when my clients decrease the level of detail in their estimate and plans. Software development seems inherently unpredictable, so spending too much time and effort trying to make detailed estimates and plans is wasteful and in some cases even counterproductive. Having detailed manuals for everything you do stifles innovation and creativity. Many of my peers and colleagues in the Agile software development community share this experience.
So my feeling is that either the Agile community has something to learn from Toyota, or Toyota has something to learn from the Agile community. Probably both.
Closing thoughts
It is deeply engrained in Toyota’s culture to be dissatisifed with the status quo. So my feeling is that even if they had a really good software development process, Ishii-san would have said that they were dissatisfied with it and wanted to improve it.
Despite all the problems Ishii-san mentioned, my first thought was that their process can’t be all bad since they are a successful company with millions of cars all around the world using their software. The Prius model was developed in record-short time despite the extreme level of innovation in that project.
I hadn’t (at that time) heard anything about quality problems with Toyota’s software. So waterfall or not, they still seemed to know what they were doing.
In fact, my conclusion after the trip was “well, now I know that there’s at least one company in the world that can succeed with the waterfall model” and I decided to stop bashing the waterfall model as hard as I usually do.
Now, however, with all the problems Toyota are having, I’m starting to reconsider. Are these problems software related? Could they have been avoided if Toyota used Lean & Agile software development instead of the waterfall model? I can’t sure, but I suspect yes.
Acknowledgements
Many people helped make this trip (and article) possible. I’m grateful to you all, and I’d especially like to thank:
- Satoshi Ishii-san and Toyota for finally telling the world how they do software development
- Mary and Tom Poppendieck for facilitating a Hansei (reflection) after each day on the tour. This helped us sort out our thoughts.
- Kenji Hiranabe for enabling the visit to Toyota.
- Bent Jensen and Best Brains for organizing the first Roots of Lean Study tour. They are doing it again, click here if you want to join!
The lean study tour was extremely inspiring and well-organized! I’ve lived in Japan for 16 years so I wasn’t sure I’d learn anything new – but going back there with ‘Lean glasses’ on I met so many interesting people and saw so many interesting things that I’ve realized my Lean journey has only just begun.
Hej Henrik, just wanted to thank you for an excellent write-up! I’ve never been to Japan myself, but would love to go there. See you! /Tobbe
Henrik,
Thank you for publishing this! It is excellently presented and extremely interesting.
I think it is great both that (a) Toyota was so willing to openly share their “warts and all” approach and aspirations with you, and (b) you wrote about it.
My father went on a similar study tour Japan in 1986 on a trip with Brian Joiner, Myron Tribus, and Peter Scholtes to meet with W. Edwards Deming and executives at many Japanese companies for a week and a half. Scholtes wrote an 18 page article about his impressions and lessons learned from the trip. It can be found here: http://cqpi.engr.wisc.edu/system/files/r005.pdf
While their trip focused primarily on manufacturing companies, it is still quite an interesting read. I’d recommend pages 7-8 in which W. Edwards Deming delivers a “scolding” lecture.
– Justin Hunter
Hi Henrik, I developed software a few years back at Toyota’s European headquarters in Brussels – and yep, pure waterfall there too. This was business process software and nothing to do with on-board car software and yet they still insisted on waterfall.
14 million lines of code on the Lexus eh? Seems a little on the extreme side to me. No wonder the Lexus range is so expensive!
henrik
Great post, it/s really interesting to see that toyota, were so much of lean thinking has come from is doing waterfall development.
who would/ve thought…
You talk about spontaneity and innovation, but maybe they just don’t need those in their embedded software at Toyota. Maybe getting tens of parts and systems that have to work together but each take time to build, out on time, takes planning. You can’t just ‘push back’ a car’s release date, or drop features. You surely aren’t going to try to fit all 1,000 engineers in the same Agile room and get them to ‘just talk’. I think this is a strong case for planning and waterfall; Agile isn’t a panacea.
Developing a next-gen car like the Prius takes A LOT of innovation. And the waterfall method was not working well for them. And there is no conflict between planning, innovation, and agile.
Great post Henrik. Thanks for sharing with the community. Why did it took you one year to write about this?
I wonder that myself :o)
I think I had to get my thoughts sorted out. And I had other more immediate priorities competing for my time. Anyway glad to finally get this article out!
Thank you very much for this feedback.
People: I would really like to know the Toyota context about people.
In agile development, we talk a lot about the iterative/flow process, but people maturity is not our concern enough.
Do you know if their software team are more stable than in the traditional western countries software industries?
About Standardization and metrics: Having them all long a car product development is probably demotivating.
Should we really have a pre-production with these metrics not correlated to the software innovation team production?
Ishii-san heavily emphasized the importance of the people aspect – the maturity and motivation of the people doing the development. I didn’t actually get to meet the teams and see their workspace though, that would have been interesting.
Hi Henrik,
One of the big dysfunctions I have seen in large organizations is how poorly communication flows up the management hierarchy/chain of command, even in companies which are trying to do Lean. Did you get any feel for how well the information actually flowed at Toyota?
Cheers,
Peter
Unfortunately not.
I’ve heard the estimate of tens of millions of lines of code in modern automobiles and that number has always seemed extremely high to me. I’ve also heard explanations that the “lines of code” in this case is more like assembly language instructions rather than a high-level language like Java or C#, which if true would make 14 million lines sound more reasonable. Can anyone with actual knowledge comment on that?
Ishii-san said that the 14 million lines of code for the Lexus LS460 include the navigation system (which I assume isn’t specific to that model)
Nice article.
Waterfall or Lean? Pick the one that works for you. Clearly Toyota got it wrong this time and learned a valuable lesson. In my experience, the more you try to reduce the cost of software development, the more expensive it becomes in the end.
If that, “eliminating waste” can not be achieved? or “eliminating waste” activities won’t result in “cost saving” at all?
Hi Henrik, thanks for posting this. I always love your clarity and balanced approach. It is interesting to see how even the most prominent Lean enterprises haven’t yet realized how to pursue flow in software development.
As Lean-Agile software development matures and gains more and more visibility, the potential for cooperation and synergies will be greater than ever 😀
Hi Henrik,
Thanks for sharing your experience. Micromanagement of tasks has its own disadvantages and biggest of them all is decreasing the motivation of developers. IMO, one should leave the deadlines to developers for them to mangage it and solely focus on the quality/testing of deliverables.
Seems pretty egotistical to think you’re going to teach Toyota how to implement TPS in software. I’d say you still have a lot to learn. As for Waterfall being a failure, how do you explain the thousands of banking systems, defense systems, telecommunications systems, etc that we rely on every day, all developed using waterfall?
Waterfall is useful for systems that Gartner describes as Systems of record (e.g. core banking system) with a life span of 10 or more years. While the agile approach is useful for solutions with shorter lifespan (speaking in Gartner terms: Systems of innovations or competitiveness).
Does this answer your question?
Great article! I completely agree with the product development comment; ‘A defect found in the production phase is about 50 times more expensive than if it is found during prototyping.’ Thanks for this.
I visited Toyota to learn, not teach. Ishii-san was the one saying that their current SW development process wasn’t working well, not me.
The fact that many companies use the waterfall method doesn’t mean it is good. On the contrary, there is an overwhelming amount of evidence of the high failure rate of waterfall projects, for example in the CHAOS reports from the Standish group.
Hi,
There is no doubt that Waterfall model has delivered as there are living examples. Now the question is under what circumstances waterfall model fails? Waterfall model works perfectly IF, the solution requirements can be defined with absolute clarity. If development starts before the required level of clarity is available (learning happens during development) costly rework becomes imperative over and above throwing the carefully planned schedule to winds. Result will be utter chaos, cost escalation and project failure.
Agile approach allows changes during development. As there is no overall plan made, there is no chaos created due to change. However, there could be rework due to changes (fed back into the product backlog in SCRUM). Careful planning and sequencing of product back log to attain READY status is still important.
By the way, there is also at least one article questioning the reliability of
CHAOS report published by Standish Group (Ref. The Rise and Fall of the
Chaos Report Figures by J. Laurenz Eveleens and Chris Verhoef, Vrije Universiteit Amsterdam).
Cheers,
-Sreekumar
Hi,
There is no doubt that Waterfall model has delivered as there are living examples. Now the question is under what circumstances waterfall model fails? Waterfall model works perfectly IF, the solution requirements can be defined with absolute clarity. If development starts before the required level of clarity is available (learning happens during development) costly rework becomes imperative over and above throwing the carefully planned schedule to winds. Result will be utter chaos, cost escalation and project failure.
Agile approach allows changes during development. As there is no overall plan made, there is no chaos created due to change. However, there could be rework due to changes (fed back into the product backlog in SCRUM). Careful planning and sequencing of product back log to attain READY status is still important.
By the way, there is also at least one article questioning the reliability of
CHAOS report published by Standish Group (Ref. The Rise and Fall of the
Chaos Report Figures by J. Laurenz Eveleens and Chris Verhoef, Vrije Universiteit Amsterdam).
Cheers,
-Sreekumar
I am glad to inform you of the publication of my new book about Toyota Production System.
The book is titled “The truth about Toyota and TPS” and can be found at the following link: http://amzn.com/2917260025
Regards,
E. Kobayashi.
Looks like a very interesting book!
Hey man, at first I would like to thank you for sharing what you’ve learned, that’s great. Second, I have to confess that I got a little bit disapointed now that I know that they are using waterfall over there. I really appreciate all the Toyota Way and Lean, I’m sure that their principles are very well defined and also really aligned and complementary to agile methods. They have a lot of empirical processes in their culture and when it comes to software their are using a deterministic approach? It makes me have feel that at some point they lost their relation with their own roots, and maybe the legacy that they left us is better than what they are currently doing and the last results that they showed us… What do you think about that?
Before making further conclusions people should realize some important things:
1) Embedded software is a different kind of animal than e.g. web sw.
2) Most of the software in Toyota cars is not made by Toyota company.
3) Automotive industry relies on supplier chains, having tier-1, tier-2 etc. suppliers rather than one company making all.
4) V-model is widely used in embedded software development. The right-hand side (test) can be done in parallel (or even before if you truly believe in test-first approach) so it does not need to be followed in waterfall way.
5) For most software, Toyota or some another brand company, is not involved in bottom part of V at all.
6) Supplier’s development method can be scrum, waterfall, spiral etc. model.
7) Unit costs is still the driving force for the automotive industry – and that fits poorly to software.
Henrik,
Thanks you for sharing this. I am impressed you can write it so vivid after nearly a year. Being there myself I find it a very accurate description of what we heard. I am though not so sure we can blame Toyotas current problems on not using agile/lean software development. Unfortunately they have much deeper problems that can not be solved by a quick transition to agile development. See for instance these two recent articles:http://bit.ly/b0THrW and http://bit.ly/dcfC65
Very interesting articles! Thanks for pointing them out.
I think this article is very interesting. However I disagree with some of your observations.
The V model does not necessarily translate to a waterfall model. It’s the length of the iterations that is the important – the feedback time based on the amount of “noise” in the process.
Or as Schwaber describes it:
“Scrum is based in empirical process control theory. As the degree of complexity rises, the number of inspections must be increased. Because of the increased frequency of inspections, the opportunity for adaptation also increases”
It does not matter if you have the V-model or not, it’s how often you verify the results that is important. Also, how much adaptation that is needed when developing a car will vary between the various components. In some cases you will have a very defined set of inputs and a defined set of outputs from that software (as in TDD). TDD is actually a way of strictly determining the requirements for a piece of software – but the verification is done according to the V-model.
And having third party suppliers, you need a process for agile development that supports procurement of agile development – which for example would exclude fixed price contracts.
In the (car) manufacturing industry, it is also much harder to ignore depenendencies, and you might have to introduce network diagrams to track or resolve them (which according to Schwaber is one of the major impediments to being agile) – but what is the alternative? Designing the cars iteratively? Including design, procurement and manufacturing?
The problem is that when it comes to cars, percieved quality includes grade. That is – quality is not only conformance to specification, but also “fitness for purpose”.
If I would make a process for software development for car manufacturers, I would not use a process like Scrum directly. I would look at the costs of change, the knowledge of changes (both what and how to change things), the dependencies (hardware, software and procurement), and then I would make a prioritization on the most important constraints (for example quality). Only then it is possible to say at what stages they should work iteratively – how long the iterations should be and how to manage and resolve dependencies.
Hi Klas! Thanks for your comment. I agree that the V model doesn’t always mean waterfall. But in this case it apparently did (Ishii-san even referred to it as a waterfall model at one point, if I remember correctly). The problems he described were also very typical waterfall symptoms (for example big test & fix cycle at end of project). Scrum or not, any suitably adapted combination of Agile methods (for example Scrum/XP/Kanban) would probably serve better than their current model.
Hi Henrik, this study is really interesting. I haven’t had the opportunity to go to the roots of Toyota but according to my studies, I suspect that Toyota uses a waterfall like development method due to their own product development culture.
AFAIK, they have a phase called Kentou which is the period at the beginning of the process of developing a product, where they develop studies and designs that will define in up-front what the final product will be. At the end of this phase they deliver a document called Sijisho, which means “document with direct commandment”. Once delivered, the Sijisho becomes law at Toyota. Since this is deeply rooted in their culture, and they must know how software components will interact with the other car’s components, they need to define things up-front. Nevertheless, I might be wrong.
I also don’t like their “micro management” style with so detailed metrics. I believe that there is a lot in the agile way that could really make improvements in the way they develop software, specially with respect to people.
I liked your experience and the more important is that you have shared this useful and important information with us. I greatly appreciate your post.
Major thankies for the post. Much obliged.
Good initiative, step to move ahead.
can someone provide 3 projects based on lean software development
Thanks for sharing!
Having third party suppliers, you need a process for agile development that supports procurement of agile development – which for example would exclude fixed price contracts.
13 years after still incredibely interesting, did Toyota filled the gap on the software side ?