Tag Archives: waterfall

Toyota’s journey from Waterfall to Lean software development

Posted on by


Toyota’s journey from Waterfall to Lean software development

Guess what. Toyota uses the waterfall method for software development – and now they’re trying to figure out how to go Lean.

Surprised? So was I!

My lean study tour to Japan in April 2009

One of the core tenets of Lean Thinking is Genchi Genbutsu – go and see for yourself. After years of experimenting with how to apply Lean principles in software development I decided to apply Genchi Genbutsu and go the source – visit Toyota and find out how they do it.

Despite all the books and articles written on Toyota Production System and Lean Thinking, very little has been published about their product development process and just about nothing about how they do software development.

So In april 2009 I was part of a small group that went on a “lean study tour” to Japan to visit Toyota and other companies in the Lean & Agile space. The group included several colleagues from Crisp, Tom and Mary Poppendieck, consultants from Best Brains in Denmark, and other mostly Scandinavian lean enthusiasts. I grew up in Japan so it was extra fun to visit the country again, this time wearing “lean glasses”.

We learned many interesting things, for example we met Katayama-san (chief engineer of several luxury and sports car models) and learned about Toyota’s product development process. But in this article I’m going to focus on the biggest surprise – how Toyota develops software.

Meet Satoshi Ishii, manager of automative sofware engineering dept.

We had the great honour of meeting Satoshi Ishii, manager of the embedded software division – i.e. the software that goes into the cars. His English was rather halting and I didn’t take detailed notes, so some of the quotes and conversations below are paraphrased.

First surprise was when he opened up by saying “I think you know more about Lean software development than we do”, and after that it just got more and more interesting.

All in all it I was impressed by Ishii-san and his presentation. He started with Toyota’s goal and vision (“we aim at the realization of sustainable mobility society”), then went into how software development plays into this, then described the problems they are having today and their strategy for solving them. We realized afterward that his presentation (and in fact all presentations we saw at Toyota) pretty followed the A3 problem solving format, confirming that this approach is deeply engrained in the Toyota culture.

Why Toyota needs to become an IT company

Toyota builds cars (duh). In the past that didn’t involve much software, and the little software that was needed was mostly developed by suppliers and embedded in isolated components. Toyota assembled them and didn’t much care about the software inside. But “The importance of automatic electronic control system has been increasing dramatically year by year” said Ishii-san.

 A modern car is pretty much a computer on wheels! In a hybrid car about half of the development cost is software, it contains millions of lines of code as all the different subsystems have to integrate with each other.  He mentioned that a Lexus contains 14 million lines of code, comparable to banking and airplane software systems.

Ishi-san concluded that “Therefore Toyota needs to become an IT company”.

Toyota’s current development process: Waterfall

Imagine our surprise when he pulled up this picture:

 

 

It was all there. The big V. Architects that don’t code. Distinct phases – all requirements to be complete and signed off before before coding, all code complete before testing, etc. He even called it a waterfall method.

We asked if he was happy with this way of developing. The answer was No. He told us about all kinds of problems they are having, most of them familiar to anyone that has experienced waterfall-style software development. There are countless case studies illustrating why the waterfall process just isn’t appropriate for software development. Even the original paper on the waterfall model in 1970 (”Managing the development of large software projects” by Dr Winston Royce) says “the implementation described above is risky and invites failure”.

Toyota is moving towards lean and agile software development

We asked Ishii-san if he had considered Agile software development. He was aware of Agile and liked the ideas, and said they will probably move in that direction. But they will do it in the Toyota Way – patiently and methodically, as Agile is not a goal in itself. I couldn’t agree stronger.

He said that “we are trying to learn how to apply TPS (= what we call Lean in the west) to software development”. Imagine the look on our faces. We came there to learn from what we thought would be the holy grail of lean software development, most of us were expecting to be dazzled and impressed.

He showed us this version of the Toyota house:

He said that they are currently at the bottom of the “house” (see the text highlighted in red, with numbers). Management, Process, and Making People. Without that, you can forget the other more technical aspects of lean such as Flow and Waste elimination.

He talked a lot about transparency – “How can you manage something that is invisible? How can we visualize software development? The key to success is mutual understanding between managers and engineers.”

 He also talked a lot about engineer motivation and skills – that software is a creative process and motivation is key. “If engineers feel their own skills are improving by ‘visualization’, they can get high motivation for their work. The most important thing in the project is not only to develop software but also to cultivate engineers.”

One of the big impediments is their current software architecture. He didn’t get into details, but mentioned that they need to make significant changes in their architecture to enable Lean and Agile software development. My belief is that it is the other way around – that Lean and Agile software development provides a way to implement the architectural change in an iterative & incremental way.

He emphasized the importance of testing & fixing defects early. A defect found in the production phase is about 50 times more expensive than if it is found during prototyping. If the defect is found after production it will be about 1,000 – 10,000 times more expensive! I’ve seen other studies showing similar numbers. He showed some data on this, visualized in the bar chart below:

Reality turned out to be worse, a lot worse! As Toyota’s current problems with the Prius braking systems is costing them over $2 000 000 000 (2 billion!) to fix because of all the recalls and lost sales. “Toyota announced that a glitch with the software program that controls the vehicle’s anti-lock braking system was to blame".

Standardization and metrics – is Toyota going too far?

Most of Toyota’s ideas about how to do Lean software development resonated well with me. My feeling was that they are on the right track.

One thing bothered me though – the extreme focus on detailed metrics. I agree with the value of visualization, standardization, and data-driven process improvement – but only if used at a high level. My feeling was that Toyota was going to far. They say engineer motivation is critical, but how motivating is it to work in an organization that plans and measures everything you do – every line of code, every hour, every defect, how many minutes it takes to do an estimate, etc? I saw very thick and detailed process manuals about this – not only at Toyota, but at other Japanese companies that claimed to be implementing Lean software development.

The thought “ugh… micromanagement” popped into my head several times during the tour. Maybe it’s a cultural thing.

I’ve spent years helping IT companies implement this stuff in practice. I’ve found that in many cases both motivation and predictability improves when my clients decrease the level of detail in their estimate and plans. Software development seems inherently unpredictable, so spending too much time and effort trying to make detailed estimates and plans is wasteful and in some cases even counterproductive. Having detailed manuals for everything you do stifles innovation and creativity. Many of my peers and colleagues in the Agile software development community share this experience.

So my feeling is that either the Agile community has something to learn from Toyota, or Toyota has something to learn from the Agile community. Probably both.

Closing thoughts

It is deeply engrained in Toyota’s culture to be dissatisifed with the status quo. So my feeling is that even if they had a really good software development process, Ishii-san would have said that they were dissatisfied with it and wanted to improve it.

Despite all the problems Ishii-san mentioned, my first thought was that their process can’t be all bad since they are a successful company with millions of cars all around the world using their software. The Prius model was developed in record-short time despite the extreme level of innovation in that project.

 I hadn’t (at that time) heard anything about quality problems with Toyota’s software. So waterfall or not, they still seemed to know what they were doing.

In fact, my conclusion after the trip was “well, now I know that there’s at least one company in the world that can succeed with the waterfall model” and I decided to stop bashing the waterfall model as hard as I usually do.

Now, however, with all the problems Toyota are having, I’m starting to reconsider. Are these problems software related? Could they have been avoided if Toyota used Lean & Agile software development instead of the waterfall model? I can’t sure, but I suspect yes.

Acknowledgements

Many people helped make this trip (and article) possible. I’m grateful to you all, and I’d especially like to thank:

  • Satoshi Ishii-san and Toyota for finally telling the world how they do software development
  • Mary and Tom Poppendieck for facilitating a Hansei (reflection) after each day on the tour. This helped us sort out our thoughts.
  • Kenji Hiranabe for enabling the visit to Toyota.
  • Bent Jensen and Best Brains for organizing the first Roots of Lean Study tour. They are doing it again, click here if you want to join!

The lean study tour was extremely inspiring and well-organized! I’ve lived in Japan for 16 years so I wasn’t sure I’d learn anything new – but going back there with ‘Lean glasses’ on I met so many interesting people and saw so many interesting things that I’ve realized my Lean journey has only just begun.

Lean = Mini waterfall?

Posted on by

Min gode kollega Hans Brattberg ställde denna provocerande fråga:

Vad i Lean hindrar oss från att göra en kejda av
Req Spec -> Architecture -> Design -> Program -> Test -> Production

Med handovers hela vägen?
Utan kommunikation?
Utan Cross Functional?
Utan Feedback?

Kedjan "Req Spec -> Architecture -> Design -> Program -> Test -> Production" som Hasse beskriver är visserligen ett vattenfall i den bemärkelsen att man gör saker i en viss ordning. Det gör man ju alltid i en verksamhet med litet koll på läget; "bäst att tänka litet först innan jag kodar" eller "det går ju inte att driftsätta innan koden finns".

Det är alltså inte dåligt utan bra att  göra saker och ting i rätt ordning.

Det som gör en kedja till en vattenfallsprocess av det dåliga slaget är:

  • om varje steg i kedjan innebär ett (del-)projekt i sig. Man gör t.ex. design av hela systemet innan man börjar programmera.
  • om varje steg i sig innebär en överlämning. Det är inte samma människor som gör efterföljande steg och man kommunicerar t.ex. via dokument (designspecar) istället för ansikte mot ansikte.
  • om det finns få tillfällen där man kan ändra kraven, innehållet eller konstruktionen. När t.ex. designen av systemet är satt och inte får ändras eftersom tidsplanen kommer att gå åt skogen då.
  • om det finns få återkopplingstillfällen att förbättra processen. Exempelvis en projektavslutning var 18:e månad.

Så vitt jag vet motverkar lean varenda en av ovanstående punkter. I lean handlar det om kontinuerligt flöde, små köer och korta cykeltider. För att uppnå detta försöker man:

  • bara göra saker som är kopplade till den aktuella produktelementet (ett försök till översättning av "feature") man jobbar med. Allt annat (t.o.m. andra produktelement = parallellt pågående arbete) betraktas ju som waste.
  • minimera antalet överlämningar och effekten av detta. Bara för att man gör saker i en viss ordningeng behöver man ju inte ha olika stationer med olika kompetenser på varje station. Till skillnad från biltillverkning där man måste ha olika stationer pga att maskinerna man använder i många fall bara står på en fysisk plats behöver vi mjukvaruutvecklare inte lida av detta. Vi kan använda samma dator till alla steg i kedjan om vi vill. Skall man slippa överlämningar är tvärkompetenta team (en ny lyckad översättning av "cross functional teams" 🙂 att föredra.
  • eftersom man bara bygger ett produktelement i taget och cykeltiden för denna är kort kan man innan man startar nästa element prioritera och välja vilket man vill göra härnäst. Kanban är ju ett exempel på detta då PO får prioritera varje dag vad som skall göras.
  • I Scrum har man återkoppling (retrospective) minst en gång i månaden. I lean skall man ju ha det varje dag hela tiden, eftersom det är katastrof om man inte gör saker bättre idag än man gjorde igår (efter vad jag hört om Toyota).

Sammanfattningsvis tycker jag inte det blir vattenfall bara för att man gör saker i en viss ordning. I så fall är allt utom kaos vattenfall. Lean motverkar det som vi anser är problem med traditionella vattenfallsprocesser på ett rätt explicit sätt.