Trying to make a document that's self contained and explains everything someone needs to know about a system is doomed to failure.
Don't accuse me of being anti documentation here, because I'm not. Documentation is very important. The important thing is to not expect unreasonable things from documentation, the people trying to write it, or the people trying to read it. Think about the capabilities of an ideal person or team that is capable of producing clear, conscise, perfect and thorough documentation. Think about how much time they need to produce that documentation. Now think about what those people could accomplish if they were working on the product instead of churning out documents. Or think about what your organization could do with the money you spent producing these documents.
Let's say you have a fairly complex system that's been running for a while, and you want to bring in a new team or a new developer. You want to hand this new developer a document that describes the system, how it works, where we want to go, what its problems are, how we might solve those, and so on. Because you're a disciplined development team, you want to have good documentation. And you want to be able to hand this document over to the new people and have them ready to rock and roll on the project after reading the document.
It ain't gonna happen. If you do try to make it happen, you're going to waste a lot of time and effort. That's because either the effort will succeed, in which case you've used your best talent spending a tremendous amount of time to produce something that's obsolete by the time it's done. Or else it will fail, and you've wasted time that way. Or a mixture of both. A project where even success is a failure is a good kind of project to avoid, and there are many such endeavors available in software engineering.
Documentation is not useless. The problem is that for almost all projects, the effort involved in producing documentation that fulfills the lofty goal of being self-contained and explaining everything well is not attainable using a level of effort that is worth spending. Of course there are exceptions to this rule of not trying to document overmuch, such as systems that don't change
much or have a whole lot of engineers working on them. But for typical projects, too much documentation is a fool's errand. There area couple of reasons for this:
1. The volume of info you need to communicate is immense. For a reasonably complex system, the size of this document would be comparable to a typical complete book about a programming language. You probably have a book about a programming language on your bookshelf in sight of you right now. Look at that book and think about it. Everything that book does, your document is going to need to do. It starts from step zero, talks about the basics first-- they've put a lot of thought in to how to structure and sequence the information so that it comes in digestible chunks, where each part is understandable based on the information that has arrived before. It'll need lots of examples and diagrams. Most people learn better from doing than from reading, so it'll need lots of examples and exercises that the reader can execute.
Anyone capable of actually doing this kind of instructional design is one of your best people. Probably more than one of your best people, who will need to collaborate in order to get the right skillset together at the same time. Or more likely still, you don't even have people that can be reasonably expected to do well at this. But if you do have such people-- Think of what they could do to your system if you let them actually do real work. But it gets worse.
2. The document won't accomplish as much as you hope. How many times have you seen huge documents that nobody ever read? Maybe it's because the document was too big, there wasn't time to read it. Maybe it's because the document was too small, it wasn't complete enough. Maybe it was overwhelming. There's a million reasons why people don't read documents or why when they do it's not all that enriching. These reasons have to do with deficiencies in the reader, the document, or the writer, or difficulties and complexities in the subject matter itself.
A system that works perfectly, and whose code is simple and understandable, and doesn't need further attention: such a system probably doesn't need a whole lot of documentation anyway. So spend that time making the system work, making it trouble free, making it simple and understandable.
Documentation is a vital part of many projects, but it's not an end unto itself. Keep the docs the as small as possible, and keep your people moving the product forward.
