← notes

Civilizations and Knowledge

• Preventing the Collapse of Civilization
• Civilizations: Institutions, Knowledge and the future

Principles

• Simplification helps the perpetuation and generation of knowledge.
• Simplification requires understanding.
• Good development values are sustainability, efficiency and robustness.
• Accidental complexity is not knowledge.
• Perpetuating knowledge is a social endeavour.
• Renewing knowledge requires functional institutions. This is, an institution that understands the knowledge.
• Between perpetuating and renewing, renewing is the high value operation. Perpetuating is a side effect, if you are able to renew, you are perpetuating the foundational knowledge.

Takeaways

• Having a lineage of knowledge (transfer) is a social endeavour.
• Institutions are the social constructs responsible for maintaining and renewing traditions of knowledge.
• Severance of lineage, make institutions unable to renew themselves, and decay.
• Decay may be hidden due to the reliance of other functional institutions.
• The understanding of knowledge is required to generate AND perpetuate it.
  • Reaching good understanding requires substantial amount of energy.
  • The better the understanding:
    • Simplification is enabled.
    • The amount of energy required to transfer knowledge gets decreased.
      • Decreased from the individual's perspective, but transferring knowledge is still a social endeavour.
  • Incomplete/incorrect understanding leads to incomplete/incorrect transfer, which degrades the knowledge.
• Lack of understanding begets miscommunication.
  • Layering attempts to remove the not understood components, because we are not able to easily communicate them
  • Ego makes it hard to deal with gaps in knowledge

Conclusion

A sustainable software industry is that in which every layer is well understood, which makes the layer's knowledge easily transferable. It would be feasible for anyone to make their own hardware/os/library/language. This needs constant simplification at all layers.

Creating functional institutions, that is, institutions that can renew themselves, is key to achieve things that last longer than a single lifetime.

Related (random) thoughts

• If simplifying every layer is the path to perpetuating knowledge, my question now is, do we have the knowledge in the first place? Does someone know how to do hardware from scratch? ASM and how CPUs work is not the foundational knowledge. Electric engineering is? (*)we do, to some extent, there's even games around this. ie. Turing Complete
• Is there a way an institution can detect the decay, and self correct?
• Are institutions created to preserve traditions of knowledge, or traditions of knowledge create the institutions to preserve them? (Egg and chicken problem?)
• What is accidental complexity then?
  • Is it an apparatus that facilitates the transfer of knowledge? Albeit, incomplete/incorrect knowledge.
• Using libraries is ok, but, find good libraries that actually solve a problem, and don't defer anything to you.
  • Harder than it sounds
• On Layering to remove not understood components, and facilitate communication:
  • How does this correlate with OOP and Clean Code being culturally dominant?
    • If I think about this in the "What are you trying to sell me?" lense, then selling this becomes easier, because we have better vocabulary to distribute it. Everyone can agree that "clean" is a good thing. Even if you don't understand what it really means.
    • OOP lets you think in language not related to the platform. Creating categories of things that let us make sense of components.
    • Perhaps it can even say something about functional, but I'm not sure what.
• Making things simpler therefore would imply generating knowledge?

Notes

On technology degrading

• Technology does not automatically improve, it degrades.
• Generating AND perpetuating knowledge requires energy.
• When people don't understand something, they easily dismiss it. Probably more than being dismissive, it boils down to the previous point, it also requires energy to understand.
  • The mere existence of knowledge, or information, requires energy.
  • Information is the semantic ordering of data? This order pulls energy.
• Ego could also play a factor in the perpetuation of knowledge?
• If there is no good/reliable way to transfer knowledge, technologies are lost.
  • What are the most efficient (cost efficient, low energy) ways to transfer knowledge?
• There are software advancements, but in a small niche?
  • Is hardware really advancing, or is it just a factor of density. Being able to handle more and more in less space. What happens when hardware can't do this anymore?
  • Machine learning has the best knowledge of the trade? Game developers have a better understanding of the physical constraints of a computer?
  • Somehow this feels incomplete, as in, they are not contained in niche industries, rather, there's a handful of individuals that actually know what they are doing. Where they work at is accidental. But how did this individuals came to be? What nurtured them?
• We are used to low quality software. Our tolerance levels have increased, but why? Is it due to us not understanding the underlying systems at play?
• Lots of developers are working on top of systems they don't fully understand. This made me think of the talk from Mike Acton. Solving the right Problems.
• If you choose to defer something to the caller/user, you are not solving a problem, you are making someone else solve it while managing your api, meaning, you not only did not solve a problem, you created a new one.
• Producing reliable software has increased costs?
  • Do these costs come from not understanding the underlying systems?
  • Are this costs even real? Web developers, or even people who heavily rely on someone elses infrastructure/libraries seem to deliver things faster. But when something breaks and they don't understand, they take time to resolve issues, and maintenance costs and annoyance from users may become a heavier cost, I think? But even, not relying on someone elses things, may become difficult, since there are a big number of things to understand before being effective developing in isolation.
  • Is this number of things really big? Or are there core concepts to understand that would make it easier to develop your own toolset?
• Companies do not generate tech (read knowledge), they use tech to deliver digital products and services.
• He limits the abstraction layering to language. Like ASM to C to Python.
  • But even after that, I don't think developers even think in terms of programming language, but of domain language.
  • Perhaps thats what the subsequent abstractions are trying to do, reach a point where domain language could be translated all the way down to machine code.
  • But the cost of all this abstractions is the lost knowledge of how the computer actually works, which hasn't really changed in decades. There are added/increased capabilities, but the underlying concepts are exactly the same.
• Productivity is not being increased, because we are all deferring solving the real problem to the caller/user. So a product developer ends up having to deal with tons of accidental complexity. The CPU is not accidental, the library is, IF it is deferring the real problem to the caller.
• Robustness and productivity is declining, with localized exceptions. Are this localized exceptions not relying on so many layers of abstraction? Perhaps that is what makes them effective.
• How many layers can a person remove to become as productive, reliable and ___.
  • I was going to write fast, but is that something worth pursuing? The perceived value here is an economical incentive. Dumb way of putting it: Going faster to market, getting the money faster.
  • Is this a topic of going further over faster? What would further entail in this context? Sustainable?
• Does the accidental complexity of transfering knowledge come from a lack of understanding of the topic, which then generates degradation?
  • I'm thinking about this given the comp enhance course. People (me include) think ASM is complicated, CPUs are hard, whatever. But Casey's deep understanding of the topic, simplifies the teaching material and lets us have a better/deeper understanding of the topic at hand.
• Relying on so many layers without understanding the foundations becomes a very fragile ecosystem.
• Simplification is the route to reliability.
  • Hardware, OS, libraries, application, network, compilation, debugging, distribution, interface
  • As a developer, where do I draw the line in things out of my control? I think the OS is the answer. We have little control over the hardware and the OS, everything else is on us software developers. Does virtualization help? Can you ship a piece of software that runs on a virtualized "custom" OS? (*)Things used to be like this many years ago, but is it possible now with so many layers?
• Our thought process gets driven by the underlying tools.
  • If there is an inherent limit to where we can simplify, what happens to the knowledge on the underlying layers?
  • There is also an inherent irreducible complexity to problems. What would simplification mean here?
    • This made me think of a talk from Casey about a paper regarding Quarternions. Quarternions are really complicated, but someone figured out a simple way to reason about them and transfer that knowledge more easy.

On institutions and traditions of knowledge

• We live in an ecosystem we did not create. We just inherited them.
• Someone has to know how things work.
  • Casey has mentioned something similar in the past. It is not necessary that everyone becomes an engine programmer, but some people DO need to know, otherwise, who will maintain those systems?
• Work is needed to learn. Blind optimism/cynicism deters us from gaining knowledge.
• What if cynical ploys for power in one generation, becomes the next generation intellectual inheritance? Severance of traditions of knowledge and institution of memory.
• Civillization enables harnessing social and material forces towards human ends.
  • We aim for things that last longer than a single lifetime, which requires us to build things that we can actually hand off to the next generation
  • We always rely on other people for the knowledge that we have because we are by nature social animals
• We construct our belief system, not on first-principles, but rather on who we trust and the environment we develop in.
• Convenience in life, makes us take things for granted.
• Functional institutions out-perform competition.
  • Dysfunctional ones, have underlying social problems, people not being able to coordinate, not knowing what they are doing. They just rely on a broken tradition of knowledge, what they've inherited.
  • Dysfunctional institutions just have a group narrative. Its easy to have a tribal costume.
• In an environment of plenty, dysfunctional institutions can seem like they are thriving, successful, when in reality, they are relying on other functional institutions.
  • This made me think about developing solely relying on AWS services, when you don't really know how anything works, you just rely on how AWS says something works. The functional institution is AWS, everyone else, not so much.
  • If AWS ceased to exist, if they were no longer functional, what would happen to the entire web industry (digital products/services)?
  • It is not necessary to go that far, AWS services break all the time, and it always makes the news how "half" the internet was down.
• Traditions of knowledge:
  • Dead: When nobody understands why things are the way they are. Knowledge might be around, but we don't know how to improve/renew.
  • Living (Lineage): Its not just about using the tribal costume, knowing the words. Imitation is not living tradition of knowledge. Contribution is.
  • Lost: Knowledge is no longer around.
  • This comes down to what the relationship of knowledge and understanding is, if there is no understanding, the tradition of knowledge is dead.
  • It also requires good social constructs (institutions) to perpetuate the knowledge and understanding.
• Communication is key in maintaining a tradition of knowledge. What makes it hard is that it becomes a social problem.
  • Its easy to trick people to think you understand, because questioning that requires conflict.
• Intellectual dark matter, is knowledge we can no longer see or attain but we know it exists
  • We have symbols of power from Greek civilization, even though we only have a small percentage of their knowledge. We don't really understand them.
• The dead of traditions of knowledge generates intellectual dark matter, which can lead us to an Intellectual Dark Age:
  • Failure of critical systems.
  • Lost tradition of knowledge
  • Squandering of capital. Immediate opportunity costs
  • Lives never lived
• Are we really more reliable than civilizations in the bronze collapse? (1200-1150 AC)
• Most institutions these days rely on other institutions, which make them seem functional.
  • Institutions or even extremely capable individuals.