# Software No digital system would be complete without software. We briefly spoke about this when we discussed the [[Printed Circuit Boards#Design process|board design process]], in particular the role of [[Printed Circuit Boards#Embedded Software|embedded software]] and how it co-evolves during hardware design. But that is at the board level, and digital systems of even a minimum complexity are seldom about one single electronic board but about a collection of those. It is hard to talk about software as a single, indivisible unit when it comes to complex systems that are made of many boards, backplanes, units, and racks. In software-intensive complex systems, software is, then, more of an aggregation of different software running across the system hierarchy, all hopefully aligned and well-choreographed for a common goal. Take, for instance, a surveillance radar: seen from a distance, it is a sensor to detect targets. When zooming in, a radar is composed of a front end, a control unit, a processing unit, and probably more subsystems. Zooming in further, these blocks will have tens of boards each, running a plethora of digital signal processing routines, filters, tracking algorithms, estimators, anti-jamming measures, and whatnot. Also, there will be management software to ensure the radar can communicate with a supervising entity—for instance, an aerospace surveillance command—and will probably include a user interface. Therefore, the software architecture of a radar cannot be defined bottom-up, that is, from the chips or the boards up. It needs to be thought top-bottom and developed knowing what kind of target application it will serve. The same goes for any other complex digital system, for instance, aircraft avionics. A commercial airliner reportedly has several millions of lines of code executing in a myriad of subsystems like Navigation Systems, Flight Management Systems, Approach and Landing Systems, Flight Control Systems, Fuel Management Systems, and many, many more. None of these systems, not the parts that compose these systems, can be developed unaware of the fact they will be executing on a flying machine. The application software in an airliner is not one single executable but a coordinated, intricate network of them. We can safely extend this axiom to any other complex digital system: application software is an aggregation of smaller application software across the system hierarchy. ## Only the source code tells the full story Not long ago, while browsing the wiki of an open-source project (a 2M lines of code kind of thing), I came across this gem:  Amen, brothers and sisters. Truer words have never been spoken or written: only the source code tells the full story. Full stop. Just in case it was not clear enough: > [!attention] Only the source code tells the full story. No document, no wiki, no README file, no UML model; nothing will tell the story as accurately as the recipe itself. If you need to bake a cake, go, and draw as many diagrams as you want, but nothing will beat the actual recipe telling you exactly how to make the damn cake. Software is like cooking; if you want to produce something that is remotely enjoyable, you must pay attention to the recipe. Model-based techniques of doing things have proliferated in the last two decades or so. But here's an unspoken truth: a piece of source code *is* a model, and its syntax is the artifacts and constructs we have at hand to model the semantics and the behavior we want to implement. Therefore, writing code is also model-based, the syntax being the lines and arrows to connect entities together. Someone could argue UML[^4] is easier to understand and follow than the source code. Only that, it is not. If you really, really want to know how a piece of software works, you will sooner than later end up browsing source code, compiling it, and putting it to run. UML diagrams can be, and will be, out of sync from the source code in no time because a software developer is always going to the source code when fixing bugs or adding new functionalities, while they only eventually go to update the diagrams. Keeping the graphical depictions that describe a software's design up to date is a highly manual process unless you still believe in that ideal world, a world of fairy tales, where source code completely autogenerated from models would keep the integrity between source and diagrams. I have seen engineers preaching about the marvels of auto-coding, and while I do recognize auto-coding has improved dramatically in the last decade or so, I have also seen the same engineers having a hard time debugging or optimizing auto-coded embedded software. Experience shows that most of the UML diagrams around have been created following the inverse logic: *from* existing source code. Often, it feels simpler to implement something directly in object-oriented language rather than drawing a sequence diagram of what needs to be done. And if it is easier to describe the problem in source code and the diagram proves to be "optional", what is the value of spending time drawing the diagram anyway? That's the dilemma of many software architects out there struggling to communicate software structure to the ones having to code it. There is no way around it: only source code tells the full story. If you want to hear partial stories, apocryphal stories, or fairy tales, go for it, but just be aware that you are only seeing an incomplete side of the plot when you look at a block diagram with colored boxes and arrows. ### The Myth of Code Readability Sure, no one wants to read cryptic code, as much as no one wants to read a badly written book. One of my favorite technical books about space was written by a German man whose rudimentary English makes it somewhat tricky to follow from time to time. Still, the book checks out in the sense that it conveys useful ideas, regardless of the grammar. We all want to communicate perfectly, and we try our best, but some people succeed more than others when putting their thoughts into words, be it poetry or a multi-threaded TCP/IP client. But comprehending complex source code requires going beyond that and working at a higher level of abstraction than the actual syntax. If you need to focus on how good or badly someone has named their variables, then you are probably putting your magnifier at the wrong zoom level. My argument is simple: there is not bad code, but there are bad source code analysts. Perhaps not bad, but lazy. Analysts who do not want to spend much mental energy and want everything distilled up front, who pretend the comments will meticulously explain how things work like they're primary schoolers. Regardless of the code readability (which largely focuses on syntax), comprehending thousands of lines of code needs to focus on the bigger picture and on understanding what are the salient building blocks of what you are analyzing and how those blocks relate with each other. Systems become actual systems thanks to the interactions between the constituent parts. Architecture readability is the real challenge we are dealing with here. As a source code analyst, refactorer, or (re)architect, you must walk past the minutiae of tooling, indentation levels, and camel cases. We all have our choices when it comes to coding style, build systems, and IDEs, and don't get me wrong: being tidy is a good thing. Naming functions consistently definitely helps searching (source codes with a mix of `camelCase`, `CamelCase` or `snake_case` can make your life harder when you're looking around for clues). But code is code, and a proper source code reader must be able to deal with it, come what may. ### Comprehending Code in Embedded, Distributed Systems Complex systems such as airliners have tens of millions of lines of code[^5] on board. What is more, those lines of code are not running all on the same computing devices but distributed in a collection of specialized computers all across their structure, connected by kilometers of cables, hubs, and routers. Recently, also running in several partitions inside isolated virtualized containers. Comprehending source code is greatly shaped by the contextual boundaries of the software. In simpler English: it is not the same story to comprehend 2 million lines of code supposed to execute all together in one single process than comprehend those 2 million lines of code spread in 10 different embedded computers running 200K lines of code each. The complexity of the interaction between the air gaps, the requests, responses, protocols, and handshakes, makes the comprehension process exponentially more challenging. So, next time you inherit legacy code, make sure you understand how cohesively (or not) those lines of code are supposed to run, in which computing contexts those lines are supposed to execute, and in case you inherit a distributed monster that executes here and there, deeply understand the interactions between the parts; sniff the protocols, listen to the packets and handshakes coming and going, and grow the overall picture. Debugging and running distributed software step-by-step is a complex matter because a lot is happening: cross-compilation, networks, delays, noise, etc. ### The Science Behind Code Comprehension Turns out, there is science behind code comprehension. Analyzing source code is a psychologically intensive activity. It is one brain against hundreds of thousands of lines of mysterious text ideated and written by one or more other brains. And our brains, as marvelous biological machines as they surely are, can meet certain limits in terms of processing complexity. There is a certain number of lines of code beyond which code comprehension becomes extremely difficult for a single brain. Why? Not because of any innate lack of initiative or laziness in our minds, but mostly because the whole process of analyzing such an amount of code becomes extremely cumbersome. Not only the act of keeping up with thousands of objects and data structures (and their relationship), but also because the sole act of compiling 1 million lines of code or more may take several minutes in a decent development laptop. IDEs' indexers can be overloaded and take a long time to converge and eat up your RAM in the process, and crash without notice. If you are debugging or refactoring a monstrously big code base, having to deal with such dynamics can become unbearable. Code comprehension—or more specifically architecture comprehension—should be a design factor when architecting software, driving the splitting of the code base in a way its comprehension would always be assured; at least until artificial intelligence finally makes a glorious appearance and can distill for us billions of lines of code in a matter of seconds. While we wait for AI, it would be great to see more "comprehension-driven design" of software. Therefore, comprehending large amounts of source code can be frustrating, and frustration is not something that really helps when it comes to mental clarity. Source code comprehension is an essential part of the software maintenance and/or re-architecting process, and, according to research[^6], it can be classified into two classes: - Functional approach: is interested in what the source code does. - Control-flow approach: is interested in how the source code works. The interesting part is that you cannot survive as a source code analyst with either one or the other. If you really need to dig into how a program works and do something tangible with it, you must embrace both: see what it does and how it does it. ### Code comprehension models The process of source code comprehension can be classified into four cognitive models[^7]: - Top-down code comprehension model; the knowledge of the program domain is restructured and mapped to the source code in a top-down manner. The process starts with a general hypothesis about the nature of the program, which represents a high-level abstraction or concepts of the program. Then, the general hypothesis is examined, refined, and verified to form subsidiary hypotheses in a hierarchical layout. Each hypothesis represents a segment or chunk of program code. The low levels are generated continuously until the comprehension model is achieved. This model is used when the analysts are familiar with the code. - Bottom-up code comprehension model; the analysts read the complete source lines of code, and then group these lines into chunks of higher-level abstractions. The elementary program chunks are specified based on the control flow model, and the procedural relations among chunks are defined based on functional relations. The knowledge of high-level abstractions is incrementally grouped until the highest level of program understanding is achieved. The comprehension is enhanced using refactoring of code functionalities. Bottom-up comprehension chunks the micro-structure of the program into macro-structure and cross-referencing these structures. Bottom-up comprehension is less risky than a bottom-up strategy, where the lower-level hypotheses can be identified directly from concrete source code. - Hybrid or Knowledge-based code comprehension model; the comprehension knowledge about the code is grasped from the integration of bottom-up and top-down models because the maintainer navigates through the source line of code and jumps through different chunks when searching the code to find the links to the intended block of code. The understanding of the program evolves using the analysts' expertise and background knowledge together with source lines of code and documentation. - Systematic and as-needed strategies; on which the analysts focus only on the code that is related to a particular evolution task. The analysts or (re)architects use a systematic method to extract the static knowledge about the structure of the program, and the causal knowledge about interfaces between different parts of the program at execution time. In the systematic macro-strategy, the programmer traces the flow of the whole program. This strategy is less feasible for large programs, more mistakes could occur because the maintainer misses some important interactions. The comprehension models described above come across as highly intertwined and insufficient when taken in isolation. In this text, we follow a combination of the Knowledge-based and Systematic approaches. ### Code Comprehension Typical Activities At any time during a program comprehension process, the analyst can apply any code comprehension model, when the size and the complexity of the program structure are varied from one program to another and from block to another. To achieve comprehension the following activities, are typically followed: - Read the Code and Documentation (if any): Read the source code line-by-line or in any arbitrary order to understand the workflow and the application behavior of the program, and assist in locating the code where the change should be performed. Also, read and reexamine the application documentation, which is not always consistent and up to date; requirements and design models and specifications. - Execute White-box and/or Black-box experiments of the program to inspect the input, sequence of implemented functions, output, and consequences. It depends on the application type and the analyst's skills and knowledge. Dynamic runtime information is acquired from the executed program. - Extract block: the source code is partitioned conceptually into coherent blocks of code (i.e., segments of code) that share relevant functionality. Then analyze how each block works (functional approach) and how each block works (control approach). Partitioning into blocks depends on the continuous statements that share common functionality and code attributes. The block may have more than one related function. The structure of functions within the block of code is defined. This will facilitate the allocation of particular functions and their statements within a block. Partitioning helps in improving the readability of source code, filtering irrelevant code, locating data structures, assisting maintainers when locating the intended code in one area, and saving maintenance effort and time. - Analyze The Internal Structure Dependencies: The inheritance model and functional dependencies between areas are analyzed to preserve the intended behavior of the source code. - Generate Program Graphs; by transforming the source code of the program into functional graphs. The dataflow graph and control-flow graph (that is similar to that produced in white-box testing) assist in identifying the data and control dependencies in the source code. They make it easier for maintainers to read, understand, and find which parts are needed to maintain and which parts are affected by the maintenance. - Refactoring: this technique is used in code comprehension by analyzing the implicit structural dependencies and finding out the data and control interrelationships. The refactoring technique improves interactively the software structure, renames the methods and attributes, eliminates conflicts, and reserves the functionality of the program. The refactoring process also eliminates the code clones. The software architecture is standardized to have low coupling and high comprehension to decrease the complexity of source code. Using a meaningful and readable naming of functions, methods, and data helps maintainers to facilitate the understanding ability of the code and to locate the intended parts that need maintenance. These steps read as a bit sequential and isolated. Rearchitecting code involves mixing all these steps, with a great dose of experimentation which includes creating small, separate projects to try things out. ### The Necessary Mental Flow Analyzing source code is an investigative endeavor, and as such it requires concentration; it can only 'tick' after reaching reasonably deep mental states. The source code detectivesque activity is usually rewarded by dopamine bombs provided upon finding knowledge nuggets as the activity unfolds, and any disruption of this mental state can be impactful for the overall performance and take a very long time to regain. If the average incoming email takes five minutes to read and reply and your re-immersion period is fifteen minutes, the total cost of that email in terms of flow time (actual brain work time) lost is twenty minutes. A dozen emails per day will use up half a day. A dozen other interruptions and the rest of the workday are gone, let alone getting to figure out what the source code does. Just as important as the loss of effective time is the accompanying frustration. The source code analyst who tries and tries to get into _flow_ and is interrupted each time is not a happy person. She gets tantalizingly close to involvement only to be bounced back into awareness of her surroundings. Instead of the deep mindfulness that she craves, she is continually channeled into the promiscuous changing of direction that the modern office tries to force upon her. A few days like that and anybody is ready to look for a new job. If you're a manager, you may be relatively unsympathetic to the frustrations of being in no flow. After all, you do most of your own work in interrupt mode—that's management—but the people who work for you comprehending code for rearchitecting and refactoring need to get into the flow. Anything that keeps them from it will reduce their effectiveness and the satisfaction they take in their work. It will also increase the cost of getting the work done. ## Build Systems & Tools ==Building software is essentially different than writing and comprehending code in its motivation, objectives, and dynamics.== In fact, my argument is that both are at odds with each other. On one hand, building software is about defining the dependencies and code structure once, only to then—after this one-time configuration stage—invoke in as few steps as possible the sequence necessary for building the binaries and just repeat the process n number of times. Eventually, building software becomes a matter of executing a selected set of commands or scripts. Building software has a clear goal of giving you binaries; to decrease the time between refactoring and coding and runtime. Building software in highly automated manners makes a lot of sense. This way, software is more replicable, more portable, and can also be integrated into other systems such as build servers for a full-blown software engineering suite including testing and the like. Comprehending software, on the other hand, is in a way about reconstructing what the build system is doing rapidly and automatically. Imagine that you enter a highly automated factory floor full of robots, and someone tells you that you must redefine the factory layout for better efficiency, or because now the company wants to assemble a new product. If you are new to the current layout, you will need to observe what the robots are doing, which will most likely require you to slow them down to get the process right. Automation brings speed and abstraction, which are great when in production, but detrimental when in need of comprehension when you need access to detail. The thesis is clear and transpires as somewhat obvious: build systems do not necessarily help in code comprehension. But this is not a terrible problem, we can live with this. We only need to be aware. Let's maybe dive a bit into build systems and how they work, how they may make the rearchitecting process somewhat challenging, and let's check the most popular build systems out there. This will be only an introduction; for more details, I recommend you check their websites and repositories and play with them accordingly to gain the necessary familiarity. ### Autotools and the GNU Build System I am sure you have delved into software packages full of files named configure, `configure.ac`, `Makefile.in`, `Makefile.am`, `aclocal.m4`, and so forth, some of them generated by Autoconf or Automake. But the exact purpose of these files and their relations is probably tricky to understand at first. The goal of this section is to briefly introduce you to this machinery. In the Unix world, a build system is traditionally achieved using the command `make`. You express the recipe to build your package in a Makefile. This file is a set of rules to build the files in the package. For instance, the program code may be built by running the linker on the files main.o, `foo.o`, and `bar.o`; the file `main.o` may be built by running the compiler on `main.c`; etc. Each time `make` is run, it reads the Makefile, checks the existence and modification time of the files mentioned, decides what files need to be built (or rebuilt), and runs the associated commands. When a package needs to be built on a different platform than the one it was developed on, its Makefile usually needs to be adjusted. For instance, the compiler may have another name or require more options. In 1991, David J. MacKenzie got a bit sick of customizing Makefiles for the 20 platforms he had to deal with. Instead, he handcrafted a little shell script called configure to automatically adjust the Makefile. Compiling his package was now as simple as running said configure script and then invoking `make`. Nowadays this process has been standardized in the GNU project. The GNU Coding Standards recommends that each package of the GNU project should have a `configure` script and the minimal interface it should have. The Makefile too should follow some established conventions. The result? A unified build system that makes all packages almost indistinguishable by the installer. In its simplest scenario, all the installer has to do is to unpack the package, run configure, `make` finally `make install`, and repeat with the next package to install. This is called the GNU Build System since it was grown out of the GNU project. However, it is used by a vast number of other packages: following any existing convention has its advantages. The Autotools are tools that will create a GNU Build System for your package. Autoconf mostly focuses on `configure` and Automake on Makefiles. It is entirely possible to create a GNU Build System without the help of these tools. However, it is rather burdensome and error-prone. Let's try a "hello world" using Autotools. Create the following files in an empty directory. `src/main.c` is the source file for the hello program. We store it in the `src/` subdirectory: ```C #include <config.h> #include <stdio.h> int main (void) { puts ("Hello World!"); puts ("This is " PACKAGE_STRING "."); return 0; } ``` README must contain some limited documentation for our little package. ```Console $ cat README This is a demonstration package for GNU Automake. Type 'info Automake' to read the Automake manual. ``` `Makefile.am` and `src/Makefile.am` contain `Automake` instructions for these two directories. ```Console $ cat src/Makefile.am bin_PROGRAMS = hello hello_SOURCES = main.c ``` ```Console $ cat Makefile.am SUBDIRS = src dist_doc_DATA = README ``` Finally, `configure.ac` contains Autoconf instructions to create the configure script. ```Console $ cat configure.ac AC_INIT([amhello], [1.0], [[email protected]]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_PROG_CC AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([ Makefile src/Makefile ]) AC_OUTPUT ``` Once you have these files, it is time to run the `Autotools` to instantiate the build system. Do this using the `autoreconf` command as follows: ```Console $ autoreconf --install configure.ac: installing './install-sh' configure.ac: installing './missing' configure.ac: installing './compile' src/Makefile.am: installing './depcomp' ``` At this point the build system is complete. In addition to the three scripts mentioned in its output, you can see that `autoreconf` created four other files: `configure`, `config.h.in`, `Makefile.in`, and `src/Makefile.in`. The latter three files are templates that will be adapted to the system by `configure` under the names `config.h`, `Makefile`, and `src/Makefile`. Let's try it: ```Console $ ./configure checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... gcc3 configure: creating ./config.status config.status: creating Makefile config.status: creating src/Makefile config.status: creating config.h config.status: executing depfiles commands ``` You can see `Makefile`, `src/Makefile`, and `config.h` being created at the end after configure has probed the system. It is now possible to run all the targets we wish. For instance: ```Console $ make … $ src/hello Hello World! This is amhello 1.0. $ make distcheck … ============================================= amhello-1.0 archives ready for distribution: amhello-1.0.tar.gz ============================================= ``` Note that running `autoreconf` is only needed initially when the GNU Build System does not exist. When you later change some instructions in a `Makefile.am` or `configure.ac`, the relevant part of the build system will be regenerated automatically when you execute make. `autoreconf` is a script that calls `autoconf`, `automake`, and a bunch of other commands in the right order. If you are just beginning with these tools, it is not important to figure out in which order all of these tools should be invoked and why. However, because `Autoconf` and `Automake` have separate documentation, the important point to understand is that `autoconf` is in charge of creating `configure` from `configure.ac`, while `automake` is in charge of creating `Makefile.in` from `Makefile.am` and `configure.ac`. This should at least direct you to the right manual when seeking answers. ### Build Systems and Files Appearing out of Nowhere An issue quickly appears while describing how `Autotools` work. You saw in a previous step that a file called `config.h` was generated. This is a header file, and therefore essential part of the compilation process and part of any code comprehension endeavor. And it did not exist before the build system was invoked. What is the conclusion there? Correct. Build systems generate source code which might be essential for a software comprehension process and code analysis. So, let's put it again big enough: > [!Note] > Build systems tools frequently autogenerate code that is essential for the dissection and comprehension process. > [!important] A corollary from the note above: when analyzing code down to see what it's made of you shall never skip taking a look at the build system used by the software you are dissecting, and you shall build it and run it to obtain all the relevant autogenerated files that are needed to understand the structure of the code under study. ### CMake CMake is another tool to manage the building of source code. Originally, CMake was designed as a generator for various dialects of Makefile, but today CMake generates modern build systems such as Ninja (which we will discuss soon) as well as project files for IDEs. Important is to note that CMake does not build software, it only produces files for a variety of build tools. The most basic CMake project is an executable built from a single source code file. For simple projects like this, a CMakeLists.txt file with a few commands is all that is required. CMake can generate a native build environment that will compile source code, create libraries, generate wrappers, and build executable binaries in arbitrary combinations. CMake has support for static and dynamic library builds. Another nice feature of CMake is that it can generate a cache file that is designed to be used with a graphical editor. For example, while CMake is running, it locates include files, libraries, and executables, and may encounter optional build directives. This information is gathered into the cache, which may be changed by the user before the generation of the native build files. CMake scripts also make source management easier because they simplify build scripts into one file and a more organized, readable format. CMake is intended to be a cross-platform build process manager, so it defines it is own scripting language with certain syntax and built-in features. CMake itself is a software program, so it should be invoked with the script file to interpret and generate an actual build file. A developer can write either simple or complex building scripts using CMake language for the projects. Build logic and definitions with CMake language is written either in CMakeLists.txt or in files ending with <project_name>.cmake. As a best practice, main script is named as CMakeLists.txt instead of cmake. CMakeLists.txt file is placed at the source of the project you want to build. CMakeLists.txt is placed at the root of the source tree of any application, or library it will work for. If there are multiple modules, and each module can be compiled and built separately, CMakeLists.txt can be inserted into the subfolder. .cmake files can be used as scripts, which run cmake commands to prepare environment pre-processing or split tasks which can be written outside of CMakeLists.txt. .cmake files can also define modules for projects. These projects can be separate build processes for libraries or extra methods for complex, multi-module projects. Writing Makefiles might be harder than writing CMake scripts. CMake scripts by syntax and logic have similarities to high-level languages so it makes it easier for developers to create their cmake scripts with less effort and without getting lost in Makefiles. Let's start with a basic "Hello World!" example with CMake so we wrote the following "Hello CMake!" the main.cpp file as follows: ```C #include <iostream> int main() {         std::cout<<"Hello CMake!"<<std::endl; } ``` Our purpose is to generate a binary to print "Hello CMake!". If there is no CMake we can run a compiler to generate a target basically with only the following commands. `$ g++ main.cpp -o cmake_hello` CMake helps to generate bash commands with the instructions you gave, for this simple project, we can just use a simple `CMakeLists.txt` which creates a Makefile for you to build the binary. It is obvious that, for such a small project it is redundant to use CMake but when things get complicated it will help a lot. To build `main.cpp`, using `add_executable` would be enough, however, let's keep things in order and write it with the proper project name and cmake version requirement as below: ```Console cmake_minimum_required(VERSION 3.9.1)project(CMakeHello)add_executable(cmake_hello main.cpp) ``` When the script is ready, you can run cmake command to generate Makefile/s for the project. You will notice that cmake is identifying compiler versions and configurations with default information. ```Console $ cmake CMakeLists.txt -- The C compiler identification is AppleClang 9.0.0.9000039 -- The CXX compiler identification is AppleClang 9.0.0.9000039 -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring done -- Generating done -- Build files have been written to: /Users/User/Projects/CMakeTutorial ``` When cmake finishes its job, the Makefile will be generated together with CMakeCache.txt and some other artifacts about the build configuration. You can run make command to build the project. ```Console $ make all # or $ make cmake_hello ``` You get the point. ### Meson Meson is a modern build system designed to be fast, easy to use, and highly configurable. It is designed to build software projects in a wide range of programming languages, including C, C++, Rust, Java, Python, and more. Meson is known for its simplicity and flexibility. Unlike other build systems that use complex configuration files, Meson uses a simple domain-specific language (DSL) that is easy to read and write. This DSL allows developers to specify the build configuration, including the compiler options, libraries to link against, and other build settings. One of the key advantages of Meson is its speed. Meson uses a ninja-based backend that is highly optimized for parallel builds. This means that Meson can build large projects quickly and efficiently, even on machines with multiple cores. Meson also supports cross-compilation, which allows developers to build software for different architectures and platforms. Meson can generate build scripts for a wide range of build systems, including Make, Ninja, and Visual Studio. Overall, Meson is a powerful and flexible build system that is well-suited for modern software development. Its simplicity, speed, and flexibility make it a popular choice among developers. #### Using Meson Let's start with the most basic of programs, the classic hello example. First, we create a file main.c which holds the source. It looks like this. ```C #include <stdio.h> int main(int argc, char **argv) { printf("Hello there.\n"); return 0; } ``` Then we create a Meson build description and put it in a file called meson.build in the same directory. Its contents are the following. ```Meson project('tutorial', 'c') executable('demo', 'main.c') ``` That is all. Note that, unlike `Autotools`, you do not need to add any source headers to the list of sources. We are now ready to build our application. First, we need to initialize the build by going into the source directory and issuing the following commands. `$ meson setup builddir` We create a separate build directory to hold all of the compiler output. Meson is different from some other build systems in that it does not permit in-source builds. You must always create a separate build directory. The common convention is to put the default build directory in a subdirectory of your top-level source directory. When Meson is run it prints the following output. ```Console The Meson build system version: 0.13.0-research Source dir: /home/jpakkane/mesontutorial Build dir: /home/jpakkane/mesontutorial/builddir Build type: native build Project name is "tutorial". Using native c compiler "ccache cc". (gcc 4.8.2) Creating build target "demo" with 1 files ``` Now we are ready to build our code. ```Console $ cd builddir $ ninja ``` If your Meson version is newer than 0.55.0, you can use the new backend-agnostic build command: ```Console $ cd builddir $ meson compile ``` Once the executable is built, we can run it. `$ ./demo` This produces the expected output. `Hello there.` #### Adding dependencies Just printing text is a bit too basic. Let's update our example program to create a graphical window instead. We'll use the GTK+ widget toolkit. First, we edit the main file to use GTK+. The new version looks like this. ```C #include <gtk/gtk.h> static void activate(GtkApplication* app, gpointer user_data) { GtkWidget *window; GtkWidget *label; window = gtk_application_window_new (app); label = gtk_label_new("Hello GNOME!"); gtk_container_add (GTK_CONTAINER (window), label); gtk_window_set_title(GTK_WINDOW (window), "Welcome to GNOME"); gtk_window_set_default_size(GTK_WINDOW (window), 400, 200); gtk_widget_show_all(window); } // end of function activate // // main is where all program execution starts // int main(int argc, char **argv) { GtkApplication *app; int status; app = gtk_application_new(NULL, G_APPLICATION_FLAGS_NONE); g_signal_connect(app, "activate", G_CALLBACK(activate), NULL); status = g_application_run(G_APPLICATION(app), argc, argv); g_object_unref(app); return status; } // end of function main ``` Then we edit the Meson file, instructing it to find and use the GTK+ libraries. ```Console project('tutorial', 'c') gtkdep = dependency('gtk+-3.0') executable('demo', 'main.c', dependencies : gtkdep) ``` If your application needs to use multiple libraries, you need to use separate [dependency()](https://mesonbuild.com/Reference-manual_functions.html#dependency) calls for each, like so: `gtkdeps = [dependency('gtk+-3.0'), dependency('gtksourceview-3.0')]` We don't need it for the current example. Now we are ready to build. The thing to notice is that we do not need to recreate our build directory, run any sort of magical commands, or the like. Instead, we just type the exact same command as if we were rebuilding our code without any build system changes. `$ meson compile` Once you have set up your build directory the first time, you don't ever need to run the meson command again. You always just run meson compile. Meson will automatically detect when you have made changes to build definitions and will take care of everything, so users don't have to care. In this case, the following output is produced. ```Console [1/1] Regenerating build files The Meson build system version: 0.13.0-research Source dir: /home/test/mesontutorial Build dir: /home/test/mesontutorial/builddir Build type: native build Project name is "tutorial". Using native c compiler "ccache cc". (gcc 4.8.2) Found pkg-config version 0.26. Dependency gtk+-3.0 found: YES Creating build target "demo" with 1 files. [1/2] Compiling c object demo.dir/main.c.o [2/2] Linking target demo ``` Note how Meson noticed that the build definition has changed and reran itself automatically. The program is now ready to be run: `$ ./demo` ### Ninja Remember we said some paragraphs above that once code reaches a certain size, the compiling process becomes annoying and slow? Let's remember this famous comic:  > [!Figure] > _Building software can be really slow (credit: XKCD)_ The issue of slowness has been known for a long time. For certain code bases, you can go and make yourself a coffee and have some small talk with colleagues before the whole thing compiles. Ninja is yet another build system, in the sense that it takes as input the interdependencies of files (typically source code and output executables) and orchestrates building them, quickly. In a way, Ninja joins the sea of other build systems. But its distinguishing goal is to be fast. Ninja was born from the Chromium browser project, which has over 30,000 source files and whose other build systems (including one built from custom non-recursive Makefiles) would take ten seconds just to start building after changing one semicolon in a file. Ninja performs in under a second. The developers of Ninja use an interesting analogy: where other build systems are trying to be high-level languages, Ninja aims to be an assembler. According to the Ninja designers, build systems get slow when they try to make decisions. When you are in an edit-compile cycle you want it to be as fast as possible; you want the build system to do the minimum work necessary to figure out what needs to be built immediately. Ninja contains the barest functionality necessary to describe arbitrary dependency graphs. Its lack of syntax makes it impossible to express complex decisions. Instead, Ninja is intended to be used with a separate program generating its input files. The generator program (like the ./configure in Autotools as we saw in previous sections) can analyze system dependencies and make as many decisions as possible upfront so that incremental builds stay fast. Going beyond Autotools, even build-time decisions like "Which compiler flags should I use?" or "Should I build a debug or release-mode binary?" belong in the .ninja file generator. #### Design goals of Ninja Ninja is purposely designed to be very fast, even for very large projects. It is designed with very little policy about how code is built. You need to tell Ninja what to do; Ninja is not made to figure too many things out, and that is in order to gain speed. Ninja is designed to get dependencies correct, and in particular situations that are difficult to get right with Makefiles (e.g., outputs need an implicit dependency on the command line used to generate them. In Ninja's book, when convenience and speed are in conflict, it prefers speed. To restate, Ninja is faster than other build systems because it is purposely made simple. You must tell Ninja exactly what to do when you create your project's .ninja files. #### Using Ninja Ninja evaluates a graph of dependencies between files and runs whichever commands are necessary to make your build target up to date as determined by file modification times. A build file (default name: `build.ninja`) provides a list of rules— short names for longer commands— along with a list of build statements saying how to build files using the rules, and which rule to apply to which inputs to produce which outputs. Conceptually, build statements describe the dependency graph of your project, while rule statements describe how to generate the files along a given edge of the graph. Here's a basic .ninja file that demonstrates most of the syntax: ```Console cflags = -Wall rule cc command = gcc $cflags -c $in -o $out build foo.o: cc foo.c ``` #### Variables Ninja supports declaring shorter reusable names for strings. A declaration like the following `cflags = -g` can be used on the right side of an equal sign, dereferencing it with a dollar sign, like this: ```Console rule cc command = gcc $cflags -c $in -o $out ``` Variables can also be referenced using curly braces like `${in}`. Variables might better be called "bindings", in that a given variable cannot be changed, only shadowed. #### Rules Rules declare a short name for a command line. Rules begin with a line consisting of the `rule` keyword and a name for the rule. Then follows an indented set of variable = value lines. The basic example above declares a new rule named `cc`, along with the command to run. In the context of a rule, the command variable defines the command to run, `$in` expands to the list of input files (`foo.c`), and `$out` to the output files (`foo.o`) for the command. #### Build statements Build statements declare a relationship between input and output files. They begin with the Build keyword and have the format build outputs: rulename inputs. Such a declaration says that all of the output files are derived from the input files. When the output files are missing or when the inputs change, Ninja will run the rule to regenerate the outputs. The basic example above describes how to build foo.o, using the cc rule. In the scope of a build block (including in the evaluation of its associated rule), the variable $in is the list of inputs and the variable $out is the list of outputs. A build statement may be followed by an indented set of key = value pairs, much like a rule. These variables will shadow any variables when evaluating the variables in the command. For example: ```Console cflags = -Wall -Werror rule cc command = gcc $cflags -c $in -o $out #If left unspecified, builds get the outer $cflags. build foo.o: cc foo.c # But you can shadow variables like cflags for a particular build. build special.o: cc special.c cflags = -Wall # The variable was only shadowed for the scope of special.o; # Subsequent build lines get the outer (original) cflags. build bar.o: cc bar.c ``` ### Make vs. Meson vs. Ninja Make, Meson, and Ninja are not all exactly the same, but they share the fact they are all used in software development to automate the process of compiling source code into binaries. As it happens, each of these build systems has its own set of advantages and disadvantages. Make is the oldest and most widely used one, and while simple and easy to understand, it can become cumbersome and difficult to manage for larger projects. Additionally, Make has limited support for parallel builds, which can significantly slow down the build process. Meson tries to address some of the limitations of Make. It features a more intuitive syntax and supports more complex build scenarios, such as cross-compiling and testing. Meson also has good support for parallel builds, which can significantly speed up the build process. Meson has a steeper learning curve than Make and may require more configuration to get started. Ninja is fast and lightweight and designed for large projects with many source files. Ninja is particularly well-suited for projects that require frequent rebuilds, as it can quickly determine which parts of the codebase need to be rebuilt. Ninja supports parallel builds, making it a popular choice for large-scale software development projects. It is quite common to find Ninja nowadays in projects with more than 200 or 300K lines of code. In summary, Make is the simplest and the founding father of build systems. It's still quite popular, but tricky larger projects. Meson is more modern and offers a more intuitive syntax and better support for complex build scenarios. Ninja is lightweight and conceived for large projects with many sources that require frequent rebuilds. All in all, build systems are essential parts of the code comprehension activity for they may directly impact the content of the source code under analysis. ## Software Architecture Software is, at the most fundamental level, a mere collection of instructions put in some memory, waiting to be fetched, decoded, and executed by a CPU. On the other hand, thanks to the handful of [[Laws, Adages, Principles and Effects#Leaky Abstractions|abstractions]] we have created to make it more intuitive to develop software compared to writing machine code, software can describe component-like and network-like structures. In short, software can have an architecture. The key is: how do we define such architecture? As it is commonly said, there is more than one way to skin a cat. When it comes to software architecture, engineers may come up with different approaches to organizing the way the different entities of a certain software interact with each other. As source code analysts and rearchitects, we need to understand the underlying design choices made by those who preceded us and whom we will probably never have the pleasure of meeting, and ask questions. Software architecture describes the high-level design principles of a system, but when we are reading source code—just words and statements— these principles might be a tad difficult to visualize. Still, software engineers do not reinvent new ways of structuring software so easily. Certain patterns or approaches have been devised decades ago and are replicated all around. These common ways of 'skinning the cat' are the so-called architectural styles. The architectural styles form principles of software design in the same way as building architecture shapes the style of a building (e.g., Gothic style). In software design, we distinguish between several of such styles. Additionally, some architectural styles are close siblings. For instance, Microservices can also be thought of as Component-based, or even as a microkernel. The line easily blurs, and software may combine some of these styles in a sort of architectural cocktail that any code analyst must be able to take a sip. The typical architectural styles out there are: - Monolithic - Microservices - Layered - Component-based (a.k.a "The AppStore") - Pipes and Filters - Front-End and Back-End - Client-Server - Publisher-Subscriber - Event-Driven - Middleware - Service-based - Common lib ### Monolith The monolith is a single, chubby functional brick. Mind you: all monoliths still must have internal architecture, unless we are talking about one single spaghetti, bare-metal function doing absolutely everything, which is rarely the case. This style refers more to the high coupling and high complexity nature of the design. Monoliths are easier to comprehend but particularly difficult to deal with because refactoring them requires extensive modifications across their code base: you move one variable here, and the whole thing comes down crashing. The monolithic architecture is often chosen for implementing safety-critical software. In monoliths, and in the name of safety and reliability, it is common to find extensive amounts of global variables, static allocation, no memory or garbage management, and no dynamic structures. On the upside, monoliths are easier to manage, debug, and analyze. If they crash, you just bring the whole thing back up, and off you go. We will see further ahead why this is actually a good advantage for reliability and availability, as opposed to distributed architectures. ### Microservices A microservice architecture is a variant of the service-oriented architecture structural style we will see. It is an architectural pattern that arranges an application as a collection of loosely coupled, self-contained, fine-grained "services" (it could be processes, or threads of execution, definitions vary), communicating through lightweight protocols. One of its theoretical goals is that teams can develop and deploy their services independently of others. This is achieved by the reduction of dependencies in the code base, allowing for developers to evolve their services with limited restrictions from users, and for additional complexity to be hidden from users. As a consequence, organizations are theoretically able to develop software faster also by using off-the-shelf, third-party services more easily. These benefits come at a cost to maintaining such decoupling; interfaces need to be designed carefully and treated as a public, well-documented API of sorts.  > [!Figure] > _Microservices architectural style_ It is somewhat obvious, but still important to note, that all services in a microservice architecture still share an underlying set of other services and infrastructure which makes it possible for the services to exist in the first place. If the services run in a Linux operating system, they will still use the same system calls, filesystems, and scheduling mechanisms, therefore true decoupling is always somewhat utopic. Microservices could be also running in [[Semiconductors#Containers|containers]], in which case, should the number of microservices grow beyond some manageable limits, the architecture will require coordination and orchestration utilizing specialized tools. Needless to say, microservices bring reliability challenges galore; managing to monitor that services do not crash, or if they crash, to respawn them while making sure the system does not perceive any glitch, is no small task. In general, failover and reliability matters are pushed into the DevOps realm, where measures like failover clusters are deployed; basically adding redundancy to the system. ### Layered Software can be thought of as a Jenga tower of sorts. This is, a "stack" of things resting on top of each other, like books on a table. But the layered architectural style is a little bit more than just inanimate objects piled up because the pieces of the "Jenga" must talk to each other and exchange relevant information at runtime. A known example of this layered architecture is the AUTOSAR[^85] (AUTomotive Open System Architecture). AUTOSAR is a development partnership of automotive interested parties founded in 2003, and it pursues the objective of creating and establishing an open and standardized software architecture for automotive electronic control units (ECUs). AUTOSAR's goals include the scalability of different vehicle and platform variants, transferability of software, the consideration of availability and safety requirements, collaboration between various partners, sustainable use of natural resources, and maintainability during the product lifecycle.  > [!Figure] > _Figure 3‑222 AUTOSAR architecture_ As usually happens with any categorization or convention, the layered model tends to present a high number of variations, "gray areas" and overlaps between the boundaries of the different layers. ### Component-Based This architectural style postulates the principle that all components in the software can be interchangeable and independent of each other. All communication goes through well-defined public interfaces and each component implements a simple interface, allowing for queries about which interfaces are implemented by the component. This approach is also somewhat stupidly referred to as "the AppStore", borrowing the term from the familiar digital marketplaces where one can select, download, install, or uninstall applications without disrupting the work of our smartphones or computers. In general, when it comes to mission-critical software, the "AppStore" concept tends to fall short of its promises and end up being more PR phantasmagoria than anything tangible, mainly because the components that form such systems are not decoupled enough to enable such modularity. ### Pipes and Filters "Pipes and filters" is another architectural style that sometimes appears named as "Data Flow" or "Processing Chain", or similar naming. This pattern fits well for systems that operate based on data processing, for example, radars. This architectural style postulates that the components are connected along the flow of the data processing, which is conceptually shown in the figure below. In contemporary software, and besides military radars, this architectural style is popular in areas such as image recognition in autonomous vehicles, where large quantities of video data need to be processed in multiple stages and each component has to be independent of the other or, what is more, bypassed in runtime.  > [!Figure] > _Pipes and filters is usually depicted as a chain_ ### Front End and Back End This feels like the layered style, only that it has two very solid, clear layers: the front end and the back end. What is the deal with these two? As we saw before, in software architecture, there may be many layers between the underlying hardware and the end user. The front part is an abstraction in charge of providing a user-friendly interface for a human user, while the back end usually handles data storage, numerical computation (if any), and business logic. Using aircraft as an analogy, the front end would be the cockpit (what the user, or the pilot in this case, sees and interacts with), whereas the rest of the system (avionics, structure, hydraulics, etc.) is the back end which can only be accessed from the cockpit. It is very easy for this architectural style to quickly morph into one of the other styles in this section, depending on the communication method between the front and the back. For example, if the front end communicates to the back using some sockets, the architecture becomes client-server (see next one). If the front end communicates with the back end using an API, it becomes a middleware thing. ### Client-Server In the client-server architectural style, the principles of the design of such systems prescribe the decoupling between entities with designated roles; "servers" that are waiting for connection requests and provide resources upon the request of the clients. These requests can be done in either the pull or the push manner. Pulled requests mean that the responsibility for querying the server lies with the client, which means that the clients need to monitor changes in resources provided by the server. Pushed requests mean that the server notifies the relevant clients about changes in the resources. In client-server architected systems, there is a "dialect" between the serves and clients which must be agreed upon beforehand at the design stage, otherwise the entities will not be able to engage in communication. In certain designs, one server can provide services to one or many more clients, although each connection represents an expenditure of resources and a potential decrease in the server's performance due to this workload. Also, the protocols chosen between clients and servers can be "conversational", that is, protocols requiring acknowledging of every message back and forth, or can be "broadcasting", in the sense that messages are spouted to broader audiences and only those who are supposed to reply to such messages will react; these are all design decisions, and they vary depending on the context and the type of application.  > [!Figure] > _Client-Server architecture_ Note that in client-server architectures, there is no need for a common computing environment between the two. There could be an air gap between them, with many other routing resources in between, and still be able to communicate. ### Publisher-Subscriber The publisher-subscriber architectural style can be seen as a special case of the client-server style, although it is often perceived as a different style. This style postulates the principle of loose coupling between providers (publishers) of the information and users (subscribers) of the information. Subscribers subscribe to a central pool of information to get notifications about changes in the information. The publisher does not know the subscribers and the responsibility of the publisher is only to update the information. This is in clear contrast to the client-server architecture, where the server sends the information directly to a known client (known as it is the client that sends the request). Nothing too different—hence the name—to subscribing to a newsletter or a magazine: inform me when there is a new issue, and that's all I need to know to carry on. ### Event-Driven The event-driven architectural style has been popularized in software engineering together with graphical user interfaces and the use of buttons, text fields, labels, and other graphical elements. This architectural style postulates that the components listen for (hook into) the events that are sent from the component to the operating system. The listener components react upon receipt of the event and process the data that has been sent together with the event (e.g., the position of the mouse pointer on the screen when clicked). ### Middleware The middleware architectural style is based on the idea of the existence of a common request 'broker' which mediates the usage of resources between different components. The middleware pattern is sometimes found as a case of "accidental architecture" in the sense that brokers are not always purposely designed but "grown" to become brokers by inadvertently increasing the centrality of a piece of software in the architecture. By the time you realize you have created an accidental middleware, it is already too late to change it and many other elements of the architecture are addicted to its services. In general, this architectural style tends to be of little use in mission-critical applications considering that the broker represents a massive single point of failure. ### Service-Based Service-oriented architectural style postulates a design pattern where a loosely coupled set of components communicate using internet-based protocols, namely HTTP/2 running on top of TCP. This approach is used in the [[Non-Terrestrial and Mobile Networks#Service-Based Interfaces|5G core network]]. The architectural style emphasizes interfaces that can be accessed as web services or RESTful APIs. Said services can be added and changed on-demand during the runtime of the system. Services might be devised to be cloud-native, although this brings the challenge of vendor lock-in. This architectural pattern is not commonly found in mission-critical applications due to the heavy reliance on IP stacks and the lack of determinism in HTTP requests.  > [!Figure] > _Service-based architecture used in the 3GPP 5G core_ As we discussed when we talked about microservices, the SBA approach suffers the same problems as complexity grows: orchestration costs and reliability concerns when it comes to managing the failover of certain critical services. For instance, while observing the figure above, one thought that comes to mind is: where is the availability of the 5G core handled? As in, who guarantees the services are working as expected and in case of failure of a network function (NF), there will be a respawning or restart executed? 5G NFs are just software: either threads or containerized processes running on top of a computing environment, whether virtual or physical. Therefore, they may (and will) crash. If a UPF NF crashes, the user data flow will be greatly affected, so who's in charge of bringing back a crashed UPF and, moreover, of bringing it back maintaining the configuration the failed one had? It seems this is just pushed to DevOps to do Kubernetes magic to keep the whole thing running. Somehow, one wants to believe that something like a 5G core would have availability built into its design. Genuinely curious how 5G can achieve Ultra-Reliable, Low-latency communication (URLLC) with such an approach. ### Common Library This architectural style finds extensive use in distributed architectures where several nodes or components in an architecture share the same processor type and which allows for an easier sharing of software functionalities. Then, the idea is to create one common library that will contain functionalities all other elements will use. In this model, a set of reusable functions, routines, or classes, typically serving a specific set of common tasks, is developed as a library. This library is then used across multiple, different applications within an organization or software ecosystem. This approach has several advantages. Firstly, it promotes code reuse, which is a fundamental principle in software engineering. By having a common set of functionalities in a shared library, developers avoid the need to write the same code multiple times for different applications, leading to more efficient use of resources and time. Secondly, this can lead to better maintainability. Updates, bug fixes, or improvements made in the shared library benefit all the applications that use it. This centralized maintenance can lead to more robust and reliable codebases. Additionally, using a common library can improve consistency across applications. Since all applications use the same underlying functions, it ensures a level of uniformity in how certain tasks are performed or how data is handled. This is particularly beneficial in large organizations where disparate teams work on different applications but need to ensure some level of standardization. However, this approach also comes with its downsides. One significant issue is the risk of tight coupling. Applications dependent on a shared library can become so intertwined with it that any change in the library can have far-reaching impacts, potentially breaking functionality in the applications. This necessitates careful management and versioning of the shared library. Another challenge is the one-size-fits-all problem. A shared library, by its nature, aims to address a general set of requirements. However, individual applications might have specific needs that the shared library does not cater to efficiently. This can lead to either the library becoming bloated with application-specific features or applications having to implement workarounds. Moreover, the shared library's development cycle can become a bottleneck. If multiple teams depend on a new feature or fix in the shared library, they might be stalled until the library's update is released. This can slow down the development process, especially in large organizations where coordination and release cycles can be complex. ## 11 Software Postulates 1. The absolute minimum complexity of software that controls a physical system is given by the complexity of the underlying physical system.  2. The combined minimum complexity of frontend and backend is a conserved quantity. You can make the front end simpler by making the back end more complicated, and vice versa. But you can always over-engineer both. 3. The higher the complexity of the underlying physical system to be controlled, the lower the fidelity of the software development environment and the higher the dependence on [[Co-Simulation#Accidental Co-Simulation|disaggregated simulation]]. 4. The only testing that matters is testing at the coarsest level of granularity possible within a given functional boundary. 5. The moment you go to read the licensing details of an open-source library or framework marks the moment when it is too late to replace it and therefore the moment you are screwed. 6. Good architectures do not magically emerge bottom-up. 7. Coding style is a first-world problem. It is a problem that only becomes relevant when other more important problems are no longer relevant.  8. Code is the best modeling language. 9. Documents as milestones mark the first step of a software project death march. 10. Software that doesn’t crash is the best measure of quality. Software that doesn’t crash during a demo is the ultimate measure of quality. 11. Badly architected software may still pass all tests. [^6]: Nedhal A. Al-Saiyd “Source Code Comprehension Analysis in Software Maintenance”, Computer Science Department, Faculty of Information Technology Applied Science Private University, Amman-Jordan [^7]: M. A. Storey “Theories, Methods and Tools in Program Comprehension: Past, Present and Future”, Software Quality Journal 14, pp. 187–208, DOI 10.1007/s11219-006-9216-4.