Android编译原理之make编译过程

一、make

以高通的代码为例，高通在编译时使用如下的指令进行编译，实质是对make的封装

./build.sh --target_only

在执行一系列check的函数后，最终调用make执行执行编译。而make函数是在build/envsetup.sh时声明。

从get_make_command函数可知，make后，真正然后执行编译的入口是：build/soong/soong_ui.bash

二、soong_ui.bash

1. source microfactory.bash，得到一些函数命令， 例如：soong_build_go
2. 编译/build/soong/cmd/soong_ui/main.go，生成 out/soong_ui这个可执行程序
3. 执行命令：out/soong_ui --make-mode ，执行了make命令，会把"build/make/core/main.mk" 加到构建环境中，同时启动kati、blueprint-soong、ninja的编译。

function gettop
{
    local TOPFILE=build/soong/root.bp
    if [ -n "${TOP-}" -a -f "${TOP-}/${TOPFILE}" ] ; then
        # The following circumlocution ensures we remove symlinks from TOP.
        (cd $TOP; PWD= /bin/pwd)
    else
        if [ -f $TOPFILE ] ; then
            # The following circumlocution (repeated below as well) ensures
            # that we record the true directory name and not one that is
            # faked up with symlink names.
            PWD= /bin/pwd
        else
            local HERE=$PWD
            T=
            while [ \\( ! \\( -f $TOPFILE \\) \\) -a \\( $PWD != "/" \\) ]; do
                \\cd ..
                T=`PWD= /bin/pwd -P`
            done
            \\cd $HERE
            if [ -f "$T/$TOPFILE" ]; then
                echo $T
            fi
        fi
    fi
}

# Save the current PWD for use in soong_ui
export ORIGINAL_PWD=${PWD}
export TOP=$(gettop)
source ${TOP}/build/soong/scripts/microfactory.bash

soong_build_go soong_ui android/soong/cmd/soong_ui
soong_build_go mk2rbc android/soong/mk2rbc/cmd
soong_build_go rbcrun rbcrun/cmd

cd ${TOP}
exec "$(getoutdir)/soong_ui" "$@"

2.1 build/soong/scripts/microfactory.bash

build/soong/scripts/microfactory.bash

microfactory.bash得到build_go的函数命令,并提供 soong_build_go的函数执行方法并继续调用

build/blueprint/microfactory/microfactory.bash

GOROOT=prebuilts/go/linux-x86

2.2 build/blueprint/microfactory/microfactory.bash

build_go函数做了以下事情：

1. 第一次执行编译时调用go run out/.microfactory_Linux/intermediates/src/microfactory.go编译出

microfactory_linux，第二次执行会执行检查，如果已编译则跳过go run

1. 通过microfactory_Linux编译出soong_ui

soong_build_go soong_ui android/soong/cmd/soong_ui

• build_go soong_ui android/soong/cmd/soong_ui

实际调用逻辑为：

GOROOT=$(cd /prebuilts/go/linux-x86/; pwd) /out/microfactory_Linux
  -b "/out/microfactory_Linux" \\
  -pkg-path "github.com/google/blueprint=/build/blueprint" \\
  -trimpath "./" \\
  -pkg-path android/soong=/build/soong
  -pkg-path github.com/golang/protobuf=/external/golang-protobuf} \\
  -o "out/soong_ui" android/soong/cmd/soong_ui

从这儿可以知道soong_build_go soong_ui android/soong/cmd/soong_ui的意思就是从soong/cmd/soong_ui/下的go文件编译生成soong_ui

2.3 soong_ui的源文件main.go

soong_ui 是通过编译 build/soong/cmd/soong_ui/main.go得来

主要执行soong/ui/build/build.go，从build.go就可以看到执行soong的大体流程。

main.go中配置的toBuild为 BuildProductConfig | BuildSoong | BuildKati | BuildNinja，支持productconfig\soong\kati\ninja的构建

2.4 build.go流程

1. runMakeProductConfig 主要配置编译参数
2. runSoong 对工具进行编译，编译出blueprint等编译工具, 把*.bp 编译成 out/soong/build.ninja
3. runKatiBuild, 加载 build/make/core/main.mk， 搜集所有的Android.mk文件生成ninja文件:out/build-${product}.ninja
4. runKatiPackage, 加载build/make/packaging/main.mk, 编译生成out/build-aosp_arm-package.ninja
5. createCombinedBuildNinjaFile,将out/soong/build.ninja 、out/build-aosp_arm.ninja和out/build-aosp_arm-package.ninja， 合成为out/combined-aosp_arm.ninja
6. runNinja，运行Ninja命令， 解析combined-aosp_arm.ninja，执行编译过程

2.4.1 runMakeProductConfig

build/soong/ui/build/dumpvars.go

只是配置编译需要的参数

2.4.2 runSoong

主要功能：

1. 对工具进行编译，编译出blueprint等编译工具
2. 把*.bp 编译成 out/soong/build.ninja

2.4.2.1 Android R版本

runSoong()的执行过程：

1. 执行build/blueprint/bootstrap.bash 生成.minibootstrap/build.ninja 和.bootstrap/build.ninja
2. 生成minibp\bpglob
3. 通过ninja来编译.minibootstrap/build.ninja 和.bootstrap/build.ninja

首先执行build/blueprint/bootstrap.bash

bootstrap.bash的作用：

1. 它可以引导独立的blueprint来生成minibp二进制文件，可以直接运行 ./build/blueprint/bootstrap.bash。
2. 也可以从另一个脚本调用它来引导基于Bleprint的自定义构建系统。

func runSoong(ctx Context, config Config) {
        ctx.BeginTrace(metrics.RunSoong, "soong")
        defer ctx.EndTrace()

        func() {
        // ⽣成out/soong/.minibootstrap/build.ninja
                ctx.BeginTrace(metrics.RunSoong, "blueprint bootstrap")
                defer ctx.EndTrace()

                cmd := Command(ctx, config, "blueprint bootstrap", "build/blueprint/bootstrap.bash", "-t")
                cmd.Environment.Set("BLUEPRINTDIR", "./build/blueprint")
                cmd.Environment.Set("BOOTSTRAP", "./build/blueprint/bootstrap.bash")
                cmd.Environment.Set("BUILDDIR", config.SoongOutDir())
                cmd.Environment.Set("GOROOT", "./"+filepath.Join("prebuilts/go", config.HostPrebuiltTag()))
                cmd.Environment.Set("BLUEPRINT_LIST_FILE", filepath.Join(config.FileListDir(), "Android.bp.list"))
                cmd.Environment.Set("NINJA_BUILDDIR", config.OutDir())
                cmd.Environment.Set("SRCDIR", ".")
                cmd.Environment.Set("TOPNAME", "Android.bp")
                cmd.Sandbox = soongSandbox
// build/blueprint/bootstrap.bash⾥的逻辑很简单，就是创建out/soong/.minibootstrap⽬录，
// 把上⾯⼏个变量写⼊到out/soong/.minibootstrap/build.ninja⾥，
// 最后再写⼊ “include ./build/blueprint/bootstrap/build.ninja”
                cmd.RunAndPrintOrFatal()
        }()

        func() {
                ctx.BeginTrace(metrics.RunSoong, "environment check")
                defer ctx.EndTrace()

                envFile := filepath.Join(config.SoongOutDir(), ".soong.environment")
                envTool := filepath.Join(config.SoongOutDir(), ".bootstrap/bin/soong_env")
                if _, err := os.Stat(envFile); err == nil {
                        if _, err := os.Stat(envTool); err == nil {
                                cmd := Command(ctx, config, "soong_env", envTool, envFile)
                                cmd.Sandbox = soongSandbox

                                var buf strings.Builder
                                cmd.Stdout = &buf
                                cmd.Stderr = &buf
                                if err := cmd.Run(); err != nil {
                                        ctx.Verboseln("soong_env failed, forcing manifest regeneration")
                                        os.Remove(envFile)
                                }

                                if buf.Len() > 0 {
                                        ctx.Verboseln(buf.String())
                                }
                        } else {
                                ctx.Verboseln("Missing soong_env tool, forcing manifest regeneration")
                                os.Remove(envFile)
                        }
                } else if !os.IsNotExist(err) {
                        ctx.Fatalf("Failed to stat %f: %v", envFile, err)
                }
        }()

        var cfg microfactory.Config
        cfg.Map("github.com/google/blueprint", "build/blueprint")

        cfg.TrimPath = absPath(ctx, ".")

        func() {
// 编译minibp程序， minibp跟后⾯要讲到的soong_build都是基于Blueprint框架构建的,
// ⽤来解析Android.bp⾥定义的各种构建单元的。
// 它们俩其实就只有⼀个默认参数不同，即minibp在解析Android.bp时若遇到未知的构建类型，
// 不会报错，即它只会从Android.bp⾥解析出blueprint框架⾥预定义的⼏种构建类型，⽣成对应的
// ninja编译规则，例如minibp会解析build/soong/Aosp下所有Android.bp⾥的bootstrap_go_package;
// 而soong_build是⽤来最终编译整个rom的所有编译命令的，所以它不允许未定义的构建类型。
                ctx.BeginTrace(metrics.RunSoong, "minibp")
                defer ctx.EndTrace()

                minibp := filepath.Join(config.SoongOutDir(), ".minibootstrap/minibp")
                if _, err := microfactory.Build(&cfg, minibp, "github.com/google/blueprint/bootstrap/minibp"); err != nil {
                        ctx.Fatalln("Failed to build minibp:", err)
                }
        }()

        func() {
// 编译bpglob程序，⽤于后⾯解析Android.bp⾥定义的 glob(⽂件路径名模式匹配) 规则，查找与之匹配的⽂件
// 例如 miui/frameworks/base/Android.bp⾥定义的`core/java/com/miui/internal/**/*.java`就
// 匹配 miui/frameworks/base/core/java/com/miui/internal⽬录下和其⼦⽬录（⽆限递归！）下所有的java⽂件
                ctx.BeginTrace(metrics.RunSoong, "bpglob")
                defer ctx.EndTrace()

                bpglob := filepath.Join(config.SoongOutDir(), ".minibootstrap/bpglob")
                if _, err := microfactory.Build(&cfg, bpglob, "github.com/google/blueprint/bootstrap/bpglob"); err != nil {
                        ctx.Fatalln("Failed to build bpglob:", err)
                }
        }()

        ninja := func(name, file string) {
                ctx.BeginTrace(metrics.RunSoong, name)
                defer ctx.EndTrace()

                fifo := filepath.Join(config.OutDir(), ".ninja_fifo")
                nr := status.NewNinjaReader(ctx, ctx.Status.StartTool(), fifo)
                defer nr.Close()

                cmd := Command(ctx, config, "soong "+name,
                        config.PrebuiltBuildTool("ninja"),
                        "-d", "keepdepfile",
                        "-w", "dupbuild=err",
                        "-j", strconv.Itoa(config.Parallel()),
                        "--frontend_file", fifo,
                        "-f", filepath.Join(config.SoongOutDir(), file))
                cmd.Environment.Set("SOONG_SANDBOX_SOONG_BUILD", "true")
                cmd.Sandbox = soongSandbox
                //启动ninja程序
                cmd.RunAndStreamOrFatal()
        }
}
// 启动ninja⽣成out/soong/.minbootstrap/build.ninja⾥定义的默认⽬标，实际就是执⾏minibp程序，
// 解析aosp源码⾥所有的Android.bp，从中得到后⾯⽤于处理不同类型构建类型的插件模块的构建⽅式(后⾯会再提到)，
// 并将它们汇总输出到out/soong/.bootstrap/build.ninja
        ninja("minibootstrap", ".minibootstrap/build.ninja")
// 启动ninja⽣成out/soong/.bootstrap/build.ninja⾥定义的默认⽬标，也就是out/soong/build.ninja
        ninja("bootstrap", ".bootstrap/build.ninja")
}

2.4.2.2 Android T版本

func runSoong(ctx Context, config Config) {
        ctx.BeginTrace(metrics.RunSoong, "soong")
        defer ctx.EndTrace()

        // We have two environment files: .available is the one with every variable,// .used with the ones that were actually used. The latter is used to// determine whether Soong needs to be re-run since why re-run it if only// unused variables were changed?
        envFile := filepath.Join(config.SoongOutDir(), availableEnvFile)

        buildMode := config.bazelBuildMode()
        integratedBp2Build := buildMode == mixedBuild// This is done unconditionally, but does not take a measurable amount of time
        bootstrapBlueprint(ctx, config)

        soongBuildEnv := config.Environment().Copy()
        soongBuildEnv.Set("TOP", os.Getenv("TOP"))
        // For Bazel mixed builds.
        soongBuildEnv.Set("BAZEL_PATH", "./tools/bazel")
        soongBuildEnv.Set("BAZEL_HOME", filepath.Join(config.BazelOutDir(), "bazelhome"))
        soongBuildEnv.Set("BAZEL_OUTPUT_BASE", filepath.Join(config.BazelOutDir(), "output"))
        soongBuildEnv.Set("BAZEL_WORKSPACE", absPath(ctx, "."))
        soongBuildEnv.Set("BAZEL_METRICS_DIR", config.BazelMetricsDir())
        soongBuildEnv.Set("LOG_DIR", config.LogsDir())

        // For Soong bootstrapping testsif os.Getenv("ALLOW_MISSING_DEPENDENCIES") == "true" {
                soongBuildEnv.Set("ALLOW_MISSING_DEPENDENCIES", "true")
        }

        err := writeEnvironmentFile(ctx, envFile, soongBuildEnv.AsMap())
        if err != nil {
                ctx.Fatalf("failed to write environment file %s: %s", envFile, err)
        }

        func() {
                ctx.BeginTrace(metrics.RunSoong, "environment check")
                defer ctx.EndTrace()

                checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(soongBuildTag))

                if integratedBp2Build || config.Bp2Build() {
                        checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(bp2buildTag))
                }

                if config.JsonModuleGraph() {
                        checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(jsonModuleGraphTag))
                }

                if config.Queryview() {
                        checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(queryviewTag))
                }

                if config.SoongDocs() {
                        checkEnvironmentFile(soongBuildEnv, config.UsedEnvFile(soongDocsTag))
                }
        }()

        runMicrofactory(ctx, config, "bpglob", "github.com/google/blueprint/bootstrap/bpglob",
                map[string]string{"github.com/google/blueprint": "build/blueprint"})

        ninja := func(name, ninjaFile string, targets ...string) {
                ctx.BeginTrace(metrics.RunSoong, name)
                defer ctx.EndTrace()

                fifo := filepath.Join(config.OutDir(), ".ninja_fifo")
                nr := status.NewNinjaReader(ctx, ctx.Status.StartTool(), fifo)
                defer nr.Close()

                ninjaArgs := []string{
                        "-d", "keepdepfile",
                        "-d", "stats",
                        "-o", "usesphonyoutputs=yes",
                        "-o", "preremoveoutputs=yes",
                        "-w", "dupbuild=err",
                        "-w", "outputdir=warn",
                        "-w", "missingoutfile=warn",
                        "-j", strconv.Itoa(config.Parallel()),
                        "--frontend_file", fifo,
                        "-f", filepath.Join(config.SoongOutDir(), ninjaFile),
                }

                ninjaArgs = append(ninjaArgs, targets...)
                cmd := Command(ctx, config, "soong "+name,
                        config.PrebuiltBuildTool("ninja"), ninjaArgs...)

                var ninjaEnv Environment// This is currently how the command line to invoke soong_build finds the// root of the source tree and the output root
                ninjaEnv.Set("TOP", os.Getenv("TOP"))

                qcEnvVars := []string{
                        "TARGET_BOARD_PLATFORM",
                        "SDCLANG_AE_CONFIG",
                        "SDCLANG_CONFIG",
                        "SDCLANG_SA_ENABLED",
                        "QIIFA_BUILD_CONFIG",
                }
                for _, qcVar := range qcEnvVars {
                        ninjaEnv.Set(qcVar, os.Getenv(qcVar))
                }

                cmd.Environment = &ninjaEnv
                cmd.Sandbox = soongSandbox
                cmd.RunAndStreamOrFatal()
        }

        targets := make([]string, 0, 0)

        if config.JsonModuleGraph() {
                targets = append(targets, config.ModuleGraphFile())
        }

        if config.Bp2Build() {
                targets = append(targets, config.Bp2BuildMarkerFile())
        }

        if config.Queryview() {
                targets = append(targets, config.QueryviewMarkerFile())
        }

        if config.SoongDocs() {
                targets = append(targets, config.SoongDocsHtml())
        }

        if config.SoongBuildInvocationNeeded() {
                // This build generates <builddir>/build.ninja, which is used later by build/soong/ui/build/build.go#Build().
                targets = append(targets, config.SoongNinjaFile())
        }

        ninja("bootstrap", "bootstrap.ninja", targets...)

        var soongBuildMetrics *soong_metrics_proto.SoongBuildMetricsif shouldCollectBuildSoongMetrics(config) {
                soongBuildMetrics := loadSoongBuildMetrics(ctx, config)
                logSoongBuildMetrics(ctx, soongBuildMetrics)
        }

        distGzipFile(ctx, config, config.SoongNinjaFile(), "soong")

        if !config.SkipKati() {
                distGzipFile(ctx, config, config.SoongAndroidMk(), "soong")
                distGzipFile(ctx, config, config.SoongMakeVarsMk(), "soong")
        }

        if shouldCollectBuildSoongMetrics(config) && ctx.Metrics != nil {
                ctx.Metrics.SetSoongBuildMetrics(soongBuildMetrics)
        }
        if config.JsonModuleGraph() {
                distGzipFile(ctx, config, config.ModuleGraphFile(), "soong")
        }
}

2.4.3 runKatiBuild

主要功能：

1. 加载 build/make/core/main.mk
2. 所有的Android.mk文件生成ninja文件:out/build-aosp_arm.ninja

func runKatiBuild(ctx Context, config Config) {
        ctx.BeginTrace(metrics.RunKati, "kati build")
        defer ctx.EndTrace()

        args := []string{
                // Mark the output directory as writable.
                "--writable", config.OutDir() + "/",
                // Fail when encountering implicit rules. e.g.
                // %.foo: %.bar
                //   cp $< $@
                "--werror_implicit_rules",
                // Entry point for the Kati Ninja file generation.
                "-f", "build/make/core/main.mk", //⼊口mk⽂件，kati会从它开始解析
        }

        if !config.BuildBrokenDupRules() {
                // Fail when redefining / duplicating a target.
                args = append(args, "--werror_overriding_commands")
        }

        args = append(args, config.KatiArgs()...)
//导出下⾯4个环境变量给kati进程，这些变量在build/make/core/main.mk以及被他依赖的mk⽂件会被使⽤到
// 例如SOONG_ANDROID_MK会在build/make/core/main.mk⾥被include，
// build/make/core/main.mk也会⾃动include前⾯FindSources()步骤⾥⽣成的out/.module_paths/Android.
        args = append(args,
                // Location of the Make vars .mk file generated by Soong.
                "SOONG_MAKEVARS_MK="+config.SoongMakeVarsMk(),
                // Location of the Android.mk file generated by Soong. This
                // file contains Soong modules represented as Kati modules,
                // allowing Kati modules to depend on Soong modules.
                "SOONG_ANDROID_MK="+config.SoongAndroidMk(),
                // Directory containing outputs for the target device.
                "TARGET_DEVICE_DIR="+config.TargetDeviceDir(),
                // Directory containing .mk files for packaging purposes, such as
                // the dist.mk file, containing dist-for-goals data.
                "KATI_PACKAGE_MK_DIR="+config.KatiPackageMkDir())

        runKati(ctx, config, katiBuildSuffix, args, func(env *Environment) {})

        // compress and dist the main build ninja file.
        distGzipFile(ctx, config, config.KatiBuildNinjaFile())

        // Cleanup steps.
        cleanCopyHeaders(ctx, config)
        cleanOldInstalledFiles(ctx, config)
}

runKati中会启动Kati程序，开始解析main.mk

$(info [1/1] initializing build system ...)
#mk⾥定义的第⼀个⽬标，即默认⽬标
.PHONY: droid
DEFAULT_GOAL := droid
$(DEFAULT_GOAL): droid_targets
.PHONY: droid_targets
droid_targets:
# Set up various standard variables based on configuration
# and host information.
include build/make/core/config.mk
include $(SOONG_MAKEVARS_MK)
...
# SOONG_ANDROID_MK即out/soong/Android-{product}.mk

subdir_makefiles := $(SOONG_ANDROID_MK) $(file
<$(OUT_DIR)/.module_paths/Android.mk.list)
$(SOONG_OUT_DIR)/late-$(TARGET_PRODUCT).mk
# 统计“int $(subdir_makefiles) post finish”⾥`单词`的个数
subdir_makefiles_total := $(words int $(subdir_makefiles) post finish)
# 将out/soong/Android-{product}.mk和out/.module_paths/Android.mk.list⾥所有的Android.mk都包含进来
$(foreach mk,$(subdir_makefiles),$(info [$(call
inc_and_print,subdir_makefiles_inc)/$(subdir_makefiles_total)] including $(mk)
...)$(eval include $(mk)))
...
$(info [$(call inc_and_print,subdir_makefiles_inc)/$(subdir_makefiles_total)]
writing build rules ...)

最终是调用系统准备好的prebuilts/build-tools/linux-x86/bin/ckati参与编译，其中传入的参数有 --ninja\ --regen\--detect_android_echo 等，kati解析完build/make/core/main.mk后，就会⽣成相应的ninja编译规则，runKatiBuild()函数最后⽣成out/build-{product}.ninja⽂件

2.4.4 runKatiPackage

也是运行runKati，只不过携带的参数不同，runKatiPackage函数最后会生成out/build-{product}-package.ninja

2.4.5 createCombinedBuildNinjaFile

会将kati和soong_build两个程序⽣成的ninja⽂件都包含进来，再⽣成⼀个out/combined-${product}.ninja⽂件

2.4.6 runNinja

启动ninja程序解析out/combined-${product}.ninja⽂件，得到我们执⾏的命令⾥指定target-files-package模块的依赖⽂件,最后只要按照拓扑顺序挨个执⾏他们的rule去⽣成他们就可以了