ETCD数据库源码分析——protobuf bingdings

该篇博客的目标就是从scripts/genproto.sh脚本中看出ETCD数据库有哪些地方需要protobuf

环境变量

配置与grpc和protobuf相关GO环境变量

GOFAST_BIN=$(tool_get_bin github.com/gogo/protobuf/protoc-gen-gofast)
GRPC_GATEWAY_BIN=$(tool_get_bin github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway)
SWAGGER_BIN=$(tool_get_bin github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger)
GOGOPROTO_ROOT="$(tool_pkg_dir github.com/gogo/protobuf/proto)/.."
GRPC_GATEWAY_ROOT="$(tool_pkg_dir github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway)/.."
GOGOPROTO_PATH="${GOGOPROTO_ROOT}:${GOGOPROTO_ROOT}/protobuf"

目标

DIR变量包含了需要构建protos的目录：./server/wal/walpb ./api/etcdserverpb ./server/etcdserver/api/snap/snappb ./raft/raftpb ./api/mvccpb ./server/lease/leasepb ./api/authpb ./server/etcdserver/api/v3lock/v3lockpb ./server/etcdserver/api/v3election/v3electionpb ./api/membershippb，共计10个文件夹。
如何编译，主要是调用命令protoc，然后

for dir in ${DIRS}; do
  run pushd "${dir}"
    run protoc --gofast_out=plugins=grpc:. -I=".:${GOGOPROTO_PATH}:${ETCD_ROOT_DIR}/..:${ETCD_ROOT_DIR}:${GRPC_GATEWAY_ROOT}/third_party/googleapis" \
      --plugin="${GOFAST_BIN}" ./*.proto
    run sed -i.bak -E 's|"etcd/api/|"go.etcd.io/etcd/api/v3/|g' ./*.pb.go
    run sed -i.bak -E 's|"raft/raftpb"|"go.etcd.io/etcd/raft/v3/raftpb"|g' ./*.pb.go
    rm -f ./*.bak
    run gofmt -s -w ./*.pb.go
    run goimports -w ./*.pb.go
  run popd
done

Golang制定了统一的官方代码风格，并推出gofmt工具（go fmt）来帮助开发人员格式化代码到统一的风格。gofmt是一个命令行工具，会优先读取标准输入，若传入了文件路径则会格式化此文件。若传入是一个目录则会格式化目录中所有的.go文件。若不传入参数则会格式化当前目录下所有.go文件。
goimports工具是Go官方提供的一种工具，它能够为我们自动格式化 Go 语言代码并对所有引入的包进行管理，包括自动增删依赖的包引用、将依赖包按字母序排序并分类。我们在使用Goland IDE的时候，建议使用goimports工具。它具备包依赖管理+gofmt的功能。

对于api/etcdserverpb/rpc server/etcdserver/api/v3lock/v3lockpb/v3lock server/etcdserver/api/v3election/v3electionpb/v3election三个proto文件运行grpc_gateway proto generation，

for pb in api/etcdserverpb/rpc server/etcdserver/api/v3lock/v3lockpb/v3lock server/etcdserver/api/v3election/v3electionpb/v3election; do
  log_callout "grpc & swagger for: ${pb}.proto"
  run protoc -I. \
      -I"${GRPC_GATEWAY_ROOT}"/third_party/googleapis \
      -I"${GOGOPROTO_PATH}" \
      -I"${ETCD_ROOT_DIR}/.." \
      --grpc-gateway_out=logtostderr=true,paths=source_relative:. \
      --swagger_out=logtostderr=true:./Documentation/dev-guide/apispec/swagger/. \
      --plugin="${SWAGGER_BIN}" --plugin="${GRPC_GATEWAY_BIN}" \
      ${pb}.proto
  # hack to move gw files around so client won't include them
  pkgpath=$(dirname "${pb}")
  pkg=$(basename "${pkgpath}")
  gwfile="${pb}.pb.gw.go"

  run sed -i -E "s#package $pkg#package gw#g" "${gwfile}"
  run sed -i -E "s#import \(#import \(\"go.etcd.io/etcd/${pkgpath}\"#g" "${gwfile}"
  run sed -i -E "s#([ (])([a-zA-Z0-9_]*(Client|Server|Request)([^(]|$))#\1${pkg}.\2#g" "${gwfile}"
  run sed -i -E "s# (New[a-zA-Z0-9_]*Client\()# ${pkg}.\1#g" "${gwfile}"
  run sed -i -E "s|go.etcd.io/etcd|go.etcd.io/etcd/v3|g" "${gwfile}"
  run sed -i -E "s|go.etcd.io/etcd/v3/api|go.etcd.io/etcd/api/v3|g" "${gwfile}"
  run sed -i -E "s|go.etcd.io/etcd/v3/server|go.etcd.io/etcd/server/v3|g" "${gwfile}"
  
  run go fmt "${gwfile}"

  gwdir="${pkgpath}/gw/"
  run mkdir -p "${gwdir}"
  run mv "${gwfile}" "${gwdir}"

  swaggerName=$(basename ${pb})
  run mv  Documentation/dev-guide/apispec/swagger/${pb}.swagger.json Documentation/dev-guide/apispec/swagger/"${swaggerName}".swagger.json
done

运行swaggerrun_go_tool github.com/hexfusion/schwag -input=Documentation/dev-guide/apispec/swagger/rpc.swagger.json

生成文档

使用protodoc生成文档

  log_callout "protodoc is auto-generating grpc API reference documentation..."

  run rm -rf Documentation/dev-guide/api_reference_v3.md
  run_go_tool go.etcd.io/protodoc --directories="api/etcdserverpb=service_message,api/mvccpb=service_message,server/lease/leasepb=service_message,api/authpb=service_message" \
    --title="etcd API Reference" \
    --output="Documentation/dev-guide/api_reference_v3.md" \
    --message-only-from-this-file="api/etcdserverpb/rpc.proto" \
    --disclaimer="This is a generated documentation. Please read the proto files for more." || exit 2

  run rm -rf Documentation/dev-guide/api_concurrency_reference_v3.md
  run_go_tool go.etcd.io/protodoc --directories="server/etcdserver/api/v3lock/v3lockpb=service_message,server/etcdserver/api/v3election/v3electionpb=service_message,api/mvccpb=service_message" \
    --title="etcd concurrency API Reference" \
    --output="Documentation/dev-guide/api_concurrency_reference_v3.md" \
    --disclaimer="This is a generated documentation. Please read the proto files for more." || exit 2