Ethereum Source Code Reading

感觉看源码这种事情,还是得自己写点东西记录下,不然代码是看过了,但是代码到底在干嘛还是没能深刻理解。

以下内容是按照我的学习顺序来写的,可能稍微有那么一点跳。

环境

  • 代码:git clone https://github.com/ethereum/go-ethereum 直接从官方仓库克隆下来。

    我看的时候的版本号:ca22d0761bd772d38d229d59b3c72a2e7ca9592c

  • 软件:Vscode

VS Code 代码审计 Tips

  • F12(或者右键—>Go to Definitions)可以直接跳转到函数的定义处

  • ^- ^ : mac上的control-: 数字键0右边的那个键)可以跳回去

  • option + F12可以查阅某个函数的定义段

  • 右键—>Peek—>Peek References可以查阅这个函数的所有引用(这个函数在其他哪些地方出现过)

  • mac上有touch bar就很舒服了:

    Touch Bar Shot 2020-04-20 at 2.26.17 PM

    推荐插件:Nasc VSCode Touchbar

  • 代码审计必备神器:一个大一点的外接显示器(Dell U2720Q

geth入口

在终端中输入命令geth ...会启动geth客户端。

geth is the official command-line client for Ethereum.

入口点在cmd/geth/main.go

main.go首先会进行一些(终端命令flag相关的)变量初始化,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
var (
	// Git SHA1 commit hash of the release (set via linker flags)
	gitCommit = ""
	gitDate   = ""
	// The app that holds all commands and flags.
	app = utils.NewApp(gitCommit, gitDate, "the go-ethereum command line interface")
	// flags that configure the node
	nodeFlags = []cli.Flag{
        ...
    }
    ...
)

并通过app = utils.NewApp(...)新建一个geth 命令行应用(cli,Command Line Interface),不过这个App类型是一个外部的包(跟这个以太坊区块链本身没什么关系,只是为了方便构建这个命令行应用)。

App is the main structure of a cli application.

然后,会先于main()函数去执行init(),进行一些额外的初始化操作:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
func init() {
	// Initialize the CLI app and start Geth
	app.Action = geth
	app.HideVersion = true // we have a command to print the version
	app.Copyright = "Copyright 2013-2020 The go-ethereum Authors"
	app.Commands = []cli.Command{
        ...
    }
    ...
    app.Flags = append(app.Flags, nodeFlags...)
	app.Flags = append(app.Flags, rpcFlags...)
    ...
    app.Before = ...
    app.After = ...
}

主要是对先前new出来的app进行了一些配置,对app的命令列表添加了一些以太坊服务相关的命令,对app的Flags列表添加了一些以太坊服务相关的Flag。

接着,执行main()函数:

1
2
3
4
5
6
func main() {
	if err := app.Run(os.Args); err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}
}

很简洁,就是启动之前配置好的app。

跟进app.Run,会跳到外部包(go/pkg/mod/gopkg.in/urfave/cli.v1@1.20.0/app.go)里的逻辑,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
// Run is the entry point to the cli app. Parses the arguments slice and routes
// to the proper flag/args combination
func (a *App) Run(arguments []string) (err error) {
    // Setup runs initialization code to ensure all data structures are ready for
	// `Run` or inspection prior to `Run`.  It is internally called by `Run`, but
	// will return early if setup has already happened.
	a.Setup()
	...

    // parse flags
	set, err := flagSet(a.Name, a.Flags)
    ...

    // Run default Action
	err = HandleAction(a.Action, context)

	HandleExitCoder(err)
	return err
}

主要是对命令行参数里提供的flag位进行解析设置,例如geth --fast --cache=1024 console ,这里应该就会对--fast之类的flag标志进行解析和设置。函数的最后会执行err = HandleAction(a.Action, context),会调用上面init()中设置的app.Action = geth函数。

Screen Shot 2020-04-20 at 12.37.11 PM


geth函数在cmd/geth/main.go中:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
// geth is the main entry point into the system if no special subcommand is ran.
// It creates a default node based on the command line arguments and runs it in
// blocking mode, waiting for it to be shut down.
func geth(ctx *cli.Context) error {
	if args := ctx.Args(); len(args) > 0 {
		return fmt.Errorf("invalid command: %q", args[0])
	}
	prepare(ctx)
	node := makeFullNode(ctx)
	defer node.Close()
	startNode(ctx, node)
	node.Wait()
	return nil
}

geth函数内部主要有4个操作:

  1. prepare(ctx)
  2. node := makeFullNode(ctx)
  3. startNode(ctx, node)
  4. node.Wait() —> return —> node.close()

一个一个跟进去看:

首先调用prepare(ctx)函数,主要用来prepare manipulates memory cache allowance and setups metric system

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
// prepare manipulates memory cache allowance and setups metric system.
// This function should be called before launching devp2p stack.
func prepare(ctx *cli.Context) {
	// If we're running a known preset, log it for convenience.
	log ...

	// If we're a full node on mainnet without --cache specified, bump default cache allowance
	log ...

    // If we're running a light client on any network, drop the cache to some meaningfully low amount
	log ...

	// Cap the cache allowance and tune the garbage collector
	var mem gosigar.Mem
	// Workaround until OpenBSD support lands into gosigar
	// Check https://github.com/elastic/gosigar#supported-platforms
	if runtime.GOOS != "openbsd" {
		if err := mem.Get(); err == nil {
			allowance := int(mem.Total / 1024 / 1024 / 3)
			if cache := ctx.GlobalInt(utils.CacheFlag.Name); cache > allowance {
				log.Warn("Sanitizing cache to Go's GC limits", "provided", cache, "updated", allowance)
				ctx.GlobalSet(utils.CacheFlag.Name, strconv.Itoa(allowance))
			}
		}
	}
	// Ensure Go's GC ignores the database cache for trigger percentage
	cache := ctx.GlobalInt(utils.CacheFlag.Name)
	gogc := math.Max(20, math.Min(100, 100/(float64(cache)/1024)))

	log.Debug("Sanitizing Go's GC trigger", "percent", int(gogc))
	godebug.SetGCPercent(int(gogc))

	// Start metrics export if enabled
	utils.SetupMetrics(ctx)

	// Start system runtime metrics collection
	go metrics.CollectProcessMetrics(3 * time.Second)
}
  • 先对网络接入点(networkid,主网?测试网?本地?)、节点信息(轻节点(light node)还是全节点(full node)?)进行了log
  • 然后根据节点信息来调整memory cache allowance(缓存大小?),并与Go的GC(garbage collcetor)进行了一个同步 。
  • 最后配置了一下用来做统计的metrics,并开启了一个CollectProcessMetricsgoroutine,用来间歇性地收集关于正在运行进程的metrics (暂时不知道是干啥的)。

然后调用makeFullNode来创建一个节点对象(在ETH里node可以认为是以太坊全网的一个节点,也可以认为是一个以太坊终端):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
func makeFullNode(ctx *cli.Context) *node.Node {
	stack, cfg := makeConfigNode(ctx)

    ...

    utils.RegisterEthService(stack, &cfg.Eth)

    ...

    utils.RegisterShhService(stack, &cfg.Shh)

    ...

    utils.RegisterGraphQLService(stack, ...)

    ...

    utils.RegisterEthStatsService(stack, cfg.Ethstats.URL)

    return stack
}
  • makeConfigNode获取以太坊相关的Node, Eth, Shh的默认配置cfg,根据cfg.Node来新建stack节点对象,再根据cfg.Eth, cfg.Shhstack节点对象内的Eth, Shh服务进行了设置。

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    
    func makeConfigNode(ctx *cli.Context) (*node.Node, gethConfig) {
    	// Load defaults.
    	cfg := gethConfig{
    		Eth:  eth.DefaultConfig,
    		Shh:  whisper.DefaultConfig,
    		Node: defaultNodeConfig(),
    	}
    
    	// Load config file.
    	if file := ctx.GlobalString(configFileFlag.Name); file != "" {
    		if err := loadConfig(file, &cfg); err != nil {
    			utils.Fatalf("%v", err)
    		}
    	}
    
    	// Apply flags.
    	utils.SetNodeConfig(ctx, &cfg.Node)
    	stack, err := node.New(&cfg.Node)
    	if err != nil {
    		utils.Fatalf("Failed to create the protocol stack: %v", err)
    	}
    	utils.SetEthConfig(ctx, stack, &cfg.Eth)
    	if ctx.GlobalIsSet(utils.EthStatsURLFlag.Name) {
    		cfg.Ethstats.URL = ctx.GlobalString(utils.EthStatsURLFlag.Name)
    	}
    	utils.SetShhConfig(ctx, stack, &cfg.Shh)
    
    	return stack, cfg
    }
    

    其中Node的数据结构如下:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    
    // Node is a container on which services can be registered.
    type Node struct {
    	eventmux *event.TypeMux // Event multiplexer used between the services of a stack
    	config   *Config
    	accman   *accounts.Manager
    
    	ephemeralKeystore string            // if non-empty, the key directory that will be removed by Stop
    	instanceDirLock   fileutil.Releaser // prevents concurrent use of instance directory
    
    	serverConfig p2p.Config
    	server       *p2p.Server // Currently running P2P networking layer
    
    	serviceFuncs []ServiceConstructor     // Service constructors (in dependency order)
    	services     map[reflect.Type]Service // Currently running services
    
    	rpcAPIs       []rpc.API   // List of APIs currently provided by the node
    	inprocHandler *rpc.Server // In-process RPC request handler to process the API requests
    
    	ipcEndpoint string       // IPC endpoint to listen at (empty = IPC disabled)
    	ipcListener net.Listener // IPC RPC listener socket to serve API requests
    	ipcHandler  *rpc.Server  // IPC RPC request handler to process the API requests
    
    	httpEndpoint  string       // HTTP endpoint (interface + port) to listen at (empty = HTTP disabled)
    	httpWhitelist []string     // HTTP RPC modules to allow through this endpoint
    	httpListener  net.Listener // HTTP RPC listener socket to server API requests
    	httpHandler   *rpc.Server  // HTTP RPC request handler to process the API requests
    
    	wsEndpoint string       // Websocket endpoint (interface + port) to listen at (empty = websocket disabled)
    	wsListener net.Listener // Websocket RPC listener socket to server API requests
    	wsHandler  *rpc.Server  // Websocket RPC request handler to process the API requests
    
    	stop chan struct{} // Channel to wait for termination notifications
    	lock sync.RWMutex
    
    	log log.Logger
    }
    
  • 然后分别注册Eth, Shh, GraphQL(if requested), Ethereum Stats(if requested)服务。

    1
    2
    3
    4
    5
    6
    
    utils.RegisterEthService(stack, &cfg.Eth)
    utils.RegisterShhService(stack, &cfg.Shh)
    // if requested
    utils.RegisterGraphQLService(stack, ...)
    // if requested
    utils.RegisterEthStatsService(stack, cfg.Ethstats.URL)
    

    可以看到,Node就像一个大容器,包含了整个以太坊区块链运行所需的部件。而以太坊的各个功能则都是通过stack这个Node实例里的Service接口来实现的。

    Service接口的定义如下:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    
    // Service is an individual protocol that can be registered into a node.
    //
    // Notes:
    //
    // • Service life-cycle management is delegated to the node. The service is allowed to
    // initialize itself upon creation, but no goroutines should be spun up outside of the
    // Start method.
    //
    // • Restart logic is not required as the node will create a fresh instance
    // every time a service is started.
    type Service interface {
    	// Protocols retrieves the P2P protocols the service wishes to start.
    	Protocols() []p2p.Protocol
    
    	// APIs retrieves the list of RPC descriptors the service provides
    	APIs() []rpc.API
    
    	// Start is called after all services have been constructed and the networking
    	// layer was also initialized to spawn any goroutines required by the service.
    	Start(server *p2p.Server) error
    
    	// Stop terminates all goroutines belonging to the service, blocking until they
    	// are all terminated.
    	Stop() error
    }
    
  • stack实例返回到geth()函数中,在geth()函数里这个实例叫做node

接着调用startNode(ctx, node)启动刚刚配置好的Node模块(传入后,这个实例又叫回了stack):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
func startNode(ctx *cli.Context, stack *node.Node) {
	debug.Memsize.Add("node", stack)

	// Start up the node itself
	utils.StartNode(stack)

	// Unlock any account specifically requested
	unlockAccounts(ctx, stack)

	// Register wallet event handlers to open and auto-derive wallets
	events := make(chan accounts.WalletEvent, 16)
	stack.AccountManager().Subscribe(events)

	// Create a client to interact with local geth node.
	rpcClient, err := stack.Attach()
	if err != nil {
		utils.Fatalf("Failed to attach to self: %v", err)
	}
	ethClient := ethclient.NewClient(rpcClient)

	// Set contract backend for ethereum service if local node
	// is serving LES requests.
	if ctx.GlobalInt(utils.LightLegacyServFlag.Name) > 0 || ctx.GlobalInt(utils.LightServeFlag.Name) > 0 {
		var ethService *eth.Ethereum
		if err := stack.Service(&ethService); err != nil {
			utils.Fatalf("Failed to retrieve ethereum service: %v", err)
		}
		ethService.SetContractBackend(ethClient)
	}
	// Set contract backend for les service if local node is
	// running as a light client.
	if ctx.GlobalString(utils.SyncModeFlag.Name) == "light" {
		var lesService *les.LightEthereum
		if err := stack.Service(&lesService); err != nil {
			utils.Fatalf("Failed to retrieve light ethereum service: %v", err)
		}
		lesService.SetContractBackend(ethClient)
	}

	go func() {
		// Open any wallets already attached
		for _, wallet := range stack.AccountManager().Wallets() {
			if err := wallet.Open(""); err != nil {
				log.Warn("Failed to open wallet", "url", wallet.URL(), "err", err)
			}
		}
		// Listen for wallet event till termination
		for event := range events {
			switch event.Kind {
			case accounts.WalletArrived:
				...
			case accounts.WalletOpened:
				...
			case accounts.WalletDropped:
				...
			}
		}
	}()

	// Spawn a standalone goroutine for status synchronization monitoring,
	// close the node when synchronization is complete if user required.
	if ctx.GlobalBool(utils.ExitWhenSyncedFlag.Name) {
		go func() {
			...
	}

	// Start auxiliary services if enabled
	if ctx.GlobalBool(utils.MiningEnabledFlag.Name) || ctx.GlobalBool(utils.DeveloperFlag.Name) {
		...
		if err := ethereum.StartMining(threads); err != nil {
			utils.Fatalf("Failed to start mining: %v", err)
		}
	}
}
  1. util.StartNode(stack)会通过stack.Start()启动一个P2P节点,并开启一个用于接受信号来终止stack的goroutine。

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    
    func StartNode(stack *node.Node) {
        if err := stack.Start(); err != nil {
            Fatalf("Error starting protocol stack: %v", err)
        }
        go func() {
            ...
            go stack.Stop()
            ...
        }()
    }
    

    跟入stack.Start()

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    
    // Start creates a live P2P node and starts running it.
    func (n *Node) Start() error {
        ...
        // Initialize the p2p server. This creates the node key and
        // discovery databases.
        n.serverConfig = n.config.P2P
        n.serverConfig.PrivateKey = n.config.NodeKey()
        n.serverConfig.Name = n.config.NodeName()
        n.serverConfig.Logger = n.log
        if n.serverConfig.StaticNodes == nil {
            n.serverConfig.StaticNodes = n.config.StaticNodes()
        }
        ...
    
        running := &p2p.Server{Config: n.serverConfig}
        n.log.Info("Starting peer-to-peer node", "instance", n.serverConfig.Name)
    
        ...
    
        // Gather the protocols and start the freshly assembled P2P server
        for _, service := range services {
            running.Protocols = append(running.Protocols, service.Protocols()...)
        }
        if err := running.Start(); err != nil {
            return convertFileLockError(err)
        }
    
        // Start each of the services
        var started []reflect.Type
        for kind, service := range services {
            // Start the next service, stopping all previous upon failure
            if err := service.Start(running); err != nil {
                for _, kind := range started {
                    services[kind].Stop()
                }
                running.Stop()
    
                return err
            }
            // Mark the service started for potential cleanup
            started = append(started, kind)
        }
    
        // Lastly, start the configured RPC interfaces
        if err := n.startRPC(services); err != nil {
            for _, service := range services {
                service.Stop()
            }
            running.Stop()
            return err
        }
        // Finish initializing the startup
        n.services = services
        n.server = running
        n.stop = make(chan struct{})
        return nil
    }
    
    • 根据Node实例stack中的配置,创建p2p.Server实例running

      running := &p2p.Server{Config: n.serverConfig}

    • 调用先前注册在Node实例中ServiceConstructor,对各个Service(Eth协议、Shh协议等等都被包装在Service接口中)进行创建

    • 将刚刚创建的所有Service里的协议(protocols,Eth协议、Shh协议等),加入到running的协议列表里

    • 启动p2p服务器

      running.Start()

    • 启动刚刚创建好的所有服务service.Start(running)

    • 配置RPC接口(根据所有服务暴露出来API开启in-process, IRC, HTTP, websocket这些RPC endpoints

  2. 解锁账号。

  3. 注册钱包事件。

  4. 启动一个RPC客户端与本地节点相连。(可以通过这个客户端进行命令调用,从而获取到本地节点的相关信息)

  5. 如果本地节点正在服务The Light Ethereum Subprotocol (LES)请求或者本地节点是个轻节点,则给以太坊服务(Eth Service)设置Contract backend

  6. 开启一个用来处理钱包事件的goroutine。

  7. 开启一个在节点数据同步完了之后用于关闭节点的goroutine(如果用户设置了ExitWhenSyncedFlag)。

  8. 如果设置了挖矿flag,(必须得是全节点)则先设置交易池的gasprice,然后开启(多线程)挖矿。

  9. 返回到geth函数。

大概就是,这里会把先前Node类型的实例stack里的部分内容,转移到p2p.Server这个类型的实例running中,并开启running这个Server。节点(Node)是本地跑的,但是为了跟其他的节点连起来,必须得开启p2p server才能与其他节点进行通信;且开启了p2p server,也需要提供各种类型的服务(Eth服务、LES服务等等),以及要开启RPC服务器,为其他用户提供远程调用。

回到geth函数中,startNode(ctx, node)后,会接下去调用node.Wait()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
// Wait blocks the thread until the node is stopped. If the node is not running
// at the time of invocation, the method immediately returns.
func (n *Node) Wait() {
	n.lock.RLock()
	if n.server == nil {
		n.lock.RUnlock()
		return
	}
	stop := n.stop
	n.lock.RUnlock()

	<-stop
}

这个函数就是用来堵塞线程,不让geth函数执行下去的。如果节点停止运行了,才会解开锁,让geth函数继续。

再回到geth函数中,最后来看defer延缓了的node.Close()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
// Close stops the Node and releases resources acquired in
// Node constructor New.
func (n *Node) Close() error {
	var errs []error

	// Terminate all subsystems and collect any errors
	if err := n.Stop(); err != nil && err != ErrNodeStopped {
		errs = append(errs, err)
	}
	if err := n.accman.Close(); err != nil {
		errs = append(errs, err)
	}
	// Report any errors that might have occurred
	switch len(errs) {
	case 0:
		return nil
	case 1:
		return errs[0]
	default:
		return fmt.Errorf("%v", errs)
	}
}

node.Close()主要会用来停止节点的运行,并释放资源。

跟入n.Stop():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
// Stop terminates a running node along with all it's services. In the node was
// not started, an error is returned.
func (n *Node) Stop() error {
	n.lock.Lock()
	defer n.lock.Unlock()

	// Short circuit if the node's not running
	if n.server == nil {
		return ErrNodeStopped
	}

	// Terminate the API, services and the p2p server.
	n.stopWS()
	n.stopHTTP()
	n.stopIPC()
	n.rpcAPIs = nil
	failure := &StopError{
		Services: make(map[reflect.Type]error),
	}
	for kind, service := range n.services {
		if err := service.Stop(); err != nil {
			failure.Services[kind] = err
		}
	}
	n.server.Stop()
	n.services = nil
	n.server = nil

	// Release instance directory lock.
	if n.instanceDirLock != nil {
		if err := n.instanceDirLock.Release(); err != nil {
			n.log.Error("Can't release datadir lock", "err", err)
		}
		n.instanceDirLock = nil
	}

	// unblock n.Wait
	close(n.stop)

	// Remove the keystore if it was created ephemerally.
	var keystoreErr error
	if n.ephemeralKeystore != "" {
		keystoreErr = os.RemoveAll(n.ephemeralKeystore)
	}

	if len(failure.Services) > 0 {
		return failure
	}
	if keystoreErr != nil {
		return keystoreErr
	}
	return nil
}

image-20200420210824796

Stop这个函数会中止一个正在运行的节点及其所有服务,如果这个节点并没有开启,那么会直接返回一个错误。

可以注意到这边的In应该是If,咳咳咳咳。。

n.Stop里面,就是停止RPC的API、各种服务,以及p2p Server,并解开两把锁,移除临时用的密钥库。

回到node.Close()中,会继续执行n.accman.Close(),跟入:

1
2
3
4
5
6
// Close terminates the account manager's internal notification processes.
func (am *Manager) Close() error {
	errc := make(chan error)
	am.quit <- errc
	return <-errc
}

主要是通过channel,关闭账户管理器(accman, account manager)相关的进程。

再回到node.Close()中,这个函数最后会报告任何可能会出现的错误。

至此geth函数运行结束,geth命令行应用app也会退出。

Screen Shot 2020-04-22 at 7.13.08 PM

感觉基本就是这种配置->注册->启动->停止的一个步骤

VS Code 调试环境搭建

首先得有源码,并且在go-ethereum目录下打开VS Code。

安装Go的调试工具dlv:在VS Code里,command + shift + p,输入Go: Install/Update Tools,选择dlvok

按照VS Code官方的Go代码调试 来进行配置:

  • 在VS Code里,command + shift + p,输入Debug: Open launch.json来打开调试的配置的文件。

  • 按照上面那个链接里的指示进行配置。

  • 由于我们是从go-ethereum/cmd/geth/main.go文件进入的,所以需要在"program"处填上"${workspaceFolder}/cmd/geth"。这样调试程序就会从main.go开始执行。

  • "args"就是geth命令的一些参数,我填的是:

    1
    2
    3
    4
    5
    
    "args": [
        "--syncmode",
        "light",
        "--ropsten"
    ],
    

    轻节点模式,连接ropsten测试网。

  • 最后我的launch.json文件中的内容如下:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    
    {
        // Use IntelliSense to learn about possible attributes.
        // Hover to view descriptions of existing attributes.
        // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
        "version": "0.2.0",
        "configurations": [
            {
                "name": "Launch",
                "type": "go",
                "request": "launch",
                "mode": "auto",
                "program": "${workspaceFolder}/cmd/geth",
                "env": {},
                "args": [
                    "--syncmode",
                    "light",
                    "--ropsten"
                ],
            }
        ]
    }
    

配置完了之后就可以开始调试了:

  • main.gogeth函数处下断点

    Screen Shot 2020-04-21 at 1.48.23 PM

  • F5或者Run -> Start Debugging

  • 会停在func geth(xxx) xxx的入口处,这时geth命令行应用已经启动:

    Screen Shot 2020-04-21 at 1.52.46 PM

  • F5 continue,会在node.Wait之前停下:

    Screen Shot 2020-04-21 at 1.54.16 PM

    此时,p2p.Server已经启动(里面包含的若干个服务也已经启动),可以在Debug Console中看到log信息:

    Screen Shot 2020-04-21 at 1.55.10 PM

  • F5 再继续,会使得p2p.Server进行p2p通信,不断尝试连接其他节点,直至接收到中止信号。

    Screen Shot 2020-04-21 at 1.58.44 PM

这样,就基本完成了以太坊源码的调试环境搭建。后面的话,就可以去自己想深入了解的地方,一步一步地跟进看代码。

以太坊架构体系

Screen Shot 2020-04-21 at 5.24.57 PM

源码目录结构

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
tree -d -L 1
.
├── accounts                账号相关
├── bmt                     实现二叉merkle树
├── build                   编译生成的程序
├── cmd                     geth程序主体
├── common                  工具函数库
├── consensus               共识算法
├── console                 交互式命令
├── containers              docker 支持相关
├── contracts               合约相关
├── core                    以太坊核心部分
├── crypto                  加密函数库
├── dashboard               统计
├── eth                     以太坊协议
├── ethclient               以太坊RPC客户端
├── ethdb                   底层存储
├── ethstats                统计报告
├── event                   事件处理
├── internal                RPC调用
├── les                     轻量级子协议
├── light                   轻客户端部分功能
├── log                     日志模块
├── metrics                 服务监控相关
├── miner                   挖矿相关
├── mobile                  geth的移动端API
├── node                    接口节点
├── p2p                     p2p网络协议
├── params                  一些预设参数值
├── rlp                     RLP系列化格式
├── rpc                     RPC接口
├── signer                  签名相关
├── swarm                   分布式存储
├── tests                   以太坊JSON测试
├── trie                    Merkle Patricia实现
├── vendor                  一些扩展库
└── whisper                 分布式消息

35 directories

p2p网络服务层

p2p网络层是整个以太坊区块链架构的底层,主要负责本地节点与其他节点的网络通信功能,包括监听服务(等着别的节点来连)、节点发现(自己不断地尝试去连其他节点)、报文处理等;当有节点连接时,会先通过RLPx协议与之交换密钥,p2p握手,上层协议的握手,最后为每个协议启动goroutine执行Run函数来将控制权移交给最终的协议。

这边主要通过以太坊源码和几篇文章

来学习一下以太坊的p2p网络服务层。

p2p网络服务层本身又可以分为下面三层:

Screen Shot 2020-04-22 at 10.56.52 AM

其中最下面Golang net是由Go语言提供的网络IO层;中间的p2p通信链路则主要负责监听、节点发现、新建连接、维护连接等操作,为上层协议提供了信道;最上面则是各个协议。

或者,

img

以太坊上的网络数据传输均遵循RLP编码。

p2p网络服务启动

通过geth函数中的startNode(ctx, node)的连续调用,启动了p2p网络服务器。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
// go-ethereum/none/node.go

// Start creates a live P2P node and starts running it.
func (n *Node) Start() error {
	...
    running := &p2p.Server{Config: n.serverConfig}
    ...

    if err := running.Start(); err != nil {
        return convertFileLockError(err)
    }
    ...
}

其中p2p.Server的数据结构可以在go-ethereum/p2p/server.go中找到:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
// Server manages all peer connections.
type Server struct {
	// Config fields may not be modified while the server is running.
	Config

	// Hooks for testing. These are useful because we can inhibit
	// the whole protocol stack.
	newTransport func(net.Conn) transport
	newPeerHook  func(*Peer)
	listenFunc   func(network, addr string) (net.Listener, error)

	lock    sync.Mutex // protects running
	running bool

	listener     net.Listener
	ourHandshake *protoHandshake
	loopWG       sync.WaitGroup // loop, listenLoop
	peerFeed     event.Feed
	log          log.Logger

	nodedb    *enode.DB
	localnode *enode.LocalNode
	ntab      *discover.UDPv4
	DiscV5    *discv5.Network
	discmix   *enode.FairMix
	dialsched *dialScheduler

	// Channels into the run loop.
	quit                    chan struct{}
	addtrusted              chan *enode.Node
	removetrusted           chan *enode.Node
	peerOp                  chan peerOpFunc
	peerOpDone              chan struct{}
	delpeer                 chan peerDrop
	checkpointPostHandshake chan *conn
	checkpointAddPeer       chan *conn

	// State of run loop and listenLoop.
	inboundHistory expHeap
}

running := &p2p.Server{Config: n.serverConfig}主要是根据之前Node实例的配置来对新建的p2p.Server实例running进行了一个初始化配置。

随后,会调用running.Start()来启动这个p2p.Server实例,跟入running.Start()函数:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// Start starts running the server.
// Servers can not be re-used after stopping.
func (srv *Server) Start() (err error) {
	...
	// static fields
	...

    if err := srv.setupLocalNode(); err != nil {
		return err
	}
	if srv.ListenAddr != "" {
		if err := srv.setupListening(); err != nil {
			return err
		}
	}
	if err := srv.setupDiscovery(); err != nil {
		return err
	}
	srv.setupDialScheduler()

	srv.loopWG.Add(1)
	go srv.run()
	return nil
}

Start做了下面几件事:

  • p2p.Server结构体内的其他部分进行了一个配置(running := &p2p.Server{Config: n.serverConfig}初始化的时候只设置了Config

  • srv.setupLocalNode()启动本地节点

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    
    func (srv *Server) setupLocalNode() error {
    	// Create the devp2p handshake.
    	pubkey := crypto.FromECDSAPub(&srv.PrivateKey.PublicKey)
    	srv.ourHandshake = &protoHandshake{Version: baseProtocolVersion, Name: srv.Name, ID: pubkey[1:]}
    	for _, p := range srv.Protocols {
    		srv.ourHandshake.Caps = append(srv.ourHandshake.Caps, p.cap())
    	}
    	sort.Sort(capsByNameAndVersion(srv.ourHandshake.Caps))
    
    	// Create the local node.
    	db, err := enode.OpenDB(srv.Config.NodeDatabase)
    	if err != nil {
    		return err
    	}
    	srv.nodedb = db
    	srv.localnode = enode.NewLocalNode(db, srv.PrivateKey)
    	srv.localnode.SetFallbackIP(net.IP{127, 0, 0, 1})
    	// TODO: check conflicts
    	for _, p := range srv.Protocols {
    		for _, e := range p.Attributes {
    			srv.localnode.Set(e)
    		}
    	}
    	switch srv.NAT.(type) {
    	case nil:
    		// No NAT interface, do nothing.
    	case nat.ExtIP:
    		// ExtIP doesn't block, set the IP right away.
    		ip, _ := srv.NAT.ExternalIP()
    		srv.localnode.SetStaticIP(ip)
    	default:
    		// Ask the router about the IP. This takes a while and blocks startup,
    		// do it in the background.
    		srv.loopWG.Add(1)
    		go func() {
    			defer srv.loopWG.Done()
    			if ip, err := srv.NAT.ExternalIP(); err == nil {
    				srv.localnode.SetStaticIP(ip)
    			}
    		}()
    	}
    	return nil
    }
    

    跟自己握手?创建本地节点,为本地节点新建数据库,设置回环ip,并记录协议,最后根据NAT设置外部ip。

  • srv.setupListening()开启服务监听。先启动了一个tcp listener,并更新了本地节点的记录,根据NAT进行了一个监听端口的映射,最后开启了一个监听循环的goroutinego srv.listenLoop()。这个应该是用来接受其他节点的连接的。

  • srv.setupDiscovery()开启节点发现。

  • srv.setupDialScheduler()开启拨号计划。

  • srv.loopWG.Add(1)等待goroutine完成

  • 最后,srv.run()启动p2p网络协议,循环处理报文,直至p2p.Server服务退出,中止节点发现、与其他节点断连。

Screen Shot 2020-04-22 at 11.30.59 AM

服务监听

img

节点发现协议

Screen Shot 2020-04-22 at 9.56.29 AM

https://juejin.im/post/5d302646f265da1bcd380f14

报文处理

Screen Shot 2020-04-22 at 11.06.10 AM

p2p通信链路

Screen Shot 2020-04-22 at 11.07.21 AM

https://paper.seebug.org/642/

共享密钥

Screen Shot 2020-04-22 at 12.34.02 PM

RlPx加密握手

img

http://wangxiaoming.com/blog/2018/07/26/HPB-51-ETH-RLPX/

上层协议处理

以LES协议为例:

Screen Shot 2020-04-22 at 11.08.54 AM

先挖好坑,待以后填。

https://ethfans.org/posts/understanding-ethereums-p2p-network

Eth协议

geth入口 那一节中有提到,Node实例在启动时,会先将一系列协议注册好;以太坊各个功能都是在Service中进行实现的。在p2p网络协议层 也有提到过,p2p服务器就是为上层的一些协议进行底层的网络通信的。Eth就是其中最重要的协议,go-ethereum/eth目录下就是Eth协议的实现。

Eth协议注册

首先,在geth()—>makeFullNode()—>makeConfigNode()中,Eth协议会从eth包中加载默认配置:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
// go-ethereum/cmd/geth/config.go:108
func makeConfigNode(ctx *cli.Context) (*node.Node, gethConfig) {
	// Load defaults.
	cfg := gethConfig{
		Eth:  eth.DefaultConfig,
		Shh:  whisper.DefaultConfig,
		Node: defaultNodeConfig(),
	}
	...
}

Eth的默认配置如下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// go-ethereum/eth/config.go:36
// DefaultConfig contains default settings for use on the Ethereum main net.
var DefaultConfig = Config{
	SyncMode: downloader.FastSync,
	Ethash: ethash.Config{
		CacheDir:         "ethash",
		CachesInMem:      2,
		CachesOnDisk:     3,
		CachesLockMmap:   false,
		DatasetsInMem:    1,
		DatasetsOnDisk:   2,
		DatasetsLockMmap: false,
	},
	NetworkId:          1,
	LightPeers:         100,
	UltraLightFraction: 75,
	DatabaseCache:      512,
	TrieCleanCache:     256,
	TrieDirtyCache:     256,
	TrieTimeout:        60 * time.Minute,
	SnapshotCache:      256,
	Miner: miner.Config{
		GasFloor: 8000000,
		GasCeil:  8000000,
		GasPrice: big.NewInt(params.GWei),
		Recommit: 3 * time.Second,
	},
	TxPool: core.DefaultTxPoolConfig,
	GPO: gasprice.Config{
		Blocks:     20,
		Percentile: 60,
	},
}

随后,会根据命令行中给出的flag对配置进行更改。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// go-ethereum/cmd/geth/config.go:36
func makeConfigNode(ctx *cli.Context) (*node.Node, gethConfig) {
	...
    // Load config file.
	if file := ctx.GlobalString(configFileFlag.Name); file != "" {
		if err := loadConfig(file, &cfg); err != nil {
			utils.Fatalf("%v", err)
		}
	}
    ...
    utils.SetEthConfig(ctx, stack, &cfg.Eth)
    ...
}

// go-ethereum/cmd/geth/config.go:148
func makeFullNode(ctx *cli.Context) *node.Node {
	...
    if ctx.GlobalIsSet(utils.OverrideIstanbulFlag.Name) {
		cfg.Eth.OverrideIstanbul = new(big.Int).SetUint64(ctx.GlobalUint64(utils.OverrideIstanbulFlag.Name))
	}
    if ctx.GlobalIsSet(utils.OverrideMuirGlacierFlag.Name) {
		cfg.Eth.OverrideMuirGlacier = new(big.Int).SetUint64(ctx.GlobalUint64(utils.OverrideMuirGlacierFlag.Name))
	}
	...
}

接着,在geth()->makeFullNode()中注册Eth服务:

1
2
3
4
5
6
// go-ethereum/cmd/geth/config.go:148
func makeFullNode(ctx *cli.Context) *node.Node {
    ...
    utils.RegisterEthService(stack, &cfg.Eth)
    ...
}

跟入utils.RegisterEthService

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
// go-ethereum/cmd/utils/flags.go:1601
// RegisterEthService adds an Ethereum client to the stack.
func RegisterEthService(stack *node.Node, cfg *eth.Config) {
	var err error
	if cfg.SyncMode == downloader.LightSync {
		err = stack.Register(func(ctx *node.ServiceContext) (node.Service, error) {
			return les.New(ctx, cfg)
		})
	} else {
		err = stack.Register(func(ctx *node.ServiceContext) (node.Service, error) {
			fullNode, err := eth.New(ctx, cfg)
			if fullNode != nil && cfg.LightServ > 0 {
				ls, _ := les.NewLesServer(fullNode, cfg)
				fullNode.AddLesServer(ls)
			}
			return fullNode, err
		})
	}
	if err != nil {
		Fatalf("Failed to register the Ethereum service: %v", err)
	}
}

会根据同步模式来选择是注册轻节点还是全节点(全节点还会根据配置来选择是否带上LES服务器)。可以看到,“注册”实际上就是给Node实例中添加了一个用于创建Ethereum对象(或LightEthereum对象,轻节点,暂不讨论)的函数。

eth.New(ctx, cfg)会根据相关配置创建一个新的Ethereum对象(包括初始化)。

Ethereum数据结构如下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
// go-ethereum/eth/backend.go:66
// Ethereum implements the Ethereum full node service.
type Ethereum struct {
	config *Config

	// Handlers
	txPool          *core.TxPool
	blockchain      *core.BlockChain
	protocolManager *ProtocolManager
	lesServer       LesServer
	dialCandiates   enode.Iterator

	// DB interfaces
	chainDb ethdb.Database // Block chain database

	eventMux       *event.TypeMux
	engine         consensus.Engine
	accountManager *accounts.Manager

	bloomRequests     chan chan *bloombits.Retrieval // Channel receiving bloom data retrieval requests
	bloomIndexer      *core.ChainIndexer             // Bloom indexer operating during block imports
	closeBloomHandler chan struct{}

	APIBackend *EthAPIBackend

	miner     *miner.Miner
	gasPrice  *big.Int
	etherbase common.Address	// 挖矿的受益者

	networkID     uint64		// 网络ID,主网是1,测试网是后面的几个数字
	netRPCService *ethapi.PublicNetAPI

	lock sync.RWMutex // Protects the variadic fields (e.g. gas price and etherbase)
}

其中包含了交易池、区块链数据结构、协议管理器、Les服务器、用于拨号其他节点的迭代器、区块链数据库、共识引擎、账户管理器、bloom过滤器的索引、API后端等等。

eth.New(ctx, cfg)具体如下:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
// go-ethereum/eth/backend.go:113
// New creates a new Ethereum object (including the
// initialisation of the common Ethereum object)
func New(ctx *node.ServiceContext, config *Config) (*Ethereum, error) {
	// Ensure configuration values are compatible and sane
	if config.SyncMode == downloader.LightSync {
		return nil, errors.New("can't run eth.Ethereum in light sync mode, use les.LightEthereum")
	}
	if !config.SyncMode.IsValid() {
		return nil, fmt.Errorf("invalid sync mode %d", config.SyncMode)
	}
	if config.Miner.GasPrice == nil || config.Miner.GasPrice.Cmp(common.Big0) <= 0 {
		log.Warn("Sanitizing invalid miner gas price", "provided", config.Miner.GasPrice, "updated", DefaultConfig.Miner.GasPrice)
		config.Miner.GasPrice = new(big.Int).Set(DefaultConfig.Miner.GasPrice)
	}
	if config.NoPruning && config.TrieDirtyCache > 0 {
		config.TrieCleanCache += config.TrieDirtyCache * 3 / 5
		config.SnapshotCache += config.TrieDirtyCache * 3 / 5
		config.TrieDirtyCache = 0
	}
	log.Info("Allocated trie memory caches", "clean", common.StorageSize(config.TrieCleanCache)*1024*1024, "dirty", common.StorageSize(config.TrieDirtyCache)*1024*1024)

	// Assemble the Ethereum object
	chainDb, err := ctx.OpenDatabaseWithFreezer("chaindata", config.DatabaseCache, config.DatabaseHandles, config.DatabaseFreezer, "eth/db/chaindata/")
	if err != nil {
		return nil, err
	}
	chainConfig, genesisHash, genesisErr := core.SetupGenesisBlockWithOverride(chainDb, config.Genesis, config.OverrideIstanbul, config.OverrideMuirGlacier)
	if _, ok := genesisErr.(*params.ConfigCompatError); genesisErr != nil && !ok {
		return nil, genesisErr
	}
	log.Info("Initialised chain configuration", "config", chainConfig)

	eth := &Ethereum{
		config:            config,
		chainDb:           chainDb,
		eventMux:          ctx.EventMux,
		accountManager:    ctx.AccountManager,
		engine:            CreateConsensusEngine(ctx, chainConfig, &config.Ethash, config.Miner.Notify, config.Miner.Noverify, chainDb),
		closeBloomHandler: make(chan struct{}),
		networkID:         config.NetworkId,
		gasPrice:          config.Miner.GasPrice,
		etherbase:         config.Miner.Etherbase,
		bloomRequests:     make(chan chan *bloombits.Retrieval),
		bloomIndexer:      NewBloomIndexer(chainDb, params.BloomBitsBlocks, params.BloomConfirms),
	}

	bcVersion := rawdb.ReadDatabaseVersion(chainDb)
	var dbVer = "<nil>"
	if bcVersion != nil {
		dbVer = fmt.Sprintf("%d", *bcVersion)
	}
	log.Info("Initialising Ethereum protocol", "versions", ProtocolVersions, "network", config.NetworkId, "dbversion", dbVer)

	if !config.SkipBcVersionCheck {
		if bcVersion != nil && *bcVersion > core.BlockChainVersion {
			return nil, fmt.Errorf("database version is v%d, Geth %s only supports v%d", *bcVersion, params.VersionWithMeta, core.BlockChainVersion)
		} else if bcVersion == nil || *bcVersion < core.BlockChainVersion {
			log.Warn("Upgrade blockchain database version", "from", dbVer, "to", core.BlockChainVersion)
			rawdb.WriteDatabaseVersion(chainDb, core.BlockChainVersion)
		}
	}
	var (
		vmConfig = vm.Config{
			EnablePreimageRecording: config.EnablePreimageRecording,
			EWASMInterpreter:        config.EWASMInterpreter,
			EVMInterpreter:          config.EVMInterpreter,
		}
		cacheConfig = &core.CacheConfig{
			TrieCleanLimit:      config.TrieCleanCache,
			TrieCleanNoPrefetch: config.NoPrefetch,
			TrieDirtyLimit:      config.TrieDirtyCache,
			TrieDirtyDisabled:   config.NoPruning,
			TrieTimeLimit:       config.TrieTimeout,
			SnapshotLimit:       config.SnapshotCache,
		}
	)
	eth.blockchain, err = core.NewBlockChain(chainDb, cacheConfig, chainConfig, eth.engine, vmConfig, eth.shouldPreserve)
	if err != nil {
		return nil, err
	}
	// Rewind the chain in case of an incompatible config upgrade.
	if compat, ok := genesisErr.(*params.ConfigCompatError); ok {
		log.Warn("Rewinding chain to upgrade configuration", "err", compat)
		eth.blockchain.SetHead(compat.RewindTo)
		rawdb.WriteChainConfig(chainDb, genesisHash, chainConfig)
	}
	eth.bloomIndexer.Start(eth.blockchain)

	if config.TxPool.Journal != "" {
		config.TxPool.Journal = ctx.ResolvePath(config.TxPool.Journal)
	}
	eth.txPool = core.NewTxPool(config.TxPool, chainConfig, eth.blockchain)

	// Permit the downloader to use the trie cache allowance during fast sync
	cacheLimit := cacheConfig.TrieCleanLimit + cacheConfig.TrieDirtyLimit + cacheConfig.SnapshotLimit
	checkpoint := config.Checkpoint
	if checkpoint == nil {
		checkpoint = params.TrustedCheckpoints[genesisHash]
	}
	if eth.protocolManager, err = NewProtocolManager(chainConfig, checkpoint, config.SyncMode, config.NetworkId, eth.eventMux, eth.txPool, eth.engine, eth.blockchain, chainDb, cacheLimit, config.Whitelist); err != nil {
		return nil, err
	}
	eth.miner = miner.New(eth, &config.Miner, chainConfig, eth.EventMux(), eth.engine, eth.isLocalBlock)
	eth.miner.SetExtra(makeExtraData(config.Miner.ExtraData))

	eth.APIBackend = &EthAPIBackend{ctx.ExtRPCEnabled(), eth, nil}
	gpoParams := config.GPO
	if gpoParams.Default == nil {
		gpoParams.Default = config.Miner.GasPrice
	}
	eth.APIBackend.gpo = gasprice.NewOracle(eth.APIBackend, gpoParams)

	eth.dialCandiates, err = eth.setupDiscovery(&ctx.Config.P2P)
	if err != nil {
		return nil, err
	}

	return eth, nil
}
  • 做了一些检查工作

  • 打开了(没有的话就新建)在eth/db/chaindate目录下的区块链数据库

    chainDb, err := ctx.OpenDatabaseWithFreezer(..., "eth/db/chaindata/")

  • 设置创世区块

    chainConfig, genesisHash, genesisErr := core.SetupGenesisBlockWithOverride(...)

  • 根据传入的ctx(上下文)和cfg(配置)新建了一个Ethereum实例eth

    eth := &Ethereum { ... }

  • 根据数据库新建了一个区块链数据结构,并传给eth中的blockchain字段

    eth.blockchain, err = core.NewBlockChain(chainDb, ...)

  • BloomIndexer操作了下(不清楚是个什么东西)

    eth.bloomIndexer.Start(eth.blockchain)

  • 根据相关的配置新建了一个交易池,并传给eth中的txPool字段

    eth.txPool = core.NewTxPool(config.TxPool, chainConfig, eth.blockchain)

  • 根据相关的配置和组件新建了一个协议管理器,并创给eth中的protocolManager字段

    eth.protocolManager, err = NewProtocolManager(...)

  • 根据配置以及共识引擎新建了一个矿工对象,并传给了eth中的miner字段

    eth.miner = miner.New(...)

  • 新建了一个用于给RPC调用提供后端支持的APIBackend对象,并传给了eth中的APIBackend字段

    eth.APIBackend = &EthAPIBackend{ctx.ExtRPCEnabled(), eth, nil}

  • 根据p2p的配置为eth协议创建了一个用于节点发现的源,并传给eth中的dailCandidates字段

    eth.dialCandiates, err = eth.setupDiscovery(&ctx.Config.P2P)

  • 最后将eth返回

可见Eth协议是一个很庞大的东西

Screen Shot 2020-04-23 at 4.52.40 PM

Eth协议启动

前面是Eth的注册操作,实际上只是在Node实例中新增了一个可以用来创建Eth协议的函数(ServiceConstructor),但Eth协议并没有被真正地创建出来。

geth()->startNode()->utils.StartNode()->stack.Start()中才会依次根据Node实例中的注册好了的服务一个一个地进行创建,然后加入到p2p.Server的协议列表中,再启动。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
// go-ethereum/node/node.go:161
// Start creates a live P2P node and starts running it.
func (n *Node) Start() error {
    ...
    running := &p2p.Server{Config: n.serverConfig}
    ...

    services := make(map[reflect.Type]Service)
    for _, constructor := range n.serviceFuncs {
        ...
        // Construct and save the service
		service, err := constructor(ctx)
        ...
        kind := reflect.TypeOf(service)
		if _, exists := services[kind]; exists {
			return &DuplicateServiceError{Kind: kind}
		}
		services[kind] = service
    }
    // Gather the protocols and start the freshly assembled P2P server
	for _, service := range services {
		running.Protocols = append(running.Protocols, service.Protocols()...)
	}
    ...

    // Start each of the services
	var started []reflect.Type
	for kind, service := range services {
		// Start the next service, stopping all previous upon failure
		if err := service.Start(running); err != nil {
			for _, kind := range started {
				services[kind].Stop()
			}
			running.Stop()

			return err
		}
		// Mark the service started for potential cleanup
		started = append(started, kind)
	}
}

service.Start(running)将会调用Ethereum.Start(),启动Eth协议所需的所有goroutine。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// Start implements node.Service, starting all internal goroutines needed by the
// Ethereum protocol implementation.
func (s *Ethereum) Start(srvr *p2p.Server) error {
	s.startEthEntryUpdate(srvr.LocalNode())

	// Start the bloom bits servicing goroutines
	s.startBloomHandlers(params.BloomBitsBlocks)

	// Start the RPC service
	s.netRPCService = ethapi.NewPublicNetAPI(srvr, s.NetVersion())

	// Figure out a max peers count based on the server limits
	maxPeers := srvr.MaxPeers
	if s.config.LightServ > 0 {
		if s.config.LightPeers >= srvr.MaxPeers {
			return fmt.Errorf("invalid peer config: light peer count (%d) >= total peer count (%d)", s.config.LightPeers, srvr.MaxPeers)
		}
		maxPeers -= s.config.LightPeers
	}
	// Start the networking layer and the light server if requested
	s.protocolManager.Start(maxPeers)
	if s.lesServer != nil {
		s.lesServer.Start(srvr)
	}
	return nil
}
  • 开启ENR entry更新循环(advertises eth protocol on the discovery network.
  • 启动布隆过滤器请求处理的goroutine
  • 将Eth的net相关API加入RPC服务
  • 基于服务器的限制,算出最大同伴节点数
  • 开启Eth子协议管理器,里面启动了大量的goroutine用于广播和同步。(主要与下面的p2p网络层进行对接?)

Eth协议停止

Eth协议停止主要有以下几种情况:

  1. 在各个Service启动时,有某一个未成功启动,会导致先前已经启动好的Service停止,再使得p2p.Server实例停止,返回错误。

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    
    // go-ethereum/node/node.go:226
    // Start the next service, stopping all previous upon failure
    if err := service.Start(running); err != nil {
        for _, kind := range started {
            services[kind].Stop()
        }
        running.Stop()
    
        return err
    }
    
  2. 在各个Service开启RPC时,有未成功开启的,也会导致已经启动好的Service停止,再使得p2p.Server实例停止,返回错误。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    
    // go-ethereum/node/node.go:238
    // Lastly, start the configured RPC interfaces
    if err := n.startRPC(services); err != nil {
        for _, service := range services {
            service.Stop()
        }
        running.Stop()
        return err
    }
    
  3. p2p.Server执行Stop函数,会使得所有已经启动的Service停止。

    1
    2
    3
    4
    5
    6
    
    // go-ethereum/node/node.go:473
    for kind, service := range n.services {
        if err := service.Stop(); err != nil {
            failure.Services[kind] = err
        }
    }
    

Eth协议停止,会调用Ethereum.Stop()函数,终止所有运行在内部的goroutines,并调用Eth数据结构内的各个部件的Stop函数。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
// go-ethereum/eth/backend.go:557
// Stop implements node.Service, terminating all internal goroutines used by the
// Ethereum protocol.
func (s *Ethereum) Stop() error {
	// Stop all the peer-related stuff first.
	s.protocolManager.Stop()
	if s.lesServer != nil {
		s.lesServer.Stop()
	}

	// Then stop everything else.
	s.bloomIndexer.Close()
	close(s.closeBloomHandler)
	s.txPool.Stop()
	s.miner.Stop()
	s.blockchain.Stop()
	s.engine.Close()
	s.chainDb.Close()
	s.eventMux.Stop()
	return nil
}

ProtocolManager:以太坊P2P通讯协议管理器

区块同步

交易同步

http://wangxiaoming.com/blog/2018/08/09/HPB-54-ETH-Network-send-recv/

布隆过滤器

以太坊核心层

ETH协议是由Ethereum这个数据结构来表示的,在Ethereum中包含了核心层的区块链(BlockChain)数据结构、交易池(TxPool)数据结构和Bloom过滤器的索引,以及共识引擎。下面,就要深入到以太坊的核心层,来探究一下以太坊的核心结构——账本模型+共识算法。

BlockChain数据结构

区块链实际上也是一个数据结构,一个复杂的单向链表,一个一个的区块(block)通过hash的方式连在了一起,构成了区块链(blockchain)这样一个庞大的数据结构。

Block数据结构

先来看看单个区块(blcok)是怎么样的,Block结构体在go-ethereum/core/types/block.go中有定义。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// go-ethereum/core/types/block.go:146
// Block represents an entire block in the Ethereum blockchain.
type Block struct {
	header       *Header
	uncles       []*Header
	transactions Transactions

	// caches
	hash atomic.Value
	size atomic.Value

	// Td is used by package core to store the total difficulty
	// of the chain up to and including the block.
	td *big.Int

	// These fields are used by package eth to track
	// inter-peer block relay.
	ReceivedAt   time.Time
	ReceivedFrom interface{}
}

// go-ethereum/core/types/block.go:139
// Body is a simple (mutable, non-safe) data container for storing and moving
// a block's data contents (transactions and uncles) together.
type Body struct {
	Transactions []*Transaction
	Uncles       []*Header
}

可见一个区块(block)数据结构又被分为两个主要的部分:区块头header和区块体body(叔区块的头uncles header + 交易列表transactions),以及其他的一些微小的部件。

别人那儿 偷的一张图(这图画得也太好了吧???):

以太坊区块结构

其中header结构体的定义如下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
// go-ethereum/core/types/block.go:69
// Header represents a block header in the Ethereum blockchain.
type Header struct {
	ParentHash  common.Hash    `json:"parentHash"       gencodec:"required"`
	UncleHash   common.Hash    `json:"sha3Uncles"       gencodec:"required"`
	Coinbase    common.Address `json:"miner"            gencodec:"required"`
	Root        common.Hash    `json:"stateRoot"        gencodec:"required"`
	TxHash      common.Hash    `json:"transactionsRoot" gencodec:"required"`
	ReceiptHash common.Hash    `json:"receiptsRoot"     gencodec:"required"`
	Bloom       Bloom          `json:"logsBloom"        gencodec:"required"`
	Difficulty  *big.Int       `json:"difficulty"       gencodec:"required"`
	Number      *big.Int       `json:"number"           gencodec:"required"`
	GasLimit    uint64         `json:"gasLimit"         gencodec:"required"`
	GasUsed     uint64         `json:"gasUsed"          gencodec:"required"`
	Time        uint64         `json:"timestamp"        gencodec:"required"`
	Extra       []byte         `json:"extraData"        gencodec:"required"`
	MixDigest   common.Hash    `json:"mixHash"`
	Nonce       BlockNonce     `json:"nonce"`
}
  • parentHash:父区块的hash值,用于将新区块与前一个区块串起来,进而形成一条区块链。

    一条区块链

  • sha3Uncles:叔区块的hash值。在区块体中有一个叔区块头的列表,这里的hash值是叔块集的RLPHASH值。叔区块产生的原因,可能就是两个(或以上)矿工同时挖到了新区块,会导致软分叉,这个时候需要依靠下一个新挖出的区块来选择链在哪一个父区块上,没有被链上的那个区块就无法成为主链的一部分,成为孤块,但下一个新挖出的区块可以选择收留下这个孤块,收留的这个孤块就成为了下一个新挖出区块的叔块,这个叔块也能得到奖励(不过貌似减半?)。通过这种叔块奖励机制,可以来降低以太坊软分叉和平衡网速慢的矿工利益。

  • miner:表示挖出这个区块的矿工的账户地址。

  • stateRoot:执行完这个区块中的所有交易后整个以太坊状态的一个快照ID(以太坊状态Merkle Tree的根hash值)。

  • transactionsRoot:这个区块中所有交易生成的Merkle Tree的根hash值。

  • receiptRoot:这个区块交易完成后生成的交易回执信息所构成的Merkle Tree的根hash值。

  • logsBloom:提取自receipt,用于快速定位查找交易回执中的智能合约事件信息。

  • difficulty:表示该区块的难度系数。

  • number:表示此区块的高度

  • gasLimit:表示此区块内交易所允许消耗的gas。

  • gasUsed:表示此区块内所有交易执行所实际消耗的gas。

  • timestamp:创建此区块的UTC时间戳(单位:秒)。

  • extraData:可以由矿工自由发挥的一个地方,不定长Byte数组,最长32bytes。

  • mixHash:区块头数据不包含nonce的一个hash值,用于校验区块是否被正确挖出。

  • nonce:一个长度为8的Bytes数组(uint64),用于校验区块是否被正确挖出。(mixHash + nonce 进行PoW工作量证明)

区块体内只有两项内容:交易集合和叔区块头集合。

整个以太坊世界状态的改变就是通过交易的执行来促使的。

go-ethereum/core/types/block.go里还定义了一些跟Block结构体创建、RLP编码、Hash相关的函数。

此外,在go-ethereum/core/types/block.go中还定义了用于协议的extblock和用于数据库存储的storageblock:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
// go-ethereum/core/types/block.go:179
// "external" block encoding. used for eth protocol, etc.
type extblock struct {
	Header *Header
	Txs    []*Transaction
	Uncles []*Header
}

// [deprecated by eth/63]
// "storage" block encoding. used for database.
type storageblock struct {
	Header *Header
	Txs    []*Transaction
	Uncles []*Header
	TD     *big.Int
}

go-ethereum/core/types/block_test.go里则是一些对Block结构体进行测试的内容。

BlockChain结构体定义

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
// go-ethereum/core/blockchain.go:128
// BlockChain represents the canonical chain given a database with a genesis
// block. The Blockchain manages chain imports, reverts, chain reorganisations.
//
// Importing blocks in to the block chain happens according to the set of rules
// defined by the two stage Validator. Processing of blocks is done using the
// Processor which processes the included transaction. The validation of the state
// is done in the second part of the Validator. Failing results in aborting of
// the import.
//
// The BlockChain also helps in returning blocks from **any** chain included
// in the database as well as blocks that represents the canonical chain. It's
// important to note that GetBlock can return any block and does not need to be
// included in the canonical one where as GetBlockByNumber always represents the
// canonical chain.
type BlockChain struct {
	chainConfig *params.ChainConfig // Chain & network configuration
	cacheConfig *CacheConfig        // Cache configuration for pruning

	db     ethdb.Database // Low level persistent database to store final content in
	snaps  *snapshot.Tree // Snapshot tree for fast trie leaf access
	triegc *prque.Prque   // Priority queue mapping block numbers to tries to gc
	gcproc time.Duration  // Accumulates canonical block processing for trie dumping

	hc            *HeaderChain
	rmLogsFeed    event.Feed
	chainFeed     event.Feed
	chainSideFeed event.Feed
	chainHeadFeed event.Feed
	logsFeed      event.Feed
	blockProcFeed event.Feed
	scope         event.SubscriptionScope
	genesisBlock  *types.Block

	chainmu sync.RWMutex // blockchain insertion lock

	currentBlock     atomic.Value // Current head of the block chain
	currentFastBlock atomic.Value // Current head of the fast-sync chain (may be above the block chain!)

	stateCache    state.Database // State database to reuse between imports (contains state cache)
	bodyCache     *lru.Cache     // Cache for the most recent block bodies
	bodyRLPCache  *lru.Cache     // Cache for the most recent block bodies in RLP encoded format
	receiptsCache *lru.Cache     // Cache for the most recent receipts per block
	blockCache    *lru.Cache     // Cache for the most recent entire blocks
	txLookupCache *lru.Cache     // Cache for the most recent transaction lookup data.
	futureBlocks  *lru.Cache     // future blocks are blocks added for later processing

	quit    chan struct{} // blockchain quit channel
	running int32         // running must be called atomically
	// procInterrupt must be atomically called
	procInterrupt int32          // interrupt signaler for block processing
	wg            sync.WaitGroup // chain processing wait group for shutting down

	engine     consensus.Engine
	validator  Validator  // Block and state validator interface
	prefetcher Prefetcher // Block state prefetcher interface
	processor  Processor  // Block transaction processor interface
	vmConfig   vm.Config

	badBlocks       *lru.Cache                     // Bad block cache
	shouldPreserve  func(*types.Block) bool        // Function used to determine whether should preserve the given block.
	terminateInsert func(common.Hash, uint64) bool // Testing hook used to terminate ancient receipt chain insertion.
}

。。。。。。在里面没找到Block结构体啊,很懵逼。。

BlockChain结构体中包含了:

  • db ethdb.Database:底层数据库,存储着以太坊的所有数据
  • hc *HeaderChain:区块头链
  • genesisBlock *types.Block:创世区块
  • currentBlock atomic.Value:当前的区块链的头
  • currentFastBlock atomic.Value:当前快速同步区块链的头
  • xxxCache:最近区块的缓存
  • engine consensus.Engine:共识引擎
  • validator Validator:区块和状态验证器
  • prefetcher Prefetcher:区块状态获取器
  • processor Processor:区块交易处理器
  • 其他部件

有几点需要注意的地方:

  1. BlockChain中只存放着由区块头构成的链,区块体和区块头是分开的。
  2. 整个区块链的数据并不是全部都在内存中的(好像现在都有240G了,一般玩家内存肯定不够用),而是通过缓存的方式(缓存数目在上面的变量初始化处可以看到)来在cache中存储最近一些的区块数据。
  3. 全节点和轻节点的数据处理方式需要区分开来。

BlockChain初始化

在先前的Eth协议初始化时,会先从eth/db/chaindata导入(创建)区块链数据库chainDb,然后通过core.SetupGenesisBlockWithOverride(chainDb, ...)配置好创世区块,随后会通过core.NewBlockChain(chainDb)新建一个BlockChain对象,并传给eth中的blockchain字段。

跟入core.NewBlockChain

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
// NewBlockChain returns a fully initialised block chain using information
// available in the database. It initialises the default Ethereum Validator and
// Processor.
func NewBlockChain(db ethdb.Database, cacheConfig *CacheConfig, chainConfig *params.ChainConfig, engine consensus.Engine, vmConfig vm.Config, shouldPreserve func(block *types.Block) bool) (*BlockChain, error) {
	if cacheConfig == nil {
		cacheConfig = &CacheConfig{
			TrieCleanLimit: 256,
			TrieDirtyLimit: 256,
			TrieTimeLimit:  5 * time.Minute,
			SnapshotLimit:  256,
			SnapshotWait:   true,
		}
	}
	bodyCache, _ := lru.New(bodyCacheLimit)
	bodyRLPCache, _ := lru.New(bodyCacheLimit)
	receiptsCache, _ := lru.New(receiptsCacheLimit)
	blockCache, _ := lru.New(blockCacheLimit)
	txLookupCache, _ := lru.New(txLookupCacheLimit)
	futureBlocks, _ := lru.New(maxFutureBlocks)
	badBlocks, _ := lru.New(badBlockLimit)

	bc := &BlockChain{
		chainConfig:    chainConfig,
		cacheConfig:    cacheConfig,
		db:             db,
		triegc:         prque.New(nil),
		stateCache:     state.NewDatabaseWithCache(db, cacheConfig.TrieCleanLimit),
		quit:           make(chan struct{}),
		shouldPreserve: shouldPreserve,
		bodyCache:      bodyCache,
		bodyRLPCache:   bodyRLPCache,
		receiptsCache:  receiptsCache,
		blockCache:     blockCache,
		txLookupCache:  txLookupCache,
		futureBlocks:   futureBlocks,
		engine:         engine,
		vmConfig:       vmConfig,
		badBlocks:      badBlocks,
	}
	bc.validator = NewBlockValidator(chainConfig, bc, engine)
	bc.prefetcher = newStatePrefetcher(chainConfig, bc, engine)
	bc.processor = NewStateProcessor(chainConfig, bc, engine)

	var err error
	bc.hc, err = NewHeaderChain(db, chainConfig, engine, bc.getProcInterrupt)
	if err != nil {
		return nil, err
	}
	bc.genesisBlock = bc.GetBlockByNumber(0)
	if bc.genesisBlock == nil {
		return nil, ErrNoGenesis
	}

	var nilBlock *types.Block
	bc.currentBlock.Store(nilBlock)
	bc.currentFastBlock.Store(nilBlock)

	// Initialize the chain with ancient data if it isn't empty.
	if bc.empty() {
		rawdb.InitDatabaseFromFreezer(bc.db)
	}

	if err := bc.loadLastState(); err != nil {
		return nil, err
	}
	// The first thing the node will do is reconstruct the verification data for
	// the head block (ethash cache or clique voting snapshot). Might as well do
	// it in advance.
	bc.engine.VerifyHeader(bc, bc.CurrentHeader(), true)

	if frozen, err := bc.db.Ancients(); err == nil && frozen > 0 {
		var (
			needRewind bool
			low        uint64
		)
		// The head full block may be rolled back to a very low height due to
		// blockchain repair. If the head full block is even lower than the ancient
		// chain, truncate the ancient store.
		fullBlock := bc.CurrentBlock()
		if fullBlock != nil && fullBlock != bc.genesisBlock && fullBlock.NumberU64() < frozen-1 {
			needRewind = true
			low = fullBlock.NumberU64()
		}
		// In fast sync, it may happen that ancient data has been written to the
		// ancient store, but the LastFastBlock has not been updated, truncate the
		// extra data here.
		fastBlock := bc.CurrentFastBlock()
		if fastBlock != nil && fastBlock.NumberU64() < frozen-1 {
			needRewind = true
			if fastBlock.NumberU64() < low || low == 0 {
				low = fastBlock.NumberU64()
			}
		}
		if needRewind {
			var hashes []common.Hash
			previous := bc.CurrentHeader().Number.Uint64()
			for i := low + 1; i <= bc.CurrentHeader().Number.Uint64(); i++ {
				hashes = append(hashes, rawdb.ReadCanonicalHash(bc.db, i))
			}
			bc.Rollback(hashes)
			log.Warn("Truncate ancient chain", "from", previous, "to", low)
		}
	}
	// Check the current state of the block hashes and make sure that we do not have any of the bad blocks in our chain
	for hash := range BadHashes {
		if header := bc.GetHeaderByHash(hash); header != nil {
			// get the canonical block corresponding to the offending header's number
			headerByNumber := bc.GetHeaderByNumber(header.Number.Uint64())
			// make sure the headerByNumber (if present) is in our current canonical chain
			if headerByNumber != nil && headerByNumber.Hash() == header.Hash() {
				log.Error("Found bad hash, rewinding chain", "number", header.Number, "hash", header.ParentHash)
				bc.SetHead(header.Number.Uint64() - 1)
				log.Error("Chain rewind was successful, resuming normal operation")
			}
		}
	}
	// Load any existing snapshot, regenerating it if loading failed
	if bc.cacheConfig.SnapshotLimit > 0 {
		bc.snaps = snapshot.New(bc.db, bc.stateCache.TrieDB(), bc.cacheConfig.SnapshotLimit, bc.CurrentBlock().Root(), !bc.cacheConfig.SnapshotWait)
	}
	// Take ownership of this particular state
	go bc.update()
	return bc, nil
}
  • 创建LRU缓存
  • 新建一个BlockChain对象bc(初始化trie的gc、初始化state缓存)
  • 初始化区块和状态验证器、状态获取器、状态处理器,并传给bc
  • 初始化区块头链,并传给bc
  • 从数据库中获取创始块,并传给bc
  • 将bc中的(全节点模式)当前区块和快速同步模式下的当前区块置为nil
  • 如果链是空的,那么用老区块链数据(ancient data)来初始化数据库
  • 加载最新的状态数据
  • 验证当前区块头部
  • 查看本地区块链的长度是否比Ancient还要小,会把Ancient里数据给截短
  • 查看本地区块链上是否有硬分叉的区块,如果有就调用SetHead回到硬分叉之前
  • 加载快照
  • go bc.update()定时处理新增的区块

BlockChain结构体其他部分

go-ethereum/core/blockchain.go文件的后面,都是一些与区块链这个数据结构相关的新建、查询、修改等功能。

摘自 https://github.com/ZtesoftCS/go-ethereum-code-analysis

从测试案例来看,blockchain的主要功能点有下面几点.

  1. import.
  2. GetLastBlock的功能.
  3. 如果有多条区块链,可以选取其中难度最大的一条作为规范的区块链.
  4. BadHashes 可以手工禁止接受一些区块的hash值.在blocks.go里面.
  5. 如果新配置了BadHashes. 那么区块启动的时候会自动禁止并进入有效状态.
  6. 错误的nonce会被拒绝.
  7. 支持Fast importing.
  8. Light vs Fast vs Full processing 在处理区块头上面的效果相等.

可以看到blockchain的主要功能是维护区块链的状态, 包括区块的验证,插入和状态查询.


数据存储

http://wangxiaoming.com/blog/2018/09/03/HPB-58-ETH-Data-Stru/

https://learnblockchain.cn/books/geth/part3/statedb.html

Screen Shot 2020-04-22 at 10.25.12 PM

TxPool数据结构

https://learnblockchain.cn/books/geth/part2/txpool.html

共识算法

https://learnblockchain.cn/books/geth/part2/consensus.html

http://wangxiaoming.com/blog/2018/06/26/HPB-47-ETH-Pow/

ethash源码

consensus/consenesus.go文件中,有一个Engine接口的定义:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
// Engine is an algorithm agnostic consensus engine.
type Engine interface {
	// Author retrieves the Ethereum address of the account that minted the given
	// block, which may be different from the header's coinbase if a consensus
	// engine is based on signatures.
	Author(header *types.Header) (common.Address, error)

	// VerifyHeader checks whether a header conforms to the consensus rules of a
	// given engine. Verifying the seal may be done optionally here, or explicitly
	// via the VerifySeal method.
	VerifyHeader(chain ChainReader, header *types.Header, seal bool) error

	// VerifyHeaders is similar to VerifyHeader, but verifies a batch of headers
	// concurrently. The method returns a quit channel to abort the operations and
	// a results channel to retrieve the async verifications (the order is that of
	// the input slice).
	VerifyHeaders(chain ChainReader, headers []*types.Header, seals []bool) (chan<- struct{}, <-chan error)

	// VerifyUncles verifies that the given block's uncles conform to the consensus
	// rules of a given engine.
	VerifyUncles(chain ChainReader, block *types.Block) error

	// VerifySeal checks whether the crypto seal on a header is valid according to
	// the consensus rules of the given engine.
	VerifySeal(chain ChainReader, header *types.Header) error

	// Prepare initializes the consensus fields of a block header according to the
	// rules of a particular engine. The changes are executed inline.
	Prepare(chain ChainReader, header *types.Header) error

	// Finalize runs any post-transaction state modifications (e.g. block rewards)
	// but does not assemble the block.
	//
	// Note: The block header and state database might be updated to reflect any
	// consensus rules that happen at finalization (e.g. block rewards).
	Finalize(chain ChainReader, header *types.Header, state *state.StateDB, txs []*types.Transaction,
		uncles []*types.Header)

	// FinalizeAndAssemble runs any post-transaction state modifications (e.g. block
	// rewards) and assembles the final block.
	//
	// Note: The block header and state database might be updated to reflect any
	// consensus rules that happen at finalization (e.g. block rewards).
	FinalizeAndAssemble(chain ChainReader, header *types.Header, state *state.StateDB, txs []*types.Transaction,
		uncles []*types.Header, receipts []*types.Receipt) (*types.Block, error)

	// Seal generates a new sealing request for the given input block and pushes
	// the result into the given channel.
	//
	// Note, the method returns immediately and will send the result async. More
	// than one result may also be returned depending on the consensus algorithm.
	Seal(chain ChainReader, block *types.Block, results chan<- *types.Block, stop <-chan struct{}) error

	// SealHash returns the hash of a block prior to it being sealed.
	SealHash(header *types.Header) common.Hash

	// CalcDifficulty is the difficulty adjustment algorithm. It returns the difficulty
	// that a new block should have.
	CalcDifficulty(chain ChainReader, time uint64, parent *types.Header) *big.Int

	// APIs returns the RPC APIs this consensus engine provides.
	APIs(chain ChainReader) []rpc.API

	// Close terminates any background threads maintained by the consensus engine.
	Close() error
}

当前Ethereum用的共识算法还是基于PoW的ethash算法。

ethash结构体的定义在consensus/ethash/ethash.go文件中:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// Ethash is a consensus engine based on proof-of-work implementing the ethash
// algorithm.
type Ethash struct {
	config Config

	caches   *lru // In memory caches to avoid regenerating too often
	datasets *lru // In memory datasets to avoid regenerating too often

	// Mining related fields
	rand     *rand.Rand    // Properly seeded random source for nonces
	threads  int           // Number of threads to mine on if mining
	update   chan struct{} // Notification channel to update mining parameters
	hashrate metrics.Meter // Meter tracking the average hashrate
	remote   *remoteSealer

	// The fields below are hooks for testing
	shared    *Ethash       // Shared PoW verifier to avoid cache regeneration
	fakeFail  uint64        // Block number which fails PoW check even in fake mode
	fakeDelay time.Duration // Time delay to sleep for before returning from verify

	lock      sync.Mutex // Ensures thread safety for the in-memory caches and mining fields
	closeOnce sync.Once  // Ensures exit channel will not be closed twice.
}

由于ethash算法的实现需要用到大量的数据集,所以有两个lru缓存的指针。

ethash结构体对Engine接口内定义的各个方法的实现在consensus/ethash/consensus.go文件中:

  1. Author方法会返回挖出给定区块的矿工地址。

    1
    2
    3
    4
    5
    
    // Author implements consensus.Engine, returning the header's coinbase as the
    // proof-of-work verified author of the block.
    func (ethash *Ethash) Author(header *types.Header) (common.Address, error) {
        return header.Coinbase, nil
    }
    
  2. VerifyHeader方法会检测给定的区块头是否符合共识规则。

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    
    // VerifyHeader checks whether a header conforms to the consensus rules of the
    // stock Ethereum ethash engine.
    func (ethash *Ethash) VerifyHeader(chain consensus.ChainReader, header *types.Header, seal bool) error {
        // If we're running a full engine faking, accept any input as valid
        if ethash.config.PowMode == ModeFullFake {
            return nil
        }
        // Short circuit if the header is known, or its parent not
        number := header.Number.Uint64()
        if chain.GetHeader(header.Hash(), number) != nil {
            return nil
        }
        parent := chain.GetHeader(header.ParentHash, number-1)
        if parent == nil {
            return consensus.ErrUnknownAncestor
        }
        // Sanity checks passed, do a proper verification
        return ethash.verifyHeader(chain, header, parent, false, seal)
    }
    

    主要检测的地方:

    • 这个区块已知(存在于本地的区块链数据库中):ok

    • 这个区块的父hash值未知:bad

    • 再去调用verifyHeader函数去仔细检测这个区块头是否ok。

ethash算法实现

https://learnblockchain.cn/books/geth/part2/consensus/ethash.html

以太坊的PoW算法主要有两个目的:

  • 抗ASIC(专用于挖矿的设备)性:防止像BitCoin那样使用专门挖矿的设备会导致的一个算力趋于中心化的问题。
  • 轻客户端可验证性:矿工找到的正确的nonce,能够被轻客户端快速有效校验。

=》采用Ethash算法。

https://github.com/ethereum/wiki/wiki/Ethash

https://learnblockchain.cn/books/geth/part2/consensus/ethash_implement.html

首先是seed的生成:根据当前区块的高度,计算出当前的epoch,然后对32bytes的"\x00"*32(初始种子)进行epoch次重复的keccak256哈希运算得到seed

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
//
// seedHash is the seed to use for generating a verification cache and the mining
// dataset.
func seedHash(block uint64) []byte {
   seed := make([]byte, 32)
   if block < epochLength {
      return seed
   }
   keccak256 := makeHasher(sha3.NewLegacyKeccak256())
   for i := 0; i < int(block/epochLength); i++ {
      keccak256(seed, seed)
   }
   return seed
}

然后根据这个seed以及epoch去生成一个16MB(缓存的大小会随epoch线性增长,初始的时候是16MB)的缓存。

缓存初始顺序填充

再根据这16MB的缓存去生成1GB(数据集的大小会随epoch线性增长,初始的时候是1GB)的数据集

数据集生成流程

具体的挖矿函数:

  • sealer.go/mine:从Crypto/random里生成seed,给Go里的rand库作为种子,然后生成一个随机数,从这个随机数开始不断地尝试nonce,如果hashimotoFull的结果小于困难值,就成功;否则,失败。

  • algorithm.go/hashimoto

    Ethash Hashing Algorithm

挖到了,就通过channel发送给found,然后返回给上一层的seal函数里的locals,再返回到miner/worker.go/ResultLoop中,接着会先把这个新区块提交到本地的数据库中,再回到eth/handler.go中启动mindeBroadcastLoop()goroutine来对这个新区块进行广播。

EVM

https://blog.csdn.net/cj2094/article/details/80343004

RPC通信

https://blog.csdn.net/cj2094/article/details/80023217

https://blog.csdn.net/cj2094/article/details/80044746

账户模型

https://learnblockchain.cn/books/geth/part1/account.html

RLP

https://learnblockchain.cn/books/geth/part3/rlp.html

Trie

https://learnblockchain.cn/books/geth/part3/mpt.html

HP编码(Hex-Prefix encoding / COMPACT encoding)

https://zhuanlan.zhihu.com/p/50242014

  1. Raw编码:原生的key编码,是MPT对外提供接口中使用的编码方式,当数据项被插入到树中时,Raw编码被转换成Hex编码;
  2. Hex编码:16进制扩展编码,用于对内存中树节点key进行编码,当树节点被持久化到数据库时,Hex编码被转换成HP编码;
  3. HP编码:16进制前缀编码,用于对数据库中树节点key进行编码,当树节点被加载到内存时,HP编码被转换成Hex编码;

img

Trie keys are dealt with in three distinct encodings:

KEYBYTES encoding contains the actual key and nothing else. This encoding is the input to most API functions.

HEX encoding contains one byte for each nibble of the key and an optional trailing ’terminator’ byte of value 0x10 which indicates whether or not the node at the key contains a value. Hex key encoding is used for nodes loaded in memory because it’s convenient to access.

COMPACT encoding is defined by the Ethereum Yellow Paper (it’s called “hex prefix encoding” there) and contains the bytes of the key and a flag. The high nibble of the first byte contains the flag; the lowest bit encoding the oddness of the length and the second-lowest encoding whether the node at the key is a value node. The low nibble of the first byte is zero in the case of an even number of nibbles and the first nibble in the case of an odd number. All remaining nibbles (now an even number) fit properly into the remaining bytes. Compact encoding is used for nodes stored on disk.

nibble:4-bit

将hex编码转化为一个字节数组。

"deadbeaf" ——> [0x0, 0xde, 0xad, 0xbe, 0xaf]

转化后的字节数组的第一个字节由两部分组成:

  1. 高4位(high nibble of the first byte):两个标记位,这4位中的最低位是标志hex编码的长度是否为奇数,如果是奇数,则为1;这4位中的第二低位,则是用来标志是否有terminator(hex编码的末尾是否为16),如果有,则为1。
  2. 低4位(low nibble of the first byte):如果hex编码的长度为奇数个,那么把第一个hex编码放在这里;否则为0。这样可以保证剩余的hex编码的个数是偶数个。

剩下的hex编码就每两个组成一个字节。

Screen Shot 2020-04-26 at 8.56.49 PM

Screen Shot 2020-04-26 at 8.37.00 PM

hex编码转COMPACT编码

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
// go-ethereum/encoding.go:37

func hexToCompact(hex []byte) []byte {
	terminator := byte(0)
	if hasTerm(hex) {
		terminator = 1
		hex = hex[:len(hex)-1]
	}
	buf := make([]byte, len(hex)/2+1)
	buf[0] = terminator << 5 // the flag byte
	if len(hex)&1 == 1 {
		buf[0] |= 1 << 4 // odd flag
		buf[0] |= hex[0] // first nibble is contained in the first byte
		hex = hex[1:]
	}
	decodeNibbles(hex, buf[1:])
	return buf
}

func hasTerm(s []byte) bool {
	return len(s) > 0 && s[len(s)-1] == 16
}

func decodeNibbles(nibbles []byte, bytes []byte) {
	for bi, ni := 0, 0; ni < len(nibbles); bi, ni = bi+1, ni+2 {
		bytes[bi] = nibbles[ni]<<4 | nibbles[ni+1]
	}
}

COMPACT编码转hex编码

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
func compactToHex(compact []byte) []byte {
	if len(compact) == 0 {
		return compact
	}
	base := keybytesToHex(compact)
	// delete terminator flag
	if base[0] < 2 {
		base = base[:len(base)-1]
	}
	// apply odd flag
	chop := 2 - base[0]&1
	return base[chop:]
}

func keybytesToHex(str []byte) []byte {
	l := len(str)*2 + 1
	var nibbles = make([]byte, l)
	for i, b := range str {
		nibbles[i*2] = b / 16
		nibbles[i*2+1] = b % 16
	}
	nibbles[l-1] = 16
	return nibbles
}

Merkle Patricia Trie

https://ethfans.org/posts/merkle-patricia-tree-in-detail

https://blog.csdn.net/ITleaks/article/details/79992072

Merkle Tree:

img

Patricia Tree:

img

Merkle Patricia Tree:

world state trie

node数据结构

这里的node是指树里面的某一个节点,而非p2p网络里的node。

黄皮书中定义了三种树的节点类型:

  • Leaf(叶子节点):可以理解为根据某一个键值遍历完了后的终点,节点里面存放着这个键值所对应的数据(+前缀+键值的末尾)。
  • Extension(扩展节点):键值是一串16进制字符串。多个键值,如果会共用中间的某一段字符串,那么就可以用一个扩展节点来表示这一段键值。例如上图中,a711355a77d337,前2个字符都是a7,因此可以通过一个扩展节点来表示这一段键值。
  • Branch(分支节点):一个有17个分叉的节点,前16个用于表示十六进制数中的一个,最后1个用于来表示到这一层的键值是否有对应的数据。

Screen Shot 2020-04-27 at 6.32.45 PM


具体实现:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
// go-ethereum/trie/node.go:35
type (
	fullNode struct {
		Children [17]node // Actual trie node data to encode/decode (needs custom encoder)
		flags    nodeFlag
	}
	shortNode struct {
		Key   []byte
		Val   node
		flags nodeFlag
	}
	hashNode  []byte
	valueNode []byte
)

可以发现,只有四种Node类型,fullNode表示Branch节点,shortNode表示Extension节点和Leaf节点,hashNode表示当前节点还未从磁盘里加载到内存中,valueNode表示当前节点存放的是数据。

new, insert, get, deletetrie源码分析

黄皮书

目录结构

Todo

理解

Screen Shot 2020-04-24 at 10.33.50 AM

心得体会

  • (2020.04.22)看源码这种东西,肯定是要深入到源码里面去看的。但是又不能直接一上来就看源码,还是需要去看一看别人的一些分析文章,从整体上有个大概的了解,然后再深入源码去看一些细节的实现。然后再回过头来,将自己的理解和别人的理解进行一些比对,看看哪里有不一样的地方,然后再去仔细分析。通过这样一种看别人分析—>自己去分析->再看别人分析—>再自己分析->...循环的方式,可以不断地加深自己的理解。
  • (2020.04.22)(首席说)区块链公链审计,其实一开始应该去看核心层里的东西,先看共识机制,然后再来看看区块链的账本模型,这样就能够对这个区块链的本质有了一个了解,接着就可以去看看p2p网络层、RPC的API接口层、智能合约应用层、密码学算法等等其他的部件。但是我就是直接从以太坊的底层p2p网络直接上手的,有点像从一个车的车轱辘开始,并没有get到关键的部位。不过感觉这样也还好,因为是第一次看这种大型项目的源码,所以从整个程序的入口处看起还是容易上手的;而且先从网络架构看起,能够更容易地对后续的协议进行学习。
  • (2020.04.22)不得不说,带薪学习还是很香的。

References

[1] https://me.csdn.net/cj2094 以太坊源码深入分析

[2] https://github.com/ZtesoftCS/go-ethereum-code-analysis GitHub上的一个读源码的repo

[3] https://paper.seebug.org/642/ 知道创宇区块链团队对以太坊网络架构的分析

[4] https://github.com/Microsoft/vscode-go/wiki/Debugging-Go-code-using-VS-Code Vscode调试Go代码

[5] https://studygolang.com/articles/22554 MacOS 用vs code 调试geth(go-ethereum)

[6] https://blog.csdn.net/itcastcpp/article/details/83866145 以太坊架构详解

[7] https://juejin.im/post/5cfce8fdf265da1bbb03cdf9 以太坊源码分析之共识算法ethash

[8] https://juejin.im/post/5cd63debe51d454759351d61 以太坊架构和组成

[9] https://www.beekuaibao.com/article/669783313079844864 从网络层、共识层、数据层、 智能合约层和应用层,聊聊区块链商业的技术架构

[10] https://juejin.im/post/5d302646f265da1bcd380f14 以太坊节点发现协议

[11] http://wangxiaoming.com/blog/2018/08/09/HPB-54-ETH-Network-send-recv/ HPB54:以太坊交易收发机制

[12] http://wangxiaoming.com/blog/2018/07/26/HPB-51-ETH-RLPX/ HPB51:RLPx加密握手协议研究

[13] http://wangxiaoming.com/blog/2018/09/03/HPB-58-ETH-Data-Stru/ HPB58:以太坊数据结构与存储分析

[14] http://wangxiaoming.com/blog/2017/09/18/HPB-34-P2P-Network/ HPB34:P2P网络及节点发现机制

[15] http://wangxiaoming.com/blog/2017/12/25/HPB-38-ETH-net/ HPB38:网络服务分析

[16] http://wangxiaoming.com/blog/2018/06/29/HPB-49-ETH-P2P-Exchange/ HPB49:P2P网络数据交互

[17] http://wangxiaoming.com/blog/2018/06/26/HPB-47-ETH-Pow/ HPB47:ETH-Pow算法分析

[18] https://ethfans.org/posts/ethereum-whitepaper 以太坊白皮书(中文版)

[19] https://ethfans.org/posts/510 以太坊设计原理(中文版)

[20] https://learnblockchain.cn/books/geth/ 书籍《以太坊设计与实现》 (图很不错)

[21] https://blog.csdn.net/wo541075754/article/details/54632929 Merkle Tree(默克尔树)算法解析

[22] https://medium.com/cybermiles/diving-into-ethereums-world-state-c893102030ed Diving into Ethereum’s world state

[23] https://zhuanlan.zhihu.com/p/50242014 Ethereum以太坊源码分析(三)Trie树源码分析(上)

[24] https://ethereum.github.io/yellowpaper/paper.pdf 以太坊黄皮书