Possible Channel Blocking Issue in leader.go

[ZuhairORZaki]

Overview

In file: leader.go, there is a possible case of channel blocking. In line 126, a goroutine with function a.watchExternalNodes is started which sends data to nodeCh channel. In line 129, main goroutine blocks until nodeCh receives a value.

	nodeCh := make(chan []*api.Node)
	instanceCh := make(chan []*api.ServiceEntry)

	go a.watchExternalNodes(nodeCh, stopCh)
	go a.watchServiceInstances(instanceCh, stopCh)

	externalNodes := <-nodeCh
	healthyInstances := <-instanceCh

In function watchExternalNodes, querying for external node changes and sending latest data to nodeCh are done in a loop. The firstRun flag ensures the loop executes at least once i.e. it sends at least once to the channel nodeCh. But a.client.Catalog function call in line 240 can return error in which case the loop continues without sending to channel. On subsequent runs firstRun is false and it leads to checking stopCh channel for a stop signal. So in a case where the first iteration encounters error and also stopCh receives a signal before next iteration proceeds, the loop will return without ever sending to nodeCh which will cause the receive operation in line 129 to block indefinitely.

	firstRun := true
	for {
		if !firstRun {
			select {
			case <-stopCh:
				return
			case <-time.After(retryTime):
				// Sleep here to limit how much load we put on the Consul servers.
			}
		}
		firstRun = false

		// Do a blocking query for any external node changes
		externalNodes, meta, err := a.client.Catalog().Nodes(opts)
		if err != nil {
			a.logger.Warn("Error getting external node list", "error", err)
			continue
		}
		sort.Slice(externalNodes, func(a, b int) bool {
			return externalNodes[a].Node < externalNodes[b].Node
		})

		opts.WaitIndex = meta.LastIndex

		a.logger.Info("Updating external node list", "items", len(externalNodes))

		nodeCh <- externalNodes
	}

We aren't fully certain how probable such a situation is, but nevertheless we decided to bring it into the attention of developers to let them decide if this is something they would consider worth fixing.

Sponsorship and Support:

This work is done by the security researchers from OpenRefactory under the Project Clean Beach initiative. The researchers are proactively scanning critical open source projects and finding previously undetected bugs in them.

The bug is found by running the iCR tool by OpenRefactory and then manually triaging the results.