Adds new metric tag to Manager to denote primary

[dlmarion]

Modified MetricsInfoImpl that enables the global registry
to be re-initialized with new tags. The new reinit method
closes and removes all MeterRegistries, clears the Meters
from the global registry, and resets some other state to
allow init() to be called again. reinit calls init with an updated
list of common tags.

Modified the Manager to call reinit in setupPrimaryMetrics,
which is called when the Manager acquires the primary lock.
Before this is called the metrics being emitted by the manager
contain the tag manager.primary=false and after reinit the
tag is manager.primary=true.

Closes #6201

[dlmarion]

FWIW I started this going down the path of using MeterFilter.replaceTagValues to change the value of the manager.primary tag dynamically based on whether or not the Manager had the primary lock. This didn't work out well and I had to adjust course ending up re-init'ing the Metrics subsystem.

[keith-turner]

Instead of changing the tags on the metrics could have all manager processes have the same tags and create a new gauge that indicates if primary manager or not. Before the manager process get the primary lock the gauge reads zero and after it get the lock the gauge reads one.

[dlmarion]

Instead of changing the tags on the metrics could have all manager processes have the same tags and create a new gauge that indicates if primary manager or not. Before the manager process get the primary lock the gauge reads zero and after it get the lock the gauge reads one.

I thought about that, but I think that will make it hard when looking at some other metric to determine if it's coming from the primary or not (you have correlate it with some other metric). This would especially be hard for graphing purposes.

[dlmarion]

It might be useful to change the tag name to manager.mode with values assistant and primary.

[keith-turner]

We can also delay registration of metrics. If there are metrics that are only emitted by the primary, then we can delay registering those until after we get the lock. That way non primary managers are not emitting useless metrics.

[dlmarion]

We can also delay registration of metrics. If there are metrics that are only emitted by the primary, then we can delay registering those until after we get the lock. That way non primary managers are not emitting useless metrics.

I think it already does that. It registers the fate metrics, and then registers the rest after it gets the primary lock.

[keith-turner]

I think it already does that. It registers the fate metrics, and then registers the rest after it gets the primary lock.

ok, so if that is all working properly then only one manager process at a time should be emitting certain metrics. If that is the case then is a tag change still useful for query purposes?