To meta-trace or not to meta-trace

[gomoripeti]

Nomral tracing in Erlang can be controlled in two parts:

• erlang:trace/3 controls which processes to trace (all/none/new/existing/Pid) and global flags (like timestamp format, arity, tracer module, and which events to trace like call/proc/gc)
• erlang:trace_pattern/3 controls MFA+match-spec

In XProf the former is controlled by the singleton xprof_core_tracer
gen_server (for example to pause all tracing at once manually or automatically
in case of overload) and the later by xprof_core_trace_handler per each
MFA.

The problem is that we would like to set some of the global flags on a per MFA granularity. Eg use arity when we don't capture long calls. (This is worked around currently with some match-spec trick). With the dawn of our tracing NIF we would also like to set different state for the tracer module per MFA.

Meta-tracing traces all processes and does not care about the process trace
flags set by erlang:trace/3, the trace flags are instead fixed to [call, timestamp]. The match specification function return_trace() works with
meta-trace and sends its trace message to the same tracer.

Meta-tracing would allow to set a different tracer module with a different state
per MFA. (This would be ideal for us). (It does not allow to use the arity
flag, so this is out of question to use in the legacy solution without a tracer
module.)

On the other hand meta-tracing would always trace all the processes. (Currently
we only support tracing all the processes from the GUI anyway but tracing only 1
process from the shell works and this is used by some tests and profiling.) If
we would like to implement something else than all-the-processes we need to
implement it in the tracer module (enable function is a good candidate) and we
cannot use the VM's tracing infrastructure.

More importantly the role of XProf processes would change. The main
xprof_core_tracer would loose its global control of pausing all tracing at
once. (It could do this by calling erlang:trace_pattern/3 for each monitored
MFA.) It would also be possible to send trace messages directly to the per MFA
gen_servers instead of the singleton proxy process which would have main role
remaining to control starting/stopping monitoring MFAs. Overload protection
would be moved to the per MFA servers too. This would be a somewhat big
architectural change (To be investigated if removing the proxy is good or bad
from safety perspective.)

An alternative is to pass a map as the global tracer module state which has one entry per MFA. Unfortunately this needs to be updated whenever there is a change in any of the MFA states. Also needs to be done by the singleton xprof_core_tracer while per MFA state is the responsibility of the per MFA gen_servers.

Another similar tracing is call-counting which is also valid for all processes and controlled only by erlang:trace_pattern/3.