Tracing (software)

Software tracing is a tool for developers to gather information for debugging. This information is used both during development cycles and post-release. Unlike event logging, software tracing usually does not have the concept of a "class" of event or an "event code". Other reasons why event-logging solutions based on event codes are inappropriate for software tracing include:

• Because software tracing is low-level, there are often many more types of messages that would need to be defined, many of which would only be used at one place in the code. The event-code paradigm introduces significant development overhead for these "one-shot" messages.
• The types of messages that are logged are often less stable through the development cycle than for event logging.
• Because the tracing output is intended to be consumed by the developer, the messages don't need to be localized. Keeping tracing messages separate from other resources that need to be localized (such as event messages) is therefore important.
• There are messages that should never be seen.
• Tracing messages should be kept in the code, because they can add to the readability of the code. This is not always possible or feasible with event-logging solutions.

OpenTelemetry is a CNCF open source project that provides comprehensive support for distributed tracing.^[3] Some vendors including Datadog, New Relic, Splunk also offer tracing SaaS services.^[4]

Google and Meta have developed their own tracing frameworks namely Dapper and Canopy.^[2]

• Tracing with GNU Debugger's trace command^[5]
• Linux C/C++ application tracing with cwrap
• Linux application tracing with UST – part of the same project as LTTng
• Windows software trace preprocessor (aka WPP)
• Instruction set simulation

In operating systems, tracing can be used in situations (such as booting) where some of the technologies used to provide event logging may not be available.

Linux offers system-level and user-level tracing capabilities with kernel markers and LTTng. ftrace also supports Linux kernel tracing. syslog is another tool in various operating systems for logging and tracing system messages.

FreeBSD and SmartOS employ DTrace for tracing for the kernel and the userland.

In embedded software, tracing also requires special techniques for efficient instrumentation and logging and low CPU overhead.^[6]

Trace generation of method calls can be done with source code instrumentation, runtime information collection, or under debugger control.^[7] Tracing macros, Aspect-oriented programming and related instrumentation techniques can be employed.

Libraries used in source code send data to an agent or directly to the collection component.^[4]

To model execution trees, ISVis converts a rooted tree into a directed acyclic graph while Jinsight utilizes the call frame principle to gather and represent cumulative information about traces.^[7]

The primary visualization method is the swimlane view, which is exemplified by tools like Jaeger and often includes annotations and key-value attributes. Despite its widespread use, this design lacks rigorous justification and users frequently face challenges like missing features and confusing navigation. Alternatives to swimlane views exist, like Jaeger’s service dependency view or SkyWalking’s List, Tree, and Table views. Aggregate visualizations are also used for analyzing large volumes of traces, with systems like Canopy offering queryable metrics and Jaeger providing trace comparison features.^[8]

Event logging provides system administrators with information useful for diagnostics and auditing. The different classes of events that will be logged, as well as what details will appear in the event messages, are often considered early in the development cycle. Many event logging technologies allow or even require each class of event to be assigned a unique "code", which is used by the event logging software or a separate viewer (e.g., Event Viewer) to format and output a human-readable message. This facilitates localization and allows system administrators to more easily obtain information on problems that occur.

Because event logging is used to log high-level information (often failure information), performance of the logging implementation is often less important.

A special concern, preventing duplicate events from being recorded "too often" is taken care of through event throttling.

Difficulties in making a clear distinction between event logging and software tracing arise from the fact that some of the same technologies are used for both, and further because many of the criteria that distinguish between the two are continuous rather than discrete. The following table lists some important, but by no means precise or universal, distinctions that are used by developers to select technologies for each purpose, and that guide the separate development of new technologies in each area:

Enabling or disabling tracing during runtime often necessitates the inclusion of extra data in the binary. This can lead to performance degradation, even when tracing is not active.

If tracing is enabled or disabled at compile-time, collecting trace data from a client's system hinges on their willingness and capability to install a version of the software specifically enabled for tracing, and subsequently replicate the issue.

Tracing in software typically demands high standards of robustness, not only in the accuracy and reliability of the trace output but also in ensuring that the process being traced remains uninterrupted.

Given its low-level nature, tracing can generate a large volume of messages. To mitigate performance issues, it's often necessary to have the option to deactivate software tracing, either at the time of compilation or during run-time.

In proprietary software, tracing data may include sensitive information about the product's source code.

• Branch trace
• Debugging
• Instrumentation (computer programming)
• Logging
• Observability (software)