Werk is a C++ library of components for developing performance-sensitive, real-time applications. Each Werk component is available a-la-carte (albeit possibly with some dependencies, such as logging or configuration). A lightweight framework that combines the most common components is also available.
To facilitate development and testing, a simulator is provided which can replay historical data from a variety of formats (CSV, PCAP, etc.) as though it were happening live.
Werk uses Waf to build and automate other development
workflows like unit tests. To make things simple, a copy of Waf is included in
the distribution and may be run with ./waf, which by default will execute an
incremental build of the project.
If you try running that in a freshly cloned repository, you'll be told you need
to configure: ./waf configure. This sets settings such as debug/release. Run
./waf configure -d to configure a debug build.
To see help and additional options, run ./waf -h.
Unit tests may be run after building by running ./waf test. Because waf can
accept more than one action, you can build and test in one command:
./waf build test.
Profiling code is included for critical components to help identify performance
regressions. Run ./waf profile to profile.
To help identify memory errors, Valgrind is built into the build system.
To run tests or profiling with valgrind, add the --valgrind option:
./waf test --valgrind or ./waf profile --valgrind. When running outside of
valgrind, profiling also acts as a test that may fail if times are too high,
automatically identifying performance regressions.
The Hello World of Werk applications is included as an example of how to instantiate the framework. To run it (and format the JSON logs for human consumption), run:
> build/bin/HelloWorld | scripts/FormatLog.py
[00000] [1491267909.296769000] CONFIG - <Config> [Application.ConfigPaths] = (null) [DEFAULT]
[00002] [1491267909.296769000] SUCCESS - <Config> Initialized.
[00003] [1491267909.296769000] CONFIG - <Config> [Application.LogPath] = (null) [DEFAULT]
[00000] [1491267909.296769000] SUCCESS - <Log> Initialized.
...
Gitlab CI support is included via the .gitlab-ci.yml file in the root of the
repository. If available, it will build, unit test, profile, and functional test
(via the Hello World app) both debug and release after every push. All testing
and profiling code is run twice: once without valgrind and once with valgrind.
Although Werk is not a framework and does not enforce a particular threading model, it does have several high level design rules in order to meet its performance requirements:
- I/O and most other system calls should be deferred from any critical path thread into a background thread, and done asynchronously. Components that require I/O should be designed to support this, while still allowing synchronous execution when possible.
- In the same vein,
newgenerally should not be called after initialization. This has advantages besides avoiding a possiblesbrk()call; for example, it guarantees that the application will not fail after startup by running out of memory.
Components below are presented in roughly the order that they depend upon each
other. For a standard set of basic components, look at the ApplicationContext
class.
To enable analytics (e.g. for profiling), a variety of signal processing and statistical components are included:
ContinuousEma: Calculate an EMA with a continuous value.DiscreteDistribution: Calculates statistics about a weighted set of ordered values, including probability information.DiscreteEma: Calculate an EMA with discrete steps.OrderStatistics: Calculate fractiles and other order statistics on a set of samples.SimpleLinearRegression: Calculate a simple linear regression from two sets of samples.SummaryStatisticsandRangedSummaryStatistics: Calculate basic summary statistics (count, average, std-dev, and optionally min/max).WeightedSummaryStatistics: Calculate summary statistics on a weighted set of samples.
In order to maintain performant code and identify outliers which require remediation, a light-weight profiling implementation is provided.
Profiles keep track of the start and end times of a given sample, and then
insert the delta into an OrderStatistics instance. Once that instance has a
certain number of samples (default=100), the OrderStatistics is sampled at
various fractiles into a set SummaryStatistics instances, then cleared. This
allows for an O(1) implementation while still collecting information about the
50th, 75th, 90th, 95th, and 99th percentile times.
Profiles are named and held by a ProfileManager class which allows exporting
the summary statistics to JSON for upstream analysis.
To keep latency low on the main application thread(s), a BackgroundThread
class offers a place to defer I/O from other components. A set of Actions is
run at a configurable interval. Although actions do not have a latency budget,
they must return as all multitasking is cooperative. To ensure performance,
background actions are automatically profiled by providing a ProfileManager
to the BackgroundThread upon instantiation.
Built-in actions include:
ActionQueue: Queues otherActions to be run a single time -- useful for queueing actions to be run on another thread.ConditionalAction: Executes anotherActiononly if aLatchis set.CounterAction: Counts the number of times the action is executed.IncrementCounterAction: Increments aCounter(appropriate when a more complex counter is needed than the simple one inCounterAction).NullAction: Does nothing (useful as a placeholder or for testing).ResetAction: Resets any component that has avoid reset()method (Latch,Counter,Watchdog, etc.).SetLatchAction: Sets a flag.Watchdog: Watches a flag and executes anotherActionif it does not get before a predefined interval expires. This allows for deadlock detection and many other behaviors to be run on the background thread.
Logs are written in a machine-readable JSON lines
format. They can either be written synchronously with SyncLog or
asynchronously with AsyncLog on a BackgroundThread.
A reloadable configuration system which can read from multiple ConfigSources
(allowing integration with standard formats as well as custom and proprietary
ones) and propagate updates to a set of Configurable objects. As Configs are
read, the values are logged, making everything auditable.
To allow for easy configuration of actions, and also user input, string command
lines can be accepted and parsed by a CommandManager, which will forward
execution to the appropriate Command instance. Command managers optionally
start with a set of default, standard commands.
Built-in commands include:
echo,warning,error(EchoCommand): Logs the arguments at a certain log level.null(NullCommand): Does nothing (useful as a placeholder for testing).?,help: Logs information about commands.version: Logs the version of Werk the application is running.redo: Runs the last command again.
A CommandAction helper class allows a Command to be run whenever an Action
is needed, opening up many opportunities for connecting components.
Consoles allow users to connect to a real time run and enter commands. Currently
the only available console is the IpcConsoleServer and IpcConsoleClient pair
which send commands over an IPC queue. The client application is built as
bin/build/IpcConsoleClient.
Althought Werk does not force the use of any particular threading model or set
of components, it does offer a default combination of the most common components
via the ApplicationContext class. Bootstrapping from a background thread and a
logger that goes to stdout, it loads configs, creates a file log, sets up
subsystems like commands and profiling, then runs some startup commands. An
optional main loop may then be run.
On startup, default signal handlers are register for segfaults and bus errors. The background thread is started, then logging and configuration are bootstrapped:
Application.ConfigPaths: Only available in the primary config, this is a comma-separated list of other configs to load. Default none.Application.LogPath: Path to the actual log file. If not present or cannot be opened, logs will go to stderr. Required.
Then the following configuration options are read and their respective components initialized:
Application.InstanceID: ID of this instance of the application, for audit trail purposes.Application.ProfilesPath: Path to profiling information JSON, written on shutdown. Default none.Application.BackgroundThreadInterval: The interval of the background thread, in nanoseconds. Default 10ms.Application.Debug: Boolean indicating whether the application is in debug mode (causing it to output additional information). Default false.Application.Simulation: Boolean indicating whether the application is a simulation. Default false.Application.RealTime: Boolean indicating whether the application is running in real time. Default true.Application.IpcConsoleName: The name of the IPC queue used for a console. Default none.
Finally, the background task Scheduler is started and startup commands are
run:
Application.StartupCommands: Semicolon delimited list of command lines to run on startup.
If the optional main loop is used by calling ApplicationContext::run(), first
some configurations are read:
Application.WatchdogInterval: The interval of the foreground thread watchdog.Application.WatchdogAllowedMisses: The number of times the foreground thread can miss resetting the application watchdog.
Then a main loop is started which updates the current time, runs a main Action,
then executes any deferred Actions in the foreground queue and resets the
watchdog.
Runs without the real time flag set use a TimeSeriesReplayer to replay 1 or
more data sources (CSV's, PCAP's, etc.) as events. This allows for simple
testing against multiple sources of historical data.
Application.HistoricalDataSources: Comma-separated list of paths to historical data sources.Application.HistoricalDataTimeout: The maximum time between events (e.g.1sor10us) -- if there is a larger gap, additional events are generated at this interval.
The application context adds additional default commands:
app: Logs information about the high level state of the application.bg: Logs information about background tasks.logs: Logs information about what logs are available.reload: Reloads the configuration.quit: Quits the application cleanly, running shutdown commands and actions.
The application may be shut down cleanly at any time by running the quit
command. Sending it a SIGINT (e.g. by pressing Ctrl+C) will also cause a
clean shutdown.
On shutdown:
- Shutdown commands are run (read from
Application.ShutdownCommands). - Shutdown actions, which may be registered by any component, are run.
- The application exits with an exit code that may optionally be set (default 0).