Emma Memory and Mapfile Analyser (Emma)

Conduct static (i.e. worst case) memory consumption analyses based on arbitrary linker map files. It produces extensive .csv files which are easy to filter and post-process. Optionally .html and markdown reports as well as neat figures help you visualising your results.

Given a map file input (Green Hills map files are the default but others - like GCC - are supported via configuration options; examples are enclosed) Emma maps the addresses of sections (aka images) and/or objects (aka modules) to memory regions (all addresses given via map files must be known during compile time). Those memory regions are classified into two levels of granularity respectively. The first level defines arbitrary groups based on your personal taste (however using names similar to those defined by your microcontroller vendor makes most sense). Later each of those regions (second level) are assigned to one of four generalised predefined memory regions (those are: INT_RAM, INT_FLASH, EXT_RAM, EXT_FLASH). In case of virtual memory objects and sections lying within virtual address spaces (VASes) get translated back into physical memory. This is depicted in the figure above (lower part).

Categorisation can be used to assign consumers (a consumer would usually represent a software component) to each object or section. This is useful for subsequent steps in order to display memory consumption per consumer type. See the upper part of the figure shown above. Mechanisms are provided to batch categorise object and section names. "Objects in sections" provides ways to obtain a finer granularity of the categorisation result. Therefore categorised sections containing (smaller) objects of a different category got split up and result into a more accurate categorisation.

As a result you will get output files in form of a .csv file which sets you up to do later processing on this data easily. In this file additional information is added like:

• Overlaps (of sections/objects)
• Containments (e.g. sections containing objects)
• Duplicates
• All meta data about the origin of each section/object (mapfile, addess space, ...)
• ...

Holding the aforementioned augmented data makes it easy to detect issues in linker scripts and get an in-depth understanding of your program's memory consumption. Including a lot of additional and "corrected" data can cause confusion. Thus all original (unmodified) data is preserved in the output files simultaneously.

The Emma visualiser helps you to create nice plots and reports in a .png and .html and markdown file format.

The whole Emma tool suite contains command line options making it convenient to be run on a build server like --Werror (treat all warnings as errors) or --no-prompt (exit and fail on user prompts; user prompts can happen when ambiguous configurations appear such as multiple matches for one configured map files).

pip3 install pypiemma

Dependencies: Python 3.6 or higher; pip3 install Pygments Markdown matplotlib pandas pypiscout

python setup.py build_ext --inplace --compiler=msvc

The following figure shows a possible workflow using Emma:

Emma - as the core component - produces an intermediate .csv file. Inputs are mapfiles and JSON files (for configuration (memory layout, sizes, ...)). From this point you are very flexible to choose your own pipeline. You could

• use the Emma tools (Emma Visualiser, Emma Deltas, ...) for further processing (data aggregation and analysis),
• simply your favourite spreadsheet software (like Microsoft Excel or LibreOffice Calc) or
• use your own tool for the data analysis.

At this point we want to give you a brief overview what to do in the below two scenarios. If you want to play around go to (project files are already present) and use our example projects in ./doc/test_project*.

• if the Emma project is already set-up (JSON files were created) and you want to analyse your software with newly generated mapfiles proceed to -> Project files are already present or
• you start your analysis from scratch and need to do configure Emma before you use it then go to -> Project files have to be created.

Example projects (including Emma* outputs/results) can be found in ./doc/test_project*.

Since version 3.1 Emma can be called in two ways (if you want to run it from the installation folder) where the following variant is recommended:

python Emma.py a --project doc/test_project --mapfiles doc/test_project/mapfiles

The following table provides an overview how you call Emma:

Try python Emma.py a --help to see all possible options or refer to the documentation (./doc/*).

1. Create intermediate .csv from mapfiles with Emma:

python Emma.py a -p .\MyProjectFolder --map .\MyProjectFolder\mapfiles --dir .\MyProjectFolder\analysis --subdir Analysis_1

2. Generate reports and graphs with Emma Visualiser:

python Emma.py v -p .\MyProjectFolder --dir .\MyProjectFolder\analysis --subdir Analysis_1 -q 

To create a new project, the following files must be created:

• globalConfig.json
• budgets.json
• categories.json
• categoriesKeywords.json
• categoriesSections.json
• categoriesSectionsKeywords.json

You will find example projects in ./doc/test_project*. In-depth documentation can be found in the full documentation (see ./doc/).

A basic configuration can be short per file. For complex systems you can choose from many optional keywords/options that will provide you means to adjust your analysis as fine grained as you wish.

One main concept includes the globalConfig.json. You can see this as meta-config. Each configuration ID (configID) is a separately conducted analysis. Per configID you state individually the configuration files you want to use for this exact analysis. Herewith you can mix and match any combination of subconfigs you prefer.

A globalConfig.json could look like this:

{
    "configID1": {
        "addressSpacesPath": "addressSpaces.json",
        "sectionsPath": "sections.json",
        "patternsPath": "patterns.json"
    }
}

For the full documentation please refer to the ./doc/ directory.

We are glad if you want to participate. In ./doc/dev-guide.md you will find a guide telling you everything you need to know including coding conventions and more.

emma-dev (. at) googlegroups.com

Optional dependencies:

Utility scripts in ./doc/ need additional dependencies. As a normal user you can ignore this.

Please refer to the gprof2dot project site and install its dependencies (this has to be done even if you install Emma via pip).

Note that those modules are invoked via subprocess calls within the ./genDoc/ scripts.

Dependencies used to generate documentation for GitHub pages (separate, independent branch gh-pages):

Utility scripts used to build GitHub pages documentation. As a normal user you can ignore this.

Code snippets etc.:

For the full documentation please refer to the ./doc/ directory.