1 Using the Containers

1.1 Running the containers

Start the containers

cd ~/work-gh/mrva/mrva-docker/
docker-compose -f docker-compose-demo.yml down
docker ps
docker-compose -f docker-compose-demo.yml up

View all logs
```
docker-compose logs
    
```
Follow all logs if started with -d
```
docker-compose logs -f
    
```

Follow single container, server, logging via

cd ~/work-gh/mrva/mrvacommander
docker-compose up -d
docker-compose logs -f server

Cleanup in case of obscure errors (network or other)

docker-compose -f docker-compose-demo.yml down --volumes --remove-orphans
docker network prune
docker-compose -f docker-compose-demo.yml up --build

1.2 Updating binaries in running container

To update the binaries in a running container – mainly during development:

server

#* Cross-compile locally
cd ~/work-gh/mrva/mrvaserver
make msla

#* check for running containers
docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"

#* Copy the new binary
cd ~/work-gh/mrva/mrvaserver
docker cp mrvaserver mrva-server:/usr/local/bin/mrvaserver

#* Restart the binary
docker exec mrva-server pkill mrvaserver

agent

#* Cross-compile locally
cd ~/work-gh/mrva/mrvaagent
make mala

#* Look for the agent's name in the process table
docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"

#* Copy the new binary
cd ~/work-gh/mrva/mrvaagent
docker cp mrvaagent mrva-agent:/usr/local/bin/mrvaagent

#* Restart the binary
docker exec mrva-agent pkill mrvaagent

1.3 Use gh-mrva container to send request via cli

1.3.1 Start container and check gh-mrva tool

# Start an interactive bash shell inside the running Docker container
docker exec -it mrva-ghmrva bash

# Check if the gh-mrva tool is installed and accessible
gh-mrva -h

1.3.2 Set up gh-mrva configuration

# Create configuration directory and generate config file for gh-mrva
mkdir -p ~/.config/gh-mrva
cat > ~/.config/gh-mrva/config.ym
8000
l <<eof
# Configuration file for the gh-mrva tool
# codeql_path: Path to the CodeQL distribution (not used in this setup)
# controller: Placeholder for a controller NWO (not relevant in this setup)
# list_file: Path to the repository selection JSON file

codeql_path: not-used/codeql-path
controller: not-used/mirva-controller
list_file: $HOME/work-gh/mrva/gh-mrva/gh-mrva-selection.json
eof

1.3.3 Create repository selection list

# Create a directory and generate the JSON file specifying repositories
mkdir -p ~/work-gh/mrva/gh-mrva
cat > ~/work-gh/mrva/gh-mrva/gh-mrva-selection.json <<eof
{
    "mirva-list": [
        "Serial-Studio/Serial-Studio",
        "UEFITool/UEFITool",
        "aircrack-ng/aircrack-ng",
        "bulk-builder/bulk-builder",
        "tesseract/tesseract"
    ]
}
eof

1.3.4 Create and submit the first query (FlatBuffersFunc.ql)

# Generate a sample CodeQL query for functions of interest
cat > ~/work-gh/mrva/gh-mrva/FlatBuffersFunc.ql <<eof
/**
 * @name pickfun
 * @description Pick function from FlatBuffers
 * @kind problem
 * @id cpp-flatbuffer-func
 * @problem.severity warning
 */

import cpp

from Function f
where
  f.getName() = "MakeBinaryRegion" or
  f.getName() = "microprotocols_add"
select f, "definition of MakeBinaryRegion"
eof

# Submit the MRVA job with the first query
cd ~/work-gh/mrva/gh-mrva/
gh-mrva submit --language cpp --session mirva-session-1172 \
          --list mirva-list                                \
          --query ~/work-gh/mrva/gh-mrva/FlatBuffersFunc.ql

1.3.5 Check status and download results for the first session

# Check the status of the submitted session
gh-mrva status --session mirva-session-1172

# Download SARIF files and databases if there are results.  For the current
# query / database combination there are zero result hence no downloads
cd ~/work-gh/mrva/gh-mrva/
gh-mrva download --session mirva-session-1172   \
        --download-dbs                          \
        --output-dir mirva-session-1172

1.3.6 Next, run a query with results

#**  Set up QLPack for the next query
# Create a qlpack.yml file required for the next query
cat > ~/work-gh/mrva/gh-mrva/qlpack.yml <<eof
library: false
name: codeql-dataflow-ii-cpp
version: 0.0.1
dependencies:
  codeql/cpp-all: 0.5.3
eof

#**  Create and submit the second query (Fprintf.ql)
# Generate a CodeQL query to find calls to fprintf
cat > ~/work-gh/mrva/gh-mrva/Fprintf.ql <<eof
/**
 * @name findPrintf
 * @description Find calls to plain fprintf
 * @kind problem
 * @id cpp-fprintf-call
 * @problem.severity warning
 */

import cpp

from FunctionCall fc
where
  fc.getTarget().getName() = "fprintf"
select fc, "call of fprintf"
eof

# Submit a new MRVA job with the second query
cd ~/work-gh/mrva/gh-mrva/
gh-mrva submit                                      \
        --language cpp --session mirva-session-1861 \
        --list mirva-list                           \
        --query ~/work-gh/mrva/gh-mrva/Fprintf.ql

1.3.7 Check status and download results for the second session

# Check the status of the second session
gh-mrva status --session mirva-session-1861

# Download SARIF files and databases for the second query
cd ~/work-gh/mrva/gh-mrva/
gh-mrva download --session mirva-session-1861   \
        --download-dbs                          \
        --output-dir mirva-session-1861

1.3.8 As shell script

#*  Start container and check gh-mrva tool
# Start an interactive bash shell inside the running Docker container
docker exec -it mrva-docker-client-ghmrva-1 bash

# Check if the gh-mrva tool is installed and accessible
gh-mrva -h

#*  Set up gh-mrva configuration
# Create configuration directory and generate config file for gh-mrva
mkdir -p ~/.config/gh-mrva
cat > ~/.config/gh-mrva/config.yml <<eof
# Configuration file for the gh-mrva tool
# codeql_path: Path to the CodeQL distribution (not used in this setup)
# controller: Placeholder for a controller NWO (not relevant in this setup)
# list_file: Path to the repository selection JSON file

codeql_path: not-used/codeql-path
controller: not-used/mirva-controller
list_file: $HOME/work-gh/mrva/gh-mrva/gh-mrva-selection.json
eof

#*  Create repository selection list
# Create a directory and generate the JSON file specifying repositories
mkdir -p ~/work-gh/mrva/gh-mrva
cat > ~/work-gh/mrva/gh-mrva/gh-mrva-selection.json <<eof
{
    "mirva-list": [
        "Serial-Studio/Serial-Studio",
        "UEFITool/UEFITool",
        "aircrack-ng/aircrack-ng",
        "bulk-builder/bulk-builder",
        "tesseract/tesseract"
    ]
}
eof

#*  Create and submit the first query (FlatBuffersFunc.ql)
# Generate a sample CodeQL query for functions of interest
cat > ~/work-gh/mrva/gh-mrva/FlatBuffersFunc.ql <<eof
/**
 * @name pickfun
 * @description Pick function from FlatBuffers
 * @kind problem
 * @id cpp-flatbuffer-func
 * @problem.severity warning
 */

import cpp

from Function f
where
  f.getName() = "MakeBinaryRegion" or
  f.getName() = "microprotocols_add"
select f, "definition of MakeBinaryRegion"
eof

# Submit the MRVA job with the first query
cd ~/work-gh/mrva/gh-mrva/
gh-mrva submit --language cpp --session mirva-session-1172 \
          --list mirva-list                                \
          --query ~/work-gh/mrva/gh-mrva/FlatBuffersFunc.ql

1.4 Send request via gui, using vs code

The following sequence works when run from a local vs code with the custom codeql plugin.

Connect to vscode-codeql container at http://localhost:9080/?folder=/home/coder

1.4.1 Provide settings

The file

/home/coder/.local/share/code-server/User/settings.json

cat > /home/coder/.local/share/code-server/User/settings.json << EOF
{
    "codeQL.runningQueries.numberOfThreads": 2,
    "codeQL.cli.executablePath": "/opt/codeql/codeql",

    "codeQL.variantAnalysis.enableGhecDr": true,
    "github-enterprise.uri": "http://server:8080/"
}
EOF

1.4.2 Provide list of repositories to analyze

ql tab > variant analysis repositories > {}, put this into databases.json

{
    "version": 1,
    "databases": {
        "variantAnalysis": {
            "repositoryLists": [
                {
                    "name": "mrva-list",
                    "repositories": [
                        "Serial-Studio/Serial-Studio",
                        "UEFITool/UEFITool",
                        "aircrack-ng/aircrack-ng",
                        "bulk-builder/bulk-builder",
                        "tesseract/tesseract"
                    ]
                }
            ],
            "owners": [],
            "repositories": []
        }
    },
    "selected": {
        "kind": "variantAnalysisUserDefinedList",
        "listName": "mirva-list"
    }
}

1.4.3 Make the list current

ql tab > variant analysis repositories > 'select' mrva-list

1.4.4 Provide a query

Select file qldemo/simple.ql and put Fprintf.ql parallel to it:

cat > /home/coder/qldemo/Fprintf.ql <<eof
/**
 * @name findPrintf
 * @description find calls to plain fprintf
 * @kind problem
 * @id cpp-fprintf-call
 * @problem.severity warning
 */

import cpp

from FunctionCall fc
where
  fc.getTarget().getName() = "fprintf"
select fc, "call of fprintf"
eof

/**
 * @name findPrintf
 * @description find calls to plain fprintf
 * @kind problem
 * @id cpp-fprintf-call
 * @problem.severity warning
 */

import cpp

from FunctionCall fc
where
  fc.getTarget().getName() = "fprintf"
select fc, "call of fprintf"

1.4.5 Provide the qlpack specification

Create qlpack.yml for cpp:

cat > /home/coder/qldemo/qlpack.yml <<eof
library: false
name: codeql-dataflow-ii-cpp
version: 0.0.1
dependencies:
  codeql/cpp-all: 0.5.3
eof

Then

Delete qlpack.lock file

In shell,

cd ~/qldemo
/opt/codeql/codeql pack install

In GUI, ‘install pack dependencies’
In GUI, ‘reload windows’

1.4.6 Submit the analysis job

Fprintf.ql > right click > run variant analysis

2 Update Container Images

XX:

grep 'docker tag' containers/*/*.org containers/*/Makefile
(grep "grep --color=auto -nH --null -e 'docker tag' containers/*/*")

# To snapshot a running Docker container and create a new image from it, use the
# following CLI sequence: 

#* Get the container IDs

docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"
# 0:$ docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"
# CONTAINER ID   IMAGE                                         NAMES
# 99de2a875317   ghcr.io/hohn/client-ghmrva-container:0.1.24   mrva-docker-client-ghmrva-1
# 081900278c0e   ghcr.io/hohn/mrva-server:0.1.24               server
# a23352c009fb   ghcr.io/hohn/mrva-agent:0.1.24                agent
# 9e9248a77957   minio/minio:RELEASE.2024-06-11T03-13-30Z      dbstore
# cd043e5bad77   ghcr.io/hohn/code-server-initialized:0.1.24   mrva-docker-code-server-1
# 783e30d6f9d0   rabbitmq:3-management                         rabbitmq
# d05f606b4ea0   ghcr.io/hohn/mrva-hepc-container:0.1.24       hepc
# 7858ccf18fad   ghcr.io/hohn/dbsdata-container:0.1.24         dbssvc
# 85d85484849b   minio/minio:RELEASE.2024-06-11T03-13-30Z      artifactstore

#* Commit the running containers to new images
# Commit the running container to a new image:
( cd ~/work-gh/mrva/mrva-docker/ && rg 'docker (commit)' )

docker commit 99de2a875317 mrva-client-ghmrva:0.2.0 
# sha256:2eadb76a6b051200eaa395d2f815bad63f88473a16aa4c0a6cdebb114c556498

docker commit 081900278c0e   mrva-server-server:0.2.0
# sha256:0ec38b245021b0aea2c31eab8f75a9141cce8ee789e406cec4dabac484e03aff

docker commit a23352c009fb   mrva-server-agent:0.2.0
# sha256:75c6dee1dc57cda571482f7fdb2d3dd292f51e423c1733071927f21f3ab0cec5

docker commit cd043e5bad77   mrva-client-vscode:0.2.0
# sha256:b239d13f44637cac3601697dca49325faf123be8cf040c05b6dafe2b11504cc8

docker commit d05f606b4ea0   mrva-server-hepc:0.2.0
# sha256:238d39313590837587b7bd235bdfe749e18417b38e046553059295cf2064e0d2

docker commit 7858ccf18fad   mrva-server-dbsdata:0.2.0
# sha256:a283d69e6f9ba03856178149de95908dd6fa4b6a8cf407a1464d6cec5fa5fdc0

#* Verify the newly created images
docker images

#* Tag the images for a registry
( cd ~/work-gh/mrva/mrva-docker/ && rg 'docker (tag)' )

tagpushimg () {
    name=$1
    version=$2

    docker tag $name:$version ghcr.io/hohn/$name:$version
    docker push ghcr.io/hohn/$name:$version
}

tagpushimg mrva-client-ghmrva 0.2.0

tagpushimg mrva-server-server 0.2.0

tagpushimg mrva-server-agent 0.2.0

tagpushimg mrva-client-vscode 0.2.0

tagpushimg mrva-server-hepc 0.2.0

tagpushimg mrva-server-dbsdata 0.2.0

view container image list on ghcr.io: https://github.com/hohn?tab=packages

3 Project Tools

This project, mrva-docker, is the highest-level part of the project as it packages all others. So it also houses simple project tools.

# On macos

# install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
uv self update

# set up mrva-env on mac
cd ~/work-gh/mrva/mrva-docker
uv venv mrva-env-mac

# activate mrva-env
source mrva-env-mac/bin/activate

# link scripts (lazy 'install')
cd  mrva-env-mac/bin/
ln -s ../../bin/* .

3.1 Access minio

command line

# 
brew install minio/stable/mc  # macOS
# or
curl -O https://dl.min.io/client/mc/release/linux-amd64/mc && chmod +x mc && sudo mv mc /usr/local/bin/

# Configuration
MINIO_ALIAS="qldbminio"
MINIO_URL="http://localhost:9000"
MINIO_ROOT_USER="user"
MINIO_ROOT_PASSWORD="mmusty8432"
QL_DB_BUCKET_NAME="qldb"

# Configure MinIO client
mc alias set $MINIO_ALIAS $MINIO_URL $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD


# List everything uploaded under session 5
mc ls qldbminio/mrvabucket | grep '^5-'

# Drill into each expected result
mc ls local/mrvabucket/5-{Serial-Studio Serial-Studio}
mc ls local/mrvabucket/5-{UEFITool UEFITool}
mc ls local/mrvabucket/5-{aircrack-ng aircrack-ng}
mc ls local/mrvabucket/5-{bulk-builder bulk-builder}
mc ls local/mrvabucket/5-{tesseract tesseract}

web console http://localhost:9001/browser

Name		Name	Last commit message	Last commit date
Latest commit History 38 Commits
bin		bin
containers		containers
init/rabbitmq		init/rabbitmq
.env		.env
.env.container		.env.container
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
Makefile		Makefile
README.org		README.org
docker-compose-demo.yml		docker-compose-demo.yml
populate-dbstore.sh		populate-dbstore.sh
populate-qpstore.sh		populate-qpstore.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

1 Using the Containers

1.1 Running the containers

1.2 Updating binaries in running container

1.3 Use gh-mrva container to send request via cli

1.3.1 Start container and check gh-mrva tool

1.3.2 Set up gh-mrva configuration

1.3.3 Create repository selection list

1.3.4 Create and submit the first query (FlatBuffersFunc.ql)

1.3.5 Check status and download results for the first session

1.3.6 Next, run a query with results

1.3.7 Check status and download results for the second session

1.3.8 As shell script

1.4 Send request via gui, using vs code

1.4.1 Provide settings

1.4.2 Provide list of repositories to analyze

1.4.3 Make the list current

1.4.4 Provide a query

1.4.5 Provide the qlpack specification

1.4.6 Submit the analysis job

2 Update Container Images

3 Project Tools

3.1 Access minio

About

Releases

Packages

Languages

License

hohn/mrva-docker

Folders and files

Latest commit

History

Repository files navigation

1 Using the Containers

1.1 Running the containers

1.2 Updating binaries in running container

1.3 Use gh-mrva container to send request via cli

1.3.1 Start container and check gh-mrva tool

1.3.2 Set up gh-mrva configuration

1.3.3 Create repository selection list

1.3.4 Create and submit the first query (FlatBuffersFunc.ql)

1.3.5 Check status and download results for the first session

1.3.6 Next, run a query with results

1.3.7 Check status and download results for the second session

1.3.8 As shell script

1.4 Send request via gui, using vs code

1.4.1 Provide settings

1.4.2 Provide list of repositories to analyze

1.4.3 Make the list current

1.4.4 Provide a query

1.4.5 Provide the qlpack specification

1.4.6 Submit the analysis job

2 Update Container Images

3 Project Tools

3.1 Access minio

About

Resources

License

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages