[PATCH v2] codeql: add support for analyzing C, Python and Go

[PATCH v2] codeql: add support for analyzing C, Python and Go

[Roger Pau Monne]

Introduce CodeQL support for Xen and analyze the C, Python and Go
files.

Note than when analyzing Python or Go we avoid building the hypervisor
and only build the tools.

Requested-by: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v1:
 - Rename to note it's x86 specific right now.
 - Merge the ignored path patch.
---
It's my understanding that we need to force the checkout action to
fetch 'staging' branch, or else for the scheduled runs we would end up
picking the current default branch (master).

Maybe we want to remove the scheduled action and just rely on pushes
and manually triggered workflows?
---
 .github/codeql/codeql-config.yml |  3 ++
 .github/workflows/codeql-x86.yml | 60 ++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)
 create mode 100644 .github/codeql/codeql-config.yml
 create mode 100644 .github/workflows/codeql-x86.yml

diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml
new file mode 100644
index 0000000000..721640c2a5
--- /dev/null
+++ b/.github/codeql/codeql-config.yml
@@ -0,0 +1,3 @@
+paths-ignore:
+  - xen/tools/kconfig
+  - tools/firmware/xen-dir/xen-root/xen/tools/kconfig
diff --git a/.github/workflows/codeql-x86.yml b/.github/workflows/codeql-x86.yml
new file mode 100644
index 0000000000..a3ec6236c4
--- /dev/null
+++ b/.github/workflows/codeql-x86.yml
@@ -0,0 +1,60 @@
+name: CodeQL x86
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [staging]
+  schedule:
+    - cron: '18 10 * * WED,SUN' # Bi-weekly at 10:18 UTC
+
+jobs:
+  analyse:
+
+    strategy:
+      matrix:
+        language: [ 'cpp', 'python', 'go' ]
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Install build dependencies
+      run: |
+        sudo apt-get install -y wget git \
+          libbz2-dev build-essential \
+          zlib1g-dev libncurses5-dev iasl \
+          libbz2-dev e2fslibs-dev uuid-dev libyajl-dev \
+          autoconf libtool liblzma-dev \
+          python3-dev golang python-dev libsystemd-dev
+
+    - uses: actions/checkout@v2
+      with:
+        ref: staging
+
+    - name: Configure Xen
+      run: |
+        ./configure --with-system-qemu=/bin/true \
+                    --with-system-seabios=/bin/true \
+                    --with-system-ovmf=/bin/true
+
+    - name: Pre build stuff
+      run: |
+        make -j`nproc` mini-os-dir
+
+    - uses: github/codeql-action/init@v1
+      with:
+        config-file: ./.github/codeql/codeql-config.yml
+        languages: ${{matrix.language}}
+        queries: security-and-quality
+
+    - if: matrix.language == 'cpp'
+      name: Full Build
+      run: |
+        make -j`nproc` build-xen build-tools
+        make -j`nproc` -C extras/mini-os/
+
+    - if: matrix.language == 'python' || matrix.language == 'go'
+      name: Tools Build
+      run: |
+        make -j`nproc` build-tools
+
+    - uses: github/codeql-action/analyze@v1
-- 
2.34.1

Re: [PATCH v2] codeql: add support for analyzing C, Python and Go

[Roger Pau Monné]

Ping?

On Mon, Mar 07, 2022 at 05:45:52PM +0100, Roger Pau Monne wrote:
> Introduce CodeQL support for Xen and analyze the C, Python and Go
> files.
> 
> Note than when analyzing Python or Go we avoid building the hypervisor
> and only build the tools.
> 
> Requested-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> Changes since v1:
>  - Rename to note it's x86 specific right now.
>  - Merge the ignored path patch.
> ---
> It's my understanding that we need to force the checkout action to
> fetch 'staging' branch, or else for the scheduled runs we would end up
> picking the current default branch (master).
> 
> Maybe we want to remove the scheduled action and just rely on pushes
> and manually triggered workflows?
> ---
>  .github/codeql/codeql-config.yml |  3 ++
>  .github/workflows/codeql-x86.yml | 60 ++++++++++++++++++++++++++++++++
>  2 files changed, 63 insertions(+)
>  create mode 100644 .github/codeql/codeql-config.yml
>  create mode 100644 .github/workflows/codeql-x86.yml
> 
> diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml
> new file mode 100644
> index 0000000000..721640c2a5
> --- /dev/null
> +++ b/.github/codeql/codeql-config.yml
> @@ -0,0 +1,3 @@
> +paths-ignore:
> +  - xen/tools/kconfig
> +  - tools/firmware/xen-dir/xen-root/xen/tools/kconfig
> diff --git a/.github/workflows/codeql-x86.yml b/.github/workflows/codeql-x86.yml
> new file mode 100644
> index 0000000000..a3ec6236c4
> --- /dev/null
> +++ b/.github/workflows/codeql-x86.yml
> @@ -0,0 +1,60 @@
> +name: CodeQL x86
> +
> +on:
> +  workflow_dispatch:
> +  push:
> +    branches: [staging]
> +  schedule:
> +    - cron: '18 10 * * WED,SUN' # Bi-weekly at 10:18 UTC
> +
> +jobs:
> +  analyse:
> +
> +    strategy:
> +      matrix:
> +        language: [ 'cpp', 'python', 'go' ]
> +
> +    runs-on: ubuntu-latest
> +
> +    steps:
> +    - name: Install build dependencies
> +      run: |
> +        sudo apt-get install -y wget git \
> +          libbz2-dev build-essential \
> +          zlib1g-dev libncurses5-dev iasl \
> +          libbz2-dev e2fslibs-dev uuid-dev libyajl-dev \
> +          autoconf libtool liblzma-dev \
> +          python3-dev golang python-dev libsystemd-dev
> +
> +    - uses: actions/checkout@v2
> +      with:
> +        ref: staging
> +
> +    - name: Configure Xen
> +      run: |
> +        ./configure --with-system-qemu=/bin/true \
> +                    --with-system-seabios=/bin/true \
> +                    --with-system-ovmf=/bin/true
> +
> +    - name: Pre build stuff
> +      run: |
> +        make -j`nproc` mini-os-dir
> +
> +    - uses: github/codeql-action/init@v1
> +      with:
> +        config-file: ./.github/codeql/codeql-config.yml
> +        languages: ${{matrix.language}}
> +        queries: security-and-quality
> +
> +    - if: matrix.language == 'cpp'
> +      name: Full Build
> +      run: |
> +        make -j`nproc` build-xen build-tools
> +        make -j`nproc` -C extras/mini-os/
> +
> +    - if: matrix.language == 'python' || matrix.language == 'go'
> +      name: Tools Build
> +      run: |
> +        make -j`nproc` build-tools
> +
> +    - uses: github/codeql-action/analyze@v1
> -- 
> 2.34.1
> 

Re: [PATCH v2] codeql: add support for analyzing C, Python and Go

[Andrew Cooper]

[-- Attachment #1: Type: text/plain, Size: 4144 bytes --]

On 21/03/2022 09:54, Roger Pau Monné wrote:

Ping?

On Mon, Mar 07, 2022 at 05:45:52PM +0100, Roger Pau Monne wrote:


Introduce CodeQL support for Xen and analyze the C, Python and Go
files.

Note than when analyzing Python or Go we avoid building the hypervisor
and only build the tools.

Requested-by: Andrew Cooper <andrew.cooper3@citrix.com><mailto:andrew.cooper3@citrix.com>
Signed-off-by: Roger Pau Monné <roger.pau@citrix.com><mailto:roger.pau@citrix.com>
---
Changes since v1:
 - Rename to note it's x86 specific right now.
 - Merge the ignored path patch.
---
It's my understanding that we need to force the checkout action to
fetch 'staging' branch, or else for the scheduled runs we would end up
picking the current default branch (master).

Forcing to staging necessary due to a limitation in Coverity.

CodeQL explicitly can cope with multiple branches, so when a user asks for a specific branch, they'd better get a run on the branch they asked for, not have it forced to staging.

It also breaks any fork which has a different default branch.




Maybe we want to remove the scheduled action and just rely on pushes
and manually triggered workflows?
---
 .github/codeql/codeql-config.yml |  3 ++
 .github/workflows/codeql-x86.yml | 60 ++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)
 create mode 100644 .github/codeql/codeql-config.yml
 create mode 100644 .github/workflows/codeql-x86.yml

diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml
new file mode 100644
index 0000000000..721640c2a5
--- /dev/null
+++ b/.github/codeql/codeql-config.yml
@@ -0,0 +1,3 @@
+paths-ignore:
+  - xen/tools/kconfig
+  - tools/firmware/xen-dir/xen-root/xen/tools/kconfig

From actually running this:

Annotations
2 warnings
analyse (go)
The "paths"/"paths-ignore" fields of the config only have effect for JavaScript, Python, and Ruby
analyse (cpp)
The "paths"/"paths-ignore" fields of the config only have effect for JavaScript, Python, and Ruby

So this obviously can't be used like this.  You'll have to add them to the prebuild step.



diff --git a/.github/workflows/codeql-x86.yml b/.github/workflows/codeql-x86.yml
new file mode 100644
index 0000000000..a3ec6236c4
--- /dev/null
+++ b/.github/workflows/codeql-x86.yml
@@ -0,0 +1,60 @@
+name: CodeQL x86
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [staging]
+  schedule:
+    - cron: '18 10 * * WED,SUN' # Bi-weekly at 10:18 UTC
+
+jobs:
+  analyse:
+
+    strategy:
+      matrix:
+        language: [ 'cpp', 'python', 'go' ]
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Install build dependencies
+      run: |
+        sudo apt-get install -y wget git \
+          libbz2-dev build-essential \
+          zlib1g-dev libncurses5-dev iasl \
+          libbz2-dev e2fslibs-dev uuid-dev libyajl-dev \
+          autoconf libtool liblzma-dev \
+          python3-dev golang python-dev libsystemd-dev
+
+    - uses: actions/checkout@v2
+      with:
+        ref: staging
+
+    - name: Configure Xen
+      run: |
+        ./configure --with-system-qemu=/bin/true \
+                    --with-system-seabios=/bin/true \
+                    --with-system-ovmf=/bin/true
+
+    - name: Pre build stuff
+      run: |
+        make -j`nproc` mini-os-dir
+
+    - uses: github/codeql-action/init@v1
+      with:
+        config-file: ./.github/codeql/codeql-config.yml
+        languages: ${{matrix.language}}
+        queries: security-and-quality

This generates 1117 alerts, lots of which are of dubious utility.  I'd drop the queries line and go with the default, to reduce the triage initially.

~Andrew



+
+    - if: matrix.language == 'cpp'
+      name: Full Build
+      run: |
+        make -j`nproc` build-xen build-tools
+        make -j`nproc` -C extras/mini-os/
+
+    - if: matrix.language == 'python' || matrix.language == 'go'
+      name: Tools Build
+      run: |
+        make -j`nproc` build-tools
+
+    - uses: github/codeql-action/analyze@v1
--
2.34.1




[-- Attachment #2: Type: text/html, Size: 5272 bytes --]

Re: [PATCH v2] codeql: add support for analyzing C, Python and Go

[Roger Pau Monné]

On Mon, Mar 21, 2022 at 01:02:30PM +0000, Andrew Cooper wrote:
> On 21/03/2022 09:54, Roger Pau Monné wrote:
> 
> Ping?
> 
> On Mon, Mar 07, 2022 at 05:45:52PM +0100, Roger Pau Monne wrote:
> 
> 
> Introduce CodeQL support for Xen and analyze the C, Python and Go
> files.
> 
> Note than when analyzing Python or Go we avoid building the hypervisor
> and only build the tools.
> 
> Requested-by: Andrew Cooper <andrew.cooper3@citrix.com><mailto:andrew.cooper3@citrix.com>
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com><mailto:roger.pau@citrix.com>
> ---
> Changes since v1:
>  - Rename to note it's x86 specific right now.
>  - Merge the ignored path patch.
> ---
> It's my understanding that we need to force the checkout action to
> fetch 'staging' branch, or else for the scheduled runs we would end up
> picking the current default branch (master).
> 
> Forcing to staging necessary due to a limitation in Coverity.
> 
> CodeQL explicitly can cope with multiple branches, so when a user asks for a specific branch, they'd better get a run on the branch they asked for, not have it forced to staging.
> 
> It also breaks any fork which has a different default branch.
> 
> 
> 
> 
> Maybe we want to remove the scheduled action and just rely on pushes
> and manually triggered workflows?
> ---
>  .github/codeql/codeql-config.yml |  3 ++
>  .github/workflows/codeql-x86.yml | 60 ++++++++++++++++++++++++++++++++
>  2 files changed, 63 insertions(+)
>  create mode 100644 .github/codeql/codeql-config.yml
>  create mode 100644 .github/workflows/codeql-x86.yml
> 
> diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml
> new file mode 100644
> index 0000000000..721640c2a5
> --- /dev/null
> +++ b/.github/codeql/codeql-config.yml
> @@ -0,0 +1,3 @@
> +paths-ignore:
> +  - xen/tools/kconfig
> +  - tools/firmware/xen-dir/xen-root/xen/tools/kconfig
> 
> From actually running this:
> 
> Annotations
> 2 warnings
> analyse (go)
> The "paths"/"paths-ignore" fields of the config only have effect for JavaScript, Python, and Ruby
> analyse (cpp)
> The "paths"/"paths-ignore" fields of the config only have effect for JavaScript, Python, and Ruby
> 
> So this obviously can't be used like this.  You'll have to add them to the prebuild step.

Right, paths-ignore can only be used for interpreted languages, so
not really useful in order to ignore the content in Kconfig.

Pre-building the Kconfig in tools/firmware/ will be complicated. I
will leave ignoring those paths to a further patch, we can always
filter from the queries.

Thanks, Roger.