Scanning in the CI

[ashyoung]

One of the things we need to consider is how we want to actually scan for security and/or license related issues.

For example, we can scan as we're ready to finalize a build, which is what we did for Colorado 1.0. In my last conversation with David McBride, he asked that we not save things up for the last minute, which is understandable.

We could also integrate directly into the CI and scan projects as they change. Or we could do some type of regular batch process.

Thoughts? What does the CI/CD team prefer here?

[fdegir]

I normally prefer to do things when things happen rather than periodically/on a timer. This will make it possible for us to catch this and other type of issues right when they are introduced, closer to design and when the delta is smaller - single commit. This would also help us increase the awareness of the community with regards the security issues.

So my suggestion to this is to run scanning when a new patch is sent for review to Gerrit and before it is merged. We could start with non-voting scans meaning that if we find a security/license issue, we just post findings as feedback to Gerrit. Once we see that things work as expected tooling-wise, we let the community know the intention/reasons and then we enable voting, blocking changes from being submitted to master. This will help us to get these issues fixed right at that moment rather than sending a report once a while and asking people to fix them altogether, which might not be that straightforward depending on the complexity of the issues and the effort needed to fix them.

Above suggestion depends on the time it takes to run the scan as well. If it is like 10 minutes or something, I think noone will object.

[ashyoung]

I tend to agree with this. Really quick question, are we intending to have the scanning setup as a regular part of our CI? In other words, we're not going to do tear down and buildup? If we're going to leave it up, then things should be fairly straight forward. We just need to figure out where we want the results placed for the CI to get them before allowing Gerrit to move forward.

Any thoughts on this? The scan takes seconds to run. I suppose it could be longer for something like KVM, but I will run a live scan and let you know how long it took.

[fdegir]

are we intending to have the scanning setup as a regular part of our CI?

Yes.

CI just needs scripts to run the validation against to the proposed change/patches. The feedback will be put on Gerrit together with the Verified+1 or Verified-1 votes like how opnfv-lint-verify-master job did on below change;

https://gerrit.opnfv.org/gerrit/#/c/25759/

https://build.opnfv.org/ci/job/opnfv-lint-verify-master/1879/ : SUCCESS

As I mentioned, if it is like couple of minutes, it is no problem to run for each and every relevant patch.

[ashyoung]

I ran a scan of kvmfornfv and it took approximately 6 seconds to complete. It took 1:57 to scan ovno. And to scan all of the projects took 2:47.

So, in total, it wasn't bad at all. I'm looking into how to incorporate this into the pipeline.

[fdegir]

If you want, I can create simple Jenkins jobs to show how it would look like and what we need from Security Group.

[ashyoung]

That would be great!!!

[lhinds]

Hi,

Sorry for lateness on the topic.

So getting this as a votable gate is going to take a fair amount of work. Currently the tool does the following:

Walk through directories and folders to discover what language is used. It then matches up a scanning tool based on the findings. You can see this in the following conditional:

    if py and not (java or c):
        run_bandit(reports_dir, project, projdir)
    # Project contains c files
    if c and not (java):
        run_rats(reports_dir, project, projdir)
    # Project contains only java files
    if java and not (py or c):
        run_pmd(reports_dir, project, projdir)
    # Project contains a mix of c and python
    if c and py and not (java):
        run_rats(reports_dir, project, projdir)
    if rb or php or perl:
        run_rats(reports_dir, project, projdir)

Whereby we have three scanners (rats, bandit and PMD):

Bandit (Python) https://github.com/openstack/bandit
PMD (Java) https://pmd.github.io/
Rats (C / C++, Ruby, Perl) https://code.google.com/archive/p/rough-auditing-tool-for-security/

Each of these tools generates a html report (but using their own formatting) - this means we would need three sets of seralizers to parse out FAIL / PASS results.

An exception to the above is Bandit, which being an openstack developed tool can more easily be wired into gerrit / jenkins (but it only lints python code).

Another key consideration, is false positives, as in some FAILS maybe be acceptable when considered by a human.

So with this in mind I think the following would be a good approach.

Generate reports as above, and provide them as Jenkins check non votable.

Extend the tool to look for private keys, blobs, binaries and other nasty stuff and have those as gates with a vote.

[lhinds]

I spoke with some folk on irc and found a good library for searching out binaries

from binaryornot.check import is_binary
is_binary('/usr/bin/ping')
True
is_binary('/etc/passwd')
False

This seems like a good candidate as they have most edge cases covered.

I will put something together that performs an os.walk through a repo.

Is there any guidance around on how to signal to the gate a failure / pass ?

[lhinds]

Actually, I am thinking we should maybe have a goto meeting over this, and myself and ash can share how the tool currently runs. We can then make some design decisions on how best to get this to fit CI.

Some key points would be:

1. Fail / Pass criteria and how to wire this into jenkins / gerrit.
2. Whereabouts anteater sits (vm, container, virtualenv on the gerrit / git server?).
3. Where to put the security scan lint reports and link those in gerrit.

I am also thinking we might be able to rip out some of my CLI args (which ash has already seen are quite bloated perhaps for our needs), but new args will be contingent upon how jenkins wants to call anteater, and where anteater should fine the opnfv repos etc.

Also another discussion point, is if we should move the code over to an OPNFV repo, as its currently on github:

lukehinds/anteater

I will set up a g2meeting for tomorrows security group @ 14:00 UTC.

[fdegir]

Here is an example job configuration: https://gerrit.opnfv.org/gerrit/#/c/25849/

We can have similar jobs for other languages as well as a start and depending on what the change contains, we can do the scanning based on that. As you said, this requires some work but we can start with one language and non-voting and find how best we can do this.

[ashyoung]

The issue I have right now is that the tests are generating results with "High" severity levels. If these are in fact "high", then we need to act upon them. If we don't think they're high, then we need to adjust the rules so they don't get flagged. I have a real problem with ignoring findings. If they're good enough to ignore, they're good enough not to run and consume resources. So that's something we really need to take into consideration.

[fdegir]

We had (and probably still have) similar cases with the doc verification.

Some issues were critical, giving -1 on Gerrit and blocking the change from getting submitted.
The other minor issues were posted as comments (warnings) and it was left to developers to act on them and the job voted +1, not blocking the change from getting submitted.

So please do not think resource as an issue as I believe we have enough build resources where these jobs can be run. If we ever get into resource issue, I am sure I can find some resources.

[lhinds]

I see what you mean ash and agree, but its real tough to decide on a highs actual real world threat without a manual review - this is the reason that Bandit is a non voting gate in openstack currently.

I suggest the following to get us started at least:

If highs are find, we put a red failure (but non voting),

If binaries (non whitelisted) are found, failure (and have it voting)

if private keys are found - failure (and have it voting).

Eventually we can then look to refine he highs for the secure lint scans (and have an ignore list for those threats already triaged)

p.s I just added some code to check for any binary files:

lukehinds/anteater/blob/5dae303ee5f5e8ee9e274547ceea703175763694/anteater/src/scan_tasks.py#L114

There is also an ignore list:

lukehinds/anteater/blob/master/ignorelist.yaml

This way, we can ignore stuff like images (png, jpeg) or pdf's and also white list a binary (if a good justification is made for it being there).

[ashyoung]

I'm good with that suggestion. @fdegir You okay with what Luke suggested?

[fdegir]

@ashyoung @lhinds I'm fine with it as well.

Please note that the voting based on the type of the finding will require some thinking, trial and error which is good.

I should have thought this earlier but never too late; another thing we could do is to use sandbox repo instead of functest so we can propose changes to it on gerrit without disturbing a real project. With this repo, we can freely propose changes that cause different problems and we see if we get different types of errors/votes on Gerrit in the end.

https://gerrit.opnfv.org/gerrit/gitweb?p=sandbox.git;a=summary

I can send a new change, switching from functest to sandbox.

[ashyoung]

I think that's fine as well. From my perspective, it doesn't really matter where we implement it. I originally assumed sandbox.

[lhinds]

One addition we will want to make in time, is perhaps being able to pull PMD down from its source (as I currently have it embedded in anteater which has .jar files in PMD/libs).

[ashyoung]

That's actually what I'm putting in the Ant and Maven support for. Will have PMD added in the next day or so.

[lhinds]

@ashyoung cool, I thought I heard you mention those (connection was a bit choppy on g2m).