Scanning in the CI

[lhinds]

So I have something like this now, will port it into anteater shortly.

$ python secrets.py
Found what looks like a sensitive file: /home/luke/repos/nursery/findsecrets/testrepo/private.gpg
Found what looks like a sensitive file: /home/luke/repos/nursery/findsecrets/testrepo/private.sig
Found what looks like a sensitive file: /home/luke/repos/nursery/findsecrets/testrepo/private.key
Found what looks like a sensitive file: /home/luke/repos/nursery/findsecrets/testrepo/private.asc
The file `somefile.py` contains the string "password = 'password'" which is blacklisted
The file `id_rsa` contains the string "-----BEGIN RSA PRIVATE KEY-----" which is blacklisted 
The file `aws_key` contains the string "aws_secret_access_key'" which is blacklisted

$ cat secrets.yaml
secrets:
  file_contents: [-----begin\S*rsa\S*private\S*key-----,password,secret,ssh_key,private_key,aws_access_key_id,aws_secret_access_key]
  file_names: [.gpg,.key,.asc,.sig]

[ashyoung]

Yeah, that'll work. Is it just looking at file endings? Can it discern based on what's in the file?

[lhinds]

@ashyoung full string, so these would match:

file_names: [secret]

my_secret_file
secret_key
secret.key
secrets.txt

file_contents searches within the file.

[rpaik]

Jumping on this a bit late... Reading through the thread, it looks like we're focused on Security only at least for now? I saw a note earlier mentioning both Security & Licensing...

[ashyoung]

Hi Ray, we can easily pull in licensing, but need to agree on a tool to use for the scanning.

[ashyoung]

Hi Fatih, when it comes to the Sandbox, what is your preference for passing the Gerrit tag to scan? Right now, there is a framework that builds the sandbox. Once the sandbox is built, you need to invoke it and initialize the virtualenv. This does the following:

1) Builds dependencies
2) Clones the repo, if first time/new
3) Checks out the gerrit tag/patch
4) Scans
5) Reports

Also, do I need to add email capabilities to forward the report to PTL and security team? I can use what you have, or I can add something.

Ash

[fdegir]

Hi Ash,

You don't need to worry about cloning the repo or checking out the patch that comes to OPNFV Gerrit for any OPNFV Project repo as this is handled by Jenkins Gerrit Trigger Plugin. The only thing you need to do is your parts (1, 4, 5).

If you think we will need to run security audit against the releases (or specific sha1s/tags), we can have a separate job which can be run automatically (or manually), taking in the project name and the tag/sha1 and doing the rest same way.

The notifications can also be configured directly on Jenkins job.

Apart from this, what do you mean by
1. there is a framework that builds the sandbox - what is sandbox here? Is it the OPNFV Gerrit Project called sandbox or?
2. clones the repo - I suppose this is the repo that belongs to one of the OPNFV projects which will be scanned?
3. builds dependencies - I suppose this is related to security scanning stuff.

I can update the example job we have and configure it for the OPNFV Gerrit sandbox repo so you can see things for real and adjust/test your steps.

https://gerrit.opnfv.org/gerrit/gitweb?p=releng.git;a=blob;f=jjb/securityaudit/opnfv-security-audit.yml
https://build.opnfv.org/ci/view/All/job/opnfv-security-audit-verify-master/

[ashyoung]

Then I will to know how to indicate the location of the repo. There is a configuration file that gets read during initialization. I currently have it pointing as follows, but can easily change:

anteaterfw/
- repos/
- reports/
- src/
- build/

When we initialize the virtualenv to run the scan, it currently sets the anteaterfw as the "root" of the environment. Once the scan completes, it deactivates the virtualenv. This is what I mean by the "sandbox". It's the virtualenv that runs the scan.

The build dependencies has to do with Anteater has a number of prerequisites (e.g., Java, maven, ant, PMD, RAT, etc). This is pretty version specific on some of the components and the consistency has not been good from OS to OS. So, I created an environment and a build script that builds the whole thing up, regardless of the OS you're running on. It is also used to manage any custom rules we come up with.

[fdegir]

Hi Ash,

If you fix your scripts like a human is running them, we can easily integrate it to Jenkins by creating a wrapper in releng so your scripts can then be used manually or Jenkins with the help of the wrapper.

The location to where things could end up also depends on how we configure jobs. We can use Jenkins build workspace. There is an environment variable called WORKSPACE which points to certain location on Jenkins slave where the build is running on and is used by the Jenkins jobs. This directory is unique to each execution of a given Jenkins job.

For example, if you have a jenkins job called securityscan and if we assume you have 2 concurrent builds of this job running on same Jenkins slave, the WORKSPACE environment variable will be set to /path/to/slave/root/workspace/securityscan and /path/to/slave/root/workspace/securityscan@2 respectively. The environment variable WORKSPACE is also injected to these builds which then becomes available to all the scripts that is executed during the build.

So in your setup above, the repos can be specified as WORKSPACE/repos and we can clone the scanned project as WORKSPACE/repos/project_to_scan. If your script can take a command line argument, Jenkins job can run it as /path/to/your/script.sh --repos $WORKSPACE/repos --project project_to_scan

Apart from this, where does your framework come from? Do we need to clone the repo the framework resides? Do we need to build things for your framework for each of the security scan we run on Jenkins?

It would really help if you can share what you prepared? Is it in OPNFV Gerrit?

[ashyoung]

Okay, so I've made some changes in a couple of locations. You can checkout the code here.

Run the build.sh script to setup everything. It takes a while to build maven and even longer to build pmd. Don't try to run this on Ubuntu yet, as I am trying to figure out why the JDK 7 isn't building maven. This will pick up the $WORKSPACE variable if it's set. If it isn't, it's not a problem. The repos and reports directories will be placed in the build/anteater directory. This is just the installer and it should work on CentOS, RHEL, OpenSuse, or SLES.

To execute the scan, I presume there is a repo cloned in the $WORKSPACE/repos location. Again, if the variable isn't set, then I am looking for a cloned repo in build/anteater/repos. All of this gets detected by the run_scan.sh script, which sets up and initializes the virtualenv for anteater to run.

The reports will be placed in either $WORKSPACE/reports or in build/anteater/reports.

For now, if you wish to provide an alert, I would simply set the location for the reports as a variable on your side and then do something like:

warnings="$(cat $REPORTS_PATH/$PROJECT_report.html | grep HIGH | wc -l)" 
if [ "$warnings" -gt 0 ]; then
    [sends some type of alert or email]
fi

As for the question about framework origin, all I did was write an installer around Luke's anteater project. And then I created a script to basically wrap anteater, since it requires a virtualenv to run. The script can easily be called by any external program.

I hope this helps and is what you're looking for. If not, just let me know and I'll adjust.

Ash

---

[fdegir]

Thanks Ash.

I will give this a try directly on our Jenkins to see how it goes. I'll keep you updated with anything I do so you can also see what's happening where and comment on it.

Is JDK7 is requirement? We have JDK installed on our build servers but they might either be JDK7 or JDK8. (most probably 8)

[ashyoung]

Unfortunately, for now. Right now, we using PMD. And while the website says it requires at least Java 7, it doesn't want to build with Java 8. Now, the good news is we should be able to simply build the PMD jar files on 7, or I can give them to you, and then run everything else on Java 8.

I've read the forums and everything seems to point to this really has a dependency on 7 and that only SOME components can compile with 8. I am still working on a solution to this, in hopes that this will get me running on Ubuntu 16.04.

BTW, if I haven't said it yet, I hate Java!!!

Updates will be greatly appreciated. I'm very interested in how we integrate with Jenkins. Thanks!

[lhinds]

How is this going now @fdegir ? Any progress?

[fdegir]

I've been traveling and haven't had time to fix this.

My plan is to have something this week. Apologies for the delay.

[ashyoung]

Any luck? Is there anything I can do to help you out? I'm happy to jump in, wherever you need.

[fdegir]

I'm trying the build.sh and run_scan.sh from anteaterfw github like dumb Jenkins and here are some questions/comments. (apologies if they sound dumb since I'm trying to behave like jenkins.)

build.sh

Tool Installations: I understand you want to make sure that the right/needed versions of ant, maven, pmd, rats installed. But for me the responsibility of build script is to build stuff rather than installing dependencies. So I think the installation of the tools can be left outside of the script and users can be asked to install these beforehand. (Or a separate script to install dependencies can be created.)

Nonstandard installation locations: Another thing about these is that they are installed in non-standard locations. (in build directory) Every time someone (including jenkins user) tries to build stuff, the tools need to be installed over and over again since the build directories can potentially be removed, not leaving the installations behind. (for jenkins this is what we are doing - we always wipe the build directories out before starting the builds, leaving nothing behind.) Installing stuff in "random" locations is not something I prefer either.

User input: Too much user interaction will not work in Jenkins setup as we have no chance to provide input. So if something is missing (and if the script is running as part of Jenkins job), it should just exit non-zero, failing the job.

When to build stuff: How frequently we need to build stuff?

run_scan.sh

I reached this point and when I run run_scan.sh, no scan happens, except seeing the ascii art logo.

The result was same when I tried anteater scan all, I see the ascii logo again but scan doesn't happen. I checked help and saw all this pull/scan etc options but none of them moved me forward. How am I supposed to run the scan?

Another thing is that when we put this into Jenkins, the repo we are going to scan will be cloned automatically so we need to be able to do something like anteater scan [path to repo on filesystem]

I hope my questions make sense.

[ashyoung]

Hi Fatih,

I have no problem with you not wanting to use the build script. If you want to install things yourself, that's totally fine. I had already said that build.sh isn't meant to be built by Jenkins. It's only there to make sure the build environment is consistent.

For run_scan.sh, where is your project downloaded? You need to make sure it's reflected properly in the config file for anteater. This was discussed already. Per your input, I've implemented a check for $WORKSPACE. It's expecting the project to be in $WORKSPACE/repos.

Hope this helps.

Ash

[fdegir]

Sorry Ash, it's been quite a while and I've forgotten what we discussed earlier.

What I understood now is that I can basically go ahead and do the build at once on build server(s) manually to prepare stuff so we don't need to rebuild it until you want it to be updated.

I then tried running it again after cloning the repo in WORKSPACE/repos and I have progress.
(Actually I have more than progress as I got some stuff reported.)

I quickly/messily put this on Jenkins and ran against releng repo to show how it would look like. Here are the links.

Jenkins job log: https://build.opnfv.org/ci/view/All/job/securityscan-prototype/4/console
HTML report: https://build.opnfv.org/ci/view/All/job/securityscan-prototype/HTML_Report/utils_report.html

Some more comments/findings;

• Depending on if we want these checks to be blocker or not, we need run_tests.sh script to exit with non-zero if it finds something. In the beginning we can use this exit code to post WARNING to Gerrit which can then be turned into blocker.
• I don't seem to be able to put WORKSPACE/repos outside of anteaterfw. Some stuff seem to be expecting things relative to where I execute the script run_tests.sh.
• And the y/n confirmation as I mentioned earlier.

jenkins@ci-jenkins-build-3:~/opnfv$ pwd
/home/jenkins/opnfv
jenkins@ci-jenkins-build-3:~/opnfv$ mkdir WORKSPACE
jenkins@ci-jenkins-build-3:~/opnfv$ export WORKSPACE=/home/jenkins/opnfv/WORKSPACE
jenkins@ci-jenkins-build-3:~/opnfv$ git clone https://gerrit.opnfv.org/gerrit/p/releng.git $WORKSPACE/repos
Cloning into '/home/jenkins/opnfv/WORKSPACE/repos'...
remote: Counting objects: 739, done
remote: Finding sources: 100% (40/40)
remote: Total 18295 (delta 10), reused 18277 (delta 10)
Receiving objects: 100% (18295/18295), 8.69 MiB | 590.00 KiB/s, done.
Resolving deltas: 100% (7490/7490), done.
Checking connectivity... done.
jenkins@ci-jenkins-build-3:~/opnfv$ /home/jenkins/opnfv/anteaterfw/run_scan.sh
You are running run_scan.sh script Version: 0.3
Last modified on January 31, 2017, by Ashlee Young.

Checking build environment dependencies...

cat: /home/jenkins/opnfv/build/anteater/anteater.conf: No such file or directory
Your WORKSPACE variable is set, but not reflected in your anteater.conf file
Would you like us to change it? [y/n] n
/home/jenkins/opnfv/anteaterfw/run_scan.sh: line 116: cd: /home/jenkins/opnfv/build/anteater: No such file or directory
mkdir: cannot create directory ‘/home/jenkins/opnfv/build/anteater/repos’: No such file or directory
/home/jenkins/opnfv/anteaterfw/run_scan.sh: line 120: cd: /home/jenkins/opnfv/build/anteater: No such file or directory
/home/jenkins/opnfv/anteaterfw/run_scan.sh: line 121: env/bin/activate: No such file or directory
/home/jenkins/opnfv/anteaterfw/run_scan.sh: line 122: anteater: command not found

And with y to the confirmation this time.

jenkins@ci-jenkins-build-3:~/opnfv$ /home/jenkins/opnfv/anteaterfw/run_scan.sh
You are running run_scan.sh script Version: 0.3
Last modified on January 31, 2017, by Ashlee Young.

Checking build environment dependencies...

cat: /home/jenkins/opnfv/build/anteater/anteater.conf: No such file or directory
Your WORKSPACE variable is set, but not reflected in your anteater.conf file
Would you like us to change it? [y/n] y
cp: cannot stat '/home/jenkins/opnfv/configs/anteater.conf': No such file or directory
sed: can't read /home/jenkins/opnfv/build/anteater/anteater.conf: No such file or directory
/home/jenkins/opnfv/anteaterfw/run_scan.sh: line 116: cd: /home/jenkins/opnfv/build/anteater: No such file or directory
mkdir: cannot create directory ‘/home/jenkins/opnfv/build/anteater/repos’: No such file or directory
/home/jenkins/opnfv/anteaterfw/run_scan.sh: line 120: cd: /home/jenkins/opnfv/build/anteater: No such file or directory
/home/jenkins/opnfv/anteaterfw/run_scan.sh: line 121: env/bin/activate: No such file or directory
/home/jenkins/opnfv/anteaterfw/run_scan.sh: line 122: anteater: command not found

[ashyoung]

https://build.opnfv.org/ci/view/All/job/securityscan-prototype/4/console

I can definitely get rid of "y/n", and just make it default to resolving the issue, on run_scan.sh.

I have been able to get $WORSPACE/repos to work outside of the underlying stack. But that was a while ago. Let me look into it.

[lhinds]

Hi Faith / Ash,

I will be at the plugfest..shall we block a few hours out to sit down and finish this integration?

Luke