Merge pull request #93 from cybermaggedon/boolean-logic

Boolean logic - update docs

.gitignore

@@ -11,7 +11,7 @@ Makefile.bak
 src/cybermon
 src/cyberprobe
 src/cyberprobe-cli
-src/eventstream_service
+src/eventstream-service
 src/etsi-rcvr
 src/nhis11-rcvr
 src/stream-rcvr

Makefile.am

@@ -1,5 +1,6 @@
 
-SUBDIRS = src include config docs stix www tests init subscribers utils
+SUBDIRS = src include config docs indicators stix www tests init subscribers \
+	utils
 
 ACLOCAL_AMFLAGS = -I m4
 

configure.ac

@@ -147,6 +147,7 @@ AC_CHECK_FUNCS([gethostbyname gettimeofday socket strdup uname])
 AC_CONFIG_FILES([Makefile src/Makefile include/Makefile config/Makefile
 	docs/Makefile stix/Makefile www/Makefile tests/Makefile tests/atlocal
 	init/Makefile subscribers/Makefile pkg.mk utils/Makefile
+	indicators/Makefile
  	cyberprobe.spec])
 
 AC_OUTPUT

docs/Makefile.am

@@ -14,7 +14,8 @@ cyberprobe_TEXINFOS = architecture.texi aws-mirroring.texi		\
 	qs-threat-indicators.texi qs-top.texi qs-visualisation.texi	\
 	ref-cybermon-alert.texi ref-cybermon-bigquery.texi		\
 	ref-cybermon-cassandra.texi ref-cybermon-configuration.texi	\
-	ref-cybermon-detector.texi ref-cybermon-dump.texi		\
+	ref-cybermon-detector.texi ref-indicators.texi                  \
+        ref-cybermon-dump.texi		                                \
 	ref-cybermon-elasticsearch.texi					\
 	ref-cybermon-example-configs.texi ref-cybermon-gaffer.texi	\
 	ref-cybermon-geoip.texi ref-cybermon-invocation.texi		\

docs/cyberprobe.texi

@@ -186,6 +186,10 @@ Texts.
 
 @comment ----------------------------------------------------------------------
 
+@include ref-indicators.texi
+
+@comment ----------------------------------------------------------------------
+
 @include ref-cybermon-dump.texi
 
 @comment ----------------------------------------------------------------------

docs/qs-threat-indicators.texi

@@ -1,133 +1,46 @@
 
-@node Threat indicators using STIX
-@section Threat indicators using STIX
-@cindex STIX
-@cindex TAXII
-@cindex STIX indicators
-@cindex Threat indicators
-@cindex Cyber threat indicators
+@node Threat indicators
+@section Threat indicators
+@cindex IOC
+@cindex Indicator of Compromise
+@cindex Indicator
+@cindex Threat indicator
 @cindex JSON
 
-We've been experimenting with an open model for describing cyber
-threats. STIX is a community-driven effort to standardise a model for cyber
-theat information. TAXII defines a set of services for distributing STIX
-information. There's some support in @command{Cyberprobe}, but you should
-know that this is very prototype at the moment.
+Cyberprobe includes a subscriber which apply indicators to events.  When
+an event matches an indicator, information about the indicator is tagged
+into the event.
 
-This is what we've got so far:
+Indicator support is present in the @command{cybermon-detector} subscriber
+which reads indicators from a JSON file.
 
-@itemize
+@heading Indicator files
 
-@item
-There's a simple CSV file format we've created to describe cyber
-threats. This is just for convenience.
+The installation bundle includes a a file containing some
+sample indicators, search for @file{indicators.json}, which may be installed
+at @file{@value{PREFIX}/share/doc/cyberprobe}.
 
-@cindex @command{stix-create}
-@item
-A script, @command{stix-create} which reads the above configuration file,
-and converts into a STIX document containing Indicator objects.
 
-@cindex @command{taxii-server}
-@item
-A script, @command{taxii-server} which acts as a very simple TAXII server,
-serving up STIX documents.
 
-@cindex @command{taxii-client}
-@item
-A script, @command{taxii-client} which connects to a TAXII server, gets STIX
-documents and dumps some stuff out.
 
-@cindex @command{taxii-sync-json}
-@item
-A script @command{taxii-sync-json} which connects to a TAXII server, gets
-STIX documents, massages the whole lot into a single JSON form, and dumps
-that to a file.  This is intended to be used with the
-@file{cybermon-detector} subscriber.
-See @ref{@command{cybermon-detector} invocation}.
 
-@item
-A configuration file for @command{cybermon} which reads the JSON threat
-information and reports when theats are observed.
 
-@end itemize
-
-@cindex @code{pyOpenSSL}
-@cindex @code{libtaxii}
-@cindex @code{stix}
-@cindex @code{pip}
-
-Before taking this any further, you need to have Python installed, along
-with various dependencies (@code{pyOpenSSL}, @code{libtaxii} and
-@code{stix}). The easiest way to install the dependencies is to install
-@code{pip}, and issue this command:
-
-@example
-sudo pip install libtaxii pyOpenSSL stix
-@end example
-
-@heading A STIX document service
-
-The installation bundle includes a couple of CSV files containing some
-fictional cyber theats. Search for @file{example1.txt} and
-@file{example2.txt}. They may be in @file{@value{PREFIX}/share/doc/cyberprobe}
-once you've installed everything. You need to create a data area, and
-convert these files into STIX ready for serving:
-
-@example
-mkdir /tmp/stix
-cd /tmp/stix
-mkdir -p data/default
-stix-create @value{PREFIX}/share/doc/cyberprobe/example1.txt \
-        data/default/1 -i ex:1
-stix-create @value{PREFIX}/share/doc/cyberprobe/example2.txt \
-        data/default/2 -i ex:2
-@end example
-
-Check that you have two new XML files in data/default directory. If they're
-there, you're ready to start a STIX server. This will run on port 8080, so
-you'll need to use a different port number if you don't like this one. It's
-important that this is run from the directory where you just created the
-data directory.
-
-@example
-taxii-server --port 8080
-@end example
-
-If that works, use the test client to communicate:
-
-@example
-taxii-client --port 8080 --poll
-@end example
-
-And you should see some stuff that looks like cyber threat information
-dumped on the screen.
 
 @heading Deploying theat information to @command{cybermon}
 
-@cindex JSON
-
-Now, we use @command{taxii-sync-json} to fetch the STIX information in a
-JSON form I can easily ingest into the LUA code:
-
-@example
-taxii-sync-json --port 8080
-@end example
-
-This will create a JSON file called @file{stix-default-combined.json}.
-
-Finally, run processing.  Stop any running @command{cybermon} and
-@command{cybermon-elasticsearch} processes.  Then run @command{cybermon}
-to publish to a queue on RabbitMQ:
+To run using your existing processing pipeline, stop any running
+@command{cybermon} and @command{cybermon-elasticsearch} processes.  Then run
+@command{cybermon} to publish to a queue on RabbitMQ:
 
 @example
 cybermon -p 10000 -c @value{SYSCONFDIR}/cyberprobe/amqp-topic.lua
 @end example
 
-Next run @command{cyberprobe-detector} to apply STIX rules.  By default,
+Next run @command{cyberprobe-detector} to apply indicator rules.  By default,
 this will subscribe to @samp{cyberprobe} and publish to @samp{ioc}:
 
 @example
-STIX_INDICATORS=stix-default-combined.json cybermon-detector \
+env INDICATORS=/path/to/indicators.json cybermon-detector \
     cyberprobe ioc
 @end example
 
@@ -148,33 +61,32 @@ cybermon-dump ioc | jq --unbuffered .indicators
 This activity should trigger a theat:
 
 @example
-wget -q -O- http://www.malware.com/malware.dat
+wget -q -O- http://www.malware.org/malware.dat
 @end example
 
 If this works, you should see the following output:
 
 @example
 [
   @{
+    "description": "URL of a page serving malware",
+    "category": "malware",
+    "author": "someone@@example.com",
+    "source": "id:3245edd9-e0f3-4982-9406-fbf93b874555",
     "type": "url",
-    "id": "example1:7",
-    "value": "http://www.malware.com/malware.dat",
-    "description": "URL of a page serving malware"
+    "value": "http://malware.org/malware.dat"
   @}
 ]
 @end example
 
 This hits on a number of theat indicators. The hostname www.malware.com is
 present in a theat indicator, and it is detected in the HTTP request, and
 both the DNS query and response. Also, the URL
-@code{http://www.malware.com/malware.dat} is in a threat indicator and it is
+@code{http://www.malware.org/malware.dat} is in a threat indicator and it is
 detected in both the HTTP request and response.
 
 @command{cybermon-detector} updates its state if
-the JSON configuration file has changed. So, you can do a round-trip update
-by changing the input files, re-running stix-create, using
-@command{taxii-sync-json} to fetch the updates, and all without stopping the
-monitoring.
+the JSON configuration file has changed.
 
 If you want to load the output of @command{cybermon-detector} into
 ElasticSearch, you can, but you need to subscribe to @samp{ioc}:
@@ -183,6 +95,12 @@ ElasticSearch, you can, but you need to subscribe to @samp{ioc}:
 cybermon-elasticsearch ioc
 @end example
 
+This results in indicator hit information being loaded into ES.
+
+For more information on indicators, see
+@ref{@command{cybermon-detector} invocation} and
+@ref{Cyberprobe indicator format}.
+
 @heading Conclusion
 
 All done, I hope you enjoyed the tutorial! Any comments on the software, or

docs/qs-top.texi

@@ -12,5 +12,5 @@
 * Using @command{cybermon}::
 * Writing your own configuration file::
 * Visualisation::
-* Threat indicators using STIX::
+* Threat indicators::
 @end menu

docs/ref-cybermon-detector.texi

@@ -3,9 +3,9 @@
 @section @command{cybermon-detector} invocation
 
 @cindex @command{cybermon-detector}, invocation
-@cindex STIX
 @cindex IOC
 @cindex Indicator of compromise
+@cindex Indicator
 
 @command{cybermon-detector} subscribes to a RabbitMQ pub/sub queue for
 @command{cybermon} events, inspects them for IOCs, and adds detection
@@ -14,6 +14,11 @@ the elaborated events.  This effectively creates a processing
 chain.  The event subscription and publishing events should be different in
 order to avoid creating an infinite loop.
 
+The indicator file is specified by the @samp{INDICATORS} environment variable,
+the default is @file{indicators.json}.  The file format is the
+@ref{Cyberprobe indicator format}.  When the file changes, the indicators are
+reloaded without the needed to restart @command{cybermon-detector}.
+
 Synopsis:
 
 @example