Merge branch 'master' of github.com:krebscode/painload

15 files changed, 286 insertions, 0 deletions

diff --git a/sites/buildbot.krebsco.de/INSTALLATION.md b/sites/buildbot.krebsco.de/INSTALLATION.md
new file mode 100644
index 00000000..b31a3989
--- /dev/null
+++ b/sites/buildbot.krebsco.de/INSTALLATION.md
@@ -0,0 +1,17 @@
+#?/bin/sh
+# something like this
+
+useradd ci
+punani install python-virtualenv 
+su ci
+virtualenv buildbot
+echo ". $HOME/buildbot/bin/activate" >~/.bashrc
+pip install buildbot-slave buildbot
+buildbot create-master master
+# tahoe cp krebs:master.conf master/master.conf
+buildbot reconf master
+# or reconfigure as many slaves as you wish
+buildslave create-slave slave localhost "ubuntu1204-local-slave" <PWD>
+buildbot start master
+buildslave start slave
+# now make sure that docker is up and working
diff --git a/sites/buildbot.krebsco.de/README.md b/sites/buildbot.krebsco.de/README.md
new file mode 100644
index 00000000..187c54f8
--- /dev/null
+++ b/sites/buildbot.krebsco.de/README.md
@@ -0,0 +1,29 @@
+# buildbot.krebsco.de
+The buildbot is configured to run all of the fancy test cases in painload (and
+possibly more project).
+
+# Testing the Painload
+Subprojects may contain folders called t/ which may contain executables which
+will be called by running `make test` in the respective folder.
+A sample `make test` may look like `//krebs/ship/Makefile`.
+The buildbot master may include these paths into the test chain.
+
+# Master & Slave
+Buildbot contains of a master with all the configuration magic and n slaves
+which will be building. Both the master and the slave are started at system
+startup as the user ci (see INSTALLATION.md).
+The configuration file is currently stored at tahoe:
+
+    krebs:ci/buildbot/master/master.cfg
+    # and 
+    krebs:ci/buildbot/slave/buildbot.tac
+
+# Docker
+For more flexibility in testing the painload contains test which are using
+docker virtual environments. These have the advantage of providing a new
+environment at every run.
+
+Docker access must be made available to the CI user.
+
+For a Sample Docker Test, see /krebs/ship/t/docker/docker_remote_punani.sh
+
diff --git a/sites/gold.krebsco.de/README.md b/sites/gold.krebsco.de/README.md
new file mode 100644
index 00000000..6839631a
--- /dev/null
+++ b/sites/gold.krebsco.de/README.md
@@ -0,0 +1,18 @@
+# gold
+gold.krebsco.de is a simple file share which provides plugins for
+firefox/chromium to rewrite referers.
+
+# Installation:
+copy the nginx config at etc/nginx/sites-available, edit before use
+
+# Plugins
+All krebsgold plugins are stored in /krebs/gold/affiliate/
+For installation follow the instruction at /krebs/gold/affiliate/README.md
+
+## Chromium
+uses userscripts, a modified version of the fsf amazon userscript
+
+## Affiliatefox
+Affiliatefox is a dodgy plugin by some weird german dudes at
+http://www.the-angelz.net . it may be removed in the future as the chromium
+userscript also works for firefox+greasemonkey
diff --git a/sites/gold.krebsco.de/etc/nginx/sites-available/gold.krebsco.de b/sites/gold.krebsco.de/etc/nginx/sites-available/gold.krebsco.de
new file mode 100644
index 00000000..1c565bbd
--- /dev/null
+++ b/sites/gold.krebsco.de/etc/nginx/sites-available/gold.krebsco.de
@@ -0,0 +1,12 @@
+server {
+  listen <external-ip>:80;
+  server_name gold.krebsco.de;
+  access_log /var/log/nginx/log/gold.krebsco.log main;
+  default_type text/plain;
+  autoindex on;
+  location / {
+    # path to //gold/affiliate/ ,may be a symlink or something
+    root  /var/www/gold.krebsco.de;
+  }
+}
+
diff --git a/sites/graph.krebsco.de/README.md b/sites/graph.krebsco.de/README.md
new file mode 100644
index 00000000..2cc43197
--- /dev/null
+++ b/sites/graph.krebsco.de/README.md
@@ -0,0 +1,98 @@
+# Retiolum graphs
+Tinc provides detailed informations about hosts in the mesh network. We are
+using this information to build graphs. 
+
+## Requirements
+
+- tinc-pre (tinc and tincd binaries)
+- python2 or python3
+- all the python dependencies in /krebs/retiolum/scripts/adv_graphgen/DEPS
+    via `pip install -r DEPS`
+
+
+## Types of Graphs
+Currently two types of graphs are generated:
+
+  1. Anonymous Graphs
+    - only fancy lines between dots
+    - this should be made available to the world via graph.krebsco.de
+  2. Detailed Graphs
+    - with all the stuff we know
+    - contain name, ip address, uptime, different coloring for supernodes and
+      hosts which die when supernodes die.
+    - these graphs should only be availabe by hosts in the retiolum darknet
+    - currently these are published by pigststarter/ but the hostname graph/
+      shall be used for this in the future
+
+In addition a Graph DB will be created which contains all the cool infos from
+the detailed graph plus the geolocation.
+This database is used by map.html which positions all the hosts on a world map.
+
+The graph generation host should be a super node with tinc-pre as tinc seems to
+be blocking when building graphs with `GraphDumpFile`.
+
+# Code
+Source Code is in /krebs/retiolum/scripts/adv_graphgen/
+
+## all_the_graphs.sh
+This script is used for booting all the graph generation magic.
+This scripy may be run as a cronjob every 5 to 10 minutes by a user which has
+the right to use the tincctl and can write to the WWW directory.
+
+    0/5 * * * * /krebs/retiolum/scripts/adv_graphgen/all_the_graphs.sh
+
+The script also writes geo_coordinates for the nodes with the help of
+tinc_stats/Geo.py.
+it contains most of the hardcoded paths which may be changed (like
+INTERNAL_FOLDER and EXTERNAL_FOLDER, see anonytize&sanitize) as well as a path
+to the geolitecity ip database
+
+## tinc_stats/Log2JSON.py
+
+This script creates a giant json file from the current tinc informations and
+writes it to stdout. It only contains the information retrieved by the tinc
+daemon. 
+
+## tinc_stats/Graph.py
+
+This script takes the json file created by Log2JSON as input. It can be either
+run as `$0 complete` to create a detailed graph or  `$0 anonymous` to create
+minimal graphs.
+
+When run as anonymous no additional information will be added to the graph.
+When run in complete mode, the script will determine the availability (see
+tinc_stats/Availability) which nodes are supernodes (tinc_stats/Supernodes).
+
+it writes a graphviz graph to stdout. This can be used to create graphs with
+dot by graphviz.
+
+## tinc_stats/Geo.py
+
+Geo.py takes the json file generated by Log2JSON as input and populates this
+graph with geo-coordinates with the help of GeoIP. This database can be used by
+map.html if put in the same directory.
+
+## tinc_stats/Supernodes
+
+This script provides functionality find out if a node is a supernode or not. 
+This will be done by checking if the tinc port of the host in the json file is reachable or not. if called directly it will return the name of the host, a space, and an array of tuples of ip-addresses which were reachable in the run.
+This script is used by Graph.py via import.
+
+## tinc_stats/Availability
+
+This modules provides functionality to generate availability information for
+each node configured in /etc/tinc/retiolum/hosts. This is done by tracking each
+request in a file called /krebs/db/availability (currently hardcoded in
+Graph.py). The Script will not append a new line of hosts by itself, Graph.py
+does this. 
+
+## anonytize & sanitize
+These two scripts handle the building of the graphs as well as the conversion
+from graphviz to svg and png. They work pretty much the same in principle, one
+is calling Graph.py complete and the other anonymous.
+
+Both scripts are called with $1 being the path where to write the graphs into.
+e.g.:
+./anonytize.sh /var/www/graph.krebsco.de
+./sanitze.sh /var/www/graph.retiolum
+
diff --git a/sites/graph.krebsco.de/etc/nginx/sites-available/graph.conf b/sites/graph.krebsco.de/etc/nginx/sites-available/graph.conf
new file mode 100644
index 00000000..1b74f53a
--- /dev/null
+++ b/sites/graph.krebsco.de/etc/nginx/sites-available/graph.conf
@@ -0,0 +1,11 @@
+server {
+        # graph hosting internal
+        listen <internal-ip>:80;
+        server_name graph graph.retiolum; # or your hostname
+        access_log /var/log/nginx/log/graph.log main;
+        default_type text/plain;
+        location / {
+                root    /var/www/graph.retiolum;
+        }
+}
+
diff --git a/sites/graph.krebsco.de/etc/nginx/sites-available/graph.krebsco.de.conf b/sites/graph.krebsco.de/etc/nginx/sites-available/graph.krebsco.de.conf
new file mode 100644
index 00000000..0619908e
--- /dev/null
+++ b/sites/graph.krebsco.de/etc/nginx/sites-available/graph.krebsco.de.conf
@@ -0,0 +1,10 @@
+server {
+        listen <external-addr>:80;
+        server_name graph.krebsco.de;
+        access_log /var/log/nginx/log/graph.krebsco.de.access.log main;
+        default_type text/plain;
+        location / {
+                root    /var/www/graph.krebsco.de;
+        }
+}
+
diff --git a/sites/paste.retiolum/README.md b/sites/paste.retiolum/README.md
new file mode 100644
index 00000000..3c634f95
--- /dev/null
+++ b/sites/paste.retiolum/README.md
@@ -0,0 +1,31 @@
+# paste.retiolum
+
+paste is a minimalistic pastebin with sprunge.us in mind.
+This paste may be a supplement to all the 'open' pastebins as the punching
+lemma applies to this installation.
+The installation always runs on a higher port (4000), to get a really short
+hostname, the host which provides this service should have a short name as well
+and have an nginx or apache which translates all request to hostname:80 to
+localhost:4000. see Nginx Configuration.
+
+# Sources
+
+- https://github.com/makefu/bump
+
+# Installation
+
+## Environment
+
+    git clone https://github.com/makefu/bump
+    useradd -a bump -m -d /opt/bump
+    cd /opt/paste
+    virtualenv .
+    pip install -r deps.txt
+
+## Nginx
+
+see etc/nginx/
+
+## Supervisor
+
+see etc/supervisor.d/
diff --git a/sites/paste.retiolum/bump b/sites/paste.retiolum/bump
new file mode 160000
+Subproject 119d2723b510be392ca03d5bca7e1573e533f84
diff --git a/sites/paste.retiolum/etc/nginx/sites-available/paste.conf b/sites/paste.retiolum/etc/nginx/sites-available/paste.conf
new file mode 100644
index 00000000..b83abf95
--- /dev/null
+++ b/sites/paste.retiolum/etc/nginx/sites-available/paste.conf
@@ -0,0 +1,14 @@
+server {
+	listen <internal-ip>:80;
+	server_name paste paste.retiolum;
+	access_log /var/log/nginx/log/paste.log main;
+	error_log /var/log/nginx/log/paste_error.log;
+	default_type text/plain;
+	location / {
+    proxy_pass http://127.0.0.1:4000/;
+    proxy_redirect off;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+	}
+}
diff --git a/sites/paste.retiolum/etc/supervisor.d/bump.supervisor.conf b/sites/paste.retiolum/etc/supervisor.d/bump.supervisor.conf
new file mode 100644
index 00000000..6b83d5f2
--- /dev/null
+++ b/sites/paste.retiolum/etc/supervisor.d/bump.supervisor.conf
@@ -0,0 +1,5 @@
+[program:bump]
+command=sh run.sh
+user=bump
+directory=/opt/bump
+autorestart=true
diff --git a/sites/tahoe.retiolum/README.md b/sites/tahoe.retiolum/README.md
new file mode 100644
index 00000000..01e8ed03
--- /dev/null
+++ b/sites/tahoe.retiolum/README.md
@@ -0,0 +1,16 @@
+# Tahoe in Retiolum
+For all the secret stuff, krebsco is using a tahoe installation with 1
+introducer and a number of tahoe bricks. 
+
+# Adding new bricks & clients
+see //cholerab/tahoe/{brick,client}\_installation
+
+# Replacing the introducer
+if the introducer may die off, all crypto material is saved in 
+krebs:tahoe/introducer. There will be a backup somewhere, but bootstrapping
+always sucks.
+
+Follow the generic brick installation,
+use the configuration file at conf/tahoe.cfg and copy the crypto material in
+the private folder of the installation. 
+autostart that shit
diff --git a/sites/tahoe.retiolum/conf/tahoe.cfg b/sites/tahoe.retiolum/conf/tahoe.cfg
new file mode 100644
index 00000000..f70a94da
--- /dev/null
+++ b/sites/tahoe.retiolum/conf/tahoe.cfg
@@ -0,0 +1,4 @@
+[node]
+nickname = intro
+web.port = 
+web.static = public_html
diff --git a/sites/tinc.krebsco.de/README.md b/sites/tinc.krebsco.de/README.md
new file mode 100644
index 00000000..3d3d2faa
--- /dev/null
+++ b/sites/tinc.krebsco.de/README.md
@@ -0,0 +1,11 @@
+# tinc.krebsco.de
+tinc.krebsco.de is a curl | sh tinc bootstrapper. For more information see
+//retiolum/scripts/tinc_setup/README.
+
+The nginx is pointed to the folder //boot in painload.
+
+In order to avoid adding an addition file name a special nginx configuration
+based on the try_files directive. 
+
+# Installation
+See etc/nginx/sites-available/tinc.krebsco.de.conf
diff --git a/sites/tinc.krebsco.de/etc/nginx/sites-available/tinc.krebsco.de.conf b/sites/tinc.krebsco.de/etc/nginx/sites-available/tinc.krebsco.de.conf
new file mode 100644
index 00000000..15749985
--- /dev/null
+++ b/sites/tinc.krebsco.de/etc/nginx/sites-available/tinc.krebsco.de.conf
@@ -0,0 +1,10 @@
+server {
+        listen <external-ip>:80;
+        server_name tinc.krebsco.de;
+        access_log /var/log/nginx/log/boot.krebsco.log main;
+        default_type text/plain;
+        # this path should be //boot
+        root    /var/www/boot.krebsco.de;
+        try_files $uri $uri/retiolum.sh ;
+}
+