Wendelin Exanalytics Libre

WENDELIN combines Scikit Learn machine learning and NEO distributed storage for out-of-core data analytics in python

Wendelin standalone

Note: A binary cache has been uploaded to shacache as well as all of the required python modules for Debian 8.1, 64 bit. This should speed up install / setup time a lot as there will be no need to wait for long downloads or compilations. If you are using a different platform/arch, this cache cannot be used and all software will be downloaded from source and compiled locally which would mean much longer setup time!

Requirements

  • 4 GB of RAM
  • At least 20GB of disk space.
  • A virtual machine running Debian 8, 64bit as only it has been thoroughly tested. Debian 8 images are available here (64bit)
  • Note 1: openblast may fail to compile if CPU type of the VM is unknown. This is an openblast issue which we can not resolve and it's needed that you properly configure your VM's CPU type. You can see some example below, unfortunately this is different for every Virtual machine provider. In our example we use '-cpu host' to use host machine's CPU type.
  • /usr/bin/qemu-kvm -cpu host -m 4096 -hda /home/vm/debian8.1.img -net nic,vlan=0,model=virtio,macaddr=00:16:35:AF:94:4B  -net tap,vlan=0,ifname=$tap,script=no,downscript=no -vnc :1 -no-reboot -enable-kvm -smp 2

  • Note 3: If something went wrong or didn't work correctly, do not be afraid to re-run the ansible script using bash wendelin-standalone. This script is completely safe to rerun as many as times as you like and doing so can resolve issues.

Initial setup:

Once you have your virtual machine up and running, execute these three steps (root permissions are needed):

  root@debian8:~# wget http://deploy.nexedi.cn/wendelin-standalone && bash wendelin-standalone


Monitoring progress of Wendelin installation

To check up on the status of your build automatically please run:

  root@debian8:~# watch -n 30 erp5-show -s  

This way if something goes wrong you will be notified immediately. The total time for install and setup should be around less than an hour, however it doesn't need any interaction from user.

Another approach to monitor is to tail Wendelin installation status like so:

   root@debian8:~# tail -f /opt/slapos/log/*.log

When is my Wendelin instance ready ?

You can check if your Wendelin instance is ready using

  root@debian8:~#  erp5-show -s

If your instance is ready the script will printout

root@debian8:~# erp5-show -s
   Build successful, connect to: https://zope:insecure@[2001::bd4d]:16001

Please note that this IPv6 is an internal one for your machine and the only way to access is to run browser inside the VM machine for this see next chapter.

Access Wendelin instance

In case one needs to access the Wendelin instance from outside the VM machine there are two ways described below:

Access through socat open ports on public VM's IP address (recommended)

We have developed a script which use socat command to make all internal Wendelin services ports available through a public VM's IP address. Just run

root@debian8:~# wget https://lab.nexedi.com/nexedi/wendelin/raw/master/utils/wendelin-standalone-bind.sh && bash wendelin-standalone-bind.sh

This script will allow you to connect to following services

  • Jupiter node at https://<YOUR_VM_IP>:20000/
  • Zope node 1 at http://<YOUR_VM_IP>:20001/
  • Zope node 2 at http://<YOUR_VM_IP>:20002/
  • Zope node 3 at http://<YOUR_VM_IP>:20003/

Access VM through ssh tunnelling to it (not recommended)

  1. Find the internal IPv4 on whcih Wendelin is listening to
    • root@debian8:~# ps xa | grep runzope
    • root@debian8:~# vi /srv/slapgrid/slappart0/etc/zope.conf (and fine block with
      <http-server>
        address 10.0.210.201:12001
      </http-server>)
  2.  Create a ssh tunnel like above from host linux machine
    •    Host debian8_tun
         HostName debian8
         User root
         LocalForward 2200    10.0.210.201:12001
  3. Access locally http://localhost:2200/ and use for username: zope and for password: insecure

That's a temporary approach until we provide a front end solution.

Configure Wendelin instance

The process is described in this How-To.

===================================================

Issues with Wendelin Standalone (read only if you need to trouble shoot):

Currently there are a few issues:

  • OpenBLAS fails during compile as it cannot detect the CPU type. A long error log will be spit out when this happens... again, you can check the status of your build with erp5-show -s and if there is an error, view /opt/slapos/log/slapos-node-software.log. Here you will find the output of slapos node software. To make it easier to read and follow, use tail -f.

    • FIX: TARGET=CORE2 slapos node software
  • Sometimes when accessing Wendelin site you might get: """

    Error Type: OperationalError 

    Error Value: (1129, "Host '10.0.221.219' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'") """

    To fix one must restart MariaDB (in command we restart all just in case):

    • root@debian8w:~# slapos node restart all
  • If after your VM gets restarted not all processes are started (slapos node reports them as EXITED) then one can do (see below). Same command will fix "[3122] ERROR No IPv6 found on interface No valid IPv6 found on lo. to construct IPv6 with" message (visible in /opt/slapos/log/slapos-node-format.log)
  • root@debian8:~# slapos node format --now
  • root@debian8:~# slapos node restart all