R and non-standard evaluation

What puzzled me the most about R was that you can write sth like

subset(df, quality > 6)

where “quality” is the name of a column in data frame df. Why does this not lead to an exception when evaluating the call, because “quality” on its own is not a defined variable? How can subset internally re-map “quality” to a column name of df? This won’t work in the eager evaluation semantics I am used to in languages like Python, Java, C.

The answer is more complex, but the key insight is that R uses “non-standard” evaluation. The whole concept is explained pretty nicely here.
The main upshot is that when you pass something to a function it is not evaluated fully yet, and only a so-called promise is passed to the function. This promise will get evaluated once its value is called for.

This behaviour is demonstrated by the following code:

> f <- function (w) { 1 }
> a
Error: object 'a' not found
> f(a)
[1] 1
Posted in Dev

Create AWR snapshot and report

dw@xe oracle$ cat create_snapshot.sql 
exec DBMS_WORKLOAD_REPOSITORY.create_snapshot;
quit

dw@xe oracle$ cat awrrpt.sh 
sqlplus system/system@XE @$ORACLE_HOME/rdbms/admin/awrrpt.sql
Posted in Dev

Set up AUTOTRACE in SQL*Plus

The PLAN_TABLE that is required for SQL*Plus AUTOTRACE to work already exists as a public synonym:

DW@XE> select table_owner, table_name from all_synonyms where synonym_name = 'PLAN_TABLE';

TABLE_OWNER		       TABLE_NAME
------------------------------ ------------------------------
SYS			       PLAN_TABLE$

Now let’s create and grant the PLUSTRACE role:

sudo su oracle
cd $ORACLE_HOME/sqlplus/admin
sqlplus / as sysdba
@plustrce
GRANT PLUSTRACE TO PUBLIC;
Posted in Dev

SQL*Plus useful login.sql setup

The .bashrc of SQL*Plus is called login.sql, and SQL*Plus looks for it in the directory specified in environment variable $SQLPATH. Here’s a useful setup, taken from Mr Kyte’s excellent book “Expert Oracle Database Architecture”:


dw@localhost admin$ echo $SQLPATH
/home/dw/etc
dw@localhost admin$ cat $SQLPATH/login.sql
-- Taken from Tom Kyte's EODA book
define _editor=vi
set serveroutput on size 1000000
set trimspool on
set long 5000
set linesize 100
set pagesize 9999
column plan_plus_exp format a80
set sqlprompt '&_user.@&_connect_identifier.> '

Posted in Dev

Build git 2.11.0 on raspberry pi

My pi runs Raspbian (debian jessie 8.0). The officially supported git version you can get via apt-get is 2.1.4. from Dec 2014. For me, that’s a tad too old. So I just built git from source on my raspi.

To avoid cloning ~100MB onto it, I cloned the Git repo onto my regular PC, and created a source tarball for the pi:


git clone https://github.com/git/git
cd git/
git checkout -b my_v2.11.0 v2.11.0
tar cvzf ../git-2.11.0-src.tar.gz --exclude="*/.git/*" .

I uploaded the tarball to my pi, unpacked it, and built Git. It took a couple of minutes, but ran without any issues:


mkdir git-2.11.0
cd git-2.11.0
tar xzf ../git-2.11.0-src.tar.gz
sudo apt-get install libcurl4-gnutls-dev libexpat1-dev gettext libz-dev libssl-dev build-essential
make prefix=/usr/local
sudo prefix=/usr/local make install

If you have the old version of git on your pi already, you should remove it first.

Posted in Dev

Set up internet via wlan0 and static ip address for eth0

I want to integrate my Raspberry Pi 2 into a routerless LAN, and to use the WiFi interface to connect to the internet. I’m running Raspbian Jessie (8.0).

I added the network= part to wpa_supplicant.conf, as described here:


$ lsb_release -a
Description: Raspbian GNU/Linux 8.0 (jessie)
$ sudo cat /etc/wpa_supplicant/wpa_supplicant.conf
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid="THE_SSID"
psk="THE_PASSWORD"
}
$ sudo ifdown wlan0
$ sudo ifup wlan0

Static IP address is configured in /etc/dhcpcd.conf, as described here


interface eth0

static ip_address=192.168.99.3/24

Posted in Dev

Install Oracle XE on CentOS 7 Minimal

Download Oracle 11g XE RPM here: http://www.oracle.com/technetwork/database/database-technologies/express-edition/downloads/index.html

Download CentOS 7 Minimal ISO here: https://www.centos.org/download/

Install CentOS 7 as a virtual machine in Oracle VirtualBox.

Setup networking:

  • Change VirtualBox network config: add “Host-only Adapter”
  • Make sure network is started properly in CentOS (both NAT for internet access, and Host-only adapter):
    • in “/etc/sysconfig/network-scripts/ifcfg-enp0s3” (or whatever your adapter name is for the NAT case): ONBOOT=yes
    • the adapter for Host-only seems to magically appear without such a config file. In my case, it’s enp0s8, with inet 192.168.56.101/24

Setup sudo:

su -
visudo
# Find root ALL=(ALL) ALL
# Add line:
dw ALL=(ALL) ALL

Install prerequisites:

sudo yum install libaio bc flex net-tools

NOTE: Oracle is pretty picky when it comes to hostnames. Basically, the hostname your machine has when you install Oracle should not change later. The CentOS machine you just created uses localhost.localdomain as hostname, but VirtualBox DHCP might later give it a different name. If that might happen to you, google how to set a fixed hostname (IIRC, I just set it in /etc/hostname), and set it before installing Oracle. If you have problems later, the first files to look into are listener.ora and tnsnames.ora at /u01/app/oracle/product/11.2.0/xe/network/admin

Install Oracle XE RPM:

sudo rpm -ivh oracle-xe-11.2.0-1.0.x86_64.rpm

If you see a silly message such as

This system does not meet the minimum requirements for swap space. Based on
the amount of physical memory available on the system, Oracle Database 11g
Express Edition requires 2048 MB of swap space. This system has 1639 MB
of swap space. Configure more swap space on the system and retry the
installation.

then rebuild the rpm and remove the check for swap space, as described at https://www.elastichosts.com/blog/oracle-database-installation-on-a-container-running-centos/:
Install rpm-build:

sudo yum install rpm-build

Download rpmrebuild from
https://sourceforge.net/projects/rpmrebuild/files/rpmrebuild/2.11/rpmrebuild-2.11-1.noarch.rpm/download
Install with

sudo rpm -ivh rpmrebuild-2.11-1.noarch.rpm

Now edit the pre-build script of the Oracle XE RPM, removing the check for swap space:

rpmrebuild --edit-pre -p oracle-xe-11.2.0-1.0.x86_64.rpm

Find the check for

# check and disallow install, if swap space is less than Min( 2047, 2 * RAM)

and modify the if statement, adding and 6 equals 9

if [ ... exisiting check ... -a "6" = "9" ]

After a while, the new RPM is ready:

result: /home/dw/rpmbuild/RPMS/x86_64/oracle-xe-11.2.0-1.0.x86_64.rpm

Finally, install it

sudo rpm -ivh /home/dw/rpmbuild/RPMS/x86_64/oracle-xe-11.2.0-1.0.x86_64.rpm

To complete the setup, you have to run

sudo /etc/init.d/oracle-xe configure

which allows you to specify the used ports, the sys and system password, and to enable Oracle server startup at boot time.

In my case, all those funny steps got the me desired

Installation completed successfully.

😉

To enable use of SQL*Plus inside the CentOS VM, add the following line to your ~/.bashrc:

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
export ORACLE_SID=XE

SQL*Plus itself is found at

/u01/app/oracle/product/11.2.0/xe/bin/sqlplus

I created a symlink in ~/bin for it.

If you want to be able to access your DB from your host machine (and not just inside the guest VM), then you should create a firewall rule for this. Else, you won’t be able to establish a connection:

$ sudo firewall-cmd --permanent --zone=public --add-port=1521/tcp
success
$ sudo firewall-cmd --zone=public --list-ports --permanent
1521/tcp

Posted in Dev

Oracle exp dump, compressed with gzip

Oracle exp does not support immediate compression of dump files, AFAIK. A neat way to achieve this, and to avoid filling your HD with unnecessary GBs, is to use Unix/Linux FIFO special files:

mknod my_schema.dmp p
gzip < my_schema.dmp > my_schema.dmp.gz &
exp system/system@XE owner=my_schema file=my_schema.dmp
rm my_schema.dmp
Posted in Dev

Poor man’s password encryption in OpenShift

We all know that Spring’s config server is the latest hype. Some people don’t find it very amusing to store credentials for production environments as plain text in a Git repo that also hosts dev/test environment configs and is hence readable by anyone.

The solution Spring uses (in the simplest variant), is to store such credentials in encrypted form in Git, then serve them to clients unencrypted. The encrypted values look like this: {cipher}U2FsdGVkX18ZpAC0I1eoChSMTU/2p/jsp2KoHpnnVFI=, and they are decrypted with a (symmetric) key provided as an environment variable ENCRYPT_KEY.

We don’t use Spring config server (yet), however we face a similar problem in our OpenShift environment. Currently, all .yml config files for OpenShift projects are stored as plain text in Git. And DB credentials are passed to our Pods as environment variables which are stored in those .yml files.

Here’s an idea borrowing from Spring’s config server to solve this issue. What we achieve:

  • We store the environment variable DB_PASS in encrypted form in the .yml file.
  • We decrypt its value in a shell script used for starting our app, and pass it to the app unencrypted.

And here’s some Bash code:

$ CIPHER_KEY=123abc
$ DB_PASS="{cipher}$(echo affenscheisse | openssl aes-256-cbc -a -salt -pass pass:$CIPHER_KEY)"
$ echo $DB_PASS
{cipher}U2FsdGVkX18ZpAC0I1eoChSMTU/2p/jsp2KoHpnnVFI=
$ if [[ $DB_PASS == {cipher}* ]]; then DB_PASS=$(echo ${DB_PASS#\{cipher\}} | openssl aes-256-cbc -a -salt -d -pass pass:$CIPHER_KEY); fi
$ echo $DB_PASS
affenscheisse

Now here’s the question: aren’t we back at square one? How do we pass CIPHER_KEY into the Pod without storing it in plain text in the .yml file? 😀

Posted in Dev

Simple encrypt and decrypt with openssl

pi@raspi2:~ $ echo affenscheisse | openssl aes-256-cbc -a -salt -pass pass:123 | openssl aes-256-cbc -d -a -pass pass:123
affenscheisse

pi@raspi2:~ $ echo affenscheisse > /tmp/secret; openssl aes-256-cbc -a -salt -pass pass:123 -in /tmp/secret -out /tmp/secret.enc; openssl aes-256-cbc -in /tmp/secret.enc -d -a -pass pass:123
affenscheisse
Posted in Dev