Marrying SOGo and Gentoo: part 2

It's been a while!

I know I know... I promised this over two years ago and I neglected this blog for too long. This is no excuse, because you already know I will likely neglect it again.

However, I did not want to miss this chance to tell you, reader, that I did manage to complete the ebuilds I mentioned in my previous post. I even made a kind of overlay for it:

https://github.com/godlike64/sogo-overlay

It will likely be outdated (now that I check SOGo's webpage, latest stable is 2.3.2. Well, I can't be in every place at the same time), but SOGo 3 is about to come out, which implies a lot more fun for me :D.Once the final version is out I will make sure to update these ebuilds.

In the meantime, feel free to check my github (link above) for some joy in checking completely unfinished projects.

Cheers!

Getting multi-head to work on Optimus laptops

Howdy. In recent times, my employer saw fit to provide me and my teammates with a Thinkpad W520. I have a W530 at home so there's not much difference between the two and I knew the beauties that we would be getting.

Also in recent times, thanks to an event I was part of, I found out that the VGA output connector bundled in these laptops is hard-wired to the NVIDIA card. Effectively, booting in integrated mode (thanks to Lenovo's BIOSes which allow you to choose between Intel integrated - NVIDIA discrete - Optimus mode) would render the output connectors completely useless. Initially I did not give this much thought (I have 4 monitors on my desktop PC at home, and whenever I use the W530 it is from the coffee table, while I watch a movie or a series' episode).

After receiving my W520, I quickly switched to NVIDIA mode from the BIOS, corrected the system (KMS off, nvidia-drivers merged) booted it up and tried to use the three displays (LVDS, VGA and DisplayPort). Sadly, the card in this model (1000M) does not seem to be able to handle more than two simultaneous screens (LVDS + VGA OR DP). I read that Kepler cards (like the K1000M the W530 has) should be able to run three displays, but this was not the case. The only way out for me was Optimus.

It came to my surprise that now Optimus has a nice level of support on Linux. And I'm not talking about running a specific app on a separate X server running on the NVIDIA card, drawing the app contents onto the Intel card's framebuffer. I mean real support, being able to use the external display connectors while running on the Intel card.

Here follow some guidelines as to how I got this working.

Requirements

You will need fairly recent versions of involved packages:

• x11-drivers/nvidia-drivers-337.12 (should also work on 334.21)
• x11-apps/xrandr-1.4.1 (needed to get the randr extensions for VIRTUAL displays)
• x11-base/xorg-server-1.15 (should also work on 1.14)
• x11-drivers/xf86-video-intel-9999 (sadly, even though the latest bundles should be apt, they do not include[1] needed[2] bugfixes[3]).

Caveats

On the NVIDIA side, there's a recent issue (with driver versions >304.xx) where the EDID of the connected display is not read properly, and it must be specified directly on the xorg.conf file, so as to be read later by xrandr. For reference, here's my current /etc/bumblebee/xorg.conf.nvidia, prepared for two external displays (one on VGA-0, one on DP-0):

Section "ServerLayout"
    Identifier  "Layout0"
    Option      "AutoAddDevices" "false"
    Option      "AutoAddGPU" "false"
EndSection

Section "Device"
    Identifier  "DiscreteNvidia"
    Driver      "nvidia"
    VendorName  "NVIDIA Corporation"

#   If the X server does not automatically detect your VGA device,
#   you can manually set it here.
#   To get the BusID prop, run `lspci | egrep 'VGA|3D'` and input the data
#   as you see in the commented example.
#   This Setting may be needed in some platforms with more than one
#   nvidia card, which may confuse the proprietary driver (e.g.,
#   trying to take ownership of the wrong device). Also needed on Ubuntu 13.04.
    BusID "PCI:01:00:0"

#   Setting ProbeAllGpus to false prevents the new proprietary driver
#   instance spawned to try to control the integrated graphics card,
#   which is already being managed outside bumblebee.
#   This option doesn't hurt and it is required on platforms running
#   more than one nvidia graphics card with the proprietary driver.
#   (E.g. Macbook Pro pre-2010 with nVidia 9400M + 9600M GT).
#   If this option is not set, the new Xorg may blacken the screen and
#   render it unusable (unless you have some way to run killall Xorg).
    Option "ProbeAllGpus" "false"

    Option "NoLogo" "true"
    Option "UseEDID" "true"
    Option "Monitor-DP" "Monitor0"
    Option "Monitor-VGA" "Monitor1
    #Option "UseDisplayDevice" "none"
    #Option "AllowSHMPixmaps" "true" <-- No matter what you read on the Internet, do NOT enable this. It will cause black output on the external monitors.
EndSection

Section "Monitor"
    Identifier "Monitor1"
    #Option "ModeDebug" "TRUE"  #Only needed to debug and to get the DFP-X number, is written to /var/log/Xorg.8.log
    Option "ConnectedMonitor" "VGA-0"
    HorizSync   35-81
    VertRefresh  35-76
        Option "ModeValidation" "AllowNonEdidModes"
    Option "ModeValidation" "NoDFPNativeResolutionCheck"
    Modeline "1440x900_60.00"  106.50  1440 1528 1672 1904  900 903 909 934 -hsync +vsync
EndSection


Section "Monitor"
    Identifier "Monitor0"
    #Option "ModeDebug" "TRUE"  #Only needed to debug and to get the DFP-X number, is written to /var/log/Xorg.8.log
    Option "ConnectedMonitor" "DP-0"
    HorizSync   35-81
    VertRefresh  35-76
        Option "ModeValidation" "AllowNonEdidModes"
    Option "ModeValidation" "NoDFPNativeResolutionCheck"
    Modeline "1280x1024_50.00"   88.50  1280 1352 1480 1680  1024 1027 1034 1057 -hsync +vsync 
EndSection

On the Intel side, recent builds (2.99.xxx) of the driver bundle the necessary intel-virtual-output binary to configure the virtual displays automagically. Bear in mind that this binary does not provide any debug output at all by default (even though you might have the debug USE activated). In order to have debug output in case you need to troubleshoot, you have to first edit the tools/virtual.c file:

diff --git a/tools/virtual.c b/tools/virtual.c
index 5883950..c5316a2 100644
--- a/tools/virtual.c
+++ b/tools/virtual.c
@@ -65,7 +65,7 @@
 #include <fcntl.h>
 #include <assert.h>

-#if 0
+#if 1
 #define DBG(x) printf x
 #define EXTRA_DBG 1
 #else

Also, beware some configuration needed on Bumblebee's side (/etc/bumblebee/bumblebee.conf):

[bumblebeed]
VirtualDisplay=:8
KeepUnusedXServer=true
ServerGroup=bumblebee
TurnCardOffAtExit=false
NoEcoModeOverride=false
Driver=nvidia
XorgConfDir=/etc/bumblebee/xorg.conf.d

[optirun]
Bridge=auto
VGLTransport=proxy
PrimusLibraryPath=/usr/lib/primus:/usr/lib32/primus
AllowFallbackToIGC=false

[driver-nvidia]
KernelDriver=nvidia
PMMethod=none
LibraryPath=/usr/lib64/opengl/nvidia/lib:/usr/lib32/opengl/nvidia/lib:/usr/lib/opengl/nvidia/lib
XorgModulePath=/usr/lib64/opengl/nvidia/lib,/usr/lib64/opengl/nvidia/extensions,/usr/lib64/xorg/modules/drivers,/usr/lib64/xorg/modules
XorgConfFile=/etc/bumblebee/xorg.conf.nvidia

[driver-nouveau]
KernelDriver=nouveau
PMMethod=none
XorgConfFile=/etc/bumblebee/xorg.conf.nouveau

Steps

Make sure you are booting into Optimus mode, using the Intel card to drive the laptop's main display (LVDS1).
Log on to your desktop environment of choice, open up a shell.
/etc/init.d/bumblebee start
modprobe bbswitch (also, encure /proc/acpi/bbswitch reports the NVIDIA card as being ON)
optirun true (this is to ensure the card is turned on and does not get turned off. I can omit this and it still works).
intel-virtual-output (I have to run this as root, did not spend too much time trying to see how could I run it from my user without it blowing up on my face).
By now (it takes a few seconds) you should be getting your desktop environment popup, notifying you of the new display(s) that has/have been attached.
You can now fire up the xrandr frontend of your choosing, and configure your VIRTUALX (where X is a number) displays to your delight.
Even if you do not plan to use a triple-head setup

[1]: https://bugs.freedesktop.org/show_bug.cgi?id=71345
[2]: https://bugs.freedesktop.org/show_bug.cgi?id=73816
[3]: https://bugs.freedesktop.org/show_bug.cgi?id=76721

Your daily nightmare with ABI_X86

Let's take a break here from the previous post. I don't know if you've read about it, but Portage (hence, Gentoo) is on its way to becoming a true multilib distro, the Gentoo way. This means that in the near future you won't need precompiled 32 bits libraries anymore (thus, effectively removing emul-linux-x86-* packages). This is a good thing...

... is it?

No, seriously, this is a good thing. Gentoo's philosophy of compiling and adjusting packages to your needs clashes with what we've had over the years in the emul-linux-x86 packages (not being able to change anything since they're precompiled binaries, having to wait for a new snapshot if something isn't working properly, etc.); so we're finally bringing the Gentoo philosophy to the multilib part. And this will apply not only to 32bits binaries from the x86 architecture. I've already seen hints of other ABIs for other archs.

This will be implemented with an expand USE: ABI_X86. All you have to do is define this in your make.conf:

ABI_X86="64 32"

Supposing you'll be wanting 32bits copies of any library that supports them so as to properly run e.g. wine. Another way is specifying individual USE for packages:

category/package abi_x86_32 abi_x64_64

However, the road is long and there are many perils. First of all, this is all in ~arch, so if you have a stable system, you can bother to read this when the ABI changes are pushed to stable, hopefully with more documentation. Second of all, if you have a completely unmasked system for ~, you won't notice any problems likely. The thing is, if you have a mixed system (stable but a few things unmasked). This is where all hell breaks loose.

I've already fought twice with this on my PC and one on my laptop, both mixed stable and ~ amd64 systems. Whenever I have one of these wrestling sessions with any of my Gentoo systems, I tend to work fast and loose, trying many things at the same time whenever possible. That is why I don't have a fixed walkthrough that can get you to this, but nevertheless my memory serves me right on some points which I'll write below:

• You will get many slot blocks the first time around (and probably most of the following times until you've fixed this). This is normal. Don't panic.
• Consider removing libreoffice, wine and your currently installed emul-linux libraries. You can install them again later.
• Unmask abi_x86_32 by putting the following into /etc/portage/profile/use.mask (seriously, I'm not joking you in this. I've spent three days fighting with my laptop until this solved everything. The creepy symptom if you don't do this is that most packages will build fine with the 32 ABI, but some Xorg libraries will refuse to merge, and will show 32 as hard negated in ABI_X86 ("(-32)"):

-abi_x86_32

• If this gets paired with a big world update (as is almost always the case with me and my systems), consider dividing the upgrade by sections and doing those sections first before trying to update world. As an example, on my laptop I first tried to upgrade kde-meta to the newest version and solved first the blockers for that package, before tackling world. This removes around ~300 packages from the world update afterwards.
• Whenever you have a slot conflict, this is due to the ABI change. Most of them can be fixed by individually upgrading the offending package to the newest version (keep in mind that most likely the newest version will be the one supporting the ABI changes. And also keep in mind that you might need to unmask said newest version of the package).

That's all I can remember for now. I seriously hope more documentation to come before the ABI move to stable. And I seriously hope noone has to go through this hell again.

Cheers!

Marrying SOGo and Gentoo: part 1

Long time no see! Since I've been away I've been fiddling with many many things (personal projects, life, etc.). One of my latest adventures was trying to make SOGo work on Gentoo. But first, a little bit of background:

My personal pain

6 years ago I was building my very first email server. It was not much but it is still working, albeit with a few modifications. I've kept postfix but replaced courier with dovecot for the IMAP part, and replaced MySQL with OpenLDAP for the authentication part. Back then one of the missing things was a calendaring solution to work on top of it. Keep in mind this was 2007.

There was no good solution available, given the self-imposed features that the system had to have in order to satisfy my needs:

1. It had to be FLOSS.
2. It had to work seamlessly with and reuse my current setup (no starting from scratch).
3. It had not to use Java.
4. It had to look like it was made after 1995.

#1 left most commercial solutions out and #2 left things like Kolab out. Meanwhile, #3 left Zimbra and OpenXchange out and #4 left Horde out.

Note: I did try to install Kolab one time in a VM. Let's just say the end result was not pretty at all.

Back in that day I found things like DAVical which at first glance seemed to work (basically it was DAV thru HTTP on a PHP webapp). Sadly, the state of PIM solutions at that time was not good (KDE4 was either coming out or had just come out, and GNOME had the ever-depressing Evolution). After several tries to make it work with my partner, we eventually gave up.

A sign of hope

A few months ago a colleague pointed me to SOGo, a solution which seemed to fullfil my criteria (it was FLOSS, it was not Java, it looked REALLY nice and it integrated seamlessly with an existing SMTP/IMAP/authentication backend). I decided to try it out...

... sadly, the ebuilds I found in the gnustep overlay were quite outdated. The last stable one was 1.2.6 (SOGo is now at version 2.0.7) and the live ones which point to their monotone server did not work (it seems their server falls down from time to time). I had to compile it by hand, for which SOGo provides some simple instructions. After editing SOGo's configuration file, which is also heavily documented (quite a bit hard to get the hang of GNUstep, but once you do it's simple enough), I had created a user which did not have a home directory. That kind of messed up starting SOGo.

I dug a bit more and created the needed directories by hand and fixed the permissions (remember: no initscript or anything):

# mkdir /var/spool/sogo
# chown sogo:sogo /var/spool/sogo
# mkdir /home/sogo
# chown sogo:sogo /home/sogo
# mkdir /var/log/sogo
# chown sogo:sogo /var/log/sogo
# mkdir /var/run/sogo
# chown sogo:sogo /var/run/sogo

Also, I found a nice Nginx sample configuration file which I used to proxy SOGo thru my web server (SOGo runs on port 20000 by default and is designed to work behind a web server). After enabling memcache (both the initscript and the Nginx module), I prayed and tested it...

... and it works beautifully. Next steps are making proper initscripts and conf.d files and after that proper ebuilds for SOGo and its main library, SOPE.

Stay tuned.

Change in IPTables: state vs conntrack

A quick one so as not to lose the rhythm :) it appears that the state match in iptables has been deprecated in favor of conntrack.

A quick sed on your firewall script should suffice:

s/-m state --state/-m conntrack --ctstate/g

To Hell and Back Again (or "How to update an old Gentoo without reinstalling")

Recently (ok, not recently, it was 5 months ago) I decided to set up a dedicated server. The hosting company stated that Gentoo was one of the distros which the server could be delivered with, so of course I chose that one. The server itself is not a big deal, but the catch is that the hosting company provides a symmetric, uncapped, unmetered 100mbps link for the server (the uncapped part comes with a little increase in price, as do additional public IPs, should you need them), which is a connection I can absolutely not get in my country. So, great, right? After paying the first bill I was handed a precious server connected to the Internet like if it was hooked up on the thing itself, carrying a precious Gentoo system in its womb.

NO

I log in to the server, start digging around and I notice something odd. Hmmm eselect does not work, eix reports very old versions (>4 years old) of the packages installed. So right then I remember that a friend a couple of weeks ago had told me that there was a time (ages past) when the eix-update command was actually update-eix. Shuddering, I type it and, like the worst of my nightmares coming true, I see that command exists. A quick cat /usr/portage/metadata/timestamp.chk confirms it: last portage sync was ON 2008.

Desperately, I check the kernel version: 3.2. Ok so... wait, wha? 3.2? Yes. The kernel is 3.2, some custom crappy, hosting-company-provided kernel based on 3.2 mainline, sitting atop a userspace from... 2008. The server does not have the udev support we know and love. It even has CONFIG_IDE enabled. While I start tearing my brains, my eyes and my common sense out, I ask the hosting company to see if they have an updated Gentoo version to plant on the server. The obvious reply was that no, that they work with what the master company provides ("master company" in this case being the company who actually owns the datacenter(s)). So I had to make a choice: either install some RHEL or Debian-based distro or stick to this mutant and bring it up to date (you already know what I did).

What followed was partly documented (since in the rush, sadly some of the details were lost) in a file called gentoo.hell. The name could not be farther from the truth. I have tried to keep my notes as useful as possible, so here follows more or less what you should do/take into account should you ever come upon a similar situation:

• python, failed modules: when compiling Python and/or Python extensions, you may get a message for "failed modules". Check for similar packages (e.g. dbm -> gdbm) and recompile those first (follow logical order)
• gcc/glibc: this is the hardest part to pull off. The server had gcc-3.4 and an according glibc version. This of course limits you on what you can install as part of the upgrade process. However, you cannot upgrade both freely to the latest available version since they are circularily-dependent. The upgrade path that should work is: gcc-4.1 --> glibc-2.10 -->gcc-4.5 --> glibc-2.14 (this upgrade path should merge gcc-4.5 on its own when installing glibc-2.10). Take care of portage tree versions which have the ebuilds you need.
• portage/python: under NO CIRCUMSTANCES should you upgrade portage tree right away! You will lose important ebuilds that are needed for this particular upgrade path and will have to dig around the Gentoo CVS in order to get those, which can be a real PITA. Upgrade path for this is more or less as follows: portage-2.1.6 --> python-2.7 (maybe use 2.6 first if you cannot compile/install 2.7) --> portage-2.2
• automake and related failures: make sure latest automake point release versions for each slot are installed. Also make sure latest automake-wrapper is installed.
• ssmtp, groupadd error ('r'): update shadow
• bash errors: may need updating too before some other things

Remember:

• gentoolkit/portage-utils are your friends
• Remember to run python-updater when updating Python versions. Same goes for perl-updater.
• Save all the eselect news and keep them in mind for later use
• stage4: as much as you can get of those (squashfs is the best). This will allow you to set "savepoints" to which you can always roll back if you're stuck.

There it goes, more or less. It was the most epic learning opportunity I ever had regarding GNU/Linux and Gentoo. However, beforehand, think twice if you really want to do this ;).

NOTE: no kittens were harmed during the upgrade of the beast. At least not by me.

KVM + network bridging

Recently I discovered the joy of having a CPU with VT extensions and using KVM thru libvirt. One of the not-soy-joyful things is that libvirt does not support creating a bridged environment for you (much as VMWare/VirtualBox do). After migrating my windows vbox VM to KVM I started hating NAT networking (my KVM host is behind an iptables gateway, so I had to choose between two iptables on my network, or tunneling via SSH. Of course I chose the later).

So after some digging I finally configured my KVM environment using network bridging. Here are the simple steps.

NOTE: libvirt creates a virbr0 bridge for you, under which all vnet* ifaces will reside if using NAT. Under no circumstances should you add eth0 to this bridge. Doing so will leave you with no network connection.

1. Create a new bridge:
   # brctl addbr br0
2. Add your physical interface to the bridge (eth0 in this example):
   # brctl addif br0 eth0
3. Create initscript:
   # ln -s /etc/init.d/net.lo /etc/init.d/net.br0
4. Modify /etc/conf.d/net accordingly (sample extract below):
   

   modules="dhclient"
   bridge_br0="eth0"
   config_br0="dhcp"
   config_eth0="null"
   

5. Correct rc:
   # rc-update del eth0 [runlevel]
   # rc-update add br0 boot
6. Either reboot or:
   # /etc/init.d/net.eth0 stop
   # /etc/init.d/net.br0 start

You could perform step 6 concatenated inside a screen so if everything works correctly, your network will come back up normally.

As a last step, inside your KVM VMs, go to networking and choose "Specify shared device name" under "Source device" and enter "br0" in the text box.

It may take your VMs a couple of reboots to load successfully with the new IP (you'll note the VM was loaded fine because the IP of its network card will be one in the range of your network, not in a NAT range).

Using vpnc with iptables

Another quick one after all this time.

If you use a vpn like vpnc (Cisco VPN) and also iptables on a gateway server, and you are having trouble and/or would like to make the vpn available to your private network, don't forget to include this on your iptables script:

(replace "tun0" with however your tunnel device is named)

iptables -A FORWARD -o tun0 -j ACCEPT
iptables -A FORWARD -i tun0 -j ACCEPT
iptables -t nat -A POSTROUTING -o tun0 -j MASQUERADE

New Gentoo Wiki

Just read this today on planet.gentoo.org after taking a week off from work. There's a new (I think more official too) Gentoo Wiki, available at: http://wiki.gentoo.org/wiki/Main_Page

A word of advice on x11-libs/vte

You might want to keyword-unmask x11-libs/vte. There are two slots: 0 for gtk+-2 and 2.90 for gtk+-3. Right now Portage pulls in both GTK+ versions, so you'll likely need both vte versions, one for each slot. And apparently the ~ masked builds cause portage not to complain.

If you get errors like:

<x11-libs/vte-0.28.2-r201:0 ("<x11-libs/vte-0.28.2-r201:0" is blocking x11-libs/gtk+-2.24.8-r1)

Then most likely you'll be better off unmasking vte.