Message Dialog from outside the main gui thread

python blue and yellow logoIf you’ve ever wanted to show a dialog from a gtk app, but needed wanted to do it from outside the main thread, you quickly run into problems. Spawning things in gobject idle loops is usually fine for getting gui updates to happen in the main thread, but running a full blown dialog is not one of these easy feats.

So here’s a simple standalone program that fetches the dialog’s title and message before displaying it. You can also specify an image file path and icon size to override the default message info dialog icon.

From your worker thread, launch it using subprocess, system, or whatever you want, and immediately write the fields to it’s standard input. To simulate the usual followed by dialog.destroy blocking behavior where you’re waiting for the user to close the dialog, just wait for the process to complete, die, or check its pid if you really don’t trust your io pipes. The side bonus is it’s not blocking if you just spawn it asynchronously and you will know the cleanup is handled since the dialog is not a part of your calling process other than the handle you create for spawning it.

import gtk
import sys

# read stdin lines for data
title = sys.stdin.readline().strip() # Dialog Title
msg = sys.stdin.readline().strip()  # Dialog Label
img = sys.stdin.readline().strip() # Dialog Image Filename
size = int(sys.stdin.readline().strip()) # Desired image pixel size

# create dialog
dialog = gtk.MessageDialog(None, gtk.DIALOG_DESTROY_WITH_PARENT, gtk.MESSAGE_INFO, gtk.BUTTONS_OK, msg)

# override default dialog icon
    pixbuf = gtk.gdk.pixbuf_new_from_file_at_size(img, size, size)
    image = gtk.Image()

# run
response =

Dirty hack or part of the ingenious gtk masterplan?

Shrink PNG file size from the console

How to decrease png file size dramatically

ape in a dress dancingPngquant bills itself as a command-line utility and a library for lossy compression of PNG images. Over on Sourceforge, optipng claims to be a PNG optimizer that recompresses image files to a smaller size, without losing any information. Combining optipng’s optimization with pngquant’s compression can drastically decrease png file sizes.

WordPress, not happy with just being a blogging software, has grown into an extremely popular do-it-all framework. Let’s use some of the basic icons and logo images provided with every installation to showoff the power of optipng and pngquant. The wp-admin/images directory contains a bunch of images, utility stuff.

Let’s look at two files, the largest in the directory and another somewhere in the middle that already seems pretty small to begin with.

Major Reduction

icons32-vs-2x.png is 37,994 bytes. Squash it with pngquant’s lossy image compression, then re-optimize it with optipng and it’s down to 21,803 bytes. That’s almost half the original size. Yet to the naked eye, the data loss is, for most intents and purposes, unnoticeable.

Minor Reduction Army

Let’s examine the w-logo-blue.png for instance. This is used as the stylized ‘W’ icon that shows up on the login page.
Provided by WordPress, it’s a whopping 3,112 bytes. After lossy image compression and re-optimization, it’s only 3,093 bytes. A waste of time for a minor reduction of 1%, but not when you repeat this process for hundreds of images. The real power is obvious when you shrink a few larger files but every bit counts, literally. Multiply that squad of itty-bitty images by the number of times your pages are served and now we’re getting somewhere.

So let’s crush a whole directory at once.


Run pngquant on all the files ending in ‘png’ and do it in place instead of creating new files by specifying the extension with ‘-ext’.

for x in `ls *.png`; do pngquant -f -ext .png $x; done


Squeeze out a little more by optimizing the new files with optipng.

for x in `ls *.png`; do optipng -o7 $x; done

You can use the most exhaustive search “not generally recommended” by optipng’s help page, probably because it takes so long and doesn’t make much difference. May as well if you don’t care how long it takes.

for x in `ls *.png`; do optipng -zc1-9 -zm1-9 -zs0-3 -f0-5 $x; done

Binary Buffers and Python Array Performance

Pre-allocate and Re-use for Performance

shipping containerWorking with binary packed data is typically used for high performance situations or passing data in/out of extensions. You can optimize by avoiding the overhead of allocating a new python array for each structure. The pack_into() and unpack_from() methods allow you to write to pre-allocated ctypes string buffers directly.

Here’s a common example of packing that I’ve fleshed out to include a few extra data types. You should probably use ctypes for the container whenever performance is critical, but the difference between python arrays and ctypes string_buffer doesn’t seem that bad for most cases.

#!/usr/bin/python -Ott
import array
import ctypes
import struct
import binascii

s = struct.Struct('c ? I f s')
values = ('z', True, 4, 2.54, 'word')
print 'ctypes.string_buffer:'

b = ctypes.create_string_buffer(s.size)
print 'Before packing  :', binascii.hexlify(b.raw)

s.pack_into(b, 0, *values)
print 'After packing  :', binascii.hexlify(b.raw)
print 'Unpacked:', s.unpack_from(b, 0)
print 'array:'

a = array.array('c', '\0' * s.size)
print 'Before packing :', binascii.hexlify(a)

s.pack_into(a, 0, *values)
print 'After packing  :', binascii.hexlify(a)
print 'Unpacked:', s.unpack_from(a, 0)

Performance Testing:

I tested both buffer types by looping through and filling a python array and a ctypes string_buffer using pack_into and unpack_from 250,000 times with an incremental double. It’s a silly example usage, but the python array took a little over 25 seconds to fill, while the ctypes took 22 seconds.

For comparison, I also testing with reallocation of the array on each iteration of the loop. The total time jumped up to a minute and a half.

A more specific example:

Binaries used for decoding on two ends of a messaging interface can convey a lot of information and is quite a bit more efficient than sending structures full of other larger things like more arrays, doubles, etc. The payload of a binary buffer saves a lot in terms of bandwidth when you’re sending messages very frequently.

And besides, you can’t always control the incoming data type when you need to talk to another piece of code through a messaging interface.

Start KVM virtual machines from the console with virsh

Here’s what I see when I run virsh and type list:

# virsh
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
   'quit' to quit

virsh # list
 Id Name                 State

I’m in the middle of working from the command line in a remote ssh session on a KVM virtual machine host that runs two virtual machines. For reasons unclear at this time, I can’t list domains, interfaces, volumes, or anything from virsh on the host. However, I can still start and stop virtual machines and flag them for autostart and all I need to do right now is start a vm.
kvm logo with tux juggling
Nothing there. I was expecting to see the same two virtual machines by name that I see from virt-manager. But i’m stuck on a console with no display available right now. virt-manager is not an option. I know there are other vm managers available that may be browser based or whatever, but I don’t have them installed.

Although virsh won’t let me query much information, I do know the names of the machines I wish to start and I have a simple ssh shell on the host machine. So lets give it a shot.

dontpanic@host:~# virsh autostart vm1
Domain vm1 marked as autostarted

dontpanic@host:~# virsh start vm1
Domain vm1 started

Everything went better than expected. So if you’re used to working in a virtual machine manager GUI but for whatever reason you can’t/won’t, give the virtual shell “virsh” a try.

If you’re on a debian based system, virsh is found in the libvirt-bin package. If you’re on an rpm based system, it’s in libvirt-client.

Pygtk: Check to see if a button is sensitive

Setting and unsetting flags is a no-brainer, but how do you just check to see if a flag such as ‘gtk.SENSITIVE’ is set for something like a button? Here’s an example of how easy it is.

Use the bitwise and operator, ‘&’ with the flag you’re interested in.

It’s pretty simple but hard to find this answer when you’re looking for a method to extract a bool value. I wish it was as easy as using ‘is_sensitive()’ that returns true if the widget flag ‘gtk.SENSITIVE’ is set. But use the ampersand with the flags and you’re all set.

two power buttons on silver

This is specifically for pygtk and only applicable to buttons, really you should be moving to QT because everything from documentation to implementation of a button is that just much smoother.

Here’s a short example of checking for a save button’s sensitivity before proceeding to quit the main loop. This is something you may use when a quit button is clicked but changes have been made to a document and you want to give the user one last chance to save before exiting your application.

def quit(self, widget):
  if self.button_save.flags() & gtk.SENSITIVE:
    msg = "Save changes before closing?"
    dialog = gtk.MessageDialog(None, gtk.DIALOG_DESTROY_WITH_PARENT,
            gtk.MESSAGE_INFO, gtk.BUTTONS_YES_NO, msg)
    dialog.set_title("There are unsaved Changes")

    response =
    if response != gtk.RESPONSE_YES:
      return False


Download and build proxytunnel in Fedora

This program connects stdin and stdout to a server somewhere on the network, through a standard HTTPS proxy. We mostly use it to tunnel SSH sessions through HTTP(S) proxies, allowing us to do many things that wouldn’t be possible without it.

* Create tunnels using HTTP and HTTPS proxies (That understand the HTTP CONNECT command).
* Work as a back-end driver for an OpenSSH client, and create SSH connections through HTTP(S) proxies.
* Work as a stand-alone application, listening on a port for connections, and then tunneling these connections to a specified destination.

This was originally posted on rootninja back in December 2009 and dug up from the Internet graveyard that is the wayback machine. Versions of Fedora and this app have changed significantly. Use this howto as-is at your own risk. As of Fedora 21 (at least), you’re better off using yum to download the prebuilt and configured binary version instead of building it (as far as just getting it to work goes anyway!) But hey, maybe you’re still using Fedora 12 for one reason or another. No judgement.

Get it

$ svn co

    A trunk
    A trunk/website
    A tags/v1-6-0-rc1/proxytunnel/Makefile.cygwin
    A tags/v1-6-0-rc1/proxytunnel/Makefile
    Checked out revision 248. 

Build it

Getting it built seems to be pretty straight forward stuff…

$ make

    make -C docs
    make[1]: Entering directory /home/svn/proxytunnel/trunk/proxytunnel/docs'
    asciidoc -b docbook -d manpage proxytunnel.1.txt
    make[1]: asciidoc: Command not found
    make[1]: *** [proxytunnel.1.xml] Error 127
    make[1]: Leaving directory /home/svn/proxytunnel/trunk/proxytunnel/docs’
    make: *** [docs] Error 2 

$ yum install asciidoc

    Running Transaction
    Installing : asciidoc-8.4.5-4.fc12.noarch 1/1
    asciidoc.noarch 0:8.4.5-4.fc12

$ make

    make -C docs
    make[1]: Entering directory /home/svn/proxytunnel/trunk/proxytunnel/docs'
    asciidoc -b docbook -d manpage proxytunnel.1.txt
    xmlto man proxytunnel.1.xml
    make[1]: xmlto: Command not found
    make[1]: *** [proxytunnel.1] Error 127
    rm proxytunnel.1.xml
    make[1]: Leaving directory /home/svn/proxytunnel/trunk/proxytunnel/docs’
    make: *** [docs] Error 2

$ yum install xmlto

    Running Transaction
    Installing : xmlto-0.0.23-2.fc12.x86_64 1/1
    xmlto.x86_64 0:0.0.23-2.fc12

$ make

    Note: Writing proxytunnel.1
    asciidoc -d manpage proxytunnel.1.txt
    asciidoc proxytunnel-paper.txt
    rm proxytunnel.1.xml
    make[1]: Leaving directory `/home/svn/proxytunnel/trunk/proxytunnel/docs’

$ ./proxytunnel –help

    proxytunnel 1.9.0 (rev 248) Copyright 2001-2008 Proxytunnel Project
    Usage: proxytunnel [OPTIONS]…
    Build generic tunnels trough HTTPS proxies using HTTP authentication

    Standard options:
    -i, –inetd Run from inetd (default: off)
    -a, –standalone=INT Run as standalone daemon on specified port
    -p, –proxy=STRING Local proxy host:port combination
    -r, –remproxy=STRING Remote proxy host:port combination (using 2 proxies)
    -d, –dest=STRING Destination host:port combination
    -e, –encrypt SSL encrypt data between local proxy and destination
    -E, –encrypt-proxy SSL encrypt data between client and local proxy
    -X, –encrypt-remproxy SSL encrypt data between local and remote proxy

    Additional options for specific features:
    -F, –passfile=STRING File with credentials for proxy authentication
    -P, –proxyauth=STRING Proxy auth credentials user:pass combination
    -R, –remproxyauth=STRING Remote proxy auth credentials user:pass combination
    -N, –ntlm Use NTLM based authentication
    -t, –domain=STRING NTLM domain (default: autodetect)
    -H, –header=STRING Add additional HTTP headers to send to proxy
    -x, –proctitle=STRING Use a different process title

    Miscellaneous options:
    -v, –verbose Turn on verbosity
    -q, –quiet Suppress messages
    -h, –help Print help and exit
    -V, –version Print version and exit

And there you have it, enjoy.

Freighter Ganking in Eve Online

A “day in the life of” short story told through my eyes as an Eve Online player in late 2012.

freighter dominix daredevil eve online

We’re waiting a few jumps down the space trade lane for one of our scanners to squawk on coms. I’m parked near a planet in a 0.5 security system, just off-grid from the warp gates in a suicide-gank fit shooter with seven other friendly shooters in fleet. Tonight my weapon of choice is a Dominix. We’re loaded to the teeth with offense, no defensive mods at all. Our guns aren’t even fully loaded as these ships will soon meet their fate as space debris when the gank goes down.

While we’re sitting idle, the scanners are busy, actively sizing up the payload of neutral players passing by in various cargo vessels. We’re just waiting for one of them to call out a target. We have a few haulers and a freighter with us, empty, ready to scoop up the goodies when we crack open a pinata.

Coms are laid back, there’s an ongoing discussion about Twinkies or something, I don’t really know. I’m only half paying attention, listening for a scanner to call a target. All of a sudden we hear the words “check, check”, then silence. “Three billion. A Freighter. Hydrogen Isotops all in one stack”, a scanner calls out. I could literally hear the grin on his face.

Our “bumpers” immediately warp down to the gate. They’re in Machariel’s, pirate battleships purposely fit for bumping into a target and giving it massive inertia out of line with its warp destination, effectively making it impossible for him to leave.

The freighter appears on grid and they begin ramming their ships into him, one after the other, pushing the massive hauler well off the gate. He’s not going anywhere any time soon, but the player just logged out and we don’t have aggression yet. We need to shoot soon before his ship disappears.

After a few solid bumps I hear the call for shooters and we fleet-warp down. Whether the goods get blown up with the freighter or drop like candy is an all-or-nothing roll of the dice since it’s just one stack.

“Drones out!”, the fleet commander yells. “Preheat your guns! 3… 2… 1…, Fire!” We engage our drones and we fire our Heavy Neutron Blasters in unison. Everyone’s flashing red and explosions are going off in every direction as we pop, one by one.

A few seconds later, Kablah! The target freighter rips apart.

At this point I’m in my pod along with all the other shooters as our battleships were systematically disabled and destroyed by CONCORD, the non-player space police. But as a shooter, my job is over.

There’s a freighter wreck on the field and I can see through the explosion it’s solid blue. Jackpot, the loot dropped.

Friendly haulers and our empty freighter are landing at the wreck, spamming their open-cargo and loot buttons. Milliseconds later the wreck goes from full to empty on my overview and our freighter pilot is sounding fat and happy. We have the prize, but it’s still on grid.

Real time begins to dilate. “Align… Align… hurry up and warp already!”, is all I can think. And then it happens… Goonies drop out of warp in Machariel’s of their own and begin ramming our freighter even farther from the gate!

As shooters, we’re helpless to watch as the 15 minute global criminal timer starts trickling down. We’re labeled kill-on-site and stuck in the station or off somewhere in space in escape pods. We can’t reship to help in any way. Some of us still on grid in pods begin to disappear as neutrals give us the clone express. On to the alts!

My main is not far away in a Daredevil, a fast little frigate with advanced stasis webbing capabilities. I quickly ditch the tank for nanofibres in the lows. I’m 11 jumps out, banging through the gates as quick as I can. 3 jumps out, 2 jumps, 1 jump away, finally landing on the gate in no time. No one’s shooting yet as they’re still bumping our guy into harms way.

I enter system and warp to our freighter at zero meters. In my tiny frigate hull, surrounded by enemy battleships and one expensive freighter, everything looks massive. One after another they land their bumps. They’re pushing him away from CONCORD and buying themselves time for their own shooters to arrive so they can 3,2,1 him.

With the help of another teammate, we quickly web down the freighter. It’s not helping with the inertia the bumpers are putting into him, but it seems to be keeping his top speed down, keeping him on grid longer, so we keep at it. Every once in awhile a bumper misses and it looks like the freighter will enter warp. But then another bump lands and the freighter goes flying again.

Are we making it easier for them to bump or slowing down the overall drift and buying time? Unknown. They’re still missing some bumps and there’s debate on coms, but one thing is for sure, we need grid-fu. I burn away from the action and drop a can, desperately trying to keep the grid intact.

We’re hoping they miss the bumps a few times in a row so our webs can get the fat cow into warp, but half an hour goes by and we’re still at it; it’s not looking good. We switch up tactics, only webbing when the freighter pilot calls for it. Still no warp.

Our freighter pilot is beginning to sound a little shaken on coms. He’s calling for friendly bumps from the other direction to cancel some of the inertia, more webs to slow him down, more pilots to reship, and logistics to get in system with armor reps in preparation for the inevitable. They’re bumping him in all directions, but he’s a pro and keeps it cool. At one point he notices the inertia has changed in the direction of the gate. He quickly realigns and enters warp back to zero on the gate. We just bought ourselves more time.

Should he jump through to the other side and try to run before they catch him? Log off and hope to disappear before aggression is established, or stay here to keep trying the warp to the station. At 0.6, the next system is higher security space, but the police aren’t there and the goons might be waiting for us on the other side. No one is in position on the other side and we’re still surrounded by space police here, so we’re safe… for now.

A few seconds pass and it’s too late to jump anyway. Our cash cow is already bouncing off again. He’s well off the gate in no time. Two guardians appear on the field but they’re not blue to us. One of our pilots convos them and finds out they want nothing other than to disrupt this operation. Our spirits are rising. The enemy of our enemy is our friend.

Enemy shooters are arriving. I see a ball of neutrals land close to our man. The grid is getting really crowded. My overview is filling with flashing red followed by explosions in space. New wrecks appear where ships once were. Catalyst down. Talos down. I lock up anything flashing and cycle my guns. My microwarpdrive is overheated and will burn out soon, but it’s all climaxing now. I’m long pointed by a Talos. I hope he uses the guns on me instead of the freighter because I’m feeling invincible as I tear through the field with high transversal on everyone. No one can keep on me for more than a second before CONCORD puts them down.

The frieghter is still in tact, but I hear his warning sirens going off in the background as he announces the shields are gone. We need more logistics ships to appear right now. The two guardians are staggering their reps.

Another group of high damage enemy ships drop out of warp right next to us. I’m red boxed by a catalyst but he’s out of range. Catalyst down. Everything is on fire and exploding all around me. And then it’s over. Freighter down.

But this time, that single stack of cargo went up in smoke.

Cluster service wont start with rgmanager fixed

Cluster service wont start because it’s in a failed state caused from the service attempting to start but failing too many times in too short of a time period (based on your cluster.conf retries and remember policy) or, more likely it’s failing to shutdown cleanly.

Manually enabling the service fails, right?

# clusvcadm -e host1service

Checking syslog, you will see the service refusing to start and the reason why. In this case it’s complaining it failed to stop cleanly, but the road block is really that it’s in a failed state.

# tail /var/log/messages
rgmanager: Service:host1service has failed; can not start.
rgmanager: Service:host1service failed to stop cleanly

four people with a thumbs up
Assuming cluster.conf is valid because you ran ccs_config_validate and it told you so, the problem is probably pretty simple. If a service is in a failed state, you can’t enable it unless you disable it first!

Perhaps this functionality is to stop you from starting a failed service because it’s already tried to start up a number of times and couldn’t already, so what’s the point of another try? In my case, what’s the harm? It’s already failed again and again. Why not just retry it when I step in and manually ask for the service to start?!

Oh well, it’s just one extra step.

Disable the service, then enable it. If you really have fixed the problem that caused the failure in the first place, the service should start right up.

# clusvcadm -d host1service
# clusvcadm -e host1service

Checking syslog once again, you should see the service disable, renable, and fire off the startup script.

Stopping service service:host1service
rgmanager: [script] Executing /etc/init.d/ClusterService stop
rgmanager: [script] Executing /etc/init.d/ClusterService start
rgmanager: Service service:host1service started

Check the cluster status to make sure it started.

# clustat
Cluster Status for hacluster @ Fri Nov 9 02:15:00 2012
Member Status: Quorate
Member Name ID Status
------ ---- -- ------
host1 1 Online, Local, rgmanager
host2 2 Online, rgmanager
host3 3 Online, rgmanager
Service Name Owner State
------- ---- ----- -----
service:host1service host1 started

If the service is relocatable, now is a good time to test relocation to make sure it works properly and doesn’t fail on the other available nodes.

# clusvcadm -r dnsmasq -m host2
Trying to relocate service:dnsmasq to host2…Success
service:dnsmasq is now running on host2

If relocation fails on any particular node, fix the problem or take that node out of the failoverdomain in cluster.conf.