Static Linking?

From time to time, I have this moments when I cannot deploy my application properly and decide that I want to link it statically (then I generally give up, because it requires me to link the Qt libraries statically…). But is it really better to prefer static over dynamic linking?

As in so many other cases, it depends on what you want to do. I read that in terms of performance, there are trade-offs in both approaches, so in the end it really does not matter so much. From my point of view, the biggest advantage of static linking is the fact that you can ship one single file with your application, removing the risk of “broken” dependencies. That is, in terms of deployment, quite an advantage!

On the other hand, if everybody would link statically, we would literally have “thousands” of libraries “repeated” inside our system, packed inside “huge” binaries. It does not make much sense, does it?

Dynamic libraries are also “cool”, because we can (till a certain extent) replace them by newer (improved) versions, without having to recompile our application. That is like a huge benefit, in terms of “bug fixing” of third party libraries.

After removing the performance issue, my verdict would be:

  • For myself, I would like to minimize resource consumption by using as much as possible, shared libraries (dynamic linking).
  • For “bullet proof” systems, where users are not experienced in installing software, and are likely to “mess up” the system by removing parts of it, I would consider providing them statically compiled versions of the software, instead. The software will likely be “bigger “(although there are tools to minimize this, such as UPX), and a bit more “hungry” of resources, but this is also the only way to prevent the DLL hell.

Finally, it is important to mention that the type of linking may be conditioned by licensing issues.  For instance due to the “nature” of the license, GPL libraries would “contaminate” any software statically linked with them.

Advertisements

“Hello World” Cross-Compiling

I recently switched to work on a Linux environment, in a project where most of the potential users will be using Windows.
Theoretically that is not a problem, since I am sticking to standard C++, and a few other libraries that are fully cross-platform (e.g.: Qt[1], Boost[1])
Of course in practice, things are always slightly more complicated 🙂

Since my original project was created in Windows, using the nmake compiler, I still have a functional environment that I can use. However this solution requires me to:

  1. start Windows;
  2. checkout the latest source code;
  3. tweak the configuration for the Windows environment;

It is mainly 1. and 3. that bother me; I don’t start Windows that often, and there are quite a few things to tweak, when you change from GNU g++ to MS nmake[3].

There are many options to make things a bit “smoother”, and one of them would be to change the compiler to Jom[4] or MinGW, so that it could be use in both OS, with minimal tweaks. Another option would be to create a repository only for the source code, and leave the configuration files unchanged, both in Windows in Linux. These are all things that I want to explore, but for the moment I ceased to my “fascination” with cross-compiling, and decided to investigate it a bit further. Wouldn’t it be so cool to be able to generate a Windows binary, straight from Linux? It seems like the perfect option, if it was that easy!

To get a functional cross-compiling environment, you basically will need Wine[5] and MinGW[6]. In a nutshell, Wine is a is a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems; the acronym says everything: “Wine Is Not an Emulator”. MinGW is a minimalist development environment for native Microsoft Windows applications. Installing these software in Ubuntu, is as easy as:

sudo apt-get install wine mingw32 mingw32-binutils mingw32-runtime

This should be it; so I started with a simple example that I grabbed from [7]:

#include 

int APIENTRY WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance,
    LPSTR lpCmdLine, int nCmdShow)
{
  MessageBox(NULL,
    "Cette fenêtre prouve que le cross-compilateur est fonctionnel !",
    "Hello World", MB_OK);
  return 0;
}

I compiled it like this, using MingGW:

i586-mingw32msvc-g++ -o essai.exe essai.cpp

To execute the binary, you will need the mingw library: mingwm10.dll, that can be copied like this to the current directory:

gunzip -c /usr/share/doc/mingw32-runtime/mingwm10.dll.gz > mingwm10.dll

Then the application can be launched with:

wine essai.execute

hello2

References:
[1] http://qt-project.org/
[2] http://www.boost.org/
[3] http://msdn.microsoft.com/en-us/library/dd9y37ha.aspx
[4] http://qt-project.org/wiki/jom
[5] http://www.winehq.org/
[6] http://mingw.org/
[7] http://retroshare.sourceforge.net/wiki/index.php/Ubuntu_cross_compilation_for_Windows

Developing with Qt Creator

I recently decided to switch to develop in Linux, and switch from MS Visual Studio to Qt’s official cross-platform IDE: Qt Creator. I have to say it is quite a responsibility to replace Visual Studio, because from my personal experience Visual Studio is one of the best IDE’s (and also the best Windows application).

First things: to migrate a QT based project from Visual Studio to Qt Creator is not really an headache. First of all, generate the project file (.pro) in Visual Studio (if you don’t have it already), and be sure to “tune” it as you like! Next thing, you can import into QT Creator as a project file, and everything should run out-of-the-box.

One thing that deserves attention are the building configurations. You need to add at least one building configuration, for the Qt Version in use. You can do that by going to “Projects” on the left side pane. By default, Qt will create a “debug” and a “release” mode, so you will only need to setup one configuration. “Shadow builds”, are builds located outside the source directory, which as a matter of principle is a “tidy” idea; to enable them, you just have to check the “shadow build” checkbox, under each mode. Regarding the run configuration, I would think that is a good idea to point to the debug binary, if you ever want to debug your application (I bet you will). And this brings me the last topic: debugging in Qt Creator.

The fact that there is not so much documentation about Qt Creator (apart from the fancy “welcome wizards”) and the lack of profile and analysis tools, when compared to MS Visual C++ does not bother me too much. But what really bothered me was to be able to setup the debug, and since I almost spend a morning on that, I will write it here and hope it will be useful to someone.

In a nutshell, Qt Creator uses the Linux “official” compiler g++, and official debugger,gdb. As a general concept I think that is great: why develop new tools, when you have working (great) ones?
Now to have the gdb working correctly with Qt Creator is another story. I installed the latest version, from the ubuntu repositories:

sudo apt-get install gdb

It did not really worked very well. Then I read that Qt Creator uses python for debugging and that gdb had to be compiled with Python enabled, so I found a version of gdb released by the Qt Creator project, and switched to that one. It did not work very well either. Finally, after many combinations, I discovered that it was actually Python that had to be “gdb-enabled”. So I installed this version of Python:

sudo apt-get install python2.7-dbg

And “asked” my system to use that one as a default:

sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.7-dbg 30
sudo update-alternatives --config python

(the exact code differ for your system).
Finally, I tested the python support in gdb by calling it from the command line and entering a python command.

GNU gdb (GDB) 7.5-ubuntu
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
For bug reporting instructions, please see:
.
(gdb) python print 123
123
(gdb)

If you see the “123” on the gdb prompt, everything should be ok. Finally I tested gdb with my application, running it from the command line:

gdb faocasd

If all is running ok, then you can go back to Qt Creator and start the application in debug mode (by pressing F5). I encountered a few other small problems, but I was able to debug. Important to mention that the debugger does not always pick the changes, for which I advise you to always recompile your project! (I know, it may be annoying for a large project).

All is well, when it finishes well: here is my “hello world” Qt Creator debug! 🙂

Hello Debug! :-)

#%^~***!~@_@?

Recently I contacted an Hacker group, and I have been excited about joining their meetings (I’ll disclose the name later).

When I say “hacker”, I mean it in the good and original sense of the word, mostly “white hat”;

I have been thinking about a couple of topics that I would like to bring up in a meeting:

  • Backing up data: what is the best way to back up data? a RAID configuration, an external USB drive, the cloud, a computer “hotel”? I am looking for the characteristics of different solutions: price, efficiency, safety, privacy level, etc.
  • Online Privacy: Google spies on you, Twitter spies on you, Facebook spies on you. I basically would like to assemble a set of tools (preferentially FOSS) that would allow a normal person to have an online presence and use, safeguarding their privacy; e.g.: use DuckDuck instead of Google Search, use Opera instead of GMail, etc. I think these tools/recommendations should cover at least one case in each of the following categories: blog, micro blog, social network, webmail, pop mail, search engine (more recommendations are welcome!)

I am looking forward to hear comments and ideas about this. Please leave your comments here, or tweet me @doublebyte 🙂

>REGEX!W\

For a long time, I have been thinking about learning about of Regex, and today I just found the perfect excuse.

I have 277 strings like this:

ALTER TABLE Sampled_Strata_Vessels ADD CONSTRAINT DF_Sampled_Strata_Vessels_description DEFAULT(('missing')) FOR description

And I want to turn them into something like this:

ALTER TABLE Sampled_Strata_Vessels ALTER description SET DEFAULT ('missing');

(yes, I am converting constraints from SQL Server to PostgreSQL!)

I had a couple of options here:

– change them by hand (that is: 277 times!)

– write a C++ program to parse the strings and generate the results (or even better: a Python program!)

Somehow I did not felt like doing any of those things, so I decided to do something that took even longer, but hopefully made me gain something: learn how to do a regular expression and apply it in in Notepad++.

Regular expressions look both scary and magic;  it took me a couple of hours to understand what are groups, how to search for literals, etc.

In the end of the day I ended up with a smile on my face, and these two expressions:

search:

(ADD)(.*)(DEFAULT)(.*)(FOR)(.*)

replace:

ALTER \6 SET DEFAULT \4;

It looks stupidily easy, but I cannot express how usefull this sentences were for me, allowing me to modified 277 lines of text, in a milisecond!

The first expression parses my sentence and separates it into groups. The first group starts with the word “ADD”, then I have anything up to “Default”, the literal “default”, anything up to “FOR”, “For” itself and then the rest of the sentence (which is a last word). With these, I have all the blocks that I need to build my string.

So the replace expression starts just after ADD (it does not replace the beginning of the sentece) and I ask it to replace it by “ALTER” and the last group – the column name – (\6) then “set default” and the 4th group (\4), the default value.

That’s it 🙂

Replication with PostgreSQL

PostgreSQL does not offer any in built support for asynchronous multi-master replication. According to this official page, some providers implemented this functionality, so I decided to have a go with the only two that advertise lazy multi-master replication: rubyrep and bucardo.
Rubyrep offers two versions: one using ruby and another using ruby and java. It is adverstised as being “really easy to use and database independent”, which are undoubtedly some cool features 🙂 I started with the most simple one (ruby only) and sadly I was unable to run it under linux 😦
I followed the instructions on this page, and in fact the installation is really easy, but when I try to scan the databases for differences, with this command:

rubyrep scan -c myrubyrep.conf

I kept running into this exception (and various warnings):

Exception caught: PG::ConnectionBad: connection is closed: SELECT tablename
FROM pg_tables
WHERE schemaname IN ('"$user"','public')

This happened, even if I reduce the conditions to the simplest: two databases in the same server, and the same user!!

After reading a bit, I realized there are some issues with the versions of Ruby (and other tools) and apparently the project has been inactive for a while (discontinued?). It is a shame because it looks really simple, but I don’t think it is really worth it to put a lot of effort on a software that has been discontinued 😦

Bucardo is perl-based and it is an absolute nightmare to install. I followed the instructions on this page, and as they warn the install script does not work, because it tries to create a “bucardo” user on the fly, but does not ask you to set a password! 😦
After fiddling with it for a while, I found a website that suggests that you edit the script (bucardo_ctl) and edit the default password.
Some things that I would recommend to get bucardo running:

  • set “permissive” authentications for the bucardo on pg_hba.conf
  • add a system bucardo user, and set the credentials on .pgpass

After that, you can add the databases that you want to syhcnronize with:

bucardo_ctl --dbhost=192.168.1.11 --dbpass=password add database test

Note that I was only able to run bucardo_ctl, passing the entire row of arguments (host, password).
Next step is to add the sync; bucardo supports three types of sync algorithms: fullcopy, pushdelta and swap.

bucardo_ctl add sync sync-delta source=test targetdb=test2 type=pushdelta tables=distributors

After that, you can start the bucardo daemon, and wait for the synchronization:

bucardo_ctl --dbhost=192.168.1.11 --dbpass=password start

The only method that supports circular (multi-master) replication is “swap”, and it has some conflict detection/resolution. Sadly for me, it works exactly with two databases 😦

My conclusion about the add-ons for supporting lazy replication in PosgreSQL is that they are both hard to install/configure and Bucardo (which seems a more mature and lively project) is still limited to two masters. I did not attempt more complicated scenarios, such as running databases at different hosts.

Replication using SymmetricDS

SymmetricDS is a a Java-based database synchronization software, supporting asynchronous multi-master database synchronization. If you want to have the GUI (and scripts) you may have to be prepared to spend some $$$ for the Pro version, but if you are happy with the command line, this one is fully functional.
Apart from being cross-platform, this tool has the extraordinary ability of being database-independent, working on a abstraction layer that covers a large number of databases; including: MySQL, PostGreSQL and SQLite (that’s right, SQLite); chances are: it will cover many more in the future.
I decided to try this promising software and I will leave here my impressions.
The first thing I have to say, is that SymmetricDS is a really complete tool, perhaps as complete or more than Microsoft Sync Framework (I did not check properly, because I am not really interested in using Microsoft products). Being so complete, it has to be said it is quite hard to configure, but once you get understand the basics, it is not so bad.
The underlying model is that you have a server and a client for replication. However, far from a simple client-server relationship, the server (root) is actually a group, that may include many nodes, and the client (store) can also be a group with many nodes; you can even have several types of stores, each one with different characteristics (for instance updating only certain tables).
AFAIK, the distinction between server and client is that the server is constantly listening (which requires a sort of a daemon), while the clients are configured to pull and push changes as required (push when they happen, and pull at certain intervals). This makes sense, if you accept that the server should have a more or less reliable internet connection, but it is not so with the clients.
Contrarily to other subscription systems, SymmetricDS actually does support changes on the server, and it does support changes on the schema, so we have some kind of powerful bi-directional synchronization.
Another area where it scores points is in conflict resolution (conflicts are unfortunately inevitable in lazy systems). It does present detection and solution methods for conflicts on a table basis.
Although I did not try the conflict management system, I am quite happy with the framework, after configuring it and running with MySQL databases, running in two different hosts (a slightly more complicated, but more realistic scenario than the one described on the Quick-Start). However, as it happens with MySQL, the scenario of offline-only synchronization still goes without solution.
Things I would like to try next, with SymmetricDS:

  • system with three nodes;
  • system of SQLite databases;

Replication in MySQL

One of the issues that I have in my hands right now, is to implement database replication.

To be more specific, we are talking about multi-master, asynchronous, or lazy replication.

Could it be any more complicated? yes it could, but I will leave that to another article 🙂

After searching and even asking for help in stack overflow, I was left on my own, so I decided to explore the different alternatives/approaches, starting with the popular non-FOSS database: MySQL.

For the people who are not familiar with replication and did not read the links above, what I want to do in a nutshell is to have a system of databases, that are both master and slave from each other; this is also called “circular replication”, as the changes are passed “around” the nodes. Contrarily to a master-slave system, where the modifications only occur on the master, in this case we can have changes occurring in any node of the system.

The fact that the replication is asynchronous, means that we may have nodes disconnected from the network (working offline), and changes will be passed around, as soon as they connect. Asynchronous replication is very convenient, but to use it we must accept that the system will be many times in an inconsistent state as changes are not necessarily propagated immediately to every node (and this is why it is called “lazy”). Another consequence of lazy replication, that is quite “ugly”, is the fact that conflicts may arise (but I will leave that for another post)

MySQL5 supports master-server replication “out of the box”.  The mechanism behind it uses an identifier for each node, and a binary log for the master, where every change is stored. To configure master replication, you need to setup some parameters in the [mysqld] section of my.cnf , on the master side:

server-id               = 1
log_bin                 = /var/log/mysql/mysql-bin.log

You may want to create an SQL user for replication (with appropriated privileges) and then you just need to setup the master settings on the slave side:

mysql> CHANGE MASTER TO
    ->     MASTER_HOST='master_host_name',
    ->     MASTER_USER='replication_user_name',
    ->     MASTER_PASSWORD='replication_password',
    ->     MASTER_LOG_FILE='recorded_log_file_name',
    ->     MASTER_LOG_POS=recorded_log_position;

The “master_log_file” and “master_log_pos” parameters are very important as they tell you at which point of the master you want to start to replicate (current) and the path to the log where all the changes are stored; you can retrieve this values, by typing on the master:

show master status;

after that, you may start the replication listener (on the slave), by typing:

start slave;

The slave will be now listening to every change on the master, and will update accordingly. If you go offline, it will immediately start updating once you go online again.

Of course, if you want to setup circular replication, you will need to make both nodes “master” and “slave” at the same time, by reproducing the configurations above for both. In that case, you will both servers listening to each other.

If you do something like this, you want to be careful about conflicts, which may easily arise when we use auto increment keys. A good solution for that is presented on this article.

This works quite well after fiddling around with some network and permission configurations, to make sure everything is smooth. However, I am not completely satisfied with this solution for two main reasons:

  • Each server listens “blindly” to the other one as master, rather than trying to conciliate the changes with its own changes (what would be a kind of a merge); this may result in an inconsistent system. As an example, I switched the network on both servers, and updated the same row with different values; once they connected, each one grab the changes of the other, which originated two different values for the same row.
  • I am still not sure how to setup circular replication with more than two servers, which would have to fork the “change master” statement to multiple servers.

 

Tweetting in Linux

This may be a bit off topic but still, I think, useful.

Recently I had to dig a little bit into social networks (more specifically Twitter) due to a Greenpeace project.

The Twitter webpage is simple and good, and I’ve been using it for ages. However, if you want to be a little bit more sophisticated, like for instance by having multiple accounts, you may have to go for a desktop application.

Tweetdeck is known to be the best (or one of the best) for this purpose, but unfortunately the applications runs on a proprietary runtime: AIR.

Proprietary apps are usually not a good idea in Linux, the main problem being that the provider (in this case Adobe) may decide to discontinue it; this is the message you get when you try to download it in Adobe’s website:

Adobe AIR for Linux is no longer supported. To access older, unsupported versions, please read the AIR archive.
Different operating system?

First of all, I decided to look for open-source alternatives. I tried two that seemed to have good reviews: Choqok and Gwibber. Unfortunately for me, they were buggy and slow :-/ so I went back to try installing Tweetdeck on Linux.

The first thing you need to know, is that you need to install an old version. The latest runtime provided by Adobe for Linux is the 2.6 (being the latest one 3.7):

http://airdownload.adobe.com/air/lin/download/2.6/AdobeAIRInstaller.bin

After downloading it, you need to change the permissions with chmod:

chmod +x AdobeAIRInstaller.bin

And then you should be able to run it by typing:

./AdobeAIRInstaller.bin

If you don’t run it as root, you will need to provide the root password.

If you encounter an error during installation (I did!), you may want to export your library path to pickup the right libs:

LD_LIBRARY_PATH=/usr/lib/i386-linux-gnu

The next step is to download the Tweetdeck desktop version. Unfortunately they removed the Linux version, so you will have to go for an old one. Fortunately, you can find it here.

After installing the runtime, it should be automatically added to the system, so that you can run applications directly with it; unfortunately, the refresh was not so easy, so I opted for a restart. The next time I started, I was able to right click the Tweetdeck binary and select “install with AIR”; then you just need to follow the screens for install.

That was it! I am happily running Tweetdeck on my Ubuntu now 🙂 I hope this can also help you!

Image