Running BOINC uppercase Demo Standalone

Running the BOINC uppercase demo standalone requires a few things.

• Create a text file called in that will serve as the input file for upper_case.
• Create an empty file named init_data.xml.
  
• Set the LD_LIBRARY_PATH environment variable to point to the current directory.
  

For the first requirement I just copied upper_case.C to in.

$ cp upper_case.C in

Then init_data.xml may be created using touch.

$ touch init_data.xml

Finally, I just set the LD_LIBRARY_PATH variable when I run upper_case.

$ LD_LIBRARY_PATH=. ./upper_case

At this point the upper_case client should run standalone and display a graphics window. The window should display a sphere and cylinder configuration that bounces around as the application runs.

If a logo.jpg file exists in the current directory, it will be displayed as a texture at the back of the area where the "ball" bounces. I just copied the boinc logo file over. Also, if you copy (or link) the Helvetica.txf from the BOINC distribution to the uppercase directory some statistics will be displayed as the application runs.

Compiling BOINC uppercase Sample on Linux

In order to compile uppercase.C with gcc version 4.1.1, I had to change some function signatures to match their definitions, since C99 defines bool. Hopefully blogger doesn't mangle the patches given here. Just in case, there is a patch to correct the function signatures and another patch to fix OpenGL linking.

diff --git a/uppercase/uc_graphics.C b/uppercase/uc_graphics.C
index 0563dfa..4f165b9 100644
--- a/uppercase/uc_graphics.C
+++ b/uppercase/uc_graphics.C
@@ -189,7 +189,7 @@ void app_graphics_reread_prefs(){
  parse_project_prefs(uc_aid.project_preferences);
}

-void boinc_app_mouse_move(int x, int y, int left, int middle, int right) {
+void boinc_app_mouse_move(int x, int y, bool left, bool middle, bool right) {
  if (left) {
      pitch_angle += (y-mouse_y)*.1;
      roll_angle += (x-mouse_x)*.1;
@@ -205,7 +205,7 @@ void boinc_app_mouse_move(int x, int y,
  }
}

-void boinc_app_mouse_button(int x, int y, int which, int is_down) {
+void boinc_app_mouse_button(int x, int y, int which, bool is_down) {
  if (is_down) {
      mouse_down = true;
      mouse_x = x;

Also, I was trying to build uppercase, with graphics enabled. There are some files that uppercase needs to link with that are not in any of the BOINC libraries, so the must be built and linked with directly. I also changed the library handling to not be hard-wired to a given directory:

diff --git a/uppercase/Makefile b/uppercase/Makefile
index 546b748..0d275c0 100644
--- a/uppercase/Makefile
+++ b/uppercase/Makefile
@@ -17,30 +17,31 @@ CXXFLAGS = -g     -L$(BOINC_LIB_DIR)     -L /usr/X11R6/lib     -L.
+CFLAGS = $(CXXFLAGS)

# the following should be freeglut; use nm to check
-LIBGLUT = /usr/local/lib/libglut.a
-LIBGLU = /usr/X11R6/lib/libGLU.a
-LIBJPEG = /usr/lib/libjpeg.a
+LIBGLUT = -lglut
+LIBGLU = -lGLU
+LIBJPEG = -ljpeg

PROGS = upper_case upper_case.so

all: $(PROGS)

clean:
-       rm $(PROGS)
+       rm $(PROGS) upper_case.o uc_graphics.o

# the -Wl,--export-dynamic causes the main program's symbols
# to be exported to the graphics library

upper_case: upper_case.o $(BOINC_API_DIR)/libboinc_api.a $(BOINC_API_DIR)/libboinc_graphics_lib.a $(BOINC_LIB_DIR)/libboinc.a
-       g++ $(CXXFLAGS) -Wl,--export-dynamic -o upper_case upper_case.o libstdc++ -pthread -lboinc_api -lboinc -lboinc_graphics_lib -ldl
+       g++ $(CXXFLAGS) -Wl,--export-dynamic -o upper_case upper_case.o -lstdc++ -pthread -lboinc_api -lboinc -lboinc_graphics_lib -ldl

-upper_case.so: uc_graphics.o $(BOINC_LIB_DIR)/libboinc.a $(BOINC_API_DIR)/libboinc_graphics_impl.a
+upper_case.so: uc_graphics.o $(BOINC_API_DIR)/txf_util.o $(BOINC_API_DIR)/texfont.o $(BOINC_LIB_DIR)/libboinc.a $(BOINC_API_DIR)/libboinc_graphics_impl.a
     g++ $(CXXFLAGS) -o upper_case.so     -shared -fPIC -pthread -    uc_graphics.o -    libstdc++ +    uc_graphics.o $(BOINC_API_DIR)/txf_util.o $(BOINC_API_DIR)/texfont.o +    -lstdc++     -lboinc_graphics_impl -lboinc     $(LIBGLUT) $(LIBGLU) $(LIBJPEG)     -lGL -lX11 -lXmu -lm 

Shared Memory Errors in Standalone BOINC Applications

Seeing an "error" message in stderr.out along the lines of the following

Can't set up shared mem: -1

is not unusual when running a BOINC client standalone. As a matter of fact, it is not an error, it is the expected behavior.

This occurs because the client is attempting to communicate with the BOINC manager, but is not being run under the manager. So, don't panic.

Standalone BOINC Application dlopen Failure

When running a BOINC application standalone, you may see an error along the lines of

dlopen() failed: upper_case.so: cannot open shared object file: No such file or directory No graphics.

This happens when the application tries to load its graphics library. The dynamic loader does not know to look in the local directory. Running the standalone application this way:

$ LD_LIBRARY_PATH=.:$(LD_LIBRARY_PATH) ./upper_case

will solve the problem, by telling the dynamic loader to look in the current directory.

Cannot Restore Segment Prot after Reloc Error

When running a BOINC application on Linux you may find the following in stderr.txt:

Error: cannot restore segment prot after reloc: Permission denied

ITTVIS.com pointed me in the right direction on this one. It is an interaction with SELinux.

The most secure solution is to run a command like:

$ chcon -t texrel_shlib_t upper_case.so

This allows the BOINC application's graphics library to be relocated in memory, which should be safe, but I make no warrantees of any sort with respect to this. This command may need to be run again each time the graphics library is relinked.

The final filename given above will vary depending upon the name of your application, but this example works for the uppercase BOINC sample.

There are other solutions, but they involve either turning off this protection system-wide or turning off SELinux entirely, which I do not recommend.

SIGSEGV Running Standalone BOINC Application

Here is part of an email I sent to the boinc-projects mailing list:

I am beginning work on a BOINC project and have built and installed BOINC stable. It seems to work fine, attaching to projects and doing work. But, when I try to compile and run a BOINC app standalone I invariably get a seg. fault in diagnostics_init. This has happened both with Eric Myer's 'hello' program and with 'uppercase'. Running uppercase produces a stderr.txt with the following contents:

SIGSEGV: segmentation violationStack trace (9 frames):
./upper_case[0x8051224]
[0x462420]
/lib/libc.so.6(index+0x63)[0x4be0ac33]
./upper_case(_ZN7MIOFILE5fgetsEPci+0x61)[0x8053ba1]
./upper_case(diagnostics_init+0x27a)[0x805169a]
./upper_case(boinc_init_diagnostics+0x34)[0x8051934]
./upper_case(main+0x2d)[0x804cd45]
/lib/libc.so.6(__libc_start_main+0xdc)[0x4bdb4f2c]
./upper_case(__gxx_personality_v0+0xdd)[0x804cc41]

Exiting...

...
Basically buf in MIOFILE is NULL, because because diagnostics_init has done an fopen and passed the results to MIOFILE::init_file. This sets f but not buf.

The response I received from Nicolas Alvarez was:

I'm not sure why this happens in standalone mode, but try creating an
empty file 'init_data.xml' and running app again.

This seemed to get me past this first problem.

Improving compile speeds on Linux

I rediscovered an old friend today, ccache. It is a compiler cache written by Andrew Tridgell of SAMBA fame.

It enhances (re-)compilation by maintaining a cache of previously built object files. This comes in very handy if you ever do:

make clean; make

Or as I do, switch between development lines or versions of a program to perform a test and then have to do a full recompile each time.

After installing it on Fedora:

# yum install ccache

I did:

# ln -s $(which ccache) /usr/local/bin/gcc
# ln -s $(which ccache) /usr/local/bin/g++
# ln -s $(which ccache) /usr/local/bin/cc

so that it would automatically be invoked in place of the non-caching compiler.

Configuring BOINC for Linux from CVS

This is not very well documented. After trying many permutations of autoconf, autoreconf, etc. I found Building BOINC on Unix, which mentions using _autosetup. This performs the proper sequence of auto... operations to allow configure to run.

I was able to build BOINC (client and server) from the latest development version in the BOINC CVS repository using the following command:

$ ./_autosetup && ./configure --disable-unicode && make

This is step one. From what I know of our requirements a UNICODE build will be necessary.

UPDATE: While I do not know, as yet, if UNICODE is required. I do know that clientgui will not build on Fedora Core 5 without it. This appears to be because gtk2-unicode-release-2.6 is the only wxWidgets configuration that is available as reported by:

$ wx-config --list

Default config is gtk2-unicode-release-2.6

Default config will be used for output

UPDATE: Installing the packages shown below allows a UNICODE build to be performed. The mysql-server package only needs to be installed if the test/test_sanity.py script is going to be run, or if the BOINC server will be run from this machine. That applies to the chkconfig and service commands as well.

# yum install mysql-devel freeglut-devel wxGTK-devel MySQL-python mysql-server
# /sbin/chkconfig --level 2345 mysqld on
# /sbin/service mysqld start

This must be done logged in as root, the superuser account, or using the su or sudo commands. Once these are installed BOINC may be built using:

$ ./_autosetup
$ ./configure
$ make

NOTE: The blogger post editor seems to have a bug, it keeps wanting to insert "amp;" after the double-ampersands above. The only thing that should appear between each command is two ampersands.

This will leave a couple files in the sea subdirectory. The names will be like boinc_5.4.11_i686-pc-linux-gnu.sh and boinc_5.4.11_i686-pc-linux-gnu_debug.sh, although the numbers may vary. Running one of these will create a BOINC subdirectory that contains the BOINC software and gives instructions on starting it. The files in this directory may be used for testing a new version of BOINC and its libraries, without loading it over the production version that should be installed in /usr/local. The first program installs a non-debugging version, while the second installs a debugging version.

If, while running configure, messages appear indicating that GL, GLU, GLUT, MySQL or wxWidgets are not installed, but they are, loading ccache may correct the problem.

If you are going to build a BOINC project on this machine, the libraries and header files will need to be installed:

# make install

This will place the necessary files under the /usr/local hierarchy.

BOINC for intra-organizational computing

Just a snippet from ACM Queue - A Conversation with David Anderson - The director of SETI@home discusses his work and the volunteer computing movement:

"BOINC works fine for intra-organizational computing also. CERN, for example, is experimenting with BOINC for a data-intensive application on its internal PCs."

Perhaps this is something to consider. Although, I suppose the good reasons for doing so would have to be related to security issues.

Scrolling wide pre-formatted text

There are places on this blog, where I have had to post overly wide pre-formatted text. Normally these are code, command-line or file snippets. By default in the theme/template I am using this causes the text to run into the sidebar.

To correct this, I added the following CSS code to the blogger template just above the footer formatting information.

pre {
overflow: auto;
line-height: 1;
margin: 1em;
padding: 1em;
background: #cdf;
color: #330;
border: 1px solid #fff; }

The idea for this came from pre-formated text with scroll bars at 100% width - HTML. It may not be perfect and well-styled, which is my fault, but it works.

Adding repositories to Fedora 5

While testing Fedora Core 5, FC5, on coLinux I added some repositories in search of TightVNC and other software. I added:

• freshrpms
• dries
  

If we switch to Red Hat Enterprise Linux, RHEL, I will add Dag Wieers' RPMforge.net repository. It is one of the few that supports RHEL releases. However, he does not have support for FC5. So, I added Dries' repository, since they interoperate.

Freshrpms
To add freshrpms, download the configuration RPM:

wget http://ftp.freshrpms.net/pub/freshrpms/fedora/linux/5/freshrpms-release/freshrpms-release-1.1-1.fc.noarch.rpm

and install it using

rpm -Uhv freshrpms-release-1.1-1.fc.noarch.rpm

Dries
To add the Dries repository create an /etc/yum.repos.d/dries.repo file that contains

[dries]
name=Extra Fedora rpms dries - $releasever - $basearch
baseurl=http://ftp.belnet.be/packages/dries.ulyssis.org/fedora/linux/$releasever/$basearch/dries/RPMS/

Then import the GPG key for the repository:

wget http://dries.ulyssis.org/rpm/RPM-GPG-KEY.dries.txt
rpm --import RPM-GPG-KEY.dries.txt

The next time you access yum, it should update to include these repositories.

NB: There is some indication that the Extras (fedora.us) repository may need to be disabled.

Fixing coLinux Fedora's passwd program

When I loaded the Fedora 5 image for coLinux, I got an error running passwd to reset the password for root. It looked like I needed to reinstall cracklab-dicts.i386, but I didn't know how to force this. I could not uninstall it, that would uninstall all most all of Fedora, which I discovered the hard way.

Someone had a similar need, only for their kernel, as reported in the e-mail [Yum] Re: yum install -force package. The solution was to run:

rpm -e --justdb --nodeps kernel-2.4.22-2g

then,

yum update kernel

would "do the right thing" and reinstall the package. Well, I had to do something slightly different:

rpm -e --justdb --nodeps cracklib-dicts.i386
yum install cracklib-dicts.i386

If that does not work try using:

rpm -e --nodeps cracklib-dicts.i386

instead. I did both and I'm not sure which actually permitted yum to do its thing.

Dvorak on Fedora Core Linux

Typing

$ loadkeys dvorak

is enough to load the dvorak keyboard layout on Fedora. But, to load it at boot time, edit the /etc/sysconfig/keyboard file and change the KEYTABLE=... line to KEYTABLE="dvorak".

Building BOINC Applications on Windows

Eric Myers has an an excellent walk-through of setting up, configuring and Building BOINC applications on Windows.

JPEG Library

Eric's solution and project files should be automatically converted to Visual Studio 2005 format upon loading.

If you want to build a release version of the JPEG library, you will need to change a couple of the project's properties:

First, turn off the precompiled headers for the Release configuration.

Then change the Configuration Type to Static Library (.lib).

After building the JPEG library Eric suggests copying the headers and library file somewhere. So far I have not done that. I am trying to set up a single Visual Studio Solution file for BOINC development that uses dependencies and relative paths for everything.

OpenGL and GLUT

Following Eric's directions, I got the freeglut-2.4.0 distribution and decompressed it in a directory alongside my jpeg-6b directory. Then I opened the Visual Studio 6 workspace that comes with the distribution and allowed Visual Studio 2005 to automatically upgrade it. Once the Workspace has been opened as a new Solution it is set freeglut_static to be the StartUp Project (love those StudlyCaps):

Other than selecting the static library, freeglut seems to build out of the box.

wxWidgets

The wxWidgets library is not used for BOINC application development. It is only needed to build the BOINC Manager, which can be downloaded in binary form.

Making a Large Sparse coLinux Disk Image

The directions given for ExpandingRoot - coLinux are pretty good. Instead of fsutil I used:

dd if=/dev/zero of=colinux_new bs=1 seek=10737418240 count=0

in Cygwin to create a spase cygwin_new file. Then I started coLinux, logged in as root and zero'd out most of the freespace in cygwin_old so that mkSparse could reduce the size of cygwin_new

later:

# e2fsck /dev/cobd4
# df
# dd if=/dev/zero of=/tmp/fill count=...
# rm /tmp/fill

This zero'd out most of the free space in cygwin_old. Then I did what ExpandingRoot - coLinux said to do:

# dd if=/dev/cobd4 of=/dev/cobd5
# resize2fs -p /dev/cobd5
# e2fsck /dev/cobd5
# shutdown now -h

After downloading mkSparse from the coLinux file-utils described on the NiceTools page of the coLinux Wiki, I ran:

C:\coLinux> mkSparse colinux_new

This turns colinux_new into a sparse file that takes up much less room on the drive. After this is done I finished up by continuing at step 7 on the ExpandingRoot page.

Spy Hill Research BOINC Developers' Notes

Spy Hill Research has a excellent set of BOINC Developers' Notes. They are written by Eric Myer, who was part of the Einstein@Home development team. His notes should help get a developer rapidly up to speed on developing for BOINC.

There are instructions for:

• Building BOINC applications on Windows
• Building BOINC and BOINC Applications on Linux
• Building BOINC and BOINC applications on MacOS X

Much of the notes is in the form of examples to be worked through:

• Hello, World -- the simplest BOINC program
• concat - concatenates two or more input files into one output file (from BOINC)
• Yello, World -- the simplest graphics program for BOINC,
• cube - simplest non-trivial 3D graphics application for BOINC
• uppercase - convert an input file to uppercase (from BOINC) - now with graphics
• The Jack LaLanne program - an API Exerciser
• The 'scroll' application for Pirates@Home
• The "Sicilian Roulette" application for BOINC
• Sextant - the Einstein@Home screensaver graphics thread
• Starboard! - xscreensaver GL graphics suite

These, sort of, form a self-guided tutorial. There are also links related to BOINC Graphics, Project Management, and external developer links.

BOINC Server Planning

The MyISAM engine requires the least amount of computer resources can be used where there is a low DB activity requirement. For example with query rates lower that 5/sec this table type may be adequate. Also if one does not have a dedicated DB server this may be a good choice for all the tables since it consumes much less computer resources. It has the advantages of allowing long text indices against tables which Innodb does not allow.

...

Innodb tables/indices are usually stored in large OS physical files and the tables and indices are managed internally within these OS/Innodb files. It is important that these files are located on high performance devices. The transaction log files should be located on independent high performance media (away from the Innodb files) for sustained high transaction rates. At DB shutdown all modified buffers have to be flushed into the transaction logs before MySQL goes away, so slow performance drives for the transaction log could delay shutdown for over 30 minutes when there are a large number of .modified buffers. to be flushed.

Configuring MySQL for BOINC

While the page quoted above is called Configuring MySQL for BOINC, the title should have something about server capacity planning. There is much information on the physical requirements of the server hardware given the loads placed upon it by SETI@home. These loads are also documented. So the planner should be able to extrapolate something about the hardware that should be needed. The SETI@home Server Status page also lists the various servers used, as well as, their software and hardware configurations.

In contrast to SETI@home, Einstein@Home uses a single server, of unknown configuration. There is an Einstein@Home Server Status page as well. The EAH status page also makes reference to four download mirror sites. This may account for the project being able to use a single server, if the data driving the project is distributed across five different sites (including the main site).

BOINC server bottlenecks

MySQL can be the bottleneck in a BOINC server. To optimize its performance, read about configuring MySQL for BOINC.

Software prerequisites

Pertinent BOINC Server Software Prerequisities

Some parts of the BOINC server (the feeder and scheduling server) use shared memory. On hosts where these run, the operating system must have shared memory enabled, with a maximum segment size of at least 32 MB. How to do this depends on the operating system; some information is [given in the PostgreSQL documentation under Managing Kernel Resources].

This is according to the BOINC Software prerequisites page. Basically it entails setting the SHMMAX parameter.

If the server is going to be run on Windows using Cygwin then the Setting SHMMAX in Cygwin mailing list conversation is pertinent. At this point in time it has not been determined whether the server will be run on Windows (with or without Cygwin), Linux, UNIX or some other platform.

Anyone responsible for maintaining or managing a BOINC server should read the Software prerequisites page. It contains information on setting up the server, backup procedures and other important tasks.

BOINC Suitability

As a rule of thumb, if your application produces or consumes more than a gigabyte of data per day of CPU time, then it may be cheaper to use in-house cluster computing rather than volunteer computing.

Which applications are suitable for BOINC?

BOINC

SciLINC will use the BOINC framework to distribute the work to volunteer's machines. A very good resource describing BOINC can be found on the Papers related to BOINC web page. Especially:

BOINC: A System for Public-Resource Computing and Storage. David P. Anderson.5th IEEE/ACM International Workshop on Grid Computing, November 8, 2004, Pittsburgh, USA.

BOINC implements Public-Resource Computing. This differs from Grid-Computing in terms of security and available network bandwidth. To make efficient use of PRC the compute time of the client must be relatively large compared to the data-transfer time required.

A research scientist can create and operate a large PRC project with about a week of initial time investment and approximately an hour per week of maintenance.

"On a particular computer, the CPU might work for one project while the network is transferring files for another."

It seems to me that this is where a large bandwidth/computation project might still succeed. If the CPU is working for a computationally intensive project and the network pipe is transfering data for the data-intensive application. This would work best given a large D/L to U/L ratio, given that most consumer internet providers offer a much large pipe downstream than upstream.

(The typical server platform is LAMP with Python.)

SciLINC - Scientific Literature Indexing on Networked Computers

This blog is all about the development of SciLINC - Scientific Literature Indexing on Networked Computers the Missouri Botanical Garden's public-resource computing application that will automatically index large amounts of digitized scientific literature.

You will find information on development progress, background information on the project and developer notes. Much of what is here will simply be a collection of information that I want to be able to refer back to as development progresses.