Update: I tested a kernel 2.6.31-12 (from kernel.org) and had the same (if not even worse) behaviour as with 2.6.31-19. So, I apologize. It has nothing to do with Ubuntu per se, but probably with the linux-kernel; that makes me wonder if I haven’t done something faulty: that behaviour should have attracted the attention of some other Haskell developers in the meantime, but I couldn’t find any information on it. So, take in mind that the following could be incorrect…

finding this strange behaviour had cost me a lot of hours (see also my post on the haskell-cafe mailing list), because honestly, who do you blame if your parallel programm behaves strangely:

1. yourself
2. the compiler
3. the special patches of your OS vendor to your linux kernel

Since the probabilities of an error in 2. are quite low (at lot of people use GHC and its parallelization techniques) and — at least I thought that — those in 3. are low, too (some of those people from 1. should be using my OS with GHC for parallel development), I chose 1. Hence, I spent a lot of time trying to find synchronisation and performance problems in my code, which in fact, weren’t there!

The current setup of a standard Ubuntu 9.10 (64 bit) is a vanilla kernel (2.6.31-19-generic) , used with GHC 6.12.1. Look at this small and seemingly innocent program

-- File Pi.hs
-- you need the numbers package from hackage.
module Main where
import Data.Number.CReal
import System.Environment
import GHC.Conc

main = do
    digits <- (read . head) `fmap` getArgs :: IO Int
    showCReal (fromEnum digits) pi `pseq` return ()

that you can compile with

ghc -- make -O2 -threaded Pi.hs -o pi

Benchmarking with

for i in `seq 1 5`;do time pi 5000 +RTS -N1;done
for i in `seq 1 5`;do time pi 5000 +RTS -N2;done

I received the following exemplary runtimes for the vanilla ubuntu kernel.

And this is the threadscope profile of a particular bad run:

As you can see, nearly all time is spend in the Garbage Collector, doing…things, GCs do…

I then compiled my own kernel2.633 from kernel.org and received these results:

Conclusion: At least for me and my parallel programs (and even some innocent toy programs like Pi.hs) does the standard Ubuntu 64bit kernel harm parallel performance (especially that of the GC) a lot.

I find this highly surprising:

• Ubuntu is a widely used distribution
• 64 bit versions are also becoming quite common
• in my perception Haskell developers use parallelism a lot

Question. Either the intersection of these properties is quite small or my measurements have some serious flaw. I’d be glad if someone could disprove my results (and give me the opportunity to learn a bit) or support my measurements.

1. Wait, until it is safe to change the capabilities structure.
2. Allocate memory for number_of_capabilities + 1
3. Copy old capabilities structure to new allocated one
4. Relink to new memory location

To be honest, I don’t think it’s so easy; but at least this is a starting point I can use. Currently, I have to things to think about and which have to be solved before I can hack around in the code:

1. How can I call C code in the RTS from Haskell?
2. When is it safe to modify the capabilities structure?

Happy for any suggestions,
Michael

Extension  Lines
.lhs       300482
.hs        473479
.c         184039
.h         448394

combined  1406394

Found using wc -l `find . -name \*.`

Thankfully I only need a tiny fraction of it to add the dynamic-capabilities-feature: according to the commentary on the runtime system it consist of only about 50000 lines, and again, I probably need to modify / extend just a fraction of it. The code of the runtime system has an extra page on the wiki; it describes

• the purpose of the single files. Unfortunately these are currently only one liners and links to the particular files. Maybe I will add a much more extensive description for the Capabilities-files.
• the coding conventions for this subsystem. They are quite extensive but (in my humble opinion) very good and describe not only syntactical conventions but also some general remarks, debugging and robustness tricks.

For seeing how the capabilites are initialized I think that the file RtsFlag.c is helpful:

• RtsFlag.c defines parsing for -N<…>, hence I see where the capabilites are set and how they fit in the big puzzle of the RTS.
• RtsFlag.c uses and sets RtsFlags.ParFlags.nNodes (line 958). The structure is included in includes/Rts.h which includes include/rts/Flags.h (line 189 defines the important parts).
• RtsMessage.c would probably help me in the future for printing some debugging messages.

I’m curious if I’m on the right track. By the way, take all the informations with a grain of salt, since I’m currently in the beginning of understanding the code of the runtime system.

Minor update: After a look at the commentary I’m pretty sure I need to understand much of the scheduling subsystem to extend the capabilities-system. Thankfully the wiki seems to document it.

For the examplary unoptimized code

-- Compile with ghc-6.10.4 -O2 -threaded --make Example.hs -o example
module Main where
import GHC.Conc
import Control.Concurrent
import Control.Monad
import System.Environment
import Data.List
------------------------------------------------------------
main :: IO ()
main = do
    args <- getArgs
    let threads = numCapabilities    -- threads determined by -N...
        taskDur = 1000000            -- big enough magic number
        taskNum = (read . head) args -- Number of tasks is 1st parameter
 
    queue    <- newChan
    finished <- newChan
    writeList2Chan queue (replicate taskNum taskDur)
    replicateM_ threads (forkIO (thread queue finished))
    replicateM_ taskNum (readChan finished)
------------------------------------------------------------
thread :: Chan Int -&gt; Chan Int -&gt; IO ()
thread queue finished =
    forever $ do
        task <- readChan queue
        workFor task
        writeChan finished 1
  where workFor n = foldl' (\a b -> a + sqrt (2^b)) 1 [1..n] `pseq`
           return ()

we get the following runtimes on my Dualcore Notebook:

$ time e-6.12 +RTS -N1 -RTS 16

real	0m43.994s
user	0m43.630s
sys	0m0.040s
$ time e-6.12 +RTS -N2 -RTS 16

real	0m25.987s
user	0m48.790s
sys	0m0.230s
$ time e-6.10 +RTS -N1 -RTS 16

real	0m45.883s
user	0m44.500s
sys	0m0.400s
$ time e-6.10 +RTS -N2 -RTS 16

real	0m38.930s
user	0m46.040s
sys	0m0.490s

As you can see, e-6.12 (the version compiled with ghc-6.12-20091010) delivers a good speedup whereas ghc-6.10 does not use my two cores very much. This makes testing, benchmarking and developing my parallel Haskell code currently quite difficult.