But we would like to get it a step further. We are currently considering the idea of a French hackathon. We already have been offered a room for 3 days in June 2012, in conjunction with a French Perl event. So one of the reasons behind this blog post is to get some feedback about this, in addition to letting people know about the French community. Who would be interested in attending such an event? This would be in Strasbourg, France for the moment. Note that this would not necessarily be restricted to French haskellers! The format would be a pretty much classical Hackathon: potential talks and a lot of Haskell hacking. So please let us know if you’d be interested in attending this (by commenting this post, the IRC channel or the mailing list).

On a side note, these past few days, I updated one of my libraries: statistics-linreg and published one too: kmeans-vector. It performs the k-means clustering algorithm on a list of points, and can lead to “pretty” graphics like the following.

Performing k-means on 10,000 points with 5 clusters

Getting, patching and building LLVM

It is as simple as executing the following commands:

$ svn co http://llvm.org/svn/llvm-project/llvm/trunk llvm
$ cd llvm
$ patch -p0 -i ~/llvm-ghc.patch
$ ./configure –enable-optimized # probably also want to set –prefix
$ make
$ make install

where the given llvm-ghc.patch file can be downloaded here : http://www.cse.unsw.edu.au/~davidt/downloads/llvm-ghc.patch.

Getting GHC HEAD and Patching it to add the LLVM backend

First, get the latest archive matching that pattern : ghc-HEAD-AAAA-MM-JJ-ghc-corelibs-testsuite.tar.{gz or bz2} at http://darcs.haskell.org/, the latest currently being ghc-HEAD-2009-10-23-ghc-corelibs-testsuite.tar.gz. It ships GHC with all its dependencies, also including a testsuite.

Put the archive file in (on my computer) /home/alp/haskell/. Uncompress it with for example:

$ tar xzf ghc-HEAD-2009-10-23-ghc-corelibs-testsuite.tar.gz

You should get a /home/alp/haskell/ghc/ directory. Now we have to fetch the latest patches applied to GHC since the archive you downloaded got published on darcs.haskell.org. To do this, simply do:

$ chmod +x ./darcs-all
$ ./darcs-all pull -a

(the former may not be necessary though)
It can take some time, there may be a lot of patches to fetch and apply, don’t worry.

If you are not interested in having the LLVM backend, you can stop reading that section and go straight to the next one.

Once done with that, you have to download the patch to add the support to the LLVM backend to GHC. Just download the following file : http://www.cse.unséw.edu.au/~davidt/downloads/ghc-llvmbackend-full.gz (it actually just is a darcs patch, not an archive or anything else — you may want to rename it with a .patch extension), put it in ghc’s directory (/home/alp/haskell/ghc/ here) and apply the patch :

$ darcs apply ghc-llvmbackend-full.patch

(or .gz)

You just have one more patch to apply (isn’t life about applying patches?), written by Don Stewart, to be able to pass LLVM options to GHC without conflicting with other options. You can get it at http://www.galois.com/~dons/add-new-llvm-code-generator-to-ghc_.dpatch. Just move it to your ghc directory and darcs apply it, like you just did for the previous patch. (note: this step might not be necessary if the previous patch or the ghc repository contain it)

Ok, we are almost done! Now create a file named “build.mk” in (on my computer) /home/alp/haskell/ghc/mk/, and put the following content in that file:

GhcWithLlvmCodeGen = YES
GhcEnableTablesNextToCode = NO

The first, obviously, enables LLVM code generation in GHC. The second disables a feature currently being in conflict between the LLVM and the C code generators, if I remember correctly. Now, let’s build GHC, hooray!

Building and using the patched GHC non-intrusively

Once all the patches are fetched and applied, you just have to do:

$ sh boot
$ ./configure
$ make

Now, have a cup of coffee, read newspapers, try to solve P vs NP, whatever. It takes some time.

Once done with ‘make’, there should be many binaries in (on my computer) /home/alp/haskell/ghc/inplace/bin/. The ghc binary is called ghc-stage2

To check that the LLVM backend is enabled, as explained on Don’s post, just do the following (in the your ghc/inplace/bin/ directory):

$ ghc-stage2 –info

and verify that it gives you (“Have llvm code generator”,”YES”).

To make use of them properly without installing your brand new GHC HEAD system-wide, here is one possibility; add the following to your ~/.bashrc (or whatever you use, it’s just about defining variables):

GHC613 = /home/alp/haskell/ghc/inplace/bin
GHC613BIN=$GHC613/ghc-stage2
GHC613PKG=$GHC613/ghc-pkg

And now, when you will want to install libraries via cabal for your new GHC, just execute commands close to the following (to install say the vector package):

$ cabal install vector –with-compiler=$GHC613BIN –with-hc-pkg=$GHC613PKG

And it will build and install the vector package using GHC HEAD and its library directories (in ~/.ghc/).
Moreover, if you want to build packages with the llvm backend, just ask cabal gently:

$ cabal install vector –with-compiler=$GHC613BIN –with-hc-pkg=$GHC613PKG –ghc-options=-fllvm

You can even chain options this way, passing for example -optlo-03. For more informations on that, check Don’s posts:

• Smoking fast Haskell code using GHC’s new LLVM codegen
• Evolving Faster Haskell Programs (now with LLVM!)

But be careful, -fvia-C and -fllvm aren’t much friends, you may sometimes have to either edit the .cabal file of the package you want to install and removing -fvia-C from there, or via the –flags option of cabal install. For more informations, cabal help install.

Here are few links that you may find to be of interest:

• Home of the LLVM backend section of GHC’s trac wiki
• The Building and Porting section of GHC’s trac wiki
• David Terei’s thesis about the LLVM backend for GHC [PDF] — thank you David !

I hope this post will be of help and that it will lead some of you to contribute to GHC !

Firstly, the code is no longer evolving at github. There are at least 2 other persons working on HNN for the moment (Thomas Bereknyei and Jorden Mauro — thank you guys for joining the project) thus we decided it was high time having a bit more than a little github place. So we now have a darcs repository, a mailing list and a trac wiki / bug report system.

As said on the trac home page, we are rewriting HNN from scratch to make it more generic and efficient. We will of course still make the API as simple as possible, fitting one of the most important goals of the original version of HNN.

Stay tuned !

Pour ceux qui ont assisté au talk sur la présentation fonctionnelle avec illustration en Haskell au Provence Linux User Group et qui voudraient revivre ce grand moment, je mets donc à disposition les slides ainsi que le code du bot IRC plugfrbot. Si j’ai le temps, un jour j’en ferai un superbot, avec protection contre toutes les attaques possibles… Donc probablement jamais.

Slides : Programmation Fonctionnelle (Présentation – PDF)

Code : Programmation Fonctionnelle – Bot plugfrbot (Code Haskell – .hs)

Peut-être à bientôt pour une présentation en lien avec la PF.

Few weeks ago, I released a first (experimental) version of my Haskell Neural Network library.

And since then, I have been thinking about the design of this library (and got some feedbacks from haskellers) : how to improve it, how to let people easily plug their stuffs in it, etc. In this post I’ll try to share my thoughts about how HNN can be improved (without anything neural network specific). I hope it will somehow be useful to someone.

Typeclasses — “Is it really worth creating a typeclass for … ?”

Let’s first replace this in its context. I like very much the typeclass feature of Haskell, I already used it for many things. But hey, HNN is a project for the Haskell community, I just can’t throw some typeclasses in it without actually thinking about why it would be good, and why it wouldn’t. Wouldn’t it be overkill and thus make haskell code using HNN less nice ?

In my case, it is about the list-of-value storage. I started to store the weights and the inputs/outputs with lists of double. Pretty inefficient yeah, but I was rather prototyping than seriously writing the library. Then I moved to the uvector package, on Hackage. Way more efficient. But, you may wonder the following : “what would he do if we have an even nicer and more efficient container ?” — and you’re right. For the moment, intern stuffs are stored in an UArr and I provide both a list-based interface and an UArr-based one. But, hell, it’s already annoying to maintain both of these ! But I couldn’t at that moment look at my code with enough hindsight to get the good and the bad and actually operate the necessary changes to make it for the release of the 0.1 version.

A possible plan for the next version is to have a typeclass representing Storage related functions, and instanciate it for UArr and List. Then I’d abstract the functions that I have to call differently for List, UArr and all in the Storage typeclass and thus would only require an instance of Storage to be given to the functions I need. This way, if you have some container in some other part of your project, or if you just want to have a more performant / specific container store your values, it’ll be possible, provided you will give the necessary functions at the typeclass instanciation.

Now that I shared my reasonning path, yeah, obviously, there will be a typeclass for value storing. I even have some ideas about other uses for typeclasses, in particular one being suggested by Thomas Bereknyei : it would represent a computation. Why would it be interesting ? So that we could have complex computation units within neural networks, or interacting with neural networks. I’m particularly thinking of Kohonen Maps or recurrent neural networks.

About “exposed function granularity”

Maybe you already wondered whether you should expose few functions covering most of the tasks you library accomplishes or if you should rather provide many functions for little tasks. First, I don’t think there’s an universal answer to that question. It highly depends on the “combinatorial” capacity of the underlying concepts. For example, parsec provides many little functions because using parsec consists in combining these little functions to create more complex parsers. With neural networks, we don’t have that much of a combinatorial power. I mean, you create your neural network, apply training functions on it, and then use it. And the library does not have (yet) enough maturity to provide features like neural network combinations or so. So what’s the best for HNN ?

For the moment, we only have few functions doing almost all the job, giving them parameters to have the necessary flexibility to train the neural network with the smoothness we want, etc. I think it might be interesting to provide something similar to data structure folding for the neural networks, or even consider offering a graph-oriented interface (after all, that’s what neural networks are) and provide graph traversal functions to operate on the neural network. The key point here is to – whatever way I choose – provide higher order functions to operate on the neural networks, in addition to the simple and straighforward functions we can find in the library for the moment.

Conclusions

I will try to introduce some typeclasses to offer more flexibility to the future users, the number one priority being for the value storage. In addition to that, I have to provide users functions offering them to do nearly anything they want on the neural network, like we have folds for lists, trees and all. If these both goals are satisfied in a close future, HNN should definitely be more interesting for serious use. Another point I have to consider is to offer even better performances since neural networks “just” compute stuffs. UArr helped a lot for that, but I’m pretty sure I can introduce other improvements on that aspect of HNN. I just need to read again some pages of the GHC manual

I hope these thoughts will be useful to someone. At least, they have been to me.

I just released the 0.1 version of my Haskell Neural Network library on Hackage.
Instead of writing a long blog post, I created a page on the Haskell wiki that you can find here : HNN describing what is HNN, how to get it, showing a sample and all.

There is an online version of the documentation here : hnn documentation
You can also consult hnn’s hackage page : hnn at hackage (the documentation should be generated soon there)

Here is a sample showing how you can use HNN :

module Main where

import AI.HNN.Net
import AI.HNN.Layer
import AI.HNN.Neuron
import Data.Array.Vector
import Control.Arrow
import Data.List

alpha = 0.8 :: Double — learning ratio
epsilon = 0.001 :: Double — desired maximal bound for the quad error

layer1, layer2 :: [Neuron]

layer1 = createSigmoidLayer 4 0.5 [0.5, 0.5, 0.5] — the hidden layer

layer2 = createSigmoidLayer 1 0.5 [0.5, 0.4, 0.6, 0.3] — the output layer

net = [layer1, layer2] — the neural network

finalnet = train alpha epsilon net [([1, 1, 1],[0]), ([1, 0, 1],[1]), ([1, 1, 0],[1]), ([1, 0, 0],[0])] — the trained neural network

good111 = computeNet finalnet [1, 1, 1]
good101 = computeNet finalnet [1, 0, 1]
good110 = computeNet finalnet [1, 1, 0]
good100 = computeNet finalnet [1, 0, 0]

main = do
putStrLn $ "Final neural network : \n" ++ show finalnet
putStrLn " —- "
putStrLn $ "Output for [1, 1, 1] (~ 0): " ++ show good111
putStrLn $ "Output for [1, 0, 1] (~ 1): " ++ show good101
putStrLn $ "Output for [1, 1, 0] (~ 1): " ++ show good110
putStrLn $ "Output for [1, 0, 0] (~ 0): " ++ show good100

Output :

$ ./xor-3inputs
Final neural network :
[[Threshold : 0.5
Weights : toU [1.30887603787326,1.7689534867644316,2.2908214981696453],Threshold : 0.5
Weights : toU [-2.4792430791673947,4.6447786039112655,-4.932860802255383],Threshold : 0.5
Weights : toU [2.613377735822592,6.793687725768354,-5.324081206358496],Threshold : 0.5
Weights : toU [-2.5134194114492585,4.730152273922408,-5.021321916827272]],[Threshold : 0.5
Weights : toU [4.525235803191061,4.994126671590998,-8.2102354168462,5.147655509585701]]]
—-
Output for [1, 1, 1] (~ 0): [2.5784449476436315e-2]
Output for [1, 0, 1] (~ 1): [0.9711209812630944]
Output for [1, 1, 0] (~ 1): [0.9830499812666017]
Output for [1, 0, 0] (~ 0): [1.4605247804272069e-2]

Don’t hesitate to try it, play with it and give some feedback ! For any feedback or question, see the end of the HNN wiki page.

Thanks, and enjoy !

Few months ago, I started working on a neural network library, in Haskell. The result wasn’t that bad, but needed some additional work. Past days, I’ve worked a bit on that code again to get a releasable and usable 0.1 version of HNN up and working. For example, the weights, inputs and outputs are of type UArr Double now (they were of type [Double] before).

You can find the source code there : http://github.com/alpmestan/HNN.
Also, I’ve built a minimal documentation. You can find it here : http://mestan.fr/haskell/hnn/.

To get the current code and test it :

$ git clone git://github.com/alpmestan/HNN.git
$ cd HNN
$ cabal configure
$ cabal build
$ cabal install (to add it in your ghc package list, etc)
$ cabal haddock (to generate the documentation)

I plan to put this on Hackage soon, but I would like to get some feedback & reviews about it before.

Thank you !

Have you ever heard about typelists in C++ ? That just consists in using the functional way of defining lists, but with templates.
It looks like that :

template <typename Head, typename Tail>
struct TypeList
{
  typedef Head head;
  typedef Tail tail;
};

However, we’ll need a type representing an empty type list. Ours will be the following.

struct EmptyList { };

How to write metafunctions (compile-time functions, working over types and not values — actually, the types are in this context like “values”) for these type lists now ?

Let’s start with a metafunction computing the length of a type list :

// declaration
template <typename Typelist>
struct Length;

// the normal case : the head element of the list (it's a type), and the tail, which is itself a type list
template <typename H, typename T>
struct Length<TypeList<H, T> >
{
  static const int value = 1 + Length<T>::value ;
};

// the terminal case : our typelist is the empty list, we're at the end of the list, so we won't add 1 neither we will go on with the recursion
template <>
struct Length<EmptyList>
{
  static const int value = 0 ;
};

Now, calling it on a given typelist will return us the good result :

Length< TypeList<int, TypeList<char, TypeList<bool, EmptyList> > > >::value // equals 3

Do you want more ? I guess you do, of course. Let’s tackle a more complicated one : Map. It maps a type list to another one, computing the result of the application of a metafunction on each type of the type list. Ok, let’s start with the declaration.

template <typename TL, template <typename> class Func>
struct Map;

Now, the basical case, with a head element and the tail, will consist in computing the result of the application of the metafunction of the head element, and appending to it the result of Map on the tail. Looks like that :

template <typename H, typename T, template <typename> class Func>
struct Map<TypeList<H, T>, Func>
{
  typedef TypeList< typename Func<H>::type, typename Map<T, Func>::type > type;
};

And the trivial case, on the empty list :

template <template <typename> class Func>
struct Map<EmptyList, Func>
{
  typedef EmptyList type;
};

And we’re done with Map !

Let’s see one more interesting function over type lists : Filter. It’ll filter (really ?!) the type list according to a compile-time predicate, and return the original type list without the types that didn’t match the predicate.
Here we go !

template <typename TypeList, template <typename> class Pred>
struct Filter;

template <typename H, typename T, template <typename> class Pred, bool result>
struct FilterAux
{
  typedef typename Filter<T, Pred>::type type;
};

template <typename H, typename T, template <typename> class Pred>
struct FilterAux<H, T, Pred, true>
{
  typedef TypeList<H, typename Filter<T, Pred>::type> type;
};

template <typename H, typename T, template <typename> class Pred>
struct Filter<TypeList<H, T>, Pred>
{
  typedef typename FilterAux<H, T, Pred, Pred<H>::value>::type type;
};

template <template <typename> class Pred>
struct Filter<EmptyList, Pred>
{
  typedef EmptyList type;
};

This one was trickier, because we needed an auxiliary template structure to have a bool against which we could specialize, to either go on without keeping the type in the type list (in case it doesn’t match the predicate) or keeping it.

Now, I’ll leave as exercise the following functions :
- Repeat : takes a type, a number, and returns a type list containing n times the given type
- Take : takes a type list and a number, and returns a type list contaning the first n elements of the typelist if possible, less otherwise.
- Interleave : it takes 2 two lists, say l1 = [T1, T2, T3] and l2 = [U1, U2, U3] and returns the list [T1, U1, T2, U2, T3, U3]
- Zip : takes two lists and returns a list of component-wise pairs of the types
- ZipWith : takes two lists and a function itself taking two types and returning a type, and returns a list compound of the component-wise application of the given function on both lists simultaneously

I’ll try to post these during the upcoming days.

Good functional metaprogramming to all

a = m1 + m2 + m3

then the computation will look like the following tree :

Evaluation tree of m1 + m2 + m3

Thus, there will be a first for loop to compute m1 + m2 which will result in a matrix we’ll call t, then another one to compute t + m3. For 3×3 matrices, it’s ok. Imagine n is actually 40000. Doing two for loops of 40000 iterations where we would do a single one… quite annoying, isn’t it ? Actually, we would rather want an evaluation tree like the following :

The desired evaluation tree of m1 + m2 + m3

Here comes expression templates

It’ll need a bit of refactoring though. First, let’s say our matrix type is the following (we’ll only deal with float to avoid ambedding an additional typename template parameter representing the number type we use) :

template <unsigned int N>
class matrix
{
    float data[N*N];

public:
    float operator()(unsigned int row, unsigned int col) const
    {
        return data[row + N*col];
    }

    float& operator()(unsigned int row, unsigned int col)
    {
        return data[row + N*col];
    }
};

template <unsigned int N>
std::ostream & operator<<(std::ostream& o, const matrix<N>& m)
{
    o << "[";
    for(unsigned int row = 0; row < N; row++)
    {
        for(unsigned int col = 0; col < N; col++)
        {
            o << m(row,col);
            if(col != N-1) o << ";";
        }
        if(row != N-1) o << "\n";
    }
    o << "]";
    return o;
}

Now, we’ll have to define a Domain Specific Embedded Language for matrix operations. Like in any language people design, we will have a tree representing what’s happening in the code. In our case, it’ll represent the evaluation tree of the matrix operations we’re dealing with. Thus, like in any expression tree, we need an Expression type. Ours will look like this :

template <typename LeftOperandType, typename OperationTag, typename RightOperandType>
struct Expression
{
    const LeftOperandType& l;
    const RightOperandType& r;

    Expression(const LeftOperandType& l_, const RightOperandType& r_) : l(l_), r(r_) { }

    float operator() (unsigned int row, unsigned int col) const
    {
        return OperationTag::apply(l(row, col), r(row,col));
    }
};

Ok, now, what should an OperationTag look like ? Well, we’ll implement the operation tag corresponding to additions :

struct plus
{
    static float apply(float a, float b)
    {
        return a+b;
    }
};

and the + operator that’ll let us create an expression with a ‘plus’ operation.

template <typename L, typename R>
Expression<L, plus, R> operator+(const L& l, const R& r)
{
    return Expression<L, plus, R>(l, r);
}

Thanks to the definition of Expression, we can embed matrix operations in Expressions, but for the moment we can’t do the reverse way, that is we can’t convert an Expression to a matrix. So let’s write an operator= in the matrix class.

    // inside the matrix class
    template <typename ExprType>
    matrix<N>& operator=(const ExprType& e)
    {
        for(unsigned int row = 0; row < N; row++)
        {
            for(unsigned int col = 0; col < N; col++)
            {
                (*this)(row, col) = e(row, col);
            }
        }
        return (*this);
    }

So you see, this is the only moment where we call the operator()(unsigned int, unsigned int) of the Expression type. Thus, this is the only moment when the whole expression is being evaluated. Let’s study the following code.

#include <iostream>
#include "matrix.h"
#include "expression.h"

int main()
{
    matrix<2> m1;
    m1(0,0) = 1.0;
    m1(1,0) = 0.0;
    m1(0,1) = 4.0;
    m1(1,1) = 1.0;

    matrix<2> m2;
    m2(0,0) = 0.0;
    m2(1,0) = -1.0;
    m2(0,1) = 1.0;
    m2(1,1) = 2.0;

    matrix<2> m3;
    m3(0,0) = 1.0;
    m3(1,0) = -2.0;
    m3(0,1) = 3.0;
    m3(1,1) = 5.0;

    matrix<2> a;
    a = m1 + m2 + m3;
    std::cout << a << std::endl;
}

Here, “m1 + m2 + m3″ crates an

Expression< Expression<matrix<2>, plus, matrix<2> >, plus, matrix<2> >

instance. The coefficients aren’t computed until operator= is called. Indeed, we have the following expression tree.

Expression tree

That is, we know which operations we have to call, on which matrices, but nothing is evaluated. The only place where we call operator() (unsigned int, unsigned int) on a value of type Expression is in matrix<N>::operator=, and this call actually computes each coefficient, one by one, applying the whole computation tree (here, two calls to ‘+’) for each coefficient. This way, we only execute the two for loops once, and it’ll remain the same whatever the number of computations is. Moreover, the only change we had to make to our matrix class was to add an operator= to be able to assign an expression to a matrix.

By the way, the output of our main function is the following.

[2;8
-3;8]

I hope you enjoyed this post

Basically, it’s just about one template structure, using very basical templates techniques : specialization, metaprogramming-level recursion and nested enum for the computation. Either a familiarity with template metaprogramming or with functional programming is enough to understand the following code.

If you’re familiar with any scheme of functional recursion, or recursion in its mathematical sense, you’ll get it quite easily. Mathematically, we’re somehow defining a recursive function generate_id this way :

Which, in C++, becomes :

template <typename T>
struct id_generator;

template <typename T>
struct id_generator<T*>
{
    enum { result = id_generator<T>::result*2 };
};

template <typename T>
struct id_generator<T&>
{
    enum { result = id_generator<T>::result*3 };
};

template <typename T>
struct id_generator< std::vector<T> >
{
    enum { result = id_generator<T>::result*11 };
};

template <>
struct id_generator<int>
{
    enum { result = 5 };
};

template <>
struct id_generator<char>
{
    enum { result = 7 };
};

And a minimal example of use :

int main()
{
    std::cout << "int : " << id_generator< int >::result << std::endl
              << "char**** : " << id_generator< char**** >::result << std::endl
              << "vector<int> : " << id_generator< std::vector<int> >::result << std::endl;
    return 0;
}
/* Output :
int : 5
char**** : 112
vector<int> : 55
*/

To make it really usable, you’ll have to add much (much much much) more :

• base types, like int and char here, which help for terminating the recursive call
• composition rules, like vector, references and pointers here, which help for propagating the recursion

To ensure the uniqueness here, I use prime numbers, simply [1]. Indeed, for the base types, I give a value being a prime number. For the different compositions, I multiply by a different prime number for each. However, most of you may have noticed that the id only lets you know about what type compositions and base types are used, but gives nothing about the order. If we add a rule for std::list, multiplying by 13 for example, then a list of vectors of ints will have the same id as a vector of lists of int. It’d need additional code and workarounds to take the order of the compositions in account — I guess it is possible though. If any reader here comes up with a solution, let us know. A possible solution would be to drop prime numbers, since they rely on a commutative (abelian) ring [2].

[1] http://en.wikipedia.org/wiki/Fundamental_theorem_of_arithmetic
[2] http://en.wikipedia.org/wiki/Commutative_ring