NixDev Open Source Blog

I want to announce that russian part of my blog was moved to separate blog at One FOSS developer story.

If you want to read my blog on Russian — please add new feed

I’m already write some articles about problems in running Trac on Jython. Today i describe successfully installation of popular Django-based blog engine Byteflow on Jython.

Step 1: Install Java, Jython, Django

To setup Byteflow on Jython you must install JDK, and Jython compiler. I’m recommend to use Sun JDK version 1.6 and latest Jython from 2.5 branch (Now latest version of Jython is 2.5b0 and final 2.5 version will released soon).

Note: if you install Jython on computer without graphical interface you must run console version of installer: $java -jar jython_installer-2.5b0.jar --console

otherwise you get error:

java.awt.HeadlessException: No X11 DISPLAY variable was set, but this program performed an operation which requires it.

Note: After Jython installation you must set executable flag on jython binary: $chmod +x <path to Jython home>/bin/jython

After it you need to install Django. Django from version 1.0.0 work out of box with Jython, therefore you can get latest version of 1.0.x branch (Now latest version of Django is 1.0.2).

$wget http://www.djangoproject.com/download/1.0.2/tarball/ $tar xjvf Django-1.0.2-final.tar.gz $cd Django-1.0.2-final $jython setup.py install

Step 2: Install Byteflow requirements Byteflow depends on some libraries which not distributed with Jython, therefore you must download and install it manually.

Install database wrapper: Jython can’t use standard DBAPI-2.0 modules because they use low level C-implementation of driver, instead it use JDBC database drivers and zxjdbc wrapper.

You must download and install django-jython module which provide Django database wrappers for jdbc drivers: $svn co http://django-jython.googlecode.com/svn/trunk/ django-jython $cd django-jython $jython setup.py install

Currently it support only postgresql and sqlite3.but work on other databases (including mysql) in progress.

Also you must download appropriate jdbc driver, for Postgresql you must to download it from http://jdbc.postgresql.org/

Install BeautifulSoup: You must download and compile BeautifulSoup module for Jython: $wget http://www.crummy.com/software/BeautifulSoup/download/BeautifulSoup.tar.gz $tar xzvf BeautifulSoup.tar.gz $cd BeautifulSoup-3.0.7a $jython setup.py install

Install ElementTree: Byteflow use ElementTree ant you need to install in by standard way: $svn checkout http://jython-elementtree.googlecode.com/svn/trunk elementtree $cd elementtree $jython setup.py install --stdlib

Install OpenID: Download and install python-openid version2: $wget "http://openidenabled.com/files/python-openid/packages/python-openid-2.2.1.tar.bz2" $jar xzvf python-openid-2.2.1.tar.bz2 $cd python-openid-2.2.1 $jython setup.py install

Step 3: Download, configure and run Byteflow: You can download Byteflow from mercurial repository and follow simple installation guide: $hg clone http://hg.piranha.org.ua/byteflow/ $cd byteflow

After this you must change some configuration values in settings_local.py according of you preferences. You must change value in DATABASE_ENGINE field to use django-jython wrapper: DATABASE_ENGINE = 'doj.backends.zxjdbc.postgresql' # variants: mysql, sqlite3

Also you must change value of CAPTCHA variable — set recaptcha if you have account on recaptcha or disable captcha by specify empty value. You must do this because captcha module depends on PIL which is not ported to Jython.

Now you are ready to create Byteflow database. You need specify path to jdbc jar file in classpath because otherwise Jython can’t resolve this: $export JYTHON_HOME=<path to Jython home> $java -classpath $JYTHON_HOME/jython-complete.jar:<path to dowloaded jdbc jar>/postgresql-8.3-604.jdbc3.jar -Dpython.home="$JYTHON_HOME" org.python.util.jython manage.py syncdb

After this you can finally run Byteflow instance with django internal http server: $export JYTHON_HOME=<path to Jython home> $java -classpath $JYTHON_HOME/jython-complete.jar:<path to dowloaded jdbc jar>/postgresql-8.3-604.jdbc3.jar -Dpython.home="$JYTHON_HOME" org.python.util.jython manage.py runserver

Reference:
 Distributions:

• Jython
• Byteflow
• Django
• JDBC PostgreSQL driver

Modules:

• BeautifulSoup
• OpenId for Python
• ElementTree for Jython

Articles:

• Byteflow instalation
• Django On Jython
• Database Backends
• zxJDBC Documentation
• Django On Jython progress report

Some days ago i start discussion about current status of Python frameworks on Jython (only Russian version available). In short summary — many people work on this problem and Django was first.

Now i want to talk about Trac on Jython status. Google don’t provide any useful results on query about Trac on Jython, therefore i start more detailed research.

If you try to run Trac 0.11.1 on Jython you need to run all Trac dependencies, which includes Setuptools (already worked fine on Jython) and Genshi. Main problem there is Genshi library which dependence on xml.parsers.expat which in python bindings to native expat library. This require writing JNI wrapper for expat library. Also Genshi use some low-level functions of CPyhton like: AST rewriting, and CPython bytecode generation. (Some discussions about Genshi on Jython).

Genshi also used in TurboGears 2 framework and some people work on Genshi porting to Jython. This work was started as GSoC 2008 project: Improving TurboGears2 support on Jython. You can read some status reports about progress: Weekly status report, TG2 (already partly working) on Jython — status update,

There is some facts about current OVAL support in deb-based distributions (Debian and Ubuntu):

• Ubuntu 8.04 Hardy released with oval-interpretator 5.3 in universe section
• oval-interpretator 5.4 already migrated to Debian testing and ready to ship with Lenny
• oval-interpretator 5.4 migrated to Ubuntu unstable repository and will include in next releas

In next phase we plan to add official support status to OVAL definitions for Debian and Ubuntu distributions.

I’m start series of articles devoted to download, build and install latest version of OpenJDK 7 under Debian Lenny. And in first article i want to describe how we can get latest source code from official mercurial repository.
At first stage we must read Repository Guide and install mercurial >= 0.9.4. Lenny already include mercurial 0.9.5 (1.0 version can be included in release). Also i create separate dir for openjdk repository in /opt/repos/mercurial/openjdk/openjdk7
Please be attentive, to proper clone openjdk repository you must use fclone commant (this command provided by extension called forest and not included in official mercurial package and Debian repository). Therefore you must manual install this extension:

     mkdir -p /opt/repos/mercurial/mercurial/forest/
     cd /opt/repos/mercurial/mercurial/forest/
     hg clone http://hg.akoha.org/hgforest/
     cp hgforest/forest.py /usr/share/python-support/mercurial/hgext/
   

And the configure your .hgrc to enable the extension by adding following lines:

     [extensions]
     hgext.forest=
     # or, if forest.py is not in the hgext dir:
     # forest=/usr/share/python-support/mercurial/hgext/forest.py
   

After this we are ready to clone openjdk repository:

     mkdir -p /opt/repos/mercurial/openjdk/openjdk7
     cd /opt/repos/mercurial/openjdk/openjdk7
     hg fclone http://hg.openjdk.java.net/jdk7/jdk7
   

This action require around 70 Mb and around 10-20 minutes of time. But after this you would have own copy of main openjdk development tree.
ps: You need fclone because openjdk repository use nested mercurial trees inside main tree. If you try to use clone instead of fclone you dowload only some documentation files without source code.

Added on 23 October 2008: Now OpenJDK 6 is official part of Debian repository: openjdk-6

Swarm is a platform for agent-based models ABMs that includes:

• A conceptual framework for designing, describing, and conducting experiments on ABMs;
• Software implementing that framework and providing many handy tools; and
• A community of users and developers that share ideas, software, and experience.

It mostly written in Objective-C but give ability to develop model in Java. In this article i give first introduction in Java Swarm and show how to run one of examples — Heatbugs. Heatbugs — is one of our canonical Swarm demonstrations, an example of how simple agents acting only on local information can produce complex global behavior.

All commands in this article given for Debian GNU/Linux distribution (and must work similar on most other moder Linux distributions). Good article about building and running Swarm on Debian written by Jakson Aquino (but their article don’t contain information about Java Swarm).

Java installation

Currently i don’t have information about supporting GNU Java by Swarm, therefore i use Sun Java SE 1.6.0_03 as java virtual machine. To install Sun Java on Debian you have two options:

• Use Debian backports and install sun-java6-jdk
• Download binary distribution from official Sun Java site and install it manually
  

Build Swarm with Java support

1. Install the software necessary to download and compile swarm:

   sudo apt-get install gobjc gperf libxpm-dev libpng12-dev automake1.9 emacs21-nox libhdf5-serial-dev blt-dev cvs autoconf libtool make xfonts-75dpi xfonts-100dpi

2. Swarm needs automake1.9. Thus, run update-alternatives and select automake1.9 if it was not the default yet (For Debian Etch — automake 1.9 is already default version):

   sudo update-alternatives —config automake

3. Now, you can download the swarm source code. There are two options:

   • Paul Johnson’s web site:

     wget http://pj.freefaculty.org/Swarm/swarm-2.2.3.tar.gz tar -xzf swarm-2.2.3.tar.gz cd swarm-2.2.3

   • The official cvs repository:

     cvs -z3 -d:pserver:anonymous@cvs.savannah.nongnu.org:/sources/swarm co swarm cd swarm ./autogen.sh

4. Then, you can compile swarm with java support:

   • If you install Java from backport package:

     ./configure —with-jdkdir=/usr/lib/jvm/java-6-sun —with-gnu-ld —with-tclinclude=/usr/include/tcl8.4/ —with-tclscriptdir=/usr/lib/tcl8.4/ —with-tkscriptdir=/usr/lib/tk8.4/ —with-tcl=/usr/lib/tcl8.4/ —with-tk=/usr/lib/tk8.4 —with-tkinclude=/usr/include/tcl8.4

   • If you install Java from binary distribution archive — replace java_path on real Java installation path:

     For swarm from cvs

     ./configure —with-jdkdir=java_path —with-gnu-ld —with-tclinclude=/usr/include/tcl8.4/ —with-tclscriptdir=/usr/lib/tcl8.4/ —with-tkscriptdir=/usr/lib/tk8.4/ —with-tcl=/usr/lib/tcl8.4/ —with-tk=/usr/lib/tk8.4

     make

     sudo make install

     For swarm 2.2.3

     ./configure —with-jdkdir=java_path —with-gnu-ld —with-tclincludedir=/usr/include/tcl8.4/ —with-tclscriptdir=/usr/lib/tcl8.4/ —with-tkscriptdir=/usr/lib/tk8.4/ —with-tcldir=/usr/lib/tcl8.4/ —with-tk=/usr/lib/tk8.4

     make sudo make install

5. Swarm is already installed. You should now put this line at the end of your ~/.bashrc, because, by default, the Makefiles that come with swarm applications are configured to look for swarm at /usr, but since we installed it at /usr/local, we need to inform the Makefiles about this:

   export SWARMHOME=/usr/local

   In addition to putting the above line in your ~/.bashrc, copy and paste it in your terminal too. When you logoff and login again the .bashrc will be re-read and the SWARMHOME environment variable automatically will be active.

Build and run Heatbugs

You can download Heatbugs from Swarm ftp:

> wget http://ftp.swarm.org/pub/swarm/apps/java/sdg/jheatbugs-2.1.tar.gz
> tar -xvzf jheatbugs-2.1.tar.gz
> cd jheatbugs-2.1/
> make

If build failed with error like:

> HeatbugModelSwarm.java:297: cannot find symbol
> symbol  : constructor FArgumentsImpl(HeatbugModelSwarm,swarm.Selector,boolean)
> location: class swarm.defobj.FArgumentsImpl
>    new FCallImpl (this, proto, sel, new FArgumentsImpl (this, sel, true)));
>                                                 ^
> 1 error

Then you must download and apply my patch for jheatbugs:

> wget http://nixdev.net/projects/swarm/patches/jheatbug-2.1-FArgumentsImpl-parameters.diff
> patch -p1 -l < jheatbug-2.1-FArgumentsImpl-parameters.diff

After this you can repeate you build. If all finished OK — you can finally run this demo:

> javaswarm StartHeatbugs

Please note that javaswarm is a script written on top of java, which set proper environment settings and run swarm java application. Now you can play with Heatbug and see it evolution.

References

• Additional information about swarm can be found on Swarm wiki and Swarm application examples.
• Information about build and run ObjectiveC version of Swarn available on Jakson Aquino swarm page.
• Debian packages for Swarm 2.2.5: Masayuki Hatta is packaged latest Swarm version for Debian and Ubuntu distrbution. This packages is available at http://www.mhatta.org/debian/swarm and will included in Official Debian repository soon.

My task for GSoC 2007 was further integration of OVAL language and infrastructure with Debian project. OVAL language already include scheme for support Debian package system (dpkg), but this functionality was don’t implemented in referenced OVAL interpretator. Therefore my first task was implement this support, create Debian package for referenced OVAL interpretator, and submit resulted patches to upstream for inclusion in next official release.

My second task was improve script that convert DSA to OVAL definitions. This script was initially developed by my mentor Javier Fernandez-Sanguino, but was based on old OVAL version and generate only simple OVAL definition that not pass validation by OVAL definition scheme.

Implementing of this two task will allow to use OVAl interpretator to evaluate security status of Debian hosts according to issued DSA. Both of this task is finished now. During work on implementing DPKG support in referenced OVAL interpretator i was implement DPKGInfoProbe and TextFileContentProbe classes. This patches was included in ovaldi Debian package and sent to upstream. Currently this patches not fully applied to upstream source, but i will continue interaction with OVAL developers to include this feature in next release.

DSA to OVAL definitions converter was fully rewrite on python language and now produce valid OVAL definitions for most of DSA in Debian repository (some old DSA is ignored because have different naming scheme). We plan to use this script as part of oval-server package for in-place conversion of DSA and on security.debian.org web-site to provide Debian OVAL definitions feed corresponding to issued DSA. This converter require some updates to reduce memory usage on handling large DSA repository (like full DSA repository).

Second and very important task was implementing agent-server architecture over the OVAL interpretator to centralized handle security status of whole Debian networks. This task include development of oval-server program that must generate OVAL definitions repository (generate from DSA or download from official debian.org feed), generate per-client definitions to evaluate security status of Debian client hosts and generate security status reports for visual presentation of security status. Oval-agent tools was developed to work on client side and it purpose include receive OVAL definitions from server, evaluate them with OVAL interpretator and send results back to server.

At the first stage of project i will plan implement both tools in C++, but during work on this task i change my choice to python language due to time constraint and better integration with DSA2OVAL converter. Oval-server use SQLite as database engine to store information about clients and definitions, HTTP protocol to interact with oval-agents and currently support synchronization DSA repository over http, ftp, local fs (in plan mail and rss support, and support fetch prepared OVAL definitions from debian.org website).

Both, oval-agent and oval-server in working prototype stage. They implements most of required functional, but have some important issues which must be fixed before inclusion in Debian repository. I plan to keep list of this issues on DebianOval wiki page.

I plan to continue working and prepare oval-agent and oval-server packages to the end of this year or early.

During last month my life was greatly changed. I moved into other town, and set up my new work. Now i will work on Java EE stack and continue write my Ph.D. thesis on Multi-Agent Solutions. As for Oval Agent project — i will continue it development as SAMS project.

For Oval project i plan to focus on fixing bugs in oval-agent and oval-server. Improve performance and memory requirements for converting Debian DSA. After this i will update Debian packages for this module and ask my summer mentor Javier Fernandez-Sanguino to sponsoring this packages.

For SAMS project we plan to release SAMS 1.0 in next days after this i can focus on updating Debian package and improve analog log parser for squid access.log. Also we need to focus on updating English version of documentation.

This goals planed to the end of current year and i hope that i can finish this until Christmas.

Today i finished most work on adopting OVAL interpretator for work with Debian OVAL definitions. This include implementation of TextFileContentProbe DPKGInfoProbe classes. Now we have both —

• OVAL definitions generator
• and special version of OVAl interpretator

Next targets:

• Bugfixing
• Building Debian packages
• Merging patches in upstream

Second part of my work in SoC was implement support of DEB-based distribution in OVAL interpreter. After discussion with my mentor i choice libapt for interacting with Debian package system.

But i have a troubles with this library because it have poor documentation. I start use regression test, which i found in library source but some of them seems broken. After some attempt to find problem i move to apt-get source and use them to build my first package querying program. apt-get use dpkg cache file to obtain information about all available packages. Therefor source for querying packages from dpkg cache file seems like:

  bool checkExist (string package)
     {
           bool exist = false;

    pkgCacheFile *Cache = new pkgCacheFile();

    OpTextProgress Prog(*_config);
    if (Cache->Open(Prog, true) == false) {
            cerr << "I need more priveleges." << endl;
    }

    pkgCache::PkgIterator Pkg = (*Cache)->FindPkg (package);

    if (strcmp (Pkg.Name(), package.c_str()) == 0 and (Pkg.CurrentVer()))
            exist = true;

    Cache->Close();

    return (exist);
  }

I think this approach is good and compact but have one lack in size of querying cache. Because we query over all available packages, not only installed. And for my machine this number about 20000.

My mentor Javier propose another approach based on apt-sort source code. This approach use dpkg status file for querying information about installed packages. There another version of checkExist function:

  bool DPKGCheckExist (string name)
     {
           FileFd Fd(StatusFile, StatusFile::ReadOnly);
           pkgTagFile Tags(&Fd);

    bool found = false;
    if (_error->PendingError() == true)
            return "false";

    // Parse.
    vector List;
    pkgTagSection Section;
    unsigned long Offset = Tags.Offset();

    while (Tags.Step(Section) == true && found == false)
    {
            PkgName Tmp;
            /* Fetch the name, auto-detecting if this is a source file or a package file */
            Tmp.Name = Section.FindS("Package");

            if (Tmp.Name.empty() == true)
                    //return _error->Error("Unknown package record!");
                    return "false";

            if ( stringcasecmp(Tmp.Name,name) == 0 )
            {
                    if (stringcasecmp(Tmp.Stat,"install ok installed") == 0 )
                    {
                            return true;
                    } else {
                            cout << "Package not fully installed" << endl;
                            return false;
                    }

                    found = true;
            }

            Tmp.Offset = Offset;
            Tmp.Length = Section.size();

            Offset = Tags.Offset();
    }

    if (found == false)
            return "ERROR";
    if (_error->PendingError() == true)
            return "ERROR";
  }

This example contain more code, but also allow to extract more information about installed packages. And this fragments would move to other functions which used to extraction DPKG package information. Currently i use this example in my work.

If you would plan use of libapt in you project this two approach may help you get started in rigth way.