Thursday, February 21, 2013

Becoming unaware of the Dependency Injection Framework. Encouraging standarization

Intro

The previous post showed up how to combine Spring Framework, Mockito and Lombok in TDD fashion. But it's far from be the simplest and portable solution if such thing even exists. Let's try to make the whole example more unaware of the DI Framework, in this case Spring, but still use it for testing purposes.

Java 6 brought us several improvements, CDI among them. This refcardz is self-explanatory. IMO the main advantage of CDI is standardization, but CDI is pure specification, not an implementation, and its RI is Weld.

Results that Spring honors CDI since version 3.0, at least in Java Injection Standard (JSR-330). There is a bunch of posts and articles talking about the Spring support for CDI specifications, and which one is better to use. But this is not the goal of this post.

Goals

  • Become unaware of the DI Framework at design time
  • Support Spring at test time
  • Support Spring at runtime
  • Support others DI Framework at runtime

How to

  • By replacing @Autowired and using injection by name with @Resource
  • By naming beans using @Name and scanning packages for candidates instead of explicitly create the bean in application context configuration file.

Let's do it

As I said before, it's just an example. Real world scenarios are more complex, and you will end for sure combining many APIs from different sources.

  1. Use the example from the previous post.
  2. Include a couple of dependencies in your pom.xml.
  3. 
      javax.annotation
      jsr250-api
      1.0
    
    
      javax.inject
      javax.inject
      1
    
    
    JSR-250 provide us various standard annotations: @Resource, @PostConstruct, @PreDestroy , etc; these annotations are honored by Spring. In the other hand  with have JSR-333 (javax.inject) that give us @Inject, @Named, @Provider, @Qualifier, @Scope and @Singleton.
    Lets explain some of these annotations:
    • @Resource
      • Matching: First matches by name, then matches by type and filters by qualifiers if not name-based match gets found.
      • Applicable to: Fields, Classes and Methods.
      • Package: javax.annotation
    • @PostConstruct
      • Runs: After the bean construction and wiring
      • Applicable to: Methods.
      • Package: javax.annotation
    • @PreDestroy
      • Runs: Before the release of the bean
      • Applicable to: Methods.
      • Package: javax.annotation
    • @Inject
      • Matching:  First matches by type, filters by qualifiers, then matches be name.
      • Applicable to: Methods, Constructors and Fields.
      • Package: javax.inject
    • @Named
      • Description:  Its a qualifier that denotes a named bean. Annotated classes are considered for auto-detection on classpath scanning. It's similar to Spring's @Component
      • Applicable to: Classes and Fields.
      • Package: javax.inject
  4. Spring dependencies are now on test scope, since they are only required on test time. But in a real world scenario you will need a DI Framework, so don't forget to include Spring or Weld dependencies for production stage.
  5. 
      ...
     
      org.springframework
      spring-test
      ${spring.version}
      test
     
     
      org.springframework
      spring-beans
      ${spring.version}
      test
     
     
      org.springframework
      spring-context
      ${spring.version}
      test
     
      ...
    
    
  6. The test is an hybrid between Spring and more standard stuff, that is:
  7. package demos.sf.editor.test;
    
    import ... ;
    
    @RunWith(SpringJUnit4ClassRunner.class)
    @ContextConfiguration(locations = { "/test-applicationContext.xml" })
    @TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
      DirtiesContextTestExecutionListener.class })
    @DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
    public class EditorTest {
    
     @Resource
     private Editor editor;
    
     @Resource
     private SpellChecker spellChecker;
    
     @Test
     public void testPasteSuccess() {
      editor.paste("Hello everybody!");
    
      String expected = "Hello everybody!";
      String actual = editor.getText();
      assertEquals(expected, actual);
     }
    
     @Test
     public void testAddParagraphSpellIsChecked() {
      editor.paste("Hello everybody!");
      verify(spellChecker, only()).check("Hello everybody!");
     }
    }
    
  8. The SUT (Editor) is now DI Framework unaware:
  9. package demos.sf.editor.impl;
    
    import ...;
    
    @Named
    public class Editor {
    
     @Getter
     protected String text;
    
     @Resource
     private SpellChecker spellChecker;
    
     public void paste(String cut) {
      if (text == null) {
       text = "";
      }
      text += cut;
      spellChecker.check(cut);
     }
    }
    
  10. The application context configuration file for testing stage file is now simplified. Only package scanning is needed to create the SUT (Editor):
  11. 
    
     
    
     
      
     
    
    
    
    NOTE:Observe that some minor package relocation were made to organize tests, interfaces and classes.
The final snapshot can be found here under the spring-hello-word-cdi directory. Enjoy it!

Wednesday, December 19, 2012

A TDD approach using Spring Framework + Mockito + Lombok

Intro

This is simple Hello World Java application that successfully combines Spring Framework with Mockito and Lombok. It's just for demonstrative purposes, so the presented example is very simple and used many times before in the literature.

The final goal is to show up how simple is to combine some of the best features of each involved technology with Test-Driven-Development in mind:

  • Spring Framework: wiring
  • Mockito: isolation
  • Lombok: simplicity

Example

A text editor uses and spell checker among other things to make the life of user easier. Depending on the target language, an spell editor may change is behavior  and its dependencies as well, for example dictionaries and tokenizers. Simplifying the whole idea you may encounter a contract-driven (interface based) solution like this:

with many possible combinations of spell checkers, dictionaries and tokenizers. For example this one:
where every contract was replaced by a concrete instance. Combinations are multiple, and sometimes you cannot predict all them in a classes/interfaces design.

You want to start engaging TDD for sure, and you want to do it in a top-down fashion. That means, start testing the text editor first, then the spell checker, then the remaining pieces. You also want to do it in isolation, so spell checker and the remaining implementations aren't involved in editor tests, and it's your desire to obtain zero infrastructure code (no constructor calls). As a plus no getter nor setter nor trivial constructor implementations.

What to test?

Our test will be very simple as well, here is the top-down ordered list (remember it's a simplification of the reality):

1- Given an Editor, when a new text is pasted it should be added.
2- Given an Editor, when a new text is pasted the spell should be checked against the appended text.
3- Given an English spell checker, when a word is not found in a dictionary during the spell check then it must be signaled as misspelled and returned back.


You may be wondering, why should I test these  obvious use cases? Trust me, it's very important to cover all possible scenarios if you really want to delivery software with built-in quality. This is the hidden synergy behind TDD.


Hands on Spring Tool Suite

  1. Open STS and create a new Maven project.
  2. Check 'Create a simple project (skip archetype selection)'.
  3. Enter a Group Id = demos.sf.editor, Artifact Id = spring-hello-world and Version = 1.0.
  4. Edit your pom.xml and append all the required dependencies. It should ends like:
  5. 
      4.0.0
      demos.sf.editor
      spring-hello-word
      1.0
      Spring Framework Hello World
      Spring Framework Hello World demo w/ Unit Tests
      
        UTF-8
        3.2.0.RELEASE
      
      
        
          junit
          junit
          4.10
        
        
        
          org.springframework
          spring-test
          ${spring.version}
        
        
        
          org.springframework
          spring-beans
          ${spring.version}
        
        
        
          org.springframework
          spring-context
          ${spring.version}
        
        
          org.mockito
          mockito-all
          1.9.5
        
        
          org.projectlombok
          lombok
          0.11.6
        
      
    
    
  6. Start testing! A golden rule, don't forget it. Create a new JUnit 4 Test Case at src/test/java. Name it EditorTest in package demos.sf.editor. 
  7. Delete the existing test, called test and append a new failing test for the first use case. Its name testPasteSuccess:
  8. package demos.sf.editor;
    
    import static org.junit.Assert.fail;
    
    import org.junit.Test;
    
    public class EditorTest {
    
     @Test
     public void testPasteSuccess() {
      fail();
     }
    
    }
    
    Making the test fail it's very important, if you start failing you won't forget it until its fixed and the test pass. So the "last thing" to do is to remove the failing statement.
  9. Write the assertion first. How's that? See use case 1: "... the text should be added":
  10. ...
    import static org.junit.Assert.assertEquals;
    
    public class EditorTest {
    
     @Test
     public void testPasteSuccess() {
      String expected = "Hello everybody!";
      String actual = editor.getText();
      assertEquals(expected, actual);
      fail();
     }
    }
    The code above will fail to compile, because there isn't and variable/field called editor. This is perfectly normal in TDD, guide your design only by needs (the tests).
  11. Right-click on editor and choose "Create local variable...". Change its type from Object to Editor, a non-existing class:
  12. public class EditorTest {
    
     @Test
     public void testPasteSuccess() {
      Editor editor;
    
      String expected = "Hello everybody!";
      String actual = editor.getText();
      assertEquals(expected, actual);
      fail();
     }
    }
    
    The code above, still without compiling due to the non-existing class Editor. Right click on Editor and "Create a new class...".
  13. Annotate the Editor class for auto-implementing the getter using Lombok annotations:
  14. package demos.sf.editor;
    
    import lombok.Getter;
    
    public class Editor {
     @Getter
     private String text;
    }
    
    NOTE: Lombok must be attached to Spring Tool Suite (or Eclipse) for completion at development time. Copy your lombok.jar to STS installation folder and append the following settings to your STS.ini (eclipse.ini):
    -javaagent:/home/lago/Soft/springsource2.9.2/sts-2.9.2.RELEASE/lombok.jar
    -Xbootclasspath/a:/home/lago/Soft/springsource2.9.2/sts-2.9.2.RELEASE/lombok.jar
    
    Alternative open a terminal/command prompt and run:
    $ java -jar ~/.m2/repository/org/projectlombok/lombok/0.11.6/lombok-0.11.6.jar
    
    An install wizard gets launched. Choose your STS.ini (eclipse.ini) and press Install/Update. Finally restart your IDE.
  15. Go back to the test, and perform the call to a non-existing paste() method:
  16. public class EditorTest {
    
     @Test
     public void testPasteSuccess() {
      Editor editor;
      editor.paste("Hello everybody!");
    
      String expected = "Hello everybody!";
      String actual = editor.getText();
      assertEquals(expected, actual);
      fail();
     }
    }
    
  17. Ctrl+1 on top the editor.paste("...") call and choose "Create method paste()....". A new method is generated:
  18. public class Editor {
     @Getter
     private String text;
    
     public void paste(String cut) {
     }
    }
    
    Go back to the test, don't waste your time implementing anything at this moment, the test will drive you to that point later.
  19. In the test, the only missing thing is the editor initialization. Let's inject the Editor at test time. Convert the editor local variable to a field declaration and annotate it as @Autowired:
  20. ...
    import org.springframework.beans.factory.annotation.Autowired;
    
    public class EditorTest {
    
     @Autowired
     private Editor editor;
    
     @Test
     public void testPasteSuccess() {
      editor.paste("Hello everybody!");
    
      String expected = "Hello everybody!";
      String actual = editor.getText();
      assertEquals(expected, actual);
      fail();
     }
    }
    
  21. Create a new Spring Bean Configuration file at src/test/resources/test-applicationContext.xml and declare the editor bean on it:
  22. 
      
      
    
    
    
  23. Use the Spring test runner and load the application context via annotations:
  24. ...
    import org.junit.runner.RunWith;
    import org.springframework.test.context.ContextConfiguration;
    import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
    
    @RunWith(SpringJUnit4ClassRunner.class)
    @ContextConfiguration(locations = { "/test-applicationContext.xml" })
    public class EditorTest {
    ...
    }
    
  25. Right click on test case and Run As JUnit test. The test will fail but corner stones are ready to support more agile tests. Make the test pass by implementing the paste() method. Oops! remember to remove the fail() from the test.
    public class Editor {
     @Getter
     private String text;
    
     public void paste(String cut) {
      if(text == null) {
       text = "";
      }
      text += cut;
     }
    }
    
  26. Re-run the test, this time also using a Maven run configuration like this:
  27. mvn test
    
  28. The spring default behavior is to cache the application context between different tests in the same test case. They also introduced the annotations @TestExecutionListeners and @DirtiesContext to mark the application context as dirty after each test:
  29. ...
    import org.springframework.test.context.TestExecutionListeners;
    import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;
    import org.springframework.test.context.support.DirtiesContextTestExecutionListener;
    import org.springframework.test.annotation.DirtiesContext;
    import org.springframework.test.annotation.DirtiesContext.ClassMode;
    
    @RunWith(SpringJUnit4ClassRunner.class)
    @ContextConfiguration(locations = { "/test-applicationContext.xml" })
    @TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
     DirtiesContextTestExecutionListener.class })
    @DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
    public class EditorTest {
    ...
    }
    
  30. Now starts the use case 2. First append the test in failing mode:
  31. ...
    public class EditorTest {
     ...
     @Test
     public void testAddParagraphSpellIsChecked() {
      fail();
     }
    }
    
  32. Using Mockito style, we need to verify that the spell is checked against the pasted text. The syntax is straightforward it means that check() method must be called only once with "Hello everybody!":
  33. ...
    import static org.mockito.Mockito.only;
    import static org.mockito.Mockito.verify;
    
    @RunWith(SpringJUnit4ClassRunner.class)
    @ContextConfiguration(locations = { "/test-applicationContext.xml" })
    @TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
      DirtiesContextTestExecutionListener.class })
    @DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
    public class EditorTest {
    
     ...
    
     @Autowired
     private SpellChecker spellChecker;
    
     @Test
     public void testAddParagraphSpellIsChecked() {
      editor.paste("Hello everybody!");
      verify(spellChecker, only()).check("Hello everybody!");
      fail();
     }
    }
    
  34. At this time we need to create a new SpellChecker interface:
  35. package demos.sf.editor;
    
    public interface SpellChecker {
    
     void check(String text);
    
    }
    
  36. The next step is to provide an spell checker mock to the application context:
  37. 
      
      
      
        
      
    
    
  38. Remove the fail() from the test and run all tests and you will get a test failure saying: 'Wanted but no invoked: spellChecker.check("Hello everybody!")'. That makes sense since we are not invoking the SpellChecker.check() from Editor.paste(). In fact haven't create any kind of dependency Editor -> SpellChecker. Let's do it at class level and force its injection at construction time using Lombok's @RequiredArgsConstructor:
  39. package demos.sf.editor;
    
    import lombok.Getter;
    import lombok.RequiredArgsConstructor;
    
    @RequiredArgsConstructor
    public class Editor {
     @Getter
     private String text;
    
     private final SpellChecker spellChecker;
    
     public void paste(String cut) {
      if (text == null) {
       text = "";
      }
      text += cut;
     }
    }
    
  40. Declare the constructor injection in the application context as well and re-run the tests:
  41. 
      
        
      
      
        
      
    
    
  42. Oops! The same failure: 'Wanted but no invoked: spellChecker.check("Hello everybody!")'. Implement the SpellChecker.check() call and run the tests again:
  43. package demos.sf.editor;
    
    import lombok.Getter;
    import lombok.RequiredArgsConstructor;
    
    @RequiredArgsConstructor
    public class Editor {
     @Getter
     private String text;
    
     private final SpellChecker spellChecker;
    
     public void paste(String cut) {
      if (text == null) {
       text = "";
      }
      text += cut;
      spellChecker.check(cut);
     }
    }
    
    All tests should now pass:

Final Snapshot

  • The test case at : src/test/java/demos/sf/editor/EditorTest.java:
  • package demos.sf.editor;
    
    import static org.junit.Assert.assertEquals;
    import static org.mockito.Mockito.only;
    import static org.mockito.Mockito.verify;
    
    import org.junit.Test;
    import org.junit.runner.RunWith;
    import org.springframework.beans.factory.annotation.Autowired;
    import org.springframework.test.annotation.DirtiesContext;
    import org.springframework.test.annotation.DirtiesContext.ClassMode;
    import org.springframework.test.context.ContextConfiguration;
    import org.springframework.test.context.TestExecutionListeners;
    import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
    import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;
    import org.springframework.test.context.support.DirtiesContextTestExecutionListener;
    
    @RunWith(SpringJUnit4ClassRunner.class)
    @ContextConfiguration(locations = { "/test-applicationContext.xml" })
    @TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
      DirtiesContextTestExecutionListener.class })
    @DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
    public class EditorTest {
    
     @Autowired
     private Editor editor;
    
     @Autowired
     private SpellChecker spellChecker;
    
     @Test
     public void testPasteSuccess() {
      editor.paste("Hello everybody!");
    
      String expected = "Hello everybody!";
      String actual = editor.getText();
      assertEquals(expected, actual);
     }
    
     @Test
     public void testAddParagraphSpellIsChecked() {
      editor.paste("Hello everybody!");
      verify(spellChecker, only()).check("Hello everybody!");
     }
    }
    
  • The test case at : src/main/java/demos/sf/editor/Editor.java:
  • package demos.sf.editor;
    
    import lombok.Getter;
    import lombok.RequiredArgsConstructor;
    
    @RequiredArgsConstructor
    public class Editor {
     @Getter
     private String text;
    
     private final SpellChecker spellChecker;
    
     public void paste(String cut) {
      if (text == null) {
       text = "";
      }
      text += cut;
      spellChecker.check(cut);
     }
    }
    
  • The test case at : src/main/java/demos/sf/editor/SpellChecker.java:
  • package demos.sf.editor;
    
    public interface SpellChecker {
    
     void check(String text);
    
    }
    
  • The test case at : src/main/resources/test-applicationContext.xml:
  • 
      
        
      
      
        
      
    
    
You can obtain the source code from here. Enjoy it!

Tuesday, December 11, 2012

Running webtoolkit application as nginx FastCGI on CentOS 6.x

Wt (pronounced as witty) is a powerful C++ library for developing web applications. Witty-based apps can be integrated as FastCGI with nginx and other web servers. This guide is about the integration of witty w/ nginx on CentOS 6.x OS.

On this how to, we'll be using the simplest witty example: hello

I assumed you already had installed a CentOS 6.x minimal x86_64.

The Steps

  1. Login as a sudoer user.
  2. Append EPEL repository by creating the file /etc/yum.repos.d/epel.repo with the following content:
  3. [epel]
    name=Extra Packages for Enterprise Linux 6 - $basearch
    #baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
    mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
    failovermethod=priority
    enabled=1
    gpgcheck=0
    
  4. Install required packages: nginx, witty and CentOS development kit:
  5. $ sudo yum install nginx 
    $ sudo yum install wt
    $ sudo yum install fcgi
    $ sudo yum install spawn-fcgi
    $ sudo yum install wt-devel wt-examples      # Only for development env
    $ sudo yum groupinstall "Development Tools"  # Only for development env
    $ sudo yum install nano                      # Only for development env
    
  6. Go to Wt's examples directory and edit the CMakeLists.txt archive:
  7. $ cd /usr/lib64/Wt/examples/hello
    $ sudo nano CMakeLists.txt
    
    and replace
    WT_ADD_EXAMPLE(hello.wt hello.C)
    by:
    ADD_EXECUTABLE(hello.wt hello.C)
    TARGET_LINK_LIBRARIES(hello.wt ${EXAMPLES_CONNECTOR})
    
  8. Run CMake specifying FastCGI support & copy resulting binary to nginx document root:
  9. $ sudo rm -rf target
    $ sudo mkdir -p target
    $ cd target
    $ sudo cmake ../ -DEXAMPLES_CONNECTOR=wtfcgi -DCONNECTOR_FCGI=yes -DCONNECTOR_HTTP=no 
    $ sudo make
    $ sudo cp -a hello.wt /usr/share/nginx/html/
    
  10. Create a new archive /etc/sysconfig/spawn-fcgi-hello.wt with the following content:
  11. FCGI_SOCKET=/var/run/hello.wt.socket
    FCGI_PROGRAM=/usr/share/nginx/html/hello.wt
    FCGI_USER=nginx
    FCGI_GROUP=nginx
    FCGI_EXTRA_OPTIONS="-M 0700"
    OPTIONS="-u $FCGI_USER -g $FCGI_GROUP -s $FCGI_SOCKET -S $FCGI_EXTRA_OPTIONS -F 1 -P /var/run/hello.wt.socket.pid -- $FCGI_PROGRAM"
  12. Allow nginx to write at /var/spool/wt/run/:
  13. $ sudo chgrp nginx /var/spool/wt/run/
    $ sudo chmod g+w /var/spool/wt/run/
  14. Launch hello app via spawn-fcgi:
  15. $ source /etc/sysconfig/spawn-fcgi-hello.wt
    $ spawn-fcgi $OPTIONS
    
    now the FastCGI is running and waiting.
  16. Create a new file /etc/nginx/conf.d/wt.conf w/ the following content:
  17. server {
        listen  9091;
        server_name  _;
    
        # by default relative to /usr/share/nginx/html
        location / {
          access_log /var/log/nginx/nginx-fastcgi-access.log;
          gzip off;
    
          # the full path /usr/share/nginx/html/hello.wt
          if ($uri !~ "^/hello.wt/$") {
            fastcgi_pass unix:/var/run/hello.wt.socket;
          }
          include /etc/nginx/fastcgi_params;
          fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        }
    }
    
  18. Restart nginx:
  19. $ service ngnix restart
    
  20. Visit http://HOST:9091 and enjoy it!

Wednesday, August 8, 2012

Packaging Csync2 in RPM on CentOS 6.X

This is a how to package in RPM format, the Csync2 synchronization tool on CentOS 6.x.

Steps

  1. Append RPM Forge repository by creating the file /etc/yum.repos.d/rpmforge.repo with the following content:
  2. $ sudo dd of=/etc/yum.repos.d/rpmforge.repo << EOT
    [rpmforge]
    name=CentOS-$releasever - Rpmforge
    baseurl=http://apt.sw.be/redhat/el6/en/$basearch/rpmforge
    gpgcheck=0
    enabled=1
    EOT
    
    
  3. Install building dependencies:
  4. $ sudo yum -y groupinstall "Development Tools"
    $ sudo yum -y install openssl-devel 
    $ sudo yum -y install librsync librsync-devel
    $ sudo yum install gnutls openssl libtasn1 gnutls-devel
    
    
  5. Create a temporal cache and define a downloader command:
  6. $ mkdir -p /tmp/cache
    
    #/** 
    # * Downloads a file to the cache if doesn't exists
    # *
    # * @param $1 the file to download
    # * @param $2 the url where the file is located
    # */
    $ get() {
     [ -f /tmp/cache/$1 ] || wget -t inf -w 5 -c $2/$1 -O /tmp/cache/$1
    }
    
    
  7. Download latest csync2 tarball with sources:
  8. $ lastver=1.34
    $ cs=csync2-$lastver.tar.gz
    $ get $cs http://oss.linbit.com/csync2/$cs 
    
    
  9. Download an sqlite compatible version:
  10. $ sqver=2.8.16
    $ sq=sqlite-$sqver.tar.gz
    get $sq http://pkgs.fedoraproject.org/repo/pkgs/sqlite/$sq/9c79b461ff30240a6f9d70dd67f8faea/$sq
    
    
  11. Create a clean RPM build environment:
  12. $ rm -rf ~/rpmbuild ~/.rpmmacros
    $ mkdir -p ~/rpmbuild/{BUILD,RPMS,S{OURCE,PEC,RPM}S}
    $ cat > ~/.rpmmacros <<< "%_topdir $HOME/rpmbuild"
    
    
  13. Copy the sources tar balls to SOURCES and BUILD:
  14. $ cd ~/rpmbuild
    $ cp /tmp/cache/$cs SOURCES/
    $ cp /tmp/cache/$sq BUILD/
    
    
  15. Extract the csync2.spec archive from tar ball and modify it to user sqlite sources and include some missing files into the final RPM:
  16. $ cd SPECS
    $ tar --strip-components=1 -x csync2-$lastver/csync2.spec -vzf ../SOURCES/$cs
    $ sed -i \
      -e 's/^%changelog/%files\n%defattr(-,root,root,-)\n\/usr\/sbin\/csync2-compare\n\/etc\/csync2.cfg\n\/etc\/xinetd.d\/csync2\n\/usr\/sbin\/csync2\n\/usr\/share\/man\/man1\/csync2.1.gz\n\n&/' \
      -e 's/\(^%configure\)/\1  --with-libsqlite-source=..\/sqlite-2.8.16.tar.gz  --disable-gnutls/' csync2.spec
    $ cd ..
    
    
  17. Build and install the csync2 RPM packages:
  18. $ rpmbuild -bb SPECS/csync2.spec
    $ sudo rpm -ivh RPMS/x86_64/csync2-$lastver-1.x86_64.rpm
    
    

Enjoy it!

Thursday, July 5, 2012

Building and packaging the latest Gearman server in CentOS 6.2

This is a how to package, install and test the latest version of Gearman server in RPM format using CentOS 6.2.

Motivation

The latest version available of Gearman in the Fedora repository is very outdated, CentOS is even more. So, if you plan to use the latest Gearman features, you have two choices 1) compile it using the tarball; 2) package it (hence compile it) in RPM format.
This guide uses the second approach, but not from scratch. Instead using the latest available SRPM German's package and performing some minor changes.

At this moment, the latest Gearman version is 0.33 and the latest Fedora-based SRPM is 0.23.

Hands on bash

  1. Append EPEL repository by creating the file /etc/yum.repos.d/epel.repo with the following content:
  2. [epel]
    name=Extra Packages for Enterprise Linux 6 - $basearch
    #baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
    mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
    failovermethod=priority
    enabled=1
    gpgcheck=0
    
    [epel-debuginfo]
    name=Extra Packages for Enterprise Linux 6 - $basearch - Debug
    #baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch/debug
    mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-6&arch=$basearch
    failovermethod=priority
    enabled=0
    gpgcheck=0
    
    [epel-source]
    name=Extra Packages for Enterprise Linux 6 - $basearch - Source
    #baseurl=http://download.fedoraproject.org/pub/epel/6/SRPMS
    mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-6&arch=$basearch
    failovermethod=priority
    enabled=0
    gpgcheck=0
    
  3. Install building dependencies:
  4. $ sudo yum groupinstall "Development Tools"
    $ sudo yum install libevent-devel libuuid-devel
    $ sudo yum install boost-devel
    $ sudo yum install libmemcached-devel memcached google-perftools-devel 
    
  5. Create a temporal cache and define a downloader command:
  6. $ mkdir -p /tmp/cache
    
    #/** 
    # * Downloads a file to the cache if doesn't exists
    # *
    # * @param $1 the file to download
    # * @param $2 the url where the file is located
    # */
    $ get() {
     [ -f /tmp/cache/$1 ] || wget -t inf -w 5 -c $2/$1 -O /tmp/cache/$1
    }
    
  7. Download latest Gearman tarball with sources and the latest SRPM package provided by Fedora
  8. $ lastver=0.33
    $ gm=gearmand-$lastver.tar.gz
    $ get $gm https://launchpadlibrarian.net/104788829/$gm 
    
    $ srcver=0.23
    $ srpm=gearmand-$srcver-1.fc16.src.rpm
    $ get $srpm http://www.muug.mb.ca/mirror/fedora/linux/releases/16/Everything/source/SRPMS/$srpm
    
    
  9. Create a clean RPM build environment and install the SRPM package on it:
  10. $ rm -rf ~/rpmbuild ~/.rpmmacros
    $ mkdir -p ~/rpmbuild/{BUILD,RPMS,S{OURCE,PEC,RPM}S}
    $ cat > ~/.rpmmacros <<< "%_topdir $HOME/rpmbuild"
    
    $ rpm -ivh /tmp/cache/$srpm
    
    
  11. Remove the unpacked old Gearman tarball and copy the new one to SOURCES:
  12. $ cd ~/rpmbuild
    $ cp /tmp/cache/gearmand-$lastver.tar.gz SOURCES/
    $ rm SOURCES/gearmand-$srcver.tar.gz
    
  13. Now the trick. The sed command below performs some changes on gearmand.spec file: 1) replaces the old version number with the new one; 2) Comments dependencies from systemd packages not available yet on CentOS; and 3) Adds various file/directory entries only available on the latest version of Gearman.
  14. $ sed -i \
      -e 's/\(Version:[[:space:]]\+\)'${srcver}'/\1'${lastver}'/' \
      -e 's/^BuildRequires:[[:space:]]\+systemd-units/#&/' \
      -e 's/^Requires(post):[[:space:]]\+systemd-sysv/#&/' \
      -e 's/^Requires(post):[[:space:]]\+systemd-units/#&/' \
      -e 's/^Requires(preun):[[:space:]]\+systemd-units/#&/' \
      -e 's/^Requires(postun):[[:space:]]\+systemd-units/#&/' \
      -e 's/install -m 0644 %{SOURCE3} %{buildroot}%{_unitdir}\/%{name}.service/#&/' \
      -e 's/^%changelog/%files\n%defattr(-,root,root,-)\n\/usr\/include\/libgearman-1.0\n\/etc\/rc.d\/init.d\/gearmand\n\/etc\/sysconfig\/gearmand\n\/usr\/bin\/gearadmin\n\/usr\/bin\/gearman\n\/usr\/sbin\/gearmand\n\/usr\/share\/man\n\n&/' \
      SPECS/gearmand.spec
    
  15. Build and install the Gearman RPM packages:
  16. $ rpmbuild -bb SPECS/gearmand.spec
    
    $ sudo rpm -ivh RPMS/x86_64/libgearman-$lastver-1.el6.x86_64.rpm RPMS/x86_64/gearmand-$lastver-1.el6.x86_64.rpm
    
    
  17. Register gearmand as a daemon on standard runleves:
  18. $ sudo chkconfig --add gearmand
    $ sudo chkconfig gearmand on
    
  19. Create an emtpy log archive with permissions for gearmand user amd group (created by the RPM installer), crate pid directory and start the daemon:
  20. $ sudo mkdir -p /usr/local/var
    $ sudo ln -s /var/log /usr/local/var/log 
    
    $ sudo touch /var/log/gearmand.log
    $ sudo chown gearmand:gearmand /var/log/gearmand.log
    
    $ sudo mkdir -p /var/run/gearmand/
    
    $ sudo /etc/init.d/gearmand start
    

Testing

  1. Use the examples provided by Gearman to test. Enter examples directory and run the reverse_worker example:
  2. $ cd BUILD/gearmand-$lastver/examples
    
    $ ./reverse_worker 
    
    
  3. Open a new terminal, enter examples directory and run the reverse_client:
  4. cd ~/rpmbuild/BUILD/gearmand-$lastver/examples
    
    $ ./reverse_client "Hello Gearman"
    

    in the reverse_worker terminal you should see:
    Recieved 12 bytes
    Job=H:centosarq62.localdomain:1  Reversed=HelloGearman
    
    

Now your Gearman with the latest features is ready for battle.

Enjoy it!