Ed's tech corner

Friday, October 7, 2016

Spring Sandbox Welcome

Just a set of examples of the state-of-the-art Spring ecosystem. Mostly Spring Boot and Spring Platform IO.

Check it out @ https://github.com/eduardo-lago-aguilar/spring-sandbox

Thursday, July 28, 2016

JAngular launched

This is set of matchers for TDD practicing developers using AngularJS + UI-Router.

JAngular

Friday, April 11, 2014

How to install collectd & graphite on Ubuntu and CentOS

This post is a direct approach to install collectd & graphite on Ubuntu, by direct I mean using the available repositories:

Requeriments

- Ubuntu 13.10 installed and configured with standard repositories
- CentOS 6.3 + EPEL repository configured
- Login as root

Ubuntu guide

(i) Install collectd daemon:

# apt-get install collectd

(here you may modify the collectd config file and select the metrics to harvest)

(ii) Install apache2 & graphite, disable default site and enable graphite site:

# apt-get install apache2 libapache2-mod-wsgi
# apt-get install graphite-carbon graphite-web

(iii) Config graphite site, disable default site and enable graphite site:

# cd /etc/apache2/sites-available/
# ln -s /usr/share/graphite-web/apache2-graphite.conf
# a2dissite 000-default
# a2ensite apache2-graphite

(iv) Initial graphite DB creation:

# cd /usr/share/pyshared/graphite
# python manage.py syncdb

(v) Configure carbon cache:

# nano /etc/default/graphite-carbon

(and change flag to true)

CARBON_CACHE_ENABLED=true

(vi) Setup carbon-cache to autostart, then start it:

# chkconfig --levels 235 carbon-cache on
# /etc/init.d/carbon-cache start

(vii) Setup collectd to autostart:

# chkconfig --levels 235 collectd on
# /etc/init.d/collectd start

(viii) Restart apache2:

# service apache2 restart

CentOS guide

(i) Install & config collectd daemon:

# yum install collectd pyhton-bucky httpd
# cp /etc/collectd.conf /etc/collectd.conf.org

(here you may modify the collectd config file and select the sts to harvest)

(ii) Install apache2 & graphite, config apache to autostart:

# yum install graphite-web python-carbon
# chkconfig --levels 235 httpd on
# /etc/init.d/httpd start

(iii) Initial graphite DB creation:

# cd /usr/lib/python2.6/site-packages/graphite
# python manage.py syncdb

(iv) configure carbon cache:

# chkconfig --levels 235 carbon-cache on
# /etc/init.d/carbon-cache start

(v) restart apache:

# /etc/init.d/httpd restart

(vii) Setup collectd to autostart:

# chkconfig --levels 235 collectd on
# /etc/init.d/collectd start

At this point you are already collecting metrics!

Enjoy it!

Friday, August 9, 2013

Spring CDI Mocks: Separating Unit Tests and Integration Tests. The right things in the right place

INTRO

In the last post I clearly became more independently of Spring, not because I hate him, in fact I like Spring a lot. But to postpone the choice of the wiring/infrastructure technology until the production stage, and to encourage standardization and simplicity.

Just for that? No!

In many domains, all bleeding-edge Java framework/toolkit seems to honor the JSR specifications, at the same time JSR specifications very often receive continuous feedback from concrete implementations, some examples are: JPA (JSR-220) with Hibernate, Bean Validation (JSR-303) with Hibernate Validator and Apache Beans Validator, and our main focus CDI (JSR-299) and Spring. Spring actually honors CDI annotations, and if the leading framework on DI (Spring) honors the standard, then there is a big chance for both to be aligned with the future.

As you may noticed I used Spring [before] on the testing phase to wire up the SUT and dependencies with a relative small complexity. Nevertheless simple and small is better, that's why I'll show you how achieve a very similar wiring scenario w/o Spring (in testing phase) with a more simple approach IMO.

There's an open question: Do we still need Spring? Where does it fit?

The trivial answer is: Yes there's no a DI framework as mature as Spring. It fits in the Integration-Test (IT) phase very well.

NOTE: Take these things into account when you read this post: 1) it's a simplified scenario, 2) there MANY other phases besides of compilation, test, integration tests, etc. where Spring also shines, 3) Spring is much more than an DI framework.

HANDS ON LAB

So we need to replace the wiring benefits of Spring in the early testing stage (don't confuse with IT) showed on the [last post] with a worthy substitute. What we mean with worthy? Two things: 1) it solves the wiring at this stage, and 2) it's easy to use and reproduce.

Testing Phase

In the Java world many mocking frameworks exists, I performed a research on this topic and found up a mighty combination in the mocking area: PowerMock w/ Mockito.

PowerMock is the nearest similar thing to a silver bullet I have ever seen in the mocking area, you can combine it with Mockito. By the other hand Mockito is very user friendly, simple and easy to learn. I recommend both in combination for more elaborated tests, and only Mockito for simple scenarios like this.

pom.xml


      org.mockito
      mockito-all
      1.9.5
      test

To achieve the same goal simplifying things we need to inject the SUT's dependencies:

EditorTest.java

package demos.sf.editor.test;

import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.only;
import static org.mockito.Mockito.verify;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.runners.MockitoJUnitRunner;

import demos.sf.editor.SpellChecker;
import demos.sf.editor.impl.Editor;

@RunWith(MockitoJUnitRunner.class)
public class EditorTest {

 @InjectMocks
 private Editor editor;

 @Mock
 private SpellChecker spellChecker;

 @Test
 public void testPasteSuccess() {
  editor.paste("Hello everybody!");

  String expected = "Hello everybody!";
  String actual = editor.getText();
  assertEquals(expected, actual);
 }

 @Test
 public void testAddParagraphSpellIsChecked() {
  editor.paste("Hello everybody!");
  verify(spellChecker, only()).check("Hello everybody!");
 }
}

Observe the introduction of:

@Mock: annotates a dependency to be mocked.

@InjectMocks: annotates the SUT to be instantiated and its @Mocks dependencies to get injected.

@RunWith(MockitoJUnitRunner.class): uses the Mockito runner for tests.

Therefore, there's no need for DI framework (neither application context configuration file) in test phase at least for scenarios like this. Of course the injection and mocking capacities of Mockito are limited compared to Spring.

IT Phase

In the IT phase you replace mock objects with real ones, followed by a verification of the real objects' combination performing as you expect. Let's show one of such possible object graph where I use an English implementation for spell checking, dictionary and tokenizer:

By replacing every mock for a real object at time, you gradually introduce a full IT. You can achieve a complete mock-replacement using a top-down approach, there may be other ways of course but I took this one:

(1) test Editor using SpellChecker's mock

(2) replace SpellChecker's mock for EnglishSpellChecker and use Dictionary and Tokenizer mocks, then test Editor

(3) replace Dictionary's mock for OxfordDictionary, then test Editor

(4) replace Tokenizer's mock for EnglishTokenizer, then test Editor

Two new interfaces appear in the demos.sf.editor package, they represent the abstractions mentioned above:

public interface Dictionary {
 String lookup(String word);
}

public interface Tokenizer {
 Collection tokenize(String text);
}

Three new classes appear in the demos.sf.editor.english package, they represent and English wiring combination:

@Named
public class EnglishSpellChecker implements SpellChecker {

 @Resource
 private Tokenizer tokenizer;

 @Resource
 private Dictionary dictionary;

 public String check(String text) {
        ...
 }
}

@Named
public class EnglishTokenizer implements Tokenizer {

 public Collection tokenize(String text) {
  // TODO Auto-generated method stub
  return null;
 }
}

@Named
public class OxfordDictionary implements Dictionary {

 public String lookup(String word) {
  // TODO Auto-generated method stub
  return null;
 }
}

The beauty behind is that out Editor remains the same:

@Named
public class Editor {

 @Getter
 protected String text;

 @Resource
 private SpellChecker spellChecker;

 public void paste(String cut) {
  if (text == null) {
   text = "";
  }
  text += cut;
  spellChecker.check(cut);
 }
}

And there is a change for introducing and test for integration by only changing the IT application context, observe the component-scan with demos.sf.editor.english package which instantiates the desired combination of beans:

it-applicationContext.xml

There are additional aliasing entries due to the fact that the injection is performed on-resource-basis (by bean name), remember @Resource and @Named annotations:

@Named
public class Editor {

 // the following injects a bean named spellChecker
 @Resource
 private SpellChecker spellChecker;
 ...
}

// the following produces a bean named englishSpellChecker
@Named
public class EnglishSpellChecker implements SpellChecker {

 // the following injects a bean named tokenizer
 @Resource
 private Tokenizer tokenizer;

 // the following injects a bean named dictionary
 @Resource
 private Dictionary dictionary;
 ...
}

// the following produces a bean named englishTokenizer
@Named
public class EnglishTokenizer implements Tokenizer {
 ...
}

// the following produces a bean named oxfordDictionary
@Named
public class OxfordDictionary implements Dictionary {
 ...
}

You may want to inject based on type instead of name, it's easy to achieve by replacing the @Resource annotations with @Inject. Lets resume again the meaning of these annotations:

* @Resource:

Matching: First matches by name, then matches by type and filters by qualifiers if not name-based match gets found.
Applicable to: Fields, Classes and Methods.
Package: javax.annotation P { margin-bottom: 0.08in; }A:link { }

@Inject:
- Matching: First matches by type, filters by qualifiers, then matches be name.
- Applicable to: Methods, Constructors and Fields.
- Package: javax.inject

@Named:
- Description: Its a qualifier that denotes a named bean. Annotated classes are considered for auto-detection on classpath scanning. It's similar to Spring's @Component
- Applicable to: Classes and Fields.
- Package: javax.inject

How to change it to Spanish?

1- First implement the three corresponding classes in demos.sf.editor.spanish package, they will represent the desired wiring combination:

@Named
public class SpanishSpellChecker implements SpellChecker {

 @Resource
 private Tokenizer tokenizer;

 @Resource
 private Dictionary dictionary;

 public String check(String text) {
        ...
 }
}

@Named
public class SpanishTokenizer implements Tokenizer {

 public Collection tokenize(String text) {
  // TODO Auto-generated method stub
  return null;
 }
}

@Named
public class RAEDictionary implements Dictionary {

 public String lookup(String word) {
  // TODO Auto-generated method stub
  return null;
 }
}

Create a new IT application context, or change the existing one with a component-scan with demos.sf.editor.spanish package which instantiates the desired combination of Spanish beans :)

A final step (really the first one)

Create the integration test in the package demos.sf.editor.it, it's very similar to the unit test case in the [previous post], the main differences are the use of it-applicationContext.xml as context configuration, and the IT suffix on the test case name:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/it-applicationContext.xml" })
@TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
  DirtiesContextTestExecutionListener.class })
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
public class EditorIT {

 @Resource
 private Editor editor;

 @Test
 public void testPasteWorks() {
  editor.paste("Hello everybody!");

  String expected = "Hello everybody!";
  String actual = editor.getText();
  assertEquals(expected, actual);
 }

}

To honor the IT test case only in the integration test phase the maven-failsafe-plugin may be used, hence the mvn test execution only runs the traditional JUnit test cases and ignores the IT test case classes suffixed with IT:
pom.xml


        maven-failsafe-plugin
        2.13
        
          
            failsafe-integration-tests
            integration-test
            
              integration-test
              verify

To run IT test then execute:

$ mvn verify

The source code can be found here [https://bitbucket.org/eduardo_lago_aguilar/spring-demos] under the directory: spring-hello-word-cdi-mocks.

REFERENCES

CDI / JSR-299:

Tuesday, May 7, 2013

Creating a REST web service using webtoolkit (aka witty)

I previously introduced witty as a powerful web framework for C++ developers, mostly those one that don't want to re-invent the wheel and become productive. Witty uses a resource-based approach to register services that listen on specific paths (resource paths), these resources are registered on the witty server first and the server is started later.

So the steps of an standalone witty w/ custom resource server:

create a server
register a custom resource on a desired path
start the server, hence its resources
wait for server shutdown signal
stop the server

Here is the snippet of the main.cpp, ignore all log-related plumbing:

#include <Wt/WServer>
#include <iostream>
#include "MyResource.h"
#include "log.h"

using namespace std;
using namespace Wt;

int main(int argc, char **argv) {
 WLogger logger;
 configLogger(logger);
 try {
  WServer server(argv[0], "");
  try {
   server.setServerConfiguration(argc, argv);
   MyResource dr;
   server.addResource(&dr, "/resource");
   info(logger, "Starting resource server.");
   if (server.start()) {
    WServer::waitForShutdown();
    server.stop();
   } else {
    fatal(logger, "Fatal error starting resource server.");
    return 1;
   }
   return 0;
  } catch (std::exception& e) {
   fatal(logger, "Fatal error starting resource server.", e.what());
   return 1;
  }
 } catch (WServer::Exception& e) {
  fatal(logger, "Fatal error creating WServer.", e.what());
  return 1;
 } catch (exception& e) {
  fatal(logger, "Fatal error occurred.", e.what());
  return 1;
 }
}

You may notice the creation and registration of MyResource, this is place where you provide the service implementation, specifically on the handleRequest() method, take a look at MyResource.h:

#ifndef MYRESOURCE_H_
#define MYRESOURCE_H_

#include <Wt/WResource>

using namespace Wt;
using namespace Wt::Http;

class MyResource: public WResource {
 public:
  MyResource();
  virtual ~MyResource();
 protected:
  virtual void handleRequest(const Request &request, Response &response);
};

#endif /* MYPRESOURCE_H_ */

The implemantation is very trivial, it's just an echo of the request content also including some extra data like content type, content length and request method: GET, POST, etc. Enjoy MyResource.cpp:

#include <iostream>
#include <Wt/Http/Response>
#include "MyResource.h"

using namespace std;
using namespace Wt::Http;

MyResource::MyResource() {
}

MyResource::~MyResource() {
}

void MyResource::handleRequest(const Request& request, Response& response) {
 string method = request.method();
 string contentType = request.contentType();
 int contentLength = request.contentLength();
 char* buffer = new char[contentLength + 1];
 request.in().read(buffer, contentLength);
 buffer[contentLength] = 0;
 response.setMimeType("application/xml");
 ostream& out = response.out();
 out << "<?xml version='1.0' encoding='utf-8' ?>" << endl;
 out << "<reply>" << endl;
 out << "<method>" << method << "</method>" << endl; 
 out << "<contentType>" << contentType << "</contentType>" << endl;
 out << "<contentLenght>" << contentLength << "</contentLenght>" << endl;
 out << "<body>" << buffer << "</body>" << endl;
 out << "</reply>";
 delete[] buffer;
}

The reamining stuff is just plumbing as I said before, here you got log.h:

#ifndef LOG_H_
#define LOG_H_

#include <Wt/WLogger>

using namespace std;
using namespace Wt;

void info(WLogger& logger, const string& message);
void fatal(WLogger& logger, const string& message, const char* what);
void fatal(WLogger& logger, const string& message);
void configLogger(WLogger& logger);

#endif /* LOG_H_ */

And log.cpp:

#include "log.h"
#include <iostream>

void info(WLogger& logger, const string& message) {
 WLogEntry entry = logger.entry("info");
 entry << WLogger::timestamp << WLogger::sep << WLogger::sep << '[' << "info"
   << ']' << WLogger::sep << message;
}
void fatal(WLogger& logger, const string& message, const char* what) {
 WLogEntry entry = logger.entry("fatal");
 entry << WLogger::timestamp << WLogger::sep << WLogger::sep << '['
  << "fatal" << ']' << WLogger::sep << message << what;
}

void fatal(WLogger& logger, const string& message) {
 fatal(logger, message, "");
}

void configLogger(WLogger& logger) {
 logger.addField("datetime", false);
 logger.addField("type", false);
 logger.addField("message", true);
 logger.setFile("/var/log/resource.log");
}

Now compile an run the server, here in this case using standalone server mode instead of fast cgi mode, but also works w/ fast cgi variant:

$ g++ log.cpp MyResource.cpp main.cpp -lwthttp -oresource

Run the server:

$ ./resource --http-address 0.0.0.0 --http-port 80 --docroot=.
INFO: Opened log file (/var/log/resource.log).
[2013-May-07 19:52:11.046985] 9658 - [info] "config: reading Wt config file: /etc/wt/wt_config.xml (location = './resource')"
[2013-May-07 19:52:11.047757] 9658 - [info] "WServer/wthttp: initializing built-in wthttpd"
[2013-May-07 19:52:11.048027] 9658 - [info] "wthttp: started server: http://0.0.0.0:80"

Finally test the service using a simple call:

$ curl -X POST -H "Content-Type: application/xml" -d"<payload>PAYLOAD GOES HERE!</payload>" http://localhost/resource

<?xml version='1.0' encoding='utf-8' ?>
<reply>
<method>POST</method>
<contentType>application/xml</contentType>
<contentLenght>37</contentLenght>
<body><payload>PAYLOAD GOES HERE!</payload></body>
</reply>

Very simple as you can see!

Thursday, May 2, 2013

ImageMagick convert to raw RGB format in C++

This is a code snippet to convert from any supported ImageMagick format to raw RGB format, it's just an example on how to use the conversion features of ImageMagick libraries. The example was tested on both: Windows and Linux.

#include <Magick++.h>
#include <iostream>
#include <boost/filesystem.hpp>

using namespace std;
using namespace Magick;
using namespace boost::filesystem;

int main(int argc, char **argv) {
 Image img;
 string src_path;
 string tgt_path;
 Blob blob;
 FILE* tgt_file;

 InitializeMagick(*argv);

 // read image
 src_path = argv[1];
 if(exists(src_path)) {
  img.read(src_path);
  cout << "Detected format: " << img.format() << endl;

  // set raw RGBS output format & convert it into a Blob
  img.magick("RGB");
  img.write(&blob);

  // dump blob to disk
  tgt_path = src_path + ".RGB.raw";
  tgt_file = fopen(tgt_path.c_str(), "wb");
  fwrite(blob.data(), blob.length(), 1, tgt_file);
  fclose(tgt_file);
  cout << "Converted to raw RGB: " << tgt_path << endl;
  exit(0);
 } else {
  cout << "Could not load image, file not found " << src_path << endl;
  exit(1);
 }
}

Recording dialog-based interactive user input using expect on Linux

This is a how to successfully combine the shell script dialog tool with expect in order to record all user interactions and repeat them later. You can apply expect to record almost anything on a terminal, I applied it on a very complex dialog-based installation wizard with 100% success. There's nothing special here, nor a magic spell. The solution is pretty straightforward: start recording, launch dialogs + interact, stop record and finally replay it on a hit.

Show the example dialogs below:

#!/bin/bash
#
# dlgs.sh : all dialogs in one script, 1- a radio list, 2- a Yes/No dialog, 3-An input box


  
radiolist_result=$(dialog --stdout --clear --backtitle "Combining dialog w/ expect" \
   --radiolist "Select distro:" 10 40 3 1 "CentOS" off 2 "Ubuntu" on 3 "Debian" off)

dialog --stdout --clear --title "Simple question" --backtitle "Combining dialog w/ expect" \
  --yesno "Are you having fun?" 6 25 && yesno_result=Yes || yesno_result=No

inputbox_result=$(dialog --stdout --clear  --backtitle "Combining dialog w/ expect" \
  --inputbox "Enter your name:" 8 40)

I use CentOS 6 for this demo, but it seems to work on other distros:

$ yum -y install expect

Assign execution permissions to dlgs.sh:

$ chmod +x dlgs.sh

Start recording using autoexpect:

$ autoexpect
autoexpect started, file is script.exp

Call dlgs.sh and interact:

$ ./dlgs.sh
...
3
No
Eduardo Lago Aguilar

By pressing Ctrl+D end the session watch. Then verify the recording by calling the generated script:

$ ./script.exp
...
3
No
Eduardo Lago Aguilar

Very simple isn't? See more.

Thursday, February 21, 2013

Becoming unaware of the Dependency Injection Framework. Encouraging standarization

Intro

The previous post showed up how to combine Spring Framework, Mockito and Lombok in TDD fashion. But it's far from be the simplest and portable solution if such thing even exists. Let's try to make the whole example more unaware of the DI Framework, in this case Spring, but still use it for testing purposes.

Java 6 brought us several improvements, CDI among them. This refcardz is self-explanatory. IMO the main advantage of CDI is standardization, but CDI is pure specification, not an implementation, and its RI is Weld.

Results that Spring honors CDI since version 3.0, at least in Java Injection Standard (JSR-330). There is a bunch of posts and articles talking about the Spring support for CDI specifications, and which one is better to use. But this is not the goal of this post.

Goals

Become unaware of the DI Framework at design time
Support Spring at test time
Support Spring at runtime
Support others DI Framework at runtime

How to

By replacing @Autowired and using injection by name with @Resource
By naming beans using @Name and scanning packages for candidates instead of explicitly create the bean in application context configuration file.

Let's do it

As I said before, it's just an example. Real world scenarios are more complex, and you will end for sure combining many APIs from different sources.

Use the example from the previous post.
Include a couple of dependencies in your pom.xml.


  javax.annotation
  jsr250-api
  1.0


  javax.inject
  javax.inject
  1

JSR-250

@Resource, @PostConstruct, @PreDestroy

JSR-333

@Inject, @Named, @Provider, @Qualifier, @Scope and @Singleton

@Resource:

Matching: First matches by name, then matches by type and filters by qualifiers if not name-based match gets found.
Applicable to: Fields, Classes and Methods.
Package: javax.annotation

@PostConstruct:

Runs: After the bean construction and wiring
Applicable to: Methods.
Package: javax.annotation

@PreDestroy:

Runs: Before the release of the bean
Applicable to: Methods.
Package: javax.annotation

@Inject:

Matching: First matches by type, filters by qualifiers, then matches be name.
Applicable to: Methods, Constructors and Fields.
Package: javax.inject

@Named:

Description: Its a qualifier that denotes a named bean. Annotated classes are considered for auto-detection on classpath scanning. It's similar to Spring's @Component
Applicable to: Classes and Fields.
Package: javax.inject

Spring dependencies are now on test scope, since they are only required on test time. But in a real world scenario you will need a DI Framework, so don't forget to include Spring or Weld dependencies for production stage.


  ...
 
  org.springframework
  spring-test
  ${spring.version}
  test
 
 
  org.springframework
  spring-beans
  ${spring.version}
  test
 
 
  org.springframework
  spring-context
  ${spring.version}
  test
 
  ...

The test is an hybrid between Spring and more standard stuff, that is:

package demos.sf.editor.test;

import ... ;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/test-applicationContext.xml" })
@TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
  DirtiesContextTestExecutionListener.class })
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
public class EditorTest {

 @Resource
 private Editor editor;

 @Resource
 private SpellChecker spellChecker;

 @Test
 public void testPasteSuccess() {
  editor.paste("Hello everybody!");

  String expected = "Hello everybody!";
  String actual = editor.getText();
  assertEquals(expected, actual);
 }

 @Test
 public void testAddParagraphSpellIsChecked() {
  editor.paste("Hello everybody!");
  verify(spellChecker, only()).check("Hello everybody!");
 }
}

The SUT (Editor) is now DI Framework unaware:

package demos.sf.editor.impl;

import ...;

@Named
public class Editor {

 @Getter
 protected String text;

 @Resource
 private SpellChecker spellChecker;

 public void paste(String cut) {
  if (text == null) {
   text = "";
  }
  text += cut;
  spellChecker.check(cut);
 }
}

The application context configuration file for testing stage file is now simplified. Only package scanning is needed to create the SUT (Editor):

NOTE

The final snapshot can be found here under the spring-hello-word-cdi directory. Enjoy it!

Wednesday, December 19, 2012

A TDD approach using Spring Framework + Mockito + Lombok

Intro

This is simple Hello World Java application that successfully combines Spring Framework with Mockito and Lombok. It's just for demonstrative purposes, so the presented example is very simple and used many times before in the literature.

The final goal is to show up how simple is to combine some of the best features of each involved technology with Test-Driven-Development in mind:

Spring Framework: wiring
Mockito: isolation
Lombok: simplicity

Example

A text editor uses and spell checker among other things to make the life of user easier. Depending on the target language, an spell editor may change is behavior and its dependencies as well, for example dictionaries and tokenizers. Simplifying the whole idea you may encounter a contract-driven (interface based) solution like this:

with many possible combinations of spell checkers, dictionaries and tokenizers. For example this one:

where every contract was replaced by a concrete instance. Combinations are multiple, and sometimes you cannot predict all them in a classes/interfaces design.

You want to start engaging TDD for sure, and you want to do it in a top-down fashion. That means, start testing the text editor first, then the spell checker, then the remaining pieces. You also want to do it in isolation, so spell checker and the remaining implementations aren't involved in editor tests, and it's your desire to obtain zero infrastructure code (no constructor calls). As a plus no getter nor setter nor trivial constructor implementations.

What to test?

Our test will be very simple as well, here is the top-down ordered list (remember it's a simplification of the reality):

1- Given an Editor, when a new text is pasted it should be added.
2- Given an Editor, when a new text is pasted the spell should be checked against the appended text.
3- Given an English spell checker, when a word is not found in a dictionary during the spell check then it must be signaled as misspelled and returned back.

You may be wondering, why should I test these obvious use cases? Trust me, it's very important to cover all possible scenarios if you really want to delivery software with built-in quality. This is the hidden synergy behind TDD.

Hands on Spring Tool Suite

Open STS and create a new Maven project.
Check 'Create a simple project (skip archetype selection)'.
Enter a Group Id = demos.sf.editor, Artifact Id = spring-hello-world and Version = 1.0.
Edit your pom.xml and append all the required dependencies. It should ends like:


  4.0.0
  demos.sf.editor
  spring-hello-word
  1.0
  Spring Framework Hello World
  Spring Framework Hello World demo w/ Unit Tests
  
    UTF-8
    3.2.0.RELEASE
  
  
    
      junit
      junit
      4.10
    
    
    
      org.springframework
      spring-test
      ${spring.version}
    
    
    
      org.springframework
      spring-beans
      ${spring.version}
    
    
    
      org.springframework
      spring-context
      ${spring.version}
    
    
      org.mockito
      mockito-all
      1.9.5
    
    
      org.projectlombok
      lombok
      0.11.6

Start testing! A golden rule, don't forget it. Create a new JUnit 4 Test Case at src/test/java. Name it EditorTest in package demos.sf.editor.
Delete the existing test, called test and append a new failing test for the first use case. Its name testPasteSuccess:

package demos.sf.editor;

import static org.junit.Assert.fail;

import org.junit.Test;

public class EditorTest {

 @Test
 public void testPasteSuccess() {
  fail();
 }

}

Write the assertion first. How's that? See use case 1: "... the text should be added":

...
import static org.junit.Assert.assertEquals;

public class EditorTest {

 @Test
 public void testPasteSuccess() {
  String expected = "Hello everybody!";
  String actual = editor.getText();
  assertEquals(expected, actual);
  fail();
 }
}

Right-click on editor and choose "Create local variable...". Change its type from Object to Editor, a non-existing class:

public class EditorTest {

 @Test
 public void testPasteSuccess() {
  Editor editor;

  String expected = "Hello everybody!";
  String actual = editor.getText();
  assertEquals(expected, actual);
  fail();
 }
}

Editor

Annotate the Editor class for auto-implementing the getter using Lombok annotations:

package demos.sf.editor;

import lombok.Getter;

public class Editor {
 @Getter
 private String text;
}

NOTE

-javaagent:/home/lago/Soft/springsource2.9.2/sts-2.9.2.RELEASE/lombok.jar
-Xbootclasspath/a:/home/lago/Soft/springsource2.9.2/sts-2.9.2.RELEASE/lombok.jar

$ java -jar ~/.m2/repository/org/projectlombok/lombok/0.11.6/lombok-0.11.6.jar

Go back to the test, and perform the call to a non-existing paste() method:

public class EditorTest {

 @Test
 public void testPasteSuccess() {
  Editor editor;
  editor.paste("Hello everybody!");

  String expected = "Hello everybody!";
  String actual = editor.getText();
  assertEquals(expected, actual);
  fail();
 }
}

Ctrl+1 on top the editor.paste("...") call and choose "Create method paste()....". A new method is generated:

public class Editor {
 @Getter
 private String text;

 public void paste(String cut) {
 }
}

In the test, the only missing thing is the editor initialization. Let's inject the Editor at test time. Convert the editor local variable to a field declaration and annotate it as @Autowired:

...
import org.springframework.beans.factory.annotation.Autowired;

public class EditorTest {

 @Autowired
 private Editor editor;

 @Test
 public void testPasteSuccess() {
  editor.paste("Hello everybody!");

  String expected = "Hello everybody!";
  String actual = editor.getText();
  assertEquals(expected, actual);
  fail();
 }
}

Create a new Spring Bean Configuration file at src/test/resources/test-applicationContext.xml and declare the editor bean on it:

Use the Spring test runner and load the application context via annotations:

...
import org.junit.runner.RunWith;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/test-applicationContext.xml" })
public class EditorTest {
...
}

Right click on test case and Run As JUnit test. The test will fail but corner stones are ready to support more agile tests. Make the test pass by implementing the paste() method. Oops! remember to remove the fail() from the test.
```
public class Editor {
 @Getter
 private String text;

 public void paste(String cut) {
  if(text == null) {
   text = "";
  }
  text += cut;
 }
}
```
Re-run the test, this time also using a Maven run configuration like this:

mvn test

The spring default behavior is to cache the application context between different tests in the same test case. They also introduced the annotations @TestExecutionListeners and @DirtiesContext to mark the application context as dirty after each test:

...
import org.springframework.test.context.TestExecutionListeners;
import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;
import org.springframework.test.context.support.DirtiesContextTestExecutionListener;
import org.springframework.test.annotation.DirtiesContext;
import org.springframework.test.annotation.DirtiesContext.ClassMode;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/test-applicationContext.xml" })
@TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
 DirtiesContextTestExecutionListener.class })
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
public class EditorTest {
...
}

Now starts the use case 2. First append the test in failing mode:

...
public class EditorTest {
 ...
 @Test
 public void testAddParagraphSpellIsChecked() {
  fail();
 }
}

Using Mockito style, we need to verify that the spell is checked against the pasted text. The syntax is straightforward it means that check() method must be called only once with "Hello everybody!":

...
import static org.mockito.Mockito.only;
import static org.mockito.Mockito.verify;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/test-applicationContext.xml" })
@TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
  DirtiesContextTestExecutionListener.class })
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
public class EditorTest {

 ...

 @Autowired
 private SpellChecker spellChecker;

 @Test
 public void testAddParagraphSpellIsChecked() {
  editor.paste("Hello everybody!");
  verify(spellChecker, only()).check("Hello everybody!");
  fail();
 }
}

At this time we need to create a new SpellChecker interface:

package demos.sf.editor;

public interface SpellChecker {

 void check(String text);

}

The next step is to provide an spell checker mock to the application context:

Remove the fail() from the test and run all tests and you will get a test failure saying: 'Wanted but no invoked: spellChecker.check("Hello everybody!")'. That makes sense since we are not invoking the SpellChecker.check() from Editor.paste(). In fact haven't create any kind of dependency Editor -> SpellChecker. Let's do it at class level and force its injection at construction time using Lombok's @RequiredArgsConstructor:

package demos.sf.editor;

import lombok.Getter;
import lombok.RequiredArgsConstructor;

@RequiredArgsConstructor
public class Editor {
 @Getter
 private String text;

 private final SpellChecker spellChecker;

 public void paste(String cut) {
  if (text == null) {
   text = "";
  }
  text += cut;
 }
}

Declare the constructor injection in the application context as well and re-run the tests:

Oops! The same failure: 'Wanted but no invoked: spellChecker.check("Hello everybody!")'. Implement the SpellChecker.check() call and run the tests again:

package demos.sf.editor;

import lombok.Getter;
import lombok.RequiredArgsConstructor;

@RequiredArgsConstructor
public class Editor {
 @Getter
 private String text;

 private final SpellChecker spellChecker;

 public void paste(String cut) {
  if (text == null) {
   text = "";
  }
  text += cut;
  spellChecker.check(cut);
 }
}

Final Snapshot

The test case at : src/test/java/demos/sf/editor/EditorTest.java:

package demos.sf.editor;

import static org.junit.Assert.assertEquals;
import static org.mockito.Mockito.only;
import static org.mockito.Mockito.verify;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.test.annotation.DirtiesContext;
import org.springframework.test.annotation.DirtiesContext.ClassMode;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.TestExecutionListeners;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.support.DependencyInjectionTestExecutionListener;
import org.springframework.test.context.support.DirtiesContextTestExecutionListener;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = { "/test-applicationContext.xml" })
@TestExecutionListeners({ DependencyInjectionTestExecutionListener.class,
  DirtiesContextTestExecutionListener.class })
@DirtiesContext(classMode = ClassMode.AFTER_EACH_TEST_METHOD)
public class EditorTest {

 @Autowired
 private Editor editor;

 @Autowired
 private SpellChecker spellChecker;

 @Test
 public void testPasteSuccess() {
  editor.paste("Hello everybody!");

  String expected = "Hello everybody!";
  String actual = editor.getText();
  assertEquals(expected, actual);
 }

 @Test
 public void testAddParagraphSpellIsChecked() {
  editor.paste("Hello everybody!");
  verify(spellChecker, only()).check("Hello everybody!");
 }
}

The test case at : src/main/java/demos/sf/editor/Editor.java:

package demos.sf.editor;

import lombok.Getter;
import lombok.RequiredArgsConstructor;

@RequiredArgsConstructor
public class Editor {
 @Getter
 private String text;

 private final SpellChecker spellChecker;

 public void paste(String cut) {
  if (text == null) {
   text = "";
  }
  text += cut;
  spellChecker.check(cut);
 }
}

The test case at : src/main/java/demos/sf/editor/SpellChecker.java:

package demos.sf.editor;

public interface SpellChecker {

 void check(String text);

}

The test case at : src/main/resources/test-applicationContext.xml:

You can obtain the source code from here. Enjoy it!

Tuesday, December 11, 2012

Running webtoolkit application as nginx FastCGI on CentOS 6.x

Wt (pronounced as witty) is a powerful C++ library for developing web applications. Witty-based apps can be integrated as FastCGI with nginx and other web servers. This guide is about the integration of witty w/ nginx on CentOS 6.x OS.

On this how to, we'll be using the simplest witty example: hello

I assumed you already had installed a CentOS 6.x minimal x86_64.

The Steps

Login as a sudoer user.
Append EPEL repository by creating the file /etc/yum.repos.d/epel.repo with the following content:

[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0

Install required packages: nginx, witty and CentOS development kit:

$ sudo yum install nginx 
$ sudo yum install wt
$ sudo yum install fcgi
$ sudo yum install spawn-fcgi
$ sudo yum install wt-devel wt-examples      # Only for development env
$ sudo yum groupinstall "Development Tools"  # Only for development env
$ sudo yum install nano                      # Only for development env

Go to Wt's examples directory and edit the CMakeLists.txt archive:

$ cd /usr/lib64/Wt/examples/hello
$ sudo nano CMakeLists.txt

WT_ADD_EXAMPLE(hello.wt hello.C)

ADD_EXECUTABLE(hello.wt hello.C)
TARGET_LINK_LIBRARIES(hello.wt ${EXAMPLES_CONNECTOR})

Run CMake specifying FastCGI support & copy resulting binary to nginx document root:

$ sudo rm -rf target
$ sudo mkdir -p target
$ cd target
$ sudo cmake ../ -DEXAMPLES_CONNECTOR=wtfcgi -DCONNECTOR_FCGI=yes -DCONNECTOR_HTTP=no 
$ sudo make
$ sudo cp -a hello.wt /usr/share/nginx/html/

Create a new archive /etc/sysconfig/spawn-fcgi-hello.wt with the following content:

FCGI_SOCKET=/var/run/hello.wt.socket
FCGI_PROGRAM=/usr/share/nginx/html/hello.wt
FCGI_USER=nginx
FCGI_GROUP=nginx
FCGI_EXTRA_OPTIONS="-M 0700"
OPTIONS="-u $FCGI_USER -g $FCGI_GROUP -s $FCGI_SOCKET -S $FCGI_EXTRA_OPTIONS -F 1 -P /var/run/hello.wt.socket.pid -- $FCGI_PROGRAM"

Allow nginx to write at /var/spool/wt/run/:

$ sudo chgrp nginx /var/spool/wt/run/
$ sudo chmod g+w /var/spool/wt/run/

Launch hello app via spawn-fcgi:

$ source /etc/sysconfig/spawn-fcgi-hello.wt
$ spawn-fcgi $OPTIONS

Create a new file /etc/nginx/conf.d/wt.conf w/ the following content:

server {
    listen  9091;
    server_name  _;

    # by default relative to /usr/share/nginx/html
    location / {
      access_log /var/log/nginx/nginx-fastcgi-access.log;
      gzip off;

      # the full path /usr/share/nginx/html/hello.wt
      if ($uri !~ "^/hello.wt/$") {
        fastcgi_pass unix:/var/run/hello.wt.socket;
      }
      include /etc/nginx/fastcgi_params;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    }
}

Restart nginx:

$ service ngnix restart

Visit http://HOST:9091 and enjoy it!

Wednesday, August 8, 2012

Packaging Csync2 in RPM on CentOS 6.X

This is a how to package in RPM format, the Csync2 synchronization tool on CentOS 6.x.

Steps

Append RPM Forge repository by creating the file /etc/yum.repos.d/rpmforge.repo with the following content:

$ sudo dd of=/etc/yum.repos.d/rpmforge.repo << EOT
[rpmforge]
name=CentOS-$releasever - Rpmforge
baseurl=http://apt.sw.be/redhat/el6/en/$basearch/rpmforge
gpgcheck=0
enabled=1
EOT

Install building dependencies:

$ sudo yum -y groupinstall "Development Tools"
$ sudo yum -y install openssl-devel 
$ sudo yum -y install librsync librsync-devel
$ sudo yum install gnutls openssl libtasn1 gnutls-devel

Create a temporal cache and define a downloader command:

$ mkdir -p /tmp/cache

#/** 
# * Downloads a file to the cache if doesn't exists
# *
# * @param $1 the file to download
# * @param $2 the url where the file is located
# */
$ get() {
 [ -f /tmp/cache/$1 ] || wget -t inf -w 5 -c $2/$1 -O /tmp/cache/$1
}

Download latest csync2 tarball with sources:

$ lastver=1.34
$ cs=csync2-$lastver.tar.gz
$ get $cs http://oss.linbit.com/csync2/$cs

Download an sqlite compatible version:

$ sqver=2.8.16
$ sq=sqlite-$sqver.tar.gz
get $sq http://pkgs.fedoraproject.org/repo/pkgs/sqlite/$sq/9c79b461ff30240a6f9d70dd67f8faea/$sq

Create a clean RPM build environment:

$ rm -rf ~/rpmbuild ~/.rpmmacros
$ mkdir -p ~/rpmbuild/{BUILD,RPMS,S{OURCE,PEC,RPM}S}
$ cat > ~/.rpmmacros <<< "%_topdir $HOME/rpmbuild"

Copy the sources tar balls to SOURCES and BUILD:

$ cd ~/rpmbuild
$ cp /tmp/cache/$cs SOURCES/
$ cp /tmp/cache/$sq BUILD/

Extract the csync2.spec archive from tar ball and modify it to user sqlite sources and include some missing files into the final RPM:

$ cd SPECS
$ tar --strip-components=1 -x csync2-$lastver/csync2.spec -vzf ../SOURCES/$cs
$ sed -i \
  -e 's/^%changelog/%files\n%defattr(-,root,root,-)\n\/usr\/sbin\/csync2-compare\n\/etc\/csync2.cfg\n\/etc\/xinetd.d\/csync2\n\/usr\/sbin\/csync2\n\/usr\/share\/man\/man1\/csync2.1.gz\n\n&/' \
  -e 's/\(^%configure\)/\1  --with-libsqlite-source=..\/sqlite-2.8.16.tar.gz  --disable-gnutls/' csync2.spec
$ cd ..

Build and install the csync2 RPM packages:

$ rpmbuild -bb SPECS/csync2.spec
$ sudo rpm -ivh RPMS/x86_64/csync2-$lastver-1.x86_64.rpm

Enjoy it!

Thursday, July 5, 2012

Building and packaging the latest Gearman server in CentOS 6.2

This is a how to package, install and test the latest version of Gearman server in RPM format using CentOS 6.2.

Motivation

The latest version available of Gearman in the Fedora repository is very outdated, CentOS is even more. So, if you plan to use the latest Gearman features, you have two choices 1) compile it using the tarball; 2) package it (hence compile it) in RPM format.
This guide uses the second approach, but not from scratch. Instead using the latest available SRPM German's package and performing some minor changes.

At this moment, the latest Gearman version is 0.33 and the latest Fedora-based SRPM is 0.23.

Hands on bash

Append EPEL repository by creating the file /etc/yum.repos.d/epel.repo with the following content:

[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=1
gpgcheck=0

[epel-debuginfo]
name=Extra Packages for Enterprise Linux 6 - $basearch - Debug
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch/debug
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-debug-6&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=0

[epel-source]
name=Extra Packages for Enterprise Linux 6 - $basearch - Source
#baseurl=http://download.fedoraproject.org/pub/epel/6/SRPMS
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-source-6&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=0

Install building dependencies:

$ sudo yum groupinstall "Development Tools"
$ sudo yum install libevent-devel libuuid-devel
$ sudo yum install boost-devel
$ sudo yum install libmemcached-devel memcached google-perftools-devel

Create a temporal cache and define a downloader command:

$ mkdir -p /tmp/cache

#/** 
# * Downloads a file to the cache if doesn't exists
# *
# * @param $1 the file to download
# * @param $2 the url where the file is located
# */
$ get() {
 [ -f /tmp/cache/$1 ] || wget -t inf -w 5 -c $2/$1 -O /tmp/cache/$1
}

Download latest Gearman tarball with sources and the latest SRPM package provided by Fedora

$ lastver=0.33
$ gm=gearmand-$lastver.tar.gz
$ get $gm https://launchpadlibrarian.net/104788829/$gm 

$ srcver=0.23
$ srpm=gearmand-$srcver-1.fc16.src.rpm
$ get $srpm http://www.muug.mb.ca/mirror/fedora/linux/releases/16/Everything/source/SRPMS/$srpm

Create a clean RPM build environment and install the SRPM package on it:

$ rm -rf ~/rpmbuild ~/.rpmmacros
$ mkdir -p ~/rpmbuild/{BUILD,RPMS,S{OURCE,PEC,RPM}S}
$ cat > ~/.rpmmacros <<< "%_topdir $HOME/rpmbuild"

$ rpm -ivh /tmp/cache/$srpm

Remove the unpacked old Gearman tarball and copy the new one to SOURCES:

$ cd ~/rpmbuild
$ cp /tmp/cache/gearmand-$lastver.tar.gz SOURCES/
$ rm SOURCES/gearmand-$srcver.tar.gz

Now the trick. The sed command below performs some changes on gearmand.spec file: 1) replaces the old version number with the new one; 2) Comments dependencies from systemd packages not available yet on CentOS; and 3) Adds various file/directory entries only available on the latest version of Gearman.

$ sed -i \
  -e 's/\(Version:[[:space:]]\+\)'${srcver}'/\1'${lastver}'/' \
  -e 's/^BuildRequires:[[:space:]]\+systemd-units/#&/' \
  -e 's/^Requires(post):[[:space:]]\+systemd-sysv/#&/' \
  -e 's/^Requires(post):[[:space:]]\+systemd-units/#&/' \
  -e 's/^Requires(preun):[[:space:]]\+systemd-units/#&/' \
  -e 's/^Requires(postun):[[:space:]]\+systemd-units/#&/' \
  -e 's/install -m 0644 %{SOURCE3} %{buildroot}%{_unitdir}\/%{name}.service/#&/' \
  -e 's/^%changelog/%files\n%defattr(-,root,root,-)\n\/usr\/include\/libgearman-1.0\n\/etc\/rc.d\/init.d\/gearmand\n\/etc\/sysconfig\/gearmand\n\/usr\/bin\/gearadmin\n\/usr\/bin\/gearman\n\/usr\/sbin\/gearmand\n\/usr\/share\/man\n\n&/' \
  SPECS/gearmand.spec

Build and install the Gearman RPM packages:

$ rpmbuild -bb SPECS/gearmand.spec

$ sudo rpm -ivh RPMS/x86_64/libgearman-$lastver-1.el6.x86_64.rpm RPMS/x86_64/gearmand-$lastver-1.el6.x86_64.rpm

$ sudo chkconfig --add gearmand
$ sudo chkconfig gearmand on

Create an emtpy log archive with permissions for gearmand user amd group (created by the RPM installer), crate pid directory and start the daemon:

$ sudo mkdir -p /usr/local/var
$ sudo ln -s /var/log /usr/local/var/log 

$ sudo touch /var/log/gearmand.log
$ sudo chown gearmand:gearmand /var/log/gearmand.log

$ sudo mkdir -p /var/run/gearmand/

$ sudo /etc/init.d/gearmand start

Testing

Use the examples provided by Gearman to test. Enter examples directory and run the reverse_worker example:

$ cd BUILD/gearmand-$lastver/examples

$ ./reverse_worker

Open a new terminal, enter examples directory and run the reverse_client:

cd ~/rpmbuild/BUILD/gearmand-$lastver/examples

$ ./reverse_client "Hello Gearman"

Recieved 12 bytes
Job=H:centosarq62.localdomain:1  Reversed=HelloGearman

Now your Gearman with the latest features is ready for battle.

Enjoy it!

Wednesday, July 4, 2012

The NGINX's RAM-only Jail. Adding nginx to RAM-only diskless Linux box

Without any doubt, nginx is one of the best ever created HTTP servers, it solves the C10K problem with an elegant event-drive architecture. Many big sites around the WWW use it: Wikipedia, Hyves, etc.

This is a how to prepare your rambox to run nginx smoothly. It makes sense only after the preparation of the RAM-only PXE boot in my previous post. This post isn't about any kind of nginx optimization stuff beside of its compilation in a single binary rock. At the end of this guide, you will be running the nginx very much like in a jail.

Run these steps BEFORE the step "Change ownership to everything" at RAM-only PXE boot in my previous post.

Download latest stable version nginx and upack it.

$ pushd /tmp/wrk
$ ngx=nginx-1.2.1.tar.gz
$ cache $ngx http://nginx.org/download/$ngx
$ tar -xvzf /tmp/cache/$ngx -C .
$ mv $ngx nginx

Add options depending on the features that you want to support. Beside of the default options I only add static compilation options:

$ pushd nginx
$ ./configure --with-ld-opt="-static -static-libgcc" \
  --with-cc-opt="-static -static-libgcc"

$ make -j2

$ ldd objs/nginx
 not a dynamic executable
$ popd

Copy the nginx executable to rambox's /sbin :

$ popd 
$ pushd sbin
$ chmod +w .
$ cp ../../nginx/objs/nginx .
$ chmod -w .
$ popd

Create /usr/local/nginx directories:

$ mkdir -p -m 0755 usr/local/nginx/{conf,logs}

Copy some needed libs:

$ chmod +w lib
$ cp /lib64/{ld-2.12.so,ld-linux-x86-64.so.2} lib/
$ cp /lib64/{libc-2.12.so,libc.so.6} lib/
$ cp /lib64/{libnsl-2.12.so,libnsl.so.1} lib/
$ cp /lib64/{libnss_compat-2.12.so,libnss_compat.so.2} lib/
$ chmod -w lib

Copy mime conf file:

$ cp ../nginx/conf/mime.types usr/local/nginx/conf/

Create the nginx.conf file with some basic settings:

$ dd of=usr/local/nginx/conf/nginx.conf << EOT
# user and group to run nginx
user www www;

# numbers of dedicated CPUs
worker_processes 1;

# pid archive
pid /var/run/nginx.pid;

events {
    # max connections in WAIT 
    worker_connections  128;
  
    # accept & enqueue NEW connections, put them in WAIT
    multi_accept        on;
}

http {
        include       mime.types;
        default_type  application/octet-stream;

        server {
                listen 80;

                autoindex on;
 
                location / {
                        root /var/www;
                        index  index.php index.html index.htm;
                }
        }
}
EOT

Resume the rambox creation process and once it gets started run:

$ nginx

Enjoy it by browsing at http://HOSTADDRESS

Post install steps

Daemonize nginx by running it at /init script!

Tuesday, July 3, 2012

The Parking Permit Demo. README and Sources for Season 01

Here I let you the README and Sources for the Season 01 of The Parking Permit Demo with Oracle BPM/SOA suite.

Enjoy it!

Thursday, June 7, 2012

RAM-only PXE boot & the "smallest" diskless Linux box

This is a how to easily create a very small Linux box purely running on RAM memory that boots using PXE. This is an introductory topic (not new at all), for further posts about scalability, load balancing and high availability. That's why I mention clustering very often and also start simple & small by preparing a single node that boots smoothly in a controlled environment.

Why RAM-only & diskless?

I must say that such configuration of a Linux box can bring many advantages if you plan to assemble a cheap cluster without persistence in mind and with low maintenance costs. Consider the following:

ram availability: RAM memory is cheaper & fast wrt. the pass years
modern hardware, big ram: A lot of hardware support a great amount RAM memory installed, from 1 GB to 128 GB, and go on
Linux rocks: 64-bits Linux systems are able to manage a lot of RAM memory efficiently
HDD & the planet: Electromechanical HDD have serious implications in energy consumption, recycling, NOISE and are more susceptible to failures
SSD & your wallet: SSD are more advanced wrt. their electromechanical counterparts (less susceptible to physical shock, are silent, and have lower access time and latency) but at present market prices, more expensive per unit of storage. So if you problem is not the storage, just processing, caching and networking, you are in the right place!
less is sometimes cheaper: It's not a bad idea, if you have a good chance to buy cheaper nodes by parts/complete w/o HDD
crashing doesn't matter: if a misbehaving node crashes you just need to restart it and it'll wake up again in a healthy state. A single node state doesn't wander in time
scaling better: adding a node to the cluster is easy, just connect it, enable PXE boot and add an entry in DHCP config
network congestion is reduced: the RAM filesystem is copied once per boot to the target node

Life is easy and cluster's maintenance costs are reduced, but remember that this is only if you don't need persistence in every single node, just CPU power, networking and RAM memory.

When don't I need persistence?

I don't have a full inventory of persistence-less & memory-network-only scenarios, but a practical and discrete list, I'm sure you can see the benefits:

cryptographic stuff, privacy: you need to run a cryptographic algorithm and ensure a full cleanup of private keys after the execution is complete, a HDD formating is not enough sometimes, and recovery data from a RAM memory after a full power of is very difficult if not impossible. Also an encrypted filesystem on top of the RAM shall be challenging for hackers
caching efficiently: if your RAM is enough and your backend cluster is under a constantly growing demand for static content. You can delegate all your caching needs to a dedicated frontend cluster running purely in RAM and release the load of backend servers by processing only dynamic content on this physical layer
time only algorithms: many algorithms have only processing power needs and low/medium memory foot print, some of them even only need volatile (non-persistence) memory for allocating data structures
display only apps: some software solutions only need for displaying incoming data via graphs, video streaming, etc.. So a good display, a RAM-only system and a network is enough

What will I obtain at the end of this guide? A Linux box, named it rambox, purely running in RAM memory, that means a root (/) filesystem mounted in RAM, that's why memory preservation is a priority as well as avoidance of a filesystem full of never-used archives which also increase the memory usage.

We'll also make a customized Kernel compilation to shrink it, with a "minimal" set of features incorporated. Keeping it simple small! At this point you should be careful about omitting mandatory kernel features, there's another set of features that are not mandatory but useful to obtain the best performance. They mainly depend on your hardware, so take care of them.

What's a RAM filesystem? A filesystem mounted on RAM isn't a new invention, is a awesome Kernel feature mostly used to load firmware/modules before starting the normal boot process. It's called initrd or initramfs, there are differences between both (see references) and we'll be using initramfs.

What do I need?

For this guide I use two KVM-virtualized computers, running in a CentOS 6.2 host with bridged networking. For simplicity, the host and the two guests are in the same subnetwork

pxe: a server computer with CentOS 6.2 amd64 installed, w/ 16 GB HDD, 1 GB RAM, no GUI, networking. With DHCP and TFTP role. Static IP = 192.168.24.202, subnet =192.168.24.0/24
rambox: a RAM-only computer, w/o HDD installed, w/ networking. With cluster node role. Dynamic DHCP-designated IP = 192.168.24.203, subnet =192.168.24.0/24

NOTE: Notice that BIOS used for QEMU and KVM virtual machines (the SeaBIOS) supports an open source implementation of PXE, named gPXE. So KVM-based virtual machine is able to boot via network. Now days almost any motherboard should have a BIOS with PXE support. Ensure that your rambox support it by checking the BIOS setup.

How does it work?

In summary, when the rambox with PXE boot activated wake ups:

the BIOS PXE boot loader requests an address to DHCP server
the DHCP server offers an IP address, a TFPT server IP address (himself), and the Linux PXE boot loader's location on the TFTP server
the BIOS PXE boot loader downloads the Linux PXE boot loader from the TFPT server
the Linux PXE boot loader takes control and uses the same IP configuration to connect to TFTP server and fetch two archives: the kernel and the ramdisk
the Kernel takes control and configures its network interface, statically or by performing a second round of DHCP request, it depends
the Kernel uncompress the ramdisk in memory
the RAM disk is mounted on / and the /init script gets invoked

What do we have to configure and where? In pxe server computer is where everything takes place:

Install and configure a DHCP server with support for PXE extensions
Install and configure a TFTP server
Create a reduced ramdisk with a minimal set of utils and programs
Compile and optionally shrink the Kernel to include support for Kernel-level IP configuration, including NIC drivers
Locate all the stuff in the correct place and wake up the rambox!

There are several detailed explanations of the Linux boot process, some of them are outdated but still useful. At the moment, I won't make a full description of every single step of the boot process, ramdisk, PXE, Kernel-level IP, etc. (see references)

Hands on Bash

Installing phase

Install the dhcp, tftp-server and syslinux packages, syslinux contains the Linux PXE boot loader:

$ sudo yum install dhcp tftp-server syslinux

Additionally, install some tools:

$ sudo yum install bc wget

Finally install kernel packages for kernel compilation. These packages ensure that you have all the required tools for the build:

$ sudo yum install kernel-devel
$ sudo yum groupinstall "Development Tools"

# This is required to enable a make *config command to execute correctly. 
$ sudo yum install ncurses-devel

# These are required when building a CentOS-6 kernel. 
$ sudo yum install hmaccalc zlib-devel binutils-devel elfutils-libelf-devel 

# These are required when working with the full Kernel source
$ sudo yum install rpm-build redhat-rpm-config unifdef

# These are needed by kernel-2.6.32-220.el6
$ sudo yun install xmlto asciidoc newt-devel python-devel perl-ExtUtils-Embed

Configure DHCP

Ensure that the dhcpd starts at boot time:

$ sudo chkconfig --level 35 dhcpd on
$ chkconfig --list dhcpd
dhcpd              0:off    1:off    2:off    3:on    4:off    5:on    6:off

Edit dhcp.conf adding PXE specific options:

$ sudo nano /etc/dhcp/dhcpd.conf

# dhcpd.conf
#
# DHCP configuration file for ISC dhcpd
#

# Use this to enble / disable dynamic dns updates globally.
ddns-update-style none;

# Definition of PXE-specific options
# Code 1: Multicast IP address of boot file server
# Code 2: UDP port that client should monitor for MTFTP responses
# Code 3: UDP port that MTFTP servers are using to listen for MTFTP requests
# Code 4: Number of seconds a client must listen for activity before trying
#         to start a new MTFTP transfer
# Code 5: Number of seconds a client must listen before trying to restart
#         a MTFTP transfer
option space PXE;
option PXE.mtftp-ip               code 1 = ip-address;  
option PXE.mtftp-cport            code 2 = unsigned integer 16;
option PXE.mtftp-sport            code 3 = unsigned integer 16;
option PXE.mtftp-tmout            code 4 = unsigned integer 8;
option PXE.mtftp-delay            code 5 = unsigned integer 8;
option PXE.discovery-control      code 6 = unsigned integer 8;
option PXE.discovery-mcast-addr   code 7 = ip-address;

subnet 192.168.24.0 netmask 255.255.255.0 {

  class "pxeclients" {
    match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
    option vendor-class-identifier "PXEClient";
    vendor-option-space PXE;

    # At least one of the vendor-specific PXE options must be set in
    # order for the client boot ROMs to realize that we are a PXE-compliant
    # server.  We set the MCAST IP address to 0.0.0.0 to tell the boot ROM
    # that we can't provide multicast TFTP (address 0.0.0.0 means no
    # address).
    option PXE.mtftp-ip 0.0.0.0;

    # This is the name of the file the boot ROMs should download.
    filename "pxelinux.0";

    # This is the name of the server they should get it from.
    next-server 192.168.24.202;
  }

  pool {
    max-lease-time 86400;
    default-lease-time 86400;
    range 192.168.24.203 192.168.24.203;
    deny unknown clients;
  }

  host rambox {
    hardware ethernet 08:00:07:26:c0:a5;
    fixed-address 192.168.24.203;
    hostname rambox01.home.dev;
  }

}

NOTE

"deny unknown clients"

"allow unknown clients"

Configuring TFPT

To enable the TFTP server, edit /etc/xinetd.d/tftp replacing the word yes on the disable line with the word no. Then save the file and exit the editor:

$ sudo nano /etc/xinetd.d/tftp

# default: off
# description: The tftp server serves files using the trivial file transfer \
# protocol.  The tftp protocol is often used to boot diskless \
# workstations, download configuration files to network-aware printers, \
# and to start the installation process for some operating systems.
service tftp
{
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /var/lib/tftpboot
        disable                 = no
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

Restart the xinetd daemon to reload configuration files:

$ sudo service xinetd restart

Verify if xinetd is started at boot time, it should be, if not then use chkconfig like the previous step:

$ chkconfig --list xinetd
xinetd             0:off    1:off    2:off    3:on    4:on    5:on    6:off

Concerning the firewall

Allow access to TFTP via standard ports:

$ sudo iptables -I INPUT -p udp --dport 69 -j ACCEPT
$ sudo iptables -I INPUT -m state --state NEW -m tcp -p tcp --dport 21 -j ACCEPT
$ sudo service iptables save
$ sudo service iptables restart

Configuring the PXE environment

Copy the Linux PXE boot loader pxelinux.0 to tftpboot published root directory:

$ sudo cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot

Create PXE config directory on TFP root, this directory will contains a single configuration file per node or per subnet:

$ sudo mkdir -p /var/lib/tftpboot/pxelinux.cfg

The Linux PXE boot loader uses its own IP address in hexadecimal format to look for a single configuration file under pxelinux.cfg directory, if its not found it will remove the last octet and try again, repeating until it runs out of octets. That's why I define a helper function to convert IPv4 decimal to an hexadecimal string:

#/**
# * converts an IPv4 address to hexadecimal format completing the missing 
# * leading zero
# * 
# * @example:
# *   $ hxip 10.10.24.203
# *   0A0A18CB
# *
# * @param $1: the IPv4 address 
# */
hxip() {
  ( bc | sed 's/^\([[:digit:]]\|[A-F]\)$/0\1/' | tr -d '\n' ) <<< "obase=16; ${1//./;}"
}

$ hxip 192.168.24.203
C0A818CB

Create PXE Linux config file using the designated IPv4 address in hexadecimal format:
```
$ sudo nano /var/lib/tftpboot/pxelinux.cfg/$(hxip 192.168.24.203)
```
with the following content:
```
DEFAULT bzImage
APPEND initrd=initramfs.cpio.gz rw ip=dhcp shell
```
or if you prefer to avoid the second round of DHCP issued by the Kernel:
```
DEFAULT bzImage
APPEND initrd=initramfs.cpio.gz rw ip=192.168.24.203:192.168.24.202:192.168.24.1:255.255.252.0:rambox:eth0:off shell
```
where DEFAULT provides the Kernel archive and APPEND the Kernel parameters passed on boot:
- bzImage: is the name of the compressed Kernel image
- initrd=initramfs.cpio.gz: tells to Linux PXE boot loader to download this file and pass it to the Kernel later which will interpret it to be a compressed ramdisk filesystem image
- rw: Kernel mounts the ramdisk filesystem in read-write mode
- ip=dhcp: a Kernel-level IP parameter indicating to perform a DHCP request to obtain a valid network parameters, or alternative you can used a fixed network configuration
- ip=192.168.24.203:192.168.24.202:192.168.24.1:255.255.252.0:rambox:eth0:off
- shell: a custom parameter added by me to run a shell

Creating a compressed root filesystem

The Kernel support for initramfs allow us to create a customizable boot process to load modules and provide a minimalistic shell that runs on RAM memory. An initramfs disk is nothing else than a compressed cpio archive, that is then either embedded directly into your kernel image, or stored as a separate file which can be loaded by the Linux PXE boot loader. Embedded or not, it should always contains at least:

a minimum set of directories:
- /sbin -- Critical system binaries
- /bin -- Essential binaries considered part of the system
- /dev -- Device files, required to perform I/O
- /etc -- System configuration files
- /lib, /lib32, /lib64 -- Shared libraries to provide run-time support
- /mnt -- A mount point for maintenance and use after the boot/root system is running
- /proc -- Directory stub required by the proc filesystem. The /proc directory is a stub under which the proc filesystem is placed
- /root -- the root's home directory
- /sys --
- /tmp -- Temporal directory
- /usr -- Additional utilities and applications
- /var -- Variable files whose content is expected to continually change during normal operation of the system—such as logs, spool files, and temporary e-mail files.
basic set of utilities: sh, ls, cp, mv, etc
minimum set of config files: rc, inittab, fstab, etc
devices: /dev/hd*, /dev/tty*, etc
runtime libraries to provide basic functions used by utilities

Is there any other simple method to create the RAM disk? Creating an initramfs can be also achieved by copying the content of an already installed Linux distro into an empty directory then package it, but you must be aware of carrying undesired and/or useless archives. There other methods, some of them simple, some of them not, but they are outside of the scopte of this guide which aims to show you a handy approch to obtain a lightweight RAM disk and Kernel

Use the following steps to create the initramfs:

Creating a download cache & working zone. Also defining a helper command to download and cache archives:

$ mkdir -p /tmp/cache
$ mkdir /tmp/wrk
$ pushd /tmp/wrk

#/** 
# * Downloads a file to the cache if doesn't exists
# *
# * @param $1 the file to download
# * @param $2 the url where the file is located
# */
$ get() {
 [ -f /tmp/cache/$1 ] || wget -t inf -w 5 -c $2/$1 -O /tmp/cache/$1
}

Creating and entering to initramfs root directory:

$ mkdir initramfs
$ pushd initramfs

Creating filesystem's base directories:

$ mkdir -p -m 0755 dev etc/{,init,sysconfig} mnt sys usr/{,local} var/{,www,log,lib,cache} run
$ mkdir -p -m 0555 {,s}bin lib{,32,64} proc usr/{,s}bin
$ mkdir -p -m 0700 root
$ mkdir -p -m 1777 tmp
$ pushd var
$ ln -s ../run run
$ popd

Creating /etc/profile to exports environment variables:

$ dd of=etc/profile << EOT
## /etc/profile

export PATH="/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/sbin"
EOT

Creating /etc/fstab with various mount points:

$ dd of=etc/fstab << EOT
devpts  /dev/pts  devpts  nosuid,noexec,gid=5,mode=0620  0 0
tmpfs   /dev/shm  tmpfs   nosuid,nodev,mode=0755  0 0
sysfs   /sys      sysfs   nosuid,nodev,noexec  0 0
proc    /proc     proc    nosuid,nodev,noexec  0 0
EOT
$ chmod 0644 etc/fstab

Configure passwd & group settings:

$ dd of=etc/passwd << EOT
root:x:0:0:root:/root:/bin/sh
nobody:x:99:99:NoBody:/none:/bin/false
www:x:33:33:HTTP Server:/var/www:/bin/false
EOT
$ dd of=etc/group << EOT
root:x:0:
nobody:x:99:
www:x:33:
EOT

Configure some host related settings:

$ dd of=etc/host.conf <<< "multi on"
$ dd of=etc/hostname <<< "rambox"
$ dd of=etc/hosts << EOT
127.0.0.1 localhost.localdomain localhost
127.0.1.1 $(cat etc/hostname)
EOT

Configure timezone:

$ dd of=etc/timezone <<< "America/New_York"
$ cp /usr/share/zoneinfo/$(cat etc/timezone) etc/localtime

Busybox is a handful tool used very often in ramdisks and small devices with very limited resources, providing a self-contained and minimal set of POSIX compatible unix tools in a single executable archive. I'll be using busybox on this guide. Getting busybox and create sh symbolic link:

$ pushd bin
$ chmod +w .
$ bb=busybox-x86_64 && get $bb http://www.busybox.net/downloads/binaries/latest/busybox-x86_64
$ cp /tmp/cache/$bb busybox && chmod +x busybox
$ ln -s busybox sh
$ chmod -w . 
$ popd

Additionally we MAY need an DHCP configuration script, so we'll use busybox's udhcp and simple.script. The we'll create an script named renew_ip that performs all the job:

$ pushd bin
$ chmod +w .
$ ss=simple.script
$ get $ss http://git.busybox.net/busybox/plain/examples/udhcp/$ss
$ cp /tmp/cache/$ss . && chmod +x $ss 

$ dd of=renew_ip << EOT
#!/bin/sh

ifconfig eth0 up
udhcpc -t 5 -q -s /bin/simple.script
EOT

$ chmod +x renew_ip
$ chmod -w . 
$ popd

One of the most important phases is /init script execution, this is a simple shell script file that performs all initialization process on the ramdisk. It usually mounts all filesystems listed on fstab, creates device nodes (like udev device manager), loads device firmware and finally remounts another root (/) directory in other device and relaunches the new mounted /sbin/init. This is the point where we intervened, by just launching the shell or by executing our own /sbin/init w/o remounting the root (/). So edit init script and add the following content:

$ nano init

#!/bin/sh

# Make all core utils reachable 
. /etc/profile

# Create all busybox's symb links
/bin/busybox --install -s

# Create some devices statically

# pts: pseudoterminal slave 
mkdir dev/pts

# shm
mkdir dev/shm
chmod 1777 dev/shm

# Mount the fstab's filesystems.
mount -av

# Some things don't work properly without /etc/mtab.
ln -sf /proc/mounts /etc/mtab

# mdev is a suitable replacement of the udev device node creator for loading 
# firmware
touch /etc/mdev.conf 
echo /sbin/mdev > /proc/sys/kernel/hotplug
mdev -s

# Only renew the IP address via DHCP if you need it. Not needed if Kernel-level 'ip=...' 
# was used. 
#renew_ip 

# set hostname
hostname $(cat /etc/hostname)

# shell launcher 
shell() {
 echo "${1}Launching shell..." && exec /bin/sh
}

# launch the shell if the 'shell' parameter was supplied
grep -q 'shell' /proc/cmdline && shell

# parse kernel command and obtain the init & root parameters
# if not then use default values
for i in $(cat /proc/cmdline); do
 par=$(echo $i | cut -d "=" -f 1)
 val=$(echo $i | cut -d "=" -f 2)
 case $par in
  root)
   root=$val
   ;;
  init)
   init=$val
   ;;
 esac
done
init=${init:-/sbin/init}
root=${root:-/dev/hda1}

# if rambox parameter is supplied then keep the ramdisk mounted, ignore root parameter 
# and run the other init script. Located at /sbin/init by default
if grep -q 'rambox' /proc/cmdline ; then
  [ -e ${init} ] || shell "Not init found on ramdisk at '${init}'... " 

  echo "Keeping the ramdisk since rambox param was supplied & executing init... "
  exec ${init}
  
  #This will only be run if the exec above failed
  shell "Failed keeping the ramdisk and executing '${init}'... "
fi

# Neither shell nor rambox parameters were supplied then, try to switch to the new
# root and launch init 
mkdir /newroot
mount ${root} /newroot || shell "An error ocurred mounting '${root}' at /newroot... "
[ -e /newroot${init} ] || shell "Not init found at '${init}'... " 

echo "Resetting kernel hotplugging... "
:> /proc/sys/kernel/hotplug
echo "Umounting all... "
umount -a
echo "Switching to the new root and executing init... "
exec switch_root /newroot ${init}

#This will only be run if the exec above failed
mount -av
mdev -s
shell "Failed to switch_root... "

/etc/profile is sourced to export PATH variable and make all executables reachable
All busybox's symbolic links are created
Some special devices are created by hand
All /etc/fstab filesystems are mounted
The rest of the devices are discovery and created by busybox's mdev
The Kernel command line located at /proc/cmdline is parsed to see if the shell parameter was supplied, is so the shell is immediately launched replacing the current process instance, hence everything else is ignored
The Kernel command line is checked again to see if rambox parameter was supplied, indicating that we want to keep the ramdisk mounted at / and launch the normal /sbin/init process
If neither shell nor rambox parameters were supplied then, try to mount the new root (/) and launch the /sbin/init on this new location
Finally if neither the new root cannot be mounted nor the /sbin/init script cannot be executed, then a shell is launched indicating this situation
If on any of these shell launching steps an error is produced, then a Kernel panic is issued

Append execution permissions to /init:

$ chmod +x init

Change ownership to everything:

$ sudo chown -R root:root *

Create the initramfs.cpio.gz compressed archive and copy it to tftp's root directory:

$ sudo find . -print0 | sudo cpio --null -ov --format=newc | gzip -9 > ../initramfs.cpio.gz
$ sudo cp ../initramfs.cpio.gz /var/lib/tftpboot

Go back to working directory:

$ popd

Now the Kernel stuff:

What I am about to do with the Kernel is very simple, compile it using a minimal set of features that makes it boot and recognize MY hardware, mainly the NIC device. Hence, depending on your hardware you should probably use a different selection of features for Kernel compiling. So I recommend first to do a once-time installation of any modern Linux distribution (like I did) like CentOS, Gentoo, Fedora, Debian or Ubuntu with a modern Kernel version and check the modules loaded on boot using /sbin/lsmod. Then using this modules list, look for the corresponding Kernel options and INCLUDE them all in the Kernel, making it a solid rock!. That's what I did.

NOTE: In our journey for making the Kernel simple and small, we should be careful in omitting some Kernel critical features and lost the hardware advantages, for example SMP features. So if we really want to use it in a production environment, then a deep research and customization must be done before.

Start Kerneling...

Download the Kernel sources from the sky:

$ krn=linux-2.6.39
$ get $krn.tar.xz http://www.kernel.org/pub/linux/kernel/v2.6/$krn.tar.xz

Uncompress it into a working directory, named it linux:

$ tar xvf /tmp/cache/$krn.tar.xz -C .
$ mv $krn linux
$ pushd linux

Clean all configuration settings and enter the menu:

$ make clean
$ make allnoconfig
$ make menuconfig

An ncurses menu dialog should be opened. Now check a "minimal" set of features, and uncheck the unneeded ones, I'll only list what changes wrt the clean configuration settings. So [*] means explicitly checked to be EMBEDDED it into the Kernel, and [ ] means explicitly unchecked to be not included

General Setup (here the RAM filesystem is the most important feature)

[*] Prompt for development and/or incomplete code/drivers
(-minimal) Local version - append to kernel release
[*] Initial RAM filesystem and RAM disk (initramfs/initrd) support

Bus options (PCI etc.) ---> (Enable support for PCI devices, you may add support for your PCI hardware here)

[*] PCI support

Executable file formats / Emulations ---> (An important piece!, you won't be able to execute almost anything if you don't check it)

[*] Kernel support for ELF binaries

[*] Networking support (Beside of enabling TCP/IP and disable Wireless, IPSec, etc. The most important feature to check here is the IP-Kernel level auto configuration with DHCP support)

  [ ]   Wireless  ---> 
 Networking options 
    [*] Packet socket                                                                                 
    [*]   Packet socket: mmapped IO                                                                     
    [*] Unix domain sockets
    [*] Transformation sub policy support (EXPERIMENTAL)
    [*] Transformation migrate database (EXPERIMENTAL)
    [*] PF_KEY sockets
    [*]   PF_KEY MIGRATE (EXPERIMENTAL)                                                                                  
    [*] TCP/IP networking
    [*]   IP: multicasting                              
    [*]   IP: advanced router                           
        Choose IP: FIB lookup algorithm (choose FIB_HASH if unsure) (FIB_HASH  
    [*]   IP: policy routing                            
    [*]   IP: equal cost multipath                      
    [*]   IP: verbose route monitoring                                                                               
    [*]   IP: kernel level autoconfiguration                                                           
    [*]     IP: DHCP support
    [*]   IP: tunneling                                 
    [*]   IP: GRE tunnels over IP                       
    [*]     IP: broadcast GRE over IP                   
    [*]   IP: multicast routing                         
    [*]     IP: PIM-SM version 1 support                
    [*]     IP: PIM-SM version 2 support                
    [*]   IP: ARP daemon support                        
    [*]   IP: TCP syncookie support (disabled per default)                         
    [*]   IP: AH transformation                         
    [*]   IP: ESP transformation                        
    [*]   IP: IPComp transformation                                                                            
    [ ]   IP: IPsec transport mode                                                                     
    [ ]   IP: IPsec tunnel mode                                                                        
    [ ]   IP: IPsec BEET mode
    [*]   TCP: advanced congestion control  --->                                                                           
     [*]   CUBIC TCP (NEW) (only cubic)
    [*]   TCP: MD5 Signature Option support (RFC2385) (EXPERIMENTAL) 
    [ ]   The IPv6 protocol  --->

Device Drivers ---> (RAM block device support and Network device support + Ethernet are the most important things, the remaining stuff is related to my current hardware)

  [*] Block devices  --->
    [*]   RAM block device support
  [*] Multiple devices driver support (RAID and LVM)  --->
    [*]   Device mapper support
  [*] Network device support  ---> 
    [*]   Ethernet (10 or 100Mbit)  --->
    [ ]   Wireless LAN  ---> 
 Character devices  --->
    [*] /dev/kmem virtual device support
    [*] Hardware Random Number Generator Core support
  [*] I2C support  --->
    [*]   I2C device interface
    I2C Hardware Bus support  ---> 
      [*] Intel PIIX4 and compatible (ATI/AMD/Serverworks/Broadcom/SMSC)
 Serial ATA (prod) and Parallel ATA (experimental) drivers (ATA [=n])
    [*]   ATA SFF support (NEW) 
      [*]    Intel ESB, ICH, PIIX3, PIIX4 PATA/SATA support
      [*]    Generic ATA support

File systems ---> (File systems are very important, they support depend on what's your final goal: mount an NFS remotely for a shared storage? use a GlusterFS / Ceph filesystem in top of a NAS? The configuration I used is the simplest one, only support for initramfs and other pseudo filesystem. I recommend to start with this one, then gradually embed your filesystems)

  [ ] Network File Systems  --->
  Pseudo filesystems  --->
    [*] Virtual memory file system support (former shm fs)
    [*]   Tmpfs POSIX Access Control Lists

[*] Virtualization ---> (As I mentioned earlier, I'm using a KVM-virtualized hardware with a wide usage of Virtio paravirtualization technology. Virtio adds supports for a paravirtual Ethernet card, a paravirtual disk I/O controller, a balloon device for adjusting guest memory usage, and a VGA graphics interface using SPICE drivers. Virtio drivers for guest machines are included in the Kernel >= 2.6.25, see details here)

  [*]   PCI driver for virtio devices (EXPERIMENTAL)
  [*]   Virtio balloon driver (EXPERIMENTAL)

Device Drivers ---> [for virtualization]

 
  [*] Block devices  --->
    [*]   Virtio block driver (EXPERIMENTAL)
  [*] Network device support  ---> 
    [*]   Virtio network driver (EXPERIMENTAL)
  Character devices  --->
    [*] Virtio console
    [*] Hardware Random Number Generator Core support
    [*]   VirtIO Random Number Generator support

Exit the Kernel configuration menu and don't forget to save the settings file.

Compile the Kernel (-j4 means 4 threads devoted to compilation), copy it to TFPT's root directory:

$ make -j4 bzImage
$ sudo cp arch/x86/boot/bzImage /var/lib/tftpboot/

Do cleanup:

$ popd
$ sudo rm -rf /tmp/wrk

Power on the rambox and enjoy it! It should boot smoothly and launch the busybox's shell.

You will find the basic tools at /bin, /sbin, /usr/bin, /usr/sbin, /usr/local/sbin, all these tools are indeed in the PATH environment variable. To renew your IP address just run renew_ip. Finally notice that any Kernel module is loaded since all that you need is embedded.

Enjoy it!

Post install

Perform some checks after install to ensure that everything is OK and measure for resource consumption:

Free memory, as you may notice approximately only 12Mb are used:

$ free -m
                total    used    free    shared    buffers
Mem:             1255      12    1243         0          0 
-/+ buffers:               12    1243
Swap:               0       0       0

Network configuration/connectivity, both interfaces should be listed, eth0 and lo:

$ ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:07:26:c0:a5  
          inet addr:192.168.24.203  Bcast:192.168.24.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:378 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:43745 (42.7 KiB)  TX bytes:1180 (1.1 KiB)
          
lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0 B)  TX bytes:0 (0 B)

$ ping -c 3 192.168.24.202
PING 192.168.24.202 (172.26.24.202) 56(84) bytes of data.
64 bytes from 192.168.24.202: icmp_req=1 ttl=63 time=0.774 ms
64 bytes from 192.168.24.202: icmp_req=2 ttl=63 time=0.639 ms
64 bytes from 192.168.24.202: icmp_req=3 ttl=63 time=0.574 ms

--- 192.168.24.202 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.574/0.662/0.774/0.085 ms

Mounted partitions:

$ mount
rootfs on / type rootfs (rw)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,nosuid,nodev,relatime,mode=0755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

Disk usage, about 1.1Mb:

$ du -chs /*
1.1M    /bin
0       /dev
28.0K   /etc
4.0K    /init
0       /lib
0       /lib32
0       /lib64
0       /linuxrc
0       /mnt
0       /proc
0       /root
36.0K   /sbin
0       /sys
0       /tmp
20.0K   /usr
0       /var
1.1M    total

Loaded modules, since no module was loaded an empty list or a 'No such a file or directory' message is issued

$ lsmod
lsmod: can't open '/proc/modules': No such file or directory

Device nodes created by device manager(mdev/udev). The result depends on many factors:

$ find /dev | wc -l
106

Friday, October 7, 2016

Spring Sandbox Welcome

Thursday, July 28, 2016

JAngular launched

Friday, April 11, 2014

Friday, August 9, 2013

Tuesday, May 7, 2013

Thursday, May 2, 2013

Thursday, February 21, 2013

Intro

Goals

How to

Let's do it

Wednesday, December 19, 2012

Intro

Example

What to test?

Hands on Spring Tool Suite

Final Snapshot

Tuesday, December 11, 2012

The Steps

Wednesday, August 8, 2012

Steps

Thursday, July 5, 2012

Motivation

Hands on bash

Testing

Enjoy it!

Wednesday, July 4, 2012

Post install steps

Tuesday, July 3, 2012

Thursday, June 7, 2012

Why RAM-only & diskless?

When don't I need persistence?

What do I need?

How does it work?

Hands on Bash

Installing phase

Configure DHCP

Configuring TFPT

Concerning the firewall

Configuring the PXE environment

Creating a compressed root filesystem

Now the Kernel stuff:

Post install

References