continuousbuild

  • A strategy for Integrations versions with maven...

     apache_maven

    We are currently asking ourselves at INNOVEO, if we need to keep a version of integration versions.
    Integration versions main objective is to be integrated with other modules to make and test an
    application or a framework. This question is quickly becoming essential when working with several
    modules, where you will have to to rely on intermediate, non finalized versions of modules.

    Since we are also following  the continuous integration paradigm for all our modules, Thanks to
    Apache MAVEN, these integration versions are produced by a continuous integration server
    (Team City from JetBrain), very frequently.

    So, how can you deal with these, possibly numerous, integration versions? The response is coming
    from this extract from IVY documentation

    There are basically two ways to deal with them,

    Use a naming convention
    The idea is pretty simple, each time you publish a new integration of your module you give the same
    name to the version (in maven world this is for example 1.0-SNAPSHOT). The dependency manager
    should then be aware that this version is special because it changes over time, so that it does not
    trust its local cache if it already has the version, but check the date of the version on the repository
    and see if it has changed.

     
    Create automatically a new version for each
    in this case you use either a build number or a timestamp to publish each new integration version
    with a new version name. Then you can use one of the numerous ways in Ivy to express a version
    constraint. Usually selecting the very latest one (using 'latest.integration' as version constraint) is
    enough.

    But usually we recommend to use the second one, because using a new version each time you publish
    a new version better fits the version identity paradigm, and can make all your builds reproducible,
    even integration one. And this is interesting because it enables, with some work in your build system,
    to introduce a mechanism to promote an integration build to a more stable status, like a milestone
    or a release.

    The example given is very interesting...

    Imagine you have a customer which comes on a Monday morning and asks your latest version of your
    software, for testing or demonstration purpose. Obviously he needs it for the afternoon :-) Now if
    you have a continuous integration process and a good tracking of your changes and your artifacts, it
    may occur that you are actually able to fulfill his request without needing the use of a dolorean to
    give you some more time :-) But it may occur also that your latest version stable enough to be used
    for the purpose of the customer was actually built a few days ago, because the very latest just break
    a feature or introduce a new one you don't want to deliver. In this case, you can deliver this 'stable'
    integration build if you want, but be sure that a few days, or weeks, or even months later, the
    customer will ask for a bug fix on this demo only version. Why? Because it's a customer, and we
    all know how they are :-)

    So, with a build promotion feature of any build in your repository, the solution would be pretty easy:
    when the customer ask for the version, you not only deliver the integration build, but you also
    promote it to a milestone status, for example. this promotion indicates that you should keep track of
    this version in a long period, to be able to come back to it and create a branch if needed.

    Note this is the strategy at Eclipse.org, where a nightly build (N20080420) can be promoted to an Maintenance
    release if quality is good enough. Below I've put an extract of a presentation document from © 2006 by Alex Blewitt;
    made available under the EPL v1.0 |  2006-03-20  |  http://www.rcpapps.org/

    We are now using the same naming convention at INNOVEO for our product

    Eclipse_logo_white
    Eclipse builds are of different type:

    (N) Nightly
    • Built every night (whether successful or not)
    • Used to run quality metrics and whether tests have passed
    (I) Integration
    • Used to ensure that code works together
    • Used to run quality metrics
    (M) Maintenance
    • Released at the end of each build cycle
    (R) Release
    • Released at the end of each release cycle

    Each product is given a build id,

    • Build Type (N, I, M or R)
    • Build ID (M20060118)
    • Build Label (M20060118-1600)
    • Timestamp of build (16:00 on the 18th Jan, 2006)
    • Each release corresponds to a specific build label
    • May also be known as other aliases in CVS
    • R3_1_2, vI20060118-1000, R3_1_Maintenance

    To keep the Eclipse ecosystem in step, everything is tagged

    • Part of the build process tags the current code with vI20060320
    • A build is only promoted from N->I if there are no build failures
    • A build is promoted from I->M if there are no failures and all the
      functionality works to a satisfactory level
    • A build is promoted from M->R at the end of a release cycle and
      the quality is suitably high

     On the other hand, the main drawback of this solution is that it can produce a lot of intermediate
    versions, and you will have to run some cleaning scripts in your repository...

    I will present You later how you can achieve this goal with MAVEN and Team City

  • Adding Google analytics to Atlassian Bamboo

    2742514705_fe8fd07b14

    As I found no better tutorial on Internet, here is a very very short how to add Google analytics to AtlassianBamboo, it require a bit of hacking, and these kind of changes will be lost after each upgrade of Bamboo..

    Edit the file webapps/ROOT/start.ftlNow put the usual code you get after creating a new analytics profile just before the </body>

    <script type="text/javascript">
    var gaJsHost = (("https:" == document.location.protocol) ? "
    https://ssl." : "http://www.");
    document.write(unescape("%3Cscript src='" + gaJsHost +
        "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
    </script>
    <script type="text/javascript">
    try {
    var pageTracker = _gat._getTracker("UA-88600-10");
    pageTracker._trackPageview();
    } catch(err) {}</script>

    While not ideal, it simply work as expected and let you get insight about Atlassian Bamboo usage using Google analytics.

    My Bamboo continuous integration server is available at  http://bamboo.waltercedric.com/

    You can also insert Google Adsense using the same trick, but don’t forget that all your changes may be lost if you upgrade to a  new version. I will investigating further if there is not a plugin or an other way to do this. Stay Tuned!

  • Apache Maven 3 Cookbook

     

    First a big thanks toPackt Publishing for having sent me this book to review! I did enjoy going through this book, while I did not learn a lot of new stuff (I am using Apache Maven daily since 2006!), I found it to be concise and would recommend it anytime to any of my colleagues. But let’s go through my review of this cookbook of over 50 recipes towards optimal Java Software Engineering with Maven 3:

    Apache Maven 3 Cookbook is a clear, precise, well-written book that gives readers clear recipes for the release process using Apache Maven 3. The authors give a step-by-step account of expectations and hurdles for software development.

    The first few chapters quickly bring you to the point to be comfortable using Maven on straightforward projects, and the later chapters provide even more recipes examples on subjects like running a Repository Manager, Writing Plugins, and details on various techniques. The book also covers numerous real world software delivery issues such as multi-module projects, web/enterprise projects, dependency management, automatic testing and documentation.

    To sum up key points from this 224 pages book in a few bullets:

  • Chapter 1: Basics of Apache Maven: Setting up Apache Maven on Windows/Linux/Mac, Creating a new project, Understanding the Project Object Model, build lifecycle and build profiles,
  • Chapter 2: Software Engineering Techniques: Build automation, modularization, Dependency management, Source code quality check, Test Driven Development (TDD), Acceptance testing automation and Deployment automation,
  • Chapter 3: Agile Team Collaboration: Creating centralized remote repositories, Performing continuous integration with Hudson, Integrating source code management, Team integration with Apache Maven, Implementing environment integration, Distributed development and Working in offline mode,
  • Chapter 4: Reporting and Documentation: javadocs, unit tests, coverage reports and Maven dashboard setup,
  • Chapter 5: Java Development with Maven: Java web application, J2EE, Spring, Hibernate and JBoss SEAM development,
  • Chapter 6: Google Development with Maven: Android and GWT (Google Web Toolkit), Google App Engine deployment,
  • Chapter 7: Scala, Groovy, and Adobe Flex
  • Chapter 8: IDE Integration
  • Chapter 9: Extending Apache Maven: creating plugins using Java, Apache ANT or Ruby,
  • The author Srirangan go into detail in describing each of these themes. 

    I recommend you this book if

  • If you need to learn Apache Maven quickly, you can go through the recipes and examples and come away with a good knowledge of Maven.
  • If you are currently implementing Apache Maven for the first time in your development process and feel a bit lost by the lack of clear examples that just run.
  • If you want to use proven solutions to real common engineering challenges: this book will save you a lot of time!
  •  

    if you want to be able to deliver your software to any target environment, using continuous delivery processes, chances are high that Apache Maven is the right tool for this job, and this book should be part of your technical library, beside also of course the free online book of Sonatype Maven: The Complete Reference

  • Apache Maven 3 Cookbook Review

    Thanks to Packt Publishing for having sent me this book to review. I will publish a review in the next coming days

    • Grasp the fundamentals and extend Apache Maven 3 to meet your needs
    • Implement engineering practices in your application development process with Apache Maven
    • Collaboration techniques for Agile teams with Apache Maven
    • Use Apache Maven with Java, Enterprise Frameworks, and various other cutting-edge technologies
    • Develop for Google Web Toolkit, Google App Engine, and Android Platforms using Apache Maven

    Apache_Maven3_Cookbook

    You may also consider reading all my articles related to Apache Maven

  • Apache Maven Archetype for Joomla

    Got this email from Cyprian Sniegota, he did develop a Maven Archetype for easing development of Joomla extensions. His archetype currently support the creation of a skeleton for components, modules, plugins and templates.

    I noticed some time ago that you described combination of Joomla! and Maven. Few weeks ago i wrote joomla-maven-plugin with skeleton projects (sources: bitbucket.org/deviapps) based on php-maven.org work.
    Here is short description http://deviapps.com/create-joomla-extension-with-maven and 5 min video (in Polish so far) http://www.youtube.com/watch?v=aE8w9EZciTg
    I hope you will be interested.

    Thanks to him for having written this project. I will also try to Maven-ize what Joomla has done with Ant in the future (I prefer now crystal clear software lifecycle )

  • Apache Maven BEA Weblogic 10.3 remote deployment

     apache_maven

    In this small post I will show you how to deploy automatically some artifacts of your build into bea_logo1Weblogic 10.3 by using the weblogic-maven-plugin

    This plugin will support various tasks within the Weblogic 8.1 and 9.x environment. Such tasks as deploy, undeploy,clientgen,servicegen, and appc are supported as well as many others. The plugin uses exposed API's that are subject to change but have been tested in 8.1 SP 4-6 and 9.0 - 9.2 MP3. There are two versions of the plugin to support the two environments based on differences in the JDK. The 9.x version is currently being refactored to support the standard JSR supported deployment interface

  • Apache Maven books

    apache_maven

    Questions for the official certification.

     JavaBlackBelt is a community for Java & open source skills assessment. It is dedicated to technical quizzes about Java related technologies. This is the place where Java developers have their technology knowledge and development abilities recognized. Everybody is welcome to take existing and build new exams.

     

    maven.the.definitive.guide BetterBuildsWithMaven

    Maven: The Definitive Guide (Readable HTML alpha release)

    Better Builds with Maven (Free PDF)

    • Covers:Maven 2.0.4
    • Publisher:DevZuz
    • Published:March 2006
    • Authors: John Casey, Vincent Massol, Brett Porter, Carlos Sanchez

      Better Builds with Maven is a comprehensive 'How-to' guide for using Maven 2.0 to better manage the build, test and release cycles associated with software development. The chapters include:

      • An introduction to Maven 2.0
      • Creating, compiling and packaging your first project
      • Best practices and real-world examples
      • Building J2EE Applications
      • Extending builds by creating your own Maven plugins
      • Monitoring the health of source code, testing, dependencies and releases
      • Team collaboration and utilising Continuum for continuous integration
      • Converting existing Ant builds to Maven
    0596007507_cat  

    Maven: A Developer's Notebook

     

     

     

  • Apache Maven Cargo deploy with Tomcat 7

    apache_maven

    cargo-banner-left

    Following the post about Deploy to Tomcat 6 using Maven, here is a ready to use example with the main differences explained in the table below

      Tomcat 7 Tomcat 6
    containerId <containerId>tomcat7x</containerId> <containerId>tomcat6x</containerId>
    Url of Tomcat manager <cargo.remote.uri> <cargo.tomcat.manager.url>
    example http://host..com/manager/text/ http://host..com/manager/
    tomcat-users.xml

    <tomcat-users>
    <role rolename="manager-gui"/>
    <role rolename="manager-script"/>
    <role rolename="manager-jmx"/>
    <role rolename="manager-status"/>
    <user username="admin" password="admin" roles="manager-gui,manager-script"/>
    </tomcat-users>

    <tomcat-users>
      <role rolename="manager"/>
      <user username="admin" password="admin" roles="manager"/>
    </tomcat-users>

    And finally a snippet of an Apache Maven pom.xml ready to use in a profile, so you can reuse this profile like a method call

    <profile>
     <id>deployTomcat</id>
    <activation>
      <activeByDefault>false</activeByDefault>
    </activation>
    <build>
     <plugins>
        <plugin>
         <groupId>org.codehaus.cargo</groupId>
         <artifactId>cargo-maven2-plugin</artifactId>
         <version>1.1.0</version>
        <configuration>
         <wait>true</wait>
         <container>
          <containerId>tomcat7x</containerId>
          <type>remote</type>
         </container>
         <configuration>
          <type>runtime</type>
          <properties>
           <cargo.remote.uri>
             ${tomcat.url}
           </cargo.remote.uri>
           <cargo.remote.username>
              ${tomcat.user}     
           </cargo.remote.username>
            <cargo.remote.password>
              ${tomcat.pwd}
            </cargo.remote.password>
          </properties>
          </configuration>
          <deployer>
           <type>remote</type>
           <deployables>
           <deployable>
            <groupId>${deploy.groupid}</groupId>
            <artifactId>${deploy.artifactid}</artifactId>
            <type>war</type>
            <properties>
             <context>${deploy.context}</context>
            </properties>
           </deployable>
          </deployables>
         </deployer>
        </configuration>
        <executions>
         <execution>
          <id>verify-deploy</id>
          <phase>pre-integration-test</phase>
          <goals>
           <goal>deployer-undeploy</goal>
           <goal>deployer-deploy</goal>
          </goals>
         </execution>
        </executions>
        </plugin>
     </plugins>
    </build>
    </profile>

    Place as many profiles as you have machine to deploy in settings.xml and declare some variables as properties, as shown below:

    <profile>
     <id>serverA</id>
     <activation>
        <activeByDefault>false</activeByDefault>
     </activation>
     <properties>
        <tomcat.url>http://host.com/manager/text</tomcat.url>
        <tomcat.user>admin</tomcat.user>
        <tomcat.pwd>admin</tomcat.pwd>
        <!-- these properties must be defined
           as system property or -D -->
        <!-- - deployable.artifactid:
             artifactId of web application to be deployed -->
        <!-- - deployable.context: web context name -->
     </properties>
    </profile>

    So you can run, and traget multiple host by just exchanging the name of the profile serverA to something else.

    mvn integration-test –PdeployTomcat,serverA
       –Ddeployable.artifactid=demo
       -Ddeploy.groupid=com.mycompany
       –Ddeployable.context=showcase
  • Apache Maven copy local file to a remote server server using SSH

    I will show you in an Apache Maven configuration file how to copy files to server each time the package phase is executed.

    Solution with Ant SCP task

    This snippet of code is a ready to use code that make use of Apache Ant task scp, Just put this snippet of code in your Maven module where the assembly is executed or anywhere else to push all tar.gz files to a server just run a maven mvn package, you can add as many ant task and push to many server the same file during the reactor build.

    <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-antrun-plugin</artifactId>
    <version>1.7</version>
    <executions>
    <execution>
        <id>server-copy</id>
        <goals>
            <goal>run</goal>
        </goals>
        <phase>package</phase>
        <configuration>
            <target>
                <echo message="Push to server/home/"/>
                <scp trust="yes"
                    todir="user:password@server:/home/">
                    <fileset dir="${basedir}/target">
                        <include name="**/*.tar.gz"/>
                    </fileset>
                </scp>
            </target>
        </configuration>
    </execution>
    </executions>
    <dependencies>
    <dependency>
        <groupId>org.apache.ant</groupId>
        <artifactId>ant-jsch</artifactId>
        <version>1.8.2</version>
    </dependency>
    </dependencies>
    </plugin>

    Solution with maven-deploy-plugin

    The maven-deploy-plugin allows you to configure the deploy phase to deploy to a server using scp. There is a page in the documentation that describes how it can be done.

    Deploy maven artifact using Maven Wagon SCP

    Another alternative would be to use Maven Wagon SCP like described in this post for example

  • Apache Maven manifest and enabling agile testing

    apache_maven

    In this small post i will present you how to

    1. Track and identify across your Apache Maven multi project builds all components, their versions, and class path if needed
    2. Get valuable information from your testing team, and add it to the GUI of your web applications

    To solve the problem number 1, we will use Manifest files

    On the Java platform, a manifest file is a specific file contained within a JAR archive. It is used to define extension and package related data. It is a metadata file that contains name-value pairs organized in different sections. If a JAR file is intended to be used as an executable file, the manifest file specifies the main class of the application. The manifest file is named MANIFEST.MF. [Wikipedia]

    If you do nothing special in Maven., you will see across your jar files the following in all META-INF/MANIFEST.MF

    Manifest-Version: 1.0
    Archiver-Version: Plexus Archiver
    Created-By: Apache Maven

     

    The solution I propose here will write all these META-INF/MANIFEST.MF the following  content:

    Manifest-Version: 1.0
    Archiver-Version: Plexus Archiver
    Created-By: Apache Maven
    Built-By: agent1
    Build-Jdk: 1.6.0_20
    Specification-Title: Unnamed - com.waltercedric:web:war:1.0.0-SNAPSHOT
    Specification-Version: 1.0.0-SNAPSHOT
    Specification-Vendor: waltercedric
    Implementation-Title: Unnamed - com.waltercedric:web:war:1.0.0-SNAPSHOT
    Implementation-Version: 1.0.0-SNAPSHOT
    Implementation-Vendor-Id: com.waltercedric
    Implementation-Vendor: waltercedric
    buildDate: 2010.09.22-14:12
    svnrevision: 18625
    Class-Path: spring-ws-core-1.5.6.jar spring-aop-2.5.6.jar spring-
     web-2.5.6.jar spring-webmvc-2.5.6.jar spring-context-support-2.5.6.jar
    .. .. ..

     

  • Apache Maven profiles order in multi modules projects

    apache_maven

    In which order are Apache Maven profiles executed? are Apache Maven profiles ordered? how can you insured that Apache Maven profiles are activated in the right order?

    You normally don’t end up with these questions, issues may only appear if

    • Some profiles are dependent each other,
    • Some profiles can not run in any order,

    The use case behind this article is very simple, as I have a a continuous build were:

    • 5 web applications have  to be deployed into a remote tomcat in phase pre-integration-test,
    • 2 databases are created for test cases in phase generate-test-resources
    • 1 more database is created and needed for runtime, done in phase pre-integration-test
    • One of these web applications is able to inject data into database using web services, a profile do this in a profile in phase pre-integration-test
    • Selenium test cases are run in phase integration-test

    All these steps are done using several Apache Maven pom profiles.

    As it is a bit complicated to explain, lets first refresh some Apache Maven concepts

    Apache Maven Goals

    First you’ll have to keep in the mind Apache Maven lifecycle of modules, 21 goals out of the box:

    • Validate: validate the project is correct and all necessary information is available 
    • generate-sources: generate any source code for inclusion in compilation    
    • process-sources: process the source code, for example to filter any values    
    • generate-resources: generate resources for inclusion in the package    
    • process-resources: copy and process the resources into the destination directory, ready for packaging  
    • compile: compile the source code of the project    
    • process-classes: post-process the generated files from compilation, for example to do byte code enhancement on Java classes    
    • generate-test-sources: generate any test source code for inclusion in compilation
    • process-test-sources: process the test source code, for example to filter any values    
    • generate-test-resources : create resources for testing 
    • process-test-resources: copy and process the resources into the test destination directory    
    • test-compile: compile the test source code into the test destination directory    
    • test: run tests using a suitable unit testing framework. These tests should not require the code be packaged or deployed
    • prepare-package: perform any operations necessary to prepare a package before the actual packaging. This often results in an unpacked, processed version of the package    
      package     take the compiled code and package it in its distributable format, such as a JAR    
      pre-integration-test: perform actions required before integration tests are executed. This may involve things such as setting up the required environment   
    • integration-test: process and deploy the package if necessary into an environment where integration tests can be run     (selenium test cases for example)
      post-integration-test: perform actions required after integration tests have been executed. This may including cleaning up the environment
    • verify     run any checks to verify the package is valid and meets quality criteria    
    • install     install the package into the local repository, for use as a dependency in other projects locally
    • deploy    code is deployed in artifactory or copied with ftp/scp for distribution

    if you run the goal compile

    mvn compile

    on a simple multi module project, EVERY modules, one after the others,  will go through these phases
    validate –> generate-sources –> process-sources –> generate-resources –> process-resources –> compile

    Apache Maven reactor

    The reactor is the part of Apache Maven that allows to execute a goal on a set of modules. As mentioned in the Apache Maven 1.x documentation on multi-modules builds, while modules are discreet unit of work, they can be gathered together using the reactor to build them simultaneously and:

    The reactor determines the correct build order from the dependencies stated by each project in their respective project descriptors, and will then execute a stated set of goals. It can be used for both building projects and other goals, such as site generation.

    The reactor is what makes multi-modules build possible: it computes the oriented graph of dependencies between modules, derive the build order from this graph and then execute goals on the modules. In other words, a "multi-modules build" is a "reactor build" and a "reactor build" is a "multi-modules build".

    A simple multi modules project

    For the sake of the exmaple, it has modules and profiles dependencies, in myProject/pom.xml


    remoting
    web
    monitoring
    common
    services

    or if you prefer the directory layout

    myProject
        |_ pom.xml
        |_common
                     |_src
                     |_pom.xml
        |_ web
                     |_src
                     |_pom.xml
        |_ remoting
                     |_src
                     |_pom.xml
        |_ services
                     |_src
                     |_pom.xml
        |_ web
                     |_src
                     |_pom.xml

    Lets assume also I would like to apply a list of profiles named

    • deployWeb, deploy the war module using cargo to a running tomcat instance
    • createDatabase, create a mysql database from scratch
    • runSelenium, run selenium test in phase integration test against web, assume database is created first
    • deployMonitoring, deploy the war module using cargo to a running tomcat instance, query the web at startup to get some infos.

    Maven calculate the module order in reactor based on dependencies, as seen in logs file after running

    mvn compile

    [INFO] Reactor build order:  Unnamed - com.waltercedric:myproject:pom:0.0.1-SNAPSHOT
    Unnamed - com.waltercedric:common:jar:0.0.1-SNAPSHOT
    Unnamed - com.waltercedric:services:jar:0.0.1-SNAPSHOT
    Unnamed - com.waltercedric:remoting:ear:0.0.1-SNAPSHOT
    Unnamed - com.waltercedric:web:war:0.0.1-SNAPSHOT
    Unnamed - com.waltercedric:monitoring:war:0.0.1-SNAPSHOT

    Example

    It start to be complicated when you provide a list of profile using Apache Maven command line like this

    mvn post-integration-test –PdeployWeb,createDatabase,runSelenium,deployMonitoring

    Chances are high that you will get profile executed in wrong order, too early or too late..

    Rule #1 profiles are activated (if found) following reactor modules order

    The first rule is that profiles are activated in module reactor order first, if myProject is first it will go through all 18 phases of  Apache Maven (from validate to post-integration-test in my example). Keep in mind also that the list of profiles will be applied to EVERY modules in EVERY phase starting at the top most module in reactor.

    • On modules myproject:
      •  Apache Maven will activate profiles PdeployWeb,createDatabase,runSelenium,deployMonitoring if one or more in the list are present in myproject/pom.xml
    • On modules common,
      • Apache Maven will activate profiles PdeployWeb,createDatabase,runSelenium,deployMonitoring if one or more in the list are present in common/pom.xml
    • and so on….

    Rule #2  Reactor modules order “may” be changed

    And now the tricky part, you can normally NOT change the module order in reactor, that’s ok but….

    The order you define in myProject/pom.xml for   (=module aggregation) is still kept if the maven dependencies resolver don't see any problems

    Not clear enough? look at the 2 examples below:

    myProject/pom.xml mvn post-integration-test
    Reactor build order (seen in logs)
    Remarks

    remoting
    web
    monitoring

    common
    services
    1. myProject
    2. common
    3. services
    4. remoting
    5. web
    6. monitoring
    Maven adapt the order based on oriented graph of dependencies between modules.

    remoting
    monitoring
    web

    common
    services
    1. myProject
    2. common
    3. services
    4. remoting
    5. monitoring
    6. web
    Swapping module having no direct connections each others and having no conflicting dependencies to other may result in a different order in reactor!!!! and also different profile execution order.

    Since Apache Maven has detected that the module monitoring and web have no connections, it accept the “human/natural” order found in myproject/pom.xml.

    You may have to use this technique to distribute your profiles in pom.xml while still keeping the profile order execution under control.

    Rule #3 Maven profile order is not taken from command line

    The order of profile in the Apache Maven command line  –P list is not taken into account, running the expected profiles order

    mvn post-integration-test –PdeployWeb,createDatabase,runSelenium,deployMonitoring

    is equivalent to

    mvn post-integration-test –PcreateDatabase,deployMonitoring, deployWeb,runSelenium

     

     

    It is a good things, as it  simply make no sense across all modules and all Maven phase all combined together.

    Rule #4 You can force profiles to run in an order if you SORT them accordingly into ONE pom.xml

    Apache Maven recommend to place profiles into the module where they are acting.

    If I want to insure that profiles deployWeb, createDatabase are run before the profiles runSelenium you have to keep that order in the pom.xml even if these profiles are acting in different Maven phase

    • createDatabase  may run in phase generate-test-resources 
    • deployWeb run in phase pre-integration-test
    • runSelenium run in phase integration-test

    Considering the module ordering in reactor, a good pom.xml candidate could be web/pom.xml like this



        createDatabase
     


        deployWeb
     


        runSelenium
     

    References

    http://maven.apache.org/pom.html#Profiles

  • Auto deployment of Maven artifacts to Oracle Weblogic

    apache_maven

    I found  this time a  new way to deploy Maven artefacts using the Oracle Weblogic Ant API!

    If you remember my previous post, there is many ways to deploy your war/ear to Oracle Weblogic

    1. Using Oracle Weblogic development mode, a mode in which a simple copy of your files in a specific autodeploy directory trigger the update/install of these
    2. Using Maven Cargo, this work only if your Oracle Weblogic container is local (see here) on the same machine, where Apache Maven is running
    3. Using a very old Maven plugin (2008), local and remote container are supported, but our builds were sometimes hanging during pre integration phase for no apparent reasons.

    And now using the official ANT API of Oracle, by far the MOST stable of all!

  • Behavior Driven Development with JBehave and Apache Maven

    apache_maven

    jbehave-logo

    I won’t explain you how to write any JBehave tests as the online documentation is more than complete.

    I prefer to show you how to make them run in eclipse, and in Apache Maven as the example were not easy to run (scenario are wrongly in src/main/java).

     

    JBehave is a framework for Behaviour-Driven Development
    Behaviour-driven development (BDD) is an evolution of test-driven development (TDD) and acceptance-test driven design, and is intended to make these practices more accessible and intuitive to newcomers and experts alike.
    It shifts the vocabulary from being test-based to behaviour-based, and positions itself as a design philosophy.
    You can find out more about behaviour-driven development on the BDD wiki, or in the article Introducing BDD

    Features of JBehave include:

    • Pure Java implementation, which plays well with Java-based enterprises.
    • Users can specify and run text-based user stories, which allows "out-in" development.
    • Annotation-based binding of textual steps to Java methods, with auto-conversion of string arguments to any parameter type (including generic types) via custom parameter converters.
    • Annotation-based configuration and Steps class specifications
    • Dependency Injection support allowing both configuration and Steps instances composed via your favourite container (Guice, PicoContainer, Spring).
    • Extensible story reporting: outputs stories executed in different human-readable file-based formats (HTML, TXT, XML). Fully style-able view.
    • Auto-generation of pending steps so the build is not broken by a missing step, but has option to configure breaking build for pending steps.
    • Localisation of user stories, allowing them to be written in any language.
    • IDE integration: stories can be run as JUnit tests or other annotation-based unit test frameworks, providing easy integration with your favourite IDE.
    • Ant integration: allows stories to be run via Ant task
    • Maven integration: allows stories to be run via Maven plugin at given build phase

    To make the online sample run easily without having to check out the whole tree of JBehave, I will show you that by slightly altering the pom.xml of a sample (Trader), you can run them against a fix version of JBehave.

    The whole pom.xml

    <profiles>
    <profile>
        jbehave
        <activation>
            <activebydefault>false</activebydefault>
        </activation>
        <build>
          <plugins>
            <plugin>
                <groupid>org.apache.maven.plugins</groupid>
                <artifactid>maven-dependency-plugin</artifactid>
                <executions>
                    <execution>
                        unpack-jbehave-site-resources
                        <phase>generate-resources</phase>
                        <goals>
                            <goal>unpack</goal>
                        </goals>
                        <configuration>
                            <overwritereleases>false</overwritereleases>
                            <overwritesnapshots>true</overwritesnapshots>
                            <artifactitems>
                                <artifactitem>
                                    <groupid>org.jbehave.site</groupid>
                                    <artifactid>jbehave-site-resources</artifactid>
                                    <version>2.0.2</version>
                                    <outputdirectory>
                                        ${project.build.directory}/jbehave-reports/rendered
                                    </outputdirectory>
                                </artifactitem>
                            </artifactitems>
                        </configuration>
                    </execution>
                    <execution>
                        unpack-jbehave-reports-resources
                        <phase>generate-resources</phase>
                        <goals>
                            <goal>unpack</goal>
                        </goals>
                        <configuration>
                            <overwritereleases>false</overwritereleases>
                            <overwritesnapshots>true</overwritesnapshots>
                            <artifactitems>
                                <artifactitem>
                                    <groupid>org.jbehave</groupid>
                                    <artifactid>jbehave-core</artifactid>
                                    <version>${jbehave.version}</version>
                                    <outputdirectory>
                                        ${project.build.directory}/jbehave-reports/rendered 
                                    </outputdirectory>
                                    <includes>
                                        **\/*.css,**\/*.ftl,**\/*.js
                                    </includes>
                                </artifactitem>
                            </artifactitems>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupid>org.jbehave</groupid>
                <artifactid>jbehave-maven-plugin</artifactid>
                <version>${jbehave.version}</version>
                <executions>
                    <execution>
                        run-scenarios-found
                        <phase>integration-test</phase>
                        <configuration>
                            <scenarioincludes>
                                <scenarioinclude>
                                    **/scenarios/*.java
                                </scenarioinclude>
                            </scenarioincludes>
                            <scenarioexcludes>
                                <scenarioexclude>
                                    **/i18n/scenarios/*.java
                                </scenarioexclude>
                            </scenarioexcludes>
                            <batch>false</batch>
                            <ignorefailure>
                                true
                            </ignorefailure>
                            <classloaderinjected>false
                            </classloaderinjected>
                            <scope>test</scope>
                        </configuration>
                        <goals>
                            <goal>run-scenarios</goal>
                        </goals>
                    </execution>
                    <execution>
                        run-i18n-scenarios-found
                        <phase>integration-test</phase>
                        <configuration>
                            <scenarioincludes>
                                <scenarioinclude>
                                    **/i18n/scenarios/*.java
                                </scenarioinclude>
                            </scenarioincludes>
                            <skip>false</skip>
                            <classloaderinjected>
                                true
                            </classloaderinjected>
                            <scope>test</scope>
                        </configuration>
                        <goals>
                            <goal>run-scenarios</goal>
                        </goals>
                    </execution>
                    <execution>
                        stepdoc
                        <phase>integration-test</phase>
                        <configuration>
                            <scenarioincludes>
                                <scenarioinclude>
                                    **/scenarios/*.java
                                </scenarioinclude>
                            </scenarioincludes>
                            <scenarioexcludes>
                                <scenarioexclude>
                                    **/scenarios/None.java
                                </scenarioexclude>
                            </scenarioexcludes>
                            <skip>false</skip>
                            <scope>test</scope>
                            <classloaderinjected>false
                            </classloaderinjected>
                        </configuration>
                        <goals>
                            <goal>stepdoc</goal>
                        </goals>
                    </execution>
                    <execution>
                        render-reports-generated
                        <phase>post-integration-test</phase>
                        <configuration>
                            <formats>
                                <format>txt</format>
                                <format>html</format>
                                <format>xml</format>
                            </formats>
                            <scope>test</scope>
                            <ignorefailure>true</ignorefailure>
    
                        </configuration>
                        <goals>
                            <goal>render-reports</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            </plugins>
        </build>
      <reporting>
        <excludedefaults>true</excludedefaults>
        <plugins>
            <plugin>
                <groupid>org.jbehave</groupid>
                <artifactid>jbehave-maven-plugin</artifactid>
                <version>${jbehave.version}</version>
                <configuration> </configuration>
                <reportsets>
                    <reportset>
                        <reports>
                            <report>render-reports</report>
                        </reports>
                    </reportset>
                </reportsets>
            </plugin>
        </plugins>
      </reporting>
    </profile>
    </profiles>
    
  • Break Maven build when there is a dependency conflict

    apache_maven_thumb

    Scenarios

    • You want to control Maven during dependency resolution and break the build if some conditions are not met,
    • You want to detect dependencies conflict early during the build,
    • You want to avoid anybody in your team to use the dependency x in the version y

    This is where the Maven Enforcer Plugin will assist you:

    The Enforcer plugin provides goals to control certain environmental constraints such as Maven version, JDK version and OS family along with many more standard rules and user created rules.

    Add in your pom.xml the following to configure the plugin

  • Code generation from XSD with JAXB and Maven

    apache_maven

    What you will learn in this small post

    • How to create JAXB proxies at build time using maven-jaxb2-plugin in a continuous build environment (TeamCity / Bamboo)
    • How to generate from an XSD file (XML-Schema-Definitions) Java code.

    Requirements

    • We will use JAXB2 (see JSR 222 and JAXB 2.x).
    • We use Maven 2.2.1, the latest available version

    Settings things up

    The only difficulties is to add to your Maven proxy (Archiva, artifactory) the Maven repository of Sun. The example below use an inline repositories definition in pom.xml. So it work out of the box.

    <repositories>
        <repository>
          <id>maven2-repository.dev.java.net</id>
          <name>Java.net Maven 2 Repository</name>
          <url>http://download.java.net/maven/2</url>
        </repository>
      </repositories>

    and the special Sun Maven plugin repository

      <pluginRepositories>
        <pluginRepository>
          <id>maven2-repository.dev.java.net</id>
          <url>http://download.java.net/maven/2</url>
        </pluginRepository>
      </pluginRepositories>

    Here is how your pom should look like

    <project xmlns=http://maven.apache.org/POM/4.0.0 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
      <modelVersion>4.0.0</modelVersion>
      <groupId>com.waltercedric.maven.examples</groupId>
      <artifactId>jaxb</artifactId>
      <version>0.1.0-SNAPSHOT</version>
      <packaging>jar</packaging>
      <name>jaxb</name>
      <build>
        <plugins>
          <plugin>
            <groupId>org.jvnet.jaxb2.maven2</groupId>
            <artifactId>maven-jaxb2-plugin</artifactId>
            <version>0.7.1</version>
            <executions>
              <execution>
                <goals>
                  <goal>generate</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
              <source>1.6</source>
              <target>1.6</target>
            </configuration>
          </plugin>
        </plugins>
      </build>
      <dependencies>
        <dependency>
          <groupId>com.sun.xml.bind</groupId>
          <artifactId>jaxb-impl</artifactId>
          <version>2.1.12</version>
        </dependency>
        <dependency>
          <groupId>junit</groupId>
          <artifactId>junit</artifactId>
          <version>4.7</version>
          <scope>test</scope>
        </dependency>
      </dependencies>
      <repositories>
        <repository>
          <id>maven2-repository.dev.java.net</id>
          <name>Java.net Maven 2 Repository</name>
          <url>http://download.java.net/maven/2</url>
        </repository>
      </repositories>  
    <pluginRepositories> <pluginRepository> <id>maven2-repository.dev.java.net</id> <name>Java.net Maven 2 Repository</name> <url>http://download.java.net/maven/2</url> </pluginRepository> </pluginRepositories>
    </
    project>

    All you have to do now is to place your XSD shema in src/main/resources and run mvn package

    The JAXB proxies will be created in target\generated-sources\xjc\generated so you can use them in src/main/java and src/test/java

  • Configuring TeamCity, Maven for PHP for Joomla continuous build

    apache_maven

    Doxygen phpDocumentator phpunit-logo teamcity512 maven4php

    Maven for PHP uses the power of Maven for building, reporting on and creating documentations of PHP projects. It adapts the Maven build lifecycle to the PHP world while fully supporting PHP 4 and PHP 5. PHP for Maven uses PHPUnitfor unit testing and doxygenfor creating the api documentation.
    Use a PHP library project to create a library that can be used by other PHP libraries or PHP web projects. Use a PHP web project to create a standalone web project.

    So I quickly describe what I did install on my root server (OpenSuse 11.X)

    My Objectives: being able to build all my Joomla! component using best agile development practices

    “Specific tools and techniques such as continuous integration, automated or xUnit test, pair programming, test driven development, design patterns, domain-driven design, code refactoring and other techniques are often used to improve quality and enhance project agility.”

    Maven

    While not needed as TeamCity has an integrated Maven engine, I would like to use an external MAVEN version, in order to have the latest version and living dangerously on the edge!

    So I download

    # wgethttp://apache.mirror.testserver.li/maven/binaries/apache-maven-2.1.0-bin.tar.gz

    And unpack

    # tar xvf apache-maven-2.1.0-bin.tar.gz

    Since I would like to avoid having a version number in my configuration build path, I create a symlink

    # ln -s apache-maven-2.1.0 maven

    I just now tell TeamCity to use my own Maven version, by specifying the Maven Home Path

    teamcity.maven4php 

    phpDocumentor

    phpDocumentor is an open source documentation generator written in PHP. It automatically parses PHP source code and produces readable API and source code documentation in a variety of formats. phpDocumentor generates documentation based on PHPDoc-formatted comments and the structure of the source code itself. It supports documentation of both object-oriented and procedural code. phpDocumentor can create documentation in HTML, PDF, CHM or Docbook formats.

    Can be installed using PEAR, simply run

    # pear upgrade PhpDocumentor
    downloading PhpDocumentor-1.4.2.tgz...
    Starting to download PhpDocumentor-1.4.2.tgz (2,421,028 bytes)
    ..............................................................................done: 2,421,028 bytes
    upgrade ok: channel://pear.php.net/PhpDocumentor-1.4.2

    PHPUnit

    PHPUnit is a unit testing framework for the PHP programming language. Created by Sebastian Bergmann, PHPUnit is one of the xUnit family of frameworks that originated with Kent Beck's SUnit.

    Can be installed using PEAR, simply run

    # pear upgrade PHPunit

    DOxygen

    Doxygen is a documentation generator for C++, C, Java, Objective-C, Python, IDL (Corba and Microsoft flavors), Fortran, VHDL, PHP, C#, and to some extent D. It runs on most Unix-like systems, including Mac OS X, as well as on Windows. The first version of Doxygen borrowed some code of an old version of DOC++; later, the Doxygen code was rewritten by Dimitri van Heesch. from www.doxygen.org

    # zypper se doxy
    Lese installierte Pakete...

    S | Name       | Zusammenfassung                                    | Typ 
    --+------------+----------------------------------------------------+------
      | doxygen    | Automated C, C++, and Java Documentation Generator | Paket
      | doxywizard | Graphical User Interface for Doxygen               | Paket

    # zypper in doxygen

    Herunterladen von Paket doxygen-1.5.5-17.1.x86_64 (1/1), 2,3 M (6,2 M installiert)
    Lade herunter: doxygen-1.5.5-17.1.x86_64.rpm [fertig] 
    Installiere: doxygen-1.5.5-17.1 [fertig]

    Artifactory

    Prepare Artifactory by adding new Maven for PHP repositories

    At http://maven.waltercedric.com/artifactory/webapp/repositoryconfig.html

    As Admin user (you cant go to that links without being an admin!), add 2 new repositories

    teamcity.maven4php.artifactory

    In Maven settings.xml

    In order to use Artifactory at his best (proxy and caching of remote repositories), I have a Maven settings.xml that contains ONLY

    # vi /home/teamcity/.m2/settings.xml

    <activeProfiles>
      <activeProfile>cedric-profile</activeProfile>
    </activeProfiles>

    <profiles>
       <profile>
            <id>cedric-profile</id>
            <activation>
            <activeByDefault>true</activeByDefault>
            </activation>
    <repositories>
        <repository>
            <id>central</id>
            <url>http://maven.waltercedric.com/artifactory/repo/</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>snapshots</id>
            <url>http://maven.waltercedric.com/artifactory/repo/</url>
            <releases>
                <enabled>false</enabled>
            </releases>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>central</id>
            <url>http://maven.waltercedric.com/artifactory/plugins-releases/</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </pluginRepository>
        <pluginRepository>
            <id>snapshots</id>
            <url>http://maven.waltercedric.com/artifactory/plugins-snapshots/</url>
            <releases>
                <enabled>false</enabled>
            </releases>
        </pluginRepository>
    </pluginRepositories>
        </profile>
      </profiles>
    </settings>

    Attention: Artifactory WIKI tell to use http://maven.waltercedric.com/artifactory/plugins-releases and not http://maven.waltercedric.com/artifactory/plugins-releases/ on my HOST with mod_proxy it made an error 404 If I do not add a slash at the end. Try with your host before, you will gain a lot of time by checking if the URL is valid!

    Note that http://url:port/artifactory/repo/ is a virtual repositories that proxy all external repositories

     

    Eclipse

    Use SolarJoomla (hopefully to be distributed this week) to have a running Eclipse, PDT, Maven 4 PHP environment, Mylyn, TeamCity in no time

    Lets build!

    First I create a new Maven Project with Archetype “Maven for PHP” “PHP5 libraries”

    In TeamCity I did create a new project “Joomla 15 components plugins and modules” and a new Build “MyGuestbook”.

    teamcity.maven4php.myguestbook

    The first build failed with

    [INFO] PHP Warning: require_once(PHPUnit/TextUI/TestRunner.php): failed to open stream: Operation not permitted

    This is because of my PHP security restrictions, I only allow file to be opened from /home/teamcity/

    So I just add

    /home/teamcity/TeamCity/buildAgent/ to my open_basedirin my php.ini

    ; open_basedir, if set, limits all file operations to the defined directory
    ; and below.  This directive makes most sense if used in a per-directory
    ; or per-virtualhost web server configuration file. This directive is
    ; *NOT* affected by whether Safe Mode is turned On or Off.
    open_basedir = /srv/www/vhosts:/tmp:/home/teamcity/TeamCity/buildAgent/

    To be continued

    So long an empty PHP project is building successfully, tomorrow I will try to make a REAL Joomla! component build there!

    As soon as It works, and all my Joomla! components are running in TeamCity, I will try to achieve the same goal in Bamboo, why? because it is simply . . .fun!

    Links

  • Continuous Build for Joomla

    Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage. Martin Fowler about continuous integration

    Things get clearer in my mind...I will use a set of Open Sources software to reach this ambitious goal.

    Brainstorming

    Maintain a Single Source Repository.

    Either Subversion running locally on windows/Linux, but I will stay with www.joomlaCode.org at the beginning

    Automate the Build

    • I am currently evaluation Maven for PHP but already have ANT script to build components and plugin for Joomla 1.5 (they are not generic enough at the moment)

    Make Your Build Self-Testing

    • PHP UNIT tests if available with be run at this stage using PHP command line.
    • Seleniumto automate components testing across many platforms, as it runs in many browsers and operating systems, and can be controlled by many programming languages and testing frameworks.
    • Some standard utilities to perform standard Joomla! processes: login, logout for Apache ANT or Selenium
    • I will develop either a ANT plugin or a set of Selenium test cases for deploying and removing
      • component: install, publish, remove
      • plugin: install, publish, remove
    • All these utilities will be available under GPL v3 and as such free to reuse, and improve.

    Everyone Commits Every Day

    • This is reserved to developer having a www.joomlaCode.org account and part of the development team.

    Every Commit Should Build the Mainline on an Integration Machine

    I will use TeamCity continuous build server as it is free for personal use (limited to 20 concurrent build), run on Windows and Linux but require a java VM to run (may not be wanted in a PHP environment). Anyway with ANT, it will be possible to use another build server like Cruise Control or PHP Under Control.

    Keep the Build Fast

    That is an objective :-)

    Test in a Clone of the Production Environment

    Joomla! Build farm

    • I can imagine a set of Joomla instances, ideally 5 of each version, aka Joomla! 1.5.3 to Joomla! 1.5.8 and Joomla! 1.0.10 to 1.0.15
    • Joomla instances will be recreated at build time (files and databases), that mean Joomla! will get newly installed and removed in case of successful build
    • All Joomla! instances will be running with XAMPP, ideally on port not available to the outside world for security reasons

    Make it Easy for Anyone to Get the Latest Executable

    Successful build (Artifacts) are only available if build is successful. Team City provide this with less effort (configuration)

    Everyone can see what's happening

    • A guest account will be available or a free public area with limited access to see the result of builds.
    • RSS feeds, emails and Instant messaging (Jabber) out of the box for end users or developers!

    Automate Deployment

    That will be, auto publish to some demo site in a configurable way. At the moment, at http://demo.waltercedric.com and http://demo2.waltercedric.com for me :-)

    Final words

    • I will provide a ready to use package for Windows and Linux and all scripts, so anybody will be able to run it also on your own.
    • Critical part will be documented in my WIKI at http://wiki.waltercedric.com direct link HERE

    It seem that nobody is providing such a package as I am after only one day at the top of search results in Google "continuous build joomla"

  • Continuous build for Joomla! part1/x

    Automatic installation of Joomla! runtime environments

    Main ideas

    Build is scalable

    Distributed build management optimize hardware resources utilization by parallelizing product builds within the build agents grid. With build chains support, it is even possible to break down a single build procedure into several parts to run them on different build agents — both in sequence and in parallel — using the same set of sources in all of them.

    I want to be able to test my components against many versions of Joomla!

    All versions of Joomla! are in subversion as zip files in an own SVN repository

    For example:

    • ${JOOMLA15_VCS_ROOT}  is svn:\\localhost\joomla1.5\trunk
    • ${JOOMLA10_VCS_ROOT}  is svn:\\localhost\joomla1.0\trunk

    These repository ${JOOMLAxx_VCS_ROOT} are connected to all build as supplementary VCS root in TeamCity and thus content get checked out as part of the build in the build temporary directory of one agent. ($AGENT_BUILD_DIR)

    joomla1.5\trunk
                                Joomla_1.5.4-Stable-Full_Package.zip
                                Joomla_1.5.5-Stable-Full_Package.zip
                                Joomla_1.5.6-Stable-Full_Package.zip
                                Joomla_1.5.7-Stable-Full_Package.zip
                                Joomla_1.5.8-Stable-Full_Package.zip

    So after the checkout, the file system will look like

    ($AGENT_BUILD_DIR)\                     

                                     Joomla_1.5.4-Stable-Full_Package.zip
                                Joomla_1.5.5-Stable-Full_Package.zip
                                Joomla_1.5.6-Stable-Full_Package.zip
                                Joomla_1.5.7-Stable-Full_Package.zip
                                Joomla_1.5.8-Stable-Full_Package.zip

    If you don't want to provide support a a specific version of Joomla! just remove it from the trunk! or add new ones on purpose. That's easy.

    Ant tasks/Maven MOJO

    • Are responsible for unpacking all these zip files to the build temporary agent directory. ($AGENT_BUILD_DIR).
    • Filenames are found with a configurable regular expression,
    • All settings will be committed in \joomla1.5\trunk\build.deploy.properties

    Another ant script/task will for each zip,

    • Start a Selenium test cases that will create a virtual user that use the regular Joomla! installer and drive installation till the end.
    • All settings which have to be Joomla! and build independent will be randomly generated, preferably UUID for password and database name for example.
    • Login and Admin password may be the same (admin:admin) at the beginning but can also be generated and written to a file on disk in ($BUILD_DIR)/{joomlarootversion}/build.install.properties.
    • Directory installation ($BUILD_DIR)/{joomlarootversion}/installation will be renamed to ($BUILD_DIR)/{joomlarootversion}/installation.old or simply deleted
    • Selenium/PHP Unit test that are committed in \joomla1.5\trunk\Installation.Checks will perform basic checks (login, logout, navigate) to ensure that installation of Joomla! has been successful.
      If everything succeed, we will have a set of Joomla! versions ready for our components regression testing.

    Remarks:

    • No build temporary directory. ($AGENT_BUILD_DIR) will be deleted by Ant or Maven but by the build server itself. This will let developer look at the issues on file system and in database.
    • New scripts may be developed to extract from the build server or Joomla! farm easily the non running Joomla! instance files + database) so developers can install the broken setup locally.

    Automatic deployment of Joomla! components

    Your component is typically shared and many developer committed regularly in a different VCS root... For SecurityImages 5.x.y, subversion root may be  svn:\\localhost\securityimages5\trunk

    This VCS root is also attached to the build and get check out at build time by TeamCity.

    Packaging

    if a build.xml is present in {VCS_ROOT}\build.xml then it is executed prior to any further operations, purpose of build.xml is to produce a component binary distribution (zip or tar.gz) that can be then installed to ALL Joomla install in the agent root directory.

    Deployment

    if a deploy.xml is present in {VCS_ROOT}\deploy.xml then it is executed, purpose of deploy.xml is to deploy one or many component binary distribution (zip or tar.gz) to ALL Joomla install in the agent root directory.

    Why one or many component?

    I want to be able to track also component dependencies issues.

    Lets say that SecurityImages does not play well with VirtueMart, I may want to test also that combination across Joomla! instances, that's why VirtueMart may have to be deployed with SecurityImages or not.

    prerequisites:

    • Running SVN server, see HERE for installing it on windows
    • Installed JVM, latest JDK 1.6u10
    • Running TeamCity server
    • Running XAMPP with HTTP root directory at TeamCity agent root directory.
    • Apache ANT with additional library for more control (if, case, for loop)

    This articles will be available in my WIKI soon http://wiki.waltercedric.com/index.php?title=ContinuousBuildforJoomla so any reader or developer can participate to the discussion, next step is to implement the above and that will e documented as well :-)

  • Continuous build of Apache projects

    Apache foundation is using Hudson for continuous build (and also JBOSS)

    Hudson monitors executions of repeated jobs, such as building a software project or jobs run by cron.
    Among those things, current Hudson focuses on the following two jobs:

    1. Building/testing software projects continuously, just like CruiseControl or DamageControl.
      In a nutshell, Hudson provides an easy-to-use so-called continuous integration system,
      making it easier for developers to integrate changes to the project, and making it easier
      for users to obtain a fresh build. The automated, continuous build increases the productivity.
    2. Monitoring executions of externally-run jobs, such as cron jobs and procmail jobs, even those
      that are run on a remote machine. For example, with cron, all you receive is regular e-mails that
      capture the output, and it is up to you to look at them diligently and notice when it broke.
      Hudson keeps those outputs and makes it easy for you to notice when something is wrong.

    This is a public build and test server for various Apache Projects.

     


     

    An Eclipse plugin is also available the online WIKI is HERE

    If you are shopping for a build server, and you are not on a budget, I can recommend you the
    excellent Team City from JetBrains. Maven integration is also good, but wont probably never reach the level
    of Hudson as it is made by Apache and tailored for their frameworks needs.

  • Continuous build server for Joomla!

    continuous.server.toolbox.for.php Starting from now on, in order to

    • Increase quality of my components and other (JoomlaComment :-))
    • Reduce time between releases,
    • Avoid subtle or recurrent issues

    I will set up a continuous integration  build server at 

    http://continuousbuildserver.waltercedric.com

    Continuous integration describes a set of software engineering practices that speed up the delivery of software by decreasing integration times.

    • Maintain a code repository, the code will stay at JoomlaCode.org subversion
    • Automate the build, with Maven for PHP/Ant and either teamcity or phpundercontrol.org
    • Make my build self-testing with PHP Unit and Selenium IDE
    • Everyone commits every day,
    • Every commit (to mainline) will be built with the help of triggers,
    • The build will be fast,
    • Test are done in production environment, by deploying code to  old and new version of Joomla! 1.0 and 1.5 "not web accessible" (for obvious security reasons),
    • It will be easy to get the latest deliverables by visiting http://continuousbuildserver.waltercedric.com
    • Everyone will see the results of the latest build by visiting http://continuousbuildserver.waltercedric.com
    • Deliverables will be automatically deployed.

    In short (for all non developer), this program will

    • Monitor any changes of code
    • Trigger a build,
    • Deploy the build to different versions of Joomla! 1.5 (5 versions should be enough)
    • Run a set of tests against these Joomla! versions and
    • Make the result available to all of you.

    Soon as there is enough test cases, it will be safe to download any new release from there.

  • Continuous build with Apache Maven

    apache_maven

    Maven is a software tool for Java project management and build automation created by Jason van Zyl in 2002. It is similar in functionality to the Apache Ant tool (and to a lesser extent, PHP's PEAR and Perl's CPAN), but has a simpler build configuration model, based on an XML format. Maven is hosted by the Apache Software Foundation, where it was formerly part of the Jakarta Project.

    Maven uses a construct known as a Project Object Model (POM) to describe the software project
    being built, its dependencies on other external modules and components, and the build order.
    It comes with pre-defined targets for performing certain well defined tasks such as compilation
    of code and its packaging.

    A key feature of Maven is that it is network-ready.The core engine can dynamically download
    plug-ins from a repository
    , the same repository that provides access to many versions of different
    Open Source Java projects, from Apache and other organizations and developers. This repository
    and its reorganized successor, the Maven 2 repository, strives to be the de facto distribution
    mechanism for Java applications, but its adoption has been slow. Mavenprovides built in support
    not just for retrieving files from this repository, but to upload artifacts at the end of the build.
    A local cache of downloaded artifacts acts as the primary means of synchronizing the output of
    projects on a local system.

    Mavenis based on a plugin-based architecture that allows it to make use of any application
    controllable through standard input. Theoretically, this would allow anyone to write plugins to
    interface with build tools (compilers, unit test tools, etc.) for any other language.
    from WikiPedia

  • Continuous integration server Bamboo up and running

    atlassian.bamboo.logo My Bamboo continuous integration server is now fully functional and available at  http://bamboo.waltercedric.com/

     

     

     

    Remember Atlassian is providing free license for Open Source Projects:

    Atlassian supports and believes in the Open Source movement -Bamboo utilizes a number of good Open Source components, and Atlassian developers are committers on a large number of Open Source projects.

    To give back to the open source community (and hopefully improve the quality of those projects!),Bamboo is free for any Open Source project to use.

    There are a few requirements for an Open Source license, the main ones being:

    • Established code base
    • Publicly available project website
    • Using an approved open source license
    • YourBamboo instance will be publicly accessible

    My objective is to make Joomla! and all my projects also running in Bamboo (not only in TeamCity as the limit of 20 builds will be rapidly reached)

    Visit it by clicking on the picture

    Very quick Bamboo install how to

    Install a fresh Tomcat 6 runtime,

    Move the war into the ROOT web context of tomcat

    Choose free port for HTTP, AJP, and Tomcat server port  in conf/server.xml

    <Server port="8050" shutdown="SHUTDOWN">
    
    <Connector port="8051" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443"
               enableLookup="false"         
               useBodyEncodingForURI="true" 
    
    <Connector port="8030" protocol="AJP/1.3" redirectPort="8443" />

    Copy mysqldriver.jar into /lib

    Configure the data source into server.xml

    <Host name="localhost"  appBase="webapps"
                unpackWARs="true" autoDeploy="true"
                xmlValidation="false" xmlNamespaceAware="false">
            <Resource name="jdbc/BambooDS" auth="Container" type="javax.sql.DataSource"
                username="user"
                password="pwd"
                driverClassName="com.mysql.jdbc.Driver"            
                url="jdbc:mysql://localhost/schema?autoReconnect=true"            
                maxActive="20"
                validationQuery="select 1" />

    Edit /etc/apache/worker.properties

    worker.list=ajp13, teamcity, jira, bamboo
    worker.bamboo.port=8030
    worker.bamboo.host=localhost
    worker.bamboo.type=ajp13

     

    Create a vhost.conf in the subdomains

    # vi /srv/www/vhosts/waltercedric.com/subdomains/bamboo/conf/vhost.conf

    ServerName bamboo.waltercedric.com
    
    ProxyPass /  ajp://bamboo.waltercedric.com:8030/
    <Proxy *>
       Order Allow,Deny
       Allow from all
    </Proxy>
    
    <Directory />
      Options FollowSymLinks
      AllowOverride None
    </Directory>

    To tell plesk to include an overridden vhost.conf, run

    # /usr/local/psa/admin/sbin/websrvmng

    Restart Apache2

    rcapache2 restart

  • ContinuousBuild4Joomla project submitted to JoomlaCode.org

    I will commit soon a first draft (alpha) of what is expected to bring continuous build to any Joomla! component (or event to Joomla! core itself ;-))

    You are free to join the project, all documentation effort stay at the moment in my WIKI

     

  • Deploy to Tomcat 6 using Maven

    apache_maven

     cargo-banner-left
    A ready to use example on how you can deploy your web application to a Tomcat 6 container using Maven Cargo. Cargo is a thin wrapper that allows you to manipulate Java EE containers in a standard way. 

    Cargo provides the following Tools and APIs:

    • A Java API to start/stop/configure Java Containers and deploy modules into them.
    • A Java API to parse/create/merge Java EE Modules
    • Ant tasks, Maven 1, Maven 2 plugins.
    • Intellij IDEA and Netbeans plugins are in the sandbox.

    First you have to decide if your tomcat server run locally or remotely as this influence the way you’ll configure your pom.xml

    Below is an example of a standard architecture

    |---MyApplication
    |           |- ear       (ear)
    |           |- service (jar)
    |           |- client    (jar)
    |           |- web      (war)
    |           |- integration (jar)

    The most interesting Maven module, which will be the subject of this article, and the next one is describing how to automate the deployment of a war to Tomcat and later on running integration tests using Selenium.

    'Integration testing'  is the activity of software testing in which individual software modules are combined and tested as a group. It occurs after unit testing and before system testing. Integration testing takes as its input modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing. [WikiPedia]

    Deploying to Tomcat running locally

    Locally mean running on your development machine, at localhost 8080 or on any other port. Maven has the concept of a 'phase' which can be thought of a collection of goals. Hence here we are specifying that during the
    ’pre-integration-test’ phase first deploy the web app to the container specific folder and then start the container. Both 'deployer-deploy' and 'start' are cargo specific goals. The code below is already ready for integration testing!

    <!—Example of Configuration –>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.cargo</groupId>
                <artifactId>cargo-maven2-plugin</artifactId>
                <configuration>
                    <wait>true</wait>
                    <container>
                        <containerId>tomcat6x</containerId>
                        <type>installed</type>
                        <home>${catalina.home}</home>
                    </container>
                    <configuration>
                        <type>existing</type> 
                        <home>${catalina.home}</home>
                    <properties>
                            <cargo.tomcat.manager.url>https://yourhost/manager</cargo.tomcat.manager.url>
                            <cargo.remote.username>ManagerAdminLogin</cargo.remote.username>
                            <cargo.remote.password>ManagerAdminPassword</cargo.remote.password>
                        </properties>
                    </configuration>
                    <deployer>
                        <type>installed</type>
                        <deployables>
                            <deployable>
                                <groupId>com.waltercedric</groupId>
                                <artifactId>myapplication-web</artifactId>
                                <type>war</type>
                            </deployable>
                        </deployables>
                    </deployer>
                </configuration>
                <executions>
                    <execution>
                        <id>start-container</id>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>deployer-deploy</goal>
                            <!--  Only local containers can be started 
                            <goal>start</goal>
                            -->
                        </goals>
                    </execution>
                    <execution>
                        <id>stop-container</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>deployer-undeploy</goal>
                            <!--  Only local containers can be started 
                            <goal>stop</goal>
                            -->
                        </goals>
                    </execution>
                    <execution>
                        <id>verify-deploy</id>
                        <phase>install</phase>
                        <goals>
                            <goal>deployer-deploy</goal>
                        </goals>
                    </execution> 
                    <execution>
                        <id>clean-undeploy</id>
                        <phase>pre-clean</phase>
                        <goals>
                            <goal>deployer-undeploy</goal>
                            <!--  Only local containers can be started 
                            <goal>stop</goal>
                            -->
                        </goals>
                    </execution> 
    
                </executions>
            </plugin>
        </plugins>
    </build>

    Deploying to Tomcat running remotely

    The code slightly change:

    • You can NOT start and stop Tomcat running remotely, only deploy and un deploy your web application
    • ‘installed’ is replaced by ‘remote’
    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.cargo</groupId>
                <artifactId>cargo-maven2-plugin</artifactId>
                <configuration>
                    <wait>true</wait>
                    <container>
                        <containerId>tomcat6x</containerId>
                        <type>remote</type>
                    </container>
                    <configuration>
                        <type>remote</type> 
                        <properties>
                            <cargo.tomcat.manager.url>https://yourhost/manager</cargo.tomcat.manager.url>
                            <cargo.remote.username>ManagerAdminLogin</cargo.remote.username>
                            <cargo.remote.password>ManagerAdminPassword</cargo.remote.password>
                        </properties>
                    </configuration>
                    <deployer>
                        <type>installed</type>
                        <deployables>
                            <deployable>
                                <groupId>com.waltercedric</groupId>
                                <artifactId>myapplication-web</artifactId>
                                <type>war</type>
                            </deployable>
                        </deployables>
                    </deployer>
                </configuration>
                <executions>
                    <execution>
                        <id>start-container</id>
                        <phase>pre-integration-test</phase>
                        <goals>
                            <goal>deployer-deploy</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>stop-container</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>deployer-undeploy</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>verify-deploy</id>
                        <phase>install</phase>
                        <goals>
                            <goal>deployer-deploy</goal>
                        </goals>
                    </execution> 
                    <execution>
                        <id>clean-undeploy</id>
                        <phase>pre-clean</phase>
                        <goals>
                            <goal>deployer-undeploy</goal>
                        </goals>
                    </execution> 
                </executions>
            </plugin>
        </plugins>
    </build>

    If you don’t want to let cargo deploy your web application artefact under the default name myapplication-web-0.0.1.SNAPSHOT.war, you can add the following to the deployable section of cargo

    <plugin>
        <groupId>org.codehaus.cargo</groupId>
        <artifactId>cargo-maven2-plugin</artifactId>
        <configuration>
            <deployer>
                <deployables>
                    <deployable>
                        <properties>
                            <context>mywebapp</context>
                        </properties>
                    </deployable>
                </deployables>
            </deployer>
        </configuration>
    </plugin>

    So you’ll be able to access your web application with http://localhost/mywebapp instead of http://localhost/myapplication-web-0.0.1.SNAPSHOT

    From now on, any phase higher than ’pre-integration-test’ will trigger the deployment to your web application to any tomcat, jboss or weblogic container. As a reminder, here are the major phase of Maven, You can put many of them just separate by a space in run as - goals

    • validate - validate the project is correct and all necessary information is available
    • generate-sources - generate any source code for inclusion in compilation
    • process-sources - process the source code, for example to filter any values
    • generate-resources - generate resources for inclusion in the package
    • process-resources - copy and process the resources into the destination directory, ready for packaging
    • compile - compile the source code of the project
    • process-classes - post-process the generated files from compilation, for example to do byte code enhancement on Java classes
    • generate-test-sources - generate any test source code for inclusion in compilation
    • process-test-sources - process the test source code, for example to filter any values
    • generate-test-resources - create resources for testing
    • process-test-resources - copy and process the resources into the test destination directory
    • test-compile - compile the test source code into the test destination directory
    • test - run tests using a suitable unit testing framework. These tests should not require the code be packaged or deployed
    • prepare-package - perform any operations necessary to prepare a package before the actual packaging.
    • package - take the compiled code and package it in its distributable format, such as a JAR
    • pre-integration-test - perform actions required before integration tests are executed. This may involve things such as setting up the required environment
    • integration-test - process and deploy the package if necessary into an environment where integration tests can be run
    • post-integration-test - perform actions required after integration tests have been executed. This may including cleaning up the environment
    • verify - run any checks to verify the package is valid and meets quality criteria
    • install - install the package into the local repository, for use as a dependency in other projects locally
    • deploy - done in an integration or release environment, copies the final package to the remote repository for sharing with other developers and projects

    I recommend you also to use:

    • Maven profiles (triggered by environment, Linux, Windows, DEV, TEST, PROD)
    • Key value pair for sensitive data (login, password)
    • Key value pair for contextual data (URL’s of management console)

    All of these data can be saved in your Maven settings.xml or given by system variables at build time.

  • Development shift in the way I deliver my latest Joomla! extensions

    agile.development

     

    Starting from now on, I will deliver all my latest (unstable) extensions versions through  my continuous build server. All request or bugs discovery that are requested in my forums and solved will lead to a new build that you will be able to download a lot faster than before.

    Thanks to Maven for PHP, I can now commit, 60 seconds later, unit test run and  the result is a direct download for my extensions snapshots.

     

     

     

     

     

     

    Here is an example with the module mod_related_thumb_items

    Head to http://teamcity.waltercedric.com/teamcity/guestLogin.html?guest=1

    Locate the module or component you are interested in:

    HowTODownloadLatest

    Click on the latest build, must be  Success

    HowTODownloadLatest.01  

    If this build is a direct answer to a support request in my forums, or solve an issue, You should be able to see in changes the commit description, and even which file have been changed after and before the commit.

    http://teamcity.waltercedric.com/teamcity/viewLog.html?buildId=217&buildTypeId=bt3&tab=buildChangesDiv

    HowTODownloadLatest.05

    But Hey! you want to download this latest build now, go to artifact

    HowTODownloadLatest.02

    Staying  on the edge by using RSS

    You can monitor any build by using the RSS icon in your browser toolbar, or example with this module, it would be

    http://teamcity.waltercedric.com/guestAuth/feed.html?buildTypeId=bt20&itemsType=builds&userKey=guest

    This way of downloading the latest extensions do not replace the page http://www.waltercedric.com/joomla-releases-mainmenu-269.html where there is there only stable versions.

    The next step is to make the maven phase “site” work (I have issue with phpdocumentor not found), this will create automatically a internet site in one of my sub-domains ad hide this complexity.