items tagged with subversion

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily - leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage. Martin Fowler about continuous integration

Things get clearer in my mind...I will use a set of Open Sources software to reach this ambitious goal.

Brainstorming

Maintain a Single Source Repository.

Either Subversion running locally on windows/Linux, but I will stay with www.joomlaCode.org at the beginning

Automate the Build

• I am currently evaluation Maven for PHP but already have ANT script to build components and plugin for Joomla 1.5 (they are not generic enough at the moment)

Make Your Build Self-Testing

• PHP UNIT tests if available with be run at this stage using PHP command line.
• Selenium to automate components testing across many platforms, as it runs in many browsers and operating systems, and can be controlled by many programming languages and testing frameworks.
• Some standard utilities to perform standard Joomla! processes: login, logout for Apache ANT or Selenium
• I will develop either a ANT plugin or a set of Selenium test cases for deploying and removing
  • component: install, publish, remove
  • plugin: install, publish, remove
• All these utilities will be available under GPL v3 and as such free to reuse, and improve.

Everyone Commits Every Day

• This is reserved to developer having a www.joomlaCode.org account and part of the development team.

Every Commit Should Build the Mainline on an Integration Machine

I will use TeamCity continuous build server as it is free for personal use (limited to 20 concurrent build), run on Windows and Linux but require a java VM to run (may not be wanted in a PHP environment). Anyway with ANT, it will be possible to use another build server like Cruise Control or PHP Under Control.

Keep the Build Fast

That is an objective :-)

Test in a Clone of the Production Environment

Joomla! Build farm

• I can imagine a set of Joomla instances, ideally 5 of each version, aka Joomla! 1.5.3 to Joomla! 1.5.8 and Joomla! 1.0.10 to 1.0.15
• Joomla instances will be recreated at build time (files and databases), that mean Joomla! will get newly installed and removed in case of successful build
• All Joomla! instances will be running with XAMPP, ideally on port not available to the outside world for security reasons

Make it Easy for Anyone to Get the Latest Executable

Successful build (Artifacts) are only available if build is successful. Team City provide this with less effort (configuration)

Everyone can see what's happening

• A guest account will be available or a free public area with limited access to see the result of builds.
• RSS feeds, emails and Instant messaging (Jabber) out of the box for end users or developers!

Automate Deployment

That will be, auto publish to some demo site in a configurable way. At the moment, at http://demo.waltercedric.com and http://demo2.waltercedric.com for me :-)

Final words

• I will provide a ready to use package for Windows and Linux and all scripts, so anybody will be able to run it also on your own.
• Critical part will be documented in my WIKI at http://wiki.waltercedric.com direct link HERE

It seem that nobody is providing such a package as I am after only one day at the top of search results in Google "continuous build joomla"

In order to better develop for Joomla by having your own subversion local repositories.

My second objective is to version different version of Joomla!/my components so I can
run a continuous build strategy with XINC...but more on that later when I will have enough regression tests :-)

Download the Subversion distribution for you operating system at: http://subversion.tigris.org/getting.html

Subversion Binary Packages
AIX	openCollabNet	Debian	Debian Project apt-get install subversion
HP-UX	HP Porting Centre	Fedora	Fedora Project yum install subversion
Mac OS X	Fink package (Requires Fink) MacPorts (Requires MacPorts) openCollabNet (Universal)	FreeBSD	FreeBSD Project cd /usr/ports/devel/subversion; make install
Red Hat	CollabNet (Certified binaries) SummerSoft (Red Hat 8,9 +)	NetBSD	pkgsrc cd /usr/pkgsrc/devel/subversion; make install clean
Solaris	CollabNet (Certified binaries, SPARC) Sunfreeware (SPARC & i386)	OpenBSD	OpenBSD Project cd /usr/ports/devel/subversion; make install
Windows	CollabNet (Certified binaries) Tigris.org: Apache 2.0 or Apache 2.2 VisualSVN (VisualSVN Server)	Ubuntu	Ubuntu Packages apt-get install subversion

The first thing you'll need is a new repository. A repository is where all of your individual projects will be stored.

Create a new directory using the file explorer or with

# mkdir e:\xsubrversion

Initialize it by issuing this command:

# svnadmin create e:\xsubrversion

Inside your new repository (e:\xsubrversion ) there is a directory called /conf. There you'll find a file
svnserve.conf. Open this file in a text editor.

Just uncomment the following lines by removing the #:

In the same directory, you'll find another text file called passwd, if it do not exist, just create it and put all
users authorize to access subversion in a [users] section. It is a pure key value pair file in the format
NAME = PASSWORD

Restart the service "Subversion Server"

In Eclipse PDT/PHP Eclipse, you can now connect to Subversion with Subversive client (Subclipse is not part of
Eclipse consortium)

URL: svn://localhost
User: cedric.walter
Password: xxxxxxxxxxxx

  and be able to commit and browse you code

I am developing for Joomla! now since the beginning but I was always disturbed by
the way people are using the code repository of Joomla! (www.joomlacode.org).
I am doing professional java development but was never able to reproduce the way
I am working nor really did take the time to really do it correctly (help by the fact that I
was developing always alone). Things changes... and now I am part of the effort of
bringing JoomlaComment to run natively in Joomla! 1.5 in a team of 3 people.

What people are doing, most of the time, is committing the code they write to Joomla
SVN/CVS in a flat way, it is just a set of files, or sometime even only the deliverable
(a set of zip that user can install)

So these kind of subversion layout are not uncommon

All of these layout are highly not efficient...The point of SVN/CVS
is to  compare and synchronize code easily (you merge code nearly 40% of your time
as a developer), and resolve code conflicts. SVN only store differences so sync is fast
and repository size is very low.

Subversion® is an open-source version control system. Subversion® allows developers to share
there projects on the repositories, where they are stored afterwards. Repository is much similar
to a file server, except the fact, that it not only stores the copy of the file system, but its previous
states and changing history. Subversion® access its repositories using network, so it provides a
probability for a person to work over some shared files and watch for every possible changes made
by other developers.

With all of the above layout A, B or C, the code committed is not connected to the runtime
where you are developing!

The developer runtime is a running Joomla instance, and developer change their code directly
in the file system of Joomla!. If you work in a group, you'll be force to connect another project
with the layout  A, B or C and merge code change manually from the local Joomla! instance to the shared
directory.

Recommended SVN layout

I am now using the recommended SVN layout:  /trunk /tags /branches

Trunk is a main (head) line of development. That's where you share your project and do initial commit.
The code in trunk is considered not stable or in development. Usually nobody  should commit code which
break the trunk (to know it there is nothing else better than regression testing and continuous build server)

Tag is a snapshot of the project state. You can create a tag of the one specified revision or a tag, containing
resources of different revisions. Tags are a kind of specific labels for a set of files each with its own revision
number. Used to track the important events if project's life cycle.

Branching means creating a new line of development on the repository location. This may be useful in
a different cases, for example if different clients wish to get the same product but with some differences
in functionality. Of course it's not convenient to create both products from the beginning to the end
separately, so the developers create branches. Branches are the additional lines of development.
Used when branching for different versions from one initial is needed or when each developer has his
own development line and plan.

I recommend you using Eclipse PDT (free),  So I have created 2 new PHP project in eclipse:
securityimages4 (Joomla! 1.4) and securityimages5 (Joomla! 1.5). For eaxmple securityimages5

• Contains Joomla 1.5.3, my code is sometimes highly dependent on a specific Joomla API version,
  or may break if Joomla! release a new version, so it has to be tested against some specific version of it
• Has the latest version of securityimages installed  (5.0.0Rc1)
• Has a MYSQL dump securityimages.sql which create a DB securityimages4 with its data, so anybody can have
  a running environment just by doing a checkout from trunk
• Has an ANT build file, more on it later.

This project is fully versioned (so even Joomla! and MYSQL dump). What could be disturbing at first but make sense
to version also the container (Joomla!) where your code run.

Carefully commit

While working in a team, not all file has to be committed to /trunk

The configuration.php for example is a default one (mine), anybody can make change but shall not commit it to the trunk! ->
Not all developer use XAMPP at the same location or may use another DB user (this is another topic among a team of developer:
standardize developer environment). I've commit dummy login/password for administrator panel is admin:admin
The login for MYSQL in  configuration.php is root: empty.

Development

When I develop, alone or in a team, I do daily before every change a Team synchronize on project securityimages4 in order
to see what has changed or who has done what and on which lines.

I it highly recommended not to update without looking at what is coming from SVN, so anybody can give give feedback on
code quality better if you look at it), you get update from the trunk, or commit your new code change to the trunk.

As soon as I have no visible bug and all functionalities, I create a new release

Create a new release

When I want to make a new release, I launch in eclipse an ant build  (right click – run as ant build on / securityimages4/build.xml)

This create the deliverable (a component zip or N files as zip with a version number) that I can publish/version also in another project
or NOT version at all since I can at any time load a project tag of securityimages4 and recreate the deliverable build!

The deliverable are what I publish on my site.

The ANT build file look like:

   1:  <?xml version="1.0" encoding="UTF-8"?>

   2:  <project name="securityimages" default="_build">

   3:  <description>Create a new release of joomla component</description>

   4:   

   5:  <property name="workspace.dir" value="." />

   6:  <property name="component.version" value="5.0.0RC2" />

   7:  <property name="component.name" value="securityimages" />

   8:   

   9:  <property name="destination.dir" value="c:\temp\${component.name}-${component.version}" />

  10:   

  11:  <target name="_preparedirs">

  12:  <mkdir dir="${destination.dir}" />

  13:  </target>

  14:   

  15:  <target name="_build" depends="_preparedirs" description="deploy joomla component">

  16:  <copy todir="${destination.dir}/components" filtering="on">

  17:  <fileset dir="${workspace.dir}/administrator/components/com_${component.name}" includes="**/*.*" />

  18:  <fileset dir="${workspace.dir}/components/com_${component.name}" includes="**/*.*" />

  19:  <fileset dir="${workspace.dir}/plugins/seystem" includes="${component.name}.*" />

  20:  </copy>

  21:  <copy todir="${destination.dir}/plugin" filtering="on">

  22:  <fileset dir="${workspace.dir}/plugins/seystem" includes="${component.name}.*" />

  23:  </copy>

  24:  </target>

  25:  </project>

Its purpose it to build a new deliverable in one click without going through the file system over and over.

I also version the project securityimages5 with the same version number, so I can then continue developing in trunk.

Branching an existing release

If someone report an issue, lets say in securityimages5.0.1.zip and code in /trunk is too far away in trunk or still unstable
to be released, I make a branch on project securityimages5 

I load the tag version 5.0.1 of the project securityimages5  in my eclipse workspace (Team replace with - tags) and open
a branch (Team – open branch)  and this project is now suddenly similar to another trunk:

• A branch is not unique and is reacting like a trunk
• The tag 5.0.1 still exist and is read only
• I can now can commit new code in the branch (= to a trunk)
• The main /trunk still exist and is not touched

When all issues are solved, after many commit to the branch (= to a trunk), I version the branch with a …. Tag, for example
5.0.2  which is a normal tag and represent a new stable state in the lifetime of the software.

Nasty back porting

Somehow a bug in version 5.0.1 has a lot of chance of also being in the latest version (the code in /trunk). So I am forced to
load the trunk (= current code) and make Team – compare with – another branch – tag version - 5.0.2 and back port all code
changes in trunk if it make sense.

Remember a tag should never be change, it is possible to overwrite tag, but this should occur 0.000001% of the developer lifetime.
A tag is a freeze code status from the past and the past can not be change ;-)

Joomla Releasing new versions

Joomla release a new version? I install the new version of Joomla! in my eclipse project securityimages5 If securityimages still
work, nice! if not I make changes for that version of Joomla! and create a … new tag of securityimages.

Conclusions

You are now able to compare the trunk with any working/non working versions (tag or branches) of the past. See what has committed what
and where in the file system of Joomla! and this by using the merging client of Eclipse SVN/CVS client which is way more efficient than any diff
for merging code. This were before with version directory in Subversion (SVN/CVS) not possible. 

These things have disturbed us (The developer Team at Innoveo.com) a lot in the past months. We did solve them recently, and I would like to publish them now here to help more people

Someone create a new maven module, after updating from SVN the module is not visible as a separate project.

Scenario:

1. User A checks out a maven project from SVN using "Checkout as Maven Project". All modules are now listed as separate projects in Eclipse
2. User B creates a new module in the project, and checks it into SVN
3. User A updates project
4. New module does not show up as a separate project.

Solution:

1. Work around found by selecting the parent project
2. Do Import -> Maven.
3. Select then the same name template and most modules should be grayed out because of name conflicts. But you should see the missing module in the list
4. Tick the one (new) module and import it.

In SVN perspective, when I choose "Checkout as Maven Project" on maven projectA, I get an exception saying that maven.123457896 can not be renamed

Scenario:

This error occur sometime, especially if you ever kill eclipse during a previous maven checkout (as sometimes it seems to hang forever). In fact the error message can be misleading,as M2Eclipse can not renamed maven.1234567896 to projectA as it may partially exist on disk.

Solution:

1. Stop eclipse
2. Go to the workspace location {workpsace_loc} 
3. Delete the directory maven.1234567896 or any directory starting with maven.xxxxxxxx
4. Delete also the temporary created Maven project  {workpsace_loc}\projectA directory you were trying to checkout if it exist.
5. Restart eclipse, and in SVN perspective, on maven project A, retry and  select "Checkout as maven project"

.classpath or .project are not committed in SVN, how to add them?

Scenario:

You may have add a svn:ignore on some directories, or some someone may have committed a recursive svn:ignore properties on some module in the hierarchy. While we should never commit any .classpath to SVN, there is some rare case where it is still needed. For example if you ever add special runtime server libraries that may not be coming from Maven Dependencies.

Solution:

Even if there is a svn:ignore on a maven module, or if a module has applied svn:ignore properties to all its child, you can always put a file under version control by doing the following

1. Go in SVN perspective
2. Drill down to the Maven module location or directory in which you would like to add a file
3. Right click New... then choose File,
4. A pop up will open letting you choose a file on disk
5. Don’t forget to Enter a commit comment

Maven Surefire runs multiple times our test case when using goal site

This is neither a bug or an issue of Maven, it look like even to be a feature!

Some reporting plugin are modifying (instrumentation) the java byte code of test cases like with Cobertura (goal: cobertura:cobertura).

The Cobertura tool is a free and easy to use source code test coverage analyses. It helps you to discover where your source-code lacks in test coverage.

In some rare case scenario (multi threaded test cases for example), it may be worth to run the code twice as instrumentation may modify behaviors and outcome of tests. So to resume, maven Surefire run them once, then Cobertura one more time but instrumented. One solution among other to escape this is to use Maven profiles, and to rely on another build that do not use reporting to run tests without instrumentation.

More to come ..

I was receiving some strange errors (HTTP error 501) while committing to my public subversion (http://svn.waltercedric.com), all my problems were related to mod security

Yes but..