analysis

Analysis is the process of breaking a complex topic or substance into smaller parts to gain a better understanding of it. read more at WikiPedia

  • logo_acunetix

    I will use it on my host very soon, if you have your own root server, this tool must be part of your administrator toolbox. Joomla! team use it to test the core framework, so we should be on the safe side, unfortunately we are are all using too many plug-ins that may be unsecure.. Here is how a report generated using Acunetix WVS look like (PDF - 1.5MB).

    Acunetix Web Vulnerability Scanner automatically scans your web applications / website (shopping carts, forms, dynamic content, etc.) and web services for vulnerabilities such as SQL injection, Blind SQL Injection, Cross site scripting, Google hacking, CRLF Injection & other web attacks. Acunetix crawls and analyzes websites including flash content, AJAX / Web 2.0. Also includes reporting for PCI Compliance, OWASP & more

    Out of the 100,000 websites scanned by Acunetix WVS, 42% were found to be vulnerable to Cross Site Scripting. XSS is extremely dangerous and the number of the attacks is on the rise. Hackers are manipulating these vulnerabilities to steal organizations’ sensitive data. Can you afford to be next?

    Cross Site Scripting allows an attacker to embed malicious JavaScript, VBScript, ActiveX, HTML, or Flash into a vulnerable dynamic page to fool the user, executing the script on his machine in order to gather data. Exploited Cross Site Scripting is commonly used to achieve the following malicious results:

    • Identity theft
    • Accessing sensitive or restricted information
    • Gaining free access to otherwise paid for content
    • Spying on user’s web browsing habits
    • Altering browser functionality
    • Public defamation of an individual or corporation
    • Web application defacement
    • Denial of Service attacks

    Scan your website for Cross Site Scripting Vulnerabilities at no cost NOW

    Get an insight into Acunetix Manual

  • apache

    The acronym LAMP refers to a solution stack of software, usually free and open source software, used to run dynamic Web sites or servers. It stand for:

    • Linux, for the operating system;
    • Apache, the Web server;
    • MySQL, the database management system (or database server);
    • Perl, Python, and PHP, the programming languages.

    &160;ab is a tool for benchmarking your Apache Hypertext Transfer Protocol (HTTP) server. It is designed to give you an impression of how your current Apache installation performs. This especially shows you how many requests per second your Apache installation is capable of serving.
    Apache-utils package contains utility programs for webservers and some add-on programs useful for any webserver. These include:

    • ab (Apache benchmark tool)
    • Logresolve (Resolve IP addresses to hostname in logfiles)
    • htpasswd (Manipulate basic authentication files)
    • htdigest (Manipulate digest authentication files)
    • dbmmanage (Manipulate basic authentication files in DBM format, using perl)
    • htdbm (Manipulate basic authentication files in DBM format, using APR)
    • rotatelogs (Periodically stop writing to a logfile and open a new one)
    • split-logfile (Split a single log including multiple vhosts)
    • checkgid (Checks whether the caller can setgid to the specified group)
    • check_forensic (Extract mod_log_forensic output from apache log files)

    This package Apache-Utils can be install through apt or YaST depending if you are using a Debian base distro or OpenSuse

    Prerequistes

    • Define realistic objectives, do not create too much virtual clients if you do not have usually that kind of user traffic..
    • For example an objective could be: number of users served, or percentage of the requests served within a certain time
    • This tool ab do not simulate realistic user behavior, it just hit a page without being able to simulate a complex workflow (like login, navigate and do things users usually do)
    • Try to monitor at the same time the CPU/Memory consumed in order not to make false assumption on apache settings (use top d 1)

    Attention

    It is an iterative process!

    1. Benchmark,
    2. Change settings and
    3. Restart benchmark.

    It is very important to only change a setting a time in order to better identify what is really bringing something! By changing only one settings at a time, you can:

    • Better see the influence on CPU, memory (you must look also at resources, a server swapping to disk is never good)
    • There is not so much universal settings bringing a speed kick (except DNSlookup off, keep alive small), some settings are depending on your Linux kernel version, CPU class, disk speed, network latency

    Other components

    mysql While tuning apache, you will see that most of the time is used in PHP/MySQL, for MySQL recommend to run at the same time tuning-primer.sh, read more here


    Usage

    ab [ -A auth-username:password ] [ -c concurrency ] [ -C cookie-name=value ] [ -d ] [ -e csv-file ] [ -g gnuplot-file ] [ -h ] [ -H custom-header ] [ -i ] [ -k ] [ -n requests ] [ -p POST-file ] [ -P proxy-auth-username:password ] [ -q ] [ -s ] [ -S ] [ -t timelimit ] [ -T content-type ] [ -v verbosity] [ -V ] [ -w ] [ -x <table>-attributes ] [ -X proxy[:port] ] [ -y <tr>-attributes ] [ -z <td>-attributes ] [http://]hostname[:port]/path

    Options

    -A auth-username:password
    Supply BASIC Authentication credentials to the server. The username and password are separated by a single : and sent on the wire base64 encoded. The string is sent regardless of whether the server needs it (i.e., has sent an 401 authentication needed).
    -c concurrency
    Number of multiple requests to perform at a time. Default is one request at a time.
    -C cookie-name=value
    Add a Cookie: line to the request. The argument is typically in the form of a name=value pair. This field is repeatable.
    -d
    Do not display the "percentage served within XX [ms] table". (legacy support).
    -e csv-file
    Write a Comma separated value (CSV) file which contains for each percentage (from 1% to 100%) the time (in milliseconds) it took to serve that percentage of the requests. This is usually more useful than the 'gnuplot' file; as the results are already 'binned'.
    -g gnuplot-file
    Write all measured values out as a 'gnuplot' or TSV (Tab separate values) file. This file can easily be imported into packages like Gnuplot, IDL, Mathematica, Igor or even Excel. The labels are on the first line of the file.
    -h
    Display usage information.
    -H custom-header
    Append extra headers to the request. The argument is typically in the form of a valid header line, containing a colon-separated field-value pair (i.e., "Accept-Encoding: zip/zop;8bit").
    -i
    Do HEAD requests instead of GET.
    -k
    Enable the HTTP KeepAlive feature, i.e., perform multiple requests within one HTTP session. Default is no KeepAlive.
    -n requests
    Number of requests to perform for the benchmarking session. The default is to just perform a single request which usually leads to non-representative benchmarking results.
    -p POST-file
    File containing data to POST.
    -P proxy-auth-username:password
    Supply BASIC Authentication credentials to a proxy en-route. The username and password are separated by a single : and sent on the wire base64 encoded. The string is sent regardless of whether the proxy needs it (i.e., has sent an 407 proxy authentication needed).
    -q
    When processing more than 150 requests, ab outputs a progress count on stderr every 10% or 100 requests or so. The -q flag will suppress these messages.
    -s
    When compiled in (ab -h will show you) use the SSL protected https rather than the http protocol. This feature is experimental and very rudimentary. You probably do not want to use it.
    -S
    Do not display the median and standard deviation values, nor display the warning/error messages when the average and median are more than one or two times the standard deviation apart. And default to the min/avg/max values. (legacy support).
    -t timelimit
    Maximum number of seconds to spend for benchmarking. This implies a -n 50000 internally. Use this to benchmark the server within a fixed total amount of time. Per default there is no timelimit.
    -T content-type
    Content-type header to use for POST data.
    -v verbosity
    Set verbosity level - 4 and above prints information on headers, 3 and above prints response codes (404, 200, etc.), 2 and above prints warnings and info.
    -V
    Display version number and exit.
    -w
    Print out results in HTML tables. Default table is two columns wide, with a white background.
    -x <table>-attributes
    String to use as attributes for <table>. Attributes are inserted <table here >.
    -X proxy[:port]
    Use a proxy server for the requests.
    -y <tr>-attributes
    String to use as attributes for <tr>.
    -z <td>-attributes
    String to use as attributes for <td>.


    Some real examples

    time /usr/sbin/ab2 -n 500 -c 30 http://www.waltercedric.com
    This will make 500 requests on them and hammering localhost for 30 seconds

    After tuning Before tuning
    Benchmarking www.waltercedric.comCompleted 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Finished 500 requests
    Server Software:&160;&160;&160;&160;&160;&160;&160; NOYB
    Server Hostname:&160;&160;&160;&160;&160;&160;&160; www.waltercedric.com
    Server Port:&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 80
    Document Path:&160;&160;&160;&160;&160;&160;&160;&160;&160; /index.php
    Document Length:&160;&160;&160;&160;&160;&160;&160; 45532 bytes
    Concurrency Level:&160;&160;&160;&160;&160; 30
    Time taken for tests:&160;&160; 38.576375 seconds
    Complete requests:&160;&160;&160;&160;&160; 500
    Failed requests:&160;&160;&160;&160;&160;&160;&160; 19&160;
    &160;&160; (Connect: 0, Length: 19, Exceptions: 0)
    Write errors:&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0
    Total transferred:&160;&160;&160;&160;&160; 23000106 bytes
    HTML transferred:&160;&160;&160;&160;&160;&160; 22762106 bytes
    Requests per second:&160;&160;&160; 12.96 [#/sec] (mean)
    Time per request:&160;&160;&160;&160;&160;&160; 2314.582 [ms] (mean)
    Time per request:&160;&160;&160;&160;&160;&160; 77.153 [ms] (mean, across all concurrent requests)
    Transfer rate:&160;&160;&160;&160;&160;&160;&160;&160;&160; 582.25 [Kbytes/sec] received
    Connection Times (ms)
    &160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; min&160; mean[+/-sd] median&160;&160; max
    Connect:&160;&160;&160;&160;&160;&160;&160; 0&160;&160;&160; 8&160; 36.9&160;&160;&160;&160;&160; 0&160;&160;&160;&160; 207
    Processing:&160;&160; 394 2239 345.3&160;&160; 2237&160;&160;&160; 6223
    Waiting:&160;&160;&160;&160;&160; 379 2197 340.9&160;&160; 2190&160;&160;&160; 6173
    Total:&160;&160;&160;&160;&160;&160;&160; 397 2247 344.2&160;&160; 2239&160;&160;&160; 6223
    Percentage of the requests served within a certain time (ms)
    &160; 50%&160;&160; 2239
    &160; 66%&160;&160; 2294
    &160; 75%&160;&160; 2327
    &160; 80%&160;&160; 2357
    &160; 90%&160;&160; 2457
    &160; 95%&160;&160; 2560
    &160; 98%&160;&160; 2973
    &160; 99%&160;&160; 3341
    100%&160;&160; 6223 (longest request)
    real&160;&160;&160; 0m38.617s
    user&160;&160;&160; 0m0.024s
    sys&160;&160;&160;&160; 0m0.240s

    Benchmarking www.waltercedric.com
    Completed 100 requests
    Completed 200 requests
    Completed 300 requests
    Completed 400 requests
    Finished 500 requests

    Server Software:&160;&160;&160;&160;&160;&160;&160; NOYB
    Server Hostname:&160;&160;&160;&160;&160;&160;&160; www.waltercedric.com
    Server Port:&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 80

    Document Path:&160;&160;&160;&160;&160;&160;&160;&160;&160; /index.php
    Document Length:&160;&160;&160;&160;&160;&160;&160; 45532bytes

    Concurrency Level:&160;&160;&160;&160;&160; 30
    Time taken for tests:&160;&160; 108.897481 seconds
    Complete requests:&160;&160;&160;&160;&160; 500
    Failed requests:&160;&160;&160;&160;&160;&160;&160; 19
    &160;&160; (Connect: 0, Length: 19, Exceptions: 0)
    Write errors:&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; 0
    Total transferred:&160;&160;&160;&160;&160; 23000106bytes
    HTML transferred:&160;&160;&160;&160; 23000106bytes
    Requests per second:&160;&160;&160; 4.59 [#/sec] (mean)
    Time per request:&160;&160;&160;&160;&160;&160; 6533.849 [ms] (mean)
    Time per request:&160;&160;&160;&160;&160;&160; 217.795 [ms] (mean, across all concurrent requests)
    Transfer rate:&160;&160;&160;&160;&160;&160;&160;&160;&160; 178.41 [Kbytes/sec] received

    Connection Times (ms)
    &160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160;&160; min&160; mean[+/-sd] median&160;&160; max
    Connect:&160;&160;&160;&160;&160;&160;&160; 0&160; 114 478.9&160;&160;&160;&160;&160; 0&160;&160;&160; 2276
    Processing:&160;&160; 336 6186 1665.2&160;&160; 6108&160;&160; 16189
    Waiting:&160;&160;&160; -5148 5982 1982.8&160;&160; 6066&160;&160; 16009
    Total:&160;&160;&160;&160;&160;&160;&160; 391 6301 1580.2&160;&160; 6120&160;&160; 17093

    Percentage of the requests served within a certain time (ms)
    &160; 50%&160;&160; 6120
    &160; 66%&160;&160; 6453
    &160; 75%&160;&160; 6778
    &160; 80%&160;&160; 7046
    &160; 90%&160;&160; 7861
    &160; 95%&160;&160; 8516
    &160; 98%&160; 10110
    &160; 99%&160; 12418
    100%&160; 17093 (longest request)

    real&160;&160;&160; 1m48.905s
    user&160;&160;&160; 0m0.024s
    sys&160;&160;&160;&160; 0m0.152s

    &160;

    time /usr/sbin/ab2 -kc 10 -t 30 http://www.waltercedric.com
    This will open 10 connections, using Keep-Alive on them and hammering localhost for 30 seconds


    Same tests but without mod_security

    • Mod_security is a module for Apache which act like a software firewall
    • Depending on the number of rules, can greatly affect through output speed


    time /usr/sbin/ab2 -kc 10 -t 30 http://www.waltercedric.com
    This will open 10 connections, using Keep-Alive on them and hammering localhost for 30 seconds

    real&160;&160;&160; 0m39.040s
    user&160;&160;&160; 0m0.020s
    sys&160;&160;&160;&160; 0m0.208s

    Nearly one second more with mod_security gotroot rules, worth the added security!

    If you want to know more options and how to use apache ab check the apache ab/ab2 man page click here for this man page

    How to optimize Apache/Joomla/PHP

    I forward You to some of my previous articles:

    And more ideas here Secure, Safe, Fast Linux Hosting

  • google_logospeedapache_logo

    mod_pagespeed is the latest addition to our family of products, performing on-the-fly optimization in the Apache™ 2 HTTP Server. It provides webmasters and web developers that manage their Apache 2 configuration with technology that applies Page Speed performance best practices automatically. Webmasters can get the benefits of the optimizations without having to manually going back and fixing the performance issues that Page Speed detects.

    This solution is valid if you do not want to carefully optimize your site, and it come for now at huge initial CPU costs till the cache is filled up. The load on my server has proven to be unacceptable (for me) and I was forced to switch it off. But it may work on your server depending on the page size, number of visitors. Authors are working hard and communicating a lot on the official mailing list to reduce that load and improve the code in the coming weeks

    If you trust me, you can download the library module libmod_pagespeed.so for Apache 2.2

    from http://drivers.waltercedric.com/

    Check before installing the MD5 keys of this file

    # md5sum -b mod_pagespeed.so
    187995e3623a222ec5b54c331ee7ffaa *mod_pagespeed.so

    If it matches, drop it into your Apache library folder /usr/lib64/apache2/

    Build Apache module mod_pagespeed yourself

    On my OpenSuSE system, I was forced to install first

    zypper in gcc-c++

    Then read  http://code.google.com/p/modpagespeed/wiki/HowToBuild

    you should get after a successful build

    • ./out/Release/libmod_pagespeed.so
    • ./out/Release/install/common/pagespeed.conf

    Install the module library

    cp ./out/Release/libmod_pagespeed.so  /usr/lib64/apache2/mod_pagespeed.so

    Configuration

    After installing mod_pagespeed, it can be customized and configured by editing the Apache configuration file

    /etc/apache2/conf.d/pagespeed.conf

    so you can run

    # cp ./out/Release/install/common/pagespeed.conf /etc/apache2/conf.d/pagespeed.conf

    Create some directories

    # mkdir /var/mod_pagespeed/
    # mkdir /var/mod_pagespeed/cache/
    # mkdir /var/mod_pagespeed/files

    Has to be writtable by apache

    chown -R wwwrun:www /var/mod_pagespeed/

    and finally restart apache

     apache2ctl restart

    There is a lot of available settings that are all well explained in this page http://code.google.com/speed/page-speed/docs/using_mod.html

    Here is my /etc/apache2/conf.d/pagespeed.conf stripped of all comments as an example:

    LoadModule pagespeed_module /usr/lib64/apache2/mod_pagespeed.so
    
        SetOutputFilter MOD_PAGESPEED_OUTPUT_FILTER
        ModPagespeed on
    
        ModPagespeedDomain www.waltercedric.com
        ModPagespeedUrlPrefix                "http://www.waltercedric.com/mod_pagespeed/"
        ModPagespeedFileCachePath "/var/mod_pagespeed/cache/"
        ModPagespeedGeneratedFilePrefix "/var/mod_pagespeed/files/"
        ModPagespeedRewriteLevel CoreFilters
        ModPagespeedEnableFilters collapse_whitespace
        ModPagespeedEnableFilters extend_cache
        ModPagespeedEnableFilters combine_css
        ModPagespeedEnableFilters rewrite_css
        ModPagespeedEnableFilters rewrite_javascript
        ModPagespeedEnableFilters rewrite_images
        ModPagespeedEnableFilters remove_comments

    # This page lets you view statistics about the mod_pagespeed module. Order allow,deny # You may insert other "Allow from" lines to add hosts you want to # allow to look at generated statistics. Another possibility is # to comment out the "Order" and "Allow" options from the config # file, to allow any client that can reach your server to examine # statistics. This might be appropriate in an experimental setup or # if the Apache server is protected by a reverse proxy that will # filter URLs in some fashion. Allow from localhost SetHandler mod_pagespeed_statistics

    Troubleshooting

    You may be forced to remove mod_deflate

    You can disable mod_pagespeed by adding in the url  ?mod_page_speed=0

    Some mod_pagespeed filters

    Links

  • Audio Signals Quality Diagnostics with Image Analysis. 
    by Vasily Tolkachev, ZHAW

    In this technical talk I present a number of interesting findings from a project with our industrial partner. The goal was to build a decent discriminative model which is able to distinguish between working and broken sound emitters based on the sound files produced by them. 
    We approached the problem with various image analysis tools by applying different classifiers on spectrograms of these files. A technique called t-SNE, which led to the key findings in the project, is going to be introduced. Having faced a number of data artefacts such as erroneous labels and class imbalance, sufficiently good performance was already achieved with Random Forest after a number of important transformations. In conclusion, a comparison to variational autoencoders will be exemplified.

     

    Powerpoint presentation

    Audio Based Bird Species Identification Using Deep Learning Techniques 
    by Elias Sprengel, ETHZ

    Accurate bird species identification is essential for biodiversity conservation and acts as an important tool in understanding the impact of cities and commercial areas on ecosystems. Therefore, many attempts have been made to identify bird species automatically. These attempts usually rely on audio recordings because images of birds are harder to obtain. They work well when the number of bird species is low and the recordings contain little background noise, but they quickly deteriorate when employed in any real world scenario.

    In this talk, we present a new audio classification approach based on recent advances in the domain of deep learning. With novel pre-processing and data augmentation methods, we train a neural network on the biggest publicly available dataset. This dataset contains crowd-sourced recordings of 999 bird species, providing an excellent way of evaluating our system in a more realistic scenario. Our convolutional neural network is able to surpass current state of the art results and won this year’s international BirdCLEF 2016 Recognition Challenge.

  • I have some major issues trying to complete this document: http://waltercedric.com/Mambo/Java/Security/Typical-issues-with-webapplications-v1.5-4.html
    Mambo do not let me add new text in the content of this article.  that is why I now propose it as a PDF download (this PDF contains much more example of we applications security issues than the html version). Enjoy.

    Download Typical issues with webapplications v1.5 .pdf

  • I’ve been&160; granted a free professional license of XDepend, thanks to Mat Huston, XDepend lead developer.

    XDepend is a static analysis tool for JAVA developers that provides 82 code metrics, several real-time code visualization panels, code base snapshots comparison, architectural and quality rules (edition and real-time validation). The tool is a frontend to support the Code Query Language (CQL) to query a code base the same way you would query a relational database. You can using CQL, write and design your own rules and conventions for your code base.

    XDepend

    • Analyses your java byte code, your test reports and your source files to extract structural information and 82 base metrics via static analysis.
    • Provides complementary and interactive views on the same information. A Tree-Map view helps you easily identify the big one from the small one. The dependency matrix, the graph view and the detailed view help you gain insight in your code base.
    • Has a Code Query Language (CQL) is a specific XDepend language, very similar to SQL, that helps you dynamically find what you are looking for.

    What is also interesting me a lot is the possibility to make XDepend part of the Maven lifecycle, but that will be part of a new post. For now I am trying to understand the added value on our company software solution Innoveo Skye®

    Watch the screen cast

    Spring analyzed by XDepend

    More to come later

    Note: .Net is also having a similar tools NDepend, build on the same engine developed by Patrick Smacchia.