Panopticode Blog: Category Metrics

A Beautiful Visualization of Source Repository Activity

Julias Shaw — Mon, 16 Jun 2008 23:07:00 -0700

Michael Ogawa has created a mesmerizing animated visualization of the activity within source code repositories named code_swarm. This is truly a case of a picture being worth (at least) a thousand words.

Go check it out at http://vis.cs.ucdavis.edu/~ogawa/codeswarm/

New Metric: C.R.A.P.

Julias Shaw — Sun, 07 Oct 2007 16:17:00 -0700

The good folks at Agitar have proposed a new metric, C.R.A.P and an Eclipse plugin crap4j for executing it.

C.R.A.P. combines a method’s complexity with its coverage to come up with a single number that can help you determine if somebody will say, “Oh crap!” when they have to work with it. The formula is:

CRAP(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m)

Read their article for details on deciding if a system is crappy or not.

To test the flexibility of Panopticode I decided to implement a C.R.A.P. Supplement. It took a little over an hour.

Looking at the results I was pleased to find that only one method in Panopticode exceeds Agitar’s C.R.A.P. threshold of 30. Unfortunately that leads me to believe that 30 is too high. Right now, there is a lot of spike code in Panopticode right now that I consider crappy ;->

Teaser: Breaking the Build in Panopticode 0.2

Julias Shaw — Sun, 16 Sep 2007 01:24:00 -0700

In the last post we learned that Panopticode 0.2 will allow us to create arbitrary reports using a SPARQL SELECT query. Another feature in Panopticode 0.2 is to use a SPARQL ASK query to break the build.

An ASK query looks very similar to a SELECT but without any elements to return. If the query matches any data it returns true, otherwise it returns false.

Rewriting last post’s SELECT query as an ASK query would look like:

 PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
 PREFIX panopticode: <http://www.panopticode.org/ontologies/panopticode#>
 PREFIX java: <http://www.panopticode.org/ontologies/technology/java#>
 PREFIX emma: <http://www.panopticode.org/ontologies/supplement/emma/1#>
 PREFIX javancss: <http://www.panopticode.org/ontologies/supplement/javancss/1#>

 ASK WHERE
 {
   ?package         rdf:type                       java:Package           .
   ?package         panopticode:name               ?packageName           .
   ?package         java:hasFile                   ?file                  .
   ?file            panopticode:filePath           ?filePath              .
   ?file            java:hasType                   ?class                 .
   ?class           panopticode:name               ?className             .
   ?class           java:hasExecutableMember       ?method                .
   ?method          java:methodSignature           ?methodSignature       .
   ?method          emma:hasLineCoverage           ?lineCoverage          .
   ?method          javancss:cyclomaticComplexity  ?ccn                   .
   ?lineCoverage    emma:coveredPercent            ?lineCoveragePercent   .

   FILTER (?ccn > 1) .
   FILTER (?lineCoveragePercent <= 80.0)
 }

Panopticode 0.2 will come with an Ant task that automatically breaks the build when an ASK query returns true.

Teaser: Creating Ad Hoc Reports in Panopticode 0.2

Julias Shaw — Wed, 12 Sep 2007 08:12:00 -0700

In Metrics Must be Interpreted In Context I described one of my preferences when creating rules around metrics. Namely, that one should not look at metrics independently, but within the context of other metrics. I described a rule that said unit test line coverage must be greater then 80% for all code with a cyclomatic complexity over 1. While you could enforce this rule in Panopticode 0.1 by creating a custom report, this is not ideal. Who wants to write Java code every time you make a new rule? This will get much easier in Panopticode 0.2.

Panopticode 0.2 has been re-architected to use RDF as it’s file format and internal data store. This enables you to write queries using SPARQL. Here is a query to find violators of the rule mentioned above:

 PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
 PREFIX panopticode: <http://www.panopticode.org/ontologies/panopticode#>
 PREFIX java: <http://www.panopticode.org/ontologies/technology/java#>
 PREFIX emma: <http://www.panopticode.org/ontologies/supplement/emma/1#>
 PREFIX javancss: <http://www.panopticode.org/ontologies/supplement/javancss/1#>

 SELECT ?packageName ?filePath ?className ?methodSignature ?ccn ?lineCoveragePercent
 WHERE
 {
   ?package         rdf:type                       java:Package           .
   ?package         panopticode:name               ?packageName           .
   ?package         java:hasFile                   ?file                  .
   ?file            panopticode:filePath           ?filePath              .
   ?file            java:hasType                   ?class                 .
   ?class           panopticode:name               ?className             .
   ?class           java:hasExecutableMember       ?method                .
   ?method          java:methodSignature           ?methodSignature       .
   ?method          emma:hasLineCoverage           ?lineCoverage          .
   ?method          javancss:cyclomaticComplexity  ?ccn                   .
   ?lineCoverage    emma:coveredPercent            ?lineCoveragePercent   .

   FILTER (?ccn > 1) .
   FILTER (?lineCoveragePercent <= 80.0)
 }
 ORDER BY DESC(?ccn) ?lineCoveragePercent ?packageName ?filePath ?className ?methodSignature

New Tool: Complexian

Julias Shaw — Sun, 04 Mar 2007 06:55:00 -0800

Marty Andrews has just released version 0.12.0 of Complexian. This version adds the ability to output to plain or XML files.

Complexian is a tool for very quickly measuring NPATH and CCN of Java projects. It will be free for use until the first major release.

Support for executing Complexian is already in Panopticode’s Subversion repository and full support will be in release 0.2.

Metrics Must be Interpreted In Context

Julias Shaw — Mon, 26 Feb 2007 20:38:00 -0800

James Newkirk points out that the absolute measurement of code coverage is not a very interesting number by itself, but must be viewed as part of the overall trend of coverage. He also points out that it is acceptable for some categories of code to have lower, or even zero, test coverage.

These are examples of one of my fundamental rules of metrics.

Rule #1: Metrics Must Be Interpreted In Context

A few examples of context are:

Within a time series. A code coverage of 20% could be very good if the team is just beginning to add automated tests to a legacy system. On the other hand a code coverage of 85% could be poor if was at 90% coverage at the last release.
By category of code. James uses the example of web service wrappers generated by Visual Studio and and views within a Model-View-Presenter pattern. I agree with James that generated code does not need unit tests. However, you should be particlarly thorough in testing any code generators you write.
Correlations with other metrics. If pressed to come up with an arbitrary threshold for a metric I like to do it within the context of another metric. For example, I might say that the unit test line coverage must be greater than 80% for all methods with a cyclomatic complexity greater than one. A key metric to correlate with is defects. Code that has proven to be buggy in production should get closer attention.
Compared to other projects in your organization. If every other software project at your company has 90% code coverage you had better have a good reason for only having 70%.

There is a lot more to say on this subject. Stay Tuned.