LibreOffice Test Marathon Results

As the year comes to an end there are plenty of accomplishments that the LibreOffice community can be proud of, and a week ago we added another success — the end of our 6 day testing marathon[1] against the upcoming release of LibreOffice Version 4.0 (scheduled for February of 2013). While the Quality Assurance (QA) team didn’t set any goals for the week other than to “get as many people as possible involved with testing LibreOffice Version 4.0 Beta 1”, the statistics speak a great deal about how great our growing community is and far exceed the results that I personally was expecting. Any time “Version 4” is referenced it includes the master build, Beta 1 build as well as the Alpha build.

Between December 14th and December 19th 445 bugs on FreeDesktop.Org (FDO) were modified in some way. These modifications included confirming bugs, adding details in comments, fixing bugs, confirming fixes, and bibisecting regressions.

On December 14th at the beginning of our test marathon 353 bugs were reported against Version 4, by the end of the marathon on December 19th, we were at 461 bugs (up 108 bugs in 6 days). Despite the fast increase of bug reports during the week, our number of UNCONFIRMED bugs only grew by 20 (from 57 to 77) — this showing that not only were users testing the suite extensively, users and QA staff were dedicated to triaging bugs as quickly as possible. The percent of fixed bugs (status of FIXED & VERIFIED) went from 45.61% to 48.16% of the total bugs reported against Version 4 (again this incorporates the 108 bug increase during the period).

The chart below gives a good idea of bug status’ during the period of the marathon:

The biggest spike was clearly in RESOLVED bugs but NEW (confirmed bugs) rose by nearly 20% and as already mentioned, UNCONFIRMED bugs rose nearly 35%.

The breakdown of bugs per component offers a good insight of what components are most popular and there weren’t many surprises. WRITER consistently was the most reported against component of LibreOffice – beginning at 102 of 353 bugs and ending at 127 of 461 bugs. Spreadsheet was second most popular but saw the largest spike in bugs reported against it during the week (rising from 56 bugs to 84 bugs, almost 50% increase). The last three components were in the following order (Presentation, Database, Drawing), non component bugs hovered around 40% of total bugs throughout the week (starting at 148 bugs, ending at 186 bugs) — unfortunately the general “LibreOffice” bug falls into this category and many of these bugs are about a specific component.

The charts below shows bugs reported against each component during the period of the marathon:

The final “stats” that I want to point out before finishing is on regressions which will get tremendous focus by developers as we near release date. On December 14th 91 bugs (34.7%) of bugs reported against Version 4 were marked as regression, on December 19th this number rose to 135 bugs (41.4%). While this number could be interpreted as bad for our product, in reality, this is tremendous news and a great success for the marathon. It is PRECISELY these bugs that we hope to discover and fix before stable release, in order to do this we must first report the bugs which is what we see here.

As a last important note, during the week we saw a tremendous increase of activity on our quality assurance IRC channel (#libreoffice-qa) on freenode. Every day developers, quality assurance contributors and users from around the world could be seen in the chat room, discussing strategies, future events, etc…Ultimately it was this aspect that I would say was the greatest success during the week. There were particular users (you know who you are 😉 ) that I am sure will continue to contribute substantially going into the future and whom I give a tremendous amount of thanks to. It is these contributors who continue to make LibreOffice a fantastic product for everyone, and I urge users and contributors alike to give thanks to one another for our combined dedication.

So as a recap — Between December 14th and December 19th:

• 445 bugs modified
• # of NEW (confirmed) bugs rose nearly 20%
• # of UNCONFIRMED bugs rose by 20 (57 to 77)
• Resolved & Verified bugs account for nearly 50% of bugs (up approximately 3%)
• Writer continues to be the most reported against component (consistently hovering above 25% of total bug reports)
• Spreadsheet saw the biggest spike in bugs (about 50% more bugs at end of marathon vs. beginning)
• Regressions rose from 95 bugs to 135 – rising from 34.7% of total to 41.4% [3]

All of the statistics have been posted online for everyone to see [2].

[1] http://wiki.documentfoundation.org/QA/Test_Marathon_LibreOffice_4.0

[2] https://wiki.documentfoundation.org/File:QA_Test_Marathon_Stats.ods

[3] These are not confirmed regressions, just bugs with regression in keyword field

Thanks for reading my first post 🙂