Archive | Statistics RSS for this section

Rough #’s

These are very very rough #’s – I already know of a couple issues with the count:
a) a single bug can be counted for multiple people with this method, Joren and I will be manually checking these
b) this only includes bugs that were UNCONFIRMED on the first second of the contest and new bugs from that point on – this means bugs that went from some other status to UNCONFIRMED in between are omitted, Joren and I will also work on this

So – don’t panic if you don’t see the #’s you were hoping to see 😉 VERY ROUGH COUNT

Name Points
Joel 137
Joren 103
Thomas Hackert 75
SteveBell 43
Thomas(pje) 37
Cor Nouws 35
Ken 14
Florian Reisinger 11
JBF 6
Ralph Aichinger 4
Tim Passingham 4
Brenda Granados 2
Jan Koberstein 2
Ahitagni Mandal 1
Robinson Tryon 1

Overall very nice work 🙂 Some other good signs:
-249 UNCONFIRMED bugs in the time frame
337 bugs were reported during this time frame
This means that 586 bugs have been triaged
New people are still joining 🙂

We have a few days left, let’s push through the end to get to the 25% goal – about 60 more bugs gets us there.

Thanks all for your tremendous dedication 😀

UNCONFIRMED Bugs

Well I’ve procrastinated doing stats for March (and now I’ll be behind in April as well) – but I opted to do a quick post about our unconfirmed bugs right now. Quick overview of Jan 15th – April 27th. I pulled bugs *almost* every day. In my opinion our biggest duty as QA is to keep the unconfirmed bug count down and try to tackle triaging bugs as fast as possible. While we continue to press towards this goal the increasing number of users – and more specifically users who are willing to report bugs – continues to challenge our small QA team but, these stats highlight just how great our team is and continues to be. Tremendous thanks to each one of you who even take a few minutes out of your busy days to triage a bug or two.

Without further delay:

Between January 15th & April 27th 3,002 bugs were reported – I’m serious, no lie, over 3,000 bugs reported! Thanks (partly sarcastic?) to our fantastic users for reporting problems when you find them. This period covers 103 days (so about 29 bugs per day reported). Just maintaining our unconfirmed count would have been a tremendous success…..but, WE DID BETTER!

Week Last Count of Week
3 1414
4 1449
5 1395
6 1457
7 1487
8 1507
9 1475
10 1475
11 1554
12 1508
13 1482
14 1503
15 1481
16 1357
17 1313

The week # is determined by Spreadsheet’s “WEEKNUM” function, the last pull of the week is the # of bugs on the right.  As can easily be seen we DECREASED the overall UNCONFIRMED bug count by 101 bugs in 103 days – so putting the numbers together (number reported – (net change)) we get a whopping 3,103 bugs triaged during this period!

Truly incredible, again, thank you all for your hard work 🙂

February Bug Stats

Well it’s a few weeks late (apologies) but, isn’t there a saying about “better late than never” 😀

Below is a list of daily (almost at least) change of total number of bugs reported against LibreOffice on FreeDesktop.Org (FDO). During the month 1,043 bugs were reported, averaging out at just over 37 bugs per day. While this could be looked on as a negative, I tend to be excited that we have enough user base and involvement that we are getting this kind of activity on FDO. On our most active days we were hitting over 80 bugs reported in a single day which is incredible (please keep in mind that some portion of these are duplicates). This being said, the other important note is that our QA team is way under staffed to handle this number of bugs, assuming there are 10 of us triaging somewhat frequently we’d need to triage over 100 bugs a month along with our other duties — including “real” (paid) work, school, and of course our personal lives. This is just further proof that we need to reach out to the community and get more people involved with QA work – ideally IMO we would be at no more than 30 bugs per month per person on average which means we need to triple our current number of regular triagers (again assuming we currently have 10 “regularly involved”)

Total Bug Count on FDO

Date

Bug Count

Change

01/31/13

16011

02/01/13

16032

+ 21

02/02/13

16055

+ 23

02/03/13

16094

+ 39

02/05/13

16146

+ 52

02/06/13

16174

+ 28

02/07/13

16205

+ 31

02/08/13

16266

+ 61

02/10/13

16354

+ 88

02/11/13

16390

+ 36

02/12/13

16439

+ 49

02/13/13

16491

+ 52

02/14/13

16524

+ 33

02/16/13

16604

+ 80

02/17/13

16630

+ 26

02/18/13

16678

+ 48

02/19/13

16711

+ 33

02/20/13

16742

+ 31

02/21/13

16790

+ 48

02/22/13

16830

+ 40

02/23/13

16878

+ 48

02/24/13

16920

+ 42

02/25/13

16938

+ 18

02/26/13

16974

+ 36

02/27/13

17007

+ 33

02/28/13

17054

+ 47

Total

+ 1043

Then we have the number of bugs that were in status NEW & RESOLVED.

A majority of the NEW bugs came from either UNCONFIRMED to NEW or from NEEDINFO to NEW. Both of these are telling as they mean the bug was confirmed with the given information and therefore can move forward in the process of trying to get the issue fixed. The majority of this work can be credited to our awesome QA team who do a tremendous job ensuring that the information that is needed is on the bug report as well as manually testing each bug one at a time.

With the RESOLVED bugs, the majority of the credit goes to our amazing Development team who move bugs from NEW & REOPENED status to RESOLVED. This being said some of these bugs are closed as RESOLVED for other reasons including WORKSFORME (QA mostly) and DUPLICATE (QA & Dev team).

New & Resolved Counts
Date New New – Change Resolved Resolved – Change
01/31/13 4261 7884  
02/01/13 4277 + 16 7904 + 20
02/02/13 4294 + 17 7920 + 16
02/03/13 4302 + 8 7928 + 8
02/05/13 4297 – 5 7956 + 28
02/06/13 4294 – 3 7975 + 19
02/07/13 4306 + 12 7999 + 24
02/08/13 4303 – 3 8040 + 41
02/10/13 4349 + 46 8068 + 28
02/11/13 4364 + 15 8091 + 23
02/12/13 4371 + 7 8114 + 23
02/13/13 4385 + 14 8135 + 21
02/14/13 4391 + 6 8151 + 16
02/16/13 4422 + 31 8194 + 43
02/17/13 4429 + 7 8211 + 17
02/18/13 4413 – 16 8242 + 31
02/19/13 4432 + 19 8259 + 17
02/20/13 4444 + 12 8296 + 37
02/21/13 4464 + 20 8315 + 19
02/22/13 4475 + 11 8332 + 17
02/23/13 4498 + 23 8354 + 22
02/24/13 4523 + 25 8376 + 22
02/25/13 4524 + 1 8386 + 10
02/26/13 4534 + 10 8427 + 41
02/27/13 4541 + 7 8451 + 24
02/28/13 4538 – 3 8472 + 21
Total   + 277   + 588

During the course of the month there was an increase of 277 bugs in NEW status on a day over day basis while there was an increase of nearly 600 on a day over day basis to RESOLVED.

NOTE: Just to be accurate, this does not accurately portray exactly how many bugs were put into NEW or RESOLVED in a given day because a bug could have gone from NEW to RESOLVED in a single day, counting for 1 on “RESOLVED” column but not 1 on “NEW” status – remember this is only a snapshot of FDO.

As stated above, DUPLICATES are a part of the “RESOLVED” category and offer a lot of insite for individual bugs – most notably it’s a way to gauge how many people are affected by a single bug. In the long run we hope that more bug reporters are able to leave comments on existing bugs instead, but for the time being, a DUPLICATE bug reflects well on QA/Devs (for identifying the dupe), our users (for reporting the bug) and on our product (for offering insite on importance of a bug).

Duplicates
Date All Versions – Total Count All Versions – Change Version 4.0+ – Total Count Version 4.0+ – Change
01/31/13 1916 85
02/01/13 1923 + 7 87 + 2
02/02/13 1925 + 2 88 + 1
02/03/13 1925 0 88 0
02/05/13 1931 + 6 90 + 2
02/06/13 1935 + 4 91 + 1
02/07/13 1941 + 6 94 + 3
02/08/13 1951 + 10 100 + 6
02/10/13 1963 + 12 107 + 7
02/11/13 1967 + 4 107 0
02/12/13 1974 + 7 111 + 4
02/13/13 1982 + 8 116 + 5
02/14/13 1987 + 5 117 + 1
02/16/13 2003 + 16 124 + 7
02/17/13 2010 + 7 129 + 5
02/18/13 2014 + 4 129 0
02/19/13 2018 + 4 132 + 3
02/20/13 2037 + 19 148 + 16
02/21/13 2044 + 7 152 + 4
02/22/13 2050 + 6 158 + 6
02/23/13 2056 + 6 163 + 5
02/24/13 2065 + 9 172 + 9
02/25/13 2066 + 1 173 + 1
02/26/13 2072 + 6 178 + 5
02/27/13 2074 + 2 178 0
02/28/13 2078 + 4 182 + 4
Total   + 162   + 97

The release of Version 4.0 was an important milestone, because of this I’ve divided the columns between “all versions” and just 4.0+ bugs. As you can see over 160 open bug reports were closed as DUPLICATE during the month, more than ½ of which were against Version 4.0.

Now on to some regressions. With the release of 4.0.0.x release we had to account for how many regressions were reported against releases prior to 4 and those reported against 4. The below table has this break down as well as showing how many regressions were in status NEW – this means that the bug was a) confirmed and b) not fixed yet.

Regression Stats
Date All Versions – Total Count All Versions –Status NEW Version 4.0+ – Total Count Version 4.0+ Status NEW
01/31/13 1199 174 244 46
02/01/13 1203 176 250 48
02/02/13 1204 178 252 50
02/03/13 1211 182 257 54
02/05/13 1219 183 260 54
02/06/13 1222 182 259 50
02/07/13 1224 184 263 51
02/08/13 1230 187 267 52
02/10/13 1250 193 286 59
02/11/13 1254 194 289 60
02/12/13 1259 192 292 58
02/13/13 1262 193 293 59
02/14/13 1267 193 297 59
02/16/13 1282 194 308 58
02/17/13 1289 197 314 60
02/18/13 1292 197 317 60
02/19/13 1301 200 323 63
02/20/13 1307 201 329 65
02/21/13 1311 199 331 65
02/22/13 1314 199 334 66
02/23/13 1319 200 339 68
02/24/13 1324 200 342 68
02/25/13 1327 200 344 68
02/26/13 1333 199 348 69
02/27/13 1337 200 352 71
02/28/13 1340 195 355 68

While the number of regressions grew quite rapidly ruing the period (climbing by 141 bugs in the month) the number of NEW bugs remained relatively stable (increasing by only 22), most of this is credit to our development team who made sure to tackle regressions efficiently & effectively.

FIXED bugs are obviously one of our “shining lights” of the project, they indicate a complete and total workflow for bugs, from reporting, to triaging, to fixing – this of course includes dev team, users and QA team and often times other teams get involved in the process to help ensure all information is there, offer opinions/alternative solutions, etc….

Marked as FIXED
Date All Versions – Total Count (FIXED) All Versions –Status FIXED Version 4.0+ – Total Count (FIXED) Version 4.0+ – FIXED % Version 4 Fixed/Total Fixed (CHANGE)
01/31/13 3313   262    
02/01/13 3319 6 266 4 40.00%
02/02/13 3328 9 270 4 30.77%
02/03/13 3334 6 271 1 14.29%
02/05/13 3352 18 276 5 21.74%
02/06/13 3360 8 280 4 33.33%
02/07/13 3367 7 282 2 22.22%
02/08/13 3388 21 291 9 30.00%
02/10/13 3395 7 293 2 22.22%
02/11/13 3404 9 297 4 30.77%
02/12/13 3417 13 305 8 38.10%
02/13/13 3421 4 304 -1 -33.33%
02/14/13 3427 6 308 4 40.00%
02/16/13 3444 17 315 7 29.17%
02/17/13 3446 2 316 1 33.33%
02/18/13 3451 5 320 4 44.44%
02/19/13 3454 3 323 3 50.00%
02/20/13 3462 8 325 2 20.00%
02/21/13 3469 7 328 3 30.00%
02/22/13 3474 5 331 3 37.50%
02/23/13 3479 5 334 3 37.50%
02/24/13 3485 6 334 0 0.00%
02/25/13 3490 5 335 1 16.67%
02/26/13 3503 13 341 6 31.58%
02/27/13 3514 11 347 6 35.29%
02/28/13 3525 11 355 8 42.11%

Again in order to highlight the importance of 4.0 we’ve split the columns to “all versions” and just version 4. As you can see the number of fixed bugs per day were impressive, more than 10 bugs were fixed on a single day. Also of importance is the fact that ~45% of fixes were against 4.0, showing that we are both dedicated to our new version as well as maintaining our older releases.

Last fun stat for this blog, # of comments. I thought it’d be interesting to look at number of comments as an indicator of how active FDO is.

Total # Of Comments
Date All Versions – Total Count All Versions –Status Version 4.0+ – Total Count Version 4.0+ % Version 4 Fixed/Total Fixed (CHANGE)
01/31/13 102,158   5,781    
02/01/13 102,342 184 5,851 70 0
02/02/13 102,511 169 5,924 73 0
02/03/13 102,658 147 5,985 61 0
02/05/13 102,987 329 6,131 146 0
02/06/13 103,164 177 6,182 51 0
02/07/13 103,376 212 6,269 87 0
02/08/13 103,721 345 6,462 193 0
02/10/13 104,256 535 6,787 325 0
02/11/13 104,450 194 6,881 94 0
02/12/13 104,750 300 7,033 152 0
02/13/13 105,033 283 7,179 146 0
02/14/13 105,256 223 7,291 112 0
02/16/13 105,704 448 7,506 215 0
02/17/13 105,915 211 7,633 127 0
02/18/13 106,154 239 7,749 116 0
02/19/13 106,375 221 7,884 135 0
02/20/13 106,658 283 8,042 158 0
02/21/13 106,957 299 8,222 180 0
02/22/13 107,183 226 8,368 146 0
02/23/13 107,412 229 8,518 150 0
02/24/13 107,681 269 8,670 152 0
02/25/13 107,799 118 8,741 71 0
02/26/13 108,145 346 8,927 186 0
02/27/13 108,372 227 9,058 131 0
02/28/13 108,598 226 9,202 144 0

As you can clearly see, we’re quite popular 🙂 6,440 comments were posted during the month (yes that is OVER 6,000 comments. Incredible. Of these 3,421 were reported against version 4 (just over 50%).

NOTE: Comments include the auto comments for fixed bugs, impossible (or nearly impossible) for me to figure out what % of the total number of comments came from bots but, 6,440 comments is none the less very impressive IMO.

With that I’ll end this months (or last months, or the month before that) stats blog. I know that I didn’t do the same stats as the previous month, trying to keep it “fun and new”.

Feel free to ask for specific stats or suggest anything that comes to mind.

Best to all 😀

Joel Madero

P.S. Apologies for now pretty charts, wordpress is being slightly difficult today, might add them later if requested 🙂

FreeDesktop.Org at a Glance — Resolve & Fixed Bugs

Well as I tend to lean towards lots of stats, I am at it again. I’ve been pulling a snapshot of all FDO bugs daily (missed one day) for the past couple weeks, I hope to use these snapshots to get a core set of stats for QA purposes, here is my first “snapshot of a snapshot” 😉

One of the most telling statistics for LibreOffice is the number of bugs which are marked RESOLVED (encompassing a few sub categories) and more importantly RESOLVED – FIXED which is a designation which says “yes indeed there was a bug, but a patch has been committed to fix it”. Below is a table which shows data pertaining to these two things (RESOLVED & RESOLVED – FIXED) since 1/15/2013.

DATE Total # of Bugs Marked “RESOLVED” Resolved Bugs Per Day Sum of Resolved Bugs Per Day Total # of Bugs Marked “FIXED” Fixed Bugs Per Day Sum of Fixed Bugs Per Day
January 15th, 2013 7657 3226
January 16th, 2013 7676 19 19 3233 7 7
January 17th, 2013 7697 21 40 3242 9 16
January 18th, 2013 7706 9 49 3247 5 21
January 19th, 2013 7724 18 67 3249 2 23
January 20th, 2013 7732 8 75 3250 1 24
January 21st, 2013 75 24
January 22nd, 2013 7756 24 99 3258 8 32
January 23rd, 2013 7767 11 110 3261 3 35
January 24th, 2013 7788 21 131 3272 11 46
January 25th, 2013 7806 18 149 3278 6 52
January 26th, 2013 7815 9 158 3283 5 57
January 27th, 2013 7824 9 167 3285 2 59
January 28th, 2013 7842 18 185 3293 8 67

The information above is a bit messy but all in all here are some high points:

  • In the past 2 weeks or so we’ve marked 185 bugs as RESOLVED
  • Of those 185, 67 were marked as FIXED — again, this means that there was a confirmed bug that was fixed
  • Every day there was at least 1 bug fixed
  • The best day was January 24th, where 11 bugs were fixed in approximately a 24 hour period

While numbers alone can’t prove the tremendous amount of work done by developers on a given day — as example consider a day where only 1 bug is fixed but that single bug is hundreds of lines of code — the numbers are promising that we are daily achieving incredible progress towards our goal to make LibreOffice the best, most comprehensive, office suite available.

Just to add a nice looking chart to this data, here is a visual of bugs marked RESOLVED & bugs marked FIXED per day.

I’ll end it here with saying a big THANK YOU to all of the developers, QA, Marketing, UI, locale, documentation, etc.. etc… that make this project possible – without each of you, we would be left with nothing.

Thanks all, until next time

Best Regards,
Joel

LibreOffice Test Marathon Results

As the year comes to an end there are plenty of accomplishments that the LibreOffice community can be proud of, and a week ago we added another success — the end of our 6 day testing marathon[1] against the upcoming release of LibreOffice Version 4.0 (scheduled for February of 2013). While the Quality Assurance (QA) team didn’t set any goals for the week other than to “get as many people as possible involved with testing LibreOffice Version 4.0 Beta 1”, the statistics speak a great deal about how great our growing community is and far exceed the results that I personally was expecting. Any time “Version 4” is referenced it includes the master build, Beta 1 build as well as the Alpha build.

Between December 14th and December 19th 445 bugs on FreeDesktop.Org (FDO) were modified in some way. These modifications included confirming bugs, adding details in comments, fixing bugs, confirming fixes, and bibisecting regressions.

On December 14th at the beginning of our test marathon 353 bugs were reported against Version 4, by the end of the marathon on December 19th, we were at 461 bugs (up 108 bugs in 6 days). Despite the fast increase of bug reports during the week, our number of UNCONFIRMED bugs only grew by 20 (from 57 to 77) — this showing that not only were users testing the suite extensively, users and QA staff were dedicated to triaging bugs as quickly as possible. The percent of fixed bugs (status of FIXED & VERIFIED) went from 45.61% to 48.16% of the total bugs reported against Version 4 (again this incorporates the 108 bug increase during the period).

The chart below gives a good idea of bug status’ during the period of the marathon:

BugStatusChart

The biggest spike was clearly in RESOLVED bugs but NEW (confirmed bugs) rose by nearly 20% and as already mentioned, UNCONFIRMED bugs rose nearly 35%.

The breakdown of bugs per component offers a good insight of what components are most popular and there weren’t many surprises. WRITER consistently was the most reported against component of LibreOffice – beginning at 102 of 353 bugs and ending at 127 of 461 bugs. Spreadsheet was second most popular but saw the largest spike in bugs reported against it during the week (rising from 56 bugs to 84 bugs, almost 50% increase). The last three components were in the following order (Presentation, Database, Drawing), non component bugs hovered around 40% of total bugs throughout the week (starting at 148 bugs, ending at 186 bugs) — unfortunately the general “LibreOffice” bug falls into this category and many of these bugs are about a specific component.

The charts below shows bugs reported against each component during the period of the marathon:

BugComponentChart

BugComponentDateChart

The final “stats” that I want to point out before finishing is on regressions which will get tremendous focus by developers as we near release date. On December 14th 91 bugs (34.7%) of bugs reported against Version 4 were marked as regression, on December 19th this number rose to 135 bugs (41.4%). While this number could be interpreted as bad for our product, in reality, this is tremendous news and a great success for the marathon. It is PRECISELY these bugs that we hope to discover and fix before stable release, in order to do this we must first report the bugs which is what we see here.

RegressionChart

As a last important note, during the week we saw a tremendous increase of activity on our quality assurance IRC channel (#libreoffice-qa) on freenode. Every day developers, quality assurance contributors and users from around the world could be seen in the chat room, discussing strategies, future events, etc…Ultimately it was this aspect that I would say was the greatest success during the week. There were particular users (you know who you are 😉 ) that I am sure will continue to contribute substantially going into the future and whom I give a tremendous amount of thanks to. It is these contributors who continue to make LibreOffice a fantastic product for everyone, and I urge users and contributors alike to give thanks to one another for our combined dedication.

So as a recap — Between December 14th and December 19th:

  • 445 bugs modified
  • # of NEW (confirmed) bugs rose nearly 20%
  • # of UNCONFIRMED bugs rose by 20 (57 to 77)
  • Resolved & Verified bugs account for nearly 50% of bugs (up approximately 3%)
  • Writer continues to be the most reported against component (consistently hovering above 25% of total bug reports)
  • Spreadsheet saw the biggest spike in bugs (about 50% more bugs at end of marathon vs. beginning)
  • Regressions rose from 95 bugs to 135 – rising from 34.7% of total to 41.4% [3]

All of the statistics have been posted online for everyone to see [2].

[1] http://wiki.documentfoundation.org/QA/Test_Marathon_LibreOffice_4.0

[2] https://wiki.documentfoundation.org/File:QA_Test_Marathon_Stats.ods

[3] These are not confirmed regressions, just bugs with regression in keyword field

Thanks for reading my first post 🙂