putting it all together

Last post was about Rainbow testing. So that's old news, and I lost count of the number of bugs it found, and have been fixed. Probably 30+ crashes.

Now, I have to put together all these odds & ends of code I wrote disparately over the last 18 months into a single collection to enable complete end-to-end testing.

I have now got roughly the following:

o) random table maker
o) query maker based on any tables, (using predefined rules)
o) data generator for any tables
o) complete charset collection for each valid character
o) database of mysql functions, data types
o) multi-threaded testing environment for any queries
o) query results comparer for any queries
o) rainbow, which will provide help in making good coverage of functions, etc.

What's needed still? Well, I would like 56 hours in a day ... More posts later when v0.001 of the integrated is semi-functional :)

mysql crash log analyzer ?

So with over 1000 crashes in my logs from just one night of simple queries, I have a painful time to find new crashes in the 90MB file. So I'm taking a detour on the rainbow query generator for 2 days. Will be writing an error log analyzer instead :)

step 1: upload the mysqld and the mysqld.sym file for the version you're working with.
step 2: import the binary and symbols into a mysql table
step 3: upload the error log
step 4: parse the error log into seperate crashes
step 5: find the stack traces for each crash, and resolve them using the symbols
step 6: determine if this crash is matching any existing crashes.

With thousands of crashes on various builds of mysqld, the above system can be useful to me. Also, I'd import all the crashes from reports on bugs.mysql.com for searching purposes. This toy will allow me to identify whether a crash is new, or it's been seen before.

some results of rainbow

ok folks. here's some results:

mysql> select last_errno,count(*) from 
queryqueue group by last_errno;
+------------+----------+
| last_errno | count(*) |
+------------+----------+
|          0 |  1600796 |
|       1048 |     1971 |
|       1053 |        1 |
|       1139 |       35 |
|       1267 |    19722 |
|       1270 |     4243 |
|       1271 |     8944 |
|       1416 |     2284 |
|       1580 |    23225 |
|       2003 |       28 |
|       2013 |     1606 |
+------------+----------+
11 rows in set (0.00 sec)

error 2013 means lost connection to server (read: server crashed).
so there are many bugs found already. 1606 crashes out of 1.6 million
executed queries, is great.

check my rss feed for the exact bugs ...

good news!

my girlfriend's grandfather wants a blog!

sql rainbow tables

I thought of a brilliant QA plan. Create sql rainbow tables! Just trust me, this system of QA will be very powerful in bug finding. Especially important will be the comparison of old/new versions of mysql. Writing a prototype now, on my day off.

Just during prototype development, I've discovered a handful of bugs. I'm testing all functions that are documented in the manual. This is alot that I've written up:

mysql> select count(*),category from func group by category;
+----------+--------------+
| count(*) | category     |
+----------+--------------+
|        6 | arithmetic   |
|        7 | bit          |
|       60 | casting      |
|       47 | comparison   |
|       52 | datetime     |
|       18 | encryption   |
|      127 | geometry     |
|       17 | information  |
|        7 | logical      |
|       32 | mathematical |
|        9 | misc         |
|       47 | string       |
|        2 | xml          |
+----------+--------------+
13 rows in set (0.00 sec)

Let me explain this rainbow tables concept with a simple example.
Suppose we have a function, like "GREATEST()".

I run a query like this:

SELECT GREATEST(col1,col2,col3) FROM table0;

The columns col1,col2,col3 will form any combination of column type. So we have int, date, string, blob, spatial! Given a combination of all datatypes and all function parameters, it leaves us with a good few million queries. Of course, a few million queries is not so much for a machine to execute quickly..