debugging mysqld corefile on AIX

I recently had the pleasure of logging into an AIX 5.3 machine for the first time ever, to debug a corefile.

Firstly, having the mysqld binary and core is not enough, unless you have an identical machine on which to study the corefile. Library mismatches can be a problem.. IBM was kind enough to provide the snapcore utility to solve this easily.

Snapcore will gather all the libraries and create a single archive contain libs, binary, core.

So we now have a file called something like: snapcore_555060.pax

On our dev box, extract the pax archive:

gunzip snapcore_555060.pax.Z

pax -r -f snapcore_555060.pax

On your dev AIX box, make sure you have DBX installed!!

bash-3.00# lslpp -l | grep bos.adt.debug

bos.adt.debug 5.3.8.0 COMMITTED Base Application Development

Now, we are ready to debug a core. But we have to instruct DBX to read the libraries

that we got from the pax archive, instead of the default libraries on this system.

dbx -F -p /usr=/home/sbester/core/usr ./mysqld ./core

And since this isn't a DBX tutorial, I'll stop here, but you'll use normal DBX commands to print a stack trace, move up/down frames, print variables, list the source code.

a little challenge

How do you make mysqld write a DELETE to the binlog just by entering a SELECT statement ?
No triggers and no stored routines/functions are involved.

partitioning + auto_increment is buggy!

Folks, I just want to warn you how buggy partitioned tables with negative values are. Never use negative values for InnoDB auto_increment columns!!!!

• Assertion failed: next_insert_id >= auto_inc_interval_for_cur_row.minimum()
• partitioned innodb tables end up with duplicate primary key values!
• AUTO_INCREMENT errors with partitioned InnoDB tables in 5.1.31
• Assertion failure in file row/row0mysql.c line 1386

There are more bugs than I listed here..

5.1 doesn't solve all merge table hell from 5.0.

This week I've had to revisit merge tables once again due to customers experiencing problems. Although 5.1 merge table implementation is a huge improvement over 5.0, there still remains some critical bugs.

My list is still growing:

bug #45800: crash when replacing into a merge table and there is a duplicate
bug #45781: infinite hang/crash in "opening tables" after handler tries to open merge table
bug #45796: invalid memory reads and writes when altering merge and base tables
bug #45777: check table doesn't show all problems for merge table compliance in 5.1

Not to mention a few feature requests, and even documentation clarification for some manual sections.

some useful additions to query generator

I've been on vacation this week, and decided to fine-tune some old QA code. Opened the manual to see the syntax for a select statement, and afterwards added to my random select generator the following:

• all index hints (force, use, ignore, for join, for order by, for group by)
• lock in share mode, for update
• key_block_size for individual indexes
• hash, btree, rtree for individual indexes
• unique, fulltext, spatial for indexes

Especially important is the 'lock in share mode' addition. The reason is InnoDB
has many serious bugs with this locking mode (insert ... select, and others) in read committed mode.

So, I don't need multitable delete or update to reproduce those bugs, since I can just do a simple select locking in share mode. For example, the following bugs previously went without proper testcase until I discovered this:

assert btr/btr0pcur.c line 217 -innodb_locks_unsafe_for_binlog or read committed
5.1.35 crashes with Failing assertion: index->type & DICT_CLUSTERED
Strange error messages about locks from InnoDB