Oops, we did it again (MySQL 5.1 released as GA with crashing bugs)

MySQL 5.1 is now released as "GA".

In this blog I will try to describe my opinions about this release and also try to set the expectations right for anyone trying out MySQL 5.1 GA.

What should you then expect from MySQL 5.1?

• If you are using MySQL 5.1 just as a 'better' version of MySQL 5.0 and you don't plan to use any of the new features in MySQL 5.1 then you are probably fine to try out MySQL 5.1. You should however not put it into production without testing it fully, preferably by running it on a couple of slaves for some weeks. It may even be the best to wait for a couple of minor/patch releases before putting the MySQL 5.1 server into production.
• Don't expect that all critical bugs that you may have encountered in 5.0 to be fixed in 5.1. Even if we have fixed a big majority of the bugs from 5.0 some really critical ones still haven't been addressed.
• If you plan to use any of the new features of MySQL 5.1, regard these as if they would be of beta quality. Test any usage of these features extensively for in close-to-live scenarios before putting them onto a production server.
• If you are a new user trying out MySQL for the first time, you should use MySQL 5.1; At least it's better than the MySQL 5.0 community version which has not been updated for some time.
  

The reason I am asking you to be very cautious about MySQL 5.1 is that there are still many known and unknown fatal bugs in the new features that are still not addressed.

To prove my points, here is some metrics and critical bugs for 5.1:

• We still have 20 known and tagged crashing and wrong result bugs in 5.1 35 more if we add the known crashing bugs from 5.0 that are likely to also be present in 5.1.
• We still have more than 180 serious bugs (P2) in 5.1. Some of these can be found here.
• We have more than 300 known and verified less critical bugs that are not going to be addressed soon. (The total reported number of bugs to the MySQL server is of course much larger)
  

Some examples of older bugs that *should* have been fixed in 5.1 before GA:

• Bug #989 "If DROP TABLE while there's an active transaction, wrong binlog order". This is a bug that has been known since August 2003, and has been discussed an referred to in several public places, including Wikipedia and my last talk at the MySQL users conference. It allows in effect anyone with rights to any database that is replicated to take down all slaves (either by accident or intentionally). This is also a bug that has been hit by several of our users in the past.
• Bug #33082 Stored Procedure: crash if table replaced with a view in a loop
• Bug #33094 Error in upgrading from 5.0 to 5.1 when table contains triggers
• Bug #34110 Crash in InnoDB when used "embedded"
• Bug #34502 mysqladmin debug causes a crash when server is creating/dropping many tmp tables
• Bug #34660 crash when federated table loses connection during insert ... select
• Bug #37756 enabling fulltext indexes with MyISAM_repair_threads > 1 causes crash
• Bug #37936 "Crash when executing a query containing date expressions"
• Bug #38816 kill + flush tables with read lock + stored procedures causes crashes!
• Bug #39178 Server crash in YaSSL with non-RSA-requesting client if server uses RSA key
• Bug #40386 Not flushing query cache after truncate
• Bug #40675 MySQL 5.1 crash with index merge algorithm and Merge tables
• Bug #32868 Stored routines do not detect changes in meta-data. Note that this will not be fixed until 6.1 !
• Bug #39526 sql_mode not retained in binary log for CREATE PROCEDURE
• The federated engine is not enabled by default. It was disabled during a previous MySQL-5.1 "RC" release because of bugs filed against the Federated engine that MySQL developers didn't have time to fix. This solution was deemed to be easier than upgrading the Federated engine to a newer version of the engine. This means that people that have problems with the federated engine are better off using a FederatedX plugin, compiling MySQL them self together with FederatedX or use the ourdelta MySQL distribution which contains FederatedX.
• MySQL-Cluster bugs are not fixed in MySQL 5.1; Instead the Cluster engine is moved from the MySQL 5.1 release to a separate MySQL-Cluster release.
  

When it comes to "quality" of the new features in MySQL:

Partitioning:

• 20 open bugs of which at least 7 are targeted to be fixed in later MySQL 5.1 releases.
• Partitioning in MySQL 5.1 should be regarded as a step to a full partitioning feature with parallel query. Parallel query is however not scheduled even for MySQL 6.0.
• For now partitioning is mainly useful in the case where you need to frequently drop a well defined part of a table (like one month of data) and when MERGE tables are too cumbersome to use.
• If one partitioned table crashes it's very hard (sometimes impossible to repair it.
• If you get a server crash during ALTER TABLE of a partitioned table you may loose all your data for that table.
• Partitioning is very slow and can become unusable if you have a large number of partitions. This happens even if you only use a few of the underlying tables in your query.
• Bug #40954 "Crash in MyISAM index code with concurrency test using partitioned tables"
• Bug #40827 Killing insert-select from InnoDB to partitioned MyISAM can cause table corruption
• Bug #30102 rename table does corrupt tables with partition files on failure

Row-based and mixed replication:

Row based replication has been regarded as one of the most wanted feature in 5.1. However, because of several problems with the implementation of row based and mixed mode replication it's not enabled by default. These problems are:

• At least 28 open bugs of which 26 are verified and at least 11 are targeted to be fixed in later MySQL 5.1 releases.
• Row based replication errors on the slave can be hard to debug as you can't see exactly what statement caused the problem. A new feature in MySQL 5.1.28 allows you to see what rows was changed, but this is usually not enough to find out the exact query that failed to replicate.
• For bulk operations on non transactional tables, the data may appear inconsistent during selects on the slave (source MySQL manual)
• Bug #40221 Replication failure on RBR + UPDATE the primary key. This bug is such a serious issue that it should have stopped a GA release!
• Bug #38205 Row-based Replication (RBR) causes inconsistencies: HA_ERR_FOUND_DUPP_KEY. This causes wrong data on slave if you do slave start/stop at the wrong time.
• Bug #40116 Uncommitted changes are replicated and stay on slave after rollback on master
• Bug #40276 Assertion trx_data->empty() in binlog_close_connection
• Bug #31240 load data infile replication between (4.0 or 4.1) and 5.1 fails

Built in job scheduler (Events):
It's hard to find a number of bugs on events as there is no easy way to search for them in the bugs system. In general events are regarded to be one of the more stable features in MySQL, but it's not totally free from problems:

• Bug #40915 "Events takes mutex in wrong order which can easily lead to deadlocks"

New SQL diagnostic aids and performance utilities:

This was part of the announcement but I don't know that we mean with this. I couldn't find anything about this on the "What's New in MySQL 5.1" page. I assume this refers to the SHOW PROFILE patch from MySQL 5.0 community which is now in MySQL 5.1

Logging to tables:

This major feature is unfortunately so slow (30% + slowdown) that it's unusable for busy sites. Ref: Bug #30414: "Slowdown (related to logging) in 5.1.21 vs. 5.1.20". I assume this was why it was left out from the MySQL announcement of MySQL 5.1

Some general crashing/wrong data bugs (not all, just enough to prove a point):

• Bug #40770 Server Crash when running with triggers including variable settings (rpl_sys)
• Bug #37016 TRUNCATE TABLE removes some rows but not all

Most if not alll of the above are things that could and should be been fixed before 5.1 was declared as "GA". Note that this was just a short list of known bugs to prove a point. The real list of serious bugs is much longer. To know if a features is stable enough for your usage, please check the features you plan to use in the MySQL bugs system!

So what went wrong with MySQL 5.1 ?

This is surprisingly not because our developers don't do a good job. On the contrary we have an excellent dedicated team of developers that are very good in what they are doing. However, even an excellent team can't work if the conditions are not right.

Here follows some of the main reasons why MySQL development department again
got a quality problem with a GA release:

• MySQL 5.1 was declared beta and RC way too early. The reason MySQL 5.1 was declared RC was not because we thought it was close to being GA, but because the MySQL manager in charge *wanted to get more people testing MySQL 5.1*. This didn't however help much, which is proved by the fact that it has taken us 14 months and 7 RC's before we could do the current "GA". This caused problems for developers as MySQL developers have not been able to do any larger changes in the source code since February 2006!
• We have changed the release model so that instead of focusing on quality and features our release is now defined by timeliness and features. Quality is not regarded to be that important. To quote Mårten Mickos: "MySQL 5.1 will be release as GA in or before December because I say so". Mårten's reasons for this is that he needs something he can sell and a release marked "GA" is much easier to sell than a release marked "RC".
• The MySQL core developers have been split into too many teams and only a small part of the core developers have been working on MySQL 5.1 to get the bugs fixed. Some of the core developers have also recently left the MySQL organization which is a serious issue as there is not many of of them.
• Too many new developers without a thorough knowledge of the server have been put on the product trying to fix bugs. This in combined with a failing review process have introduced of a lot new bugs while trying to fix old bugs.
• Bug fixing and development processes are not systematic and not persistent.
• We have not been giving the MySQL community enough opportunities to test MySQL 5.1 (too few releases). The reason few releases was made was that if we would have done a release every month, as we have done in the past, we would have got 14 RC releases which would have looked silly and proved that the first RC was made too early. In addition, the MySQL current development model doesn't in practice allow the MySQL community to participate in the development of the MySQL server.
• The MySQL organization doesn't have a release criteria for the MySQL server that is followed; Both the external one and the internal one have not been followed when it comes to declaring MySQL 5.1 as GA. You can read more about our release policy in Kaj's blog.
• Internal QA on the MySQL server was started very late in the process. Now when the process have started to show results, the found bugs have largely being ignored as fixing these they would delayed the MySQL 5.1 GA date.
• The MySQL server team have a bug fixing policy where a bug that has existed a long time has a lower priority 'because people know about them'. This is supposedly one of the reasons why the Bug#989 mentioned above has not been fixed.
  

One would have thought that MySQL AB (now the MySQL department at Sun) should have learned something from our too early release of MySQL 5.0 but unfortunately this is not the case. The main argument I have heard for why MySQL 5.1 was declared as GA now is that it's better than MySQL 5.0 was when it was declared as GA. In my opinion, this is not a good reason to declare something GA, especially as 5.0 GA was in terrible shape when it was released. What is worse is that the new features in MySQL 5.1 are of no better quality than new features in MySQL 5.0 was at the time MySQL 5.0 was declared GA.

What should then have been done before declaring MySQL GA ?

It's of course impossible to get all issues fixed, but we should at least have tried to ensure that all issues important to a lot of MySQL developers and MySQL users should have been discussed, fixed and/or addressed in a public manner! We should also never have a single serious crashing/wrong data bug in a GA release.

There should also be, from MySQL management, an independent release criteria committee that would be the one deciding when the MySQL server is ready to be declared beta, RC and GA. This is something that Sun usually has for their other products.

As I said in my talk at the MySQL users conference, I think it's time to seriously review how the MySQL server is being developed and change the development model to be more like Drizzle and PostgreSQL where the community has a driving role in what gets done!

I would like to point out that the current release is not something that can be said to be fault of Sun. The decisions to do a GA release was solely been made by the MySQL management in Sun. The only thing Sun can be blamed of is to not start fixing the MySQL development organization soon enough to ensure that things like this can't happen.

I still have some hopes that Sun will come in and fix the MySQL development organization, but with MySQL server releases like this one my hopes have started to fade a bit.

There is however some good news in getting MySQL 5.1 released as GA:

• MySQL community users that have not got an update for MySQL 5.0 for 4 months should be able to switch to MySQL 5.1 and now finally get some of their bugs fixed! What still worries me is that the MySQL organization has not yet clearly defined how future MySQL 5.1 versions will be released to the community. This is however a large topic of it's own...
• The MySQL embedded library is back in a supported release. (Not a big thing, but still important for some part of our community).

So what to do next?

Install and test MySQL 5.1; If it works, feel lucky. If not, report a bug at http://bugs.mysql.com/. Don't forget to blog about your experiences with MySQL 5.1!

There is two ways things can go:

• If MySQL 5.1 works for a lot of people and not too many get serious crashes and losses data, then I was concerned without a good reason and everything is fine.
• If MySQL 5.1 does have some serious problems and people report them, the bugs will be fixed and the MySQL & Sun management will have more information to not repeat the same thing with MySQL 6.0.

Good luck with your MySQL 5.1 usage and keep us posted about it!

PS: For those interested, the picture in the blog is decidicated to those in charge of releasing MySQL 5.1 as GA. The statue itself was bought in Riga this year during the internal MySQL developers conference and was presented to the MySQL 6.0 managers as a symbol for the MySQL 6.0 server release planning.

Investing in an open source company (IT mill)

I just made an investment into the Finnish open source company IT mill.

I was elected to be in the IT mill board to be able to efficiently help with the companies business and community strategies. I have also introduced IT mill to Ralf Wahlsten, who will act as an advisor for IT mill's board when needed. Ralf Wahlsten was also an investor in MySQL AB.

IT mill is a still quite small but it has many of the the characteristics of a company that is meant to succeed:

• The company is profitable and has been so for a long time.
• The company knows how to growing organically according to it's profit. (This is very important to know even if one later decides to grow exponentially with the help of investments)
• The company It's has been growing rapidly the last few years (50 % growth per year for 3 years)
• IT mills founder and CEO, Joonas Lehtinen, is burning with a passion for his company, his team and his product.
• The staff seams to be really dedicated to their job and to their leaders (at least as far as I can tell from my visit to their head quarter and spending time with a lot of them in a hot Sauna with *some* vodka to get out the whole truth about the company dynamics)
• Last, but not least, they have well working stable product, that is well received by their customers and that may well be a trend setter for how advanced web applications may be built in the future.
  

(MySQL old-timers may recognize that MySQL AB had many similar characteristics in the early year)

IT mill produces RIA (Rich Internet Applications) programming tools. This allows one to run applications trough the web browser that looks and feels like traditional desktop applications; In other words it allows for much better user experience than traditional web applications but still is as easy to deploy as web applications.

This is the second investment I have done in a Finnish company. My first investment was in Finsor earlier this year. Finsor is not an open source company but their product was so interesting that I couldn't ignore the opportunity I got to invest into it.

My rule for investment is that I want to provide 'smart money'. This means that I will only invest into a few companies where I believe that my technical expertise and/or my experience in building an open source/free software community provides a notable difference for the company.

Still going strong

Gnu is 25 years. How time flies...

I still remember my first contact with the free software people at the Stockholm-Helsinki cruise 1991 (where Minix was first released).

I was back then, of course, already an emacs and gcc user, but that trip was when I first felt an urge to release something as free software. (The concept 'open source' didn't exist back then). It did however take me until 1995 before David and I felt we had something that was good enough to be released.

We haven't yet achieved world domination in all categories, but we are definitely getting there; Some things take time, but we have time on our side.

As a way to celebrate, the actor and humorist Steven Fry has made a video entitled 'Happy Birtday to Gnu'. Please check it out!

Happy birthday to Gnu!

What if

For the last 2-3 years, Brian Aker and I have had many discussions about how to refactor MySQL. Brian has been the one driving these discussions by asking why some things in MySQL were done in a certain way and in a true "what if" manner asked what would happen if we would do things in another way.

Being tired of not being able to get critically needed reconstruction work done in the MySQL server, Brian started to work on Drizzle to search for answers to these questions.

So what is Drizzle?

• Drizzle is what MySQL would be with a more interactive community involvement in the design of the software itself, and had targeted website deployments.

• Drizzle is a version of MySQL that is driven by Brian and the community, attempting to solve practical problems that a large group of MySQL users are facing.
  

• Drizzle is a smaller, slimmer and (hopefully) faster version of MySQL; Features that the broad Drizzle community does not want or need are now removed or in the process of being removed (This includes stored procedures, views, triggers, grants, some non-pluggable storage engines and more).

• Drizzle is the 3rd fork of MySQL server code base, but is the one that has (for now) the most developers working on it. The two other forks are:
• ProvenScaling's fork on mirror.provenscaling.com provided by Jeremy Cole. You should read Peter Zaitsev's blog about this.

• The MySQL-5.1-maria fork, provided by the Maria team lead by me. (Brian did beat us when it comes to opening up the tree for outside development; We are still at least a month away from doing this).

• You can find out more about Drizzle at Brian's blog at http://krow.livejournal.com/.
  

Why is it that the people working on Drizzle are extremely enthusiastic about Drizzle?

This is because Drizzle solves many of the problems that MySQL's development has had for years:

• It opens up MySQL development for the community; You no longer have to wait years to get your patches and resonable extensions into the server.

• Critical bugs that have existed for years can finally get fixed as the development is no longer constrained by unrealistic release schedules that put artificial constraints on things that can be fixed.

• Drizzle will put some MySQL server differentiation on a true test; A bit like Fedora does to RedHat.

• Drizzle has created new excitement in the MySQL developer community; A lot of people seem to be very enthusiastic to work on it in a true community-oriented manner.

• Developers working on Drizzle is doing drastic refactoring of the server, something that MySQL planned to do years ago but never happened.

• Development decisions is again driven by people that are using the server daily; This will ensure that Drizzle will be faster and more stable than what can be done with current MySQL development model

• Drizzle will target the MySQL core users, the web users, whose requirements have been ignored for years while the core MySQL developers have added features that they don't need.

• In addition Drizzle will include the latest InnoDB code; You don't have to wait for MySQL 6.0 or go to the trouble of annually downloadoing and installing the InnoDB plugin from Oracle just to get access to the latest and fastest InnoDB version.

That said, Drizzle is not here to replace the normal MySQL server; Drizzle targets a limited but important market and will thus help us the enhance the MySQL based offerings. Think of Drizzle as the microkernel server around which other offerings/features can be developed.

Drizzle has no release schedule or timeline, but will follow a true open source and agile methodology of releasing early and releasing often; Brian expects that there will be a usable, reasonably stable version of drizzle within 3-4 months, but there are no promises.

Drizzle will also be of great help for the 'normal' MySQL developers; By looking at how Drizzle is solving things and by constant benchmarking against Drizzle, we will get a better insight into the weaknesses of the current codebase and have a better idea of what needs to be fixed; Friendly competition is good!

Drizzle is one of the good things that have been made possible by Sun acquiring MySQL. Brian has been working on Drizzle with the blessing and encouragement from Sun's upper management. We are finding Sun to be open and encouraging of innovation, this has been a good aspect of the acquisition.

My personal reaction to Drizzle is that I am very enthusiastic for this new kid on the MySQL block. I don't agree with everything that is done, but most things makes a lot of sense. I am looking forward to a lot of very interesting discussions about solutions in Drizzle that will help improve both MySQL versions.

A bugs life

This is a request to all MySQL users to help mysql developers, by providing information, so that we can help you, by providing a more stable MySQL server for your needs.

As you may know, MySQL 5.1 has been in state of release candidate (RC) for some time. The last RC was announced as the last RC and is supposed to be followed by a GA release. The GA release is planned to be the exact same code as the last GA, only with the label changed.

The question we, who are developing and supporting the MYSQL server have been asking ourselves is, "Are really now in shape to do a proper GA release?".

We would like you as a MySQL User to help us out with deciding this.

We don't want to repeat the mistake we did with MySQL 5.0 GA and then again with MySQL 5.1 RC, by releasing a MySQL 5.1 GA too early.

Our external criteria for General Availability (GA) or Production release can be found here.

What you may not know is that we have as part of our internal GA criteria, a requirement that we should not have any serious bugs, crashing or wrong result bugs, that affects a notable amount of users. The criteria states that it is ok to postpone fixes for bugs that have a low impact (ie affects few users).

This has the following implications:

• Bugs for which we don't have many user/customer reports for are not likely to get fixed. (In the worst case not even in 6.0 !)
• It's ok to go out with bugs in new feature in the GA as long as we don't have many users/customers that have reported problems with these features.
  

In other words, if you have an issue with a serious bug that exists in 5.1 that you *really* would like to have fixed soon now is your chance to influence our development!

Note that it's ok, and we want, you to also report bugs after we release the MySQL 5.1 as GA. The more users/customers commenting on a bug the more chance it will be fixed!

What I would like for people to do:

• Report every single bug that you encounter or know about in 5.1 into our bugs system.

• If the bug is already reported, please add a comment to the bug report that it affects you too.
• If you are a MySQL support customer, add either a note to the bug report that you are a customer and the bug affects you or send a request as a customer directly to the MySQL support personal (they are happy to take your reports!)
• If you have reported a bug a long time ago that has not been fixed and this is still important for you to get fixed, please reopen the bug/add a note in the bugs database that the bug is still relevant for you!
  

Note that you should also do this for bugs that you know of in MySQL 5.0 that are still open for MySQL 5.1 (or any bugs that are labelled to be fixed in a future release). Bugs that are in earlier version are also very likely to also be in 5.1 if the bugs database doesn't say otherwise.

Lets make the MySQL server bugs life's harder by giving us more information about which bugs are really important for us to fix. More information will help us make better decisions about how and when a bug should be fixed.

Bringing up the baby

At last we are code complete with Maria 1.5, the so called crash safe version of MyISAM. Note that code complete means that all features are coded, not that the code is 100 % bug free!

The code should be available soon in a bzr repository near you. We will also shortly (as soon as we pass all build tests) make a normal source and binary release of MySQL-5.1-Maria from here.

There is still a couple of minor bugs that we know about and we will fix them shortly, but nothing that should stop you from testing/using Maria. See the file KNOWN_BUGS.txt in the source release for details.

The current release plan for MySQL-5.1-Maria is as follows:

- The current release will be an alpha release. (Technically it could have been a beta release but because we have added so much new code we wanted to signal this by making another alpha release).

- If there is not any significant bugs found the would require a major rewrite of the code, we will for the next release fix the found bugs and make a beta release. This should hopefully be done August.

- We will continue doing one beta releases per month until there is no more redesign done and the new bug reports are down to a minimum. Then we will make a final RC and then a GA release (assuming everything goes well).

We have soo far been able to keep the opens bugs in Maria down to a minimum. This together with a quite large test suite make us, the Maria team, reasonable confident that we don't need many beta releases before we can come to RC and then GA. The more people that is downloading Maria to test and use it, the faster it will get stable. Please report any bugs you find and we will do our best to fix them!

At the same time we are merging MySQL-5.1-Maria into the standard MySQL-6.0 tree, where Maria will be a core part of the server.

The next step is to span of a MySQL-6.0-Maria tree in which we will start working on making Maria fully transactional and more concurrent.

Bug fixes that affects MySQL-5.1-Maria will of course be done in this tree and then be merged up to the MySQL-6.0 and MySQL-6.0-maria trees.

Those that are interested in known more about Maria, now and future, are welcome to my Maria talk and/or the Maria BOF at Oscon on July 24'th.

At Oscon Tim O'Reilly will interview Brian Aker and me on a keynote. . We will all do our best to make this a fun and unforgettable event!

Now it's time for me to go and have my well earned vacation. (I actually planned to go already last Monday, but I wanted to finish my part of the Maria project before going and now it's next Sunday...). At least I will get some rest from computers during the next two weeks as we don't have any electricity in my summer cottage up in Vasa.

Hope to see many of you at Oscon!

For a few dollars more

It's nice to be able to make a difference.

Since David Axmark and I started to work on MySQL we also took a strong stand against software patents. MySQL AB have been sponsoring several efforts to prevent software patents in Europa.

Now David and I are continuing to do this outside of MySQL AB. We just gave an economic sponsorship to the Patent Lens, who recently lost their main sponsor, so that it can continue it's important work in making the patents system more accessible.

Here is a short description of Patent Lens from it's founder, Richard A Jefferson:

We are working to generalize the Patent Lens as the informatics platform of the Initiative for Open Innovation, which aspires to render worldwide patent systems understandable and navigable, and to guide policy and practice reform. We intend to create a sector and jurisdiction agnostic resource that is open source, open access and free of cost.

The funding you may provide will keep a small core team going long enough to make it through these lean times and build new momentum.

If you are a philanthropist and if you care about open source software and don't have a love for the current patent system, I encourage you to join us in sponsoring Patent Lens. You can also try to get your company to sponsor.

David and I have also recently sponsored another company that works for Free and open source software, the Software Freedom Law Center.

We did it because we believe in what they are doing and we are greatful for all the help that Eben Moglen gave us in the past when we got into trouble.

Honor award at HP Linux forum 2008

I got yesterday a honor award at the HP Linux Forum 2008 in Helsinki, from FLUG (Finnish Linux user group) for "important work for the good of Linux" and for "working to keep the code open even when part of MySQL management has been of different opinion".

The major award went to Ubuntu Suomi for their good work in translating Ubuntu to Finnish and helping Finnish users setting up and using Ubuntu.

Linux.fi got the other honor award for their work on creating a Finnish Wiki for Linux users.

Arto Teräs, the spokesman of FLUG, said among other things, that "MySQL was one of the killer applications that brought Linux to the business world and to other places where Linux was not used before". You can find a video of his presentation here.

I am honored and happy for the recognition from FLUG. Thanks FLUG for the award and for HP hosting the event!

It was a well attended event (for being Finland); Based on the number of cars on the parking slot I would estimate the number of persons to at least 500. The one thing that I found missing was that Sun was not present at the event. (Note to myself; Need to fix this for next year)

Back to the future

I was yesterday attending the "Open Tuesday" Sun & MySQL event. One of the first questions I got from the audience during my questions & answer session was what is my take of the recent MySQL proposition of having closed source parts/modules in the server.

I was very happy to be able to say that Mårten some hours earlier had announced on CommuntyOne that the MySQL server is and is always going to be open source.

It looks like the community is also quite happy about the outcome, as can be seen on the responses from Slashdot when this was announced.

Mårten did also say on the CommunityOne panel that, that we should "expect Sun/MySQL to continue experimenting with the business model, and with what’s offered for the community and what’s offered commercial-only."

My hope is that the experiment when it comes to closed source extensions developed by Sun is now ended. As far as I know, there is no existing plans for any closed source extensions to the MySQL server. The strong commitments made now by Mårten and by Jonathan Schwartz in his interview by Tim O'Reilly where Jonathan said "Everything Sun delivers will be freely available, via a free and open license (either GPL, LGPL or Mozilla/CDDL), to the community. Everything. No exception." should ensure that things stays this way.

Kaj Arnö, who attended the event, has in his blog this in detail and his post is worth reading.

I would however like to clarify one thing Kaj said, in an attempt to avoid future confusion:

"Then for MySQL’s business model. To financially support MySQL’s free and open source platform, we have a business model which allows both community and commercial add-ons, and we remain committed to it."

I interpret this, in the context of Mårten's and Jonathan's announcements, that we will continue to support and make available commercial addons to the MySQL server from third party, like the Infobright storage engine. Things that we develop ourselves at Sun, at least on the server, will continue to be open source.

It's very good to see that Mårten is continuing to be responsive to the MySQL community and to the MySQL customers. Thanks to Mårten for doing the right thing! Thanks to the MySQL community for expressing their opinions!

For those that wonder what the title of the blog has to do with this post, I just want to say Open Source is the Future and the MySQL server is now back on track for the Future.

MySQL conference; The good, the bad and the ugly

The "2008 MySQL Conference and Expo" previously known as the "MySQL Users Conference" is over and I finally have time to blog.

What I really like with the MySQL conference is that it shows the loyalty that MySQL has from it's users and community. Year after year I get the change to meet the same friendly (!) faces and every year the group grows notable bigger (more than 2000 attendees this year). It really feels amazing for me to that I have had a small share in making this happen!

A lot of things happened during the conference, some good, some bad and some ugly.

Lets start with the bad parts and then go to the good stuff.

The Ugly

The ugly part was of course the announcement that MySQL was planning to change the MySQL server from open source/free software to crippleware by only giving out key parts of MySQL online backup (a server component) as closed source within the Enterprise server offering.

This announcement was not taken well by the MySQL community, as can be seen by the several hundreds of comments at slashdot

I would like to emphasis that this decision was made by the MySQL management team and *NOT* influenced in any way by Sun Management! All the Sun people I have met so far, including Jonathan Schwartz and my boss Greg Papadopoulos, CTO of Sun, has made it very clear for me that Sun is for open/free software and they was caught by surprise by this announcement. I think this happened because the integration process between MySQL and Sun is yet not complete, and MySQL AB is still functioning more or less as an atomic entity inside of Sun.

Mårten showed at his keynote a photo of where they were burning the IPO Prospectus for MySQL AB. This was a very cool thing to do!

What the MySQL management team forgot to burn, was all the plans they had of how to make more money when MySQL would be a public company. They have apparently not yet realized that when MySQL AB was acquired by Sun, things changed. As I understand, Sun is not interested in closing MySQL up, in fact quite the contrary. Sun bought MySQL to expand their open source/free software offerings and also to use the expertise we have in MySQL AB to transform Sun to be even more successful in this software space. More about this later.

The Bad

MySQL 5.1.24RC binaries (but not the source) was released without federated tables compiled in. Federated tables is a feature that is used by thousands of MySQL users, including MySQL customers. A lot of people within MySQL development, the MySQL community team, product management and myself, tried to stop this from happening but unfortunately we where not successful. I find it outrageous that something like this can be done within an RC release and see this as yet another indication that 5.1 is far from being ready to be released as GA. I think we are still 4-6 months from having a stable 5.1 GA.

The conference was a very good one, but one can clearly see the commercial interests are starting to take over the conference. There were fewer talks from the MySQL community than before, talks from people that could be seen as MySQL competitors was regulated and some talks was censored. If the conference is going even more into this direction the next year, the community should start thinking about organising a MySQL developer conference that would be targeting those that are contributing to MySQL server or developing solutions with MySQL.

The Good

The next release of MySQL 5.1RC binaries will again have federated enabled, hopefully with most of the federated issues addressed.

I agreed to join Sun one week ago. I will be working at Sun Labs directly under Greg Papadopoulos. My tasks, among other things will include:

• Finishing and Releasing the Maria storage engine as per current schedule
  
• Continue to do architectural work on the MySQL server and in general help with MySQL development
• Help Sun to be even more successful in the open source/free software world by actively helping different open source projects within Sun to be more involved with the community and more community driven.
  

I am very excited in joining Sun and working on making Sun the leading open source/free software company! I will of course make it a personal goal to keep MySQL development on track and to ensure that the MySQL server in the future will continue to be developed in an open source friendly fashion.

A new InnoDB version with a lot of new features was released under GPL at the conference for MySQL 5.1. Thanks to Heikki Tuuri and the InnoDB team!

My first talk "Maria engine" was quite well received. This included a short presentation of the XDB indexes that I have hinted about in one of my earlier blogs. A description of this will soon appear at http://forge.mysql.com/worklog/task.php?id=3537. We hope to finish the Maria 1.5 version, which includes concurrent inserts and selects to tables with versioning, within a few weeks. After that we will work on stabilising the MySQL-5.1-Maria release and then start working on the MySQL-6.0-Maria release that will include transactions.

You can find the slides to this talk at http://www.scribd.com/doc/2575706/Maria-Engine

The Maria BOF was even better received. Here we had a demo that proved that Maria was indeed crash safe and to celebrate the successful demo we then consumed 3 liter of black vodka.

My second talk "Future Design Hurdles to Tackle in the MySQL Server" or "The Future of MySQL (The Project)" was even better received. During the talk we had spontaneous chanting of "We don't ship crippleware" and afterwards I got many people coming to say to me that it was my best talk ever! (This doesn't of course say anything else than that I may be improving or that they have never heard me present before).

You can find the slides to this talk here.

I have been contacted to give this presentation again at other events, and I look forward to be working on MySQL with Sun, and with the MySQL Community.

Maria build & public MySQL architecture meeting

First 'normal' build of Maria is now released; This is a build that went trough our build farm and compiled and passed all tests on many different OS and machine combinations.

You can find the release at http://dev.mysql.com/downloads/maria/index.html

I am quite happy with the Maria releases so far. The Maria team has been able to fix almost all reported bugs (a couple of not repeatable bugs remains) but the best news is that there have been very few bugs.

Now we have started to work on the few remaining features that is needed to be able to shift from Alpha to Beta: The main one is many concurrent insert and many concurrent selects on the same table. We hope to have this done in good time before the MySQL Users conference in April.

For anyone attending the MySQL users conference, I would like to take the opportunity to invite them interested to an open MySQL architecture meeting on the day after the conference, Friday April the 18'th at 10.00 to 14.00 somewhere in the conference hotel.

The idea is to have some MySQL architects in a panel and then freely discuss the challenges in the MySQL architecture and how the MySQL server needs to be changed and evolve. We want to get advice in whatis important for other developers and also give advice for anyone wanting to do work on the server.

This is one of the many initiatives we are planing to open up the MySQL server development for outside participants. (Yes, I know that the current initiatives for doing this has not worked out very well, but I hope that this is about to change).

I hope that within Sun we will get resources to change our current polices, priorities and in some ways the whole engineering organization to make the development model much friendlier to outside participants. It should be as easy for an outsider to get a patch into the MySQL server as someone working for MySQL. This is one of the things I would like to spend my time on inside of Sun!

Maria specifications are available

Hi!

You can now find the specifications, flow charts and implementation details for all (except one) parts of Maria at:

http://forge.mysql.com/worklog/search.php?k=Maria&t=tds

This makes Maria the best documented project in the history of MySQL AB. Yes, I know that this doesn't say much, but I think we have done a really decent job in this case.

The public worklog doesn't yet display the dependences between tasks, like our internal worklog does, which makes it a bit hard to navigate between tasks and to understand in which order tasks will be done. I hope this will be fixed soon.

So, you may ask, what is the one task that we have not published.? This is a new very cool way to index data to make some searches extremely fast. The reason for not releasing it is just that I
wanted to save this to my talk on the MySQL users conference in April as a nice surprise.

Regards,
Monty

The Maria engine is released

Hi!

We have now made the bitkeeper tree of Maria public at http://mysql.bkbits.com


Maria is a new storage engine that Guilhem, Sanja, Sergei and I have been working on for the last 2 years.

(Before anyone comments that this is not 'that much' for 2 years work, I would lke to note that the first 1 1/2 this was done part time while we have finished other projects; It's only during the last 4 months we have been working full time on this. From now on we Maria is our main focus.)

We are still discussing in MySQL AB how and when we will start to make binary distributions of Maria; Until then, please use the bk tree to test the Maria engine! We are now in a state of Maria 1.0 with 'no known bugs' (which of course doesn't mean that there isn't any bugs just that we belive we are in 'reasonable good shape') and we need the help from other developers to find the hidden bugs so that we can quickly fix them and make Maria stable!

Here follows an FAQ.txt that I have written and to which the current Maria developers have committed. Note that this is NOT a commitment from MySQL AB but a personal commitment from me and the other developers working on the Maria project.

If you have questions, comments bugs or patches, please send them to the soon-to-be made Maria mailing list and the Maria forum).

Enjoy!
Monty

--------------

This FAQ explains what to expect of the Maria storage engine
(below referred to only as 'Maria') and of the mysql -5.1-maria release.

NOTE: The opinions and promises stated in this FAQ is by the Maria development team and not promises by MySQL AB.

Overview of questions:

Q: Why is the engine called Maria ?
Q: What is goal of Maria in current (MySQL 5.1) release ?
Q: How do you plan to reach this goal ?
Q: What will be in the next major release of Maria ?
Q: What is ultimate goal of Maria ?
Q: What are the 'design goals' in Maria ?
Q: Where can I find documentation and help about Maria ?
Q: Who is behind Maria ?
Q: What is the release policy/schedule of Maria ?
Q: How does Maria 1.0 Compare to MyISAM ?
Q: What is the differences between the MySQL-5.1-Maria release and the normal MySQL-5.1 release ?
Q: Why do you use the TRANSACTIONAL keyword now when Maria is not yet transactional ?
Q: What are the currently known problems with the MySQL-5.1-Maria release ?
Q: What things are going to change in later Maria main releases ?

Usage questions:
Q: How can I create a MyISAM like (non-transactional) table in Maria ?
Q: Whats the proper way to copy a Maria table from on place to another ?
Q: When is it safe to remove old log files ?
---------
Q: Why is the engine called Maria?
A:

Monty, the creator of MySQL, named MySQL after his first child 'My'. His second child, Max, gave his name to MaxDB and the MySQL-Max distributions. His third and youngest child is named Maria...

---------

Q: What is goal of Maria in current (MySQL 5.1-Maria) release ? 
A:

To make a crash-safe alternative to MyISAM. That is, when mysqld restarts
after a crash occurs, Maria will recover all tables to the state as of the
start of a statement or at the start of a previous LOCK TABLES.

Another primary goal for this release is make Maria of good quality; performance is not a primary goal for this release! We will start to work on performance when we know we have a good basis for Maria!

The goal is more clearly defined in WorkLog at http://forge.mysql.com/worklog/task.php?id=3871

We are now at Maria 1.0 in the worklog schedule and we will take it to Maria 1.5 during the alpha period of Maria-5.1-Maria-alpha.

For things that still need to be done for Maria 1.5, the first Maria main release, check section "Missing features that is planned to fix before Beta" in the KNOWN_BUGS.txt file

--------

Q: How do you plan to reach this goal?
A:

Continuously improve the MySQL-5.1-Maria release until we reach the goal above. The improvements between releases should be made in such a way that upgrades are trivial (that is, no requirement to dump and reload data and no application changes).

The Maria development tree will be open and we will do monthly source and binary releases until the Maria engine reaches a stable state. After first release, for at least one year, we will
provide bug fixes and source and binary releases of the mysql-5.1-maria tree. (See also Q: What is the release policy/schedule of Maria)

When Maria 5-1-Maria gets to be declared Beta, we will start implementing the new features for Maria 2.0 into a MySQL-6.X-Maria-alpha branch.

-------------

Q: What will be in the next major release of Maria?
A:

This is defined in http://forge.mysql.com/worklog/ for Maria 2.0. Here follows a short description:

• ACID
• Commit/Rollback
• Concurrent selects (thanks to MVCC)
• Row locking
• Group commit

--------------

Q: What is ultimate goal of Maria?
A:

NOTE: The following goals are the *technical goals* of the project team. How Maria in the end will be incorporated in the MySQL Server product and whether it will replace MyISAM is subject to decision by MySQL Product Management.

• To create a new, ACID and multi-version concurrency Control (MVCC), transactional storage engine that can function as the default non-transactional an the default transactional storage engine for MySQL.
  
• To be a MyISAM replacement. This is possible because Maria can also be run in non-transactional mode, supports all row formats as MyISAM, and supports or will support all major features of MyISAM.
• Maria to be standard part of MySQL 6.X

-------------

Q: What are the 'design goals' in Maria?
A:

• Multi-version concurrency Control (MVCC) and ACID storage engine
• Optionally not transactional tables that should be 'as fast and as compact' as MyISAM tables.
• Be able to use Maria for internal temporary tables in MySQL (instead of MyISAM)
• All indexes should have equal speed (clustered index is not on current road map).
• Allow 'any' length transactions to work (Having long running transactions will cause more logs space to be used)
• Allow log shipping; Ie, you can do incremental backups of Maria tables just by copying the maria logs.
• Allow copying of Maria tables between different Maria servers (under some well-defined constraints)
• Better blob handling (than that in MyISAM at least):
• No memory copying or extra memory used for blobs on insert/update.
• Blobs allocated in big sequential blocks - Less fragmentation over time
• Blobs are stored so that Maria can easily be extended to have access to any part of a blob with a single fetch in the future.
• Efficient storage on disk (that is, Low row data overhead, low page data overhead and little lost space on pages). Note: There is still some more work to succeed with this goal. The disk layout is fine, but we need more in-memory caches to ensure that we get a higher fill factor on the pages.
• Small footprint, to make MySQL + Maria suitable for desktop and embedded applications.
• Flexible memory allocation and scalable algorithms to use huge memory efficient, if available.

--------

Q: Where can I find documentation and help about Maria?
A:

Documentation about Maria can be found at: http://forge.mysql.com/wiki/Maria_Docs

You can find also use the Maria email list at: maria@lists.mysql.com or at the Maria forum at: http://forums.mysql.com, section Maria

You can report bugs and check bugs in Maria in the MySQL bugs system at http://bugs.mysql.com/

---------

Q: Who is behind Maria?
A:

The current MySQL AB Maria team is:

Technical lead:
Michael "Monty" Widenius - Creator of MySQL and MyISAM

Core developers (in alphabetical order)
Guilhem Bichot - Replication expert, on line backup for MyISAM, etc.
Oleksandr Byelkin - Query cache, sub-queries, views
Sergei Golubchik - Server Architect, Full text search,
keys for MyISAM-Merge, Plugin architecture, etc.

Other developers
Jani Tolonen - Long term all employee of MySQL AB; Worked with
Monty since 1997.

Managers
Patrik Backman - Director of Software Engineering at MySQL AB

---------

Q: What is the release policy/schedule of Maria?
A:

The Maria team can only promise to cover bugs in Maria. For bugs in MySQL, we can't give any guarantees. However, bugs in MySQL that directly affect Maria or are related to Maria are likely to be fixed or worked around by the Maria team.

The Maria development release policy follows the MySQL release criteria at http://dev.mysql.com/doc/refman/5.1/en/choosing-version.html. In addition to this we have the following project commitments for the maturity stages:

The release policy for all Maria storage engine releases:

• All Maria releases should be free from bugs that can cause problem in 'normal' operation for most users.
• We aim for one release per month until Maria is stable. This will include a source release and binary release for all machines in our build farm for which Maria passes all tests.
• When Maria becomes stable we will do few releases monthly, but later only when critical bugs have been found and fixed.
• The MySQL-Maria release will include all changes from the main MySQL tree up to at least the latest stable build of MySQL.
• If we make a release with known critical bugs (for example, if there is a really nasty bug we want to fix at once and get the fix out, while we are fixing other bugs), they will be documented in the KNOWN_BUGS.txt file.
• Maria index and data file formats should be backward and forward compatible to make it easy to upgrade/downgrade.
• We will try to keep the log file format compatible, but for this we don't want make any guarantees yet. In other words, in some cases when upgrading, you must remove the old maria_log.######## files before restarting mysqld.

Commitment of Beta

• Maria is feature complete according to specification for Maria 1.5 in worklog at http://forge.mysql.com/worklog/task.php?id=4158
• No known bugs, except for those marked as feature requests
• No bugs fixed since last release that caused a notable code changes.

Commitment of RC/Gamma:

• No known bugs, except for those marked as feature requests
• No bugs fixed since last release that caused a notable code changes.
• We belive Maria code is ready for general usage (based on bug inflow), but we want more testing before calling it stable

Commitment of GA/stable (and all releases after this)

• No known bugs, except for those marked as feature requests
• No bugs fixed since last release that caused a notable code changes.
• We belive Maria code is ready for general usage (based on bug inflow).

In the rare case when there is a bug that can't be fixed in a specific release (either because it's a design bug or the bug fix is likely to cause other, possible worse bugs), we will document it in the KNOWN_BUGS.txt section. However, we will try to keep these kinds of open bugs at a minimum.

--------

Q: How does Maria 1.0 Compare to MyISAM?
A:

Maria 1.0 is basically a crash-safe non transactional version of MyISAM.

Maria supports all aspects of MyISAM, except as noted below. This includes external and internal check/repair/compressing of rows, different row formats, different index compress formats, maria_check etc. After a normal shutdown one can copy Maria files between servers.

Advantages of Maria (Compared to MyISAM)

• Data and indexes are crash safe.
• On a crash, changes will be rolled back to state of the start of a statement or a last LOCK TABLES commands.
• Maria can replay almost everything from the log. (Including create/drop/rename/truncate tables).  Therefore, you make a backup of Maria by just copying the log. The things that can't be replayed (yet) are:

• Batch INSERT into an empty table (This includes LOAD DATA INFILE, SELECT ... INSERT and INSERT (many rows))
• ALTER TABLE. Note that .frm tables are NOT recreated!
• LOAD INDEX can skip index blocks for unwanted indexes

• Supports all MyISAM row formats + new PAGE format where data is stored in pages. (default size is 8K)
• When using PAGE format (default) row data is cached by page cache.
• Maria has unit tests of most parts
• Supports both crash-safe (soon to be transactional) and not transactional tables. (Non-transactional tables are not logged and rows uses less space): CREATE TABLE foo (...) TRANSACTIONAL=0|1 ENGINE=Maria
• PAGE is the only crash-safe/transactional row format.
• PAGE format should give a notable speed improvement on systems which have bad data caching. (For example windows).

Differences between Maria and MyISAM:

• Maria uses BIG (1G by default) log files.
• Maria has a log control file (maria_log_control) and log files ( maria_log.???????). The log files can be automatically purged when not needed or purged on demand (after backup).
• Maria uses 8K pages by default (MyISAM uses 1K). This makes Maria a bit faster when using keys of fixed size, but slower when using variable-length packed keys (until we add a directory to index pages)

Disadvantages of Maria (compared to MyISAM), that will be fixed soon

• Maria 1.0 has one writer or many readers. (MyISAM can have one inserter and many readers when using concurrent inserts).
• Maria doesn't support INSERT DELAYED.
• Maria does not support multiple key caches.

Disadvantages of Maria (compared to MyISAM), that will be fixed in later releases

• Storage of very small rows (<>
• MERGE tables don't support Maria (should be very easy to add later).

Differences that are not likely to be fixed.

• Maria data pages in block format have an overhead of 10 bytes/page and 5 bytes/row. Transaction and multiple concurrent-writer support will use an extra overhead of 7 bytes for new rows, 14 bytes for deleted rows and 0 bytes for old compacted rows.
• No external locking (MyISAM has external locking, but is not much used)
• Maria has one page size for both index and data (defined when Maria is used the first time). MyISAM supports different page sizes per index.
• Index number requires one extra byte per index page.
• Maria doesn't support MySQL internal RAID (disabled in MyISAM too, it's a deprecated feature)
• Minimum data file size for PAGE format is 16K (with 8K pages)

------------

Q: What are the differences between the MySQL-5.1-Maria release and the normal
MySQL-5.1 release?
A:

• Maria is compiled in by default and required to be 'in use' when mysqld is started.
• Internal on-disk tables are in Maria table format instead of MyISAM table format. This should speed up some GROUP BY and DISTINCT queries because Maria has better caching than MyISAM.

New options to CREATE TABLE:

• TRANSACTIONAL= 0 | 1 ; Transactional means crash-safe for Maria <>
• PAGE_CHECKSUM= 0 | 1 ; If index and data should use page checksums for extra safety.
• TABLE_CHECKSUM= 0 | 1 ; Same as CHECKSUM in MySQL 5.1
• ROW_FORMAT=PAGE ; The new cacheable row format for Maria tables. Default row format for Maria tables and only row format that can be used if TRANSACTIONAL=1. To emulate MyISAM, use ROW_FORMAT=FIXED or ROW_FORMAT=DYNAMIC
• CHECKSUM TABLE now ignores values in NULL fields. This makes CHECKSUM TABLE faster and fixes some cases where same table definition could give different checksum values depending on row format. The disadvantage is that the value is now different compared to other MySQL installations. The new checksum calculation is fixed for all table engines that uses the default way to calculate and MyISAM which does the calculation internally. Note: Old MyISAM tables with internal checksum will return the same checksum as before. To fix them to calculate according to new rules you have to do an ALTER TABLE. You can use the old ways to calculate checksums by using the option --old to mysqld or set the system variable '@@old' to 1 when you do CHECKSUM TABLE ... EXTENDED;
• At startup Maria will check the Maria logs and automatically recover the tables from last checkpoint if mysqld was not taken down correctly.
• There are some improvements to DBUG code to make its execution faster when debug is compiled in but not used.

Q: Why do you use the TRANSACTIONAL keyword now when Maria is not yet transactional?
A:

In the current development phase Maria tables created with TRANSACTIONAL=1 are crashsafe and atomic but not transactional because changes in Maria tables can't be rolled back with the ROLLBACK command. As we  will make Maria tables fully transactional in a relatively short time frame we think it's better to use  the TRANSACTIONAL keyword already now so that applications don't need to be changed later.

Tables marked with TRANSACTIONAL=1 will for each Maria release get more and more transactional aspects and when we reach Maria 2.0 they will be fully transactional in the traditional sense.

----------

Q: What are the currently known problems with the MySQL-5.1-Maria release?
A: None (just kidding, this is software we are talking about...)

• See KNOWN_BUGS.txt for open/design bugs
• See http://bugs.mysql.com/ for newly reported bugs. Please report anything you can't find here!
• If there is a bug in the Maria recovery code or in the code that generates the logs, or if the logs become corrupted, then mysqld may fail to start because Maria can't execute the logs at start up.

FIX: Remove the maria_log.???????? files from the data directory, restart mysqld and run CHECK TABLE / REPAIR TABLE or mysqlcheck on your Maria tables.

Alternative remove logs and run maria_chk on your *.MAI files

----------

Q: What things are going to change in later Maria main releases?
A:

LOCK TABLES will not start a crash-safe segment. You should use BEGIN/COMMIT instead. To make things future safe, you could do this:

BEGIN;
LOCK TABLES ....
UNLOCK TABLES;
COMMIT;

And later you can just remove the LOCK/UNLOCK part.

------------

Q: How can I create a MyISAM like (non-transactional) table in Maria?
A:

Example:

CREATE TABLE t1 (a int) ROW_FORMAT=FIXED TRANSACTIONAL=0 PAGE_CHECKSUM=0;
CREATE TABLE t2 (a int) ROW_FORMAT=DYNAMIC TRANSACTIONAL=0 PAGE_CHECKSUM=0;
SHOW CREATE TABLE t1;
SHOW CREATE TABLE t2;

Note that the rows are not cached in the page cache for FIXED or DYNAMIC format. If you want to have the data cached (something MyISAM doesn't support) you should use ROW_FORMAT=PAGE:

CREATE TABLE t3 (a int) ROW_FORMAT=PAGE TRANSACTIONAL=0 PAGE_CHECKSUM=0;
SHOW CREATE TABLE t1;

You can use PAGE_CHECKSUM=1 also for non-transactional tables; This puts a page checksums on all index pages. It also puts a checksum on data pages if you use ROW_FORMAT=PAGE.

-----------

Q: Whats the proper way to copy a Maria table from on place to another?
A:

A Maria table consists of 3 files:
XXX.frm ; The definition for the table, used by MySQL
XXX.MYI ; Maria internal information about the structure of the data
and index and data for all indexes.
XXX.MAD ; The data

It's safe to copy all the maria files to another directory or MySQL instance
if any of the following holds:

- If you shutdown mysqld properly with 'mysqladmin shutdown', so that there is nothing for Maria to recover when it starts.

or

- You have done 'flush tables' and not accessed the table using SQL from that time until the tables have been copied.


In addition, you must adhere the following rule for transactional tables:

You can't copy the table to a location within the same MySQL server if the new table has existed before and the new table is still active in the Maria recovery log (that is, Maria may need to access the old data during recovery). If you are unsure whether the old name existed, run 'maria_chk --zerofill' on the table before you use it.

After copying a transactional table and before you use the table, we recommend that you run the command:

'maria_chk --zerofill table_name'

This will overwrite all references to the logs (LSN), all transactional references (TRN) and all unused space with 0. It also marks the table as 'movable'.

Maria will automatically notice if have copied a table from another system and do 'zerofill' for the first access of the table. The reason for using maria_chk is that you avoid a delay in the MySQL server for the first access of the table. Note that this automatic detection doesn't work if you copy tables within the same MySQL server!

------------

Q: When is it safe to remove old log files ?
A:

If you want  to remove the Maria log files with 'rm' or delete, then you must first shut down MySQL cleanly (for example, with 'mysqladmin shutdown') before deleting the old files.

The same rules apply when upgrading MySQL; When upgrading, first take down MySQL in a clean way and then upgrade. This will allow you to remove the old log files if there are incompatible problems between releases.

------------

Q: How does the other Maria look like:
A:

Sun buys MySQL AB

Hi!

Recent events have convinced me that it's finally time for me to start blogging.

The first thing I would like to talk about is about Sun buying MySQL AB. In a second blog entry I will then talk about the Maria storage engine, a project that I have been working on for the past two year (first half time and now full time).

Personally I think that the deal with Sun is the best thing that could have happened to us. When David, Allan and I originally took on investors on MySQL we did know that there was only two ways out; Either we would make an IPO or we would be bought by some large company.

I was always a bit afraid of the IPO route, as in this case there was no guarantees long term that MySQL AB would be a company that would focus on open source. I am quite confident that With Sun, who has made strong commitments to open source with things like Open Solaris, Open Office and making Java open source, our open source inheritance will be taken good care of.

I think Sun is the best possible buyer, because of the following reasons:
(Note that this is of course my interpretation)

• Sun is committed to open source.
• Sun doesn't have an database of their own; In other words, no risk of internal conflicts between similar products.
• Sun understands what it means to be a virtual company where people work from home.
• Sun has a good understanding of developers needs and there is a good chance that the integration of the two companies will be relative smooth.
• Sun has said they will let the MySQL developers continue work as before in their own unit and without big changes (except of course changes for the better!).

Another reason I believe we are good for each other is the different history:

Sun is a hardware company who has been for a long time in a transition to also be a software company. In their software space they where first closed source but has lately started to change most of their software to open source/free software.

MySQL AB on the other hand is a company that was originally totally committed to free software / open source but who has lately changed to be more closed.

This deal will allow both companies to learn from each others successes and failures and build a stronger company than we would have been able to do separately.

A match truly made in heaven (sorry Sun :). Lets hope that we here on Earth will be able to make the best of it!

Regards,
Monty