MariaDB 5.3.3 RC released.

Just in time for Christmas, we have made MariaDB 5.3.3 Release candidate available for download.

Looking at the number of changes and new features, MariaDB 5.3 is probably the biggest milestone release in MySQL history since 5.0:

• Fast sub queries and better optimizer
• Faster and better replication
• NoSQL extensions
• New important types (Microseconds)

And much more.

MariaDB 5.3 is also important as MariaDB 5.5 is based on this. The merge of the code bases are already completed and as soon as our testing is complete (estimated to a couple of weeks) we will release MariaDB 5.5-beta.

A merry holiday and happy new year to you all!

Commercial use of MariaDB & Webinar with SkySQL

A lot of things is happening in the MariaDB world; We just had our developers conference in Athens, we are about to bring out MariaDB 5.3 -RC and MariaDB 5.5-beta. MariaDB 5.5 will include all the closed source extensions that Oracle have announced for MySQL 5.6. It's nice to see that the open source community can continue to produce open source software to rival and exceed closed source offerings.

At the same time, we need commercial goals to support the growth of free software. The development crew in Monty Program Ab, all ex-MySQL developers, are happy to develop features based on commercial agreements. These will of course be released as open source to the benefit of the entire community of users. The support staff of SkySQL is happy to give you commercial 24x7 support of MariaDB, so you have the same level of support that you werea used to for MySQL in the days of MySQL AB: The first person you talk to should know more than you about MySQL!

The fact that we at Monty Program focus on development should by no means deter you from relying on MariaDB in commercial settings. Our work with SkySQL is intended to accelerate adoption for users and customers who need mission critical services.

To learn how you best extend your usage of MariaDB inside a commercial organization, consider taking part in the SkySQL webinar on Thursday 8 Dec 2011 at 8am PT (11am ET, 16:00 UK, 17:00 CET, or 18:00 EET).

Unfortunately I will not be there to share Black Vodka with you, but other familiar voices will present for you

• Colin Charles, the Chief Evangelist at Monty Program, on when and why to use MariaDB in a commercial setting
• Rasmus Johansson, the Monty Program COO, on the offerings of Monty Program Ab and how to interact with us commercially
• Kaj Arnö, EVP Products at SkySQL, on how SkySQL supports MariaDB users technically and commercially

The webinar is a good time to give us feedback on what we need to do to make it easier for you to adopt MariaDB and/or get more of MySQL in critical and high performance environments.

You can register to the seminar here.

MariaDB developer meeting in Athens

It's time for the next MariaDB developer meeting. This time it's in Athens from 11-November to 14'th of November.

The reason for choosing Athens was that the previous MariaDB developer meetings has been in places like Iceland, Spain and Portugal (all countries with some financial issues) and we thought it was no more than right to support Greece next.

As before, the meeting is open for everyone that wants to attend.

The agenda can be found here. This is also the place where all proposed and agreed to plans will be be posted.

We will try to keep the #maria IRC channel on Freenode up to date about what is happening. If you have any questions about a session, you can ask questions on the channel and we will do our best to answer them.

The main topic of the meeting is to finalise the plans for MariaDB 5.6 now when MariaDB 5.5 is close to completion. (MariaDB 5.3 + MySQL 5.5 -> MariaDB 5.5 merge is done and we are now fixing the final issues found by our test systems so that we can do the first alpha binary release of MariaDB 5.5).

If you have any ideas or suggestions for what we should do in 5.6, please update either the plans page or send an email to maria-developers at lists.launchpad.net list.

Oracle adding close source extensions to MySQL

Oracle has now officially announced that MySQL is not a full free software project anymore and that they will go for an Open Core model.

This is inconsistent to the original MySQL business model. The main strength of MySQL has always been that all MySQL source code was freely available. It was thanks to this that MySQL got widely adopted and got to be used by tens of millions of users.

This was a fact that the MySQL board and Sun understood. Closing the MySQL server code base in any manner would be bad for long term business.

It's true that we in MySQL Ab did have some people, notable persons who came from the commercial world and did not understand or believe in free software, that spoke strongly for an open core model where some parts would be open and others closed. Initially they could not do this, mainly thanks to the founders shareholder agreement which guaranteed that the MySQL server would be open source. When Sun bought MySQL, and the shareholder agreement expired, they saw their chance and announced that backup would be a commercial closed source extensions. This initiative was however quickly killed by Sun's management who did understand the true value of MySQL as an open source project and by making it closed source they would have made it less valuable.

One of the reasons MySQL AB did not need an open core model for it's business was that since MySQL is an infrastructure product that is easily embeddable into other products, we could sell licenses to those who wanted to embed MySQL into their product but did not want to make their product open source. (Dual licensing).

So what's so wrong with Open Core?

As a business model, especially to bootstrap a business, Open Core is not a bad idea. It gives you some leverage with your users to get part of your development paid for. This is however done at the cost of fewer users and a less adopted project. Many projects however abandon Open Core models when they grow as open source gives them more users and thus more value.

What is most important to understand about an Open Core project is that it has nothing to do with an open source project. If you are depending on a single closed source component then you have to regard the whole project as a closed source project as you lose all the benefits of open source:

• You are depending on one vendor.
• You can't do any bug fixes yourself and you can't contract anyone except the original vendor to get things fixed.
• You can't examine and improve the product (not any part of if).
• You can't use any open source or commercial extensions from anyone else. This is true also for any plugins that access any internal parts of the product (which is true for most MySQL plugins).
• You are limited to the platforms that the original vendors makes available. (In MySQL's case you can't for example use the new commercial features on RedHat 6 as it's not a supported platform).
• The builds you are using are not tested anymore by millions of users (more bugs).
• The product is not checked by the open source community for security problems or back doors.
• The new features will not be adopted by (and may even cause conflicts with) other open source projects.
• You are subject to one vendor's price politics which they can change at their convenience.
  

What is interesting with the new commercial features is that parts of them where not developed by Oracle. The thread pool was originally developed by Ebay for MySQL 5.0 and contributed to MySQL to be include in MySQL 5.1. Only the new scheduler interface code was added to MySQL 5.1 while the thread pool itself was added (but accidentally with a slower implementation) into MySQL 6.0. Oracle never back ported the thread pool code to the MySQL 5.5 community version and now also the MySQL 6.0 tree is deleted.
The new pluggable authentication, which makes the new PAM authentication possible, was developed and contributed to Oracle by Sergei Golubchik at Monty Program Ab.

The good news is however that the most important feature, the improved thread pool, is easily duplicated in MariaDB. We have had a thread pool in MariaDB since 5.1 (better than was in MySQL 5.1) and we are working on a new one for MariaDB 5.5 which we expect will have similar speed gains as MySQL enterprise. MariaDB don't yet have PAM authentication but the MariaDB developers are prepared to develop that if there is a need for it or if someone contributes the code to us. This should be relatively easy as we have the same pluggable authentication protocol as MySQL.

Read also Stewart Smith's thoughts and the comments to his blog about this subject.

What is happening with the MySQL conference?

What is happening with the MySQL conference?

I'm at the moment perplexed, confused and disappointed in what's going on around the traditional MySQL user conference. Percona yesterday announced Percona MYSQL conference that they will be organizing their own event during the same time and in the same place where this great conference has taken place for years.

The reason for my state of mind is that although there have been rumors about discontinuance of the O'Reilly arranged conference there hasn't been any announcement about this.

In fact, I have been working with O'Reilly to try to setup next year's O'Reilly MySQL conference with the intention of having it 'exactly like before', even if Oracle would not participate. The intention was to have Percona, SkySQL and Monty Program Ab and some other sponsors helping O'Reilly to drive the conference.

This is something that Percona has been very well aware of when announcing this (I personally talked with Percona about this a few weeks ago).

One of the main reasons we at MySQL Ab originally liked to work with O'Reilly is not only that O'Reilly knows how to do a good conference. We also saw the importance to have the conference being organized by a neutral entity to make it easier for competing vendors to be part of the conference. A conference called the "Percona MySQL conference" is not going to be seen as vendor neutral, which will cause problems in getting different vendors to attend. Giuseppe Maxia and Sheeri Cabral seem to think in a similar ways.

Personally I will wait and see what will happen with the O'Reilly MySQL Conference & Expo 2012, or equivalent neutral conference for the MySQL community. The best option, if O'Reilly is not doing it, would of course be a community conference where Oracle/MySQL participates actively, both as a sponsor and as content provider.

MariaDB Crash Course released

I am happy to announce that the first MariaDB book is released!

The book is called MariaDB Crash Course and is written by Ben Forta, who also wrote the MySQL Crash Course book.

Quoting the book description:

"This book will teach you all you need to know to be immediately productive with MySQL. By working through 30 highly focused hands-on lessons, your MySQL Crash Course will be both easier and more effective than you'd have thought possible"

This is great news for new users to SQL and to MariaDB as it makes it easier for them to get things going quickly!

You can find a link to this book and other recommended MariaDB / MySQL books here.

Enjoy!

MariaDB 5.3 released as beta

MariaDB 5.3 is now released as beta (feature freeze).

MariaDB 5.3 radically improves performance for subqueries as well as for joins and single-table queries over large data sets. MariaDB 5.3 is based on MySQL 5.1.

For a detailed description of MariaDB 5.3, see the What is MariaDB 5.3 page.

In most respects MariaDB will work exactly as MySQL: all commands, interfaces, libraries and APIs that exist in MySQL also exist in MariaDB.

A few highlights:

One of the major performance improvements in MariaDB 5.3.0 is that subqueries are finally usable in practice. It is no longer necessary to rewrite subqueries manually into joins or separate queries. The optimizer of MariaDB 5.3 will do this, and more automatically. EXPLAIN for derived tables and materialized views is instantaneous. Both derived tables and views can be optimized by indexes created on the fly.

MariaDB now supports many join optimizations, including utilizing the Block Nested Loop algorithm that can be used for outer joins, Classic Hash Join which can be used for equi-joins, and Batch Key Access joins are supported providing the benefits of ordered retrievals for primary and secondary keys provided by the new implementation of Multi-Range-Read (MRR) optimizations. MariaDB also features a new version of Index Condition Pushdown.

NoSQL is all the rage these days and MariaDB 5.3.0 ships with the HandlerSocket plugin and improved HANDLER commands. Dynamic columns allows you to have a different set of "virtual columns" for each row in your table. You can at any time add or remove columns from a row.

MariaDB 5.3 implements group commit which gives you a big speedup when using XtraDB with the binary log enabled and you have many concurrent running updates.

Microsoft Windows performance improvements from MySQL 5.5 have been backported, including benefits to the XtraDB storage engine. MariaDB 5.3 also uses the new Windows installer and includes the HediSQL GUI client for Windows.

There is also a lot of other "goodies", like progress reporting, checksum for binary logs, mytop, microsecond support in DATETIME, TIMESTAMP and TIME, more status variables etc. See What is MariaDB 5.3 for details.

When it comes to MariaDB 5.5 we have just finished the first big merge between MariaDB 5.3 and MySQL 5.5 and we are now working on merging the recent changes. This should be released shortly (you can follow the work on launchpad). We will continue to fix bugs in both version for some time! The reason we are releasing 5.3 and 5.5 is that we don't want to delay users the features in 5.3 while we work to merge and stabilize 5.5. (We don't expect any problems with 5.5 but as one never knows we want to play safe.).

Please test MariaDB 5.3 and report bugs to launchpad! The more input we can ge the quicker we can do stable (GA) release.

If you want to know more, visit the MariaDB booth at OSCON or come to the MariaDB BOF on Wednesday at 8.00 PM!

Progress reporting for ALTER TABLE

I just finished my last MariaDB 5.3 feature before we go beta:

Progress reporting for ALTER TABLE, LOAD DATA INFILE etc.

This is a feature that I think every MySQL user who has ever waited for an ALTER TABLE to finish has wanted for a long time! I know people who have written scripts to monitor the size of the result files to try to estimate how much an ALTER TABLE has progressed.

Progress reporting means that:

• There is a new column Progress in SHOW PROCESSLIST which shows the total progress (0-100 %)
• INFORMATION_SCHEMA.PROCESSLIST has three new columns which allow you to see in which process stage we are and how much of that stage is completed:
• STAGE
• MAX_STAGE
• PROGRESS_DONE (within current stage).
• The client receives out-of-band progress messages which it can display to the user to indicate how long the command will take!

Progress reporting is enabled by default in the new mysql client. When enabled, for every supported command you get a progress report like:

MariaDB [test]> alter table my_mail engine=maria;
Stage: 1 of 2 'copy to tmp table' 5.37% of stage done

MariaDB 5.3 also includes a modified mytop which shows progress for the running commands.

This feature is now pending a review. It will be pushed into MariaDB 5.3 tomorrow.

You can find a full documentation of this feature in the knowledgebase.

Open Oceans new fund has closed

Open Ocean's new Fund celebrated with pitching competition - I'll give the winner Black vodka as much as he/she can carry (inside and outside)!

I am delighted that we now have been able to launch our new Open Ocean Fund. With 40 M euro (60M USD) in first closing, the race is now on to identify new disruptive start ups, that we with Community and Open Source strategy and proper funding, can help drive for similar success as achieved with MySQL.

In my mind a vital part of entrepreneurship is to be able to combine work with fun!

For this reason we decided to launch a pitching competition, where we bring the winning team to Finland, both for good worktime with our team, but also for some social time and nice summer activities.

On top of the boat trip and crayfish party already mentioned in the news item, I add here an additional personal price. During the evening party I will offer our guests with a chance to taste some (or a lot) of one of my personal favorites: The Finnish Black Vodka. And, the next morning, as a gift to take home, I will provide the winner with Black Vodka - as much as he/she can carry - to take home as a souvenir. Something that should help to remember the trip, and something enabling the entrepreneur(s) to share the fun with family, friends and colleagues!

Dynamic columns as a bridge between SQL and NoSQL

Last week Oleksandr "Sanja" Byelkin and I pushed Dynamic columns into MariaDB 5.3.

Dynamic columns allows you to store a different set of columns for every row in the table. This allows you to solve things that you can't normally do with a relational database, like the web store problem where you want to store attributes for different things, like a computer (memory, speed, weight) and t-shirt (size, color).

Dynamic columns works by storing the extra columns in a blob and having a small set of functions to manipulate it. The functions exist both in SQL and in the MariaDB client library to allow you to manipulate the data where it suits you best. The client code is BSD to make it easy to port the code to other languages.

You can use a dynamic column in SQL anywhere you can use a column or function:

Some examples:
SELECT name, COLUMN_GET(blob, 1 as char(10)) FROM t1 WHERE COLUMN_EXISTS(blob, 1);

SELECT COLUMN_GET(blob, 1 as char(10)) as color, count(*) FROM t1 group by COLUMN_GET(blob, 1 as char(10));

COLUMN_GET() is used above to access the dynamic column stored in the table colum 'blob'.

The full set of SQL functions are:

Create data for a dynamic column:

COLUMN_CREATE(column_nr, value [as type], [column_nr, value [as type]]...)

Add a new column or update an old one:

COLUMN_ADD(dynamic_col, column_nr, value [as type], [column_nr, value [as type]]...)

Delete a dynamic column:
COLUMN_DELETE(dynamic_column, column_nr, column_nr...);

Get the value from a dynamic column casted to a given type. (The cast is required as SQL is a strongly typed language and we need to send the result type to the client before we have executed the query)

COLUMN_GET(dynamic_column, column_nr as type)

Check if a column exists:

COLUMN_EXISTS(dynamic_column, column_nr);

Get a list of the existing dynamic columns:

COLUMN_LIST(dynamic_column);

You can find more information in the MariaDB /MySQL knowledge base.

It's very nice to finally get this work done. I got the idea for Dynamic columns 8 years ago and always wanted to do this in MySQL but never managed to find the time to do it.

The recent trend of combining NoSQL with SQL finally pushed me to get this done as this also opens doors to do new existing things with MariaDB. There are people already working on using Dynamic columns as a building block for creating connections to NoSQL databases like HBase.

At the O'Reilly MySQL conference 2011 my talk about Dynamic columns was well attended. A large portion of the attendees said this is something they need and want to start testing / using right away as it can replace some things they are already doing / planning to do. We will have another talk about dynamic columns and combining NoSQL and SQL at Open Source bridge on June 21-24 in Portland, Oregon.

Please try this out and send your feedback to the MariaDB email list or come to Open Source bridge and discuss it with us in person.

Welcome to Percona Live in New York

Percona is organizing another Percona Live event, this time a one-day MySQL summit in New York.

This is a 100% technical conference with no marketing allowed, perfect for those that are only interested in the real stuff or those that want to get answers to problems they have *right now*.

The previous summit, held in San Francisco, was very well attended and I have heard a lot of good things about it from people that were there.

In San Francisco we had one of the MariaDB optimizer gurus holding a talk about all the advanced optimization we have added to MariaDB 5.3.

In NYC we have Kurt von Finck giving a talk about What's New In MariaDB.

Unfortunately I can't be there, even if I would like to attend :(

I was in the USA last month at the MySQL Conference and expo, and I will be in the USA again in June for Open Source Bridge. Then again in July for OSCon.

Even if I like to travel to the USA, once a month is a little too often when you live in Europe. Hope that Percona will host a summit in Europe soon ...

However, don't worry; Kurt will, of course, have with him in NYC some of the black stuff everyone is expecting from a Monty Program Ab employee.

Last not but least, for all the readers of the monty-says blog, you can get a 50$ discount to the Percona Live event by using the MONTYSAYS discount code. One never knows in what kind of places this discount code may work... ;)

Update from the MySQL users conference

I just finished the MariaDB future planning BoF session, which was more about discussing the distinct flavor of the home made black vodka than about future MariaDB features. It was still a very good BoF and there is still time for those interested to add their future request to the plans.

I have never before spent such much time preparing a keynote, and it looks like my attempts to present something a bit different were appreciated. You can find a copy of the keynote and video of it at the MySQL conference web site.

I would, however, like to offer a correction. A couple of days before the conference I had been told by someone (someone that I had all reasons to believe was a reliable source) that there were only 54 of the original 400-450 MySQL Ab people left at Oracle. I asked a lot of former Oracle employees if this figure could be accurate and everyone told me that that the figure sounded low, but they could believe it. I could not find anyone in Oracle willing to comment upon it before the keynote. Now I have finally been able to verify this and there is still closer to 200 original MySQL Ab people left at Oracle working with MySQL.

My statement that it's only the InnoDB, NDB and most part of the replication team that are intact and that there is only 2 original core MySQL developers left is however accurate. My apologies for the wrong initial number of total people left, but I was in good belief regarding it!

To make things clear, I am not in any way trying to downplay the hard work the MySQL developers (and other MySQL people) are doing. They are doing an amazing job, with the resources they have at their disposal.

My point was that I am worried that the MySQL developer and support ecosystem is slowly falling apart because people are leaving Oracle and going to other companies where they are not anymore participating in the MySQL development. That is why companies like Monty Program Ab and SkySQL are important as we help keep the ecosystem together by having people working on the same thing they did before!

Apart from this, it has been a great conference and it's been a true pleasure to meet all of the 'old MySQL conference gang' yet another time!

Planning for MariaDB 5.6 at the MySQL Conference 2011

Planning for MariaDB 5.6 at the MySQL Conference 2011

A couple of weeks ago we had a Monty Program Ab and MariaDB developers meeting in Lisbon.

MariaDB 5.3 / 5.5 are now almost in beta (the last features will be pushed directly after the MySQL conference) and it was time to start considering what features should be in the next MariaDB release.

The meeting was open to anyone and we were lucky to have several other companies than just Monty Program Ab involved in the discussions!

During the meeting we created a list of features we would like to see in the MariaDB 5.6 release.

Some of the features are already allocated to developers. Others are waiting for developers willing to do the development or sponsors willing to pay for getting things done. If you are interested in doing or sponsoring some MariaDB development, check the list out and assign your name to the features!

This list is still open for new additions until 5.3 / 5.5 is in gamma. (In practice probably a few months). If you think something critical for you is missing, feel free add the things you want to see done. If you can commit to do them, then it's of course even better!

The MySQL users conference 2011 is starting next week. I plan to organize a MariaDB BoF where we can together discuss the plans for 5.6 and collect requests and suggestions. Black drinks and chocolate will be served, so come prepared!

Most of the developers of Monty Program Ab will be at the conference and we will also have a booth so you should have no problem in finding someone to talk to about MariaDB there.

Monty Program Ab is also organizing a MySQL/MariaDB/Drizzle store engine summit. Follow the link if you are interested in attending!

Don't miss my keynote on Wednesday morning! There should be a couple of surprises for everyone in it...

Looking forward to another interesting conference!

Come to Lisbon and meet MariaDB developers

The next Monty Program Ab company and MariaDB developer meeting will be in Lisbon, Portugal between Friday 11 March and Monday March 14, 2011.

The topic of the meeting is to go through what has been done in the MariaDB and MySQL code recently and to plan the features for the next main MariaDB release.

The developer meeting is open to anyone who wants to participate. If you want to know about, affect, or participate in future development of MariaDB or if you just want to hang around and have a good time with the MariaDB developers, feel free to come and join us in Lisbon!

We are just now in the process of booking a hotel and will publish the details on our MariaDB Developer Meetings page as soon as it's decided.

What is still not decided is what restaurants to visit, where to have the actual meetings (we are just now talking with some hotels about this, but we are also open to suggestions) and what to do on the Monday "excursion day'. If you know Lisbon and have a suggestion that would work with a group of 20-40 people who enjoy good food, good company, and want to experience 'new (unexpected?) things', please let us know!

You can reach us at 'community at askmonty dot org'.

SQL 99 Complete, Really now freely available

I am very happy that Daniel Bartholomew recently finished his task of getting the excellent book SQL 99 Complete, Really into the askmonty.org Knowledge base (KB). Even if the book is a bit old, it's still one of the best (maybe even the best?) reference books about SQL.

A big thanks to my friends Peter Gulutzan and Trudy Pelzer for making this possible!

This is however just the beginning. Now we will start cross referencing the KB and the book and also moving appropriate sections from the book to the KB to create better KB articles. (We also need to add some links to make it easier to read trough the book online).

One of the goals with the KB is that over time it will become a full reference and user manual, not only about MariaDB and MySQL, but also about other databases, database systems and database languages.

If you own the rights to some database documentation that you think would benefit the users of the KB, please contact us about getting it included. We are especially interested in MariaDB and MySQL related books and are even willing purchase the rights for these to get them published in the KB.

One of the great benefits of the KB is that if you can't find an answer, you can ask a question at precisely the point where you think the answer should be and get an answer there from the people working on or using the KB. This will allow others to find the answer without having to ask the same question again. The KB is thus not only a manual, but also a replacement for forums.

When it comes to MariaDB in the KB, I am quite happy about the progress. The documentation is growing rapidly and we have been able to quickly create new articles to answer most questions that have been asked in the KB, in the #maria IRC channel or on the Launchpad MariaDB mailing list.

If you want to participate in creating free database documentation and/or help others by answering database questions please join us in working on the KB.