Progress reporting for ALTER TABLE

I just finished my last MariaDB 5.3 feature before we go beta:

Progress reporting for ALTER TABLE, LOAD DATA INFILE etc.

This is a feature that I think every MySQL user who has ever waited for an ALTER TABLE to finish has wanted for a long time! I know people who have written scripts to monitor the size of the result files to try to estimate how much an ALTER TABLE has progressed.

Progress reporting means that:

• There is a new column Progress in SHOW PROCESSLIST which shows the total progress (0-100 %)
• INFORMATION_SCHEMA.PROCESSLIST has three new columns which allow you to see in which process stage we are and how much of that stage is completed:
• STAGE
• MAX_STAGE
• PROGRESS_DONE (within current stage).
• The client receives out-of-band progress messages which it can display to the user to indicate how long the command will take!

Progress reporting is enabled by default in the new mysql client. When enabled, for every supported command you get a progress report like:

MariaDB [test]> alter table my_mail engine=maria;
Stage: 1 of 2 'copy to tmp table' 5.37% of stage done

MariaDB 5.3 also includes a modified mytop which shows progress for the running commands.

This feature is now pending a review. It will be pushed into MariaDB 5.3 tomorrow.

You can find a full documentation of this feature in the knowledgebase.

Open Oceans new fund has closed

Open Ocean's new Fund celebrated with pitching competition - I'll give the winner Black vodka as much as he/she can carry (inside and outside)!

I am delighted that we now have been able to launch our new Open Ocean Fund. With 40 M euro (60M USD) in first closing, the race is now on to identify new disruptive start ups, that we with Community and Open Source strategy and proper funding, can help drive for similar success as achieved with MySQL.

In my mind a vital part of entrepreneurship is to be able to combine work with fun!

For this reason we decided to launch a pitching competition, where we bring the winning team to Finland, both for good worktime with our team, but also for some social time and nice summer activities.

On top of the boat trip and crayfish party already mentioned in the news item, I add here an additional personal price. During the evening party I will offer our guests with a chance to taste some (or a lot) of one of my personal favorites: The Finnish Black Vodka. And, the next morning, as a gift to take home, I will provide the winner with Black Vodka - as much as he/she can carry - to take home as a souvenir. Something that should help to remember the trip, and something enabling the entrepreneur(s) to share the fun with family, friends and colleagues!

Dynamic columns as a bridge between SQL and NoSQL

Last week Oleksandr "Sanja" Byelkin and I pushed Dynamic columns into MariaDB 5.3.

Dynamic columns allows you to store a different set of columns for every row in the table. This allows you to solve things that you can't normally do with a relational database, like the web store problem where you want to store attributes for different things, like a computer (memory, speed, weight) and t-shirt (size, color).

Dynamic columns works by storing the extra columns in a blob and having a small set of functions to manipulate it. The functions exist both in SQL and in the MariaDB client library to allow you to manipulate the data where it suits you best. The client code is BSD to make it easy to port the code to other languages.

You can use a dynamic column in SQL anywhere you can use a column or function:

Some examples:
SELECT name, COLUMN_GET(blob, 1 as char(10)) FROM t1 WHERE COLUMN_EXISTS(blob, 1);

SELECT COLUMN_GET(blob, 1 as char(10)) as color, count(*) FROM t1 group by COLUMN_GET(blob, 1 as char(10));

COLUMN_GET() is used above to access the dynamic column stored in the table colum 'blob'.

The full set of SQL functions are:

Create data for a dynamic column:

COLUMN_CREATE(column_nr, value [as type], [column_nr, value [as type]]...)

Add a new column or update an old one:

COLUMN_ADD(dynamic_col, column_nr, value [as type], [column_nr, value [as type]]...)

Delete a dynamic column:
COLUMN_DELETE(dynamic_column, column_nr, column_nr...);

Get the value from a dynamic column casted to a given type. (The cast is required as SQL is a strongly typed language and we need to send the result type to the client before we have executed the query)

COLUMN_GET(dynamic_column, column_nr as type)

Check if a column exists:

COLUMN_EXISTS(dynamic_column, column_nr);

Get a list of the existing dynamic columns:

COLUMN_LIST(dynamic_column);

You can find more information in the MariaDB /MySQL knowledge base.

It's very nice to finally get this work done. I got the idea for Dynamic columns 8 years ago and always wanted to do this in MySQL but never managed to find the time to do it.

The recent trend of combining NoSQL with SQL finally pushed me to get this done as this also opens doors to do new existing things with MariaDB. There are people already working on using Dynamic columns as a building block for creating connections to NoSQL databases like HBase.

At the O'Reilly MySQL conference 2011 my talk about Dynamic columns was well attended. A large portion of the attendees said this is something they need and want to start testing / using right away as it can replace some things they are already doing / planning to do. We will have another talk about dynamic columns and combining NoSQL and SQL at Open Source bridge on June 21-24 in Portland, Oregon.

Please try this out and send your feedback to the MariaDB email list or come to Open Source bridge and discuss it with us in person.

Welcome to Percona Live in New York

Percona is organizing another Percona Live event, this time a one-day MySQL summit in New York.

This is a 100% technical conference with no marketing allowed, perfect for those that are only interested in the real stuff or those that want to get answers to problems they have *right now*.

The previous summit, held in San Francisco, was very well attended and I have heard a lot of good things about it from people that were there.

In San Francisco we had one of the MariaDB optimizer gurus holding a talk about all the advanced optimization we have added to MariaDB 5.3.

In NYC we have Kurt von Finck giving a talk about What's New In MariaDB.

Unfortunately I can't be there, even if I would like to attend :(

I was in the USA last month at the MySQL Conference and expo, and I will be in the USA again in June for Open Source Bridge. Then again in July for OSCon.

Even if I like to travel to the USA, once a month is a little too often when you live in Europe. Hope that Percona will host a summit in Europe soon ...

However, don't worry; Kurt will, of course, have with him in NYC some of the black stuff everyone is expecting from a Monty Program Ab employee.

Last not but least, for all the readers of the monty-says blog, you can get a 50$ discount to the Percona Live event by using the MONTYSAYS discount code. One never knows in what kind of places this discount code may work... ;)

Update from the MySQL users conference

I just finished the MariaDB future planning BoF session, which was more about discussing the distinct flavor of the home made black vodka than about future MariaDB features. It was still a very good BoF and there is still time for those interested to add their future request to the plans.

I have never before spent such much time preparing a keynote, and it looks like my attempts to present something a bit different were appreciated. You can find a copy of the keynote and video of it at the MySQL conference web site.

I would, however, like to offer a correction. A couple of days before the conference I had been told by someone (someone that I had all reasons to believe was a reliable source) that there were only 54 of the original 400-450 MySQL Ab people left at Oracle. I asked a lot of former Oracle employees if this figure could be accurate and everyone told me that that the figure sounded low, but they could believe it. I could not find anyone in Oracle willing to comment upon it before the keynote. Now I have finally been able to verify this and there is still closer to 200 original MySQL Ab people left at Oracle working with MySQL.

My statement that it's only the InnoDB, NDB and most part of the replication team that are intact and that there is only 2 original core MySQL developers left is however accurate. My apologies for the wrong initial number of total people left, but I was in good belief regarding it!

To make things clear, I am not in any way trying to downplay the hard work the MySQL developers (and other MySQL people) are doing. They are doing an amazing job, with the resources they have at their disposal.

My point was that I am worried that the MySQL developer and support ecosystem is slowly falling apart because people are leaving Oracle and going to other companies where they are not anymore participating in the MySQL development. That is why companies like Monty Program Ab and SkySQL are important as we help keep the ecosystem together by having people working on the same thing they did before!

Apart from this, it has been a great conference and it's been a true pleasure to meet all of the 'old MySQL conference gang' yet another time!

Planning for MariaDB 5.6 at the MySQL Conference 2011

Planning for MariaDB 5.6 at the MySQL Conference 2011

A couple of weeks ago we had a Monty Program Ab and MariaDB developers meeting in Lisbon.

MariaDB 5.3 / 5.5 are now almost in beta (the last features will be pushed directly after the MySQL conference) and it was time to start considering what features should be in the next MariaDB release.

The meeting was open to anyone and we were lucky to have several other companies than just Monty Program Ab involved in the discussions!

During the meeting we created a list of features we would like to see in the MariaDB 5.6 release.

Some of the features are already allocated to developers. Others are waiting for developers willing to do the development or sponsors willing to pay for getting things done. If you are interested in doing or sponsoring some MariaDB development, check the list out and assign your name to the features!

This list is still open for new additions until 5.3 / 5.5 is in gamma. (In practice probably a few months). If you think something critical for you is missing, feel free add the things you want to see done. If you can commit to do them, then it's of course even better!

The MySQL users conference 2011 is starting next week. I plan to organize a MariaDB BoF where we can together discuss the plans for 5.6 and collect requests and suggestions. Black drinks and chocolate will be served, so come prepared!

Most of the developers of Monty Program Ab will be at the conference and we will also have a booth so you should have no problem in finding someone to talk to about MariaDB there.

Monty Program Ab is also organizing a MySQL/MariaDB/Drizzle store engine summit. Follow the link if you are interested in attending!

Don't miss my keynote on Wednesday morning! There should be a couple of surprises for everyone in it...

Looking forward to another interesting conference!

Come to Lisbon and meet MariaDB developers

The next Monty Program Ab company and MariaDB developer meeting will be in Lisbon, Portugal between Friday 11 March and Monday March 14, 2011.

The topic of the meeting is to go through what has been done in the MariaDB and MySQL code recently and to plan the features for the next main MariaDB release.

The developer meeting is open to anyone who wants to participate. If you want to know about, affect, or participate in future development of MariaDB or if you just want to hang around and have a good time with the MariaDB developers, feel free to come and join us in Lisbon!

We are just now in the process of booking a hotel and will publish the details on our MariaDB Developer Meetings page as soon as it's decided.

What is still not decided is what restaurants to visit, where to have the actual meetings (we are just now talking with some hotels about this, but we are also open to suggestions) and what to do on the Monday "excursion day'. If you know Lisbon and have a suggestion that would work with a group of 20-40 people who enjoy good food, good company, and want to experience 'new (unexpected?) things', please let us know!

You can reach us at 'community at askmonty dot org'.