Author Archives: Mike Rooney

Automating translation template generation and check-ins for Launchpad

I previously wrote about how excited I was for the automated translation import and export features of Launchpad. Launchpad will automatically notice when you commit a translation template and import it, making it available for translation online. Generous translators will then contribute translations, and Launchpad will commit them back to your project.


Okay, so this is pretty good! However for this process to work well, the translation template needs to be generated manually by a developer whenever there are changes to strings. Otherwise, translators are working on potentially outdated strings; some are perhaps not in the application anymore, and there are likely some strings which aren’t in the template yet.

After forgetting to generate and commit my template until shortly before a release more than once (and thus having poor translation coverage on newly added strings), I decided to automate this part of the process as well. All it took was a relatively small script to generate the template, and then if there were any changes, commit and push to the branch configured for automatic import in Launchpad. The following script does just that, by searching for any files using gettext calls starting with _(” or _(‘, and passing them to xgettext.


#!/bin/bash
set -e
ack-grep "_\([\"\']" -l | xargs xgettext --output=wxbanker/po/wxbanker.pot
ACTUALCHANGES="`bzr diff | grep \"^[\-\+]msg\" | wc -l`"
if [[ "$ACTUALCHANGES" != "0" ]]; then
bzr ci -m "automated generation of translation template"
bzr push :parent
fi


Note that if you aren’t using Python, you may need to tweak the regular expression supplied to ack-grep. Once the template is generated, the diff is piped through grep to grab any changes to actual messages and make sure there was at least one. Otherwise there would always be changes due to the timestamp in the template, causing useless commits.

I then threw this script into a Hudson job, the Continuous Integration server I use for wxBanker. I configured this particular job to run nightly, pulling down the latest bzr branch beforehand, and emailing me on any failures.

It seems to be working quite well and ensures translators are always translating the latest strings and leaves nothing for me to forget, smoothing out the release process.

If this sounds interesting to you but you’re not familiar with Launchpad as a translation system, check out http://blog.launchpad.net/general/trying-out-launchpad-translations and feel free to ask any questions here. If you do have experience with translations, how do you handle generating translation templates and then integrating the translations?

wxBanker 0.6 preview available, now with recurring transactions!

I’ve just released wxBanker 0.5.9 in the wxbanker-testing PPA, which is a preview release for 0.6. If you aren’t familiar with wxBanker already, it is a cross-platform, lightweight personal finance application, and you can find more info at https://launchpad.net/wxbanker.

While there are many improvements and fixes in 0.6, the main feature is recurring transactions, allowing you to automate repetitive transactions. They are functional in 0.5.9 with one caveat and the purpose of this preview release: there’s no way (besides sqlitebrowser :) to modify existing recurring transactions. As such I’d love to get feedback on your impressions of recurring transactions, ideas on a nice configuration UI for them, and of course general feedback. With this I can implement the configuration UI and release 0.6.

Here’s an example of a simple quarterly transaction:



and here is a more complex one, with specific days chosen, as well as the transaction being a transfer from another account:



When you start up wxBanker and there is a recurring transaction due, you will see something like:



I didn’t reinvent recurrence rules but instead (luckily) found the dateutil module for python which includes rrule, an “implementation of the recurrence rules documented in the iCalendar RFC”, so it should behave quite well.

As far as the needed configuration UI goes, there are a couple ideas that I’ve had so far. Below is the current top-left portion of the UI, for reference:



One idea was to have a third “Recurring Transactions” tab after the “Transactions” and “Summary” tabs, which appears if and only if recurring transactions exist. This would provide a list of recurring transactions and allow editing them via the same UI used for creation, as well as changing the account or removing them altogether. A second idea is to add another button next to the Add/Rename/Remove account buttons in the upper-left, for account configuration (this will be necessary for future features anyway), and allow modifying recurring transactions for that account there.

Another perhaps complementary way would be to provide a right-click action on existing transactions which were recurred, for editing. I’d also like to implement functionality similar to Google Calendar where editing a value on an existing transaction caused by a recurring transaction will ask you if you want to apply that change to just this transaction, all existing, or all future.

If you have ideas, please feel free to leave comments here or drop by #wxbanker on irc.freenode.net any time this week after 10AM PDT, to have a more interactive chat about them.

Please do let me know what you think from either just the screenshots here or actually playing around with the application!

Launchpad is now an automatic, magical translation factory!

I’ve been using Launchpad to host my personal finance application wxBanker for a few years now. The thing I was hearing most often was that it wasn’t localized; people wanted currencies to look the way they should in their country, and the application to be in their language. Let me explain how Launchpad helped me provide translations for my application, and how much of an utterly slothful breeze it has recently become.

Image courtesy of shirt.woot.com
Normally to handle translations, an application has to wrap strings with gettext, create a template, find translators and give the template to them, collect translation files back, and integrate them into the project. This is painful and is why many applications aren’t localized, and shut out most of the world as a result. One of the amazing features of Launchpad however, happens to be Rosetta, which brings translators TO your translations via a simple web interface, and then makes those translations available to the developer. With Rosetta, translators don’t need to understand gettext or translation file formats; they just need to know two languages!



So that’s what a translator sees. Notice how Launchpad even shows how other applications translated the same string. So once you generate a template and upload it, you can assign a translation group to your project such as “Ubuntu Translators” so that your strings will be translated by volunteers on the Ubuntu project; if your project isn’t relevant to the Ubuntu project, you can use the more generic Launchpad Translators group. Now all you have to do is wait for some translations, then download them and check it in to your code. Not too bad, right?

It isn’t, but Launchpad has recently made it so much better. They started by adding an option to automatically import translation templates from your project. This means as you are developing, all you have to do is regenerate the template and commit, and new strings will show up for translators in Rosetta and be translated automatically (from the developer’s perspective). Then today, they announced the other side of this, which is automatically committing those translations back into your code on a daily basis. This means that all I have to do is commit templates as I change strings, and Launchpad handles everything else. This is a profound tool for developers.

What’s the next step? Well, from a developer’s perspective the translation template is a tool to give to the translators or in this case Launchpad. In the future Launchpad could eliminate this by generating the template itself from the code (this is what developers are doing themselves, after all), so that truly all you have to do after you set up the initial i18n/l10n framework is commit code as normal, and Launchpad magically commits back translations.

All this work Launchpad is doing gives developers more time to develop while still having localized applications at a very minimal cost. This is continuous translation integration, and boy is it cool!

Comments

Sneca
Hi Michael.

Rosetta has gone a long way, and it is a much appreciated tool. However, as a professional translator that sometimes collaborates on free software translations, I must say it is *not yet* a perfect tool. It lacks better concordance support with easier searches, for instance.
I would like to point out that for translating, one needs to know a bit more than just two languages. It is necessary to have a whole lotta of things in mind, and use your brainz in a very complex way :) Whether I do not want to discourage any contributors, l18n is serious business; terminology consistency, some linguistic common sense (and even style) should be stressed. In the business world, bad translations leave bad impressions on your clients. In GNUlinux realmz, where there are quite a few different desktop managers, so many programs and sometimes different names for similar things, bad translations may end up confusing people (esp. newcomers) and eventually turning them, at least, nervous. English prevails largely on the GNULinux world, and the more languages AND the better quality of translations, the closer we will get to squash bug nº1.

We will keep on working on it. Cheers!
Michael
Thanks for the comment Jeroen! I mentioned Ubuntu Translators because my package is in Ubuntu and it seemed most relevant to readers of my blog on the Ubuntu planet, but mentioning LP Translators for generic use is good to know as well.

The problems with automatic translation are definitely true, and it will be interesting to see if any of them can be mitigated. Perhaps allowing someone to import all translations with known good copyrights for untranslated strings, marking them as "Needs Review".
Jeroen Vermeulen
Hi, I'm one of the Launchpad Translations developers. Glad you like the new stuff we built!

A very minor point: if you want to invite translations for your project, we now have a Launchpad translation group as well as the Ubuntu translation group. The Ubuntu group is to focus more on Ubuntu (surprise), whereas the Launchpad group can serve as the "generical" translation group for other projects.

And a comment on the "automatically use translations from elsewhere" idea: it's very tempting, but there are a few obstacles.

First, there may be a copyright risk from automatically copying other projects' translations. And once we automatically start avoiding some translations, there's a bigger risk of choosing bad translations.

Second, not every translation you find will be correct, let alone consistent. Sometimes people import entire translations to the wrong language!

So ideally there should still be a human review step for those suggestions before they definitely go into a project. And that's exactly how it works now. For some programs, the "global suggestions" from other projects let you do a large chunk of the work very quickly, even in languages you only just well enough to see which translations make sense.
Michael
Tom, I'm not a Launchpad developer, but I have talk of it before I think.

I checked out transifex and it seems pretty interesting. I know LP has done work in the past to make Rosetta easier to use for upstreams and sync the two, and I am sure it will only get easier and better.
Tom
I love the "Automatically use suggestions for otherwise untranslated strings" option, will it come? Launchpad should get a big language model like Google Wave that would certainly help. Maybe Mark should hire some cheap Google engineers ;)

It would be cool if Rosetta could work directly with upstream as http://transifex.org/ can. That also would be a cool addition for people who prefer git etc.
Michael
Thanks simpsus :) The tagging branch is waiting on some work from the author, as well as some conceptual work I need to do (how can tagging be hidden/out of the way for users who don't need that, but easy to discover and turn on?) I'd like to get 0.6 by the end of August with recurring transactions and perhaps reporting (https://launchpad.net/wxbanker/+milestone/0.6), and target tagging/categories for 0.7 not much later.
Michael
Tom, that could be cool and I wouldn't mind an "Automatically use suggestions for otherwise untranslated strings" checkbox, but there are going to be a bunch of previously untranslated strings that will still need translation by a person. And for short phrases there can be multiple translations. For example, the word "read" in English can be both a verb and an adjective, and "import" can be a verb and noun. In another language there might be two different words for each use though, and only a person with context (provided by translation comments) can know which one to use. Certainly though, it seems like it could help you grab some low-hanging translation fruit that's better than nothing.

As far as your second question, I'm not sure what you mean by "hosted". If the project isn't registered on Launchpad at all, it can't use Launchpad features obviously. But there is nothing stopping a project hosted anywhere from creating a Launchpad account and using it for Rosetta. If the code is hosted elsewhere (and Launchpad allows you to specify external SCMs and bug trackers, as well as import from them) you could still have a branch in Launchpad solely for Launchpad to import and export from, and handle merges back and forth on your end yourself.
simpsus_science
Hi Michael,
nice to see you give wxBanker some publicity. Are there any news when the transactions tagging and advanced categories will be merged? When is the 0.6 release planned?

Cheers

Simpsus
Tom
Couldn't it automatically translate without users? Just take the most used strings from other apps?

And how about projects that are not hosted on LP?

Simple timing of Python code

Often when I am writing in Python, I find that I want to see how long a particular function call or set of statements are taking to execute. Let’s say I have the following code that gets executed frequently:

for i in range(10000000):
x = 934.12 ** 32.61 * i / 453.12 ** 0.23

and I want to know how long it takes to execute to see if it is slowing down my app and should be optimized. Previously I would surround it as such:

import time; x = time.time()
for i in range(10000000):
x = 934.12 ** 32.61 * i / 453.12 ** 0.23
print time.time() - x

This will print out the duration in seconds of that code segment, but is more work and typing than I want, and more cleaning up later. I realized that the new “with” statement in Python could probably help me out. Let’s create a little timer class that cooperates with it:

class Timer():
def __enter__(self): self.start = time.time()
def __exit__(self, *args): print time.time() - self.start

Now all we have to do is:

with Timer():
for i in range(1000000):
x = 934.12 ** 32.61 * i / 453.12 ** 0.23

You can also try:

with Timer():
time.sleep(1.5)


For these, 0.28738 and 1.50169 are what I get, respectively. While something like this couldn’t really replace application-wide profiling via a module like cProfile, it can be an extremely useful and quick way to see if your prototype is scalable or not. I usually end up having a debug.py or helpers.py file in my larger projects with little tools like this, and I’ll probably end up adding this one as well.

Let me know if you are doing something similar, or if I’ve reinvented something that already exists. I’d also love to hear from people profiling their python code and what techniques they are using, as I am just starting to learn about it.

Comments

flimm
Got here after reading wxbanker's source code. You're right, it does look handy.
mgedmin
Using the with statement for this is a brilliant idea!

Check out profilehooks on PyPI for something like this at function level, implemented using decorators. I wonder how difficult it would be to have the same object (or function) act both as a decorator and as a context manager.

Timeit is a great module, but it serves a different purpose: very accurate microbenchmarks. Sometimes you just need to know whether a for loop takes 1 or 20 seconds; repeating that for loop 1000 times to get its accurate speed up to the nearest microsecond is overkill. Plus, timeit's API is cumbersome and inconvenient, or at least I can never remember offhand how to use it, which is the same thing in my book.
Paddy3118
… The timeit module documentation states that it stops you repeating some common errors when doing this kind of task.
Michael
Thanks liw.fi for pointing that out, I wasn't aware of it! However while convenient, I don't think it is as trivially injectable into existing code paths. It looks great for testing a function or one-liner, but not for a random segment of code that might be a part of a large function / sequence.
liw.fi
See also the timeit module in the standard library.

Karmic Desktop UDS run-down!

I just got back from a wonderful UDS in beautiful Barcelona and thought I would provide a summary of what we can expect in the Karmic Koala 9.10 desktop. Keep in mind that I don’t speak for Canonical and what follows is just my understanding of what is on the table for Karmic.

Overall it is gearing up to be a pretty radical and exciting release; there are some changes to the default application set as well as some major version upgrades of existing core components. We are trying to be fairly aggressive in terms of new stuff so that if Canonical wants Karmic+1 to be an LTS (Long Term Support) release, we can have fairly stabilized new technologies by then (thanks to 6 months of stabilization in the Karmic cycle) instead of having to wait until after the LTS (Karmic+2) to introduce them. Since many of these changes would be too radical to first appear in an LTS, if we don’t upgrade now we may not be able to for a year, and have to maintain old versions for 3-5 years in the LTS.

On the messaging front, Ekiga will be dropped from the CD to save 10 megs, and Empathy will likely replace Pidgin due to a responsive upstream, voice/video support, and better GNOME integration. It also now has the ability to import accounts from Pidgin, so this should help out with migration. I checked it out a bit at UDS and was impressed with how useful it is with absolutely zero configuration. It will pull your name from the system and enable avahi (auto-discovery of people nearby, like bonjour) with no set up, which made it quite easy to get in contact with people at the conference. You can also supply your email and Jabber ID to the avahi interface to allow other people to discover that info as well.

It also sounds likely that Banshee will replace Rhythmbox as the default media player, and it is the official default of UNR (Ubuntu Netbook Remix) Karmic. This will bring a snazzier interface, better device support including iPods and Androids, and quite importantly an active and responsive upstream. I will admit to not being a huge fan of this transition for Karmic as it seemed too early for me (the lack of a folder watch is quite a regression for me, and it has been promised for the last 3 releases or so, so I’m not holding my breath), but after checking out 1.5 for a bit I will admit that it is growing on me. The user interface does seem nicer, and the lightweight video library which keeps track of what you haven’t watched is nice. However, it does seem to use 3-10x more memory than RB which is very troubling (60-300MB compared to RBs fairly consistent 25MB), especially on the netbook scene. I’ve also had issues with it skipping occasionally, which is very unfortunate. Hopefully the UNR switch will put pressure on better memory management for Banshee.

Banshee syncing with an Android G1

Empathy and Banshee will probably replace their predecessors around alpha 2 of Karmic, and will be either left as default or reverted based on reported regressions and bugs. Keep in mind that if you end up not preferring these applications, the other ones still exist and you can continue to use them.

There are also going to be a bunch of underlying speed improvements, with the boot speed goal being 10-12 seconds. When Ubuntu talks about boot times, we are referring to the time from when grub starts (when Linux first gets control of the machine) to when the user is at a fully loaded desktop with no I/O. The main test machines being used by Canonical here are Dell Mini 9s, with auto-login enabled to get a consistent log-in time. This is pretty impressive as the boot goal was 25 seconds in Jaunty, which was met, and was aggressive itself as Intrepid booted in about 65 seconds on the Dell Mini 9.

grub2 is likely to be default for new installations (upgrades will have grub1 chainloaded to grub2), with ext4 as the default filesystem. The boot process will also be streamlined, with the grub timeout set to 0 and the boot menu hidden. There will instead be two new ways to boot into a different system now. First, there will be a key that can be pressed while booting to bring up an OS chooser, which will halt the current boot and restart into the chosen one. Another goal is to have the restart menu item in GNOME aware of installed OSes and allow the choice there, so you could select for example “Restart into OSX”. All in all this means no racing to select the OS for dual-booters, and a faster boot process as well. /tmp is also hopefully going to be made a tmpfs, which means it will reside in RAM and overflow to a swapfile (which in recent Linuxes have on par performance with swap partitions). This means power savings, less disk I/O (especially great for SSDs), and of course blazingly fast performance which should help out a lot especially when, say, loading files from inside an archive The Gnome Display Manager (GDM, which handles the login screen) will also likely be upgraded to GDM2.

Finally let me fire off a few more changes. Power management is being improved all around, with one change already landed being that audio cards will be automatically powered down after 10 seconds of no sound. Encrypted Home directories will hopefully be easier to set up now with an option right in the graphical installer, and I’m working on a UI for managing this and encrypted Private directories in Karmic, more on that later. Firefox 3.5 should be the default version of Firefox. For notifications which want to display actions if the user is interested, there is work on morphing windows:


Ubuntu is also working on being social from the start (see desktop-karmic-social-from-the-start on gobby.ubuntu.com), perhaps installing Gwibber by default and asking the user if they want to integrate social sites (twitter, facebook) into the desktop when they visit them in Firefox, via an extension. There has also been work in looking for a better scanning application to replace xsane (perhaps GnomeScan), some look into using Gnome Control Center, and a common printing dialog.

Okay phew, that’s what I’ve got to report! Let me know what you think of these decisions and changes, and if there is anything you were hoping for that didn’t make it, or really anything else you’ve got to say!

Comments

News Blog
Nice Post
Steven Spurrier
Michael
Regarding the last anonymous post, I'm not even sure if you read my post. KMS support IS planned for the next release, and I specifically mentioned it. Delta updates were also discussed at the summit as well.

Since Ubuntu and Fedora have different release schedules, they are going to have differences as different technologies and features mature at different times. Both distributions have had certain features before the other and that is surely the way it will continue.
Anonymous
what a boring release it will be…
app1 changed to app2…
ubuntu is really getting years behind fedora:
- no multiseat (MDM) support
- no delta updates
- no smooth loading (KMS) support
and it's not even planned… dispite the fact that some of these features were requested for years!
Anonymous
"Ekiga will be dropped from the CD to save 10 megs"

why not save lots of megs by leaving out mono/banshee/tomboy?

there's gnote to replace tomboy, and banshee doesnt appear to be fully functional..
Anonymous
Banshee? Mono fail!

Empathy has no UI for dialing out on voice calls, doesn't have metacontacts, and its SIP and IRC support are underdeveloped. I've been tracking development and testing it for years. It's still crap for usability, though I understand it's an abstraction-obsessed programmer's wet dream.
The Mad Hatter
We don't use VLC for legal reasons. The same reasons apply to Mono applications. Mono needs to be purged from Ubuntu, along with the applications that use it (FSpot, Tomboy, Banshee). This will also save a considerable amount of disk space.

I would also suggest changing out Evolution for Thunderbird. The amount of integration between Evolution and the Gnome Desktop environment reminds me to much of the way Internet Exploder is integrated into Windows, and the problems caused by that.

As a final note, I think that we should consider not using any software which isn't GPL/BSD/MIT/Apache/Mozilla licensed. License proliferation is becoming a serious issue, which can lock code from reuse. The above licenses are (at least in my opinion) the best ones from a point of sharing.tr
Anonymous
Agreed with the above poster not thrilled with Banshee on two counts: 1) it is not nearly as lean as RB, which is quite a snazzy app in of itself, and 2) it is encumbered by .NET and Mono infested. I think it would be good for the distro’s to heed R. Stallman’s advice and stick to GNU applications, and if closed source must be included make sure its from a originator without a history of exploits of such.
Blimundus
Thanks for the run-down! These are very interesting developments to hear about.

I am particularly interested in the debate on the scanning application.

Etienne’s work on GnomeScan started off with the right design choices, and with the goal to make scanning as ubiquitous as printing. In other words, as I understand it, GnomeScan is all about integration with other programs, the file chooser etc… I think it would be great to see this becoming a reality.

GnomeScan would benefit from Gnome/Ubuntu giving it some attention though.
Bayger
Michael: I have added a blueprint for the idea suggested by me. It’s called desktop-karmic-color-themes. I don’t know what to do next with this. This is my first blueprint. I don’t even know if this is the usual place to post such ideas.
Jeffrey Seguerra
I read lot of rants about mono… was mono discussed in Karmic UDS?

http://tinyurl.com/pycr9u
Jef Spaleta
You had sessions at UDS about gnome 3.0…but is Canonical talking directly with upstream about integrating Canonical built technologies like libindicate?

UDS is a walled garden. Don’t misintepret talking “amongst yourselves” as a substitute for discussion with upstream.

Packagekit adoption would be an example. packagekit-gnome is a gnome module now. It’s moving along a path towards deep integration into the GNOME desktop. How much work has Canonical invested release after release into refining its distro-specific update manager experience? How much of that manpower expenditure work is directly working against adoption of PackageKit as a common cross-desktop solution?

PackageKit adoption has originally brought up for intrepid UDS. I’m not holding my breath for Karmic. Especially in light of the AppCenter proposal which is an even more aggressive divergence path than the current gnome-app-install tool.
https://wiki.ubuntu.com/AppCenter
Michael
Alexander, I think there was a PackageKit session, I’d check summit.ubuntu.com and look at the schedules. If there is one it should be linked to a blueprint / wiki page.

Jef, what you are pointing out is definitely true. If we diverge too much and upstream stops maintaining what we are patching, we are either left maintaining it ourselves or porting it, either of which is extra manpower as you say and is suboptimal. We are looking long term and there were some sessions at UDS on GNOME 3.0, Shell, Zeitgeist, Control Center and such. The plan is to have as many GNOME 3.0 components as possible easily installable in Karmic, and solicit some early feedback on user experience, regressions, and integration with other Ubuntu aspects.

There is definitely some exciting work going on to make it really easy to use the latest upstream versions of packages in future Ubuntus, which should really help upstreams get better feedback from us and make development upstream easier as well. Plus it should eliminate some surprises by pointing out integration issues early.
Jef Spaleta
Do you know if upstream gdm has incorporated the presence functionality Canonical introduced in Intrepid? Is Canonical going to forward port that or is essentially duplicating functionality in the gdm trunk with a custom implementation?

The differentiated guest and presence features that Canonical developed against the 2.20 codebase was one of the reasons holding back from using gnome 2.26 in Jaunty. And it appears to be something Canonical is going to continue to spend manpower on.

You have to be careful, the more you differentiate and rely on those differentiated features, the harder it is for you to continue to move forward as upstrem executes its own feature roadmap. The fact that Canonical is having to spend time refactoring its customized guest session and customized fast-user-session code..code that was created against the older 2.20 gdm codebase..means Canonical has less manpower resources to help finish off the gdm rewrite.

And looking even longer term, how much of the differentiated work that Canonical is doing across the Gnome desktop more generally is going to make it into the vision for Gnome 3.0? Is all the effort that continues to go into making libindicate and notify-osd work across the application space going to be incorporated into what gnome 3.0 is? Will it even fit in with gnome 3.0 concept of a shell interface? How much is Canonical engaged in upstream roadmapping for new functionality? You can’t really argue an “agree to disagree” attitude if Canonical isn’t taking part in the roadmapping discussion and making the case for integration of their technology implementations.

-jef
Anonymous
kubuntu has packagekit now, so i imagine there is at least SOME work being put into it
Alexander
Any news on integrating PackageKit as the principal software installation mechanism?
Michael
Thanks for reminding me of that feathertail. I didn’t follow that too much but the blueprint is here: https://blueprints.launchpad.net/ubuntu/+spec/desktop-karmic-integrating-with-ubuntu-one . The plan is to make it as easy to use as possible (and potentially integrate into migration-assistant if you already have it) without being invasive such as popping anything up or adding an unconditional step to the installer. The client is not proprietary, it is open-source and in fact I have already used the source to help me in another project so I am quite grateful for it!

Jef, that is a great question. I am not an authority on this topic (you might try dbarth and seb128?) but my understanding is we want the bleeding edge of GDM, so definitely 2.26, I assume 2.28. Since that is the case it should be easy to upstream, but whether we are doing the work upstream or downstream I don’t know. Upstream is the preferred approach so I imagine if upstream wants this feature, we would. Ubuntu gets knocked for doing too much work downstream and while some of this is deserved (there was in fact a general session on this), in some cases upstream is not interested in our direction, and that is the purpose of being a different distro after all. If we weren’t doing anything different there wouldn’t be a point to existing!
Jef Spaleta
What version of gdm is Canonical committed to shipping in Karmic?

Jaunty is still shipping the gdm 2.20 release while the last upstream release was gdm 2.26.
http://ftp.acc.umu.se/pub/GNOME/sources/gdm/

Is the restart option for gdm going to be developed against gdm upstream trunk in collaboration with upstream gdm? Or is it going to be a downstream patch built against the old gdm that Ubuntu ships like some of the previous gdm patchset that Canonical developed in the Intrepid development time period?
feathertail
What about the Ubuntu One integration that I have been hearing about? And if it’s being included, isn’t that kind of distressing when it’s proprietary?
Michael
Anonymous, upstream GNOME isn’t switching to Banshee as far as I know (but Banshee would eventually like them to) but as andreasn points out below (thanks!), apparently GNOME doesn’t ship one at all except totem. So this isn’t a divergence from upstream, just a change on the Ubuntu side. And in fact going to Empathy is in fact a convergence bringing us closer to GNOME! As far as upstreaming the boot stuff, last I heard (last UDS) Debian was not interested due to how wide-reaching the changes are, but maybe they’ll come around. This also could have changed recently.

nailor, you can find the blueprint at https://blueprints.launchpad.net/ubuntu/+spec/desktop-karmic-gnomescan. I’ll add this to the post.

gilir, I agree and that was one of the first things I noticed as well that was disappointing about Banshee; no fade in/out.

Alex, I think Mono is more modern and should allow easier extensibility and maintenance as well as make new contributors more likely. Plus tomboy and gnome-do are already using it. As far as PackageKit goes, I’m not sure, but you can check out summit.ubuntu.com for the full schedule!

Mackenzie, thanks for the info and I have heard that, but I doubt that a custom widget takes THAT much more memory; something else is definitely going on. Plus, my library is about 50 gigs and 8,000 songs, so I feel like I am on the upper end of the average desktop user anyway. I don’t think we should be harming the experience of people with libraries of this size, so someone can have 220GB of music with less problems :) I’ve also had issues with it skipping on my Core 2 Duo, which RB never does, so that is unfortunate. That said, I do think Banshee is headed in a cool direction and I’m excited to see where it goes.

Bayger, I don’t know of that being considered but it could be cool; I’d personally love for a “look and feel” choice in the installer which lets you enable a dock instead of a bottom gnome-panel as well.

Pro, the idea with boot choice is to make it fairly simple (I think pressing something like shift/alt/ctrl) and make it as discoverable as possible, but with Windows or OSX you just “have to know” so we don’t have much to go by as far as presenting this. Also, the restart option should make it fairly easy.

Pēteris, I’m up in the air about some of the changes as well, but we’ll see how feedback goes. If upstream is as responsive and interested as claimed, the kinks should be worked out. And indeed, gnomescan does seem to be what is being considered.
Pēteris Krišjānis
Pidgin replacement - good, RB replacement - bad, for scanning - Gnome Scan please. Everything else sound good :)
Pro
The new boot menu which makes grub time set to 0 might cause a headache for new users to ubuntu who are just trying it and would like to have easy access to their win/osx os also.
it seems this step needs to be executed really well for it to be good for both experienced and new uwsers.
Anonymous
Great news indeed! Thanks for your report.
It seems that Karmic will be awesome.

(Even if I don’t feel completely trustworthy Banshee for Mono…)

@Bayger
Usually I keep the brown colour but I like your idea.
andreasn
Nice to hear Banshee is moving closer to getting included. The player have a really good and slick UI, and hopefully it means we can convert more people to Ubuntu.

Anonymous: GNOME upstream don’t have a default music player included in the desktop module set [1] (unless you count Totem)

1. http://live.gnome.org/TwoPointTwentyfive/Desktop
Bayger
Great news! By the way, have you considered adding desktop theme selection during installation of Ubuntu 9.10? I am sure that a lot of people would be very grateful to have a choice between standard brownish theme and 2 or 3 other colour schemes (green, blue). This would be great option, especially for beginers. And yes, people are a little bored of all those browns everywhere.
Mackenzie
Banshee has a higher memory overhead as a baseline, but it does not require more memory for larger libraries the way Rhythmbox does. This is due to a Banshee-only widget which only needs to store the visible items in the song/album/artist list plus 10 above and 10 below. Rhythmbox’s stock GTK list viewer must hold ALL items, so if you have a very large library, it could easily use hundreds of megs of RAM.

If RB consistently uses 25MB for you, you must have a small library. For people like my friend with 219GB of music, RB uses much, much more.
Alex Eftimie
I don’t like the idea of Banshee either, mostly because of .NET implications.

Anything else sounds great!

How about PackageKit integration? Have you discussed that?
gilir
Banshee lacks also the gapless/crossfading playback, which is a NO-GO for me. Sure Banshee have cool features, but it should not hide the problems Banshee have (memory, crash, slow …).
Also RB is not so inactive, the 0.12.2 was released some days ago and there are 2 GSOC on RB.
nailor
Michael, can you provide a pointer to more info about the ongoing investigation of scanning applications?
Anonymous
Is GNOME upstream switching to banshee too?

I hope you push those speed improvements back to Debian.

A teaser: Desktop integration with encrypted directories for Karmic

Recently I’ve been working on desktop integration with ecryptfs. Dustin Kirkland has done some awesome work enabling encrypted Private directories, as well as entirely encrypted Home directories, and I want to bring a UI to that goodness for the Karmic desktop.

UbuntuOne displays a banner at the top of its shares, and this inspired me to borrow the code for use with encrypted Private directories. After a bunch of hacking and debugging, I finally got something to show up:


Pretty exciting! There is much work to be done behind the scenes but this is an encouraging start. After I get this working I plan on making a UI for installing ecryptfs-utils (the necessary package), setting up an encrypted Private directory, and managing/configuring one (or an encrypted Home). This UI would perhaps be available from System -> Administration -> Encrypted Directories, and would allow a user to have a directory of encrypted files available in a few clicks, which is mounted/unmounted transparently at login/logout.

What do you think? Are you currently using an encrypted Home or Private directory? Would you be more likely to if there was a UI to set it up? Please share your thoughts and comments :) I’ll be at UDS and can schedule a session on this if there is interest, as well.

Comments

Perpetual
Hi, I really need something like this. My family multimedia center, based on ubuntu, contains a lot of movies (horror and such) that my small children better not click on by accident. Still, I want them to be able to access the cartoons section in the Movies directory. Everybody just uses my account, it is always logged in, so decrypt at login does not cut it. Instead I need something that pops up an “enter decrypt password” dialog when someone tries to access the grown-up movies.
Michael
Since I got some upstream questions I’ll give an update there. The projects this would touch are ecryptfs-utils and potentially seahorse. ecryptfs-utils has stated an interest in this so that means the nautilus extension, python API, and setup/configuration UI are likely to land upstream there.

As far as seahorse I haven’t heard anything (I think I contacted them but am not sure), but I’ll hopefully meet someone involved in that project at UDS and discuss it with them. Perhaps I’ll end up putting an “Encrypted Directories” tab in that utility which allows the installation of ecryptfs-utils and from there allows you to launch the ecryptfs UI. I don’t think embedding a whole UI devoted entirely to ecryptfs makes that much sense in seahorse.
gourgi
i really like this approach!
nautilus intergration looks very cool!
As for the menu approach, i strongly believe that you should talk/work in collaboration with upstream (seahorse?) before implementing yet another system-> administration menu item. Encryption and keyrings is the right way to go IMHO…

Looking forward to have this in karmic!
Gourgi
Anonymous
What about implementing this nice feature upstream?
Wirelizard
I love the fact that the open-sourced UbuntuOne client has already help produce cool improvements like this.

All the closed-source UbuntuOne server software has produced in the same timeframe is controversy in the community!

Go open source!
Nathan Nutter
A UI is definitely necessary if desktop users are to take advantage of it. On Mac OS X there is FileVault and it is super easy to setup. However, I don’t currently use it because of concerns of stability/corruption. Of course, I have no evidence of this it’s just something that got baked into my head. I think it’s important to address the “safety” of using a new feature and I hope Canonical/Ubuntu will have some information about using encrypted home folders and/or encrypted directories.

Things like:
* How do I know some “minor” corruption won’t end up locking me out of all my data?
Michael
Thanks for all the encouraging comments everyone!

meastp, for 2-factor encryption with a USB stick I'll point you to Dustin's post on how to do exactly what you say! http://blog.dustinkirkland.com/2009/03/ubuntu-encrypted-home-with-2-factor.html

wiflye81, the theme is shiki-colors (wise) available on gnome-look and potentially will be added to the community themes package in Karmic.

Jim, I am not doing any graphical work for other desktop environments but I have created a python API for ecryptfs and a generic controller for my UI which other DEs can leverage to implement their equivalent quite easily. I'll probably be talking about this at UDS.

Philipp, filename encryption is supported since Jaunty and is the default, though you can specify to not encrypt filenames if you like.

Andrew, System -> Preferences does make sense in some ways, and ideally it would be a tab in the Encryption and Keyrings entry there, although that application is done in C++ and Glade, neither of which I am familiar with so I am not sure it will end up there. I suggested Administration simply because it is a somewhat advanced feature, and will lead to data loss if you forget your password. Though ideally this risk can be mitigated with the proper prompts to record your encryption passphrase.
Andrew
Would System > Preferences not be better, as it is on a per user basis, rather than a system wide thing.
Dread Knight
I will use encryption for sure with a GUI. :) Keep up the good work!
Philipp Kern
Are filenames encrypted nowadays?
Philipp Kern
This comment has been removed by the author.
Alan
An excellent development, but one which I might not end up using.

Reason being - I need to share an encrypted local space with Windows machines on the same network (& a MacBook possible soon …). I can (and have for some time) do this with TrueCrypt, although with less desktop integration.

So my ideal would be integrated AND Truecrypt compatible.
Jim
Hi There,

Thanks for doing this work. :) If possible, would any of this desktop integration be available for Kubuntu or Xubuntu?

Per this bug ( https://bugs.launchpad.net/ecryptfs/+bug/365796 ), the ecryptfs functionality was a bit less fleshed-out in Ubuntu’s non-GNOME distros for Jaunty, so it would be good to include Kubuntu and Xubuntu folks on this.

Thanks again!

Jim
wiflye81
Great work, love to see it in karmic.

Is this the official new theme for karmic (green looks good for koala and eucalyptus ;)) ?
Jeff
+1 for the GUI to set up encrypted directories!
MadsRH
Currently I’m not using encrypted directories, but would be more likely to if there was a slick UI to set it up.

Fantastic work :-)
Jon Dowland
I think encrypted-home-by-default for installs is a good goal to reach for. This is the first I’ve heard of ecryptfs though and I wonder if it’s the best tech for the job. I currently use libpam-mount and dm-crypt/luks.
clivew
Excellent concept
Love to see it in Karmic

Best of luck!
Anonymous
I am really looking forward to using this, especially since the new laws came in in the states saying that when you (re)enter the states they can take any paper or electronic storage device from you for “examination” without any kind of warrant or even suspicion of wrongdoing?! I mean, I have nothing illegal to hide (not even ripped music or movies) but I would prefer customs officials weren’t looking at my private documents and photos without me knowing. If they need to ask for a password, they need to give an explanation as to why they want it. :)

Anyway, I’m very grateful for your work on this, as the only thing holding me back from using an encrypted directory is a) not knowing how without a gui and b) just wanting to know it’s stable first.

Given a little time, b. will take care of itself, so I’m very happy to see work on a nice gui!

Thank you!
meastp
I would love this!

I used encrypted fs a couple of cycles ago, but it was a hassle to have to type the password on start up. Since I am using a laptop, I would like to encrypt at least some of my data, in case of theft etc. This solution will make a breeze, it seems! :)

Could you perhaps consider a feature for two-phase security (e.g passphrase-thing on a usb-pen ) on login / unlock ?
Anonymous
Great, i would love it.
Kind of EncFSVault wich replaces Apple’s FileVault with good encryption.
Hope to see this soon

Counting the number of Ubuntu users

There have been a few articles recently trying to estimate the number of Linux users, which is apparently a challenging problem. However I have to wonder why it can’t be figured out at least at the distro level by simply storing hashes of IP addresses that hit Canonical’s update site, and looking at the number of unique ones each week/month.

There are going to be people using mirrors, but this is a small percent to lose to at least get something in the right magnitude, and the most popular mirrors could probably do a similar thing and contribute their numbers anyway. The only other main drawback would be multiple Ubuntu machines under the same IP, which again seems like it would only result in a slight inaccuracy. You’d also lose a small percent to users infrequently using their computers such that they aren’t updated on a monthly basis, but yearly results would pull back in any of these people using their computers frequently enough to warrant counting.

Alternatively, as others have suggested as well, if Google would just release their numbers for browsers hitting google.com, we’d probably have a solid idea as well.

Are there already accurate numbers for Ubuntu and if not, am I missing something with my proposal?

UPDATE: Jef pointed out that Fedora is already doing this at http://fedoraproject.org/wiki/Statistics#Yum_Data, which is pretty awesome! That shows about 14 million unique repository connections, so making a VERY rough, not remotely scientific estimate, we could use distrowatch to estimate that Ubuntu has 1.68 times the number of users as Fedora, and get something around the order of 24 million users that have connected.

Comments

silver
What about universities, schools and companies using ubuntu all under the same ip-address?. This could have a pretty big influence on the numbers!
Tommi
I would either count at all the Distrowatch site. Even the sites admin says that should not be used anyway to calculate the amounts of users.

And I would like to see too that MAC address is hashed to upgrades, but Fedora is already using the smolt what generates unique ID of the system specs and sends it to Fedora. Sadly (good thing) it is not default so not all enable it. But by those figures says that Fedora has more installs than Ubuntu. Sorry about that, you ain’t special, only gaining the media attention ;-)

I dont think that market share figures means anything for us. (OK, mayby some like to extend their virtual penis).

We all - what ever distribution we use - are using same OS. It does not matter what is your opinion about system, packagemanager, brand or even freedom. All we use Linux OS to power all the other software. Without Linux, we would not be here. So kudos to Linus Torvalds to code the OS in first place and kudos for Richard Stallman to start GNU project what gave us GPL and so on free software and we got the Linux OS licended under it (GPLv2).

I see more other distributions than Ubuntu in Finland, (the homeland of the Linux OS). Same thing around on the European or Asia where I have traveled.
Ubuntu has smaller share on technical oriented users (science labs, computer stores, universities etc) but that does not matter at all. We are all using same OS, the Linux (kernel)!
scottritchie
If you want a rough estimate for the proportion of Ubuntu vs Fedora users, a better back-of-the-envelope would be the Desktop Linux Survey: http://www.desktoplinux.com/cgi-bin/survey/survey.cgi?view=archive&id=0813200712407

Now, it's 2 years out of date, but back then Fedora had 6% and Ubuntu 30%; if that's still the case today this would put Ubuntu at 5 times Fedora rather than 1.68 times Fedora.

Another metric might be Google Trends: http://www.google.com/trends?q=fedora%2C+ubuntu – suggests that the Ubuntu:Fedora ratio of searches has been increasing; so either the ratio of usage has been changing or Ubuntu users are becoming even more likely to search for Ubuntu for some reason.

Now, Google Trends and Desktop Linux surveys can be inaccurate for a variety of reasons. I'll note, however, that the google trends data is consistent with the desktop Linux survey data; both imply roughly 5x - and both are doubtlessly better proxies for usage share than distro watch.
Jef Spaleta
Micheal:
At no point did I say that all “fun” things can’t be “meaningful.”

What I said was very specific. The distrowatch metric is not meaningful. The google trends metric is not meaningful. I making absolutely no claim about the meaningfulness of any other “fun” activity. I will say that making global maps of client connections to MirrorManager using GeoIP is both “fun for me” and “meaningiful” as it gives Fedora an easy to understand snapshot of how globally used Fedora is.

You’ve extrapolated what i said and attempted to apply beyond the bounds of the original context. Is that “fun” for you as well..making gross generalization about what other people say? That’s neither friendly nor healthy. You want to keep this friendly..you want to keep this constructive? Then take more care and rein in your tendency to generalize.

I find it really amazing that you can so easily discount accurate ShipIt numbers as a useful rough metric and yet… you reached for distrowatch as a scaling metric. Stop putting the cart before the horse. Make accuracy the primary importance.. then worry about interpretation. Don’t waste your time trying to interpret the meaning of numbers that aren’t even accurate measure.

If LoCos have hundreds of cds collecting dust every release…that’s also something you could get accurate stats on…you just have to survey LoCos and ask them. if they are requesting CDs and not giving them out..that is a drain on Canonical resources. It benefits everyone by making sure that’s not happening too much.

-jef
Michael
Anonymous, hashing the MAC addresses instead is a pretty good idea! I just suggested IPs because that is something the mirrors already have access to so it wouldn’t involve any extra data sending or controversy. It is a great idea though, it would fix the cases of multiple users behind one IP, and also one user one multiple IPs.

soren, it sounds like you are right, sorry :) The more I look into this, the more I see that mirrors are more common than I thought. Thanks for enlightening me and sharing your knowledge! Though, I still think it would be feasible if enough mirrors participated. Combined with hashed MAC addresses, it has a decent accuracy potential.

Jef, ShipIt is certainly countable but I am not convinced how useful it would be. Surely some people order CDs and never use them, and someone in a shop might order one and install it a hundred times. And each LoCo could have hundreds on hand that never get used. Also, I’m not sure I could agree that something which is fun can have no meaning; fun IS meaning!

Lots of knowledgeable people have shared great stuff here, so that’s awesome! Let’s just try to keep it friendly and healthy :)
Jef Spaleta
Are you suggesting that Shipit is uncountable? It most certainly is countable. If anything its the most accurate statistic you have available to you.

Canonical could tell us tomorrow exactly the number of ShipIt disks they have paid for AND the number of disks purchased directly for the Canonical shop. Have they ever done that? Have they ever put any hard numbers out with regard to how active ShipIt is? I haven’t found them. If they haven’t that’s a pretty remarkable lack of transparency.

How about you press your leadership to publish the no-guesswork numbers associated with the amount of media sent via ShipIt for Intrepid on a monthly basis since the release of Intrepid. What is it maybe 1% of the total number of Ubuntu users Canonical employees have publicly claimed exist?

You want to haggle over a statistic that below the noise floor of any overall estimate fine..go right ahead…noise seems to be pretty important for Ubuntu supporters…much more than accuracy.
soren
Michael, I know what “order of magnitude” means, and I’d be deeply surprised if people using archive.ubunut.com as their mirror wasn’t the minority. We have an extensive mirror network, and we configure Ubuntu systems be default to use country specific mirrors. Hence, metrics based on web requests to the primary archive mirror at archive.ubuntu.com would be completely useless to even come close to the order of magnitude of users. The only thing you can use it for is to get a rough idea about trends, but even then, it’s a stretch.

Jef, you’re missing the fact that Ubuntu is freely available from ShipIt, no matter where in the world you are, and no matter what sort of network connectivity you have (as long as you can actually get to ShipIt, of course). Hence, users who can only get Linux by these means will never be counted by a service such as MirrorManager. Since a similar service to ShipIt does not (to my knowledge) exist for any other Linux distro, number from MirrorManager-like services will be biased in favour of non-Ubuntu distros. Let’s face it: There’s no way to get hard numbers for number of Linux installs.

I’m sure there’s also organisations that for whatever reason do not want to publish their use of Linux at all, even to an (alleged) anonymous service like MirrorManager. I don’t say “alleged” because I don’t believe it to be anonymous, but because there’s no way for me to know whether that’s the case or not, and for some organisations, that’s simply not good enough.
Anonymous
Couldn’t apt send the MAC-address (or some hash of it) to some kind of MirrorManager-like service? You would then get unique machines and I don’t think people would feel it would violate their privacy (I don’t feel my MAC-address is something private)…
Jef Spaleta
I’m offended.. because those are exactly the sort of numbers which the laypress picks up on. By doing the calculation and publishing a calculation you know has no merit..you are not helping make the case for solid methodology for counting that can be reused by all distros..so we can get good solid numbers out to the press. Because god knows the press don’t care about accuracy.

Do you want solid numbers or not for total linux usage? If you do, then don’t publish goofball numbers yourself.

distrowatch and google trends…while “fun” to look at..have no meaning..have no value..in any well understood sense. You might as well just generate random numbers between 1 and 10 million for all distros and call them a rough estimate with a +/- 9 million errorbar on all the numbers.

MirrorManager and the statistics it generates is a methodological approach that everyone can use..we could get solid consistent numbers across pretty much all linux distributions if they adopted the MirrorManager approach to handing out mirror information to clients dynamically. There’s real value for everyone in this tech. Users, network admins, and distributors. We don’t have to rely on CEO’s making up deployment numbers in press interviews.

-jef
Michael
Jef, I completely agree! That’s why I said it was “a VERY rough, not remotely scientific estimate”. It was just meant as a fun number. I am not at all serious about it; I thought it would be obvious from my disclaimer. Obviously the ratio obtained on distrowatch is flawed in numerous ways. I’m sorry if I have offended you by multiplying some numbers together :) If you really don’t think my disclaimer was sufficient, feel free to let me know why it isn’t VERY rough or not remotely scientific, and I can adjust it.

pfrields, thanks for leaving a comment! It is good to hear that you guys are somewhat serious about metrics like this, and are certainly quite ahead of Ubuntu (as far as I can tell) in terms of collecting the data and being open about it. Like Jef said, it is certainly an aspect that is missing from Canonical/Ubuntu which is unfortunate considering their other marketing efforts.

Anyway I definitely appreciate some Fedora folks chiming in here and didn’t mean to offend anyone with my wild extrapolations.
Jef Spaleta
Your distrowatch metric is absolute crap. I refuse to let that stand without comment.

Xandros is ranked pretty low.. and yet it has a significant number of pre-installs via being the linux Asus uses on its EEE netbooks…for like what is it now 2 whole years. In fact from netbook sales estimates Xandros is crushing Ubuntu netbook pre-installs.

In no way whatsoever can you reasonably argue that the distrowatch metric correctly places Xandros compared to the 30 or so other distros in front of it. No way.

The distrowatch scaling metric does not stand up to scrutiny.

To understand how to use the distrowatch metric you have to understand why people are going to distrowatch. You also have to understand that distrowatch’s own ranking system has a nonlinear affect on the ranking. Higher ranked distros are going to get more interests from new distrowatch visitors..because they are highly ranked. It’s a feedback loop in the methodology. And it makes for an absolutely crap metric of anything at all.

If you are serious about this you need to find a metric that actually measures what you are interested in.

What you need to do is demand Shuttleworth or any other Canonical employee who has so far been quoting userbase numbers in the press for the last 3 years..that they actually describe how they get those numbers.

http://www.theregister.co.uk/2008/10/27/shuttleworth_ubuntu_commitment/
“Precise Ubuntu installed base numbers are impossible to obtain, but Shuttleworth said the most recent estimate is about 8 million users for the Linux variant. Ubuntu does not have any call-home features to help Canonical count installations. That’s because Shuttleworth does not want to violate users’ privacy or put up any barriers to adoption for the software. “We actually have no idea,” Shuttleworth admitted.”

Numbers have contexts… methodology has meaning. You can’t just make up numbers and scaling factors just because they seem to fit the argument you are making. You have to test them for sanity. The distrowatch scaling factor is not a sane metric.
pfrields
I do look at total uniqueness once in a while. As with all the measurement scenarios, there are some flaws – the office I used to work in had many users on different versions, but they would be counted as one IP address since they were behind a firewall. And if the object is to capture number of *machine* installations, then you’re also missing users who have more than one version on separate machines at home (as I do).

Nevertheless, the count I just found from the already collected IP lists was over 12.5 million totally unique IP addresses, out of around 14 million current IP addresses found through simple summing. Obviously this count doesn’t include Fedora derivatives such as CentOS, Scientific Linux, Red Hat Enterprise Linux, and so on.
Michael
Anonymous, I am familiar with popcon but that is a purely opt-in system so as such is only useful for relative comparisons between packages, not for getting a good count.

PriceChild, using the security repo seems like a good idea, although it appears that for me I am hitting my mirror for security too. Some users/admins probably disable everything but that anyway.

Jef, that is awesome, thanks for the resource! Do you know if the total across versions is unique across them? That is, if I have Fedora 9 installed and then upgrade to Fedora 10, do I get counted twice? That seems like a common place where people are going to get double counted over the course of a year.

pochu, hopefully getting the main and US repository picks up a large percentage, and making it easy for the other mirrors to participate would make it fairly accurate.
pochu
IIRC the installer sets apt to use mirrors by default, so people using the canonical repo are likely the minority
Jef Spaleta
What you mean you don’t trust Canonical’s statement has been using every year since 2006 that Ubuntu has 8 million users? I don’t trust it either.

Fedora already has a way to estimate users via IP. We have the dynamic MirrorManager service..and it has logs…

http://fedoraproject.org/wiki/Statistics#Yum_Data

People have put a lot of thought in to what is actually achievable and what is not with regard to Fedora metrics:
http://fedoraproject.org/wiki/Infrastructure/Metrics

There’s absolutely no reason Canonical couldn’t take the MirrorManager codebase and adapt it for Ubuntu’s needs. Unlike Canonical, which spends a lot of time building closed web services codebases they are reluctant to share. All of Fedora infrastructure is done in the open..including the MirrorManager service.


MirrorManager is important enough to talk more about. Every Fedora client by default contacts the MirrorManager service asking for which mirrors to use. The MirrorManager service even lets admins on large private networks redirect fedora clients in their network block to a local private mirror..without client reconfiguration. We still count those clients because they contact MirrorManager instead of having to be manually reconfigured to point to the local mirror. Our MirrorManager service is a benefit to both the user and the local network admin who is trying to conserve bandwidth….and its enabled by default.


-jef
PriceChild
perhaps those hitting the security repo (central, not mirrored and used by default unless sa chooses not to) could be used… infact I’m pretty sure they’ve looked at that before.
Anonymous
You know that http://popcon.ubuntu.com/ exists?
Michael
soren, it doesn’t quite sound like you understand orders of magnitude if I understand your comment correctly. Unless people using the default mirror are in the MINORITY (I highly, highly doubt this is the case), it would easily be on the same order. And second, as I mentioned in the post, the most popular mirrors would likely contribute their counts to Canonical as well, making it irrelevant.

Stefano, I agree, but again this number would be better than throwing around download numbers and random guesses. And conversely to your point, as Drew mentions, there are going to be users on netbooks/laptops updating from many different IPs which will partially offset that issue.
Stefano
Yes,
counting unique IPs is the first whing that can be done, but clearly this is not accurate due to NAT.

Behind a single IP you might have N users, and you don’t know how many they are.
soren
I think you’re greatly underestimating the percentage of users using mirrors. I don’t think we’d be anywhere near the order of magnitude of users that way.
Drew Stephens
This sounds like good idea to me - each distribution’s package repositories (well, except for Red Hat/Fedora with it’s decidedly lack-luster and ill-used repos) has a good handle on the number of actively used machines. There will be some that won’t get counted, because they update from someone’s local mirror, and a few that will get counted multiple times when the check form different IPs, but it should be a close-enough number.

Extending Java’s Semaphore to Allow Dynamic Resizing

Marshall wrote an excellent post explaining how to dynamically change the number of permits when using semaphores in Java, which I thought I’d share for anyone interested. This can be particularly useful if you have a long-running daemon which you don’t wish to restart for changes such as this. If you are using semaphores in Java, or if you don’t even use Java but just want to learn more about semaphores, I’d recommend giving it a read.

Email Deliverability & RFC 2142: Everything you wanted to know and never dared to ask

Today Franck Martin wrote an interesting post regarding RFC 2142 (“Mailbox names for common services, roles and functions”) and how it relates to email deliverability. If you are running your own email server or own a domain, you may interested in reading it as it sums up what email addresses are expected to be manned at any domain and for what purposes.

For example, did you know that “if an Internet service provider’s domain name is COMPANY.COM, then the ABUSE@COMPANY.COM address must be valid and supported”? And are you manning your postmaster and hostmaster addresses?