|
|
This one’s for filesystems geeks.
Eric Sandeen posted The world wants 32-bit inodes on his blog back in 2008, and recently followed up with another comment. His conclusion is that it’s (still) not safe to use 64-bit inode numbers (e.g. the inode64 option in XFS) because of compatibility issues with userspace programs.
Eric’s original post grew out of an email discussion we had back in 2007, and he used a script that I wrote back then to gather data. He’s carefully preserved that script – thanks Eric! – so I downloaded it and ran it on a more modern, fast-moving desktop/laptop distro to see if I could get another data point.
Here’s the output of the old script on a 64b Ubuntu 12.10 install.
$ ./summarise_stat.pl /usr/bin /bin /usr/sbin /sbin
58099 76.9% don’t use any stat() family calls at all
12493 16.5% use 32-bit stat() family interfaces only
4498 6.0% use 64-bit stat64() family interfaces only
491 0.6% use both 32-bit and 64-bit stat() family interfaces
So it turns out that script had a few problems.
- I don’t have 70000 executables! The script had a bug which mis-handled symlinks and in particular the symlink /usr/bin/X11 -> . which Ubuntu sets up, and massively overcounted executables.
- The script doesn’t take into account the difference between 32b and 64b executables, in particular that a 64b executable can use the old stat interface without problems because the st_ino field is wider.
Here’s an updated script with that and some other quirks fixed. I also added an explicit list of all the broken executables. Here’s the result of the fixed script on the same machine.
$ ./summarise_stat.pl /usr/bin /bin /usr/sbin /sbin
Summary by status
—————–
672 33.4% are scripts (shell, perl, whatever)
1338 66.5% are 64-bit executables
1 0.0% use both 32-bit and 64-bit stat() family interfaces [BROKEN]
1 0.0% BROKEN
List of broken files
——————–
These use both 32-bit and 64-bit stat() family interfaces
/usr/bin/skype
So there’s one single executable which is doing the wrong thing. This is really good because most recent machines are 64 bit, even desktops and laptops. For comparison, here’s the output on a 32 bit Ubuntu 12.10 box, edited for brevity
$ ./summarise_stat.pl /usr/bin /bin /usr/sbin /sbin
Summary by status
—————–
460 28.3% are scripts (shell, perl, whatever)
687 42.3% don’t use any stat() family calls at all
165 10.1% use 32-bit stat() family interfaces only [BROKEN]
305 18.8% use 64-bit stat64() family interfaces only
9 0.6% use both 32-bit and 64-bit stat() family interfaces [BROKEN]
174 10.7% BROKEN
List of broken files
——————–
These use 32-bit stat() family interfaces only
…
/usr/bin/nautilus
…
/usr/bin/ischroot
…
/usr/bin/file-roller
…
/bin/mountpoint
…
/sbin/e2fsck
/sbin/mke2fs
…
/sbin/e2image
…
/sbin/cryptsetup
These use both 32-bit and 64-bit stat() family interfaces
[...nothing interesting...]
Note how there’s a larger proportion of 32 bit programs which use stat64() (i.e. do the right thing) than the sample from 2008. Note also that very few of the programs which use stat() look like they might care about st_ino at all. This looks to me like application developers and distro vendors have been addressing this problem by fixing applications.
Of course this is only a single machine install, not the entire software repo, so there may be sampling issues.
To go back to the theory, the executables having problems will be the intersection of 3 sets:
- 32b programs (every program on a 32bit machine, hardly any on a 64b machine), and
- programs using the old stat interface, and
- programs caring about the value stat() reports for st_ino.
In 2007 set 1 was the majority of programs, now it’s a minority and shrinking thanks to the march of hardware progress. Set 2 has always been a small minority of all programs, and it looks like it’s shrinking due to the march of software progress. Set 3 is a very small minority. I would say the trend is towards problematic programs basically drying up. This is excellent news, if my data can be believed.
I think the right thing to do is for distros to force the issue. One way is to make the 32 bit glibc wrapper for the stat() family put a fixed invalid value into st_ino, probably 0xffffffff would be best. We could then document this field as being invalid for stat() on 32 bit machines, please use stat64. That way 32 bit programs which use the old stat() but ignore st_ino will keep working just fine instead of the current situation where they get spurious EOVERFLOW errors.
Another perfectly reasonable approach would be to remove the overflow check for st_ino, just report the lower 32 bits, and document the problem in the stat(2) manpage. Currently EOVERFLOW is documented only to occur when st_size overflows 32b, not st_ino, so this documentation change would be a clarification and not a change in documented behaviour.
It seems that Tim Bray has a draft RFC for a new HTTP status code.
New status codes aren’t exactly common things because the guys who developed HTTP 1.1 filled in the handful of holes left by the very first HTTP standard. I did not expect to see another one, and yet here it is.
451 Unavailable For Legal Reasons
That’s right, it’s now so important and common an event that the Law interferes with the technical operation of the World Wide Web, that we need a status code baked into the HTTP protocol itself.
Tim’s example of usage is humourous in the best tradition of RFCs, but the inescapable fact is that the fun time for techies is over. The Internet’s Wild West phase where anything goes, where sheer competence and bravado won respect, where fortunes were made, where men were men horses were horses and women were schoolmarms, is done. Finished. “Tamed”. Succumbed to the Rule of Law. Or more precisely, the Rule of Lawyers.
This is terrifying. What happens to the Type A/nonconformist/rugged individualist personalities who built the place? Superannuated. Marginalised. Incarcerated. Shot in the back during a poker game. Well maybe not that last one, the analogy is not that strong.
I wonder if Elon Musk needs help building his Mars colony.
Today was my last day at FastMail. It was great working with those guys and I would recommend it to anyone. But now I’m heading off to work for SGI in the Bay Area. It’s going to be a great challenge!
In the last few days, various folks on social media have replaced their profile pictures with a red rendering of the = symbol, mathematical talk for “equals”. Or various amusing riffs on that, such as the Equals-Dalek.
This is meant to indicate that the person involved supports the right of other people to choose to be married, without any level of government saying saying something silly like “no you can’t , because the person you love and want want to marry has the wrong set of chromosomes”.
Just for the record, I support that freedom.
I also support the freedom to breathe. It’s more or less the same thing, just operating at a different level on Mazlow’s hierarchy. It’s not something you can morally justify denying people, absent real damage caused to third parties.
I don’t feel the need to replace my profile picture with an = for the same reason I don’t feel the need to replace my profile picture with a big red O2. It should be obvious, duh.
The Oatmeal said it best
I’m pleased to announce that a new version of NovaProva, the new generation unit test framework for C, has been released. NovaProva is available for download at http://www.novaprova.org now.
Changes in version 1.1:
- ported to Linux x86_64
- all the meta-tests pass now
- minor cleanups
For more information, see Getting Started.
One of the things I really don’t enjoy collecting is sentences that are always a lie, i.e. they are not spoken if they’re true. For example:
“Please don’t get me wrong, I’m not here to sell you anything”
Yes, I have an iPhone again.
Yesterday I went out and bought a shiny new iPhone 4S. Apple had a sale on, and the Apple Store in Chadstone was extraordinarily busy. Of course the sale didn’t apply to the device I wanted…sigh…so the only effect of the sale was to make it harder to find a redshirted somebody to take my money. Quite a lot of my money.
Why didn’t I buy an iPhone 5? Because there’s nothing that compelling about it. A faster processor is nice, but my old iPhone was a 3GS so any current model will be an improvement. The 5′s camera is no better than the 4S’. LTE might be nice, but I really don’t need to be giving Telstra any more money when I can use WiFi at home and in the office and Instapaper for most of my reading on the train. The 5 does have one large negative: a different connector. We are an iPhone household; three days ago there were 3 30-pin chargers around the house, two at work, and one in the car. If we were going to spend time and money changing all these over or fitting adaptors, I don’t see any good reason to go to yet another proprietary connector instead of the EU standard Micro-USB connector.
So now I’m in a position to describe what it’s like to go back to an iPhone after several weeks of exile in Windows Phone land.
The very first thing I noticed: I actually had a choice of cases, instead of just one naff rubber thing from Nokia in my choice of two ugly fluorescent colours. Yay for form factor that people actually make accessories for!
The very best thing I noticed: there are actual apps again! I can play Boggle. And Words with Friends. And use Urbanspoon. Yay for a platform that people actually write apps for!
The most significant accumulation of small effects: Mobile Safari is about a bajillion times better than mobile IE. Pages actually load reliably, instead of crapping out at the 80% mark about 20% of the time. Pages load much faster, it feels about two to three times as fast. Sometimes I barely have time to notice that the new page has loaded. Typically the page title and the first few lines of content appear on screen in under a second, even before the page has finished loading, instead of making we wait a minute or two for the whole thing to download. Touching clickable screen items doesn’t randomly highlight completely irrelevant rectangles on the screen before failing to notice the touch. Sites actually format themselves half decently, instead of deciding that you must be on a desktop because “Windows” is in the user agent string. Most amusingly, The Huff doesn’t decide I’m viewing from a Blackberry and go completely useless (this made me pity Blackberry users almost as much as Windows Phone users).
Best difference for a network engineer: Safari’s page loading progress bar moves in fits and starts as the page loads and keeps going until the end. This might look inelegant but at least it’s honest. Mobile IE on the other hand is obviously lying to me. It shows progress going linearly with time and smoothly from 0% to about 80%, always, for every page. Even when I’m in a Metro tunnel and I know that there is no reception. Even when the immediately upstream DNS server is not responding. Seriously Microsoft, it’s insulting.
Worst surprise about cleaning up a Windows phone: There is no obvious way to upload SMS conversations to somewhere else. WTF? Still exploring this one.
Least surprising surprise about cleaning up a Windows phone: The Pictures tile on the main screen shows a slideshow of randomly selected photos from your Camera Roll and Saved Photos albums. Even after you’ve “deleted” them. I saw this coming and avoided taking any embarrassing photos with the phone.
The one feature I actually miss from the Windows phone: The Windows virtual keyboard shows the virtual keys in the case that will actually be typed. So in shift mode, you see Q W E R T Y, but otherwise q w e r t y. The iOS virtual keyboard shows capitals always. It’s a small thing, but it’s surprising to see Microsoft actually get something more ‟right” than Apple.
Today I want to talk about one of the project management practices that I find valuable, the post-mortem review.
The post-mortem review is a review of a project or a phase of a project, a product release, or a service rollout, performed soon after the conclusion of the work. It is in some ways analagous to it’s namesake in medicine, the post-mortem or autopsy. In both cases the purpose of the exercise is twofold.
- To establish from evidence what if anything went wrong.
- To recommend measures to avoid those things going wrong again in the future.
The analogy starts to break apart on further examination, but then all analogies do. I originally thought in this post that I would compare and contrast the medical and project management post-mortem procedures. I think I know what I’m talking about with reviews; I’ve never been a project manager but I’ve been involved with several reviews since my first in 1996, and most recently attended one Friday. But I know absolutely nothing worth saying about autopsies. Sure I watch CSI, but I don’t confuse it with reality. So…no.
So here are the most significant aspects of the post-mortem review, from my experience.
- Timing
-
A post-mortem review needs to happen soon after the conclusion of the work it covers, because it draws a lot of its value from eyewitness evidence and that evidence needs to be fresh.
However it is possible to start a post-mortem review too early. After the typical project end, the people involved are often too busy with fallout, or too tired, or too emotionally affected to contribute properly. The two factors need to be balanced out with a judgement call.
The review I attended Friday was ten days after delivery. This is a little later than usual but I think it was timed about right for the circumstances.
- Participants
-
The people in the team who were responsible for the project management of the work just concluded, absolutely must attend the post-mortem review. It is vital for their experience to understand the effects of decisions they made, and some of those effects may not be visible until the post-mortem.
If new people are taking over that responsibility for the next phase, they should attend as observers, so that can get a better understanding of how the team got to where it is.
As many people as possible who worked in the team should attend, so that their view is heard and is known by everyone to have been heard, and to avoid the review becoming a project management echo chamber.
Finally, absolutely nobody outside the team should attend: not upper management, not rival or sibling teams, not anyone from partner organisations, and absolutely not customers. The review needs to be brutally frank, and without recrimination; the presence of anyone outside the team will prevent that.
- Tone
-
That brings me to the tone of the review. It is vitally important that the meeting proceed calmly, quietly, honestly, and frankly. Any departure from this tone will most likely wreck the exercise. Accusations and recriminations will just make people defensive or argumentative. Intimidation will just hide problems and make them fester.
If there’s any doubt, the people running the meeting should explain the ground rules right at the start and enforce them, including especially for themselves.
I find that three important guidelines help keep the tone respectful:
- Stay calm and polite.
- Collective ownership: say “when we did X” instead of “when you did X”, i.e. preemptively assume a portion of the blame for things that you were not directly involved in. This helps ease the pain when people realise they screwed up. Remember, one day you too will screw up.
- Acknowledge the arrow of time: some decisions are only obviously wrong well after they were made, so be careful to use wording like “in hindsight” when appropriate.
- Evidence
-
I’ve never been in a review where most of the problems weren’t just known to everybody. The real point of the procedure is to give people a forum to say what everyone knows. Sometimes this can be cathartic, sometimes just plain dull. Be grateful for the dull ones.
I can imagine instances where you might feel the need to bring in some kind of additional evidence, like bug closure metrics. One day my dream will come true and I’ll be in a project so well run that all the problems are subtle enough to need data. It hasn’t happened yet.
- Deliverables
-
The only deliverables that matter are realistic ideas that will improve the process next time, and a belief in their importance by the people who need to implement them.
However I have seen a useful technique for organising the ideas that bubble up during the review. Make three columns and write down ideas as they are aired (don’t stop to argue, just record first). The columns are pretty self-explanatory:
- Things we did right
- Things we did wrong
- What we can do to improve
- Value proposition
-
Post-mortem reviews provide the feedback loop that allows a software development process to optimise itself. Or in plain English, it helps us screw up less next time. That is totally well worth a few hours of your time.
That’s a lot of advice. If I were to boil it down to one sentence, it would be: be full and frank but don’t let the review become a blamestorm.
Meanwhile, I’m pleased to report that my last post-mortem review was calm friendly and productive. I hope that the team in the post-mortem review for project Orca, which should be happening around about now, has that experience too.
I’m writing this in the eleventh hour of the eleventh day of the eleventh month of 2012. This is the time to remember the 1918 Armistice that ended the First World War.
It’s a time to remember the many millions who died in the preceding four years, from machine gun bullets, shells, bombs, and bayonets, from starvation, and from the influenza pandemic. The four horsemen roamed the face of the Europe and the Middle East.
It’s time to remember the brave soldiers who volunteered to kill or be killed for their countries. It’s time to remember those who were conscripted to fight, and those who were innocent victims as their towns and farmhouses were chewed up in an argument they didn’t start.
It’s also time to remember the countries born from the ashes of the conflict. In Australia, we like to think of WWI as the time that forged a national identity, but for many countries this was far more literally true. Australia existed as a legal entity before WWI, we elected most of our own leaders, we learned our own language in schools, we had our own army and navy. A great many of today’s nations didn’t have any of those freedoms in 1914.
For the Great War was extraordinarily destructive not just of people but of empires. Four of the biggest 7 combatant nations started 1914 as large, prosperous and aggressive empires and then limped out of 1918 as defeated, exhausted, and massively shrunken republics. The territories lost in Eastern Europe and the Middle East gained their own lives as their own nations.
If they could have foreseen the outcome, would those four emperors have gone to war in those crazy thirteen days after the assassination of the Archduke? Or would Wilhelm, Franz Joseph, Nikolai, and Mehmed have calmed down and sought a better solution? Maybe not. A mere twenty-one years later three of those shrunken republics were back again. Angry, puffed up and overconfident, they hoovered up lost territories and went straight back to slaughtering anyone they could reach for dominion over their neighbours.
This makes me think that in the long run we won’t as a species learn from history. Aggression and overconfidence way well be inherent characteristics of the way we organise ourselves into groups. Yes, even democracies.
The charming 2 year old daughter of the folks whose house I’m at today, just called a laptop computer a “booter”. This seems terribly appropriate for a Windows computer.
|
|
What You Said: