Eavesdropping on a September 2015 discussion about Chain Replication

Silly me, one day I wrote a tweet that said: Just wrote 1,400 words about chain replication in 3 email messages to [@**cmeik**](https://twitter.com/cmeik) to make certain he's occupied when he wakes up in the morning. It got two retweets and 13 favorites and a few replies, including and including: [@**slfritchie**](https://twitter.com/slfritchie) Publish it? You do have a larger audience :) [@**cmeik**](https://twitter.com/cmeik) [@**slfritchie**](https://twitter.com/slfritchie)[@**swvist**](https://twitter.com/swvist)[@**cmeik**](https://twitter.com/cmeik) I'd be interested, and I imagine [@**zeit_geist**](https://twitter.com/zeit_geist) (who has also worked on a chain replication system) would be too!

The first sake/nihonshu that I loved

Wow, I’m so happy and excited tonight. I was doing some exploration in an old photo archive, and I found the photos that I’d been looking for … for more than two years. I will try to tell a story about sake (in Japanese, nihonshu/日本酒 ). It’s definitely a love story, from the beginning. And a few photos.2015 February, a sake brewery tour Louise and I had moved to Tokyo in late summer of 2013.

Moving to Tokyo and making choices about language study

"It's odd, not everyone speaking English here. It's almost as if this weren't America at all." -- Me, joking very intentionally while thinking of [The Onion news article, "Perky 'Canada' has its own government, laws".](http://www.theonion.com/article/perky-canada-has-own-government-laws-19927) Joking aside, my work environment doesn’t force me to use Japanese, so I have the luxury of choice. I have nothing but deep respect for émigrés, ex-pats, refugees, and migrants who find themselves in a sink-or-swim language “school” of hard-knocks-life.

A third year in Japan

This week marks the beginning of our third year living and working in Japan. Thanks, Louise, for pushing me to leave Minneapolis for a while. The last two years have been gratifying, tremendously delicious, educational, routine, and … aw, enough of the adjective festival. Many thanks, Uncle Basho, for sending us here and giving me and opportunity to work with amazing people! I’ve got a lot of wish list items added to my “todo before leaving Japan” list.

Humming consensus, an animated example with an asymmetric network partition

Hi, all. It’s been a happy & productive day working on some new code. I’ve done it brute-force style, but it’s nice to see results. Visual images are cool, and animation is very cool. My animation quality is horrible, so that reduces coolness to something mediocre. But I’m still excited because it’s a first! It’s my first animated case study of humming consensus. It’s also my first description of a case study, please don’t let that stop you from reading more.

On “Humming Consensus”, an allegory, part 2

Part 2, you ask? There isn’t a part 2. The Lamport fan fiction ends at part 1, I’m sorry. The good news is that there’s something more concrete than allegory to follow up on. The work is still in progress, but the description over at the Machi source repository at GitHub is place to look. That link is fixed at a version as of mid-March 2015. If you’d like to see the latest version instead, give it a try.

On “Humming Consensus”, an allegory

"We reject kings, presidents, and voting. We believe in rough consensus and running code." -- David Clark, about the IETF And to quote a paragraph from the introduction of https://tools.ietf.org/html/rfc7282: To reinforce that we do not vote, we have also adopted the tradition of "humming": When, for example, we have face-to-face meetings and the chair of the working group wants to get a "sense of the room", instead of a show of hands, sometimes the chair will ask for each side to hum on a particular question, either "

Hand-crafted IP packets with Form RFC 791

In a fit of silliness, I wrote a Tweet earlier today: Hand-crafted IP packets with Form RFC 791: "Print legibly and press hard. You are making up to 255 copies." It’s probably the most popular Tweet I’ve ever written. The image to go along with it is an oldie but a goodie, which probably explains why it’s been such a big hit. Here is a link to the original Tweet. Edit 2014-05-26: I am flabbergasted.


Lots of irises, last June at the Koishikawa Korakuen garden, near the Tokyo Dome stadium. (a.k.a., a demo try explore the use of the WordPress gallery feature). [caption id=“attachment_259” align=“alignnone” width=“300”] Photographing the photographer….[/caption]

TCP incast: What is it? How can it affect Erlang applications?

So, what’s the deal with the “TCP incast” pattern? Never heard of it? Join the club. I’ve been wearing a developer hat for too long and not wearing my sysadmin and network manager hats. And the publications about Ethernet microbursts and the TCP incast pattern have been hiding in conference proceedings & journals that I don’t follow. (Note to self: change reading habits.) If you’d rather read a paper about the problem, go read one or more of these:

RSS clients, please, stop pinging every 60 seconds

Hi, everyone. I know that y’all is simply craving to hear everything that I have to say, as quickly as I say it. That’s why you use an RSS client, to keep track of my blog. You keep tabs on me, because I’m worth keeping tabs on. I’m so worth it that your RSS client is printing my blog’s RSS feed every 60 seconds. I might say something. At any time.

DTrace, FreeBSD 9.0, and Erlang

While working on the DTrace probes for the Erlang R15 release (which I’ve blogged about earlier this month), I discovered that the build recipe for FreeBSD 9.0 was broken. Oh no, my favorite OS doesn’t work, sound the alarm! FreeBSD has supported kernel-space DTrace probes for a while. However, support for user-space probes, USDT, has not arrived until FreeBSD 9.0RC1. ”RC1” means “Release Candidate #1”. As of this writing, RC2 is the newest release.

Been a while since I've written

Life has been busy. Very good, but also busy. Professionally, things at Basho are very good. It’s a great bunch of smart, ego-firmly-in-check folks. Basho sent me to the Erlang User Conference in Stockholm a couple weeks ago to present my work on adding DTrace probes to the Erlang virtual machine. Flying from Stockholm to San Francisco was a very long Sunday, but I managed to survive. The meetings in San Francisco with my fellow Bashonians were great, as was a lunch with the old crowd at Gemini Mobile in Foster City.

SystemTap and Erlang: a tutorial

As mentioned in my previous posting, DTrace and Erlang: a status report, it’s also possible to use Linux’s SystemTap to watch the inner workings of the Erlang virtual machine. This is possible via a DTrace compatibility layer built in to SystemTap. In theory, any Linux system that supports user-space SystemTap should work just fine. In practice, I highly recommend that you use a CentOS 5 machine. Why? * CentOS 5 is what I use for my Linux + SystemTap testing * CentOS 6 ought to work, but I've noticed strange problems when trying to run some SystemTap scripts .

DTrace and Erlang: a status report

I thought it’d be a good idea to write a status report about my work with Erlang, DTrace, and some SystemTap. Quick, I Want an Overview and Some Slides…. I gave a presentation at the Erlang User Conference in Stockholm a couple weeks ago. The topic was about adding DTrace probes to the Erlang virtual machine. The presentation slides (and soon video, I hope) are available online at the Erlang Factory website.

Blog Roused from Slumber: the MSPIFF 2011

Yep, I haven’t written for a while. One excuse is that I’ve started using Twitter. After being a Twitter skeptic for quite a while, but I guess I’ve decided that Twitter’s 140 character limit is indeed enough for some things. For those three loyal readers of this blog, here is the link to my Twitter messages of near-infinite wisdom. Meanwhile, it’s about time for the 2011 Minneapolis/St. Paul International Film Festival to start.

Hibari mentioned in InfoWorld article

This article on Oct 25th mentions Hibari in the “Erlang” section. As a Basho guy nowadays, I’m very slightly sad that the author didn’t also mention Riak, but as a Hibari fan, I’m quite happy. Seven Programming Languages on the Rise: From Ruby to Erlang, once niche programming language are gaining converts in today’s enterprise. http://infoworld.com/d/developer-world/7-programming-languages-the-rise-620

Setting Up Hibari from Scratch in a CentOS 5 Virtual Machine

The following is based on a message that I’d sent a few weeks ago to the hibari-questions-en mailing list over at SourceForge. Since then, I’ve streamlined things slightly and added instructions on how to set up a CentOS 5 (inside a VMware virtual machine, though using “real” hardware is certainly OK also) to act as the host for a Hibari server node. Installing CentOS 5 From Scratch I was installing on virtual hardware, using VMware Fusion for MacOS.

basho_bench for Hibari

Howdy, all. I’ve written a first draft of a basho_bench driver for Hibari. Instructions on how to use it follow. NOTE: You’ll need Git and Mercurial in order to check out the basho_bench source. You’ll also need the (very cool) statistics package R installed in order to create basho_bench’s graphs. For CentOS 5 users, I don’t have any good advice, sorry. For Mac OS, once I’d installed MacPorts, I think it was just a matter of “port install R”.

Hibari News: Presentations in Chicago and Tokyo

As mentioned in an earlier blog posting, I was in Chicago on August 25th to talk to the Chicago Erlang User Group about the Hibari key value store. (Not the manga character or the announced-a-few-weeks-ago Twitter application by the same name.) The slides for my Chicago presentation were in two pieces. The first set is available via SlideShare. The second are available from my home Web server. 1. First half: Hibari -- [Key-Value store for big data](http://www.

2010 ACM Erlang Workshop approaching quickly

September 30th is quickly approaching. The 2010 ACM Erlang Workshop will be held on 9⁄30 in Baltimore, Maryland, not too far from Washington, DC. There are plenty of seats still available, so make your registration and travel plans now. :-) The URL http://www.erlang.org/workshop/2010/ has the full program & schedule. We’re happy to have David Smith from Basho Technologies as our invited speaker. The title of his talk is “Concurrent Life, the Universe, and Everything Rebar”

Hibari Presentation to Chicago Erlang User Group: Wednesday 25 August, 2010

I’ll be giving a presentation about Hibari to the Chicago Erlang User Group (and hopefully some ChicagoDB folks) on Wednesday, 25 August, 2010 at 6pm. Orbitz will be hosting the event, please see the CEUG event at MeetUp for more time and location details. It looks like I’ll be arriving in O’Hare shortly after noon on Wednesday and will be returning Thursday morning. Here’s a copy of the abstract that I submitted for the talk:

Hibari Database Released Under Apache Public License v2

It was difficult to leave Gemini, a decision that I’d made back in May. I didn’t know if this day would arrive: would Gemini proceed with its plans to release the core of Hibari publicly? Once I was on the outside, I couldn’t see in again. Would there be another delay? Perhaps a change of business strategy? It’s now official: Gemini Mobile Technologies has release the Hibari key-value distributed database into the wild.

Final MSPIFF 2010: Monday May 3: USA, Ethiopia, Finland

Well, it’s been a while since the MSPIFF “Best of the Fest” series finished. I did manage to see three films that I’d wanted to catch but couldn’t, due to work conflicts or sleep or whathaveyou. All three were on Monday, May 3rd: * [No More Smoke Signals](http://www.imdb.com/title/tt1339133/), USA. This is a Swiss film about the [Pine Ridge Indian Reservation](http://en.wikipedia.org/wiki/Pine_Ridge_Indian_Reservation) in southwest South Dakota. The central character is [a radio station, KILI, 90.

Hibari update: still writing documentation

I don’t have any big news yet to announce regarding Hibari (Gemini link). We’re still waiting on some negotiations to finish before making a big public announcement. That announcement is planned for mid-May. That’s still the goal, though it may slip to a bit later in the month in the worst case. Meanwhile, I’ve been writing a lot of documentation these past three weeks. Between the “Sysadmin Guide” and the “Developer’s Guide”, I’ve got almost 130 pages of A4-formatted PDF files.

MSPIFF: Thursday 4/29: Italy, USA

Thursday was Day 14 of the Minneapolis/St. Paul International Film Festival, the last full day with 5 screens running. It doesn’t seem like things have been running for two weeks now. Louise was able to escape from grading long enough to dash to Punch for some tasty, tasty pizza and the first of these two films: * [I am Love](http://www.imdb.com/title/tt1226236/), Italy. This film had got a 4-of-4 review from the Star Tribune, if I'm not mistaken.

MSPIFF: Wednesday 4/28: Austria, USA

Wednesday was Day 13 of the Minneapolis/St. Paul International Film Festival. It was a day of contrasts, the concrete and the surreal. * [Pianomania](http://www.imdb.com/title/tt1331115/), Austria. This documentary follows a year in the professional life of Stefan Knüpfer, a piano technician and tuner who works in concert halls and venues in Austria. [(MSPIFF link)](http://mspfilmfest.org/MMX/content/pianomania) His pursuit of the ideal piano sound is truly geeky, in large part because "ideal" depends on the pianist, the music, the piano (we learn quite a bit about Steinway serial numbers #109 and #245), the piano's many parts, and the concert hall.

MSPIFF: Tuesday 4/27: Canada, USA, and Best of Fest Schedule

Tuesday was Day 12 of the Minneapolis/St. Paul International Film Festival. Mom left for home today, so I didn’t have a movie buddie today. * [Father and Guns](http://www.imdb.com/title/tt1258134/), Canada. I was expecting a comedy, and I got one. ["De père en flic" (MSPIFF link)](http://mspfilmfest.org/MMX/content/father-and-guns-de-p%C3%A8re-en-flic) is the highest-grossing French Canadian film. What I didn't expect was the acting. There are some serious, emotional moments throughout the film, and the film had my attention and everyone else's in the theater.

MSPIFF: Monday 4/26: Canada, Italy

Monday was Day 11 of the Minneapolis/St. Paul International Film Festival. Mom stayed with us for a few days, to catch up on the worst of jet lag before driving home on Tuesday. Movies are a great cure for jet lag…. * [Genius Within: The Inner Life of Glenn Gould](http://www.imdb.com/title/tt1503774/), Canada. I'd known a bit about Glenn Gould, but [this bio-pic (MSPIFF link)](http://mspfilmfest.org/MMX/content/genius-within-inner-life-glenn-gould) helped explain why he was such a sensation in his day.

MSPIFF of yore: 21st Annual Festival, 2003

It was 2003. I’d been working for Caspian Networks for over two years. We were going to make an IP router that would surpass Juniper Networks and Cisco, “eating the lunch” of every other top-end router maker in the world. Oh, the market projections were overinflated by orders of magnitude. Oops. Sorry, we have to let you go. It was layoff #2 for me. Layoff #1 hit me when I was at Sendmail, Inc.

Speaking at Erlang Factory 2010 London

According to http://www.erlang-factory.com/news/list, my proposal for a presentation at the Erlang Factory London 2010 in mid-June has been accepted. Good news indeed! I’ll be speaking about Hibari, Gemini’s key-value database that provides strong consistency and micro-transaction support. Hibari was mentioned as part of my Erlang Factory SF Bay 2010 presentation called “Webmail for millions, powered by Erlang”. The new presentation in London will focus entirely on Hibari. Fingers crossed for luck, we’ll have the source code and lots of documentation available a few weeks before the conference date.

MSPIFF: Sunday 4/25: Jordan & Syria, France

Sunday was Day 10 of the Minneapolis/St. Paul International Film Festival. How does a person who’s just returned from vacation in Europe (asking hypothetically, of course) deal with jet lag? Movies! Today Mom and I saw two documentaries: * [The Unreturned](http://www.imdb.com/title/tt1527784/), Jordan/Syria/Minnesota, [MSPIFF link](http://mspfilmfest.org/MMX/content/unreturned). The director is originally from Minneapolis; this documentary was his master's thesis work in film at The New School in New York. He wanted a project that could make a difference outside of school.

MSPIFF: Saturday 4/24: Spain

Saturday was Day 9 of the Minneapolis/St. Paul International Film Festival. It was also concert day (with god-daughter Viveka and Ian singing) and pick-up-Mom-from-the-airport-returning-from-holiday-in-Europe day, so I only made it to one film: * [Forever Waiting](http://www.imdb.com/title/tt1338612/), Spain. The reviews at IMDB are quite low, but at the moment there are only 9 total votes, so take it with a grain of salt. [This film (MSPIFF link)](http://mspfilmfest.org/MMX/content/forever-waiting-hoy-no-se-fia-manana-si) is set in Spain in 1953 under General Franco.

MSPIFF: Friday 4/23: USA

Friday was Day 8 of the Minneapolis/St. Paul International Film Festival. The blossoms on the crab apple trees smelled even better than they did on Thursday, so I had to flee to the movies to escape them. It turned out that Friday was director’s and producer’s day. * [Ghosts of the 7th Cavalry](http://www.imdb.com/title/tt1473037/), USA. Director Tom Roberts was present for the US premiere of this documentary. Despite being an American film about American history about American soldiers, and despite the critical acclaim at film festivals around the world (I forget how many he named, but it was at least 5), this film has yet to receive any showing by HBO, ShowTime, or any other US TV distributor or distribution company.

MSPIFF: Thursday 4/22: U.K.

Thursday was Day 7 of the Minneapolis/St. Paul International Film Festival. The blossoms on the crab apple trees were intolerably lovely, forcing me indoors to see movies instead. Thursday got me to only one film, though. * [Skeletons](http://www.imdb.com/title/tt1425933/), U.K. The MSPIFF volunteer who introduced [this film (link to MSPIFF)](http://mspfilmfest.org/MMX/content/skeletons) said that it had a history of being disliked by film critics and liked by fans of quirky films. I'm definitely one of the latter, and I definitely loved this film.

MSPIFF: Wednesday 4/21: Turkey, Mexico, USA

Wednesday was Day 6 of the Minneapolis/St. Paul International Film Festival. The sudden burst of 58F cold weather forced me indoors for another day of cinema. It turns out that it was “music and Islam” day; read on for the scoop. * [Muezzin](http://www.imdb.com/title/tt1513183/), Turkey. This documentary follows the fourth (?) annual Call to Prayer competition for muezzins all over Turkey. Most of the film follows contenders from Istanbul, which has about 3000 mosques within the city (and perhaps suburbs?

MSPIFF: Tuesday 4/20: Italy and the U.K.

Tuesday was Day 5 of the Minneapolis/St. Paul International Film Festival. Nice weather, again? Well, I braved the sunshine and the mild wind to get into the movie theater for two films on Tuesday: * [Mid-August Lunch](http://www.imdb.com/title/tt1277728/), Italy. Ferragosto is Italy's biggest holiday in the holiday month of August. Everyone is out on the beaches or in the mountains. Rome's honking traffic jams vanish. Gianni is staying home this year with his elderly mother.

MSPIFF: Monday 4/19: Russia and China

Monday was Day 4 of the Minneapolis/St. Paul International Film Festival. What is it with the fabulous weather? Couldn’t it be cold and rainy for once? I struggled quite mightily through what turned out to be “transportation documentary day”. * [Pink Taxi](http://www.imdb.com/title/tt1509201/), Russia. This was its American premiere, hooray. I really liked this film. Of the 3.5 million cars in Moscow, and of the I-forget-how-many taxis in Moscow, about 22 of them are pink.

MSPIFF: Sunday 4/18: Norway, France, much of eastern Europe, Japan

Sunday was Day 3 of the Minneapolis/St. Paul International Film Festival. Despite the absolutely gorgeous sunny spring weather, I bravely fought my way into: * [Kirsten Flagstad's Square](http://mspfilmfest.org/MMX/content/kirsten-flagstads-square), shown with [Fjord](http://mspfilmfest.org/MMX/content/fjord), both documentaries from Norway. Kirsten Flagstad was a famous operatic soprano before and after World War II. There was some sort of smear campaign against her and her husband, accusing them of Nazi sympathies. The story was told in a disjointed way, leaving Louise and me to try to figure out what the accusations were.

MSPIFF Saturday 4/17: Pakistan, Ireland, and Finland

Saturday was Day 2 of the Minneapolis/St. Paul International Film Festival. I managed to get to these movies: * “[Son of a Lion](http://www.imdb.com/title/tt1198220/)” (Pakistan). If we hear about reluctance to send kids to school in Pakistan, it's more often that not about sending girls. This movie is a bit of Three Cups of Tea territory, from the view of the son (who wants to go to school and perhaps study music) and the gun-making father who sees school as useless ("Did I go to school?

MSPIFF Friday 4/16: Finland and Argentina

Day 1 of 13. I made it to the two films that I’d predicted yesterday: * “[Forbidden Fruit](http://www.imdb.com/title/tt1188998/)” (Finland). The Amish practice of going out into the secular world, as an adult, before committing yourself to the Amish way makes a lot of sense. (The fact that I married a PK (Pastor's Kid) is neither here nor there.) **Recommended**. Showing again on the 20th. * “[The Wind Journeys](http://www.imdb.com/title/tt1426374/)” (Argentina). I'm puzzled by the relationship between the two main characters: master and student isn't it, though Fermin definitely wanted to study.

2010 ACM Erlang Workshop CFP is ready to go

The last of our program committee has accepted his duties, privileges, and responsibilities … so my role as the workshop chair of the Ninth ACM SIGPLAN Erlang Workshop will now start in earnest. I started asking around and twisting a few arms (gently, gently) when I was at the Erlang Factory San Francisco 2010 conference a few weeks ago. Now it’s time to start pestering mailing lists, emailing the folks I’d spoken with and exchanged cards with, and beating the bushes for innovative Erlang work and people willing to write up a paper.

MSPIFF: Pages 16&17 tell the story

Story, what story? Oh, what I’ll be doing (on and off) for the next couple of week(nights). I got a copy of this year’s MSPIFF catalog. Last year, the festival organizers ran out of printed catalogs quite early. Louise and I went to the Norwegian WWII “Max Manus” on opening night (Thursday 4⁄15), so we were able to scoop up a copy of the printed catalog. The catalog is … wonderful.

MSPIFF: Opening night: Max Manus

Tonight Louise and I went to the MSPIFF’s opening film: Max Manus. It was well worth hustling ourselves out the door after a hasty dinner to get to St. Anthony Main for the opener. Based on World War II history in Norway, a guy by the name of Max Manus (Wikipedia) was one of a number of young men in Oslo … kids, really … Manus (in the film) protrayed as a school dropout, and if any his compatriots were older than 25, then the film lied to me.

Beautiful spring weather, or movies? ... Movies.

It’s sad to say, but in a contest of fine spring weather or movies, movies win. Good news, Louise helped me find lots of old catalogs and notes from previous MSPIFF festivals. So odds are high I can fritter away my valuable spare time by looking back at movies shown in previous years: what I’d seen and loved, the stuff that wasn’t so good, and the movies that got away.

New MSPIFF (Minneapolis/St. Paul International Film Festival)

Now we know where all of my free time for the next few weeks is going to go…. The 2010 Minneapolis/St. Paul International Film Festival starts on Thursday, April 15th. I’ve got my pass, and I’m ready to see a whole bunch of movies. It looks like all films this year, like it was in 2009, will be shown at the St. Anthony Main Theater in southeast Minneapolis, across the Mississippi River from downtown Minneapolis and within sight and roar of the St.

Humans and Hippo (Sculptures)

Does the name really make sense? Not really, unless you’ve been to Matsue City in west central Japan. The sculpture I’m photographed with is outside of the Shimane Art Museum. If you’re ever in Matsue, perhaps visiting Matz, creator of Ruby, I recommend that you take the time to visit the museum. The photo of me, Scott Lystig Fritchie, was taken by my beautiful wife, Louise Lystig Fritchie. While pondering the relationship between humans and (speaking hypothetically, of course) house cats, I’ll interrupt my reverie to post on topics of international & indie film, FreeBSD and sometimes Linux, food and restaurants (if you don’t eat, you die) and distributed data storage and algorithms of many kinds.