Erlang and my inferiority complex against functional programming

A few weeks ago in the middle of October 2016, I had some interesting exchange of tweets inspired by an Alvaro Videla's tweet. Alvaro wrote he always told people Functional Programming (FP) in Erlang was a necessity, not part of the original design (emphasis mine).

I later found Robert Virding, one of the founders of Erlang language and concurrent system, explicitly stated in this talk (video at Functional Conf 2016) (at 4m23s) as follows:

We were not out to implement a functional language
We were not out to implement the actor model
We were trying to solve the problem

Robert's statements in the quote above are the exact reason why I started and still like Erlang.

I've already erased the tweets (I keep most of my tweets ephemeral) of my response, but I would like to write my story about FP.

My first exposure to FP was through the books of Lisp in 1980s. I didn't understand the FP principles, but I got rather interested in the symbolic processing capability and the ease of handling data structures.

I'm not interested in slow (i.e., not sufficiently fast for a purpose) systems. In 1980s Lisp systems were running mostly on interpreters, so I have a strong recognition bias that tells functional languages are slow, which is incorrect as of 2016; a counter evidence is shown as a production level documentation tool written in Haskell called Pandoc, and by compiler-based FP systems such as Common Lisp. Nevertheless, in 1980s, FP systems were mostly slow. Until the late 2000s, I hadn't been really interested in FP again.

I've got a very strong inferiority complex when I tried to study something about FP and Lisp from 1988 to 1990 at a university laboratory where the main research topic was about the programming languages and paradigms. I utterly failed, because my main interest then was how to connect the people in Japan and the world, through email, USENET (NetNews), UUCP, and affordable TCP/IP links.

My inferiority complex against FP is still not gone yet. Three PC member roles, Two accepted papers, and one workshop chair role (in 2011) for ACM SIGPLAN Erlang Workshop events were something I would have least expected in the late 1980s or early 1990s. FP people are generally much smarter than me and they are amazing. I have been lucky to meet the talented and productive researchers, including John Hughes, Simon Thompson, Kostis Sagonas, and many others (which I'm very sorry to omit their names here).

On the other hand, however, I still have some feelings that some FP people are too theoretical and not respecting well on practicality of currently programmed system, when I am watching on (mostly Japanese) FP communities, especially those on Twitter. My gut feelings are like this: Theory is nice. Theory is beautiful. Beauty is important and often essential in programming. I think that's what academic and research communities for software are for. But the real virtue of programming prevails on how it solves the real-world issues and problems. There's no side-effect-free world so long as having to deal with I/O devices. Memory is limited. CPU speed is limited. Network bandwidth is limited. Isn't FP a way to deal with these real-world limitations while maximizing the ease of abstraction on programming? (I think Object-Oriented Programming (OOP) is also trying to solve the same issues, but that's another issue; I still need to understand why Java/C++/C#-like OOP languages are simply that much complex.)

Erlang has been one of the rare language systems which didn't impose or force me to do the things in "functional" way, though the FP principles are embedded deeply in the core, such as the immutability and no-shared object/process principles. I think Elixir follows these principles well, though I see some interesting FP technologies in Elixir such as macros and metaprogramming. In short: Erlang (and Elixir) didn't trigger my inferiority complex against FP!

So some trivial lessons I've learned:

• You are never too late to learn new paradigms.
• You are never too late to re-learn things once you failed to learn.
• There's something fits for you even for a category you don't feel like to study.

I think my inferiority complex against FP will stay until I die, but now I don't have to feel FP is not my stuff. And that is good.

My new startup: Kenji Rikitake Professional Engineer's Office

This is an announcement of my new startup business.

I've started my own business, as a Japanese government-licensed Professional Engineer of Information Technology (in Japanese it's called Gijyutsushi), since April 21, 2014. It's called Kenji Rikitake Professional Engineer's Office (KRPEO).

KRPEO provides the consulting and engineering services on information technology in general, including but not limited to: information and network security, the design, deployment, implementation and performance tuning of Erlang/OTP, FreeBSD, and Riak. KRPEO will provide all services in both Japanese and English.

I had been looking for a full-time job since October 2013, and I made a conclusion that the job market near Osaka was mostly for energy-consuming legacy programming tasks due to the fact that most of the high-end software engineering companies were only located in Tokyo. Most of the employers in Japan do not accept remote teleworking due to the cultural and legal reasons either. So I had to lean out from the traditional corporate culture of Japan, for a more sustainable business model, making myself, my family, and my customers happy at the same time. I understand this is a hard challenge, but I will definitely take it.

My involvement in the open-source developer and network operator communities in Japan and the world will remain the same and unchanged.

The English announcement site for KRPEO is at http://www.k2r.org/krpeo/. The Japanese site URL is: http://www.k2r.org/gijyutsushi/.

Update 28-APR-2014: add the URL of the English site, correct the URL of the Japanese site.

Erlang Factory SF Bay 2014: list of some notable talks

(Photo: a shot during my talk by Yoshihiro Tanaka, used by permission)

(Disclaimer: there were too many talks I wanted to listen to, but I couldn't.)

Here's the list of talks I thought intriguing for Erlang Factory SF Bay 2014 (links are to the slides or videos):

• Mike Williams on What Programming is Really About
• Brian Troutwine on Monitoring Real-Time Bidding at AdRoll
• Kostis Sagonas on Erlang VM and Term Storage scalability
• Fred Hebert on Planning for Overload
• Natalia Chechina on Scaling Distributed Erlang at a supercomputing environment
• Garrett Smith on Erlang and CZMQ, and the choice of using Erlang Port for CZMQ
• Dave Thomas and Jose Valim dropped an atomic bomb to the audience on how to catalyze the change to get more newcomer and other community's attention towards Erlang/Elixir ecosystem
• Rick Reed on Scaling at WhatsApp

I also wanted to listen to the following talks, and found the slides intriguing:

• Jay Nelson on eliminating single process bottlenecks with ETS concurrency patterns
• Lukas Larsson on Memory Management battle stories
• Louis-Philippe Gauthier on Performance Optimization 101

I will post my impressions for the above talks in later articles. I would like to note some personal impressions for the audiences this year:

• Erlang is no longer an exotic language or system. The audiences want the real solutions and hints.
• Elixir is gaining popularity, and will surely contribute to reduce resentment against BEAM (Erlang VM) and the ecosystem.
• The implementation talks were getting more detailed and hard core, and the questions were also more specific.

Video quality

Thanks to the hard-working video and audio recording and editing team, this year's video quality is very high in overall. While the live streaming was not possible due to the prohibitive cost, some videos were made available within six hours from the end of the talk. I think this was impressive and a practical solution to make a trade-off between the turn-around time for the availability and the quality of video of the talks.

[To be continued in another article]

Erlang Factory SF Bay 2014: erltrek and the beginning

(Badge photo by Kenji Rikitake)

It's already been near a month since Erlang Factory SF Bay 2014 as I'm writing this blog article. It was my fifth Erlang Factory, consecutive once a year since 2010. The topic I've talked are so diverse:

• 2010: SSH RPC
(sponsored by NSIRG, NICT, Japan)
• 2011: SFMT on Erlang
(sponsored by ACCMS, Kyoto University)
• 2012: IPv6 example of DNS simultaneous resolution with IPv4
(sponsored by IIMC/ACCMS, Kyoto University)
• 2013: Riak on amateur radio data analysis
(sponsored by Basho Technologies)
• 2014: Star Trek game revisited
(this year no sponsorship - I paid all by myself - thanks Erlang Solutions for accepting my talk proposal!)

The topic list shows how broad range of subjects and topics which Erlang and OTP can cover.

Highlights of this year

There are so many things I have to write about for Erlang Factory in this year 2014. Listing the topics (before I forget) here:

• My erltrek implementation of the Star Trek game (immediately modified into a completely new thing, with a great help and massive contribution from Andreas Stenius)
• Erlang Foundation and Intermediate Certificates
(Foundation: passed, Intermediate: failed.)
• Concuerror (invitation-only) workshop by Kostis Sagonas
• Elixir tutorial by one of the Erlang Gang of Four, Robert Virding
• Fréd Hébert's presentation and autograph on LYSE
• ... and many more

So let's start from the erltrek.

Why Star Trek?

The ultimate reason is: no one seemed to did it. Period. It was a completely improvised idea, after I was browsing the FreeBSD old games library, reading the BSD Trek code, written by Eric Allman (one of the Sendmail people), in ANSI C89. The BSD trek's last update date was in 1993, so it was a 20-year old code. I decided to rebuild the game in the following procedure:

• Read the C code
• Port it to Lua so that I can at least read through all the C functions
• Rebuild the Lua code (luatrek) into more Erlang-ish one

Was the strategy successful? I think it was not bad, but the part of rebuilding into Erlang was revealed utterly insufficient, soon after I opened up the code. I will describe the reason later in another article, but the main reason of failure was that my thinking and design of the code were completely procedural, neither fully functional, nor Erlang process-based. I have to write that I've been doing the coding and reviewing of the erltrek software even after I come back to Japan from the Erlang Factory event; that is one of the reason why this article gets so late. It's still a work in progress, as of 31-MAR-2014.

Venue

Marines' Memorial Club and Hotel again hosted the Erlang Factory. In fact they also hosted a part of JavaScript event series called JSFest at the theatre (a very nice and large one) on Sunday 9-MAR-2014. There was no fire alarm this year (as it happened in the 2011 and 2013 events.) I didn't socialize much, as I don't as usual (I'm not really an extrovert person), but this year I had a few dinner hangouts with the Erlang all stars and I really enjoyed them. The hotel service was very good and my favorite chicken-flavored S.O.S. Oatmeal was always there at the breakfast; and the catering was very good as well.

People

I met a lot of people. Monika, Andra, and Alison of Erlang Solutions, and all the staff members including the volunteers, kept running the conference and all the other events very smoothly. Meeting with old Erlang friends, especially the Francophones including Fréd Hébert, Loïc Hoguin, and Benoit Chesneau, was always intriguing (especially when the three were having a very heated discussion in French.) Greeting Basho people was also nice. I will not mention all the people in this article, but it was nice to meet a newcomer from Japan, Keisuke Takahashi; he showed a strong interest into the Elixir language and I'm sure he'll drive a community in Tokyo soon.

[To be continued in another article]

Learn You Some Erlang for Great Good! (a review)

Learn You Some Erlang for Great Good! (LYSE) is one of the best books to learn about the programming language and the system Erlang/OTP. This article includes a brief review about LYSE of No Starch Press version, my impression about the author Fred Hébert (Fred), and related miscellaneous things.

Fred is a young person. I assume he was born in the late 1980s. That itself makes LYSE extraordinary. The writing style of LYSE is purely casual and conversational; it's quite different from other Erlang classics, such as Joe Armstrong's Programming Erlang (I'm eagerly waiting for the Second Edition!) and Erlang Programming of Francesco Cesarini and Simon Thompson (I'm also eagerly waiting for a new book from the authors with Robert Virding this time!), let alone the first book of Erlang called Concurrent Programming in Erlang published in the 1990s. Those classic books are written by much older people, and they follow the traditional textbook format. LYSE does not. You will also be surprised by the illustrations drawn by Fred. Those drawings are so creative that I've even got distracted with them while reading the otherwise very technical and detailed contents. Maybe I've got too old. Nevertheless, you need to know LYSE is not an ordinary textbook.

I've seen no book on Erlang about the complete coverage of the language and the OTP library other than the online manual at erlang.org and the source code repository at GitHub. LYSE is not a manual either; it is rather a collection of live stories and practical examples of Erlang based on the hard-earned experience of Fred himself. It's written for those who actually write the code, develop the packages, and release the products. For those who want to study from the very beginning, I recommend Simon St. Laurent's Introducing Erlang.

I will avoid digging into explaining the whole contents of LYSE, because the explanation will take the same amount of words the book has. LYSE is a thick book which has approximately 600 pages, so it's not something for an easy reading. I have to confess there are many things I didn't know and I haven't tested yet in the book. The book hyper-comprehensively covers the necessary topics for dealing with the day-to-day tasks on Erlang/OTP development, from the language basics to gen_server, (the rage against the) finite state machines, package release, Mnesia and the OTP internal database modules, and a proposal of how to read all the Erlang punctuations in English.

In LYSE, the semantic details of the language and system elements of Erlang/OTP are meticulously well-written. One of the most impressive contents is about the intentional avoidance of tail-call optimization in the try-catch exception handling; Fred simply writes:

The protected part of an exception can't be tail recursive.
(Chapter 7, “Protecting The Right Thing")

I will not explain the reason here, but this sort of caremad attitude in the details makes LYSE a professional handbook, if not a textbook (and it is not, I repeat.)

Fred is a very talkative and energetic person. I really admire the level of his English fluency especially when he makes a thunderstorm-like (real) lightning talk. He is also helpful and has assisted a lot of newcomers as his erlang-questions mailing list articles show. The book represents his personality very well. If you are an experienced programmer, LYSE is also an interesting book as a general reading, because the book will remind you of how you have been self-teaching yourself.

I recommend LYSE to all those who want to learn Erlang and OTP, especially to those who want to experiment and self-learn the language and system. If you want to know the contents first, check out the free online version.

Appendix: about the book title

When I saw the title of LYSE first time, and when I saw the title of the cousin book of LYSE about Haskell called Learn You a Haskell for Great Good! (LYAH), I thought there was something severely wrong with the grammar, even from a speaker of English as a second language. I've learned at school that the verb learn will not take a person as the object, and so does my New Oxford American Dictionary built into OS X say. I had a hard time to explain the meaning of this sentence to some Japanese experts who are not familiar with this type of non-standard English sentence. And I have to confess the phrase “Great Good” looked non-standard to me too.

On the other hand, the title of LYSE looks much more understandable to me if I see it as a literal translation of a non-English Indo-European language. Fred is a proud francophone Quebecker, and he is also an excellent English speaker and writer. I can imagine Fred reads and understands the title of LYSE in a complete different manner than I do. So I will complain no more about the title of LYSE or LYAH.

Disclaimers

I've been sent a review copy of LYSE from No Starch Press on January 2013. In this article I mostly refer to the No Starch Press version, though I also have read the online free version. Note well: I have ordered and purchased my own copy of LYSE from No Starch by myself before receiving the review copy!

I should also note that I've met Fred at least three times at Erlang Factory SF Bay Area 2011, 2012, and 2013, so this review may be heavily biased. (I hope this review is not too late, Fred!)

Acknowledgment

Thanks to Jessica Miller of No Starch Press for sending me a copy of LYSE and helping me about what to do with the 600 pages of paper :)

The Setup

(Photo: Kenji Rikitake, taken by himself, at his home workstation, 30-MAY-2013, while watching Joe Armstrong's article of The Setup)

Disclaimer: this post has been inspired by the following website: The Setup.

Who are you, and what do you do?

I'm Kenji Rikitake, network and software engineer. I'm a developer at Basho Japan KK, a group company of Basho Technologies, since February 2013. People expect me to write Erlang code and I like doing it, but I also write programs in C, Python, and other languages, documents, corporate blog articles, and doing almost everything I can to help my coworkers, and of course for helping our customers and friends.

I've been working on computer networking, internet, and the distributed systems since 1988, from VAX/VMS SMGRTL screen management library development, to the campus network security management as a full professor at Kyoto University. I decided to get back to where the action is on November 2012, when I experienced a mental breakdown, after spending 12 years at the research and academic ivory towers. I became a professor dropout spending unrewarded time of 2-year-and-10-month non-tenured full professor from April 2010 to January 2013.

I am also an amateur radio enthusiast and electronic device builder since 1976. My Twitter handle @jj1bdx is the Japanese primary callsign. I also have the US FCC amateur radio license with the callsign N6BDX. Morse code telegraphy and shortwave long-distance contacts (DXing) are my favorites, and I'm also interested in digital radio communications technologies, as I did in the late 1980s with UHF TCP/IP links.

What hardware do you use?

Since February 2013, Apple MacBook Air 11" on the move and Mac Mini at home, both with SSD, are my workstations. Before then I hadn't owned any Apple Macintosh computer since 1980 for myself. My first computer was Apple ][ in 1979 which I abandoned in 1982; I now realize many things have changed. I have even bought an Apple TV for my living room display and it's quite entertaining.

I've been solely using PFU's Happy Hacking Keyboard (HHKB) and the family products since 1997. Prof. Eiiti Wada, who was my advisor when I barely got out of University of Tokyo Graduate School of Computer Engineering, designed the keyboard. Currently I use both HHKB Pro and HHKB Pro 2. I'm a conservative guy and still stick to my Logitech/Logicool Anywhere MX wireless mice, both on the move and at home, while I'm quite satisfied with MacBook Air's trackpad with a click.

Recent addition to my desk is Ergotron's Workfit-S Sit-Stand Workstation, combined with EIZO EV2436W 1920x1200 pixel monitor. Being able to work both on standing and sitting positions, with adjustable keyboard and display heights, makes my legs and shoulders much confortable and reduces a lot of stress. While this desk attachment is not the perfect solution, this is definitely worth trying. I also add a small Home Erecta metal rack to change a sitting desk to a standing writing desk; this is also good when you want to write on a paper at the standing position.

I still have Windows 7 and Ubuntu 13.04 dual-OS notebook client PC, which was the main workstation before I joined Basho. For the software experiments and development of both work and play, I rely on FreeBSD PCs since 1990s; now I run two 9-STABLE machines. I've got rid of tower PCs, and the recent Intel DC3217IYE micro desktop is working smoothly with the FreeBSD OS.

I'm not an audiophile fanatic, but I like good sounds and music. Styleaudio's CARAT-PERIDOT is one of my favorite USB headphone amplifiers. AKG K240 Studio is so far the finest headphones I mostly use at home, with a custom cable from Oyaide in Akihabara. I also use Sennheiser HD580 and Etymotic Research ER-4P. My spouse Kyoko wants to play her Super-Audio CD (SACD) disks, so I also set up a Pioneer D6 SACD player at our living room. Mackie 1202-VLZ3 is also one of my favorite audio mixers.

While on the move, a 6th gen iPod Nano, and Audio-Technica ATH-CKS90NC noise-cancelling earphones (available only from Japan) are the must, both in the airplanes and train cars. Recently I start storing music without compression, that is, with plain WAV files. The sound has got far better than those encoded in 320kbps MP3. I was carrying around a CD Walkman in 1987, and technology has made a lot of progress.

On amateur radio, ICOM IC-7200 has been my favorite shortwave transceiver, rock-solid to stably pump out 100W to the antenna. I also use Idiom Press K-5 keyer for sending Morse Code with the paddles. I sometime hand-send the code with a Czech Morse Straight Key made in 1950s. I only use a temporary antenna system at the balcony of my house, with a 6.5m-length vertical telescopic fiber pole with the antenna wire attached, and ICOM AH-4 antenna tuner. I can still work the six continents and many remote places in the world. The ionosphere is always amazing.

And I also have a Sony Ericsson Android smartphone (but not used as a voice telephone in Japan due to the subscription clumsiness and legal hyper-restriction towards SIM-card-only sales), and a Sony Ericsson feature phone for the carrier au (by KDDI). They have replaced most of the needs for a dedicated camera, but I still have a compact digital camera called Nikon COOLPIX P310.

I like pens, pencils, and papers, too. Moleskine is good, though not so good for writing with fountain pens. LAMY ballpoint pens and fountain pens are very good. Pentel's PRESS MAN 0.9mm mechanical/sharp pencils are robust and my favorite since 1980s, with JIS 2B-grade lead refills.

And what software?

I've been a BSD UNIX fan since 1986. Some of the real BSD-based UNIXes I've used are VAX 4.2/4.3BSD, DEC ULTRIX, SunOS 4, BSD/OS, and FreeBSD. BSD always rocks and rules, especially since AT&T finally loosened up the license for the general public (and accepted what they stole from the Berkeley CSRG team) in 1990s. I really admire FreeBSD core developers and the port maintainers. While I do not register myself as a committer, I still occasionally help maintaining Erlang/OTP port for FreeBSD.

Linux is not bad, and Ubuntu is a nice distro, but I think the kernel is cluttered with too many drivers, though this is inevitable when thinking about how many devices it has to support. And the lack of usable man pages is the primary reason why I don't want to use Linux for programming unless absolutely needed.

I had long been a serious hater of Macintosh computers before Apple decided to introduce Mach/UNIX kernel onto it. I believe Mac OS 9 and older kernels were practically useless for any real-world production systems. And I also had long been a hater of Windows, before the multi-tasking stability was realized in Windows 2000. Having said those rants, I have very little problems on using Windows XP and Windows 7, and Mac OS X Mountain Lion (10.8.3), so long as I don't write native programs on those machines. I intentionally stayed away from Windows Vista and I'm very happy I didn't have to touch it at all. And I will not touch Windows 8 either, because Mac and the OS X look much better as a desktop.

I still have some nostalgia towards OpenVMS, which has a very consistent and rigid command line interface and internal system service architecture. I would get one if Hewlett-Packard would make it an Open Source operating system, which I also believe what would never happen.

About editors, I use both vim and Emacs. I don't want to commit to the meaningless wars of editor preferences. Currently I don't use other dedicated programming editors, though for writing this article, I solely use a web application called Writebox, which is nicely designed to avoid distraction as possible. Windows' Notepad is not bad either, provided choosing the appropriate font and screen size.

For Japanese input system, I solely use SKK, since 1990s. OS X has AquaSKK, and Windows has SKKIME. Emacs has DDSKK, and vim has skk.vim. Until recently I was using EUC/Japanese for the internal encoding system; I decided to convert to UTF-8 finally after I decided to leave Kyoto University on November 2012.

On web browsers and services: Chrome is almost always used at any time on OS X, Windows, and Ubuntu. Or even Chromium on FreeBSD if needed. Diigo has been my primary bookmarking service. GitHub is extensively used for both work and play. Also Google Drive, Calendar, and Mail, though I still maintain a primitive mail processing environment at home. Recently I'm using Feedly as an RSS reader. Music services: Soma FM and SoundCloud. (YouTube and the Amazon sites are intentionally excluded from this list :))

Microsoft Office Suite (Word, Excel, and Powerpoint) are necessary evil. I install them on my Windows 7 machine and the Macs. I believe you cannot substitute by non-MS products, if you have to deal with the Japanese governmental agencies and large corporations, who are solely dependent on the MS products (and know virtually nothing about anything else.)

I still feel LaTeX does much better than MS Word.

Apple stuff: iTunes is not bad, especially for OS X. Keynote looks very good and easy to use.

Adobe stuff: Acrobat is installed on my Windows 7 machine, to proofread/check the PDF document structure and embedded fonts. I once used to play around with the Photoshop Elements, but now I feel OS X's Preview software is enough for playing around with the photo files.

Social services: Twitter: @jj1bdx - no need to explain, isn't it? Tweetdeck is also nice. I quit Facebook and LinkedIn because those two services were too socially demanding (and attracting too many spammers.) HipChat and Yammer are two nice things at work, and I feel those services should be outsourced as much as possible, because the providers are almost always doing better than us. And about blogging: Blogger and Tumblr.

Finally on programming languages and environment: Erlang/OTP for the obvious reason - Riak runs on it. kerl is a good stuff to switch between different Erlang/OTP execution environments. Python (with virtualenvwrapper), Perl, and awk are my favorites too. And of course, C, for fixing the FreeBSD kernel code. Programming tools: rebar, git, make, and other trivial ones (e.g., GNU Autotools). Shells: /bin/sh (NO bash assumption please though I use it too) and zsh.

What would be your dream setup?

Systems that don't need regular human intervention for maintenance. Period.

Frankly speaking, I don't want to provide unpaid support or even paid support regardless of the price, especially to my spouse Kyoko, and other relatives, for broken computer and electronic systems. It takes too much time, very interruptive and distracting, and it is indeed a very stressful work. Of course this doesn't mean I want to leave them alone in panic - I want to help them, but doing so simply makes me weary. I believe the supplier and vendor of those products should also take the responsibility to build the systems self-recover, self-restart and self-fix, at least for the software.

While I'm OK with fixing systems and I think I like doing so by referring to my professional records of being a system administrator for more than 20 years, most of the so-called appliances are quite badly designed and have to have a lot of insider skills and expertise just to use them, let alone to fix them.

I believe in the hacker movement and I think hackers should be well respected in the society and that the status quo is quite against the well-behaved hackers by assuming all hackers are malicious which is absolutely incorrect, but this does not mean everybody has to be assumed to have necessary skills to repair a broken machine or system. Risk dumping of technologies or incomplete systems towards users by the manufacturers is not an ethically right thing to do.

I hope Erlang/OTP and the philosophy/perspectives on the general programming will make the world a better place, for building a robust system, not only for the servers but all software-based products.

On amateur radio: I want to have an unlimited time and access to a legal-limit transmission system and a broadcasting-station-grade shortwave antenna, at a place with the least man-made noise. I want to listen to my voice bounced back from the moon too.

About my disability: I've lost my right eye's lens for more than 30 years. I wish I can connect the mostly-working retina of my right eye to the computers so that I don't have to tolerate the optical blurriness. The Google Glass in May 2013 is not enough yet. And I hope things will get better soon.

Update: Fixing typo and adding product URLs (last update 1-JUN-2013)

Marunage - a dark side of omakase

If Rails is omakase, Japanese construction and IT projects are marunage.

When I first heard about the popular readout and the written article of Rails is Omakase, I immediately thought about another common Japanese word called marunage 丸投げ (pronounced mah-rue-nah-gay), which literally means passing everything onto someone else and completely getting rid of the things which you passed onto.

The actual usage of the word marunage is more vicious and exploiting; large corporations and government organizations often do the marunage to their contractors. When you do the marunage of a project, it means you are getting rid of all the responsibilities and liabilities from the project and letting someone else take care of the mess and the aftermath. In this sense, marunage is a form of risk dumping, or simply a bad form of delegation. The analogy of marunage is also quite common for the national projects. For example, the ministries do the marunage to the research subsidiaries.

I believe the word marunage originally comes from the construction industry, which is very popular in Japan for multi-level delegation from a large general contractor aka zenekon ゼネコン (pronounced zeh-neh-con) to the smaller and powerless contractors aka shitauke 下請け (pronounced shi-tah-ooh-keh). The business model here is that the shitauke companies do the most if not all the actual tasks, and the zenecon does little things other than taking brokerage fees. Multi-level sub-contracts are also popular; for example, in Fukushima Dai-ichi Nuclear Power Plant cleanup project after the meltdown incident on March 2011, ten levels of sub-contracts were revealed.

The mindset of zenekons enslaving the shitauke companies is also common in IT industry in Japan. The IT zenekons, also known as shisutemu integureitaa ("systems integrator" aka "SIer"), have been historically accused of making questionable practices on winning large-scale project contract bids, such as winning a contract by proposing unrealistically budget. For example, in 1989, there was a case of major SIer repetitively tried to win bids by proposing one (1) yen (JPY) as the budget for a whole project of municipal governmental systems. Also, those large SIers or IT zenekons maintain strong relationships with the large-scale customers, and gain profits by doing the marunage of the projects to the shitauke companies. I won't dig deeper in this dark side of Japanese IT history here, but you may find many similar cases.

Getting back to the definition of marunage, I believe that the mindset of marunage represents the feverish risk-aversion towards everything among Japanese people. Still many if not most workers in Japan think a single failure means the professional death which cannot be recovered later, and the legal and economic systems are very harsh against people who have the failure records, for example, a bankruptcy. This kind of culture forces the decision makers to do the marunage to avoid risks. And I see the culture of marunage shows how Japan is more and more closely like the former Soviet Union, one of the most bureaucratic and stagnated nations, disbanded by the economic failure.

So when you hear about marunage, you've got to be very careful; don't become the victim, and try your best to avoid becoming an oppressor by doing the marunage to the powerless subordinates, colleagues, coworkers, or even your family members or bosses.

Computing with only one eye

My right eye has not been really functioning since 1982, when I was 17.

I wore cataract first in May 1982 while I frequently hit my right eye due to the strong itchiness caused by chronic skin allergy. Eventually I lost my sight from the right eye, and after a few eye check the retina detachment was discovered. I was immediately thrown into a hospital room, locked onto the bed with keeping my face up against the gravity (so that the gravitational force helped preventing further detachment) for three weeks, in June 1982. It was an awful experience for a young teenage boy.

I had to have another surgery on September 1982 because of the unfixed detachment again. Later I had another surgery to fix the squint in 1983, caused by the previous surgeries. And finally in March 1986, the swollen and white-colored lens cell (yes, the lens is a single cell) which had lost the transparency in my right eye finally got taken away.

Fortunately, the retina (not the Apple's high-resolution display) in my right eye is mostly functioning OK, though I really can't read anything with it, and I can no longer accurately identify the 3D depth any more. I usually ignore the sight coming from my right eye other than the level of the light.

I've been with the usability problem caused by the broken right eye for many years, long before I started professional programming. Fortunately I've learned how things get large and small when their distance changes, so I can sway to avoid getting hit from most of the things. I still can enjoy 3D movies, though I won't accurately focus anymore.

Things coming towards me from the right side, however, are hard to recognize. I've stopped operating any vehicles including bicycles; I don't have a driver's license either. You don't want to get into a trouble which you can easily expect, especially where the driver's seat is on the left side of the car, like in the USA or Canada. Japan is not, but that doesn't alleviate the problem.

During the age of 80x25 character terminals when DEC VT100 or VT220 terminals were popular in 1980s, my right eye disability didn't cause much trouble, because the whole screen fit very well in the visual space I could recognize. Even in the age of where 14" or 17" CRT or LCD displays, where the default aspect radio was 4:3 or 5:4, I didn't have much problem. I've written a lot of document and many pieces of code; fortunately none of them mandated me to tackle the three-dimensional loss of perception.

I have a few problems regarding my sight disability since the beginning of the 21st century, unfortunately. One of them is the color design; most web sites choose white as the background color, which interferes the black foreground characters I read. I try to choose darker background colors and brighter foreground colors as possible, but not all web applications allow to do so.

Another one of the 21st century problems is the change of default aspect ratio of the screen to 16:9; this is too wide for my rather narrow sight space. If you are watching a movie this wouldn't be a problem. I welcome high-resolution displays with crispier pixels and accurate positions. But when you write a text, this will be a serious problem, because you can't see everything at once. I always choose narrower horizontal size to keep the screen in my left eye's sight range.

So far I can manage the environmental change of computing in the past 31 years. I expect, however, the future computing models, including those require wearing glasses or assume three-dimensional recognition ability, may eventually reject me from the cutting edge of computing. Maybe I can ask Google to make a special Google Glass to compensate the loss of my right eye's lens and focus directly into my right eye's retina, with proper size adjustment so that I can see the things in two eyes again.

And I can assure everyone that you can write the code with sight disabilities; I know a professional programmer who has completely lost the sight too.

(Thanks Tyler Hannan, who suggested me to write this. I had been hesitating to write about my weakness on the web, but I thought this would be helpful to others.)

I've joined Basho Japan from February 2013

I've started my new career as a Senior Software Engineer at Basho Japan KK, a 100% subsidiary of Basho Technologies, Inc., effective 1-FEB-2013. I'm very happy that my colleagues are all very much supportive for my startup as a new developer. I will try my best to catch up the lean software development process and to get the most out of both the elementary and advanced technologies for contributing to the company.

Basho is a wolf pack of the bleeding-edge Erlang/OTP and distributed system engineers, who have made Riak as an open source standard of highly-distributed and fault-tolerant databases. Catching up with the skilled engineers is a really tough job, but it's also very exciting to take the new challenges for making the actual software products to the customers. This is completely different from what I've experienced in the past 12 years as an academic/government researcher.

Basho's customers have a lot of demands for Riak already, and the company including myself has to meet the expectation while shipping the products on time and maintaining the high product standard. I will contribute my 22-year skills of network engineer, sysadmin, and an OS developer to the software development process, in the most effective way.

My participation to Erlang/OTP activities and ACM Erlang Workshop will remain the same.

So stay tuned for my pull requests and pieces of code!

My technical interests for Erlang/OTP as of April 2011

This is a list of my technical interests for Erlang/OTP as of April 2011:

• Catching up the latest cryptographic algorithms, including hash functions, shared-key cryptography, public-key cryptography, pseudo random number generators (PRNGs), while making them efficient in parallel/concurrent execution environment;
• Delay tolerance network technologies which can handle days, weeks, months of message delivery delays;
• Fundamental support functions of IPv6, especially for Distributed Erlang and epmd, and cryptographic transport support with SSL/TLS;
• DNSSEC-compatible resolver implementation for inet_res module;
• Large binary heap handling of BEAM, and the programming methodologies for the faster NIF implementation, including the performance measurement;
• PRNGs with small context/internal-state size, e.g., Wichmann-Hill 1982/2006, LShift, XOR32;
• PRNGs with large context/internal-state size, e.g., SFMT;
• TAI or leap-second handling by NIF;
• Privilege separation techniques between BEAM instances (e.g., virtualization, FreeBSD Jail); and
• the effective usage and choice of port drivers, linked-in drivers, and NIFs

And projects I'm interested in (but not limited to):

• Disco Project, a MapReduce implementation
• ZeroMQ and other general queueing systems
• UBF, BERT-RPC, and other messaging formats
• CouchDB

And don't forget I'll be hosting the ACM Erlang Workshop 2011.

(This article is a revised version of http://heikou-konton.blogspot.com/2011/02/erlangotp.html (originally written in Japanese.)

3-FEB-2011: IPv4 address pool in IANA is used up

As of 3 February 2011, the central pool of available IPv4 addresses managed by the Internet Assigned Numbers Authority (IANA) has been depleted. See ARIN's FAQ for the further details. Since July 1994 when IPv6 was first recommended by the IETF, Internet engineers are periodically warning people that IPv4 address space is finite. And the day has come. Of course this is not the end of the Internet. We need to consider, however, some new issues:

• Internet is no longer a unified network of single protocol; it will become a multi-protocol network of IPv4 and IPv6 shortly.
• While quite a new technologies have been developed and are under deployment for the migration to IPv6 (i.e., dumping the old technology of IPv4 and replace it by IPv6), we will still have to endure the fact that IPv4 will be staying around for another few years, or even more than 10 years.
• Promoting IPv6 is essential. I think, however, maintaining the inter-connectivity of the two networks and the services provided there from both networks is even more critical. We don't want two split networks; we still want one unified Internet, don't we?

Erlang and Github

Erlang/OTP is now officially maintained under the Github repository, since the release R13B03. I think this is a milestone for the language, because the Ericsson development team finally decided to show the interim results of what they are doing for the time being.

One of the characteristics I like about Erlang is that the language specification and libraries have been maintained by a single entity called Ericsson's Erlang/OTP Development Team. I do not want an anarchy for computer language and operating systems. I prefer BSDism than Linuxism in this sense; I think pieces of code should be rather controlled by the core people while sufficiently accepting improvements from the other developers.

The old Erlang/OTP daily snap archives, however, are no longer sufficient to catch up with the daily development cycles. And many non-Ericsson authors have put in their patches into Erlang, including mine. So there had to be some systems to accept user feedbacks.

Using an open repository system such as Github is a wise idea for incorporating new code into Erlang/OTP, and showing the official status of modifications. Git is flexible enough to allow per-user and per-purpose branches. And Github allows forking between the users. The Ericsson's Team doesn't have to build and publicize its own code repository system for Erlang/OTP, which will cost them significant amount of human and financial resources.

And now I have an official requirement to learn Git; to catch up with the Erlang/OTP development cycles.

Current and outdated references of secure C programming

C is the modern assembly language for many architectures, and still the most useful computer language for me. C does not have a rigid grammar and has a lot of variants and local dialects, and have revised a few times including the old UNIX C, ANSI C 1989 which first introduced prototypes, and C99. Finding out the de-facto standard elements of C is a complicated work. You can find a bunch of different indentation and writing styles on C code. I do not recommend a specific coding style in this article; I can only recommend you need to follow the mainstream style when working in a project. Sometimes you have to read the books for discovering what is the most right thing to do. I recommend following books for C programming now:

• C: A reference Manual (5th edition) (2002), which has a lot of precise and detailed examples on ambiguous usage;
• Secure Programming Cookbook for C and C++ (2003), which explains what to do and what not to do to write secure code; and
• Advanced Programming in the UNIX Environment (2nd Edition) (2005), which explains usages of modern UNIX system calls and libraries.

For practical programming, however, depending on books is not enough. Actually those books I recommended above are 5 to 7 years old as of 2009, so if you want to know the cutting-edge details of programming, you should read the latest software. Consulting a C compiler manual and well-written source code such as that of BSD kernels is a must if you want to write an efficient code (those are freely available). One thing to which you've got to pay special attention is that books are eventually but surely getting outdated. Books are not the Web articles; they are static and will not change. The lifespan of a reference book for computer science is typically very short these days, due to the rapid change of technologies. Books about C is not an exception either. And I should confess that a few days ago I decided to sell the following old worn-out books because I found out them simply outdated (and I no longer recommend the following two books any more):

• The C Programming Language (2nd edition) (1988)
• The Standard C Library (1991)

The reason that I found them outdated were as follows:

• They are old, written in approx. 20 years ago, and they do not reflect the changes of C99 and other additional elements;
• Not mentioning secure programming at all, including
  • avoiding reference to non-existent data objects,
  • preventing buffer overflows,
  • limiting the length of a string;
  and
• The C library structure and source have been changed a lot for these 20 years.

Frankly speaking, I loved those old books, especially which I referred to the most during my apprentice time of learning the language in the late 1980s. Those books were the only source before the Web. I had to repeatedly read the old bestsellers many times to discover the details. I do respect the authors of those books. They are pioneers of UNIX and C programming. Nothing is eternal, however; and I suggest use to stop using outdated reference books ASAP for every subject, not only for programming.

The risks of systems left alone and untested

Computer systems left alone unmaintained are a premier source of risks. Those systems may cause a serious crisis and a major service disruption.

On September 14, 2008, All Nippon Airways (ANA), a major Japanese airline, caused the disruption of the ticketing service due to the cryptographic function software expiration (as they announced in the Japanese press release), which is logically assumable about the PKI certificate, according to the other Japanese-written press reports like ITmedia's and Nikkei ITpro's.

The chilling fact revealed was that the ANA left the cryptographic function unused for 2 years and did not make a review about expiration at all when they activated the function for the terminals used by the ticketing agents. This is an awful example of software development indeed.

I wrote about the service disruption for RISKS-DIGEST 25.34 just after the incident occurred.  And recently I knew the article was quoted by another blog article yesterday.

The expiration issue is not only about the PKI certificate; domain name registration is another source of expiration risks. An expired domain can be abused for phishing and overtaken by attackers.  Software license is another good example.  In general, Expiration is a part of overall misconfiguration.  So when did you review the expiration date of the system resources under your control last time?

Why choosing Erlang for concurrent processing

A few weeks ago I was reading a blog article called Erlang, the next Java. While I agreed with the author's views, I felt something was bugging me in the article, which eventually lead me to write this blog article.

I didn't choose Erlang just because it's a functional language. I have chosen Erlang to learn because Erlang programming requires and has to enforce the programmers to write the concurrent code. Concurrent programing should minimize the number and size of critical regions and bottlenecks. And I think writing such programs is very difficult without the assistance of the programming language.

Many programming languages claim or plan to be capable of running a part of code based on concurrent processing. For example, Ruby will incorporate distributed storage mechanism called Roma and the task administration subsystem called Fairy. Another good example is Haskell concurrency: it is implemented as a language feature called Parallel Haskell.

I think, however, introducing concurrency while allowing programmers to write code to use shared memory will cause a lot of problems. Joe Armstrong has already described his concern on shared memory in his blog. I support his arguments; those arguments an important part of the reason why I decided to learn Erlang.

I should add another problem programmers will face when dealing with code allowing shared memory; rewriting the code for removing shared access to run it efficiently on concurrent environment will be an incredibly difficult task.

Unfortunately, most existing languages have already had a lot of code written assuming shared memory areas. For example, C code with extern variables implicitly assume those are shared between the functions in all source code files linked together. I assume I can hardly find any set of C code without using an extern declaration. I have learned that even Common Lisp has special declaration for the variables, which allows multiple functions share the same object, out of lexical scopes.

Another example of shared-memory concurrency is the operating system (OS) threads. C/C++/Java threads inherently share the parent OS process address space and environment in common. Python has the Thread Objects. While OS threads often ease implementation of concurrent servers by reducing the task switching time, the semantics is implicit and error-prone.

I understand and agree that sharing objects itself cannot be completely eliminated under the read-world constraints of processing and communication timings between programming language functions. I think, however, that programming languages should help the programmers to minimize writing code including shared memory areas, which will turn themselves into critical regions.

Erlang imposes necessary restriction on avoiding implicit data sharing between functions by:

• prohibiting multiple variable assignments in a function;


• enforcing and helping the programmers to conduct message-passing programming between functions, by not providing any implicit data-sharing facility between the functions;


• providing fast task-switching capabilities, by giving the definition that functions are the minimum concurrent execution units; and


• restricting the usage of data-sharing facilities between functions to the minimum, such as process dictionaries, ETS, DETS, Mnesia, and the global naming service shared between connected Erlang nodes, by requring explicitly writing so in the code.
  

In short, I think adding concurrent features is not enough for concurrent programming; prohibiting non-concurrent habits and enforcing writing concurrent programs are necessary as part of the programming language specifications. I believe concurrent programming under coming multi-processor environment is only possible under such a hard-liner attitude to the programmers. I feel programmers including myself are very conservative and rigid to change their sequential-programming habits.

And actually, I was one of those programmers who did not recognize the urgency of learning concurrent programming in 2007.

Programming language design and security

I suspect quite a few programming language designers just don't care about the security of the languages and the systems built by them. Maybe I'm too pessimistic, but a recent experience brought me the idea.

I was an audience at an event called Lightweight Language Future yesterday in Nakano, Tokyo, Japan. I only attended the first two sessions. The second session was a panel discussion about the programming language 100 years later. While the discussion was technically intriguing, I heard not a word about security at all during the panel. Two of the panelists (names withheld because I don't want this article to be a personal attack to them) were the principal designers of popular lightweight languages, so I asked them in the Q&A session about how they would deal with the security issues which would become more serious in the future. One of the answers surprised me; the person told me that VMware would do.

I understand virtualization in various operating system layers, including VMware and FreeBSD jail, is surely a good hack to control the damage area in the whole system to run a public access system. I think it's not sufficient at all, however, to prevent future security incidents; if a programming language is potentially vulnerable to attacks, the systems built by the language will be also equally vulnerable.

I think programming language designers are not exempted from the responsibility of (at least trying to) making computer systems secure. If I could hear more constructive ways of solving the security problems in the answer from one of the panelists, such as:

• protecting the language from the buffer-overflow bugs;
  
• preventing the garbage collector from crashing even in a hostile environment where the external attackers try to deliberately manipulate the pointers or variables to hack into the protected memory area;
  
• implementing a syntax or semantics validation system in the programming language so that the programmers can apply it to validate the external data, such as those from the network-connected users; or
  
• preventing the race condition, deadlocks, or any sort of resource starvation by the intentional or unintentional programming;
  

then I would have been much more convinced. But now I should suspect that quite a few programming language designers just don't care about the security consequences of the features they build into the language.

I don't have a definitive answer to solve the issues of unsecure systems built from unsecure programming languages. All I can write now is that a security-aware programming language will surely help making a secure system. Note that security is not just about privacy and secrecy; it's also about the reliability and fault-tolerance.

Note on 4-SEP-2008: one of my friends later reviewed the panel recording and told me that it was not VMware but VM layer that the panelist answered to me. I still think solely relying on the VM layer is insufficient for building a security structure in a programming language, though.

nsplit available as a Google Code project

My Erlang code nsplit is now available as a part of Google Code projects.

Link on this blog about the code is here.

Learning Erlang

I've been learning the Erlang programming language.

I've tested an IPv6 address-to-domain-name conversion for the reverse lookups. I made a presentation at the 1000speakers:4 conference on April 26, 2008. The presentation slide set in PDF is here.

Teleworking technology: we already have it

I've been thinking about months for what technology can help people to discover and get the most out of teleworking. I'm getting closer and closer to the following conclusion:

We've already built teleworking infrastructures, at least in my home country Japan, and many other nations and regions which have already deployed the Internet. Period. We need to focus on how to break the barriers and remove the impediments to people to embrace teleworking technologies as their own tools.

Let me simply put in this way: teleworking is no longer something special. For example, writing this blog itself is a very good example of teleworking technologies. All I need is a decent Web browser capable to run JavaScript for Blogger's writing environment; and I can publish and show you what I write, like what you read now.

Collaboration is getting more and more easier also. LUNARR allows you to flip back an electronic document and let you and other people write something on the back. This is a rather intuitive way to co-author a document, which I had never thought about until I saw it. Geeks will do the whole things in more geeky ways using SSH, rsync, CVS, subversion, or whatever else for software development, but it's not only for geeks anymore. Anyone who has to write a document can do it on a teleworking environment.

So let's stop saying "I'm not allowed to work from (wherever you want other than your office) because blah blah blah..." and talk how you can leverage your productivity from introducing teleworking into your lifestyle. If you and your employer think teleworking is something restricted for special people, that mindset is archaic, and no longer applicable to the 21st-century version of modern lifestyle. And let's not take commuting as a duty; it should be a choice and it has to be, to reduce the gross amount of time and energy wasted by being forced to commute.

But beware that teleworking is not necessarily a duty either; don't do it like this as in an xkcd article. Don't leave your partner alone just because "something is wrong on/with the Internet" (grin).

The art and limit of dependencies

If you want to get a job done, you need to list up the necessary things and tools, and procedures to use and apply them. In other words, you have many dependencies on them.

In a modern computer software development, no piece of software can withstand without any dependency to other pieces. You do not want to make a C program without the standard I/O library. Dependencies to the tools are also important and critical; if your codes include those written in FORTRAN, C, C++, and Java, you need the four compilers and language execution environments.

Computer programmers have been making the tools to automatically resolve the dependencies. make is a popular one, derived from the UNIX programming environment, which parses the rulesets called Makefile and determine whether if you need to rebuild a result from the source files, by comparing the timestamps. If make finds out one of the source files are newer than the result, it will invoke the command to rebuild the result.

Version control systems such as subversion, CVS, and RCS, are another good examples of dependency management tools. You can save the history of changes on a file, a directory, or a set of directories. You can make a software package by checking out a set of tagged files; you can even make multiple branches of a code.

FreeBSD operating system has its own dependency management system of externally-contributed programs called ports and packages. A port means a set of rules, configuration files, and source codes necessary to build a program. A package means the derivative of the port, built by another computer. Many essential parts of FreeBSD subsystems, including the X Window and Perl programming language, are installed as packages, because they are not considered as the core parts of FreeBSD.

I'm always pleasantly surprised when a very complicated port, such as Japanese version of LaTeX, a typesetting and documentation set of programs, can be built without major glitches, including the automatic installation of depending programs, such as Ghostscript. For most of the ports, FreeBSD volunteers are always doing the outstanding jobs.

I feel very much annoyed, however, when I have to untangle the web of dependencies when the installation/building of a software toolset from the scratch. Unfortunately, Xorg 7.3, a free implementation set of X Window programs, fonts, and tools, was not able to build in my environment. I had to copy three prebuilt sets of files: include files of C/C++, shared/static libraries, and the very basic fonts, to finish building the necessary programs.

In my case of Xorg kitbuilding, some very old fonts existing since late 1980s were not successfully compiled and converted into various ISO8859 part codesets. And I could not build the font handling libraries. This glitch killed the whole automatic compilation task, which was supposed to untangle the enormous list of dependencies. I also found that the target directory name was merged to /usr/local from the traditional /usr/X11R6, and I wasn't sure whether if I moved the old files and subdirectories under the old directory safely to the new directory.

Untangling the web of dependency is a very hard task, since the parameters you need to examine are scattering around all the directories, in all the ports, and you often also have to set an environment variable to do something extraordinary. While I can guess what I should do because I've done a lot of this kind of tasks, I never want to do this at any time because scanning your memory and poking around the files are very much mentally painful. In a recent hostile computer environment, vulnerabilities can easily sneak into such a complicated and unformalized tasks.

How do you manage your computers? Do you care about the dependencies of the tools, programs, configuration files, and other objects?