Erlang and my inferiority complex against functional programming

A few weeks ago in the middle of October 2016, I had some interesting exchange of tweets inspired by an Alvaro Videla's tweet. Alvaro wrote he always told people Functional Programming (FP) in Erlang was a necessity, not part of the original design (emphasis mine).

I later found Robert Virding, one of the founders of Erlang language and concurrent system, explicitly stated in this talk (video at Functional Conf 2016) (at 4m23s) as follows:

We were not out to implement a functional language
We were not out to implement the actor model
We were trying to solve the problem

Robert's statements in the quote above are the exact reason why I started and still like Erlang.

I've already erased the tweets (I keep most of my tweets ephemeral) of my response, but I would like to write my story about FP.

My first exposure to FP was through the books of Lisp in 1980s. I didn't understand the FP principles, but I got rather interested in the symbolic processing capability and the ease of handling data structures.

I'm not interested in slow (i.e., not sufficiently fast for a purpose) systems. In 1980s Lisp systems were running mostly on interpreters, so I have a strong recognition bias that tells functional languages are slow, which is incorrect as of 2016; a counter evidence is shown as a production level documentation tool written in Haskell called Pandoc, and by compiler-based FP systems such as Common Lisp. Nevertheless, in 1980s, FP systems were mostly slow. Until the late 2000s, I hadn't been really interested in FP again.

I've got a very strong inferiority complex when I tried to study something about FP and Lisp from 1988 to 1990 at a university laboratory where the main research topic was about the programming languages and paradigms. I utterly failed, because my main interest then was how to connect the people in Japan and the world, through email, USENET (NetNews), UUCP, and affordable TCP/IP links.

My inferiority complex against FP is still not gone yet. Three PC member roles, Two accepted papers, and one workshop chair role (in 2011) for ACM SIGPLAN Erlang Workshop events were something I would have least expected in the late 1980s or early 1990s. FP people are generally much smarter than me and they are amazing. I have been lucky to meet the talented and productive researchers, including John Hughes, Simon Thompson, Kostis Sagonas, and many others (which I'm very sorry to omit their names here).

On the other hand, however, I still have some feelings that some FP people are too theoretical and not respecting well on practicality of currently programmed system, when I am watching on (mostly Japanese) FP communities, especially those on Twitter. My gut feelings are like this: Theory is nice. Theory is beautiful. Beauty is important and often essential in programming. I think that's what academic and research communities for software are for. But the real virtue of programming prevails on how it solves the real-world issues and problems. There's no side-effect-free world so long as having to deal with I/O devices. Memory is limited. CPU speed is limited. Network bandwidth is limited. Isn't FP a way to deal with these real-world limitations while maximizing the ease of abstraction on programming? (I think Object-Oriented Programming (OOP) is also trying to solve the same issues, but that's another issue; I still need to understand why Java/C++/C#-like OOP languages are simply that much complex.)

Erlang has been one of the rare language systems which didn't impose or force me to do the things in "functional" way, though the FP principles are embedded deeply in the core, such as the immutability and no-shared object/process principles. I think Elixir follows these principles well, though I see some interesting FP technologies in Elixir such as macros and metaprogramming. In short: Erlang (and Elixir) didn't trigger my inferiority complex against FP!

So some trivial lessons I've learned:

• You are never too late to learn new paradigms.
• You are never too late to re-learn things once you failed to learn.
• There's something fits for you even for a category you don't feel like to study.

I think my inferiority complex against FP will stay until I die, but now I don't have to feel FP is not my stuff. And that is good.

My new startup: Kenji Rikitake Professional Engineer's Office

This is an announcement of my new startup business.

I've started my own business, as a Japanese government-licensed Professional Engineer of Information Technology (in Japanese it's called Gijyutsushi), since April 21, 2014. It's called Kenji Rikitake Professional Engineer's Office (KRPEO).

KRPEO provides the consulting and engineering services on information technology in general, including but not limited to: information and network security, the design, deployment, implementation and performance tuning of Erlang/OTP, FreeBSD, and Riak. KRPEO will provide all services in both Japanese and English.

I had been looking for a full-time job since October 2013, and I made a conclusion that the job market near Osaka was mostly for energy-consuming legacy programming tasks due to the fact that most of the high-end software engineering companies were only located in Tokyo. Most of the employers in Japan do not accept remote teleworking due to the cultural and legal reasons either. So I had to lean out from the traditional corporate culture of Japan, for a more sustainable business model, making myself, my family, and my customers happy at the same time. I understand this is a hard challenge, but I will definitely take it.

My involvement in the open-source developer and network operator communities in Japan and the world will remain the same and unchanged.

The English announcement site for KRPEO is at http://www.k2r.org/krpeo/. The Japanese site URL is: http://www.k2r.org/gijyutsushi/.

Update 28-APR-2014: add the URL of the English site, correct the URL of the Japanese site.

Erlang Factory SF Bay 2014: list of some notable talks

(Photo: a shot during my talk by Yoshihiro Tanaka, used by permission)

(Disclaimer: there were too many talks I wanted to listen to, but I couldn't.)

Here's the list of talks I thought intriguing for Erlang Factory SF Bay 2014 (links are to the slides or videos):

• Mike Williams on What Programming is Really About
• Brian Troutwine on Monitoring Real-Time Bidding at AdRoll
• Kostis Sagonas on Erlang VM and Term Storage scalability
• Fred Hebert on Planning for Overload
• Natalia Chechina on Scaling Distributed Erlang at a supercomputing environment
• Garrett Smith on Erlang and CZMQ, and the choice of using Erlang Port for CZMQ
• Dave Thomas and Jose Valim dropped an atomic bomb to the audience on how to catalyze the change to get more newcomer and other community's attention towards Erlang/Elixir ecosystem
• Rick Reed on Scaling at WhatsApp

I also wanted to listen to the following talks, and found the slides intriguing:

• Jay Nelson on eliminating single process bottlenecks with ETS concurrency patterns
• Lukas Larsson on Memory Management battle stories
• Louis-Philippe Gauthier on Performance Optimization 101

I will post my impressions for the above talks in later articles. I would like to note some personal impressions for the audiences this year:

• Erlang is no longer an exotic language or system. The audiences want the real solutions and hints.
• Elixir is gaining popularity, and will surely contribute to reduce resentment against BEAM (Erlang VM) and the ecosystem.
• The implementation talks were getting more detailed and hard core, and the questions were also more specific.

Video quality

Thanks to the hard-working video and audio recording and editing team, this year's video quality is very high in overall. While the live streaming was not possible due to the prohibitive cost, some videos were made available within six hours from the end of the talk. I think this was impressive and a practical solution to make a trade-off between the turn-around time for the availability and the quality of video of the talks.

[To be continued in another article]

Erlang Factory SF Bay 2014: erltrek and the beginning

(Badge photo by Kenji Rikitake)

It's already been near a month since Erlang Factory SF Bay 2014 as I'm writing this blog article. It was my fifth Erlang Factory, consecutive once a year since 2010. The topic I've talked are so diverse:

• 2010: SSH RPC
(sponsored by NSIRG, NICT, Japan)
• 2011: SFMT on Erlang
(sponsored by ACCMS, Kyoto University)
• 2012: IPv6 example of DNS simultaneous resolution with IPv4
(sponsored by IIMC/ACCMS, Kyoto University)
• 2013: Riak on amateur radio data analysis
(sponsored by Basho Technologies)
• 2014: Star Trek game revisited
(this year no sponsorship - I paid all by myself - thanks Erlang Solutions for accepting my talk proposal!)

The topic list shows how broad range of subjects and topics which Erlang and OTP can cover.

Highlights of this year

There are so many things I have to write about for Erlang Factory in this year 2014. Listing the topics (before I forget) here:

• My erltrek implementation of the Star Trek game (immediately modified into a completely new thing, with a great help and massive contribution from Andreas Stenius)
• Erlang Foundation and Intermediate Certificates
(Foundation: passed, Intermediate: failed.)
• Concuerror (invitation-only) workshop by Kostis Sagonas
• Elixir tutorial by one of the Erlang Gang of Four, Robert Virding
• Fréd Hébert's presentation and autograph on LYSE
• ... and many more

So let's start from the erltrek.

Why Star Trek?

The ultimate reason is: no one seemed to did it. Period. It was a completely improvised idea, after I was browsing the FreeBSD old games library, reading the BSD Trek code, written by Eric Allman (one of the Sendmail people), in ANSI C89. The BSD trek's last update date was in 1993, so it was a 20-year old code. I decided to rebuild the game in the following procedure:

• Read the C code
• Port it to Lua so that I can at least read through all the C functions
• Rebuild the Lua code (luatrek) into more Erlang-ish one

Was the strategy successful? I think it was not bad, but the part of rebuilding into Erlang was revealed utterly insufficient, soon after I opened up the code. I will describe the reason later in another article, but the main reason of failure was that my thinking and design of the code were completely procedural, neither fully functional, nor Erlang process-based. I have to write that I've been doing the coding and reviewing of the erltrek software even after I come back to Japan from the Erlang Factory event; that is one of the reason why this article gets so late. It's still a work in progress, as of 31-MAR-2014.

Venue

Marines' Memorial Club and Hotel again hosted the Erlang Factory. In fact they also hosted a part of JavaScript event series called JSFest at the theatre (a very nice and large one) on Sunday 9-MAR-2014. There was no fire alarm this year (as it happened in the 2011 and 2013 events.) I didn't socialize much, as I don't as usual (I'm not really an extrovert person), but this year I had a few dinner hangouts with the Erlang all stars and I really enjoyed them. The hotel service was very good and my favorite chicken-flavored S.O.S. Oatmeal was always there at the breakfast; and the catering was very good as well.

People

I met a lot of people. Monika, Andra, and Alison of Erlang Solutions, and all the staff members including the volunteers, kept running the conference and all the other events very smoothly. Meeting with old Erlang friends, especially the Francophones including Fréd Hébert, Loïc Hoguin, and Benoit Chesneau, was always intriguing (especially when the three were having a very heated discussion in French.) Greeting Basho people was also nice. I will not mention all the people in this article, but it was nice to meet a newcomer from Japan, Keisuke Takahashi; he showed a strong interest into the Elixir language and I'm sure he'll drive a community in Tokyo soon.

[To be continued in another article]

Learn You Some Erlang for Great Good! (a review)

Learn You Some Erlang for Great Good! (LYSE) is one of the best books to learn about the programming language and the system Erlang/OTP. This article includes a brief review about LYSE of No Starch Press version, my impression about the author Fred Hébert (Fred), and related miscellaneous things.

Fred is a young person. I assume he was born in the late 1980s. That itself makes LYSE extraordinary. The writing style of LYSE is purely casual and conversational; it's quite different from other Erlang classics, such as Joe Armstrong's Programming Erlang (I'm eagerly waiting for the Second Edition!) and Erlang Programming of Francesco Cesarini and Simon Thompson (I'm also eagerly waiting for a new book from the authors with Robert Virding this time!), let alone the first book of Erlang called Concurrent Programming in Erlang published in the 1990s. Those classic books are written by much older people, and they follow the traditional textbook format. LYSE does not. You will also be surprised by the illustrations drawn by Fred. Those drawings are so creative that I've even got distracted with them while reading the otherwise very technical and detailed contents. Maybe I've got too old. Nevertheless, you need to know LYSE is not an ordinary textbook.

I've seen no book on Erlang about the complete coverage of the language and the OTP library other than the online manual at erlang.org and the source code repository at GitHub. LYSE is not a manual either; it is rather a collection of live stories and practical examples of Erlang based on the hard-earned experience of Fred himself. It's written for those who actually write the code, develop the packages, and release the products. For those who want to study from the very beginning, I recommend Simon St. Laurent's Introducing Erlang.

I will avoid digging into explaining the whole contents of LYSE, because the explanation will take the same amount of words the book has. LYSE is a thick book which has approximately 600 pages, so it's not something for an easy reading. I have to confess there are many things I didn't know and I haven't tested yet in the book. The book hyper-comprehensively covers the necessary topics for dealing with the day-to-day tasks on Erlang/OTP development, from the language basics to gen_server, (the rage against the) finite state machines, package release, Mnesia and the OTP internal database modules, and a proposal of how to read all the Erlang punctuations in English.

In LYSE, the semantic details of the language and system elements of Erlang/OTP are meticulously well-written. One of the most impressive contents is about the intentional avoidance of tail-call optimization in the try-catch exception handling; Fred simply writes:

The protected part of an exception can't be tail recursive.
(Chapter 7, “Protecting The Right Thing")

I will not explain the reason here, but this sort of caremad attitude in the details makes LYSE a professional handbook, if not a textbook (and it is not, I repeat.)

Fred is a very talkative and energetic person. I really admire the level of his English fluency especially when he makes a thunderstorm-like (real) lightning talk. He is also helpful and has assisted a lot of newcomers as his erlang-questions mailing list articles show. The book represents his personality very well. If you are an experienced programmer, LYSE is also an interesting book as a general reading, because the book will remind you of how you have been self-teaching yourself.

I recommend LYSE to all those who want to learn Erlang and OTP, especially to those who want to experiment and self-learn the language and system. If you want to know the contents first, check out the free online version.

Appendix: about the book title

When I saw the title of LYSE first time, and when I saw the title of the cousin book of LYSE about Haskell called Learn You a Haskell for Great Good! (LYAH), I thought there was something severely wrong with the grammar, even from a speaker of English as a second language. I've learned at school that the verb learn will not take a person as the object, and so does my New Oxford American Dictionary built into OS X say. I had a hard time to explain the meaning of this sentence to some Japanese experts who are not familiar with this type of non-standard English sentence. And I have to confess the phrase “Great Good” looked non-standard to me too.

On the other hand, the title of LYSE looks much more understandable to me if I see it as a literal translation of a non-English Indo-European language. Fred is a proud francophone Quebecker, and he is also an excellent English speaker and writer. I can imagine Fred reads and understands the title of LYSE in a complete different manner than I do. So I will complain no more about the title of LYSE or LYAH.

Disclaimers

I've been sent a review copy of LYSE from No Starch Press on January 2013. In this article I mostly refer to the No Starch Press version, though I also have read the online free version. Note well: I have ordered and purchased my own copy of LYSE from No Starch by myself before receiving the review copy!

I should also note that I've met Fred at least three times at Erlang Factory SF Bay Area 2011, 2012, and 2013, so this review may be heavily biased. (I hope this review is not too late, Fred!)

Acknowledgment

Thanks to Jessica Miller of No Starch Press for sending me a copy of LYSE and helping me about what to do with the 600 pages of paper :)

My Interview on Erlang by Paolo d'Incau and my Medium posts

A brief introduction on my interview about Erlang/OTP and other tech topics, by Paolo d'Incau. (Thanks Paolo!)
http://pdincau.wordpress.com/2013/06/27/an-interview-with-kenji-rikitake-jj1bdx/

I've also posted some non-engineering essays to my Medium page:

• Things I'm not interested in
• Electric appliance woes

Enjoy reading.

I've joined Basho Japan from February 2013

I've started my new career as a Senior Software Engineer at Basho Japan KK, a 100% subsidiary of Basho Technologies, Inc., effective 1-FEB-2013. I'm very happy that my colleagues are all very much supportive for my startup as a new developer. I will try my best to catch up the lean software development process and to get the most out of both the elementary and advanced technologies for contributing to the company.

Basho is a wolf pack of the bleeding-edge Erlang/OTP and distributed system engineers, who have made Riak as an open source standard of highly-distributed and fault-tolerant databases. Catching up with the skilled engineers is a really tough job, but it's also very exciting to take the new challenges for making the actual software products to the customers. This is completely different from what I've experienced in the past 12 years as an academic/government researcher.

Basho's customers have a lot of demands for Riak already, and the company including myself has to meet the expectation while shipping the products on time and maintaining the high product standard. I will contribute my 22-year skills of network engineer, sysadmin, and an OS developer to the software development process, in the most effective way.

My participation to Erlang/OTP activities and ACM Erlang Workshop will remain the same.

So stay tuned for my pull requests and pieces of code!

Coming change in 2013

(Photo by Kenji Rikitake, with a book cover of "Concurrent Programming in Erlang", which is an on-demand printed version via amazon.co.uk)

I've been away from this blog for quite a long time. This will be a short entry, though I'm writing this to make a mark at the beginning of the year 2013.

I'll be on a big transition in this year 2013. I'll write about the details when available. I can promise that I'll keep focusing on Erlang/OTP and distributed systems, as I have been for the past 5 years. Stay tuned for the details.

Lots of things happened last year. A good news is that I joined the 11th ACM SIGPLAN Erlang Workshop held at Copenhagen on September 2012. Meeting a lot of great people and nice friends there was a precious experience.

A bad news is that I have been taking a long sick leave since November 2012. I will write about the details later when I feel I should, but for the time being I can only conclude that a long-distance commuting literally killed me again, as it had done during 1990 to 1992. I definitely have to change my lifestyle.

On new year's resolution: I won't declare it this year, because I need to drastically restructure my professional life anyway. Of course I've got to get myself into a good shape. And I need to spend more time on code writing and development, which I could not for the past three years.

I hope things will work out in a good way this year; it's a year of Snake in Japanese/Chinese 12-year holoscope, and it's my year too.

Why Erlang people need the Erlang Factory events

Disclaimer: I am writing this blog article on my way home from SFO. The reduced air pressure in the passenger cabin might have affected the contents of this article.

I spent all three days of Erlang Factory SF Bay 2012, with two more days to cope with the jet lag. This was my third consecutive time to participate in the event as an invited speaker.

One of the most impressive things about this conference is that Francesco Cesarini, the leader of this event and CTO of the hosting company Erlang Solutions, carefully treats all delegates (which stands for the participants) equally with kindness and hospitality. Francesco's principles are well understood by all the other Erlang Solutions staff members as well.

I think Erlang Factory provides integrated feelings of satisfaction to all the delegated by letting them exchange the serious but friendly discussions. This is not something you can find over teleconferences or any forms of distant communication either synchronous or asynchronous. Erlang Factory is not a decision-making event; it is about sharing ideas and hanging out with each other, supporting both mental and emotional needs of the delegates.

Erlang Factory conference accepts a large number of versatile topics including:

• Distributed databases
• Case studies in production systems
• Detailed debugging into the library and the host operating systems
• Crash courses for programming and testing the language system, and
• Official announcements from Ericsson's OTP Team, who owns the final responsibility of managing and directing the language system's future.

An opportunity to discuss various complex issues with the speakers and the audiences is itself rare and precious, especially when the issues share the same core interests: Erlang/OTP.

Erlang Factory also accepts non-mainstream topics as a talk proposal. In the past three years, I talked about RPC over SSH, Mersenne Twister implementation with the NIFs, and in this year 2012 it was about IPv6 readiness and programming tips of the language system. Those are not necessarily the mainstream topics in the Erlang/OTP community, though I find quite a few problem reports on the online discussion places, such as in the erlang-questions mailing list.

My style of talk at the Erlang Factory is nothing extraordinary:

• First I analyze the problem and define the detailed problem domain to solve;
• then I write some example code to solve them and put them on the Web including GitHub and the other sharing places before finalizing the talk so that I could make it better with the wisdoms of others;
• I also experiment and evaluate the results while I'm finalizing the code and the presentation materials;
• Then at the Erlang Factory talks I discuss the conclusions found and propose the unresolved issues which may become another interesting talk topic.

The important points are sticking to the facts, no exaggeration, distinguishing assumptions from the facts, and listening to what the audiences ask and comment. Actually this is the same procedure I will take for the ACM Erlang Workshops, but for the Erlang Factory I try to make the presentation including more practical aspects and open questions.

I usually assign three months as a part-time project for an Erlang Factory presentation. The toughest part is to decide what to talk about. It's actually Francesco who suggested me to give a talk about IPv6 for this year 2012. I had some experience dealt with the technical issues of IPv6, so I decided to take the suggestion. The important point is that what you are going to talk is negotiable with the Erlang Factory staff.

This year 2012's events of Erlang Factory SF Bay Area were very intense, both in formal and casual senses. I've got an impression that it has become a truly international-class conference, while retaining the casual and friendly style of the past conferences.

I believe people go for conferences and events to meet people and share the feelings. It's a sort of festival, or Matsuri in Japanese, though also with the practical exchange of ideas. I strongly suggest all the Erlang/OTP enthusiasts to participate in this superb Matsuri.

Erlang VM memory size and ERL_MAX_PORTS

Erlang VM has the ports to communicate with other programs and objects (files, drivers, etc.) The ports are seen as the Erlang processes. The OS environment variable ERL_MAX_PORTS sets the default value of maximum number of ports. (See the manual for the function erlang:open_port/2 for the further details.)

I discovered by accident that Erlang VM of R15B on FreeBSD/amd64 9.0 in my test environment sets the value of maximum file descriptors assigned to the OS process as the default value for ERL_MAX_PORTS. This means ERL_MAX_PORTS is set to 800000, far larger than 1024, specified in the manual. This excessive value of the ERL_MAX_PORTS leads into unnecessary memory consumption of the Erlang VM. An easy way to test this is to execute the following code on the shell:

env ERL_MAX_PORTS=[given value] erl -noinput -eval 'io:format("~p~n",[erlang:memory(system)]).' -s erlang halt

How memory consumption changes just after the Erlang VM startup, as {ERL_MAX_PORTS, memory size in bytes}: {1024, ~65M}, {102400, ~124M}, {800000, ~543M}.

So I decided to set the ERL_MAX_PORTS for my home Yaws server to 4096. This significantly reduced the memory consumption.

I suggest you to tune the value of ERL_MAX_PORTS to optimize the number of concurrently opening ports of an Erlang VM, especially when you run a server on a memory-restricted host.

Update 11-MAR-2013: on R16B, the amount of memory consumed by the ports will be significantly reduced at the Erlang VM startup.

References

• Manual of erlang:memory/0
• [erlang-questions] About Environment variable ERL_MAX_PORTS
• [erlang-questions] Max open files and erlang memory

Kernel module functions for IPv6 on Erlang/OTP R15B

I've been reading the source code of Erlang/OTP R15B to find out how it handles the IPv6-related functions.

Erlang VM has the most functions embedded into the inet_drv linked-in driver, so programmers don't really have to worry about many things except for the address tuple format difference between IPv4 ({192, 168, 0, 1}) and IPv6 ({8193,3512,4660,22136,51966,47806,57005,48879} for 2001:db8:1234:5678:cafe:babe:dead:beef). You can find a lot of gems by reading the kernel module source code.

Here is a list of notable functions inside the kernel module for IPv6 handling on Erlang/OTP R15B. I suggest you to read these functions before writing your own ones.

(Note: this is a work-in-progress article of my Erlang Factory SF Bay Area 2012 presentation.)

%%% A partial list of IPv6 keywords in Erlang/OTP R15B source code

%%% atom "inet6" is an important keyword

inet_parse:address/1
inet_parse:ipv6_address/1
inet_parse:ipv6strict_address/1 % review RFC4291
inet_parse:ntoa/1

inet_parse:hosts/2 % internal, now ignoring "%if" suffix

-type inet:ip6_address()

inet:getifaddrs/0
inet:getifaddrs/1
inet:gethostbyaddr/1
inet:getaddrs/2 % second arg: inet6
inet:gethostbyname_string/2 % second arg: inet6

inet:i/0
inet:i/1

%%% interesting module results
%%% (no significant difference other than inet/inet6 flags):
%%%     diff --unified inet_tcp.erl inet6_tcp.erl
%%%     diff --unified inet_udp.erl inet6_udp.erl
%%%     diff --unified inet_sctp.erl inet6_sctp.erl
%%%     diff --unified inet_tcp_dist.erl inet6_tcp_dist.erl

inet_config:set_hostname/0 % internal, dependent on inet_udp:open/2

%%% inet_db.erl has tcp/udp/sctp module config params
%%% for the gen_server callbacks

inet_db:set_inet6/1
inet_db:dn_ip6_int/8 % internal, for 8 elements of an IPv6 address tuple
inet_db:make_hostent(Name, Addrs, Aliases, ?S_AAAA)
inet_db:handle_call({add_host, IPv6address, Names})
inet_db:handle_call({del_host, IPv6address})

inet_dns.erl: ?S_AAAA related lines

inet_dns.hrl: --define(T_AAAA), --define(S_AAAA)

inet_int.hrl: --define(INET_AF_INET6), --define(SCTP_FLAG_PMTUD_*),
              --define(ip6(A,B,C,D,E,F,G,H)

-type inet_res:res_option({inet6, boolean()})
-type inet_res:dns_data() % inet:ip6_address()

inet_res:gethostbyaddr/1
inet_res:gethostbyaddr_tm/2
inet_res:gethostbyname/1
inet_res:gethostbyname_tm/3
inet_res:udp_open/2 % internal (record #sock)
inet_res:udp_connect/2 % internal (record #sock)

% There will be more...

Leap second, again

News: There will be another leap second on 23:59:60UTC 31-JUN-2012.

Nevertheless, I quitted supporting leap second on the time zone file at home last year on December 2011, which had been introduced for more than 12 years. Reason: I gain very little while I lose quite a lot by setting the system time based on TAI.

I think POSIX should have defined the timezone functions to handle leap seconds properly, by setting the TAI-based monotonically-increasing value to the local clock, and by correcting the difference between UTC and TAI with the timezone file. But POSIX doesn't. So TAI-based local time is not interchangeable. Most applications assume the value of time_t represents UTC without the leap seconds, which is not the case when the local clock is set based on TAI.

I suspect that many sysadmins are not really aware of the leap seconds, unfortunately, including those provide NTP services. I've experienced one of the upper-stratum NTP servers of my ISP did not synchronize on 1-JAN-2009, when the last leap second was added.

Very few language systems can explicitly handle leap seconds either. For example, I had to post a patch for Erlang for handling TAI. (Fixed on Erlang/OTP R13A and later.)

So you should be very careful on the de-synchronization of your machine's local clocks on June 30th or July 1st this year on 2012.

Useful links:

• The Future of Leap Seconds
• Leap Seconds by Markus Kuhn
• Communications ACM May 2011: The One-Second War
• The NTP Timescale and Leap Seconds
• D. J. Bernstein: UTC, TAI, and UNIX time

(Correction 8-JAN-2012: gmtime() was ambiguous. Corrected to time_t.)

Preparing FreeBSD Port of Erlang/OTP

Updating an Erlang/OTP Port (files under the Ports tree) for FreeBSD is not a complicated task, with the latest pieces of the Port files. You can find out the latest pieces at the FreeBSD CVSweb page for Erlang Port.

You need to do the following things:

• Updating the source file names
• Preparing the library list
• Preparing the man file list
• Removing/editing/adding patches
• Updating the checksum file
• Preparing the package list
• Test building
• References

Updating the source file names

The following parts of the main Makefile must be modified to specify the related source file names (in a unified diff-like format):

    -PORTVERSION=   r14b04
    +PORTVERSION=   r15b

    -DISTNAME=      otp_src_R14B04
    +DISTNAME=      otp_src_R15B

    -ERLANG_MAN=    otp_doc_man_R14B04.tar.gz
    +ERLANG_MAN=    otp_doc_man_R15B.tar.gz

    -ERLANG_DOCS=   otp_doc_html_R14B04.tar.gz
    +ERLANG_DOCS=   otp_doc_html_R15B.tar.gz

Preparing the library list

Makefile.lib must be supplied to provide all the version numbers of the OTP library modules.

To extract the version number, use the following shell command:

    (cd work/otp_src_VERSION; \
     find . -name vsn.mk -print | xargs grep VSN) \
     > OTP-vsnlist.txt

The content of OTP-vsnlist.txt will be something like this:

    ./erts/vsn.mk:VSN = 5.9
    ./erts/vsn.mk:SYSTEM_VSN = R15B
    ./lib/appmon/vsn.mk:APPMON_VSN = 2.1.14
    ./lib/asn1/vsn.mk:ASN1_VSN = 1.6.19
    ./lib/common_test/vsn.mk:COMMON_TEST_VSN = 1.6
    ./lib/compiler/vsn.mk:COMPILER_VSN = 4.8
    ./lib/cosEvent/vsn.mk:COSEVENT_VSN = 2.1.12
    ./lib/cosEventDomain/vsn.mk:COSEVENTDOMAIN_VSN = 1.1.12
    ./lib/cosFileTransfer/vsn.mk:COSFILETRANSFER_VSN = 1.1.13
    ./lib/cosNotification/vsn.mk:COSNOTIFICATION_VSN = 1.1.18
    [...]
    ./lib/wx/vsn.mk:WX_VSN = 0.99.1
    ./lib/xmerl/vsn.mk:XMERL_VSN = 1.3

You have to manually edit the Makefile.lib with the content of OTP-vsnlist.txt. The format of Makefile.lib is (beware that the last line has no backslash):

    ERTS_VSN=       5.9
    TOOLS_VSN=      2.6.6.6
    OTP_LIBS=       appmon-2.1.14 \
                    asn1-1.6.19 \
                    common_test-1.6 \
                    compiler-4.8 \
                    cosEvent-2.1.12 \
                    cosEventDomain-1.1.12 \
                    cosFileTransfer-1.1.13 \
                    cosNotification-1.1.18 \
                    [...]
                    wx-0.99.1 \
                    xmerl-1.3

The contents of Makefile.lib is a set of Makefile variables, so all the module names must be listed with the backslash character. (Note: ERTS_VSN is no longer needed, but I will include it here just FYI.)

Preparing the man file list

Makefile.man must be supplied to provide all the version numbers of the OTP library modules.

To extract the man filenames, use the following shell command:

    tar ztf /usr/ports/distfiles/erlang/otp_doc_man_VERSION.tar.gz | \
    sort | awk '{FS="/"; print $3;}' > OTP-manlist.txt

You have to manually edit the Makefile.man with the content of OTP-manlist.txt. For OTP R15B, Makefile.man only needs man sections 1, 3, 4, and 6, so man section 7 must not be included.

The content of OTP-manlist.txt will be something like this:

COPYRIGHT
PR.template
README

man1/
man1/ct_run.1
man1/diameter_compile.1
man1/epmd.1
man1/erl.1
man1/erl_call.1
man1/erlc.1
[...]
man7/SNMP-VIEW-BASED-ACM-MIB.7
man7/SNMPv2-MIB.7
man7/SNMPv2-TM.7
man7/STANDARD-MIB.7
man7/TRANSPORT-ADDRESS-MIB.7

You have to manually edit the Makefile.man with the content of OTP-manlist.txt. The format of Makefile.man is (beware that the last line of each section has no backslash):

    MAN1=   ct_run.1 \
            diameter_compile.1 \
            epmd.1 \
            erl.1 \
            erl_call.1 \
            erlc.1 \
            erlsrv.1 \
            escript.1 \
            run_erl.1 \
            snmpc.1 \
            [...]
            start_webtool.1 \
            werl.1

    MAN3=   CosEventChannelAdmin.3 \
            CosEventChannelAdmin_ConsumerAdmin.3 \
            [...]
            zlib.3 \
            zlib_stub.3

    MAN4=   app.4 \
            appup.4 \
            [...]
            relup.4 \
            script.4

    MAN6=   common_test.6 \
            crypto.6 \
            [...]
            stdlib.6 \
            test_server.6

The contents of Makefile.man is a set of Makefile variables, so all the man file names must be listed with the backslash character.

Removing/editing/adding patches

FreeBSD-specific patch files are provided under the Port directory files. You need to remove the obsolete patches first, by make patch. You should edit the existing ones and add the new ones as well.

The naming convention of the patch file is:

    patch-lib_et_src_et__gs__contents__viewer.erl

for

    lib/et/src/et_gs_contents_viewer.erl

(Conversion of / -> _ and _ -> __ is needed.)

Updating the checksum file

make makesum is the convenient way to update the distinfo checksum data.

Preparing the package list

New binary files are occasionally added in the OTP distributions. The pkg-plist file must be updated as well.

Test building

The building process will be like this (note the umask 022):

    # I use zsh, but use whatever you want
    # as root
    sudo zsh
    umask 022
    make fetch
    exit
    # as your own account
    umask 022
    make
    # as root
    sudo zsh
    umask 022
    make deinstall && make reinstall
    exit

(Note added 5-MAR-2012: I've posted a send-pr (ports/163711) to FreeBSD issue tracker system on 30-DEC-2011. I've made an Evernote page for the port tar.gz archive.)

References

• Erlang/OTP repository at GitHub
• FreeBSD Porter's Handbook

My technical interests for Erlang/OTP as of April 2011

This is a list of my technical interests for Erlang/OTP as of April 2011:

• Catching up the latest cryptographic algorithms, including hash functions, shared-key cryptography, public-key cryptography, pseudo random number generators (PRNGs), while making them efficient in parallel/concurrent execution environment;
• Delay tolerance network technologies which can handle days, weeks, months of message delivery delays;
• Fundamental support functions of IPv6, especially for Distributed Erlang and epmd, and cryptographic transport support with SSL/TLS;
• DNSSEC-compatible resolver implementation for inet_res module;
• Large binary heap handling of BEAM, and the programming methodologies for the faster NIF implementation, including the performance measurement;
• PRNGs with small context/internal-state size, e.g., Wichmann-Hill 1982/2006, LShift, XOR32;
• PRNGs with large context/internal-state size, e.g., SFMT;
• TAI or leap-second handling by NIF;
• Privilege separation techniques between BEAM instances (e.g., virtualization, FreeBSD Jail); and
• the effective usage and choice of port drivers, linked-in drivers, and NIFs

And projects I'm interested in (but not limited to):

• Disco Project, a MapReduce implementation
• ZeroMQ and other general queueing systems
• UBF, BERT-RPC, and other messaging formats
• CouchDB

And don't forget I'll be hosting the ACM Erlang Workshop 2011.

(This article is a revised version of http://heikou-konton.blogspot.com/2011/02/erlangotp.html (originally written in Japanese.)

Erlang Workshop 2011 will be held as scheduled

This is a short notice that Erlang Workshop 2011 aka The Tenth ACM SIGPLAN Erlang Workshop will be held as scheduled at National Center of Sciences, National Institute of Informatics, Tokyo, Japan, on Friday, September 23, 2011.

See my presentation slides at ErlLounge of Erlang Factory SF Bay 2011 for the further details (in PDF). I should emphasize that the distance between the troubled Fukushima nuclear power plants and Tokyo is more than 200 kilometers, and also that the venue building of the workshop will not be affected by the scheduled rolling blackouts.

I also suggest you to take a look at ICFP 2011 Web site for the details of the ICFP 2011 related events.

Persistent Erlang processes or process pairs

Erlang's processes are the minimal unit of execution of Erlang BEAM virtual machine. Each process has its own ID, and can send/receive messages, register the name to the BEAM it's running, and can link with another process for error handling and monitoring. It even has its own process dictionary.

I've been thinking about a question for a few days: can you make a computer holding a process for more than 100 years? The word process here does not necessarily have to be the Erlang one, but the Erlang process will be a good candidate because the working environment it has to carry around with it is minimal and much smaller than that of UNIX process.

Keeping a process alive for a long time would not be possible if a process is confined to a single machine; the machine's failure means the immediate death of the process. So the process should be able to move around between multiple machines and make its clone on its own. Using shared memory should be avoided as possible. Realizing these characteristics with Erlang is less difficult than in other computer language systems.

I discovered the idea of having a persistent computer process is actually not my original. Google search engine tell me that Jim Gray has already published a technical report when he was in Tandem in 1985 (PDF copy from HP Labs) (a scanned text file of the report) with an idea of persistent process-pairs, as a part of his model of transactions for simple fault-tolerant execution.

In Gray's report, he describes a much smarter approach of making two persistent processes a pair to realize the persistency. If one of the pair fails, another one will show up and take over the actions of the failed one. This idea is much wiser than trying to keep a single process alive.

So now I find a way to realize persistent processes; next I need to learn how to implement them. It'll be a part of my new year's resolution for year 2010. A happy new year to you all the readers.

Erlang and Github

Erlang/OTP is now officially maintained under the Github repository, since the release R13B03. I think this is a milestone for the language, because the Ericsson development team finally decided to show the interim results of what they are doing for the time being.

One of the characteristics I like about Erlang is that the language specification and libraries have been maintained by a single entity called Ericsson's Erlang/OTP Development Team. I do not want an anarchy for computer language and operating systems. I prefer BSDism than Linuxism in this sense; I think pieces of code should be rather controlled by the core people while sufficiently accepting improvements from the other developers.

The old Erlang/OTP daily snap archives, however, are no longer sufficient to catch up with the daily development cycles. And many non-Ericsson authors have put in their patches into Erlang, including mine. So there had to be some systems to accept user feedbacks.

Using an open repository system such as Github is a wise idea for incorporating new code into Erlang/OTP, and showing the official status of modifications. Git is flexible enough to allow per-user and per-purpose branches. And Github allows forking between the users. The Ericsson's Team doesn't have to build and publicize its own code repository system for Erlang/OTP, which will cost them significant amount of human and financial resources.

And now I have an official requirement to learn Git; to catch up with the Erlang/OTP development cycles.

Why choosing Erlang for concurrent processing

A few weeks ago I was reading a blog article called Erlang, the next Java. While I agreed with the author's views, I felt something was bugging me in the article, which eventually lead me to write this blog article.

I didn't choose Erlang just because it's a functional language. I have chosen Erlang to learn because Erlang programming requires and has to enforce the programmers to write the concurrent code. Concurrent programing should minimize the number and size of critical regions and bottlenecks. And I think writing such programs is very difficult without the assistance of the programming language.

Many programming languages claim or plan to be capable of running a part of code based on concurrent processing. For example, Ruby will incorporate distributed storage mechanism called Roma and the task administration subsystem called Fairy. Another good example is Haskell concurrency: it is implemented as a language feature called Parallel Haskell.

I think, however, introducing concurrency while allowing programmers to write code to use shared memory will cause a lot of problems. Joe Armstrong has already described his concern on shared memory in his blog. I support his arguments; those arguments an important part of the reason why I decided to learn Erlang.

I should add another problem programmers will face when dealing with code allowing shared memory; rewriting the code for removing shared access to run it efficiently on concurrent environment will be an incredibly difficult task.

Unfortunately, most existing languages have already had a lot of code written assuming shared memory areas. For example, C code with extern variables implicitly assume those are shared between the functions in all source code files linked together. I assume I can hardly find any set of C code without using an extern declaration. I have learned that even Common Lisp has special declaration for the variables, which allows multiple functions share the same object, out of lexical scopes.

Another example of shared-memory concurrency is the operating system (OS) threads. C/C++/Java threads inherently share the parent OS process address space and environment in common. Python has the Thread Objects. While OS threads often ease implementation of concurrent servers by reducing the task switching time, the semantics is implicit and error-prone.

I understand and agree that sharing objects itself cannot be completely eliminated under the read-world constraints of processing and communication timings between programming language functions. I think, however, that programming languages should help the programmers to minimize writing code including shared memory areas, which will turn themselves into critical regions.

Erlang imposes necessary restriction on avoiding implicit data sharing between functions by:

• prohibiting multiple variable assignments in a function;


• enforcing and helping the programmers to conduct message-passing programming between functions, by not providing any implicit data-sharing facility between the functions;


• providing fast task-switching capabilities, by giving the definition that functions are the minimum concurrent execution units; and


• restricting the usage of data-sharing facilities between functions to the minimum, such as process dictionaries, ETS, DETS, Mnesia, and the global naming service shared between connected Erlang nodes, by requring explicitly writing so in the code.
  

In short, I think adding concurrent features is not enough for concurrent programming; prohibiting non-concurrent habits and enforcing writing concurrent programs are necessary as part of the programming language specifications. I believe concurrent programming under coming multi-processor environment is only possible under such a hard-liner attitude to the programmers. I feel programmers including myself are very conservative and rigid to change their sequential-programming habits.

And actually, I was one of those programmers who did not recognize the urgency of learning concurrent programming in 2007.

nsplit available as a Google Code project

My Erlang code nsplit is now available as a part of Google Code projects.

Link on this blog about the code is here.

Erlang: parallelism without modifying code

The Erlang programming language has been getting popular among programmers, for the capability of handling massive number of connections, such as the chat server used in Facebook.

One of the basic ideas of Erlang are that minimizing the side effect of assignments by prohibiting multiple assignments into the same variable. At first this looks a rigid restriction and memory-consuming practice, but once you learn the native list structure of Erlang (mostly the same as in the Lisp language) and the tail-recursion control flow, writing the Erlang code is not too difficult, while retaining the conciseness and the performance.

I've written a code of parallel mapping of a function to a list. The word mapping means applying the same function to each member of the source list and obtaining the results which retains the same sequence as in the source list. If the function does not have side effects (such as changing the values of shared data structures), the operation can be parallelized by splitting the list into the smaller sublists and invoking the mapping process for each sublist.

Erlang has the supporting libraries of invoking a process in a distributed Erlang node running in multiple computers, so the parallel mapping function is the simplest but very powerful tool to experience the collective computing power of parallelism.

The code is available here as a tar archive, of one file of Erlang source code.