Monday, April 2, 2012

Why Erlang people need the Erlang Factory events

Disclaimer: I am writing this blog article on my way home from SFO. The reduced air pressure in the passenger cabin might have affected the contents of this article.

I spent all three days of Erlang Factory SF Bay 2012, with two more days to cope with the jet lag. This was my third consecutive time to participate in the event as an invited speaker.

One of the most impressive things about this conference is that Francesco Cesarini, the leader of this event and CTO of the hosting company Erlang Solutions, carefully treats all delegates (which stands for the participants) equally with kindness and hospitality. Francesco's principles are well understood by all the other Erlang Solutions staff members as well.

I think Erlang Factory provides integrated feelings of satisfaction to all the delegated by letting them exchange the serious but friendly discussions. This is not something you can find over teleconferences or any forms of distant communication either synchronous or asynchronous. Erlang Factory is not a decision-making event; it is about sharing ideas and hanging out with each other, supporting both mental and emotional needs of the delegates.

Erlang Factory conference accepts a large number of versatile topics including:

  • Distributed databases
  • Case studies in production systems
  • Detailed debugging into the library and the host operating systems
  • Crash courses for programming and testing the language system, and
  • Official announcements from Ericsson's OTP Team, who owns the final responsibility of managing and directing the language system's future.

An opportunity to discuss various complex issues with the speakers and the audiences is itself rare and precious, especially when the issues share the same core interests: Erlang/OTP.

Erlang Factory also accepts non-mainstream topics as a talk proposal. In the past three years, I talked about RPC over SSH, Mersenne Twister implementation with the NIFs, and in this year 2012 it was about IPv6 readiness and programming tips of the language system. Those are not necessarily the mainstream topics in the Erlang/OTP community, though I find quite a few problem reports on the online discussion places, such as in the erlang-questions mailing list.

My style of talk at the Erlang Factory is nothing extraordinary:

  • First I analyze the problem and define the detailed problem domain to solve;
  • then I write some example code to solve them and put them on the Web including GitHub and the other sharing places before finalizing the talk so that I could make it better with the wisdoms of others;
  • I also experiment and evaluate the results while I'm finalizing the code and the presentation materials;
  • Then at the Erlang Factory talks I discuss the conclusions found and propose the unresolved issues which may become another interesting talk topic.
The important points are sticking to the facts, no exaggeration, distinguishing assumptions from the facts, and listening to what the audiences ask and comment. Actually this is the same procedure I will take for the ACM Erlang Workshops, but for the Erlang Factory I try to make the presentation including more practical aspects and open questions.

I usually assign three months as a part-time project for an Erlang Factory presentation. The toughest part is to decide what to talk about. It's actually Francesco who suggested me to give a talk about IPv6 for this year 2012. I had some experience dealt with the technical issues of IPv6, so I decided to take the suggestion. The important point is that what you are going to talk is negotiable with the Erlang Factory staff.

This year 2012's events of Erlang Factory SF Bay Area were very intense, both in formal and casual senses. I've got an impression that it has become a truly international-class conference, while retaining the casual and friendly style of the past conferences.

I believe people go for conferences and events to meet people and share the feelings. It's a sort of festival, or Matsuri in Japanese, though also with the practical exchange of ideas. I strongly suggest all the Erlang/OTP enthusiasts to participate in this superb Matsuri.

Monday, March 5, 2012

Erlang VM memory size and ERL_MAX_PORTS

Erlang VM has the ports to communicate with other programs and objects (files, drivers, etc.) The ports are seen as the Erlang processes. The OS environment variable ERL_MAX_PORTS sets the default value of maximum number of ports. (See the manual for the function erlang:open_port/2 for the further details.)

I discovered by accident that Erlang VM of R15B on FreeBSD/amd64 9.0 in my test environment sets the value of maximum file descriptors assigned to the OS process as the default value for ERL_MAX_PORTS. This means ERL_MAX_PORTS is set to 800000, far larger than 1024, specified in the manual. This excessive value of the ERL_MAX_PORTS leads into unnecessary memory consumption of the Erlang VM. An easy way to test this is to execute the following code on the shell:

env ERL_MAX_PORTS=[given value] erl -noinput -eval 'io:format("~p~n",[erlang:memory(system)]).' -s erlang halt

How memory consumption changes just after the Erlang VM startup, as {ERL_MAX_PORTS, memory size in bytes}: {1024, ~65M}, {102400, ~124M}, {800000, ~543M}.

So I decided to set the ERL_MAX_PORTS for my home Yaws server to 4096. This significantly reduced the memory consumption.

I suggest you to tune the value of ERL_MAX_PORTS to optimize the number of concurrently opening ports of an Erlang VM, especially when you run a server on a memory-restricted host.

Update 11-MAR-2013: on R16B, the amount of memory consumed by the ports will be significantly reduced at the Erlang VM startup.


Friday, March 2, 2012

Kernel module functions for IPv6 on Erlang/OTP R15B

I've been reading the source code of Erlang/OTP R15B to find out how it handles the IPv6-related functions.

Erlang VM has the most functions embedded into the inet_drv linked-in driver, so programmers don't really have to worry about many things except for the address tuple format difference between IPv4 ({192, 168, 0, 1}) and IPv6 ({8193,3512,4660,22136,51966,47806,57005,48879} for 2001:db8:1234:5678:cafe:babe:dead:beef). You can find a lot of gems by reading the kernel module source code.

Here is a list of notable functions inside the kernel module for IPv6 handling on Erlang/OTP R15B. I suggest you to read these functions before writing your own ones.

(Note: this is a work-in-progress article of my Erlang Factory SF Bay Area 2012 presentation.)

%%% A partial list of IPv6 keywords in Erlang/OTP R15B source code

%%% atom "inet6" is an important keyword

inet_parse:ipv6strict_address/1 % review RFC4291

inet_parse:hosts/2 % internal, now ignoring "%if" suffix

-type inet:ip6_address()

inet:getaddrs/2 % second arg: inet6
inet:gethostbyname_string/2 % second arg: inet6


%%% interesting module results
%%% (no significant difference other than inet/inet6 flags):
%%%     diff --unified inet_tcp.erl inet6_tcp.erl
%%%     diff --unified inet_udp.erl inet6_udp.erl
%%%     diff --unified inet_sctp.erl inet6_sctp.erl
%%%     diff --unified inet_tcp_dist.erl inet6_tcp_dist.erl

inet_config:set_hostname/0 % internal, dependent on inet_udp:open/2

%%% inet_db.erl has tcp/udp/sctp module config params
%%% for the gen_server callbacks

inet_db:dn_ip6_int/8 % internal, for 8 elements of an IPv6 address tuple
inet_db:make_hostent(Name, Addrs, Aliases, ?S_AAAA)
inet_db:handle_call({add_host, IPv6address, Names})
inet_db:handle_call({del_host, IPv6address})

inet_dns.erl: ?S_AAAA related lines

inet_dns.hrl: --define(T_AAAA), --define(S_AAAA)

inet_int.hrl: --define(INET_AF_INET6), --define(SCTP_FLAG_PMTUD_*),

-type inet_res:res_option({inet6, boolean()})
-type inet_res:dns_data() % inet:ip6_address()

inet_res:udp_open/2 % internal (record #sock)
inet_res:udp_connect/2 % internal (record #sock)

% There will be more...

Saturday, February 25, 2012

Small pieces of happiness that I pursue

The small pieces of happiness I've been pursuing since 1992 are as follows:

  • Making money by doing what I want and by minimizing doing what I don't want to do;
  • Living in a house of my own control with my partner without interference;
  • Working at a place where I want and at a place where I can minimize the mental stress; and
  • Working at a period of time in the most productive way, including taking enough time to rest.

I feel I've been devoting myself to pursue this small happiness at my best effort.

And I'm happy with it.

Sunday, January 8, 2012

Leap second, again

News: There will be another leap second on 23:59:60UTC 31-JUN-2012.

Nevertheless, I quitted supporting leap second on the time zone file at home last year on December 2011, which had been introduced for more than 12 years. Reason: I gain very little while I lose quite a lot by setting the system time based on TAI.

I think POSIX should have defined the timezone functions to handle leap seconds properly, by setting the TAI-based monotonically-increasing value to the local clock, and by correcting the difference between UTC and TAI with the timezone file. But POSIX doesn't. So TAI-based local time is not interchangeable. Most applications assume the value of time_t represents UTC without the leap seconds, which is not the case when the local clock is set based on TAI.

I suspect that many sysadmins are not really aware of the leap seconds, unfortunately, including those provide NTP services. I've experienced one of the upper-stratum NTP servers of my ISP did not synchronize on 1-JAN-2009, when the last leap second was added.

Very few language systems can explicitly handle leap seconds either. For example, I had to post a patch for Erlang for handling TAI. (Fixed on Erlang/OTP R13A and later.)

So you should be very careful on the de-synchronization of your machine's local clocks on June 30th or July 1st this year on 2012.

Useful links:

(Correction 8-JAN-2012: gmtime() was ambiguous. Corrected to time_t.)

Wednesday, January 4, 2012

Preparing FreeBSD Port of Erlang/OTP

Updating an Erlang/OTP Port (files under the Ports tree) for FreeBSD is not a complicated task, with the latest pieces of the Port files. You can find out the latest pieces at the FreeBSD CVSweb page for Erlang Port.

You need to do the following things:

Updating the source file names

The following parts of the main Makefile must be modified to specify the related source file names (in a unified diff-like format):

    -PORTVERSION=   r14b04
    +PORTVERSION=   r15b

    -DISTNAME=      otp_src_R14B04
    +DISTNAME=      otp_src_R15B

    -ERLANG_MAN=    otp_doc_man_R14B04.tar.gz
    +ERLANG_MAN=    otp_doc_man_R15B.tar.gz

    -ERLANG_DOCS=   otp_doc_html_R14B04.tar.gz
    +ERLANG_DOCS=   otp_doc_html_R15B.tar.gz

Preparing the library list

Makefile.lib must be supplied to provide all the version numbers of the OTP library modules.

To extract the version number, use the following shell command:

    (cd work/otp_src_VERSION; \
     find . -name -print | xargs grep VSN) \
     > OTP-vsnlist.txt

The content of OTP-vsnlist.txt will be something like this:

    ./erts/ = 5.9
    ./erts/ = R15B
    ./lib/appmon/ = 2.1.14
    ./lib/asn1/ = 1.6.19
    ./lib/common_test/ = 1.6
    ./lib/compiler/ = 4.8
    ./lib/cosEvent/ = 2.1.12
    ./lib/cosEventDomain/ = 1.1.12
    ./lib/cosFileTransfer/ = 1.1.13
    ./lib/cosNotification/ = 1.1.18
    ./lib/wx/ = 0.99.1
    ./lib/xmerl/ = 1.3

You have to manually edit the Makefile.lib with the content of OTP-vsnlist.txt. The format of Makefile.lib is (beware that the last line has no backslash):

    ERTS_VSN=       5.9
    OTP_LIBS=       appmon-2.1.14 \
                    asn1-1.6.19 \
                    common_test-1.6 \
                    compiler-4.8 \
                    cosEvent-2.1.12 \
                    cosEventDomain-1.1.12 \
                    cosFileTransfer-1.1.13 \
                    cosNotification-1.1.18 \
                    wx-0.99.1 \

The contents of Makefile.lib is a set of Makefile variables, so all the module names must be listed with the backslash character. (Note: ERTS_VSN is no longer needed, but I will include it here just FYI.)

Preparing the man file list must be supplied to provide all the version numbers of the OTP library modules.

To extract the man filenames, use the following shell command:

    tar ztf /usr/ports/distfiles/erlang/otp_doc_man_VERSION.tar.gz | \
    sort | awk '{FS="/"; print $3;}' > OTP-manlist.txt

You have to manually edit the with the content of OTP-manlist.txt. For OTP R15B, only needs man sections 1, 3, 4, and 6, so man section 7 must not be included.

The content of OTP-manlist.txt will be something like this:



You have to manually edit the with the content of OTP-manlist.txt. The format of is (beware that the last line of each section has no backslash):

    MAN1=   ct_run.1 \
            diameter_compile.1 \
            epmd.1 \
            erl.1 \
            erl_call.1 \
            erlc.1 \
            erlsrv.1 \
            escript.1 \
            run_erl.1 \
            snmpc.1 \
            start_webtool.1 \

    MAN3=   CosEventChannelAdmin.3 \
            CosEventChannelAdmin_ConsumerAdmin.3 \
            zlib.3 \

    MAN4=   app.4 \
            appup.4 \
            relup.4 \

    MAN6=   common_test.6 \
            crypto.6 \
            stdlib.6 \

The contents of is a set of Makefile variables, so all the man file names must be listed with the backslash character.

Removing/editing/adding patches

FreeBSD-specific patch files are provided under the Port directory files. You need to remove the obsolete patches first, by make patch. You should edit the existing ones and add the new ones as well.

The naming convention of the patch file is:




(Conversion of / -> _ and _ -> __ is needed.)

Updating the checksum file

make makesum is the convenient way to update the distinfo checksum data.

Preparing the package list

New binary files are occasionally added in the OTP distributions. The pkg-plist file must be updated as well.

Test building

The building process will be like this (note the umask 022):

    # I use zsh, but use whatever you want
    # as root
    sudo zsh
    umask 022
    make fetch
    # as your own account
    umask 022
    # as root
    sudo zsh
    umask 022
    make deinstall && make reinstall

(Note added 5-MAR-2012: I've posted a send-pr (ports/163711) to FreeBSD issue tracker system on 30-DEC-2011. I've made an Evernote page for the port tar.gz archive.)