Working CW stations at the end of the year

I've set up an amateur radio station again this week, and worked some stations, including some friends. The transmission feeder is a temporary one, so I will disassemble and reassemble the system soon, after the new year vacation time is over.

Amateur radio for me since 2002 is mostly CW (Morse Code). CW is the most primitive but still practical digital communication form. The bit rate is about 10 to 25 bps (no kilo, mega, or giga) (that is 12 to 30WPM) and it's very slow indeed. You still can send something meaningful over CW, and it has been used for more than 100 years. See how people are using Twitter these days.

CW has already been phased out from the mainstream of professional communication systems, but it's still a viable backup, and ham radio operators still use it over shortwaves to exploit the maximum possibility of communication, to overcome the difficulties of natural and artificial noises, and significant path loss. On the other hand, the latency is minimal; only the path distance between the peers determines the delay time. It's the oldest chat system in the world.

The hardest part of CW is that you need to learn and have a lot of practice to listen to it. It's like learning another foreign language. Some eccentric people including me, however, choose this road less traveled, for many reasons.

Many old operators may find out their younger days of enthusiasm towards radio itself. And many hardware hackers will use CW for realizing the simplest but useful radio system with a pair of home-made transmitter and receiver. Whatever the reason it might be, still quite a lot of people devote themselves to CW. And I am one of them.

Working on CW is simply a good time for me. Learning for a faster CW listening stimulates my mind a lot. And CW reminds me of the very basic issues of digital communication; reliability, bandwidth, stability, latency, and practicality. The procedure of sending and receiving letters by yourself gives you more time to think.

I'm a computer engineer and scientist, and technologically CW is an archaic thing. The principle embedded, however, is still effective to be a fundamental motivation of pursuing engineering and scientific skills; to explore the road less traveled and discover the new world.

The following article of Jeff, KE9V may help you to understand why some old amateur radio operators are still enthusiastic to operate on CW: The Road Less Traveled

73 and have a happy new year 2009.

Kenji Rikitake, JJ1BDX(/3), JO3FUO, N6BDX and JQ2KST

My Influences

My influences on programming, Internet, and computer skills:

1. Tsuneji Rikitake, my father: who taught me basic math, FORTRAN, and financed my startup (of my professional career, indeed)
2. Paul Vixie and DEC NSL people: for the primary idea of firewalls and Internet systems administration, and the proper engineering attitudes toward problem solving
   
3. Daniel J. Bernstein: on re-thinking Internet protocols from the very beginning, especially on DNS
4. Bruce Schneier: for the basic philosophy and principles of security, not only for computer systems
5. UNIX gurus on Bell Labs and BSD communities: for the programming suite and styles, including C, awk, and Bourne shell
   
6. Joe Armstrong and the Erlang programming communities: for letting me know a practical message-passing-based concurrency
   

I know there are many other people I should put on the list, but I will just keep this list small.

My salutation to Chad Fowler and Kevin Smith for reminding me of this topic.

A quiet life without a TV tuner

I've been spending time without TV at home since this (2008) September. And I recommend you to do so too for living a quiet life.

I decided to stop watching TV regularly; no airwave, no satellite, no cable channel at all. Actually the condo building I live is cable-TV ready. But I decided to quit. I gave our LCD TV to my wife's Mom. We instead listen to the radio when we're at home and having a dinner or lunch and we want some news. Most of airwave TV programs are junk. And most of non-airwave (that is, paid) TV programs are also junk. And we know most of airwave radio programs are junk. But some of them, including news and classic music programs of our national broadcaster (in Japan), are fortunately not. I carry around a pocket radio when I travel in Japan to listen to the same night program called Radio Shin-ya-bin, a rather-quiet-and-calm program without ads.

We still want to see the DVD contents. We have a DVD player. So I decide to newly buy and install a new PC display instead. It accepts DVI, VGA, component and NTSC video, and even HDMI. It has 1920x1200 pixel resolution; an excessive spec for DVD or an old PC. It doesn't have a TV tuner, however. It doesn't have speakers either, but that is OK, because I installed a pair of speakers and an amplifier with AM/FM radio tuner installed. So no problem for enjoying audio.

The good thing about not watching TV while on a family meal is that we rather talk and make a lot of conversations. And we started to read a lot of books and talk about them. It's like living in the world of 1950s or 1960s, though we still use PCs.

We have also quit subscribing newspapers since this April. I decided so because even Nikkei Shimbun, let alone other papers, were carrying junk articles these days. The New York Times and The Economist have already allowed people to read their articles online freely, as other media companies follow. So I told my wife we didn't need a newspaper subscription at home anyway. She first complained, but eventually she also learned to enjoy reading something instead, or to spend the time for being exposed to mass media to something else.

We don't have any game machines such as Sony's PlayStation or Microsoft's Xbox. Our eyes can no longer follow or respond to artificial graphic pictures.

We mostly use PC to read letters and design presentations; we don't make active 3D graphics and we will not unless we need to visualize something for our works. And of course we see visual presentations on YouTube and elsewhere. The good news is, however, they are on-demand only. Unless you explicitly tell PCs to show them, you don't have to watch at those on-demand videos.

Why choosing Erlang for concurrent processing

A few weeks ago I was reading a blog article called Erlang, the next Java. While I agreed with the author's views, I felt something was bugging me in the article, which eventually lead me to write this blog article.

I didn't choose Erlang just because it's a functional language. I have chosen Erlang to learn because Erlang programming requires and has to enforce the programmers to write the concurrent code. Concurrent programing should minimize the number and size of critical regions and bottlenecks. And I think writing such programs is very difficult without the assistance of the programming language.

Many programming languages claim or plan to be capable of running a part of code based on concurrent processing. For example, Ruby will incorporate distributed storage mechanism called Roma and the task administration subsystem called Fairy. Another good example is Haskell concurrency: it is implemented as a language feature called Parallel Haskell.

I think, however, introducing concurrency while allowing programmers to write code to use shared memory will cause a lot of problems. Joe Armstrong has already described his concern on shared memory in his blog. I support his arguments; those arguments an important part of the reason why I decided to learn Erlang.

I should add another problem programmers will face when dealing with code allowing shared memory; rewriting the code for removing shared access to run it efficiently on concurrent environment will be an incredibly difficult task.

Unfortunately, most existing languages have already had a lot of code written assuming shared memory areas. For example, C code with extern variables implicitly assume those are shared between the functions in all source code files linked together. I assume I can hardly find any set of C code without using an extern declaration. I have learned that even Common Lisp has special declaration for the variables, which allows multiple functions share the same object, out of lexical scopes.

Another example of shared-memory concurrency is the operating system (OS) threads. C/C++/Java threads inherently share the parent OS process address space and environment in common. Python has the Thread Objects. While OS threads often ease implementation of concurrent servers by reducing the task switching time, the semantics is implicit and error-prone.

I understand and agree that sharing objects itself cannot be completely eliminated under the read-world constraints of processing and communication timings between programming language functions. I think, however, that programming languages should help the programmers to minimize writing code including shared memory areas, which will turn themselves into critical regions.

Erlang imposes necessary restriction on avoiding implicit data sharing between functions by:

• prohibiting multiple variable assignments in a function;


• enforcing and helping the programmers to conduct message-passing programming between functions, by not providing any implicit data-sharing facility between the functions;


• providing fast task-switching capabilities, by giving the definition that functions are the minimum concurrent execution units; and


• restricting the usage of data-sharing facilities between functions to the minimum, such as process dictionaries, ETS, DETS, Mnesia, and the global naming service shared between connected Erlang nodes, by requring explicitly writing so in the code.
  

In short, I think adding concurrent features is not enough for concurrent programming; prohibiting non-concurrent habits and enforcing writing concurrent programs are necessary as part of the programming language specifications. I believe concurrent programming under coming multi-processor environment is only possible under such a hard-liner attitude to the programmers. I feel programmers including myself are very conservative and rigid to change their sequential-programming habits.

And actually, I was one of those programmers who did not recognize the urgency of learning concurrent programming in 2007.

Programming language design and security

I suspect quite a few programming language designers just don't care about the security of the languages and the systems built by them. Maybe I'm too pessimistic, but a recent experience brought me the idea.

I was an audience at an event called Lightweight Language Future yesterday in Nakano, Tokyo, Japan. I only attended the first two sessions. The second session was a panel discussion about the programming language 100 years later. While the discussion was technically intriguing, I heard not a word about security at all during the panel. Two of the panelists (names withheld because I don't want this article to be a personal attack to them) were the principal designers of popular lightweight languages, so I asked them in the Q&A session about how they would deal with the security issues which would become more serious in the future. One of the answers surprised me; the person told me that VMware would do.

I understand virtualization in various operating system layers, including VMware and FreeBSD jail, is surely a good hack to control the damage area in the whole system to run a public access system. I think it's not sufficient at all, however, to prevent future security incidents; if a programming language is potentially vulnerable to attacks, the systems built by the language will be also equally vulnerable.

I think programming language designers are not exempted from the responsibility of (at least trying to) making computer systems secure. If I could hear more constructive ways of solving the security problems in the answer from one of the panelists, such as:

• protecting the language from the buffer-overflow bugs;
  
• preventing the garbage collector from crashing even in a hostile environment where the external attackers try to deliberately manipulate the pointers or variables to hack into the protected memory area;
  
• implementing a syntax or semantics validation system in the programming language so that the programmers can apply it to validate the external data, such as those from the network-connected users; or
  
• preventing the race condition, deadlocks, or any sort of resource starvation by the intentional or unintentional programming;
  

then I would have been much more convinced. But now I should suspect that quite a few programming language designers just don't care about the security consequences of the features they build into the language.

I don't have a definitive answer to solve the issues of unsecure systems built from unsecure programming languages. All I can write now is that a security-aware programming language will surely help making a secure system. Note that security is not just about privacy and secrecy; it's also about the reliability and fault-tolerance.

Note on 4-SEP-2008: one of my friends later reviewed the panel recording and told me that it was not VMware but VM layer that the panelist answered to me. I still think solely relying on the VM layer is insufficient for building a security structure in a programming language, though.

About myself: 1965-2000

This self introduction is a reprint of an article once posted to my own website http://www.k2r.org/kenji/ as a part of my self introduction.

---

I was born in 1965. I grew up in a Japanese urban community in Setagaya-ku, Tokyo. The whole things changed in 1974 when my father decided to go to Boulder, CO, USA for a visiting professor job at Cooperative Institute for Research in Environmental Sciences of University of Colorado at Boulder.

For the first 6 months of my American life, I didn't speak English at all. We lived there for 15 months, so I was a lousy kid speaking broken English just before when we returned to Japan. I still think the life in the USA was the best of my times. I've never lived abroad since then. I've been keeping up my English proficiency by listening to and watching English-speaking broadcasts, and reading English Web articles.

After coming back to Tokyo, I found out myself I was fatally incompatible with and just way too outspoken than other kids around.The totalitarian and socialist society and elementary schools of Japan did not and still do not accept me. They told and still tell me that I was and I am so-called contaminated with individualism and American way of thoughts. So I had to make a difference to survive.

I didn't have much advantage from the other kids, except for hacking around the radio and audio devices, writing some code on the papers (no desktop PC existed in 1975 in Japan), playing around with electronic circuits, and having no fear for speaking English. I was just a geek. My father Tsuneji bought me a FORTRAN book and a shortwave radio in 1973, the slide rules and electronic calculators in 1974, and a bunch of math books. I learned them mostly by myself.

I went to Musashi high school, a private school well-known for the liberalism. Soon I became a programmer of Apple II in 1979. My programming experience was mostly on 6502 assembly language, 6K Integer Basic, and the GAME programming language. ASCII Magazine bought my GAME-APPLE compiler in 1980 and I earned about JPY200,000 by the sales.During that time I met a lot of talented hackers. Many of them are active in the Japanese Internet R&D scene.

In 1982, my right eye was badly damaged with the retina detachment so I had to stay away from computers. I started to make techno music instead of computer programs, using later well-known techno gears such as TB-303 and TR-909, until 1986. I loved making music, but I didn't feel I belonged to the music industry. I think I made the right decision,after discovering the whole mess of the ripping-off business models.

I went to University of Tokyo in 1984 and there I experienced the harshest part of Japanese career competition. I managed to be accepted by the faculty of Engineering to learn instrumentation physics. I later studied information engineering for my Master's degree. Life in the university was no fun; hanging around with real hackers and real systems were much better.

In 1985 after NTT finally opened up the phone lines to public, I discovered international computer networks and online systems. I soon became an e-mail and BBS junkie. In 1987, I started working with Jeffrey Shapard, who was running an English-speaking system called TWICS, for the VAX/VMS system administration, until 1989. During the TWICS days I met Joichi "Joi" Ito, an English-speaking multi-talented Japanese activist.

Many hackers went for UNIX, BSD, VAX and Sun in 1980s. I was also one of the root-privileged users of a packet-radio UNIX systems, learning vi and hacking with the code. Having an account on an UNIX system was a cool thing. What made my difference was that I ported the UUPC/Extended software package for NEC's PC-9801 computers. It was a version of UUCP software,which performed automatic file exchange and email address relay between computers. And I was a licensed radio amateur (as JJ1BDX since 1976), so I made some gateway software packages linking ham radio TCP/IP mail/NetNews systems and the UUCP-based systems.

In 1990, I joined Digital Equipment Corporation (DEC) Japan, to write the core software of VAX/VMS. During that period, I had a chance to exchange messages to many Internet gurus, including Paul Vixie, a DNS guru and Internet leading technologist, who was working for DEC then. It was really nice to learn the philosophy and principles of Internet from those hard-core hackers.

During my 2 years of Digital career, I was struggling with the corporate bureaucrats, and too many overworking and commuting hours which were eventually killing me. I met Kyoko in 1991, and we decided to get together and married. She told me she had a new job in Osaka from April 1992, so I decided to move to Osaka too. I was looking for an Internet job there, and Kazuo Hirono hired me for one of his research team at TDI Co., Ltd., a Japanese software service firm. We started to build a research lab in Kyoto and the lab was operational from 1993 to 2000, also as a part of WIDE Project Kyoto NOC (POP). Hirono-san taught me a lot of things, and he was very tolerate on my rather individualistic and outspoken communication style. (Hirono-san passed away on June 2007. RIP.)

(Originally written in March 2008)

nsplit available as a Google Code project

My Erlang code nsplit is now available as a part of Google Code projects.

Link on this blog about the code is here.

Outsourcing my DNS lookup service to OpenDNS

DNS subsystem is a real headache to manage. You will realize that once you have to tweak the system, especially if you want to incorporate your own filtering/access-blocking rules.

And the DNS Cache Poisoning. It's a real threat.

While I believe my DNS subsystem is quite safe under the good-old djbdns servers, I am now testing the OpenDNS, a DNS cache service provider. One of the good things about OpenDNS is that they even allow a single-IPv4 address network to be individually managed, even it's dynamically allocated, as in most of the cases for non-static-IPv4 users.

If you can trust your ISP for the DNS management, you are on your own. But if you can't or don't, OpenDNS is a good alternative. I notice many ISPs still have not changed their DNS cache servers to prevent the poisoning attack as of today (July 24, 2008); using OpenDNS from such a mobile networking environment will make the whole DNS access much secure.

My allegation of spamdexing is finally dropped

It seems that the Blogger admin finally dropped the allegation of spamdexing from http://macrofield.blogspot.com/. It took more than 2 months, but less than 3 months, to solve this issue.

Of course, I removed the suspicious META refresh tag from the alleged page.

Links:
The report of false detection of spamming
Meta refreshing considered spamdexing

Erlang: parallelism without modifying code

The Erlang programming language has been getting popular among programmers, for the capability of handling massive number of connections, such as the chat server used in Facebook.

One of the basic ideas of Erlang are that minimizing the side effect of assignments by prohibiting multiple assignments into the same variable. At first this looks a rigid restriction and memory-consuming practice, but once you learn the native list structure of Erlang (mostly the same as in the Lisp language) and the tail-recursion control flow, writing the Erlang code is not too difficult, while retaining the conciseness and the performance.

I've written a code of parallel mapping of a function to a list. The word mapping means applying the same function to each member of the source list and obtaining the results which retains the same sequence as in the source list. If the function does not have side effects (such as changing the values of shared data structures), the operation can be parallelized by splitting the list into the smaller sublists and invoking the mapping process for each sublist.

Erlang has the supporting libraries of invoking a process in a distributed Erlang node running in multiple computers, so the parallel mapping function is the simplest but very powerful tool to experience the collective computing power of parallelism.

The code is available here as a tar archive, of one file of Erlang source code.

Installing FreeBSD 6.3 and 7.0 to Panasonic CF-R4

I had difficulties installing FreeBSD 6.3-RELEASE and 7.0-RELEASE to Panasonic Note PC CF-R4. This machine has been known for having an incompatible BIOS which prevents FreeBSD BTX 1.01 or earlier to boot.

Recent FreeBSD 8.0-CURRENT snapshot ISO images are with BTX 1.02, which allows CF-R4 booting from USB CD-ROM drives. So I tested the following procedure for the 6.3 and 7.0 releases and the both releases were successfully installed:

• boot from a CD-R(W) with 8.0-CURRENT-200805-i386-bootonly.iso (or later)
  
• exit the boot prompt screen to the loader prompt (by pressing 6) before the 8.0 kernel boots itself
  
• enter unload to clear the loaded (8.0) kernel
  
• replace the CD-R(W) in the drive to the target installation image (e.g., Disc 1 of 6.3/7.0-RELEASE)
  
• enter boot-conf to load the target kernel and install
  

I think this is the compatibility problem of BTX. I also think this is so far the easiest way to boot from the USB CD-ROM drive for CF-R4, because generating ISO images for previous releases with the compatible BTX loader will be a big hassle. You may give it a try if you want to boot from a USB flash memory.

Learning Erlang

I've been learning the Erlang programming language.

I've tested an IPv6 address-to-domain-name conversion for the reverse lookups. I made a presentation at the 1000speakers:4 conference on April 26, 2008. The presentation slide set in PDF is here.

The days of Neuro Net Recordings

Neuro Net Recordings (NNR), an online techno-music distribution project, is one of the most important music projects of my life. Many music files of NNR have been available at archive.org as a part of the open source music archive. It was founded by Tomotoshi "Ackie" Akiyama and other core members including myself, in 1994. I was the Internet Archivist, or the one who handled the online distribution channels over Internet.

NNR had the free online distribution policy from the beginning: NNR had the non-exclusive distibution rights of the music files in the various available formats on the Internet. It was quite similar to the Creative Commons BY-NC-ND license, which means preserving the author's credit, non-profit use only, and changes not allowed during redistribution.

NNR was not really a for-profit label, but most of the albums of NNR Active Potential Series were distributed by cassette tapes with reasonable price (JPY500 for Active Potential Number 0). NNR also released one CD album called Neural Networks Vol. 1 in 1996. So NNR was really an independent label, though NNR consistently distributed the music files towards the potential listeners on the Internet. NNR once had a redistribution agreement with FranceMP3.com. Jean-François Fecteau has been a long-time supporter of NNR activities, and he has played NNR tunes many times in his program Le Vestibule from CFOU-FM in Trois-Rivières, Québec, Canada.

The distribution format NNR chosen were RealAudio, MPEG-1, Ogg Vorbis, and MP3 finally, because MP3 technology license owners allowed exemption of the MP3 technology royalty fee to the business with USD100000 or less annual revenue. Ackie kindly sent me the past DATs for mastering NNR cassette tapes, so I rebuilt and remastered the CD-R master discs from the DATs. I learned the remastering skills using Syntrillium's Cool Edit (now Adobe Audition) and other tools.

I also performed real-time streaming events called Neuro Net Radio twice on 1998 and 1999, which were actually DJing from my house over RealAudio streaming, playing only NNR tunes to avoid dealing copyright issues with the copyright holders.

Many professional music creators contributed to the NNR music archive before and during they became really professional. Some of the artists are: NISH, Leopaldon, Takuya Kitani, and Warping Allergen. I find many NNR contributors are still actively seeking their other opportunities in their lives.

In the age of iTunes and Amazon.com MP3, no professional musician can survive without distributing their music online. Streaming music radio stations like Soma FM are doing competitive business. Many people in the so-called music industry, however, still do not accept online media, and I feel quite sad about it. I've already been fed up with the stagnated copyright issues in Japanese music scene either. So I decided to quit distributing NNR files on my own in 2004. Fortunately, archive.org generously provides the storage space and other goodies to the free-music distributors, so I decided to put NNR and my music pieces there in 2007.

I know my talent on music is quite limited and is not competitive enough to survive as a professional musician. But I also believe that does not limit my rights to freely distribute my own music. I've learned a lot through my NNR activities and I appreciate all those who helped NNR.

Fumi Yamazaki suggested me to write this, as a case study of Creative Commons activities in Japan. Thanks, Fumi.

META refreshing considered spamdexing

I am writing this as a memo for myself, and a message to Blogger.com reviewers of my old (blocked) blog at macrofield.blogspot.com.

I realize the whole issue is about Meta refreshing of HTML. This has been a common practice for a long time to redirect users from an old page to a new page. The problem of this practice, however, is that users will have difficulties on intervening the redirection.

I embedded a Meta-refreshing tag in my old (now blocked) blog for an automatic redirection. As seen in the Blogger.com's definition of spam blog, this redirection is considered spamdexing, a violation of Blogger.com service terms.

I think no one likes spamming any more and neither do I, but I should confess that I didn't study well on how Meta-refreshing is considered abusive these days in the HTML writing practice. I will remove the embedded META tag as soon as Blogger.com allows me to reedit the contents, or to be specific, to reedit the layout HTML tags of the old blog of mine.

And I suggest you all to stop using Meta refreshing. Being misunderstood as a spamdexing is doing nothing good for you, whatever your intention might be.

Possible false detection of spamming on Blogger.com

This is an administrative announcement.

Due to the automatic classification of possible blog abuse on Blogger.com, the old URL of this blog, macrofield.blogspot.com. has been disabled by Blogger.com. I've already requested Blogger.com to unlock the contents under the old URL. I will make a necessary fix ASAP. I'm sorry for the inconvenience.

Note: on July 2008, I have confirmed this blockage has been dropped.

Gauche: yet another Scheme-a-tic diagram of programming (self-)education

I feel programmers in the 21th century prefer so-called dynamic, scripting or light-weight programming languages to the good-old static, compilation-based, and heavy-weight ones. In this sense, I belong to the old-style programmers, who like C, awk, and simple Bourne Shell. So far the most complex light-weight language I've used is Perl.

I like definitive or pre-defined syntax and keywords than those which may change the meanings by the programmer's definition, because the dynamic definition will make the program too complicated. Maybe my brain is losing flexibility, but I'd rather want to read something which can be easily decoded than the cryptic ones.

I didn't have a chance to formally learn programming, however, during my education. This is both good and bad: the good thing is that you don't have to spend time on learning subjects which you won't use throughout the rest of your life; and the bad thing is that you may have no chance to know the coding practice much efficient than what you usually write.

Some programming languages still irritate me. I should confess I don't like Prolog. I will be a very poor programmer if I have to write only rules between the objects than the way of moving around and cutting and pasting them. Thinking about the backtracking facility is a nightmare for me. I'm 99.9% (if not 100%) filled with the idea that every statement of a programming language will have a side effect, or may cause changes of the contents of the preallocated memory space. I still don't believe in the idea that no assignment is needed for a real-world computing.

I still think, however, the idea of functional languages which is focused on the relationship of data than copying and manipulating them back and forth, is an essential part of the modern computer programming. Abstraction work of data often simplifies the complex problems. Writing things in recursion than an arbitrary loop will simplify the code and often makes it efficient.

Recently I've learned many young programmers are learning a lot from LISP and Scheme, and I've heard some of them have learned Scheme in their formal education process. I welcome that trend and I hope it continues to grow, because in the real-world programming paradigm there's large room for optimization by introducing Scheme-style abstraction, such as describing an XML document in an S expression.

Gauche, an implementation of Scheme by Shiro Kawai and other developers, is a well-balanced mixture of the LISP-Scheme tradition of simplicity and a Perl-like feature, such as being able to handle Unicode, regular expressions, network sockets, and simple byte-aligned buffer structures which is essential for network programming.

Gauche is a viable alternative to scripting languages like Perl; and it's still a Scheme implementation, so an old-minded programmer like me has many things to learn and enjoy. Gauche has a lot of modules or external libraries to handle real-world problems, such as manipulating database engines and HTML/XML data structures. It took me only three hours to rewrite a Japanese input method server program in C by Gauche, thanks to the modules. Gauche and Scheme designers also know that programmers want to use assignments, so an old-headed guy like me didn't have to scratch the head forever.

In 1989, when I first met a Symbolics LISP machine, it didn't attract me much because the machine was something completely different from the UNIX workstations, which I wanted to use to communicate with my friends and gurus. I realize, however, things have changed a lot now; you can play around and even solve the real-world issues on a LISP-Scheme processing environment like Gauche. The vast increase of processing power including the CPU speed and the amount of main memory make this possible for everyone. And I think it's a new Scheme-a-tic diagram for a competitive programmer; you will become much more productive after learning practical data abstraction.

Gotchas, gotchas, gotchas

I've learned a lot of hard cold facts on how to deal with multimedia systems. Here are the examples:

• White fungi can grow on a CD medium surface and even often render the media unreadable, though in most cases the medium can be rescued by wiping out the fungi. (Be careful not to damage the acrylic surface!)
  
• Windows XP goes into an infinite loop (100% usage of a CPU core) to try to read an unreadable CD medium. If you don't have a multi-core machine, you'll be getting into a trouble. Why the CD/DVD drive cannot tell you that the media seems to have a problem, rather than just simply retry and wait for the timeout periods?
  
• iPod 5G firmware Version 1.2.3 can crash and reboot forever, just because the album title field of an MP3 file contains an unreadable/irregular character. The iTunes 7.6.1 for Windows is running smoothly without any error message even with the MP3 file, so you can't tell what's really wrong until you try-and-fail the whole possible cases.
  
• You cannot easily attach still pictures to an audio file without external software. And I realize that's the meaning of the word authoring of a streaming video.
  
• You have to make a set of still pictures into motion pictures or a video stream first first, before syncing with an audio stream and make them a video stream, to be broadcast on the Internet, such as on YouTube. (A good news is that you can do this by a set of the open source software, such as mjpegtools and Avidemux.)
  
• Gracenote CDDB has been broken for WinAmp, so I have to migrate into iTunes, just to pick up the metadata of each song I'm going to put into my lovely iPod. I like my iPod very much, but iTunes for Windows is really a big slow program and I hope it was a bit smaller and faster.
  

And I even learned some security and systems administration tips:

• Many of PCI Serial-ATA cards cannot be used for operating systems other than Windows.
  
• A 13-year old hard disk can spin up and is readable even if you once put it in your closet and left it unplugged for 10 years.
  
• As of 2008, you can hardly find a SCSI interface card for a PC, at least in Osaka.
  

Then I remember the truth: your computer could be the number one device to consume your time and kill your productivity. So how many gotchas do you get every day? Ten? Or a hundred? And how many hours or even days do you lose?

Teleworking technology: we already have it

I've been thinking about months for what technology can help people to discover and get the most out of teleworking. I'm getting closer and closer to the following conclusion:

We've already built teleworking infrastructures, at least in my home country Japan, and many other nations and regions which have already deployed the Internet. Period. We need to focus on how to break the barriers and remove the impediments to people to embrace teleworking technologies as their own tools.

Let me simply put in this way: teleworking is no longer something special. For example, writing this blog itself is a very good example of teleworking technologies. All I need is a decent Web browser capable to run JavaScript for Blogger's writing environment; and I can publish and show you what I write, like what you read now.

Collaboration is getting more and more easier also. LUNARR allows you to flip back an electronic document and let you and other people write something on the back. This is a rather intuitive way to co-author a document, which I had never thought about until I saw it. Geeks will do the whole things in more geeky ways using SSH, rsync, CVS, subversion, or whatever else for software development, but it's not only for geeks anymore. Anyone who has to write a document can do it on a teleworking environment.

So let's stop saying "I'm not allowed to work from (wherever you want other than your office) because blah blah blah..." and talk how you can leverage your productivity from introducing teleworking into your lifestyle. If you and your employer think teleworking is something restricted for special people, that mindset is archaic, and no longer applicable to the 21st-century version of modern lifestyle. And let's not take commuting as a duty; it should be a choice and it has to be, to reduce the gross amount of time and energy wasted by being forced to commute.

But beware that teleworking is not necessarily a duty either; don't do it like this as in an xkcd article. Don't leave your partner alone just because "something is wrong on/with the Internet" (grin).

The art and limit of dependencies

If you want to get a job done, you need to list up the necessary things and tools, and procedures to use and apply them. In other words, you have many dependencies on them.

In a modern computer software development, no piece of software can withstand without any dependency to other pieces. You do not want to make a C program without the standard I/O library. Dependencies to the tools are also important and critical; if your codes include those written in FORTRAN, C, C++, and Java, you need the four compilers and language execution environments.

Computer programmers have been making the tools to automatically resolve the dependencies. make is a popular one, derived from the UNIX programming environment, which parses the rulesets called Makefile and determine whether if you need to rebuild a result from the source files, by comparing the timestamps. If make finds out one of the source files are newer than the result, it will invoke the command to rebuild the result.

Version control systems such as subversion, CVS, and RCS, are another good examples of dependency management tools. You can save the history of changes on a file, a directory, or a set of directories. You can make a software package by checking out a set of tagged files; you can even make multiple branches of a code.

FreeBSD operating system has its own dependency management system of externally-contributed programs called ports and packages. A port means a set of rules, configuration files, and source codes necessary to build a program. A package means the derivative of the port, built by another computer. Many essential parts of FreeBSD subsystems, including the X Window and Perl programming language, are installed as packages, because they are not considered as the core parts of FreeBSD.

I'm always pleasantly surprised when a very complicated port, such as Japanese version of LaTeX, a typesetting and documentation set of programs, can be built without major glitches, including the automatic installation of depending programs, such as Ghostscript. For most of the ports, FreeBSD volunteers are always doing the outstanding jobs.

I feel very much annoyed, however, when I have to untangle the web of dependencies when the installation/building of a software toolset from the scratch. Unfortunately, Xorg 7.3, a free implementation set of X Window programs, fonts, and tools, was not able to build in my environment. I had to copy three prebuilt sets of files: include files of C/C++, shared/static libraries, and the very basic fonts, to finish building the necessary programs.

In my case of Xorg kitbuilding, some very old fonts existing since late 1980s were not successfully compiled and converted into various ISO8859 part codesets. And I could not build the font handling libraries. This glitch killed the whole automatic compilation task, which was supposed to untangle the enormous list of dependencies. I also found that the target directory name was merged to /usr/local from the traditional /usr/X11R6, and I wasn't sure whether if I moved the old files and subdirectories under the old directory safely to the new directory.

Untangling the web of dependency is a very hard task, since the parameters you need to examine are scattering around all the directories, in all the ports, and you often also have to set an environment variable to do something extraordinary. While I can guess what I should do because I've done a lot of this kind of tasks, I never want to do this at any time because scanning your memory and poking around the files are very much mentally painful. In a recent hostile computer environment, vulnerabilities can easily sneak into such a complicated and unformalized tasks.

How do you manage your computers? Do you care about the dependencies of the tools, programs, configuration files, and other objects?

Accountability of an SNS user

In a major Japanese SNS mixi, I feel I have been always exposed to a strong peer pressure, which I find common in Japanese society, that you should not argue with others unless you really want to break up with those people. This sort of fuzzy feelings covers over the whole society of Japan, including that on the Internet.

I really don't like the fuzzy Japanese social atmosphere, though I don't favor the hostile and negative environment which I always find in anonymous and open bulletin boards and blogs in Japan nowadays either. I always want to have a creative discussion with a constructive criticism. People tend to go the opposite way, however, in most of the times. This is what I've learned both from the real and Internet communications.

One of my friends claimed to me that the warm fuzziness surrounded him in 1980s in the Japanese online community of bulletin boards was something completely different from that after the popularization of Internet in Japan. He told me the fuzziness which I don't like actually helped the self-governance of the users. I asked him why.

He told me that the bulletin board operators could individually locate and even persecute each account holder, by exchanging the written agreement in paper, and charging access fee individually from each user's bank account, automatically drawn each month. He said he had many anonymous or pseudonymous friends there, because he could trust those people even without knowing the real names, by the guarantee of financial and social accountabilities for each users that the BBS operator provided. In other words, the participants were supposed to behave nicely online, and the participants could prevent each other from the extreme cases of disputes.

The friend and I agreed, however, that you could no longer expect such a high degree of individual responsibility on the current SNSes and other Internet communities. Many people belong to many different systems, and those systems are usually supposed to keep the personal information undisclosed except for a legal request for the authorities. This means all disputes between Internet participants have to become legal issues, or lawsuits.

And even if you win a legal battle, you can't really collect the compensation justified by the law, unless the law enforcement officers really pursue the defendant to do so. The legally-responsible individual of one of the largest anonymous BBSes in Japan keeps refusing paying the money he has been requested from the court, though he loses on multiple lawsuits, and he is actually winning the battle by not adhering to the legal requests, because those lawsuits are all civil law issues and not the criminal ones. This is a good example of the limitation of depending resolution of individual disputes over Internet on the legal procedures. Forcing people to use "real" names will not solve this problem at all.

How can an SNS user be held accountable, for his/her legal and social activities? This is not a direct technology issue, but technology can and must help the issue to be resolved. And I think tagging individuals and restricting online identifiers do not necessarily effectively work on this issue.

CALLing disaster during MySQL upgrade

I've been upgrading one of my servers for the daily use, by migrating the running environment between two PCs. It's not a mission critical server because I don't run a public service there, but I need to do the upgrade carefully anyway because the version number of running FreeBSD and other applications have been changed. I want to keep the old environment as long as possible to use it as a reference, so I'm doing the migration manually by recompiling and reconfiguring the software.

One of the glitches I faced was about MySQL. It's not about the bugs in MySQL, because the database server load is very small. I was trying to transfer a dataset from the old version 4.0 to a new version 5.1 software of MySQL. The mysqldump result of the old 4.0 output didn't get through and reloaded into the 5.1 server. I could not even perform CREATE TABLE. The reason: the column and database identifiers were not properly backquoted.

The database was for my amateur radio activity. Amateur radio stations have callsigns, and in a popular contacting log exchange format called ADIF, the other party's callsign is represented by the identifier CALL. Unfortunately, MySQL 5.1 made the word CALL a reserved word, while in the 4.0 version the word was apparently not. This is a tragedy for an amateur radio enthusiast, and a careless programmer like me who tends to omit proper (back-)quotation.

After an investigation for a few minutes, an idea of referring to the result SHOW CREATE TABLE from the 4.0 server came into my mind. I did it and fortunately the table definition was properly backquoted, so at least I could rebuild the database skeleton. The dumped data was a set of INSERT INTO statements with the VALUES and they were all properly quoted, so I could rebuild the database.

The new mysqldump command's output of MySQL 5.1 looks much better and properly backquoted and quoted all the necessary strings, and even put the SQL statements to lock and unlock the database. The entire dataset is represented by a single INSERT INTO statement VALUES set, so you've got to be careful when you want to use the data set not in the full contents.

I should note that I had to rewrite all the software which generated the SQL statements for the proper (back-)quotation. This was a handful of complicated tasks of fixing the Bourne shell and Perl scripts.

And I realize why SQL injection is so popular for attacking the database servers. Parsing SQL correctly is a non-trivial process. A word can be either a part of a directive or a target identifier, depending on the position where it is in an SQL statement.

So when you want to store CALLs into a MySQL database, you've got to do it carefully with (back-)quotation.

Data erasing woes

Clearing up the used disk has been a real burden for sysadmins, as exploitation of carelessly-discarded information becomes popular. In Japan, people are extremely sensitive against how their personal data records are kept secret and controlled. The reality is, however, that data disclosure incidents keep happening, no matter the government enforces the law, and companies make frequent apologies for those incidents. Once the data get out, you won't be able to take them back.

I think a part of the reason of continuing data disclosure incidents is that erasing data is simply a difficult task. For example, even if you store only 10G bytes into a 160G-byte disk, you need to thoroughly sweep the whole 160G bytes, to guarantee all data are cleared. Also, you've got to be careful to deal with the hidden data, which is not accessible by the data erasing software. If you really want to make the data not recoverable at all, you need to erase the trace of electromagnetic residue on the hard disk platters, which is hard to perform in a usual business or office environment. Destroying a usable device is not an environmentally-friendly practice either.

Another problem is that the sweeping process is slow. The physical writing speed governs the whole performance. For example, one of my old portable 40G-byte hard disk can be written in only 20M bytes per second. So it will take at least 2000 seconds or about 34 minutes to sweep out the whole disk once. You need to do this multiple times to ensure the residue of data is not easily detectable, so the whole process may take 2 or 3 hours. 3 hours for just 40G bytes. 12 hours for 160G bytes. (sigh)

The data storage which an individual has to manage is getting bigger and bigger every year. I wonder how people can cope with this. Do you sweep and erase the used disk data before you resell or give it away to somebody?

Simple access-control status display

I had a chance to talk to my business partners about computer networking security. One of the participants told me that a secure networks should not be mixed up with the unsecure ones. He said in a trusted operating system environment, you would see a red window and a green window, when you handle the different security levels of the information in the windows. You can't copy-and-paste the pieces between the red and green windows, unless an overriding procedure is accepted, and recorded. While I think you can do something more to assist visually-handicapped people, this simple definition of security models should be incorporated with many Web and other Internet applications as soon as possible.

A few years ago I had a chance to see a presentation of similar ideas [1], which described the difference between Microsoft and Symantec products on telling the security level of the Web pages to the users. I thought Symantec's simple color-based access control looked much better than the IE's massive pairs of (not necessarily trivial) parameters.

Access control is often very complex, but the ideas should easily get through to the users. This simple principle has not been popular yet in the world of computer software and Internet applications.

Related link:
[1]Steven Furnell: Fostering the usability of information security solutions, at OECD-APEC Workshop on Security of Information Systems and Networks, Seoul, Korea, 6 September 2005.

A speculation: SMTP may disappear from the global Internet soon

Long time ago in the late 1990s, I was a system administrator, maintaining a bunch of SMTP (mail transfer agent) servers. Since 2007 I have quitted running SMTP servers at home. The number of SPAM messages have rapidly increased since then; 95% of my received mail messages are considered SPAM and unread now. I've been watching this on multiple mailboxes using different SPAM-filtering algorithms, and the trend is mostly the same. Only 5% of mail messages are meaningful to me, including those from my business partners and mailing lists.

On a DNS operator meeting last year, I was joking to one of the participants that the current SMTP (email) and DNS will cease to exist much sooner than we expect, due to the complexity of the running systems. People tend to believe they are always up and running, but the reality is that a lot of sysadmins are killing themselves to get the systems up and running.

I think DNS surely has a lot of economic incentives to keep it alive as it is now. For example, domain names are protected under intellectual property rights and the international treaties. So somebody will keep the system alive, although the implementation details may change.

On the other hand, many people have already seen the limitation of email; they are not pervasive as they were a decade ago. People are fed up with SPAM, and try not to rely on email too much. Many of them have already been using other messaging systems, such as the one implemented in various SNSes. In other words, they put a lower priority to email in general.

Some people say the easiest way to deal with SPAMmers is to constantly change the mail addresses, which I do not think so. I think SPAMmers nowadays exploit all the traceable resources of mail addresses, so the leading time you can gain by changing the email address is getting smaller and smaller every day. The hardest part to fix, and the impossible part to fix, is the promiscuity of SMTP, to accept everything without authentication. Many people have said this and I know this is a mere repetition, but it is a fact we must recognize.

Recently I tend to feel more than ever that I really don't care about the medium or protocol, as long as I can stay connected to the people who I want to. Of course it might get harder to meet new unknown people and communities, but I think that role has already been replaced by open blogs and the update notifiers or RSS readers.

The fall and decline of Mixi

How Alexa finds the difference between mixi.jp, facebook.com, and myspace.com.

You will see how the Japanese SNS giant Mixi is smaller than the rest of the world. Even worse, Mixi is losing the users, at least from the PC/Internet domain.

Mixi's markering target is shifting towards the Keitai (cell phone) users, which will not show up on the Alexa pageview statistics. I think this is a bad change of tactics, at least for their business. The number of Keitai users (of cell-phone users in and only in Japan) has already been saturated and does not have much room to grow any more. If Mixi depends on the ad business, the revenue will decline.

And I feel the emptiness of Mixi every day as one of the users; less number of diary articles, less frequent updates, and the persistent scanning by bots. It's getting more and more like a deserted city.

New Year Resolution 2008

Dear Kenji: this is your new year resolution for 2008.

a) Love your job. Love computers, coding, experimenting, and hacking.

b) Redesign your financial portfolio, and how to earn money.

c) Re-learn abstraction and principles of Internet and electronics.

d) Continue reducing home computer system; downsize and outsource.

e) Optimize online activities; publicize your works and minimize rants.

f) Do not waste time on DXing; spend time for more creative activities.

g) Practice DJ and recordings; stop wasting time on thinking about music.

h) Always STAY FIT; eat less, go out of your house and move around.

i) Interact with real people; love your family, respect your friends, and nurture productive relationships.