Showing posts with label sysadmin. Show all posts
Showing posts with label sysadmin. Show all posts

Monday, March 5, 2012

Erlang VM memory size and ERL_MAX_PORTS

Erlang VM has the ports to communicate with other programs and objects (files, drivers, etc.) The ports are seen as the Erlang processes. The OS environment variable ERL_MAX_PORTS sets the default value of maximum number of ports. (See the manual for the function erlang:open_port/2 for the further details.)

I discovered by accident that Erlang VM of R15B on FreeBSD/amd64 9.0 in my test environment sets the value of maximum file descriptors assigned to the OS process as the default value for ERL_MAX_PORTS. This means ERL_MAX_PORTS is set to 800000, far larger than 1024, specified in the manual. This excessive value of the ERL_MAX_PORTS leads into unnecessary memory consumption of the Erlang VM. An easy way to test this is to execute the following code on the shell:

env ERL_MAX_PORTS=[given value] erl -noinput -eval 'io:format("~p~n",[erlang:memory(system)]).' -s erlang halt

How memory consumption changes just after the Erlang VM startup, as {ERL_MAX_PORTS, memory size in bytes}: {1024, ~65M}, {102400, ~124M}, {800000, ~543M}.

So I decided to set the ERL_MAX_PORTS for my home Yaws server to 4096. This significantly reduced the memory consumption.

I suggest you to tune the value of ERL_MAX_PORTS to optimize the number of concurrently opening ports of an Erlang VM, especially when you run a server on a memory-restricted host.

Update 11-MAR-2013: on R16B, the amount of memory consumed by the ports will be significantly reduced at the Erlang VM startup.

References

Sunday, January 8, 2012

Leap second, again

News: There will be another leap second on 23:59:60UTC 31-JUN-2012.

Nevertheless, I quitted supporting leap second on the time zone file at home last year on December 2011, which had been introduced for more than 12 years. Reason: I gain very little while I lose quite a lot by setting the system time based on TAI.

I think POSIX should have defined the timezone functions to handle leap seconds properly, by setting the TAI-based monotonically-increasing value to the local clock, and by correcting the difference between UTC and TAI with the timezone file. But POSIX doesn't. So TAI-based local time is not interchangeable. Most applications assume the value of time_t represents UTC without the leap seconds, which is not the case when the local clock is set based on TAI.

I suspect that many sysadmins are not really aware of the leap seconds, unfortunately, including those provide NTP services. I've experienced one of the upper-stratum NTP servers of my ISP did not synchronize on 1-JAN-2009, when the last leap second was added.

Very few language systems can explicitly handle leap seconds either. For example, I had to post a patch for Erlang for handling TAI. (Fixed on Erlang/OTP R13A and later.)

So you should be very careful on the de-synchronization of your machine's local clocks on June 30th or July 1st this year on 2012.

Useful links:

(Correction 8-JAN-2012: gmtime() was ambiguous. Corrected to time_t.)

Wednesday, January 4, 2012

Preparing FreeBSD Port of Erlang/OTP

Updating an Erlang/OTP Port (files under the Ports tree) for FreeBSD is not a complicated task, with the latest pieces of the Port files. You can find out the latest pieces at the FreeBSD CVSweb page for Erlang Port.

You need to do the following things:

Updating the source file names

The following parts of the main Makefile must be modified to specify the related source file names (in a unified diff-like format):

    -PORTVERSION=   r14b04
    +PORTVERSION=   r15b

    -DISTNAME=      otp_src_R14B04
    +DISTNAME=      otp_src_R15B

    -ERLANG_MAN=    otp_doc_man_R14B04.tar.gz
    +ERLANG_MAN=    otp_doc_man_R15B.tar.gz

    -ERLANG_DOCS=   otp_doc_html_R14B04.tar.gz
    +ERLANG_DOCS=   otp_doc_html_R15B.tar.gz

Preparing the library list

Makefile.lib must be supplied to provide all the version numbers of the OTP library modules.

To extract the version number, use the following shell command:

    (cd work/otp_src_VERSION; \
     find . -name vsn.mk -print | xargs grep VSN) \
     > OTP-vsnlist.txt

The content of OTP-vsnlist.txt will be something like this:

    ./erts/vsn.mk:VSN = 5.9
    ./erts/vsn.mk:SYSTEM_VSN = R15B
    ./lib/appmon/vsn.mk:APPMON_VSN = 2.1.14
    ./lib/asn1/vsn.mk:ASN1_VSN = 1.6.19
    ./lib/common_test/vsn.mk:COMMON_TEST_VSN = 1.6
    ./lib/compiler/vsn.mk:COMPILER_VSN = 4.8
    ./lib/cosEvent/vsn.mk:COSEVENT_VSN = 2.1.12
    ./lib/cosEventDomain/vsn.mk:COSEVENTDOMAIN_VSN = 1.1.12
    ./lib/cosFileTransfer/vsn.mk:COSFILETRANSFER_VSN = 1.1.13
    ./lib/cosNotification/vsn.mk:COSNOTIFICATION_VSN = 1.1.18
    [...]
    ./lib/wx/vsn.mk:WX_VSN = 0.99.1
    ./lib/xmerl/vsn.mk:XMERL_VSN = 1.3

You have to manually edit the Makefile.lib with the content of OTP-vsnlist.txt. The format of Makefile.lib is (beware that the last line has no backslash):

    ERTS_VSN=       5.9
    TOOLS_VSN=      2.6.6.6
    OTP_LIBS=       appmon-2.1.14 \
                    asn1-1.6.19 \
                    common_test-1.6 \
                    compiler-4.8 \
                    cosEvent-2.1.12 \
                    cosEventDomain-1.1.12 \
                    cosFileTransfer-1.1.13 \
                    cosNotification-1.1.18 \
                    [...]
                    wx-0.99.1 \
                    xmerl-1.3

The contents of Makefile.lib is a set of Makefile variables, so all the module names must be listed with the backslash character. (Note: ERTS_VSN is no longer needed, but I will include it here just FYI.)

Preparing the man file list

Makefile.man must be supplied to provide all the version numbers of the OTP library modules.

To extract the man filenames, use the following shell command:

    tar ztf /usr/ports/distfiles/erlang/otp_doc_man_VERSION.tar.gz | \
    sort | awk '{FS="/"; print $3;}' > OTP-manlist.txt

You have to manually edit the Makefile.man with the content of OTP-manlist.txt. For OTP R15B, Makefile.man only needs man sections 1, 3, 4, and 6, so man section 7 must not be included.

The content of OTP-manlist.txt will be something like this:

COPYRIGHT
PR.template
README

man1/
man1/ct_run.1
man1/diameter_compile.1
man1/epmd.1
man1/erl.1
man1/erl_call.1
man1/erlc.1
[...]
man7/SNMP-VIEW-BASED-ACM-MIB.7
man7/SNMPv2-MIB.7
man7/SNMPv2-TM.7
man7/STANDARD-MIB.7
man7/TRANSPORT-ADDRESS-MIB.7

You have to manually edit the Makefile.man with the content of OTP-manlist.txt. The format of Makefile.man is (beware that the last line of each section has no backslash):

    MAN1=   ct_run.1 \
            diameter_compile.1 \
            epmd.1 \
            erl.1 \
            erl_call.1 \
            erlc.1 \
            erlsrv.1 \
            escript.1 \
            run_erl.1 \
            snmpc.1 \
            [...]
            start_webtool.1 \
            werl.1

    MAN3=   CosEventChannelAdmin.3 \
            CosEventChannelAdmin_ConsumerAdmin.3 \
            [...]
            zlib.3 \
            zlib_stub.3

    MAN4=   app.4 \
            appup.4 \
            [...]
            relup.4 \
            script.4

    MAN6=   common_test.6 \
            crypto.6 \
            [...]
            stdlib.6 \
            test_server.6

The contents of Makefile.man is a set of Makefile variables, so all the man file names must be listed with the backslash character.

Removing/editing/adding patches

FreeBSD-specific patch files are provided under the Port directory files. You need to remove the obsolete patches first, by make patch. You should edit the existing ones and add the new ones as well.

The naming convention of the patch file is:

    patch-lib_et_src_et__gs__contents__viewer.erl

for

    lib/et/src/et_gs_contents_viewer.erl

(Conversion of / -> _ and _ -> __ is needed.)

Updating the checksum file

make makesum is the convenient way to update the distinfo checksum data.

Preparing the package list

New binary files are occasionally added in the OTP distributions. The pkg-plist file must be updated as well.

Test building

The building process will be like this (note the umask 022):

    # I use zsh, but use whatever you want
    # as root
    sudo zsh
    umask 022
    make fetch
    exit
    # as your own account
    umask 022
    make
    # as root
    sudo zsh
    umask 022
    make deinstall && make reinstall
    exit

(Note added 5-MAR-2012: I've posted a send-pr (ports/163711) to FreeBSD issue tracker system on 30-DEC-2011. I've made an Evernote page for the port tar.gz archive.)

References

Sunday, January 30, 2011

Cloudsourcing from home

Managing the information and services which I provide has been a persistent source of headache since I set up my own Web server at home in 1997. So far I've been successfully managing the contents, such as:

  • External authoritative DNS (though you still need to maintain the internal authoritative DNS and the cache servers)
  • Mail receiving servers (or more formally called the message transfer agents or MTAs) and the complex forwarding schemes for my family, namely me and Kyoko
  • Static web servers solely for the public contents, including this blog

I still have to pursue moving out the servers from home to the external sites, however, because managing new protocols from home such as IPv6 and other new applications is increasingly getting difficult, due to the constraints of available bandwidth, IP address space, and my own time.

Frankly speaking, I do not want to buy any more computer at home. More accurately, I do not want to install and maintain operating systems and the application software at home unless I really have to. Things have already been getting too complex, and I need to offset or outsource the complexity to external sites and services.

The problem I'm facing is that cloudsourcing or moving out services provided at my home servers adds many levels of indirection and layers of things to consider: availability and redundancy issues, association of the services, how and where to consistently collect the backup data, and the contingency plan and the recovery procedure in case of service disruption.

I know the cloudsourcing issue is a matter of business reengineering process, and to redesign the workflows of the various activities, in my family. It's not only about the computer systems. We need to change our way of living.

And I finally started to learn what kind of things you can do on Google Apps, Google AppEngine, and Amazon Web Services.

Friday, December 31, 2010

My new challenge in 2010: security issues at university network systems

Since April 2010, I've been working on the network security management of Kyoto University. There are many things to consider. I've already published a report in PDF about the status and issues.

The fundamental principle which always haunts me is that adding security to a not-so-secure system is far more difficult than making the security component built-in into the system. In a large organization, you cannot change the policies and configuration of the running systems overnight; you need to negotiate with a lot of stakeholders to reach a consensus or an agreement. Consolidation takes a lot of time, especially for a well-established system such as the email servers.

Another difficult issue to handle is how not to restrict users from the legitimate use of the campus systems. University is an organization of research and education. While leaving a system vulnerable for the known attack vectors is not an option anymore, allowing failures for learning from the try-and-error process should not be prohibited. Without a firm security policy, you cannot really decide what is right or wrong.

Catching up with new technologies is also a tough requirement to meet for a large network system. The global IPv4 address space will be used up in next year 2011, so introducing IPv6 is a must, though not urgent; and the organizational DNS subsystems should be DNSSEC-ready, for both the cache resolvers and authoritative servers. And I have to be able to explain those changes will surely benefit the users and are worth paying for.

I'm sure I'm going to face a lot of existing and new problems on next year 2011. And I'm hoping I will be able to solve at least some of them.

Monday, December 21, 2009

DNS operation is utterly neglected by many people

Twitter outage via DNS hijacking showed another case of common symptom: DNS operation is simply neglected by people doing business on the Internet.

I was doing research on DNS transport security from 2002 to 2008. One of the reason I quit focusing on the research was that most, if not all, of the DNS problems are caused by operation failures, not necessarily due to technical deficiency of the DNS protocols and systems. In short, it's too political and social to do the technological experiments over DNS.

I still think DNS transport protocol issues are critical for stable Internet operation. But solving those issues does not help recovering human errors, such as lame delegation (missing link) between the domain name hierarchy. And stable operation of DNS systems is very difficult to maintain without stable hardware, software, networks, and operators.

I notice many small companies (especially in Japan) keep their authoritative servers inside their office, which is not good from the stability point of view. Actually, for many small Internet sites, including mine, not so many DNS zone records have to be exposed to the public. So I've already outsourced the DNS authoritative servers, while I periodically watch whether those servers do the right thing.

DNS is by definition a distributed system; and the management standard is much lower than what people (and even Internet engineers) believe. For the further details of how DNS is not well-managed, I suggest you to read a more detailed commentary on how important DNS is as an asset, by Danny McPherson of Arbor Networks.

Saturday, January 10, 2009

The risks of systems left alone and untested

Computer systems left alone unmaintained are a premier source of risks. Those systems may cause a serious crisis and a major service disruption.

On September 14, 2008, All Nippon Airways (ANA), a major Japanese airline, caused the disruption of the ticketing service due to the cryptographic function software expiration (as they announced in the Japanese press release), which is logically assumable about the PKI certificate, according to the other Japanese-written press reports like ITmedia's and Nikkei ITpro's.

The chilling fact revealed was that the ANA left the cryptographic function unused for 2 years and did not make a review about expiration at all when they activated the function for the terminals used by the ticketing agents. This is an awful example of software development indeed.

I wrote about the service disruption for RISKS-DIGEST 25.34 just after the incident occurred.  And recently I knew the article was quoted by another blog article yesterday.

The expiration issue is not only about the PKI certificate; domain name registration is another source of expiration risks. An expired domain can be abused for phishing and overtaken by attackers.  Software license is another good example.  In general, Expiration is a part of overall misconfiguration.  So when did you review the expiration date of the system resources under your control last time?

Thursday, July 24, 2008

Outsourcing my DNS lookup service to OpenDNS

DNS subsystem is a real headache to manage. You will realize that once you have to tweak the system, especially if you want to incorporate your own filtering/access-blocking rules.

And the DNS Cache Poisoning. It's a real threat.

While I believe my DNS subsystem is quite safe under the good-old djbdns servers, I am now testing the OpenDNS, a DNS cache service provider. One of the good things about OpenDNS is that they even allow a single-IPv4 address network to be individually managed, even it's dynamically allocated, as in most of the cases for non-static-IPv4 users.

If you can trust your ISP for the DNS management, you are on your own. But if you can't or don't, OpenDNS is a good alternative. I notice many ISPs still have not changed their DNS cache servers to prevent the poisoning attack as of today (July 24, 2008); using OpenDNS from such a mobile networking environment will make the whole DNS access much secure.

Saturday, May 17, 2008

Installing FreeBSD 6.3 and 7.0 to Panasonic CF-R4

I had difficulties installing FreeBSD 6.3-RELEASE and 7.0-RELEASE to Panasonic Note PC CF-R4. This machine has been known for having an incompatible BIOS which prevents FreeBSD BTX 1.01 or earlier to boot.

Recent FreeBSD 8.0-CURRENT snapshot ISO images are with BTX 1.02, which allows CF-R4 booting from USB CD-ROM drives. So I tested the following procedure for the 6.3 and 7.0 releases and the both releases were successfully installed:

  • boot from a CD-R(W) with 8.0-CURRENT-200805-i386-bootonly.iso (or later)
  • exit the boot prompt screen to the loader prompt (by pressing 6) before the 8.0 kernel boots itself
  • enter unload to clear the loaded (8.0) kernel
  • replace the CD-R(W) in the drive to the target installation image (e.g., Disc 1 of 6.3/7.0-RELEASE)
  • enter boot-conf to load the target kernel and install

I think this is the compatibility problem of BTX. I also think this is so far the easiest way to boot from the USB CD-ROM drive for CF-R4, because generating ISO images for previous releases with the compatible BTX loader will be a big hassle. You may give it a try if you want to boot from a USB flash memory.

Monday, March 3, 2008

Gotchas, gotchas, gotchas

I've learned a lot of hard cold facts on how to deal with multimedia systems. Here are the examples:

  • White fungi can grow on a CD medium surface and even often render the media unreadable, though in most cases the medium can be rescued by wiping out the fungi. (Be careful not to damage the acrylic surface!)
  • Windows XP goes into an infinite loop (100% usage of a CPU core) to try to read an unreadable CD medium. If you don't have a multi-core machine, you'll be getting into a trouble. Why the CD/DVD drive cannot tell you that the media seems to have a problem, rather than just simply retry and wait for the timeout periods?
  • iPod 5G firmware Version 1.2.3 can crash and reboot forever, just because the album title field of an MP3 file contains an unreadable/irregular character. The iTunes 7.6.1 for Windows is running smoothly without any error message even with the MP3 file, so you can't tell what's really wrong until you try-and-fail the whole possible cases.
  • You cannot easily attach still pictures to an audio file without external software. And I realize that's the meaning of the word authoring of a streaming video.
  • You have to make a set of still pictures into motion pictures or a video stream first first, before syncing with an audio stream and make them a video stream, to be broadcast on the Internet, such as on YouTube. (A good news is that you can do this by a set of the open source software, such as mjpegtools and Avidemux.)
  • Gracenote CDDB has been broken for WinAmp, so I have to migrate into iTunes, just to pick up the metadata of each song I'm going to put into my lovely iPod. I like my iPod very much, but iTunes for Windows is really a big slow program and I hope it was a bit smaller and faster.


And I even learned some security and systems administration tips:


  • Many of PCI Serial-ATA cards cannot be used for operating systems other than Windows.
  • A 13-year old hard disk can spin up and is readable even if you once put it in your closet and left it unplugged for 10 years.
  • As of 2008, you can hardly find a SCSI interface card for a PC, at least in Osaka.


Then I remember the truth: your computer could be the number one device to consume your time and kill your productivity. So how many gotchas do you get every day? Ten? Or a hundred? And how many hours or even days do you lose?

Sunday, February 17, 2008

The art and limit of dependencies

If you want to get a job done, you need to list up the necessary things and tools, and procedures to use and apply them. In other words, you have many dependencies on them.

In a modern computer software development, no piece of software can withstand without any dependency to other pieces. You do not want to make a C program without the standard I/O library. Dependencies to the tools are also important and critical; if your codes include those written in FORTRAN, C, C++, and Java, you need the four compilers and language execution environments.

Computer programmers have been making the tools to automatically resolve the dependencies. make is a popular one, derived from the UNIX programming environment, which parses the rulesets called Makefile and determine whether if you need to rebuild a result from the source files, by comparing the timestamps. If make finds out one of the source files are newer than the result, it will invoke the command to rebuild the result.

Version control systems such as subversion, CVS, and RCS, are another good examples of dependency management tools. You can save the history of changes on a file, a directory, or a set of directories. You can make a software package by checking out a set of tagged files; you can even make multiple branches of a code.

FreeBSD operating system has its own dependency management system of externally-contributed programs called ports and packages. A port means a set of rules, configuration files, and source codes necessary to build a program. A package means the derivative of the port, built by another computer. Many essential parts of FreeBSD subsystems, including the X Window and Perl programming language, are installed as packages, because they are not considered as the core parts of FreeBSD.

I'm always pleasantly surprised when a very complicated port, such as Japanese version of LaTeX, a typesetting and documentation set of programs, can be built without major glitches, including the automatic installation of depending programs, such as Ghostscript. For most of the ports, FreeBSD volunteers are always doing the outstanding jobs.

I feel very much annoyed, however, when I have to untangle the web of dependencies when the installation/building of a software toolset from the scratch. Unfortunately, Xorg 7.3, a free implementation set of X Window programs, fonts, and tools, was not able to build in my environment. I had to copy three prebuilt sets of files: include files of C/C++, shared/static libraries, and the very basic fonts, to finish building the necessary programs.

In my case of Xorg kitbuilding, some very old fonts existing since late 1980s were not successfully compiled and converted into various ISO8859 part codesets. And I could not build the font handling libraries. This glitch killed the whole automatic compilation task, which was supposed to untangle the enormous list of dependencies. I also found that the target directory name was merged to /usr/local from the traditional /usr/X11R6, and I wasn't sure whether if I moved the old files and subdirectories under the old directory safely to the new directory.

Untangling the web of dependency is a very hard task, since the parameters you need to examine are scattering around all the directories, in all the ports, and you often also have to set an environment variable to do something extraordinary. While I can guess what I should do because I've done a lot of this kind of tasks, I never want to do this at any time because scanning your memory and poking around the files are very much mentally painful. In a recent hostile computer environment, vulnerabilities can easily sneak into such a complicated and unformalized tasks.

How do you manage your computers? Do you care about the dependencies of the tools, programs, configuration files, and other objects?

Friday, February 8, 2008

CALLing disaster during MySQL upgrade

I've been upgrading one of my servers for the daily use, by migrating the running environment between two PCs. It's not a mission critical server because I don't run a public service there, but I need to do the upgrade carefully anyway because the version number of running FreeBSD and other applications have been changed. I want to keep the old environment as long as possible to use it as a reference, so I'm doing the migration manually by recompiling and reconfiguring the software.

One of the glitches I faced was about MySQL. It's not about the bugs in MySQL, because the database server load is very small. I was trying to transfer a dataset from the old version 4.0 to a new version 5.1 software of MySQL. The mysqldump result of the old 4.0 output didn't get through and reloaded into the 5.1 server. I could not even perform CREATE TABLE. The reason: the column and database identifiers were not properly backquoted.

The database was for my amateur radio activity. Amateur radio stations have callsigns, and in a popular contacting log exchange format called ADIF, the other party's callsign is represented by the identifier CALL. Unfortunately, MySQL 5.1 made the word CALL a reserved word, while in the 4.0 version the word was apparently not. This is a tragedy for an amateur radio enthusiast, and a careless programmer like me who tends to omit proper (back-)quotation.

After an investigation for a few minutes, an idea of referring to the result SHOW CREATE TABLE from the 4.0 server came into my mind. I did it and fortunately the table definition was properly backquoted, so at least I could rebuild the database skeleton. The dumped data was a set of INSERT INTO statements with the VALUES and they were all properly quoted, so I could rebuild the database.

The new mysqldump command's output of MySQL 5.1 looks much better and properly backquoted and quoted all the necessary strings, and even put the SQL statements to lock and unlock the database. The entire dataset is represented by a single INSERT INTO statement VALUES set, so you've got to be careful when you want to use the data set not in the full contents.

I should note that I had to rewrite all the software which generated the SQL statements for the proper (back-)quotation. This was a handful of complicated tasks of fixing the Bourne shell and Perl scripts.

And I realize why SQL injection is so popular for attacking the database servers. Parsing SQL correctly is a non-trivial process. A word can be either a part of a directive or a target identifier, depending on the position where it is in an SQL statement.

So when you want to store CALLs into a MySQL database, you've got to do it carefully with (back-)quotation.

Monday, February 4, 2008

Data erasing woes

Clearing up the used disk has been a real burden for sysadmins, as exploitation of carelessly-discarded information becomes popular. In Japan, people are extremely sensitive against how their personal data records are kept secret and controlled. The reality is, however, that data disclosure incidents keep happening, no matter the government enforces the law, and companies make frequent apologies for those incidents. Once the data get out, you won't be able to take them back.

I think a part of the reason of continuing data disclosure incidents is that erasing data is simply a difficult task. For example, even if you store only 10G bytes into a 160G-byte disk, you need to thoroughly sweep the whole 160G bytes, to guarantee all data are cleared. Also, you've got to be careful to deal with the hidden data, which is not accessible by the data erasing software. If you really want to make the data not recoverable at all, you need to erase the trace of electromagnetic residue on the hard disk platters, which is hard to perform in a usual business or office environment. Destroying a usable device is not an environmentally-friendly practice either.

Another problem is that the sweeping process is slow. The physical writing speed governs the whole performance. For example, one of my old portable 40G-byte hard disk can be written in only 20M bytes per second. So it will take at least 2000 seconds or about 34 minutes to sweep out the whole disk once. You need to do this multiple times to ensure the residue of data is not easily detectable, so the whole process may take 2 or 3 hours. 3 hours for just 40G bytes. 12 hours for 160G bytes. (sigh)

The data storage which an individual has to manage is getting bigger and bigger every year. I wonder how people can cope with this. Do you sweep and erase the used disk data before you resell or give it away to somebody?